* [RFC PATCH bpf-next v7 06/11] mm: memcontrol: Add BPF struct_ops for memory controller
From: Hui Zhu @ 2026-05-26 2:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Hui Zhu
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Hui Zhu <zhuhui@kylinos.cn>
Introduce BPF struct_ops support to the memory controller, enabling
custom and dynamic control over memory pressure via a new struct_ops
type, `memcg_bpf_ops`.
The `memcg_bpf_ops` interface exposes the following hooks:
- `memcg_charged`: Called on the synchronous blocking charge path after
pages have been charged to the cgroup. Returns a custom throttling
delay in milliseconds. This value is used as a lower bound for the
penalty passed to `__mem_cgroup_handle_over_high()` and applies even
when `memory.high` is not breached, allowing BPF programs to impose
proactive back-pressure on any charge event. Return 0 for no delay.
- `memcg_uncharged`: Called when pages are uncharged from a cgroup,
allowing BPF programs to track or react to memory releases.
- `below_low`: Overrides the `memory.low` protection check. Receives
the effective low threshold (elow) and current usage as arguments.
If it returns true, the cgroup is treated as protected regardless of
the standard elow >= usage comparison. Returning false continues
to the normal kernel check.
- `below_min`: Same as `below_low`, but for `memory.min` protection.
Receives emin and usage as arguments.
- `handle_cgroup_online`/`offline`: Callbacks invoked when a cgroup
with an attached program comes online or goes offline, allowing BPF
programs to manage per-cgroup state.
These hooks are integrated into core memory control logic.
`memcg_charged` is consulted in `try_charge_memcg` on the synchronous
blocking path. To avoid losing the originally charged cgroup pointer as
the charge loop walks up the ancestor chain, `orig_memcg` is saved
before the loop begins. After the loop, the BPF hook is called with
`orig_memcg` and the actual batch size, and its result (converted from
milliseconds to jiffies) is stored as `bpf_high_delay`.
`__mem_cgroup_handle_over_high()` is then invoked when either
`bpf_high_delay` is non-zero or `memcg_nr_pages_over_high` exceeds
MEMCG_CHARGE_BATCH. Inside the function, the current task's memcg is
obtained independently via `get_mem_cgroup_from_mm()`. Reclaim is
attempted first; if reclaim makes forward progress or retries remain,
the function loops back to reclaim again rather than throttling
immediately. `bpf_high_delay` serves as a lower bound for the final
penalty via `max(penalty_jiffies, bpf_high_delay)`: when
`memcg_nr_pages_over_high` is zero (memory.high not breached),
the kernel overage calculation is skipped and `bpf_high_delay` alone
sets the penalty. In all cases, throttling only occurs if the resulting
penalty exceeds HZ/100; a BPF-requested delay below this threshold
causes no sleep. The deferred user-return path (via
`mem_cgroup_handle_over_high()`) always passes bpf_high_delay=0 since
BPF delay is evaluated exactly once, on the synchronous charge path.
`below_low` and `below_min` are inserted in their respective inline
functions after the unprotected check. The pre-read elow/emin and usage
values are forwarded to the BPF hook; on false return the standard
kernel comparison (elow >= usage) proceeds as normal.
Support for `BPF_F_ALLOW_OVERRIDE` is included. When a program is
registered with this flag, a descendant cgroup may later attach its own
`memcg_bpf_ops` to override the inherited program. Without this flag,
attaching to a cgroup that already has a program (whether attached
directly or inherited from an ancestor) will fail with -EBUSY.
On registration, ops are propagated to the cgroup itself and all its
descendants via `mem_cgroup_iter`. A `bpf_ops_flags` field is added to
`struct mem_cgroup` to persist the attachment flags, which are inherited
during `css_online` and restored to the parent's flags on
unregistration. On unregistration, rather than unconditionally clearing
`bpf_ops` to NULL throughout the subtree, each descendant that still
holds the unregistered ops pointer has its `bpf_ops` and
`bpf_ops_flags` restored to the values the registering cgroup's parent
held at that time. This correctly handles the override case where a
descendant had re-attached over an inherited program.
Lifecycle management ensures programs are inherited by child cgroups
on `css_online` and cleaned up on `css_offline`. SRCU (`memcg_bpf_srcu`)
protects concurrent read access to the `memcg->bpf_ops` pointer; all
writes are serialized under `cgroup_mutex`.
Signed-off-by: Barry Song <baohua@kernel.org>
Signed-off-by: Geliang Tang <geliang@kernel.org>
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
include/linux/memcontrol.h | 250 ++++++++++++++++++++++++++++++-
mm/bpf_memcontrol.c | 298 ++++++++++++++++++++++++++++++++++++-
mm/memcontrol.c | 43 ++++--
3 files changed, 574 insertions(+), 17 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dc3fa687759b..30b7b8558ccb 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -23,6 +23,7 @@
#include <linux/writeback.h>
#include <linux/page-flags.h>
#include <linux/shrinker.h>
+#include <linux/srcu.h>
struct mem_cgroup;
struct obj_cgroup;
@@ -192,6 +193,59 @@ struct obj_cgroup {
bool is_root;
};
+#ifdef CONFIG_BPF_SYSCALL
+/*
+ * struct memcg_bpf_ops - BPF callbacks for memory cgroup operations
+ *
+ * @handle_cgroup_online: Called when a cgroup comes online. May be used
+ * by a BPF program to initialize per-cgroup state.
+ * @handle_cgroup_offline: Called when a cgroup goes offline. May be used
+ * to release per-cgroup state allocated in the
+ * online callback.
+ * @below_low: Override the memory.low protection check.
+ * Receives the effective low threshold @elow and the current
+ * memory usage @usage (both in pages). If the callback returns
+ * true, mem_cgroup_below_low() returns true immediately,
+ * treating the cgroup as protected regardless of the standard
+ * elow >= usage comparison. Returning false continues to
+ * the normal kernel check.
+ * @below_min: Same as @below_low, but for the memory.min protection check.
+ * Receives @emin and @usage. Returning true short-circuits the
+ * standard emin >= usage comparison.
+ * @memcg_charged: Called on the synchronous blocking charge path after
+ * pages have been charged to the cgroup. Returns a custom
+ * throttle delay in milliseconds. This delay is taken as
+ * a lower bound for the penalty in
+ * __mem_cgroup_handle_over_high() and applies even when
+ * memory.high is not breached. Return 0 for no extra delay.
+ * @memcg_uncharged: Called when pages are uncharged from the cgroup.
+ * Allows BPF programs to track memory releases or update
+ * accounting state. No return value.
+ *
+ * This structure defines the interface for BPF programs to customize
+ * memory cgroup behavior through struct_ops programs. All callbacks are
+ * non-sleepable. Concurrent readers are protected by SRCU
+ * (memcg_bpf_srcu); writers hold cgroup_mutex.
+ */
+struct memcg_bpf_ops {
+ void (*handle_cgroup_online)(struct mem_cgroup *memcg);
+
+ void (*handle_cgroup_offline)(struct mem_cgroup *memcg);
+
+ bool (*below_low)(struct mem_cgroup *memcg, unsigned long elow,
+ unsigned long usage);
+
+ bool (*below_min)(struct mem_cgroup *memcg, unsigned long emin,
+ unsigned long usage);
+
+ unsigned int (*memcg_charged)(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+
+ void (*memcg_uncharged)(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+};
+#endif /* CONFIG_BPF_SYSCALL */
+
/*
* The memory controller data structure. The memory controller controls both
* page cache and RSS per cgroup. We would eventually like to provide
@@ -323,6 +377,11 @@ struct mem_cgroup {
spinlock_t event_list_lock;
#endif /* CONFIG_MEMCG_V1 */
+#ifdef CONFIG_BPF_SYSCALL
+ struct memcg_bpf_ops *bpf_ops;
+ u32 bpf_ops_flags;
+#endif
+
struct mem_cgroup_per_node *nodeinfo[];
};
@@ -533,6 +592,165 @@ static inline bool mem_cgroup_disabled(void)
return !cgroup_subsys_enabled(memory_cgrp_subsys);
}
+#ifdef CONFIG_BPF_SYSCALL
+
+/* SRCU for protecting concurrent access to memcg->bpf_ops */
+extern struct srcu_struct memcg_bpf_srcu;
+
+/*
+ * BPF_MEMCG_CALL - Safely invoke a BPF memcg callback with return value
+ * @memcg: The memory cgroup whose bpf_ops to invoke
+ * @op: The callback name (struct member of memcg_bpf_ops)
+ * @default_val: Value to return if no BPF program is attached or the
+ * specific callback is not implemented
+ * @...: Additional arguments forwarded to the callback
+ *
+ * Uses a two-phase READ_ONCE() pattern:
+ * 1. An initial lockless READ_ONCE() provides a fast-path check.
+ * If bpf_ops is NULL the SRCU lock is never taken, keeping the
+ * common no-BPF path free of synchronization overhead.
+ * 2. A second READ_ONCE() after srcu_read_lock() ensures a consistent
+ * view of the pointer under the SRCU read section, guarding against
+ * a concurrent bpf_memcg_ops_unreg() that may be in progress.
+ */
+#define BPF_MEMCG_CALL(memcg, op, default_val, ...) ({ \
+ typeof(default_val) __ret = (default_val); \
+ struct memcg_bpf_ops *__ops; \
+ int __idx; \
+ \
+ if (unlikely(READ_ONCE((memcg)->bpf_ops))) { \
+ __idx = srcu_read_lock(&memcg_bpf_srcu); \
+ __ops = READ_ONCE((memcg)->bpf_ops); \
+ if (__ops && __ops->op) \
+ __ret = __ops->op(memcg, ##__VA_ARGS__);\
+ srcu_read_unlock(&memcg_bpf_srcu, __idx); \
+ } \
+ __ret; \
+})
+
+/*
+ * BPF_MEMCG_CALL_VOID - Safely invoke a void BPF memcg callback
+ * @memcg: The memory cgroup whose bpf_ops to invoke
+ * @op: The callback name (struct member of memcg_bpf_ops)
+ * @...: Additional arguments forwarded to the callback
+ *
+ * Same SRCU fast-path pattern as BPF_MEMCG_CALL but for callbacks
+ * that have no return value.
+ */
+#define BPF_MEMCG_CALL_VOID(memcg, op, ...) do { \
+ struct memcg_bpf_ops *__ops; \
+ int __idx; \
+ \
+ if (unlikely(READ_ONCE((memcg)->bpf_ops))) { \
+ __idx = srcu_read_lock(&memcg_bpf_srcu); \
+ __ops = READ_ONCE((memcg)->bpf_ops); \
+ if (__ops && __ops->op) \
+ __ops->op(memcg, ##__VA_ARGS__); \
+ srcu_read_unlock(&memcg_bpf_srcu, __idx); \
+ } \
+} while (0)
+
+static inline bool
+bpf_memcg_below_low(struct mem_cgroup *memcg, unsigned long elow,
+ unsigned long usage)
+{
+ return BPF_MEMCG_CALL(memcg, below_low, false, elow, usage);
+}
+
+static inline bool
+bpf_memcg_below_min(struct mem_cgroup *memcg, unsigned long emin,
+ unsigned long usage)
+{
+ return BPF_MEMCG_CALL(memcg, below_min, false, emin, usage);
+}
+
+static inline unsigned long
+bpf_memcg_charged(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ unsigned int ret;
+
+ /*
+ * Retrieve the BPF-specified throttle delay in milliseconds and
+ * convert to jiffies for use in __mem_cgroup_handle_over_high().
+ */
+ ret = BPF_MEMCG_CALL(memcg, memcg_charged, 0U, nr_pages);
+ return msecs_to_jiffies(ret);
+}
+
+static inline void
+bpf_memcg_uncharged(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ BPF_MEMCG_CALL_VOID(memcg, memcg_uncharged, nr_pages);
+}
+
+#undef BPF_MEMCG_CALL
+#undef BPF_MEMCG_CALL_VOID
+
+/*
+ * memcontrol_bpf_online - Inherit BPF ops for a newly online cgroup.
+ * @memcg: The memory cgroup coming online.
+ *
+ * Called under cgroup_mutex from mem_cgroup_css_online(). Inherits the
+ * parent's bpf_ops pointer and bpf_ops_flags into @memcg so that
+ * BPF-based memory control policies propagate down the hierarchy
+ * automatically.
+ *
+ * If the parent has no bpf_ops, this is a no-op. If it does, the ops
+ * pointer is copied and, if an online handler is implemented, it is
+ * invoked to allow the BPF program to initialize per-cgroup state for
+ * the new child.
+ *
+ * Locking: cgroup_mutex is held by the caller. Because bpf_memcg_ops_reg()
+ * and bpf_memcg_ops_unreg() also hold cgroup_mutex when writing
+ * memcg->bpf_ops, no additional lock on memcg_bpf_srcu is required here.
+ */
+extern void memcontrol_bpf_online(struct mem_cgroup *memcg);
+
+/*
+ * memcontrol_bpf_offline - Run BPF cleanup for a cgroup going offline.
+ * @memcg: The memory cgroup going offline.
+ *
+ * Called under cgroup_mutex from mem_cgroup_css_offline(). If a BPF
+ * program is attached and implements a handle_cgroup_offline callback,
+ * it is invoked so the program can release any per-cgroup state before
+ * the memcg is freed.
+ *
+ * Locking: same as memcontrol_bpf_online() — cgroup_mutex is held.
+ */
+extern void memcontrol_bpf_offline(struct mem_cgroup *memcg);
+
+#else /* CONFIG_BPF_SYSCALL */
+
+static inline unsigned long
+bpf_memcg_charged(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ return 0;
+}
+
+static inline void
+bpf_memcg_uncharged(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+}
+
+static inline bool
+bpf_memcg_below_low(struct mem_cgroup *memcg, unsigned long elow,
+ unsigned long usage)
+{
+ return false;
+}
+
+static inline bool
+bpf_memcg_below_min(struct mem_cgroup *memcg, unsigned long emin,
+ unsigned long usage)
+{
+ return false;
+}
+
+static inline void memcontrol_bpf_online(struct mem_cgroup *memcg) { }
+static inline void memcontrol_bpf_offline(struct mem_cgroup *memcg) { }
+
+#endif /* CONFIG_BPF_SYSCALL */
+
static inline void mem_cgroup_protection(struct mem_cgroup *root,
struct mem_cgroup *memcg,
unsigned long *min,
@@ -603,21 +821,35 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target,
static inline bool mem_cgroup_below_low(struct mem_cgroup *target,
struct mem_cgroup *memcg)
{
+ unsigned long elow, usage;
+
if (mem_cgroup_unprotected(target, memcg))
return false;
- return READ_ONCE(memcg->memory.elow) >=
- page_counter_read(&memcg->memory);
+ elow = READ_ONCE(memcg->memory.elow);
+ usage = page_counter_read(&memcg->memory);
+
+ if (bpf_memcg_below_low(memcg, elow, usage))
+ return true;
+
+ return elow >= usage;
}
static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
struct mem_cgroup *memcg)
{
+ unsigned long emin, usage;
+
if (mem_cgroup_unprotected(target, memcg))
return false;
- return READ_ONCE(memcg->memory.emin) >=
- page_counter_read(&memcg->memory);
+ emin = READ_ONCE(memcg->memory.emin);
+ usage = page_counter_read(&memcg->memory);
+
+ if (bpf_memcg_below_min(memcg, emin, usage))
+ return true;
+
+ return emin >= usage;
}
int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp);
@@ -890,12 +1122,18 @@ unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec,
return READ_ONCE(mz->lru_zone_size[zone_idx][lru]);
}
-void __mem_cgroup_handle_over_high(gfp_t gfp_mask);
+void __mem_cgroup_handle_over_high(gfp_t gfp_mask,
+ unsigned long bpf_high_delay);
static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
{
if (unlikely(current->memcg_nr_pages_over_high))
- __mem_cgroup_handle_over_high(gfp_mask);
+ /*
+ * Deferred user-return path: no BPF delay lookup here.
+ * BPF-provided delay is injected from try_charge_memcg()
+ * on the synchronous blocking charge path.
+ */
+ __mem_cgroup_handle_over_high(gfp_mask, 0);
}
unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg);
diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
index 716df49d7647..1f726a7b22e3 100644
--- a/mm/bpf_memcontrol.c
+++ b/mm/bpf_memcontrol.c
@@ -8,6 +8,9 @@
#include <linux/memcontrol.h>
#include <linux/bpf.h>
+/* Protects memcg->bpf_ops pointer for read and write. */
+DEFINE_SRCU(memcg_bpf_srcu);
+
__bpf_kfunc_start_defs();
/**
@@ -179,15 +182,306 @@ static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = {
.set = &bpf_memcontrol_kfuncs,
};
+/**
+ * memcontrol_bpf_online - Inherit BPF programs for a new online cgroup.
+ * @memcg: The memory cgroup that is coming online.
+ *
+ * When a new memcg is brought online, it inherits the BPF programs
+ * attached to its parent. This ensures consistent BPF-based memory
+ * control policies throughout the cgroup hierarchy.
+ *
+ * After inheriting, if the BPF program has an online handler, it is
+ * invoked for the new memcg.
+ */
+void memcontrol_bpf_online(struct mem_cgroup *memcg)
+{
+ struct memcg_bpf_ops *ops;
+ struct mem_cgroup *parent_memcg;
+
+ /* The root cgroup does not inherit from a parent. */
+ if (mem_cgroup_is_root(memcg))
+ return;
+
+ /*
+ * Because only functions bpf_memcg_ops_reg and bpf_memcg_ops_unreg
+ * write to memcg->bpf_ops and memcg->bpf_ops_flags under the
+ * protection of cgroup_mutex, ensuring that cgroup_mutex is already
+ * locked here allows safe reading and writing of memcg->bpf_ops and
+ * memcg->bpf_ops_flags without needing to acquire a lock on
+ * memcg_bpf_srcu.
+ */
+ lockdep_assert_held(&cgroup_mutex);
+
+ parent_memcg = parent_mem_cgroup(memcg);
+
+ /* Inherit the BPF program from the parent cgroup. */
+ ops = READ_ONCE(parent_memcg->bpf_ops);
+ if (!ops)
+ return;
+ WRITE_ONCE(memcg->bpf_ops, ops);
+ memcg->bpf_ops_flags = parent_memcg->bpf_ops_flags;
+
+ /*
+ * If the BPF program implements it, call the online handler to
+ * allow the program to perform setup tasks for the new cgroup.
+ */
+ if (ops->handle_cgroup_online)
+ ops->handle_cgroup_online(memcg);
+}
+
+/**
+ * memcontrol_bpf_offline - Run BPF cleanup for an offline cgroup.
+ * @memcg: The memory cgroup that is going offline.
+ *
+ * If a BPF program is attached and implements an offline handler,
+ * it is invoked to perform cleanup tasks before the memcg goes
+ * completely offline.
+ */
+void memcontrol_bpf_offline(struct mem_cgroup *memcg)
+{
+ struct memcg_bpf_ops *ops;
+
+ /* Same locking rules as memcontrol_bpf_online(). */
+ lockdep_assert_held(&cgroup_mutex);
+
+ ops = READ_ONCE(memcg->bpf_ops);
+ if (!ops || !ops->handle_cgroup_offline)
+ return;
+
+ ops->handle_cgroup_offline(memcg);
+}
+
+static int memcg_ops_btf_struct_access(struct bpf_verifier_log *log,
+ const struct bpf_reg_state *reg,
+ int off, int size)
+{
+ return -EACCES;
+}
+
+static bool memcg_ops_is_valid_access(int off, int size, enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}
+
+const struct bpf_verifier_ops bpf_memcg_verifier_ops = {
+ .get_func_proto = bpf_base_func_proto,
+ .btf_struct_access = memcg_ops_btf_struct_access,
+ .is_valid_access = memcg_ops_is_valid_access,
+};
+
+static void cfi_handle_cgroup_online(struct mem_cgroup *memcg)
+{
+}
+
+static void cfi_handle_cgroup_offline(struct mem_cgroup *memcg)
+{
+}
+
+static bool
+cfi_below_low(struct mem_cgroup *memcg, unsigned long elow,
+ unsigned long usage)
+{
+ return false;
+}
+
+static bool
+cfi_below_min(struct mem_cgroup *memcg, unsigned long emin,
+ unsigned long usage)
+{
+ return false;
+}
+
+static unsigned int cfi_memcg_charged(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ return 0;
+}
+
+static void cfi_memcg_uncharged(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+}
+
+static struct memcg_bpf_ops cfi_bpf_memcg_ops = {
+ .handle_cgroup_online = cfi_handle_cgroup_online,
+ .handle_cgroup_offline = cfi_handle_cgroup_offline,
+ .below_low = cfi_below_low,
+ .below_min = cfi_below_min,
+ .memcg_charged = cfi_memcg_charged,
+ .memcg_uncharged = cfi_memcg_uncharged,
+};
+
+static int bpf_memcg_ops_init(struct btf *btf)
+{
+ return 0;
+}
+
+static int bpf_memcg_ops_check_member(const struct btf_type *t,
+ const struct btf_member *member,
+ const struct bpf_prog *prog)
+{
+ u32 moff = __btf_member_bit_offset(t, member) / 8;
+
+ switch (moff) {
+ case offsetof(struct memcg_bpf_ops, handle_cgroup_online):
+ case offsetof(struct memcg_bpf_ops, handle_cgroup_offline):
+ case offsetof(struct memcg_bpf_ops, below_low):
+ case offsetof(struct memcg_bpf_ops, below_min):
+ case offsetof(struct memcg_bpf_ops, memcg_charged):
+ case offsetof(struct memcg_bpf_ops, memcg_uncharged):
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (prog->sleepable)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int bpf_memcg_ops_init_member(const struct btf_type *t,
+ const struct btf_member *member,
+ void *kdata, const void *udata)
+{
+ return 0;
+}
+
+static int bpf_memcg_ops_reg(void *kdata, struct bpf_link *link)
+{
+ struct bpf_struct_ops_link *ops_link;
+ struct memcg_bpf_ops *ops = kdata, *old_ops;
+ struct cgroup_subsys_state *css;
+ struct mem_cgroup *memcg, *iter;
+ int err = 0;
+
+ if (!link)
+ return -EOPNOTSUPP;
+ ops_link = container_of(link, struct bpf_struct_ops_link, link);
+ if (!ops_link->cgroup)
+ return -EINVAL;
+
+ cgroup_lock();
+
+ css = cgroup_e_css(ops_link->cgroup, &memory_cgrp_subsys);
+ if (!css) {
+ err = -EINVAL;
+ goto unlock_out;
+ }
+ memcg = mem_cgroup_from_css(css);
+
+ /*
+ * Check if memcg has bpf_ops and whether it is inherited from
+ * parent.
+ * If inherited and BPF_F_ALLOW_OVERRIDE is set, allow override.
+ */
+ old_ops = READ_ONCE(memcg->bpf_ops);
+ if (old_ops) {
+ struct mem_cgroup *parent_memcg = parent_mem_cgroup(memcg);
+
+ if (!parent_memcg ||
+ !(memcg->bpf_ops_flags & BPF_F_ALLOW_OVERRIDE) ||
+ READ_ONCE(parent_memcg->bpf_ops) != old_ops) {
+ err = -EBUSY;
+ goto unlock_out;
+ }
+ }
+
+ /* Check for incompatible bpf_ops in descendants. */
+ iter = NULL;
+ while ((iter = mem_cgroup_iter(memcg, iter, NULL))) {
+ struct memcg_bpf_ops *iter_ops = READ_ONCE(iter->bpf_ops);
+
+ if (iter_ops && iter_ops != old_ops) {
+ /* cannot override existing bpf_ops of sub-cgroup. */
+ mem_cgroup_iter_break(memcg, iter);
+ err = -EBUSY;
+ goto unlock_out;
+ }
+ }
+
+ iter = NULL;
+ while ((iter = mem_cgroup_iter(memcg, iter, NULL))) {
+ WRITE_ONCE(iter->bpf_ops, ops);
+ iter->bpf_ops_flags = ops_link->flags;
+ }
+
+unlock_out:
+ cgroup_unlock();
+ return err;
+}
+
+/* Unregister the struct ops instance */
+static void bpf_memcg_ops_unreg(void *kdata, struct bpf_link *link)
+{
+ struct bpf_struct_ops_link *ops_link;
+ struct memcg_bpf_ops *ops = kdata;
+ struct cgroup_subsys_state *css;
+ struct mem_cgroup *memcg;
+ struct mem_cgroup *iter;
+ struct memcg_bpf_ops *parent_bpf_ops = NULL;
+ u32 parent_bpf_ops_flags = 0;
+
+ if (!link)
+ return;
+ ops_link = container_of(link, struct bpf_struct_ops_link, link);
+ if (!ops_link->cgroup)
+ return;
+
+ cgroup_lock();
+
+ css = cgroup_e_css(ops_link->cgroup, &memory_cgrp_subsys);
+ if (!css)
+ goto unlock_out;
+ memcg = mem_cgroup_from_css(css);
+
+ /* Get the parent bpf_ops and bpf_ops_flags */
+ iter = parent_mem_cgroup(memcg);
+ if (iter) {
+ parent_bpf_ops = READ_ONCE(iter->bpf_ops);
+ parent_bpf_ops_flags = iter->bpf_ops_flags;
+ }
+
+ iter = NULL;
+ while ((iter = mem_cgroup_iter(memcg, iter, NULL))) {
+ if (READ_ONCE(iter->bpf_ops) == ops) {
+ WRITE_ONCE(iter->bpf_ops, parent_bpf_ops);
+ iter->bpf_ops_flags = parent_bpf_ops_flags;
+ }
+ }
+
+unlock_out:
+ cgroup_unlock();
+ synchronize_srcu(&memcg_bpf_srcu);
+}
+
+static struct bpf_struct_ops bpf_memcg_bpf_ops = {
+ .verifier_ops = &bpf_memcg_verifier_ops,
+ .init = bpf_memcg_ops_init,
+ .check_member = bpf_memcg_ops_check_member,
+ .init_member = bpf_memcg_ops_init_member,
+ .reg = bpf_memcg_ops_reg,
+ .unreg = bpf_memcg_ops_unreg,
+ .name = "memcg_bpf_ops",
+ .owner = THIS_MODULE,
+ .cfi_stubs = &cfi_bpf_memcg_ops,
+};
+
static int __init bpf_memcontrol_init(void)
{
- int err;
+ int err, err2;
err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
&bpf_memcontrol_kfunc_set);
if (err)
pr_warn("error while registering bpf memcontrol kfuncs: %d", err);
- return err;
+ err2 = register_bpf_struct_ops(&bpf_memcg_bpf_ops, memcg_bpf_ops);
+ if (err2)
+ pr_warn("error while registering memcontrol bpf ops: %d\n",
+ err2);
+
+ return err ? err : err2;
}
late_initcall(bpf_memcontrol_init);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d466..ec912d19ef87 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2085,6 +2085,8 @@ static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
page_counter_uncharge(&memcg->memory, nr_pages);
if (do_memsw_account())
page_counter_uncharge(&memcg->memsw, nr_pages);
+
+ bpf_memcg_uncharged(memcg, nr_pages);
}
/*
@@ -2473,8 +2475,12 @@ static unsigned long calculate_high_delay(struct mem_cgroup *memcg,
* Reclaims memory over the high limit. Called directly from
* try_charge() (context permitting), as well as from the userland
* return path where reclaim is always able to block.
+ *
+ * @bpf_high_delay is caller-provided extra delay. Callers that do
+ * not evaluate BPF delay (e.g. deferred return-path handling) pass 0.
*/
-void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
+void
+__mem_cgroup_handle_over_high(gfp_t gfp_mask, unsigned long bpf_high_delay)
{
unsigned long penalty_jiffies;
unsigned long pflags;
@@ -2516,11 +2522,15 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
* memory.high is breached and reclaim is unable to keep up. Throttle
* allocators proactively to slow down excessive growth.
*/
- penalty_jiffies = calculate_high_delay(memcg, nr_pages,
- mem_find_max_overage(memcg));
+ if (nr_pages) {
+ penalty_jiffies = calculate_high_delay(
+ memcg, nr_pages, mem_find_max_overage(memcg));
- penalty_jiffies += calculate_high_delay(memcg, nr_pages,
- swap_find_max_overage(memcg));
+ penalty_jiffies += calculate_high_delay(
+ memcg, nr_pages, swap_find_max_overage(memcg));
+ } else
+ penalty_jiffies = 0;
+ penalty_jiffies = max(penalty_jiffies, bpf_high_delay);
/*
* Clamp the max delay per usermode return so as to still keep the
@@ -2578,6 +2588,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
bool raised_max_event = false;
unsigned long pflags;
bool allow_spinning = gfpflags_allow_spinning(gfp_mask);
+ struct mem_cgroup *orig_memcg;
+ unsigned long bpf_high_delay;
retry:
if (consume_stock(memcg, nr_pages))
@@ -2704,6 +2716,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
if (batch > nr_pages)
refill_stock(memcg, batch - nr_pages);
+ orig_memcg = memcg;
/*
* If the hierarchy is above the normal consumption range, schedule
* reclaim on returning to userland. We can perform reclaim here
@@ -2746,6 +2759,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
}
} while ((memcg = parent_mem_cgroup(memcg)));
+ bpf_high_delay = bpf_memcg_charged(orig_memcg, batch);
+
/*
* Reclaim is set up above to be called from the userland
* return path. But also attempt synchronous reclaim to avoid
@@ -2753,10 +2768,17 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
* kernel. If this is successful, the return path will see it
* when it rechecks the overage and simply bail out.
*/
- if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
- !(current->flags & PF_MEMALLOC) &&
- gfpflags_allow_blocking(gfp_mask))
- __mem_cgroup_handle_over_high(gfp_mask);
+ if (!(current->flags & PF_MEMALLOC) &&
+ gfpflags_allow_blocking(gfp_mask)) {
+ /*
+ * BPF high-delay is evaluated only on the synchronous
+ * blocking path. The deferred user-return path calls
+ * __mem_cgroup_handle_over_high() with bpf_high_delay == 0.
+ */
+ if (bpf_high_delay ||
+ current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH)
+ __mem_cgroup_handle_over_high(gfp_mask, bpf_high_delay);
+ }
return 0;
}
@@ -4151,6 +4173,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
*/
xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL);
+ memcontrol_bpf_online(memcg);
+
return 0;
free_objcg:
for_each_node(nid) {
@@ -4188,6 +4212,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
zswap_memcg_offline_cleanup(memcg);
+ memcontrol_bpf_offline(memcg);
memcg_offline_kmem(memcg);
reparent_deferred_split_queue(memcg);
/*
--
2.43.0
^ permalink raw reply related
* [RFC PATCH bpf-next v7 05/11] bpf: Pass flags in bpf_link_create for struct_ops
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Hui Zhu
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Hui Zhu <zhuhui@kylinos.cn>
To support features like allowing overrides in cgroup hierarchies,
we need a way to pass flags from userspace to the kernel when
attaching a struct_ops.
Extend `bpf_struct_ops_link` to include a `flags` field. This field
is populated from `attr->link_create.flags` during link creation. This
will allow struct_ops implementations, such as the upcoming memory
controller ops, to interpret these flags and modify their attachment
behavior accordingly.
The flags validation in bpf_struct_ops_link_create() is updated
to explicitly permit BPF_F_ALLOW_OVERRIDE in addition to the
already-allowed BPF_F_CGROUP_FD. Any other flag combination
will still be rejected with -EINVAL.
UAPI Change:
This patch updates the comment in include/uapi/linux/bpf.h to reflect
that the cgroup-bpf attach flags (such as BPF_F_ALLOW_OVERRIDE) are
now applicable to both BPF_PROG_ATTACH and BPF_LINK_CREATE commands.
Previously, these flags were only documented for BPF_PROG_ATTACH.
The actual flag definitions remain unchanged, so this is a compatible
extension of the existing API. Older userspace will continue to work
(by not passing flags), and newer userspace can opt-in to the new
functionality by setting appropriate flags.
Signed-off-by: Barry Song <baohua@kernel.org>
Signed-off-by: Geliang Tang <geliang@kernel.org>
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
include/linux/bpf.h | 1 +
include/uapi/linux/bpf.h | 2 +-
kernel/bpf/bpf_struct_ops.c | 4 +++-
tools/include/uapi/linux/bpf.h | 2 +-
4 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 743b4f0546b5..aae7f9837944 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1915,6 +1915,7 @@ struct bpf_struct_ops_link {
bool cgroup_removed;
struct list_head list;
wait_queue_head_t wait_hup;
+ u32 flags;
};
struct bpf_link_primer {
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index f547613986cc..85ab5bdf81ac 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1194,7 +1194,7 @@ enum bpf_perf_event_type {
BPF_PERF_EVENT_EVENT = 6,
};
-/* cgroup-bpf attach flags used in BPF_PROG_ATTACH command
+/* cgroup-bpf attach flags used in BPF_PROG_ATTACH and BPF_LINK_CREATE command
*
* NONE(default): No further bpf programs allowed in the subtree.
*
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 5333290957cb..1d15c667a300 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -1389,7 +1389,8 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
struct cgroup *cgrp;
int err;
- if (attr->link_create.flags & ~BPF_F_CGROUP_FD)
+ if (attr->link_create.flags & ~(BPF_F_CGROUP_FD |
+ BPF_F_ALLOW_OVERRIDE))
return -EINVAL;
map = bpf_map_get(attr->link_create.map_fd);
@@ -1427,6 +1428,7 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
goto err_out;
}
}
+ link->flags = attr->link_create.flags;
err = bpf_link_prime(&link->link, &link_primer);
if (err)
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index fa075dc3b7eb..8a2b1f865d2b 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1194,7 +1194,7 @@ enum bpf_perf_event_type {
BPF_PERF_EVENT_EVENT = 6,
};
-/* cgroup-bpf attach flags used in BPF_PROG_ATTACH command
+/* cgroup-bpf attach flags used in BPF_PROG_ATTACH and BPF_LINK_CREATE command
*
* NONE(default): No further bpf programs allowed in the subtree.
*
--
2.43.0
^ permalink raw reply related
* [RFC PATCH bpf-next v7 04/11] libbpf: introduce bpf_map__attach_struct_ops_opts()
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Roman Gushchin <roman.gushchin@linux.dev>
Introduce bpf_map__attach_struct_ops_opts(), an extended version of
bpf_map__attach_struct_ops(), which takes additional struct
bpf_struct_ops_opts argument.
This allows to pass a target_fd argument and the BPF_F_CGROUP_FD flag
and attach the struct ops to a cgroup as a result.
Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
---
tools/lib/bpf/libbpf.c | 20 +++++++++++++++++---
tools/lib/bpf/libbpf.h | 14 ++++++++++++++
tools/lib/bpf/libbpf.map | 1 +
3 files changed, 32 insertions(+), 3 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 1e8688975d16..a1b54da1ded2 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -13683,11 +13683,18 @@ static int bpf_link__detach_struct_ops(struct bpf_link *link)
return close(link->fd);
}
-struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
+struct bpf_link *bpf_map__attach_struct_ops_opts(const struct bpf_map *map,
+ const struct bpf_struct_ops_opts *opts)
{
+ DECLARE_LIBBPF_OPTS(bpf_link_create_opts, link_opts);
struct bpf_link_struct_ops *link;
+ int err, fd, target_fd;
__u32 zero = 0;
- int err, fd;
+
+ if (!OPTS_VALID(opts, bpf_struct_ops_opts)) {
+ pr_warn("map '%s': invalid opts\n", map->name);
+ return libbpf_err_ptr(-EINVAL);
+ }
if (!bpf_map__is_struct_ops(map)) {
pr_warn("map '%s': can't attach non-struct_ops map\n", map->name);
@@ -13724,7 +13731,9 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
return &link->link;
}
- fd = bpf_link_create(map->fd, 0, BPF_STRUCT_OPS, NULL);
+ link_opts.flags = OPTS_GET(opts, flags, 0);
+ target_fd = OPTS_GET(opts, target_fd, 0);
+ fd = bpf_link_create(map->fd, target_fd, BPF_STRUCT_OPS, &link_opts);
if (fd < 0) {
free(link);
return libbpf_err_ptr(fd);
@@ -13736,6 +13745,11 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
return &link->link;
}
+struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
+{
+ return bpf_map__attach_struct_ops_opts(map, NULL);
+}
+
/*
* Swap the back struct_ops of a link with a new struct_ops map.
*/
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index bba4e8464396..18af178547ad 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -945,6 +945,20 @@ bpf_program__attach_cgroup_opts(const struct bpf_program *prog, int cgroup_fd,
struct bpf_map;
LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map);
+
+struct bpf_struct_ops_opts {
+ /* size of this struct, for forward/backward compatibility */
+ size_t sz;
+ __u32 flags;
+ __u32 target_fd;
+ __u64 expected_revision;
+ size_t :0;
+};
+#define bpf_struct_ops_opts__last_field expected_revision
+
+LIBBPF_API struct bpf_link *
+bpf_map__attach_struct_ops_opts(const struct bpf_map *map,
+ const struct bpf_struct_ops_opts *opts);
LIBBPF_API int bpf_link__update_map(struct bpf_link *link, const struct bpf_map *map);
struct bpf_iter_attach_opts {
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index dfed8d60af05..6105619b5ecf 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -454,6 +454,7 @@ LIBBPF_1.7.0 {
bpf_prog_assoc_struct_ops;
bpf_program__assoc_struct_ops;
btf__permute;
+ bpf_map__attach_struct_ops_opts;
} LIBBPF_1.6.0;
LIBBPF_1.8.0 {
--
2.43.0
^ permalink raw reply related
* [RFC PATCH bpf-next v7 03/11] libbpf: fix return value on memory allocation failure
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Yafang Shao
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Roman Gushchin <roman.gushchin@linux.dev>
bpf_map__attach_struct_ops() returns -EINVAL instead of -ENOMEM
on the memory allocation failure. Fix it.
Fixes: 590a00888250 ("bpf: libbpf: Add STRUCT_OPS support")
Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
---
tools/lib/bpf/libbpf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index ab2071fdd3e8..1e8688975d16 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -13701,7 +13701,7 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
link = calloc(1, sizeof(*link));
if (!link)
- return libbpf_err_ptr(-EINVAL);
+ return libbpf_err_ptr(-ENOMEM);
/* kern_vdata should be prepared during the loading phase. */
err = bpf_map_update_elem(map->fd, &zero, map->st_ops->kern_vdata, 0);
--
2.43.0
^ permalink raw reply related
* [RFC PATCH bpf-next v7 02/11] bpf: allow attaching struct_ops to cgroups
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Roman Gushchin <roman.gushchin@linux.dev>
Introduce an ability to attach bpf struct_ops'es to cgroups.
>From user's standpoint it works in the following way:
a user passes a BPF_F_CGROUP_FD flag and specifies the target cgroup
fd while creating a struct_ops link. As the result, the bpf struct_ops
link will be created and attached to a cgroup.
The cgroup.bpf structure maintains a list of attached struct ops links.
If the cgroup is getting deleted, attached struct ops'es are getting
auto-detached and the userspace program gets a notification.
This change doesn't answer the question how bpf programs belonging
to these struct ops'es will be executed. It will be done individually
for every bpf struct ops which supports this.
Please, note that unlike "normal" bpf programs, struct ops'es
are not propagated to cgroup sub-trees.
Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
---
include/linux/bpf-cgroup-defs.h | 3 ++
include/linux/bpf-cgroup.h | 16 +++++++++
include/linux/bpf.h | 3 ++
include/uapi/linux/bpf.h | 3 ++
kernel/bpf/bpf_struct_ops.c | 59 ++++++++++++++++++++++++++++++---
kernel/bpf/cgroup.c | 46 +++++++++++++++++++++++++
tools/include/uapi/linux/bpf.h | 1 +
7 files changed, 127 insertions(+), 4 deletions(-)
diff --git a/include/linux/bpf-cgroup-defs.h b/include/linux/bpf-cgroup-defs.h
index c9e6b26abab6..6c5e37190dad 100644
--- a/include/linux/bpf-cgroup-defs.h
+++ b/include/linux/bpf-cgroup-defs.h
@@ -71,6 +71,9 @@ struct cgroup_bpf {
/* temp storage for effective prog array used by prog_attach/detach */
struct bpf_prog_array *inactive;
+ /* list of bpf struct ops links */
+ struct list_head struct_ops_links;
+
/* reference counter used to detach bpf programs after cgroup removal */
struct percpu_ref refcnt;
diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h
index b2e79c2b41d5..88b643568012 100644
--- a/include/linux/bpf-cgroup.h
+++ b/include/linux/bpf-cgroup.h
@@ -423,6 +423,11 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+ struct bpf_struct_ops_link *link);
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+ struct bpf_struct_ops_link *link);
+
const struct bpf_func_proto *
cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
#else
@@ -451,6 +456,17 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
return -EINVAL;
}
+static inline int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+ struct bpf_struct_ops_link *link)
+{
+ return -EINVAL;
+}
+
+static inline void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+ struct bpf_struct_ops_link *link)
+{
+}
+
static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 01c0bf5a9cd0..743b4f0546b5 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1911,6 +1911,9 @@ struct bpf_raw_tp_link {
struct bpf_struct_ops_link {
struct bpf_link link;
struct bpf_map __rcu *map;
+ struct cgroup *cgroup;
+ bool cgroup_removed;
+ struct list_head list;
wait_queue_head_t wait_hup;
};
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index aec171ccb6ef..f547613986cc 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1246,6 +1246,7 @@ enum bpf_perf_event_type {
#define BPF_F_AFTER (1U << 4)
#define BPF_F_ID (1U << 5)
#define BPF_F_PREORDER (1U << 6)
+#define BPF_F_CGROUP_FD (1U << 7)
#define BPF_F_LINK BPF_F_LINK /* 1 << 13 */
/* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
@@ -6793,6 +6794,8 @@ struct bpf_link_info {
} xdp;
struct {
__u32 map_id;
+ __u32 :32;
+ __u64 cgroup_id;
} struct_ops;
struct {
__u32 pf;
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index cf3c604d48ef..5333290957cb 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -13,6 +13,8 @@
#include <linux/btf_ids.h>
#include <linux/rcupdate_wait.h>
#include <linux/poll.h>
+#include <linux/bpf-cgroup.h>
+#include <linux/cgroup.h>
struct bpf_struct_ops_value {
struct bpf_struct_ops_common_value common;
@@ -1220,6 +1222,10 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
bpf_map_put(&st_map->map);
}
+
+ if (st_link->cgroup)
+ cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+
kfree(st_link);
}
@@ -1228,6 +1234,7 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
{
struct bpf_struct_ops_link *st_link;
struct bpf_map *map;
+ u64 cgrp_id = 0;
st_link = container_of(link, struct bpf_struct_ops_link, link);
rcu_read_lock();
@@ -1235,6 +1242,14 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
if (map)
seq_printf(seq, "map_id:\t%d\n", map->id);
rcu_read_unlock();
+
+ cgroup_lock();
+ if (st_link->cgroup)
+ cgrp_id = cgroup_id(st_link->cgroup);
+ cgroup_unlock();
+
+ if (cgrp_id)
+ seq_printf(seq, "cgroup_id:\t%llu\n", cgrp_id);
}
static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
@@ -1242,6 +1257,7 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
{
struct bpf_struct_ops_link *st_link;
struct bpf_map *map;
+ u64 cgrp_id = 0;
st_link = container_of(link, struct bpf_struct_ops_link, link);
rcu_read_lock();
@@ -1249,6 +1265,13 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
if (map)
info->struct_ops.map_id = map->id;
rcu_read_unlock();
+
+ cgroup_lock();
+ if (st_link->cgroup)
+ cgrp_id = cgroup_id(st_link->cgroup);
+ cgroup_unlock();
+
+ info->struct_ops.cgroup_id = cgrp_id;
return 0;
}
@@ -1327,6 +1350,9 @@ static int bpf_struct_ops_map_link_detach(struct bpf_link *link)
mutex_unlock(&update_mutex);
+ if (st_link->cgroup)
+ cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+
wake_up_interruptible_poll(&st_link->wait_hup, EPOLLHUP);
return 0;
@@ -1339,6 +1365,9 @@ static __poll_t bpf_struct_ops_map_link_poll(struct file *file,
poll_wait(file, &st_link->wait_hup, pts);
+ if (st_link->cgroup_removed)
+ return EPOLLHUP;
+
return rcu_access_pointer(st_link->map) ? 0 : EPOLLHUP;
}
@@ -1357,8 +1386,12 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
struct bpf_link_primer link_primer;
struct bpf_struct_ops_map *st_map;
struct bpf_map *map;
+ struct cgroup *cgrp;
int err;
+ if (attr->link_create.flags & ~BPF_F_CGROUP_FD)
+ return -EINVAL;
+
map = bpf_map_get(attr->link_create.map_fd);
if (IS_ERR(map))
return PTR_ERR(map);
@@ -1378,11 +1411,26 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_map_lops, NULL,
attr->link_create.attach_type);
+ init_waitqueue_head(&link->wait_hup);
+
+ if (attr->link_create.flags & BPF_F_CGROUP_FD) {
+ cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
+ if (IS_ERR(cgrp)) {
+ err = PTR_ERR(cgrp);
+ goto err_out;
+ }
+ link->cgroup = cgrp;
+ err = cgroup_bpf_attach_struct_ops(cgrp, link);
+ if (err) {
+ cgroup_put(cgrp);
+ link->cgroup = NULL;
+ goto err_out;
+ }
+ }
+
err = bpf_link_prime(&link->link, &link_primer);
if (err)
- goto err_out;
-
- init_waitqueue_head(&link->wait_hup);
+ goto err_put_cgroup;
/* Hold the update_mutex such that the subsystem cannot
* do link->ops->detach() before the link is fully initialized.
@@ -1393,13 +1441,16 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
mutex_unlock(&update_mutex);
bpf_link_cleanup(&link_primer);
link = NULL;
- goto err_out;
+ goto err_put_cgroup;
}
RCU_INIT_POINTER(link->map, map);
mutex_unlock(&update_mutex);
return bpf_link_settle(&link_primer);
+err_put_cgroup:
+ if (link && link->cgroup)
+ cgroup_bpf_detach_struct_ops(link->cgroup, link);
err_out:
bpf_map_put(map);
kfree(link);
diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c
index 876f6a81a9b6..b593ebb30a4e 100644
--- a/kernel/bpf/cgroup.c
+++ b/kernel/bpf/cgroup.c
@@ -16,6 +16,7 @@
#include <linux/bpf-cgroup.h>
#include <linux/bpf_lsm.h>
#include <linux/bpf_verifier.h>
+#include <linux/poll.h>
#include <net/sock.h>
#include <net/bpf_sk_storage.h>
@@ -307,12 +308,23 @@ static void cgroup_bpf_release(struct work_struct *work)
bpf.release_work);
struct bpf_prog_array *old_array;
struct list_head *storages = &cgrp->bpf.storages;
+ struct bpf_struct_ops_link *st_link, *st_tmp;
struct bpf_cgroup_storage *storage, *stmp;
+ LIST_HEAD(st_links);
unsigned int atype;
cgroup_lock();
+ list_splice_init(&cgrp->bpf.struct_ops_links, &st_links);
+ list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+ st_link->cgroup = NULL;
+ st_link->cgroup_removed = true;
+ cgroup_put(cgrp);
+ if (IS_ERR(bpf_link_inc_not_zero(&st_link->link)))
+ list_del(&st_link->list);
+ }
+
for (atype = 0; atype < ARRAY_SIZE(cgrp->bpf.progs); atype++) {
struct hlist_head *progs = &cgrp->bpf.progs[atype];
struct bpf_prog_list *pl;
@@ -346,6 +358,11 @@ static void cgroup_bpf_release(struct work_struct *work)
cgroup_unlock();
+ list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+ st_link->link.ops->detach(&st_link->link);
+ bpf_link_put(&st_link->link);
+ }
+
for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
cgroup_bpf_put(p);
@@ -525,6 +542,7 @@ static int cgroup_bpf_inherit(struct cgroup *cgrp)
INIT_HLIST_HEAD(&cgrp->bpf.progs[i]);
INIT_LIST_HEAD(&cgrp->bpf.storages);
+ INIT_LIST_HEAD(&cgrp->bpf.struct_ops_links);
for (i = 0; i < NR; i++)
if (compute_effective_progs(cgrp, i, &arrays[i]))
@@ -2755,3 +2773,31 @@ cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return NULL;
}
}
+
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+ struct bpf_struct_ops_link *link)
+{
+ int ret = 0;
+
+ cgroup_lock();
+ if (percpu_ref_is_zero(&cgrp->bpf.refcnt)) {
+ ret = -EBUSY;
+ goto out;
+ }
+ list_add_tail(&link->list, &cgrp->bpf.struct_ops_links);
+out:
+ cgroup_unlock();
+ return ret;
+}
+
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+ struct bpf_struct_ops_link *link)
+{
+ cgroup_lock();
+ if (link->cgroup == cgrp) {
+ list_del(&link->list);
+ link->cgroup = NULL;
+ cgroup_put(cgrp);
+ }
+ cgroup_unlock();
+}
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 37142e6d911a..fa075dc3b7eb 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1246,6 +1246,7 @@ enum bpf_perf_event_type {
#define BPF_F_AFTER (1U << 4)
#define BPF_F_ID (1U << 5)
#define BPF_F_PREORDER (1U << 6)
+#define BPF_F_CGROUP_FD (1U << 7)
#define BPF_F_LINK BPF_F_LINK /* 1 << 13 */
/* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
--
2.43.0
^ permalink raw reply related
* [RFC PATCH bpf-next v7 01/11] bpf: move bpf_struct_ops_link into bpf.h
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Matt Bobrowski, Yafang Shao
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Roman Gushchin <roman.gushchin@linux.dev>
Move struct bpf_struct_ops_link's definition into bpf.h,
where other custom bpf links definitions are.
It's necessary to access its members from outside of generic
bpf_struct_ops implementation, which will be done by following
patches in the series.
Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
Acked-by: Matt Bobrowski <mattbobrowski@google.com>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
---
include/linux/bpf.h | 6 ++++++
kernel/bpf/bpf_struct_ops.c | 6 ------
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 1b28cacc3075..01c0bf5a9cd0 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1908,6 +1908,12 @@ struct bpf_raw_tp_link {
u64 cookie;
};
+struct bpf_struct_ops_link {
+ struct bpf_link link;
+ struct bpf_map __rcu *map;
+ wait_queue_head_t wait_hup;
+};
+
struct bpf_link_primer {
struct bpf_link *link;
struct file *file;
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 521cb9d7e8c7..cf3c604d48ef 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -55,12 +55,6 @@ struct bpf_struct_ops_map {
struct bpf_struct_ops_value kvalue;
};
-struct bpf_struct_ops_link {
- struct bpf_link link;
- struct bpf_map __rcu *map;
- wait_queue_head_t wait_hup;
-};
-
static DEFINE_MUTEX(update_mutex);
#define VALUE_PREFIX "bpf_struct_ops_"
--
2.43.0
^ permalink raw reply related
* [RFC PATCH bpf-next v7 00/11] mm: BPF struct_ops for dynamic memory protection and async reclaim
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
Overview:
This series introduces BPF struct_ops support for the memory controller,
enabling userspace BPF programs to implement custom, dynamic memory
management policies per cgroup. The feature allows BPF programs to hook
into the core reclaim and charge paths without requiring kernel
modifications, providing a flexible alternative to static knobs such as
memory.low and memory.min.
The series enables two complementary use cases.
Dynamic memory protection: static memory protection thresholds
(memory.low, memory.min) are poor fits for workloads whose actual memory
activity varies over time. A high-priority cgroup holding a large working
set but temporarily idle will still suppress reclaim on its siblings,
wasting available memory. A BPF-driven approach can observe real workload
activity -- page faults, charge/uncharge events -- and activate or
withdraw protection dynamically. The test results at the end of this
letter quantify the difference: in a scenario where the high-priority
cgroup is idle, the BPF-controlled low-priority cgroup achieves roughly
37x higher throughput than with static memory.low.
Asynchronous proactive reclaim: the memcg_charged and memcg_uncharged
hooks, combined with the BPF workqueue mechanism and the new
bpf_try_to_free_mem_cgroup_pages() kfunc, enable BPF programs to perform
proactive background reclaim without blocking the charge path. The
pattern works as follows: the memcg_charged callback tracks accumulated
memory usage; when usage crosses a configurable threshold, it enqueues an
asynchronous work item via bpf_wq_start() and returns immediately without
throttling the charging task. The workqueue callback then invokes
bpf_try_to_free_mem_cgroup_pages() to reclaim pages from the target
cgroup; if usage remains elevated after reclaim, the callback re-enqueues
itself to continue. This allows a BPF program to keep a cgroup's
footprint below its hard limit (memory.max) entirely in the background,
avoiding the OOM killer or direct-reclaim stalls that would otherwise
occur. The selftest for this feature (patch 10/11) validates the
mechanism concretely: a workload that writes and mmaps a 64 MB file inside
a 32 MB cgroup reliably triggers memory.events "max" events without BPF;
with the async reclaim program attached, the "max" counter does not
increase at all across the same workload.
In this patch series, I've incorporated a portion of Roman's patch in
[1] to ensure the entire series can be compiled cleanly with bpf-next.
Patch Breakdown:
Patches 1-4 are from Roman Gushchin's series [1], included here to
provide the necessary BPF infrastructure for attaching struct_ops to
cgroups.
Patches 5-11 are the new work in this series:
05/11 bpf: Pass flags in bpf_link_create for struct_ops
Stores attr->link_create.flags in struct bpf_struct_ops_link
and extends the validation to allow BPF_F_ALLOW_OVERRIDE.
Also updates the UAPI comment to reflect that cgroup-bpf attach
flags now apply to BPF_LINK_CREATE in addition to
BPF_PROG_ATTACH.
06/11 mm: memcontrol: Add BPF struct_ops for memory controller
The core feature patch. Introduces the memcg_bpf_ops struct_ops
type with the following hooks:
- memcg_charged(memcg, batch): called on the synchronous charge
path. Returns a throttling delay in milliseconds; used as a
lower bound for __mem_cgroup_handle_over_high(), effective
even when memory.high is not breached.
- memcg_uncharged(memcg, batch): called on uncharge, allowing
BPF programs to track memory releases.
- below_low(memcg, elow, usage): overrides the memory.low
protection check. Returns true to treat the cgroup as
protected regardless of the elow >= usage comparison.
- below_min(memcg, emin, usage): same as below_low but for
memory.min protection.
- handle_cgroup_online/offline(memcg): lifecycle callbacks for
per-cgroup state management in BPF programs.
BPF_F_ALLOW_OVERRIDE is supported: when a program is registered
with this flag, descendant cgroups may attach their own
memcg_bpf_ops to override the inherited policy. Registration
propagates ops down through the subtree via mem_cgroup_iter;
unregistration restores each descendant to the ops its
registering ancestor's parent held, correctly preserving
override chains.
07/11 mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
Exposes try_to_free_mem_cgroup_pages() to BPF programs as a
KF_SLEEPABLE kfunc. A swappiness parameter controls the
override value passed to the core reclaim path
(effective only when MEMCG_RECLAIM_PROACTIVE is set in
reclaim_options).
08/11 selftests/bpf: Add tests for memcg_bpf_ops
Adds prog_tests/memcg_ops.c covering three scenarios:
memcg_charged-only throttling, below_low + memcg_charged
interaction, and below_min + memcg_charged interaction. A
tracepoint on memcg:count_memcg_events (PGFAULT) is used to
detect memory pressure and trigger hooks accordingly.
09/11 selftests/bpf: Add test for memcg_bpf_ops hierarchies
Validates BPF_F_ALLOW_OVERRIDE attachment semantics across a
three-level cgroup hierarchy: attach with ALLOW_OVERRIDE at the
root, override at the middle level without the flag, then assert
that attaching to the leaf correctly fails with -EBUSY.
10/11 selftests/bpf: Add selftest for memcg async reclaim via BPF
Demonstrates and validates asynchronous memory reclaim: a BPF
program uses the memcg_charged/memcg_uncharged hooks to track
accumulated usage and, when a threshold is exceeded, enqueues a
bpf_wq_start() workqueue item that calls
bpf_try_to_free_mem_cgroup_pages() without blocking the charge
path. The test asserts that with the BPF program active,
memory.events "max" events do not increase under a workload
that would otherwise exceed the hard limit.
11/11 samples/bpf: Add memcg priority control and async reclaim example
Adds a complete sample (samples/bpf/memcg.bpf.c + memcg.c)
demonstrating both features. The BPF side monitors PGFAULT
events on a high-priority cgroup; when the per-second fault
count crosses a configurable threshold, it activates below_low
or below_min protection for the high-priority cgroup and/or
applies a charge delay to the low-priority cgroup. Six
struct_ops variants are exported so userspace can attach only
the hooks needed. Async reclaim is optionally combined with
priority throttling via a shared low-cgroup ops map.
Test Environment:
The following examples run on x86_64 QEMU (10 CPUs, 2 GB RAM), using
a tmpfs-backed file on the host as a swap device to reduce I/O impact.
Two cgroups are created -- high (high-priority) and low (low-priority)
-- and each test runs two concurrent stress-ng workloads, one per
cgroup, each requesting 3 GB of memory.
# mkdir /sys/fs/cgroup/high /sys/fs/cgroup/low
# free -h
total used free shared buff/cache available
Mem: 1.9Gi 317Mi 1.6Gi 1.0Mi 144Mi 1.6Gi
Swap: 4.0Gi 0B 4.0Gi
Baseline: no memory priority policy:
Both cgroups run without any reclaim protection. Results are roughly
equal, as expected:
cgroup bogo ops/s
high 4,979
low 4,927
Test 1: memory.low protection:
Setting memory.low on the high-priority cgroup protects it from
reclaim, at the cost of pushing reclaim pressure onto the low-priority
cgroup:
# echo $((3 * 1024 * 1024 * 1024)) > /sys/fs/cgroup/high/memory.low
cgroup bogo ops/s
high 450,290
low 11,307
The high-priority cgroup benefits significantly, but memory.low relies
on static usage thresholds and cannot adapt to actual workload
behavior.
Test 2: memory.low with an idle high-priority task:
Here the high-priority cgroup runs a Python script that allocates 3 GB
and then sleeps, simulating a low-activity but memory-holding workload.
Because the process is idle, it generates no page faults and does not
actively use its memory. Yet memory.low still protects it, continuing
to suppress the low-priority cgroup's performance:
cgroup bogo ops/s
low 14,757
The low-priority cgroup remains significantly throttled despite the
high-priority cgroup being effectively idle -- a clear limitation of
static memory.low control.
Test 3: memcg eBPF -- dynamic priority control:
memcg is a sample program introduced in this patch series
(samples/bpf/memcg.c + memcg.bpf.c). It loads a BPF program that
monitors PGFAULT events in the high-priority cgroup. When the
per-second fault count exceeds a configured threshold, the hook
activates below_min protection for one second; otherwise the cgroup
receives no special treatment.
# ./memcg --low_path=/sys/fs/cgroup/low \
--high_path=/sys/fs/cgroup/high \
--threshold=1 --use_below_min
Successfully attached!
3a. Both cgroups under active memory pressure:
When both cgroups run stress-ng, the high-priority cgroup generates
frequent page faults and the BPF hook activates protection, matching
the behavior of memory.low:
cgroup bogo ops/s
high 404,392
low 11,404
3b. High-priority cgroup is idle (Python + sleep):
Because the sleeping Python process generates no page faults, the BPF
hook never activates, and the low-priority cgroup is free to reclaim
memory normally:
cgroup bogo ops/s
low 551,083
This is a ~37x improvement over the equivalent memory.low scenario
(Test 2), demonstrating that eBPF-driven dynamic control can
accurately reflect actual workload activity and avoid unnecessary
protection of idle high-priority tasks.
Summary:
Scenario low-cgroup bogo ops/s
Baseline (no policy) ~4,927
memory.low, both active ~11,307
memory.low, high idle ~14,757
memcg eBPF, both active ~11,404
memcg eBPF, high idle ~551,083
References:
[1] https://patchew.org/linux/20260127024421.494929-1-roman.gushchin@linux.dev/
Changelog:
v7:
Change base commits of "mm: BPF OOM" to v3.
Some fixes according to the comments of bpf-ci.
Rename get_high_delay_ms hook to memcg_charged; add memcg_uncharged
hook for tracking uncharge events.
Update below_low and below_min hooks to receive elow/emin and usage
as explicit arguments.
Add bpf_try_to_free_mem_cgroup_pages kfunc to expose cgroup reclaim
to BPF programs.
Add selftest for BPF-driven asynchronous page reclaim.
Extend samples/bpf/memcg to support async reclaim in addition to
priority throttling.
v6:
Based on the bot+bof-ci comments, fixed the following issues.
Added fast-path check with unlikely() before SRCU lock acquisition to
optimize the no-BPF case in BPF_MEMCG_CALL.
Add missing newline in pr_warn message to bpf_memcontrol_init.
Added comprehensive child process exit status checking with WIFEXITED()
and WEXITSTATUS(), and added zombie process prevention in
real_test_memcg_ops.
Changed malloc() to calloc() for BSS data allocation in all test
functions and samples main function.
Change srcu_read_lock(&memcg_bpf_srcu) to
lockdep_assert_held(&cgroup_mutex) in function memcontrol_bpf_online
and memcontrol_bpf_offline.
v5:
Based on the bot+bof-ci comments, fixed the following issues.
Fixed issues in memcg_ops.c and memcg.bpf.c by moving variable
declaration to the beginning of need_threshold() function.
The 'u64 current_ts' variable must be declared before any
executable statements
Improved input validation in samples/bpf/memcg.c by adding a new
parse_u64() helper function. This function properly handles errors
from strtoull() and provides better error messages when parsing
threshold and over_high_ms command-line arguments.
Move check for prog->sleepable after validating member offsets in
mm/bpf_memcontrol.c bpf_memcg_ops_check_member.
Fixed sscanf return value checking in prog_tests/memcg_ops.c.
Changed the condition from 'sscanf() < 0' to 'sscanf() != 1' because
sscanf returns the number of successfully matched items, not a negative
value on error. This makes the test more reliable when reading timing
data from temporary files.
v4:
Fix the issues according to the comments from bot+bof-ci.
According to JP Kobryn's comments, move exit(0) from
real_test_memcg_ops_child_work to real_test_memcg_ops.
Fix issues in the bpf_memcg_ops_reg function.
v3:
According to the comments from Michal Koutný and Chen Ridong, update hooks
to get_high_delay_ms, below_low, below_min, handle_cgroup_online, and
handle_cgroup_offline.
According to Michal Koutný's comments, add BPF_F_ALLOW_OVERRIDE
support to memcg_bpf_ops.
v2:
According to Tejun Heo's comments, rebased on Roman Gushchin's BPF
OOM patch series [1] and added hierarchical delegation support.
According to the comments from Roman Gushchin and Michal Hocko, designed
concrete use case scenarios and provided test results.
Hui Zhu (7):
bpf: Pass flags in bpf_link_create for struct_ops
mm: memcontrol: Add BPF struct_ops for memory controller
mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
selftests/bpf: Add tests for memcg_bpf_ops
selftests/bpf: Add test for memcg_bpf_ops hierarchies
selftests/bpf: Add selftest for memcg async reclaim via BPF
samples/bpf: Add memcg priority control and async reclaim example
Roman Gushchin (4):
bpf: move bpf_struct_ops_link into bpf.h
bpf: allow attaching struct_ops to cgroups
libbpf: fix return value on memory allocation failure
libbpf: introduce bpf_map__attach_struct_ops_opts()
MAINTAINERS | 6 +
include/linux/bpf-cgroup-defs.h | 3 +
include/linux/bpf-cgroup.h | 16 +
include/linux/bpf.h | 10 +
include/linux/memcontrol.h | 250 ++++++-
include/uapi/linux/bpf.h | 5 +-
kernel/bpf/bpf_struct_ops.c | 67 +-
kernel/bpf/cgroup.c | 46 ++
mm/bpf_memcontrol.c | 355 +++++++++-
mm/memcontrol.c | 43 +-
samples/bpf/.gitignore | 1 +
samples/bpf/Makefile | 8 +-
samples/bpf/memcg.bpf.c | 380 +++++++++++
samples/bpf/memcg.c | 411 ++++++++++++
tools/include/uapi/linux/bpf.h | 3 +-
tools/lib/bpf/libbpf.c | 22 +-
tools/lib/bpf/libbpf.h | 14 +
tools/lib/bpf/libbpf.map | 1 +
tools/testing/selftests/bpf/cgroup_helpers.c | 41 ++
tools/testing/selftests/bpf/cgroup_helpers.h | 2 +
.../bpf/prog_tests/memcg_async_reclaim.c | 333 +++++++++
.../selftests/bpf/prog_tests/memcg_ops.c | 634 ++++++++++++++++++
.../selftests/bpf/progs/memcg_async_reclaim.c | 203 ++++++
tools/testing/selftests/bpf/progs/memcg_ops.c | 132 ++++
24 files changed, 2952 insertions(+), 34 deletions(-)
create mode 100644 samples/bpf/memcg.bpf.c
create mode 100644 samples/bpf/memcg.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_ops.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops.c
--
2.43.0
^ permalink raw reply
* Re: [PATCH v6 2/4] mm: swap: associate swap devices with tiers
From: Baoquan He @ 2026-05-25 23:04 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-3-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
> This patch connects swap devices to the swap tier infrastructure,
> ensuring that devices are correctly assigned to tiers based on their
> priority.
>
> A `tier_mask` is added to identify the tier membership of swap devices.
> Although tier-based allocation logic is not yet implemented, this
> mapping is necessary to track which tier a device belongs to. Upon
> activation, the device is assigned to a tier by matching its priority
> against the configured tier ranges.
>
> The infrastructure allows dynamic modification of tiers, such as
> splitting or merging ranges. These operations are permitted provided
> that the tier assignment of already configured swap devices remains
> unchanged.
>
> This patch also adds the documentation for the swap tier feature,
> covering the core concepts, sysfs interface usage, and configuration
> details.
>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> ---
> Documentation/mm/index.rst | 1 +
> Documentation/mm/swap-tier.rst | 159 +++++++++++++++++++++++++++++++++
> MAINTAINERS | 1 +
> include/linux/swap.h | 1 +
> mm/swap_state.c | 2 +-
> mm/swap_tier.c | 101 ++++++++++++++++++---
> mm/swap_tier.h | 13 ++-
> mm/swapfile.c | 2 +
> 8 files changed, 266 insertions(+), 14 deletions(-)
> create mode 100644 Documentation/mm/swap-tier.rst
LGTM,
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
^ permalink raw reply
* Re: [PATCH v6 1/4] mm: swap: introduce swap tier infrastructure
From: Baoquan He @ 2026-05-25 22:57 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-2-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
...snip...
> +bool swap_tiers_validate(void)
> +{
> + struct swap_tier *tier;
> +
> + /*
> + * Initial setting might not cover DEF_SWAP_PRIO.
> + * Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
> + */
If so, do we need check if the upmost boundary SHRT_MAX is covered?
> + if (swap_tier_is_active()) {
> + tier = list_last_entry(&swap_tier_active_list,
> + struct swap_tier, list);
> +
> + if (tier->prio != DEF_SWAP_PRIO)
> + return false;
> + }
> +
> + return true;
> +}
...snip...
^ permalink raw reply
* Re: [PATCH v6 1/4] mm: swap: introduce swap tier infrastructure
From: Baoquan He @ 2026-05-25 21:49 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-2-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
> This patch introduces the "Swap tier" concept, which serves as an
> abstraction layer for managing swap devices based on their performance
> characteristics (e.g., NVMe, HDD, Network swap).
>
> Swap tiers are user-named groups representing priority ranges.
> Tier names must consist of alphanumeric characters and underscores.
> These tiers collectively cover the entire priority space from -1
> (`DEF_SWAP_PRIO`) to `SHRT_MAX`.
>
> To configure tiers, a new sysfs interface is exposed at
> /sys/kernel/mm/swap/tiers. The input parser evaluates commands from
> left to right and supports batch input, allowing users to add or remove
> multiple tiers in a single write operation.
>
> Tier management enforces continuous priority ranges anchored by start
> priorities. Operations trigger range splitting or merging, but overwriting
> start priorities is forbidden. Merging expands lower tiers upwards to
> preserve configured start priorities, except when removing `DEF_SWAP_PRIO`,
> which merges downwards.
>
> Suggested-by: Chris Li <chrisl@kernel.org>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
This looks good to me.
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
While there's only one tiny concern, please see the inline comment.
> diff --git a/mm/swap_tier.c b/mm/swap_tier.c
> new file mode 100644
> index 000000000000..9490e891c5fe
> --- /dev/null
> +++ b/mm/swap_tier.c
> @@ -0,0 +1,302 @@
......
> +/*
> + * Naming Convention:
> + * swap_tiers_*() - Public/exported functions
> + * swap_tier_*() - Private/internal functions
> + */
> +
> +static bool swap_tier_is_active(void)
> +{
> + return !list_empty(&swap_tier_active_list) ? true : false;
The above line seems like generated by AI. Whatever else I have seen is
"return !list_empty(&swap_tier_active_list);" which is enough.
> +}
> +
...snip...
^ permalink raw reply
* Re: [PATCH 1/7 v3] mm/page_counter: introduce per-page_counter stock
From: Joshua Hahn @ 2026-05-25 19:45 UTC (permalink / raw)
To: Joshua Hahn
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, cgroups, linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-2-joshua.hahnjy@gmail.com>
On Mon, 25 May 2026 12:04:48 -0700 Joshua Hahn <joshua.hahnjy@gmail.com> wrote:
> In order to avoid expensive hierarchy walks on every memcg charge and
> limit check, memcontrol uses per-cpu stocks (memcg_stock_pcp) to cache
> pre-charged pages and introduce a fast path to try_charge_memcg.
>
> However, there are a few quirks with the current implementation that
> could be improved upon.
>
> First, each memcg_stock_pcp can only cache the charges of 7 memcgs
> (defined as NR_MEMCG_STOCK), which means that once a CPU starts handling
> the charging of more than 7 memcgs, it randomly selects a victim memcg
> to evict and drain from the cpu, which can cause unnecessarily increased
> latencies and thrashing as memcgs continually evict each others' stock.
>
> Second, stock is tightly coupled with memcg, which means that all page
> counters in a memcg share the same resource. This may simplify some of
> the charging logic, but it prevents new page counters from being added
> and using a separate stock.
>
> We can address these concerns by pushing the concept of stock down to
> the page_counter level, which addresses the random eviction problem by
> getting rid of the 7 slot limit, and makes enabling separate stock
> caches for other page_counters simpler.
>
> Introduce a generic per-cpu stock directly in struct page_counter.
> Stock can optionally be enabled per-page_counter, limiting the overhead
> increase for page_counters who do not benefit greatly from caching
> charges.
>
> This patch introduces the page_counter_stock struct and its
> enable/disable/free functions, but does not use these yet.
>
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> ---
> include/linux/page_counter.h | 13 ++++++++
> mm/page_counter.c | 64 ++++++++++++++++++++++++++++++++++++
> 2 files changed, 77 insertions(+)
>
> diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
> index d649b6bbbc871..c7e3ab3356d20 100644
> --- a/include/linux/page_counter.h
> +++ b/include/linux/page_counter.h
> @@ -5,8 +5,15 @@
> #include <linux/atomic.h>
> #include <linux/cache.h>
> #include <linux/limits.h>
> +#include <linux/local_lock.h>
> +#include <linux/percpu.h>
> #include <asm/page.h>
>
> +struct page_counter_stock {
> + local_trylock_t lock;
> + unsigned long nr_pages;
> +};
> +
> struct page_counter {
> /*
> * Make sure 'usage' does not share cacheline with any other field in
> @@ -41,6 +48,8 @@ struct page_counter {
> unsigned long high;
> unsigned long max;
> struct page_counter *parent;
> + struct page_counter_stock __percpu *stock;
> + unsigned int batch;
> } ____cacheline_internodealigned_in_smp;
>
> #if BITS_PER_LONG == 32
> @@ -99,6 +108,10 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
> counter->watermark = usage;
> }
>
> +int page_counter_enable_stock(struct page_counter *counter, unsigned int batch);
> +void page_counter_disable_stock(struct page_counter *counter);
> +void page_counter_free_stock(struct page_counter *counter);
> +
> #if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
> void page_counter_calculate_protection(struct page_counter *root,
> struct page_counter *counter,
> diff --git a/mm/page_counter.c b/mm/page_counter.c
> index 661e0f2a5127a..a1a871a9d5c49 100644
> --- a/mm/page_counter.c
> +++ b/mm/page_counter.c
> @@ -8,6 +8,7 @@
> #include <linux/page_counter.h>
> #include <linux/atomic.h>
> #include <linux/kernel.h>
> +#include <linux/percpu.h>
> #include <linux/string.h>
> #include <linux/sched.h>
> #include <linux/bug.h>
> @@ -289,6 +290,69 @@ int page_counter_memparse(const char *buf, const char *max,
> return 0;
> }
>
> +int page_counter_enable_stock(struct page_counter *counter, unsigned int batch)
> +{
> + struct page_counter_stock __percpu *stock;
> + int cpu;
> +
> + stock = alloc_percpu(struct page_counter_stock);
> + if (!stock)
> + return -ENOMEM;
> +
> + for_each_possible_cpu(cpu) {
> + struct page_counter_stock *s = per_cpu_ptr(stock, cpu);
> +
> + local_trylock_init(&s->lock);
> + }
> + counter->stock = stock;
> + counter->batch = batch;
> +
> + return 0;
> +}
> +
> +static void page_counter_drain_stock_nolock(struct page_counter *counter)
> +{
> + unsigned long stock_to_drain = 0;
> + int cpu;
> +
> + for_each_possible_cpu(cpu) {
> + struct page_counter_stock *stock;
> +
> + stock = per_cpu_ptr(counter->stock, cpu);
> + stock_to_drain += stock->nr_pages;
> + stock->nr_pages = 0;
> + }
> +
> + if (stock_to_drain)
> + page_counter_uncharge(counter, stock_to_drain);
> +}
> +
> +void page_counter_disable_stock(struct page_counter *counter)
> +{
> + if (!counter->stock)
> + return;
> +
> + /* This prevents future charges from trying to deposit pages */
> + WRITE_ONCE(counter->batch, 0);
> +
> + /*
> + * Charges can still be in-flight at this time. Instead of locking here,
> + * do the majority of the drains here without locking to free up pages
> + * now. Any remaining stock will be drained in page_counter_free_stock.
> + */
> + page_counter_drain_stock_nolock(counter);
Sashiko raised a concern here.
Allowing racy charges is OK, but the problem is that writing stock->nr_pages = 0
with no lock here can race with the reading of that value, and lead to
double-uncharging when racing with concurrent charges.
I think that this can be fixed by not draining in disable_stock and
reordering the callsite:
Before:
drain_all_stock(memcg);
page_counter_disable_stock(&memcg->memory);
After:
page_counter_disable_stock(&memcg->memory);
drain_all_stock(memcg);
This way, the WRITE_ONCE(counter->batch, 0); should prevent any
future charges from trying to land, before we drain stock.
Despite not allowing any more racy charges, we still need to keep the drain
in free_stock since drain_all_stock() uses a mutex_trylock and can fail;
if that happens we need to still drain the stock at a later time, when we
can guarantee that there will be no more contention.
Thanks, Sashiko! I'll keep my eyes posted on the rest of the series as it
goes through the review pipeline.
Joshua
> +}
> +
> +void page_counter_free_stock(struct page_counter *counter)
> +{
> + if (!counter->stock)
> + return;
> +
> + page_counter_drain_stock_nolock(counter);
> + free_percpu(counter->stock);
> + counter->stock = NULL;
> +}
> +
^ permalink raw reply
* Re: [PATCH v2 0/4] mm/zswap: Implement per-cgroup proactive writeback
From: Andrew Morton @ 2026-05-25 19:24 UTC (permalink / raw)
To: Hao Jia
Cc: tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
On Mon, 25 May 2026 20:22:38 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
> Zswap currently writes back pages to backing swap reactively, triggered
> either by the shrinker or by the pool reaching its size limit. Although
> proactive memory reclaim can automatically write back a portion of zswap
> pages via the shrinker, it cannot explicitly control the amount of
> writeback for a specific memory cgroup. Moreover, proactive memory reclaim
> may not always be triggered during a steady state.
>
> In certain scenarios, it is desirable to trigger writeback in advance to
> free up memory. For example, users may want to prepare for an upcoming
> memory-intensive workload by flushing cold memory to the backing storage
> when the system is relatively idle.
>
> This patch series introduces a "zswap_writeback_only" key to memory.reclaim
> cgroup interface, allowing users to proactively write back cold compressed
> pages from zswap to the backing swap device. When specified, this key
> bypasses standard memory reclaim and exclusively performs proactive zswap
> writeback up to the requested budget. If omitted, the default reclaim
> behavior remains unchanged.
Thanks. AI review found a few things to complain about, one of them
described as "preexisting".
^ permalink raw reply
* [PATCH 7/7 v3] mm/memcontrol: remove unused memcg_stock code
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Now that all memcg_stock logic has been moved to page_counter_stock, we
can remove all code related to handling memcg_stock. Note that obj_stock
is untouched and is still needed. FLUSHING_CACHED_CHARGE is preserved
so that it can be used by obj_stock as well.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 186 ------------------------------------------------
1 file changed, 186 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 64b82f1782720..5319219d0dcb5 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1998,25 +1998,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
pr_cont(" are going to be killed due to memory.oom.group set\n");
}
-/*
- * The value of NR_MEMCG_STOCK is selected to keep the cached memcgs and their
- * nr_pages in a single cacheline. This may change in future.
- */
-#define NR_MEMCG_STOCK 7
#define FLUSHING_CACHED_CHARGE 0
-struct memcg_stock_pcp {
- local_trylock_t lock;
- uint8_t nr_pages[NR_MEMCG_STOCK];
- struct mem_cgroup *cached[NR_MEMCG_STOCK];
-
- struct work_struct work;
- unsigned long flags;
- uint8_t drain_idx;
-};
-
-static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
- .lock = INIT_LOCAL_TRYLOCK(lock),
-};
/*
* NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
@@ -2056,47 +2038,6 @@ static void drain_obj_stock(struct obj_stock_pcp *stock);
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg);
-/**
- * consume_stock: Try to consume stocked charge on this cpu.
- * @memcg: memcg to consume from.
- * @nr_pages: how many pages to charge.
- *
- * Consume the cached charge if enough nr_pages are present otherwise return
- * failure. Also return failure for charge request larger than
- * MEMCG_CHARGE_BATCH or if the local lock is already taken.
- *
- * returns true if successful, false otherwise.
- */
-static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
-{
- struct memcg_stock_pcp *stock;
- uint8_t stock_pages;
- bool ret = false;
- int i;
-
- if (nr_pages > MEMCG_CHARGE_BATCH ||
- !local_trylock(&memcg_stock.lock))
- return ret;
-
- stock = this_cpu_ptr(&memcg_stock);
-
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- if (memcg != READ_ONCE(stock->cached[i]))
- continue;
-
- stock_pages = READ_ONCE(stock->nr_pages[i]);
- if (stock_pages >= nr_pages) {
- WRITE_ONCE(stock->nr_pages[i], stock_pages - nr_pages);
- ret = true;
- }
- break;
- }
-
- local_unlock(&memcg_stock.lock);
-
- return ret;
-}
-
static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
page_counter_uncharge(&memcg->memory, nr_pages);
@@ -2104,51 +2045,6 @@ static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
page_counter_uncharge(&memcg->memsw, nr_pages);
}
-/*
- * Returns stocks cached in percpu and reset cached information.
- */
-static void drain_stock(struct memcg_stock_pcp *stock, int i)
-{
- struct mem_cgroup *old = READ_ONCE(stock->cached[i]);
- uint8_t stock_pages;
-
- if (!old)
- return;
-
- stock_pages = READ_ONCE(stock->nr_pages[i]);
- if (stock_pages) {
- memcg_uncharge(old, stock_pages);
- WRITE_ONCE(stock->nr_pages[i], 0);
- }
-
- css_put(&old->css);
- WRITE_ONCE(stock->cached[i], NULL);
-}
-
-static void drain_stock_fully(struct memcg_stock_pcp *stock)
-{
- int i;
-
- for (i = 0; i < NR_MEMCG_STOCK; ++i)
- drain_stock(stock, i);
-}
-
-static void drain_local_memcg_stock(struct work_struct *dummy)
-{
- struct memcg_stock_pcp *stock;
-
- if (WARN_ONCE(!in_task(), "drain in non-task context"))
- return;
-
- local_lock(&memcg_stock.lock);
-
- stock = this_cpu_ptr(&memcg_stock);
- drain_stock_fully(stock);
- clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
-
- local_unlock(&memcg_stock.lock);
-}
-
static void drain_local_obj_stock(struct work_struct *dummy)
{
struct obj_stock_pcp *stock;
@@ -2165,88 +2061,6 @@ static void drain_local_obj_stock(struct work_struct *dummy)
local_unlock(&obj_stock.lock);
}
-static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
-{
- struct memcg_stock_pcp *stock;
- struct mem_cgroup *cached;
- uint8_t stock_pages;
- bool success = false;
- int empty_slot = -1;
- int i;
-
- /*
- * For now limit MEMCG_CHARGE_BATCH to 127 and less. In future if we
- * decide to increase it more than 127 then we will need more careful
- * handling of nr_pages[] in struct memcg_stock_pcp.
- */
- BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S8_MAX);
-
- VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg));
-
- if (nr_pages > MEMCG_CHARGE_BATCH ||
- !local_trylock(&memcg_stock.lock)) {
- /*
- * In case of larger than batch refill or unlikely failure to
- * lock the percpu memcg_stock.lock, uncharge memcg directly.
- */
- memcg_uncharge(memcg, nr_pages);
- return;
- }
-
- stock = this_cpu_ptr(&memcg_stock);
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- cached = READ_ONCE(stock->cached[i]);
- if (!cached && empty_slot == -1)
- empty_slot = i;
- if (memcg == READ_ONCE(stock->cached[i])) {
- stock_pages = READ_ONCE(stock->nr_pages[i]) + nr_pages;
- WRITE_ONCE(stock->nr_pages[i], stock_pages);
- if (stock_pages > MEMCG_CHARGE_BATCH)
- drain_stock(stock, i);
- success = true;
- break;
- }
- }
-
- if (!success) {
- i = empty_slot;
- if (i == -1) {
- i = stock->drain_idx++;
- if (stock->drain_idx == NR_MEMCG_STOCK)
- stock->drain_idx = 0;
- drain_stock(stock, i);
- }
- css_get(&memcg->css);
- WRITE_ONCE(stock->cached[i], memcg);
- WRITE_ONCE(stock->nr_pages[i], nr_pages);
- }
-
- local_unlock(&memcg_stock.lock);
-}
-
-static bool is_memcg_drain_needed(struct memcg_stock_pcp *stock,
- struct mem_cgroup *root_memcg)
-{
- struct mem_cgroup *memcg;
- bool flush = false;
- int i;
-
- rcu_read_lock();
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- memcg = READ_ONCE(stock->cached[i]);
- if (!memcg)
- continue;
-
- if (READ_ONCE(stock->nr_pages[i]) &&
- mem_cgroup_is_descendant(memcg, root_memcg)) {
- flush = true;
- break;
- }
- }
- rcu_read_unlock();
- return flush;
-}
-
static void schedule_drain_work(int cpu, struct work_struct *work)
{
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 6/7 v3] mm/memcontrol: optimize stock usage for cgroup v2
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
In cgroup v2, it is unlikely for memcg charges to happen directly on
non-leaf cgroups. There are a few exceptions, such as processes that
have yet to be migrated to children, and tasks that are reparented on
memcg destruction, that charge to the parent.
Because it is rare for parent cgroups to receive direct charges, stock
that remains in them are wasted memory.
Drain parent stocks when the first child is created to return those
pages for other memcgs to use.
This optimization is not for cgroup v1, where tasks can be attached to
any cgroup in the hierarchy, meaning stock can be consumed & refilled
for non-leaf cgroups as well.
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f20c9b829224a..64b82f1782720 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4280,6 +4280,21 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
*/
xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL);
+ /*
+ * It is unlikely for non-leaf memcgs to get direct charges on v2.
+ * Drain the parent's stock if we are the first child.
+ */
+ if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
+ struct mem_cgroup *parent = parent_mem_cgroup(memcg);
+ int cpu;
+
+ if (parent && !mem_cgroup_is_root(parent) &&
+ !css_has_online_children(&parent->css)) {
+ for_each_online_cpu(cpu)
+ work_on_cpu(cpu, drain_stock_on_cpu, parent);
+ }
+ }
+
return 0;
free_objcg:
for_each_node(nid) {
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 5/7 v3] mm/memcontrol: optimize memsw stock for cgroup v1
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Previously, each memcg had its own stock, which was shared by all page
counters within it. Specifically in try_charge_memcg, the stock limit
check would occur before the memsw and memory page_counters were
charged hierarchically. Now that the memcg stock was folded into the
page_counter level, and we have replaced try_charge_memcg's stock check
against the memory page_counter's stock, this leaves no fast path available
for cgroup v1's memsw check.
Introduce a new stock for the memsw page_counter, charged independently
from the memory page_counter. This provides better caching on cgroup v1:
The best case scenario is when both the memsw and memory page_counters
can use their cached stock charge; this is the old behavior.
The halfway scenario is when either the memsw or memory page_counter
is within the stock size, but the other isn't. This requires one
hierarchical charge.
The worst case scenario is when both memsw and memory page_counters
are over their limit, and must walk two page_counter hierarchies. This
is the same as the old behavior.
By introducing an independent stock for memsw, we can avoid the worst
case scenario more often and can fail or succeed separately from the
memory page counter.
One user-visible change is that reported memsw usage may transiently
be lower than memory usage. This happens because each counter
independently batches the stock charges, so the visible values can
differ by up to the stock batch size (MEMCG_CHARGE_BATCH) pages.
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 952c6f7430395..f20c9b829224a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2265,8 +2265,11 @@ static long drain_stock_on_cpu(void *arg)
struct mem_cgroup *root_memcg = arg;
struct mem_cgroup *memcg;
- for_each_mem_cgroup_tree(memcg, root_memcg)
+ for_each_mem_cgroup_tree(memcg, root_memcg) {
page_counter_drain_stock_local(&memcg->memory);
+ if (do_memsw_account())
+ page_counter_drain_stock_local(&memcg->memsw);
+ }
return 0;
}
@@ -2313,8 +2316,11 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu)
/* no need for the local lock */
drain_obj_stock(&per_cpu(obj_stock, cpu));
- for_each_mem_cgroup(memcg)
+ for_each_mem_cgroup(memcg) {
page_counter_drain_stock_cpu(&memcg->memory, cpu);
+ if (do_memsw_account())
+ page_counter_drain_stock_cpu(&memcg->memsw, cpu);
+ }
return 0;
}
@@ -4259,6 +4265,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
/* failure is nonfatal, charges fall back to direct hierarchy */
page_counter_enable_stock(&memcg->memory, MEMCG_CHARGE_BATCH);
+ if (do_memsw_account())
+ page_counter_enable_stock(&memcg->memsw, MEMCG_CHARGE_BATCH);
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
@@ -4323,6 +4331,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
drain_all_stock(memcg);
page_counter_disable_stock(&memcg->memory);
+ if (do_memsw_account())
+ page_counter_disable_stock(&memcg->memsw);
mem_cgroup_private_id_put(memcg, 1);
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 4/7 v3] mm/memcontrol: convert memcg to use page_counter_stock
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Now with all of the memcg_stock handling logic replicated in
page_counter_stock, switch memcg to use the page_counter_stock.
There are a few details that have changed:
First, the old special-casing for the !allow_spinning check to avoid
refilling and flushing of the old stock is removed. This special casing
was important previously, because refilling the stock could do a lot of
extra work by evicting one of 7 random victim memcgs in the percpu
memcg_stock slots. In the new per-counter design, refilling stock just adds
pages to the counter's own local cache without affecting other memcgs,
so the original reason for the special case no longer applies.
Also, we can now fail during page_counter_enable_stock(), if there is
not enough memory to allocate a percpu page_counter_stock. This failure
is rare and nonfatal; the system can continue to operate, with the page
counter working without stock and falling back to walking the hierarchy.
Finally, drain_all_stock is restructured to iterate CPUs in the outer
loop (rather than memcgs) to be able to schedule draining all memcgs
via a single work_on_cpu call. It reduces the number of synchronous
per-CPU work calls from O(memcgs * CPUs) to just O(CPUs).
Note that obj_stock remains untouched by these changes.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 78 +++++++++++++++++++++----------------------------
1 file changed, 34 insertions(+), 44 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 368efc1455e35..952c6f7430395 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2260,6 +2260,17 @@ static void schedule_drain_work(int cpu, struct work_struct *work)
queue_work_on(cpu, memcg_wq, work);
}
+static long drain_stock_on_cpu(void *arg)
+{
+ struct mem_cgroup *root_memcg = arg;
+ struct mem_cgroup *memcg;
+
+ for_each_mem_cgroup_tree(memcg, root_memcg)
+ page_counter_drain_stock_local(&memcg->memory);
+
+ return 0;
+}
+
/*
* Drains all per-CPU charge caches for given root_memcg resp. subtree
* of the hierarchy under it.
@@ -2271,28 +2282,16 @@ void drain_all_stock(struct mem_cgroup *root_memcg)
/* If someone's already draining, avoid adding running more workers. */
if (!mutex_trylock(&percpu_charge_mutex))
return;
- /*
- * Notify other cpus that system-wide "drain" is running
- * We do not care about races with the cpu hotplug because cpu down
- * as well as workers from this path always operate on the local
- * per-cpu data. CPU up doesn't touch memcg_stock at all.
- */
+
+ for_each_online_cpu(cpu)
+ work_on_cpu(cpu, drain_stock_on_cpu, root_memcg);
+
+ /* Drain obj_stock on all online CPUs */
migrate_disable();
curcpu = smp_processor_id();
for_each_online_cpu(cpu) {
- struct memcg_stock_pcp *memcg_st = &per_cpu(memcg_stock, cpu);
struct obj_stock_pcp *obj_st = &per_cpu(obj_stock, cpu);
- if (!test_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags) &&
- is_memcg_drain_needed(memcg_st, root_memcg) &&
- !test_and_set_bit(FLUSHING_CACHED_CHARGE,
- &memcg_st->flags)) {
- if (cpu == curcpu)
- drain_local_memcg_stock(&memcg_st->work);
- else
- schedule_drain_work(cpu, &memcg_st->work);
- }
-
if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) &&
obj_stock_flush_required(obj_st, root_memcg) &&
!test_and_set_bit(FLUSHING_CACHED_CHARGE,
@@ -2309,9 +2308,13 @@ void drain_all_stock(struct mem_cgroup *root_memcg)
static int memcg_hotplug_cpu_dead(unsigned int cpu)
{
+ struct mem_cgroup *memcg;
+
/* no need for the local lock */
drain_obj_stock(&per_cpu(obj_stock, cpu));
- drain_stock_fully(&per_cpu(memcg_stock, cpu));
+
+ for_each_mem_cgroup(memcg)
+ page_counter_drain_stock_cpu(&memcg->memory, cpu);
return 0;
}
@@ -2586,7 +2589,6 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
unsigned int nr_pages)
{
- unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
int nr_retries = MAX_RECLAIM_RETRIES;
struct mem_cgroup *mem_over_limit;
struct page_counter *counter;
@@ -2599,31 +2601,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
bool allow_spinning = gfpflags_allow_spinning(gfp_mask);
retry:
- if (consume_stock(memcg, nr_pages))
- return 0;
-
- if (!allow_spinning)
- /* Avoid the refill and flush of the older stock */
- batch = nr_pages;
-
reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
if (!do_memsw_account() ||
- page_counter_try_charge(&memcg->memsw, batch, &counter)) {
- if (page_counter_try_charge(&memcg->memory, batch, &counter))
+ page_counter_try_charge(&memcg->memsw, nr_pages, &counter)) {
+ if (page_counter_try_charge(&memcg->memory, nr_pages, &counter))
goto done_restock;
if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, batch);
+ page_counter_uncharge(&memcg->memsw, nr_pages);
mem_over_limit = mem_cgroup_from_counter(counter, memory);
} else {
mem_over_limit = mem_cgroup_from_counter(counter, memsw);
reclaim_options &= ~MEMCG_RECLAIM_MAY_SWAP;
}
- if (batch > nr_pages) {
- batch = nr_pages;
- goto retry;
- }
-
/*
* Prevent unbounded recursion when reclaim operations need to
* allocate memory. This might exceed the limits temporarily,
@@ -2720,9 +2710,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
return 0;
done_restock:
- if (batch > nr_pages)
- refill_stock(memcg, batch - nr_pages);
-
/*
* If the hierarchy is above the normal consumption range, schedule
* reclaim on returning to userland. We can perform reclaim here
@@ -2759,7 +2746,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
* and distribute reclaim work and delay penalties
* based on how much each task is actually allocating.
*/
- current->memcg_nr_pages_over_high += batch;
+ current->memcg_nr_pages_over_high += nr_pages;
set_notify_resume(current);
break;
}
@@ -3064,7 +3051,7 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg,
account_kmem_nmi_safe(memcg, -nr_pages);
memcg1_account_kmem(memcg, -nr_pages);
if (!mem_cgroup_is_root(memcg))
- refill_stock(memcg, nr_pages);
+ memcg_uncharge(memcg, nr_pages);
css_put(&memcg->css);
}
@@ -4096,6 +4083,8 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)
static void mem_cgroup_free(struct mem_cgroup *memcg)
{
+ page_counter_free_stock(&memcg->memory);
+ page_counter_free_stock(&memcg->memsw);
lru_gen_exit_memcg(memcg);
memcg_wb_domain_exit(memcg);
__mem_cgroup_free(memcg);
@@ -4268,6 +4257,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
refcount_set(&memcg->id.ref, 1);
css_get(css);
+ /* failure is nonfatal, charges fall back to direct hierarchy */
+ page_counter_enable_stock(&memcg->memory, MEMCG_CHARGE_BATCH);
+
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
*
@@ -4330,6 +4322,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
lru_gen_offline_memcg(memcg);
drain_all_stock(memcg);
+ page_counter_disable_stock(&memcg->memory);
mem_cgroup_private_id_put(memcg, 1);
}
@@ -5524,7 +5517,7 @@ void mem_cgroup_sk_uncharge(const struct sock *sk, unsigned int nr_pages)
mod_memcg_state(memcg, MEMCG_SOCK, -nr_pages);
- refill_stock(memcg, nr_pages);
+ page_counter_uncharge(&memcg->memory, nr_pages);
}
void mem_cgroup_flush_workqueue(void)
@@ -5577,12 +5570,9 @@ int __init mem_cgroup_init(void)
memcg_wq = alloc_workqueue("memcg", WQ_PERCPU, 0);
WARN_ON(!memcg_wq);
- for_each_possible_cpu(cpu) {
- INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work,
- drain_local_memcg_stock);
+ for_each_possible_cpu(cpu)
INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work,
drain_local_obj_stock);
- }
memcg_size = struct_size_t(struct mem_cgroup, nodeinfo, nr_node_ids);
memcg_cachep = kmem_cache_create("mem_cgroup", memcg_size, 0,
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 3/7 v3] mm/page_counter: introduce stock drain APIs
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Introduce page_counter variants to replace memcg stock draining
functions.
page_counter_drain_stock_local() drains the stock of the local CPU,
taking a local stock lock to serialize against concurrent charges.
page_counter_drain_stock_cpu() does the same, but without taking a local
lock. This is possible because it will only be called from the CPU
hotplug path, where the CPU is dead and there cannot be any more charges.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
include/linux/page_counter.h | 3 +++
mm/page_counter.c | 34 ++++++++++++++++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index c7e3ab3356d20..ffe13224213c9 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -111,6 +111,9 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
int page_counter_enable_stock(struct page_counter *counter, unsigned int batch);
void page_counter_disable_stock(struct page_counter *counter);
void page_counter_free_stock(struct page_counter *counter);
+void page_counter_drain_stock_local(struct page_counter *counter);
+void page_counter_drain_stock_cpu(struct page_counter *counter,
+ unsigned int cpu);
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
void page_counter_calculate_protection(struct page_counter *root,
diff --git a/mm/page_counter.c b/mm/page_counter.c
index e002688bf7f1a..fbfe9a1b29d2e 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -389,6 +389,40 @@ void page_counter_free_stock(struct page_counter *counter)
counter->stock = NULL;
}
+void page_counter_drain_stock_local(struct page_counter *counter)
+{
+ struct page_counter_stock *stock;
+ unsigned long nr_pages;
+
+ if (!counter->stock)
+ return;
+
+ local_lock(&counter->stock->lock);
+ stock = this_cpu_ptr(counter->stock);
+ nr_pages = stock->nr_pages;
+ stock->nr_pages = 0;
+ local_unlock(&counter->stock->lock);
+
+ if (nr_pages)
+ page_counter_uncharge(counter, nr_pages);
+}
+
+void page_counter_drain_stock_cpu(struct page_counter *counter,
+ unsigned int cpu)
+{
+ struct page_counter_stock *stock;
+ unsigned long nr_pages;
+
+ if (!counter->stock)
+ return;
+
+ stock = per_cpu_ptr(counter->stock, cpu);
+ nr_pages = stock->nr_pages;
+ if (nr_pages) {
+ stock->nr_pages = 0;
+ page_counter_uncharge(counter, nr_pages);
+ }
+}
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 2/7 v3] mm/page_counter: use page_counter_stock in page_counter_try_charge
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Make page_counter_try_charge() stock-aware. We preserve the same
semantics as the existing stock handling logic in try_charge_memcg:
1. Limit-check against the stock. If there is enough, charge to the
stock (non-hierarchical) and return immediately.
2. Greedily attempt to fulfill the charge request and fill the stock up
at the same time via a hierarchical charge.
3. If we fail with this charge, retry again (once) with the exact number
of pages requested.
4. If we succeed with the greedy attempt, then try to add those extra
pages to the stock. If that fails (trylock), then uncharge those
surplus pages hierarchically.
As of this patch, the page_counter_stock is unused, as it has not been
enabled on any memcg yet. No functional changes intended.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/page_counter.c | 42 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/mm/page_counter.c b/mm/page_counter.c
index a1a871a9d5c49..e002688bf7f1a 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -121,9 +121,25 @@ bool page_counter_try_charge(struct page_counter *counter,
struct page_counter **fail)
{
struct page_counter *c;
+ unsigned long charge = nr_pages;
+ unsigned long batch = READ_ONCE(counter->batch);
bool protection = track_protection(counter);
bool track_failcnt = counter->track_failcnt;
+ if (counter->stock && local_trylock(&counter->stock->lock)) {
+ struct page_counter_stock *stock = this_cpu_ptr(counter->stock);
+
+ if (stock->nr_pages >= charge) {
+ stock->nr_pages -= charge;
+ local_unlock(&counter->stock->lock);
+ return true;
+ }
+ local_unlock(&counter->stock->lock);
+ }
+
+ charge = max_t(unsigned long, batch, nr_pages);
+
+retry:
for (c = counter; c; c = c->parent) {
long new;
/*
@@ -140,9 +156,9 @@ bool page_counter_try_charge(struct page_counter *counter,
* we either see the new limit or the setter sees the
* counter has changed and retries.
*/
- new = atomic_long_add_return(nr_pages, &c->usage);
+ new = atomic_long_add_return(charge, &c->usage);
if (new > c->max) {
- atomic_long_sub(nr_pages, &c->usage);
+ atomic_long_sub(charge, &c->usage);
/*
* This is racy, but we can live with some
* inaccuracy in the failcnt which is only used
@@ -163,11 +179,31 @@ bool page_counter_try_charge(struct page_counter *counter,
WRITE_ONCE(c->watermark, new);
}
}
+
+ /* charge > nr_pages implies this page_counter has stock enabled */
+ if (charge > nr_pages) {
+ if (local_trylock(&counter->stock->lock)) {
+ struct page_counter_stock *stock;
+
+ stock = this_cpu_ptr(counter->stock);
+ stock->nr_pages += charge - nr_pages;
+ local_unlock(&counter->stock->lock);
+ } else {
+ page_counter_uncharge(counter, charge - nr_pages);
+ }
+ }
+
return true;
failed:
for (c = counter; c != *fail; c = c->parent)
- page_counter_cancel(c, nr_pages);
+ page_counter_cancel(c, charge);
+
+ if (charge > nr_pages) {
+ /* Retry without trying to grab extra pages to refill stock */
+ charge = nr_pages;
+ goto retry;
+ }
return false;
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 1/7 v3] mm/page_counter: introduce per-page_counter stock
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
In order to avoid expensive hierarchy walks on every memcg charge and
limit check, memcontrol uses per-cpu stocks (memcg_stock_pcp) to cache
pre-charged pages and introduce a fast path to try_charge_memcg.
However, there are a few quirks with the current implementation that
could be improved upon.
First, each memcg_stock_pcp can only cache the charges of 7 memcgs
(defined as NR_MEMCG_STOCK), which means that once a CPU starts handling
the charging of more than 7 memcgs, it randomly selects a victim memcg
to evict and drain from the cpu, which can cause unnecessarily increased
latencies and thrashing as memcgs continually evict each others' stock.
Second, stock is tightly coupled with memcg, which means that all page
counters in a memcg share the same resource. This may simplify some of
the charging logic, but it prevents new page counters from being added
and using a separate stock.
We can address these concerns by pushing the concept of stock down to
the page_counter level, which addresses the random eviction problem by
getting rid of the 7 slot limit, and makes enabling separate stock
caches for other page_counters simpler.
Introduce a generic per-cpu stock directly in struct page_counter.
Stock can optionally be enabled per-page_counter, limiting the overhead
increase for page_counters who do not benefit greatly from caching
charges.
This patch introduces the page_counter_stock struct and its
enable/disable/free functions, but does not use these yet.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
include/linux/page_counter.h | 13 ++++++++
mm/page_counter.c | 64 ++++++++++++++++++++++++++++++++++++
2 files changed, 77 insertions(+)
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index d649b6bbbc871..c7e3ab3356d20 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -5,8 +5,15 @@
#include <linux/atomic.h>
#include <linux/cache.h>
#include <linux/limits.h>
+#include <linux/local_lock.h>
+#include <linux/percpu.h>
#include <asm/page.h>
+struct page_counter_stock {
+ local_trylock_t lock;
+ unsigned long nr_pages;
+};
+
struct page_counter {
/*
* Make sure 'usage' does not share cacheline with any other field in
@@ -41,6 +48,8 @@ struct page_counter {
unsigned long high;
unsigned long max;
struct page_counter *parent;
+ struct page_counter_stock __percpu *stock;
+ unsigned int batch;
} ____cacheline_internodealigned_in_smp;
#if BITS_PER_LONG == 32
@@ -99,6 +108,10 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
counter->watermark = usage;
}
+int page_counter_enable_stock(struct page_counter *counter, unsigned int batch);
+void page_counter_disable_stock(struct page_counter *counter);
+void page_counter_free_stock(struct page_counter *counter);
+
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
void page_counter_calculate_protection(struct page_counter *root,
struct page_counter *counter,
diff --git a/mm/page_counter.c b/mm/page_counter.c
index 661e0f2a5127a..a1a871a9d5c49 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -8,6 +8,7 @@
#include <linux/page_counter.h>
#include <linux/atomic.h>
#include <linux/kernel.h>
+#include <linux/percpu.h>
#include <linux/string.h>
#include <linux/sched.h>
#include <linux/bug.h>
@@ -289,6 +290,69 @@ int page_counter_memparse(const char *buf, const char *max,
return 0;
}
+int page_counter_enable_stock(struct page_counter *counter, unsigned int batch)
+{
+ struct page_counter_stock __percpu *stock;
+ int cpu;
+
+ stock = alloc_percpu(struct page_counter_stock);
+ if (!stock)
+ return -ENOMEM;
+
+ for_each_possible_cpu(cpu) {
+ struct page_counter_stock *s = per_cpu_ptr(stock, cpu);
+
+ local_trylock_init(&s->lock);
+ }
+ counter->stock = stock;
+ counter->batch = batch;
+
+ return 0;
+}
+
+static void page_counter_drain_stock_nolock(struct page_counter *counter)
+{
+ unsigned long stock_to_drain = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ struct page_counter_stock *stock;
+
+ stock = per_cpu_ptr(counter->stock, cpu);
+ stock_to_drain += stock->nr_pages;
+ stock->nr_pages = 0;
+ }
+
+ if (stock_to_drain)
+ page_counter_uncharge(counter, stock_to_drain);
+}
+
+void page_counter_disable_stock(struct page_counter *counter)
+{
+ if (!counter->stock)
+ return;
+
+ /* This prevents future charges from trying to deposit pages */
+ WRITE_ONCE(counter->batch, 0);
+
+ /*
+ * Charges can still be in-flight at this time. Instead of locking here,
+ * do the majority of the drains here without locking to free up pages
+ * now. Any remaining stock will be drained in page_counter_free_stock.
+ */
+ page_counter_drain_stock_nolock(counter);
+}
+
+void page_counter_free_stock(struct page_counter *counter)
+{
+ if (!counter->stock)
+ return;
+
+ page_counter_drain_stock_nolock(counter);
+ free_percpu(counter->stock);
+ counter->stock = NULL;
+}
+
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 0/7 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged
allocations, allowing small allocations to avoid walking the expensive
mem_cgroup hierarchy traversal and atomic operations on each charge.
This design introduces a fastpath, but there is room for improvement:
1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When
more than 7 mem_cgroups are actively charging on a single CPU, a
random victim is evicted and its associated stock is drained.
2. Stock management is tightly coupled to struct mem_cgroup, which makes
it difficult to add a new page_counter to mem_cgroup and have
multiple sources of stock management, which is required when trying
to introduce fastpaths to multiple hard limit checks.
This series moves the per-cpu stock down into the page_counter which
consolidates stock limit checking and page_counter limit checking into
page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit,
the random evictions (drain & refill), and slot traversal.
In turn, we can add independent stock management for additional
page_counters in each memcg, which is used in my tiered memory limits
series to add a new page_counter to track toptier usage [1].
The resulting code in memcg is also easier to follow, as the caching
becomes transparent from memcg's perspective and managed entirely within
page_counter.
There are, however, a few tradeoffs.
First, the bound on how much memory can be overcharged (and remain stale
as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages.
Now, it becomes nr_leaf_cgroups x nr_cpus x 64 pages. On large machines
with many cgroups, this could be significant. There are three qualifying
points: (1) larger machines should be able to tolerate the additional
overhead, (2) the stock should not remain stale as long as the
cgroups are actively charging memory, and (3) a process would have to
migrate across all CPUs to incur this upper bound on overhead.
Secondly, we introduce some additional memory footprint. The new struct
page_counter_stock adds 2 words of extra overhead per-(cpu x memcg).
A small change is that for cgroupv1, reported memsw usage can be lower
than reported memory usage, if the memsw page_counter overcharges to its
stock whereas the memory page_counter does not.
Finally, to keep the above memory footprint limited, I opted to not
embed a work_struct into page_counter_stock, but rather decided to
trigger synchronous stock draining, since the drain operation is rarer
now, and only happens under memory pressure and on cgroup death.
Performance testing across single-cgroup, as well as 4-cgroup (under the
7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system
shows negligible performance differences. In the tests, I repeatedly
fault and release anonymous pages using madvise(MADV_DONTNEED) to
stress the charge/uncharge path, across 40 trials of 50 iterations.
Metric here is time it took across each iteration (ms).
There are two testing versions below; the only difference is that v3
is based on top of mm-new, and v2 is based on top of mm-stable. The
"after" on both sides are similar, but mm-new and mm-stable have
different perforamnces.
v3, tested against mm-new
+----------+--------+-------+-----------+
| #cgroups | mm-new | after | delta (%) |
+----------+--------+-------+-----------+
| 1 | 357 | 358 | +0.283 |
| 4 | 1245 | 1214 | -2.430 |
| 32 | 9281 | 8970 | -3.470 |
+----------+--------+-------+-----------+
v2, tested against mm-stable
+----------+-----------+-------+-----------+
| #cgroups | mm-stable | after | delta (%) |
+----------+-----------+-------+-----------+
| 1 | 352 | 353 | +0.283 |
| 4 | 1198 | 1217 | +1.585 |
| 32 | 8980 | 9027 | +0.526 |
+----------+-----------+-------+-----------+
Further testing on other stress-ng microbenchmarks also agreed with
these results.
v2 --> v3:
- Rebased on top of latest mm-new, May 25, 2026, since the previous
version could not be applied for Sashiko review.
- Re-ran test numbers
v1 --> v2:
- Dropped stock returning on uncharge to preserve same behavior as memcg
stock. This resolves some race conditions present in v1.
- Fixed many race conditions between disabling page_counter_stock and
in-flight charges
- Restructured drain_all_stock to iterate over all CPUs first before
memcgs, to reduce the number of synchronous CPU work scheduling
- Optimized cgroup v2 further to drain only on the first child and skip
the root mem_cgroup
- Dropped RFC
- Wordsmithing cover letter
[1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/
Joshua Hahn (7):
mm/page_counter: introduce per-page_counter stock
mm/page_counter: use page_counter_stock in page_counter_try_charge
mm/page_counter: introduce stock drain APIs
mm/memcontrol: convert memcg to use page_counter_stock
mm/memcontrol: optimize memsw stock for cgroup v1
mm/memcontrol: optimize stock usage for cgroup v2
mm/memcontrol: remove unused memcg_stock code
include/linux/page_counter.h | 16 ++
mm/memcontrol.c | 289 +++++++----------------------------
mm/page_counter.c | 140 ++++++++++++++++-
3 files changed, 212 insertions(+), 233 deletions(-)
--
2.53.0-Meta
^ permalink raw reply
* Re: [PATCH v2 0/4] memcg: shrink obj_stock_pcp and cache multiple objcgs
From: Shakeel Butt @ 2026-05-25 18:53 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260522193440.40e20563422afcc69b8445dd@linux-foundation.org>
On Fri, May 22, 2026 at 07:34:40PM -0700, Andrew Morton wrote:
> On Thu, 21 May 2026 18:19:04 -0700 Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> > Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
> > per-node type") split a memcg's single obj_cgroup into one per NUMA
> > node so that reparenting LRU folios can take per-node lru locks. As a
> > side effect, the per-CPU obj_stock_pcp -- which caches a single
> > cached_objcg pointer -- thrashes on workloads where threads of the
> > same memcg run on different NUMA nodes. The kernel test robot reported
> > a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.
> >
> > Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg
> > pointer") landed as a temporary fix by treating sibling per-node
> > objcgs as equivalent for the cache lookup, intended to be reverted
> > once per-node kmem accounting is introduced. This series takes a more
> > general approach: cache multiple objcgs per CPU using the multi-slot
> > pattern memcg_stock_pcp already uses, so the per-node objcg variants
> > of one memcg can all coexist in the stock without ever forcing a
> > drain. The temporary fix can then be reverted.
> >
> > To avoid increasing the per-CPU cache footprint, the first three
> > patches shrink the existing single-slot obj_stock_pcp fields.
> > The final patch converts cached_objcg and nr_bytes into
> > NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire
> > consume/refill/account hot path fits within a single 64-byte cache
> > line on non-debug 64-bit builds (verified with pahole).
>
> Thanks, I added this to mm.git's mm-new branch, along with a couple of
> possible todo notes from the review.
>
> Sashiko asked a thing:
> https://sashiko.dev/#/patchset/20260522011908.1669332-1-shakeel.butt@linux.dev
>
> Did you already see this? The footers there indicate that an email was
> sent out but I don't know if it works?
Yes, I saw that comment. It is kind of very specific to archs with 256KiB base
page sizes. Anyways, I have a simple fix for that and there were minor
suggestions from others for simple changes, I will send v3 with the requested
changes.
^ permalink raw reply
* Re: [PATCH v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: patchwork-bot+netdevbpf @ 2026-05-25 13:53 UTC (permalink / raw)
To: Qing Ming
Cc: tj, josef, axboe, hannes, mkoutny, mhocko, roman.gushchin,
shakeel.butt, muchun.song, akpm, ast, haoluo, yosry, cgroups,
linux-block, linux-kernel, linux-mm, bpf
In-Reply-To: <20260516070849.106141-1-a0yami@mailbox.org>
Hello:
This patch was applied to bpf/bpf.git (master)
by Tejun Heo <tj@kernel.org>:
On Sat, 16 May 2026 15:08:49 +0800 you wrote:
> css_rstat_updated() is exposed as a BPF kfunc and accepts a
> caller-provided cpu argument. The function uses cpu for per-cpu rstat
> lookups without checking whether it refers to a valid possible CPU.
>
> A BPF iter/cgroup program with CAP_BPF and CAP_PERFMON can pass an
> invalid cpu value. On an unfixed UBSCAN_BOUNDS test kernel, cpu ==
> 0x7fffffff triggers:
>
> [...]
Here is the summary with links:
- [v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
https://git.kernel.org/bpf/bpf/c/8817005efbdf
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply
* Re: [RFC PATCH rdma-next 0/5] cgroup/rdma: add per-type resource accounting for QP, MR and MR memory
From: Jason Gunthorpe @ 2026-05-25 13:43 UTC (permalink / raw)
To: Tao Cui; +Cc: tj, hannes, mkoutny, leon, linux-rdma, cgroups
In-Reply-To: <20260525055506.2002985-1-cuitao@kylinos.cn>
On Mon, May 25, 2026 at 01:55:01PM +0800, Tao Cui wrote:
> Currently the RDMA cgroup only tracks two aggregate counters:
> hca_handle and hca_object. This is too coarse for real-world
> deployment: a tenant can exhaust all HCA objects by creating nothing
> but QPs, while the administrator has no way to impose separate limits
> on QP count, MR count, or the cumulative memory registered through
> MRs.
This was a deliberate choice.
> - qp - Queue Pair count
> - mr - Memory Region count
> - mr_mem - Cumulative MR memory size in bytes
I would agree to mr_mem as a reasonable extension, but not splitting
out objects to finer grains. There are endless objects we don't want a
100 different cgroup knobs, it is not usable.
Jason
^ permalink raw reply
* [PATCH v2 4/4] selftests/cgroup: Add tests for zswap proactive writeback
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Add test_zswap_proactive_writeback() to cover the new memory.reclaim
"zswap_writeback_only" key. The test populates a memory cgroup zswap
pool, triggers proactive writeback, and verifies the behavior by
observing the change in zswpwb_proactive. Invalid input combinations
are also covered.
Extend test_zswap_writeback_one() to assert that the existing
non-proactive writeback path leaves zswpwb_proactive at zero.
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
tools/testing/selftests/cgroup/test_zswap.c | 161 +++++++++++++++++++-
1 file changed, 153 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/cgroup/test_zswap.c b/tools/testing/selftests/cgroup/test_zswap.c
index a7bdcdd09d62..b80ed13bc5e2 100644
--- a/tools/testing/selftests/cgroup/test_zswap.c
+++ b/tools/testing/selftests/cgroup/test_zswap.c
@@ -57,6 +57,11 @@ static long get_cg_wb_count(const char *cg)
return cg_read_key_long(cg, "memory.stat", "zswpwb");
}
+static long get_cg_pwb_count(const char *cg)
+{
+ return cg_read_key_long(cg, "memory.stat", "zswpwb_proactive");
+}
+
static long get_zswpout(const char *cgroup)
{
return cg_read_key_long(cgroup, "memory.stat", "zswpout ");
@@ -323,11 +328,17 @@ static int attempt_writeback(const char *cgroup, void *arg)
static int test_zswap_writeback_one(const char *cgroup, bool wb)
{
- long zswpwb_before, zswpwb_after;
+ long wb_cnt, pwb_cnt;
+
+ wb_cnt = get_cg_wb_count(cgroup);
+ if (wb_cnt != 0) {
+ ksft_print_msg("zswpwb_before = %ld instead of 0\n", wb_cnt);
+ return -1;
+ }
- zswpwb_before = get_cg_wb_count(cgroup);
- if (zswpwb_before != 0) {
- ksft_print_msg("zswpwb_before = %ld instead of 0\n", zswpwb_before);
+ pwb_cnt = get_cg_pwb_count(cgroup);
+ if (pwb_cnt != 0) {
+ ksft_print_msg("zswpwb_proactive_before = %ld instead of 0\n", pwb_cnt);
return -1;
}
@@ -335,13 +346,24 @@ static int test_zswap_writeback_one(const char *cgroup, bool wb)
return -1;
/* Verify that zswap writeback occurred only if writeback was enabled */
- zswpwb_after = get_cg_wb_count(cgroup);
- if (zswpwb_after < 0)
+ wb_cnt = get_cg_wb_count(cgroup);
+ if (wb_cnt < 0)
return -1;
- if (wb != !!zswpwb_after) {
+ if (wb != !!wb_cnt) {
ksft_print_msg("zswpwb_after is %ld while wb is %s\n",
- zswpwb_after, wb ? "enabled" : "disabled");
+ wb_cnt, wb ? "enabled" : "disabled");
+ return -1;
+ }
+
+ /*
+ * attempt_writeback() does not use the proactive writeback path, so
+ * zswpwb_proactive must stay at zero regardless of whether writeback
+ * was enabled.
+ */
+ pwb_cnt = get_cg_pwb_count(cgroup);
+ if (pwb_cnt != 0) {
+ ksft_print_msg("zswpwb_proactive_after is %ld, expected 0\n", pwb_cnt);
return -1;
}
@@ -709,6 +731,128 @@ static int test_zswap_incompressible(const char *root)
return ret;
}
+/*
+ * Trigger proactive zswap writeback with the following steps:
+ * 1. Allocate memory.
+ * 2. Push allocated memory into zswap.
+ * 3. Proactively write back zswap pages to swap
+ * using "zswap_writeback_only".
+ */
+static int proactive_writeback_workload(const char *cgroup, void *arg)
+{
+ long pagesize = sysconf(_SC_PAGESIZE);
+ size_t memsize = MB(4);
+ char reclaim_cmd[64];
+ char buf[pagesize];
+ int ret = -1;
+ char *mem;
+
+ mem = (char *)malloc(memsize);
+ if (!mem)
+ return ret;
+
+ for (int i = 0; i < pagesize; i++)
+ buf[i] = i < pagesize / 2 ? (char)i : 0;
+ for (int i = 0; i < memsize; i += pagesize)
+ memcpy(&mem[i], buf, pagesize);
+
+ /* Evict allocated memory into zswap. */
+ if (cg_write_numeric(cgroup, "memory.reclaim", memsize)) {
+ ksft_print_msg("Failed to push pages into zswap\n");
+ goto out;
+ }
+
+ /* Trigger proactive zswap writeback for the same amount. */
+ snprintf(reclaim_cmd, sizeof(reclaim_cmd), "%zu zswap_writeback_only", memsize);
+ if (cg_write(cgroup, "memory.reclaim", reclaim_cmd)) {
+ ksft_print_msg("memory.reclaim zswap_writeback_only failed\n");
+ goto out;
+ }
+
+ ret = 0;
+out:
+ free(mem);
+ return ret;
+}
+
+static int check_writeback_invalid_inputs(const char *cgroup)
+{
+ static char * const bad_inputs[] = {
+ "zswap_writeback_only",
+ "1M zswap_writeback_only swappiness=60",
+ "1M swappiness=60 zswap_writeback_only",
+ "1M zswap_writeback_only swappiness=max",
+ "1M swappiness=max zswap_writeback_only",
+ };
+ int i, rc;
+
+ for (i = 0; i < ARRAY_SIZE(bad_inputs); i++) {
+ rc = cg_write(cgroup, "memory.reclaim", bad_inputs[i]);
+ if (rc != -EINVAL) {
+ ksft_print_msg("memory.reclaim '%s': returned %d, expected %d\n",
+ bad_inputs[i], rc, -EINVAL);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static int test_zswap_proactive_writeback(const char *root)
+{
+ long pwb_before, wb_before, pwb_after, wb_after;
+ long pwb_delta, wb_delta;
+ int ret = KSFT_FAIL;
+ char *test_group;
+
+ if (cg_read_strcmp(root, "memory.zswap.writeback", "1"))
+ return KSFT_SKIP;
+
+ test_group = cg_name(root, "zswap_proactive_test");
+ if (!test_group)
+ return KSFT_FAIL;
+ if (cg_create(test_group))
+ goto out;
+ if (check_writeback_invalid_inputs(test_group))
+ goto out;
+
+ pwb_before = get_cg_pwb_count(test_group);
+ wb_before = get_cg_wb_count(test_group);
+ if (pwb_before < 0 || wb_before < 0)
+ goto out;
+
+ if (cg_run(test_group, proactive_writeback_workload, NULL))
+ goto out;
+
+ pwb_after = get_cg_pwb_count(test_group);
+ wb_after = get_cg_wb_count(test_group);
+ if (pwb_after < 0 || wb_after < 0)
+ goto out;
+
+ pwb_delta = pwb_after - pwb_before;
+ wb_delta = wb_after - wb_before;
+
+ if (pwb_delta <= 0) {
+ ksft_print_msg("zswpwb_proactive did not increase: delta=%ld\n",
+ pwb_delta);
+ goto out;
+ }
+ if (wb_delta <= 0) {
+ ksft_print_msg("zswpwb did not increase: delta=%ld\n", wb_delta);
+ goto out;
+ }
+ if (pwb_delta > wb_delta) {
+ ksft_print_msg("zswpwb_proactive delta (%ld) > zswpwb delta (%ld)\n",
+ pwb_delta, wb_delta);
+ goto out;
+ }
+
+ ret = KSFT_PASS;
+out:
+ cg_destroy(test_group);
+ free(test_group);
+ return ret;
+}
+
#define T(x) { x, #x }
struct zswap_test {
int (*fn)(const char *root);
@@ -722,6 +866,7 @@ struct zswap_test {
T(test_no_kmem_bypass),
T(test_no_invasive_cgroup_shrink),
T(test_zswap_incompressible),
+ T(test_zswap_proactive_writeback),
};
#undef T
--
2.34.1
^ permalink raw reply related
* [PATCH v2 3/4] mm/zswap: Add per-memcg stat for proactive writeback
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Currently, zswap writeback can be triggered by either the pool limit
being hit or by the proactive writeback mechanism. However, the
existing 'zswpwb' metric in memory.stat and /proc/vmstat counts all
written back pages, making it difficult to distinguish between pages
written back due to the pool limit and those written back proactively.
Add a new statistic 'zswpwb_proactive' to memory.stat and /proc/vmstat.
This counter tracks the number of pages written back due to proactive
writeback. This allows users to better monitor and tune the proactive
writeback mechanism.
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 +++
include/linux/vm_event_item.h | 1 +
mm/memcontrol.c | 1 +
mm/vmstat.c | 1 +
mm/zswap.c | 41 ++++++++++++++++++-------
5 files changed, 37 insertions(+), 11 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6564abf0dec5..7d65aef83f7b 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1748,6 +1748,10 @@ The following nested keys are defined.
zswpwb
Number of pages written from zswap to swap.
+ zswpwb_proactive
+ Number of pages written from zswap to swap by proactive
+ writeback. This is a subset of zswpwb.
+
zswap_incomp
Number of incompressible pages currently stored in zswap
without compression. These pages could not be compressed to
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 03fe95f5a020..7a5bee0a20b6 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -138,6 +138,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
ZSWPIN,
ZSWPOUT,
ZSWPWB,
+ ZSWPWB_PROACTIVE,
#endif
#ifdef CONFIG_X86
DIRECT_MAP_LEVEL2_SPLIT,
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 409c41359dc8..67de71b2a659 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -571,6 +571,7 @@ static const unsigned int memcg_vm_event_stat[] = {
ZSWPIN,
ZSWPOUT,
ZSWPWB,
+ ZSWPWB_PROACTIVE,
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
THP_FAULT_ALLOC,
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..66fd06d1bb01 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1452,6 +1452,7 @@ const char * const vmstat_text[] = {
[I(ZSWPIN)] = "zswpin",
[I(ZSWPOUT)] = "zswpout",
[I(ZSWPWB)] = "zswpwb",
+ [I(ZSWPWB_PROACTIVE)] = "zswpwb_proactive",
#endif
#ifdef CONFIG_X86
[I(DIRECT_MAP_LEVEL2_SPLIT)] = "direct_map_level2_splits",
diff --git a/mm/zswap.c b/mm/zswap.c
index 947507b9a185..78190631e2c4 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -160,6 +160,11 @@ struct zswap_pool {
char tfm_name[CRYPTO_MAX_ALG_NAME];
};
+struct zswap_shrink_walk_arg {
+ bool proactive;
+ bool encountered_page_in_swapcache;
+};
+
/* Global LRU lists shared by all zswap pools. */
static struct list_lru zswap_list_lru;
@@ -1042,7 +1047,8 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
* freed.
*/
static int zswap_writeback_entry(struct zswap_entry *entry,
- swp_entry_t swpentry)
+ swp_entry_t swpentry,
+ bool proactive)
{
struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
@@ -1102,6 +1108,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (entry->objcg)
count_objcg_events(entry->objcg, ZSWPWB, 1);
+ if (proactive) {
+ count_vm_event(ZSWPWB_PROACTIVE);
+ if (entry->objcg)
+ count_objcg_events(entry->objcg, ZSWPWB_PROACTIVE, 1);
+ }
+
zswap_entry_free(entry);
/* folio is up to date */
@@ -1151,7 +1163,8 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
void *arg)
{
struct zswap_entry *entry = container_of(item, struct zswap_entry, lru);
- bool *encountered_page_in_swapcache = (bool *)arg;
+ struct zswap_shrink_walk_arg *walk_arg = arg;
+ bool proactive_wb = walk_arg && walk_arg->proactive;
swp_entry_t swpentry;
enum lru_status ret = LRU_REMOVED_RETRY;
int writeback_result;
@@ -1206,7 +1219,7 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
*/
spin_unlock(&l->lock);
- writeback_result = zswap_writeback_entry(entry, swpentry);
+ writeback_result = zswap_writeback_entry(entry, swpentry, proactive_wb);
if (writeback_result) {
zswap_reject_reclaim_fail++;
@@ -1217,9 +1230,9 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
* into the warmer region. We should terminate shrinking (if we're in the dynamic
* shrinker context).
*/
- if (writeback_result == -EEXIST && encountered_page_in_swapcache) {
+ if (writeback_result == -EEXIST && walk_arg) {
ret = LRU_STOP;
- *encountered_page_in_swapcache = true;
+ walk_arg->encountered_page_in_swapcache = true;
}
} else {
zswap_written_back_pages++;
@@ -1231,8 +1244,11 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
static unsigned long zswap_shrinker_scan(struct shrinker *shrinker,
struct shrink_control *sc)
{
+ struct zswap_shrink_walk_arg walk_arg = {
+ .proactive = false,
+ .encountered_page_in_swapcache = false,
+ };
unsigned long shrink_ret;
- bool encountered_page_in_swapcache = false;
if (!zswap_shrinker_enabled ||
!mem_cgroup_zswap_writeback_enabled(sc->memcg)) {
@@ -1241,9 +1257,9 @@ static unsigned long zswap_shrinker_scan(struct shrinker *shrinker,
}
shrink_ret = list_lru_shrink_walk(&zswap_list_lru, sc, &shrink_memcg_cb,
- &encountered_page_in_swapcache);
+ &walk_arg);
- if (encountered_page_in_swapcache)
+ if (walk_arg.encountered_page_in_swapcache)
return SHRINK_STOP;
return shrink_ret ? shrink_ret : SHRINK_STOP;
@@ -1714,7 +1730,10 @@ static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
return -ENOENT;
for_each_node_state(nid, N_NORMAL_MEMORY) {
- bool encountered_page_in_swapcache = false;
+ struct zswap_shrink_walk_arg walk_arg = {
+ .proactive = true,
+ .encountered_page_in_swapcache = false,
+ };
unsigned long nr_to_scan, nr_scanned = 0;
/*
@@ -1748,12 +1767,12 @@ static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
nr_written += list_lru_walk_one(&zswap_list_lru, nid, memcg,
&shrink_memcg_cb,
- &encountered_page_in_swapcache,
+ &walk_arg,
&nr_to_walk);
if (nr_written >= nr_to_write)
return nr_written;
- if (encountered_page_in_swapcache)
+ if (walk_arg.encountered_page_in_swapcache)
break;
cond_resched();
--
2.34.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox