* Re: [PATCH] memcg: use round-robin victim selection in refill_stock
From: Michal Hocko @ 2026-05-25 9:07 UTC (permalink / raw)
To: Shakeel Butt
Cc: Andrew Morton, Johannes Weiner, Roman Gushchin, Muchun Song,
Harry Yoo, Meta kernel team, linux-mm, cgroups, linux-kernel
In-Reply-To: <20260521223751.3794625-1-shakeel.butt@linux.dev>
On Thu 21-05-26 15:37:51, Shakeel Butt wrote:
> Harry Yoo reported that get_random_u32_below() is not safe to call in
> the nmi context and memcg charge draining can happen in nmi context.
>
> More specifically get_random_u32_below() is neither reentrant- nor
> NMI-safe: it acquires a per-cpu local_lock via local_lock_irqsave() on
> the batched_entropy_u32 state. An NMI that lands on a CPU mid-update of
> the ChaCha batch state and recurses into the random subsystem would
> corrupt that state. The memcg_stock local_trylock prevents re-entry
> on the percpu stock itself, but cannot protect an unrelated
> subsystem's per-cpu lock.
>
> Replace the random pick with a per-cpu round-robin counter stored in
> memcg_stock_pcp and serialized by the same local_trylock that already
> guards cached[] and nr_pages[]. No atomics, no random calls, no extra
> locks needed.
>
> Fixes: f735eebe55f8f ("memcg: multi-memcg percpu charge cache")
> Reported-by: Harry Yoo <harry@kernel.org>
> Closes: https://lore.kernel.org/4e20f643-6983-4b6e-b12d-c6c4eb20ae0c@kernel.org/
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!
> ---
> mm/memcontrol.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 0eb50e639f0a..6392a2704441 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2031,6 +2031,7 @@ struct memcg_stock_pcp {
>
> struct work_struct work;
> unsigned long flags;
> + uint8_t drain_idx;
> };
>
> static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
> @@ -2214,7 +2215,9 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
> if (!success) {
> i = empty_slot;
> if (i == -1) {
> - i = get_random_u32_below(NR_MEMCG_STOCK);
> + i = stock->drain_idx++;
> + if (stock->drain_idx == NR_MEMCG_STOCK)
> + stock->drain_idx = 0;
> drain_stock(stock, i);
> }
> css_get(&memcg->css);
> --
> 2.53.0-Meta
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [PATCH RFC] memcg: add per-cgroup dirty page controls (dirty_ratio, dirty_min)
From: Jan Kara @ 2026-05-25 12:20 UTC (permalink / raw)
To: Alireza Haghdoost
Cc: Jan Kara, Johannes Weiner, Michal Hocko, Roman Gushchin,
Shakeel Butt, Muchun Song, Andrew Morton, Matthew Wilcox (Oracle),
cgroups, linux-mm, linux-kernel, linux-fsdevel, Kshitij Doshi
In-Reply-To: <CA+w=M=RkewNB0Fc=KNXTrOnt-YABFy+7ZALLmp1wQOP5k_=13Q@mail.gmail.com>
Thanks for the details and sorry for a delayed reply. I had two conferences
lately...
On Wed 13-05-26 21:10:10, Alireza Haghdoost wrote:
> On Wed 06-05-26 16:21:00, Jan Kara wrote:
> > Things like motivation actually belong to the changelog itself, measured
> > results how the patch helps as well. On the other hand stuff like history
> > is largely irrelevant here, frankly I don't have a bandwidth to carefully
> > read the huge amount of text LLM has generated below so please try to make
> > it more concise for next time.
>
> Understood. Will trim for the non-RFC posting; apologies for the
> volume.
>
> > ... I quite don't see how a multisecond stalls you are describing would
> > happen [...] If we are below freerun in the memcg, the task dirtying
> > folios from that memcg shouldn't be throttled at all, once we get above
> > freerun we throttle by maximum of throttling delay decided from global
> > and memcg situation [...]
>
> The stall is reachable even with the victim's memcg well below its
> own freerun. The freerun shortcut in balance_dirty_pages() is an AND,
> not OR:
>
> if (gdtc->freerun && (!mdtc || mdtc->freerun))
> goto free_running;
True but this is mostly a performance optimization.
> Once gdtc is over freerun (because the noisy neighbour pushed it
> there) the shortcut does not fire, even when mdtc->freerun is true.
Below in balance_dirty_pages() we also have:
/*
* Calculate global domain's pos_ratio and select the
* global dtc by default.
*/
balance_wb_limits(gdtc, strictlimit);
if (gdtc->freerun)
goto free_running;
sdtc = gdtc;
if (mdtc) {
/*
* If memcg domain is in effect, calculate its
* pos_ratio. @wb should satisfy constraints from
* both global and memcg domains. Choose the one
* w/ lower pos_ratio.
*/
balance_wb_limits(mdtc, strictlimit);
if (mdtc->freerun)
goto free_running;
which is the key logic. So unless you have strictlimit enabled (which you
didn't mention you'd have), being under freerun limit in your memcg is
enough to protect you from a noisy neighbor.
> After the shortcut fails, the per-task pause is computed from the
> dtc with the smaller pos_ratio:
>
> if (mdtc->pos_ratio < gdtc->pos_ratio)
> sdtc = mdtc;
>
> When global is the worse domain, the victim sleeps against global
> state, not memcg state.
*If* you are above freerun in your memcg (or have strictlimit enabled) then
yes, I agree.
> > So can you perhaps share more details about the configuration where
> > you observe these delays to innocent tasks due to another task
> > dirtying a lot of memory? How many page cache in total and dirty
> > pages are there in each memcg [...]? Is the delayed task really
> > throttled in balance_dirty_pages()?
>
> Yes. Re-ran the reproducer: stock 7.0-rc5, ext4 on virtio-blk
> throttled to 256 KB/s, dirty_bytes=32M, dirty_background_bytes=16M
> (freerun = 24 MB), noisy = single fio job doing unlimited buffered
> randwrite, victim = single fio job doing 4 KiB sequential write
> rate-limited to 500 KB/s.
>
> Per-memcg snapshot during the contended phase, ~10 s into the run:
>
> noisy memcg victim memcg global
> memory.current 47 MB 21 MB --
> file (cache) 38 MB 14 MB --
> file_dirty 26 MB 1.7 MB 27 MB
> file_writeback 1.5 MB 4.0 MB 5.3 MB
>
> Victim memcg holds 1.7 MB dirty, far below any reasonable per-memcg
> freerun. Global dirty (NR_FILE_DIRTY + NR_WRITEBACK ~ 32 MB) is over
> the 24 MB freerun ceiling, driven entirely by noisy.
Ah, OK. I think I see what's going on. How much page cache does the machine
have in total and what are memory limits for the noisy and victim memcgs?
Because there's this somewhat surprising behavior when you configure dirty
limits in bytes in domain_dirty_limits() - the memcg dirty limit will
roughly be dirty_bytes / global_available_memory * memcg_available (where
memcg_available is memcg page cache size + how much memcg can grow from the
current size until it hits memory limit). Since you set dirty_bytes to 32M,
your machine presumably has gigabytes of memory, then it's possible victim
memcg dirty limits end up really low.
> The victim writer (fio with psync) is in fact sleeping in
> balance_dirty_pages(). One stack snapshot during a stall:
>
> [<0>] balance_dirty_pages+0x5c5/0xac0
> [<0>] balance_dirty_pages_ratelimited_flags+0x2a1/0x380
> [<0>] generic_perform_write+0x194/0x280
> [<0>] ext4_buffered_write_iter+0x63/0x110
> [<0>] vfs_write+0x28d/0x450
> [<0>] __x64_sys_pwrite64+0x8c/0xc0
> [<0>] do_syscall_64+0xfa/0x520
> [<0>] entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Sampling /proc/<pid>/wchan at 100 Hz across the contended phase
> yields the histogram:
>
> 104 balance_dirty_pages
> 88 hrtimer_nanosleep (the fio rate-limit sleep between writes)
> 12 RUNNING
> 4 p9_client_rpc (virtfs, host-guest filesystem RPC)
> 3 d_alloc_parallel
>
> The vast majority of non-rate-limit samples have the writer parked
> in balance_dirty_pages(). Victim per-IO clat in this run reaches a
> 3 s tail (worst single 4 KiB pwrite blocked ~3.0 s) while its own
> memcg holds < 2 MB dirty.
If dirty limits for the victim memcg end up really low, then yes, this is
what I'd expect.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply
* [PATCH v2 0/4] mm/zswap: Implement per-cgroup proactive writeback
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
From: Hao Jia <jiahao1@lixiang.com>
Zswap currently writes back pages to backing swap reactively, triggered
either by the shrinker or by the pool reaching its size limit. Although
proactive memory reclaim can automatically write back a portion of zswap
pages via the shrinker, it cannot explicitly control the amount of
writeback for a specific memory cgroup. Moreover, proactive memory reclaim
may not always be triggered during a steady state.
In certain scenarios, it is desirable to trigger writeback in advance to
free up memory. For example, users may want to prepare for an upcoming
memory-intensive workload by flushing cold memory to the backing storage
when the system is relatively idle.
This patch series introduces a "zswap_writeback_only" key to memory.reclaim
cgroup interface, allowing users to proactively write back cold compressed
pages from zswap to the backing swap device. When specified, this key
bypasses standard memory reclaim and exclusively performs proactive zswap
writeback up to the requested budget. If omitted, the default reclaim
behavior remains unchanged.
Example usage:
# Write back 100MB of pages from zswap to the backing swap
echo "100M zswap_writeback_only" > memory.reclaim
Patch 1: Move the global zswap shrink cursor into struct mem_cgroup as a
per-memcg zswap_wb_iter, so patch 2 can scope writeback to a given memcg
and make forward progress across its subtree on repeated invocations.
Patch 2: Extend the memory.reclaim cgroup v2 interface with a new
"zswap_writeback_only" key, allowing users to trigger proactive zswap
writeback up to a requested budget.
Patch 3: Add a zswpwb_proactive counter to memory.stat and /proc/vmstat
to track the number of writebacks triggered by proactive writeback.
Patch 4: Add tests for zswap proactive writeback.
v1->v2:
- As suggested by Yosry and Nhat, extend the memory.reclaim cgroup v2
interface with a "zswap_writeback_only" key instead of adding a new
dedicated cgroup interface.
- Update the zswap documentation and add selftests for proactive writeback.
[v1] https://lore.kernel.org/all/20260511105149.75584-1-jiahao.kernel@gmail.com
Hao Jia (4):
mm/zswap: Make shrink_worker writeback cursor per-memcg
mm/zswap: Implement proactive writeback
mm/zswap: Add per-memcg stat for proactive writeback
selftests/cgroup: Add tests for zswap proactive writeback
Documentation/admin-guide/cgroup-v2.rst | 22 +-
Documentation/admin-guide/mm/zswap.rst | 11 +-
include/linux/memcontrol.h | 3 +
include/linux/vm_event_item.h | 1 +
include/linux/zswap.h | 16 ++
mm/memcontrol.c | 4 +
mm/vmscan.c | 14 +
mm/vmstat.c | 1 +
mm/zswap.c | 292 +++++++++++++++++---
tools/testing/selftests/cgroup/test_zswap.c | 161 ++++++++++-
10 files changed, 470 insertions(+), 55 deletions(-)
--
2.34.1
^ permalink raw reply
* [PATCH v2 1/4] mm/zswap: Make shrink_worker writeback cursor per-memcg
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
The zswap background writeback worker shrink_worker() uses a global
cursor zswap_next_shrink, protected by zswap_shrink_lock, to round-robin
across the online memcgs under root_mem_cgroup.
Proactive writeback also wants a similar per-memcg cursor that is
scoped to the specified memcg, so that repeated invocations against
the same memcg make forward progress across its descendant memcgs
instead of restarting from the first child memcg each time.
Naturally, group the cursor and its protecting spinlock into a
zswap_wb_iter struct, and make it a member of struct mem_cgroup to
realize per-memcg cursor management. Accordingly, shrink_worker() now
uses the lock and cursor in root_mem_cgroup->zswap_wb_iter.
Because the cursor is now per-memcg, the offline cleanup must visit
every ancestor that could be holding a reference to the dying memcg.
Factor out __zswap_memcg_offline_cleanup() and walk from dead_memcg up
to the root.
No functional change intended for shrink_worker().
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
include/linux/memcontrol.h | 3 +
include/linux/zswap.h | 9 +++
mm/memcontrol.c | 3 +
mm/zswap.c | 119 ++++++++++++++++++++++++++-----------
4 files changed, 98 insertions(+), 36 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dc3fa687759b..b8323c8d6565 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -228,6 +228,9 @@ struct mem_cgroup {
* swap, and from being swapped out on zswap store failures.
*/
bool zswap_writeback;
+
+ /* Per-memcg writeback cursor */
+ struct zswap_wb_iter zswap_wb_iter;
#endif
/* vmpressure notifications */
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 30c193a1207e..efa6b551217e 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -11,6 +11,15 @@ extern atomic_long_t zswap_stored_pages;
#ifdef CONFIG_ZSWAP
+/* Iteration cursor for zswap writeback over a memcg's subtree. */
+struct zswap_wb_iter {
+ /* protects @pos against concurrent advances */
+ spinlock_t lock;
+ struct mem_cgroup *pos;
+};
+
+void zswap_wb_iter_init(struct zswap_wb_iter *iter);
+
struct zswap_lruvec_state {
/*
* Number of swapped in pages from disk, i.e not found in the zswap pool.
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d466..409c41359dc8 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4022,6 +4022,9 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)
INIT_LIST_HEAD(&memcg->memory_peaks);
INIT_LIST_HEAD(&memcg->swap_peaks);
spin_lock_init(&memcg->peaks_lock);
+#ifdef CONFIG_ZSWAP
+ zswap_wb_iter_init(&memcg->zswap_wb_iter);
+#endif
memcg->socket_pressure = get_jiffies_64();
#if BITS_PER_LONG < 64
seqlock_init(&memcg->socket_pressure_seqlock);
diff --git a/mm/zswap.c b/mm/zswap.c
index 4b5149173b0e..6519f646b496 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -163,9 +163,6 @@ struct zswap_pool {
/* Global LRU lists shared by all zswap pools. */
static struct list_lru zswap_list_lru;
-/* The lock protects zswap_next_shrink updates. */
-static DEFINE_SPINLOCK(zswap_shrink_lock);
-static struct mem_cgroup *zswap_next_shrink;
static struct work_struct zswap_shrink_work;
static struct shrinker *zswap_shrinker;
@@ -717,28 +714,88 @@ void zswap_folio_swapin(struct folio *folio)
}
}
-/*
- * This function should be called when a memcg is being offlined.
+void zswap_wb_iter_init(struct zswap_wb_iter *iter)
+{
+ spin_lock_init(&iter->lock);
+}
+
+#ifdef CONFIG_MEMCG
+/**
+ * zswap_mem_cgroup_iter - advance the writeback cursor
+ * @root: subtree root whose cursor to advance
+ *
+ * Advance @root->zswap_wb_iter.pos to @root itself or the next online
+ * descendant. Passing root_mem_cgroup yields a global walk.
*
- * Since the global shrinker shrink_worker() may hold a reference
- * of the memcg, we must check and release the reference in
- * zswap_next_shrink.
+ * The cursor is retained across invocations, so successive calls walk
+ * @root's subtree cyclically in pre-order and, after %NULL, restart
+ * from the beginning.
*
- * shrink_worker() must handle the case where this function releases
- * the reference of memcg being shrunk.
+ * The returned memcg carries an extra reference; release it with
+ * mem_cgroup_put().
+ *
+ * Return: the next online memcg in @root's subtree, or @root itself,
+ * with an extra reference, or %NULL after a full round-trip.
*/
-void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg)
+static struct mem_cgroup *zswap_mem_cgroup_iter(struct mem_cgroup *root)
{
- /* lock out zswap shrinker walking memcg tree */
- spin_lock(&zswap_shrink_lock);
- if (zswap_next_shrink == memcg) {
+ struct mem_cgroup *memcg;
+
+ if (mem_cgroup_disabled())
+ return NULL;
+
+ spin_lock(&root->zswap_wb_iter.lock);
+ do {
+ memcg = mem_cgroup_iter(root, root->zswap_wb_iter.pos, NULL);
+ root->zswap_wb_iter.pos = memcg;
+ } while (memcg && !mem_cgroup_tryget_online(memcg));
+ spin_unlock(&root->zswap_wb_iter.lock);
+
+ return memcg;
+}
+
+/*
+ * If @root's cursor currently points at @dead_memcg, advance it to the
+ * next online descendant so @dead_memcg can be freed.
+ */
+static void __zswap_memcg_offline_cleanup(struct mem_cgroup *root,
+ struct mem_cgroup *dead_memcg)
+{
+ spin_lock(&root->zswap_wb_iter.lock);
+ if (root->zswap_wb_iter.pos == dead_memcg) {
do {
- zswap_next_shrink = mem_cgroup_iter(NULL, zswap_next_shrink, NULL);
- } while (zswap_next_shrink && !mem_cgroup_online(zswap_next_shrink));
+ root->zswap_wb_iter.pos =
+ mem_cgroup_iter(root,
+ root->zswap_wb_iter.pos, NULL);
+ } while (root->zswap_wb_iter.pos &&
+ !mem_cgroup_online(root->zswap_wb_iter.pos));
}
- spin_unlock(&zswap_shrink_lock);
+ spin_unlock(&root->zswap_wb_iter.lock);
+}
+
+/*
+ * Called when a memcg is being offlined. If @memcg or any of its
+ * ancestors has a cursor pointing at @memcg, it must be advanced
+ * past @memcg before @memcg can be freed. Walk the chain and
+ * release such references.
+ */
+void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg)
+{
+ struct mem_cgroup *parent = memcg;
+
+ do {
+ __zswap_memcg_offline_cleanup(parent, memcg);
+ } while ((parent = parent_mem_cgroup(parent)));
+}
+#else /* !CONFIG_MEMCG */
+static struct mem_cgroup *zswap_mem_cgroup_iter(struct mem_cgroup *root)
+{
+ return NULL;
}
+void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) { }
+#endif /* CONFIG_MEMCG */
+
/*********************************
* zswap entry functions
**********************************/
@@ -1328,38 +1385,28 @@ static void shrink_worker(struct work_struct *w)
* - No writeback-candidate memcgs found in a memcg tree walk.
* - Shrinking a writeback-candidate memcg failed.
*
- * We save iteration cursor memcg into zswap_next_shrink,
+ * We save the iteration cursor in root_mem_cgroup->zswap_wb_iter.pos,
* which can be modified by the offline memcg cleaner
* zswap_memcg_offline_cleanup().
*
* Since the offline cleaner is called only once, we cannot leave an
- * offline memcg reference in zswap_next_shrink.
+ * offline memcg reference in root_mem_cgroup->zswap_wb_iter.pos.
* We can rely on the cleaner only if we get online memcg under lock.
*
* If we get an offline memcg, we cannot determine if the cleaner has
* already been called or will be called later. We must put back the
* reference before returning from this function. Otherwise, the
- * offline memcg left in zswap_next_shrink will hold the reference
- * until the next run of shrink_worker().
+ * offline memcg left in root_mem_cgroup->zswap_wb_iter.pos will hold
+ * the reference until the next run of shrink_worker().
*/
do {
/*
- * Start shrinking from the next memcg after zswap_next_shrink.
- * When the offline cleaner has already advanced the cursor,
- * advancing the cursor here overlooks one memcg, but this
- * should be negligibly rare.
- *
- * If we get an online memcg, keep the extra reference in case
- * the original one obtained by mem_cgroup_iter() is dropped by
- * zswap_memcg_offline_cleanup() while we are shrinking the
- * memcg.
+ * Start shrinking from the next memcg after
+ * root_mem_cgroup->zswap_wb_iter.pos. When the offline cleaner
+ * has already advanced the cursor, advancing the cursor here
+ * overlooks one memcg, but this should be negligibly rare.
*/
- spin_lock(&zswap_shrink_lock);
- do {
- memcg = mem_cgroup_iter(NULL, zswap_next_shrink, NULL);
- zswap_next_shrink = memcg;
- } while (memcg && !mem_cgroup_tryget_online(memcg));
- spin_unlock(&zswap_shrink_lock);
+ memcg = zswap_mem_cgroup_iter(root_mem_cgroup);
if (!memcg) {
/*
--
2.34.1
^ permalink raw reply related
* [PATCH v2 2/4] mm/zswap: Implement proactive writeback
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Zswap currently writes back pages to backing swap reactively, triggered
either by the shrinker or when the pool reaches its size limit. There is
no mechanism to control the amount of writeback for a specific memory
cgroup. However, users may want to proactively write back zswap pages,
e.g., to free up memory for other applications or to prepare for
memory-intensive workloads.
Introduce a "zswap_writeback_only" key to the memory.reclaim cgroup
interface. When specified, this key bypasses standard memory reclaim
and exclusively performs proactive zswap writeback up to the requested
budget. If omitted, the default reclaim behavior remains unchanged.
Example usage:
# Write back 100MB of pages from zswap to the backing swap
echo "100M zswap_writeback_only" > memory.reclaim
Note that the actual amount written back may be less than requested due
to the zswap second-chance algorithm: referenced entries are rotated on
the LRU on the first encounter and only written back on a second pass.
The interface returns -EAGAIN if no pages were successfully written back.
Internally, extend user_proactive_reclaim() to parse the new
"zswap_writeback_only" token and invoke the dedicated handler. Add
zswap_proactive_writeback() to walk the target memcg subtree via the
per-memcg writeback cursor, draining per-node zswap LRUs through
list_lru_walk_one() with the shrink_memcg_cb() callback.
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Suggested-by: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
Documentation/admin-guide/cgroup-v2.rst | 18 +++-
Documentation/admin-guide/mm/zswap.rst | 11 +-
include/linux/zswap.h | 7 ++
mm/vmscan.c | 14 +++
mm/zswap.c | 138 ++++++++++++++++++++++++
5 files changed, 185 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed99..6564abf0dec5 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1425,9 +1425,10 @@ PAGE_SIZE multiple when read back.
The following nested keys are defined.
- ========== ================================
+ ==================== ==================================================
swappiness Swappiness value to reclaim with
- ========== ================================
+ zswap_writeback_only Only perform proactive zswap writeback
+ ==================== ==================================================
Specifying a swappiness value instructs the kernel to perform
the reclaim with that swappiness value. Note that this has the
@@ -1437,6 +1438,19 @@ The following nested keys are defined.
The valid range for swappiness is [0-200, max], setting
swappiness=max exclusively reclaims anonymous memory.
+ The zswap_writeback_only key skips ordinary memory reclaim and
+ writes back pages from zswap to the backing swap device until
+ the requested amount has been written or no further candidates
+ are found. This is useful to proactively offload cold pages from
+ the zswap pool to the swap device. It is only available if
+ zswap writeback is enabled. zswap_writeback_only cannot be combined
+ with swappiness; specifying both returns -EINVAL.
+
+ Example::
+
+ # Write back up to 100MB of pages from zswap to the backing swap
+ echo "100M zswap_writeback_only" > memory.reclaim
+
memory.peak
A read-write single value file which exists on non-root cgroups.
diff --git a/Documentation/admin-guide/mm/zswap.rst b/Documentation/admin-guide/mm/zswap.rst
index 2464425c783d..1c0598e77958 100644
--- a/Documentation/admin-guide/mm/zswap.rst
+++ b/Documentation/admin-guide/mm/zswap.rst
@@ -131,7 +131,16 @@ User can enable it as follows::
echo Y > /sys/module/zswap/parameters/shrinker_enabled
This can be enabled at the boot time if ``CONFIG_ZSWAP_SHRINKER_DEFAULT_ON`` is
-selected.
+selected. Once enabled, the shrinker automatically writes back zswap pages to
+backing swap during memory reclaim.
+
+If users want to explicitly trigger proactive zswap writeback for a specific
+memory cgroup without invoking standard page reclaim, it can be done as follows::
+
+ echo "100M zswap_writeback_only" > /sys/fs/cgroup/<cgroup-name>/memory.reclaim
+
+Both of the methods mentioned above are subject to the ``memory.zswap.writeback``
+control. This means that ``memory.zswap.writeback`` can reject all zswap writeback.
A debugfs interface is provided for various statistic about pool size, number
of pages stored, same-value filled pages and various counters for the reasons
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index efa6b551217e..98434d39339a 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -44,6 +44,7 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);
void zswap_folio_swapin(struct folio *folio);
bool zswap_is_enabled(void);
bool zswap_never_enabled(void);
+int zswap_proactive_writeback(struct mem_cgroup *memcg, unsigned long nr_to_writeback);
#else
struct zswap_lruvec_state {};
@@ -78,6 +79,12 @@ static inline bool zswap_never_enabled(void)
return true;
}
+static inline int zswap_proactive_writeback(struct mem_cgroup *memcg,
+ unsigned long nr_to_writeback)
+{
+ return -EOPNOTSUPP;
+}
+
#endif
#endif /* _LINUX_ZSWAP_H */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index bd1b1aa12581..6249176b9886 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -64,6 +64,7 @@
#include <linux/swapops.h>
#include <linux/sched/sysctl.h>
+#include <linux/zswap.h>
#include "internal.h"
#include "swap.h"
@@ -7894,11 +7895,13 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask,
enum {
MEMORY_RECLAIM_SWAPPINESS = 0,
MEMORY_RECLAIM_SWAPPINESS_MAX,
+ MEMORY_RECLAIM_ZSWAP_WRITEBACK_ONLY,
MEMORY_RECLAIM_NULL,
};
static const match_table_t tokens = {
{ MEMORY_RECLAIM_SWAPPINESS, "swappiness=%d"},
{ MEMORY_RECLAIM_SWAPPINESS_MAX, "swappiness=max"},
+ { MEMORY_RECLAIM_ZSWAP_WRITEBACK_ONLY, "zswap_writeback_only"},
{ MEMORY_RECLAIM_NULL, NULL },
};
@@ -7908,6 +7911,7 @@ int user_proactive_reclaim(char *buf,
unsigned int nr_retries = MAX_RECLAIM_RETRIES;
unsigned long nr_to_reclaim, nr_reclaimed = 0;
int swappiness = -1;
+ bool zswap_writeback_only = false;
char *old_buf, *start;
substring_t args[MAX_OPT_ARGS];
gfp_t gfp_mask = GFP_KERNEL;
@@ -7938,11 +7942,21 @@ int user_proactive_reclaim(char *buf,
case MEMORY_RECLAIM_SWAPPINESS_MAX:
swappiness = SWAPPINESS_ANON_ONLY;
break;
+ case MEMORY_RECLAIM_ZSWAP_WRITEBACK_ONLY:
+ zswap_writeback_only = true;
+ break;
default:
return -EINVAL;
}
}
+ if (zswap_writeback_only) {
+ /* zswap_writeback_only and swappiness are mutually exclusive. */
+ if (swappiness != -1)
+ return -EINVAL;
+ return zswap_proactive_writeback(memcg, nr_to_reclaim);
+ }
+
while (nr_reclaimed < nr_to_reclaim) {
/* Will converge on zero, but reclaim enforces a minimum */
unsigned long batch_size = (nr_to_reclaim - nr_reclaimed) / 4;
diff --git a/mm/zswap.c b/mm/zswap.c
index 6519f646b496..947507b9a185 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1684,6 +1684,144 @@ int zswap_load(struct folio *folio)
return 0;
}
+/*
+ * Maximum LRU scan limit:
+ * number of entries to scan per page of remaining budget.
+ */
+#define ZSWAP_PROACTIVE_WB_SCAN_RATIO 16UL
+/*
+ * Batch size for proactive writeback:
+ * - As the per-memcg writeback target in the outer memcg loop.
+ * - As the per-walk budget passed to list_lru_walk_one().
+ */
+#define ZSWAP_PROACTIVE_WB_BATCH 128UL
+
+/*
+ * Walk the per-node LRUs of @memcg to write back up to @nr_to_write pages.
+ * Returns the number of pages written back, or -ENOENT if @memcg is a
+ * zombie or has writeback disabled.
+ */
+static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
+ unsigned long nr_to_write)
+{
+ unsigned long nr_written = 0;
+ int nid;
+
+ if (!mem_cgroup_zswap_writeback_enabled(memcg))
+ return -ENOENT;
+
+ if (!mem_cgroup_online(memcg))
+ return -ENOENT;
+
+ for_each_node_state(nid, N_NORMAL_MEMORY) {
+ bool encountered_page_in_swapcache = false;
+ unsigned long nr_to_scan, nr_scanned = 0;
+
+ /*
+ * Cap by LRU length: bounds rewalks when referenced
+ * entries keep rotating to the tail.
+ */
+ nr_to_scan = list_lru_count_one(&zswap_list_lru, nid, memcg);
+ if (!nr_to_scan)
+ continue;
+
+ /*
+ * Cap by SCAN_RATIO * remaining budget: bounds scan cost
+ * to the remaining writeback budget.
+ */
+ nr_to_scan = min(nr_to_scan,
+ (nr_to_write - nr_written) * ZSWAP_PROACTIVE_WB_SCAN_RATIO);
+
+ while (nr_scanned < nr_to_scan) {
+ unsigned long nr_to_walk = min(ZSWAP_PROACTIVE_WB_BATCH,
+ nr_to_scan - nr_scanned);
+
+ if (signal_pending(current))
+ return nr_written;
+
+ /*
+ * Account for the committed budget rather than the walker's
+ * actual delta. If the list is emptied concurrently, the
+ * walker visits nothing and nr_scanned would never advance.
+ */
+ nr_scanned += nr_to_walk;
+
+ nr_written += list_lru_walk_one(&zswap_list_lru, nid, memcg,
+ &shrink_memcg_cb,
+ &encountered_page_in_swapcache,
+ &nr_to_walk);
+
+ if (nr_written >= nr_to_write)
+ return nr_written;
+ if (encountered_page_in_swapcache)
+ break;
+
+ cond_resched();
+ }
+ }
+
+ return nr_written;
+}
+
+int zswap_proactive_writeback(struct mem_cgroup *memcg,
+ unsigned long nr_to_writeback)
+{
+ struct mem_cgroup *iter_memcg;
+ unsigned long nr_written = 0;
+ int failures = 0, attempts = 0;
+
+ if (!memcg)
+ return -EINVAL;
+ if (!nr_to_writeback)
+ return 0;
+
+ /*
+ * Writeback will be aborted with -EAGAIN if @nr_written is still
+ * zero and we encounter the following MAX_RECLAIM_RETRIES times:
+ * - No writeback-candidate memcgs found in a subtree walk.
+ * - A writeback-candidate memcg wrote back zero pages.
+ */
+ while (nr_written < nr_to_writeback) {
+ unsigned long batch_size;
+ long shrunk;
+
+ if (signal_pending(current))
+ return -EINTR;
+
+ iter_memcg = zswap_mem_cgroup_iter(memcg);
+
+ if (!iter_memcg) {
+ /*
+ * Continue without incrementing failures if we found
+ * candidate memcgs in the last subtree walk.
+ */
+ if (!attempts && ++failures == MAX_RECLAIM_RETRIES)
+ goto out;
+ attempts = 0;
+ continue;
+ }
+
+ batch_size = min(nr_to_writeback - nr_written,
+ ZSWAP_PROACTIVE_WB_BATCH);
+ shrunk = zswap_proactive_shrink_memcg(iter_memcg, batch_size);
+ mem_cgroup_put(iter_memcg);
+
+ /* Writeback-disabled or offline: skip without counting. */
+ if (shrunk == -ENOENT)
+ continue;
+
+ ++attempts;
+ if (shrunk > 0)
+ nr_written += shrunk;
+ else if (++failures == MAX_RECLAIM_RETRIES)
+ goto out;
+
+ cond_resched();
+ }
+out:
+ return nr_written ? 0 : -EAGAIN;
+}
+
void zswap_invalidate(swp_entry_t swp)
{
pgoff_t offset = swp_offset(swp);
--
2.34.1
^ permalink raw reply related
* [PATCH v2 3/4] mm/zswap: Add per-memcg stat for proactive writeback
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Currently, zswap writeback can be triggered by either the pool limit
being hit or by the proactive writeback mechanism. However, the
existing 'zswpwb' metric in memory.stat and /proc/vmstat counts all
written back pages, making it difficult to distinguish between pages
written back due to the pool limit and those written back proactively.
Add a new statistic 'zswpwb_proactive' to memory.stat and /proc/vmstat.
This counter tracks the number of pages written back due to proactive
writeback. This allows users to better monitor and tune the proactive
writeback mechanism.
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 +++
include/linux/vm_event_item.h | 1 +
mm/memcontrol.c | 1 +
mm/vmstat.c | 1 +
mm/zswap.c | 41 ++++++++++++++++++-------
5 files changed, 37 insertions(+), 11 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6564abf0dec5..7d65aef83f7b 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1748,6 +1748,10 @@ The following nested keys are defined.
zswpwb
Number of pages written from zswap to swap.
+ zswpwb_proactive
+ Number of pages written from zswap to swap by proactive
+ writeback. This is a subset of zswpwb.
+
zswap_incomp
Number of incompressible pages currently stored in zswap
without compression. These pages could not be compressed to
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 03fe95f5a020..7a5bee0a20b6 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -138,6 +138,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
ZSWPIN,
ZSWPOUT,
ZSWPWB,
+ ZSWPWB_PROACTIVE,
#endif
#ifdef CONFIG_X86
DIRECT_MAP_LEVEL2_SPLIT,
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 409c41359dc8..67de71b2a659 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -571,6 +571,7 @@ static const unsigned int memcg_vm_event_stat[] = {
ZSWPIN,
ZSWPOUT,
ZSWPWB,
+ ZSWPWB_PROACTIVE,
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
THP_FAULT_ALLOC,
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..66fd06d1bb01 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1452,6 +1452,7 @@ const char * const vmstat_text[] = {
[I(ZSWPIN)] = "zswpin",
[I(ZSWPOUT)] = "zswpout",
[I(ZSWPWB)] = "zswpwb",
+ [I(ZSWPWB_PROACTIVE)] = "zswpwb_proactive",
#endif
#ifdef CONFIG_X86
[I(DIRECT_MAP_LEVEL2_SPLIT)] = "direct_map_level2_splits",
diff --git a/mm/zswap.c b/mm/zswap.c
index 947507b9a185..78190631e2c4 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -160,6 +160,11 @@ struct zswap_pool {
char tfm_name[CRYPTO_MAX_ALG_NAME];
};
+struct zswap_shrink_walk_arg {
+ bool proactive;
+ bool encountered_page_in_swapcache;
+};
+
/* Global LRU lists shared by all zswap pools. */
static struct list_lru zswap_list_lru;
@@ -1042,7 +1047,8 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
* freed.
*/
static int zswap_writeback_entry(struct zswap_entry *entry,
- swp_entry_t swpentry)
+ swp_entry_t swpentry,
+ bool proactive)
{
struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
@@ -1102,6 +1108,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (entry->objcg)
count_objcg_events(entry->objcg, ZSWPWB, 1);
+ if (proactive) {
+ count_vm_event(ZSWPWB_PROACTIVE);
+ if (entry->objcg)
+ count_objcg_events(entry->objcg, ZSWPWB_PROACTIVE, 1);
+ }
+
zswap_entry_free(entry);
/* folio is up to date */
@@ -1151,7 +1163,8 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
void *arg)
{
struct zswap_entry *entry = container_of(item, struct zswap_entry, lru);
- bool *encountered_page_in_swapcache = (bool *)arg;
+ struct zswap_shrink_walk_arg *walk_arg = arg;
+ bool proactive_wb = walk_arg && walk_arg->proactive;
swp_entry_t swpentry;
enum lru_status ret = LRU_REMOVED_RETRY;
int writeback_result;
@@ -1206,7 +1219,7 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
*/
spin_unlock(&l->lock);
- writeback_result = zswap_writeback_entry(entry, swpentry);
+ writeback_result = zswap_writeback_entry(entry, swpentry, proactive_wb);
if (writeback_result) {
zswap_reject_reclaim_fail++;
@@ -1217,9 +1230,9 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
* into the warmer region. We should terminate shrinking (if we're in the dynamic
* shrinker context).
*/
- if (writeback_result == -EEXIST && encountered_page_in_swapcache) {
+ if (writeback_result == -EEXIST && walk_arg) {
ret = LRU_STOP;
- *encountered_page_in_swapcache = true;
+ walk_arg->encountered_page_in_swapcache = true;
}
} else {
zswap_written_back_pages++;
@@ -1231,8 +1244,11 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
static unsigned long zswap_shrinker_scan(struct shrinker *shrinker,
struct shrink_control *sc)
{
+ struct zswap_shrink_walk_arg walk_arg = {
+ .proactive = false,
+ .encountered_page_in_swapcache = false,
+ };
unsigned long shrink_ret;
- bool encountered_page_in_swapcache = false;
if (!zswap_shrinker_enabled ||
!mem_cgroup_zswap_writeback_enabled(sc->memcg)) {
@@ -1241,9 +1257,9 @@ static unsigned long zswap_shrinker_scan(struct shrinker *shrinker,
}
shrink_ret = list_lru_shrink_walk(&zswap_list_lru, sc, &shrink_memcg_cb,
- &encountered_page_in_swapcache);
+ &walk_arg);
- if (encountered_page_in_swapcache)
+ if (walk_arg.encountered_page_in_swapcache)
return SHRINK_STOP;
return shrink_ret ? shrink_ret : SHRINK_STOP;
@@ -1714,7 +1730,10 @@ static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
return -ENOENT;
for_each_node_state(nid, N_NORMAL_MEMORY) {
- bool encountered_page_in_swapcache = false;
+ struct zswap_shrink_walk_arg walk_arg = {
+ .proactive = true,
+ .encountered_page_in_swapcache = false,
+ };
unsigned long nr_to_scan, nr_scanned = 0;
/*
@@ -1748,12 +1767,12 @@ static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
nr_written += list_lru_walk_one(&zswap_list_lru, nid, memcg,
&shrink_memcg_cb,
- &encountered_page_in_swapcache,
+ &walk_arg,
&nr_to_walk);
if (nr_written >= nr_to_write)
return nr_written;
- if (encountered_page_in_swapcache)
+ if (walk_arg.encountered_page_in_swapcache)
break;
cond_resched();
--
2.34.1
^ permalink raw reply related
* [PATCH v2 4/4] selftests/cgroup: Add tests for zswap proactive writeback
From: Hao Jia @ 2026-05-25 12:22 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Add test_zswap_proactive_writeback() to cover the new memory.reclaim
"zswap_writeback_only" key. The test populates a memory cgroup zswap
pool, triggers proactive writeback, and verifies the behavior by
observing the change in zswpwb_proactive. Invalid input combinations
are also covered.
Extend test_zswap_writeback_one() to assert that the existing
non-proactive writeback path leaves zswpwb_proactive at zero.
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
tools/testing/selftests/cgroup/test_zswap.c | 161 +++++++++++++++++++-
1 file changed, 153 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/cgroup/test_zswap.c b/tools/testing/selftests/cgroup/test_zswap.c
index a7bdcdd09d62..b80ed13bc5e2 100644
--- a/tools/testing/selftests/cgroup/test_zswap.c
+++ b/tools/testing/selftests/cgroup/test_zswap.c
@@ -57,6 +57,11 @@ static long get_cg_wb_count(const char *cg)
return cg_read_key_long(cg, "memory.stat", "zswpwb");
}
+static long get_cg_pwb_count(const char *cg)
+{
+ return cg_read_key_long(cg, "memory.stat", "zswpwb_proactive");
+}
+
static long get_zswpout(const char *cgroup)
{
return cg_read_key_long(cgroup, "memory.stat", "zswpout ");
@@ -323,11 +328,17 @@ static int attempt_writeback(const char *cgroup, void *arg)
static int test_zswap_writeback_one(const char *cgroup, bool wb)
{
- long zswpwb_before, zswpwb_after;
+ long wb_cnt, pwb_cnt;
+
+ wb_cnt = get_cg_wb_count(cgroup);
+ if (wb_cnt != 0) {
+ ksft_print_msg("zswpwb_before = %ld instead of 0\n", wb_cnt);
+ return -1;
+ }
- zswpwb_before = get_cg_wb_count(cgroup);
- if (zswpwb_before != 0) {
- ksft_print_msg("zswpwb_before = %ld instead of 0\n", zswpwb_before);
+ pwb_cnt = get_cg_pwb_count(cgroup);
+ if (pwb_cnt != 0) {
+ ksft_print_msg("zswpwb_proactive_before = %ld instead of 0\n", pwb_cnt);
return -1;
}
@@ -335,13 +346,24 @@ static int test_zswap_writeback_one(const char *cgroup, bool wb)
return -1;
/* Verify that zswap writeback occurred only if writeback was enabled */
- zswpwb_after = get_cg_wb_count(cgroup);
- if (zswpwb_after < 0)
+ wb_cnt = get_cg_wb_count(cgroup);
+ if (wb_cnt < 0)
return -1;
- if (wb != !!zswpwb_after) {
+ if (wb != !!wb_cnt) {
ksft_print_msg("zswpwb_after is %ld while wb is %s\n",
- zswpwb_after, wb ? "enabled" : "disabled");
+ wb_cnt, wb ? "enabled" : "disabled");
+ return -1;
+ }
+
+ /*
+ * attempt_writeback() does not use the proactive writeback path, so
+ * zswpwb_proactive must stay at zero regardless of whether writeback
+ * was enabled.
+ */
+ pwb_cnt = get_cg_pwb_count(cgroup);
+ if (pwb_cnt != 0) {
+ ksft_print_msg("zswpwb_proactive_after is %ld, expected 0\n", pwb_cnt);
return -1;
}
@@ -709,6 +731,128 @@ static int test_zswap_incompressible(const char *root)
return ret;
}
+/*
+ * Trigger proactive zswap writeback with the following steps:
+ * 1. Allocate memory.
+ * 2. Push allocated memory into zswap.
+ * 3. Proactively write back zswap pages to swap
+ * using "zswap_writeback_only".
+ */
+static int proactive_writeback_workload(const char *cgroup, void *arg)
+{
+ long pagesize = sysconf(_SC_PAGESIZE);
+ size_t memsize = MB(4);
+ char reclaim_cmd[64];
+ char buf[pagesize];
+ int ret = -1;
+ char *mem;
+
+ mem = (char *)malloc(memsize);
+ if (!mem)
+ return ret;
+
+ for (int i = 0; i < pagesize; i++)
+ buf[i] = i < pagesize / 2 ? (char)i : 0;
+ for (int i = 0; i < memsize; i += pagesize)
+ memcpy(&mem[i], buf, pagesize);
+
+ /* Evict allocated memory into zswap. */
+ if (cg_write_numeric(cgroup, "memory.reclaim", memsize)) {
+ ksft_print_msg("Failed to push pages into zswap\n");
+ goto out;
+ }
+
+ /* Trigger proactive zswap writeback for the same amount. */
+ snprintf(reclaim_cmd, sizeof(reclaim_cmd), "%zu zswap_writeback_only", memsize);
+ if (cg_write(cgroup, "memory.reclaim", reclaim_cmd)) {
+ ksft_print_msg("memory.reclaim zswap_writeback_only failed\n");
+ goto out;
+ }
+
+ ret = 0;
+out:
+ free(mem);
+ return ret;
+}
+
+static int check_writeback_invalid_inputs(const char *cgroup)
+{
+ static char * const bad_inputs[] = {
+ "zswap_writeback_only",
+ "1M zswap_writeback_only swappiness=60",
+ "1M swappiness=60 zswap_writeback_only",
+ "1M zswap_writeback_only swappiness=max",
+ "1M swappiness=max zswap_writeback_only",
+ };
+ int i, rc;
+
+ for (i = 0; i < ARRAY_SIZE(bad_inputs); i++) {
+ rc = cg_write(cgroup, "memory.reclaim", bad_inputs[i]);
+ if (rc != -EINVAL) {
+ ksft_print_msg("memory.reclaim '%s': returned %d, expected %d\n",
+ bad_inputs[i], rc, -EINVAL);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static int test_zswap_proactive_writeback(const char *root)
+{
+ long pwb_before, wb_before, pwb_after, wb_after;
+ long pwb_delta, wb_delta;
+ int ret = KSFT_FAIL;
+ char *test_group;
+
+ if (cg_read_strcmp(root, "memory.zswap.writeback", "1"))
+ return KSFT_SKIP;
+
+ test_group = cg_name(root, "zswap_proactive_test");
+ if (!test_group)
+ return KSFT_FAIL;
+ if (cg_create(test_group))
+ goto out;
+ if (check_writeback_invalid_inputs(test_group))
+ goto out;
+
+ pwb_before = get_cg_pwb_count(test_group);
+ wb_before = get_cg_wb_count(test_group);
+ if (pwb_before < 0 || wb_before < 0)
+ goto out;
+
+ if (cg_run(test_group, proactive_writeback_workload, NULL))
+ goto out;
+
+ pwb_after = get_cg_pwb_count(test_group);
+ wb_after = get_cg_wb_count(test_group);
+ if (pwb_after < 0 || wb_after < 0)
+ goto out;
+
+ pwb_delta = pwb_after - pwb_before;
+ wb_delta = wb_after - wb_before;
+
+ if (pwb_delta <= 0) {
+ ksft_print_msg("zswpwb_proactive did not increase: delta=%ld\n",
+ pwb_delta);
+ goto out;
+ }
+ if (wb_delta <= 0) {
+ ksft_print_msg("zswpwb did not increase: delta=%ld\n", wb_delta);
+ goto out;
+ }
+ if (pwb_delta > wb_delta) {
+ ksft_print_msg("zswpwb_proactive delta (%ld) > zswpwb delta (%ld)\n",
+ pwb_delta, wb_delta);
+ goto out;
+ }
+
+ ret = KSFT_PASS;
+out:
+ cg_destroy(test_group);
+ free(test_group);
+ return ret;
+}
+
#define T(x) { x, #x }
struct zswap_test {
int (*fn)(const char *root);
@@ -722,6 +866,7 @@ struct zswap_test {
T(test_no_kmem_bypass),
T(test_no_invasive_cgroup_shrink),
T(test_zswap_incompressible),
+ T(test_zswap_proactive_writeback),
};
#undef T
--
2.34.1
^ permalink raw reply related
* Re: [RFC PATCH rdma-next 0/5] cgroup/rdma: add per-type resource accounting for QP, MR and MR memory
From: Jason Gunthorpe @ 2026-05-25 13:43 UTC (permalink / raw)
To: Tao Cui; +Cc: tj, hannes, mkoutny, leon, linux-rdma, cgroups
In-Reply-To: <20260525055506.2002985-1-cuitao@kylinos.cn>
On Mon, May 25, 2026 at 01:55:01PM +0800, Tao Cui wrote:
> Currently the RDMA cgroup only tracks two aggregate counters:
> hca_handle and hca_object. This is too coarse for real-world
> deployment: a tenant can exhaust all HCA objects by creating nothing
> but QPs, while the administrator has no way to impose separate limits
> on QP count, MR count, or the cumulative memory registered through
> MRs.
This was a deliberate choice.
> - qp - Queue Pair count
> - mr - Memory Region count
> - mr_mem - Cumulative MR memory size in bytes
I would agree to mr_mem as a reasonable extension, but not splitting
out objects to finer grains. There are endless objects we don't want a
100 different cgroup knobs, it is not usable.
Jason
^ permalink raw reply
* Re: [PATCH v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: patchwork-bot+netdevbpf @ 2026-05-25 13:53 UTC (permalink / raw)
To: Qing Ming
Cc: tj, josef, axboe, hannes, mkoutny, mhocko, roman.gushchin,
shakeel.butt, muchun.song, akpm, ast, haoluo, yosry, cgroups,
linux-block, linux-kernel, linux-mm, bpf
In-Reply-To: <20260516070849.106141-1-a0yami@mailbox.org>
Hello:
This patch was applied to bpf/bpf.git (master)
by Tejun Heo <tj@kernel.org>:
On Sat, 16 May 2026 15:08:49 +0800 you wrote:
> css_rstat_updated() is exposed as a BPF kfunc and accepts a
> caller-provided cpu argument. The function uses cpu for per-cpu rstat
> lookups without checking whether it refers to a valid possible CPU.
>
> A BPF iter/cgroup program with CAP_BPF and CAP_PERFMON can pass an
> invalid cpu value. On an unfixed UBSCAN_BOUNDS test kernel, cpu ==
> 0x7fffffff triggers:
>
> [...]
Here is the summary with links:
- [v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
https://git.kernel.org/bpf/bpf/c/8817005efbdf
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply
* Re: [PATCH v2 0/4] memcg: shrink obj_stock_pcp and cache multiple objcgs
From: Shakeel Butt @ 2026-05-25 18:53 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260522193440.40e20563422afcc69b8445dd@linux-foundation.org>
On Fri, May 22, 2026 at 07:34:40PM -0700, Andrew Morton wrote:
> On Thu, 21 May 2026 18:19:04 -0700 Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> > Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
> > per-node type") split a memcg's single obj_cgroup into one per NUMA
> > node so that reparenting LRU folios can take per-node lru locks. As a
> > side effect, the per-CPU obj_stock_pcp -- which caches a single
> > cached_objcg pointer -- thrashes on workloads where threads of the
> > same memcg run on different NUMA nodes. The kernel test robot reported
> > a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.
> >
> > Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg
> > pointer") landed as a temporary fix by treating sibling per-node
> > objcgs as equivalent for the cache lookup, intended to be reverted
> > once per-node kmem accounting is introduced. This series takes a more
> > general approach: cache multiple objcgs per CPU using the multi-slot
> > pattern memcg_stock_pcp already uses, so the per-node objcg variants
> > of one memcg can all coexist in the stock without ever forcing a
> > drain. The temporary fix can then be reverted.
> >
> > To avoid increasing the per-CPU cache footprint, the first three
> > patches shrink the existing single-slot obj_stock_pcp fields.
> > The final patch converts cached_objcg and nr_bytes into
> > NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire
> > consume/refill/account hot path fits within a single 64-byte cache
> > line on non-debug 64-bit builds (verified with pahole).
>
> Thanks, I added this to mm.git's mm-new branch, along with a couple of
> possible todo notes from the review.
>
> Sashiko asked a thing:
> https://sashiko.dev/#/patchset/20260522011908.1669332-1-shakeel.butt@linux.dev
>
> Did you already see this? The footers there indicate that an email was
> sent out but I don't know if it works?
Yes, I saw that comment. It is kind of very specific to archs with 256KiB base
page sizes. Anyways, I have a simple fix for that and there were minor
suggestions from others for simple changes, I will send v3 with the requested
changes.
^ permalink raw reply
* [PATCH 0/7 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged
allocations, allowing small allocations to avoid walking the expensive
mem_cgroup hierarchy traversal and atomic operations on each charge.
This design introduces a fastpath, but there is room for improvement:
1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When
more than 7 mem_cgroups are actively charging on a single CPU, a
random victim is evicted and its associated stock is drained.
2. Stock management is tightly coupled to struct mem_cgroup, which makes
it difficult to add a new page_counter to mem_cgroup and have
multiple sources of stock management, which is required when trying
to introduce fastpaths to multiple hard limit checks.
This series moves the per-cpu stock down into the page_counter which
consolidates stock limit checking and page_counter limit checking into
page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit,
the random evictions (drain & refill), and slot traversal.
In turn, we can add independent stock management for additional
page_counters in each memcg, which is used in my tiered memory limits
series to add a new page_counter to track toptier usage [1].
The resulting code in memcg is also easier to follow, as the caching
becomes transparent from memcg's perspective and managed entirely within
page_counter.
There are, however, a few tradeoffs.
First, the bound on how much memory can be overcharged (and remain stale
as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages.
Now, it becomes nr_leaf_cgroups x nr_cpus x 64 pages. On large machines
with many cgroups, this could be significant. There are three qualifying
points: (1) larger machines should be able to tolerate the additional
overhead, (2) the stock should not remain stale as long as the
cgroups are actively charging memory, and (3) a process would have to
migrate across all CPUs to incur this upper bound on overhead.
Secondly, we introduce some additional memory footprint. The new struct
page_counter_stock adds 2 words of extra overhead per-(cpu x memcg).
A small change is that for cgroupv1, reported memsw usage can be lower
than reported memory usage, if the memsw page_counter overcharges to its
stock whereas the memory page_counter does not.
Finally, to keep the above memory footprint limited, I opted to not
embed a work_struct into page_counter_stock, but rather decided to
trigger synchronous stock draining, since the drain operation is rarer
now, and only happens under memory pressure and on cgroup death.
Performance testing across single-cgroup, as well as 4-cgroup (under the
7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system
shows negligible performance differences. In the tests, I repeatedly
fault and release anonymous pages using madvise(MADV_DONTNEED) to
stress the charge/uncharge path, across 40 trials of 50 iterations.
Metric here is time it took across each iteration (ms).
There are two testing versions below; the only difference is that v3
is based on top of mm-new, and v2 is based on top of mm-stable. The
"after" on both sides are similar, but mm-new and mm-stable have
different perforamnces.
v3, tested against mm-new
+----------+--------+-------+-----------+
| #cgroups | mm-new | after | delta (%) |
+----------+--------+-------+-----------+
| 1 | 357 | 358 | +0.283 |
| 4 | 1245 | 1214 | -2.430 |
| 32 | 9281 | 8970 | -3.470 |
+----------+--------+-------+-----------+
v2, tested against mm-stable
+----------+-----------+-------+-----------+
| #cgroups | mm-stable | after | delta (%) |
+----------+-----------+-------+-----------+
| 1 | 352 | 353 | +0.283 |
| 4 | 1198 | 1217 | +1.585 |
| 32 | 8980 | 9027 | +0.526 |
+----------+-----------+-------+-----------+
Further testing on other stress-ng microbenchmarks also agreed with
these results.
v2 --> v3:
- Rebased on top of latest mm-new, May 25, 2026, since the previous
version could not be applied for Sashiko review.
- Re-ran test numbers
v1 --> v2:
- Dropped stock returning on uncharge to preserve same behavior as memcg
stock. This resolves some race conditions present in v1.
- Fixed many race conditions between disabling page_counter_stock and
in-flight charges
- Restructured drain_all_stock to iterate over all CPUs first before
memcgs, to reduce the number of synchronous CPU work scheduling
- Optimized cgroup v2 further to drain only on the first child and skip
the root mem_cgroup
- Dropped RFC
- Wordsmithing cover letter
[1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/
Joshua Hahn (7):
mm/page_counter: introduce per-page_counter stock
mm/page_counter: use page_counter_stock in page_counter_try_charge
mm/page_counter: introduce stock drain APIs
mm/memcontrol: convert memcg to use page_counter_stock
mm/memcontrol: optimize memsw stock for cgroup v1
mm/memcontrol: optimize stock usage for cgroup v2
mm/memcontrol: remove unused memcg_stock code
include/linux/page_counter.h | 16 ++
mm/memcontrol.c | 289 +++++++----------------------------
mm/page_counter.c | 140 ++++++++++++++++-
3 files changed, 212 insertions(+), 233 deletions(-)
--
2.53.0-Meta
^ permalink raw reply
* [PATCH 1/7 v3] mm/page_counter: introduce per-page_counter stock
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
In order to avoid expensive hierarchy walks on every memcg charge and
limit check, memcontrol uses per-cpu stocks (memcg_stock_pcp) to cache
pre-charged pages and introduce a fast path to try_charge_memcg.
However, there are a few quirks with the current implementation that
could be improved upon.
First, each memcg_stock_pcp can only cache the charges of 7 memcgs
(defined as NR_MEMCG_STOCK), which means that once a CPU starts handling
the charging of more than 7 memcgs, it randomly selects a victim memcg
to evict and drain from the cpu, which can cause unnecessarily increased
latencies and thrashing as memcgs continually evict each others' stock.
Second, stock is tightly coupled with memcg, which means that all page
counters in a memcg share the same resource. This may simplify some of
the charging logic, but it prevents new page counters from being added
and using a separate stock.
We can address these concerns by pushing the concept of stock down to
the page_counter level, which addresses the random eviction problem by
getting rid of the 7 slot limit, and makes enabling separate stock
caches for other page_counters simpler.
Introduce a generic per-cpu stock directly in struct page_counter.
Stock can optionally be enabled per-page_counter, limiting the overhead
increase for page_counters who do not benefit greatly from caching
charges.
This patch introduces the page_counter_stock struct and its
enable/disable/free functions, but does not use these yet.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
include/linux/page_counter.h | 13 ++++++++
mm/page_counter.c | 64 ++++++++++++++++++++++++++++++++++++
2 files changed, 77 insertions(+)
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index d649b6bbbc871..c7e3ab3356d20 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -5,8 +5,15 @@
#include <linux/atomic.h>
#include <linux/cache.h>
#include <linux/limits.h>
+#include <linux/local_lock.h>
+#include <linux/percpu.h>
#include <asm/page.h>
+struct page_counter_stock {
+ local_trylock_t lock;
+ unsigned long nr_pages;
+};
+
struct page_counter {
/*
* Make sure 'usage' does not share cacheline with any other field in
@@ -41,6 +48,8 @@ struct page_counter {
unsigned long high;
unsigned long max;
struct page_counter *parent;
+ struct page_counter_stock __percpu *stock;
+ unsigned int batch;
} ____cacheline_internodealigned_in_smp;
#if BITS_PER_LONG == 32
@@ -99,6 +108,10 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
counter->watermark = usage;
}
+int page_counter_enable_stock(struct page_counter *counter, unsigned int batch);
+void page_counter_disable_stock(struct page_counter *counter);
+void page_counter_free_stock(struct page_counter *counter);
+
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
void page_counter_calculate_protection(struct page_counter *root,
struct page_counter *counter,
diff --git a/mm/page_counter.c b/mm/page_counter.c
index 661e0f2a5127a..a1a871a9d5c49 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -8,6 +8,7 @@
#include <linux/page_counter.h>
#include <linux/atomic.h>
#include <linux/kernel.h>
+#include <linux/percpu.h>
#include <linux/string.h>
#include <linux/sched.h>
#include <linux/bug.h>
@@ -289,6 +290,69 @@ int page_counter_memparse(const char *buf, const char *max,
return 0;
}
+int page_counter_enable_stock(struct page_counter *counter, unsigned int batch)
+{
+ struct page_counter_stock __percpu *stock;
+ int cpu;
+
+ stock = alloc_percpu(struct page_counter_stock);
+ if (!stock)
+ return -ENOMEM;
+
+ for_each_possible_cpu(cpu) {
+ struct page_counter_stock *s = per_cpu_ptr(stock, cpu);
+
+ local_trylock_init(&s->lock);
+ }
+ counter->stock = stock;
+ counter->batch = batch;
+
+ return 0;
+}
+
+static void page_counter_drain_stock_nolock(struct page_counter *counter)
+{
+ unsigned long stock_to_drain = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ struct page_counter_stock *stock;
+
+ stock = per_cpu_ptr(counter->stock, cpu);
+ stock_to_drain += stock->nr_pages;
+ stock->nr_pages = 0;
+ }
+
+ if (stock_to_drain)
+ page_counter_uncharge(counter, stock_to_drain);
+}
+
+void page_counter_disable_stock(struct page_counter *counter)
+{
+ if (!counter->stock)
+ return;
+
+ /* This prevents future charges from trying to deposit pages */
+ WRITE_ONCE(counter->batch, 0);
+
+ /*
+ * Charges can still be in-flight at this time. Instead of locking here,
+ * do the majority of the drains here without locking to free up pages
+ * now. Any remaining stock will be drained in page_counter_free_stock.
+ */
+ page_counter_drain_stock_nolock(counter);
+}
+
+void page_counter_free_stock(struct page_counter *counter)
+{
+ if (!counter->stock)
+ return;
+
+ page_counter_drain_stock_nolock(counter);
+ free_percpu(counter->stock);
+ counter->stock = NULL;
+}
+
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 2/7 v3] mm/page_counter: use page_counter_stock in page_counter_try_charge
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Make page_counter_try_charge() stock-aware. We preserve the same
semantics as the existing stock handling logic in try_charge_memcg:
1. Limit-check against the stock. If there is enough, charge to the
stock (non-hierarchical) and return immediately.
2. Greedily attempt to fulfill the charge request and fill the stock up
at the same time via a hierarchical charge.
3. If we fail with this charge, retry again (once) with the exact number
of pages requested.
4. If we succeed with the greedy attempt, then try to add those extra
pages to the stock. If that fails (trylock), then uncharge those
surplus pages hierarchically.
As of this patch, the page_counter_stock is unused, as it has not been
enabled on any memcg yet. No functional changes intended.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/page_counter.c | 42 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/mm/page_counter.c b/mm/page_counter.c
index a1a871a9d5c49..e002688bf7f1a 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -121,9 +121,25 @@ bool page_counter_try_charge(struct page_counter *counter,
struct page_counter **fail)
{
struct page_counter *c;
+ unsigned long charge = nr_pages;
+ unsigned long batch = READ_ONCE(counter->batch);
bool protection = track_protection(counter);
bool track_failcnt = counter->track_failcnt;
+ if (counter->stock && local_trylock(&counter->stock->lock)) {
+ struct page_counter_stock *stock = this_cpu_ptr(counter->stock);
+
+ if (stock->nr_pages >= charge) {
+ stock->nr_pages -= charge;
+ local_unlock(&counter->stock->lock);
+ return true;
+ }
+ local_unlock(&counter->stock->lock);
+ }
+
+ charge = max_t(unsigned long, batch, nr_pages);
+
+retry:
for (c = counter; c; c = c->parent) {
long new;
/*
@@ -140,9 +156,9 @@ bool page_counter_try_charge(struct page_counter *counter,
* we either see the new limit or the setter sees the
* counter has changed and retries.
*/
- new = atomic_long_add_return(nr_pages, &c->usage);
+ new = atomic_long_add_return(charge, &c->usage);
if (new > c->max) {
- atomic_long_sub(nr_pages, &c->usage);
+ atomic_long_sub(charge, &c->usage);
/*
* This is racy, but we can live with some
* inaccuracy in the failcnt which is only used
@@ -163,11 +179,31 @@ bool page_counter_try_charge(struct page_counter *counter,
WRITE_ONCE(c->watermark, new);
}
}
+
+ /* charge > nr_pages implies this page_counter has stock enabled */
+ if (charge > nr_pages) {
+ if (local_trylock(&counter->stock->lock)) {
+ struct page_counter_stock *stock;
+
+ stock = this_cpu_ptr(counter->stock);
+ stock->nr_pages += charge - nr_pages;
+ local_unlock(&counter->stock->lock);
+ } else {
+ page_counter_uncharge(counter, charge - nr_pages);
+ }
+ }
+
return true;
failed:
for (c = counter; c != *fail; c = c->parent)
- page_counter_cancel(c, nr_pages);
+ page_counter_cancel(c, charge);
+
+ if (charge > nr_pages) {
+ /* Retry without trying to grab extra pages to refill stock */
+ charge = nr_pages;
+ goto retry;
+ }
return false;
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 3/7 v3] mm/page_counter: introduce stock drain APIs
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Introduce page_counter variants to replace memcg stock draining
functions.
page_counter_drain_stock_local() drains the stock of the local CPU,
taking a local stock lock to serialize against concurrent charges.
page_counter_drain_stock_cpu() does the same, but without taking a local
lock. This is possible because it will only be called from the CPU
hotplug path, where the CPU is dead and there cannot be any more charges.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
include/linux/page_counter.h | 3 +++
mm/page_counter.c | 34 ++++++++++++++++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index c7e3ab3356d20..ffe13224213c9 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -111,6 +111,9 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
int page_counter_enable_stock(struct page_counter *counter, unsigned int batch);
void page_counter_disable_stock(struct page_counter *counter);
void page_counter_free_stock(struct page_counter *counter);
+void page_counter_drain_stock_local(struct page_counter *counter);
+void page_counter_drain_stock_cpu(struct page_counter *counter,
+ unsigned int cpu);
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
void page_counter_calculate_protection(struct page_counter *root,
diff --git a/mm/page_counter.c b/mm/page_counter.c
index e002688bf7f1a..fbfe9a1b29d2e 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -389,6 +389,40 @@ void page_counter_free_stock(struct page_counter *counter)
counter->stock = NULL;
}
+void page_counter_drain_stock_local(struct page_counter *counter)
+{
+ struct page_counter_stock *stock;
+ unsigned long nr_pages;
+
+ if (!counter->stock)
+ return;
+
+ local_lock(&counter->stock->lock);
+ stock = this_cpu_ptr(counter->stock);
+ nr_pages = stock->nr_pages;
+ stock->nr_pages = 0;
+ local_unlock(&counter->stock->lock);
+
+ if (nr_pages)
+ page_counter_uncharge(counter, nr_pages);
+}
+
+void page_counter_drain_stock_cpu(struct page_counter *counter,
+ unsigned int cpu)
+{
+ struct page_counter_stock *stock;
+ unsigned long nr_pages;
+
+ if (!counter->stock)
+ return;
+
+ stock = per_cpu_ptr(counter->stock, cpu);
+ nr_pages = stock->nr_pages;
+ if (nr_pages) {
+ stock->nr_pages = 0;
+ page_counter_uncharge(counter, nr_pages);
+ }
+}
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 4/7 v3] mm/memcontrol: convert memcg to use page_counter_stock
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Now with all of the memcg_stock handling logic replicated in
page_counter_stock, switch memcg to use the page_counter_stock.
There are a few details that have changed:
First, the old special-casing for the !allow_spinning check to avoid
refilling and flushing of the old stock is removed. This special casing
was important previously, because refilling the stock could do a lot of
extra work by evicting one of 7 random victim memcgs in the percpu
memcg_stock slots. In the new per-counter design, refilling stock just adds
pages to the counter's own local cache without affecting other memcgs,
so the original reason for the special case no longer applies.
Also, we can now fail during page_counter_enable_stock(), if there is
not enough memory to allocate a percpu page_counter_stock. This failure
is rare and nonfatal; the system can continue to operate, with the page
counter working without stock and falling back to walking the hierarchy.
Finally, drain_all_stock is restructured to iterate CPUs in the outer
loop (rather than memcgs) to be able to schedule draining all memcgs
via a single work_on_cpu call. It reduces the number of synchronous
per-CPU work calls from O(memcgs * CPUs) to just O(CPUs).
Note that obj_stock remains untouched by these changes.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 78 +++++++++++++++++++++----------------------------
1 file changed, 34 insertions(+), 44 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 368efc1455e35..952c6f7430395 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2260,6 +2260,17 @@ static void schedule_drain_work(int cpu, struct work_struct *work)
queue_work_on(cpu, memcg_wq, work);
}
+static long drain_stock_on_cpu(void *arg)
+{
+ struct mem_cgroup *root_memcg = arg;
+ struct mem_cgroup *memcg;
+
+ for_each_mem_cgroup_tree(memcg, root_memcg)
+ page_counter_drain_stock_local(&memcg->memory);
+
+ return 0;
+}
+
/*
* Drains all per-CPU charge caches for given root_memcg resp. subtree
* of the hierarchy under it.
@@ -2271,28 +2282,16 @@ void drain_all_stock(struct mem_cgroup *root_memcg)
/* If someone's already draining, avoid adding running more workers. */
if (!mutex_trylock(&percpu_charge_mutex))
return;
- /*
- * Notify other cpus that system-wide "drain" is running
- * We do not care about races with the cpu hotplug because cpu down
- * as well as workers from this path always operate on the local
- * per-cpu data. CPU up doesn't touch memcg_stock at all.
- */
+
+ for_each_online_cpu(cpu)
+ work_on_cpu(cpu, drain_stock_on_cpu, root_memcg);
+
+ /* Drain obj_stock on all online CPUs */
migrate_disable();
curcpu = smp_processor_id();
for_each_online_cpu(cpu) {
- struct memcg_stock_pcp *memcg_st = &per_cpu(memcg_stock, cpu);
struct obj_stock_pcp *obj_st = &per_cpu(obj_stock, cpu);
- if (!test_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags) &&
- is_memcg_drain_needed(memcg_st, root_memcg) &&
- !test_and_set_bit(FLUSHING_CACHED_CHARGE,
- &memcg_st->flags)) {
- if (cpu == curcpu)
- drain_local_memcg_stock(&memcg_st->work);
- else
- schedule_drain_work(cpu, &memcg_st->work);
- }
-
if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) &&
obj_stock_flush_required(obj_st, root_memcg) &&
!test_and_set_bit(FLUSHING_CACHED_CHARGE,
@@ -2309,9 +2308,13 @@ void drain_all_stock(struct mem_cgroup *root_memcg)
static int memcg_hotplug_cpu_dead(unsigned int cpu)
{
+ struct mem_cgroup *memcg;
+
/* no need for the local lock */
drain_obj_stock(&per_cpu(obj_stock, cpu));
- drain_stock_fully(&per_cpu(memcg_stock, cpu));
+
+ for_each_mem_cgroup(memcg)
+ page_counter_drain_stock_cpu(&memcg->memory, cpu);
return 0;
}
@@ -2586,7 +2589,6 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
unsigned int nr_pages)
{
- unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
int nr_retries = MAX_RECLAIM_RETRIES;
struct mem_cgroup *mem_over_limit;
struct page_counter *counter;
@@ -2599,31 +2601,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
bool allow_spinning = gfpflags_allow_spinning(gfp_mask);
retry:
- if (consume_stock(memcg, nr_pages))
- return 0;
-
- if (!allow_spinning)
- /* Avoid the refill and flush of the older stock */
- batch = nr_pages;
-
reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
if (!do_memsw_account() ||
- page_counter_try_charge(&memcg->memsw, batch, &counter)) {
- if (page_counter_try_charge(&memcg->memory, batch, &counter))
+ page_counter_try_charge(&memcg->memsw, nr_pages, &counter)) {
+ if (page_counter_try_charge(&memcg->memory, nr_pages, &counter))
goto done_restock;
if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, batch);
+ page_counter_uncharge(&memcg->memsw, nr_pages);
mem_over_limit = mem_cgroup_from_counter(counter, memory);
} else {
mem_over_limit = mem_cgroup_from_counter(counter, memsw);
reclaim_options &= ~MEMCG_RECLAIM_MAY_SWAP;
}
- if (batch > nr_pages) {
- batch = nr_pages;
- goto retry;
- }
-
/*
* Prevent unbounded recursion when reclaim operations need to
* allocate memory. This might exceed the limits temporarily,
@@ -2720,9 +2710,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
return 0;
done_restock:
- if (batch > nr_pages)
- refill_stock(memcg, batch - nr_pages);
-
/*
* If the hierarchy is above the normal consumption range, schedule
* reclaim on returning to userland. We can perform reclaim here
@@ -2759,7 +2746,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
* and distribute reclaim work and delay penalties
* based on how much each task is actually allocating.
*/
- current->memcg_nr_pages_over_high += batch;
+ current->memcg_nr_pages_over_high += nr_pages;
set_notify_resume(current);
break;
}
@@ -3064,7 +3051,7 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg,
account_kmem_nmi_safe(memcg, -nr_pages);
memcg1_account_kmem(memcg, -nr_pages);
if (!mem_cgroup_is_root(memcg))
- refill_stock(memcg, nr_pages);
+ memcg_uncharge(memcg, nr_pages);
css_put(&memcg->css);
}
@@ -4096,6 +4083,8 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)
static void mem_cgroup_free(struct mem_cgroup *memcg)
{
+ page_counter_free_stock(&memcg->memory);
+ page_counter_free_stock(&memcg->memsw);
lru_gen_exit_memcg(memcg);
memcg_wb_domain_exit(memcg);
__mem_cgroup_free(memcg);
@@ -4268,6 +4257,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
refcount_set(&memcg->id.ref, 1);
css_get(css);
+ /* failure is nonfatal, charges fall back to direct hierarchy */
+ page_counter_enable_stock(&memcg->memory, MEMCG_CHARGE_BATCH);
+
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
*
@@ -4330,6 +4322,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
lru_gen_offline_memcg(memcg);
drain_all_stock(memcg);
+ page_counter_disable_stock(&memcg->memory);
mem_cgroup_private_id_put(memcg, 1);
}
@@ -5524,7 +5517,7 @@ void mem_cgroup_sk_uncharge(const struct sock *sk, unsigned int nr_pages)
mod_memcg_state(memcg, MEMCG_SOCK, -nr_pages);
- refill_stock(memcg, nr_pages);
+ page_counter_uncharge(&memcg->memory, nr_pages);
}
void mem_cgroup_flush_workqueue(void)
@@ -5577,12 +5570,9 @@ int __init mem_cgroup_init(void)
memcg_wq = alloc_workqueue("memcg", WQ_PERCPU, 0);
WARN_ON(!memcg_wq);
- for_each_possible_cpu(cpu) {
- INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work,
- drain_local_memcg_stock);
+ for_each_possible_cpu(cpu)
INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work,
drain_local_obj_stock);
- }
memcg_size = struct_size_t(struct mem_cgroup, nodeinfo, nr_node_ids);
memcg_cachep = kmem_cache_create("mem_cgroup", memcg_size, 0,
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 5/7 v3] mm/memcontrol: optimize memsw stock for cgroup v1
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Previously, each memcg had its own stock, which was shared by all page
counters within it. Specifically in try_charge_memcg, the stock limit
check would occur before the memsw and memory page_counters were
charged hierarchically. Now that the memcg stock was folded into the
page_counter level, and we have replaced try_charge_memcg's stock check
against the memory page_counter's stock, this leaves no fast path available
for cgroup v1's memsw check.
Introduce a new stock for the memsw page_counter, charged independently
from the memory page_counter. This provides better caching on cgroup v1:
The best case scenario is when both the memsw and memory page_counters
can use their cached stock charge; this is the old behavior.
The halfway scenario is when either the memsw or memory page_counter
is within the stock size, but the other isn't. This requires one
hierarchical charge.
The worst case scenario is when both memsw and memory page_counters
are over their limit, and must walk two page_counter hierarchies. This
is the same as the old behavior.
By introducing an independent stock for memsw, we can avoid the worst
case scenario more often and can fail or succeed separately from the
memory page counter.
One user-visible change is that reported memsw usage may transiently
be lower than memory usage. This happens because each counter
independently batches the stock charges, so the visible values can
differ by up to the stock batch size (MEMCG_CHARGE_BATCH) pages.
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 952c6f7430395..f20c9b829224a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2265,8 +2265,11 @@ static long drain_stock_on_cpu(void *arg)
struct mem_cgroup *root_memcg = arg;
struct mem_cgroup *memcg;
- for_each_mem_cgroup_tree(memcg, root_memcg)
+ for_each_mem_cgroup_tree(memcg, root_memcg) {
page_counter_drain_stock_local(&memcg->memory);
+ if (do_memsw_account())
+ page_counter_drain_stock_local(&memcg->memsw);
+ }
return 0;
}
@@ -2313,8 +2316,11 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu)
/* no need for the local lock */
drain_obj_stock(&per_cpu(obj_stock, cpu));
- for_each_mem_cgroup(memcg)
+ for_each_mem_cgroup(memcg) {
page_counter_drain_stock_cpu(&memcg->memory, cpu);
+ if (do_memsw_account())
+ page_counter_drain_stock_cpu(&memcg->memsw, cpu);
+ }
return 0;
}
@@ -4259,6 +4265,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
/* failure is nonfatal, charges fall back to direct hierarchy */
page_counter_enable_stock(&memcg->memory, MEMCG_CHARGE_BATCH);
+ if (do_memsw_account())
+ page_counter_enable_stock(&memcg->memsw, MEMCG_CHARGE_BATCH);
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
@@ -4323,6 +4331,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
drain_all_stock(memcg);
page_counter_disable_stock(&memcg->memory);
+ if (do_memsw_account())
+ page_counter_disable_stock(&memcg->memsw);
mem_cgroup_private_id_put(memcg, 1);
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 6/7 v3] mm/memcontrol: optimize stock usage for cgroup v2
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
In cgroup v2, it is unlikely for memcg charges to happen directly on
non-leaf cgroups. There are a few exceptions, such as processes that
have yet to be migrated to children, and tasks that are reparented on
memcg destruction, that charge to the parent.
Because it is rare for parent cgroups to receive direct charges, stock
that remains in them are wasted memory.
Drain parent stocks when the first child is created to return those
pages for other memcgs to use.
This optimization is not for cgroup v1, where tasks can be attached to
any cgroup in the hierarchy, meaning stock can be consumed & refilled
for non-leaf cgroups as well.
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f20c9b829224a..64b82f1782720 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4280,6 +4280,21 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
*/
xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL);
+ /*
+ * It is unlikely for non-leaf memcgs to get direct charges on v2.
+ * Drain the parent's stock if we are the first child.
+ */
+ if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
+ struct mem_cgroup *parent = parent_mem_cgroup(memcg);
+ int cpu;
+
+ if (parent && !mem_cgroup_is_root(parent) &&
+ !css_has_online_children(&parent->css)) {
+ for_each_online_cpu(cpu)
+ work_on_cpu(cpu, drain_stock_on_cpu, parent);
+ }
+ }
+
return 0;
free_objcg:
for_each_node(nid) {
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 7/7 v3] mm/memcontrol: remove unused memcg_stock code
From: Joshua Hahn @ 2026-05-25 19:04 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
Now that all memcg_stock logic has been moved to page_counter_stock, we
can remove all code related to handling memcg_stock. Note that obj_stock
is untouched and is still needed. FLUSHING_CACHED_CHARGE is preserved
so that it can be used by obj_stock as well.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 186 ------------------------------------------------
1 file changed, 186 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 64b82f1782720..5319219d0dcb5 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1998,25 +1998,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
pr_cont(" are going to be killed due to memory.oom.group set\n");
}
-/*
- * The value of NR_MEMCG_STOCK is selected to keep the cached memcgs and their
- * nr_pages in a single cacheline. This may change in future.
- */
-#define NR_MEMCG_STOCK 7
#define FLUSHING_CACHED_CHARGE 0
-struct memcg_stock_pcp {
- local_trylock_t lock;
- uint8_t nr_pages[NR_MEMCG_STOCK];
- struct mem_cgroup *cached[NR_MEMCG_STOCK];
-
- struct work_struct work;
- unsigned long flags;
- uint8_t drain_idx;
-};
-
-static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
- .lock = INIT_LOCAL_TRYLOCK(lock),
-};
/*
* NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
@@ -2056,47 +2038,6 @@ static void drain_obj_stock(struct obj_stock_pcp *stock);
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg);
-/**
- * consume_stock: Try to consume stocked charge on this cpu.
- * @memcg: memcg to consume from.
- * @nr_pages: how many pages to charge.
- *
- * Consume the cached charge if enough nr_pages are present otherwise return
- * failure. Also return failure for charge request larger than
- * MEMCG_CHARGE_BATCH or if the local lock is already taken.
- *
- * returns true if successful, false otherwise.
- */
-static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
-{
- struct memcg_stock_pcp *stock;
- uint8_t stock_pages;
- bool ret = false;
- int i;
-
- if (nr_pages > MEMCG_CHARGE_BATCH ||
- !local_trylock(&memcg_stock.lock))
- return ret;
-
- stock = this_cpu_ptr(&memcg_stock);
-
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- if (memcg != READ_ONCE(stock->cached[i]))
- continue;
-
- stock_pages = READ_ONCE(stock->nr_pages[i]);
- if (stock_pages >= nr_pages) {
- WRITE_ONCE(stock->nr_pages[i], stock_pages - nr_pages);
- ret = true;
- }
- break;
- }
-
- local_unlock(&memcg_stock.lock);
-
- return ret;
-}
-
static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
page_counter_uncharge(&memcg->memory, nr_pages);
@@ -2104,51 +2045,6 @@ static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
page_counter_uncharge(&memcg->memsw, nr_pages);
}
-/*
- * Returns stocks cached in percpu and reset cached information.
- */
-static void drain_stock(struct memcg_stock_pcp *stock, int i)
-{
- struct mem_cgroup *old = READ_ONCE(stock->cached[i]);
- uint8_t stock_pages;
-
- if (!old)
- return;
-
- stock_pages = READ_ONCE(stock->nr_pages[i]);
- if (stock_pages) {
- memcg_uncharge(old, stock_pages);
- WRITE_ONCE(stock->nr_pages[i], 0);
- }
-
- css_put(&old->css);
- WRITE_ONCE(stock->cached[i], NULL);
-}
-
-static void drain_stock_fully(struct memcg_stock_pcp *stock)
-{
- int i;
-
- for (i = 0; i < NR_MEMCG_STOCK; ++i)
- drain_stock(stock, i);
-}
-
-static void drain_local_memcg_stock(struct work_struct *dummy)
-{
- struct memcg_stock_pcp *stock;
-
- if (WARN_ONCE(!in_task(), "drain in non-task context"))
- return;
-
- local_lock(&memcg_stock.lock);
-
- stock = this_cpu_ptr(&memcg_stock);
- drain_stock_fully(stock);
- clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
-
- local_unlock(&memcg_stock.lock);
-}
-
static void drain_local_obj_stock(struct work_struct *dummy)
{
struct obj_stock_pcp *stock;
@@ -2165,88 +2061,6 @@ static void drain_local_obj_stock(struct work_struct *dummy)
local_unlock(&obj_stock.lock);
}
-static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
-{
- struct memcg_stock_pcp *stock;
- struct mem_cgroup *cached;
- uint8_t stock_pages;
- bool success = false;
- int empty_slot = -1;
- int i;
-
- /*
- * For now limit MEMCG_CHARGE_BATCH to 127 and less. In future if we
- * decide to increase it more than 127 then we will need more careful
- * handling of nr_pages[] in struct memcg_stock_pcp.
- */
- BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S8_MAX);
-
- VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg));
-
- if (nr_pages > MEMCG_CHARGE_BATCH ||
- !local_trylock(&memcg_stock.lock)) {
- /*
- * In case of larger than batch refill or unlikely failure to
- * lock the percpu memcg_stock.lock, uncharge memcg directly.
- */
- memcg_uncharge(memcg, nr_pages);
- return;
- }
-
- stock = this_cpu_ptr(&memcg_stock);
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- cached = READ_ONCE(stock->cached[i]);
- if (!cached && empty_slot == -1)
- empty_slot = i;
- if (memcg == READ_ONCE(stock->cached[i])) {
- stock_pages = READ_ONCE(stock->nr_pages[i]) + nr_pages;
- WRITE_ONCE(stock->nr_pages[i], stock_pages);
- if (stock_pages > MEMCG_CHARGE_BATCH)
- drain_stock(stock, i);
- success = true;
- break;
- }
- }
-
- if (!success) {
- i = empty_slot;
- if (i == -1) {
- i = stock->drain_idx++;
- if (stock->drain_idx == NR_MEMCG_STOCK)
- stock->drain_idx = 0;
- drain_stock(stock, i);
- }
- css_get(&memcg->css);
- WRITE_ONCE(stock->cached[i], memcg);
- WRITE_ONCE(stock->nr_pages[i], nr_pages);
- }
-
- local_unlock(&memcg_stock.lock);
-}
-
-static bool is_memcg_drain_needed(struct memcg_stock_pcp *stock,
- struct mem_cgroup *root_memcg)
-{
- struct mem_cgroup *memcg;
- bool flush = false;
- int i;
-
- rcu_read_lock();
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- memcg = READ_ONCE(stock->cached[i]);
- if (!memcg)
- continue;
-
- if (READ_ONCE(stock->nr_pages[i]) &&
- mem_cgroup_is_descendant(memcg, root_memcg)) {
- flush = true;
- break;
- }
- }
- rcu_read_unlock();
- return flush;
-}
-
static void schedule_drain_work(int cpu, struct work_struct *work)
{
/*
--
2.53.0-Meta
^ permalink raw reply related
* Re: [PATCH v2 0/4] mm/zswap: Implement per-cgroup proactive writeback
From: Andrew Morton @ 2026-05-25 19:24 UTC (permalink / raw)
To: Hao Jia
Cc: tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260525122242.36127-1-jiahao.kernel@gmail.com>
On Mon, 25 May 2026 20:22:38 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
> Zswap currently writes back pages to backing swap reactively, triggered
> either by the shrinker or by the pool reaching its size limit. Although
> proactive memory reclaim can automatically write back a portion of zswap
> pages via the shrinker, it cannot explicitly control the amount of
> writeback for a specific memory cgroup. Moreover, proactive memory reclaim
> may not always be triggered during a steady state.
>
> In certain scenarios, it is desirable to trigger writeback in advance to
> free up memory. For example, users may want to prepare for an upcoming
> memory-intensive workload by flushing cold memory to the backing storage
> when the system is relatively idle.
>
> This patch series introduces a "zswap_writeback_only" key to memory.reclaim
> cgroup interface, allowing users to proactively write back cold compressed
> pages from zswap to the backing swap device. When specified, this key
> bypasses standard memory reclaim and exclusively performs proactive zswap
> writeback up to the requested budget. If omitted, the default reclaim
> behavior remains unchanged.
Thanks. AI review found a few things to complain about, one of them
described as "preexisting".
^ permalink raw reply
* Re: [PATCH 1/7 v3] mm/page_counter: introduce per-page_counter stock
From: Joshua Hahn @ 2026-05-25 19:45 UTC (permalink / raw)
To: Joshua Hahn
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, cgroups, linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-2-joshua.hahnjy@gmail.com>
On Mon, 25 May 2026 12:04:48 -0700 Joshua Hahn <joshua.hahnjy@gmail.com> wrote:
> In order to avoid expensive hierarchy walks on every memcg charge and
> limit check, memcontrol uses per-cpu stocks (memcg_stock_pcp) to cache
> pre-charged pages and introduce a fast path to try_charge_memcg.
>
> However, there are a few quirks with the current implementation that
> could be improved upon.
>
> First, each memcg_stock_pcp can only cache the charges of 7 memcgs
> (defined as NR_MEMCG_STOCK), which means that once a CPU starts handling
> the charging of more than 7 memcgs, it randomly selects a victim memcg
> to evict and drain from the cpu, which can cause unnecessarily increased
> latencies and thrashing as memcgs continually evict each others' stock.
>
> Second, stock is tightly coupled with memcg, which means that all page
> counters in a memcg share the same resource. This may simplify some of
> the charging logic, but it prevents new page counters from being added
> and using a separate stock.
>
> We can address these concerns by pushing the concept of stock down to
> the page_counter level, which addresses the random eviction problem by
> getting rid of the 7 slot limit, and makes enabling separate stock
> caches for other page_counters simpler.
>
> Introduce a generic per-cpu stock directly in struct page_counter.
> Stock can optionally be enabled per-page_counter, limiting the overhead
> increase for page_counters who do not benefit greatly from caching
> charges.
>
> This patch introduces the page_counter_stock struct and its
> enable/disable/free functions, but does not use these yet.
>
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> ---
> include/linux/page_counter.h | 13 ++++++++
> mm/page_counter.c | 64 ++++++++++++++++++++++++++++++++++++
> 2 files changed, 77 insertions(+)
>
> diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
> index d649b6bbbc871..c7e3ab3356d20 100644
> --- a/include/linux/page_counter.h
> +++ b/include/linux/page_counter.h
> @@ -5,8 +5,15 @@
> #include <linux/atomic.h>
> #include <linux/cache.h>
> #include <linux/limits.h>
> +#include <linux/local_lock.h>
> +#include <linux/percpu.h>
> #include <asm/page.h>
>
> +struct page_counter_stock {
> + local_trylock_t lock;
> + unsigned long nr_pages;
> +};
> +
> struct page_counter {
> /*
> * Make sure 'usage' does not share cacheline with any other field in
> @@ -41,6 +48,8 @@ struct page_counter {
> unsigned long high;
> unsigned long max;
> struct page_counter *parent;
> + struct page_counter_stock __percpu *stock;
> + unsigned int batch;
> } ____cacheline_internodealigned_in_smp;
>
> #if BITS_PER_LONG == 32
> @@ -99,6 +108,10 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
> counter->watermark = usage;
> }
>
> +int page_counter_enable_stock(struct page_counter *counter, unsigned int batch);
> +void page_counter_disable_stock(struct page_counter *counter);
> +void page_counter_free_stock(struct page_counter *counter);
> +
> #if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
> void page_counter_calculate_protection(struct page_counter *root,
> struct page_counter *counter,
> diff --git a/mm/page_counter.c b/mm/page_counter.c
> index 661e0f2a5127a..a1a871a9d5c49 100644
> --- a/mm/page_counter.c
> +++ b/mm/page_counter.c
> @@ -8,6 +8,7 @@
> #include <linux/page_counter.h>
> #include <linux/atomic.h>
> #include <linux/kernel.h>
> +#include <linux/percpu.h>
> #include <linux/string.h>
> #include <linux/sched.h>
> #include <linux/bug.h>
> @@ -289,6 +290,69 @@ int page_counter_memparse(const char *buf, const char *max,
> return 0;
> }
>
> +int page_counter_enable_stock(struct page_counter *counter, unsigned int batch)
> +{
> + struct page_counter_stock __percpu *stock;
> + int cpu;
> +
> + stock = alloc_percpu(struct page_counter_stock);
> + if (!stock)
> + return -ENOMEM;
> +
> + for_each_possible_cpu(cpu) {
> + struct page_counter_stock *s = per_cpu_ptr(stock, cpu);
> +
> + local_trylock_init(&s->lock);
> + }
> + counter->stock = stock;
> + counter->batch = batch;
> +
> + return 0;
> +}
> +
> +static void page_counter_drain_stock_nolock(struct page_counter *counter)
> +{
> + unsigned long stock_to_drain = 0;
> + int cpu;
> +
> + for_each_possible_cpu(cpu) {
> + struct page_counter_stock *stock;
> +
> + stock = per_cpu_ptr(counter->stock, cpu);
> + stock_to_drain += stock->nr_pages;
> + stock->nr_pages = 0;
> + }
> +
> + if (stock_to_drain)
> + page_counter_uncharge(counter, stock_to_drain);
> +}
> +
> +void page_counter_disable_stock(struct page_counter *counter)
> +{
> + if (!counter->stock)
> + return;
> +
> + /* This prevents future charges from trying to deposit pages */
> + WRITE_ONCE(counter->batch, 0);
> +
> + /*
> + * Charges can still be in-flight at this time. Instead of locking here,
> + * do the majority of the drains here without locking to free up pages
> + * now. Any remaining stock will be drained in page_counter_free_stock.
> + */
> + page_counter_drain_stock_nolock(counter);
Sashiko raised a concern here.
Allowing racy charges is OK, but the problem is that writing stock->nr_pages = 0
with no lock here can race with the reading of that value, and lead to
double-uncharging when racing with concurrent charges.
I think that this can be fixed by not draining in disable_stock and
reordering the callsite:
Before:
drain_all_stock(memcg);
page_counter_disable_stock(&memcg->memory);
After:
page_counter_disable_stock(&memcg->memory);
drain_all_stock(memcg);
This way, the WRITE_ONCE(counter->batch, 0); should prevent any
future charges from trying to land, before we drain stock.
Despite not allowing any more racy charges, we still need to keep the drain
in free_stock since drain_all_stock() uses a mutex_trylock and can fail;
if that happens we need to still drain the stock at a later time, when we
can guarantee that there will be no more contention.
Thanks, Sashiko! I'll keep my eyes posted on the rest of the series as it
goes through the review pipeline.
Joshua
> +}
> +
> +void page_counter_free_stock(struct page_counter *counter)
> +{
> + if (!counter->stock)
> + return;
> +
> + page_counter_drain_stock_nolock(counter);
> + free_percpu(counter->stock);
> + counter->stock = NULL;
> +}
> +
^ permalink raw reply
* Re: [PATCH v6 1/4] mm: swap: introduce swap tier infrastructure
From: Baoquan He @ 2026-05-25 21:49 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-2-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
> This patch introduces the "Swap tier" concept, which serves as an
> abstraction layer for managing swap devices based on their performance
> characteristics (e.g., NVMe, HDD, Network swap).
>
> Swap tiers are user-named groups representing priority ranges.
> Tier names must consist of alphanumeric characters and underscores.
> These tiers collectively cover the entire priority space from -1
> (`DEF_SWAP_PRIO`) to `SHRT_MAX`.
>
> To configure tiers, a new sysfs interface is exposed at
> /sys/kernel/mm/swap/tiers. The input parser evaluates commands from
> left to right and supports batch input, allowing users to add or remove
> multiple tiers in a single write operation.
>
> Tier management enforces continuous priority ranges anchored by start
> priorities. Operations trigger range splitting or merging, but overwriting
> start priorities is forbidden. Merging expands lower tiers upwards to
> preserve configured start priorities, except when removing `DEF_SWAP_PRIO`,
> which merges downwards.
>
> Suggested-by: Chris Li <chrisl@kernel.org>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
This looks good to me.
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
While there's only one tiny concern, please see the inline comment.
> diff --git a/mm/swap_tier.c b/mm/swap_tier.c
> new file mode 100644
> index 000000000000..9490e891c5fe
> --- /dev/null
> +++ b/mm/swap_tier.c
> @@ -0,0 +1,302 @@
......
> +/*
> + * Naming Convention:
> + * swap_tiers_*() - Public/exported functions
> + * swap_tier_*() - Private/internal functions
> + */
> +
> +static bool swap_tier_is_active(void)
> +{
> + return !list_empty(&swap_tier_active_list) ? true : false;
The above line seems like generated by AI. Whatever else I have seen is
"return !list_empty(&swap_tier_active_list);" which is enough.
> +}
> +
...snip...
^ permalink raw reply
* Re: [PATCH v6 1/4] mm: swap: introduce swap tier infrastructure
From: Baoquan He @ 2026-05-25 22:57 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-2-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
...snip...
> +bool swap_tiers_validate(void)
> +{
> + struct swap_tier *tier;
> +
> + /*
> + * Initial setting might not cover DEF_SWAP_PRIO.
> + * Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
> + */
If so, do we need check if the upmost boundary SHRT_MAX is covered?
> + if (swap_tier_is_active()) {
> + tier = list_last_entry(&swap_tier_active_list,
> + struct swap_tier, list);
> +
> + if (tier->prio != DEF_SWAP_PRIO)
> + return false;
> + }
> +
> + return true;
> +}
...snip...
^ permalink raw reply
* Re: [PATCH v6 2/4] mm: swap: associate swap devices with tiers
From: Baoquan He @ 2026-05-25 23:04 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-3-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
> This patch connects swap devices to the swap tier infrastructure,
> ensuring that devices are correctly assigned to tiers based on their
> priority.
>
> A `tier_mask` is added to identify the tier membership of swap devices.
> Although tier-based allocation logic is not yet implemented, this
> mapping is necessary to track which tier a device belongs to. Upon
> activation, the device is assigned to a tier by matching its priority
> against the configured tier ranges.
>
> The infrastructure allows dynamic modification of tiers, such as
> splitting or merging ranges. These operations are permitted provided
> that the tier assignment of already configured swap devices remains
> unchanged.
>
> This patch also adds the documentation for the swap tier feature,
> covering the core concepts, sysfs interface usage, and configuration
> details.
>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> ---
> Documentation/mm/index.rst | 1 +
> Documentation/mm/swap-tier.rst | 159 +++++++++++++++++++++++++++++++++
> MAINTAINERS | 1 +
> include/linux/swap.h | 1 +
> mm/swap_state.c | 2 +-
> mm/swap_tier.c | 101 ++++++++++++++++++---
> mm/swap_tier.h | 13 ++-
> mm/swapfile.c | 2 +
> 8 files changed, 266 insertions(+), 14 deletions(-)
> create mode 100644 Documentation/mm/swap-tier.rst
LGTM,
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
^ permalink raw reply
* [RFC PATCH bpf-next v7 00/11] mm: BPF struct_ops for dynamic memory protection and async reclaim
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
Overview:
This series introduces BPF struct_ops support for the memory controller,
enabling userspace BPF programs to implement custom, dynamic memory
management policies per cgroup. The feature allows BPF programs to hook
into the core reclaim and charge paths without requiring kernel
modifications, providing a flexible alternative to static knobs such as
memory.low and memory.min.
The series enables two complementary use cases.
Dynamic memory protection: static memory protection thresholds
(memory.low, memory.min) are poor fits for workloads whose actual memory
activity varies over time. A high-priority cgroup holding a large working
set but temporarily idle will still suppress reclaim on its siblings,
wasting available memory. A BPF-driven approach can observe real workload
activity -- page faults, charge/uncharge events -- and activate or
withdraw protection dynamically. The test results at the end of this
letter quantify the difference: in a scenario where the high-priority
cgroup is idle, the BPF-controlled low-priority cgroup achieves roughly
37x higher throughput than with static memory.low.
Asynchronous proactive reclaim: the memcg_charged and memcg_uncharged
hooks, combined with the BPF workqueue mechanism and the new
bpf_try_to_free_mem_cgroup_pages() kfunc, enable BPF programs to perform
proactive background reclaim without blocking the charge path. The
pattern works as follows: the memcg_charged callback tracks accumulated
memory usage; when usage crosses a configurable threshold, it enqueues an
asynchronous work item via bpf_wq_start() and returns immediately without
throttling the charging task. The workqueue callback then invokes
bpf_try_to_free_mem_cgroup_pages() to reclaim pages from the target
cgroup; if usage remains elevated after reclaim, the callback re-enqueues
itself to continue. This allows a BPF program to keep a cgroup's
footprint below its hard limit (memory.max) entirely in the background,
avoiding the OOM killer or direct-reclaim stalls that would otherwise
occur. The selftest for this feature (patch 10/11) validates the
mechanism concretely: a workload that writes and mmaps a 64 MB file inside
a 32 MB cgroup reliably triggers memory.events "max" events without BPF;
with the async reclaim program attached, the "max" counter does not
increase at all across the same workload.
In this patch series, I've incorporated a portion of Roman's patch in
[1] to ensure the entire series can be compiled cleanly with bpf-next.
Patch Breakdown:
Patches 1-4 are from Roman Gushchin's series [1], included here to
provide the necessary BPF infrastructure for attaching struct_ops to
cgroups.
Patches 5-11 are the new work in this series:
05/11 bpf: Pass flags in bpf_link_create for struct_ops
Stores attr->link_create.flags in struct bpf_struct_ops_link
and extends the validation to allow BPF_F_ALLOW_OVERRIDE.
Also updates the UAPI comment to reflect that cgroup-bpf attach
flags now apply to BPF_LINK_CREATE in addition to
BPF_PROG_ATTACH.
06/11 mm: memcontrol: Add BPF struct_ops for memory controller
The core feature patch. Introduces the memcg_bpf_ops struct_ops
type with the following hooks:
- memcg_charged(memcg, batch): called on the synchronous charge
path. Returns a throttling delay in milliseconds; used as a
lower bound for __mem_cgroup_handle_over_high(), effective
even when memory.high is not breached.
- memcg_uncharged(memcg, batch): called on uncharge, allowing
BPF programs to track memory releases.
- below_low(memcg, elow, usage): overrides the memory.low
protection check. Returns true to treat the cgroup as
protected regardless of the elow >= usage comparison.
- below_min(memcg, emin, usage): same as below_low but for
memory.min protection.
- handle_cgroup_online/offline(memcg): lifecycle callbacks for
per-cgroup state management in BPF programs.
BPF_F_ALLOW_OVERRIDE is supported: when a program is registered
with this flag, descendant cgroups may attach their own
memcg_bpf_ops to override the inherited policy. Registration
propagates ops down through the subtree via mem_cgroup_iter;
unregistration restores each descendant to the ops its
registering ancestor's parent held, correctly preserving
override chains.
07/11 mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
Exposes try_to_free_mem_cgroup_pages() to BPF programs as a
KF_SLEEPABLE kfunc. A swappiness parameter controls the
override value passed to the core reclaim path
(effective only when MEMCG_RECLAIM_PROACTIVE is set in
reclaim_options).
08/11 selftests/bpf: Add tests for memcg_bpf_ops
Adds prog_tests/memcg_ops.c covering three scenarios:
memcg_charged-only throttling, below_low + memcg_charged
interaction, and below_min + memcg_charged interaction. A
tracepoint on memcg:count_memcg_events (PGFAULT) is used to
detect memory pressure and trigger hooks accordingly.
09/11 selftests/bpf: Add test for memcg_bpf_ops hierarchies
Validates BPF_F_ALLOW_OVERRIDE attachment semantics across a
three-level cgroup hierarchy: attach with ALLOW_OVERRIDE at the
root, override at the middle level without the flag, then assert
that attaching to the leaf correctly fails with -EBUSY.
10/11 selftests/bpf: Add selftest for memcg async reclaim via BPF
Demonstrates and validates asynchronous memory reclaim: a BPF
program uses the memcg_charged/memcg_uncharged hooks to track
accumulated usage and, when a threshold is exceeded, enqueues a
bpf_wq_start() workqueue item that calls
bpf_try_to_free_mem_cgroup_pages() without blocking the charge
path. The test asserts that with the BPF program active,
memory.events "max" events do not increase under a workload
that would otherwise exceed the hard limit.
11/11 samples/bpf: Add memcg priority control and async reclaim example
Adds a complete sample (samples/bpf/memcg.bpf.c + memcg.c)
demonstrating both features. The BPF side monitors PGFAULT
events on a high-priority cgroup; when the per-second fault
count crosses a configurable threshold, it activates below_low
or below_min protection for the high-priority cgroup and/or
applies a charge delay to the low-priority cgroup. Six
struct_ops variants are exported so userspace can attach only
the hooks needed. Async reclaim is optionally combined with
priority throttling via a shared low-cgroup ops map.
Test Environment:
The following examples run on x86_64 QEMU (10 CPUs, 2 GB RAM), using
a tmpfs-backed file on the host as a swap device to reduce I/O impact.
Two cgroups are created -- high (high-priority) and low (low-priority)
-- and each test runs two concurrent stress-ng workloads, one per
cgroup, each requesting 3 GB of memory.
# mkdir /sys/fs/cgroup/high /sys/fs/cgroup/low
# free -h
total used free shared buff/cache available
Mem: 1.9Gi 317Mi 1.6Gi 1.0Mi 144Mi 1.6Gi
Swap: 4.0Gi 0B 4.0Gi
Baseline: no memory priority policy:
Both cgroups run without any reclaim protection. Results are roughly
equal, as expected:
cgroup bogo ops/s
high 4,979
low 4,927
Test 1: memory.low protection:
Setting memory.low on the high-priority cgroup protects it from
reclaim, at the cost of pushing reclaim pressure onto the low-priority
cgroup:
# echo $((3 * 1024 * 1024 * 1024)) > /sys/fs/cgroup/high/memory.low
cgroup bogo ops/s
high 450,290
low 11,307
The high-priority cgroup benefits significantly, but memory.low relies
on static usage thresholds and cannot adapt to actual workload
behavior.
Test 2: memory.low with an idle high-priority task:
Here the high-priority cgroup runs a Python script that allocates 3 GB
and then sleeps, simulating a low-activity but memory-holding workload.
Because the process is idle, it generates no page faults and does not
actively use its memory. Yet memory.low still protects it, continuing
to suppress the low-priority cgroup's performance:
cgroup bogo ops/s
low 14,757
The low-priority cgroup remains significantly throttled despite the
high-priority cgroup being effectively idle -- a clear limitation of
static memory.low control.
Test 3: memcg eBPF -- dynamic priority control:
memcg is a sample program introduced in this patch series
(samples/bpf/memcg.c + memcg.bpf.c). It loads a BPF program that
monitors PGFAULT events in the high-priority cgroup. When the
per-second fault count exceeds a configured threshold, the hook
activates below_min protection for one second; otherwise the cgroup
receives no special treatment.
# ./memcg --low_path=/sys/fs/cgroup/low \
--high_path=/sys/fs/cgroup/high \
--threshold=1 --use_below_min
Successfully attached!
3a. Both cgroups under active memory pressure:
When both cgroups run stress-ng, the high-priority cgroup generates
frequent page faults and the BPF hook activates protection, matching
the behavior of memory.low:
cgroup bogo ops/s
high 404,392
low 11,404
3b. High-priority cgroup is idle (Python + sleep):
Because the sleeping Python process generates no page faults, the BPF
hook never activates, and the low-priority cgroup is free to reclaim
memory normally:
cgroup bogo ops/s
low 551,083
This is a ~37x improvement over the equivalent memory.low scenario
(Test 2), demonstrating that eBPF-driven dynamic control can
accurately reflect actual workload activity and avoid unnecessary
protection of idle high-priority tasks.
Summary:
Scenario low-cgroup bogo ops/s
Baseline (no policy) ~4,927
memory.low, both active ~11,307
memory.low, high idle ~14,757
memcg eBPF, both active ~11,404
memcg eBPF, high idle ~551,083
References:
[1] https://patchew.org/linux/20260127024421.494929-1-roman.gushchin@linux.dev/
Changelog:
v7:
Change base commits of "mm: BPF OOM" to v3.
Some fixes according to the comments of bpf-ci.
Rename get_high_delay_ms hook to memcg_charged; add memcg_uncharged
hook for tracking uncharge events.
Update below_low and below_min hooks to receive elow/emin and usage
as explicit arguments.
Add bpf_try_to_free_mem_cgroup_pages kfunc to expose cgroup reclaim
to BPF programs.
Add selftest for BPF-driven asynchronous page reclaim.
Extend samples/bpf/memcg to support async reclaim in addition to
priority throttling.
v6:
Based on the bot+bof-ci comments, fixed the following issues.
Added fast-path check with unlikely() before SRCU lock acquisition to
optimize the no-BPF case in BPF_MEMCG_CALL.
Add missing newline in pr_warn message to bpf_memcontrol_init.
Added comprehensive child process exit status checking with WIFEXITED()
and WEXITSTATUS(), and added zombie process prevention in
real_test_memcg_ops.
Changed malloc() to calloc() for BSS data allocation in all test
functions and samples main function.
Change srcu_read_lock(&memcg_bpf_srcu) to
lockdep_assert_held(&cgroup_mutex) in function memcontrol_bpf_online
and memcontrol_bpf_offline.
v5:
Based on the bot+bof-ci comments, fixed the following issues.
Fixed issues in memcg_ops.c and memcg.bpf.c by moving variable
declaration to the beginning of need_threshold() function.
The 'u64 current_ts' variable must be declared before any
executable statements
Improved input validation in samples/bpf/memcg.c by adding a new
parse_u64() helper function. This function properly handles errors
from strtoull() and provides better error messages when parsing
threshold and over_high_ms command-line arguments.
Move check for prog->sleepable after validating member offsets in
mm/bpf_memcontrol.c bpf_memcg_ops_check_member.
Fixed sscanf return value checking in prog_tests/memcg_ops.c.
Changed the condition from 'sscanf() < 0' to 'sscanf() != 1' because
sscanf returns the number of successfully matched items, not a negative
value on error. This makes the test more reliable when reading timing
data from temporary files.
v4:
Fix the issues according to the comments from bot+bof-ci.
According to JP Kobryn's comments, move exit(0) from
real_test_memcg_ops_child_work to real_test_memcg_ops.
Fix issues in the bpf_memcg_ops_reg function.
v3:
According to the comments from Michal Koutný and Chen Ridong, update hooks
to get_high_delay_ms, below_low, below_min, handle_cgroup_online, and
handle_cgroup_offline.
According to Michal Koutný's comments, add BPF_F_ALLOW_OVERRIDE
support to memcg_bpf_ops.
v2:
According to Tejun Heo's comments, rebased on Roman Gushchin's BPF
OOM patch series [1] and added hierarchical delegation support.
According to the comments from Roman Gushchin and Michal Hocko, designed
concrete use case scenarios and provided test results.
Hui Zhu (7):
bpf: Pass flags in bpf_link_create for struct_ops
mm: memcontrol: Add BPF struct_ops for memory controller
mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
selftests/bpf: Add tests for memcg_bpf_ops
selftests/bpf: Add test for memcg_bpf_ops hierarchies
selftests/bpf: Add selftest for memcg async reclaim via BPF
samples/bpf: Add memcg priority control and async reclaim example
Roman Gushchin (4):
bpf: move bpf_struct_ops_link into bpf.h
bpf: allow attaching struct_ops to cgroups
libbpf: fix return value on memory allocation failure
libbpf: introduce bpf_map__attach_struct_ops_opts()
MAINTAINERS | 6 +
include/linux/bpf-cgroup-defs.h | 3 +
include/linux/bpf-cgroup.h | 16 +
include/linux/bpf.h | 10 +
include/linux/memcontrol.h | 250 ++++++-
include/uapi/linux/bpf.h | 5 +-
kernel/bpf/bpf_struct_ops.c | 67 +-
kernel/bpf/cgroup.c | 46 ++
mm/bpf_memcontrol.c | 355 +++++++++-
mm/memcontrol.c | 43 +-
samples/bpf/.gitignore | 1 +
samples/bpf/Makefile | 8 +-
samples/bpf/memcg.bpf.c | 380 +++++++++++
samples/bpf/memcg.c | 411 ++++++++++++
tools/include/uapi/linux/bpf.h | 3 +-
tools/lib/bpf/libbpf.c | 22 +-
tools/lib/bpf/libbpf.h | 14 +
tools/lib/bpf/libbpf.map | 1 +
tools/testing/selftests/bpf/cgroup_helpers.c | 41 ++
tools/testing/selftests/bpf/cgroup_helpers.h | 2 +
.../bpf/prog_tests/memcg_async_reclaim.c | 333 +++++++++
.../selftests/bpf/prog_tests/memcg_ops.c | 634 ++++++++++++++++++
.../selftests/bpf/progs/memcg_async_reclaim.c | 203 ++++++
tools/testing/selftests/bpf/progs/memcg_ops.c | 132 ++++
24 files changed, 2952 insertions(+), 34 deletions(-)
create mode 100644 samples/bpf/memcg.bpf.c
create mode 100644 samples/bpf/memcg.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_ops.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops.c
--
2.43.0
^ permalink raw reply
* [RFC PATCH bpf-next v7 01/11] bpf: move bpf_struct_ops_link into bpf.h
From: Hui Zhu @ 2026-05-26 2:20 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: geliang, baohua, Matt Bobrowski, Yafang Shao
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
From: Roman Gushchin <roman.gushchin@linux.dev>
Move struct bpf_struct_ops_link's definition into bpf.h,
where other custom bpf links definitions are.
It's necessary to access its members from outside of generic
bpf_struct_ops implementation, which will be done by following
patches in the series.
Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
Acked-by: Matt Bobrowski <mattbobrowski@google.com>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
---
include/linux/bpf.h | 6 ++++++
kernel/bpf/bpf_struct_ops.c | 6 ------
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 1b28cacc3075..01c0bf5a9cd0 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1908,6 +1908,12 @@ struct bpf_raw_tp_link {
u64 cookie;
};
+struct bpf_struct_ops_link {
+ struct bpf_link link;
+ struct bpf_map __rcu *map;
+ wait_queue_head_t wait_hup;
+};
+
struct bpf_link_primer {
struct bpf_link *link;
struct file *file;
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 521cb9d7e8c7..cf3c604d48ef 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -55,12 +55,6 @@ struct bpf_struct_ops_map {
struct bpf_struct_ops_value kvalue;
};
-struct bpf_struct_ops_link {
- struct bpf_link link;
- struct bpf_map __rcu *map;
- wait_queue_head_t wait_hup;
-};
-
static DEFINE_MUTEX(update_mutex);
#define VALUE_PREFIX "bpf_struct_ops_"
--
2.43.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox