* Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask
From: Nhat Pham @ 2026-05-30 18:21 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
baoquan.he, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <CAKEwX=O-_OZ8x0UC96a_k+0eZfAE+mWMWDdn68uy1LHRq=JC0w@mail.gmail.com>
On Sat, May 30, 2026 at 10:51 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
>
> How expensive is it to add per-cpu caching for each device :(
to clarify - a percpu_swap_cluster per si for every si.
>
... or for each tier (assuming devices in each tier share the same
performance characteristics, and could be used interchangeably?).
Basically:
struct percpu_swap_cluster {
struct swap_info_struct *si[MAX_SWAPTIER][SWAP_NR_ORDERS];
unsigned long offset[MAX_SWAPTIER][SWAP_NR_ORDERS];
local_lock_t lock;
};
Seems like 4 is the default number of tier right? So the extra
overhead is just (nr cpu) * 10 * 3 * (sizeof(unsigned long) +
sizeof(*ptr)) or wev?
^ permalink raw reply
* Re: [PATCH v7 0/4] mm: swap: introduce swap tier infrastructure
From: Nhat Pham @ 2026-05-30 18:02 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
baoquan.he, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260527062247.3440692-1-youngjun.park@lge.com>
On Tue, May 26, 2026 at 11:23 PM Youngjun Park <youngjun.park@lge.com> wrote:
>
> This is v7 of the swap tier series addressing review feedback.
> The cover letter has been simplified.
>
> I revisited the design (see Design Rationale). Since our use case
> fits best with a memcg-based model, the implementation remains
> within memcg and preserves its resource accounting semantics.
>
> Alternatives considered:
>
> 1. A separate sysfs interface under swap. (Workable. But, it would still
> need to reference memcg paths, and fully decoupling it would add
> swap-layer logic to manage memcgs, making it secondary option.)
>
> 2. Making the feature non-default.
>
> Other interfaces were also reviewed. Aside from sysfs and BPF,
> the options involve trade-offs and are largely design choices.
> BPF was excluded due to possible disablement on our embedded
> platform, though future extension remains possible.
>
> Overview
> ========
>
> Swap Tiers group swap devices into performance classes (e.g. NVMe,
> HDD, Network) and allow per-memcg selection of which tiers to use.
> This mechanism was suggested by Chris Li.
>
> Design Rationale
> ================
>
> Swap tier selection is attached to memcg. A child cgroup may select a
> subset of the parent's allowed tiers.
>
> This
> - Preserves cgroup inheritance semantics (boundary at parent,
> refinement at child).
> - Reuses memcg, which already groups processes and enforces
> hierarchical memory limits.
> - Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
> - Avoids introducing a parallel swap control hierarchy.
>
> Placing tier control outside memcg (e.g., via BPF, syscalls, or
> madvise) would allow swap preference to diverge from the memcg
> hierarchy. Integrating it into memcg keeps the swap policy
> consistent with existing memory ownership semantics. There are
> also real use cases built around memcg.
>
> In the future, this can be extended to other interfaces to cover
> additional use cases.
>
> I believe a memcg-based swap control is a good starting point
> before such extensions.
>
> Use Cases
> =========
>
> #1: Latency separation (our primary deployment scenario)
> [ / ]
> |
> +-- latency-sensitive workload (fast tier)
> +-- background workload (slow tier)
>
> The parent defines the memory boundary.
> Each workload selects a swap tier via memory.swap.tiers according to
> latency requirements.
>
> This prevents latency-sensitive workloads from being swapped to
> slow devices used by background workloads.
>
> #2: Per-VM swap selection (Chris Li's deployment scenario)
> [ / ]
> |
> +-- [ Job on VM ] (tiers: zswap, SSD)
> |
> +-- [ VMM guest memory ] (tiers: SSD)
>
> The parent (job) has access to both zswap and SSD tiers.
> The child (VMM guest memory) selects SSD as its swap tier via
> memory.swap.tiers. In this deployment, swap device selection
> happens at the child level from the parent's available set.
>
> #3: Tier isolation for reduced contention (hypothetical)
> [ / ] (tiers: A, B)
> |
> +-- workload X (tiers: A)
> +-- workload Y (tiers: B)
>
> Each child uses a different tier. Since swap paths are separated
> per tier, synchronization overhead between the two workloads is
> reduced.
>
> Future extension
> ================
>
> #1: Intra-tier distribution policy:
> Currently, swap devices with the same priority are allocated in a
> round-robin fashion. Per-tier policy files under
> /sys/kernel/mm/swap/tiers/ can control how devices within a tier
> are selected (e.g. round-robin, weighted).
>
> #2: Inter-tier promotion and demotion:
> Promotion and demotion apply between tiers, not within a single
> tier. The current interface defines only tier assignment; it does
> not yet define when or how pages move between tiers. Two triggering
> models are possible:
>
> (a) User-triggered: userspace explicitly initiates migration between
> tiers (e.g. via a new interface or existing move_pages semantics).
> (b) Kernel-triggered: the kernel moves pages between tiers at
> appropriate points such as reclaim or refault.
>
> #3: Per-VMA, per-process swap and BPF:
> Not just for memcg based swap, possible to extend Per-VMA or per-process swap.
> Or we can use it as BPF program.
>
> Experimentation
> ===============
>
> Tested on our internal platform using NBD as a separate swap tier.
> Our first production's simple usecase.
>
> Without tiers:
> - No selective control over flash wear
> - Cannot selectively assign NBD to specific applications
>
> Cold launch improvement (preloaded vs. baseline):
> - App A: 13.17s -> 4.18s (68%)
> - App B: 5.60s -> 1.12s (80%)
> - App C: 10.25s -> 2.00s (80%)
>
> Performance impact with no tiers configured:
> <1% regression in kernel build and vm-scalability benchmarks
>
Bit late to the party - working on my review backlog right now :)
I see some parallels with this and memory tiering work being done. One
future line of work could be considering how to ensure fairness when
multiple cgroups share same tiers:
https://lwn.net/Articles/1073400/
I can imagine a scenario where one noisy neighbor eagerly swaps first
and occupy all the space in the faster tier(s), pushing the other
colocated tenants to the slower tier(s). We might need to figure out a
way to ensure fairness here (while letting cgroups occupy fast swap
backends opportunistically if there is no resources scarcity).
^ permalink raw reply
* Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask
From: Nhat Pham @ 2026-05-30 17:51 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
baoquan.he, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260527062247.3440692-5-youngjun.park@lge.com>
On Tue, May 26, 2026 at 11:23 PM Youngjun Park <youngjun.park@lge.com> wrote:
>
> Apply memcg tier effective mask during swap slot allocation to
> enforce per-cgroup swap tier restrictions.
>
> In the fast path, check the percpu cached swap_info's tier_mask
> against the folio's effective mask. If it does not match, fall
> through to the slow path. In the slow path, skip swap devices
> whose tier_mask is not covered by the folio's effective mask.
>
> This works correctly when there is only one non-rotational
> device in the system and no devices share the same priority.
> However, there are known limitations:
>
> - When non-rotational devices are distributed across multiple
> tiers, and different memcgs are configured to use those
> distinct tiers, they may constantly overwrite the shared
> percpu swap cache. This cache thrashing leads to frequent
> fast path misses.
>
> - Combined with the above issue, if same-priority devices exist
> among them, a percpu cache miss (overwritten by another memcg)
> forces the allocator to round-robin to the next device
> prematurely, even if the current cluster is not fully
> exhausted.
I had very similar issues when I tried hacking vswap on top of swap
table too... It's even worse over there because it's not just
performance - vswap needs special handling in certain cases, and in
some places cannot be used at all (for e.g in zswap writeback). I
ended up having to add separate caching for vswap device:
https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
How expensive is it to add per-cpu caching for each device :(
Anyway, as a first step, this LGTM. Reviewing from swap's mechanism
perspective, and leaving the cgroup side to memcg folks:
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
^ permalink raw reply
* Re: [PATCH v5 8/9] mm: memory: flatten alloc_anon_folio() retry loop
From: Dev Jain @ 2026-05-30 9:06 UTC (permalink / raw)
To: Johannes Weiner, Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Lance Yang, Nico Pache, Ryan Roberts,
cgroups, linux-mm, linux-kernel
In-Reply-To: <20260527204757.2544958-9-hannes@cmpxchg.org>
On 28/05/26 2:15 am, Johannes Weiner wrote:
> alloc_anon_folio() uses a top-level if (folio) that buries the success
> path four levels deep. This makes for awkward long lines and wrapping.
> The next patch will add more code here, so flatten this now to keep
> things clean and simple.
>
> The next label is already there, use it for !folio.
>
> No functional change intended.
>
> Suggested-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> Acked-by: Usama Arif <usama.arif@linux.dev>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> ---
Reviewed-by: Dev Jain <dev.jain@arm.com>
> mm/memory.c | 34 +++++++++++++++++-----------------
> 1 file changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 7c020995eafc..135f5c0f57bd 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -5215,24 +5215,24 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
> while (orders) {
> addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
> folio = vma_alloc_folio(gfp, order, vma, addr);
> - if (folio) {
> - if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
> - count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
> - folio_put(folio);
> - goto next;
> - }
> - folio_throttle_swaprate(folio, gfp);
> - /*
> - * When a folio is not zeroed during allocation
> - * (__GFP_ZERO not used) or user folios require special
> - * handling, folio_zero_user() is used to make sure
> - * that the page corresponding to the faulting address
> - * will be hot in the cache after zeroing.
> - */
> - if (user_alloc_needs_zeroing())
> - folio_zero_user(folio, vmf->address);
> - return folio;
> + if (!folio)
> + goto next;
> + if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
> + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
> + folio_put(folio);
> + goto next;
> }
> + folio_throttle_swaprate(folio, gfp);
> + /*
> + * When a folio is not zeroed during allocation
> + * (__GFP_ZERO not used) or user folios require special
> + * handling, folio_zero_user() is used to make sure
> + * that the page corresponding to the faulting address
> + * will be hot in the cache after zeroing.
> + */
> + if (user_alloc_needs_zeroing())
> + folio_zero_user(folio, vmf->address);
> + return folio;
> next:
> count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
> order = next_order(&orders, order);
^ permalink raw reply
* [PATCH v4] cgroup/dmem: implement dmem.high soft limit via prioritized eviction
From: Qiliang Yuan @ 2026-05-30 7:35 UTC (permalink / raw)
To: Christian Koenig, Huang Rui, Matthew Auld, Matthew Brost,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, David Airlie,
Simona Vetter, Tejun Heo, Johannes Weiner, Michal Koutný,
Natalie Vock
Cc: dri-devel, linux-kernel, cgroups, Qiliang Yuan
The dmem cgroup v2 controller currently only provides a hard "max"
limit, which causes immediate allocation failures when a cgroup's
device memory usage reaches its quota. GPU-bound AI workloads need
smoother over-subscription support: a soft limit that temporarily
allows excess usage while applying backpressure through reclaim
rather than outright failure.
Add dmem.high, a soft limit that penalizes over-limit cgroups by
evicting their buffer objects first when eviction is triggered (e.g.
due to a "max" limit hit). Unlike the rejected v1 approach which
used sleep-on-allocation throttling, this version provides a
meaningful recovery action through prioritized reclaim.
Expose "high" as a new cgroupfs control file per region via
set_resource_high() and get_resource_high(), and initialize it to
PAGE_COUNTER_MAX in reset_all_resource_limits(). Like get_resource_max(),
get_resource_high() returns PAGE_COUNTER_MAX when the pool is NULL.
Extend dmem_cgroup_state_evict_valuable() with a "try_high"
parameter. When set, the function evaluates the try_high condition
first (before the limit_pool == test_pool shortcut) so that even the
limit-hitting cgroup's own BOs are filtered by the high threshold.
It then walks the page_counter parent chain to check whether any
ancestor exceeds its high limit, and verifies that the pool is above
its effective minimum to respect dmem.min protection.
Refactor ttm_bo_evict_alloc() into a 3-pass eviction strategy.
Pass 1 uses a blocking lock and targets only BOs whose cgroup exceeds
dmem.high, ensuring over-limit cgroups are penalized even when their
BOs are actively in use. Pass 2 falls back to the standard above-elow
trylock eviction. Pass 3+ uses proper locking and repeats while
making progress with the existing low-watermark fallback.
Signed-off-by: Qiliang Yuan <realwujing@gmail.com>
---
Introduce a "high" soft limit for the dmem cgroup v2 controller.
When a "max" limit is hit and eviction is triggered, buffer objects
belonging to cgroups that exceed their dmem.high limit are targeted
first, providing a meaningful recovery action through reclaim.
The dmem cgroup currently only supports hard "max" limits, which
cause immediate allocation failures for GPU-bound workloads. A soft
limit enables smoother over-subscription by penalizing over-limit
cgroups via prioritized eviction rather than outright rejection.
The implementation adds a "high" cgroupfs control file per region,
a try_high parameter to dmem_cgroup_state_evict_valuable() for
tier-1 eviction, and a 3-pass strategy in ttm_bo_evict_alloc().
---
V3 -> V4:
- Use a blocking lock in Pass 1 instead of trylock to ensure
over-limit cgroups are penalized even when their BOs are actively
in use, as requested by Maarten Lankhorst.
- Evaluate the try_high condition before the limit_pool == test_pool
early-return so that the limit-hitting cgroup's own BOs are also
filtered by dmem.high.
- Remove the high-priority compensation retry at the start of Pass 3,
which is no longer needed now that Pass 1 uses a blocking lock.
V2 -> V3:
- Walk the page_counter parent chain in the try_high pass to prevent
child cgroups from evading the penalty when a parent cgroup exceeds
its dmem.high limit.
- Check dmem.min protection in the try_high pass to avoid evicting
BOs below the effective minimum.
- Add a properly-locked high-priority retry at the beginning of Pass 3
so that actively-used over-limit BOs (which failed trylock in Pass 1)
are not skipped while innocent cgroups are evicted.
- Fix get_resource_high(NULL) returning 0 instead of PAGE_COUNTER_MAX
to match the behavior of get_resource_max().
V1 -> V2:
- Replace sleep-on-allocation throttling with prioritized eviction.
When a "max" limit is hit, BOs from cgroups exceeding dmem.high are
evicted first in a dedicated pass. No throttling or sleeping is
performed in the charge path.
- Remove task throttling (schedule_timeout_killable, TIF_NOTIFY_RESUME,
resume_user_mode_work() integration) entirely.
- Add dmem.high cgroupfs control file per region.
- Extend dmem_cgroup_state_evict_valuable() with try_high parameter
to target over-limit cgroups as tier-1 eviction.
- Refactor ttm_bo_evict_alloc() into a 3-pass eviction strategy:
(1) trylock: evict only BOs exceeding dmem.high
(2) trylock: above-elow
(3) proper-lock: repeat with low fallback.
- Initialize high to PAGE_COUNTER_MAX in reset_all_resource_limits().
v3: https://lore.kernel.org/r/20260528-feature-dmem-high-v3-1-c642b34bcb2f@gmail.com
v2: https://lore.kernel.org/r/20260522-feature-dmem-high-v2-1-1d7d4a0fa5da@gmail.com
v1: https://lore.kernel.org/all/20260520-feature-dmem-high-v1-1-97ca0cb7f95a@gmail.com
---
drivers/gpu/drm/ttm/ttm_bo.c | 32 +++++++++++++----
include/linux/cgroup_dmem.h | 4 +--
kernel/cgroup/dmem.c | 81 +++++++++++++++++++++++++++++++++++++++++---
3 files changed, 104 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index bcd76f6bb7f02..bf06e9e4b18a3 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -505,6 +505,8 @@ struct ttm_bo_evict_walk {
/** @limit_pool: Which pool limit we should test against */
struct dmem_cgroup_pool_state *limit_pool;
+ /** @try_high: Whether to only evict BO's above the high watermark (first pass) */
+ bool try_high;
/** @try_low: Whether we should attempt to evict BO's with low watermark threshold */
bool try_low;
/** @hit_low: If we cannot evict a bo when @try_low is false (first pass) */
@@ -518,7 +520,8 @@ static s64 ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *
s64 lret;
if (!dmem_cgroup_state_evict_valuable(evict_walk->limit_pool, bo->resource->css,
- evict_walk->try_low, &evict_walk->hit_low))
+ evict_walk->try_high, evict_walk->try_low,
+ &evict_walk->hit_low))
return 0;
if (bo->pin_count || !bo->bdev->funcs->eviction_valuable(bo, evict_walk->place))
@@ -577,31 +580,46 @@ static int ttm_bo_evict_alloc(struct ttm_device *bdev,
};
s64 lret;
- evict_walk.walk.arg.trylock_only = true;
+ /*
+ * Pass 1 (blocking, high-priority): Evict only BOs whose cgroup
+ * exceeds its dmem.high soft limit. A blocking lock is used to
+ * ensure over-limit cgroups are penalized even when their BOs are
+ * actively in use.
+ */
+ evict_walk.walk.arg.trylock_only = false;
+ evict_walk.try_high = true;
lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
+ evict_walk.try_high = false;
+ if (lret)
+ goto out;
- /* One more attempt if we hit low limit? */
+ /*
+ * Pass 2 (trylock): Evict BOs above the effective low watermark.
+ * Falls back to low-priority eviction if needed.
+ */
+ lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
if (!lret && evict_walk.hit_low) {
evict_walk.try_low = true;
lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
}
+
if (lret || !ticket)
goto out;
- /* Reset low limit */
+ /*
+ * Pass 3+ (properly locked): Evict while making progress.
+ * Reset flags and retry with try_low if we hit the low watermark.
+ */
evict_walk.try_low = evict_walk.hit_low = false;
- /* If ticket-locking, repeat while making progress. */
evict_walk.walk.arg.trylock_only = false;
retry:
do {
- /* The walk may clear the evict_walk.walk.ticket field */
evict_walk.walk.arg.ticket = ticket;
evict_walk.evicted = 0;
lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
} while (!lret && evict_walk.evicted);
- /* We hit the low limit? Try once more */
if (!lret && evict_walk.hit_low && !evict_walk.try_low) {
evict_walk.try_low = true;
goto retry;
diff --git a/include/linux/cgroup_dmem.h b/include/linux/cgroup_dmem.h
index dd4869f1d736e..06115d35509b1 100644
--- a/include/linux/cgroup_dmem.h
+++ b/include/linux/cgroup_dmem.h
@@ -23,7 +23,7 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,
void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64 size);
bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
struct dmem_cgroup_pool_state *test_pool,
- bool ignore_low, bool *ret_hit_low);
+ bool try_high, bool ignore_low, bool *ret_hit_low);
void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool);
#else
@@ -54,7 +54,7 @@ static inline void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64
static inline
bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
struct dmem_cgroup_pool_state *test_pool,
- bool ignore_low, bool *ret_hit_low)
+ bool try_high, bool ignore_low, bool *ret_hit_low)
{
return true;
}
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 4753a67d0f0f2..f81fbb538cf2f 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -156,6 +156,12 @@ set_resource_low(struct dmem_cgroup_pool_state *pool, u64 val)
page_counter_set_low(&pool->cnt, val);
}
+static void
+set_resource_high(struct dmem_cgroup_pool_state *pool, u64 val)
+{
+ page_counter_set_high(&pool->cnt, val);
+}
+
static void
set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val)
{
@@ -167,6 +173,11 @@ static u64 get_resource_low(struct dmem_cgroup_pool_state *pool)
return pool ? READ_ONCE(pool->cnt.low) : 0;
}
+static u64 get_resource_high(struct dmem_cgroup_pool_state *pool)
+{
+ return pool ? READ_ONCE(pool->cnt.high) : PAGE_COUNTER_MAX;
+}
+
static u64 get_resource_min(struct dmem_cgroup_pool_state *pool)
{
return pool ? READ_ONCE(pool->cnt.min) : 0;
@@ -186,6 +197,7 @@ static void reset_all_resource_limits(struct dmem_cgroup_pool_state *rpool)
{
set_resource_min(rpool, 0);
set_resource_low(rpool, 0);
+ set_resource_high(rpool, PAGE_COUNTER_MAX);
set_resource_max(rpool, PAGE_COUNTER_MAX);
}
@@ -289,10 +301,13 @@ dmem_cgroup_calculate_protection(struct dmem_cgroup_pool_state *limit_pool,
* dmem_cgroup_state_evict_valuable() - Check if we should evict from test_pool
* @limit_pool: The pool for which we hit limits
* @test_pool: The pool for which to test
+ * @try_high: Only evict BOs whose usage exceeds the high limit (first pass)
* @ignore_low: Whether we have to respect low watermarks.
* @ret_hit_low: Pointer to whether it makes sense to consider low watermark.
*
* This function returns true if we can evict from @test_pool, false if not.
+ * When @try_high is set, only pools with usage above their high limit are
+ * evictable, enabling prioritized eviction of over-limit cgroups.
* When returning false and @ignore_low is false, @ret_hit_low may
* be set to true to indicate this function can be retried with @ignore_low
* set to true.
@@ -301,12 +316,56 @@ dmem_cgroup_calculate_protection(struct dmem_cgroup_pool_state *limit_pool,
*/
bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
struct dmem_cgroup_pool_state *test_pool,
- bool ignore_low, bool *ret_hit_low)
+ bool try_high, bool ignore_low, bool *ret_hit_low)
{
struct dmem_cgroup_pool_state *pool = test_pool;
struct page_counter *ctest;
u64 used, min, low;
+ ctest = &test_pool->cnt;
+ used = page_counter_read(ctest);
+
+ if (try_high) {
+ /*
+ * When the limit-hitting cgroup's own BOs are being
+ * considered, only evict them if their pool exceeds its
+ * own dmem.high limit. No ancestry check is needed
+ * because the limit was triggered by this pool itself.
+ */
+ if (limit_pool == test_pool)
+ return used > READ_ONCE(ctest->high);
+
+ {
+ struct page_counter *c;
+
+ /*
+ * Walk the page_counter parent chain to check
+ * whether any ancestor cgroup exceeds its
+ * dmem.high limit. This prevents child cgroups
+ * from evading the penalty when a parent cgroup
+ * is over its high limit.
+ */
+ if (used <= READ_ONCE(ctest->high)) {
+ for (c = ctest->parent; c; c = c->parent) {
+ if (page_counter_read(c) >
+ READ_ONCE(c->high))
+ break;
+ }
+ if (!c)
+ return false;
+ }
+ }
+
+ /*
+ * Respect dmem.min protection: do not evict BOs below the
+ * effective minimum even during the high-priority pass.
+ */
+ dmem_cgroup_calculate_protection(limit_pool, test_pool);
+ min = READ_ONCE(ctest->emin);
+
+ return used > min;
+ }
+
/* Can always evict from current pool, despite limits */
if (limit_pool == test_pool)
return true;
@@ -329,11 +388,8 @@ bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
{}
}
- ctest = &test_pool->cnt;
-
dmem_cgroup_calculate_protection(limit_pool, test_pool);
- used = page_counter_read(ctest);
min = READ_ONCE(ctest->emin);
if (used <= min)
@@ -835,6 +891,17 @@ static ssize_t dmem_cgroup_region_low_write(struct kernfs_open_file *of,
return dmemcg_limit_write(of, buf, nbytes, off, set_resource_low);
}
+static int dmem_cgroup_region_high_show(struct seq_file *sf, void *v)
+{
+ return dmemcg_limit_show(sf, v, get_resource_high);
+}
+
+static ssize_t dmem_cgroup_region_high_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ return dmemcg_limit_write(of, buf, nbytes, off, set_resource_high);
+}
+
static int dmem_cgroup_region_max_show(struct seq_file *sf, void *v)
{
return dmemcg_limit_show(sf, v, get_resource_max);
@@ -868,6 +935,12 @@ static struct cftype files[] = {
.seq_show = dmem_cgroup_region_low_show,
.flags = CFTYPE_NOT_ON_ROOT,
},
+ {
+ .name = "high",
+ .write = dmem_cgroup_region_high_write,
+ .seq_show = dmem_cgroup_region_high_show,
+ .flags = CFTYPE_NOT_ON_ROOT,
+ },
{
.name = "max",
.write = dmem_cgroup_region_max_write,
---
base-commit: ab5fce87a778cb780a05984a2ca448f2b41aafbf
change-id: 20260519-feature-dmem-high-16997148dc38
Best regards,
--
Qiliang Yuan <realwujing@gmail.com>
^ permalink raw reply related
* [tj-cgroup:for-7.2] BUILD SUCCESS 390f2d73bc99a888469f789f274c162da33bafe5
From: kernel test robot @ 2026-05-30 6:57 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-7.2
branch HEAD: 390f2d73bc99a888469f789f274c162da33bafe5 cgroup/cpuset: Free sched domains on rebuild guard failure
elapsed time: 734m
configs tested: 164
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc defconfig gcc-15.2.0
arc randconfig-001-20260530 gcc-14.3.0
arc randconfig-002-20260530 gcc-14.3.0
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm defconfig gcc-15.2.0
arm multi_v7_defconfig gcc-15.2.0
arm randconfig-001-20260530 gcc-14.3.0
arm randconfig-002-20260530 gcc-14.3.0
arm randconfig-003-20260530 gcc-14.3.0
arm randconfig-004-20260530 gcc-14.3.0
arm rpc_defconfig clang-18
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260530 gcc-8.5.0
arm64 randconfig-002-20260530 gcc-8.5.0
arm64 randconfig-003-20260530 gcc-8.5.0
arm64 randconfig-004-20260530 gcc-8.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260530 gcc-8.5.0
csky randconfig-002-20260530 gcc-8.5.0
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001-20260530 clang-23
hexagon randconfig-002-20260530 clang-23
i386 allmodconfig clang-20
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 buildonly-randconfig-001-20260530 clang-20
i386 buildonly-randconfig-002-20260530 clang-20
i386 buildonly-randconfig-003-20260530 clang-20
i386 buildonly-randconfig-004-20260530 clang-20
i386 buildonly-randconfig-005-20260530 clang-20
i386 buildonly-randconfig-006-20260530 clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001-20260530 clang-20
i386 randconfig-002-20260530 clang-20
i386 randconfig-003-20260530 clang-20
i386 randconfig-004-20260530 clang-20
i386 randconfig-005-20260530 clang-20
i386 randconfig-006-20260530 clang-20
i386 randconfig-007-20260530 clang-20
i386 randconfig-011-20260530 clang-20
i386 randconfig-012-20260530 clang-20
i386 randconfig-013-20260530 clang-20
i386 randconfig-014-20260530 clang-20
i386 randconfig-015-20260530 clang-20
i386 randconfig-016-20260530 clang-20
i386 randconfig-017-20260530 clang-20
loongarch allmodconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001-20260530 clang-23
loongarch randconfig-002-20260530 clang-23
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
mips qi_lb60_defconfig clang-23
nios2 allmodconfig clang-23
nios2 allnoconfig clang-23
nios2 defconfig clang-19
nios2 randconfig-001-20260530 clang-23
nios2 randconfig-002-20260530 clang-23
openrisc allmodconfig clang-23
openrisc allnoconfig clang-23
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allyesconfig clang-19
parisc defconfig gcc-15.2.0
parisc randconfig-001-20260530 gcc-8.5.0
parisc randconfig-002-20260530 gcc-8.5.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc randconfig-001-20260530 gcc-8.5.0
powerpc randconfig-002-20260530 gcc-8.5.0
powerpc64 randconfig-001-20260530 gcc-8.5.0
powerpc64 randconfig-002-20260530 gcc-8.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260530 gcc-12.5.0
riscv randconfig-002-20260530 gcc-12.5.0
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260530 gcc-12.5.0
s390 randconfig-002-20260530 gcc-12.5.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allyesconfig clang-19
sh defconfig gcc-14
sh randconfig-001-20260530 gcc-12.5.0
sh randconfig-002-20260530 gcc-12.5.0
sparc allnoconfig clang-23
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260530 gcc-9.5.0
sparc randconfig-002-20260530 gcc-9.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260530 gcc-9.5.0
sparc64 randconfig-002-20260530 gcc-9.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260530 gcc-9.5.0
um randconfig-002-20260530 gcc-9.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260530 gcc-14
x86_64 buildonly-randconfig-002-20260530 gcc-14
x86_64 buildonly-randconfig-003-20260530 gcc-14
x86_64 buildonly-randconfig-004-20260530 gcc-14
x86_64 buildonly-randconfig-005-20260530 gcc-14
x86_64 buildonly-randconfig-006-20260530 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-011-20260530 gcc-14
x86_64 randconfig-012-20260530 gcc-14
x86_64 randconfig-013-20260530 gcc-14
x86_64 randconfig-014-20260530 gcc-14
x86_64 randconfig-015-20260530 gcc-14
x86_64 randconfig-016-20260530 gcc-14
x86_64 randconfig-071-20260530 gcc-14
x86_64 randconfig-072-20260530 gcc-14
x86_64 randconfig-073-20260530 gcc-14
x86_64 randconfig-074-20260530 gcc-14
x86_64 randconfig-075-20260530 gcc-14
x86_64 randconfig-076-20260530 gcc-14
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allyesconfig clang-23
xtensa randconfig-001-20260530 gcc-9.5.0
xtensa randconfig-002-20260530 gcc-9.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* [tj-cgroup:for-next] BUILD SUCCESS ebc50c66b365d3046c7741195224d2aa7809c9b5
From: kernel test robot @ 2026-05-30 6:47 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
branch HEAD: ebc50c66b365d3046c7741195224d2aa7809c9b5 Merge branch 'for-7.2' into for-next
elapsed time: 724m
configs tested: 171
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc defconfig gcc-15.2.0
arc randconfig-001-20260530 gcc-14.3.0
arc randconfig-002-20260530 gcc-14.3.0
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm defconfig gcc-15.2.0
arm multi_v7_defconfig gcc-15.2.0
arm randconfig-001-20260530 gcc-14.3.0
arm randconfig-002-20260530 gcc-14.3.0
arm randconfig-003-20260530 gcc-14.3.0
arm randconfig-004-20260530 gcc-14.3.0
arm rpc_defconfig clang-18
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260530 gcc-8.5.0
arm64 randconfig-002-20260530 gcc-8.5.0
arm64 randconfig-003-20260530 gcc-8.5.0
arm64 randconfig-004-20260530 gcc-8.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260530 gcc-8.5.0
csky randconfig-002-20260530 gcc-8.5.0
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001-20260530 clang-23
hexagon randconfig-002-20260530 clang-23
i386 allmodconfig clang-20
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 buildonly-randconfig-001-20260530 clang-20
i386 buildonly-randconfig-002-20260530 clang-20
i386 buildonly-randconfig-003-20260530 clang-20
i386 buildonly-randconfig-004-20260530 clang-20
i386 buildonly-randconfig-005-20260530 clang-20
i386 buildonly-randconfig-006-20260530 clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001-20260530 clang-20
i386 randconfig-002-20260530 clang-20
i386 randconfig-003-20260530 clang-20
i386 randconfig-004-20260530 clang-20
i386 randconfig-005-20260530 clang-20
i386 randconfig-006-20260530 clang-20
i386 randconfig-007-20260530 clang-20
i386 randconfig-011-20260530 clang-20
i386 randconfig-012-20260530 clang-20
i386 randconfig-013-20260530 clang-20
i386 randconfig-014-20260530 clang-20
i386 randconfig-015-20260530 clang-20
i386 randconfig-016-20260530 clang-20
i386 randconfig-017-20260530 clang-20
loongarch allmodconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001-20260530 clang-23
loongarch randconfig-002-20260530 clang-23
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
mips qi_lb60_defconfig clang-23
nios2 allmodconfig clang-23
nios2 allnoconfig clang-23
nios2 defconfig clang-19
nios2 randconfig-001-20260530 clang-23
nios2 randconfig-002-20260530 clang-23
openrisc allmodconfig clang-23
openrisc allnoconfig clang-23
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allyesconfig clang-19
parisc defconfig gcc-15.2.0
parisc randconfig-001-20260530 gcc-8.5.0
parisc randconfig-002-20260530 gcc-8.5.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc randconfig-001-20260530 gcc-8.5.0
powerpc randconfig-002-20260530 gcc-8.5.0
powerpc socrates_defconfig gcc-15.2.0
powerpc64 randconfig-001-20260530 gcc-8.5.0
powerpc64 randconfig-002-20260530 gcc-8.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260530 gcc-12.5.0
riscv randconfig-002-20260530 gcc-12.5.0
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260530 gcc-12.5.0
s390 randconfig-002-20260530 gcc-12.5.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allyesconfig clang-19
sh defconfig gcc-14
sh randconfig-001-20260530 gcc-12.5.0
sh randconfig-002-20260530 gcc-12.5.0
sparc allnoconfig clang-23
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260530 gcc-9.5.0
sparc randconfig-002-20260530 gcc-9.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260530 gcc-9.5.0
sparc64 randconfig-002-20260530 gcc-9.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260530 gcc-9.5.0
um randconfig-002-20260530 gcc-9.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260530 gcc-14
x86_64 buildonly-randconfig-002-20260530 gcc-14
x86_64 buildonly-randconfig-003-20260530 gcc-14
x86_64 buildonly-randconfig-004-20260530 gcc-14
x86_64 buildonly-randconfig-005-20260530 gcc-14
x86_64 buildonly-randconfig-006-20260530 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260530 gcc-14
x86_64 randconfig-002-20260530 gcc-14
x86_64 randconfig-003-20260530 gcc-14
x86_64 randconfig-004-20260530 gcc-14
x86_64 randconfig-005-20260530 gcc-14
x86_64 randconfig-006-20260530 gcc-14
x86_64 randconfig-011-20260530 gcc-14
x86_64 randconfig-012-20260530 gcc-14
x86_64 randconfig-013-20260530 gcc-14
x86_64 randconfig-014-20260530 gcc-14
x86_64 randconfig-015-20260530 gcc-14
x86_64 randconfig-016-20260530 gcc-14
x86_64 randconfig-071-20260530 gcc-14
x86_64 randconfig-072-20260530 gcc-14
x86_64 randconfig-073-20260530 gcc-14
x86_64 randconfig-074-20260530 gcc-14
x86_64 randconfig-075-20260530 gcc-14
x86_64 randconfig-076-20260530 gcc-14
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allyesconfig clang-23
xtensa randconfig-001-20260530 gcc-9.5.0
xtensa randconfig-002-20260530 gcc-9.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH v5 1/9] mm: list_lru: fix set_shrinker_bit() call during race with cgroup deletion
From: Wei Yang @ 2026-05-30 2:38 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Liam R . Howlett, Usama Arif,
Kiryl Shutsemau, Vlastimil Babka, Kairui Song, Mikhail Zaslonko,
Vasily Gorbik, Baolin Wang, Barry Song, Dev Jain, Lance Yang,
Nico Pache, Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260527204757.2544958-2-hannes@cmpxchg.org>
On Wed, May 27, 2026 at 04:45:08PM -0400, Johannes Weiner wrote:
>When list_lru_add() races with cgroup deletion, the shrinker bit is set
>on the wrong group and lost. This can cause a shrinker run to miss the
>cgroup that actually has the object.
>
>When the passed in memcg is dead, the function finds the first non-dead
>parent from the passed in memcg and adds the object there; but the
>shrinker bit is set on the memcg that was passed in.
>
This means we just miss to reclaim some obj, but won't crash the kernel.
>This bug is as old as the shrinker bitmap itself.
>
>Fix it by returning the "effective" memcg from the locking function, and
>have the caller use that.
>
>Fixes: fae91d6d8be5 ("mm/list_lru.c: set bit in memcg shrinker bitmap on first list_lru item appearance")
>Reported-by: Usama Arif <usama.arif@linux.dev>
>Reported-by: Sashiko
>Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
So we don't want to cc stable, right?
The fix looks right, so
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
--
Wei Yang
Help you, Help me
^ permalink raw reply
* Re: [PATCH v3 2/4] mm/zswap: Implement proactive writeback
From: Yosry Ahmed @ 2026-05-30 1:40 UTC (permalink / raw)
To: Nhat Pham
Cc: Hao Jia, akpm, tj, hannes, shakeel.butt, mhocko, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <CAKEwX=MQe_KFZe2vBXQYh0aa-x+E8AzNwmyjJGJk4tDoS9ML3A@mail.gmail.com>
On Fri, May 29, 2026 at 12:58:09PM -0700, Nhat Pham wrote:
> On Tue, May 26, 2026 at 4:46 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
> >
> > From: Hao Jia <jiahao1@lixiang.com>
> >
> > Zswap currently writes back pages to backing swap reactively, triggered
> > either by the shrinker or when the pool reaches its size limit. There is
> > no mechanism to control the amount of writeback for a specific memory
> > cgroup. However, users may want to proactively write back zswap pages,
> > e.g., to free up memory for other applications or to prepare for
> > memory-intensive workloads.
> >
> > Introduce a "zswap_writeback_only" key to the memory.reclaim cgroup
> > interface. When specified, this key bypasses standard memory reclaim
> > and exclusively performs proactive zswap writeback up to the requested
> > budget. If omitted, the default reclaim behavior remains unchanged.
> >
> > Example usage:
> > # Write back 100MB of pages from zswap to the backing swap
> > echo "100M zswap_writeback_only" > memory.reclaim
>
> Hmmm, so this 100MB is the pre-compression size? i.e if this 100 MB
> compresses to 25 MB, then you're only freeing 25 MB?
>
> I'm ok-ish with this, but can you document it?
That's a good point. I think pre-compressed size doesn't make sense to
be honest. We should care about how much memory we are actually trying
to save by doing writeback here.
The pre-compressed size is only useful in determining the blast radius,
how many actual pages are going to have slower page faults now. But
then, I don't think there's a reasonable way for userspace to decide
that.
I understand passing in the compressed size is tricky because we need to
keep track of the size of the compressed pages we end up writing back,
but it should be doable.
If we really want pre-compressed size here, then yes we need to make it
very clear, and I vote that we use a separate interface in this case
because memory.reclaim having different meanings for the amount of
memory written to it is extremely counter-intuitive.
>
> The rest seems solid to me, FWIW. I'll defer to Johannes and Yosry for
> opinions on zswap-only proactive reclaim.
^ permalink raw reply
* Re: [PATCH v3 2/4] mm/zswap: Implement proactive writeback
From: Yosry Ahmed @ 2026-05-30 1:37 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260526114601.67041-3-jiahao.kernel@gmail.com>
On Tue, May 26, 2026 at 07:45:59PM +0800, Hao Jia wrote:
> From: Hao Jia <jiahao1@lixiang.com>
>
> Zswap currently writes back pages to backing swap reactively, triggered
> either by the shrinker or when the pool reaches its size limit. There is
> no mechanism to control the amount of writeback for a specific memory
> cgroup. However, users may want to proactively write back zswap pages,
> e.g., to free up memory for other applications or to prepare for
> memory-intensive workloads.
>
> Introduce a "zswap_writeback_only" key to the memory.reclaim cgroup
> interface. When specified, this key bypasses standard memory reclaim
> and exclusively performs proactive zswap writeback up to the requested
> budget. If omitted, the default reclaim behavior remains unchanged.
>
> Example usage:
> # Write back 100MB of pages from zswap to the backing swap
> echo "100M zswap_writeback_only" > memory.reclaim
>
> Note that the actual amount written back may be less than requested due
> to the zswap second-chance algorithm: referenced entries are rotated on
> the LRU on the first encounter and only written back on a second pass.
> If fewer bytes are written back than requested, -EAGAIN is returned,
> matching the existing memory.reclaim semantics.
>
> Internally, extend user_proactive_reclaim() to parse the new
> "zswap_writeback_only" token and invoke the dedicated handler. Add
> zswap_proactive_writeback() to walk the target memcg subtree via the
> per-memcg writeback cursor, draining per-node zswap LRUs through
> list_lru_walk_one() with the shrink_memcg_cb() callback.
>
> Suggested-by: Yosry Ahmed <yosry@kernel.org>
> Suggested-by: Nhat Pham <nphamcs@gmail.com>
> Signed-off-by: Hao Jia <jiahao1@lixiang.com>
> ---
> Documentation/admin-guide/cgroup-v2.rst | 18 +++-
> Documentation/admin-guide/mm/zswap.rst | 11 +-
> include/linux/zswap.h | 7 ++
> mm/vmscan.c | 14 +++
> mm/zswap.c | 138 ++++++++++++++++++++++++
> 5 files changed, 185 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 6efd0095ed99..6564abf0dec5 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -1425,9 +1425,10 @@ PAGE_SIZE multiple when read back.
>
> The following nested keys are defined.
>
> - ========== ================================
> + ==================== ==================================================
> swappiness Swappiness value to reclaim with
> - ========== ================================
> + zswap_writeback_only Only perform proactive zswap writeback
> + ==================== ==================================================
>
> Specifying a swappiness value instructs the kernel to perform
> the reclaim with that swappiness value. Note that this has the
> @@ -1437,6 +1438,19 @@ The following nested keys are defined.
> The valid range for swappiness is [0-200, max], setting
> swappiness=max exclusively reclaims anonymous memory.
>
> + The zswap_writeback_only key skips ordinary memory reclaim and
> + writes back pages from zswap to the backing swap device until
> + the requested amount has been written or no further candidates
> + are found. This is useful to proactively offload cold pages from
> + the zswap pool to the swap device. It is only available if
> + zswap writeback is enabled. zswap_writeback_only cannot be combined
> + with swappiness; specifying both returns -EINVAL.
> +
> + Example::
> +
> + # Write back up to 100MB of pages from zswap to the backing swap
> + echo "100M zswap_writeback_only" > memory.reclaim
memcg folks need to chime in about the interface here. An alternative
would be a separate interface (e.g. memory.zswap.do_writeback or
memory.zswap.writeback.reclaim or sth).
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 73e64a635690..7bcbf788f634 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -1679,6 +1679,144 @@ int zswap_load(struct folio *folio)
> return 0;
> }
>
> +/*
> + * Maximum LRU scan limit:
> + * number of entries to scan per page of remaining budget.
> + */
> +#define ZSWAP_PROACTIVE_WB_SCAN_RATIO 16UL
> +/*
> + * Batch size for proactive writeback:
> + * - As the per-memcg writeback target in the outer memcg loop.
> + * - As the per-walk budget passed to list_lru_walk_one().
> + */
> +#define ZSWAP_PROACTIVE_WB_BATCH 128UL
> +
> +/*
> + * Walk the per-node LRUs of @memcg to write back up to @nr_to_write pages.
> + * Returns the number of pages written back, or -ENOENT if @memcg is a
> + * zombie or has writeback disabled.
> + */
> +static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
> + unsigned long nr_to_write)
> +{
> + unsigned long nr_written = 0;
> + int nid;
> +
> + if (!mem_cgroup_zswap_writeback_enabled(memcg))
> + return -ENOENT;
> +
> + if (!mem_cgroup_online(memcg))
> + return -ENOENT;
> +
> + for_each_node_state(nid, N_NORMAL_MEMORY) {
> + bool encountered_page_in_swapcache = false;
> + unsigned long nr_to_scan, nr_scanned = 0;
> +
> + /*
> + * Cap by LRU length: bounds rewalks when referenced
> + * entries keep rotating to the tail.
> + */
> + nr_to_scan = list_lru_count_one(&zswap_list_lru, nid, memcg);
> + if (!nr_to_scan)
> + continue;
> +
> + /*
> + * Cap by SCAN_RATIO * remaining budget: bounds scan cost
> + * to the remaining writeback budget.
> + */
> + nr_to_scan = min(nr_to_scan,
> + (nr_to_write - nr_written) * ZSWAP_PROACTIVE_WB_SCAN_RATIO);
> +
> + while (nr_scanned < nr_to_scan) {
> + unsigned long nr_to_walk = min(ZSWAP_PROACTIVE_WB_BATCH,
> + nr_to_scan - nr_scanned);
> +
> + if (signal_pending(current))
> + return nr_written;
> +
> + /*
> + * Account for the committed budget rather than the walker's
> + * actual delta. If the list is emptied concurrently, the
> + * walker visits nothing and nr_scanned would never advance.
> + */
> + nr_scanned += nr_to_walk;
> +
> + nr_written += list_lru_walk_one(&zswap_list_lru, nid, memcg,
> + &shrink_memcg_cb,
> + &encountered_page_in_swapcache,
> + &nr_to_walk);
> +
> + if (nr_written >= nr_to_write)
> + return nr_written;
> + if (encountered_page_in_swapcache)
> + break;
> +
> + cond_resched();
> + }
> + }
> +
> + return nr_written;
> +}
> +
> +int zswap_proactive_writeback(struct mem_cgroup *memcg,
> + unsigned long nr_to_writeback)
> +{
> + struct mem_cgroup *iter_memcg;
> + unsigned long nr_written = 0;
> + int failures = 0, attempts = 0;
> +
> + if (!memcg)
> + return -EINVAL;
> + if (!nr_to_writeback)
> + return 0;
> +
> + /*
> + * Writeback will be aborted with -EAGAIN if we encounter
> + * the following MAX_RECLAIM_RETRIES times:
> + * - No writeback-candidate memcgs found in a subtree walk.
> + * - A writeback-candidate memcg wrote back zero pages.
> + */
> + while (nr_written < nr_to_writeback) {
> + unsigned long batch_size;
> + long shrunk;
> +
> + if (signal_pending(current))
> + return -EINTR;
> +
> + iter_memcg = zswap_mem_cgroup_iter(memcg);
> +
> + if (!iter_memcg) {
> + /*
> + * Continue without incrementing failures if we found
> + * candidate memcgs in the last subtree walk.
> + */
> + if (!attempts && ++failures == MAX_RECLAIM_RETRIES)
> + return -EAGAIN;
> + attempts = 0;
> + continue;
> + }
> +
> + batch_size = min(nr_to_writeback - nr_written,
> + ZSWAP_PROACTIVE_WB_BATCH);
> + shrunk = zswap_proactive_shrink_memcg(iter_memcg, batch_size);
> + mem_cgroup_put(iter_memcg);
> +
> + /* Writeback-disabled or offline: skip without counting. */
> + if (shrunk == -ENOENT)
> + continue;
> +
> + ++attempts;
> + if (shrunk > 0)
> + nr_written += shrunk;
> + else if (++failures == MAX_RECLAIM_RETRIES)
> + return -EAGAIN;
> +
> + cond_resched();
> + }
> +
> + return 0;
> +}
> +
There is a lot of copy+paste from shrink_worker() and shrink_memcg()
here. We really should be able to reuse shrink_memcg().
Is the main difference that we are scanning in batches here? I think we
can have shrink_memcg() do that too. If anything, it might make the
shrinker more efficient. Over-reclaim is ofc a concern, and especially
in the zswap_store() path as the overhead can be noticeable. Maybe we
can parameterize the batch size based on the code path.
Nhat, what do you think?
^ permalink raw reply
* Re: [PATCH v5 5/9] mm: list_lru: deduplicate lock_list_lru()
From: Wei Yang @ 2026-05-30 1:25 UTC (permalink / raw)
To: Johannes Weiner
Cc: Wei Yang, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Shakeel Butt, Michal Hocko, Dave Chinner, Roman Gushchin,
Muchun Song, Qi Zheng, Yosry Ahmed, Zi Yan, Liam R . Howlett,
Usama Arif, Kiryl Shutsemau, Vlastimil Babka, Kairui Song,
Mikhail Zaslonko, Vasily Gorbik, Baolin Wang, Barry Song,
Dev Jain, Lance Yang, Nico Pache, Ryan Roberts, cgroups, linux-mm,
linux-kernel
In-Reply-To: <ahmXqjQ0Vz4pb4u1@cmpxchg.org>
On Fri, May 29, 2026 at 09:42:02AM -0400, Johannes Weiner wrote:
>On Fri, May 29, 2026 at 09:56:28AM +0000, Wei Yang wrote:
>> On Wed, May 27, 2026 at 04:45:12PM -0400, Johannes Weiner wrote:
>> >The MEMCG and !MEMCG paths have the same pattern. Share the code.
>> >
>> >Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
>> >Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
>> >Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
>> >Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
>> >Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
>> >---
>> > mm/list_lru.c | 21 +++++++++------------
>> > 1 file changed, 9 insertions(+), 12 deletions(-)
>> >
>> >diff --git a/mm/list_lru.c b/mm/list_lru.c
>> >index 7d0523e44010..fdb3fe2ea64f 100644
>> >--- a/mm/list_lru.c
>> >+++ b/mm/list_lru.c
>> >@@ -15,6 +15,14 @@
>> > #include "slab.h"
>> > #include "internal.h"
>>
>> Hi, Johannes
>>
>> One very tiny nit below.
>>
>> >
>> >+static inline void lock_list_lru(struct list_lru_one *l, bool irq)
>>
>> Here we use @irq.
>>
>> >+{
>> >+ if (irq)
>> >+ spin_lock_irq(&l->lock);
>> >+ else
>> >+ spin_lock(&l->lock);
>> >+}
>> >+
>> > static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
>>
>> Here we use @irq_off.
>>
>> Do you think it would be nicer to unify the parameter name?
>
>Yes, I think it would be nicer.
>
>Note that I inherited this - we had irq on the lock and irq_off on the
>unlock before already. I didn't want to mix even more yak shaving prep
>patches into this series.
Reasonable.
>
>Mind sending a follow-up patch on top of mm-unstable?
Thanks, I am glad to.
Since this is trivial, I would wait until everything is settle down.
Looks good to you?
--
Wei Yang
Help you, Help me
^ permalink raw reply
* Re: [PATCH v3 1/4] mm/zswap: Make shrink_worker writeback cursor per-memcg
From: Yosry Ahmed @ 2026-05-30 1:24 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260526114601.67041-2-jiahao.kernel@gmail.com>
On Tue, May 26, 2026 at 07:45:58PM +0800, Hao Jia wrote:
> From: Hao Jia <jiahao1@lixiang.com>
>
> The zswap background writeback worker shrink_worker() uses a global
> cursor zswap_next_shrink, protected by zswap_shrink_lock, to round-robin
> across the online memcgs under root_mem_cgroup.
>
> Proactive writeback also wants a similar per-memcg cursor that is
> scoped to the specified memcg, so that repeated invocations against
> the same memcg make forward progress across its descendant memcgs
> instead of restarting from the first child memcg each time.
Is this a problem in practice?
Is the concern the overhead of scanning memcgs repeatedly, or lack of
fairness? I wonder if we should just do writeback in batches from all
memcgs, similar to how reclaim does it, then evaluate at the end if we
need to start over?
>
> Naturally, group the cursor and its protecting spinlock into a
> zswap_wb_iter struct, and make it a member of struct mem_cgroup to
> realize per-memcg cursor management. Accordingly, shrink_worker() now
> uses the lock and cursor in root_mem_cgroup->zswap_wb_iter.
If we really need to have per-memcg cursors (I am not a big fan), I
think we can minimize the overhead by making the cursor updates use
atomic cmpxchg instead of having a per-memcg lock.
>
> Because the cursor is now per-memcg, the offline cleanup must visit
> every ancestor that could be holding a reference to the dying memcg.
> Factor out __zswap_memcg_offline_cleanup() and walk from dead_memcg up
> to the root.
Another reason why I don't like per-memcg cursors. There is too much
complexity and I wonder if it's warranted. If we stick with per-memcg
cursors please do the refactoring in separate patches to make the
patches easier to review.
Thanks!
^ permalink raw reply
* [PATCH-next v4 6/6] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
In-Reply-To: <20260529212108.120506-1-longman@redhat.com>
With cgroup v2, the cgroup_taskset structure passed into the cgroup
can_attach() and attach() methods can contain task migration data with
multiple destination or source cpusets when the cpuset controller is
enabled or disabled respectively.
Since cpuset is threaded in both v1 and v2, another possible way to
cause many-to-one migration is to move the whole process with multiple
threads in different cpuset enabled threaded cgroups into another cpuset
enabled cgroup.
The current cpuset_can_attach() and cpuset_attach() functions still
expect task migration is from one source cpuset to one destination
cpuset. This has been the case since cpuset was enabled for cgroup v2
in commit 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default
hierarchy").
This problem is less an issue when enabling the cpuset controller as all
the newly created child cpusets will have exactly the same set of CPUs
and memory nodes except when deadline tasks are involved in migration
as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as
their set of CPUs and memory nodes may differ from their parent or with
the moving of multi-threaded process from different threaded cgroups.
Fix that by tracking the set of source (old) and destination cpusets
in singly linked lists and iterating them all to properly update the
internal data. Also keep the current cs and oldcs variables up-to-date
with the css and task iterators.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both
the source and destination cpusets are decremented/incremented with
their values added to nr_deadline_tasks when the migration is successful.
Fixes: 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy")
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset-internal.h | 6 +
kernel/cgroup/cpuset.c | 204 ++++++++++++++++++++++++--------
2 files changed, 158 insertions(+), 52 deletions(-)
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index f7aaf01f7cd5..4c2772a7fd5e 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -161,6 +161,12 @@ struct cpuset {
*/
bool remote_partition;
+ /*
+ * cpuset_can_attach() and cpuset_attach() specific data
+ */
+ bool attach_node_in_llist;
+ struct llist_node attach_node;
+
/*
* number of SCHED_DEADLINE tasks attached to this cpuset, so that we
* know when to rebuild associated root domain bandwidth information.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index a6506b94e60a..2add658eb288 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -37,6 +37,7 @@
#include <linux/wait.h>
#include <linux/workqueue.h>
#include <linux/task_work.h>
+#include <linux/llist.h>
DEFINE_STATIC_KEY_FALSE(cpusets_pre_enable_key);
DEFINE_STATIC_KEY_FALSE(cpusets_enabled_key);
@@ -1127,6 +1128,8 @@ static void update_sibling_cpumasks(struct cpuset *parent, struct cpuset *cs,
* matter which child cpuset is selected as cpuset_attach_old_cs.
*/
static struct cpuset *cpuset_attach_old_cs;
+static LLIST_HEAD(src_cs_head);
+static LLIST_HEAD(dst_cs_head);
static bool attach_cpus_updated;
static bool attach_mems_updated;
@@ -3017,9 +3020,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)
* Also set the boolean flag passed in by @psetsched depending on if
* security_task_setscheduler() call is needed and @oldcs is not NULL.
*/
-static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
- bool *psetsched)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs, bool *psetsched)
{
+ bool cpu_match, mem_match;
+
if (cpumask_empty(cs->effective_cpus) ||
(!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
return -ENOSPC;
@@ -3030,15 +3034,34 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
/*
* Update attach specific data
*/
- attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
- attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+ if (!cs->attach_node_in_llist) {
+ llist_add(&cs->attach_node, &dst_cs_head);
+ cs->attach_node_in_llist = true;
+ }
+ if (!oldcs->attach_node_in_llist) {
+ llist_add(&oldcs->attach_node, &src_cs_head);
+ oldcs->attach_node_in_llist = true;
+ }
+
+ cpu_match = cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+ mem_match = nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+ /*
+ * Set the updated flags whenever there is a mismatch in any of the
+ * src/dst pairs.
+ */
+ if (!attach_cpus_updated)
+ attach_cpus_updated = !cpu_match;
+
+ if (!attach_mems_updated)
+ attach_mems_updated = !mem_match;
/*
* Skip rights over task setsched check in v2 when nothing changes,
* migration permission derives from hierarchy ownership in
* cgroup_procs_write_permission()).
*/
- *psetsched = !cpuset_v2() || attach_cpus_updated || attach_mems_updated;
+ *psetsched = !cpuset_v2() || !cpu_match || !mem_match;
/*
* A v1 cpuset with tasks will have no CPU left only when CPU hotplug
@@ -3053,33 +3076,103 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
return 0;
}
-static int cpuset_reserve_dl_bw(struct cpuset *cs)
+/*
+ * If reset_dl_bw is set, reset the previous dl_bw_alloc() call. Otherwise,
+ * update nr_deadline_tasks according to nr_migrate_dl_tasks in both source
+ * and destination cpusets.
+ */
+static void clear_attach_data(bool reset_dl_bw)
+{
+ struct cpuset *cs, *next;
+
+ llist_for_each_entry_safe(cs, next, src_cs_head.first, attach_node) {
+ cs->attach_node.next = NULL;
+ cs->attach_node_in_llist = false;
+ if (cs->nr_migrate_dl_tasks && !reset_dl_bw)
+ cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
+ cs->nr_migrate_dl_tasks = 0;
+ }
+
+ llist_for_each_entry_safe(cs, next, dst_cs_head.first, attach_node) {
+ cs->attach_node.next = NULL;
+ cs->attach_node_in_llist = false;
+ if (reset_dl_bw && cs->dl_bw_cpu >= 0)
+ dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+ if (cs->nr_migrate_dl_tasks && !reset_dl_bw)
+ cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
+ cs->nr_migrate_dl_tasks = 0;
+ cs->sum_migrate_dl_bw = 0;
+ cs->dl_bw_cpu = -1;
+ }
+
+ src_cs_head.first = NULL;
+ dst_cs_head.first = NULL;
+ attach_cpus_updated = false;
+ attach_mems_updated = false;
+}
+
+static int cpuset_reserve_dl_bw(void)
{
+ struct cpuset *cs;
int cpu, ret;
- if (!cs->sum_migrate_dl_bw)
- return 0;
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node) {
+ if (!cs->sum_migrate_dl_bw)
+ continue;
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids))
- return -EINVAL;
+ cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+ if (unlikely(cpu >= nr_cpu_ids))
+ return -EINVAL;
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
- if (ret)
- return ret;
+ ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ if (ret)
+ return ret;
- cs->dl_bw_cpu = cpu;
+ cs->dl_bw_cpu = cpu;
+ }
return 0;
}
-static void reset_migrate_dl_data(struct cpuset *cs)
+static void set_attach_in_progress(void)
{
- cs->nr_migrate_dl_tasks = 0;
- cs->sum_migrate_dl_bw = 0;
- cs->dl_bw_cpu = -1;
+ struct cpuset *cs;
+
+ /*
+ * Mark attach is in progress. This makes validate_change() fail
+ * changes which zero cpus/mems_allowed.
+ */
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node)
+ cs->attach_in_progress++;
+}
+
+static void reset_attach_in_progress(void)
+{
+ struct cpuset *cs;
+
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node)
+ dec_attach_in_progress_locked(cs);
}
-/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
+/*
+ * Called by cgroups to determine if a cpuset is usable; cpuset_mutex held.
+ *
+ * With cgroup v2, enabling of cpuset controller in a cgroup subtree can
+ * cause @tset to contain task migration data from one parent cpuset to multiple
+ * child cpusets. Not much is needed to be done here other than tracking the
+ * number of DL tasks in each cpuset as the CPUs and memory nodes of the child
+ * cpusets are exactly the same as the parent.
+ *
+ * Conversely, disabling of cpuset controller can cause @tset to contain task
+ * migration data from multiple child cpusets to one parent cpuset. Here, the
+ * CPUs and memory nodes of the child cpusets may be different from the parent,
+ * but must be a subset of its parent.
+ *
+ * Another possible many-to-one migration is the moving of the whole
+ * multithreaded process with threads in different cpusets to another cpuset.
+ *
+ * For all other use cases, @tset task migration data should be from one source
+ * cpuset to one destination cpuset.
+ */
static int cpuset_can_attach(struct cgroup_taskset *tset)
{
struct cgroup_subsys_state *css;
@@ -3101,6 +3194,16 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
goto out_unlock;
cgroup_taskset_for_each(task, css, tset) {
+ struct cpuset *newcs = css_cs(css);
+ struct cpuset *new_oldcs = task_cs(task);
+
+ if ((newcs != cs) || (new_oldcs != oldcs)) {
+ cs = newcs;
+ oldcs = new_oldcs;
+ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
+ if (ret)
+ goto out_unlock;
+ }
ret = task_can_attach(task);
if (ret)
goto out_unlock;
@@ -3122,23 +3225,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
* contribute to sum_migrate_dl_bw.
*/
cs->nr_migrate_dl_tasks++;
+ oldcs->nr_migrate_dl_tasks--;
if (dl_task_needs_bw_move(task, cs->effective_cpus))
cs->sum_migrate_dl_bw += task->dl.dl_bw;
}
}
- ret = cpuset_reserve_dl_bw(cs);
+ ret = cpuset_reserve_dl_bw();
out_unlock:
- if (ret) {
- reset_migrate_dl_data(cs);
- } else {
- /*
- * Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
- cs->attach_in_progress++;
- }
+ if (ret)
+ clear_attach_data(true);
+ else
+ set_attach_in_progress();
mutex_unlock(&cpuset_mutex);
return ret;
@@ -3153,14 +3252,8 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
cs = css_cs(css);
mutex_lock(&cpuset_mutex);
- dec_attach_in_progress_locked(cs);
-
- if (cs->dl_bw_cpu >= 0)
- dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
-
- if (cs->nr_migrate_dl_tasks)
- reset_migrate_dl_data(cs);
-
+ reset_attach_in_progress();
+ clear_attach_data(true);
mutex_unlock(&cpuset_mutex);
}
@@ -3232,7 +3325,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct task_struct *task;
struct cgroup_subsys_state *css;
struct cpuset *cs;
- struct cpuset *oldcs = cpuset_attach_old_cs;
cgroup_taskset_first(tset, &css);
cs = css_cs(css);
@@ -3245,32 +3337,40 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* In the default hierarchy, enabling cpuset in the child cgroups
* will trigger a cpuset_attach() call with no change in effective cpus
* and mems. In that case, we can optimize out by skipping the task
- * iteration and update.
+ * iteration and update, but the destination cpuset list is iterated to
+ * set old_mems_sllowed.
*/
if (cpuset_v2()) {
cpuset_attach_nodemask_to = cs->effective_mems;
- if (!attach_cpus_updated && !attach_mems_updated)
+ if (!attach_cpus_updated && !attach_mems_updated) {
+ llist_for_each_entry(cs, dst_cs_head.first, attach_node)
+ cs->old_mems_allowed = cs->effective_mems;
goto out;
+ }
} else {
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
}
- cgroup_taskset_for_each(task, css, tset)
+ cgroup_taskset_for_each(task, css, tset) {
+ struct cpuset *newcs = css_cs(css);
+
+ if (newcs != cs) {
+ cs->old_mems_allowed = cpuset_attach_nodemask_to;
+ cs = newcs;
+ if (cpuset_v2())
+ cpuset_attach_nodemask_to = cs->effective_mems;
+ else
+ guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ }
cpuset_attach_task(cs, task);
+ }
-out:
if (queue_task_work)
schedule_flush_migrate_mm();
cs->old_mems_allowed = cpuset_attach_nodemask_to;
-
- if (cs->nr_migrate_dl_tasks) {
- cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
- oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks;
- reset_migrate_dl_data(cs);
- }
-
- dec_attach_in_progress_locked(cs);
-
+out:
+ reset_attach_in_progress();
+ clear_attach_data(false);
mutex_unlock(&cpuset_mutex);
}
--
2.54.0
^ permalink raw reply related
* [PATCH-next v4 5/6] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
In-Reply-To: <20260529212108.120506-1-longman@redhat.com>
The cpuset_attach_task() was introduced in commit 42a11bf5c543
("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly")
to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like
moving a task from one cpuset into another one. That commits didn't
move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader
into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new
task is its own group leader. So it is still not equivalent to moving
task between cpusets in this case. Make CLONE_INTO_CGROUP behaves
more close to cpuset_attach() by moving the mpol_rebind_mm() and
cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result,
the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 89 ++++++++++++++++++++++++------------------
1 file changed, 51 insertions(+), 38 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 0bb63a9cda0b..a6506b94e60a 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3171,9 +3171,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
*/
static cpumask_var_t cpus_attach;
static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
+ struct mm_struct *mm;
+
lockdep_assert_cpuset_lock_held();
if (cs != &top_cpuset)
@@ -3187,24 +3190,56 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
*/
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ if (cpuset_v2() && !attach_mems_updated)
+ return;
+
cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
cpuset1_update_task_spread_flags(cs, task);
+
+ if (task != task->group_leader)
+ return;
+
+ /*
+ * Change mm for threadgroup leader. This is expensive and may
+ * sleep and should be moved outside migration path proper.
+ */
+ mm = get_task_mm(task);
+ if (mm) {
+ struct cpuset *oldcs = cpuset_attach_old_cs;
+
+ mpol_rebind_mm(mm, &cs->effective_mems);
+
+ /*
+ * old_mems_allowed is the same with mems_allowed
+ * here, except if this task is being moved
+ * automatically due to hotplug. In that case
+ * @mems_allowed has been updated and is empty, so
+ * @old_mems_allowed is the right nodesets that we
+ * migrate mm from.
+ */
+ if (is_memory_migrate(cs)) {
+ cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+ &cpuset_attach_nodemask_to);
+ queue_task_work = true;
+ } else {
+ mmput(mm);
+ }
+ }
}
static void cpuset_attach(struct cgroup_taskset *tset)
{
struct task_struct *task;
- struct task_struct *leader;
struct cgroup_subsys_state *css;
struct cpuset *cs;
struct cpuset *oldcs = cpuset_attach_old_cs;
- bool queue_task_work = false;
cgroup_taskset_first(tset, &css);
cs = css_cs(css);
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
+ queue_task_work = false;
/*
* In the default hierarchy, enabling cpuset in the child cgroups
@@ -3223,38 +3258,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
cgroup_taskset_for_each(task, css, tset)
cpuset_attach_task(cs, task);
- /*
- * Change mm for all threadgroup leaders. This is expensive and may
- * sleep and should be moved outside migration path proper. Skip it
- * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
- * not set.
- */
- if (!is_memory_migrate(cs) && !attach_mems_updated)
- goto out;
-
- cgroup_taskset_for_each_leader(leader, css, tset) {
- struct mm_struct *mm = get_task_mm(leader);
-
- if (mm) {
- mpol_rebind_mm(mm, &cs->effective_mems);
-
- /*
- * old_mems_allowed is the same with mems_allowed
- * here, except if this task is being moved
- * automatically due to hotplug. In that case
- * @mems_allowed has been updated and is empty, so
- * @old_mems_allowed is the right nodesets that we
- * migrate mm from.
- */
- if (is_memory_migrate(cs)) {
- cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
- &cpuset_attach_nodemask_to);
- queue_task_work = true;
- } else
- mmput(mm);
- }
- }
-
out:
if (queue_task_work)
schedule_flush_migrate_mm();
@@ -3688,15 +3691,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)
*/
static void cpuset_fork(struct task_struct *task)
{
- struct cpuset *cs;
- bool same_cs;
+ struct cpuset *cs, *oldcs;
rcu_read_lock();
cs = task_cs(task);
- same_cs = (cs == task_cs(current));
+ oldcs = task_cs(current);
rcu_read_unlock();
- if (same_cs) {
+ if (cs == oldcs) {
if (cs == &top_cpuset)
return;
@@ -3708,7 +3710,18 @@ static void cpuset_fork(struct task_struct *task)
/* CLONE_INTO_CGROUP */
mutex_lock(&cpuset_mutex);
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ /*
+ * Assume CPUs and memory nodes are updated
+ * A CLONE_INTO_CGROUP operation should have taken the cgroup mutex
+ * and so there shouldn't be a competing cpuset_attach() operation.
+ */
+ attach_cpus_updated = attach_mems_updated = true;
+ queue_task_work = false;
+ cpuset_attach_old_cs = oldcs;
cpuset_attach_task(cs, task);
+ attach_cpus_updated = attach_mems_updated = false;
+ if (queue_task_work)
+ schedule_flush_migrate_mm();
dec_attach_in_progress_locked(cs);
mutex_unlock(&cpuset_mutex);
--
2.54.0
^ permalink raw reply related
* [PATCH-next v4 4/6] cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long,
Ridong Chen
In-Reply-To: <20260529212108.120506-1-longman@redhat.com>
There are two possible ways that migration of tasks from multiple source
cpusets to a target cpuset can happen. Either a multithread application
with threads in different cpusets is wholely moved to a new cpuset
or disabling of v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that really
matters. So cpuset_attach_old_cs should track the oldcs of the thread
leader. In the latter case, effective_mems of child cpusets must always
be a subset of the parent. So no real page migration will be necessary
no matter which child cpuset is selected as cpuset_attach_old_cs.
IOW, cpuset_attach_old_cs should be updated to match the latest task
group leader in cpuset_can_attach().
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 0f93f3d84494..0bb63a9cda0b 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1111,6 +1111,20 @@ static void update_sibling_cpumasks(struct cpuset *parent, struct cpuset *cs,
/*
* cpuset_can_attach() and cpuset_attach() specific internal data
* Protected by cpuset_mutex
+ *
+ * The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get the
+ * old_mems_allowed value. There are two ways that many-to-one cpuset migration
+ * can happen:
+ * 1) A multithread application with threads in different cpusets is wholely
+ * moved to a new cpuset.
+ * 2) Disabling v2 cpuset controller will move all the tasks in child cpusets
+ * to the parent cpuset.
+ *
+ * In the former case, it is the mm setting of the group leader that really
+ * matters. So cpuset_attach_old_cs should track the oldcs of the thread
+ * leader. In the latter case, effective_mems of child cpusets must always
+ * be a subset of the parent. So no real page migration will be necessary no
+ * matter which child cpuset is selected as cpuset_attach_old_cs.
*/
static struct cpuset *cpuset_attach_old_cs;
static bool attach_cpus_updated;
@@ -3091,6 +3105,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
if (ret)
goto out_unlock;
+ /* Update cpuset_attach_old_cs to the latest group leader */
+ if (task == task->group_leader)
+ cpuset_attach_old_cs = task_cs(task);
+
if (setsched_check) {
ret = security_task_setscheduler(task);
if (ret)
--
2.54.0
^ permalink raw reply related
* [PATCH-next v4 3/6] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
In-Reply-To: <20260529212108.120506-1-longman@redhat.com>
Expand the scope of cpuset_can_attach_check() by including the setting
of setsched flag inside cpuset_can_attach_check() with the new @oldcs
and @psetsched argument. As cpuset_can_attach_check() is also called
from cpuset_can_fork(), set the new arguments to NULL from that caller.
While at it, expose the source and destination cpuset cpu/memory check
results in the new attach_cpus_updated and attach_mems_updated static
flags so that these flags can be used directly from cpuset_attach()
without the need to do the same computations again.
Two new global attach related flags are added (attach_cpus_updated &
attach_mems_updated) which are set to indicate that CPUs or memory nodes
are updated. These 2 flags are set in cpuset_can_attach() and are used
in cpuset_attach() for optimization. Since cpuset_mutex will be released
between the 2 calls, it is possible that an intervening cpuset action
may change the CPU or node mask of the relevant cpusets, so check is
added to set these flags if the effective_cpus or effective_mems of
those cpusets is changed.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 90 ++++++++++++++++++++++++++++--------------
1 file changed, 60 insertions(+), 30 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index a6f191b48529..0f93f3d84494 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1108,6 +1108,14 @@ enum partition_cmd {
static void update_sibling_cpumasks(struct cpuset *parent, struct cpuset *cs,
struct tmpmasks *tmp);
+/*
+ * cpuset_can_attach() and cpuset_attach() specific internal data
+ * Protected by cpuset_mutex
+ */
+static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
+
/*
* Update partition exclusive flag
*
@@ -1192,6 +1200,8 @@ static void reset_partition_data(struct cpuset *cs)
}
if (!cpumask_and(cs->effective_cpus, parent->effective_cpus, cs->cpus_allowed))
cpumask_copy(cs->effective_cpus, parent->effective_cpus);
+ if (cs->attach_in_progress)
+ attach_cpus_updated = true;
}
/*
@@ -1242,6 +1252,8 @@ static void partition_xcpus_add(int new_prs, struct cpuset *parent,
xcpus);
cpumask_andnot(parent->effective_cpus, parent->effective_cpus, xcpus);
+ if (parent->attach_in_progress)
+ attach_cpus_updated = true;
}
/*
@@ -1269,6 +1281,8 @@ static void partition_xcpus_del(int old_prs, struct cpuset *parent,
cpumask_or(parent->effective_cpus, parent->effective_cpus, xcpus);
cpumask_and(parent->effective_cpus, parent->effective_cpus, cpu_active_mask);
+ if (parent->attach_in_progress)
+ attach_cpus_updated = true;
}
/*
@@ -2217,6 +2231,8 @@ static void update_cpumasks_hier(struct cpuset *cs, struct tmpmasks *tmp,
if (new_prs <= 0)
reset_partition_data(cp);
spin_unlock_irq(&callback_lock);
+ if (cp->attach_in_progress)
+ attach_cpus_updated = true;
notify_partition_change(cp, old_prs);
@@ -2720,6 +2736,8 @@ static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems)
spin_lock_irq(&callback_lock);
cp->effective_mems = *new_mems;
spin_unlock_irq(&callback_lock);
+ if (cp->attach_in_progress)
+ attach_mems_updated = true;
WARN_ON(!is_in_v2_mode() &&
!nodes_equal(cp->mems_allowed, cp->effective_mems));
@@ -2976,19 +2994,48 @@ static int update_prstate(struct cpuset *cs, int new_prs)
return 0;
}
-static struct cpuset *cpuset_attach_old_cs;
-
/*
* Check to see if a cpuset can accept a new task
* For v1, cpus_allowed and mems_allowed can't be empty.
* For v2, effective_cpus can't be empty.
* Note that in v1, effective_cpus = cpus_allowed.
+ *
+ * Also set the boolean flag passed in by @psetsched depending on if
+ * security_task_setscheduler() call is needed and @oldcs is not NULL.
*/
-static int cpuset_can_attach_check(struct cpuset *cs)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
+ bool *psetsched)
{
if (cpumask_empty(cs->effective_cpus) ||
(!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
return -ENOSPC;
+
+ if (!oldcs)
+ return 0;
+
+ /*
+ * Update attach specific data
+ */
+ attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+ attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+ /*
+ * Skip rights over task setsched check in v2 when nothing changes,
+ * migration permission derives from hierarchy ownership in
+ * cgroup_procs_write_permission()).
+ */
+ *psetsched = !cpuset_v2() || attach_cpus_updated || attach_mems_updated;
+
+ /*
+ * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
+ * brings the last online CPU offline as users are not allowed to empty
+ * cpuset.cpus when there are active tasks inside. When that happens,
+ * we should allow tasks to migrate out without security check to make
+ * sure they will be able to run after migration.
+ */
+ if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
+ *psetsched = false;
+
return 0;
}
@@ -3035,29 +3082,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
mutex_lock(&cpuset_mutex);
/* Check to see if task is allowed in the cpuset */
- ret = cpuset_can_attach_check(cs);
+ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
if (ret)
goto out_unlock;
- /*
- * Skip rights over task setsched check in v2 when nothing changes,
- * migration permission derives from hierarchy ownership in
- * cgroup_procs_write_permission()).
- */
- setsched_check = !cpuset_v2() ||
- !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
- !nodes_equal(cs->effective_mems, oldcs->effective_mems);
-
- /*
- * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
- * brings the last online CPU offline as users are not allowed to empty
- * cpuset.cpus when there are active tasks inside. When that happens,
- * we should allow tasks to migrate out without security check to make
- * sure they will be able to run after migration.
- */
- if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
- setsched_check = false;
-
cgroup_taskset_for_each(task, css, tset) {
ret = task_can_attach(task);
if (ret)
@@ -3152,7 +3180,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct cgroup_subsys_state *css;
struct cpuset *cs;
struct cpuset *oldcs = cpuset_attach_old_cs;
- bool cpus_updated, mems_updated;
bool queue_task_work = false;
cgroup_taskset_first(tset, &css);
@@ -3160,9 +3187,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
- cpus_updated = !cpumask_equal(cs->effective_cpus,
- oldcs->effective_cpus);
- mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
/*
* In the default hierarchy, enabling cpuset in the child cgroups
@@ -3172,7 +3196,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
*/
if (cpuset_v2()) {
cpuset_attach_nodemask_to = cs->effective_mems;
- if (!cpus_updated && !mems_updated)
+ if (!attach_cpus_updated && !attach_mems_updated)
goto out;
} else {
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
@@ -3187,7 +3211,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* if there is no change in effective_mems and CS_MEMORY_MIGRATE is
* not set.
*/
- if (!is_memory_migrate(cs) && !mems_updated)
+ if (!is_memory_migrate(cs) && !attach_mems_updated)
goto out;
cgroup_taskset_for_each_leader(leader, css, tset) {
@@ -3602,7 +3626,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
mutex_lock(&cpuset_mutex);
/* Check to see if task is allowed in the cpuset */
- ret = cpuset_can_attach_check(cs);
+ ret = cpuset_can_attach_check(cs, NULL, NULL);
if (ret)
goto out_unlock;
@@ -3742,6 +3766,8 @@ hotplug_update_tasks(struct cpuset *cs,
cpumask_copy(cs->effective_cpus, new_cpus);
cs->effective_mems = *new_mems;
spin_unlock_irq(&callback_lock);
+ if (cs->attach_in_progress)
+ attach_cpus_updated = attach_mems_updated = true;
if (cpus_updated)
cpuset_update_tasks_cpumask(cs, new_cpus);
@@ -3927,6 +3953,8 @@ static void cpuset_handle_hotplug(void)
}
cpumask_copy(top_cpuset.effective_cpus, &new_cpus);
spin_unlock_irq(&callback_lock);
+ if (top_cpuset.attach_in_progress)
+ attach_cpus_updated = true;
/* we don't mess with cpumasks of tasks in top_cpuset */
}
@@ -3937,6 +3965,8 @@ static void cpuset_handle_hotplug(void)
top_cpuset.mems_allowed = new_mems;
top_cpuset.effective_mems = new_mems;
spin_unlock_irq(&callback_lock);
+ if (top_cpuset.attach_in_progress)
+ attach_mems_updated = true;
cpuset_update_tasks_nodemask(&top_cpuset);
}
--
2.54.0
^ permalink raw reply related
* [PATCH-next v4 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
In-Reply-To: <20260529212108.120506-1-longman@redhat.com>
Whenever memory node mask is changed, there are 4 places where the node
mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to
the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and
cpuset_attach(). However there are inconsistencies in what node masks
are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is
hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or
an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in
node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE].
However, node in node_states[N_ONLINE] may not have memory. So
node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should only be useful for v1 where
mems_allowed is the same as effective_mems. With v2, the memory nodes
in effective_mems should always be a subset of node_states[N_MEMORY],
so guarantee_online_mems() should just return cs->effective_mems.
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems. For v1, mpol_rebind_mm()
uses cpus_allowed which may differ from what guarantee_online_mems()
returns.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 34 +++++++++++++++++++++++-----------
1 file changed, 23 insertions(+), 11 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 51327333980a..961427cd83a5 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2615,6 +2615,13 @@ static void *cpuset_being_rebound;
* Iterate through each task of @cs updating its mems_allowed to the
* effective cpuset's. As this function is called with cpuset_mutex held,
* cpuset membership stays stable.
+ *
+ * - cpuset_change_task_nodemask(): guarantee_online_mems()
+ * - mpol_rebind_mm(): effective_mems
+ * - cpuset_migrate_mm(): guarantee_online_mems()
+ * - old_mems_allowed: guarantee_online_mems()
+ *
+ * For v2, guarantee_online_mems() should just return effective_mems.
*/
void cpuset_update_tasks_nodemask(struct cpuset *cs)
{
@@ -2624,7 +2631,10 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
cpuset_being_rebound = cs; /* causes mpol_dup() rebind */
- guarantee_online_mems(cs, &newmems);
+ if (cpuset_v2())
+ newmems = cs->effective_mems;
+ else
+ guarantee_online_mems(cs, &newmems);
/*
* The mpol_rebind_mm() call takes mmap_lock, which we couldn't
@@ -2649,7 +2659,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
migrate = is_memory_migrate(cs);
- mpol_rebind_mm(mm, &cs->mems_allowed);
+ mpol_rebind_mm(mm, &cs->effective_mems);
if (migrate)
cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
else
@@ -2713,6 +2723,8 @@ static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems)
WARN_ON(!is_in_v2_mode() &&
!nodes_equal(cp->mems_allowed, cp->effective_mems));
+ WARN_ON(cpuset_v2() &&
+ !nodes_subset(cp->effective_mems, node_states[N_MEMORY]));
cpuset_update_tasks_nodemask(cp);
@@ -3147,17 +3159,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)
/*
* In the default hierarchy, enabling cpuset in the child cgroups
- * will trigger a number of cpuset_attach() calls with no change
- * in effective cpus and mems. In that case, we can optimize out
- * by skipping the task iteration and update.
+ * will trigger a cpuset_attach() call with no change in effective cpus
+ * and mems. In that case, we can optimize out by skipping the task
+ * iteration and update.
*/
- if (cpuset_v2() && !cpus_updated && !mems_updated) {
+ if (cpuset_v2()) {
cpuset_attach_nodemask_to = cs->effective_mems;
- goto out;
+ if (!cpus_updated && !mems_updated)
+ goto out;
+ } else {
+ guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
}
- guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
-
cgroup_taskset_for_each(task, css, tset)
cpuset_attach_task(cs, task);
@@ -3167,7 +3180,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* if there is no change in effective_mems and CS_MEMORY_MIGRATE is
* not set.
*/
- cpuset_attach_nodemask_to = cs->effective_mems;
if (!is_memory_migrate(cs) && !mems_updated)
goto out;
@@ -3175,7 +3187,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct mm_struct *mm = get_task_mm(leader);
if (mm) {
- mpol_rebind_mm(mm, &cpuset_attach_nodemask_to);
+ mpol_rebind_mm(mm, &cs->effective_mems);
/*
* old_mems_allowed is the same with mems_allowed
--
2.54.0
^ permalink raw reply related
* [PATCH-next v4 0/6] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
v4:
- Add a new patch 1 to fix inconsistency in node mask usage in
cpuset_update_tasks_nodemask() and cpuset_attach() and adjust
the subsequent patches accordingly.
- Update patch 3 to set the update flags whenever the CPU or node
mask is updated to address issue reported by Sashiko.
- Update patch 5 to remove unneeded setting of old_mems_allowed as
well as calling schedule_flush_migrate_mm() if queue_task_work is
set.
v3:
- Rebased to the lastest linux-next tree.
- Keep cpuset_attach_old_cs as suggested by Chen Ridong and replace
patch 3 by a new one to make it track task group leader.
Sashiko AI review of another cpuset patch had found that cpuset_attach()
and cpuset_can_attach() can be passed a cgroup_taskset with tasks
migrating from one source cpuset to multiple destination cpusets and
vice versa. Further testing of the cpuset code indicates that this is
indeed the case when the v2 cpuset controller is enabled or disabled.
Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that
there will be one source and one destinaton cpuset which may result in
inocrrect behavior.
This patch series is created to fix this issue.
Patch 1 is to fix an inconsistency in the way node mask update is being
handled in cpuset_update_tasks_nodemask() and cpuset_attach() so that
they match each other.
Patches 2 and 3 are just preparatory patches to make the remaining
patches easier to review.
Patch 4 makes cpuset_attach_old_cs to track group leader for use by
cpuset_migrate_mm().
Patch 5 moves mpol_rebind_mm() and cpuset_migrate_mm() inside
cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works
more like moving task from one cpuset to another one, while also make
supporting multiple source and destination cpusets easier.
Patch 6 makes the necessary changes to enable the support of multiple
source and destination cpusets by keeping all the source and destination
cpusets found during task iterations in two singly linked lists for
source and destination cpusets respectively.
Waiman Long (6):
cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask() and cpuset_attach()
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders
cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source/destination cpusets for
cpuset_*attach()
kernel/cgroup/cpuset-internal.h | 6 +
kernel/cgroup/cpuset.c | 424 +++++++++++++++++++++++---------
2 files changed, 308 insertions(+), 122 deletions(-)
--
2.54.0
^ permalink raw reply
* [PATCH-next v4 2/6] cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
From: Waiman Long @ 2026-05-29 21:21 UTC (permalink / raw)
To: Chen Ridong, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
In-Reply-To: <20260529212108.120506-1-longman@redhat.com>
Extract the DL bandwidth allocation code in cpuset_attach() to a new
cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------
1 file changed, 30 insertions(+), 23 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 961427cd83a5..a6f191b48529 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2992,6 +2992,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)
return 0;
}
+static int cpuset_reserve_dl_bw(struct cpuset *cs)
+{
+ int cpu, ret;
+
+ if (!cs->sum_migrate_dl_bw)
+ return 0;
+
+ cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+ if (unlikely(cpu >= nr_cpu_ids))
+ return -EINVAL;
+
+ ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ if (ret)
+ return ret;
+
+ cs->dl_bw_cpu = cpu;
+ return 0;
+}
+
static void reset_migrate_dl_data(struct cpuset *cs)
{
cs->nr_migrate_dl_tasks = 0;
@@ -3006,7 +3025,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
struct cpuset *cs, *oldcs;
struct task_struct *task;
bool setsched_check;
- int cpu, ret;
+ int ret;
/* used later by cpuset_attach() */
cpuset_attach_old_cs = task_cs(cgroup_taskset_first(tset, &css));
@@ -3062,31 +3081,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
}
}
- if (!cs->sum_migrate_dl_bw)
- goto out_success;
-
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids)) {
- ret = -EINVAL;
- goto out_unlock;
- }
-
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
- if (ret)
- goto out_unlock;
-
- cs->dl_bw_cpu = cpu;
-
-out_success:
- /*
- * Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
- cs->attach_in_progress++;
+ ret = cpuset_reserve_dl_bw(cs);
out_unlock:
- if (ret)
+ if (ret) {
reset_migrate_dl_data(cs);
+ } else {
+ /*
+ * Mark attach is in progress. This makes validate_change() fail
+ * changes which zero cpus/mems_allowed.
+ */
+ cs->attach_in_progress++;
+ }
+
mutex_unlock(&cpuset_mutex);
return ret;
}
--
2.54.0
^ permalink raw reply related
* Re: [PATCH rdma-next v2 0/3] cgroup/rdma: add MR memory size resource tracking
From: yanjun.zhu @ 2026-05-29 21:14 UTC (permalink / raw)
To: Tao Cui, tj, hannes, mkoutny, leon, jgg; +Cc: linux-rdma, cgroups, Tao Cui
In-Reply-To: <20260529090733.2242822-1-cui.tao@linux.dev>
On 5/29/26 2:07 AM, Tao Cui wrote:
> From: Tao Cui <cuitao@kylinos.cn>
>
> Currently the RDMA cgroup only tracks two aggregate counters:
> hca_handle and hca_object. The real scarce resource in multi-tenant
> deployments is pinned memory: how much physical memory gets registered
> through MRs. The existing hca_object counter is too coarse to capture
> this.
>
> This series adds a single new resource type:
>
> - mr_mem - Cumulative MR memory size in bytes
>
> The per-object-type counters (qp, mr) from RFC v1 have been removed
> per review feedback [1]: modern NICs pool objects from the same memory
> pool so the distinction between QP count and MR count is not
> meaningful for resource limiting. hca_object remains sufficient for
> coarse object accounting.
>
> After this series, an administrator can set limits like:
>
> echo "mlx5_0 mr_mem=1073741824" > rdma.max
>
Hi,
Thanks for the patchset! Introducing `mr_mem` to track and limit pinned
memory size is a very practical enhancement for multi-tenant deployments.
I have a question regarding how this new resource type interacts with
Fast Registration (FRWR / FRMR), which is widely used in production
environments (e.g., NVMe-oF, iSER) to achieve high performance.
As we know, FRWR decouples the MR object allocation (`ib_alloc_mr`) from
the actual memory page mapping (`ib_map_mr_sg`). The creation of FRWR
Memory Regions is often managed via a pre-allocated page pool.
Could you clarify how `mr_mem` accounts for FRWR in the following scenarios?
1. Accounting Granularity: Does `mr_mem` charge the maximum capacity of
the FRWR object at its allocation time (`ib_alloc_mr`), or does it
dynamically track the actual mapped bytes during the fast-reg data
path? If it's the former, it represents a "static maximum budget" per
pool, which seems more practical for performance.
2. Kernel-space vs Userspace: FRWR pools are frequently allocated by
kernel-space drivers (like NVMe-oF target/host). If these kernel
threads are not bound to a specific user cgroup, will their FRWR
allocations end up in the root cgroup, potentially bypassing the
per-tenant limits?
Don't you think it would be beneficial to explicitly document or
consider the FRWR pattern in the design section, given its prevalence in
real-world storage and networking workloads?
Thanks,
Zhu Yanjun
> Design
> ~~~~~~
>
> mr_mem is not page-level ownership tracking; it is object-based
> accounting tied to the MR lifetime:
>
> - charged at MR registration time
> - uncharged at MR destruction time
> - the charge is pinned to the cgroup that created the MR for the
> entire lifetime of the MR object
>
> This model intentionally defines accounting semantics around MR
> object lifetime rather than page ownership:
>
> 1. fork(): fork() does not duplicate MR objects. Even though the
> child inherits the uverbs fd and can access the parent's ucontext,
> the MR remains a single kernel object. The charge is tied to the
> MR object, not to the number of processes that can reach it, so
> no splitting or re-accounting is needed.
>
> 2. Cgroup migration: mr_mem follows the same semantics as the existing
> hca_object; charge at creation time against the invoking task's
> cgroup, uncharge at destruction time. The RDMA cgroup does not
> implement can_attach/attach callbacks today, so charges do not
> migrate with the task. This is a known limitation that applies
> equally to hca_handle and hca_object. mr_mem does not introduce
> any new complication here.
>
> 3. Overlap with memory cgroup: mr_mem does not count process memory
> usage; it represents a per-device DMA registration budget: the
> amount of memory this cgroup may register through a given HCA.
> This is a different dimension from what memory cgroup tracks. An
> administrator might set mr_mem limits differently per device, which
> memory cgroup cannot express.
>
> In particular, mr_mem tracks the registered memory range associated
> with the MR rather than exact dynamically pinned pages (e.g. for
> ODP MRs). This is a stable, policy-oriented approximation of
> registration footprint, not an attempt at precise physical page
> accounting.
>
> Tao Cui (3):
> cgroup/rdma: extend charge/uncharge API with s64 amount parameter
> cgroup/rdma: add MR memory size resource tracking
> cgroup/rdma: update cgroup resource list for MR_MEM
>
> Documentation/admin-guide/cgroup-v2.rst | 21 ++--
> drivers/infiniband/core/cgroup.c | 10 +-
> drivers/infiniband/core/core_priv.h | 12 +-
> drivers/infiniband/core/rdma_core.c | 20 +++-
> drivers/infiniband/core/uverbs_cmd.c | 61 +++++++++-
> drivers/infiniband/core/uverbs_std_types_mr.c | 37 ++++++
> include/linux/cgroup_rdma.h | 8 +-
> include/rdma/ib_verbs.h | 1 +
> kernel/cgroup/rdma.c | 108 +++++++++++++-----
> 9 files changed, 219 insertions(+), 59 deletions(-)
>
> ---
> Changes from RFC v1:
>
> - Removed RDMACG_RESOURCE_QP and RDMACG_RESOURCE_MR per-type
> counters following review feedback from Jason Gunthorpe [1].
> - Retained only RDMACG_RESOURCE_MR_MEM as the sole new resource.
> - Added detailed semantic notes to the commit messages addressing
> fork(), cgroup migration, and overlap with memory cgroup [2].
> - Renamed patches to reflect the narrower scope.
>
> [1] https://lore.kernel.org/all/20260525134314.GI7702@ziepe.ca/
> [2] https://lore.kernel.org/all/20260528075537.2170697-1-cuitao@kylinos.cn/
^ permalink raw reply
* Re: [PATCH 5/5] cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()
From: Mark Brown @ 2026-05-29 21:08 UTC (permalink / raw)
To: Tejun Heo
Cc: Johannes Weiner, Michal Koutný, Sebastian Andrzej Siewior,
Petr Malat, Bert Karwatzki, kernel test robot, Martin Pitt,
cgroups, linux-kernel, Aishwarya.TCV
In-Reply-To: <ahnMCQuw2K6zA3Hs@slm.duckdns.org>
[-- Attachment #1: Type: text/plain, Size: 736 bytes --]
On Fri, May 29, 2026 at 07:25:29AM -1000, Tejun Heo wrote:
> On Wed, May 27, 2026 at 11:45:54AM +0100, Mark Brown wrote:
> > On Mon, May 04, 2026 at 02:51:21PM -1000, Tejun Heo wrote:
> > with no further output and given that this is a cgroup locking change
> > this does seem like a plausible commmit, though I didn't look into it in
> > detail. Bisect log and the list of LTP tests we're running in our test
> > job below. We are running multuple tests in parallel.
> Unfortunately, I can't reproduce this in my environment. Any chance you can
> try testing on x86 tooa nd see whether it produces there?
Not readily sadly, I'll see if I can figure something out. Our rootfs
images are based on Debian Trixie if that's relevant?
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH v3 4/4] selftests/cgroup: Add tests for zswap proactive writeback
From: Nhat Pham @ 2026-05-29 20:02 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260526114601.67041-5-jiahao.kernel@gmail.com>
On Tue, May 26, 2026 at 4:46 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
> From: Hao Jia <jiahao1@lixiang.com>
>
> Add test_zswap_proactive_writeback() to cover the new memory.reclaim
> "zswap_writeback_only" key. The test populates a memory cgroup zswap
> pool, triggers proactive writeback, and verifies the behavior by
> observing the change in zswpwb_proactive. Invalid input combinations
> are also covered.
>
> Extend test_zswap_writeback_one() to assert that the existing
> non-proactive writeback path leaves zswpwb_proactive at zero.
>
> Signed-off-by: Hao Jia <jiahao1@lixiang.com>
LGTM.
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
^ permalink raw reply
* Re: [PATCH v3 3/4] mm/zswap: Add per-memcg stat for proactive writeback
From: Nhat Pham @ 2026-05-29 20:01 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260526114601.67041-4-jiahao.kernel@gmail.com>
On Tue, May 26, 2026 at 4:46 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
> From: Hao Jia <jiahao1@lixiang.com>
>
> Currently, zswap writeback can be triggered by either the pool limit
> being hit or by the proactive writeback mechanism. However, the
> existing 'zswpwb' metric in memory.stat and /proc/vmstat counts all
> written back pages, making it difficult to distinguish between pages
> written back due to the pool limit and those written back proactively.
>
> Add a new statistic 'zswpwb_proactive' to memory.stat and /proc/vmstat.
> This counter tracks the number of pages written back due to proactive
> writeback. This allows users to better monitor and tune the proactive
> writeback mechanism.
>
> Signed-off-by: Hao Jia <jiahao1@lixiang.com>
> ---
> Documentation/admin-guide/cgroup-v2.rst | 4 +++
> include/linux/vm_event_item.h | 1 +
> mm/memcontrol.c | 1 +
> mm/vmstat.c | 1 +
> mm/zswap.c | 41 ++++++++++++++++++-------
> 5 files changed, 37 insertions(+), 11 deletions(-)
>
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 6564abf0dec5..7d65aef83f7b 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -1748,6 +1748,10 @@ The following nested keys are defined.
> zswpwb
> Number of pages written from zswap to swap.
>
> + zswpwb_proactive
> + Number of pages written from zswap to swap by proactive
> + writeback. This is a subset of zswpwb.
> +
nit: I think this is specifically the zswap_writeback_only mode right?
Technically, normal proactive reclaim (memory.reclaim) can also hit zswap :)
Maybe some clarification here?
> zswap_incomp
> Number of incompressible pages currently stored in zswap
> without compression. These pages could not be compressed to
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index 03fe95f5a020..7a5bee0a20b6 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -138,6 +138,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
> ZSWPIN,
> ZSWPOUT,
> ZSWPWB,
> + ZSWPWB_PROACTIVE,
> #endif
> #ifdef CONFIG_X86
> DIRECT_MAP_LEVEL2_SPLIT,
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index e205e5de193d..7648b3fd940e 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -571,6 +571,7 @@ static const unsigned int memcg_vm_event_stat[] = {
> ZSWPIN,
> ZSWPOUT,
> ZSWPWB,
> + ZSWPWB_PROACTIVE,
> #endif
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> THP_FAULT_ALLOC,
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index f534972f517d..66fd06d1bb01 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1452,6 +1452,7 @@ const char * const vmstat_text[] = {
> [I(ZSWPIN)] = "zswpin",
> [I(ZSWPOUT)] = "zswpout",
> [I(ZSWPWB)] = "zswpwb",
> + [I(ZSWPWB_PROACTIVE)] = "zswpwb_proactive",
> #endif
> #ifdef CONFIG_X86
> [I(DIRECT_MAP_LEVEL2_SPLIT)] = "direct_map_level2_splits",
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 7bcbf788f634..b45d094f532a 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -160,6 +160,11 @@ struct zswap_pool {
> char tfm_name[CRYPTO_MAX_ALG_NAME];
> };
>
> +struct zswap_shrink_walk_arg {
> + bool proactive;
> + bool encountered_page_in_swapcache;
> +};
> +
> /* Global LRU lists shared by all zswap pools. */
> static struct list_lru zswap_list_lru;
>
> @@ -1042,7 +1047,8 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
> * freed.
> */
> static int zswap_writeback_entry(struct zswap_entry *entry,
> - swp_entry_t swpentry)
> + swp_entry_t swpentry,
> + bool proactive)
> {
> struct xarray *tree;
> pgoff_t offset = swp_offset(swpentry);
> @@ -1097,6 +1103,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
> if (entry->objcg)
> count_objcg_events(entry->objcg, ZSWPWB, 1);
>
> + if (proactive) {
> + count_vm_event(ZSWPWB_PROACTIVE);
> + if (entry->objcg)
> + count_objcg_events(entry->objcg, ZSWPWB_PROACTIVE, 1);
> + }
> +
With the above clarification, the rest LGTM.
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
^ permalink raw reply
* Re: [PATCH v3 2/4] mm/zswap: Implement proactive writeback
From: Nhat Pham @ 2026-05-29 19:58 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260526114601.67041-3-jiahao.kernel@gmail.com>
On Tue, May 26, 2026 at 4:46 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
> From: Hao Jia <jiahao1@lixiang.com>
>
> Zswap currently writes back pages to backing swap reactively, triggered
> either by the shrinker or when the pool reaches its size limit. There is
> no mechanism to control the amount of writeback for a specific memory
> cgroup. However, users may want to proactively write back zswap pages,
> e.g., to free up memory for other applications or to prepare for
> memory-intensive workloads.
>
> Introduce a "zswap_writeback_only" key to the memory.reclaim cgroup
> interface. When specified, this key bypasses standard memory reclaim
> and exclusively performs proactive zswap writeback up to the requested
> budget. If omitted, the default reclaim behavior remains unchanged.
>
> Example usage:
> # Write back 100MB of pages from zswap to the backing swap
> echo "100M zswap_writeback_only" > memory.reclaim
Hmmm, so this 100MB is the pre-compression size? i.e if this 100 MB
compresses to 25 MB, then you're only freeing 25 MB?
I'm ok-ish with this, but can you document it?
The rest seems solid to me, FWIW. I'll defer to Johannes and Yosry for
opinions on zswap-only proactive reclaim.
^ permalink raw reply
* Re: [PATCH v3 1/4] mm/zswap: Make shrink_worker writeback cursor per-memcg
From: Nhat Pham @ 2026-05-29 19:51 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260526114601.67041-2-jiahao.kernel@gmail.com>
On Tue, May 26, 2026 at 4:46 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
> From: Hao Jia <jiahao1@lixiang.com>
>
> The zswap background writeback worker shrink_worker() uses a global
> cursor zswap_next_shrink, protected by zswap_shrink_lock, to round-robin
> across the online memcgs under root_mem_cgroup.
>
> Proactive writeback also wants a similar per-memcg cursor that is
> scoped to the specified memcg, so that repeated invocations against
> the same memcg make forward progress across its descendant memcgs
> instead of restarting from the first child memcg each time.
>
> Naturally, group the cursor and its protecting spinlock into a
> zswap_wb_iter struct, and make it a member of struct mem_cgroup to
> realize per-memcg cursor management. Accordingly, shrink_worker() now
> uses the lock and cursor in root_mem_cgroup->zswap_wb_iter.
>
> Because the cursor is now per-memcg, the offline cleanup must visit
> every ancestor that could be holding a reference to the dying memcg.
> Factor out __zswap_memcg_offline_cleanup() and walk from dead_memcg up
> to the root.
>
> No functional change intended for shrink_worker().
LGTM, if the memcg maintainers are happy with the overhead.
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox