* [PATCH v2 1/2] cgroup/cpuset: Use effective_xcpus in partcmd_update add/del mask calculation
From: Sun Shaojie @ 2026-05-27 6:43 UTC (permalink / raw)
To: Waiman Long, Chen Ridong, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, zhangguopeng, Sun Shaojie
In-Reply-To: <20260527064329.640060-1-sunshaojie@kylinos.cn>
When sibling CPU exclusion occurs, a partition's user_xcpus may contain
CPUs that were never actually granted to it. These CPUs are present in
user_xcpus(cs) but not in cs->effective_xcpus.
The partcmd_update path in update_parent_effective_cpumask() uses
user_xcpus(cs) (via the local variable xcpus) to compute the addmask
(CPUs to return to parent) and delmask (CPUs to request from parent).
This is incorrect:
1) When newmask removes a CPU that was previously excluded by a
sibling, addmask incorrectly includes that CPU and tries to return
it to the parent even though the partition never actually owned it,
causing CPU overlap with sibling partitions and triggering warnings
in generate_sched_domains().
2) When newmask adds a previously excluded CPU that is now available,
delmask fails to request it from the parent because user_xcpus(cs)
already includes it.
Fix this by using cs->effective_xcpus instead of user_xcpus(cs) in all
partcmd_update paths that calculate addmask or delmask, including the
PERR_NOCPUS error handling paths.
Reproducers:
Example 1 - Removing a sibling-excluded CPU incorrectly returns it:
# cd /sys/fs/cgroup
# echo "0-1" > a1/cpuset.cpus
# echo "root" > a1/cpuset.cpus.partition
# echo "0-2" > b1/cpuset.cpus
# echo "root" > b1/cpuset.cpus.partition
# echo "2" > b1/cpuset.cpus
# cat cpuset.cpus.effective
# Actual: 0-1,3 Expected: 3
Example 2 - Expanding to a previously excluded CPU fails to request it:
# cd /sys/fs/cgroup
# echo "0-1" > a1/cpuset.cpus
# echo "root" > a1/cpuset.cpus.partition
# echo "0-2" > b1/cpuset.cpus
# echo "root" > b1/cpuset.cpus.partition
# echo "member" > a1/cpuset.cpus.partition
# echo "1-2" > b1/cpuset.cpus
# cat cpuset.cpus.effective
# Actual: 0-1,3 Expected: 0,3
Fixes: 2a3602030d80 ("cgroup/cpuset: Don't invalidate sibling partitions on cpuset.cpus conflict")
Suggested-by: Zhang Guopeng <zhangguopeng@kylinos.cn>
Signed-off-by: Sun Shaojie <sunshaojie@kylinos.cn>
---
kernel/cgroup/cpuset.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 1335e437098e..2395c5aec871 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1807,9 +1807,9 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
* Compute add/delete mask to/from effective_cpus
*
* For valid partition:
- * addmask = exclusive_cpus & ~newmask
+ * addmask = effective_xcpus & ~newmask
* & parent->effective_xcpus
- * delmask = newmask & ~exclusive_cpus
+ * delmask = newmask & ~effective_xcpus
* & parent->effective_xcpus
*
* For invalid partition:
@@ -1821,11 +1821,11 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
deleting = cpumask_and(tmp->delmask,
newmask, parent->effective_xcpus);
} else {
- cpumask_andnot(tmp->addmask, xcpus, newmask);
+ cpumask_andnot(tmp->addmask, cs->effective_xcpus, newmask);
adding = cpumask_and(tmp->addmask, tmp->addmask,
parent->effective_xcpus);
- cpumask_andnot(tmp->delmask, newmask, xcpus);
+ cpumask_andnot(tmp->delmask, newmask, cs->effective_xcpus);
deleting = cpumask_and(tmp->delmask, tmp->delmask,
parent->effective_xcpus);
}
@@ -1864,7 +1864,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
part_error = PERR_NOCPUS;
deleting = false;
adding = cpumask_and(tmp->addmask,
- xcpus, parent->effective_xcpus);
+ cs->effective_xcpus, parent->effective_xcpus);
}
} else {
/*
@@ -1886,7 +1886,8 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
part_error = PERR_NOCPUS;
if (is_partition_valid(cs))
adding = cpumask_and(tmp->addmask,
- xcpus, parent->effective_xcpus);
+ cs->effective_xcpus,
+ parent->effective_xcpus);
} else if (is_partition_invalid(cs) && !cpumask_empty(xcpus) &&
cpumask_subset(xcpus, parent->effective_xcpus)) {
struct cgroup_subsys_state *css;
--
2.43.0
^ permalink raw reply related
* [PATCH v2 0/2] cgroup/cpuset: Fix sibling CPU exclusion in partcmd_update
From: Sun Shaojie @ 2026-05-27 6:43 UTC (permalink / raw)
To: Waiman Long, Chen Ridong, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, zhangguopeng, Sun Shaojie
Fix the partcmd_update path to use effective_xcpus instead of user_xcpus
for add/del mask calculation, and add test cases to verify the fix.
Changes in v2:
- Updated comments to match the code change (suggested by Zhang Guopeng)
- Added test cases for the fix
Sun Shaojie (2):
cgroup/cpuset: Use effective_xcpus in partcmd_update add/del mask
calculation
cgroup/cpuset: Add test cases for sibling CPU exclusion on partition
update
kernel/cgroup/cpuset.c | 13 +++++++------
tools/testing/selftests/cgroup/test_cpuset_prs.sh | 10 ++++++++++
2 files changed, 17 insertions(+), 6 deletions(-)
--
2.43.0
^ permalink raw reply
* [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask
From: Youngjun Park @ 2026-05-27 6:22 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, gunho.lee, taejoon.song,
hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260527062247.3440692-1-youngjun.park@lge.com>
Apply memcg tier effective mask during swap slot allocation to
enforce per-cgroup swap tier restrictions.
In the fast path, check the percpu cached swap_info's tier_mask
against the folio's effective mask. If it does not match, fall
through to the slow path. In the slow path, skip swap devices
whose tier_mask is not covered by the folio's effective mask.
This works correctly when there is only one non-rotational
device in the system and no devices share the same priority.
However, there are known limitations:
- When non-rotational devices are distributed across multiple
tiers, and different memcgs are configured to use those
distinct tiers, they may constantly overwrite the shared
percpu swap cache. This cache thrashing leads to frequent
fast path misses.
- Combined with the above issue, if same-priority devices exist
among them, a percpu cache miss (overwritten by another memcg)
forces the allocator to round-robin to the next device
prematurely, even if the current cluster is not fully
exhausted.
These edge cases do not affect the primary use case of
directing swap traffic per cgroup. Further optimization is
planned for future work.
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
mm/swapfile.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 9a86ebe992f4..1a2d29735b71 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1365,14 +1365,18 @@ static bool swap_alloc_fast(struct folio *folio)
struct swap_cluster_info *ci;
struct swap_info_struct *si;
unsigned int offset;
+ int mask = folio_tier_effective_mask(folio);
/*
* Once allocated, swap_info_struct will never be completely freed,
* so checking it's liveness by get_swap_device_info is enough.
*/
si = this_cpu_read(percpu_swap_cluster.si[order]);
+ if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
+ return false;
+
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
- if (!si || !offset || !get_swap_device_info(si))
+ if (!offset || !get_swap_device_info(si))
return false;
ci = swap_cluster_lock(si, offset);
@@ -1392,10 +1396,14 @@ static bool swap_alloc_fast(struct folio *folio)
static void swap_alloc_slow(struct folio *folio)
{
struct swap_info_struct *si, *next;
+ int mask = folio_tier_effective_mask(folio);
spin_lock(&swap_avail_lock);
start_over:
plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+ if (!swap_tiers_mask_test(si->tier_mask, mask))
+ continue;
+
/* Rotate the device and switch to a new cluster */
plist_requeue(&si->avail_list, &swap_avail_head);
spin_unlock(&swap_avail_lock);
--
2.34.1
^ permalink raw reply related
* [PATCH v7 3/4] mm: memcontrol: add interfaces for swap tier selection
From: Youngjun Park @ 2026-05-27 6:22 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, gunho.lee, taejoon.song,
hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260527062247.3440692-1-youngjun.park@lge.com>
Integrate swap tier infrastructure with cgroup to allow selecting
specific swap devices per cgroup.
Introduce memory.swap.tiers for configuring allowed tiers, and
memory.swap.tiers.effective for exposing the effective tiers.
The effective tiers are the intersection of the configured tiers
and the parent's effective tiers.
Note that cgroups do not pin swap tiers, similar to cpuset and CPU
hotplug, allowing configuration changes regardless of usage.
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
Documentation/admin-guide/cgroup-v2.rst | 29 +++++++
include/linux/memcontrol.h | 5 ++
mm/memcontrol.c | 96 +++++++++++++++++++++
mm/swap_state.c | 5 +-
mm/swap_tier.c | 107 +++++++++++++++++++++++-
mm/swap_tier.h | 56 +++++++++++--
6 files changed, 288 insertions(+), 10 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed99..08253072a252 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1850,6 +1850,35 @@ The following nested keys are defined.
Swap usage hard limit. If a cgroup's swap usage reaches this
limit, anonymous memory of the cgroup will not be swapped out.
+ memory.swap.tiers
+ A read-write file which exists on non-root cgroups.
+ Format is similar to cgroup.subtree_control.
+
+ Controls which swap tiers this cgroup is allowed to swap
+ out to. All tiers are enabled by default.
+
+ ::
+
+ (-|+)TIER [(-|+)TIER ...]
+
+ "-" disables a tier, "+" re-enables it.
+ Entries are whitespace-delimited.
+
+ Changes here are combined with parent restrictions to
+ compute memory.swap.tiers.effective.
+
+ If a tier is removed from /sys/kernel/mm/swap/tiers,
+ any prior disable for that tier is invalidated.
+
+ memory.swap.tiers.effective
+ A read-only file which exists on non-root cgroups.
+
+ Shows the tiers this cgroup can actually swap out to.
+ This is the intersection of the parent's effective tiers
+ and this cgroup's own memory.swap.tiers configuration.
+ A child cannot enable a tier that is disabled in its
+ parent.
+
memory.swap.events
A read-only flat-keyed file which exists on non-root cgroups.
The following entries are defined. Unless specified
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index bf1a6e131eca..eb33c8e30c9e 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -287,6 +287,11 @@ struct mem_cgroup {
struct lru_gen_mm_list mm_list;
#endif
+#ifdef CONFIG_SWAP
+ int tier_mask;
+ int tier_effective_mask;
+#endif
+
#ifdef CONFIG_MEMCG_V1
/* Legacy consumer-oriented counters */
struct page_counter kmem; /* v1 only */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index e24114a4493a..cbc7a519a24d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -68,6 +68,7 @@
#include <net/ip.h>
#include "slab.h"
#include "memcontrol-v1.h"
+#include "swap_tier.h"
#include <linux/uaccess.h>
@@ -4249,6 +4250,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
refcount_set(&memcg->id.ref, 1);
css_get(css);
+ swap_tiers_memcg_inherit_mask(memcg);
+
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
*
@@ -5791,6 +5794,88 @@ static int swap_events_show(struct seq_file *m, void *v)
return 0;
}
+static int swap_tier_show(struct seq_file *m, void *v)
+{
+ struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+
+ swap_tiers_mask_show(m, READ_ONCE(memcg->tier_mask));
+ return 0;
+}
+
+static ssize_t swap_tier_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
+ char *pos, *token;
+ int ret = 0;
+ int original_mask = 0;
+
+ pos = strstrip(buf);
+
+ spin_lock(&swap_tier_lock);
+ if (!*pos) {
+ WRITE_ONCE(memcg->tier_mask, TIER_ALL_MASK);
+ goto sync;
+ }
+
+ original_mask = memcg->tier_mask;
+
+ while ((token = strsep(&pos, " \t\n")) != NULL) {
+ int mask;
+
+ if (!*token)
+ continue;
+
+ if (token[0] != '-' && token[0] != '+') {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ mask = swap_tiers_mask_lookup(token+1);
+ if (!mask) {
+ ret = -EINVAL;
+ goto err;
+ }
+ /*
+ * tier_mask can be modified independently at each memcg.
+ * However, the effective mask is restricted to a subset of
+ * the parent's mask in swap_tiers_memcg_sync_mask().
+ */
+ switch (token[0]) {
+ case '-':
+ WRITE_ONCE(memcg->tier_mask,
+ memcg->tier_mask & ~mask);
+ break;
+ case '+':
+ WRITE_ONCE(memcg->tier_mask,
+ memcg->tier_mask | mask);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ if (ret)
+ goto err;
+ }
+
+sync:
+ swap_tiers_memcg_sync_mask(memcg);
+err:
+ if (ret)
+ WRITE_ONCE(memcg->tier_mask, original_mask);
+ spin_unlock(&swap_tier_lock);
+ return ret ? ret : nbytes;
+}
+
+static int swap_tier_effective_show(struct seq_file *m, void *v)
+{
+ struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+
+ swap_tiers_mask_show(m, READ_ONCE(memcg->tier_effective_mask));
+ return 0;
+}
+
static struct cftype swap_files[] = {
{
.name = "swap.current",
@@ -5823,6 +5908,17 @@ static struct cftype swap_files[] = {
.file_offset = offsetof(struct mem_cgroup, swap_events_file),
.seq_show = swap_events_show,
},
+ {
+ .name = "swap.tiers",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = swap_tier_show,
+ .write = swap_tier_write,
+ },
+ {
+ .name = "swap.tiers.effective",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = swap_tier_effective_show,
+ },
{ } /* terminate */
};
diff --git a/mm/swap_state.c b/mm/swap_state.c
index de285b36e31c..2fda6b61e2de 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -1011,6 +1011,7 @@ static ssize_t tiers_store(struct kobject *kobj,
char *p, *token, *name, *tmp;
int ret = 0;
short prio;
+ int mask = 0;
tmp = kstrdup(buf, GFP_KERNEL);
if (!tmp)
@@ -1043,7 +1044,7 @@ static ssize_t tiers_store(struct kobject *kobj,
goto restore;
break;
case '-':
- ret = swap_tiers_remove(token + 1);
+ ret = swap_tiers_remove(token + 1, &mask);
if (ret)
goto restore;
break;
@@ -1053,7 +1054,7 @@ static ssize_t tiers_store(struct kobject *kobj,
}
}
- if (!swap_tiers_update()) {
+ if (!swap_tiers_update(mask)) {
ret = -EINVAL;
goto restore;
}
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index 6b57cadb3e95..9c180f55a4e9 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -253,7 +253,7 @@ int swap_tiers_add(const char *name, int prio)
return ret;
}
-int swap_tiers_remove(const char *name)
+int swap_tiers_remove(const char *name, int *mask)
{
int ret = 0;
struct swap_tier *tier;
@@ -276,6 +276,7 @@ int swap_tiers_remove(const char *name)
list_prev_entry(tier, list)->prio = DEF_SWAP_PRIO;
swap_tier_inactivate(tier);
+ *mask |= TIER_MASK(tier);
return ret;
}
@@ -344,7 +345,26 @@ void swap_tiers_assign_dev(struct swap_info_struct *swp)
swp->tier_mask = TIER_DEFAULT_MASK;
}
-bool swap_tiers_update(void)
+#ifdef CONFIG_MEMCG
+static void swap_tier_memcg_propagate(int mask)
+{
+ struct mem_cgroup *child;
+
+ rcu_read_lock();
+ for_each_mem_cgroup_tree(child, root_mem_cgroup) {
+ WRITE_ONCE(child->tier_mask, child->tier_mask | mask);
+ WRITE_ONCE(child->tier_effective_mask,
+ child->tier_effective_mask | mask);
+ }
+ rcu_read_unlock();
+}
+#else
+static void swap_tier_memcg_propagate(int mask)
+{
+}
+#endif
+
+bool swap_tiers_update(int mask)
{
struct swap_tier *tier;
struct swap_info_struct *swp;
@@ -375,5 +395,88 @@ bool swap_tiers_update(void)
swap_tiers_assign_dev(swp);
}
+ /*
+ * When a tier is removed, its index (bit position in the mask) becomes
+ * free for reassignment to a future tier. If a memcg had previously
+ * disabled this tier (cleared the bit in its swap.tiers file), the
+ * effective mask would keep that bit clear -- meaning the new tier at
+ * the same index would be silently unavailable, an invisible cgroup
+ * constraint left behind by a tier that no longer exists.
+ *
+ * To prevent this, OR the removed tier's mask bit into every memcg's
+ * tier_mask and tier_effective_mask. This resets the bit so the new
+ * tier is accessible by default; users who want to restrict it must
+ * explicitly disable it after the tier is re-created.
+ */
+ if (mask)
+ swap_tier_memcg_propagate(mask);
+
return true;
}
+
+#ifdef CONFIG_MEMCG
+void swap_tiers_mask_show(struct seq_file *m, int mask)
+{
+ struct swap_tier *tier;
+
+ spin_lock(&swap_tier_lock);
+ for_each_active_tier(tier) {
+ if (mask & TIER_MASK(tier))
+ seq_printf(m, "%s ", tier->name);
+ }
+ spin_unlock(&swap_tier_lock);
+ seq_puts(m, "\n");
+}
+
+int swap_tiers_mask_lookup(const char *name)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ for_each_active_tier(tier) {
+ if (!strcmp(name, tier->name))
+ return TIER_MASK(tier);
+ }
+
+ return 0;
+}
+
+static void __swap_tier_memcg_inherit_mask(struct mem_cgroup *memcg,
+ struct mem_cgroup *parent)
+{
+ int parent_mask = parent
+ ? READ_ONCE(parent->tier_effective_mask)
+ : TIER_ALL_MASK;
+
+ WRITE_ONCE(memcg->tier_effective_mask,
+ parent_mask & READ_ONCE(memcg->tier_mask));
+}
+
+/* Computes the initial effective mask from the parent's effective mask. */
+void swap_tiers_memcg_inherit_mask(struct mem_cgroup *memcg)
+{
+ spin_lock(&swap_tier_lock);
+ rcu_read_lock();
+ memcg->tier_mask = TIER_ALL_MASK;
+ __swap_tier_memcg_inherit_mask(memcg, parent_mem_cgroup(memcg));
+ rcu_read_unlock();
+ spin_unlock(&swap_tier_lock);
+}
+
+/*
+ * Called when a memcg's tier_mask is modified. Walks the subtree
+ * and recomputes each descendant's effective mask against its parent.
+ */
+void swap_tiers_memcg_sync_mask(struct mem_cgroup *memcg)
+{
+ struct mem_cgroup *child;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ rcu_read_lock();
+ for_each_mem_cgroup_tree(child, memcg)
+ __swap_tier_memcg_inherit_mask(child, parent_mem_cgroup(child));
+ rcu_read_unlock();
+}
+#endif
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
index 3e355f857363..49433dcaa1ce 100644
--- a/mm/swap_tier.h
+++ b/mm/swap_tier.h
@@ -10,22 +10,66 @@ struct swap_info_struct;
extern spinlock_t swap_tier_lock;
-#define TIER_ALL_MASK (~0)
-#define TIER_DEFAULT_IDX (31)
-#define TIER_DEFAULT_MASK (1U << TIER_DEFAULT_IDX)
-
/* Initialization and application */
void swap_tiers_init(void);
ssize_t swap_tiers_sysfs_show(char *buf);
int swap_tiers_add(const char *name, int prio);
-int swap_tiers_remove(const char *name);
+int swap_tiers_remove(const char *name, int *mask);
void swap_tiers_snapshot(void);
void swap_tiers_snapshot_restore(void);
-bool swap_tiers_update(void);
+bool swap_tiers_update(int mask);
/* Tier assignment */
void swap_tiers_assign_dev(struct swap_info_struct *swp);
+#define TIER_ALL_MASK (~0)
+#define TIER_DEFAULT_IDX (31)
+#define TIER_DEFAULT_MASK (1U << TIER_DEFAULT_IDX)
+
+#if defined(CONFIG_SWAP) && defined(CONFIG_MEMCG)
+/* Memcg related functions */
+void swap_tiers_mask_show(struct seq_file *m, int mask);
+void swap_tiers_memcg_inherit_mask(struct mem_cgroup *memcg);
+void swap_tiers_memcg_sync_mask(struct mem_cgroup *memcg);
+int swap_tiers_mask_lookup(const char *name);
+static inline int folio_tier_effective_mask(struct folio *folio)
+{
+ struct mem_cgroup *memcg;
+ int mask = TIER_ALL_MASK;
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (memcg)
+ mask = READ_ONCE(memcg->tier_effective_mask);
+ rcu_read_unlock();
+
+ return mask;
+}
+#else
+static inline void swap_tiers_mask_show(struct seq_file *m, int mask) {}
+static inline void swap_tiers_memcg_inherit_mask(struct mem_cgroup *memcg) {}
+static inline void swap_tiers_memcg_sync_mask(struct mem_cgroup *memcg) {}
+static inline int swap_tiers_mask_lookup(const char *name)
+{
+ return 0;
+}
+static inline int folio_tier_effective_mask(struct folio *folio)
+{
+ return TIER_ALL_MASK;
+}
+#endif
+
+/**
+ * swap_tiers_mask_test - Check if the tier mask is valid
+ * @tier_mask: The tier mask to check
+ * @mask: The mask to compare against
+ *
+ * Return: true if condition matches, false otherwise
+ */
+static inline bool swap_tiers_mask_test(int tier_mask, int mask)
+{
+ return tier_mask & mask;
+}
#endif /* _SWAP_TIER_H */
--
2.34.1
^ permalink raw reply related
* [PATCH v7 1/4] mm: swap: introduce swap tier infrastructure
From: Youngjun Park @ 2026-05-27 6:22 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, gunho.lee, taejoon.song,
hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260527062247.3440692-1-youngjun.park@lge.com>
This patch introduces the "Swap tier" concept, which serves as an
abstraction layer for managing swap devices based on their performance
characteristics (e.g., NVMe, HDD, Network swap).
Swap tiers are user-named groups representing priority ranges.
Tier names must consist of alphanumeric characters and underscores.
These tiers collectively cover the entire priority space from -1
(`DEF_SWAP_PRIO`) to `SHRT_MAX`.
To configure tiers, a new sysfs interface is exposed at
/sys/kernel/mm/swap/tiers. The input parser evaluates commands from
left to right and supports batch input, allowing users to add or remove
multiple tiers in a single write operation.
Tier management enforces continuous priority ranges anchored by start
priorities. Operations trigger range splitting or merging, but overwriting
start priorities is forbidden. Merging expands lower tiers upwards to
preserve configured start priorities, except when removing `DEF_SWAP_PRIO`,
which merges downwards.
Suggested-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
MAINTAINERS | 2 +
mm/Kconfig | 12 ++
mm/Makefile | 2 +-
mm/swap.h | 4 +
mm/swap_state.c | 74 ++++++++++++
mm/swap_tier.c | 302 ++++++++++++++++++++++++++++++++++++++++++++++++
mm/swap_tier.h | 20 ++++
mm/swapfile.c | 8 +-
8 files changed, 420 insertions(+), 4 deletions(-)
create mode 100644 mm/swap_tier.c
create mode 100644 mm/swap_tier.h
diff --git a/MAINTAINERS b/MAINTAINERS
index e3ee97f5474e..e3cbfedbaa5f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17049,6 +17049,8 @@ F: mm/swap.c
F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
+F: mm/swap_tier.c
+F: mm/swap_tier.h
F: mm/swapfile.c
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
diff --git a/mm/Kconfig b/mm/Kconfig
index 776b67c66e82..5343937f3da9 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -19,6 +19,18 @@ menuconfig SWAP
used to provide more virtual memory than the actual RAM present
in your computer. If unsure say Y.
+config NR_SWAP_TIERS
+ int "Number of swap device tiers"
+ depends on SWAP
+ default 4
+ range 1 31
+ help
+ Sets the number of swap device tiers. Swap devices are
+ grouped into tiers based on their priority, allowing the
+ system to prefer faster devices over slower ones.
+
+ If unsure, say 4.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061..29cb1e778285 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -75,7 +75,7 @@ ifdef CONFIG_MMU
obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
endif
-obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o
+obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o swap_tier.o
obj-$(CONFIG_ZSWAP) += zswap.o
obj-$(CONFIG_HAS_DMA) += dmapool.o
obj-$(CONFIG_HUGETLBFS) += hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
diff --git a/mm/swap.h b/mm/swap.h
index 900a539c63f0..067f0afd7a3e 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -34,6 +34,10 @@ extern int page_cluster;
#define swap_entry_order(order) 0
#endif
+#define DEF_SWAP_PRIO -1
+
+extern spinlock_t swap_lock;
+extern struct plist_head swap_active_head;
extern struct swap_info_struct *swap_info[];
/*
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 04f5ce992401..e609bbdf7e13 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -25,6 +25,7 @@
#include "internal.h"
#include "swap_table.h"
#include "swap.h"
+#include "swap_tier.h"
/*
* swapper_space is a fiction, retained to simplify the path through
@@ -997,8 +998,81 @@ static ssize_t vma_ra_enabled_store(struct kobject *kobj,
}
static struct kobj_attribute vma_ra_enabled_attr = __ATTR_RW(vma_ra_enabled);
+static ssize_t tiers_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return swap_tiers_sysfs_show(buf);
+}
+
+static ssize_t tiers_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ char *p, *token, *name, *tmp;
+ int ret = 0;
+ short prio;
+
+ tmp = kstrdup(buf, GFP_KERNEL);
+ if (!tmp)
+ return -ENOMEM;
+
+ spin_lock(&swap_lock);
+ spin_lock(&swap_tier_lock);
+ swap_tiers_snapshot();
+
+ p = tmp;
+ while ((token = strsep(&p, ", \t\n")) != NULL) {
+ if (!*token)
+ continue;
+
+ switch (token[0]) {
+ case '+':
+ name = token + 1;
+ token = strchr(name, ':');
+ if (!token) {
+ ret = -EINVAL;
+ goto restore;
+ }
+ *token++ = '\0';
+ if (kstrtos16(token, 10, &prio)) {
+ ret = -EINVAL;
+ goto restore;
+ }
+ ret = swap_tiers_add(name, prio);
+ if (ret)
+ goto restore;
+ break;
+ case '-':
+ ret = swap_tiers_remove(token + 1);
+ if (ret)
+ goto restore;
+ break;
+ default:
+ ret = -EINVAL;
+ goto restore;
+ }
+ }
+
+ if (!swap_tiers_validate()) {
+ ret = -EINVAL;
+ goto restore;
+ }
+ goto out;
+
+restore:
+ swap_tiers_snapshot_restore();
+out:
+ spin_unlock(&swap_tier_lock);
+ spin_unlock(&swap_lock);
+ kfree(tmp);
+ return ret ? ret : count;
+}
+
+static struct kobj_attribute tier_attr = __ATTR_RW(tiers);
+
static struct attribute *swap_attrs[] = {
&vma_ra_enabled_attr.attr,
+ &tier_attr.attr,
NULL,
};
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
new file mode 100644
index 000000000000..ac7a3c2a48cb
--- /dev/null
+++ b/mm/swap_tier.c
@@ -0,0 +1,302 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/swap.h>
+#include <linux/memcontrol.h>
+#include "memcontrol-v1.h"
+#include <linux/sysfs.h>
+#include <linux/plist.h>
+
+#include "swap.h"
+#include "swap_tier.h"
+
+#define MAX_SWAPTIER CONFIG_NR_SWAP_TIERS
+#define MAX_TIERNAME 16
+
+/*
+ * struct swap_tier - structure representing a swap tier.
+ *
+ * @name: name of the swap_tier.
+ * @prio: starting value of priority.
+ * @list: linked list of tiers.
+ */
+static struct swap_tier {
+ char name[MAX_TIERNAME];
+ short prio;
+ struct list_head list;
+} swap_tiers[MAX_SWAPTIER];
+
+DEFINE_SPINLOCK(swap_tier_lock);
+/* active swap priority list, sorted in descending order */
+static LIST_HEAD(swap_tier_active_list);
+/* unused swap_tier object */
+static LIST_HEAD(swap_tier_inactive_list);
+
+#define TIER_IDX(tier) ((tier) - swap_tiers)
+#define TIER_MASK(tier) (1U << TIER_IDX(tier))
+#define TIER_INACTIVE_PRIO (DEF_SWAP_PRIO - 1)
+#define TIER_IS_ACTIVE(tier) ((tier->prio) != TIER_INACTIVE_PRIO)
+#define TIER_END_PRIO(tier) \
+ (!list_is_first(&(tier)->list, &swap_tier_active_list) ? \
+ list_prev_entry((tier), list)->prio - 1 : SHRT_MAX)
+
+#define for_each_tier(tier, idx) \
+ for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
+ idx++, tier = &swap_tiers[idx])
+
+#define for_each_active_tier(tier) \
+ list_for_each_entry(tier, &swap_tier_active_list, list)
+
+#define for_each_inactive_tier(tier) \
+ list_for_each_entry(tier, &swap_tier_inactive_list, list)
+
+/*
+ * Naming Convention:
+ * swap_tiers_*() - Public/exported functions
+ * swap_tier_*() - Private/internal functions
+ */
+
+static bool swap_tier_is_active(void)
+{
+ return !list_empty(&swap_tier_active_list);
+}
+
+static struct swap_tier *swap_tier_lookup(const char *name)
+{
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (!strcmp(tier->name, name))
+ return tier;
+ }
+
+ return NULL;
+}
+
+/* Insert new tier into the active list sorted by priority. */
+static void swap_tier_activate(struct swap_tier *new)
+{
+ struct list_head *pos = &swap_tier_active_list;
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (tier->prio <= new->prio) {
+ pos = &tier->list;
+ break;
+ }
+ }
+
+ list_add_tail(&new->list, pos);
+}
+
+static void swap_tier_inactivate(struct swap_tier *tier)
+{
+ list_move(&tier->list, &swap_tier_inactive_list);
+ tier->prio = TIER_INACTIVE_PRIO;
+}
+
+void swap_tiers_init(void)
+{
+ struct swap_tier *tier;
+ int idx;
+
+ BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER);
+
+ for_each_tier(tier, idx) {
+ INIT_LIST_HEAD(&tier->list);
+ swap_tier_inactivate(tier);
+ }
+}
+
+ssize_t swap_tiers_sysfs_show(char *buf)
+{
+ struct swap_tier *tier;
+ ssize_t len = 0;
+
+ len += sysfs_emit_at(buf, len, "%-16s %-5s %-11s %-11s\n",
+ "Name", "Idx", "PrioStart", "PrioEnd");
+
+ spin_lock(&swap_tier_lock);
+ for_each_active_tier(tier) {
+ len += sysfs_emit_at(buf, len, "%-16s %-5td %-11d %-11d\n",
+ tier->name,
+ TIER_IDX(tier),
+ tier->prio,
+ TIER_END_PRIO(tier));
+ }
+ spin_unlock(&swap_tier_lock);
+
+ return len;
+}
+
+static struct swap_tier *swap_tier_prepare(const char *name, short prio)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ if (prio < DEF_SWAP_PRIO)
+ return ERR_PTR(-EINVAL);
+
+ if (list_empty(&swap_tier_inactive_list))
+ return ERR_PTR(-ENOSPC);
+
+ tier = list_first_entry(&swap_tier_inactive_list,
+ struct swap_tier, list);
+
+ list_del_init(&tier->list);
+ strscpy(tier->name, name, MAX_TIERNAME);
+ tier->prio = prio;
+
+ return tier;
+}
+
+static int swap_tier_check_range(short prio)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ for_each_active_tier(tier) {
+ /* No overwrite */
+ if (tier->prio == prio)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static bool swap_tier_validate_name(const char *name)
+{
+ int len;
+
+ if (!name || !*name)
+ return false;
+
+ len = strlen(name);
+ if (len >= MAX_TIERNAME)
+ return false;
+
+ while (*name) {
+ if (!isalnum(*name) && *name != '_')
+ return false;
+ name++;
+ }
+ return true;
+}
+
+int swap_tiers_add(const char *name, int prio)
+{
+ int ret;
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ /* Duplicate check */
+ if (swap_tier_lookup(name))
+ return -EEXIST;
+
+ if (!swap_tier_validate_name(name))
+ return -EINVAL;
+
+ ret = swap_tier_check_range(prio);
+ if (ret)
+ return ret;
+
+ tier = swap_tier_prepare(name, prio);
+ if (IS_ERR(tier)) {
+ ret = PTR_ERR(tier);
+ return ret;
+ }
+
+ swap_tier_activate(tier);
+
+ return ret;
+}
+
+int swap_tiers_remove(const char *name)
+{
+ int ret = 0;
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ tier = swap_tier_lookup(name);
+ if (!tier)
+ return -EINVAL;
+
+ /* Removing DEF_SWAP_PRIO merges into the higher tier. */
+ if (!list_is_singular(&swap_tier_active_list)
+ && tier->prio == DEF_SWAP_PRIO)
+ list_prev_entry(tier, list)->prio = DEF_SWAP_PRIO;
+
+ swap_tier_inactivate(tier);
+
+ return ret;
+}
+
+static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
+/*
+ * XXX: When multiple operations (adds and removes) are submitted in a
+ * single write, reverting each individually on failure is complex and
+ * error-prone. Instead, snapshot the entire state beforehand and
+ * restore it wholesale if any operation fails.
+ */
+void swap_tiers_snapshot(void)
+{
+ BUILD_BUG_ON(sizeof(swap_tiers_snap) != sizeof(swap_tiers));
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ memcpy(swap_tiers_snap, swap_tiers, sizeof(swap_tiers));
+}
+
+void swap_tiers_snapshot_restore(void)
+{
+ struct swap_tier *tier;
+ int idx;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ memcpy(swap_tiers, swap_tiers_snap, sizeof(swap_tiers));
+
+ INIT_LIST_HEAD(&swap_tier_active_list);
+ INIT_LIST_HEAD(&swap_tier_inactive_list);
+
+ /*
+ * memcpy copied snapshot-time list pointers into each tier's
+ * list_head. Those references are stale, so re-init every
+ * tier before re-linking into the freshly initialised global
+ * lists below.
+ */
+ for_each_tier(tier, idx) {
+ INIT_LIST_HEAD(&tier->list);
+
+ if (TIER_IS_ACTIVE(tier))
+ swap_tier_activate(tier);
+ else
+ swap_tier_inactivate(tier);
+ }
+}
+
+bool swap_tiers_validate(void)
+{
+ struct swap_tier *tier;
+
+ /*
+ * Initial setting might not cover DEF_SWAP_PRIO.
+ * Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
+ */
+ if (swap_tier_is_active()) {
+ tier = list_last_entry(&swap_tier_active_list,
+ struct swap_tier, list);
+
+ if (tier->prio != DEF_SWAP_PRIO)
+ return false;
+ }
+
+ return true;
+}
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
new file mode 100644
index 000000000000..a1395ec02c24
--- /dev/null
+++ b/mm/swap_tier.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SWAP_TIER_H
+#define _SWAP_TIER_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+
+extern spinlock_t swap_tier_lock;
+
+/* Initialization and application */
+void swap_tiers_init(void);
+ssize_t swap_tiers_sysfs_show(char *buf);
+
+int swap_tiers_add(const char *name, int prio);
+int swap_tiers_remove(const char *name);
+
+void swap_tiers_snapshot(void);
+void swap_tiers_snapshot_restore(void);
+bool swap_tiers_validate(void);
+#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index e3d126602a1e..3f7225dbc6cd 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -48,6 +48,7 @@
#include "swap_table.h"
#include "internal.h"
#include "swap.h"
+#include "swap_tier.h"
static void swap_range_alloc(struct swap_info_struct *si,
unsigned int nr_entries);
@@ -63,7 +64,8 @@ static void move_cluster(struct swap_info_struct *si,
*
* Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.
*/
-static DEFINE_SPINLOCK(swap_lock);
+DEFINE_SPINLOCK(swap_lock);
+
static unsigned int nr_swapfiles;
atomic_long_t nr_swap_pages;
/*
@@ -74,7 +76,6 @@ atomic_long_t nr_swap_pages;
EXPORT_SYMBOL_GPL(nr_swap_pages);
/* protected with swap_lock. reading in vm_swap_full() doesn't need lock */
long total_swap_pages;
-#define DEF_SWAP_PRIO -1
unsigned long swapfile_maximum_size;
#ifdef CONFIG_MIGRATION
bool swap_migration_ad_supported;
@@ -87,7 +88,7 @@ static const char Bad_offset[] = "Bad swap offset entry ";
* all active swap_info_structs
* protected with swap_lock, and ordered by priority.
*/
-static PLIST_HEAD(swap_active_head);
+PLIST_HEAD(swap_active_head);
/*
* all available (active, not full) swap_info_structs
@@ -3988,6 +3989,7 @@ static int __init swapfile_init(void)
swap_migration_ad_supported = true;
#endif /* CONFIG_MIGRATION */
+ swap_tiers_init();
return 0;
}
subsys_initcall(swapfile_init);
--
2.34.1
^ permalink raw reply related
* [PATCH v7 2/4] mm: swap: associate swap devices with tiers
From: Youngjun Park @ 2026-05-27 6:22 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, gunho.lee, taejoon.song,
hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260527062247.3440692-1-youngjun.park@lge.com>
This patch connects swap devices to the swap tier infrastructure,
ensuring that devices are correctly assigned to tiers based on their
priority.
A `tier_mask` is added to identify the tier membership of swap devices.
Although tier-based allocation logic is not yet implemented, this
mapping is necessary to track which tier a device belongs to. Upon
activation, the device is assigned to a tier by matching its priority
against the configured tier ranges.
The infrastructure allows dynamic modification of tiers, such as
splitting or merging ranges. These operations are permitted provided
that the tier assignment of already configured swap devices remains
unchanged.
This patch also adds the documentation for the swap tier feature,
covering the core concepts, sysfs interface usage, and configuration
details.
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
Documentation/mm/index.rst | 1 +
Documentation/mm/swap-tier.rst | 159 +++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
include/linux/swap.h | 1 +
mm/swap_state.c | 2 +-
mm/swap_tier.c | 101 ++++++++++++++++++---
mm/swap_tier.h | 13 ++-
mm/swapfile.c | 2 +
8 files changed, 266 insertions(+), 14 deletions(-)
create mode 100644 Documentation/mm/swap-tier.rst
diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst
index 7aa2a8886908..a0d1447c5569 100644
--- a/Documentation/mm/index.rst
+++ b/Documentation/mm/index.rst
@@ -21,6 +21,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
page_reclaim
swap
swap-table
+ swap-tier
page_cache
shmfs
oom
diff --git a/Documentation/mm/swap-tier.rst b/Documentation/mm/swap-tier.rst
new file mode 100644
index 000000000000..addbc495de8c
--- /dev/null
+++ b/Documentation/mm/swap-tier.rst
@@ -0,0 +1,159 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+:Author: Chris Li <chrisl@kernel.org> Youngjun Park <youngjun.park@lge.com>
+
+==========
+Swap Tier
+==========
+
+Swap tier is a collection of user-named groups classified by priority ranges.
+It acts as a facilitation layer, allowing users to manage swap devices based
+on their speeds.
+
+Users are encouraged to assign swap device priorities according to device
+speed to fully utilize this feature. While the current implementation is
+integrated with cgroups, the concept is designed to be extensible for other
+subsystems in the future.
+
+Use case
+---------
+
+Users can perform selective swapping by choosing a swap tier assigned according
+to speed within a cgroup.
+
+For more information on cgroup v2, please refer to
+``Documentation/admin-guide/cgroup-v2.rst``.
+
+Priority Range
+--------------
+
+The specified tiers must cover the entire priority range from -1
+(DEF_SWAP_PRIO) to SHRT_MAX.
+
+Consistency
+-----------
+
+Tier consistency is guaranteed with a focus on maximizing flexibility. When a
+swap device is activated within a tier range, the tier covering that device's
+priority is guaranteed not to disappear or change while the device remains
+active. Adding a new tier may split the range of an existing tier, but the
+active device's tier assignment remains unchanged.
+
+However, specifying a tier in a cgroup does not guarantee the tier's existence.
+Consequently, the corresponding tier can disappear at any time.
+
+Configuration Interface
+-----------------------
+
+The swap tiers can be configured via the following interface:
+
+/sys/kernel/mm/swap/tiers
+
+Operations can be performed using the following syntax:
+
+* Add: ``+"<tiername>":"<start_priority>"``
+* Remove: ``-"<tiername>"``
+
+Tier names must consist of alphanumeric characters and underscores. Multiple
+operations can be provided in a single write, separated by commas (",") or
+whitespace (spaces, tabs, newlines).
+
+When configuring tiers, the specified value represents the **start priority**
+of that tier. The end priority is automatically determined by the start
+priority of the next higher tier. Consequently, adding a tier
+automatically adjusts the ranges of adjacent tiers to ensure continuity.
+
+Examples
+--------
+
+**1. Initialization**
+
+A tier starting at -1 is mandatory to cover the entire priority range up to
+SHRT_MAX. In this example, 'HDD' starts at 50, and 'NET' covers the remaining
+lower range starting from -1.
+
+::
+
+ # echo "+HDD:50, +NET:-1" > /sys/kernel/mm/swap/tiers
+ # cat /sys/kernel/mm/swap/tiers
+ Name Idx PrioStart PrioEnd
+ HDD 0 50 32767
+ NET 1 -1 49
+
+**2. Adding a New Tier (split)**
+
+A new tier 'SSD' is added at priority 100, splitting the existing 'HDD' tier.
+The ranges are automatically recalculated:
+
+* 'SSD' takes the top range (100 to SHRT_MAX).
+* 'HDD' is adjusted to the range between 'NET' and 'SSD' (50 to 99).
+* 'NET' remains unchanged (-1 to 49).
+
+::
+
+ # echo "+SSD:100" > /sys/kernel/mm/swap/tiers
+ # cat /sys/kernel/mm/swap/tiers
+ Name Idx PrioStart PrioEnd
+ SSD 2 100 32767
+ HDD 0 50 99
+ NET 1 -1 49
+
+**3. Removal (merge)**
+
+Tiers can be removed using the '-' prefix.
+::
+
+ # echo "-SSD" > /sys/kernel/mm/swap/tiers
+
+When a tier is removed, its priority range is merged into the adjacent
+tier. The merge direction is always upward (the tier below expands),
+except when the lowest tier is removed — in that case the tier above
+shifts its starting priority down to -1 to maintain full range coverage.
+
+::
+
+ Initial state:
+ Name Idx PrioStart PrioEnd
+ SSD 2 100 32767
+ HDD 1 50 99
+ NET 0 -1 49
+
+ # echo "-SSD" > /sys/kernel/mm/swap/tiers
+
+ Name Idx PrioStart PrioEnd
+ HDD 1 50 32767 <- merged with SSD's range
+ NET 0 -1 49
+
+ # echo "-NET" > /sys/kernel/mm/swap/tiers
+
+ Name Idx PrioStart PrioEnd
+ HDD 1 -1 32767 <- shifted down to -1
+
+**4. Interaction with Active Swap Devices**
+
+If a swap device is active (swapon), the tier covering that device's
+priority cannot be removed. Splitting the active tier's range is only
+allowed above the device's priority.
+
+Assume a swap device is active at priority 60 (inside 'HDD' tier).
+
+::
+
+ # swapon -p 60 /dev/zram0
+
+ Name Idx PrioStart PrioEnd
+ HDD 0 50 32767
+ NET 1 -1 49
+
+ # echo "-HDD" > /sys/kernel/mm/swap/tiers
+ -bash: echo: write error: Device or resource busy
+
+ # echo "+SSD:60" > /sys/kernel/mm/swap/tiers
+ -bash: echo: write error: Device or resource busy
+
+ # echo "+SSD:100" > /sys/kernel/mm/swap/tiers
+
+ Name Idx PrioStart PrioEnd
+ SSD 2 100 32767
+ HDD 0 50 99 <- device (prio 60) stays here
+ NET 1 -1 49
diff --git a/MAINTAINERS b/MAINTAINERS
index e3cbfedbaa5f..c008014663b3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17041,6 +17041,7 @@ R: Youngjun Park <youngjun.park@lge.com>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/mm/swap-table.rst
+F: Documentation/mm/swap-tier.rst
F: include/linux/swap.h
F: include/linux/swapfile.h
F: include/linux/swapops.h
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 6d72778e6cc3..21286945770a 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -250,6 +250,7 @@ struct swap_info_struct {
struct percpu_ref users; /* indicate and keep swap device valid. */
unsigned long flags; /* SWP_USED etc: see above */
signed short prio; /* swap priority of this type */
+ int tier_mask; /* swap tier mask */
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
unsigned int max; /* size of this swap device */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index e609bbdf7e13..de285b36e31c 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -1053,7 +1053,7 @@ static ssize_t tiers_store(struct kobject *kobj,
}
}
- if (!swap_tiers_validate()) {
+ if (!swap_tiers_update()) {
ret = -EINVAL;
goto restore;
}
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index ac7a3c2a48cb..6b57cadb3e95 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -38,6 +38,8 @@ static LIST_HEAD(swap_tier_inactive_list);
(!list_is_first(&(tier)->list, &swap_tier_active_list) ? \
list_prev_entry((tier), list)->prio - 1 : SHRT_MAX)
+#define MASK_TO_TIER(mask) (&swap_tiers[__ffs((mask))])
+
#define for_each_tier(tier, idx) \
for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
idx++, tier = &swap_tiers[idx])
@@ -59,6 +61,26 @@ static bool swap_tier_is_active(void)
return !list_empty(&swap_tier_active_list);
}
+static bool swap_tier_prio_in_range(struct swap_tier *tier, short prio)
+{
+ if (tier->prio <= prio && TIER_END_PRIO(tier) >= prio)
+ return true;
+
+ return false;
+}
+
+static bool swap_tier_prio_is_used(short prio)
+{
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (tier->prio == prio)
+ return true;
+ }
+
+ return false;
+}
+
static struct swap_tier *swap_tier_lookup(const char *name)
{
struct swap_tier *tier;
@@ -99,6 +121,7 @@ void swap_tiers_init(void)
int idx;
BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER);
+ BUILD_BUG_ON(MAX_SWAPTIER > TIER_DEFAULT_IDX);
for_each_tier(tier, idx) {
INIT_LIST_HEAD(&tier->list);
@@ -149,17 +172,29 @@ static struct swap_tier *swap_tier_prepare(const char *name, short prio)
return tier;
}
-static int swap_tier_check_range(short prio)
+static int swap_tier_can_split_range(short new_prio)
{
+ struct swap_info_struct *p;
struct swap_tier *tier;
lockdep_assert_held(&swap_lock);
lockdep_assert_held(&swap_tier_lock);
- for_each_active_tier(tier) {
- /* No overwrite */
- if (tier->prio == prio)
- return -EINVAL;
+ plist_for_each_entry(p, &swap_active_head, list) {
+ if (p->tier_mask == TIER_DEFAULT_MASK)
+ continue;
+
+ tier = MASK_TO_TIER(p->tier_mask);
+ if (!swap_tier_prio_in_range(tier, new_prio))
+ continue;
+
+ /*
+ * Device sits in a tier that spans new_prio;
+ * splitting here would reassign it to a
+ * different tier.
+ */
+ if (p->prio >= new_prio)
+ return -EBUSY;
}
return 0;
@@ -199,7 +234,11 @@ int swap_tiers_add(const char *name, int prio)
if (!swap_tier_validate_name(name))
return -EINVAL;
- ret = swap_tier_check_range(prio);
+ /* No overwrite */
+ if (swap_tier_prio_is_used(prio))
+ return -EBUSY;
+
+ ret = swap_tier_can_split_range(prio);
if (ret)
return ret;
@@ -226,6 +265,11 @@ int swap_tiers_remove(const char *name)
if (!tier)
return -EINVAL;
+ /* Simulate adding a tier to check for conflicts */
+ ret = swap_tier_can_split_range(tier->prio);
+ if (ret)
+ return ret;
+
/* Removing DEF_SWAP_PRIO merges into the higher tier. */
if (!list_is_singular(&swap_tier_active_list)
&& tier->prio == DEF_SWAP_PRIO)
@@ -236,13 +280,15 @@ int swap_tiers_remove(const char *name)
return ret;
}
-static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
/*
- * XXX: When multiple operations (adds and removes) are submitted in a
- * single write, reverting each individually on failure is complex and
- * error-prone. Instead, snapshot the entire state beforehand and
- * restore it wholesale if any operation fails.
+ * XXX: Static global snapshot buffer for batch operations. Small
+ * and used once per write, so a static global is not bad.
+ * When multiple adds/removes are submitted in a single write,
+ * reverting each individually on failure is error-prone. Instead,
+ * snapshot beforehand and restore wholesale if any operation fails.
*/
+static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
+
void swap_tiers_snapshot(void)
{
BUILD_BUG_ON(sizeof(swap_tiers_snap) != sizeof(swap_tiers));
@@ -282,10 +328,30 @@ void swap_tiers_snapshot_restore(void)
}
}
-bool swap_tiers_validate(void)
+void swap_tiers_assign_dev(struct swap_info_struct *swp)
{
struct swap_tier *tier;
+ lockdep_assert_held(&swap_lock);
+
+ for_each_active_tier(tier) {
+ if (swap_tier_prio_in_range(tier, swp->prio)) {
+ swp->tier_mask = TIER_MASK(tier);
+ return;
+ }
+ }
+
+ swp->tier_mask = TIER_DEFAULT_MASK;
+}
+
+bool swap_tiers_update(void)
+{
+ struct swap_tier *tier;
+ struct swap_info_struct *swp;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
/*
* Initial setting might not cover DEF_SWAP_PRIO.
* Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
@@ -298,5 +364,16 @@ bool swap_tiers_validate(void)
return false;
}
+ /*
+ * If applied initially, the swap tier_mask may change
+ * from the default value.
+ */
+ plist_for_each_entry(swp, &swap_active_head, list) {
+ /* Tier is already configured */
+ if (swp->tier_mask != TIER_DEFAULT_MASK)
+ break;
+ swap_tiers_assign_dev(swp);
+ }
+
return true;
}
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
index a1395ec02c24..3e355f857363 100644
--- a/mm/swap_tier.h
+++ b/mm/swap_tier.h
@@ -5,8 +5,15 @@
#include <linux/types.h>
#include <linux/spinlock.h>
+/* Forward declarations */
+struct swap_info_struct;
+
extern spinlock_t swap_tier_lock;
+#define TIER_ALL_MASK (~0)
+#define TIER_DEFAULT_IDX (31)
+#define TIER_DEFAULT_MASK (1U << TIER_DEFAULT_IDX)
+
/* Initialization and application */
void swap_tiers_init(void);
ssize_t swap_tiers_sysfs_show(char *buf);
@@ -16,5 +23,9 @@ int swap_tiers_remove(const char *name);
void swap_tiers_snapshot(void);
void swap_tiers_snapshot_restore(void);
-bool swap_tiers_validate(void);
+bool swap_tiers_update(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+
#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 3f7225dbc6cd..9a86ebe992f4 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -3036,6 +3036,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
/* Add back to available list */
add_to_avail_list(si, true);
+
+ swap_tiers_assign_dev(si);
}
/*
--
2.34.1
^ permalink raw reply related
* [PATCH v7 0/4] mm: swap: introduce swap tier infrastructure
From: Youngjun Park @ 2026-05-27 6:22 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, gunho.lee, taejoon.song,
hyungjun.cho, mkoutny, baver.bae, matia.kim
This is v7 of the swap tier series addressing review feedback.
The cover letter has been simplified.
I revisited the design (see Design Rationale). Since our use case
fits best with a memcg-based model, the implementation remains
within memcg and preserves its resource accounting semantics.
Alternatives considered:
1. A separate sysfs interface under swap. (Workable. But, it would still
need to reference memcg paths, and fully decoupling it would add
swap-layer logic to manage memcgs, making it secondary option.)
2. Making the feature non-default.
Other interfaces were also reviewed. Aside from sysfs and BPF,
the options involve trade-offs and are largely design choices.
BPF was excluded due to possible disablement on our embedded
platform, though future extension remains possible.
Overview
========
Swap Tiers group swap devices into performance classes (e.g. NVMe,
HDD, Network) and allow per-memcg selection of which tiers to use.
This mechanism was suggested by Chris Li.
Design Rationale
================
Swap tier selection is attached to memcg. A child cgroup may select a
subset of the parent's allowed tiers.
This
- Preserves cgroup inheritance semantics (boundary at parent,
refinement at child).
- Reuses memcg, which already groups processes and enforces
hierarchical memory limits.
- Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
- Avoids introducing a parallel swap control hierarchy.
Placing tier control outside memcg (e.g., via BPF, syscalls, or
madvise) would allow swap preference to diverge from the memcg
hierarchy. Integrating it into memcg keeps the swap policy
consistent with existing memory ownership semantics. There are
also real use cases built around memcg.
In the future, this can be extended to other interfaces to cover
additional use cases.
I believe a memcg-based swap control is a good starting point
before such extensions.
Use Cases
=========
#1: Latency separation (our primary deployment scenario)
[ / ]
|
+-- latency-sensitive workload (fast tier)
+-- background workload (slow tier)
The parent defines the memory boundary.
Each workload selects a swap tier via memory.swap.tiers according to
latency requirements.
This prevents latency-sensitive workloads from being swapped to
slow devices used by background workloads.
#2: Per-VM swap selection (Chris Li's deployment scenario)
[ / ]
|
+-- [ Job on VM ] (tiers: zswap, SSD)
|
+-- [ VMM guest memory ] (tiers: SSD)
The parent (job) has access to both zswap and SSD tiers.
The child (VMM guest memory) selects SSD as its swap tier via
memory.swap.tiers. In this deployment, swap device selection
happens at the child level from the parent's available set.
#3: Tier isolation for reduced contention (hypothetical)
[ / ] (tiers: A, B)
|
+-- workload X (tiers: A)
+-- workload Y (tiers: B)
Each child uses a different tier. Since swap paths are separated
per tier, synchronization overhead between the two workloads is
reduced.
Future extension
================
#1: Intra-tier distribution policy:
Currently, swap devices with the same priority are allocated in a
round-robin fashion. Per-tier policy files under
/sys/kernel/mm/swap/tiers/ can control how devices within a tier
are selected (e.g. round-robin, weighted).
#2: Inter-tier promotion and demotion:
Promotion and demotion apply between tiers, not within a single
tier. The current interface defines only tier assignment; it does
not yet define when or how pages move between tiers. Two triggering
models are possible:
(a) User-triggered: userspace explicitly initiates migration between
tiers (e.g. via a new interface or existing move_pages semantics).
(b) Kernel-triggered: the kernel moves pages between tiers at
appropriate points such as reclaim or refault.
#3: Per-VMA, per-process swap and BPF:
Not just for memcg based swap, possible to extend Per-VMA or per-process swap.
Or we can use it as BPF program.
Experimentation
===============
Tested on our internal platform using NBD as a separate swap tier.
Our first production's simple usecase.
Without tiers:
- No selective control over flash wear
- Cannot selectively assign NBD to specific applications
Cold launch improvement (preloaded vs. baseline):
- App A: 13.17s -> 4.18s (68%)
- App B: 5.60s -> 1.12s (80%)
- App C: 10.25s -> 2.00s (80%)
Performance impact with no tiers configured:
<1% regression in kernel build and vm-scalability benchmarks
Change log
===========
v7
- Collect Baoquan's review tag
- Baoquan's feedback on fixing improper comment
- Minor code adjustments per Baoquan's feedback.
- Rebase on recent mm-new
- v6 link: https://lore.kernel.org/linux-mm/20260421055323.940344-1-youngjun.park@lge.com/
v6
- Sashiko AI review fixes
- Fix batch parsing error path to restore snapshot before exit
- Reject overlong tier names to prevent truncated duplicates
- Avoid restoring raw list_head via memcpy (stale pointer risk)
- Ensure early parse errors do not skip DEF_SWAP_PRIO validation
- Use (1U << TIER_DEFAULT_IDX) to avoid signed shift UB
- Defer tier mask inheritance to css_online() to close race window
- Add READ_ONCE()/WRITE_ONCE() for tier mask accesses
- Other fixes
- Fix build error reintroduced due to missing v5 change (sorry for that..)
- Fix WARNING in folio_tier_effective_mask by adding rcu_read_lock() (syzbot CI fix)
- default number of swap tier max (change to 32->31, for reserving last bit)
- commit message refinement.
- rebased on recently mm-new
- v5 link: https://lore.kernel.org/linux-mm/20260325175453.2523280-1-youngjun.park@lge.com/
v5
- Fixed build errors reported in v4
- rebased on up to date mm-new
- Minor cleanups
- Design docs with validation (by Shakeel Butt discussion)
- v4 link : https://lore.kernel.org/linux-mm/20260217000950.4015880-1-youngjun.park@lge.com/
v4
- Simplified control flow and indentation
- Added CONFIG option for MAX_SWAPTIER (default: 4)
- Added memory.swap.tiers.effective interface
- Reworked save/restore logic into snapshot/rollback model
- Removed tier priority modification support (deferred)
- Improved validation and fixed edge cases
- Rebased onto latest mm-new
- RFC v3 link: https://lore.kernel.org/linux-mm/20260131125454.3187546-1-youngjun.park@lge.com/
RFC v1 ~ v3
- Change the direction after discussion with Chris-Li
- apply some LPC feedback.
- RFC v2 - https://lore.kernel.org/linux-mm/20260126065242.1221862-1-youngjun.park@lge.com/
- RFC v1 - https://lore.kernel.org/linux-mm/20251109124947.1101520-1-youngjun.park@lge.com/
Earlier Approach (per cgroup swap priority)
- v1: https://lore.kernel.org/linux-mm/20250716202006.3640584-1-youngjun.park@lge.com/
- RFC: https://lore.kernel.org/linux-mm/aEvLjEInMQC7hEyh@yjaykim-PowerEdge-T330/T/#mbbb6a5e9e30843097e1f5f65fb98f31d582b973d
Youngjun Park (4):
mm: swap: introduce swap tier infrastructure
mm: swap: associate swap devices with tiers
mm: memcontrol: add interfaces for swap tier selection
mm: swap: filter swap allocation by memcg tier mask
Documentation/admin-guide/cgroup-v2.rst | 29 ++
Documentation/mm/index.rst | 1 +
Documentation/mm/swap-tier.rst | 159 ++++++++
MAINTAINERS | 3 +
include/linux/memcontrol.h | 5 +
include/linux/swap.h | 1 +
mm/Kconfig | 12 +
mm/Makefile | 2 +-
mm/memcontrol.c | 96 +++++
mm/swap.h | 4 +
mm/swap_state.c | 75 ++++
mm/swap_tier.c | 482 ++++++++++++++++++++++++
mm/swap_tier.h | 75 ++++
mm/swapfile.c | 20 +-
14 files changed, 959 insertions(+), 5 deletions(-)
create mode 100644 Documentation/mm/swap-tier.rst
create mode 100644 mm/swap_tier.c
create mode 100644 mm/swap_tier.h
base-commit: 938bf00744a1b82cefd551f848a927cc24d5fb2f
--
2.34.1
^ permalink raw reply
* Re: [PATCH v2 0/4] mm/zswap: Implement per-cgroup proactive writeback
From: Hao Jia @ 2026-05-27 2:39 UTC (permalink / raw)
To: Yosry Ahmed
Cc: Andrew Morton, tj, hannes, shakeel.butt, mhocko, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <CAO9r8zO1+brQroYufMZ2K=ZH_PBBpzYPsdYm-DT3K2GxoKJs9A@mail.gmail.com>
On 2026/5/27 02:55, Yosry Ahmed wrote:
> On Tue, May 26, 2026 at 4:56 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>>
>>
>>
>> On 2026/5/26 03:24, Andrew Morton wrote:
>>> On Mon, 25 May 2026 20:22:38 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
>>>
>>>> Zswap currently writes back pages to backing swap reactively, triggered
>>>> either by the shrinker or by the pool reaching its size limit. Although
>>>> proactive memory reclaim can automatically write back a portion of zswap
>>>> pages via the shrinker, it cannot explicitly control the amount of
>>>> writeback for a specific memory cgroup. Moreover, proactive memory reclaim
>>>> may not always be triggered during a steady state.
>>>>
>>>> In certain scenarios, it is desirable to trigger writeback in advance to
>>>> free up memory. For example, users may want to prepare for an upcoming
>>>> memory-intensive workload by flushing cold memory to the backing storage
>>>> when the system is relatively idle.
>>>>
>>>> This patch series introduces a "zswap_writeback_only" key to memory.reclaim
>>>> cgroup interface, allowing users to proactively write back cold compressed
>>>> pages from zswap to the backing swap device. When specified, this key
>>>> bypasses standard memory reclaim and exclusively performs proactive zswap
>>>> writeback up to the requested budget. If omitted, the default reclaim
>>>> behavior remains unchanged.
>>>
>>> Thanks. AI review found a few things to complain about, one of them
>>> described as "preexisting".
>>>
>>
>> Thanks Andrew. I have replied to the AI's review comments in a separate
>> email and posted v3.
>> https://lore.kernel.org/all/20260526114601.67041-1-jiahao.kernel@gmail.com
>
> Generally speaking, please give time for reviewers to take a look
> before sending a new version. Less than a day is usually too fast
> (unless you're iterating super fast with the reviewers). Review
> feedback does not have to be addressed immediately, usually wait for a
> bit to collect as much feedback as possible before spinning a new
> version.
>
Thanks for the advice, Yosry. Got it.
> I will take a look at v3 soon, thank you.
Appreciate you taking a look at v3.
Thakns,
Hao
^ permalink raw reply
* Re: [PATCH v6 4/4] mm: swap: filter swap allocation by memcg tier mask
From: YoungJun Park @ 2026-05-27 2:17 UTC (permalink / raw)
To: Baoquan He
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <ahZMHmMbhnNPspQj@MiWiFi-R3L-srv>
On Wed, May 27, 2026 at 09:42:54AM +0800, Baoquan He wrote:
> On 04/21/26 at 02:53pm, Youngjun Park wrote:
> > Apply memcg tier effective mask during swap slot allocation to
> > enforce per-cgroup swap tier restrictions.
> >
> > In the fast path, check the percpu cached swap_info's tier_mask
> > against the folio's effective mask. If it does not match, fall
> > through to the slow path. In the slow path, skip swap devices
> > whose tier_mask is not covered by the folio's effective mask.
> >
> > This works correctly when there is only one non-rotational
> > device in the system and no devices share the same priority.
> > However, there are known limitations:
> >
> > - When non-rotational devices are distributed across multiple
> > tiers, and different memcgs are configured to use those
> > distinct tiers, they may constantly overwrite the shared
> > percpu swap cache. This cache thrashing leads to frequent
> > fast path misses.
> >
> > - Combined with the above issue, if same-priority devices exist
> > among them, a percpu cache miss (overwritten by another memcg)
> > forces the allocator to round-robin to the next device
> > prematurely, even if the current cluster is not fully
> > exhausted.
> >
> > These edge cases do not affect the primary use case of
> > directing swap traffic per cgroup. Further optimization is
> > planned for future work.
> >
> > Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> > ---
> > mm/swapfile.c | 13 ++++++++++++-
> > 1 file changed, 12 insertions(+), 1 deletion(-)
> >
> > diff --git a/mm/swapfile.c b/mm/swapfile.c
> > index d5abc831cde7..8734e5d26b08 100644
> > --- a/mm/swapfile.c
> > +++ b/mm/swapfile.c
> > @@ -1352,15 +1352,22 @@ static bool swap_alloc_fast(struct folio *folio)
> > struct swap_cluster_info *ci;
> > struct swap_info_struct *si;
> > unsigned int offset;
> > + int mask = folio_tier_effective_mask(folio);
> >
> > /*
> > * Once allocated, swap_info_struct will never be completely freed,
> > * so checking it's liveness by get_swap_device_info is enough.
> > */
> > si = this_cpu_read(percpu_swap_cluster.si[order]);
> > + if (!si || !swap_tiers_mask_test(si->tier_mask, mask) ||
> > + !get_swap_device_info(si))
> > + return false;
> > +
> > offset = this_cpu_read(percpu_swap_cluster.offset[order]);
> > - if (!si || !offset || !get_swap_device_info(si))
> > + if (!offset) {
> > + put_swap_device(si);
> > return false;
> > + }
>
> The whole patch looks good to me except of one nitpick. Is it a lille
> cleaner with below tiny adjustment?
>
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index 2864cd8c2da9..cdf453bf6b80 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -1359,15 +1359,12 @@ static bool swap_alloc_fast(struct folio *folio)
> * so checking it's liveness by get_swap_device_info is enough.
> */
> si = this_cpu_read(percpu_swap_cluster.si[order]);
> - if (!si || !swap_tiers_mask_test(si->tier_mask, mask) ||
> - !get_swap_device_info(si))
> + if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
> return false;
>
> offset = this_cpu_read(percpu_swap_cluster.offset[order]);
> - if (!offset) {
> - put_swap_device(si);
> + if (!offset || !get_swap_device_info(si))
> return false;
> - }
Thanks!
Your suggested version of code is simpler than privious one.
I will apply it.
^ permalink raw reply
* Re: [PATCH v6 3/4] mm: memcontrol: add interfaces for swap tier selection
From: YoungJun Park @ 2026-05-27 2:08 UTC (permalink / raw)
To: Baoquan He
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <ahYzNVQHnWlwYq8u@MiWiFi-R3L-srv>
On Wed, May 27, 2026 at 07:56:37AM +0800, Baoquan He wrote:
> On 04/21/26 at 02:53pm, Youngjun Park wrote:
> ...snip...
> > diff --git a/mm/swap_tier.c b/mm/swap_tier.c
> > index 9019b8978770..b53d6cc67d1e 100644
> > --- a/mm/swap_tier.c
> > +++ b/mm/swap_tier.c
> ...snip...
> > @@ -374,6 +404,80 @@ bool swap_tiers_update(void)
> > break;
> > swap_tiers_assign_dev(swp);
> > }
> > + /*
> > + * XXX: Unused tiers default to ON, disabled after next tier added.
> > + * Use removed tier mask to clear settings for removed/re-added tiers.
> > + * (Could hold tier refs, but better to keep cgroup config independent)
> > + */
> > + if (mask)
> > + swap_tier_memcg_propagate(mask);
>
> Code is neat, and high efficiency, while it's not easy to understand. I
> just sat in front of my computer the whole day yesterday to recall and
> understand why it's done like this, even though I made it clear to my
> self in the past. I think more words would be helpful to decrease the
> difficulty for people to undersntand it. E.g below two paragraphes. Just
> a suggestion and for your reference.
Thanks for the review again :)
Okay I see.
This explanation seems heavily abbreviated.
> diff --git a/mm/swap_tier.c b/mm/swap_tier.c
> index b53d6cc67d1e..0a6adf14ab91 100644
> --- a/mm/swap_tier.c
> +++ b/mm/swap_tier.c
> @@ -405,9 +405,17 @@ bool swap_tiers_update(int mask)
> swap_tiers_assign_dev(swp);
> }
> /*
> - * XXX: Unused tiers default to ON, disabled after next tier added.
> - * Use removed tier mask to clear settings for removed/re-added tiers.
> - * (Could hold tier refs, but better to keep cgroup config independent)
> + * When a tier is removed, its index (bit position in the mask) becomes
> + * free for reassignment to a future tier. If a memcg had previously
> + * disabled this tier (cleared the bit in its swap.tiers file), the
> + * effective mask would keep that bit clear -- meaning the new tier at
> + * the same index would be silently unavailable, an invisible cgroup
> + * constraint left behind by a tier that no longer exists.
> + *
> + * To prevent this, OR the removed tier's mask bit into every memcg's
> + * tier_mask and tier_effective_mask. This resets the bit so the new
> + * tier is accessible by default; users who want to restrict it must
> + * explicitly disable it after the tier is re-created.
> */
The explanation seems good.
it explains why we must clear the "disabled bit",
when a new tier comes and we don't automatically diable,
it can implicitly disable "new tier" which user might not want.
This comes from that fact we don't have a cgroup refecounting on the tier land.
it is the corner case, we must sync for correct behavior.
I will change the comments as like it is.
^ permalink raw reply
* Re: [PATCH v6 3/4] mm: memcontrol: add interfaces for swap tier selection
From: YoungJun Park @ 2026-05-27 1:58 UTC (permalink / raw)
To: Baoquan He
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <ahW9UQH93jBT_VaD@MiWiFi-R3L-srv>
On Tue, May 26, 2026 at 11:33:37PM +0800, Baoquan He wrote:
> On 04/21/26 at 02:53pm, Youngjun Park wrote:
> ...snip...
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index c3d98ab41f1f..0f67572e5e3e 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -68,6 +68,7 @@
> > #include <net/ip.h>
> > #include "slab.h"
> > #include "memcontrol-v1.h"
> > +#include "swap_tier.h"
> >
> > #include <linux/uaccess.h>
> >
> > @@ -4130,6 +4131,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
> > refcount_set(&memcg->id.ref, 1);
> > css_get(css);
> >
> > + swap_tiers_memcg_inherit_mask(memcg);
> > +
> > /*
> > * Ensure mem_cgroup_from_private_id() works once we're fully online.
> > *
> > @@ -5667,6 +5670,88 @@ static int swap_events_show(struct seq_file *m, void *v)
> > return 0;
> > }
> >
> > +static int swap_tier_show(struct seq_file *m, void *v)
> > +{
> > + struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
> > +
> > + swap_tiers_mask_show(m, READ_ONCE(memcg->tier_mask));
> > + return 0;
> > +}
> > +
> > +static ssize_t swap_tier_write(struct kernfs_open_file *of,
> > + char *buf, size_t nbytes, loff_t off)
> > +{
> > + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
> > + char *pos, *token;
> > + int ret = 0;
> > + int original_mask = 0;
> > +
> > + pos = strstrip(buf);
> > +
> > + spin_lock(&swap_tier_lock);
> > + if (!*pos) {
> > + WRITE_ONCE(memcg->tier_mask, TIER_ALL_MASK);
> > + goto sync;
> > + }
> > +
> > + original_mask = memcg->tier_mask;
> > +
> > + while ((token = strsep(&pos, " \t\n")) != NULL) {
> > + int mask;
> > +
> > + if (!*token)
> > + continue;
> > +
> > + if (token[0] != '-' && token[0] != '+') {
> > + ret = -EINVAL;
> > + goto err;
> > + }
> > +
> > + mask = swap_tiers_mask_lookup(token+1);
> > + if (!mask) {
> > + ret = -EINVAL;
> > + goto err;
> > + }
> > +
> > + /*
> > + * if child already set, cannot add that tiers for hierarch mismatching.
> > + * parent compatible, child must respect parent selected swap device.
> > + */
>
> This paragraph of code comment sounds a little unnatural. We are writing
> it into memcg, the child memcg is handled in
> swap_tiers_memcg_sync_mask(), isn't it? I don't get the 2nd sentence.
> Could you help explain?
Thanks for the review.
I think this sentence is not appropriate.
Maybe I forgot to remove this line after modifying the code many times.
This sentence is not correct.
I explain one by one for clarification.
"if child already set, cannot add that tiers for hierarch mismatching."
mask can be changed (can change the tier_mask).
But the effective one (effective_mask) is recomputed with parent's effective_mask.
"parent compatible, child must respect parent selected swap device"
effective_mask must be restricted, which means child effective_mask
must be subset of parent's effective_mask.
^ permalink raw reply
* Re: [PATCH v6 4/4] mm: swap: filter swap allocation by memcg tier mask
From: Baoquan He @ 2026-05-27 1:42 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-5-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
> Apply memcg tier effective mask during swap slot allocation to
> enforce per-cgroup swap tier restrictions.
>
> In the fast path, check the percpu cached swap_info's tier_mask
> against the folio's effective mask. If it does not match, fall
> through to the slow path. In the slow path, skip swap devices
> whose tier_mask is not covered by the folio's effective mask.
>
> This works correctly when there is only one non-rotational
> device in the system and no devices share the same priority.
> However, there are known limitations:
>
> - When non-rotational devices are distributed across multiple
> tiers, and different memcgs are configured to use those
> distinct tiers, they may constantly overwrite the shared
> percpu swap cache. This cache thrashing leads to frequent
> fast path misses.
>
> - Combined with the above issue, if same-priority devices exist
> among them, a percpu cache miss (overwritten by another memcg)
> forces the allocator to round-robin to the next device
> prematurely, even if the current cluster is not fully
> exhausted.
>
> These edge cases do not affect the primary use case of
> directing swap traffic per cgroup. Further optimization is
> planned for future work.
>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> ---
> mm/swapfile.c | 13 ++++++++++++-
> 1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index d5abc831cde7..8734e5d26b08 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -1352,15 +1352,22 @@ static bool swap_alloc_fast(struct folio *folio)
> struct swap_cluster_info *ci;
> struct swap_info_struct *si;
> unsigned int offset;
> + int mask = folio_tier_effective_mask(folio);
>
> /*
> * Once allocated, swap_info_struct will never be completely freed,
> * so checking it's liveness by get_swap_device_info is enough.
> */
> si = this_cpu_read(percpu_swap_cluster.si[order]);
> + if (!si || !swap_tiers_mask_test(si->tier_mask, mask) ||
> + !get_swap_device_info(si))
> + return false;
> +
> offset = this_cpu_read(percpu_swap_cluster.offset[order]);
> - if (!si || !offset || !get_swap_device_info(si))
> + if (!offset) {
> + put_swap_device(si);
> return false;
> + }
The whole patch looks good to me except of one nitpick. Is it a lille
cleaner with below tiny adjustment?
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 2864cd8c2da9..cdf453bf6b80 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1359,15 +1359,12 @@ static bool swap_alloc_fast(struct folio *folio)
* so checking it's liveness by get_swap_device_info is enough.
*/
si = this_cpu_read(percpu_swap_cluster.si[order]);
- if (!si || !swap_tiers_mask_test(si->tier_mask, mask) ||
- !get_swap_device_info(si))
+ if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
return false;
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
- if (!offset) {
- put_swap_device(si);
+ if (!offset || !get_swap_device_info(si))
return false;
- }
ci = swap_cluster_lock(si, offset);
if (cluster_is_usable(ci, order)) {
>
> ci = swap_cluster_lock(si, offset);
> if (cluster_is_usable(ci, order)) {
> @@ -1379,10 +1386,14 @@ static bool swap_alloc_fast(struct folio *folio)
> static void swap_alloc_slow(struct folio *folio)
> {
> struct swap_info_struct *si, *next;
> + int mask = folio_tier_effective_mask(folio);
>
> spin_lock(&swap_avail_lock);
> start_over:
> plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
> + if (!swap_tiers_mask_test(si->tier_mask, mask))
> + continue;
> +
> /* Rotate the device and switch to a new cluster */
> plist_requeue(&si->avail_list, &swap_avail_head);
> spin_unlock(&swap_avail_lock);
> --
> 2.34.1
>
^ permalink raw reply related
* Re: [PATCH v6 3/4] mm: memcontrol: add interfaces for swap tier selection
From: Baoquan He @ 2026-05-26 23:56 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-4-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
...snip...
> diff --git a/mm/swap_tier.c b/mm/swap_tier.c
> index 9019b8978770..b53d6cc67d1e 100644
> --- a/mm/swap_tier.c
> +++ b/mm/swap_tier.c
...snip...
> @@ -374,6 +404,80 @@ bool swap_tiers_update(void)
> break;
> swap_tiers_assign_dev(swp);
> }
> + /*
> + * XXX: Unused tiers default to ON, disabled after next tier added.
> + * Use removed tier mask to clear settings for removed/re-added tiers.
> + * (Could hold tier refs, but better to keep cgroup config independent)
> + */
> + if (mask)
> + swap_tier_memcg_propagate(mask);
Code is neat, and high efficiency, while it's not easy to understand. I
just sat in front of my computer the whole day yesterday to recall and
understand why it's done like this, even though I made it clear to my
self in the past. I think more words would be helpful to decrease the
difficulty for people to undersntand it. E.g below two paragraphes. Just
a suggestion and for your reference.
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index b53d6cc67d1e..0a6adf14ab91 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -405,9 +405,17 @@ bool swap_tiers_update(int mask)
swap_tiers_assign_dev(swp);
}
/*
- * XXX: Unused tiers default to ON, disabled after next tier added.
- * Use removed tier mask to clear settings for removed/re-added tiers.
- * (Could hold tier refs, but better to keep cgroup config independent)
+ * When a tier is removed, its index (bit position in the mask) becomes
+ * free for reassignment to a future tier. If a memcg had previously
+ * disabled this tier (cleared the bit in its swap.tiers file), the
+ * effective mask would keep that bit clear -- meaning the new tier at
+ * the same index would be silently unavailable, an invisible cgroup
+ * constraint left behind by a tier that no longer exists.
+ *
+ * To prevent this, OR the removed tier's mask bit into every memcg's
+ * tier_mask and tier_effective_mask. This resets the bit so the new
+ * tier is accessible by default; users who want to restrict it must
+ * explicitly disable it after the tier is re-created.
*/
if (mask)
swap_tier_memcg_propagate(mask);
>
> return true;
> }
> +
...snip...
^ permalink raw reply related
* Re: [PATCH cgroup/for-next v2 0/5] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-05-26 20:12 UTC (permalink / raw)
To: Ridong Chen, Chen Ridong, Tejun Heo, Johannes Weiner,
Michal Koutný, Ingo Molnar, Peter Zijlstra, Juri Lelli,
Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall,
Mel Gorman, Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin
In-Reply-To: <3936b856-cf88-41a9-bb3f-4f48440e2692@linux.dev>
On 5/20/26 4:29 AM, Ridong Chen wrote:
>
>
> On 2026/5/16 12:24, Waiman Long wrote:
>> Sashiko AI review of another cpuset patch had found that cpuset_attach()
>> and cpuset_can_attach() can be passed a cgroup_taskset with tasks
>> migrating from one source cpuset to multiple destination cpusets and
>> vice versa. Further testing of the cpuset code indicates that this is
>> indeed the case when the v2 cpuset controller is enabled or disabled.
>>
>> Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that
>> there will be one source and one destinaton cpuset which may result in
>> inocrrect behavior.
>>
>
> Hi Longman,
>
> I am thinking whether we can use the pids subsystem's approach to
> solve this issue, which I think could be much simpler.
>
> For the DL task accounting, we can handle it the same way
> pids_can_attach() does - just call task_cs(task) for each task
> individually inside the can_attach() loop and do the nr_deadline_tasks
> adjustment right there. This eliminates the need to pass per-task
> source cpuset information to the attach() callback entirely for DL
> accounting purposes.
DL task accounting doesn't use the new oldcs stored in the task
structure which is only used for mm migration. BTW, I believe
task_cs(task) doesn't return the old cs in cpuset_attach().
>
> For cpuset_migrate_mm(), I don't think we need per-task oldcs storage
> in task_struct either. The scenarios where multiple source cpusets are
> involved are:
>
> enable cpuset controller: child cpusets inherit parent's
> effective_mems, so attach_mems_updated is false and
> cpuset_migrate_mm() is never called.
>
> disable cpuset controller: tasks move from children to parent. Since
> children's effective_mems is always a subset of parent's
> effective_mems, even if cpuset_migrate_mm() is triggered, it's
> effectively a noop (no pages need to move from a subset to its superset).
>
> cgroup.procs write with threads in different cpusets: this is a
> many-to-one migration with a single process, so there is only one
> group_leader and one mm. We only need to record the leader's oldcs,
> which a single static variable can handle.
>
> So in all cases, the migration path only needs one oldcs for the
> leader. We don't need to add a field to task_struct.
>
> What do you think?
Yes, that makes sense. I will rework the patch series.
Thanks,
Longman
>
>
>
>> This patch series is created to fix this issue. The first 2 patches are
>> just preparatory patches to make the remaining patches easier to review.
>>
>> Patch 3 adds a new attach_old_cs field into task_struct to track the
>> old cpuset to be used in case when cpuset_migrate_mm() needs to be
>> called in cpuset_attach().
>>
>> Patch 4 moves mpol_rebind_mm() and cpuset_migrate_mm() inside
>> cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works
>> more like moving task from one cpuset to another one, while also make
>> supporting multiple source and destination cpusets easier.
>>
>> Patch 5 makes the necessary changes to enable the support of multiple
>> source and destination cpusets by keeping all the source and destination
>> cpusets found during task iterations in two singly linked lists for
>> source and destination cpusets respectively.
>>
>> Waiman Long (5):
>> cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
>> cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
>> cgroup/cpuset: Replace cpuset_attach_old_cs by a new attach_old_cs
>> field in task_struct
>> cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
>> cpuset_attach_task()
>> cgroup/cpuset: Support multiple source/destination cpusets for
>> cpuset_*attach()
>>
>> include/linux/sched.h | 3 +
>> kernel/cgroup/cpuset-internal.h | 6 +
>> kernel/cgroup/cpuset.c | 358 +++++++++++++++++++++-----------
>> 3 files changed, 249 insertions(+), 118 deletions(-)
>>
>
^ permalink raw reply
* Re: [PATCH v4 8/8] mm: switch deferred split shrinker to list_lru
From: Johannes Weiner @ 2026-05-26 20:10 UTC (permalink / raw)
To: Usama Arif
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Liam R . Howlett, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260526120923.2331056-1-usama.arif@linux.dev>
On Tue, May 26, 2026 at 05:09:22AM -0700, Usama Arif wrote:
> On Thu, 21 May 2026 11:02:14 -0400 Johannes Weiner <hannes@cmpxchg.org> wrote:
> > @@ -4446,36 +4377,20 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
> > count_vm_event(THP_DEFERRED_SPLIT_PAGE);
> > count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);
> > mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1);
> > -
> > }
> > } else {
> > /* partially mapped folios cannot become non-partially mapped */
> > VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio);
> > }
> > - if (list_empty(&folio->_deferred_list)) {
> > - struct mem_cgroup *memcg;
> > -
> > - memcg = folio_split_queue_memcg(folio, ds_queue);
> > - list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
> > - ds_queue->split_queue_len++;
> > - if (memcg)
> > - set_shrinker_bit(memcg, folio_nid(folio),
> > - shrinker_id(deferred_split_shrinker));
> > - }
> > - split_queue_unlock_irqrestore(ds_queue, flags);
> > + __list_lru_add(&deferred_split_lru, lru, &folio->_deferred_list, nid, memcg);
> > + list_lru_unlock_irqrestore(lru, &flags);
> > + rcu_read_unlock();
>
> Can the shrinker bit end up on the wrong memcg here?
>
> deferred_split_folio() takes the lock via list_lru_lock_irqsave() with
> the original folio_memcg() of the folio. If that memcg is dying and
> already reparented, lock_list_lru_of_memcg() walks up parent_mem_cgroup()
> until it finds a live sublist and locks it (lru), but the memcg local
> variable still points at the dying child.
>
> __list_lru_add() then calls set_shrinker_bit(memcg, ...) with the
> original (dying / reparented) memcg, not the parent that actually owns
> the locked sublist where the folio was queued.
I think you're right. Good catch.
This looks like an existing list_lru issue, actually. Even before,
list_lru_add() would call lock_list_lru_of_memcg() which might do the
hierarchy walk. But then set_shrinker_bit() on the passed in memcg.
I'll fix this in list_lru first. It's a likely backport
candidate. Then rebase my patches on top.
> > @@ -1301,6 +1301,9 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
> > if (result != SCAN_SUCCEED)
> > goto out_nolock;
> >
> > + if (folio_memcg_alloc_deferred(folio))
> > + goto out_nolock;
> > +
>
> Over here, you will end up reporting success on allocation failure.
>
> Maybe set result to SCAN_ALLOC_HUGE_PAGE_FAIL?
Good point. Sashiko pointed that out as well. I have a follow-up fix.
Thanks for taking a look.
^ permalink raw reply
* Re: [PATCH v3] security: Expand task_setscheduler LSM hook to include CPU affinity mask
From: Aaron Tomlin @ 2026-05-26 19:53 UTC (permalink / raw)
To: tsbogend, paul, jmorris, serge, mingo, peterz, juri.lelli,
vincent.guittot, stephen.smalley.work, casey, longman, tj, hannes,
mkoutny
Cc: chenridong, dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, omosnace, kees, neelx, sean, chjohnst, steve,
mproche, nick.lange, cgroups, bpf, linux-mips, linux-fsdevel,
linux-security-module, selinux, linux-kernel
In-Reply-To: <20260526142838.774711-1-atomlin@atomlin.com>
On Tue, May 26, 2026 at 10:28:38AM -0400, Aaron Tomlin wrote:
> At present, the task_setscheduler LSM hook provides security modules
> with the opportunity to mediate changes to a task's scheduling policy.
> However, when invoked via sched_setaffinity(), the hook lacks
> visibility into the actual CPU affinity mask being requested.
> Consequently, BPF-based security modules are entirely blind to the
> target CPUs and cannot make granular access control decisions based on
> spatial isolation.
>
> In modern multi-tenant and real-time environments, CPU isolation is a
> critical boundary. The inability to audit or restrict specific CPU
> pinning requests limits the effectiveness of eBPF-driven security
> policies, particularly when attempting to shield isolated or
> cryptographic cores from unprivileged or compromised tasks.
>
> This patch expands the security_task_setscheduler() hook signature to
> include a pointer to the requested cpumask. Because this is a shared
> hook used for multiple scheduling attribute changes, call sites that do
> not modify CPU affinity are updated to safely pass NULL.
> To protect against unverified dereferences, the parameter is annotated
> with __nullable in the LSM hook definition, ensuring the BPF verifier
> mandates explicit NULL checks for attached eBPF programs.
>
> This change updates all in-tree security modules (SELinux and Smack) to
> accommodate the new parameter mechanically, whilst providing BPF LSMs
> with the necessary context to enforce strict affinity policies.
Adding BPF Core to review the use of annotation "__nullable" in the LSM
hook definition.
Kind regards,
--
Aaron Tomlin
^ permalink raw reply
* Re: [PATCH] cgroup/cpuset: Use effective_xcpus in partcmd_update add/del mask calculation
From: Waiman Long @ 2026-05-26 19:09 UTC (permalink / raw)
To: Guopeng Zhang, Sun Shaojie, Chen Ridong, Tejun Heo,
Johannes Weiner, Michal Koutný
Cc: cgroups, linux-kernel
In-Reply-To: <a1a89205-4e4b-4bb9-86fe-e106997ab1d5@kylinos.cn>
On 5/22/26 4:41 AM, Guopeng Zhang wrote:
>
> 在 2026/5/22 15:53, Sun Shaojie 写道:
>> When sibling CPU exclusion occurs, a partition's user_xcpus may contain
>> CPUs that were never actually granted to it. These CPUs are present in
>> user_xcpus(cs) but not in cs->effective_xcpus.
>>
>> The partcmd_update path in update_parent_effective_cpumask() uses
>> user_xcpus(cs) (via the local variable xcpus) to compute the addmask
>> (CPUs to return to parent) and delmask (CPUs to request from parent).
>> This is incorrect:
>>
>> 1) When newmask removes a CPU that was previously excluded by a
>> sibling, addmask incorrectly includes that CPU and tries to return
>> it to the parent even though the partition never actually owned it,
>> causing CPU overlap with sibling partitions and triggering warnings
>> in generate_sched_domains().
>>
>> 2) When newmask adds a previously excluded CPU that is now available,
>> delmask fails to request it from the parent because user_xcpus(cs)
>> already includes it.
>>
>> Fix this by using cs->effective_xcpus instead of user_xcpus(cs) in all
>> partcmd_update paths that calculate addmask or delmask, including the
>> PERR_NOCPUS error handling paths.
>>
>> Reproducers:
>>
>> Example 1 - Removing a sibling-excluded CPU incorrectly returns it:
>>
>> # cd /sys/fs/cgroup
>> # echo "0-1" > a1/cpuset.cpus
>> # echo "root" > a1/cpuset.cpus.partition
>> # echo "0-2" > b1/cpuset.cpus
>> # echo "root" > b1/cpuset.cpus.partition
>> # echo "2" > b1/cpuset.cpus
>> # cat cpuset.cpus.effective
>> # Actual: 0-1,3 Expected: 3
>>
>> Example 2 - Expanding to a previously excluded CPU fails to request it:
>>
>> # cd /sys/fs/cgroup
>> # echo "0-1" > a1/cpuset.cpus
>> # echo "root" > a1/cpuset.cpus.partition
>> # echo "0-2" > b1/cpuset.cpus
>> # echo "root" > b1/cpuset.cpus.partition
>> # echo "member" > a1/cpuset.cpus.partition
>> # echo "1-2" > b1/cpuset.cpus
>> # cat cpuset.cpus.effective
>> # Actual: 0-1,3 Expected: 0,3
>>
>> Fixes: 2a3602030d80 ("cgroup/cpuset: Don't invalidate sibling partitions on cpuset.cpus conflict")
>> Signed-off-by: Sun Shaojie <sunshaojie@kylinos.cn>
>> ---
>> kernel/cgroup/cpuset.c | 9 +++++----
>> 1 file changed, 5 insertions(+), 4 deletions(-)
>>
>> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
>> index 1335e437098e..5a5fa2481467 100644
>> --- a/kernel/cgroup/cpuset.c
>> +++ b/kernel/cgroup/cpuset.c
>> @@ -1821,11 +1821,11 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
>> deleting = cpumask_and(tmp->delmask,
>> newmask, parent->effective_xcpus);
>> } else {
>> - cpumask_andnot(tmp->addmask, xcpus, newmask);
>> + cpumask_andnot(tmp->addmask, cs->effective_xcpus, newmask);
>> adding = cpumask_and(tmp->addmask, tmp->addmask,
>> parent->effective_xcpus);
>>
>> - cpumask_andnot(tmp->delmask, newmask, xcpus);
>> + cpumask_andnot(tmp->delmask, newmask, cs->effective_xcpus);
>> deleting = cpumask_and(tmp->delmask, tmp->delmask,
>> parent->effective_xcpus);
>> }
>> @@ -1864,7 +1864,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
>> part_error = PERR_NOCPUS;
>> deleting = false;
>> adding = cpumask_and(tmp->addmask,
>> - xcpus, parent->effective_xcpus);
>> + cs->effective_xcpus, parent->effective_xcpus);
>> }
>> } else {
>> /*
>> @@ -1886,7 +1886,8 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
>> part_error = PERR_NOCPUS;
>> if (is_partition_valid(cs))
>> adding = cpumask_and(tmp->addmask,
>> - xcpus, parent->effective_xcpus);
>> + cs->effective_xcpus,
>> + parent->effective_xcpus);
>> } else if (is_partition_invalid(cs) && !cpumask_empty(xcpus) &&
>> cpumask_subset(xcpus, parent->effective_xcpus)) {
>> struct cgroup_subsys_state *css;
> Hi, Shaojie
>
> The code change looks reasonable to me, but I think the comment above
> the partcmd_update calculation should be updated as well.
>
> Maybe it can be updated like this:
>
> * Compute add/delete mask to/from effective_cpus
> *
> * For valid partition:
> - * addmask = exclusive_cpus & ~newmask
> + * addmask = cs->effective_xcpus & ~newmask
> * & parent->effective_xcpus
> - * delmask = newmask & ~exclusive_cpus
> + * delmask = newmask & ~cs->effective_xcpus
> * & parent->effective_xcpus
> *
> * For invalid partition:
>
> Does this look reasonable to you?
>
> Tested-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Shaojie, thanks for the fix. The changes look good to me. I also like
the suggested changes in the comment as suggested by Guopeng. Would you
mind sending a v2 with this additional change as well?
Thanks,
Longman
^ permalink raw reply
* Re: [PATCH v2 0/4] mm/zswap: Implement per-cgroup proactive writeback
From: Yosry Ahmed @ 2026-05-26 18:55 UTC (permalink / raw)
To: Hao Jia
Cc: Andrew Morton, tj, hannes, shakeel.butt, mhocko, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <9b2ac88c-a67f-2512-d898-3dadd50ec03e@gmail.com>
On Tue, May 26, 2026 at 4:56 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
>
>
> On 2026/5/26 03:24, Andrew Morton wrote:
> > On Mon, 25 May 2026 20:22:38 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
> >
> >> Zswap currently writes back pages to backing swap reactively, triggered
> >> either by the shrinker or by the pool reaching its size limit. Although
> >> proactive memory reclaim can automatically write back a portion of zswap
> >> pages via the shrinker, it cannot explicitly control the amount of
> >> writeback for a specific memory cgroup. Moreover, proactive memory reclaim
> >> may not always be triggered during a steady state.
> >>
> >> In certain scenarios, it is desirable to trigger writeback in advance to
> >> free up memory. For example, users may want to prepare for an upcoming
> >> memory-intensive workload by flushing cold memory to the backing storage
> >> when the system is relatively idle.
> >>
> >> This patch series introduces a "zswap_writeback_only" key to memory.reclaim
> >> cgroup interface, allowing users to proactively write back cold compressed
> >> pages from zswap to the backing swap device. When specified, this key
> >> bypasses standard memory reclaim and exclusively performs proactive zswap
> >> writeback up to the requested budget. If omitted, the default reclaim
> >> behavior remains unchanged.
> >
> > Thanks. AI review found a few things to complain about, one of them
> > described as "preexisting".
> >
>
> Thanks Andrew. I have replied to the AI's review comments in a separate
> email and posted v3.
> https://lore.kernel.org/all/20260526114601.67041-1-jiahao.kernel@gmail.com
Generally speaking, please give time for reviewers to take a look
before sending a new version. Less than a day is usually too fast
(unless you're iterating super fast with the reviewers). Review
feedback does not have to be addressed immediately, usually wait for a
bit to collect as much feedback as possible before spinning a new
version.
I will take a look at v3 soon, thank you.
^ permalink raw reply
* Re: [PATCH v2 2/2] cgroup/dmem: add dmem.memcg control file for double-charging to memcg
From: Eric Chanudet @ 2026-05-26 16:59 UTC (permalink / raw)
To: Michal Koutný
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tejun Heo, Jonathan Corbet, Shuah Khan, cgroups,
linux-mm, linux-kernel, dri-devel, T.J. Mercier,
Christian König, Maxime Ripard, Albert Esteve, Dave Airlie,
linux-doc
In-Reply-To: <ahBxB5a9sX9DEWvl@localhost.localdomain>
On Fri, May 22, 2026 at 05:26:16PM +0200, Michal Koutný wrote:
> Hello Eric.
>
> On Tue, May 19, 2026 at 11:59:02AM -0400, Eric Chanudet <echanude@redhat.com> wrote:
> > Add a root-only cgroupfs file "dmem.memcg" that lets an administrator
> > configure whether allocations in a dmem region should also be charged to
> > the memory controller.
>
> This kinda makes sense as it is not unlike io.cost.* device
> configurators.
>
> Just for my better understanding -- will there be a space for userspace
> to switch this? (No charged dmem allocations happen before responsible
> userspace runs, so that the attribute remains unlocked.)
>
> (I'm rather indifferent about the actual double charging/non-charging
> matter.)
Yes, this is intended to be configured before the user space stack that
would start allocating things is started. Once it has started (and tried
to charge something), the configuration is locked
>
> >
> > To handle inheritance, dmem adds a depends_on the memory controller,
> > unless MEMCG isn't configured in.
> >
> > Double-charging is disabled by default. Once a charge is attempted, the
> > setting is locked to prevent inconsistent accounting by a small 4-state
> > machine (off, on, locked off, locked on).
> >
> > The memcg to charge is derived from the pool's cgroup, since the pool
> > holds a reference to the dmem cgroup state that keeps the cgroup alive
> > until it gets uncharged.
> >
> > Signed-off-by: Eric Chanudet <echanude@redhat.com>
> > ---
> > Documentation/admin-guide/cgroup-v2.rst | 23 +++++
> > kernel/cgroup/dmem.c | 158 +++++++++++++++++++++++++++++++-
> > 2 files changed, 178 insertions(+), 3 deletions(-)
> >
> > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> > index 6efd0095ed995b1550317662bc1b56c7a7f3db23..1d2fa55ddf0faa17baa916a8914d3033e8e42359 100644
> > --- a/Documentation/admin-guide/cgroup-v2.rst
> > +++ b/Documentation/admin-guide/cgroup-v2.rst
> > @@ -2828,6 +2828,29 @@ DMEM Interface Files
> > drm/0000:03:00.0/vram0 12550144
> > drm/0000:03:00.0/stolen 8650752
> >
> > + dmem.memcg
> > + A readwrite nested-keyed file that exists only on the root
> > + cgroup.
>
> Strictly speaking this is not nested-keyed but flat keyed [1],
Indeed,
> which leads me to realization that this is the first instance of a boolean.
> All in call, such a composition comes to my mind (latter is RO):
>
> drm/0000:03:00.0/vram0 enable=0|1 locked=0|1
>
So per[1] 1 key, 2 sub-keys (enable RW, locked RO), that looks better
and match the documentation, thanks!
>
>
> > +static ssize_t dmem_cgroup_memcg_write(struct kernfs_open_file *of, char *buf,
> > + size_t nbytes, loff_t off)
> > +{
> > + while (buf) {
> > + struct dmem_cgroup_region *region;
> > + char *options, *name;
> > + bool flag;
> > +
> > + options = buf;
> > + buf = strchr(buf, '\n');
> > + if (buf)
> > + *buf++ = '\0';
>
> I recall there was a discussion about accepting only a single device per
> write(2) (at the same time I see this idiom is still present in other
> dmem.* files, so this is nothing to change in _this_ patch).
I would second that. When setting say dmem.max for 2 regions, with a
typo on the second, the first one is set, but write still get EINVAL.
Also, I just notice dmemcg_limit_write() returns EINVAL if the region is
not found (this patch returns ENODEV).
>
> Thanks,
> Michal
>
> [1] https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#format
--
Eric Chanudet
^ permalink raw reply
* Re: [PATCH v6 3/4] mm: memcontrol: add interfaces for swap tier selection
From: Baoquan He @ 2026-05-26 15:33 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, hannes,
mhocko, roman.gushchin, shakeel.butt, muchun.song, shikemeng,
nphamcs, bhe, baohua, gunho.lee, taejoon.song, hyungjun.cho,
mkoutny, baver.bae, matia.kim
In-Reply-To: <20260421055323.940344-4-youngjun.park@lge.com>
On 04/21/26 at 02:53pm, Youngjun Park wrote:
...snip...
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index c3d98ab41f1f..0f67572e5e3e 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -68,6 +68,7 @@
> #include <net/ip.h>
> #include "slab.h"
> #include "memcontrol-v1.h"
> +#include "swap_tier.h"
>
> #include <linux/uaccess.h>
>
> @@ -4130,6 +4131,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
> refcount_set(&memcg->id.ref, 1);
> css_get(css);
>
> + swap_tiers_memcg_inherit_mask(memcg);
> +
> /*
> * Ensure mem_cgroup_from_private_id() works once we're fully online.
> *
> @@ -5667,6 +5670,88 @@ static int swap_events_show(struct seq_file *m, void *v)
> return 0;
> }
>
> +static int swap_tier_show(struct seq_file *m, void *v)
> +{
> + struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
> +
> + swap_tiers_mask_show(m, READ_ONCE(memcg->tier_mask));
> + return 0;
> +}
> +
> +static ssize_t swap_tier_write(struct kernfs_open_file *of,
> + char *buf, size_t nbytes, loff_t off)
> +{
> + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
> + char *pos, *token;
> + int ret = 0;
> + int original_mask = 0;
> +
> + pos = strstrip(buf);
> +
> + spin_lock(&swap_tier_lock);
> + if (!*pos) {
> + WRITE_ONCE(memcg->tier_mask, TIER_ALL_MASK);
> + goto sync;
> + }
> +
> + original_mask = memcg->tier_mask;
> +
> + while ((token = strsep(&pos, " \t\n")) != NULL) {
> + int mask;
> +
> + if (!*token)
> + continue;
> +
> + if (token[0] != '-' && token[0] != '+') {
> + ret = -EINVAL;
> + goto err;
> + }
> +
> + mask = swap_tiers_mask_lookup(token+1);
> + if (!mask) {
> + ret = -EINVAL;
> + goto err;
> + }
> +
> + /*
> + * if child already set, cannot add that tiers for hierarch mismatching.
> + * parent compatible, child must respect parent selected swap device.
> + */
This paragraph of code comment sounds a little unnatural. We are writing
it into memcg, the child memcg is handled in
swap_tiers_memcg_sync_mask(), isn't it? I don't get the 2nd sentence.
Could you help explain?
> + switch (token[0]) {
> + case '-':
> + WRITE_ONCE(memcg->tier_mask,
> + memcg->tier_mask & ~mask);
> + break;
> + case '+':
> + WRITE_ONCE(memcg->tier_mask,
> + memcg->tier_mask | mask);
> + break;
> + default:
> + ret = -EINVAL;
> + break;
> + }
> +
> + if (ret)
> + goto err;
> + }
> +
> +sync:
> + swap_tiers_memcg_sync_mask(memcg);
> +err:
> + if (ret)
> + WRITE_ONCE(memcg->tier_mask, original_mask);
> + spin_unlock(&swap_tier_lock);
> + return ret ? ret : nbytes;
> +}
> +
...snip...
^ permalink raw reply
* Re: [PATCH v6 2/6] cgroup,cgroup/dmem: Add (dmem_)cgroup_common_ancestor helper
From: Thadeu Lima de Souza Cascardo @ 2026-05-26 15:28 UTC (permalink / raw)
To: Natalie Vock
Cc: Maarten Lankhorst, Maxime Ripard, Tejun Heo, Johannes Weiner,
Michal Koutný, Christian Koenig, Huang Rui, Matthew Auld,
Matthew Brost, Maarten Lankhorst, Thomas Zimmermann, David Airlie,
Simona Vetter, Tvrtko Ursulin, cgroups, dri-devel
In-Reply-To: <20260313-dmemcg-aggressive-protect-v6-2-7c71cc1492db@gmx.de>
On Fri, Mar 13, 2026 at 12:40:01PM +0100, Natalie Vock wrote:
> This helps to find a common subtree of two resources, which is important
> when determining whether it's helpful to evict one resource in favor of
> another.
>
> To facilitate this, add a common helper to find the ancestor of two
> cgroups using each cgroup's ancestor array.
>
> Signed-off-by: Natalie Vock <natalie.vock@gmx.de>
> ---
> include/linux/cgroup.h | 21 +++++++++++++++++++++
> include/linux/cgroup_dmem.h | 9 +++++++++
> kernel/cgroup/dmem.c | 28 ++++++++++++++++++++++++++++
> 3 files changed, 58 insertions(+)
>
> diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
> index bc892e3b37eea..560ae995e3a54 100644
> --- a/include/linux/cgroup.h
> +++ b/include/linux/cgroup.h
> @@ -561,6 +561,27 @@ static inline struct cgroup *cgroup_ancestor(struct cgroup *cgrp,
> return cgrp->ancestors[ancestor_level];
> }
>
> +/**
> + * cgroup_common_ancestor - find common ancestor of two cgroups
> + * @a: first cgroup to find common ancestor of
> + * @b: second cgroup to find common ancestor of
> + *
> + * Find the first cgroup that is an ancestor of both @a and @b, if it exists
> + * and return a pointer to it. If such a cgroup doesn't exist, return NULL.
> + *
> + * This function is safe to call as long as both @a and @b are accessible.
> + */
> +static inline struct cgroup *cgroup_common_ancestor(struct cgroup *a,
> + struct cgroup *b)
> +{
> + int level;
> +
> + for (level = min(a->level, b->level); level >= 0; level--)
> + if (a->ancestors[level] == b->ancestors[level])
> + return a->ancestors[level];
> + return NULL;
> +}
> +
> /**
> * task_under_cgroup_hierarchy - test task's membership of cgroup ancestry
> * @task: the task to be tested
> diff --git a/include/linux/cgroup_dmem.h b/include/linux/cgroup_dmem.h
> index 1a88cd0c9eb00..9d72457c4cb9d 100644
> --- a/include/linux/cgroup_dmem.h
> +++ b/include/linux/cgroup_dmem.h
> @@ -28,6 +28,8 @@ bool dmem_cgroup_below_min(struct dmem_cgroup_pool_state *root,
> struct dmem_cgroup_pool_state *test);
> bool dmem_cgroup_below_low(struct dmem_cgroup_pool_state *root,
> struct dmem_cgroup_pool_state *test);
> +struct dmem_cgroup_pool_state *dmem_cgroup_get_common_ancestor(struct dmem_cgroup_pool_state *a,
> + struct dmem_cgroup_pool_state *b);
>
> void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool);
> #else
> @@ -75,6 +77,13 @@ static inline bool dmem_cgroup_below_low(struct dmem_cgroup_pool_state *root,
> return false;
> }
>
> +static inline
> +struct dmem_cgroup_pool_state *dmem_cgroup_get_common_ancestor(struct dmem_cgroup_pool_state *a,
> + struct dmem_cgroup_pool_state *b)
> +{
> + return NULL;
> +}
> +
> static inline void dmem_cgroup_pool_state_put(struct dmem_cgroup_pool_state *pool)
> { }
>
> diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
> index 28227405f7cfe..9ae085a7fcb73 100644
> --- a/kernel/cgroup/dmem.c
> +++ b/kernel/cgroup/dmem.c
> @@ -756,6 +756,34 @@ bool dmem_cgroup_below_low(struct dmem_cgroup_pool_state *root,
> }
> EXPORT_SYMBOL_GPL(dmem_cgroup_below_low);
>
> +/**
> + * dmem_cgroup_get_common_ancestor(): Find the first common ancestor of two pools.
> + * @a: First pool to find the common ancestor of.
> + * @b: First pool to find the common ancestor of.
> + *
> + * Return: The first pool that is a parent of both @a and @b, or NULL if either @a or @b are NULL,
> + * or if such a pool does not exist. A reference to the returned pool is grabbed and must be
> + * released by the caller when it is done using the pool.
> + */
> +struct dmem_cgroup_pool_state *dmem_cgroup_get_common_ancestor(struct dmem_cgroup_pool_state *a,
> + struct dmem_cgroup_pool_state *b)
> +{
> + struct cgroup *ancestor_cgroup;
> + struct cgroup_subsys_state *ancestor_css;
> +
> + if (!a || !b)
> + return NULL;
> +
> + ancestor_cgroup = cgroup_common_ancestor(a->cs->css.cgroup, b->cs->css.cgroup);
> + if (!ancestor_cgroup)
> + return NULL;
> +
> + ancestor_css = cgroup_e_css(ancestor_cgroup, &dmem_cgrp_subsys);
cgroup_e_css must be called in RCU read context. Besides, a reference to
ancestor_css must be got as later on, dmem_cgroup_pool_state_put will call
css_put.
Here is my fixup, which I tested and did not cause RCU or reference
warnings whereas the original patch caused such issues.
Feel free to use it with my sign-off.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
Regards.
Cascardo.
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 72ee8f1d69ef..28adb042baca 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -773,6 +773,7 @@ struct dmem_cgroup_pool_state *dmem_cgroup_get_common_ancestor(struct dmem_cgrou
{
struct cgroup *ancestor_cgroup;
struct cgroup_subsys_state *ancestor_css;
+ struct dmem_cgroup_pool_state *pool = NULL;
if (!a || !b)
return NULL;
@@ -781,9 +782,15 @@ struct dmem_cgroup_pool_state *dmem_cgroup_get_common_ancestor(struct dmem_cgrou
if (!ancestor_cgroup)
return NULL;
+ rcu_read_lock();
ancestor_css = cgroup_e_css(ancestor_cgroup, &dmem_cgrp_subsys);
+ if (css_tryget(ancestor_css))
+ pool = get_cg_pool_unlocked(css_to_dmemcs(ancestor_css), a->region);
+ if (!pool)
+ css_put(ancestor_css);
+ rcu_read_unlock();
- return get_cg_pool_unlocked(css_to_dmemcs(ancestor_css), a->region);
+ return pool;
}
EXPORT_SYMBOL_GPL(dmem_cgroup_get_common_ancestor);
--
2.47.3
> +
> + return get_cg_pool_unlocked(css_to_dmemcs(ancestor_css), a->region);
> +}
> +EXPORT_SYMBOL_GPL(dmem_cgroup_get_common_ancestor);
> +
> static int dmem_cgroup_region_capacity_show(struct seq_file *sf, void *v)
> {
> struct dmem_cgroup_region *region;
>
> --
> 2.53.0
>
^ permalink raw reply related
* Re: [PATCH 0/7 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter
From: Joshua Hahn @ 2026-05-26 15:16 UTC (permalink / raw)
To: Joshua Hahn
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, cgroups, linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260525190455.2843786-1-joshua.hahnjy@gmail.com>
On Mon, 25 May 2026 12:04:47 -0700 Joshua Hahn <joshua.hahnjy@gmail.com> wrote:
> Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged
> allocations, allowing small allocations to avoid walking the expensive
> mem_cgroup hierarchy traversal and atomic operations on each charge.
> This design introduces a fastpath, but there is room for improvement:
Hello everyone,
Sashiko has left some great comments on the series, some of which are
things I need to address for the next iteration, while others are false
positives. There aren't a bunch, so I'll go over all of them below.
Note that the same warning brought up in 1/7 is duplicated for the rest
of the series; I've addressed my thoughts for that in [1].
> Joshua Hahn (7):
> mm/page_counter: introduce per-page_counter stock
Sashiko raised a concern about how zeroing per-cpu stock during the
nolock drain could race with in-flight charges that read the value
before it gets zeroed, leading to duplicate drains. I think this is
solvable by just changing the order in the callsites (disable, then drain).
More details can be found in [1].
> mm/page_counter: use page_counter_stock in page_counter_try_charge
Sashiko raises the same concern as [1].
> mm/page_counter: introduce stock drain APIs
Same concern as [1].
> mm/memcontrol: convert memcg to use page_counter_stock
Sashiko raises 4 concerns, of which I think 3 of them are false positives
(or not as serious as Sashiko makes it out to be).
(1) Sashiko asks whether the synchronous draining with the percpu_charge_mutex
lock held could lead to more time spent holding the lock, which means
more callers of drain_all_stock would fail the trylock and just skip draining.
To clarify, even in the original code, two tasks simultaneously calling
drain_all_stock would serialize and only one of them would schedule the
drain work, so this problem definitely existed before as well. It's just that
the window for this race is a bit longer now.
I do think that there is actually a behavior change here (for the better).
Previously, drain_all_stock had no guarantees on whether the stock was
drained before retrying. Now, if the caller can get the trylock, they have
a stronger guarantee that the stock is drained before retrying the drain.
On the note of premature OOMs, each retry loop takes much longer than the
draining itself; I would imagine that by the time the next retry loop happens,
there's a better chance that the trylock succeeds in the next iteration.
(2) Sashiko also raises another concern about a potential ABBA deadlock with
the mmap_lock. I think this concern is not really true, the synchronous
work being done (drain_stock_on_cpu) only takes a local lock. Hopefully I'm
not missing anything here.
(3) I think Sashiko's concerns about NOHZ / CPU isolation is real. But it shouldn't
be too bad, all I need is a cpu_is_isolated() check in the for_each_online_cpu
iterator. Again, not draining a CPU is not fatal here, so it shouldn't be too
big of a problem to skip some of them.
I also just wanted to note here explicitly that we don't need the
migrate_disable() for the memcg stock drain, since we don't differentiate
between local drain work & remote drain scheduling (like objcg_stock).
(4) Finally Sashiko asks if we should enable the memcg->memsw stock here.
That's included in the very next patch : -) I separated them so that they
can be reviewed separately, since they are separate ideas.
> mm/memcontrol: optimize memsw stock for cgroup v1
Both concerns here are addressed in the previous section.
> mm/memcontrol: optimize stock usage for cgroup v2
Sashiko raises 3 concerns, of which I think all of them are actually OK.
(1) If we drain the parent memcg stock on first child creation, then this would
mean that there will be additional synchronous work being done with the
cgroup_mutex lock held. I personally think this is fine, since it happens
once per parent cgroup, and the draining work is pretty cheap. But I would
appreciate it if other reviewers could chime in here.
(2) Sashiko also asks whether we need cpus_read_lock during the iteration.
I think it's fine without it; if a CPU happens to go offline during the
iteration, then that work will be scheduled on another CPU. That's fine,
duplicate draining work on 1 CPU isn't the end of the world (and preferable
to taking a cpus_read_lock here). As for the dying CPU, it will drain its
own stock during the destruction path anyways, so no stock is lost.
(3) This one is not related to this series, so I'll move on.
> mm/memcontrol: remove unused memcg_stock code
No comments for this patch.
I think that's all the comments that Sashiko raised for this patch. Most of them
had to do with performance tradeoffs, for which I hope that my testing results
in the cover letter were able to instill some confidence that a lot of these
tradeoffs aren't as bad as they seem. Regardless, I would really appreciate
reviewer feedback on whether they think it is acceptable.
There are definitely some real bugs that I want to address, so a v4 will be
incoming to address those (in a week or so).
Thank you Sashiko!
Joshua
[1] https://lore.kernel.org/linux-mm/20260525194506.3414995-1-joshua.hahnjy@gmail.com/
^ permalink raw reply
* [PATCH v3] security: Expand task_setscheduler LSM hook to include CPU affinity mask
From: Aaron Tomlin @ 2026-05-26 14:28 UTC (permalink / raw)
To: tsbogend, paul, jmorris, serge, mingo, peterz, juri.lelli,
vincent.guittot, stephen.smalley.work, casey, longman, tj, hannes,
mkoutny
Cc: chenridong, dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, omosnace, kees, atomlin, neelx, sean, chjohnst,
steve, mproche, nick.lange, cgroups, linux-mips, linux-fsdevel,
linux-security-module, selinux, linux-kernel
At present, the task_setscheduler LSM hook provides security modules
with the opportunity to mediate changes to a task's scheduling policy.
However, when invoked via sched_setaffinity(), the hook lacks
visibility into the actual CPU affinity mask being requested.
Consequently, BPF-based security modules are entirely blind to the
target CPUs and cannot make granular access control decisions based on
spatial isolation.
In modern multi-tenant and real-time environments, CPU isolation is a
critical boundary. The inability to audit or restrict specific CPU
pinning requests limits the effectiveness of eBPF-driven security
policies, particularly when attempting to shield isolated or
cryptographic cores from unprivileged or compromised tasks.
This patch expands the security_task_setscheduler() hook signature to
include a pointer to the requested cpumask. Because this is a shared
hook used for multiple scheduling attribute changes, call sites that do
not modify CPU affinity are updated to safely pass NULL.
To protect against unverified dereferences, the parameter is annotated
with __nullable in the LSM hook definition, ensuring the BPF verifier
mandates explicit NULL checks for attached eBPF programs.
This change updates all in-tree security modules (SELinux and Smack) to
accommodate the new parameter mechanically, whilst providing BPF LSMs
with the necessary context to enforce strict affinity policies.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
This patch is strictly dependent on the prior acceptance of "mips:
sched: Fix CPUMASK_OFFSTACK memory corruption in MT fpaff" (Message-ID:
20260526141651.773306-1-atomlin@atomlin.com), as expanding the LSM hook
signature requires passing the mask pointer from
mipsmt_sys_sched_setaffinity().
Changes since v2 [1]:
- Dropped patch 1. This is to be addressed by the cgroup cpuset
maintainer (Waiman Long)
- Dropped patch 3. Will be submitted as a separate patch (Paul Moore)
Changes since v1 [2]:
- Reordered the allocation and user-copy of new_mask in the MIPS
architecture's mipsmt_sys_sched_setaffinity() to occur before the
LSM hook is invoked. This ensures the security modules evaluate a fully
populated mask rather than uninitialised memory, while cleanly handling
error unwinding
- Updated cpuset_can_fork() to pass the destination cpuset's effective CPU
mask instead of NULL
[1]: https://lore.kernel.org/lkml/20260509213803.968464-1-atomlin@atomlin.com/
[2]: https://lore.kernel.org/lkml/20260509164847.939294-1-atomlin@atomlin.com/
---
arch/mips/kernel/mips-mt-fpaff.c | 2 +-
fs/proc/base.c | 2 +-
include/linux/lsm_hook_defs.h | 3 ++-
include/linux/security.h | 11 +++++++----
kernel/cgroup/cpuset.c | 4 ++--
kernel/sched/syscalls.c | 4 ++--
security/commoncap.c | 7 +++++--
security/security.c | 11 ++++++-----
security/selinux/hooks.c | 3 ++-
security/smack/smack_lsm.c | 11 +++++++++--
10 files changed, 37 insertions(+), 21 deletions(-)
diff --git a/arch/mips/kernel/mips-mt-fpaff.c b/arch/mips/kernel/mips-mt-fpaff.c
index 4fead87d2f43..c68d1676350e 100644
--- a/arch/mips/kernel/mips-mt-fpaff.c
+++ b/arch/mips/kernel/mips-mt-fpaff.c
@@ -110,7 +110,7 @@ asmlinkage long mipsmt_sys_sched_setaffinity(pid_t pid, unsigned int len,
goto out_unlock;
}
- retval = security_task_setscheduler(p);
+ retval = security_task_setscheduler(p, new_mask);
if (retval)
goto out_unlock;
diff --git a/fs/proc/base.c b/fs/proc/base.c
index d9acfa89c894..ac4096958a00 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -2619,7 +2619,7 @@ static ssize_t timerslack_ns_write(struct file *file, const char __user *buf,
}
rcu_read_unlock();
- err = security_task_setscheduler(p);
+ err = security_task_setscheduler(p, NULL);
if (err) {
count = err;
goto out;
diff --git a/include/linux/lsm_hook_defs.h b/include/linux/lsm_hook_defs.h
index 2b8dfb35caed..6ec7bc04a1b7 100644
--- a/include/linux/lsm_hook_defs.h
+++ b/include/linux/lsm_hook_defs.h
@@ -255,7 +255,8 @@ LSM_HOOK(int, 0, task_prlimit, const struct cred *cred,
const struct cred *tcred, unsigned int flags)
LSM_HOOK(int, 0, task_setrlimit, struct task_struct *p, unsigned int resource,
struct rlimit *new_rlim)
-LSM_HOOK(int, 0, task_setscheduler, struct task_struct *p)
+LSM_HOOK(int, 0, task_setscheduler, struct task_struct *p,
+ const struct cpumask *in_mask__nullable)
LSM_HOOK(int, 0, task_getscheduler, struct task_struct *p)
LSM_HOOK(int, 0, task_movememory, struct task_struct *p)
LSM_HOOK(int, 0, task_kill, struct task_struct *p, struct kernel_siginfo *info,
diff --git a/include/linux/security.h b/include/linux/security.h
index 41d7367cf403..8b74153daa43 100644
--- a/include/linux/security.h
+++ b/include/linux/security.h
@@ -196,7 +196,8 @@ extern int cap_mmap_addr(unsigned long addr);
extern int cap_task_fix_setuid(struct cred *new, const struct cred *old, int flags);
extern int cap_task_prctl(int option, unsigned long arg2, unsigned long arg3,
unsigned long arg4, unsigned long arg5);
-extern int cap_task_setscheduler(struct task_struct *p);
+extern int cap_task_setscheduler(struct task_struct *p,
+ const struct cpumask *in_mask);
extern int cap_task_setioprio(struct task_struct *p, int ioprio);
extern int cap_task_setnice(struct task_struct *p, int nice);
extern int cap_vm_enough_memory(struct mm_struct *mm, long pages);
@@ -531,7 +532,8 @@ int security_task_prlimit(const struct cred *cred, const struct cred *tcred,
unsigned int flags);
int security_task_setrlimit(struct task_struct *p, unsigned int resource,
struct rlimit *new_rlim);
-int security_task_setscheduler(struct task_struct *p);
+int security_task_setscheduler(struct task_struct *p,
+ const struct cpumask *in_mask);
int security_task_getscheduler(struct task_struct *p);
int security_task_movememory(struct task_struct *p);
int security_task_kill(struct task_struct *p, struct kernel_siginfo *info,
@@ -1392,9 +1394,10 @@ static inline int security_task_setrlimit(struct task_struct *p,
return 0;
}
-static inline int security_task_setscheduler(struct task_struct *p)
+static inline int security_task_setscheduler(struct task_struct *p,
+ const struct cpumask *in_mask)
{
- return cap_task_setscheduler(p);
+ return cap_task_setscheduler(p, in_mask);
}
static inline int security_task_getscheduler(struct task_struct *p)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 5c33ab20cc20..7b3dfccb77d8 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3033,7 +3033,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
goto out_unlock;
if (setsched_check) {
- ret = security_task_setscheduler(task);
+ ret = security_task_setscheduler(task, cs->effective_cpus);
if (ret)
goto out_unlock;
}
@@ -3591,7 +3591,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
if (ret)
goto out_unlock;
- ret = security_task_setscheduler(task);
+ ret = security_task_setscheduler(task, cs->effective_cpus);
if (ret)
goto out_unlock;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index b215b0ead9a6..68bc7e466fb1 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -540,7 +540,7 @@ int __sched_setscheduler(struct task_struct *p,
if (attr->sched_flags & SCHED_FLAG_SUGOV)
return -EINVAL;
- retval = security_task_setscheduler(p);
+ retval = security_task_setscheduler(p, NULL);
if (retval)
return retval;
}
@@ -1213,7 +1213,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
return -EPERM;
}
- retval = security_task_setscheduler(p);
+ retval = security_task_setscheduler(p, in_mask);
if (retval)
return retval;
diff --git a/security/commoncap.c b/security/commoncap.c
index 3399535808fe..d86f1c2b9210 100644
--- a/security/commoncap.c
+++ b/security/commoncap.c
@@ -1222,13 +1222,16 @@ static int cap_safe_nice(struct task_struct *p)
/**
* cap_task_setscheduler - Determine if scheduler policy change is permitted
* @p: The task to affect
+ * @in_mask: Requested CPU affinity mask (ignored)
*
* Determine if the requested scheduler policy change is permitted for the
- * specified task.
+ * specified task. The capabilities security module does not evaluate the
+ * @in_mask parameter, relying solely on cap_safe_nice().
*
* Return: 0 if permission is granted, -ve if denied.
*/
-int cap_task_setscheduler(struct task_struct *p)
+int cap_task_setscheduler(struct task_struct *p,
+ const struct cpumask *in_mask __always_unused)
{
return cap_safe_nice(p);
}
diff --git a/security/security.c b/security/security.c
index 4e999f023651..53804ee40df5 100644
--- a/security/security.c
+++ b/security/security.c
@@ -3240,17 +3240,18 @@ int security_task_setrlimit(struct task_struct *p, unsigned int resource,
}
/**
- * security_task_setscheduler() - Check if setting sched policy/param is allowed
+ * security_task_setscheduler() - Check if setting sched policy/param/affinity is allowed
* @p: target task
+ * @in_mask: requested CPU affinity mask, or NULL if not changing affinity
*
- * Check permission before setting scheduling policy and/or parameters of
- * process @p.
+ * Check permission before setting the scheduling policy, parameters, and/or
+ * CPU affinity of process @p.
*
* Return: Returns 0 if permission is granted.
*/
-int security_task_setscheduler(struct task_struct *p)
+int security_task_setscheduler(struct task_struct *p, const struct cpumask *in_mask)
{
- return call_int_hook(task_setscheduler, p);
+ return call_int_hook(task_setscheduler, p, in_mask);
}
/**
diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c
index 0f704380a8c8..5f0914db23f6 100644
--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -4557,7 +4557,8 @@ static int selinux_task_setrlimit(struct task_struct *p, unsigned int resource,
return 0;
}
-static int selinux_task_setscheduler(struct task_struct *p)
+static int selinux_task_setscheduler(struct task_struct *p,
+ const struct cpumask *in_mask __always_unused)
{
return avc_has_perm(current_sid(), task_sid_obj(p), SECCLASS_PROCESS,
PROCESS__SETSCHED, NULL);
diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c
index 3f9ae05039a2..a77143beff44 100644
--- a/security/smack/smack_lsm.c
+++ b/security/smack/smack_lsm.c
@@ -2343,10 +2343,17 @@ static int smack_task_getioprio(struct task_struct *p)
/**
* smack_task_setscheduler - Smack check on setting scheduler
* @p: the task object
+ * @in_mask: Requested CPU affinity mask (ignored)
*
- * Return 0 if read access is permitted
+ * Evaluate whether the current task has write access to the target task @p
+ * to change its scheduling policy. The Smack security module relies
+ * strictly on label-based access control and does not evaluate CPU
+ * affinity masks.
+ *
+ * Return: 0 if write access is permitted
*/
-static int smack_task_setscheduler(struct task_struct *p)
+static int smack_task_setscheduler(struct task_struct *p,
+ const struct cpumask *in_mask __always_unused)
{
return smk_curacc_on_task(p, MAY_WRITE, __func__);
}
base-commit: 5200f5f493f79f14bbdc349e402a40dfb32f23c8
prerequisite-patch-id: f9200d420002c9fd0663d0ec00c83db866889c19
--
2.51.0
^ permalink raw reply related
* Re: [RFC PATCH bpf-next v7 00/11] mm: BPF struct_ops for dynamic memory protection and async reclaim
From: Usama Arif @ 2026-05-26 13:41 UTC (permalink / raw)
To: Hui Zhu
Cc: Usama Arif, Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Kumar Kartikeya Dwivedi,
Song Liu, Yonghong Song, Jiri Olsa, Johannes Weiner, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, JP Kobryn,
Andrew Morton, Shuah Khan, davem, Jakub Kicinski,
Jesper Dangaard Brouer, Stanislav Fomichev, KP Singh, Tao Chen,
Mykyta Yatsenko, Leon Hwang, Anton Protopopov, Amery Hung,
Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo, Peter Zijlstra,
Miguel Ojeda, Nathan Chancellor, Kees Cook, Tejun Heo, Jeff Xu,
mkoutny, Jan Hendrik Farr, Christian Brauner, Randy Dunlap,
Brian Gerst, Masahiro Yamada, Willem de Bruijn, Jason Xing,
Paul Chaignon, Chen Ridong, Lance Yang, Jiayuan Chen,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest,
geliang, baohua, Hui Zhu
In-Reply-To: <cover.1779760876.git.zhuhui@kylinos.cn>
On Tue, 26 May 2026 10:20:00 +0800 Hui Zhu <hui.zhu@linux.dev> wrote:
> From: Hui Zhu <zhuhui@kylinos.cn>
>
> Overview:
> This series introduces BPF struct_ops support for the memory controller,
> enabling userspace BPF programs to implement custom, dynamic memory
> management policies per cgroup. The feature allows BPF programs to hook
> into the core reclaim and charge paths without requiring kernel
> modifications, providing a flexible alternative to static knobs such as
> memory.low and memory.min.
>
> The series enables two complementary use cases.
>
> Dynamic memory protection: static memory protection thresholds
> (memory.low, memory.min) are poor fits for workloads whose actual memory
> activity varies over time. A high-priority cgroup holding a large working
> set but temporarily idle will still suppress reclaim on its siblings,
> wasting available memory. A BPF-driven approach can observe real workload
> activity -- page faults, charge/uncharge events -- and activate or
> withdraw protection dynamically. The test results at the end of this
> letter quantify the difference: in a scenario where the high-priority
> cgroup is idle, the BPF-controlled low-priority cgroup achieves roughly
> 37x higher throughput than with static memory.low.
>
> Asynchronous proactive reclaim: the memcg_charged and memcg_uncharged
> hooks, combined with the BPF workqueue mechanism and the new
> bpf_try_to_free_mem_cgroup_pages() kfunc, enable BPF programs to perform
> proactive background reclaim without blocking the charge path. The
> pattern works as follows: the memcg_charged callback tracks accumulated
> memory usage; when usage crosses a configurable threshold, it enqueues an
> asynchronous work item via bpf_wq_start() and returns immediately without
> throttling the charging task. The workqueue callback then invokes
> bpf_try_to_free_mem_cgroup_pages() to reclaim pages from the target
> cgroup; if usage remains elevated after reclaim, the callback re-enqueues
> itself to continue. This allows a BPF program to keep a cgroup's
> footprint below its hard limit (memory.max) entirely in the background,
> avoiding the OOM killer or direct-reclaim stalls that would otherwise
> occur. The selftest for this feature (patch 10/11) validates the
> mechanism concretely: a workload that writes and mmaps a 64 MB file inside
> a 32 MB cgroup reliably triggers memory.events "max" events without BPF;
> with the async reclaim program attached, the "max" counter does not
> increase at all across the same workload.
>
Hi Hui,
Thanks for the series.
Would it not be simpler to just have another memcg knob, something like
memory.high_async.
When memory usage > memory.high_async, queue a per-memcg work item that calls
try_to_free_mem_cgroup_pages() until usage drops back below some threshold.
I am not sure I see what programability aspect from bpf you need here.
Thanks
>
> 08/11 selftests/bpf: Add tests for memcg_bpf_ops
> Adds prog_tests/memcg_ops.c covering three scenarios:
> memcg_charged-only throttling, below_low + memcg_charged
> interaction, and below_min + memcg_charged interaction. A
> tracepoint on memcg:count_memcg_events (PGFAULT) is used to
> detect memory pressure and trigger hooks accordingly.
>
> 09/11 selftests/bpf: Add test for memcg_bpf_ops hierarchies
> Validates BPF_F_ALLOW_OVERRIDE attachment semantics across a
> three-level cgroup hierarchy: attach with ALLOW_OVERRIDE at the
> root, override at the middle level without the flag, then assert
> that attaching to the leaf correctly fails with -EBUSY.
>
> 10/11 selftests/bpf: Add selftest for memcg async reclaim via BPF
> Demonstrates and validates asynchronous memory reclaim: a BPF
> program uses the memcg_charged/memcg_uncharged hooks to track
> accumulated usage and, when a threshold is exceeded, enqueues a
> bpf_wq_start() workqueue item that calls
> bpf_try_to_free_mem_cgroup_pages() without blocking the charge
> path. The test asserts that with the BPF program active,
> memory.events "max" events do not increase under a workload
> that would otherwise exceed the hard limit.
>
> 11/11 samples/bpf: Add memcg priority control and async reclaim example
> Adds a complete sample (samples/bpf/memcg.bpf.c + memcg.c)
> demonstrating both features. The BPF side monitors PGFAULT
> events on a high-priority cgroup; when the per-second fault
> count crosses a configurable threshold, it activates below_low
> or below_min protection for the high-priority cgroup and/or
> applies a charge delay to the low-priority cgroup. Six
> struct_ops variants are exported so userspace can attach only
> the hooks needed. Async reclaim is optionally combined with
> priority throttling via a shared low-cgroup ops map.
>
> Test Environment:
> The following examples run on x86_64 QEMU (10 CPUs, 2 GB RAM), using
> a tmpfs-backed file on the host as a swap device to reduce I/O impact.
> Two cgroups are created -- high (high-priority) and low (low-priority)
> -- and each test runs two concurrent stress-ng workloads, one per
> cgroup, each requesting 3 GB of memory.
>
> # mkdir /sys/fs/cgroup/high /sys/fs/cgroup/low
> # free -h
> total used free shared buff/cache available
> Mem: 1.9Gi 317Mi 1.6Gi 1.0Mi 144Mi 1.6Gi
> Swap: 4.0Gi 0B 4.0Gi
>
> Baseline: no memory priority policy:
> Both cgroups run without any reclaim protection. Results are roughly
> equal, as expected:
>
> cgroup bogo ops/s
> high 4,979
> low 4,927
>
> Test 1: memory.low protection:
> Setting memory.low on the high-priority cgroup protects it from
> reclaim, at the cost of pushing reclaim pressure onto the low-priority
> cgroup:
>
> # echo $((3 * 1024 * 1024 * 1024)) > /sys/fs/cgroup/high/memory.low
>
> cgroup bogo ops/s
> high 450,290
> low 11,307
>
> The high-priority cgroup benefits significantly, but memory.low relies
> on static usage thresholds and cannot adapt to actual workload
> behavior.
>
> Test 2: memory.low with an idle high-priority task:
> Here the high-priority cgroup runs a Python script that allocates 3 GB
> and then sleeps, simulating a low-activity but memory-holding workload.
> Because the process is idle, it generates no page faults and does not
> actively use its memory. Yet memory.low still protects it, continuing
> to suppress the low-priority cgroup's performance:
>
> cgroup bogo ops/s
> low 14,757
>
> The low-priority cgroup remains significantly throttled despite the
> high-priority cgroup being effectively idle -- a clear limitation of
> static memory.low control.
>
> Test 3: memcg eBPF -- dynamic priority control:
> memcg is a sample program introduced in this patch series
> (samples/bpf/memcg.c + memcg.bpf.c). It loads a BPF program that
> monitors PGFAULT events in the high-priority cgroup. When the
> per-second fault count exceeds a configured threshold, the hook
> activates below_min protection for one second; otherwise the cgroup
> receives no special treatment.
>
> # ./memcg --low_path=/sys/fs/cgroup/low \
> --high_path=/sys/fs/cgroup/high \
> --threshold=1 --use_below_min
> Successfully attached!
>
> 3a. Both cgroups under active memory pressure:
>
> When both cgroups run stress-ng, the high-priority cgroup generates
> frequent page faults and the BPF hook activates protection, matching
> the behavior of memory.low:
>
> cgroup bogo ops/s
> high 404,392
> low 11,404
>
> 3b. High-priority cgroup is idle (Python + sleep):
>
> Because the sleeping Python process generates no page faults, the BPF
> hook never activates, and the low-priority cgroup is free to reclaim
> memory normally:
>
> cgroup bogo ops/s
> low 551,083
>
> This is a ~37x improvement over the equivalent memory.low scenario
> (Test 2), demonstrating that eBPF-driven dynamic control can
> accurately reflect actual workload activity and avoid unnecessary
> protection of idle high-priority tasks.
>
> Summary:
> Scenario low-cgroup bogo ops/s
> Baseline (no policy) ~4,927
> memory.low, both active ~11,307
> memory.low, high idle ~14,757
> memcg eBPF, both active ~11,404
> memcg eBPF, high idle ~551,083
>
> References:
> [1] https://patchew.org/linux/20260127024421.494929-1-roman.gushchin@linux.dev/
>
> Changelog:
> v7:
> Change base commits of "mm: BPF OOM" to v3.
> Some fixes according to the comments of bpf-ci.
> Rename get_high_delay_ms hook to memcg_charged; add memcg_uncharged
> hook for tracking uncharge events.
> Update below_low and below_min hooks to receive elow/emin and usage
> as explicit arguments.
> Add bpf_try_to_free_mem_cgroup_pages kfunc to expose cgroup reclaim
> to BPF programs.
> Add selftest for BPF-driven asynchronous page reclaim.
> Extend samples/bpf/memcg to support async reclaim in addition to
> priority throttling.
> v6:
> Based on the bot+bof-ci comments, fixed the following issues.
> Added fast-path check with unlikely() before SRCU lock acquisition to
> optimize the no-BPF case in BPF_MEMCG_CALL.
> Add missing newline in pr_warn message to bpf_memcontrol_init.
> Added comprehensive child process exit status checking with WIFEXITED()
> and WEXITSTATUS(), and added zombie process prevention in
> real_test_memcg_ops.
> Changed malloc() to calloc() for BSS data allocation in all test
> functions and samples main function.
> Change srcu_read_lock(&memcg_bpf_srcu) to
> lockdep_assert_held(&cgroup_mutex) in function memcontrol_bpf_online
> and memcontrol_bpf_offline.
> v5:
> Based on the bot+bof-ci comments, fixed the following issues.
> Fixed issues in memcg_ops.c and memcg.bpf.c by moving variable
> declaration to the beginning of need_threshold() function.
> The 'u64 current_ts' variable must be declared before any
> executable statements
> Improved input validation in samples/bpf/memcg.c by adding a new
> parse_u64() helper function. This function properly handles errors
> from strtoull() and provides better error messages when parsing
> threshold and over_high_ms command-line arguments.
> Move check for prog->sleepable after validating member offsets in
> mm/bpf_memcontrol.c bpf_memcg_ops_check_member.
> Fixed sscanf return value checking in prog_tests/memcg_ops.c.
> Changed the condition from 'sscanf() < 0' to 'sscanf() != 1' because
> sscanf returns the number of successfully matched items, not a negative
> value on error. This makes the test more reliable when reading timing
> data from temporary files.
> v4:
> Fix the issues according to the comments from bot+bof-ci.
> According to JP Kobryn's comments, move exit(0) from
> real_test_memcg_ops_child_work to real_test_memcg_ops.
> Fix issues in the bpf_memcg_ops_reg function.
> v3:
> According to the comments from Michal Koutný and Chen Ridong, update hooks
> to get_high_delay_ms, below_low, below_min, handle_cgroup_online, and
> handle_cgroup_offline.
> According to Michal Koutný's comments, add BPF_F_ALLOW_OVERRIDE
> support to memcg_bpf_ops.
> v2:
> According to Tejun Heo's comments, rebased on Roman Gushchin's BPF
> OOM patch series [1] and added hierarchical delegation support.
> According to the comments from Roman Gushchin and Michal Hocko, designed
> concrete use case scenarios and provided test results.
>
> Hui Zhu (7):
> bpf: Pass flags in bpf_link_create for struct_ops
> mm: memcontrol: Add BPF struct_ops for memory controller
> mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
> selftests/bpf: Add tests for memcg_bpf_ops
> selftests/bpf: Add test for memcg_bpf_ops hierarchies
> selftests/bpf: Add selftest for memcg async reclaim via BPF
> samples/bpf: Add memcg priority control and async reclaim example
>
> Roman Gushchin (4):
> bpf: move bpf_struct_ops_link into bpf.h
> bpf: allow attaching struct_ops to cgroups
> libbpf: fix return value on memory allocation failure
> libbpf: introduce bpf_map__attach_struct_ops_opts()
>
> MAINTAINERS | 6 +
> include/linux/bpf-cgroup-defs.h | 3 +
> include/linux/bpf-cgroup.h | 16 +
> include/linux/bpf.h | 10 +
> include/linux/memcontrol.h | 250 ++++++-
> include/uapi/linux/bpf.h | 5 +-
> kernel/bpf/bpf_struct_ops.c | 67 +-
> kernel/bpf/cgroup.c | 46 ++
> mm/bpf_memcontrol.c | 355 +++++++++-
> mm/memcontrol.c | 43 +-
> samples/bpf/.gitignore | 1 +
> samples/bpf/Makefile | 8 +-
> samples/bpf/memcg.bpf.c | 380 +++++++++++
> samples/bpf/memcg.c | 411 ++++++++++++
> tools/include/uapi/linux/bpf.h | 3 +-
> tools/lib/bpf/libbpf.c | 22 +-
> tools/lib/bpf/libbpf.h | 14 +
> tools/lib/bpf/libbpf.map | 1 +
> tools/testing/selftests/bpf/cgroup_helpers.c | 41 ++
> tools/testing/selftests/bpf/cgroup_helpers.h | 2 +
> .../bpf/prog_tests/memcg_async_reclaim.c | 333 +++++++++
> .../selftests/bpf/prog_tests/memcg_ops.c | 634 ++++++++++++++++++
> .../selftests/bpf/progs/memcg_async_reclaim.c | 203 ++++++
> tools/testing/selftests/bpf/progs/memcg_ops.c | 132 ++++
> 24 files changed, 2952 insertions(+), 34 deletions(-)
> create mode 100644 samples/bpf/memcg.bpf.c
> create mode 100644 samples/bpf/memcg.c
> create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
> create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_ops.c
> create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
> create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops.c
>
> --
> 2.43.0
>
>
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-26 12:40 UTC (permalink / raw)
To: K Prateek Nayak
Cc: Zhang Qiao, mingo, longman, chenridong, juri.lelli,
vincent.guittot, dietmar.eggemann, rostedt, bsegall, mgorman,
vschneid, tj, hannes, mkoutny, cgroups, linux-kernel, jstultz,
qyousef, Hui Tang
In-Reply-To: <20260526110709.GF4149641@noisy.programming.kicks-ass.net>
On Tue, May 26, 2026 at 01:07:09PM +0200, Peter Zijlstra wrote:
> On Tue, May 26, 2026 at 04:24:32PM +0530, K Prateek Nayak wrote:
> > Hello Peter,
> >
> > On 5/26/2026 3:22 PM, Peter Zijlstra wrote:
> > > On Tue, May 26, 2026 at 02:45:45PM +0530, K Prateek Nayak wrote:
> > >
> > >> The suggested diff above solves the crash in my case but your
> > >> mileage may vary. Peter can comment if this is the right thing
> > >> to do or not :-)
> > >
> > > Is this a different issue than the one you raised before?
> >
> > Yes, this is different. Essentially, this is what is happening:
> >
> > throttle_cfs_rq_work()
> > task_rq_lock()
> >
> > dequeue_task_fair(current) /* Task is dequeued on cfs side */
> > __dequeue_task(current)
> > dequeue_hierarchy(current);
> > current->se.on_rq = 0;
> > /* update_load_sub() */
> > resched_curr(); /* Initiates a resched */
> >
> > task_rq_unlock()
> > local_irq_enable();
> >
> > =====> sched_tick()
> > task_tick_fair()
> > __calc_prop_weight()
> > /*
> > * Oops: update_load_sub() above has
> > * 0ed the weight of cfs_rq.
> > */
> > <====
> >
> > preempt_schedule_irq()
> > next = ...
> > put_prev_set_next_task() /* The runtime context is switched here */
> >
>
> Ah, right. OK, I'll go have a poke once I get these proxy patches I've
> been spending too much time on posted.
Yes, your solution seems reasonable. I'll fold that and push out a new
version a little later today.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox