* [PATCH v7 8/9] cgroup/cpuset: Support multiple source cpusets for cpuset_*attach()
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
There are 2 possible scenarios where the cgroup_taskset structure
passed into the cgroup can_attach() and attach() methods can contain
task migration data with multiple source cpusets.
- A multithread application with threads in different cpusets is
fully migrated into a new cpuset.
- Disabling v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
The current cpuset_can_attach() and cpuset_attach() functions still
expect task migration is from one source cpuset to one destination
cpuset.
Fix that by tracking the set of source (old) cpusets in singly linked
lists with the setting of attach_in_progress flag associated with the
insertion into the list. The list will be iterated when necessary to
properly update the internal data.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both
the source and destination cpusets are decremented/incremented with
their values added to nr_deadline_tasks when the migration is successful.
The setting of the global attach_cpus_updated and attach_mems_updated
flags are also moved from cpuset_attach() to cpuset_can_attach() as the
correct source cpuset can no longer be determined in cpuset_attach()
and cpuset states will not be changed between cpuset_attach() and
cpuset_can_attach() with an earlier patch.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset-internal.h | 1 +
kernel/cgroup/cpuset.c | 66 ++++++++++++++++++++++++++++-----
2 files changed, 57 insertions(+), 10 deletions(-)
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index f7aaf01f7cd5..011993b1f756 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -149,6 +149,7 @@ struct cpuset {
* Tasks are being attached to this cpuset. Used to prevent
* zeroing cpus/mems_allowed between ->can_attach() and ->attach().
*/
+ struct llist_node attach_node;
int attach_in_progress;
/* partition root state */
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 511afb077e2d..c2d172873166 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -37,6 +37,7 @@
#include <linux/wait.h>
#include <linux/workqueue.h>
#include <linux/task_work.h>
+#include <linux/llist.h>
DEFINE_STATIC_KEY_FALSE(cpusets_pre_enable_key);
DEFINE_STATIC_KEY_FALSE(cpusets_enabled_key);
@@ -584,6 +585,7 @@ static struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs)
return NULL;
trial->dl_bw_cpu = -1;
+ init_llist_node(&trial->attach_node);
/* Setup cpumask pointer array */
cpumask_var_t *pmask[4] = {
@@ -2983,9 +2985,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)
* Protected by cpuset_mutex
*
* The attach_cpus_updated/attach_mems_updated flags are set in either
- * cpuset_attach() or cpuset_fork() and used in cpuset_attach_task().
+ * cpuset_can_attach() or cpuset_fork() and used in cpuset_attach_task().
*/
static struct cpuset *cpuset_attach_old_cs;
+static LLIST_HEAD(src_cs_head);
static bool attach_cpus_updated;
static bool attach_mems_updated;
@@ -3001,6 +3004,8 @@ static bool attach_mems_updated;
static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
bool *psetsched)
{
+ bool cpus_updated, mems_updated;
+
if (cpumask_empty(cs->effective_cpus) ||
(!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
return -ENOSPC;
@@ -3008,14 +3013,25 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
if (!oldcs)
return 0;
+ if (!llist_on_list(&oldcs->attach_node)) {
+ llist_add(&oldcs->attach_node, &src_cs_head);
+ oldcs->attach_in_progress++;
+ }
+
+ cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+ mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+ if (cpus_updated)
+ attach_cpus_updated = true;
+ if (mems_updated)
+ attach_mems_updated = true;
+
/*
* Skip rights over task setsched check in v2 when nothing changes,
* migration permission derives from hierarchy ownership in
* cgroup_procs_write_permission()).
*/
- *psetsched = !cpuset_v2() ||
- !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
- !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+ *psetsched = !cpuset_v2() || cpus_updated || mems_updated;
/*
* A v1 cpuset with tasks will have no CPU left only when CPU hotplug
@@ -3056,6 +3072,26 @@ static void reset_migrate_dl_data(struct cpuset *cs)
cs->dl_bw_cpu = -1;
}
+/*
+ * Clear and optionally apply (@cancel is false) the attach related data in the
+ * source cpusets.
+ */
+static void clear_attach_data(struct llist_head *head, bool cancel)
+{
+ struct cpuset *cs, *next;
+ struct llist_node *lnode = __llist_del_all(head);
+
+ llist_for_each_entry_safe(cs, next, lnode, attach_node) {
+ init_llist_node(&cs->attach_node);
+ if (cs->nr_migrate_dl_tasks) {
+ if (!cancel)
+ cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
+ cs->nr_migrate_dl_tasks = 0;
+ }
+ dec_attach_in_progress_locked(cs);
+ }
+}
+
/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
static int cpuset_can_attach(struct cgroup_taskset *tset)
{
@@ -3071,6 +3107,8 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
cs = css_cs(css);
mutex_lock(&cpuset_mutex);
+ attach_cpus_updated = false;
+ attach_mems_updated = false;
/* Check to see if task is allowed in the cpuset */
ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
@@ -3095,6 +3133,15 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
* selected as cpuset_attach_old_cs.
*/
cgroup_taskset_for_each(task, css, tset) {
+ struct cpuset *new_oldcs = task_cs(task);
+
+ if (new_oldcs != oldcs) {
+ oldcs = new_oldcs;
+ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
+ if (ret)
+ goto out_unlock;
+ }
+
ret = task_can_attach(task);
if (ret)
goto out_unlock;
@@ -3116,6 +3163,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
* contribute to sum_migrate_dl_bw.
*/
cs->nr_migrate_dl_tasks++;
+ oldcs->nr_migrate_dl_tasks--;
if (dl_task_needs_bw_move(task, cs->effective_cpus))
cs->sum_migrate_dl_bw += task->dl.dl_bw;
}
@@ -3126,9 +3174,9 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
out_unlock:
if (ret) {
reset_migrate_dl_data(cs);
+ clear_attach_data(&src_cs_head, true);
} else {
cs->attach_in_progress++;
- oldcs->attach_in_progress++;
}
mutex_unlock(&cpuset_mutex);
@@ -3145,6 +3193,7 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
mutex_lock(&cpuset_mutex);
dec_attach_in_progress_locked(cs);
+ clear_attach_data(&src_cs_head, true);
if (cs->dl_bw_cpu >= 0)
dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
@@ -3224,7 +3273,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct task_struct *task;
struct cgroup_subsys_state *css;
struct cpuset *cs;
- struct cpuset *oldcs = cpuset_attach_old_cs;
cgroup_taskset_first(tset, &css);
cs = css_cs(css);
@@ -3232,9 +3280,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
queue_task_work = false;
-
- attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
- attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
/*
@@ -3256,10 +3301,10 @@ static void cpuset_attach(struct cgroup_taskset *tset)
if (cs->nr_migrate_dl_tasks) {
cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
- oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks;
reset_migrate_dl_data(cs);
}
+ clear_attach_data(&src_cs_head, false);
dec_attach_in_progress_locked(cs);
mutex_unlock(&cpuset_mutex);
@@ -3777,6 +3822,7 @@ int __init cpuset_init(void)
cpumask_setall(top_cpuset.effective_xcpus);
cpumask_setall(top_cpuset.exclusive_cpus);
nodes_setall(top_cpuset.effective_mems);
+ init_llist_node(&top_cpuset.attach_node);
cpuset1_init(&top_cpuset);
--
2.54.0
^ permalink raw reply related
* [PATCH v7 7/9] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
The cpuset_attach_task() was introduced in commit 42a11bf5c543
("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly")
to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like
moving a task from one cpuset into another one. That commits didn't
move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader
into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new
task is its own group leader. So it is still not equivalent to moving
task between cpusets in this case. Make CLONE_INTO_CGROUP behaves
more close to cpuset_attach() by moving the mpol_rebind_mm() and
cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result,
the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 105 ++++++++++++++++++++++++-----------------
1 file changed, 62 insertions(+), 43 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 0375dae26d0b..511afb077e2d 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2981,8 +2981,13 @@ static int update_prstate(struct cpuset *cs, int new_prs)
/*
* cpuset_can_attach() and cpuset_attach() specific internal data
* Protected by cpuset_mutex
+ *
+ * The attach_cpus_updated/attach_mems_updated flags are set in either
+ * cpuset_attach() or cpuset_fork() and used in cpuset_attach_task().
*/
static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
/*
* Check to see if a cpuset can accept a new task
@@ -3157,9 +3162,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
*/
static cpumask_var_t cpus_attach;
static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
+ struct mm_struct *mm;
+
lockdep_assert_cpuset_lock_held();
if (cs != &top_cpuset)
@@ -3173,28 +3181,60 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
*/
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ if (cpuset_v2() && !attach_mems_updated)
+ return;
+
cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
cpuset1_update_task_spread_flags(cs, task);
+
+ if ((task != task->group_leader) ||
+ (!is_memory_migrate(cs) && !attach_mems_updated))
+ return;
+
+ /*
+ * Change mm for threadgroup leader. This is expensive and may
+ * sleep and should be moved outside migration path proper.
+ */
+ mm = get_task_mm(task);
+ if (mm) {
+ struct cpuset *oldcs = cpuset_attach_old_cs;
+
+ mpol_rebind_mm(mm, &cs->effective_mems);
+
+ /*
+ * old_mems_allowed is the same with mems_allowed
+ * here, except if this task is being moved
+ * automatically due to hotplug. In that case
+ * @mems_allowed has been updated and is empty, so
+ * @old_mems_allowed is the right nodesets that we
+ * migrate mm from.
+ */
+ if (is_memory_migrate(cs)) {
+ cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+ &cpuset_attach_nodemask_to);
+ queue_task_work = true;
+ } else {
+ mmput(mm);
+ }
+ }
}
static void cpuset_attach(struct cgroup_taskset *tset)
{
struct task_struct *task;
- struct task_struct *leader;
struct cgroup_subsys_state *css;
struct cpuset *cs;
struct cpuset *oldcs = cpuset_attach_old_cs;
- bool cpus_updated, mems_updated;
- bool queue_task_work = false;
cgroup_taskset_first(tset, &css);
cs = css_cs(css);
lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */
mutex_lock(&cpuset_mutex);
- cpus_updated = !cpumask_equal(cs->effective_cpus,
- oldcs->effective_cpus);
- mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+ queue_task_work = false;
+
+ attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+ attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
/*
@@ -3203,44 +3243,12 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* and mems. In that case, we can optimize out by skipping the task
* iteration and update.
*/
- if (cpuset_v2() && !cpus_updated && !mems_updated)
+ if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated)
goto out;
cgroup_taskset_for_each(task, css, tset)
cpuset_attach_task(cs, task);
- /*
- * Change mm for all threadgroup leaders. This is expensive and may
- * sleep and should be moved outside migration path proper. Skip it
- * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
- * not set.
- */
- if (!is_memory_migrate(cs) && !mems_updated)
- goto out;
-
- cgroup_taskset_for_each_leader(leader, css, tset) {
- struct mm_struct *mm = get_task_mm(leader);
-
- if (mm) {
- mpol_rebind_mm(mm, &cs->effective_mems);
-
- /*
- * old_mems_allowed is the same with mems_allowed
- * here, except if this task is being moved
- * automatically due to hotplug. In that case
- * @mems_allowed has been updated and is empty, so
- * @old_mems_allowed is the right nodesets that we
- * migrate mm from.
- */
- if (is_memory_migrate(cs)) {
- cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
- &cpuset_attach_nodemask_to);
- queue_task_work = true;
- } else
- mmput(mm);
- }
- }
-
out:
if (queue_task_work)
schedule_flush_migrate_mm();
@@ -3689,15 +3697,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)
*/
static void cpuset_fork(struct task_struct *task)
{
- struct cpuset *cs;
- bool same_cs;
+ struct cpuset *cs, *oldcs;
rcu_read_lock();
cs = task_cs(task);
- same_cs = (cs == task_cs(current));
+ oldcs = task_cs(current);
rcu_read_unlock();
- if (same_cs) {
+ if (cs == oldcs) {
if (cs == &top_cpuset)
return;
@@ -3709,7 +3716,19 @@ static void cpuset_fork(struct task_struct *task)
/* CLONE_INTO_CGROUP */
mutex_lock(&cpuset_mutex);
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ cs->old_mems_allowed = cpuset_attach_nodemask_to;
+
+ /*
+ * Assume CPUs and memory nodes are updated
+ * A CLONE_INTO_CGROUP operation should have taken the cgroup mutex
+ * and so there shouldn't be a competing cpuset_attach() operation.
+ */
+ attach_cpus_updated = attach_mems_updated = true;
+ queue_task_work = false;
+ cpuset_attach_old_cs = oldcs;
cpuset_attach_task(cs, task);
+ if (queue_task_work)
+ schedule_flush_migrate_mm();
dec_attach_in_progress_locked(cs);
mutex_unlock(&cpuset_mutex);
--
2.54.0
^ permalink raw reply related
* [PATCH v7 6/9] cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
There are two possible ways that migration of tasks from multiple source
cpusets to a target cpuset can happen. Either a multithread application
with threads in different cpusets is wholely moved to a new cpuset
or disabling of v2 cpuset controller will move all the tasks in child
cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that really
matters. So cpuset_attach_old_cs should track the oldcs of the thread
leader. In the latter case, effective_mems of child cpusets must always
be a subset of the parent. So no real page migration will be necessary
no matter which child cpuset is selected as cpuset_attach_old_cs.
IOW, cpuset_attach_old_cs should be updated to match the latest task
group leader in cpuset_can_attach(), but fall back to that of the first
task if there is no group leader in the taskset.
Suggested-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index b7b5072f2fdd..0375dae26d0b 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2978,6 +2978,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)
return 0;
}
+/*
+ * cpuset_can_attach() and cpuset_attach() specific internal data
+ * Protected by cpuset_mutex
+ */
static struct cpuset *cpuset_attach_old_cs;
/*
@@ -3068,11 +3072,32 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
if (ret)
goto out_unlock;
+ /*
+ * The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get
+ * the old_mems_allowed value. There are two ways that many-to-one
+ * cpuset migration can happen:
+ * 1) A multithread application with threads in different cpusets is
+ * wholely migrated to a new cpuset.
+ * 2) Disabling v2 cpuset controller will move all the tasks in child
+ * cpusets to the parent cpuset.
+ *
+ * In the former case, it is the mm setting of the group leader that
+ * really matters. So cpuset_attach_old_cs should track the oldcs of the
+ * group leader. It falls back to the oldcs of the first task if there
+ * is no group leader in the taskset. In the latter case, effective_mems
+ * of child cpusets must always be a subset of the parent. So no real
+ * page migration will be necessary no matter which child cpuset is
+ * selected as cpuset_attach_old_cs.
+ */
cgroup_taskset_for_each(task, css, tset) {
ret = task_can_attach(task);
if (ret)
goto out_unlock;
+ /* Update cpuset_attach_old_cs to the latest group leader */
+ if (task == task->group_leader)
+ cpuset_attach_old_cs = task_cs(task);
+
if (setsched_check) {
ret = security_task_setscheduler(task);
if (ret)
--
2.54.0
^ permalink raw reply related
* [PATCH v7 5/9] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
Expand the scope of cpuset_can_attach_check() by including the setting
of setsched flag inside cpuset_can_attach_check() with the new @oldcs
and @psetsched argument. As cpuset_can_attach_check() is also called
from cpuset_can_fork(), set the new arguments to NULL from that caller.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 52 ++++++++++++++++++++++++------------------
1 file changed, 30 insertions(+), 22 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 2ffc66baedf3..b7b5072f2fdd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2985,12 +2985,39 @@ static struct cpuset *cpuset_attach_old_cs;
* For v1, cpus_allowed and mems_allowed can't be empty.
* For v2, effective_cpus can't be empty.
* Note that in v1, effective_cpus = cpus_allowed.
+ *
+ * Also set the boolean flag passed in by @psetsched depending on if
+ * security_task_setscheduler() call is needed and @oldcs is not NULL.
*/
-static int cpuset_can_attach_check(struct cpuset *cs)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
+ bool *psetsched)
{
if (cpumask_empty(cs->effective_cpus) ||
(!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
return -ENOSPC;
+
+ if (!oldcs)
+ return 0;
+
+ /*
+ * Skip rights over task setsched check in v2 when nothing changes,
+ * migration permission derives from hierarchy ownership in
+ * cgroup_procs_write_permission()).
+ */
+ *psetsched = !cpuset_v2() ||
+ !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
+ !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+ /*
+ * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
+ * brings the last online CPU offline as users are not allowed to empty
+ * cpuset.cpus when there are active tasks inside. When that happens,
+ * we should allow tasks to migrate out without security check to make
+ * sure they will be able to run after migration.
+ */
+ if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
+ *psetsched = false;
+
return 0;
}
@@ -3037,29 +3064,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
mutex_lock(&cpuset_mutex);
/* Check to see if task is allowed in the cpuset */
- ret = cpuset_can_attach_check(cs);
+ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
if (ret)
goto out_unlock;
- /*
- * Skip rights over task setsched check in v2 when nothing changes,
- * migration permission derives from hierarchy ownership in
- * cgroup_procs_write_permission()).
- */
- setsched_check = !cpuset_v2() ||
- !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
- !nodes_equal(cs->effective_mems, oldcs->effective_mems);
-
- /*
- * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
- * brings the last online CPU offline as users are not allowed to empty
- * cpuset.cpus when there are active tasks inside. When that happens,
- * we should allow tasks to migrate out without security check to make
- * sure they will be able to run after migration.
- */
- if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
- setsched_check = false;
-
cgroup_taskset_for_each(task, css, tset) {
ret = task_can_attach(task);
if (ret)
@@ -3616,7 +3624,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
mutex_lock(&cpuset_mutex);
/* Check to see if task is allowed in the cpuset */
- ret = cpuset_can_attach_check(cs);
+ ret = cpuset_can_attach_check(cs, NULL, NULL);
if (ret)
goto out_unlock;
--
2.54.0
^ permalink raw reply related
* [PATCH v7 4/9] cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
Extract the DL bandwidth allocation code in cpuset_attach() to a new
cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 47 ++++++++++++++++++++++++------------------
1 file changed, 27 insertions(+), 20 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 65d095dcada1..2ffc66baedf3 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2994,6 +2994,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)
return 0;
}
+static int cpuset_reserve_dl_bw(struct cpuset *cs)
+{
+ int cpu, ret;
+
+ if (!cs->sum_migrate_dl_bw)
+ return 0;
+
+ cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+ if (unlikely(cpu >= nr_cpu_ids))
+ return -EINVAL;
+
+ ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ if (ret)
+ return ret;
+
+ cs->dl_bw_cpu = cpu;
+ return 0;
+}
+
static void reset_migrate_dl_data(struct cpuset *cs)
{
cs->nr_migrate_dl_tasks = 0;
@@ -3008,7 +3027,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
struct cpuset *cs, *oldcs;
struct task_struct *task;
bool setsched_check;
- int cpu, ret;
+ int ret;
/* used later by cpuset_attach() */
cpuset_attach_old_cs = task_cs(cgroup_taskset_first(tset, &css));
@@ -3064,28 +3083,16 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
}
}
- if (!cs->sum_migrate_dl_bw)
- goto out_success;
-
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids)) {
- ret = -EINVAL;
- goto out_unlock;
- }
-
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
- if (ret)
- goto out_unlock;
-
- cs->dl_bw_cpu = cpu;
-
-out_success:
- cs->attach_in_progress++;
- oldcs->attach_in_progress++;
+ ret = cpuset_reserve_dl_bw(cs);
out_unlock:
- if (ret)
+ if (ret) {
reset_migrate_dl_data(cs);
+ } else {
+ cs->attach_in_progress++;
+ oldcs->attach_in_progress++;
+ }
+
mutex_unlock(&cpuset_mutex);
return ret;
}
--
2.54.0
^ permalink raw reply related
* [PATCH v7 3/9] cgroup/cpuset: Prevent race between task attach and cpuset state change
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
Commit e44193d39e8d ("cpuset: let hotplug propagation work wait for
task attaching") was introduced to let hotplug operation to wait
until the completion of task attaching operation. However, it is
still possible that the states of the source or destination cpuset
can be changed between the cpuset_can_attach() call and the subsequent
cpuset_attach()/cpuset_cacnel_attach() call.
As a result, data gathered during cpuset_can_attach() cannot be reliably
used in the subsequent cpuset_attach()/cpuset_cacnel_attach()
call at all. Make the task attach operation more robust
and allow the sharing of data between cpuset_can_attach() and
cpuset_attach()/cpuset_cacnel_attach() by making cpuset_write_resmask()
and cpuset_partition_write() wait for the completion of task attach
and set the attach_in_progress flag in the source cpuset as well.
The comments about validate_change() are no longer valid as it won't
be called at all if an attach operation is in progress. So the comments
can be removed.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 28 ++++++++++++++++++++--------
1 file changed, 20 insertions(+), 8 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index a1c8890d3519..65d095dcada1 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3080,11 +3080,8 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
cs->dl_bw_cpu = cpu;
out_success:
- /*
- * Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
cs->attach_in_progress++;
+ oldcs->attach_in_progress++;
out_unlock:
if (ret)
@@ -3235,10 +3232,19 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
return -EACCES;
buf = strstrip(buf);
+retry:
+ wait_event(cpuset_attach_wq, cs->attach_in_progress == 0);
+
cpuset_full_lock();
if (!is_cpuset_online(cs))
goto out_unlock;
+ /* Don't race with task attach */
+ if (cs->attach_in_progress) {
+ cpuset_full_unlock();
+ goto retry;
+ }
+
trialcs = dup_or_alloc_cpuset(cs);
if (!trialcs) {
retval = -ENOMEM;
@@ -3366,7 +3372,17 @@ static ssize_t cpuset_partition_write(struct kernfs_open_file *of, char *buf,
else
return -EINVAL;
+retry:
+ wait_event(cpuset_attach_wq, cs->attach_in_progress == 0);
+
cpuset_full_lock();
+
+ /* Don't race with task attach */
+ if (cs->attach_in_progress) {
+ cpuset_full_unlock();
+ goto retry;
+ }
+
if (is_cpuset_online(cs))
retval = update_prstate(cs, val);
cpuset_update_sd_hk_unlock();
@@ -3605,10 +3621,6 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
if (ret)
goto out_unlock;
- /*
- * Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
cs->attach_in_progress++;
out_unlock:
mutex_unlock(&cpuset_mutex);
--
2.54.0
^ permalink raw reply related
* [PATCH v7 2/9] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
Whenever memory node mask is changed, there are 4 places where the node
mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to
the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and
cpuset_attach(). However there are inconsistencies in what node masks
are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is
hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or
an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in
node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE].
However, node in node_states[N_ONLINE] may not have memory. So
node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should mostly be useful for v1
where mems_allowed is the same as effective_mems. With v2, the memory
nodes in effective_mems should be a subset of node_states[N_MEMORY]
except when a memory hot-unplug operation is in progress and a memory
node is removed from node_states[N_MEMORY] but not yet reflected in
the effective_mems's as cpuset_handle_hotplug() has not been called
from cpuset_track_online_nodes().
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems most of the time. For
v1, mpol_rebind_mm() uses mems_allowed which may differ from what
guarantee_online_mems() returns, but it conforms to what the cpuset v1
documentation says with respect to setting memory policy.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 30 ++++++++++++++++++------------
1 file changed, 18 insertions(+), 12 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index b21c31650583..a1c8890d3519 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -489,7 +489,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,
* Return in *pmask the portion of a cpusets's mems_allowed that
* are online, with memory. If none are online with memory, walk
* up the cpuset hierarchy until we find one that does have some
- * online mems. The top cpuset always has some mems online.
+ * online mems. The top cpuset always has some mems online. With v2,
+ * effective_mems should always contain online memory nodes except
+ * during the transition period where a memory node hotunplug operation
+ * is in progress.
*
* One way or another, we guarantee to return some non-empty subset
* of node_states[N_MEMORY].
@@ -2619,6 +2622,14 @@ static void *cpuset_being_rebound;
* Iterate through each task of @cs updating its mems_allowed to the
* effective cpuset's. As this function is called with cpuset_mutex held,
* cpuset membership stays stable.
+ *
+ * - cpuset_change_task_nodemask(): guarantee_online_mems()
+ * - mpol_rebind_mm(): effective_mems
+ * - cpuset_migrate_mm(): guarantee_online_mems()
+ * - old_mems_allowed: guarantee_online_mems()
+ *
+ * For v2, guarantee_online_mems() should return a node mask that is the same
+ * as the effective_mems of current cpuset.
*/
void cpuset_update_tasks_nodemask(struct cpuset *cs)
{
@@ -2627,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
struct task_struct *task;
cpuset_being_rebound = cs; /* causes mpol_dup() rebind */
-
guarantee_online_mems(cs, &newmems);
/*
@@ -3148,19 +3158,16 @@ static void cpuset_attach(struct cgroup_taskset *tset)
cpus_updated = !cpumask_equal(cs->effective_cpus,
oldcs->effective_cpus);
mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+ guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
/*
* In the default hierarchy, enabling cpuset in the child cgroups
- * will trigger a number of cpuset_attach() calls with no change
- * in effective cpus and mems. In that case, we can optimize out
- * by skipping the task iteration and update.
+ * will trigger a cpuset_attach() call with no change in effective cpus
+ * and mems. In that case, we can optimize out by skipping the task
+ * iteration and update.
*/
- if (cpuset_v2() && !cpus_updated && !mems_updated) {
- cpuset_attach_nodemask_to = cs->effective_mems;
+ if (cpuset_v2() && !cpus_updated && !mems_updated)
goto out;
- }
-
- guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
cgroup_taskset_for_each(task, css, tset)
cpuset_attach_task(cs, task);
@@ -3171,7 +3178,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
* if there is no change in effective_mems and CS_MEMORY_MIGRATE is
* not set.
*/
- cpuset_attach_nodemask_to = cs->effective_mems;
if (!is_memory_migrate(cs) && !mems_updated)
goto out;
@@ -3179,7 +3185,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
struct mm_struct *mm = get_task_mm(leader);
if (mm) {
- mpol_rebind_mm(mm, &cpuset_attach_nodemask_to);
+ mpol_rebind_mm(mm, &cs->effective_mems);
/*
* old_mems_allowed is the same with mems_allowed
--
2.54.0
^ permalink raw reply related
* [PATCH v7 1/9] cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long, stable
In-Reply-To: <20260621032816.1806773-1-longman@redhat.com>
From: Farhad Alemi <farhad.alemi@berkeley.edu>
Creating a child cpuset where cpuset.mems is never set leads to a div/0
when a VMA mempolicy with MPOL_F_RELATIVE_NODES rebinds in response to a
CPU hotplug event.
Reproduction steps:
1) Create a cgroup w/ cpuset controls (do not set cpuset.mems)
2) Move the task into the child cpuset
3) Create a VMA mempolicy for that task with MPOL_F_RELATIVE_NODES
4) unplug and hotplug a cpu
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu1/online
5) mempolicy rebind does a div/0 in mpol_relative_nodemask on the
call to __nodes_fold()
The cpuset code passes (cs->mems_allowed) which is not guaranteed to have
nodes to the rebind routine. Use cs->effective_mems instead, which is
guaranteed to have a non-empty nodemask.
Closes: https://lore.kernel.org/linux-mm/CA+0ovCgxbZkXa+OU8w3s84R3KNPNxxRfmsNR-udh+afQBbGNmw@mail.gmail.com/
Link: https://lore.kernel.org/all/CA+0ovCiEz6SP_sn3kN4Tb+_oC=eHMXy_Ffj=usV3wREdQrUtww@mail.gmail.com/
Fixes: ae1c802382f7 ("cpuset: apply cs->effective_{cpus,mems}")
Suggested-by: Gregory Price <gourry@gourry.net>
Suggested-by: Waiman Long <longman@redhat.com>
Signed-off-by: Farhad Alemi <farhad.alemi@berkeley.edu>
Acked-by: Waiman Long <longman@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/cgroup/cpuset.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 591e3aa487fc..b21c31650583 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2653,7 +2653,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
migrate = is_memory_migrate(cs);
- mpol_rebind_mm(mm, &cs->mems_allowed);
+ mpol_rebind_mm(mm, &cs->effective_mems);
if (migrate)
cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
else
--
2.54.0
^ permalink raw reply related
* [PATCH v7 0/9] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Waiman Long @ 2026-06-21 3:28 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Li Zefan, Farhad Alemi, Andrew Morton
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Gregory Price,
David Hildenbrand, Waiman Long
v7:
- Include the fix patch from Farhad Alemi to fix a div/0 crash that
was part of the old patch 1.
- Integrated v6 patch 7 into earlier patches.
- Add a new "cgroup/cpuset: Prevent race between task attach and
cpuset state change" patch to ensure that there will be no cpuset
state change between cpuset_can_attach() and cpuset_attach() or
cpuset_cancel_attach().
- Break v6 patch 6 into 2 separate patches for supporting multiple
source cpusets and multiple destination cpusets respectively and
further simplify and streamline the code.
v6:
- Make guarantee_online_mems() to only return cs->effective_mems with v2
in patch 1.
- Remove obsolete commit description text from patch 3.
- Add Reviewed-by tags.
- In patch 6, add WARN_ON_ONCE() test in cpuset_can_attach() to
confirm that cs != oldcs.
v5:
- Remove the WARN_ON() call as it can be triggered in a corner case.
- Instead of passing an attach_cpus_updated and attach_mems_updated
flags from cpuset_can_attach() to cpuset_attach(), re-evaluate the
flags at the beginning of cpuset_attach() based on data in the source &
destination cpusets in the singly linked lists to eliminate the
Time-of-Check to Time-of-Use (TOCTOU) race condition & simplify the
code changes.
- Add back the dropped optimization in patch 5.
Sashiko AI review of another cpuset patch had found that cpuset_attach()
and cpuset_can_attach() can be passed a cgroup_taskset with tasks
migrating from one source cpuset to multiple destination cpusets and
vice versa. Further testing of the cpuset code indicates that this is
indeed the case when the v2 cpuset controller is enabled or disabled.
Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that
there will be one source and one destinaton cpuset which may result in
inocrrect behavior.
This patch series is created to fix this issue.
Patch 1 is a fix that fix a cgroup v2 div/0 crash due to bug in
cpuset_update_tasks_nodemask().
Patch 2 is to fix an inconsistency in the way node mask update is being
handled in cpuset_update_tasks_nodemask() and cpuset_attach() so that
they match each other.
Patch 3 makes any cpuset state change to wait for the completion of the
pending cpuset attach operation, if any.
Patches 4 and 5 are just preparatory patches to make the remaining
patches easier to review.
Patch 6 makes cpuset_attach_old_cs to track group leader for use by
cpuset_migrate_mm().
Patch 7 moves mpol_rebind_mm() and cpuset_migrate_mm() inside
cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works
more like moving task from one cpuset to another one, while also make
supporting multiple source and destination cpusets easier.
Patch 8 makes the necessary changes to enable the support of multiple
source cpusets by keeping all the source cpusets found during task
iterations in a singly linked lists.
Patch 9 enables the support of multiple destination cpusets during the
the enabling of cpuset v2 controller.
Farhad Alemi (1):
cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed
Waiman Long (8):
cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask() and cpuset_attach()
cgroup/cpuset: Prevent race between task attach and cpuset state
change
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders
cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source cpusets for cpuset_*attach()
cgroup/cpuset: Support multiple destination cpusets for
cpuset_*attach()
kernel/cgroup/cpuset-internal.h | 2 +
kernel/cgroup/cpuset.c | 400 ++++++++++++++++++++++----------
2 files changed, 277 insertions(+), 125 deletions(-)
--
2.54.0
^ permalink raw reply
* Re: [PATCH v2] cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed
From: Waiman Long @ 2026-06-21 3:24 UTC (permalink / raw)
To: David Hildenbrand (Arm), Gregory Price
Cc: Farhad Alemi, Andrew Morton, Farhad Alemi, Yury Norov,
Joshua Hahn, Zi Yan, Matthew Brost, Rakie Kim, Byungchul Park,
Ying Huang, Alistair Popple, Rasmus Villemoes, linux-mm,
linux-kernel, cgroups, stable
In-Reply-To: <38578aea-61c3-4328-aee9-8e7421672647@kernel.org>
On 6/18/26 4:41 AM, David Hildenbrand (Arm) wrote:
> On 6/16/26 17:23, Waiman Long wrote:
>> On 6/16/26 2:59 AM, David Hildenbrand (Arm) wrote:
>>> On 6/16/26 05:43, Waiman Long wrote:
>>>> BTW, I still prefer the v2 patch. If it is decided we should use the
>>>> guarantee_online_mems() value instead, it will have to be a separate patch with
>>>> changes in the relevant documentation like Documentation/admin-guide/cgroup-v1/
>>>> cpuset.rst.
>>> newmems is "obviously" correct, so I really don't see why we should add
>>> something that needs half a page of text to explain why it is fine -- if newmems
>>> just does the trick?
>>>
>>> Please enlighten me.
>> Yes, taking newmems is a reasonable choice and there are pros and cons with each
>> options. My focus is more on not changing how v1 cpuset behaves as it is well
>> defined in the v1 cpusets.rst file:
>>
>> Requests by a task, using the sched_setaffinity(2) system call to
>> include CPUs in its CPU affinity mask, and using the mbind(2) and
>> set_mempolicy(2) system calls to include Memory Nodes in its memory
>> policy, are both filtered through that task's cpuset, filtering out any
>> CPUs or Memory Nodes not in that cpuset. The scheduler will not
>> schedule a task on a CPU that is not allowed in its cpus_allowed
>> vector, and the kernel page allocator will not allocate a page on a
>> node that is not allowed in the requesting task's mems_allowed vector.
>>
>> v2, OTOH, is more vague as to what setting cpuset.mems will mean and we
>> generally follow what v1 is doing, but we have more leeway of what we can do.
>>
>> Using newmems will make the above text not totally correct. At least the offline
>> memory nodes will be filtered out which will not be utilized by the task when
>> the offline node becomes online. That is why I am saying that we will have to
>> correct the documentation if we want to make this change.
> So IIUC:
>
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 1335e437098e..cdfc615f35a5 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -2645,7 +2645,13 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
>
> migrate = is_memory_migrate(cs);
>
> - mpol_rebind_mm(mm, &cs->mems_allowed);
> + /*
> + * For v1 we can have empty effective_mems, but we cannot
> + * attach any tasks (see cpuset_can_attach_check()). For v2,
> + * it's guaranteed to not be empty.
> + */
> + VM_WARN_ON_ONCE(nodes_empty(cs->effective_mems));
> + mpol_rebind_mm(mm, &cs->effective_mems);
> if (migrate)
> cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
> else
That is true, but I don't think we need a VM_WARN_ON_ONCE() here.
Cheers,
Longman
>
^ permalink raw reply
* Re: [PATCH V3] blk-cgroup: defer blkcg css_put until blkg is unlinked from queue
From: yu kuai @ 2026-06-20 18:29 UTC (permalink / raw)
To: Zizhi Wo, axboe, tj, josef, linux-block
Cc: cgroups, yangerkun, chengzhihao1, houtao1, yukuai
In-Reply-To: <20260616011746.2451461-1-wozizhi@huaweicloud.com>
在 2026/6/16 9:17, Zizhi Wo 写道:
> From: Zizhi Wo<wozizhi@huawei.com>
>
> [BUG]
> Our fuzz testing triggered a blkcg use-after-free issue:
>
> BUG: KASAN: slab-use-after-free in _raw_spin_lock+0x75/0xe0
> Call Trace:
> ...
> blkcg_deactivate_policy+0x244/0x4d0
> ioc_rqos_exit+0x44/0xe0
> rq_qos_exit+0xba/0x120
> __del_gendisk+0x50b/0x800
> del_gendisk+0xff/0x190
> ...
>
> [CAUSE]
> process1 process2
> cgroup_rmdir
> ...
> css_killed_work_fn
> offline_css
> ...
> blkcg_destroy_blkgs
> ...
> __blkg_release
> css_put(&blkg->blkcg->css)
> blkg_free
> INIT_WORK(xxx, blkg_free_workfn)
> schedule_work
> css_put
> ...
> blkcg_css_free
> kfree(blkcg)--------blkcg has been freed!!!
> ====================================schedule_work
> blkg_free_workfn
> __del_gendisk
> rq_qos_exit
> ioc_rqos_exit
> blkcg_deactivate_policy
> mutex_lock(&q->blkcg_mutex)
> spin_lock_irq(&q->queue_lock)
> list_for_each_entry(blkg, xxx)
> blkcg = blkg->blkcg
> spin_lock(&blkcg->lock)-------UAF!!!
> mutex_lock(&q->blkcg_mutex)
> spin_lock_irq(&q->queue_lock)
> /* Only then is the blkg removed from the list */
> list_del_init(&blkg->q_node)
>
> As a result, a blkg can still be reachable through q->blkg_list while
> its ->blkcg has already been freed.
>
> [Fix]
> Fix this by deferring the blkcg css_put() until after the blkg has been
> unlinked from q->blkg_list in blkg_free_workfn(). This ensures that the
> blkcg outlives every blkg still reachable through q->blkg_list, so any
> iterator holding q->queue_lock is guaranteed to observe a valid
> blkg->blkcg.
>
> While at it, move css_tryget_online() from blkg_create() into blkg_alloc()
> so that the css reference is owned by the alloc/free pair rather than
> straddling layers:
> blkg_alloc() <-> blkg_free()
> blkg_create() <-> blkg_destroy()
>
> Fixes: f1c006f1c685 ("blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()")
> Suggested-by: Hou Tao<houtao1@huawei.com>
> Signed-off-by: Zizhi Wo<wozizhi@huawei.com>
> Reviewed-by: Yu Kuai<yukuai@fygo.io>
> ---
> v3:
> - move css_put() after mutex_unlock() in blkg_free_workfn().
>
> v2:
> - Move css_tryget_online() from blkg_create() into blkg_alloc() so the
> css reference follows the blkg's own lifetime, making the put in
> blkg_free_workfn() symmetric with the get in blkg_alloc().
>
> v1:https://lore.kernel.org/all/20260518010932.633707-1-wozizhi@huaweicloud.com/
> block/blk-cgroup.c | 24 ++++++++++++------------
> 1 file changed, 12 insertions(+), 12 deletions(-)
Reviewed-by: Yu Kuai <yukuai@fygo.io>
--
Thanks,
Kuai
^ permalink raw reply
* [PATCH v9 6/6] selftests/cgroup: add a swap tier routing test
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260620181635.299364-1-youngjun.park@lge.com>
This commit adds a test program for the per-cgroup swap tier control
memory.swap.tiers.max. It checks the default mask, toggling a tier,
rejection of invalid input, and that recreating a tier resets the mask.
It also checks that a cgroup's pages swap only to an allowed tier,
including across the parent and child hierarchy. The routing check uses
two zram devices placed in different tiers.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
tools/testing/selftests/cgroup/.gitignore | 1 +
tools/testing/selftests/cgroup/Makefile | 2 +
tools/testing/selftests/cgroup/config | 2 +
.../selftests/cgroup/test_swap_tiers.c | 500 ++++++++++++++++++
4 files changed, 505 insertions(+)
create mode 100644 tools/testing/selftests/cgroup/test_swap_tiers.c
diff --git a/tools/testing/selftests/cgroup/.gitignore b/tools/testing/selftests/cgroup/.gitignore
index 952e4448bf07..77b8e6c3e592 100644
--- a/tools/testing/selftests/cgroup/.gitignore
+++ b/tools/testing/selftests/cgroup/.gitignore
@@ -8,5 +8,6 @@ test_kill
test_kmem
test_memcontrol
test_pids
+test_swap_tiers
test_zswap
wait_inotify
diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile
index e01584c2189a..a98e3c414cd5 100644
--- a/tools/testing/selftests/cgroup/Makefile
+++ b/tools/testing/selftests/cgroup/Makefile
@@ -16,6 +16,7 @@ TEST_GEN_PROGS += test_kill
TEST_GEN_PROGS += test_kmem
TEST_GEN_PROGS += test_memcontrol
TEST_GEN_PROGS += test_pids
+TEST_GEN_PROGS += test_swap_tiers
TEST_GEN_PROGS += test_zswap
LOCAL_HDRS += $(selfdir)/clone3/clone3_selftests.h $(selfdir)/pidfd/pidfd.h
@@ -32,4 +33,5 @@ $(OUTPUT)/test_kill: $(LIBCGROUP_O)
$(OUTPUT)/test_kmem: $(LIBCGROUP_O)
$(OUTPUT)/test_memcontrol: $(LIBCGROUP_O)
$(OUTPUT)/test_pids: $(LIBCGROUP_O)
+$(OUTPUT)/test_swap_tiers: $(LIBCGROUP_O)
$(OUTPUT)/test_zswap: $(LIBCGROUP_O)
diff --git a/tools/testing/selftests/cgroup/config b/tools/testing/selftests/cgroup/config
index 39f979690dd3..6086bb5bba97 100644
--- a/tools/testing/selftests/cgroup/config
+++ b/tools/testing/selftests/cgroup/config
@@ -4,3 +4,5 @@ CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
CONFIG_MEMCG=y
CONFIG_PAGE_COUNTER=y
+CONFIG_SWAP=y
+CONFIG_ZRAM=y
diff --git a/tools/testing/selftests/cgroup/test_swap_tiers.c b/tools/testing/selftests/cgroup/test_swap_tiers.c
new file mode 100644
index 000000000000..24420c1ef398
--- /dev/null
+++ b/tools/testing/selftests/cgroup/test_swap_tiers.c
@@ -0,0 +1,500 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/limits.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/swap.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "kselftest.h"
+#include "cgroup_util.h"
+
+#ifndef MADV_PAGEOUT
+#define MADV_PAGEOUT 21
+#endif
+
+#define TIERS_PATH "/sys/kernel/mm/swap/tiers"
+#define TIERS_MAX "memory.swap.tiers.max"
+
+static int tiers_write(const char *cmd)
+{
+ int fd, ret = 0;
+
+ fd = open(TIERS_PATH, O_WRONLY);
+ if (fd < 0)
+ return -errno;
+ if (write(fd, cmd, strlen(cmd)) < 0)
+ ret = -errno;
+ close(fd);
+ return ret;
+}
+
+static int tier_count(void)
+{
+ char buf[4096], *line, *save;
+ int fd, count = 0;
+ ssize_t n;
+
+ fd = open(TIERS_PATH, O_RDONLY);
+ if (fd < 0)
+ return -1;
+ n = read(fd, buf, sizeof(buf) - 1);
+ close(fd);
+ if (n < 0)
+ return -1;
+ buf[n] = '\0';
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ char name[64];
+ int idx, s, e;
+
+ if (sscanf(line, "%63s %d %d %d", name, &idx, &s, &e) == 4)
+ count++;
+ }
+ return count;
+}
+
+static long swap_used_kb(const char *dev)
+{
+ char line[256];
+ long used = -1;
+ FILE *f;
+
+ f = fopen("/proc/swaps", "r");
+ if (!f)
+ return -1;
+ while (fgets(line, sizeof(line), f)) {
+ char name[128], type[64];
+ long size, u, prio;
+
+ if (sscanf(line, "%127s %63s %ld %ld %ld",
+ name, type, &size, &u, &prio) >= 4 &&
+ !strcmp(name, dev)) {
+ used = u;
+ break;
+ }
+ }
+ fclose(f);
+ return used;
+}
+
+static int swap_active_count(void)
+{
+ char line[256];
+ int n = 0;
+ FILE *f;
+
+ f = fopen("/proc/swaps", "r");
+ if (!f)
+ return -1;
+ if (fgets(line, sizeof(line), f)) /* header */
+ while (fgets(line, sizeof(line), f))
+ n++;
+ fclose(f);
+ return n;
+}
+
+static int zram_add(long size)
+{
+ char path[128], val[64];
+ ssize_t n;
+ int idx, fd;
+
+ fd = open("/sys/class/zram-control/hot_add", O_RDONLY);
+ if (fd < 0)
+ return -1;
+ n = read(fd, val, sizeof(val) - 1);
+ close(fd);
+ if (n <= 0)
+ return -1;
+ val[n] = '\0';
+ idx = atoi(val);
+
+ snprintf(path, sizeof(path), "/sys/block/zram%d/disksize", idx);
+ fd = open(path, O_WRONLY);
+ if (fd < 0)
+ return -1;
+ snprintf(val, sizeof(val), "%ld", size);
+ n = write(fd, val, strlen(val));
+ close(fd);
+ return n < 0 ? -1 : idx;
+}
+
+static void zram_remove(int idx)
+{
+ char val[16];
+ int fd;
+
+ fd = open("/sys/class/zram-control/hot_remove", O_WRONLY);
+ if (fd < 0)
+ return;
+ snprintf(val, sizeof(val), "%d", idx);
+ if (write(fd, val, strlen(val)) < 0)
+ ; /* ignore: best-effort cleanup */
+ close(fd);
+}
+
+static int swap_setup(const char *dev, int prio)
+{
+ char cmd[128];
+
+ snprintf(cmd, sizeof(cmd), "mkswap %s >/dev/null 2>&1", dev);
+ if (system(cmd))
+ return -1;
+ return swapon(dev, SWAP_FLAG_PREFER | (prio & SWAP_FLAG_PRIO_MASK));
+}
+
+/* A new cgroup may use every tier ("max"). */
+static int test_default(const char *root)
+{
+ char *cg = cg_name(root, "swaptier_default");
+ int ret = KSFT_FAIL;
+
+ if (!cg || cg_create(cg))
+ goto out;
+ if (!cg_read_strstr(cg, TIERS_MAX, "fast max") &&
+ !cg_read_strstr(cg, TIERS_MAX, "slow max"))
+ ret = KSFT_PASS;
+out:
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return ret;
+}
+
+/* A tier can be disabled and re-enabled, and the change reads back. */
+static int test_toggle(const char *root)
+{
+ char *cg = cg_name(root, "swaptier_toggle");
+ int ret = KSFT_FAIL;
+
+ if (!cg || cg_create(cg))
+ goto out;
+ if (cg_write(cg, TIERS_MAX, "fast 0"))
+ goto out;
+ if (cg_read_strstr(cg, TIERS_MAX, "fast 0"))
+ goto out;
+ if (cg_write(cg, TIERS_MAX, "fast max"))
+ goto out;
+ if (cg_read_strstr(cg, TIERS_MAX, "fast max"))
+ goto out;
+ ret = KSFT_PASS;
+out:
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return ret;
+}
+
+/* An unknown tier name or a bad value must be rejected. */
+static int test_invalid(const char *root)
+{
+ char *cg = cg_name(root, "swaptier_invalid");
+ int ret = KSFT_FAIL;
+
+ if (!cg || cg_create(cg))
+ goto out;
+ if (!cg_write(cg, TIERS_MAX, "nosuchtier 0"))
+ goto out;
+ if (!cg_write(cg, TIERS_MAX, "fast bogus"))
+ goto out;
+ ret = KSFT_PASS;
+out:
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return ret;
+}
+
+/* A tier recreated by the same name is allowed again, even if disabled before. */
+static int test_recreate(const char *root)
+{
+ char *cg = cg_name(root, "swaptier_recreate");
+ int ret = KSFT_FAIL;
+
+ if (!cg || cg_create(cg))
+ goto out;
+ if (cg_write(cg, TIERS_MAX, "fast 0"))
+ goto out;
+ if (cg_read_strstr(cg, TIERS_MAX, "fast 0"))
+ goto out;
+ if (tiers_write("-fast") || tiers_write("+fast:10"))
+ goto out;
+ if (cg_read_strstr(cg, TIERS_MAX, "fast max"))
+ goto out;
+ ret = KSFT_PASS;
+out:
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return ret;
+}
+
+/* Map anon memory, fault it in, push it to swap, then wait to be killed. */
+static int swapout_child(const char *cgroup, void *arg)
+{
+ size_t size = (size_t)arg;
+ char *mem;
+ size_t i;
+ int page_size;
+
+ mem = mmap(NULL, size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (mem == MAP_FAILED)
+ return -1;
+
+ page_size = sysconf(_SC_PAGE_SIZE);
+ for (i = 0; i < size; i += page_size)
+ mem[i] = 'x';
+ if (madvise(mem, size, MADV_PAGEOUT))
+ return -1;
+ /* Hold the swap entries while the parent inspects /proc/swaps. */
+ pause();
+ return 0;
+}
+
+static int run_routing_case(const char *cg)
+{
+ char fast_dev[32], slow_dev[32];
+ int zfast = -1, zslow = -1;
+ long used_fast, used_slow;
+ int ret = KSFT_SKIP;
+ pid_t pid = -1;
+ int i;
+
+ /* Only our devices must be present, so usage is unambiguous. */
+ if (swap_active_count() != 0)
+ return KSFT_SKIP;
+
+ zfast = zram_add(MB(128));
+ zslow = zram_add(MB(128));
+ if (zfast < 0 || zslow < 0)
+ goto out;
+ snprintf(fast_dev, sizeof(fast_dev), "/dev/zram%d", zfast);
+ snprintf(slow_dev, sizeof(slow_dev), "/dev/zram%d", zslow);
+
+ /* prio 10 -> 'fast' tier [10, MAX]; prio 0 -> 'slow' tier [-1, 9]. */
+ if (swap_setup(fast_dev, 10) || swap_setup(slow_dev, 0))
+ goto out;
+
+ ret = KSFT_FAIL;
+
+ pid = cg_run_nowait(cg, swapout_child, (void *)MB(64));
+ if (pid < 0)
+ goto out;
+
+ for (i = 0; i < 50; i++) { /* up to ~5s for pageout */
+ if (swap_used_kb(slow_dev) > 0)
+ break;
+ usleep(100000);
+ }
+
+ used_fast = swap_used_kb(fast_dev);
+ used_slow = swap_used_kb(slow_dev);
+ if (used_slow > 0 && used_fast == 0)
+ ret = KSFT_PASS;
+ else
+ ksft_print_msg("routing[%s]: fast=%ldKB slow=%ldKB (want fast=0, slow>0)\n",
+ cg, used_fast, used_slow);
+out:
+ if (pid > 0) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ }
+ if (zfast >= 0) {
+ swapoff(fast_dev);
+ zram_remove(zfast);
+ }
+ if (zslow >= 0) {
+ swapoff(slow_dev);
+ zram_remove(zslow);
+ }
+ return ret;
+}
+
+/*
+ * A cgroup that disabled the high-priority 'fast' tier must swap only to the
+ * 'slow' tier's device; the fast device must stay untouched.
+ */
+static int test_routing(const char *root)
+{
+ char *cg = cg_name(root, "swaptier_routing");
+ int ret = KSFT_FAIL;
+
+ if (!cg || cg_create(cg))
+ goto out;
+ if (cg_write(cg, TIERS_MAX, "fast 0"))
+ goto out;
+ ret = run_routing_case(cg);
+out:
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return ret;
+}
+
+/* Create @name under @root and delegate the memory controller to its children. */
+static char *make_parent(const char *root, const char *name)
+{
+ char *cg = cg_name(root, name);
+
+ if (cg && !cg_create(cg) &&
+ !cg_write(cg, "cgroup.subtree_control", "+memory"))
+ return cg;
+
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return NULL;
+}
+
+/*
+ * The effective mask is the parent's intersected with the child's, so a tier
+ * the parent disabled stays disabled for the child even if the child re-enables
+ * it. Parent disables 'fast', child sets 'fast max' -> child still swaps slow.
+ */
+static int test_routing_parent_wins(const char *root)
+{
+ char *parent = make_parent(root, "swaptier_pwins");
+ char *child = NULL;
+ int ret = KSFT_FAIL;
+
+ if (!parent)
+ goto out;
+ if (cg_write(parent, TIERS_MAX, "fast 0"))
+ goto out;
+
+ child = cg_name(parent, "child");
+ if (!child || cg_create(child))
+ goto out;
+ if (cg_write(child, TIERS_MAX, "fast max")) /* child tries to re-enable */
+ goto out;
+
+ ret = run_routing_case(child);
+out:
+ if (child) {
+ cg_destroy(child);
+ free(child);
+ }
+ if (parent) {
+ cg_destroy(parent);
+ free(parent);
+ }
+ return ret;
+}
+
+/*
+ * A child can restrict below its parent: the parent leaves all tiers enabled,
+ * the child disables 'fast' on its own -> the child swaps only to slow.
+ */
+static int test_routing_child_restricts(const char *root)
+{
+ char *parent = make_parent(root, "swaptier_crestr");
+ char *child = NULL;
+ int ret = KSFT_FAIL;
+
+ if (!parent)
+ goto out;
+
+ child = cg_name(parent, "child");
+ if (!child || cg_create(child))
+ goto out;
+ if (cg_write(child, TIERS_MAX, "fast 0"))
+ goto out;
+
+ ret = run_routing_case(child);
+out:
+ if (child) {
+ cg_destroy(child);
+ free(child);
+ }
+ if (parent) {
+ cg_destroy(parent);
+ free(parent);
+ }
+ return ret;
+}
+
+/* Remove all remaining tiers, so a mid-test failure still leaves them empty. */
+static void tiers_clear(void)
+{
+ char buf[4096], *line, *save;
+ int fd;
+ ssize_t n;
+
+ fd = open(TIERS_PATH, O_RDONLY);
+ if (fd < 0)
+ return;
+ n = read(fd, buf, sizeof(buf) - 1);
+ close(fd);
+ if (n < 0)
+ return;
+ buf[n] = '\0';
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ char name[64], cmd[80];
+ int idx, s, e;
+
+ if (sscanf(line, "%63s %d %d %d", name, &idx, &s, &e) != 4)
+ continue;
+ snprintf(cmd, sizeof(cmd), "-%s", name);
+ tiers_write(cmd);
+ }
+}
+
+int main(void)
+{
+ char root[PATH_MAX];
+
+ ksft_print_header();
+ ksft_set_plan(7);
+
+ if (geteuid() != 0)
+ ksft_exit_skip("test requires root\n");
+ if (cg_find_unified_root(root, sizeof(root), NULL))
+ ksft_exit_skip("cgroup v2 isn't mounted\n");
+ if (cg_read_strstr(root, "cgroup.controllers", "memory"))
+ ksft_exit_skip("memory controller isn't available\n");
+ if (cg_read_strstr(root, "cgroup.subtree_control", "memory"))
+ if (cg_write(root, "cgroup.subtree_control", "+memory"))
+ ksft_exit_skip("failed to enable memory controller\n");
+ if (access(TIERS_PATH, F_OK))
+ ksft_exit_skip("swap tiers interface not present\n");
+ if (tier_count() != 0)
+ ksft_exit_skip("swap tiers already configured; run on a clean system\n");
+
+ /* Two tiers: fast = [10, MAX], slow = [-1, 9]. */
+ if (tiers_write("+slow:-1 +fast:10"))
+ ksft_exit_skip("failed to configure swap tiers\n");
+
+ ksft_test_result(test_default(root) == KSFT_PASS, "default mask is max\n");
+ ksft_test_result(test_toggle(root) == KSFT_PASS, "enable/disable tier\n");
+ ksft_test_result(test_invalid(root) == KSFT_PASS, "invalid input rejected\n");
+ ksft_test_result(test_recreate(root) == KSFT_PASS,
+ "recreated tier resets cgroup mask\n");
+
+ ksft_test_result_code(test_routing(root),
+ "swapout honors tier mask", NULL);
+ ksft_test_result_code(test_routing_parent_wins(root),
+ "child cannot re-enable a parent-disabled tier", NULL);
+ ksft_test_result_code(test_routing_child_restricts(root),
+ "child can restrict tiers below its parent", NULL);
+
+ tiers_clear();
+
+ ksft_finished();
+}
--
2.48.1
^ permalink raw reply related
* [PATCH v9 5/6] selftests/mm: add a swap tier configuration test
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260620181635.299364-1-youngjun.park@lge.com>
This commit adds a test program for the global swap tier interface at
/sys/kernel/mm/swap/tiers. It checks the add, split and remove
operations and the documented error and batch atomicity rules. It also
checks that a tier with an active swap device cannot be removed until
the device is swapped off. That device is a zram device, and the check
is skipped when zram is not available.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
tools/testing/selftests/mm/.gitignore | 1 +
tools/testing/selftests/mm/Makefile | 1 +
tools/testing/selftests/mm/config | 2 +
tools/testing/selftests/mm/run_vmtests.sh | 5 +
tools/testing/selftests/mm/swap_tier.c | 323 ++++++++++++++++++++++
5 files changed, 332 insertions(+)
create mode 100644 tools/testing/selftests/mm/swap_tier.c
diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore
index 9ccd9e1447e6..a6e588c7979e 100644
--- a/tools/testing/selftests/mm/.gitignore
+++ b/tools/testing/selftests/mm/.gitignore
@@ -46,6 +46,7 @@ hmm-tests
memfd_secret
soft-dirty
split_huge_page_test
+swap_tier
ksm_tests
local_config.h
local_config.mk
diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index e6df968f0971..1836127df092 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -105,6 +105,7 @@ TEST_GEN_FILES += guard-regions
TEST_GEN_FILES += merge
TEST_GEN_FILES += rmap
TEST_GEN_FILES += folio_split_race_test
+TEST_GEN_FILES += swap_tier
ifneq ($(ARCH),arm64)
TEST_GEN_FILES += soft-dirty
diff --git a/tools/testing/selftests/mm/config b/tools/testing/selftests/mm/config
index 06f78bd232e2..de3752e1bbd2 100644
--- a/tools/testing/selftests/mm/config
+++ b/tools/testing/selftests/mm/config
@@ -14,3 +14,5 @@ CONFIG_UPROBES=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_PROC_MEM_ALWAYS_FORCE=y
+CONFIG_SWAP=y
+CONFIG_ZRAM=y
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 8c296dedf047..1b0b8ec185a9 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -71,6 +71,8 @@ separated by spaces:
tests for VM_PFNMAP handling
- process_madv
test for process_madv
+- swap_tier
+ test the /sys/kernel/mm/swap/tiers configuration interface
- cow
test copy-on-write semantics
- thp
@@ -353,6 +355,9 @@ CATEGORY="process_madv" run_test ./process_madv
CATEGORY="vma_merge" run_test ./merge
+# swap tier configuration interface (/sys/kernel/mm/swap/tiers)
+CATEGORY="swap_tier" run_test ./swap_tier
+
if [ -x ./memfd_secret ]
then
if [ -f /proc/sys/kernel/yama/ptrace_scope ]; then
diff --git a/tools/testing/selftests/mm/swap_tier.c b/tools/testing/selftests/mm/swap_tier.c
new file mode 100644
index 000000000000..b4fe21b0eb5d
--- /dev/null
+++ b/tools/testing/selftests/mm/swap_tier.c
@@ -0,0 +1,323 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/swap.h>
+#include <unistd.h>
+
+#include "kselftest.h"
+
+#define TIERS_PATH "/sys/kernel/mm/swap/tiers"
+
+static int tiers_write(const char *cmd)
+{
+ int fd, ret = 0;
+
+ fd = open(TIERS_PATH, O_WRONLY);
+ if (fd < 0)
+ return -errno;
+ if (write(fd, cmd, strlen(cmd)) < 0)
+ ret = -errno;
+ close(fd);
+ return ret;
+}
+
+static int tier_range(const char *name, int *start, int *end)
+{
+ char buf[4096], *line, *save;
+ int fd;
+ ssize_t n;
+
+ fd = open(TIERS_PATH, O_RDONLY);
+ if (fd < 0)
+ return -1;
+ n = read(fd, buf, sizeof(buf) - 1);
+ close(fd);
+ if (n < 0)
+ return -1;
+ buf[n] = '\0';
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ char tname[64];
+ int idx, s, e;
+
+ /* The header line has no integer columns, so sscanf skips it. */
+ if (sscanf(line, "%63s %d %d %d", tname, &idx, &s, &e) != 4)
+ continue;
+ if (!strcmp(tname, name)) {
+ *start = s;
+ *end = e;
+ return 0;
+ }
+ }
+ return -1;
+}
+
+static bool tier_exists(const char *name)
+{
+ int s, e;
+
+ return tier_range(name, &s, &e) == 0;
+}
+
+static bool range_is(const char *name, int start, int end)
+{
+ int s, e;
+
+ if (tier_range(name, &s, &e))
+ return false;
+ return s == start && e == end;
+}
+
+static int tier_count(void)
+{
+ char buf[4096], *line, *save;
+ int fd, count = 0;
+ ssize_t n;
+
+ fd = open(TIERS_PATH, O_RDONLY);
+ if (fd < 0)
+ return -1;
+ n = read(fd, buf, sizeof(buf) - 1);
+ close(fd);
+ if (n < 0)
+ return -1;
+ buf[n] = '\0';
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ char tname[64];
+ int idx, s, e;
+
+ if (sscanf(line, "%63s %d %d %d", tname, &idx, &s, &e) == 4)
+ count++;
+ }
+ return count;
+}
+
+/*
+ * A single add at a priority above -1, from the empty set, leaves the range
+ * below it uncovered and must be rejected. the set stays empty.
+ */
+static int test_coverage(void)
+{
+ if (tiers_write("+orphan:100") != -EINVAL)
+ return KSFT_FAIL;
+ if (tier_exists("orphan"))
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+/*
+ * Add two tiers covering the full range. The end priority of a tier is the
+ * start of the next higher tier minus one.
+ */
+static int test_add(void)
+{
+ if (tiers_write("+lo:-1 +hi:50"))
+ return KSFT_FAIL;
+ if (!range_is("hi", 50, SHRT_MAX) || !range_is("lo", -1, 49))
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+/* Adding a tier inside an existing range splits it. the lower part shrinks. */
+static int test_split(void)
+{
+ if (tiers_write("+mid:100"))
+ return KSFT_FAIL;
+ if (!range_is("mid", 100, SHRT_MAX) ||
+ !range_is("hi", 50, 99) ||
+ !range_is("lo", -1, 49))
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+/* Removing a tier merges its range into the adjacent (lower) tier. */
+static int test_remove(void)
+{
+ /* Remove the top tier: 'hi' re-expands upward to SHRT_MAX. */
+ if (tiers_write("-mid"))
+ return KSFT_FAIL;
+ if (tier_exists("mid") || !range_is("hi", 50, SHRT_MAX))
+ return KSFT_FAIL;
+
+ /* Remove the lowest tier: 'hi' shifts its start down to -1. */
+ if (tiers_write("-lo"))
+ return KSFT_FAIL;
+ if (tier_exists("lo") || !range_is("hi", -1, SHRT_MAX))
+ return KSFT_FAIL;
+
+ return KSFT_PASS;
+}
+
+/* Each invalid operation must fail with its documented errno. State: hi[-1,MAX]. */
+static int test_errors(void)
+{
+ if (tiers_write("+hi:100") != -EEXIST) /* duplicate name */
+ return KSFT_FAIL;
+ if (tiers_write("+bad.name:100") != -EINVAL) /* illegal name */
+ return KSFT_FAIL;
+ if (tiers_write("+dup:-1") != -EBUSY) /* priority in use */
+ return KSFT_FAIL;
+ if (tiers_write("+low:-2") != -EINVAL) /* prio < DEF_SWAP_PRIO */
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+/*
+ * A write carrying several operations is atomic: if any operation fails, the
+ * whole batch is rolled back. The second '+a' duplicates the first and fails,
+ * so neither must take effect. State before/after: hi[-1,MAX].
+ */
+static int test_atomic(void)
+{
+ if (tiers_write("+a:50 +a:60") != -EEXIST)
+ return KSFT_FAIL;
+ if (tier_exists("a") || !range_is("hi", -1, SHRT_MAX))
+ return KSFT_FAIL;
+ return KSFT_PASS;
+}
+
+static int zram_add(long size)
+{
+ char path[128], val[64];
+ ssize_t n;
+ int idx, fd;
+
+ fd = open("/sys/class/zram-control/hot_add", O_RDONLY);
+ if (fd < 0)
+ return -1;
+ n = read(fd, val, sizeof(val) - 1);
+ close(fd);
+ if (n <= 0)
+ return -1;
+ val[n] = '\0';
+ idx = atoi(val);
+
+ snprintf(path, sizeof(path), "/sys/block/zram%d/disksize", idx);
+ fd = open(path, O_WRONLY);
+ if (fd < 0)
+ return -1;
+ snprintf(val, sizeof(val), "%ld", size);
+ n = write(fd, val, strlen(val));
+ close(fd);
+ return n < 0 ? -1 : idx;
+}
+
+static void zram_remove(int idx)
+{
+ char val[16];
+ int fd;
+
+ fd = open("/sys/class/zram-control/hot_remove", O_WRONLY);
+ if (fd < 0)
+ return;
+ snprintf(val, sizeof(val), "%d", idx);
+ if (write(fd, val, strlen(val)) < 0)
+ ; /* ignore: best-effort cleanup */
+ close(fd);
+}
+
+static int swap_setup(const char *dev, int prio)
+{
+ char cmd[128];
+
+ snprintf(cmd, sizeof(cmd), "mkswap %s >/dev/null 2>&1", dev);
+ if (system(cmd))
+ return -1;
+ return swapon(dev, SWAP_FLAG_PREFER | (prio & SWAP_FLAG_PRIO_MASK));
+}
+
+/* A tier holding an active swap device can't be removed until swapoff. */
+static int test_device_pins_tier(void)
+{
+ char dev[32];
+ int zidx, ret = KSFT_FAIL;
+
+ if (tiers_write("+top:50"))
+ return KSFT_FAIL;
+
+ zidx = zram_add(64 << 20);
+ if (zidx < 0) {
+ ret = KSFT_SKIP;
+ goto out_tier;
+ }
+ snprintf(dev, sizeof(dev), "/dev/zram%d", zidx);
+ if (swap_setup(dev, 50)) {
+ ret = KSFT_SKIP;
+ goto out_zram;
+ }
+
+ if (tiers_write("-top") == -EBUSY) { /* blocked while active */
+ swapoff(dev);
+ if (!tiers_write("-top")) /* removable after swapoff */
+ ret = KSFT_PASS;
+ } else {
+ swapoff(dev);
+ }
+out_zram:
+ zram_remove(zidx);
+out_tier:
+ tiers_write("-top");
+ return ret;
+}
+
+/* Remove all remaining tiers, so a mid-test failure still leaves them empty. */
+static void tiers_clear(void)
+{
+ char buf[4096], *line, *save;
+ int fd;
+ ssize_t n;
+
+ fd = open(TIERS_PATH, O_RDONLY);
+ if (fd < 0)
+ return;
+ n = read(fd, buf, sizeof(buf) - 1);
+ close(fd);
+ if (n < 0)
+ return;
+ buf[n] = '\0';
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ char name[64], cmd[80];
+ int idx, s, e;
+
+ if (sscanf(line, "%63s %d %d %d", name, &idx, &s, &e) != 4)
+ continue;
+ snprintf(cmd, sizeof(cmd), "-%s", name);
+ tiers_write(cmd);
+ }
+}
+
+int main(void)
+{
+ ksft_print_header();
+ ksft_set_plan(7);
+
+ if (geteuid() != 0)
+ ksft_exit_skip("test requires root\n");
+ if (access(TIERS_PATH, F_OK))
+ ksft_exit_skip("%s not present (CONFIG_SWAP/tiers)\n", TIERS_PATH);
+ if (tier_count() != 0)
+ ksft_exit_skip("swap tiers already configured; run on a clean system\n");
+
+ ksft_test_result(test_coverage() == KSFT_PASS, "coverage rule\n");
+ ksft_test_result(test_add() == KSFT_PASS, "add tiers\n");
+ ksft_test_result(test_split() == KSFT_PASS, "split tier\n");
+ ksft_test_result(test_remove() == KSFT_PASS, "remove and merge\n");
+ ksft_test_result(test_errors() == KSFT_PASS, "invalid operations\n");
+ ksft_test_result(test_atomic() == KSFT_PASS, "batch atomicity\n");
+
+ ksft_test_result_code(test_device_pins_tier(), "device pins its tier", NULL);
+
+ tiers_clear();
+
+ ksft_finished();
+}
--
2.48.1
^ permalink raw reply related
* [PATCH v9 4/6] mm: swap: filter swap allocation by memcg tier mask
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260620181635.299364-1-youngjun.park@lge.com>
Apply memcg tier effective mask during swap slot allocation to
enforce per-cgroup swap tier restrictions.
The folio's effective mask is computed once and passed to the fast,
slow and discard paths as a parameter, so all of them act on the same
mask even if the memcg's mask changes concurrently.
In the fast path, check the percpu cached swap_info's tier_mask
against the folio's effective mask. If it does not match, fall
through to the slow path. In the slow path, skip swap devices
whose tier_mask is not covered by the folio's effective mask.
The discard fallback honors the mask too: otherwise it would drain
the discard clusters of a device outside the folio's tiers and then
loop back to allocate from a tier the memcg is not allowed to use.
This works correctly when there is only one non-rotational
device in the system and no devices share the same priority.
However, there are known limitations:
- When non-rotational devices are distributed across multiple
tiers, and different memcgs are configured to use those
distinct tiers, they may constantly overwrite the shared
percpu swap cache. This cache thrashing leads to frequent
fast path misses.
- Combined with the above issue, if same-priority devices exist
among them, a percpu cache miss (overwritten by another memcg)
forces the allocator to round-robin to the next device
prematurely, even if the current cluster is not fully
exhausted.
These edge cases do not affect the primary use case of
directing swap traffic per cgroup. Further optimization is
planned for future work.
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
mm/swapfile.c | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 9a86ebe992f4..624d1ba93fd9 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1359,7 +1359,7 @@ static bool get_swap_device_info(struct swap_info_struct *si)
* Fast path try to get swap entries with specified order from current
* CPU's swap entry pool (a cluster).
*/
-static bool swap_alloc_fast(struct folio *folio)
+static bool swap_alloc_fast(struct folio *folio, int mask)
{
unsigned int order = folio_order(folio);
struct swap_cluster_info *ci;
@@ -1371,8 +1371,11 @@ static bool swap_alloc_fast(struct folio *folio)
* so checking it's liveness by get_swap_device_info is enough.
*/
si = this_cpu_read(percpu_swap_cluster.si[order]);
+ if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
+ return false;
+
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
- if (!si || !offset || !get_swap_device_info(si))
+ if (!offset || !get_swap_device_info(si))
return false;
ci = swap_cluster_lock(si, offset);
@@ -1389,13 +1392,16 @@ static bool swap_alloc_fast(struct folio *folio)
}
/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static void swap_alloc_slow(struct folio *folio, int mask)
{
struct swap_info_struct *si, *next;
spin_lock(&swap_avail_lock);
start_over:
plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+ if (!swap_tiers_mask_test(si->tier_mask, mask))
+ continue;
+
/* Rotate the device and switch to a new cluster */
plist_requeue(&si->avail_list, &swap_avail_head);
spin_unlock(&swap_avail_lock);
@@ -1429,7 +1435,7 @@ static void swap_alloc_slow(struct folio *folio)
* Discard pending clusters in a synchronized way when under high pressure.
* Return: true if any cluster is discarded.
*/
-static bool swap_sync_discard(void)
+static bool swap_sync_discard(int mask)
{
bool ret = false;
struct swap_info_struct *si, *next;
@@ -1437,6 +1443,8 @@ static bool swap_sync_discard(void)
spin_lock(&swap_lock);
start_over:
plist_for_each_entry_safe(si, next, &swap_active_head, list) {
+ if (!swap_tiers_mask_test(si->tier_mask, mask))
+ continue;
spin_unlock(&swap_lock);
if (get_swap_device_info(si)) {
if (si->flags & SWP_PAGE_DISCARD)
@@ -1736,6 +1744,7 @@ int folio_alloc_swap(struct folio *folio)
{
unsigned int order = folio_order(folio);
unsigned int size = 1 << order;
+ int mask;
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1759,13 +1768,14 @@ int folio_alloc_swap(struct folio *folio)
}
again:
+ mask = folio_tier_effective_mask(folio);
local_lock(&percpu_swap_cluster.lock);
- if (!swap_alloc_fast(folio))
- swap_alloc_slow(folio);
+ if (!swap_alloc_fast(folio, mask))
+ swap_alloc_slow(folio, mask);
local_unlock(&percpu_swap_cluster.lock);
if (!order && unlikely(!folio_test_swapcache(folio))) {
- if (swap_sync_discard())
+ if (swap_sync_discard(mask))
goto again;
}
--
2.48.1
^ permalink raw reply related
* [PATCH v9 3/6] mm: memcontrol: add interface for swap tier selection
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260620181635.299364-1-youngjun.park@lge.com>
Introduce memory.swap.tiers.max, a flat-keyed file listing each
tier defined in /sys/kernel/mm/swap/tiers with its state, "max"
(allowed, the default) or "0" (disabled). A tier is one bit in the
cgroup's tier mask, so writing "<tier> max" or "<tier> 0" sets or
clears that bit.
Since the current use case lacks amount control, it only supports
"max" (on) and "0" (off). Therefore, it does not track per-tier swap
usage, relying instead on a fast runtime bitmask check.
We maintain both `mask` and `effective_mask`. The `effective_mask` is
strictly bounded by the parent (e.g., if a parent is "0", the child's
effective state is "0" even if its `mask` is "max"). Maintaining this
separately avoids costly cgroup tree traversals to check ancestors at
runtime.
Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
Documentation/admin-guide/cgroup-v2.rst | 20 +++++
Documentation/mm/swap-tier.rst | 9 +++
include/linux/memcontrol.h | 5 ++
mm/memcontrol.c | 67 ++++++++++++++++
mm/swap_state.c | 5 +-
mm/swap_tier.c | 102 +++++++++++++++++++++++-
mm/swap_tier.h | 57 +++++++++++--
7 files changed, 255 insertions(+), 10 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed99..4843ffcfd110 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1850,6 +1850,26 @@ The following nested keys are defined.
Swap usage hard limit. If a cgroup's swap usage reaches this
limit, anonymous memory of the cgroup will not be swapped out.
+ memory.swap.tiers.max
+ A read-write flat-keyed file which exists on non-root
+ cgroups. The default is "max" for every tier.
+
+ Limits the swap tiers this cgroup may swap to. Tiers are
+ defined globally in /sys/kernel/mm/swap/tiers and listed here,
+ one per line. When read, the values are displayed in descending
+ order of the tiers (highest tier first)::
+
+ <tier_1> max
+ <tier_2> 0
+ ...
+
+ Currently, only "max" and "0" are supported. "max" allows the
+ tier, "0" disables it. Each write sets a single "<tier> max"
+ or "<tier> 0" pair.
+
+ A child may only narrow what its parent allows. A tier an
+ ancestor disabled stays disabled regardless of the value here.
+
memory.swap.events
A read-only flat-keyed file which exists on non-root cgroups.
The following entries are defined. Unless specified
diff --git a/Documentation/mm/swap-tier.rst b/Documentation/mm/swap-tier.rst
index 0fb4a1153a67..addbc495de8c 100644
--- a/Documentation/mm/swap-tier.rst
+++ b/Documentation/mm/swap-tier.rst
@@ -15,6 +15,15 @@ speed to fully utilize this feature. While the current implementation is
integrated with cgroups, the concept is designed to be extensible for other
subsystems in the future.
+Use case
+---------
+
+Users can perform selective swapping by choosing a swap tier assigned according
+to speed within a cgroup.
+
+For more information on cgroup v2, please refer to
+``Documentation/admin-guide/cgroup-v2.rst``.
+
Priority Range
--------------
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e1f46a0016fc..d53826c68562 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -283,6 +283,11 @@ struct mem_cgroup {
struct lru_gen_mm_list mm_list;
#endif
+#ifdef CONFIG_SWAP
+ int tier_mask;
+ int tier_effective_mask;
+#endif
+
#ifdef CONFIG_MEMCG_V1
/* Legacy consumer-oriented counters */
struct page_counter kmem; /* v1 only */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 56cd4af08232..63259576792a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -68,6 +68,7 @@
#include <net/ip.h>
#include "slab.h"
#include "memcontrol-v1.h"
+#include "swap_tier.h"
#include <linux/uaccess.h>
@@ -4244,6 +4245,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
refcount_set(&memcg->id.ref, 1);
css_get(css);
+ swap_tiers_memcg_inherit_mask(memcg);
+
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
*
@@ -5785,6 +5788,64 @@ static int swap_events_show(struct seq_file *m, void *v)
return 0;
}
+static int swap_tier_max_show(struct seq_file *m, void *v)
+{
+ struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+
+ swap_tiers_mask_show(m, memcg);
+ return 0;
+}
+
+static ssize_t swap_tier_max_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
+ char *pos, *name, *val;
+ bool enable;
+ int mask;
+ int ret = 0;
+
+ pos = strstrip(buf);
+ name = strsep(&pos, " \t\n");
+ if (!name || !*name)
+ return -EINVAL;
+ if (pos)
+ pos = skip_spaces(pos);
+ val = strsep(&pos, " \t\n");
+ if (!val || !*val)
+ return -EINVAL;
+ if (pos && *skip_spaces(pos))
+ return -EINVAL;
+
+ if (!strcmp(val, "max"))
+ enable = true;
+ else if (!strcmp(val, "0"))
+ enable = false;
+ else
+ return -EINVAL;
+
+ spin_lock(&swap_tier_lock);
+ mask = swap_tiers_mask_lookup(name);
+ if (!mask) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /*
+ * tier_mask is set per memcg here; the effective mask is clamped
+ * to the parent's in swap_tiers_memcg_sync_mask().
+ */
+ if (enable)
+ WRITE_ONCE(memcg->tier_mask, memcg->tier_mask | mask);
+ else
+ WRITE_ONCE(memcg->tier_mask, memcg->tier_mask & ~mask);
+
+ swap_tiers_memcg_sync_mask(memcg);
+out:
+ spin_unlock(&swap_tier_lock);
+ return ret ? ret : nbytes;
+}
+
static struct cftype swap_files[] = {
{
.name = "swap.current",
@@ -5817,6 +5878,12 @@ static struct cftype swap_files[] = {
.file_offset = offsetof(struct mem_cgroup, swap_events_file),
.seq_show = swap_events_show,
},
+ {
+ .name = "swap.tiers.max",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = swap_tier_max_show,
+ .write = swap_tier_max_write,
+ },
{ } /* terminate */
};
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 2f382d4dcbdc..712b225509cc 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -1021,6 +1021,7 @@ static ssize_t tiers_store(struct kobject *kobj,
char *p, *token, *name, *tmp;
int ret = 0;
short prio;
+ int mask = 0;
tmp = kstrdup(buf, GFP_KERNEL);
if (!tmp)
@@ -1053,7 +1054,7 @@ static ssize_t tiers_store(struct kobject *kobj,
goto restore;
break;
case '-':
- ret = swap_tiers_remove(token + 1);
+ ret = swap_tiers_remove(token + 1, &mask);
if (ret)
goto restore;
break;
@@ -1063,7 +1064,7 @@ static ssize_t tiers_store(struct kobject *kobj,
}
}
- if (!swap_tiers_update()) {
+ if (!swap_tiers_update(mask)) {
ret = -EINVAL;
goto restore;
}
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index 6b57cadb3e95..98bfee760b8d 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -253,7 +253,7 @@ int swap_tiers_add(const char *name, int prio)
return ret;
}
-int swap_tiers_remove(const char *name)
+int swap_tiers_remove(const char *name, int *mask)
{
int ret = 0;
struct swap_tier *tier;
@@ -276,6 +276,7 @@ int swap_tiers_remove(const char *name)
list_prev_entry(tier, list)->prio = DEF_SWAP_PRIO;
swap_tier_inactivate(tier);
+ *mask |= TIER_MASK(tier);
return ret;
}
@@ -344,7 +345,24 @@ void swap_tiers_assign_dev(struct swap_info_struct *swp)
swp->tier_mask = TIER_DEFAULT_MASK;
}
-bool swap_tiers_update(void)
+#ifdef CONFIG_MEMCG
+static void swap_tier_memcg_propagate(int mask)
+{
+ struct mem_cgroup *child;
+
+ for_each_mem_cgroup_tree(child, root_mem_cgroup) {
+ WRITE_ONCE(child->tier_mask, child->tier_mask | mask);
+ WRITE_ONCE(child->tier_effective_mask,
+ child->tier_effective_mask | mask);
+ }
+}
+#else
+static void swap_tier_memcg_propagate(int mask)
+{
+}
+#endif
+
+bool swap_tiers_update(int mask)
{
struct swap_tier *tier;
struct swap_info_struct *swp;
@@ -375,5 +393,85 @@ bool swap_tiers_update(void)
swap_tiers_assign_dev(swp);
}
+ /*
+ * When a tier is removed, its index (bit position in the mask) becomes
+ * free for reassignment to a future tier. If a memcg had previously
+ * disabled this tier (cleared the bit in its swap.tiers.max file), the
+ * effective mask would keep that bit clear -- meaning the new tier at
+ * the same index would be silently unavailable, an invisible cgroup
+ * constraint left behind by a tier that no longer exists.
+ *
+ * To prevent this, OR the removed tier's mask bit into every memcg's
+ * tier_mask and tier_effective_mask. This resets the bit so the new
+ * tier is accessible by default; users who want to restrict it must
+ * explicitly disable it after the tier is re-created.
+ */
+ if (mask)
+ swap_tier_memcg_propagate(mask);
+
return true;
}
+
+#ifdef CONFIG_MEMCG
+void swap_tiers_mask_show(struct seq_file *m, struct mem_cgroup *memcg)
+{
+ struct swap_tier *tier;
+ int mask;
+
+ spin_lock(&swap_tier_lock);
+ mask = READ_ONCE(memcg->tier_mask);
+
+ for_each_active_tier(tier)
+ seq_printf(m, "%s %s\n", tier->name,
+ (mask & TIER_MASK(tier)) ? "max" : "0");
+ spin_unlock(&swap_tier_lock);
+}
+
+int swap_tiers_mask_lookup(const char *name)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ for_each_active_tier(tier) {
+ if (!strcmp(name, tier->name))
+ return TIER_MASK(tier);
+ }
+
+ return 0;
+}
+
+static void __swap_tier_memcg_inherit_mask(struct mem_cgroup *memcg,
+ struct mem_cgroup *parent)
+{
+ int parent_mask = parent
+ ? READ_ONCE(parent->tier_effective_mask)
+ : TIER_ALL_MASK;
+
+ WRITE_ONCE(memcg->tier_effective_mask,
+ parent_mask & READ_ONCE(memcg->tier_mask));
+}
+
+/* Computes the initial effective mask from the parent's effective mask. */
+void swap_tiers_memcg_inherit_mask(struct mem_cgroup *memcg)
+{
+ spin_lock(&swap_tier_lock);
+ memcg->tier_mask = TIER_ALL_MASK;
+ __swap_tier_memcg_inherit_mask(memcg, parent_mem_cgroup(memcg));
+ spin_unlock(&swap_tier_lock);
+}
+
+/*
+ * Called when a memcg's tier_mask is modified. Walks the subtree
+ * and recomputes each descendant's effective mask against its parent.
+ */
+void swap_tiers_memcg_sync_mask(struct mem_cgroup *memcg)
+{
+ struct mem_cgroup *child;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ for_each_mem_cgroup_tree(child, memcg)
+ __swap_tier_memcg_inherit_mask(child, parent_mem_cgroup(child));
+}
+#endif
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
index 3e355f857363..e2f0cf32035b 100644
--- a/mm/swap_tier.h
+++ b/mm/swap_tier.h
@@ -10,22 +10,67 @@ struct swap_info_struct;
extern spinlock_t swap_tier_lock;
-#define TIER_ALL_MASK (~0)
-#define TIER_DEFAULT_IDX (31)
-#define TIER_DEFAULT_MASK (1U << TIER_DEFAULT_IDX)
-
/* Initialization and application */
void swap_tiers_init(void);
ssize_t swap_tiers_sysfs_show(char *buf);
int swap_tiers_add(const char *name, int prio);
-int swap_tiers_remove(const char *name);
+int swap_tiers_remove(const char *name, int *mask);
void swap_tiers_snapshot(void);
void swap_tiers_snapshot_restore(void);
-bool swap_tiers_update(void);
+bool swap_tiers_update(int mask);
/* Tier assignment */
void swap_tiers_assign_dev(struct swap_info_struct *swp);
+#define TIER_ALL_MASK (~0)
+#define TIER_DEFAULT_IDX (31)
+#define TIER_DEFAULT_MASK (1U << TIER_DEFAULT_IDX)
+
+#if defined(CONFIG_SWAP) && defined(CONFIG_MEMCG)
+/* Memcg related functions */
+void swap_tiers_mask_show(struct seq_file *m, struct mem_cgroup *memcg);
+void swap_tiers_memcg_inherit_mask(struct mem_cgroup *memcg);
+void swap_tiers_memcg_sync_mask(struct mem_cgroup *memcg);
+int swap_tiers_mask_lookup(const char *name);
+static inline int folio_tier_effective_mask(struct folio *folio)
+{
+ struct mem_cgroup *memcg;
+ int mask = TIER_ALL_MASK;
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (memcg)
+ mask = READ_ONCE(memcg->tier_effective_mask);
+ rcu_read_unlock();
+
+ return mask;
+}
+#else
+static inline void swap_tiers_mask_show(struct seq_file *m,
+ struct mem_cgroup *memcg) {}
+static inline void swap_tiers_memcg_inherit_mask(struct mem_cgroup *memcg) {}
+static inline void swap_tiers_memcg_sync_mask(struct mem_cgroup *memcg) {}
+static inline int swap_tiers_mask_lookup(const char *name)
+{
+ return 0;
+}
+static inline int folio_tier_effective_mask(struct folio *folio)
+{
+ return TIER_ALL_MASK;
+}
+#endif
+
+/**
+ * swap_tiers_mask_test - Check if the tier mask is valid
+ * @tier_mask: The tier mask to check
+ * @mask: The mask to compare against
+ *
+ * Return: true if condition matches, false otherwise
+ */
+static inline bool swap_tiers_mask_test(int tier_mask, int mask)
+{
+ return tier_mask & mask;
+}
#endif /* _SWAP_TIER_H */
--
2.48.1
^ permalink raw reply related
* [PATCH v9 2/6] mm: swap: associate swap devices with tiers
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260620181635.299364-1-youngjun.park@lge.com>
This patch connects swap devices to the swap tier infrastructure,
ensuring that devices are correctly assigned to tiers based on their
priority.
A `tier_mask` is added to identify the tier membership of swap devices.
Although tier-based allocation logic is not yet implemented, this
mapping is necessary to track which tier a device belongs to. Upon
activation, the device is assigned to a tier by matching its priority
against the configured tier ranges.
The infrastructure allows dynamic modification of tiers, such as
splitting or merging ranges. These operations are permitted provided
that the tier assignment of already configured swap devices remains
unchanged.
This patch also adds the documentation for the swap tier feature,
covering the core concepts, sysfs interface usage, and configuration
details.
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
Documentation/mm/index.rst | 1 +
Documentation/mm/swap-tier.rst | 150 +++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
include/linux/swap.h | 1 +
mm/swap_state.c | 2 +-
mm/swap_tier.c | 101 +++++++++++++++++++---
mm/swap_tier.h | 13 ++-
mm/swapfile.c | 2 +
8 files changed, 257 insertions(+), 14 deletions(-)
create mode 100644 Documentation/mm/swap-tier.rst
diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst
index 7aa2a8886908..a0d1447c5569 100644
--- a/Documentation/mm/index.rst
+++ b/Documentation/mm/index.rst
@@ -21,6 +21,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
page_reclaim
swap
swap-table
+ swap-tier
page_cache
shmfs
oom
diff --git a/Documentation/mm/swap-tier.rst b/Documentation/mm/swap-tier.rst
new file mode 100644
index 000000000000..0fb4a1153a67
--- /dev/null
+++ b/Documentation/mm/swap-tier.rst
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+:Author: Chris Li <chrisl@kernel.org> Youngjun Park <youngjun.park@lge.com>
+
+==========
+Swap Tier
+==========
+
+Swap tier is a collection of user-named groups classified by priority ranges.
+It acts as a facilitation layer, allowing users to manage swap devices based
+on their speeds.
+
+Users are encouraged to assign swap device priorities according to device
+speed to fully utilize this feature. While the current implementation is
+integrated with cgroups, the concept is designed to be extensible for other
+subsystems in the future.
+
+Priority Range
+--------------
+
+The specified tiers must cover the entire priority range from -1
+(DEF_SWAP_PRIO) to SHRT_MAX.
+
+Consistency
+-----------
+
+Tier consistency is guaranteed with a focus on maximizing flexibility. When a
+swap device is activated within a tier range, the tier covering that device's
+priority is guaranteed not to disappear or change while the device remains
+active. Adding a new tier may split the range of an existing tier, but the
+active device's tier assignment remains unchanged.
+
+However, specifying a tier in a cgroup does not guarantee the tier's existence.
+Consequently, the corresponding tier can disappear at any time.
+
+Configuration Interface
+-----------------------
+
+The swap tiers can be configured via the following interface:
+
+/sys/kernel/mm/swap/tiers
+
+Operations can be performed using the following syntax:
+
+* Add: ``+"<tiername>":"<start_priority>"``
+* Remove: ``-"<tiername>"``
+
+Tier names must consist of alphanumeric characters and underscores. Multiple
+operations can be provided in a single write, separated by commas (",") or
+whitespace (spaces, tabs, newlines).
+
+When configuring tiers, the specified value represents the **start priority**
+of that tier. The end priority is automatically determined by the start
+priority of the next higher tier. Consequently, adding a tier
+automatically adjusts the ranges of adjacent tiers to ensure continuity.
+
+Examples
+--------
+
+**1. Initialization**
+
+A tier starting at -1 is mandatory to cover the entire priority range up to
+SHRT_MAX. In this example, 'HDD' starts at 50, and 'NET' covers the remaining
+lower range starting from -1.
+
+::
+
+ # echo "+HDD:50, +NET:-1" > /sys/kernel/mm/swap/tiers
+ # cat /sys/kernel/mm/swap/tiers
+ Name Idx PrioStart PrioEnd
+ HDD 0 50 32767
+ NET 1 -1 49
+
+**2. Adding a New Tier (split)**
+
+A new tier 'SSD' is added at priority 100, splitting the existing 'HDD' tier.
+The ranges are automatically recalculated:
+
+* 'SSD' takes the top range (100 to SHRT_MAX).
+* 'HDD' is adjusted to the range between 'NET' and 'SSD' (50 to 99).
+* 'NET' remains unchanged (-1 to 49).
+
+::
+
+ # echo "+SSD:100" > /sys/kernel/mm/swap/tiers
+ # cat /sys/kernel/mm/swap/tiers
+ Name Idx PrioStart PrioEnd
+ SSD 2 100 32767
+ HDD 0 50 99
+ NET 1 -1 49
+
+**3. Removal (merge)**
+
+Tiers can be removed using the '-' prefix.
+::
+
+ # echo "-SSD" > /sys/kernel/mm/swap/tiers
+
+When a tier is removed, its priority range is merged into the adjacent
+tier. The merge direction is always upward (the tier below expands),
+except when the lowest tier is removed — in that case the tier above
+shifts its starting priority down to -1 to maintain full range coverage.
+
+::
+
+ Initial state:
+ Name Idx PrioStart PrioEnd
+ SSD 2 100 32767
+ HDD 1 50 99
+ NET 0 -1 49
+
+ # echo "-SSD" > /sys/kernel/mm/swap/tiers
+
+ Name Idx PrioStart PrioEnd
+ HDD 1 50 32767 <- merged with SSD's range
+ NET 0 -1 49
+
+ # echo "-NET" > /sys/kernel/mm/swap/tiers
+
+ Name Idx PrioStart PrioEnd
+ HDD 1 -1 32767 <- shifted down to -1
+
+**4. Interaction with Active Swap Devices**
+
+If a swap device is active (swapon), the tier covering that device's
+priority cannot be removed. Splitting the active tier's range is only
+allowed above the device's priority.
+
+Assume a swap device is active at priority 60 (inside 'HDD' tier).
+
+::
+
+ # swapon -p 60 /dev/zram0
+
+ Name Idx PrioStart PrioEnd
+ HDD 0 50 32767
+ NET 1 -1 49
+
+ # echo "-HDD" > /sys/kernel/mm/swap/tiers
+ -bash: echo: write error: Device or resource busy
+
+ # echo "+SSD:60" > /sys/kernel/mm/swap/tiers
+ -bash: echo: write error: Device or resource busy
+
+ # echo "+SSD:100" > /sys/kernel/mm/swap/tiers
+
+ Name Idx PrioStart PrioEnd
+ SSD 2 100 32767
+ HDD 0 50 99 <- device (prio 60) stays here
+ NET 1 -1 49
diff --git a/MAINTAINERS b/MAINTAINERS
index d1bb3b4b1e1c..4293048be1ab 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17052,6 +17052,7 @@ L: linux-mm@kvack.org
S: Maintained
F: Documentation/ABI/testing/sysfs-kernel-mm-swap
F: Documentation/mm/swap-table.rst
+F: Documentation/mm/swap-tier.rst
F: include/linux/swap.h
F: include/linux/swapfile.h
F: include/linux/swapops.h
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 6d72778e6cc3..21286945770a 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -250,6 +250,7 @@ struct swap_info_struct {
struct percpu_ref users; /* indicate and keep swap device valid. */
unsigned long flags; /* SWP_USED etc: see above */
signed short prio; /* swap priority of this type */
+ int tier_mask; /* swap tier mask */
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
unsigned int max; /* size of this swap device */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 762d9ca6ad5a..2f382d4dcbdc 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -1063,7 +1063,7 @@ static ssize_t tiers_store(struct kobject *kobj,
}
}
- if (!swap_tiers_validate()) {
+ if (!swap_tiers_update()) {
ret = -EINVAL;
goto restore;
}
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index ac7a3c2a48cb..6b57cadb3e95 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -38,6 +38,8 @@ static LIST_HEAD(swap_tier_inactive_list);
(!list_is_first(&(tier)->list, &swap_tier_active_list) ? \
list_prev_entry((tier), list)->prio - 1 : SHRT_MAX)
+#define MASK_TO_TIER(mask) (&swap_tiers[__ffs((mask))])
+
#define for_each_tier(tier, idx) \
for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
idx++, tier = &swap_tiers[idx])
@@ -59,6 +61,26 @@ static bool swap_tier_is_active(void)
return !list_empty(&swap_tier_active_list);
}
+static bool swap_tier_prio_in_range(struct swap_tier *tier, short prio)
+{
+ if (tier->prio <= prio && TIER_END_PRIO(tier) >= prio)
+ return true;
+
+ return false;
+}
+
+static bool swap_tier_prio_is_used(short prio)
+{
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (tier->prio == prio)
+ return true;
+ }
+
+ return false;
+}
+
static struct swap_tier *swap_tier_lookup(const char *name)
{
struct swap_tier *tier;
@@ -99,6 +121,7 @@ void swap_tiers_init(void)
int idx;
BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER);
+ BUILD_BUG_ON(MAX_SWAPTIER > TIER_DEFAULT_IDX);
for_each_tier(tier, idx) {
INIT_LIST_HEAD(&tier->list);
@@ -149,17 +172,29 @@ static struct swap_tier *swap_tier_prepare(const char *name, short prio)
return tier;
}
-static int swap_tier_check_range(short prio)
+static int swap_tier_can_split_range(short new_prio)
{
+ struct swap_info_struct *p;
struct swap_tier *tier;
lockdep_assert_held(&swap_lock);
lockdep_assert_held(&swap_tier_lock);
- for_each_active_tier(tier) {
- /* No overwrite */
- if (tier->prio == prio)
- return -EINVAL;
+ plist_for_each_entry(p, &swap_active_head, list) {
+ if (p->tier_mask == TIER_DEFAULT_MASK)
+ continue;
+
+ tier = MASK_TO_TIER(p->tier_mask);
+ if (!swap_tier_prio_in_range(tier, new_prio))
+ continue;
+
+ /*
+ * Device sits in a tier that spans new_prio;
+ * splitting here would reassign it to a
+ * different tier.
+ */
+ if (p->prio >= new_prio)
+ return -EBUSY;
}
return 0;
@@ -199,7 +234,11 @@ int swap_tiers_add(const char *name, int prio)
if (!swap_tier_validate_name(name))
return -EINVAL;
- ret = swap_tier_check_range(prio);
+ /* No overwrite */
+ if (swap_tier_prio_is_used(prio))
+ return -EBUSY;
+
+ ret = swap_tier_can_split_range(prio);
if (ret)
return ret;
@@ -226,6 +265,11 @@ int swap_tiers_remove(const char *name)
if (!tier)
return -EINVAL;
+ /* Simulate adding a tier to check for conflicts */
+ ret = swap_tier_can_split_range(tier->prio);
+ if (ret)
+ return ret;
+
/* Removing DEF_SWAP_PRIO merges into the higher tier. */
if (!list_is_singular(&swap_tier_active_list)
&& tier->prio == DEF_SWAP_PRIO)
@@ -236,13 +280,15 @@ int swap_tiers_remove(const char *name)
return ret;
}
-static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
/*
- * XXX: When multiple operations (adds and removes) are submitted in a
- * single write, reverting each individually on failure is complex and
- * error-prone. Instead, snapshot the entire state beforehand and
- * restore it wholesale if any operation fails.
+ * XXX: Static global snapshot buffer for batch operations. Small
+ * and used once per write, so a static global is not bad.
+ * When multiple adds/removes are submitted in a single write,
+ * reverting each individually on failure is error-prone. Instead,
+ * snapshot beforehand and restore wholesale if any operation fails.
*/
+static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
+
void swap_tiers_snapshot(void)
{
BUILD_BUG_ON(sizeof(swap_tiers_snap) != sizeof(swap_tiers));
@@ -282,10 +328,30 @@ void swap_tiers_snapshot_restore(void)
}
}
-bool swap_tiers_validate(void)
+void swap_tiers_assign_dev(struct swap_info_struct *swp)
{
struct swap_tier *tier;
+ lockdep_assert_held(&swap_lock);
+
+ for_each_active_tier(tier) {
+ if (swap_tier_prio_in_range(tier, swp->prio)) {
+ swp->tier_mask = TIER_MASK(tier);
+ return;
+ }
+ }
+
+ swp->tier_mask = TIER_DEFAULT_MASK;
+}
+
+bool swap_tiers_update(void)
+{
+ struct swap_tier *tier;
+ struct swap_info_struct *swp;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
/*
* Initial setting might not cover DEF_SWAP_PRIO.
* Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
@@ -298,5 +364,16 @@ bool swap_tiers_validate(void)
return false;
}
+ /*
+ * If applied initially, the swap tier_mask may change
+ * from the default value.
+ */
+ plist_for_each_entry(swp, &swap_active_head, list) {
+ /* Tier is already configured */
+ if (swp->tier_mask != TIER_DEFAULT_MASK)
+ break;
+ swap_tiers_assign_dev(swp);
+ }
+
return true;
}
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
index a1395ec02c24..3e355f857363 100644
--- a/mm/swap_tier.h
+++ b/mm/swap_tier.h
@@ -5,8 +5,15 @@
#include <linux/types.h>
#include <linux/spinlock.h>
+/* Forward declarations */
+struct swap_info_struct;
+
extern spinlock_t swap_tier_lock;
+#define TIER_ALL_MASK (~0)
+#define TIER_DEFAULT_IDX (31)
+#define TIER_DEFAULT_MASK (1U << TIER_DEFAULT_IDX)
+
/* Initialization and application */
void swap_tiers_init(void);
ssize_t swap_tiers_sysfs_show(char *buf);
@@ -16,5 +23,9 @@ int swap_tiers_remove(const char *name);
void swap_tiers_snapshot(void);
void swap_tiers_snapshot_restore(void);
-bool swap_tiers_validate(void);
+bool swap_tiers_update(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+
#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 3f7225dbc6cd..9a86ebe992f4 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -3036,6 +3036,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
/* Add back to available list */
add_to_avail_list(si, true);
+
+ swap_tiers_assign_dev(si);
}
/*
--
2.48.1
^ permalink raw reply related
* [PATCH v9 1/6] mm: swap: introduce swap tier infrastructure
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
In-Reply-To: <20260620181635.299364-1-youngjun.park@lge.com>
This patch introduces the "Swap tier" concept, which serves as an
abstraction layer for managing swap devices based on their performance
characteristics (e.g., NVMe, HDD, Network swap).
Swap tiers are user-named groups representing priority ranges.
Tier names must consist of alphanumeric characters and underscores.
These tiers collectively cover the entire priority space from -1
(`DEF_SWAP_PRIO`) to `SHRT_MAX`.
To configure tiers, a new sysfs interface is exposed at
/sys/kernel/mm/swap/tiers. The input parser evaluates commands from
left to right and supports batch input, allowing users to add or remove
multiple tiers in a single write operation.
Tier management enforces continuous priority ranges anchored by start
priorities. Operations trigger range splitting or merging, but overwriting
start priorities is forbidden. Merging expands lower tiers upwards to
preserve configured start priorities, except when removing `DEF_SWAP_PRIO`,
which merges downwards.
Suggested-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
MAINTAINERS | 2 +
mm/Kconfig | 12 ++
mm/Makefile | 2 +-
mm/swap.h | 4 +
mm/swap_state.c | 74 ++++++++++++
mm/swap_tier.c | 302 ++++++++++++++++++++++++++++++++++++++++++++++++
mm/swap_tier.h | 20 ++++
mm/swapfile.c | 8 +-
8 files changed, 420 insertions(+), 4 deletions(-)
create mode 100644 mm/swap_tier.c
create mode 100644 mm/swap_tier.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 65bd4328fe05..d1bb3b4b1e1c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17060,6 +17060,8 @@ F: mm/swap.c
F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
+F: mm/swap_tier.c
+F: mm/swap_tier.h
F: mm/swapfile.c
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
diff --git a/mm/Kconfig b/mm/Kconfig
index 776b67c66e82..5343937f3da9 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -19,6 +19,18 @@ menuconfig SWAP
used to provide more virtual memory than the actual RAM present
in your computer. If unsure say Y.
+config NR_SWAP_TIERS
+ int "Number of swap device tiers"
+ depends on SWAP
+ default 4
+ range 1 31
+ help
+ Sets the number of swap device tiers. Swap devices are
+ grouped into tiers based on their priority, allowing the
+ system to prefer faster devices over slower ones.
+
+ If unsure, say 4.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061..29cb1e778285 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -75,7 +75,7 @@ ifdef CONFIG_MMU
obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
endif
-obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o
+obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o swap_tier.o
obj-$(CONFIG_ZSWAP) += zswap.o
obj-$(CONFIG_HAS_DMA) += dmapool.o
obj-$(CONFIG_HUGETLBFS) += hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
diff --git a/mm/swap.h b/mm/swap.h
index 77d2d14eda42..d6c5f5d31f63 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -34,6 +34,10 @@ extern int page_cluster;
#define swap_entry_order(order) 0
#endif
+#define DEF_SWAP_PRIO -1
+
+extern spinlock_t swap_lock;
+extern struct plist_head swap_active_head;
extern struct swap_info_struct *swap_info[];
/*
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 9c3a5cf99778..762d9ca6ad5a 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -25,6 +25,7 @@
#include "internal.h"
#include "swap_table.h"
#include "swap.h"
+#include "swap_tier.h"
/*
* swapper_space is a fiction, retained to simplify the path through
@@ -1007,8 +1008,81 @@ static ssize_t vma_ra_enabled_store(struct kobject *kobj,
}
static struct kobj_attribute vma_ra_enabled_attr = __ATTR_RW(vma_ra_enabled);
+static ssize_t tiers_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return swap_tiers_sysfs_show(buf);
+}
+
+static ssize_t tiers_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ char *p, *token, *name, *tmp;
+ int ret = 0;
+ short prio;
+
+ tmp = kstrdup(buf, GFP_KERNEL);
+ if (!tmp)
+ return -ENOMEM;
+
+ spin_lock(&swap_lock);
+ spin_lock(&swap_tier_lock);
+ swap_tiers_snapshot();
+
+ p = tmp;
+ while ((token = strsep(&p, ", \t\n")) != NULL) {
+ if (!*token)
+ continue;
+
+ switch (token[0]) {
+ case '+':
+ name = token + 1;
+ token = strchr(name, ':');
+ if (!token) {
+ ret = -EINVAL;
+ goto restore;
+ }
+ *token++ = '\0';
+ if (kstrtos16(token, 10, &prio)) {
+ ret = -EINVAL;
+ goto restore;
+ }
+ ret = swap_tiers_add(name, prio);
+ if (ret)
+ goto restore;
+ break;
+ case '-':
+ ret = swap_tiers_remove(token + 1);
+ if (ret)
+ goto restore;
+ break;
+ default:
+ ret = -EINVAL;
+ goto restore;
+ }
+ }
+
+ if (!swap_tiers_validate()) {
+ ret = -EINVAL;
+ goto restore;
+ }
+ goto out;
+
+restore:
+ swap_tiers_snapshot_restore();
+out:
+ spin_unlock(&swap_tier_lock);
+ spin_unlock(&swap_lock);
+ kfree(tmp);
+ return ret ? ret : count;
+}
+
+static struct kobj_attribute tier_attr = __ATTR_RW(tiers);
+
static struct attribute *swap_attrs[] = {
&vma_ra_enabled_attr.attr,
+ &tier_attr.attr,
NULL,
};
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
new file mode 100644
index 000000000000..ac7a3c2a48cb
--- /dev/null
+++ b/mm/swap_tier.c
@@ -0,0 +1,302 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/swap.h>
+#include <linux/memcontrol.h>
+#include "memcontrol-v1.h"
+#include <linux/sysfs.h>
+#include <linux/plist.h>
+
+#include "swap.h"
+#include "swap_tier.h"
+
+#define MAX_SWAPTIER CONFIG_NR_SWAP_TIERS
+#define MAX_TIERNAME 16
+
+/*
+ * struct swap_tier - structure representing a swap tier.
+ *
+ * @name: name of the swap_tier.
+ * @prio: starting value of priority.
+ * @list: linked list of tiers.
+ */
+static struct swap_tier {
+ char name[MAX_TIERNAME];
+ short prio;
+ struct list_head list;
+} swap_tiers[MAX_SWAPTIER];
+
+DEFINE_SPINLOCK(swap_tier_lock);
+/* active swap priority list, sorted in descending order */
+static LIST_HEAD(swap_tier_active_list);
+/* unused swap_tier object */
+static LIST_HEAD(swap_tier_inactive_list);
+
+#define TIER_IDX(tier) ((tier) - swap_tiers)
+#define TIER_MASK(tier) (1U << TIER_IDX(tier))
+#define TIER_INACTIVE_PRIO (DEF_SWAP_PRIO - 1)
+#define TIER_IS_ACTIVE(tier) ((tier->prio) != TIER_INACTIVE_PRIO)
+#define TIER_END_PRIO(tier) \
+ (!list_is_first(&(tier)->list, &swap_tier_active_list) ? \
+ list_prev_entry((tier), list)->prio - 1 : SHRT_MAX)
+
+#define for_each_tier(tier, idx) \
+ for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
+ idx++, tier = &swap_tiers[idx])
+
+#define for_each_active_tier(tier) \
+ list_for_each_entry(tier, &swap_tier_active_list, list)
+
+#define for_each_inactive_tier(tier) \
+ list_for_each_entry(tier, &swap_tier_inactive_list, list)
+
+/*
+ * Naming Convention:
+ * swap_tiers_*() - Public/exported functions
+ * swap_tier_*() - Private/internal functions
+ */
+
+static bool swap_tier_is_active(void)
+{
+ return !list_empty(&swap_tier_active_list);
+}
+
+static struct swap_tier *swap_tier_lookup(const char *name)
+{
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (!strcmp(tier->name, name))
+ return tier;
+ }
+
+ return NULL;
+}
+
+/* Insert new tier into the active list sorted by priority. */
+static void swap_tier_activate(struct swap_tier *new)
+{
+ struct list_head *pos = &swap_tier_active_list;
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (tier->prio <= new->prio) {
+ pos = &tier->list;
+ break;
+ }
+ }
+
+ list_add_tail(&new->list, pos);
+}
+
+static void swap_tier_inactivate(struct swap_tier *tier)
+{
+ list_move(&tier->list, &swap_tier_inactive_list);
+ tier->prio = TIER_INACTIVE_PRIO;
+}
+
+void swap_tiers_init(void)
+{
+ struct swap_tier *tier;
+ int idx;
+
+ BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER);
+
+ for_each_tier(tier, idx) {
+ INIT_LIST_HEAD(&tier->list);
+ swap_tier_inactivate(tier);
+ }
+}
+
+ssize_t swap_tiers_sysfs_show(char *buf)
+{
+ struct swap_tier *tier;
+ ssize_t len = 0;
+
+ len += sysfs_emit_at(buf, len, "%-16s %-5s %-11s %-11s\n",
+ "Name", "Idx", "PrioStart", "PrioEnd");
+
+ spin_lock(&swap_tier_lock);
+ for_each_active_tier(tier) {
+ len += sysfs_emit_at(buf, len, "%-16s %-5td %-11d %-11d\n",
+ tier->name,
+ TIER_IDX(tier),
+ tier->prio,
+ TIER_END_PRIO(tier));
+ }
+ spin_unlock(&swap_tier_lock);
+
+ return len;
+}
+
+static struct swap_tier *swap_tier_prepare(const char *name, short prio)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ if (prio < DEF_SWAP_PRIO)
+ return ERR_PTR(-EINVAL);
+
+ if (list_empty(&swap_tier_inactive_list))
+ return ERR_PTR(-ENOSPC);
+
+ tier = list_first_entry(&swap_tier_inactive_list,
+ struct swap_tier, list);
+
+ list_del_init(&tier->list);
+ strscpy(tier->name, name, MAX_TIERNAME);
+ tier->prio = prio;
+
+ return tier;
+}
+
+static int swap_tier_check_range(short prio)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ for_each_active_tier(tier) {
+ /* No overwrite */
+ if (tier->prio == prio)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static bool swap_tier_validate_name(const char *name)
+{
+ int len;
+
+ if (!name || !*name)
+ return false;
+
+ len = strlen(name);
+ if (len >= MAX_TIERNAME)
+ return false;
+
+ while (*name) {
+ if (!isalnum(*name) && *name != '_')
+ return false;
+ name++;
+ }
+ return true;
+}
+
+int swap_tiers_add(const char *name, int prio)
+{
+ int ret;
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ /* Duplicate check */
+ if (swap_tier_lookup(name))
+ return -EEXIST;
+
+ if (!swap_tier_validate_name(name))
+ return -EINVAL;
+
+ ret = swap_tier_check_range(prio);
+ if (ret)
+ return ret;
+
+ tier = swap_tier_prepare(name, prio);
+ if (IS_ERR(tier)) {
+ ret = PTR_ERR(tier);
+ return ret;
+ }
+
+ swap_tier_activate(tier);
+
+ return ret;
+}
+
+int swap_tiers_remove(const char *name)
+{
+ int ret = 0;
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ tier = swap_tier_lookup(name);
+ if (!tier)
+ return -EINVAL;
+
+ /* Removing DEF_SWAP_PRIO merges into the higher tier. */
+ if (!list_is_singular(&swap_tier_active_list)
+ && tier->prio == DEF_SWAP_PRIO)
+ list_prev_entry(tier, list)->prio = DEF_SWAP_PRIO;
+
+ swap_tier_inactivate(tier);
+
+ return ret;
+}
+
+static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
+/*
+ * XXX: When multiple operations (adds and removes) are submitted in a
+ * single write, reverting each individually on failure is complex and
+ * error-prone. Instead, snapshot the entire state beforehand and
+ * restore it wholesale if any operation fails.
+ */
+void swap_tiers_snapshot(void)
+{
+ BUILD_BUG_ON(sizeof(swap_tiers_snap) != sizeof(swap_tiers));
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ memcpy(swap_tiers_snap, swap_tiers, sizeof(swap_tiers));
+}
+
+void swap_tiers_snapshot_restore(void)
+{
+ struct swap_tier *tier;
+ int idx;
+
+ lockdep_assert_held(&swap_lock);
+ lockdep_assert_held(&swap_tier_lock);
+
+ memcpy(swap_tiers, swap_tiers_snap, sizeof(swap_tiers));
+
+ INIT_LIST_HEAD(&swap_tier_active_list);
+ INIT_LIST_HEAD(&swap_tier_inactive_list);
+
+ /*
+ * memcpy copied snapshot-time list pointers into each tier's
+ * list_head. Those references are stale, so re-init every
+ * tier before re-linking into the freshly initialised global
+ * lists below.
+ */
+ for_each_tier(tier, idx) {
+ INIT_LIST_HEAD(&tier->list);
+
+ if (TIER_IS_ACTIVE(tier))
+ swap_tier_activate(tier);
+ else
+ swap_tier_inactivate(tier);
+ }
+}
+
+bool swap_tiers_validate(void)
+{
+ struct swap_tier *tier;
+
+ /*
+ * Initial setting might not cover DEF_SWAP_PRIO.
+ * Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
+ */
+ if (swap_tier_is_active()) {
+ tier = list_last_entry(&swap_tier_active_list,
+ struct swap_tier, list);
+
+ if (tier->prio != DEF_SWAP_PRIO)
+ return false;
+ }
+
+ return true;
+}
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
new file mode 100644
index 000000000000..a1395ec02c24
--- /dev/null
+++ b/mm/swap_tier.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SWAP_TIER_H
+#define _SWAP_TIER_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+
+extern spinlock_t swap_tier_lock;
+
+/* Initialization and application */
+void swap_tiers_init(void);
+ssize_t swap_tiers_sysfs_show(char *buf);
+
+int swap_tiers_add(const char *name, int prio);
+int swap_tiers_remove(const char *name);
+
+void swap_tiers_snapshot(void);
+void swap_tiers_snapshot_restore(void);
+bool swap_tiers_validate(void);
+#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index e3d126602a1e..3f7225dbc6cd 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -48,6 +48,7 @@
#include "swap_table.h"
#include "internal.h"
#include "swap.h"
+#include "swap_tier.h"
static void swap_range_alloc(struct swap_info_struct *si,
unsigned int nr_entries);
@@ -63,7 +64,8 @@ static void move_cluster(struct swap_info_struct *si,
*
* Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.
*/
-static DEFINE_SPINLOCK(swap_lock);
+DEFINE_SPINLOCK(swap_lock);
+
static unsigned int nr_swapfiles;
atomic_long_t nr_swap_pages;
/*
@@ -74,7 +76,6 @@ atomic_long_t nr_swap_pages;
EXPORT_SYMBOL_GPL(nr_swap_pages);
/* protected with swap_lock. reading in vm_swap_full() doesn't need lock */
long total_swap_pages;
-#define DEF_SWAP_PRIO -1
unsigned long swapfile_maximum_size;
#ifdef CONFIG_MIGRATION
bool swap_migration_ad_supported;
@@ -87,7 +88,7 @@ static const char Bad_offset[] = "Bad swap offset entry ";
* all active swap_info_structs
* protected with swap_lock, and ordered by priority.
*/
-static PLIST_HEAD(swap_active_head);
+PLIST_HEAD(swap_active_head);
/*
* all available (active, not full) swap_info_structs
@@ -3988,6 +3989,7 @@ static int __init swapfile_init(void)
swap_migration_ad_supported = true;
#endif /* CONFIG_MIGRATION */
+ swap_tiers_init();
return 0;
}
subsys_initcall(swapfile_init);
--
2.48.1
^ permalink raw reply related
* [PATCH v9 0/6] mm/swap, memcg: Introduce swap tiers for cgroup based swap control
From: Youngjun Park @ 2026-06-20 18:16 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, nphamcs, baoquan.he, baohua, yosry, gunho.lee,
taejoon.song, hyungjun.cho, mkoutny, baver.bae, matia.kim
This is the v9 series of the swap tier patchset.
The main change in this version is the addition of selftests for the tier
interfaces, requested by Nhat; see the changelog below for the other changes.
I designed the test cases and wrote the selftests with some AI assistance.
For context, the bulk of the series is unchanged since v8, with great thanks
to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it.
The main change in v8 was the interface change to use memory.swap.tiers.max
with '0' (disable) and 'max' (enable) values. This mechanism was suggested
by Shakeel and Yosry.
This change allows for future extensions to control swap between tiers and
aligns better with existing memcg interfaces. It is confined to patch #3's
user-facing interface; internally, patch #3 still uses the existing mask
processing method, which is implementation-efficient.
We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their
valuable feedback.
Here is a brief summary of our tentative conclusions. Please correct me
if anything is misrepresented (details in references):
* Zswap tiering [2]:
Tiering applies only to the vswap + zswap combo. Zswap itself will
not be tiered, as the current architecture requires a physical device
for zswap allocation.
* Vswap tiering [3]:
Vswap should be handled transparently to the user. Vswap itself will
not be tiered. But, someday supported if there is strong and real usecase.
* Relationship with zswap.writeback [4]:
If zswap tiering is introduced, it could replace the zswap-only tier.
However, since zswap cannot be tiered independently, it is still
needed for non-vswap cases. Separately, the internal logic could
potentially be integrated into the tiering logic.
* Tier demotion [5]:
A separate interface like memory.swap.tiers.demotion might be needed.
For now, we only support 0/max to enable/disable tiers. In the future,
we could introduce an "auto" mode to automatically scale the limit
based on swapfile size and memory.swap.max, similar to the direction
memory tiering is heading in.
I plan to apply the swap tier infrastructure and the first use case
(cgroup-based swap control) first, and continue following up on the
discussions above.
Overview
========
Swap Tiers group swap devices into performance classes (e.g. NVMe,
HDD, Network) and allow per-memcg selection of which tiers to use.
This mechanism was suggested by Chris Li.
Design Rationale
================
Swap tier selection is attached to memcg. A child cgroup may select a
subset of the parent's allowed tiers.
This
- Preserves cgroup inheritance semantics (boundary at parent,
refinement at child).
- Reuses memcg, which already groups processes and enforces
hierarchical memory limits.
- Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
- Avoids introducing a parallel swap control hierarchy.
Placing tier control outside memcg (e.g., via BPF, syscalls, or
madvise) would allow swap preference to diverge from the memcg
hierarchy. Integrating it into memcg keeps the swap policy
consistent with existing memory ownership semantics. There are
also real use cases built around memcg.
In the future, this can be extended to other interfaces to cover
additional use cases.
I believe a memcg-based swap control is a good starting point
before such extensions.
Use Cases
=========
#1: Latency separation (our primary deployment scenario)
[ / ]
|
+-- latency-sensitive workload (fast tier)
+-- background workload (slow tier)
The parent defines the memory boundary.
Each workload selects a swap tier via memory.swap.tiers.max according to
latency requirements.
This prevents latency-sensitive workloads from being swapped to
slow devices used by background workloads.
#2: Per-VM swap selection (Chris Li's deployment scenario)
[ / ]
|
+-- [ Job on VM ] (tiers: zswap, SSD)
|
+-- [ VMM guest memory ] (tiers: SSD)
The parent (job) has access to both zswap and SSD tiers.
The child (VMM guest memory) selects SSD as its swap tier via
memory.swap.tiers.max. In this deployment, swap device selection
happens at the child level from the parent's available set.
#3: Tier isolation for reduced contention (hypothetical)
[ / ] (tiers: A, B)
|
+-- workload X (tiers: A)
+-- workload Y (tiers: B)
Each child uses a different tier. Since swap paths are separated
per tier, synchronization overhead between the two workloads is
reduced.
Future extension (Follow up)
============================
#1: Intra-tier distribution policy:
Currently, swap devices with the same priority are allocated in a
round-robin fashion. Per-tier policy files under
/sys/kernel/mm/swap/tiers/ can control how devices within a tier
are selected (e.g. round-robin, weighted).
#2: Inter-tier promotion and demotion:
Promotion and demotion apply between tiers, not within a single
tier. The current interface defines only tier assignment; it does
not yet define when or how pages move between tiers. Two triggering
models are possible:
(a) User-triggered: userspace explicitly initiates migration between
tiers (e.g. via a new interface or existing move_pages semantics).
(b) Kernel-triggered: the kernel moves pages between tiers at
appropriate points such as reclaim or refault.
#3: Per-VMA, per-process swap and BPF:
Not just for memcg based swap, possible to extend Per-VMA or per-process
swap. Or we can use it as BPF program.
#4: Zswap and vswap tiering:
Tiering applies to the vswap + zswap combination.
#5: Vswap on/off control:
Currently not supported. If a strong use case arises where vswap needs
to be controlled by memcg, the tier interface could be used for it.
#6: Per-CPU swap allocation caching:
Per-si/per-tier per-CPU caching of allocations to reduce contention in
the tier-filtered allocation path.
Experimentation
===============
Tested on our internal platform using NBD as a separate swap tier.
Our first production's simple usecase.
Without tiers:
- No selective control over flash wear
- Cannot selectively assign NBD to specific applications
Cold launch improvement (preloaded vs. baseline):
- App A: 13.17s -> 4.18s (68%)
- App B: 5.60s -> 1.12s (80%)
- App C: 10.25s -> 2.00s (80%)
Performance impact with no tiers configured:
<1% regression in kernel build and vm-scalability benchmarks
Change log
===========
v9
- Added selftests (per Nhat's request):
- selftests/mm: swap tier configuration test for /sys/kernel/mm/swap/tiers.(#5 patch)
- selftests/cgroup: swap tier routing test for memory.swap.tiers.max. (#6 patch)
- Removed the redundant rcu_read_lock() around the memcg tier-mask tree walk;
for_each_mem_cgroup_tree() already takes RCU internally and returns each
memcg with a reference held. (#3 patch)
- Sashiko review: swap_sync_discard() now honors the memcg tier mask, so the
discard fallback no longer drains clusters on disallowed tiers. Left as-is:
the cgroup tree walk under spinlock (bounded by cgroup.max.descendants, an
admin-controlled limit, and triggered only by infrequent tier writes) and
the pre-existing swap_avail_lock drop in swap_alloc_slow(). (#4 patch)
- Dropped patch #4's Reviewed-by tags (Nhat, Kairui, Baoquan): the
swap_sync_discard() change above modifies that patch (the tier mask is now
passed as a parameter into the alloc and discard paths), so the earlier tags
no longer apply. Re-review would be welcome.
- v8 link: https://lore.kernel.org/linux-mm/20260617053447.2831896-1-youngjun.park@lge.com/
v8
- Changed the memcg interface to memory.swap.tiers.max.
Values are '0' (disable) and 'max' (enable). Default is 'max'.
- Addressed Sashiko's review: Update the mask value atomically at once and
read the mask value while grabbing lock.
- Collected review tags from Kairui and Nhat.
- Rebase on recent mm-new
- v7 link: https://lore.kernel.org/linux-mm/20260527062247.3440692-1-youngjun.park@lge.com/
v7
- Collect Baoquan's review tag
- Baoquan's feedback on fixing improper comment
- Minor code adjustments per Baoquan's feedback.
- Rebase on recent mm-new
- v6 link: https://lore.kernel.org/linux-mm/20260421055323.940344-1-youngjun.park@lge.com/
v6
- Sashiko AI review fixes
- Fix batch parsing error path to restore snapshot before exit
- Reject overlong tier names to prevent truncated duplicates
- Avoid restoring raw list_head via memcpy (stale pointer risk)
- Ensure early parse errors do not skip DEF_SWAP_PRIO validation
- Use (1U << TIER_DEFAULT_IDX) to avoid signed shift UB
- Defer tier mask inheritance to css_online() to close race window
- Add READ_ONCE()/WRITE_ONCE() for tier mask accesses
- Other fixes
- Fix build error reintroduced due to missing v5 change
- Fix WARNING in folio_tier_effective_mask by adding rcu_read_lock()
- default number of swap tier max (change to 32->31, for reserving last bit)
- commit message refinement.
- rebased on recently mm-new
- v5 link: https://lore.kernel.org/linux-mm/20260325175453.2523280-1-youngjun.park@lge.com/
v5
- Fixed build errors reported in v4
- rebased on up to date mm-new
- Minor cleanups
- Design docs with validation (by Shakeel Butt discussion)
- v4 link : https://lore.kernel.org/linux-mm/20260217000950.4015880-1-youngjun.park@lge.com/
v4
- Simplified control flow and indentation
- Added CONFIG option for MAX_SWAPTIER (default: 4)
- Added memory.swap.tiers.effective interface
- Reworked save/restore logic into snapshot/rollback model
- Removed tier priority modification support (deferred)
- Improved validation and fixed edge cases
- Rebased onto latest mm-new
- RFC v3 link: https://lore.kernel.org/linux-mm/20260131125454.3187546-1-youngjun.park@lge.com/
RFC v1 ~ v3
- Change the direction after discussion with Chris-Li
- apply some LPC feedback.
- RFC v2 - https://lore.kernel.org/linux-mm/20260126065242.1221862-1-youngjun.park@lge.com/
- RFC v1 - https://lore.kernel.org/linux-mm/20251109124947.1101520-1-youngjun.park@lge.com/
Earlier Approach (per cgroup swap priority)
- v1: https://lore.kernel.org/linux-mm/20250716202006.3640584-1-youngjun.park@lge.com/
- RFC: https://lore.kernel.org/linux-mm/aEvLjEInMQC7hEyh@yjaykim-PowerEdge-T330/T/#mbbb6a5e9e30843097e1f5f65fb98f31d582b973d
Reference
=========
[1] https://lore.kernel.org/linux-doc/aiw2p5ANjsQUCIHA@linux.dev/
[2] https://lore.kernel.org/linux-mm/CAKEwX=Nz9SWcEVQGQjHN8P8OANJY4BG0w+iQOzoNOWuteoVjAg@mail.gmail.com/
[3] https://lore.kernel.org/cgroups/CAKEwX=O23a4iWBZoewKVb8QqODte6r3Xijckw3_oCJNoiO9M5A@mail.gmail.com/
[4] https://lore.kernel.org/linux-mm/CAO9r8zOg0OP1Ak1v7CRzSfQq0D8b4Dw+_T0Jui6YTM_KwQQNOA@mail.gmail.com/
[5] https://lore.kernel.org/linux-mm/CAO9r8zNi4-QC4sUi=xXWHt9WMeG39mbyoSf8kON9vLOZ=cbCmw@mail.gmail.com/
Youngjun Park (6):
mm: swap: introduce swap tier infrastructure
mm: swap: associate swap devices with tiers
mm: memcontrol: add interface for swap tier selection
mm: swap: filter swap allocation by memcg tier mask
selftests/mm: add a swap tier configuration test
selftests/cgroup: add a swap tier routing test
Documentation/admin-guide/cgroup-v2.rst | 20 +
Documentation/mm/index.rst | 1 +
Documentation/mm/swap-tier.rst | 159 ++++++
MAINTAINERS | 3 +
include/linux/memcontrol.h | 5 +
include/linux/swap.h | 1 +
mm/Kconfig | 12 +
mm/Makefile | 2 +-
mm/memcontrol.c | 67 +++
mm/swap.h | 4 +
mm/swap_state.c | 75 +++
mm/swap_tier.c | 477 +++++++++++++++++
mm/swap_tier.h | 76 +++
mm/swapfile.c | 34 +-
tools/testing/selftests/cgroup/.gitignore | 1 +
tools/testing/selftests/cgroup/Makefile | 2 +
tools/testing/selftests/cgroup/config | 2 +
.../selftests/cgroup/test_swap_tiers.c | 500 ++++++++++++++++++
tools/testing/selftests/mm/.gitignore | 1 +
tools/testing/selftests/mm/Makefile | 1 +
tools/testing/selftests/mm/config | 2 +
tools/testing/selftests/mm/run_vmtests.sh | 5 +
tools/testing/selftests/mm/swap_tier.c | 323 +++++++++++
23 files changed, 1762 insertions(+), 11 deletions(-)
create mode 100644 Documentation/mm/swap-tier.rst
create mode 100644 mm/swap_tier.c
create mode 100644 mm/swap_tier.h
create mode 100644 tools/testing/selftests/cgroup/test_swap_tiers.c
create mode 100644 tools/testing/selftests/mm/swap_tier.c
base-commit: cdad4d4e4fc2e5acb9a8b2cac9af6ce87c92656f
--
2.48.1
^ permalink raw reply
* [PATCH] Docs/admin-guide/cgroup-v2: fix memory.stat doc details
From: Doehyun Baek @ 2026-06-20 12:27 UTC (permalink / raw)
To: Tejun Heo, Jonathan Corbet
Cc: Johannes Weiner, Michal Koutný, Andrew Morton, Shakeel Butt,
Roman Gushchin, Yosry Ahmed, Nhat Pham, cgroups, linux-doc,
linux-kernel, Doehyun Baek
Fix minor cgroup v2 memory.stat documentation issues. Correct the
vmalloc per-node marker now that vmalloc uses the native NR_VMALLOC node
stat, and document zswap_incomp as a byte-valued memory amount instead
of as a page counter.
Fixes: c466412c73c3 ("mm: memcontrol: switch to native NR_VMALLOC vmstat counter")
Fixes: 5ad41a38c364 ("mm: zswap: add per-memcg stat for incompressible pages")
Signed-off-by: Doehyun Baek <doehyunbaek@gmail.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 993446ab66d0..ce6741f78f4f 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1570,7 +1570,7 @@ The following nested keys are defined.
sock (npn)
Amount of memory used in network transmission buffers
- vmalloc (npn)
+ vmalloc
Amount of memory used for vmap backed memory.
shmem
@@ -1735,7 +1735,7 @@ The following nested keys are defined.
Number of pages written from zswap to swap.
zswap_incomp
- Number of incompressible pages currently stored in zswap
+ Amount of memory used by incompressible pages currently stored in zswap
without compression. These pages could not be compressed to
a size smaller than PAGE_SIZE, so they are stored as-is.
base-commit: 1a3746ccbb0a97bed3c06ccde6b880013b1dddc1
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v5 9/9] mm: switch deferred split shrinker to list_lru
From: Wei Yang @ 2026-06-20 7:39 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Liam R . Howlett, Usama Arif,
Kiryl Shutsemau, Vlastimil Babka, Kairui Song, Mikhail Zaslonko,
Vasily Gorbik, Baolin Wang, Barry Song, Dev Jain, Lance Yang,
Nico Pache, Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260527204757.2544958-10-hannes@cmpxchg.org>
On Wed, May 27, 2026 at 04:45:16PM -0400, Johannes Weiner wrote:
[...]
>@@ -3890,34 +3804,43 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
> struct folio *end_folio = folio_next(folio);
> struct folio *new_folio, *next;
> int old_order = folio_order(folio);
>+ struct list_lru_one *lru;
>+ bool dequeue_deferred;
> int ret = 0;
>- struct deferred_split *ds_queue;
>
> VM_WARN_ON_ONCE(!mapping && end);
>- /* Prevent deferred_split_scan() touching ->_refcount */
>- ds_queue = folio_split_queue_lock(folio);
>+ /*
>+ * If this folio can be on the deferred split queue, lock out
>+ * the shrinker before freezing the ref. If the shrinker sees
>+ * a 0-ref folio, it assumes it beat folio_put() to the list
>+ * lock and must clean up the LRU state - the same dequeue we
>+ * will do below as part of the split.
>+ */
>+ dequeue_deferred = folio_test_anon(folio) && old_order > 1;
Looking at __folio_remove_rmap(), we check !folio_is_device_private() before
deferred_split_folio(). __folio_freeze_and_split_unmapped() is used in
folio_split_unmapped(). According to its comment, it could take device-private
folio.
This means for device-private folio, we still lock lru_list and try to remove
it from deferred_split_lru. The good news is this doesn't harm the system, but
does some extra work.
Would it be better to add !folio_is_device_private() here?
The purpose to lock here is to prevent shrinker seeing ref-0 folio. Since
device-private folio is not on deferred_split_lru, shrink won't see it.
>+ if (dequeue_deferred) {
>+ struct mem_cgroup *memcg;
>+
>+ rcu_read_lock();
>+ memcg = folio_memcg(folio);
>+ lru = list_lru_lock(&deferred_split_lru,
>+ folio_nid(folio), &memcg);
>+ }
> if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
> struct swap_cluster_info *ci = NULL;
> struct lruvec *lruvec;
>
>- if (old_order > 1) {
>- if (!list_empty(&folio->_deferred_list)) {
>- ds_queue->split_queue_len--;
>- /*
>- * Reinitialize page_deferred_list after removing the
>- * page from the split_queue, otherwise a subsequent
>- * split will see list corruption when checking the
>- * page_deferred_list.
>- */
>- list_del_init(&folio->_deferred_list);
>- }
>+ if (dequeue_deferred) {
>+ __list_lru_del(&deferred_split_lru, lru,
>+ &folio->_deferred_list, folio_nid(folio));
> if (folio_test_partially_mapped(folio)) {
> folio_clear_partially_mapped(folio);
> mod_mthp_stat(old_order,
> MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
> }
>+ list_lru_unlock(lru);
>+ rcu_read_unlock();
> }
>- split_queue_unlock(ds_queue);
>+
> if (mapping) {
> int nr = folio_nr_pages(folio);
>
>@@ -4017,7 +3940,10 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
> if (ci)
> swap_cluster_unlock(ci);
> } else {
>- split_queue_unlock(ds_queue);
>+ if (dequeue_deferred) {
>+ list_lru_unlock(lru);
>+ rcu_read_unlock();
>+ }
> return -EAGAIN;
> }
>
--
Wei Yang
Help you, Help me
^ permalink raw reply
* Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
From: Chen, Yu C @ 2026-06-20 3:54 UTC (permalink / raw)
To: Peter Zijlstra
Cc: longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef,
mingo
In-Reply-To: <20260605124052.227463677@infradead.org>
On 6/5/2026 8:40 PM, Peter Zijlstra wrote:
> Change fair/cgroup to a single runqueue.
>
> Infamously fair/cgroup isn't working for a number of people; typically
> the complaint is latencies and/or overhead. The latency issue is due
> to the intermediate entries that represent a combination of tasks and
> thereby obfuscate the runnability of tasks.
>
> The approach here is to leave the cgroup hierarchy as is; including
> the intermediate enqueue/dequeue but move the actual EEVDF runqueue
> outside. This means things like the shares_weight approximation are
> fully preserved.
>
> That is, given a hierarchy like:
>
> R
> |
> se--G1
> / \
> G2--se se--G3
> / \ |
> T1--se se--T2 se--T3
>
> This is fully maintained for load tracking, however the EEVDF parts of
> cfs_rq/se go unused for the intermediates and are instead connected
> like:
>
> _R_
> / | \
> T1 T2 T3
>
> Since the effective weight of the entities is determined by the
> hierarchy, this gets recomputed on enqueue,set_next_task and tick.
>
> Notably, the effective weight (se->h_load) is computed from the
> hierarchical fraction: se->load / cfs_rq->load.
>
> Since EEVDF is now exclusively operating on rq->cfs, it needs to
> consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
> only tasks can get delayed, simplifying some of the cgroup cleanup.
>
> One place where additional information was required was
> set_next_task() / put_prev_task(), where we need to track 'current'
> both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
> (cfs_rq->curr).
>
> As a result of only having a single level to pick from, much of the
> complications in pick_next_task() and preemption go away.
>
> Since many of the hierarchical operations are still there, this won't
> immediately fix the performance issues, but hopefully it will fix some
> of the latency issues.
>
> TODO: split struct cfs_rq / struct sched_entity
> TODO: try and get rid of h_curr
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not
tainted 7.1.0-rc2+
[14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0
[14697.506799] <TASK>
[14697.507411] __dequeue_task+0x2b4/0xc70
[14697.508677] dequeue_task_fair+0x36/0x370
[14697.509047] dequeue_task+0x101/0x2f0
[14697.509426] __schedule+0x1b1/0x1a00
[14697.510868] anon_pipe_read+0x3da/0x450
[14697.511400] vfs_read+0x361/0x390
[14697.512053] __x64_sys_read+0x19/0x30
The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) {
load_sum = div_u64(gcfs_rq->avg.load_sum,
scale_load_down(gcfs_rq->load.weight));
}
gcfs_rq->load.weight is an insane large value and is truncated
to the lower 32 bits by div_u64, which happen to be 0.
Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when using
tg_tasks():
u32 new_sum, divider;
...
new_sum = se->avg.runnable_avg * divider; <-- boom
The following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable() # u32 overflow corrupts runnable_sum
__update_load_avg_cfs_rq()
___update_load_avg() # computes insane runnable_avg
update_tg_load_avg() # propagates to tg->runnable_avg
update_cfs_group()
calc_concur_shares()
tg_tasks() # long-to-int truncation, negative nr
reweight_entity() # corrupted se->load.weight
update_load_add() # corrupted cfs_rq->load.weight
propagate_entity_load_avg()
update_tg_cfs_load()
div_u64() # divide-by-zero
Fix by widening new_sum from u32 to u64(no need to force tg_tasks()
to return unsigned long after this fix)
Assisted-by: Claude:claude-opus-4.6
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
kernel/sched/fair.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d991ea85873a..99ea51448981 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5305,7 +5305,8 @@ static inline void
update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se,
struct cfs_rq *gcfs_rq)
{
long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg -
se->avg.runnable_avg;
- u32 new_sum, divider;
+ u64 new_sum;
+ u32 divider;
/* Nothing to update */
if (!delta_avg)
@@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq,
struct sched_entity *se, struct cf
/* Set new sched_entity's runnable */
se->avg.runnable_avg = gcfs_rq->avg.runnable_avg;
- new_sum = se->avg.runnable_avg * divider;
+ new_sum = (u64)se->avg.runnable_avg * divider;
delta_sum = (long)new_sum - (long)se->avg.runnable_sum;
se->avg.runnable_sum = new_sum;
--
2.45.2
^ permalink raw reply related
* [PATCH] selftests/cgroup: Adjust cpu.max quota based on HZ
From: Joe Simmons-Talbott @ 2026-06-19 21:18 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shuah Khan
Cc: Joe Simmons-Talbott, cgroups, linux-kselftest, linux-kernel
For lower HZ values a quota of 1000us is much lower than the amount
of microseconds per tick which makes the test_cpucg_max and
test_cpugc_max_nested fail. Use the amount of microseconds per tick
as the quota value.
Signed-off-by: Joe Simmons-Talbott <joest@redhat.com>
---
tools/testing/selftests/cgroup/test_cpu.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index 7a40d76b9548..4ac5d3ecae00 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -646,7 +646,7 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
static int test_cpucg_max(const char *root)
{
int ret = KSFT_FAIL;
- long quota_usec = 1000;
+ long quota_usec = USEC_PER_SEC / sysconf(_SC_CLK_TCK);
long default_period_usec = 100000; /* cpu.max's default period */
long duration_seconds = 1;
@@ -710,7 +710,7 @@ static int test_cpucg_max(const char *root)
static int test_cpucg_max_nested(const char *root)
{
int ret = KSFT_FAIL;
- long quota_usec = 1000;
+ long quota_usec = USEC_PER_SEC / sysconf(_SC_CLK_TCK);
long default_period_usec = 100000; /* cpu.max's default period */
long duration_seconds = 1;
--
2.54.0
^ permalink raw reply related
* [REGRESSION] [PATCH v2 1/2] mm: vmalloc: streamline vmalloc memory accounting
From: Aishwarya Rambhadran @ 2026-06-19 12:53 UTC (permalink / raw)
To: Johannes Weiner, Andrew Morton
Cc: Uladzislau Rezki, Joshua Hahn, Michal Hocko, Roman Gushchin,
Shakeel Butt, Muchun Song, linux-mm, cgroups, linux-kernel,
Ryan Roberts
In-Reply-To: <20260223160147.3792777-1-hannes@cmpxchg.org>
Hi Johannes,
We have observed kernel performance regressions in vmalloc benchmarks
when comparing v7.0 mainline results against later releases in the v7.1
cycle.
The regressions were detected by Fastpath, our automated kernel
performance benchmark and regression tracking framework.
Independent bisections on multiple arm64 systems consistently
identify this patch as the root cause. The regressions are reproducible
on both AWS Graviton3 & AmpereOne systems.
Fastpath bisection details :
Benchmark - micromm/vmalloc
Test - fix_size_alloc_test: p:512, h:1, l:100000
Good Kernel - v7.0
Bad Kernel - v7.1-rc4
The measured regression for the above test is approximately 32.5%
on AWS Graviton3. Similar regressions are observed across multiple
tests within the vmalloc benchmark suite as well as on AmpereOne.
Below given are the performance benchmark results of vmalloc
suite generated by Fastpath Tool, for v7.1 kernel version relative to
the base version v7.0, executed on the AWS Graviton3 SUT. Label (R)
mean statistically significant regression, where "statistically
significant"
means the 95% confidence intervals do not overlap.
v7.0 (base) | v7.1
-------------------------------------------------------------------
fix_align_alloc_test: p:1, h:0, l:500000
895106.67 | (R) -10.73%
fix_size_alloc_test: p:1, h:0, l:500000
336785.00 | (R) -7.31%
fix_size_alloc_test: p:4, h:0, l:500000
529652.83 | (R) -13.11%
fix_size_alloc_test: p:16, h:0, l:500000
1043412.50 | (R) -21.92%
fix_size_alloc_test: p:16, h:1, l:500000
1015795.83 | (R) -22.02%
fix_size_alloc_test: p:64, h:0, l:100000
643074.33 | (R) -25.91%
fix_size_alloc_test: p:64, h:1, l:100000
607604.00 | (R) -27.31%
fix_size_alloc_test: p:256, h:0, l:100000
2367906.50 | (R) -27.67%
fix_size_alloc_test: p:256, h:1, l:100000
2275464.67 | (R) -28.66%
fix_size_alloc_test: p:512, h:0, l:100000
4696069.17 | (R) -28.15%
fix_size_alloc_test: p:512, h:1, l:100000
3767292.00 | (R) -32.65%
full_fit_alloc_test: p:1, h:0, l:500000
493884.17 | (R) -12.38%
kvfree_rcu_1_arg_vmalloc_test: p:1, h:0, l:500000
354542.83 | -2.31%
kvfree_rcu_2_arg_vmalloc_test: p:1, h:0, l:500000
358082.83 | -1.53%
long_busy_list_alloc_test: p:1, h:0, l:500000
5490101.33 | (R) -25.85%
pcpu_alloc_test: p:1, h:0, l:500000
193634.00 | -1.53%
random_size_align_alloc_test: p:1, h:0, l:500000
1200206.83 | (R) -11.88%
random_size_alloc_test: p:1, h:0, l:500000
2875736.33 | (R) -24.41%
vm_map_ram_test: p:1, h:0, l:500000
81204.33 | -0.28%
-------------------------------------------------------------------
The regression signal appears stable across repeated runs.
Have you seen similar effects before, or is there an expected
behavioral change associated with the conversion from the
custom atomic accounting to vmstat counters that could
explain this result ?
We would be happy to provide additional performance data,
kernel configurations or any other details if useful.
Thank you.
Aishwarya Rambhadran
On 23/02/26 9:31 PM, Johannes Weiner wrote:
> Use a vmstat counter instead of a custom, open-coded atomic. This has
> the added benefit of making the data available per-node, and prepares
> for cleaning up the memcg accounting as well.
>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
> ---
> fs/proc/meminfo.c | 3 ++-
> include/linux/mmzone.h | 1 +
> include/linux/vmalloc.h | 3 ---
> mm/vmalloc.c | 19 ++++++++++---------
> mm/vmstat.c | 1 +
> 5 files changed, 14 insertions(+), 13 deletions(-)
>
> V2:
> - Fix mod_node_page_state() pgdat argument (Shakeel)
>
> diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
> index a458f1e112fd..549793f44726 100644
> --- a/fs/proc/meminfo.c
> +++ b/fs/proc/meminfo.c
> @@ -126,7 +126,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
> show_val_kb(m, "Committed_AS: ", committed);
> seq_printf(m, "VmallocTotal: %8lu kB\n",
> (unsigned long)VMALLOC_TOTAL >> 10);
> - show_val_kb(m, "VmallocUsed: ", vmalloc_nr_pages());
> + show_val_kb(m, "VmallocUsed: ",
> + global_node_page_state(NR_VMALLOC));
> show_val_kb(m, "VmallocChunk: ", 0ul);
> show_val_kb(m, "Percpu: ", pcpu_nr_pages());
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index fc5d6c88d2f0..64df797d45c6 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -220,6 +220,7 @@ enum node_stat_item {
> NR_KERNEL_MISC_RECLAIMABLE, /* reclaimable non-slab kernel pages */
> NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */
> NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */
> + NR_VMALLOC,
> NR_KERNEL_STACK_KB, /* measured in KiB */
> #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK)
> NR_KERNEL_SCS_KB, /* measured in KiB */
> diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h
> index e8e94f90d686..3b02c0c6b371 100644
> --- a/include/linux/vmalloc.h
> +++ b/include/linux/vmalloc.h
> @@ -286,8 +286,6 @@ int unregister_vmap_purge_notifier(struct notifier_block *nb);
> #ifdef CONFIG_MMU
> #define VMALLOC_TOTAL (VMALLOC_END - VMALLOC_START)
>
> -unsigned long vmalloc_nr_pages(void);
> -
> int vm_area_map_pages(struct vm_struct *area, unsigned long start,
> unsigned long end, struct page **pages);
> void vm_area_unmap_pages(struct vm_struct *area, unsigned long start,
> @@ -304,7 +302,6 @@ static inline void set_vm_flush_reset_perms(void *addr)
> #else /* !CONFIG_MMU */
> #define VMALLOC_TOTAL 0UL
>
> -static inline unsigned long vmalloc_nr_pages(void) { return 0; }
> static inline void set_vm_flush_reset_perms(void *addr) {}
> #endif /* CONFIG_MMU */
>
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index e286c2d2068c..a5fc7795aafd 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -1063,14 +1063,8 @@ static BLOCKING_NOTIFIER_HEAD(vmap_notify_list);
> static void drain_vmap_area_work(struct work_struct *work);
> static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
>
> -static __cacheline_aligned_in_smp atomic_long_t nr_vmalloc_pages;
> static __cacheline_aligned_in_smp atomic_long_t vmap_lazy_nr;
>
> -unsigned long vmalloc_nr_pages(void)
> -{
> - return atomic_long_read(&nr_vmalloc_pages);
> -}
> -
> static struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_root *root)
> {
> struct rb_node *n = root->rb_node;
> @@ -3463,11 +3457,11 @@ void vfree(const void *addr)
> * High-order allocs for huge vmallocs are split, so
> * can be freed as an array of order-0 allocations
> */
> + if (!(vm->flags & VM_MAP_PUT_PAGES))
> + dec_node_page_state(page, NR_VMALLOC);
> __free_page(page);
> cond_resched();
> }
> - if (!(vm->flags & VM_MAP_PUT_PAGES))
> - atomic_long_sub(vm->nr_pages, &nr_vmalloc_pages);
> kvfree(vm->pages);
> kfree(vm);
> }
> @@ -3655,6 +3649,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
> continue;
> }
>
> + mod_node_page_state(page_pgdat(page), NR_VMALLOC, 1 << large_order);
> +
> split_page(page, large_order);
> for (i = 0; i < (1U << large_order); i++)
> pages[nr_allocated + i] = page + i;
> @@ -3675,6 +3671,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
> if (!order) {
> while (nr_allocated < nr_pages) {
> unsigned int nr, nr_pages_request;
> + int i;
>
> /*
> * A maximum allowed request is hard-coded and is 100
> @@ -3698,6 +3695,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
> nr_pages_request,
> pages + nr_allocated);
>
> + for (i = nr_allocated; i < nr_allocated + nr; i++)
> + inc_node_page_state(pages[i], NR_VMALLOC);
> +
> nr_allocated += nr;
>
> /*
> @@ -3722,6 +3722,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
> if (unlikely(!page))
> break;
>
> + mod_node_page_state(page_pgdat(page), NR_VMALLOC, 1 << order);
> +
> /*
> * High-order allocations must be able to be treated as
> * independent small pages by callers (as they can with
> @@ -3864,7 +3866,6 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
> vmalloc_gfp_adjust(gfp_mask, page_order), node,
> page_order, nr_small_pages, area->pages);
>
> - atomic_long_add(area->nr_pages, &nr_vmalloc_pages);
> /* All pages of vm should be charged to same memcg, so use first one. */
> if (gfp_mask & __GFP_ACCOUNT && area->nr_pages)
> mod_memcg_page_state(area->pages[0], MEMCG_VMALLOC,
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index d6e814c82952..bc199c7cd07b 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1270,6 +1270,7 @@ const char * const vmstat_text[] = {
> [I(NR_KERNEL_MISC_RECLAIMABLE)] = "nr_kernel_misc_reclaimable",
> [I(NR_FOLL_PIN_ACQUIRED)] = "nr_foll_pin_acquired",
> [I(NR_FOLL_PIN_RELEASED)] = "nr_foll_pin_released",
> + [I(NR_VMALLOC)] = "nr_vmalloc",
> [I(NR_KERNEL_STACK_KB)] = "nr_kernel_stack",
> #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK)
> [I(NR_KERNEL_SCS_KB)] = "nr_shadow_call_stack",
^ permalink raw reply
* Re: [PATCH v3 08/13] genirq: Add explicit housekeeping callback for managed IRQ migration
From: Thomas Gleixner @ 2026-06-18 21:11 UTC (permalink / raw)
To: Jing Wu, Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Paul E. McKenney, Frederic Weisbecker,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Lai Jiangshan, Zqiang,
Anna-Maria Behnsen, Tejun Heo, Jonathan Corbet, Shuah Khan,
Shuah Khan, Waiman Long
Cc: linux-kernel, rcu, cgroups, linux-doc, linux-kselftest, Jing Wu,
Qiliang Yuan
In-Reply-To: <87cxxnegqa.ffs@fw13>
On Thu, Jun 18 2026 at 22:27, Thomas Gleixner wrote:
> On Thu, Jun 18 2026 at 11:11, Jing Wu wrote:
>> + */
>> + if (irqd_affinity_is_managed(&desc->irq_data)) {
>
> So you set the affinity even on an interrupt which is shutdown?
>
>> + const struct cpumask *mask;
>> + struct cpumask *tmp = this_cpu_ptr(&__tmp_mask);
How is this correct? You cannot get the per cpu pointer in preemptible
context. The task might be migrated and then fiddle with the wrong
per CPU data. But that's moot as this code is broken anyway.
^ permalink raw reply
* Re: [PATCH v3 05/13] cpu/hotplug: Reserve CPUHP states for nohz_full and managed IRQ down-paths
From: Thomas Gleixner @ 2026-06-18 21:01 UTC (permalink / raw)
To: Jing Wu, Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Paul E. McKenney, Frederic Weisbecker,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Lai Jiangshan, Zqiang,
Anna-Maria Behnsen, Tejun Heo, Jonathan Corbet, Shuah Khan,
Shuah Khan
Cc: linux-kernel, rcu, cgroups, linux-doc, linux-kselftest, Jing Wu,
Qiliang Yuan
In-Reply-To: <871pe3de9b.ffs@fw13>
On Thu, Jun 18 2026 at 18:06, Thomas Gleixner wrote:
> On Thu, Jun 18 2026 at 11:11, Jing Wu wrote:
>> Add CPUHP_AP_NO_HZ_FULL_DYING and CPUHP_AP_IRQ_AFFINITY_DYING to the
>> cpuhp_state enum. These dying callbacks are invoked during CPU offline
>> before the tick is stopped, enabling clean tick handover and managed
>> IRQ migration when a CPU transitions between isolated and housekeeping
>> states.
>>
>> The existing CPUHP_AP_IRQ_AFFINITY_ONLINE already handles managed IRQ
>> restoration on CPU online. The new dying callback completes the pair,
>> migrating managed interrupts away from the CPU before it goes down.
>
> What? They are migrated away today already when the CPU goes down unless
> the CPU is the last one in the affinity set of the interrupt. So why do
> you need a new step for something which already exists?
Aside of that these hotplug states are not used at all. So what is this
patch for?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox