* [PATCH 2/3] mm/zswap: Implement proactive writeback
From: Hao Jia @ 2026-05-11 10:51 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260511105149.75584-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Zswap currently writes back pages to backing swap devices reactively,
triggered either by memory pressure via the shrinker or by the pool
reaching its size limit. This reactive approach offers no precise
control over when writeback happens, which can disturb latency-sensitive
workloads, and it cannot direct writeback at a specific memory cgroup.
However, there are scenarios where users might want to proactively
write back cold pages from zswap to the backing swap device, for
example, to free up memory for other applications or to prepare for
upcoming memory-intensive workloads.
Therefore, implement a proactive writeback mechanism for zswap by
adding a new cgroup interface file memory.zswap.proactive_writeback
within the memory controller.
Users can trigger writeback by writing to this file with the following
parameters:
- max=<bytes>: The maximum amount of memory to write back (optional,
default: unlimited).
- <age>: The minimum age of the pages to write back. Only pages that
have been in zswap for at least this duration will be written back.
Example usage:
# Write back pages older than 1 hour (3600 seconds), max 10MB
echo "max=10M 3600" > memory.zswap.proactive_writeback
The implementation consists of:
1. Add store_time to struct zswap_entry to record when each entry was
inserted into zswap, used for proactive writeback age comparison.
2. Introduce struct zswap_shrink_walk_arg, passed as the cb_arg to
list_lru_walk_one() in both the shrinker and proactive paths. It
carries the per-invocation cutoff_time and proactive flag down to
shrink_memcg_cb(), and propagates the encountered_page_in_swapcache
out-signal from the callback back to the caller.
3. Modify the callback function shrink_memcg_cb() to proactively
writeback zswap_entries that meet the time threshold.
4. Add zswap_proactive_writeback() as the proactive writeback driver:
a per-node batched list_lru_walk_one() loop bounded by the
writeback budget.
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
Documentation/admin-guide/cgroup-v2.rst | 24 ++++
include/linux/zswap.h | 8 ++
mm/memcontrol.c | 76 ++++++++++
mm/zswap.c | 176 ++++++++++++++++++++++--
4 files changed, 276 insertions(+), 8 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed99..05b664b3b3e8 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1908,6 +1908,30 @@ The following nested keys are defined.
This setting has no effect if zswap is disabled, and swapping
is allowed unless memory.swap.max is set to 0.
+ memory.zswap.proactive_writeback
+ A write-only nested-keyed file which exists in non-root cgroups.
+
+ This interface allows proactive writeback of pages from the zswap
+ pool to the backing swap device. This is useful to offload cold
+ pages from the zswap pool to the slower swap device. It is only
+ available if zswap writeback is enabled.
+
+ Users can trigger writeback by writing to this file with the following
+ parameters:
+
+ - "max=<bytes>" : Optional. The maximum amount of data to write back.
+ (default: unlimited). Please note that the kernel can over or under
+ writeback this value.
+
+ - "<age>" : Required. The minimum age of the pages to write back
+ (in seconds). Only pages that have been in the zswap pool for at
+ least this amount of time will be written back.
+
+ Example::
+
+ # Write back pages older than 1 hour (3600 seconds), max 10MB
+ echo "max=10M 3600" > memory.zswap.proactive_writeback
+
memory.pressure
A read-only nested-keyed file.
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index efa6b551217e..7a51b4f95017 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -44,6 +44,8 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);
void zswap_folio_swapin(struct folio *folio);
bool zswap_is_enabled(void);
bool zswap_never_enabled(void);
+int zswap_proactive_writeback(struct mem_cgroup *root, unsigned long nr_max_writeback,
+ ktime_t cutoff);
#else
struct zswap_lruvec_state {};
@@ -78,6 +80,12 @@ static inline bool zswap_never_enabled(void)
return true;
}
+static inline int zswap_proactive_writeback(struct mem_cgroup *root,
+ unsigned long nr_max_writeback, ktime_t cutoff)
+{
+ return 0;
+}
+
#endif
#endif /* _LINUX_ZSWAP_H */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 409c41359dc8..ba7f7b1954a8 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -70,6 +70,7 @@
#include "memcontrol-v1.h"
#include <linux/uaccess.h>
+#include <linux/parser.h>
#define CREATE_TRACE_POINTS
#include <trace/events/memcg.h>
@@ -5891,6 +5892,76 @@ static ssize_t zswap_writeback_write(struct kernfs_open_file *of,
return nbytes;
}
+enum {
+ ZSWAP_WRITEBACK_MAX,
+ ZSWAP_WRITEBACK_AGE,
+ ZSWAP_WRITEBACK_ERR,
+};
+
+static const match_table_t zswap_writeback_tokens = {
+ { ZSWAP_WRITEBACK_MAX, "max=%s" },
+ { ZSWAP_WRITEBACK_AGE, "%u" },
+ { ZSWAP_WRITEBACK_ERR, NULL },
+};
+
+static ssize_t zswap_proactive_writeback_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes,
+ loff_t off)
+{
+ struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
+ unsigned long nr_max_writeback = ULONG_MAX;
+ substring_t args[MAX_OPT_ARGS];
+ unsigned int age_sec;
+ bool age_set = false;
+ ktime_t cutoff_time;
+ char *token, *end;
+ int err;
+
+ if (!mem_cgroup_zswap_writeback_enabled(memcg))
+ return -EINVAL;
+
+ buf = strstrip(buf);
+
+ while ((token = strsep(&buf, " ")) != NULL) {
+ if (!strlen(token))
+ continue;
+
+ switch (match_token(token, zswap_writeback_tokens, args)) {
+ case ZSWAP_WRITEBACK_MAX:
+ nr_max_writeback = memparse(args[0].from, &end);
+ if (*end != '\0')
+ return -EINVAL;
+ nr_max_writeback >>= PAGE_SHIFT;
+ break;
+ case ZSWAP_WRITEBACK_AGE:
+ if (age_set)
+ return -EINVAL;
+
+ if (match_uint(&args[0], &age_sec))
+ return -EINVAL;
+ age_set = true;
+ break;
+ default:
+ return -EINVAL;
+ }
+ }
+
+ if (!age_set || !age_sec || !nr_max_writeback)
+ return -EINVAL;
+
+ cutoff_time = ktime_sub(ktime_get_boottime(),
+ ns_to_ktime((u64)age_sec * NSEC_PER_SEC));
+ /* age_sec >= uptime: no entry can be that old, skip the walk. */
+ if (ktime_to_ns(cutoff_time) <= 0)
+ return nbytes;
+
+ err = zswap_proactive_writeback(memcg, nr_max_writeback, cutoff_time);
+ if (err)
+ return err;
+
+ return nbytes;
+}
+
static struct cftype zswap_files[] = {
{
.name = "zswap.current",
@@ -5908,6 +5979,11 @@ static struct cftype zswap_files[] = {
.seq_show = zswap_writeback_show,
.write = zswap_writeback_write,
},
+ {
+ .name = "zswap.proactive_writeback",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .write = zswap_proactive_writeback_write,
+ },
{ } /* terminate */
};
#endif /* CONFIG_ZSWAP */
diff --git a/mm/zswap.c b/mm/zswap.c
index 19538d6f169a..1173ac6836fa 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -36,6 +36,7 @@
#include <linux/workqueue.h>
#include <linux/list_lru.h>
#include <linux/zsmalloc.h>
+#include <linux/timekeeping.h>
#include "swap.h"
#include "internal.h"
@@ -160,6 +161,12 @@ struct zswap_pool {
char tfm_name[CRYPTO_MAX_ALG_NAME];
};
+struct zswap_shrink_walk_arg {
+ ktime_t cutoff_time;
+ bool proactive;
+ bool encountered_page_in_swapcache;
+};
+
/* Global LRU lists shared by all zswap pools. */
static struct list_lru zswap_list_lru;
@@ -183,6 +190,7 @@ static struct shrinker *zswap_shrinker;
* handle - zsmalloc allocation handle that stores the compressed page data
* objcg - the obj_cgroup that the compressed memory is charged to
* lru - handle to the pool's lru used to evict pages.
+ * store_time - Time when the entry was stored, for proactive writeback.
*/
struct zswap_entry {
swp_entry_t swpentry;
@@ -192,6 +200,7 @@ struct zswap_entry {
unsigned long handle;
struct obj_cgroup *objcg;
struct list_head lru;
+ ktime_t store_time;
};
static struct xarray *zswap_trees[MAX_SWAPFILES];
@@ -1148,10 +1157,19 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
void *arg)
{
struct zswap_entry *entry = container_of(item, struct zswap_entry, lru);
- bool *encountered_page_in_swapcache = (bool *)arg;
- swp_entry_t swpentry;
+ struct zswap_shrink_walk_arg *walk_arg = arg;
+ bool proactive_wb = walk_arg && walk_arg->proactive;
enum lru_status ret = LRU_REMOVED_RETRY;
int writeback_result;
+ swp_entry_t swpentry;
+
+ /*
+ * For proactive writeback, rotate young entries to the LRU tail
+ * so that subsequent list_lru_walk_one() batches start past
+ * them.
+ */
+ if (proactive_wb && ktime_after(entry->store_time, walk_arg->cutoff_time))
+ return LRU_ROTATE;
/*
* Second chance algorithm: if the entry has its referenced bit set, give it
@@ -1160,7 +1178,9 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
*/
if (entry->referenced) {
entry->referenced = false;
- return LRU_ROTATE;
+ /* Proactive writeback is an explicit hint; don't rotate. */
+ if (!proactive_wb)
+ return LRU_ROTATE;
}
/*
@@ -1214,9 +1234,9 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
* into the warmer region. We should terminate shrinking (if we're in the dynamic
* shrinker context).
*/
- if (writeback_result == -EEXIST && encountered_page_in_swapcache) {
+ if (writeback_result == -EEXIST && walk_arg) {
ret = LRU_STOP;
- *encountered_page_in_swapcache = true;
+ walk_arg->encountered_page_in_swapcache = true;
}
} else {
zswap_written_back_pages++;
@@ -1228,8 +1248,12 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
static unsigned long zswap_shrinker_scan(struct shrinker *shrinker,
struct shrink_control *sc)
{
+ struct zswap_shrink_walk_arg walk_arg = {
+ .cutoff_time = KTIME_MAX,
+ .proactive = false,
+ .encountered_page_in_swapcache = false,
+ };
unsigned long shrink_ret;
- bool encountered_page_in_swapcache = false;
if (!zswap_shrinker_enabled ||
!mem_cgroup_zswap_writeback_enabled(sc->memcg)) {
@@ -1238,9 +1262,9 @@ static unsigned long zswap_shrinker_scan(struct shrinker *shrinker,
}
shrink_ret = list_lru_shrink_walk(&zswap_list_lru, sc, &shrink_memcg_cb,
- &encountered_page_in_swapcache);
+ &walk_arg);
- if (encountered_page_in_swapcache)
+ if (walk_arg.encountered_page_in_swapcache)
return SHRINK_STOP;
return shrink_ret ? shrink_ret : SHRINK_STOP;
@@ -1508,6 +1532,7 @@ static bool zswap_store_page(struct page *page,
entry->swpentry = page_swpentry;
entry->objcg = objcg;
entry->referenced = true;
+ entry->store_time = ktime_get_boottime();
if (entry->length) {
INIT_LIST_HEAD(&entry->lru);
zswap_lru_add(&zswap_list_lru, entry);
@@ -1681,6 +1706,141 @@ int zswap_load(struct folio *folio)
return 0;
}
+/* Cap LRU scan to this many entries per page of remaining budget. */
+#define ZSWAP_PROACTIVE_WB_SCAN_RATIO 16UL
+/*
+ * Batch size for proactive writeback, used both as the per-memcg
+ * writeback target in the outer memcg loop and as the per-walk budget
+ * for list_lru_walk_one().
+ */
+#define ZSWAP_PROACTIVE_WB_BATCH 128UL
+
+/*
+ * Walk @memcg's per-node LRUs, writing back entries older than @cutoff
+ * up to @nr_to_write pages. Returns the number of pages written back,
+ * or -ENOENT if @memcg is a zombie or has writeback disabled.
+ */
+static long zswap_proactive_shrink_memcg(struct mem_cgroup *memcg,
+ ktime_t cutoff,
+ unsigned long nr_to_write)
+{
+ unsigned long nr_written = 0;
+ int nid;
+
+ if (!mem_cgroup_zswap_writeback_enabled(memcg))
+ return -ENOENT;
+
+ if (!mem_cgroup_online(memcg))
+ return -ENOENT;
+
+ for_each_node_state(nid, N_NORMAL_MEMORY) {
+ struct zswap_shrink_walk_arg walk_arg = {
+ .cutoff_time = cutoff,
+ .proactive = true,
+ .encountered_page_in_swapcache = false,
+ };
+ unsigned long nr_to_scan, nr_scanned = 0;
+
+ /*
+ * Cap by LRU length: bounds rewalks when entries keep
+ * rotating (young or referenced).
+ */
+ nr_to_scan = list_lru_count_one(&zswap_list_lru, nid, memcg);
+ if (!nr_to_scan)
+ continue;
+
+ /*
+ * Cap by SCAN_RATIO * remaining budget: bounds scan cost
+ * to the remaining writeback budget.
+ */
+ nr_to_scan = min(nr_to_scan,
+ (nr_to_write - nr_written) * ZSWAP_PROACTIVE_WB_SCAN_RATIO);
+
+ while (nr_scanned < nr_to_scan) {
+ unsigned long nr_to_walk = min(ZSWAP_PROACTIVE_WB_BATCH,
+ nr_to_scan - nr_scanned);
+
+ if (signal_pending(current))
+ return nr_written;
+
+ /*
+ * Account the committed budget rather than the walker's
+ * actual delta: if the list empties under us the walker
+ * visits nothing and nr_scanned would never advance.
+ */
+ nr_scanned += nr_to_walk;
+
+ nr_written += list_lru_walk_one(&zswap_list_lru, nid, memcg,
+ &shrink_memcg_cb, &walk_arg,
+ &nr_to_walk);
+
+ if (nr_written >= nr_to_write)
+ return nr_written;
+ if (walk_arg.encountered_page_in_swapcache)
+ break;
+
+ cond_resched();
+ }
+ }
+
+ return nr_written;
+}
+
+int zswap_proactive_writeback(struct mem_cgroup *root,
+ unsigned long nr_max_writeback,
+ ktime_t cutoff)
+{
+ struct mem_cgroup *memcg;
+ unsigned long nr_written = 0;
+ int failures = 0, attempts = 0;
+
+ /*
+ * Writeback will be aborted with -EAGAIN if @nr_written is still
+ * zero and we encounter the following MAX_RECLAIM_RETRIES times:
+ * - No writeback-candidate memcgs found in a subtree walk.
+ * - A writeback-candidate memcg wrote back zero pages.
+ */
+ while (nr_written < nr_max_writeback) {
+ unsigned long nr_to_write;
+ long shrunk;
+
+ if (signal_pending(current))
+ return -EINTR;
+
+ memcg = zswap_mem_cgroup_iter(root);
+
+ if (!memcg) {
+ /*
+ * Continue without incrementing failures if we found
+ * candidate memcgs in the last subtree walk.
+ */
+ if (!attempts && ++failures == MAX_RECLAIM_RETRIES)
+ goto out;
+ attempts = 0;
+ continue;
+ }
+
+ nr_to_write = min(nr_max_writeback - nr_written,
+ ZSWAP_PROACTIVE_WB_BATCH);
+ shrunk = zswap_proactive_shrink_memcg(memcg, cutoff, nr_to_write);
+ mem_cgroup_put(memcg);
+
+ /* Writeback-disabled or offline: skip without counting. */
+ if (shrunk == -ENOENT)
+ continue;
+
+ ++attempts;
+ if (shrunk > 0)
+ nr_written += shrunk;
+ else if (++failures == MAX_RECLAIM_RETRIES)
+ goto out;
+
+ cond_resched();
+ }
+out:
+ return nr_written ? 0 : -EAGAIN;
+}
+
void zswap_invalidate(swp_entry_t swp)
{
pgoff_t offset = swp_offset(swp);
--
2.34.1
^ permalink raw reply related
* [PATCH 3/3] mm/zswap: Add per-memcg stat for proactive writeback
From: Hao Jia @ 2026-05-11 10:51 UTC (permalink / raw)
To: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny, nphamcs,
chengming.zhou, muchun.song, roman.gushchin
Cc: cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260511105149.75584-1-jiahao.kernel@gmail.com>
From: Hao Jia <jiahao1@lixiang.com>
Currently, zswap writeback can be triggered by either the pool limit
being hit or by the proactive writeback mechanism. However, the
existing 'zswpwb' metric in memory.stat and /proc/vmstat counts all
written back pages, making it difficult to distinguish between pages
written back due to the pool limit and those written back proactively.
Add a new statistic 'zswpwb_proactive' to memory.stat and /proc/vmstat.
This counter tracks the number of pages written back due to proactive
writeback. This allows users to better monitor and tune the proactive
writeback mechanism.
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 ++++
include/linux/vm_event_item.h | 1 +
mm/memcontrol.c | 1 +
mm/vmstat.c | 1 +
mm/zswap.c | 11 +++++++++--
5 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 05b664b3b3e8..29a189b18efc 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1734,6 +1734,10 @@ The following nested keys are defined.
zswpwb
Number of pages written from zswap to swap.
+ zswpwb_proactive
+ Number of pages written from zswap to swap by proactive
+ writeback. This is a subset of zswpwb.
+
zswap_incomp
Number of incompressible pages currently stored in zswap
without compression. These pages could not be compressed to
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 03fe95f5a020..7a5bee0a20b6 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -138,6 +138,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
ZSWPIN,
ZSWPOUT,
ZSWPWB,
+ ZSWPWB_PROACTIVE,
#endif
#ifdef CONFIG_X86
DIRECT_MAP_LEVEL2_SPLIT,
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index ba7f7b1954a8..830d895e77c3 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -572,6 +572,7 @@ static const unsigned int memcg_vm_event_stat[] = {
ZSWPIN,
ZSWPOUT,
ZSWPWB,
+ ZSWPWB_PROACTIVE,
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
THP_FAULT_ALLOC,
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..66fd06d1bb01 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1452,6 +1452,7 @@ const char * const vmstat_text[] = {
[I(ZSWPIN)] = "zswpin",
[I(ZSWPOUT)] = "zswpout",
[I(ZSWPWB)] = "zswpwb",
+ [I(ZSWPWB_PROACTIVE)] = "zswpwb_proactive",
#endif
#ifdef CONFIG_X86
[I(DIRECT_MAP_LEVEL2_SPLIT)] = "direct_map_level2_splits",
diff --git a/mm/zswap.c b/mm/zswap.c
index 1173ac6836fa..bf23c46e838e 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1048,7 +1048,8 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
* freed.
*/
static int zswap_writeback_entry(struct zswap_entry *entry,
- swp_entry_t swpentry)
+ swp_entry_t swpentry,
+ bool proactive)
{
struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
@@ -1108,6 +1109,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (entry->objcg)
count_objcg_events(entry->objcg, ZSWPWB, 1);
+ if (proactive) {
+ count_vm_event(ZSWPWB_PROACTIVE);
+ if (entry->objcg)
+ count_objcg_events(entry->objcg, ZSWPWB_PROACTIVE, 1);
+ }
+
zswap_entry_free(entry);
/* folio is up to date */
@@ -1223,7 +1230,7 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
*/
spin_unlock(&l->lock);
- writeback_result = zswap_writeback_entry(entry, swpentry);
+ writeback_result = zswap_writeback_entry(entry, swpentry, proactive_wb);
if (writeback_result) {
zswap_reject_reclaim_fail++;
--
2.34.1
^ permalink raw reply related
* Re: [PATCH 1/3] cgroup/cpuset: Fix deadline bandwidth leak in cpuset_can_attach()
From: Aaron Tomlin @ 2026-05-11 11:08 UTC (permalink / raw)
To: Waiman Long
Cc: tsbogend, paul, jmorris, serge, mingo, peterz, juri.lelli,
vincent.guittot, stephen.smalley.work, casey, tj, hannes, mkoutny,
chenridong, dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, omosnace, kees, neelx, sean, chjohnst, steve,
mproche, nick.lange, cgroups, linux-mips, linux-fsdevel,
linux-security-module, selinux, linux-kernel
In-Reply-To: <8aaa7dd9-2426-475c-af64-85ef5f2aa855@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]
On Mon, May 11, 2026 at 01:10:02AM -0400, Waiman Long wrote:
>
> On 5/9/26 12:48 PM, Aaron Tomlin wrote:
> > During a cgroup migration, cpuset_can_attach() iterates over the
> > provided taskset. If a task within the batch is a deadline (DL) task,
> > the destination cpuset's DL metrics (i.e., nr_migrate_dl_tasks and
> > sum_migrate_dl_bw) are appropriately incremented.
> >
> > However, if a subsequent task in the same migration batch fails the
> > task_can_attach() check, the loop aborts and jumps directly to
> > out_unlock. Consequently, any DL metrics accumulated from previously
> > processed tasks in the batch remain permanently inflated in the
> > destination cpuset. Because the migration is subsequently aborted by the
> > cgroup core, cpuset_cancel_attach() is never invoked to unwind these
> > specific increments.
> >
> > This behaviour results in a permanent leak of deadline bandwidth, which
> > incorrectly restricts the admission control capacity of the destination
> > cpuset.
> >
> > To resolve this, introduce an out_unlock_reset failure path that
> > conditionally invokes reset_migrate_dl_data(). This guarantees that if a
> > batch migration is aborted for any reason, the pending DL metrics are
> > safely reset before returning the error.
> >
> > Fixes: 0a67b847e1f06 ("cpuset: Allow setscheduler regardless of manipulated task")
>
> That is not the commit that introduced the bug. Anyway, there is already
> another patch sent recently to fix this bug. See
>
> https://lore.kernel.org/lkml/20260509102031.97608-2-zhangguopeng@kylinos.cn/
>
Hi Waiman,
Thank you for the follow up.
Acknowledged. I will drop this patch in the next iteration due to [1].
Please note, the sashiko AI Review bot reported: cpuset_can_attach()
incorrectly assumes all migrating tasks originate from the same source
cpuset. At first glance, this feedback is valid. I plan to submit a patch,
if no solution was already proposed.
[1]: https://lore.kernel.org/lkml/20260509102031.97608-2-zhangguopeng@kylinos.cn/
Kind regards,
--
Aaron Tomlin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH 0/3] mm/zswap: Implement per-cgroup proactive writeback
From: Michal Koutný @ 2026-05-11 11:39 UTC (permalink / raw)
To: Hao Jia
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <20260511105149.75584-1-jiahao.kernel@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]
On Mon, May 11, 2026 at 06:51:46PM +0800, Hao Jia <jiahao.kernel@gmail.com> wrote:
> From: Hao Jia <jiahao1@lixiang.com>
>
> Zswap currently writes back pages to backing swap devices reactively,
> triggered either by memory pressure via the shrinker or by the pool
> reaching its size limit. However, this reactive approach makes writeback
> timing indeterminate and can disrupt latency-sensitive workloads when
> eviction happens to coincide with a critical execution window.
>
> Furthermore, in certain scenarios, it is desirable to trigger writeback
> in advance to free up memory. For example, users may want to prepare for
> an upcoming memory-intensive workload by flushing cold memory to the
> backing storage when the system is relatively idle.
I can imagine the zswap writeout can come at the least possible
moment...
> To address these issues, this patch series introduces a per-cgroup
> interface that allows users to proactively write back cold compressed
> pages from zswap to the backing swap device.
...but I see this series is not only per-cgroup proactive reclaim but
it's also age-based reclaim.
The per-cg consumption and limits (and regular memory reclaim) are all
measured in sizes. This age-based invocations don't seem commensurable
(e.g. how would users in practice determine what is the desired input to
here).
Could you explain more reasoning behind this design?
Thanks,
Michal
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply
* Re: [PATCH] selftests/cgroup: check malloc return value in alloc_anon functions
From: Vishal Moola @ 2026-05-11 11:59 UTC (permalink / raw)
To: Hongfu Li
Cc: hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song, tj,
mkoutny, shuah, cgroups, linux-mm, linux-kselftest, linux-kernel
In-Reply-To: <20260511021615.1768623-1-lihongfu@kylinos.cn>
On Mon, May 11, 2026 at 10:16:15AM +0800, Hongfu Li wrote:
> The alloc_anon() function calls malloc() without checking for a NULL
> return. If memory allocation fails, a NULL pointer dereference will
> occur when accessing the buffer.
>
> Add proper error handling to return -1 when malloc() fails in all
> four alloc_anon variants:
> - alloc_anon()
Just a nit, It looks like the below already have proper error handling.
> - alloc_anon_50M_check()
> - alloc_anon_noexit()
> - alloc_anon_50M_check_swap()
>
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
> ---
> tools/testing/selftests/cgroup/test_memcontrol.c | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c
> index b43da9bc20c4..8ef9c99a82eb 100644
> --- a/tools/testing/selftests/cgroup/test_memcontrol.c
> +++ b/tools/testing/selftests/cgroup/test_memcontrol.c
> @@ -61,6 +61,11 @@ int alloc_anon(const char *cgroup, void *arg)
> char *buf, *ptr;
>
> buf = malloc(size);
> + if (buf == NULL) {
> + fprintf(stderr, "malloc() failed\n");
> + return -1;
> + }
> +
> for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
> *ptr = 0;
Every malloc() call in this file has this same pattern. Maybe we'd be
better off making it a helper function?
Either way:
Reviewed-by: Vishal Moola <vishal.moola@gmail.com>
^ permalink raw reply
* Re: [PATCH] selftests/cgroup: Add NULL check after malloc in cgroup_util.c
From: Vishal Moola @ 2026-05-11 12:04 UTC (permalink / raw)
To: Hongfu Li
Cc: tj, hannes, mkoutny, shuah, jthoughton, seanjc, zhangguopeng,
cgroups, linux-kselftest, linux-kernel
In-Reply-To: <20260511060853.1873161-1-lihongfu@kylinos.cn>
On Mon, May 11, 2026 at 02:08:53PM +0800, Hongfu Li wrote:
> Add NULL checks after malloc() in three helper functions to prevent
> NULL pointer dereference on memory allocation failure.
> - cg_name()
> - cg_name_indexed()
> - cg_control()
>
> These functions allocate memory with malloc() but previously called
> snprintf() unconditionally, which would trigger undefined behavior
> if allocation fails.
>
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
> ---
Reviewed-by: Vishal Moola <vishal.moola@gmail.com>
^ permalink raw reply
* [PATCH v2 00/10] sched: Flatten the pick
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
Hi!
So cgroup scheduling has always been a pain in the arse. The problems start
with weight distribution and end with hierachical picks and it all sucks.
The problems with weight distribution are related to that infernal global
fraction:
tg->w * grq_i->w
ge_i->w = ----------------
\Sum_j grq_j->w
which we've approximated reasonably well by now. However, the immediate
consequence of this fraction is that the total group weight (tg->w) gets
fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup
weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine
with the fact that 256 CPU systems are relatively common these days, this
becomes painful.
The common 'solution' is to inflate the group weight by 'nr_cpus'; the
immediate problem with that is that when all load of a group gets concentrated
on a single CPU, the per-cpu cgroup weight becomes insanely large, easily
exceeding nice -20.
Additionally there are numerical limits on the max weight you can have before
the math starts suffering overflows. As such there is a definite limit on the
total group weight. Which has annoyed people ;-)
The first few patches add a knob /debug/sched/cgroup_mode and a few different
options on how to deal with this. My favourite is 'concur', but obviously that
is also the most expensive one :-/ It adds a tg->tasks counter which makes the
update_tg_load_avg() thing more expensive.
I have some ideas but I figured I ought to share these things before sinking
more time into it.
On to the hierarchical pick; this has been causing trouble for a very long
time. So once again an attempt at flatting it. The basic idea is to keep the
full hierarchical load tracking as-is, but keep all the runnable entities in a
single level. The immediate concequence of all this is ofcourse that we need to
constantly re-compute the effective weight of each entity as things progress.
Reweight is done on:
- enqueue
- pick -- or rather set_next_entity(.first=true)
- tick
So while the {en,de}queue operations are still O(depth) due to the full
accounting mess, the pick is now a single level. Removing the intermediate
levels that obscure runnability etc.
For testing, I've done a little experiment, I dug out what is colloqually known
as a potato. A trusty old Sandybridge 12600k with a RX 580, and ran a game on
it. From GOG, I had available 'Shadows: Awakens', a fun title that normally
runs really well on this machine (provided you stick to 1080p).
To make it interesting, I added 8 (one for each logical CPU) copies of: 'nice
spin.sh'; this results in the game becoming almost unplayable, as in proper
terrible.
I used MangoHUD to record a few minutes of playtime for statistics, and then
quit the came and re-started it with a shorter slice set (base/10). This
results in the game being entirely playable -- not great, but definiltey
playable.
Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper)
Intel Core i7-2600K
AMD Radeon RX 580
Shadows Awakening (GOG)
default slice(*)
FPS min 3.8 20.6
avg 48.0 57.2
mag 87.4 80.3
FT min 9.4 8.4
avg 34.5 19.5
max 107.4 37.2
FPS (Frames Per Second)
FT (FrameTime)
[*] Command prefix: 'chrt -o --sched-runtime 280000 0'
effectively setting 'base_slice_ns/10'
I have not compared to a kernel without flat on, just wanted to run non trivial
workloads and play with slice to make sure everything 'works'.
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
include/linux/cpuset.h | 6
include/linux/sched.h | 1
kernel/cgroup/cpuset.c | 15
kernel/sched/core.c | 47 --
kernel/sched/debug.c | 171 +++++---
kernel/sched/fair.c | 1038 ++++++++++++++++++++++---------------------------
kernel/sched/pelt.c | 6
kernel/sched/sched.h | 44 --
8 files changed, 672 insertions(+), 656 deletions(-)
---
Change since v1 ( https://patch.msgid.link/20260317095113.387450089@infradead.org ):
- various Sashiko thingies
- rebase atop curren -tip
^ permalink raw reply
* [PATCH v2 02/10] sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
Robots figured out you can read and write this concurrently and got
'upset'. Gemini even noted sched_dynamic_show() can generate
'confusing' output if it observed different values during the
printing.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/core.c | 15 ++++++++-------
kernel/sched/debug.c | 5 +++--
2 files changed, 11 insertions(+), 9 deletions(-)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7743,7 +7743,7 @@ static void __sched_dynamic_update(int m
break;
}
- preempt_dynamic_mode = mode;
+ WRITE_ONCE(preempt_dynamic_mode, mode);
}
void sched_dynamic_update(int mode)
@@ -7784,12 +7784,13 @@ static void __init preempt_dynamic_init(
}
}
-# define PREEMPT_MODEL_ACCESSOR(mode) \
- bool preempt_model_##mode(void) \
- { \
- WARN_ON_ONCE(preempt_dynamic_mode == preempt_dynamic_undefined); \
- return preempt_dynamic_mode == preempt_dynamic_##mode; \
- } \
+# define PREEMPT_MODEL_ACCESSOR(mode) \
+ bool preempt_model_##mode(void) \
+ { \
+ int mode = READ_ONCE(preempt_dynamic_mode); \
+ WARN_ON_ONCE(mode == preempt_dynamic_undefined); \
+ return mode == preempt_dynamic_##mode; \
+ } \
EXPORT_SYMBOL_GPL(preempt_model_##mode)
PREEMPT_MODEL_ACCESSOR(none);
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -239,6 +239,7 @@ static ssize_t sched_dynamic_write(struc
static int sched_dynamic_show(struct seq_file *m, void *v)
{
int i = (IS_ENABLED(CONFIG_PREEMPT_RT) || IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY)) * 2;
+ int mode = READ_ONCE(preempt_dynamic_mode);
int j;
/* Count entries in NULL terminated preempt_modes */
@@ -247,10 +248,10 @@ static int sched_dynamic_show(struct seq
j -= !IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY);
for (; i < j; i++) {
- if (preempt_dynamic_mode == i)
+ if (mode == i)
seq_puts(m, "(");
seq_puts(m, preempt_modes[i]);
- if (preempt_dynamic_mode == i)
+ if (mode == i)
seq_puts(m, ")");
seq_puts(m, " ");
^ permalink raw reply
* [PATCH v2 05/10] sched/fair: Add cgroup_mode: UP
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
Instead of calculating the proportional fraction of tg->shares for
each CPU, just give each CPU the full measure, ignoring these pesky
SMP problems.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 3 ++-
kernel/sched/fair.c | 21 ++++++++++++++++++++-
2 files changed, 22 insertions(+), 2 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -589,9 +589,10 @@ static void debugfs_fair_server_init(voi
}
#ifdef CONFIG_FAIR_GROUP_SCHED
-int cgroup_mode = 0;
+int cgroup_mode = 1;
static const char *cgroup_mode_str[] = {
+ "up",
"smp",
};
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4150,7 +4150,7 @@ static inline int throttled_hierarchy(st
*
* hence icky!
*/
-static long calc_group_shares(struct cfs_rq *cfs_rq)
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
{
long tg_weight, tg_shares, load, shares;
struct task_group *tg = cfs_rq->tg;
@@ -4185,6 +4185,25 @@ static long calc_group_shares(struct cfs
}
/*
+ * Ignore this pesky SMP stuff, use (4).
+ */
+static long calc_up_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ return READ_ONCE(tg->shares);
+}
+
+static long calc_group_shares(struct cfs_rq *cfs_rq)
+{
+ int mode = READ_ONCE(cgroup_mode);
+
+ if (mode == 0)
+ return calc_up_shares(cfs_rq);
+
+ return calc_smp_shares(cfs_rq);
+}
+
+/*
* Recomputes the group entity based on the current state of its group
* runqueue.
*/
^ permalink raw reply
* [PATCH v2 01/10] sched/debug: Use char * instead of char (*)[]
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
Some of the fancy AI robots are getting 'upset'.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -136,7 +136,7 @@ sched_feat_write(struct file *filp, cons
if (cnt > 63)
cnt = 63;
- if (copy_from_user(&buf, ubuf, cnt))
+ if (copy_from_user(buf, ubuf, cnt))
return -EFAULT;
buf[cnt] = 0;
@@ -221,7 +221,7 @@ static ssize_t sched_dynamic_write(struc
if (cnt > 15)
cnt = 15;
- if (copy_from_user(&buf, ubuf, cnt))
+ if (copy_from_user(buf, ubuf, cnt))
return -EFAULT;
buf[cnt] = 0;
^ permalink raw reply
* [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
With commit 50653216e4ff ("sched: Add support to pick functions to
take rf") removing the balance callback, the pick_task() callback is
in charge of newidle balancing.
This means pick_task_fair() should do so too. This hasn't been a
problem in practise because pick_next_task_fair() is used. However,
since we'll be removing that one shortly, make sure pick_next_task()
is up to scratch.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/fair.c | 38 +++++++++++++++-----------------------
1 file changed, 15 insertions(+), 23 deletions(-)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9215,16 +9215,18 @@ static void wakeup_preempt_fair(struct r
}
static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
+ __must_hold(__rq_lockp(rq))
{
struct sched_entity *se;
struct cfs_rq *cfs_rq;
struct task_struct *p;
bool throttled;
+ int new_tasks;
again:
cfs_rq = &rq->cfs;
if (!cfs_rq->nr_queued)
- return NULL;
+ goto idle;
throttled = false;
@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
if (unlikely(throttled))
task_throttle_setup_work(p);
return p;
+
+idle:
+ new_tasks = sched_balance_newidle(rq, rf);
+ if (new_tasks < 0)
+ return RETRY_TASK;
+ if (new_tasks > 0)
+ goto again;
+ return NULL;
}
static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
@@ -9256,12 +9266,12 @@ pick_next_task_fair(struct rq *rq, struc
{
struct sched_entity *se;
struct task_struct *p;
- int new_tasks;
-again:
p = pick_task_fair(rq, rf);
+ if (unlikely(p == RETRY_TASK))
+ return p;
if (!p)
- goto idle;
+ return p;
se = &p->se;
#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9311,29 +9321,11 @@ pick_next_task_fair(struct rq *rq, struc
#endif /* CONFIG_FAIR_GROUP_SCHED */
put_prev_set_next_task(rq, prev, p);
return p;
-
-idle:
- if (rf) {
- new_tasks = sched_balance_newidle(rq, rf);
-
- /*
- * Because sched_balance_newidle() releases (and re-acquires)
- * rq->lock, it is possible for any higher priority task to
- * appear. In that case we must re-start the pick_next_entity()
- * loop.
- */
- if (new_tasks < 0)
- return RETRY_TASK;
-
- if (new_tasks > 0)
- goto again;
- }
-
- return NULL;
}
static struct task_struct *
fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
+ __must_hold(__rq_lockp(dl_se->rq))
{
return pick_task_fair(dl_se->rq, rf);
}
^ permalink raw reply
* [PATCH v2 04/10] sched/fair: Add cgroup_mode switch
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
Since calc_group_shares() has issues with 'many' CPUs, specifically the
computed shares value gets to be roughly 1/nr_cpus, prepare to add a few
alternative methods.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 1
2 files changed, 75 insertions(+)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -588,6 +588,76 @@ static void debugfs_fair_server_init(voi
}
}
+#ifdef CONFIG_FAIR_GROUP_SCHED
+int cgroup_mode = 0;
+
+static const char *cgroup_mode_str[] = {
+ "smp",
+};
+
+static int sched_cgroup_mode(const char *str)
+{
+ for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+ if (!strcmp(str, cgroup_mode_str[i]))
+ return i;
+ }
+ return -EINVAL;
+}
+
+static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[16];
+ int mode;
+
+ if (cnt > 15)
+ cnt = 15;
+
+ if (copy_from_user(buf, ubuf, cnt))
+ return -EFAULT;
+
+ buf[cnt] = 0;
+ mode = sched_cgroup_mode(strstrip(buf));
+ if (mode < 0)
+ return mode;
+
+ WRITE_ONCE(cgroup_mode, mode);
+
+ *ppos += cnt;
+ return cnt;
+}
+
+static int sched_cgroup_show(struct seq_file *m, void *v)
+{
+ int mode = READ_ONCE(cgroup_mode);
+
+ for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+ if (mode == i)
+ seq_puts(m, "(");
+ seq_puts(m, cgroup_mode_str[i]);
+ if (mode == i)
+ seq_puts(m, ")");
+
+ seq_puts(m, " ");
+ }
+ seq_puts(m, "\n");
+ return 0;
+}
+
+static int sched_cgroup_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, sched_cgroup_show, NULL);
+}
+
+static const struct file_operations sched_cgroup_fops = {
+ .open = sched_cgroup_open,
+ .write = sched_cgroup_write,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+#endif
+
static __init int sched_init_debug(void)
{
struct dentry __maybe_unused *numa;
@@ -625,6 +695,10 @@ static __init int sched_init_debug(void)
debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
+#ifdef CONFIG_FAIR_GROUP_SCHED
+ debugfs_create_file("cgroup_mode", 0644, debugfs_sched, NULL, &sched_cgroup_fops);
+#endif
+
debugfs_fair_server_init();
#ifdef CONFIG_SCHED_CLASS_EXT
debugfs_ext_server_init();
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -565,6 +565,7 @@ static inline struct task_group *css_tg(
extern int tg_nop(struct task_group *tg, void *data);
#ifdef CONFIG_FAIR_GROUP_SCHED
+extern int cgroup_mode;
extern void free_fair_sched_group(struct task_group *tg);
extern int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent);
extern void online_fair_sched_group(struct task_group *tg);
^ permalink raw reply
* [PATCH v2 06/10] sched/fair: Add cgroup_mode: MAX
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
In order to avoid the CPU shares becoming tiny '1 / nr_cpus', assume each
cgroup is maximally concurrent and distrubute 'nr_cpus * tg->shares',
such that each CPU ends up with a 'tg->shares' sized fraction (on
average).
There is the corner case, when a cgroup is minimally loaded, eg a
single spinner, therefore limit the CPU shares to that of a nice -20
task to avoid getting too much load.
It was previously suggested to allow raising cpu.weight to '100 * nr_cpus'
to combat this same problem, but the problem there is the above corner case,
allowing multiple cgroups with such immense weight to the runqueue has
significant problems.
It would drown the kthreads, but it also risks overflowing the load values.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/cpuset.h | 6 +++++
kernel/cgroup/cpuset.c | 15 ++++++++++++++
kernel/sched/debug.c | 1
kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-----
4 files changed, 69 insertions(+), 5 deletions(-)
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern int cpuset_num_cpus(struct cgroup *cgroup);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
#define cpuset_current_mems_allowed (current->mems_allowed)
void cpuset_init_current_mems_allowed(void);
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f
return false;
}
+static inline int cpuset_num_cpus(struct cgroup *cgroup)
+{
+ return num_online_cpus();
+}
+
static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
{
return node_possible_map;
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4100,6 +4100,21 @@ bool cpuset_cpus_allowed_fallback(struct
return changed;
}
+int cpuset_num_cpus(struct cgroup *cgrp)
+{
+ int nr = num_online_cpus();
+ struct cpuset *cs;
+
+ if (is_in_v2_mode()) {
+ guard(rcu)();
+ cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
+ if (cs)
+ nr = cpumask_weight(cs->effective_cpus);
+ }
+
+ return nr;
+}
+
void __init cpuset_init_current_mems_allowed(void)
{
nodes_setall(current->mems_allowed);
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -594,6 +594,7 @@ int cgroup_mode = 1;
static const char *cgroup_mode_str[] = {
"up",
"smp",
+ "max",
};
static int sched_cgroup_mode(const char *str)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4150,12 +4150,10 @@ static inline int throttled_hierarchy(st
*
* hence icky!
*/
-static long calc_smp_shares(struct cfs_rq *cfs_rq)
+static long __calc_smp_shares(struct cfs_rq *cfs_rq, long tg_shares, long shares_max)
{
- long tg_weight, tg_shares, load, shares;
struct task_group *tg = cfs_rq->tg;
-
- tg_shares = READ_ONCE(tg->shares);
+ long tg_weight, load, shares;
load = max(scale_load_down(cfs_rq->load.weight), cfs_rq->avg.load_avg);
@@ -4181,7 +4179,48 @@ static long calc_smp_shares(struct cfs_r
* case no task is runnable on a CPU MIN_SHARES=2 should be returned
* instead of 0.
*/
- return clamp_t(long, shares, MIN_SHARES, tg_shares);
+ return clamp_t(long, shares, MIN_SHARES, shares_max);
+}
+
+static int tg_cpus(struct task_group *tg)
+{
+ int nr = num_online_cpus();
+
+ if (cpusets_enabled()) {
+ struct cgroup *cgrp = tg->css.cgroup;
+ if (cgrp)
+ nr = cpuset_num_cpus(cgrp);
+ }
+
+ return nr;
+}
+
+/*
+ * Func: min(fraction(num_cpus * tg->shares), nice -20)
+ *
+ * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
+ * nice -20, otherwise a single spinner on a 512 CPU machine would result in
+ * 512*NICE_0_LOAD, which is also crazy.
+ */
+static long calc_max_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ int nr = tg_cpus(tg);
+ long tg_shares = READ_ONCE(tg->shares);
+ long max_shares = scale_load(sched_prio_to_weight[0]);
+ return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares);
+}
+
+/*
+ * Func: fraction(tg->shares)
+ *
+ * This infamously results in tiny shares when you have many CPUs.
+ */
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ long tg_shares = READ_ONCE(tg->shares);
+ return __calc_smp_shares(cfs_rq, tg_shares, tg_shares);
}
/*
@@ -4200,6 +4239,9 @@ static long calc_group_shares(struct cfs
if (mode == 0)
return calc_up_shares(cfs_rq);
+ if (mode == 2)
+ return calc_max_shares(cfs_rq);
+
return calc_smp_shares(cfs_rq);
}
^ permalink raw reply
* [PATCH v2 09/10] sched: Remove sched_class::pick_next_task()
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
The reason for pick_next_task_fair() is the put/set optimization that
avoids touching the common ancestors. However, it is possible to
implement this in the put_prev_task() and set_next_task() calls as
used in put_prev_set_next_task().
Notably, put_prev_set_next_task() is the only site that:
- calls put_prev_task() with a .next argument;
- calls set_next_task() with .first = true.
This means that put_prev_task() can determine the common hierarchy and
stop there, and then set_next_task() can terminate where put_prev_task
stopped.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/core.c | 27 +++------
kernel/sched/fair.c | 139 +++++++++++++++++----------------------------------
kernel/sched/sched.h | 14 -----
3 files changed, 57 insertions(+), 123 deletions(-)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5980,16 +5980,15 @@ __pick_next_task(struct rq *rq, struct t
if (likely(!sched_class_above(prev->sched_class, &fair_sched_class) &&
rq->nr_running == rq->cfs.h_nr_queued)) {
- p = pick_next_task_fair(rq, prev, rf);
+ p = pick_task_fair(rq, rf);
if (unlikely(p == RETRY_TASK))
goto restart;
/* Assume the next prioritized class is idle_sched_class */
- if (!p) {
+ if (!p)
p = pick_task_idle(rq, rf);
- put_prev_set_next_task(rq, prev, p);
- }
+ put_prev_set_next_task(rq, prev, p);
return p;
}
@@ -5997,20 +5996,12 @@ __pick_next_task(struct rq *rq, struct t
prev_balance(rq, prev, rf);
for_each_active_class(class) {
- if (class->pick_next_task) {
- p = class->pick_next_task(rq, prev, rf);
- if (unlikely(p == RETRY_TASK))
- goto restart;
- if (p)
- return p;
- } else {
- p = class->pick_task(rq, rf);
- if (unlikely(p == RETRY_TASK))
- goto restart;
- if (p) {
- put_prev_set_next_task(rq, prev, p);
- return p;
- }
+ p = class->pick_task(rq, rf);
+ if (unlikely(p == RETRY_TASK))
+ goto restart;
+ if (p) {
+ put_prev_set_next_task(rq, prev, p);
+ return p;
}
}
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9214,7 +9214,7 @@ static void wakeup_preempt_fair(struct r
resched_curr_lazy(rq);
}
-static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
+struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
__must_hold(__rq_lockp(rq))
{
struct sched_entity *se;
@@ -9257,72 +9257,6 @@ static struct task_struct *pick_task_fai
return NULL;
}
-static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
-static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
-
-struct task_struct *
-pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
- __must_hold(__rq_lockp(rq))
-{
- struct sched_entity *se;
- struct task_struct *p;
-
- p = pick_task_fair(rq, rf);
- if (unlikely(p == RETRY_TASK))
- return p;
- if (!p)
- return p;
- se = &p->se;
-
-#ifdef CONFIG_FAIR_GROUP_SCHED
- if (prev->sched_class != &fair_sched_class)
- goto simple;
-
- __put_prev_set_next_dl_server(rq, prev, p);
-
- /*
- * Because of the set_next_buddy() in dequeue_task_fair() it is rather
- * likely that a next task is from the same cgroup as the current.
- *
- * Therefore attempt to avoid putting and setting the entire cgroup
- * hierarchy, only change the part that actually changes.
- *
- * Since we haven't yet done put_prev_entity and if the selected task
- * is a different task than we started out with, try and touch the
- * least amount of cfs_rqs.
- */
- if (prev != p) {
- struct sched_entity *pse = &prev->se;
- struct cfs_rq *cfs_rq;
-
- while (!(cfs_rq = is_same_group(se, pse))) {
- int se_depth = se->depth;
- int pse_depth = pse->depth;
-
- if (se_depth <= pse_depth) {
- put_prev_entity(cfs_rq_of(pse), pse);
- pse = parent_entity(pse);
- }
- if (se_depth >= pse_depth) {
- set_next_entity(cfs_rq_of(se), se, true);
- se = parent_entity(se);
- }
- }
-
- put_prev_entity(cfs_rq, pse);
- set_next_entity(cfs_rq, se, true);
-
- __set_next_task_fair(rq, p, true);
- }
-
- return p;
-
-simple:
-#endif /* CONFIG_FAIR_GROUP_SCHED */
- put_prev_set_next_task(rq, prev, p);
- return p;
-}
-
static struct task_struct *
fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
__must_hold(__rq_lockp(dl_se->rq))
@@ -9346,10 +9280,33 @@ static void put_prev_task_fair(struct rq
{
struct sched_entity *se = &prev->se;
struct cfs_rq *cfs_rq;
+ struct sched_entity *nse = NULL;
- for_each_sched_entity(se) {
+#ifdef CONFIG_FAIR_GROUP_SCHED
+ if (next && next->sched_class == &fair_sched_class)
+ nse = &next->se;
+#endif
+
+ while (se) {
cfs_rq = cfs_rq_of(se);
- put_prev_entity(cfs_rq, se);
+ if (!nse || cfs_rq->curr)
+ put_prev_entity(cfs_rq, se);
+#ifdef CONFIG_FAIR_GROUP_SCHED
+ if (nse) {
+ if (is_same_group(se, nse))
+ break;
+
+ int d = nse->depth - se->depth;
+ if (d >= 0) {
+ /* nse has equal or greater depth, ascend */
+ nse = parent_entity(nse);
+ /* if nse is the deeper, do not ascend se */
+ if (d > 0)
+ continue;
+ }
+ }
+#endif
+ se = parent_entity(se);
}
}
@@ -13896,10 +13853,30 @@ static void switched_to_fair(struct rq *
}
}
-static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
+/*
+ * Account for a task changing its policy or group.
+ *
+ * This routine is mostly called to set cfs_rq->curr field when a task
+ * migrates between groups/classes.
+ */
+static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_entity *se = &p->se;
+ for_each_sched_entity(se) {
+ struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+ if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
+ first && cfs_rq->curr)
+ break;
+
+ set_next_entity(cfs_rq, se, first);
+ /* ensure bandwidth has been allocated on our new cfs_rq */
+ account_cfs_rq_runtime(cfs_rq, 0);
+ }
+
+ se = &p->se;
+
if (task_on_rq_queued(p)) {
/*
* Move the next running task to the front of the list, so our
@@ -13919,27 +13896,6 @@ static void __set_next_task_fair(struct
sched_fair_update_stop_tick(rq, p);
}
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
-static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
-{
- struct sched_entity *se = &p->se;
-
- for_each_sched_entity(se) {
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
- set_next_entity(cfs_rq, se, first);
- /* ensure bandwidth has been allocated on our new cfs_rq */
- account_cfs_rq_runtime(cfs_rq, 0);
- }
-
- __set_next_task_fair(rq, p, first);
-}
-
void init_cfs_rq(struct cfs_rq *cfs_rq)
{
cfs_rq->tasks_timeline = RB_ROOT_CACHED;
@@ -14251,7 +14207,6 @@ DEFINE_SCHED_CLASS(fair) = {
.wakeup_preempt = wakeup_preempt_fair,
.pick_task = pick_task_fair,
- .pick_next_task = pick_next_task_fair,
.put_prev_task = put_prev_task_fair,
.set_next_task = set_next_task_fair,
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2555,17 +2555,6 @@ struct sched_class {
* schedule/pick_next_task: rq->lock
*/
struct task_struct *(*pick_task)(struct rq *rq, struct rq_flags *rf);
- /*
- * Optional! When implemented pick_next_task() should be equivalent to:
- *
- * next = pick_task();
- * if (next) {
- * put_prev_task(prev);
- * set_next_task_first(next);
- * }
- */
- struct task_struct *(*pick_next_task)(struct rq *rq, struct task_struct *prev,
- struct rq_flags *rf);
/*
* sched_change:
@@ -2789,8 +2778,7 @@ static inline bool sched_fair_runnable(s
return rq->cfs.nr_queued > 0;
}
-extern struct task_struct *pick_next_task_fair(struct rq *rq, struct task_struct *prev,
- struct rq_flags *rf);
+extern struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf);
extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_CHECK 0x01
^ permalink raw reply
* [PATCH v2 03/10] sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 92 ++++++++++++++++++++++++---------------------------
1 file changed, 44 insertions(+), 48 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -446,6 +446,8 @@ static const struct file_operations fair
.release = single_release,
};
+static struct dentry *debugfs_sched;
+
#ifdef CONFIG_SCHED_CLASS_EXT
static ssize_t
sched_ext_server_runtime_write(struct file *filp, const char __user *ubuf,
@@ -478,75 +480,92 @@ static const struct file_operations ext_
.llseek = seq_lseek,
.release = single_release,
};
-#endif /* CONFIG_SCHED_CLASS_EXT */
static ssize_t
-sched_fair_server_period_write(struct file *filp, const char __user *ubuf,
- size_t cnt, loff_t *ppos)
+sched_ext_server_period_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
{
long cpu = (long) ((struct seq_file *) filp->private_data)->private;
struct rq *rq = cpu_rq(cpu);
return sched_server_write_common(filp, ubuf, cnt, ppos, DL_PERIOD,
- &rq->fair_server);
+ &rq->ext_server);
}
-static int sched_fair_server_period_show(struct seq_file *m, void *v)
+static int sched_ext_server_period_show(struct seq_file *m, void *v)
{
unsigned long cpu = (unsigned long) m->private;
struct rq *rq = cpu_rq(cpu);
- return sched_server_show_common(m, v, DL_PERIOD, &rq->fair_server);
+ return sched_server_show_common(m, v, DL_PERIOD, &rq->ext_server);
}
-static int sched_fair_server_period_open(struct inode *inode, struct file *filp)
+static int sched_ext_server_period_open(struct inode *inode, struct file *filp)
{
- return single_open(filp, sched_fair_server_period_show, inode->i_private);
+ return single_open(filp, sched_ext_server_period_show, inode->i_private);
}
-static const struct file_operations fair_server_period_fops = {
- .open = sched_fair_server_period_open,
- .write = sched_fair_server_period_write,
+static const struct file_operations ext_server_period_fops = {
+ .open = sched_ext_server_period_open,
+ .write = sched_ext_server_period_write,
.read = seq_read,
.llseek = seq_lseek,
.release = single_release,
};
-#ifdef CONFIG_SCHED_CLASS_EXT
+static void debugfs_ext_server_init(void)
+{
+ struct dentry *d_ext;
+ unsigned long cpu;
+
+ d_ext = debugfs_create_dir("ext_server", debugfs_sched);
+ if (!d_ext)
+ return;
+
+ for_each_possible_cpu(cpu) {
+ struct dentry *d_cpu;
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "cpu%lu", cpu);
+ d_cpu = debugfs_create_dir(buf, d_ext);
+
+ debugfs_create_file("runtime", 0644, d_cpu, (void *) cpu, &ext_server_runtime_fops);
+ debugfs_create_file("period", 0644, d_cpu, (void *) cpu, &ext_server_period_fops);
+ }
+}
+#endif /* CONFIG_SCHED_CLASS_EXT */
+
static ssize_t
-sched_ext_server_period_write(struct file *filp, const char __user *ubuf,
- size_t cnt, loff_t *ppos)
+sched_fair_server_period_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
{
long cpu = (long) ((struct seq_file *) filp->private_data)->private;
struct rq *rq = cpu_rq(cpu);
return sched_server_write_common(filp, ubuf, cnt, ppos, DL_PERIOD,
- &rq->ext_server);
+ &rq->fair_server);
}
-static int sched_ext_server_period_show(struct seq_file *m, void *v)
+static int sched_fair_server_period_show(struct seq_file *m, void *v)
{
unsigned long cpu = (unsigned long) m->private;
struct rq *rq = cpu_rq(cpu);
- return sched_server_show_common(m, v, DL_PERIOD, &rq->ext_server);
+ return sched_server_show_common(m, v, DL_PERIOD, &rq->fair_server);
}
-static int sched_ext_server_period_open(struct inode *inode, struct file *filp)
+static int sched_fair_server_period_open(struct inode *inode, struct file *filp)
{
- return single_open(filp, sched_ext_server_period_show, inode->i_private);
+ return single_open(filp, sched_fair_server_period_show, inode->i_private);
}
-static const struct file_operations ext_server_period_fops = {
- .open = sched_ext_server_period_open,
- .write = sched_ext_server_period_write,
+static const struct file_operations fair_server_period_fops = {
+ .open = sched_fair_server_period_open,
+ .write = sched_fair_server_period_write,
.read = seq_read,
.llseek = seq_lseek,
.release = single_release,
};
-#endif /* CONFIG_SCHED_CLASS_EXT */
-
-static struct dentry *debugfs_sched;
static void debugfs_fair_server_init(void)
{
@@ -569,29 +588,6 @@ static void debugfs_fair_server_init(voi
}
}
-#ifdef CONFIG_SCHED_CLASS_EXT
-static void debugfs_ext_server_init(void)
-{
- struct dentry *d_ext;
- unsigned long cpu;
-
- d_ext = debugfs_create_dir("ext_server", debugfs_sched);
- if (!d_ext)
- return;
-
- for_each_possible_cpu(cpu) {
- struct dentry *d_cpu;
- char buf[32];
-
- snprintf(buf, sizeof(buf), "cpu%lu", cpu);
- d_cpu = debugfs_create_dir(buf, d_ext);
-
- debugfs_create_file("runtime", 0644, d_cpu, (void *) cpu, &ext_server_runtime_fops);
- debugfs_create_file("period", 0644, d_cpu, (void *) cpu, &ext_server_period_fops);
- }
-}
-#endif /* CONFIG_SCHED_CLASS_EXT */
-
static __init int sched_init_debug(void)
{
struct dentry __maybe_unused *numa;
^ permalink raw reply
* [PATCH v2 07/10] sched/fair: Add cgroup_mode: CONCUR
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
A variation of MAX; where instead of assuming maximal concurrent, this scales
with 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more
gracefully, with the exception of CPU affnity.
Note: the tracking of tg->tasks is somewhat expensive :-/
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 1 +
kernel/sched/fair.c | 39 ++++++++++++++++++++++++++++++++++++---
kernel/sched/sched.h | 3 +++
3 files changed, 40 insertions(+), 3 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -594,6 +594,7 @@ int cgroup_mode = 1;
static const char *cgroup_mode_str[] = {
"up",
"smp",
+ "concur",
"max",
};
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4211,6 +4211,30 @@ static long calc_max_shares(struct cfs_r
return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares);
}
+static inline int tg_tasks(struct task_group *tg)
+{
+ return max(1, atomic_long_read(&tg->tasks));
+}
+
+/*
+ * Func: min(fraction(num * tg->shares), nice -20); where
+ * num = min(nr_tasks, nr_cpus)
+ *
+ * Similar to max, except scale with min(nr_tasks, nr_cpus), which gives
+ * a far more natural distrubution. Can still create edge case using CPU
+ * affinity.
+ */
+static long calc_concur_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ int nr_cpus = tg_cpus(tg);
+ int nr_tasks = tg_tasks(tg);
+ int nr = min(nr_tasks, nr_cpus);
+ long tg_shares = READ_ONCE(tg->shares);
+ long max_shares = scale_load(sched_prio_to_weight[0]);
+ return __calc_smp_shares(cfs_rq, nr * tg_shares, max_shares);
+}
+
/*
* Func: fraction(tg->shares)
*
@@ -4240,6 +4264,9 @@ static long calc_group_shares(struct cfs
return calc_up_shares(cfs_rq);
if (mode == 2)
+ return calc_concur_shares(cfs_rq);
+
+ if (mode == 3)
return calc_max_shares(cfs_rq);
return calc_smp_shares(cfs_rq);
@@ -4385,7 +4412,7 @@ static inline bool cfs_rq_is_decayed(str
*/
static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
{
- long delta;
+ long delta, dt;
u64 now;
/*
@@ -4407,16 +4434,19 @@ static inline void update_tg_load_avg(st
return;
delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
- if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
+ dt = cfs_rq->h_nr_queued - cfs_rq->tg_tasks_contrib;
+ if (dt || abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
atomic_long_add(delta, &cfs_rq->tg->load_avg);
+ atomic_long_add(dt, &cfs_rq->tg->tasks);
cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg;
+ cfs_rq->tg_tasks_contrib = cfs_rq->h_nr_queued;
cfs_rq->last_update_tg_load_avg = now;
}
}
static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq)
{
- long delta;
+ long delta, dt;
u64 now;
/*
@@ -4427,8 +4457,11 @@ static inline void clear_tg_load_avg(str
now = sched_clock_cpu(cpu_of(rq_of(cfs_rq)));
delta = 0 - cfs_rq->tg_load_avg_contrib;
+ dt = 0 - cfs_rq->tg_tasks_contrib;
atomic_long_add(delta, &cfs_rq->tg->load_avg);
+ atomic_long_add(dt, &cfs_rq->tg->tasks);
cfs_rq->tg_load_avg_contrib = 0;
+ cfs_rq->tg_tasks_contrib = 0;
cfs_rq->last_update_tg_load_avg = now;
}
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -491,6 +491,8 @@ struct task_group {
* will also be accessed at each tick.
*/
atomic_long_t load_avg ____cacheline_aligned;
+ atomic_long_t tasks;
+
#endif /* CONFIG_FAIR_GROUP_SCHED */
#ifdef CONFIG_RT_GROUP_SCHED
@@ -720,6 +722,7 @@ struct cfs_rq {
#ifdef CONFIG_FAIR_GROUP_SCHED
u64 last_update_tg_load_avg;
unsigned long tg_load_avg_contrib;
+ unsigned long tg_tasks_contrib;
long propagate;
long prop_runnable_sum;
^ permalink raw reply
* [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-11 11:31 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511113104.563854162@infradead.org>
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically
the complaint is latencies and/or overhead. The latency issue is due
to the intermediate entries that represent a combination of tasks and
thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including
the intermediate enqueue/dequeue but move the actual EEVDF runqueue
outside. This means things like the shares_weight approximation are
fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |
T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of
cfs_rq/se go unused for the intermediates and are instead connected
like:
_R_
/ | \
T1 T2 T3
Since the effective weight of the entities is determined by the
hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the
hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusive operating on rq->cfs, it needs to
consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was
set_next_task() / put_prev_task(), where we need to track 'current'
both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
(cfs_rq->curr).
As a result of only having a single level to pick from, much of the
complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't
immediately fix the performance issues, but hopefully it will fix some
of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/sched.h | 1
kernel/sched/core.c | 5
kernel/sched/debug.c | 9
kernel/sched/fair.c | 789 +++++++++++++++++++++-----------------------------
kernel/sched/pelt.c | 6
kernel/sched/sched.h | 26 -
6 files changed, 366 insertions(+), 470 deletions(-)
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -575,6 +575,7 @@ struct sched_statistics {
struct sched_entity {
/* For load-balancing: */
struct load_weight load;
+ struct load_weight h_load;
struct rb_node run_node;
u64 deadline;
u64 min_vruntime;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
*/
static inline void prefetch_curr_exec_start(struct task_struct *p)
{
-#ifdef CONFIG_FAIR_GROUP_SCHED
- struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
prefetch(curr);
prefetch(&curr->exec_start);
}
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rq
else
SEQ_printf(m, " %c", task_state_to_char(p));
- SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
+ SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
p->comm, task_pid_nr(p),
+ p->se.h_load.weight,
SPLIT_NS(p->se.vruntime),
- entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+ entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
SPLIT_NS(p->se.deadline),
p->se.custom_slice ? 'S' : ' ',
SPLIT_NS(p->se.slice),
@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,
SEQ_printf(m, "\n");
SEQ_printf(m, "runnable tasks:\n");
- SEQ_printf(m, " S task PID vruntime eligible "
+ SEQ_printf(m, " S task PID weight vruntime eligible "
"deadline slice sum-exec switches "
"prio wait-time sum-sleep sum-block"
#ifdef CONFIG_NUMA_BALANCING
@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in
cfs_rq->tg_load_avg_contrib);
SEQ_printf(m, " .%-30s: %ld\n", "tg_load_avg",
atomic_long_read(&cfs_rq->tg->load_avg));
+ SEQ_printf(m, " .%-30s: %lu\n", "h_load",
+ cfs_rq->h_load);
#endif /* CONFIG_FAIR_GROUP_SCHED */
#ifdef CONFIG_CFS_BANDWIDTH
SEQ_printf(m, " .%-30s: %d\n", "throttled",
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -296,8 +296,8 @@ static u64 __calc_delta(u64 delta_exec,
*/
static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
{
- if (unlikely(se->load.weight != NICE_0_LOAD))
- delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
+ if (se->h_load.weight != NICE_0_LOAD)
+ delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
return delta;
}
@@ -427,38 +427,6 @@ static inline struct sched_entity *paren
return se->parent;
}
-static void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
- int se_depth, pse_depth;
-
- /*
- * preemption test can be made between sibling entities who are in the
- * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
- * both tasks until we find their ancestors who are siblings of common
- * parent.
- */
-
- /* First walk up until both entities are at same depth */
- se_depth = (*se)->depth;
- pse_depth = (*pse)->depth;
-
- while (se_depth > pse_depth) {
- se_depth--;
- *se = parent_entity(*se);
- }
-
- while (pse_depth > se_depth) {
- pse_depth--;
- *pse = parent_entity(*pse);
- }
-
- while (!is_same_group(*se, *pse)) {
- *se = parent_entity(*se);
- *pse = parent_entity(*pse);
- }
-}
-
static int tg_is_idle(struct task_group *tg)
{
return tg->idle > 0;
@@ -502,11 +470,6 @@ static inline struct sched_entity *paren
return NULL;
}
-static inline void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-}
-
static inline int tg_is_idle(struct task_group *tg)
{
return 0;
@@ -685,7 +648,7 @@ static inline unsigned long avg_vruntime
static inline void
__sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
s64 w_vruntime, key = entity_key(cfs_rq, se);
w_vruntime = key * weight;
@@ -702,7 +665,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
s64 key, tmp;
again:
- weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
key = entity_key(cfs_rq, se);
if (check_mul_overflow(key, weight, &key))
@@ -748,7 +711,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
static void
sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
s64 key = entity_key(cfs_rq, se);
cfs_rq->sum_w_vruntime -= key * weight;
@@ -790,7 +753,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
s64 runtime = cfs_rq->sum_w_vruntime;
if (curr) {
- unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
+ unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
runtime += entity_key(cfs_rq, curr) * w;
weight += w;
@@ -861,8 +824,6 @@ bool update_entity_lag(struct cfs_rq *cf
u64 avruntime = avg_vruntime(cfs_rq);
s64 vlag = entity_lag(cfs_rq, se, avruntime);
- WARN_ON_ONCE(!se->on_rq);
-
if (se->sched_delayed) {
/* previous vlag < 0 otherwise se would not be delayed */
vlag = max(vlag, se->vlag);
@@ -898,7 +859,7 @@ static int vruntime_eligible(struct cfs_
long load = cfs_rq->sum_weight;
if (curr && curr->on_rq) {
- unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+ unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
avg += entity_key(cfs_rq, curr) * weight;
load += weight;
@@ -1039,6 +1000,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
*/
static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+ WARN_ON_ONCE(!entity_is_task(se));
+
sum_w_vruntime_add(cfs_rq, se);
se->min_vruntime = se->vruntime;
se->min_slice = se->slice;
@@ -1048,6 +1012,9 @@ static void __enqueue_entity(struct cfs_
static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+ WARN_ON_ONCE(!entity_is_task(se));
+
rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
&min_vruntime_cb);
sum_w_vruntime_sub(cfs_rq, se);
@@ -1144,7 +1111,7 @@ static struct sched_entity *pick_eevdf(s
* We can safely skip eligibility check if there is only one entity
* in this cfs_rq, saving some cycles.
*/
- if (cfs_rq->nr_queued == 1)
+ if (cfs_rq->h_nr_queued == 1)
return curr && curr->on_rq ? curr : se;
/*
@@ -1391,8 +1358,6 @@ static s64 update_se(struct rq *rq, stru
return delta_exec;
}
-static void set_next_buddy(struct sched_entity *se);
-
/*
* Used by other classes to account runtime.
*/
@@ -1412,7 +1377,7 @@ static void update_curr(struct cfs_rq *c
* not necessarily be the actual task running
* (rq->curr.se). This is easy to confuse!
*/
- struct sched_entity *curr = cfs_rq->curr;
+ struct sched_entity *curr = cfs_rq->h_curr;
struct rq *rq = rq_of(cfs_rq);
s64 delta_exec;
bool resched;
@@ -1424,26 +1389,29 @@ static void update_curr(struct cfs_rq *c
if (unlikely(delta_exec <= 0))
return;
+ account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+ if (!entity_is_task(curr))
+ return;
+
+ cfs_rq = &rq->cfs;
+
curr->vruntime += calc_delta_fair(delta_exec, curr);
resched = update_deadline(cfs_rq, curr);
- if (entity_is_task(curr)) {
- /*
- * If the fair_server is active, we need to account for the
- * fair_server time whether or not the task is running on
- * behalf of fair_server or not:
- * - If the task is running on behalf of fair_server, we need
- * to limit its time based on the assigned runtime.
- * - Fair task that runs outside of fair_server should account
- * against fair_server such that it can account for this time
- * and possibly avoid running this period.
- */
- dl_server_update(&rq->fair_server, delta_exec);
- }
-
- account_cfs_rq_runtime(cfs_rq, delta_exec);
+ /*
+ * If the fair_server is active, we need to account for the
+ * fair_server time whether or not the task is running on
+ * behalf of fair_server or not:
+ * - If the task is running on behalf of fair_server, we need
+ * to limit its time based on the assigned runtime.
+ * - Fair task that runs outside of fair_server should account
+ * against fair_server such that it can account for this time
+ * and possibly avoid running this period.
+ */
+ dl_server_update(&rq->fair_server, delta_exec);
- if (cfs_rq->nr_queued == 1)
+ if (cfs_rq->h_nr_queued == 1)
return;
if (resched || !protect_slice(curr)) {
@@ -1454,7 +1422,10 @@ static void update_curr(struct cfs_rq *c
static void update_curr_fair(struct rq *rq)
{
- update_curr(cfs_rq_of(&rq->donor->se));
+ struct sched_entity *se = &rq->donor->se;
+
+ for_each_sched_entity(se)
+ update_curr(cfs_rq_of(se));
}
static inline void
@@ -1530,7 +1501,7 @@ update_stats_enqueue_fair(struct cfs_rq
* Are we enqueueing a waiting task? (for current tasks
* a dequeue/enqueue event is a NOP)
*/
- if (se != cfs_rq->curr)
+ if (se != cfs_rq->h_curr)
update_stats_wait_start_fair(cfs_rq, se);
if (flags & ENQUEUE_WAKEUP)
@@ -1548,7 +1519,7 @@ update_stats_dequeue_fair(struct cfs_rq
* Mark the end of the wait period if dequeueing a
* waiting task:
*/
- if (se != cfs_rq->curr)
+ if (se != cfs_rq->h_curr)
update_stats_wait_end_fair(cfs_rq, se);
if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
@@ -3875,6 +3846,7 @@ static inline void update_scan_period(st
static void
account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
update_load_add(&cfs_rq->load, se->load.weight);
if (entity_is_task(se)) {
struct rq *rq = rq_of(cfs_rq);
@@ -3888,6 +3860,7 @@ account_entity_enqueue(struct cfs_rq *cf
static void
account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
update_load_sub(&cfs_rq->load, se->load.weight);
if (entity_is_task(se)) {
account_numa_dequeue(rq_of(cfs_rq), task_of(se));
@@ -3965,7 +3938,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
static void
rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
{
- unsigned long old_weight = se->load.weight;
+ long old_weight = se->h_load.weight;
/*
* VRUNTIME
@@ -4065,16 +4038,17 @@ rescale_entity(struct sched_entity *se,
se->vprot = div64_long(se->vprot * old_weight, weight);
}
-static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
- unsigned long weight)
+static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
+ unsigned long weight, bool on_rq)
{
bool curr = cfs_rq->curr == se;
bool rel_vprot = false;
u64 avruntime = 0;
- if (se->on_rq) {
- /* commit outstanding execution time */
- update_curr(cfs_rq);
+ if (se->h_load.weight == weight)
+ return;
+
+ if (on_rq) {
avruntime = avg_vruntime(cfs_rq);
se->vlag = entity_lag(cfs_rq, se, avruntime);
se->deadline -= avruntime;
@@ -4084,46 +4058,90 @@ static void reweight_entity(struct cfs_r
rel_vprot = true;
}
- cfs_rq->nr_queued--;
+ cfs_rq->h_nr_queued--;
if (!curr)
__dequeue_entity(cfs_rq, se);
- update_load_sub(&cfs_rq->load, se->load.weight);
}
- dequeue_load_avg(cfs_rq, se);
rescale_entity(se, weight, rel_vprot);
- update_load_set(&se->load, weight);
+ update_load_set(&se->h_load, weight);
- do {
- u32 divider = get_pelt_divider(&se->avg);
- se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
- } while (0);
-
- enqueue_load_avg(cfs_rq, se);
- if (se->on_rq) {
+ if (on_rq) {
if (rel_vprot)
se->vprot += avruntime;
se->deadline += avruntime;
se->rel_deadline = 0;
se->vruntime = avruntime - se->vlag;
- update_load_add(&cfs_rq->load, se->load.weight);
if (!curr)
__enqueue_entity(cfs_rq, se);
- cfs_rq->nr_queued++;
+ cfs_rq->h_nr_queued++;
}
}
+static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
+ unsigned long weight)
+{
+ if (se->load.weight == weight)
+ return;
+
+ if (se->on_rq) {
+ WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
+ update_load_sub(&cfs_rq->load, se->load.weight);
+ }
+ dequeue_load_avg(cfs_rq, se);
+
+ update_load_set(&se->load, weight);
+
+ do {
+ u32 divider = get_pelt_divider(&se->avg);
+ se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
+ } while (0);
+
+ enqueue_load_avg(cfs_rq, se);
+
+ if (se->on_rq)
+ update_load_add(&cfs_rq->load, se->load.weight);
+}
+
+/*
+ * weight = NICE_0_LOAD;
+ * for_each_entity_se(se)
+ * weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+ */
+static __always_inline
+unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
+ unsigned long weight)
+{
+ weight *= se->load.weight;
+ if (parent_entity(se))
+ weight /= cfs_rq->load.weight;
+ else
+ weight /= NICE_0_LOAD;
+
+ return max(weight, MIN_SHARES);
+}
+
static void reweight_task_fair(struct rq *rq, struct task_struct *p,
const struct load_weight *lw)
{
struct sched_entity *se = &p->se;
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
- struct load_weight *load = &se->load;
+ unsigned long weight = NICE_0_LOAD;
+
+ if (se->on_rq)
+ update_curr_fair(rq);
+
+ reweight_entity(cfs_rq_of(se), se, lw->weight);
+ se->load.inv_weight = lw->inv_weight;
+
+ if (!se->on_rq)
+ return;
+
+ for_each_sched_entity(se)
+ weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
- reweight_entity(cfs_rq, se, lw->weight);
- load->inv_weight = lw->inv_weight;
+ reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
}
static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
@@ -4331,7 +4349,6 @@ static long calc_group_shares(struct cfs
static void update_cfs_group(struct sched_entity *se)
{
struct cfs_rq *gcfs_rq = group_cfs_rq(se);
- long shares;
/*
* When a group becomes empty, preserve its weight. This matters for
@@ -4340,9 +4357,7 @@ static void update_cfs_group(struct sche
if (!gcfs_rq || !gcfs_rq->load.weight)
return;
- shares = calc_group_shares(gcfs_rq);
- if (unlikely(se->load.weight != shares))
- reweight_entity(cfs_rq_of(se), se, shares);
+ reweight_entity(cfs_rq_of(se), se, calc_group_shares(gcfs_rq));
}
#else /* !CONFIG_FAIR_GROUP_SCHED: */
@@ -4460,7 +4475,7 @@ static inline bool cfs_rq_is_decayed(str
* differential update where we store the last value we propagated. This in
* turn allows skipping updates if the differential is 'small'.
*
- * Updating tg's load_avg is necessary before update_cfs_share().
+ * Updating tg's load_avg is necessary before update_cfs_group().
*/
static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
{
@@ -4926,7 +4941,7 @@ static void migrate_se_pelt_lag(struct s
* The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
* avg. The immediate corollary is that all (fair) tasks must be attached.
*
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
*
* Return: true if the load decayed or we removed load.
*
@@ -5475,6 +5490,7 @@ static void
place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
{
u64 vslice, vruntime = avg_vruntime(cfs_rq);
+ unsigned int nr_queued = cfs_rq->h_nr_queued;
bool update_zero = false;
s64 lag = 0;
@@ -5482,6 +5498,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
se->slice = sysctl_sched_base_slice;
vslice = calc_delta_fair(se->slice, se);
+ if (flags & ENQUEUE_QUEUED)
+ nr_queued -= 1;
+
/*
* Due to how V is constructed as the weighted average of entities,
* adding tasks with positive lag, or removing tasks with negative lag
@@ -5490,7 +5509,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
*
* EEVDF: placement strategy #1 / #2
*/
- if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
+ if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
struct sched_entity *curr = cfs_rq->curr;
long load, weight;
@@ -5550,9 +5569,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
*/
load = cfs_rq->sum_weight;
if (curr && curr->on_rq)
- load += avg_vruntime_weight(cfs_rq, curr->load.weight);
+ load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
- weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
lag *= load + weight;
if (WARN_ON_ONCE(!load))
load = 1;
@@ -5611,22 +5630,8 @@ static void check_enqueue_throttle(struc
static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
static void
-requeue_delayed_entity(struct sched_entity *se);
-
-static void
enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
{
- bool curr = cfs_rq->curr == se;
-
- /*
- * If we're the current task, we must renormalise before calling
- * update_curr().
- */
- if (curr)
- place_entity(cfs_rq, se, flags);
-
- update_curr(cfs_rq);
-
/*
* When enqueuing a sched_entity, we must:
* - Update loads to have both entity and cfs_rq synced with now.
@@ -5645,13 +5650,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
*/
update_cfs_group(se);
- /*
- * XXX now that the entity has been re-weighted, and it's lag adjusted,
- * we can place the entity.
- */
- if (!curr)
- place_entity(cfs_rq, se, flags);
-
account_entity_enqueue(cfs_rq, se);
/* Entity has migrated, no longer consider this task hot */
@@ -5660,8 +5658,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
check_schedstat_required();
update_stats_enqueue_fair(cfs_rq, se, flags);
- if (!curr)
- __enqueue_entity(cfs_rq, se);
se->on_rq = 1;
if (cfs_rq->nr_queued == 1) {
@@ -5679,21 +5675,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
}
}
-static void __clear_buddies_next(struct sched_entity *se)
+static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- for_each_sched_entity(se) {
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
- if (cfs_rq->next != se)
- break;
-
- cfs_rq->next = NULL;
- }
+ if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
+ return;
+ if (se_is_idle(se))
+ return;
+ cfs_rq->next = se;
}
static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
if (cfs_rq->next == se)
- __clear_buddies_next(se);
+ cfs_rq->next = NULL;
}
static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
@@ -5704,7 +5698,7 @@ static void set_delayed(struct sched_ent
/*
* Delayed se of cfs_rq have no tasks queued on them.
- * Do not adjust h_nr_runnable since dequeue_entities()
+ * Do not adjust h_nr_runnable since __dequeue_task()
* will account it for blocked tasks.
*/
if (!entity_is_task(se))
@@ -5737,37 +5731,11 @@ static void clear_delayed(struct sched_e
}
}
-static bool
+static void
dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
{
- bool sleep = flags & DEQUEUE_SLEEP;
int action = UPDATE_TG;
- update_curr(cfs_rq);
- clear_buddies(cfs_rq, se);
-
- if (flags & DEQUEUE_DELAYED) {
- WARN_ON_ONCE(!se->sched_delayed);
- } else {
- bool delay = sleep;
- /*
- * DELAY_DEQUEUE relies on spurious wakeups, special task
- * states must not suffer spurious wakeups, excempt them.
- */
- if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
- delay = false;
-
- WARN_ON_ONCE(delay && se->sched_delayed);
-
- if (sched_feat(DELAY_DEQUEUE) && delay &&
- !entity_eligible(cfs_rq, se)) {
- update_load_avg(cfs_rq, se, 0);
- update_entity_lag(cfs_rq, se);
- set_delayed(se);
- return false;
- }
- }
-
if (entity_is_task(se) && task_on_rq_migrating(task_of(se)))
action |= DO_DETACH;
@@ -5785,14 +5753,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
update_stats_dequeue_fair(cfs_rq, se, flags);
- update_entity_lag(cfs_rq, se);
- if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
- se->deadline -= se->vruntime;
- se->rel_deadline = 1;
- }
-
- if (se != cfs_rq->curr)
- __dequeue_entity(cfs_rq, se);
se->on_rq = 0;
account_entity_dequeue(cfs_rq, se);
@@ -5801,9 +5761,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
update_cfs_group(se);
- if (flags & DEQUEUE_DELAYED)
- clear_delayed(se);
-
if (cfs_rq->nr_queued == 0) {
update_idle_cfs_rq_clock_pelt(cfs_rq);
#ifdef CONFIG_CFS_BANDWIDTH
@@ -5816,15 +5773,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
}
#endif
}
-
- return true;
}
static void
-set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
+set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- clear_buddies(cfs_rq, se);
-
/* 'current' is not kept within the tree. */
if (se->on_rq) {
/*
@@ -5833,16 +5786,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
* runqueue.
*/
update_stats_wait_end_fair(cfs_rq, se);
- __dequeue_entity(cfs_rq, se);
update_load_avg(cfs_rq, se, UPDATE_TG);
-
- if (first)
- set_protect_slice(cfs_rq, se);
}
update_stats_curr_start(cfs_rq, se);
- WARN_ON_ONCE(cfs_rq->curr);
- cfs_rq->curr = se;
+ WARN_ON_ONCE(cfs_rq->h_curr);
+ cfs_rq->h_curr = se;
/*
* Track our maximum slice length, if the CPU's load is at
@@ -5862,23 +5811,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
se->prev_sum_exec_runtime = se->sum_exec_runtime;
}
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
-/*
- * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
- */
static struct sched_entity *
-pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
+pick_next_entity(struct rq *rq, bool protect)
{
+ struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
se = pick_eevdf(cfs_rq, protect);
if (se->sched_delayed) {
- dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+ __dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
/*
* Must not reference @se again, see __block_task().
*/
@@ -5903,13 +5846,11 @@ static void put_prev_entity(struct cfs_r
if (prev->on_rq) {
update_stats_wait_start_fair(cfs_rq, prev);
- /* Put 'current' back into the tree. */
- __enqueue_entity(cfs_rq, prev);
/* in !on_rq case, update occurred at dequeue */
update_load_avg(cfs_rq, prev, 0);
}
- WARN_ON_ONCE(cfs_rq->curr != prev);
- cfs_rq->curr = NULL;
+ WARN_ON_ONCE(cfs_rq->h_curr != prev);
+ cfs_rq->h_curr = NULL;
}
static void
@@ -6062,7 +6003,7 @@ static void __account_cfs_rq_runtime(str
* if we're unable to extend our runtime we resched so that the active
* hierarchy can be throttled
*/
- if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
+ if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->h_curr))
resched_curr(rq_of(cfs_rq));
}
@@ -6420,7 +6361,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
assert_list_leaf_cfs_rq(rq);
/* Determine whether we need to wake up potentially idle CPU: */
- if (rq->curr == rq->idle && rq->cfs.nr_queued)
+ if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
resched_curr(rq);
}
@@ -6761,7 +6702,7 @@ static void check_enqueue_throttle(struc
return;
/* an active group must be handled by the update_curr()->put() path */
- if (!cfs_rq->runtime_enabled || cfs_rq->curr)
+ if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
return;
/* ensure the group is not already throttled */
@@ -7156,7 +7097,7 @@ static void hrtick_start_fair(struct rq
resched_curr(rq);
return;
}
- delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+ delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
/*
* Correct for instantaneous load of other classes.
@@ -7256,10 +7197,8 @@ static int choose_idle_cpu(int cpu, stru
}
static void
-requeue_delayed_entity(struct sched_entity *se)
+requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
/*
* se->sched_delayed should imply: se->on_rq == 1.
* Because a delayed entity is one that is still on
@@ -7269,19 +7208,58 @@ requeue_delayed_entity(struct sched_enti
WARN_ON_ONCE(!se->on_rq);
if (update_entity_lag(cfs_rq, se)) {
- cfs_rq->nr_queued--;
+ cfs_rq->h_nr_queued--;
if (se != cfs_rq->curr)
__dequeue_entity(cfs_rq, se);
place_entity(cfs_rq, se, 0);
if (se != cfs_rq->curr)
__enqueue_entity(cfs_rq, se);
- cfs_rq->nr_queued++;
+ cfs_rq->h_nr_queued++;
}
update_load_avg(cfs_rq, se, 0);
clear_delayed(se);
}
+static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
+{
+ unsigned long weight = NICE_0_LOAD;
+ int task_new = !(flags & ENQUEUE_WAKEUP);
+ struct sched_entity *se = &p->se;
+ int h_nr_idle = task_has_idle_policy(p);
+ int h_nr_runnable = 1;
+
+ if (task_new && se->sched_delayed)
+ h_nr_runnable = 0;
+
+ for_each_sched_entity(se) {
+ struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+ update_curr(cfs_rq);
+
+ if (!se->on_rq) {
+ enqueue_entity(cfs_rq, se, flags);
+ } else {
+ update_load_avg(cfs_rq, se, UPDATE_TG);
+ se_update_runnable(se);
+ update_cfs_group(se);
+ }
+
+ cfs_rq->h_nr_runnable += h_nr_runnable;
+ cfs_rq->h_nr_queued++;
+ cfs_rq->h_nr_idle += h_nr_idle;
+
+ if (cfs_rq_is_idle(cfs_rq))
+ h_nr_idle = 1;
+
+ weight = __calc_prop_weight(cfs_rq, se, weight);
+
+ flags = ENQUEUE_WAKEUP;
+ }
+
+ return weight;
+}
+
/*
* The enqueue_task method is called before nr_running is
* increased. Here we update the fair scheduling stats and
@@ -7290,13 +7268,12 @@ requeue_delayed_entity(struct sched_enti
static void
enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
{
- struct cfs_rq *cfs_rq;
- struct sched_entity *se = &p->se;
- int h_nr_idle = task_has_idle_policy(p);
- int h_nr_runnable = 1;
- int task_new = !(flags & ENQUEUE_WAKEUP);
int rq_h_nr_queued = rq->cfs.h_nr_queued;
- u64 slice = 0;
+ int task_new = !(flags & ENQUEUE_WAKEUP);
+ struct sched_entity *se = &p->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
+ unsigned long weight;
+ bool curr;
if (task_is_throttled(p) && enqueue_throttled_task(p))
return;
@@ -7308,10 +7285,10 @@ enqueue_task_fair(struct rq *rq, struct
* estimated utilization, before we update schedutil.
*/
if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
- util_est_enqueue(&rq->cfs, p);
+ util_est_enqueue(cfs_rq, p);
if (flags & ENQUEUE_DELAYED) {
- requeue_delayed_entity(se);
+ requeue_delayed_entity(cfs_rq, se);
return;
}
@@ -7323,57 +7300,22 @@ enqueue_task_fair(struct rq *rq, struct
if (p->in_iowait)
cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
- if (task_new && se->sched_delayed)
- h_nr_runnable = 0;
-
- for_each_sched_entity(se) {
- if (se->on_rq) {
- if (se->sched_delayed)
- requeue_delayed_entity(se);
- break;
- }
- cfs_rq = cfs_rq_of(se);
-
- /*
- * Basically set the slice of group entries to the min_slice of
- * their respective cfs_rq. This ensures the group can service
- * its entities in the desired time-frame.
- */
- if (slice) {
- se->slice = slice;
- se->custom_slice = 1;
- }
- enqueue_entity(cfs_rq, se, flags);
- slice = cfs_rq_min_slice(cfs_rq);
-
- cfs_rq->h_nr_runnable += h_nr_runnable;
- cfs_rq->h_nr_queued++;
- cfs_rq->h_nr_idle += h_nr_idle;
-
- if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = 1;
-
- flags = ENQUEUE_WAKEUP;
- }
-
- for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
-
- update_load_avg(cfs_rq, se, UPDATE_TG);
- se_update_runnable(se);
- update_cfs_group(se);
+ /*
+ * XXX comment on the curr thing
+ */
+ curr = (cfs_rq->curr == se);
+ if (curr)
+ place_entity(cfs_rq, se, flags);
- se->slice = slice;
- if (se != cfs_rq->curr)
- min_vruntime_cb_propagate(&se->run_node, NULL);
- slice = cfs_rq_min_slice(cfs_rq);
+ if (se->on_rq && se->sched_delayed)
+ requeue_delayed_entity(cfs_rq, se);
- cfs_rq->h_nr_runnable += h_nr_runnable;
- cfs_rq->h_nr_queued++;
- cfs_rq->h_nr_idle += h_nr_idle;
+ weight = enqueue_hierarchy(p, flags);
- if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = 1;
+ if (!curr) {
+ reweight_eevdf(cfs_rq, se, weight, false);
+ place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
+ __enqueue_entity(cfs_rq, se);
}
if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
@@ -7404,105 +7346,107 @@ enqueue_task_fair(struct rq *rq, struct
hrtick_update(rq);
}
-/*
- * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
- * failing half-way through and resume the dequeue later.
- *
- * Returns:
- * -1 - dequeue delayed
- * 0 - dequeue throttled
- * 1 - dequeue complete
- */
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
+static void dequeue_hierarchy(struct task_struct *p, int flags)
{
- bool was_sched_idle = sched_idle_rq(rq);
+ struct sched_entity *se = &p->se;
bool task_sleep = flags & DEQUEUE_SLEEP;
bool task_delayed = flags & DEQUEUE_DELAYED;
bool task_throttled = flags & DEQUEUE_THROTTLE;
- struct task_struct *p = NULL;
- int h_nr_idle = 0;
- int h_nr_queued = 0;
int h_nr_runnable = 0;
- struct cfs_rq *cfs_rq;
- u64 slice = 0;
+ int h_nr_idle = task_has_idle_policy(p);
+ bool dequeue = true;
- if (entity_is_task(se)) {
- p = task_of(se);
- h_nr_queued = 1;
- h_nr_idle = task_has_idle_policy(p);
- if (task_sleep || task_delayed || !se->sched_delayed)
- h_nr_runnable = 1;
- }
+ if (task_sleep || task_delayed || !se->sched_delayed)
+ h_nr_runnable = 1;
for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
+ struct cfs_rq *cfs_rq = cfs_rq_of(se);
- if (!dequeue_entity(cfs_rq, se, flags)) {
- if (p && &p->se == se)
- return -1;
+ update_curr(cfs_rq);
- slice = cfs_rq_min_slice(cfs_rq);
- break;
+ if (dequeue) {
+ dequeue_entity(cfs_rq, se, flags);
+ /* Don't dequeue parent if it has other entities besides us */
+ if (cfs_rq->load.weight)
+ dequeue = false;
+ } else {
+ update_load_avg(cfs_rq, se, UPDATE_TG);
+ se_update_runnable(se);
+ update_cfs_group(se);
}
cfs_rq->h_nr_runnable -= h_nr_runnable;
- cfs_rq->h_nr_queued -= h_nr_queued;
+ cfs_rq->h_nr_queued--;
cfs_rq->h_nr_idle -= h_nr_idle;
if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = h_nr_queued;
+ h_nr_idle = 1;
if (throttled_hierarchy(cfs_rq) && task_throttled)
record_throttle_clock(cfs_rq);
- /* Don't dequeue parent if it has other entities besides us */
- if (cfs_rq->load.weight) {
- slice = cfs_rq_min_slice(cfs_rq);
-
- /* Avoid re-evaluating load for this entity: */
- se = parent_entity(se);
- /*
- * Bias pick_next to pick a task from this cfs_rq, as
- * p is sleeping when it is within its sched_slice.
- */
- if (task_sleep && se)
- set_next_buddy(se);
- break;
- }
flags |= DEQUEUE_SLEEP;
flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
}
+}
- for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
+/*
+ * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
+ *
+ * Returns:
+ * true - dequeued
+ * false - delayed
+ */
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
+{
+ struct sched_entity *se = &p->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
+ bool was_sched_idle = sched_idle_rq(rq);
+ bool task_sleep = flags & DEQUEUE_SLEEP;
+ bool task_delayed = flags & DEQUEUE_DELAYED;
- update_load_avg(cfs_rq, se, UPDATE_TG);
- se_update_runnable(se);
- update_cfs_group(se);
+ clear_buddies(cfs_rq, se);
- se->slice = slice;
- if (se != cfs_rq->curr)
- min_vruntime_cb_propagate(&se->run_node, NULL);
- slice = cfs_rq_min_slice(cfs_rq);
+ if (flags & DEQUEUE_DELAYED) {
+ WARN_ON_ONCE(!se->sched_delayed);
+ } else {
+ bool delay = task_sleep;
+ /*
+ * DELAY_DEQUEUE relies on spurious wakeups, special task
+ * states must not suffer spurious wakeups, excempt them.
+ */
+ if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
+ delay = false;
- cfs_rq->h_nr_runnable -= h_nr_runnable;
- cfs_rq->h_nr_queued -= h_nr_queued;
- cfs_rq->h_nr_idle -= h_nr_idle;
+ WARN_ON_ONCE(delay && se->sched_delayed);
- if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = h_nr_queued;
+ if (sched_feat(DELAY_DEQUEUE) && delay &&
+ !entity_eligible(cfs_rq, se)) {
+ update_load_avg(cfs_rq_of(se), se, 0);
+ set_delayed(se);
+ return false;
+ }
+ }
- if (throttled_hierarchy(cfs_rq) && task_throttled)
- record_throttle_clock(cfs_rq);
+ dequeue_hierarchy(p, flags);
+
+ update_entity_lag(cfs_rq, se);
+ if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
+ se->deadline -= se->vruntime;
+ se->rel_deadline = 1;
}
+ if (se != cfs_rq->curr)
+ __dequeue_entity(cfs_rq, se);
- sub_nr_running(rq, h_nr_queued);
+ sub_nr_running(rq, 1);
/* balance early to pull high priority tasks */
if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
rq->next_balance = jiffies;
- if (p && task_delayed) {
+ if (task_delayed) {
+ clear_delayed(se);
+
WARN_ON_ONCE(!task_sleep);
WARN_ON_ONCE(p->on_rq != 1);
@@ -7514,7 +7458,7 @@ static int dequeue_entities(struct rq *r
__block_task(rq, p);
}
- return 1;
+ return true;
}
/*
@@ -7533,11 +7477,11 @@ static bool dequeue_task_fair(struct rq
util_est_dequeue(&rq->cfs, p);
util_est_update(&rq->cfs, p, flags & DEQUEUE_SLEEP);
- if (dequeue_entities(rq, &p->se, flags) < 0)
+ if (!__dequeue_task(rq, p, flags))
return false;
/*
- * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
+ * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
*/
return true;
}
@@ -9021,19 +8965,6 @@ static void migrate_task_rq_fair(struct
static void task_dead_fair(struct task_struct *p)
{
struct sched_entity *se = &p->se;
-
- if (se->sched_delayed) {
- struct rq_flags rf;
- struct rq *rq;
-
- rq = task_rq_lock(p, &rf);
- if (se->sched_delayed) {
- update_rq_clock(rq);
- dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
- }
- task_rq_unlock(rq, p, &rf);
- }
-
remove_entity_load_avg(se);
}
@@ -9067,21 +8998,10 @@ static void set_cpus_allowed_fair(struct
set_task_max_allowed_capacity(p);
}
-static void set_next_buddy(struct sched_entity *se)
-{
- for_each_sched_entity(se) {
- if (WARN_ON_ONCE(!se->on_rq))
- return;
- if (se_is_idle(se))
- return;
- cfs_rq_of(se)->next = se;
- }
-}
-
enum preempt_wakeup_action {
PREEMPT_WAKEUP_NONE, /* No preemption. */
PREEMPT_WAKEUP_SHORT, /* Ignore slice protection. */
- PREEMPT_WAKEUP_PICK, /* Let __pick_eevdf() decide. */
+ PREEMPT_WAKEUP_PICK, /* Let pick_eevdf() decide. */
PREEMPT_WAKEUP_RESCHED, /* Force reschedule. */
};
@@ -9098,7 +9018,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
if (cfs_rq->next && entity_before(cfs_rq->next, pse))
return false;
- set_next_buddy(pse);
+ set_next_buddy(cfs_rq, pse);
return true;
}
@@ -9188,7 +9108,6 @@ static void wakeup_preempt_fair(struct r
if (!sched_feat(WAKEUP_PREEMPTION))
return;
- find_matching_se(&se, &pse);
WARN_ON_ONCE(!pse);
cse_is_idle = se_is_idle(se);
@@ -9216,8 +9135,7 @@ static void wakeup_preempt_fair(struct r
if (unlikely(!normal_policy(p->policy)))
return;
- cfs_rq = cfs_rq_of(se);
- update_curr(cfs_rq);
+ update_curr_fair(rq);
/*
* If @p has a shorter slice than current and @p is eligible, override
* current's slice protection in order to allow preemption.
@@ -9261,18 +9179,15 @@ static void wakeup_preempt_fair(struct r
}
pick:
- nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
- /* If @p has become the most eligible task, force preemption */
- if (nse == pse)
- goto preempt;
-
- /*
- * Because p is enqueued, nse being null can only mean that we
- * dequeued a delayed task. If there are still entities queued in
- * cfs, check if the next one will be p.
- */
- if (!nse && cfs_rq->nr_queued)
- goto pick;
+ if (cfs_rq->h_nr_queued) {
+ nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
+ if (unlikely(!nse))
+ goto pick;
+
+ /* If @p has become the most eligible task, force preemption */
+ if (nse == pse)
+ goto preempt;
+ }
if (sched_feat(RUN_TO_PARITY))
update_protect_slice(cfs_rq, se);
@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
__must_hold(__rq_lockp(rq))
{
+ struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
- struct cfs_rq *cfs_rq;
struct task_struct *p;
- bool throttled;
int new_tasks;
again:
- cfs_rq = &rq->cfs;
- if (!cfs_rq->nr_queued)
+ if (!cfs_rq->h_nr_queued)
goto idle;
- throttled = false;
-
- do {
- /* Might not have done put_prev_entity() */
- if (cfs_rq->curr && cfs_rq->curr->on_rq)
- update_curr(cfs_rq);
-
- throttled |= check_cfs_rq_runtime(cfs_rq);
+ /* Might not have done put_prev_entity() */
+ if (cfs_rq->curr && cfs_rq->curr->on_rq)
+ update_curr(cfs_rq);
- se = pick_next_entity(rq, cfs_rq, true);
- if (!se)
- goto again;
- cfs_rq = group_cfs_rq(se);
- } while (cfs_rq);
+ se = pick_next_entity(rq, true);
+ if (!se)
+ goto again;
p = task_of(se);
- if (unlikely(throttled))
+ if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
task_throttle_setup_work(p);
return p;
@@ -9353,7 +9259,7 @@ void fair_server_init(struct rq *rq)
static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
{
struct sched_entity *se = &prev->se;
- struct cfs_rq *cfs_rq;
+ struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *nse = NULL;
#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9363,7 +9269,7 @@ static void put_prev_task_fair(struct rq
while (se) {
cfs_rq = cfs_rq_of(se);
- if (!nse || cfs_rq->curr)
+ if (!nse || cfs_rq->h_curr)
put_prev_entity(cfs_rq, se);
#ifdef CONFIG_FAIR_GROUP_SCHED
if (nse) {
@@ -9382,6 +9288,14 @@ static void put_prev_task_fair(struct rq
#endif
se = parent_entity(se);
}
+
+ /* Put 'current' back into the tree. */
+ cfs_rq = &rq->cfs;
+ se = &prev->se;
+ WARN_ON_ONCE(cfs_rq->curr != se);
+ cfs_rq->curr = NULL;
+ if (se->on_rq)
+ __enqueue_entity(cfs_rq, se);
}
/*
@@ -9390,8 +9304,8 @@ static void put_prev_task_fair(struct rq
static void yield_task_fair(struct rq *rq)
{
struct task_struct *curr = rq->donor;
- struct cfs_rq *cfs_rq = task_cfs_rq(curr);
struct sched_entity *se = &curr->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
/*
* Are we the only task in the tree?
@@ -9432,11 +9346,11 @@ static bool yield_to_task_fair(struct rq
struct sched_entity *se = &p->se;
/* !se->on_rq also covers throttled task */
- if (!se->on_rq)
+ if (!se->on_rq || se->sched_delayed)
return false;
/* Tell the scheduler that we'd really like se to run next. */
- set_next_buddy(se);
+ set_next_buddy(&task_rq(p)->cfs, se);
yield_task_fair(rq);
@@ -9762,15 +9676,10 @@ static inline long migrate_degrades_loca
*/
static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
{
- struct cfs_rq *dst_cfs_rq;
+ struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#ifdef CONFIG_FAIR_GROUP_SCHED
- dst_cfs_rq = task_group(p)->cfs_rq[dest_cpu];
-#else
- dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#endif
- if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
- !entity_eligible(task_cfs_rq(p), &p->se))
+ if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
+ !entity_eligible(&task_rq(p)->cfs, &p->se))
return 1;
return 0;
@@ -10240,7 +10149,7 @@ static void update_cfs_rq_h_load(struct
while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
load = cfs_rq->h_load;
load = div64_ul(load * se->avg.load_avg,
- cfs_rq_load_avg(cfs_rq) + 1);
+ cfs_rq_load_avg(cfs_rq) + 1);
cfs_rq = group_cfs_rq(se);
cfs_rq->h_load = load;
cfs_rq->last_h_load_update = now;
@@ -13459,7 +13368,7 @@ static inline void task_tick_core(struct
* MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
* if we need to give up the CPU.
*/
- if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
+ if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
__entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
resched_curr(rq);
}
@@ -13668,30 +13577,8 @@ bool cfs_prio_less(const struct task_str
WARN_ON_ONCE(task_rq(b)->core != rq->core);
-#ifdef CONFIG_FAIR_GROUP_SCHED
- /*
- * Find an se in the hierarchy for tasks a and b, such that the se's
- * are immediate siblings.
- */
- while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
- int sea_depth = sea->depth;
- int seb_depth = seb->depth;
-
- if (sea_depth >= seb_depth)
- sea = parent_entity(sea);
- if (sea_depth <= seb_depth)
- seb = parent_entity(seb);
- }
-
- se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
- se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
-
- cfs_rqa = sea->cfs_rq;
- cfs_rqb = seb->cfs_rq;
-#else /* !CONFIG_FAIR_GROUP_SCHED: */
cfs_rqa = &task_rq(a)->cfs;
cfs_rqb = &task_rq(b)->cfs;
-#endif /* !CONFIG_FAIR_GROUP_SCHED */
/*
* Find delta after normalizing se's vruntime with its cfs_rq's
@@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct
*/
static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
{
- struct cfs_rq *cfs_rq;
struct sched_entity *se = &curr->se;
+ unsigned long weight = NICE_0_LOAD;
+ struct cfs_rq *cfs_rq;
for_each_sched_entity(se) {
cfs_rq = cfs_rq_of(se);
entity_tick(cfs_rq, se, queued);
+
+ weight = __calc_prop_weight(cfs_rq, se, weight);
}
+ se = &curr->se;
+ reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+
if (queued)
return;
@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, struct
if (p->prio == oldprio)
return;
- if (rq->cfs.nr_queued == 1)
+ if (rq->cfs.h_nr_queued == 1)
return;
/*
@@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq *
}
}
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_entity *se = &p->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
+ unsigned long weight = NICE_0_LOAD;
+ bool on_rq = se->on_rq;
+
+ clear_buddies(cfs_rq, se);
+
+ if (on_rq)
+ __dequeue_entity(cfs_rq, se);
for_each_sched_entity(se) {
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
+ cfs_rq = cfs_rq_of(se);
- if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
- first && cfs_rq->curr)
- break;
+ if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
+ !first || !cfs_rq->h_curr)
+ set_next_entity(cfs_rq, se);
- set_next_entity(cfs_rq, se, first);
/* ensure bandwidth has been allocated on our new cfs_rq */
account_cfs_rq_runtime(cfs_rq, 0);
+
+ if (on_rq)
+ weight = __calc_prop_weight(cfs_rq, se, weight);
}
se = &p->se;
+ cfs_rq->curr = se;
+
+ if (on_rq) {
+ reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+ if (first)
+ set_protect_slice(cfs_rq, se);
+ }
if (task_on_rq_queued(p)) {
/*
@@ -14054,17 +13958,8 @@ void unregister_fair_sched_group(struct
struct sched_entity *se = tg->se[cpu];
struct rq *rq = cpu_rq(cpu);
- if (se) {
- if (se->sched_delayed) {
- guard(rq_lock_irqsave)(rq);
- if (se->sched_delayed) {
- update_rq_clock(rq);
- dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
- }
- list_del_leaf_cfs_rq(cfs_rq);
- }
+ if (se)
remove_entity_load_avg(se);
- }
/*
* Only empty task groups can be destroyed; so we can speculatively
--- a/kernel/sched/pelt.c
+++ b/kernel/sched/pelt.c
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
/*
* running is a subset of runnable (weight) so running can't be set if
* runnable is clear. But there are some corner cases where the current
- * se has been already dequeued but cfs_rq->curr still points to it.
+ * se has been already dequeued but cfs_rq->h_curr still points to it.
* This means that weight will be 0 but not running for a sched_entity
* but also for a cfs_rq if the latter becomes idle. As an example,
* this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
{
if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
- cfs_rq->curr == se)) {
+ cfs_rq->h_curr == se)) {
___update_load_avg(&se->avg, se_weight(se));
cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
if (___update_load_sum(now, &cfs_rq->avg,
scale_load_down(cfs_rq->load.weight),
cfs_rq->h_nr_runnable,
- cfs_rq->curr != NULL)) {
+ cfs_rq->h_curr != NULL)) {
___update_load_avg(&cfs_rq->avg, 1);
trace_pelt_cfs_tp(cfs_rq);
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -528,21 +528,8 @@ struct task_group {
};
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
#define ROOT_TASK_GROUP_LOAD NICE_0_LOAD
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- * limitation from this.)
- */
-#define MIN_SHARES (1UL << 1)
-#define MAX_SHARES (1UL << 18)
-#endif
-
typedef int (*tg_visitor)(struct task_group *, void *);
extern int walk_tg_tree_from(struct task_group *from,
@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain
#endif /* !CONFIG_CGROUP_SCHED */
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ * limitation from this.)
+ */
+#define MIN_SHARES (1UL << 1)
+#define MAX_SHARES (1UL << 18)
+
extern void unregister_rt_sched_group(struct task_group *tg);
extern void free_rt_sched_group(struct task_group *tg);
extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -707,6 +705,7 @@ struct cfs_rq {
/*
* CFS load tracking
*/
+ struct sched_entity *h_curr;
struct sched_avg avg;
#ifndef CONFIG_64BIT
u64 last_update_time_copy;
@@ -2509,6 +2508,7 @@ extern const u32 sched_prio_to_wmult[40
#define ENQUEUE_MIGRATED 0x00040000
#define ENQUEUE_INITIAL 0x00080000
#define ENQUEUE_RQ_SELECTED 0x00100000
+#define ENQUEUE_QUEUED 0x00200000
#define RETRY_TASK ((void *)-1UL)
^ permalink raw reply
* Re: [PATCH] cgroup: Keep favordynmods enabled once per-threadgroup rwsem is active
From: Chen Ridong @ 2026-05-11 12:08 UTC (permalink / raw)
To: Guopeng Zhang, Tejun Heo, Johannes Weiner, Michal Koutný
Cc: Yi Tao, cgroups, linux-kernel
In-Reply-To: <85eaa9ae-1558-41a8-bf12-999a9b44bfa9@kylinos.cn>
On 2026/5/11 17:53, Guopeng Zhang wrote:
>
>
> 在 2026/5/11 17:05, Chen Ridong 写道:
>>
>>
>> On 2026/5/11 16:16, Guopeng Zhang wrote:
>>> cgroup_enable_per_threadgroup_rwsem is a one-way switch. Once it is
>>> enabled, cgroup.procs writes use the per-threadgroup rwsem and
>>> cgroup_threadgroup_change_begin()/end() use the same global state to
>>> decide whether to take and release the per-threadgroup rwsem.
>>>
>>> The disable path warned that the per-threadgroup rwsem mechanism could not
>>> be disabled but still called rcu_sync_exit() and cleared
>>> CGRP_ROOT_FAVOR_DYNMODS. That partially disabled favordynmods while the
>>> global per-threadgroup rwsem mode remained enabled: cgroup.procs writes
>>> would continue to use the per-threadgroup rwsem, while
>>> cgroup_threadgroup_change_begin()/end() could observe the exited rcu_sync
>>> state. The root would also no longer report favordynmods.
>>>
>>> Make the transition match the documented one-way semantics. Call
>>> rcu_sync_enter() only for the first favordynmods enable, and make later
>>> disable attempts a no-op after warning once the per-threadgroup rwsem mode
>>> has been enabled.
>>>
>>> Fixes: 0568f89d4fb8 ("cgroup: replace global percpu_rwsem with per threadgroup resem when writing to cgroup.procs")
>>> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
>>> ---
>>> Manual AB test:
>>>
>>> Before this patch:
>>> enable favordynmods:
>>> cgroup2 opts: rw,relatime,favordynmods
>>> disable attempt:
>>> cgroup2 opts: rw,relatime
>>> dmesg:
>>> cgroup: cgroup favordynmods: per threadgroup rwsem mechanism can't be disabled
>>>
>>> After this patch:
>>> enable favordynmods:
>>> cgroup2 opts: rw,relatime,favordynmods
>>> disable attempt:
>>> cgroup2 opts: rw,relatime,favordynmods
>>> dmesg:
>>> cgroup: cgroup favordynmods: per threadgroup rwsem mechanism can't be disabled
>>>
>>> kernel/cgroup/cgroup.c | 11 +++++------
>>> 1 file changed, 5 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
>>> index 6152add0c5eb..fd10fb5b3598 100644
>>> --- a/kernel/cgroup/cgroup.c
>>> +++ b/kernel/cgroup/cgroup.c
>>> @@ -1297,14 +1297,13 @@ void cgroup_favor_dynmods(struct cgroup_root *root, bool favor)
>>> */
>>> percpu_down_write(&cgroup_threadgroup_rwsem);
>>> if (favor && !favoring) {
>>> - cgroup_enable_per_threadgroup_rwsem = true;
>>> - rcu_sync_enter(&cgroup_threadgroup_rwsem.rss);
>>> + if (!cgroup_enable_per_threadgroup_rwsem) {
>>
>> Is this branch redundant? I think if (favor && !favoring) alone should suffice —
>> or can the outer condition be true twice (i.e., can this block be entered
>> multiple times)?
>> Hi Ridong,
>
> Thanks for taking a look.
>
> I don't think the inner check is redundant. `favoring` is per-root, while
> `cgroup_enable_per_threadgroup_rwsem` is global.
>
> For example, root A may have already enabled favordynmods:
>
This functionality is only available for cgroup v2, right?
Never mind — I see Tj has already replied.
--
Best regards,
Ridong
^ permalink raw reply
* Re: [PATCH v3 05/12] mm, swap: unify large folio allocation
From: David Hildenbrand (Arm) @ 2026-05-11 12:57 UTC (permalink / raw)
To: kasong, linux-mm
Cc: Andrew Morton, Zi Yan, Baolin Wang, Barry Song, Hugh Dickins,
Chris Li, Kemeng Shi, Nhat Pham, Baoquan He, Johannes Weiner,
Youngjun Park, Chengming Zhou, Roman Gushchin, Shakeel Butt,
Muchun Song, Qi Zheng, linux-kernel, cgroups, Yosry Ahmed,
Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko, Michal Hocko,
Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <20260421-swap-table-p4-v3-5-2f23759a76bc@tencent.com>
On 4/21/26 08:16, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> Now that direct large order allocation is supported in the swap cache,
> both anon and shmem can use it instead of implementing their own methods.
> This unifies the fallback and swap cache check, which also reduces the
> TOCTOU race window of swap cache state: previously, high order swapin
> required checking swap cache states first, then allocating and falling
> back separately. Now all these steps happen in the same compact loop.
>
> Order fallback and statistics are also unified, callers just need to
> check and pass the acceptable order bitmask.
>
> There is basically no behavior change. This only makes things more
> unified and prepares for later commits. Cgroup and zero map checks can
> also be moved into the compact loop, further reducing race windows and
> redundancy
>
You should spell out the rename from swapin_folio() to swapin_entry() [and why
it is done].
swapin_readahead() vs. swapin_entry() looks a bit odd, fiven that both consume
an entry.
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/memory.c | 77 ++++++------------------------
> mm/shmem.c | 94 +++++++++---------------------------
> mm/swap.h | 30 ++----------
> mm/swap_state.c | 145 ++++++++++----------------------------------------------
> mm/swapfile.c | 3 +-
> 5 files changed, 67 insertions(+), 282 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index ea6568571131..404734a5bcff 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -4593,26 +4593,6 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
> return VM_FAULT_SIGBUS;
> }
>
> -static struct folio *__alloc_swap_folio(struct vm_fault *vmf)
> -{
> - struct vm_area_struct *vma = vmf->vma;
> - struct folio *folio;
> - softleaf_t entry;
> -
> - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address);
> - if (!folio)
> - return NULL;
> -
> - entry = softleaf_from_pte(vmf->orig_pte);
> - if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
> - GFP_KERNEL, entry)) {
> - folio_put(folio);
> - return NULL;
> - }
> -
> - return folio;
> -}
> -
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> /*
> * Check if the PTEs within a range are contiguous swap entries
> @@ -4642,8 +4622,6 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
> */
> if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))
> return false;
> - if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages))
> - return false;
>
This should also be pointed out in the patch description. (and why it is ok)
> return true;
> }
> @@ -4671,16 +4649,14 @@ static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
> return orders;
> }
>
> -static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> +static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> unsigned long orders;
> - struct folio *folio;
> unsigned long addr;
> softleaf_t entry;
> spinlock_t *ptl;
> pte_t *pte;
> - gfp_t gfp;
> int order;
>
> /*
> @@ -4688,7 +4664,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> * maintain the uffd semantics.
> */
> if (unlikely(userfaultfd_armed(vma)))
> - goto fallback;
> + return 0;
>
> /*
> * A large swapped out folio could be partially or fully in zswap. We
> @@ -4696,7 +4672,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> * folio.
> */
> if (!zswap_never_enabled())
> - goto fallback;
> + return 0;
>
> entry = softleaf_from_pte(vmf->orig_pte);
> /*
> @@ -4710,12 +4686,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> vmf->address, orders);
>
> if (!orders)
> - goto fallback;
> + return 0;
>
> pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
> vmf->address & PMD_MASK, &ptl);
> if (unlikely(!pte))
> - goto fallback;
> + return 0;
>
> /*
> * For do_swap_page, find the highest order where the aligned range is
> @@ -4731,29 +4707,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
>
> pte_unmap_unlock(pte, ptl);
>
> - /* Try allocating the highest of the remaining orders. */
> - gfp = vma_thp_gfp_mask(vma);
> - while (orders) {
> - addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
> - folio = vma_alloc_folio(gfp, order, vma, addr);
> - if (folio) {
> - if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
> - gfp, entry))
> - return folio;
> - count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
> - folio_put(folio);
> - }
> - count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
> - order = next_order(&orders, order);
> - }
> -
> -fallback:
> - return __alloc_swap_folio(vmf);
> + return orders;
> }
> #else /* !CONFIG_TRANSPARENT_HUGEPAGE */
> -static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> +static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
> {
> - return __alloc_swap_folio(vmf);
> + return 0;
> }
> #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
>
> @@ -4859,21 +4818,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
> if (folio)
> swap_update_readahead(folio, vma, vmf->address);
> if (!folio) {
> - if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
> - folio = alloc_swap_folio(vmf);
> - if (folio) {
> - /*
> - * folio is charged, so swapin can only fail due
> - * to raced swapin and return NULL.
> - */
> - swapcache = swapin_folio(entry, folio);
> - if (swapcache != folio)
> - folio_put(folio);
> - folio = swapcache;
> - }
> - } else {
> + /* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
> + if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
> + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE,
> + thp_swapin_suitable_orders(vmf),
> + vmf, NULL, 0);
> + else
> folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
> - }
>
> if (!folio) {
> /*
Nothing else jumped at me in memory.c
--
Cheers,
David
^ permalink raw reply
* Re: [PATCH] cgroup/dmem: return -ENOMEM on failed pool preallocation
From: Michal Koutný @ 2026-05-11 13:03 UTC (permalink / raw)
To: Guopeng Zhang
Cc: Maarten Lankhorst, Maxime Ripard, Natalie Vock, Tejun Heo,
Johannes Weiner, cgroups, dri-devel, linux-kernel
In-Reply-To: <20260511013150.7235-1-zhangguopeng@kylinos.cn>
[-- Attachment #1: Type: text/plain, Size: 1596 bytes --]
On Mon, May 11, 2026 at 09:31:50AM +0800, Guopeng Zhang <zhangguopeng@kylinos.cn> wrote:
> get_cg_pool_unlocked() handles allocation failures under dmemcg_lock by
> dropping the lock, preallocating a pool with GFP_KERNEL, and retrying the
> locked lookup and creation path.
>
> If the fallback allocation fails too, pool remains NULL. Since the loop
> condition is while (!pool), the function can keep retrying instead of
> propagating the allocation failure to the caller.
This implies that it's OK when the function keeps retrying with
allocpool != NULL (and repeated WARN_ON()s)?
> Set pool to ERR_PTR(-ENOMEM) when the fallback allocation fails so the
> loop exits through the existing common return path. The callers already
> handle ERR_PTR() from get_cg_pool_unlocked(), so this restores the
> expected error path.
If the callers can handle it, maybe there's no need to retry at all.
Perhpas dmem fellows can step in here.
>
> Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup")
> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
> ---
> kernel/cgroup/dmem.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
> index 1ab1fb47f271..4753a67d0f0f 100644
> --- a/kernel/cgroup/dmem.c
> +++ b/kernel/cgroup/dmem.c
> @@ -602,6 +602,7 @@ get_cg_pool_unlocked(struct dmemcg_state *cg, struct dmem_cgroup_region *region)
> pool = NULL;
This 2nd pool zeroing seems pointless.
> continue;
> }
> + pool = ERR_PTR(-ENOMEM);
> }
> }
HTH,
Michal
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply
* Re: [PATCH] cgroup/cpuset: skip hardwall ancestor scan in v2 mode in cpuset_current_node_allowed()
From: Michal Koutný @ 2026-05-11 13:09 UTC (permalink / raw)
To: Wandun; +Cc: Tejun Heo, longman, chenridong, hannes, cgroups, linux-kernel
In-Reply-To: <92eebc24-9612-4ea3-9bd3-da4d437b4d81@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 530 bytes --]
On Sat, May 09, 2026 at 05:36:39PM +0800, Wandun <chenwandun1@gmail.com> wrote:
> > is_in_v2_mode() is also true for v1 mounted with cpuset_v2_mode, where
> > cpuset.mem_exclusive / cpuset.mem_hardwall are still settable. Would
> > that be a problem here? cpuset_v2() looks like a tighter fit.
> You're right, it is a problem.
>
> Under v1 + cpuset_v2_mode, CS_MEM_HARDWALL/CS_MEM_EXCLUSIVE can be set
> on non-root cpuset cgroup, so can't directly return true;
Ah, sorry missd that.
> I will fix it in v2.
Thx.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply
* Re: [PATCH v3 05/12] mm, swap: unify large folio allocation
From: Kairui Song @ 2026-05-11 14:37 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: linux-mm, Andrew Morton, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <675e9027-9fb5-47b5-9a2d-c9a416a27d0d@kernel.org>
On Mon, May 11, 2026 at 8:58 PM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 4/21/26 08:16, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> >
> > Now that direct large order allocation is supported in the swap cache,
> > both anon and shmem can use it instead of implementing their own methods.
> > This unifies the fallback and swap cache check, which also reduces the
> > TOCTOU race window of swap cache state: previously, high order swapin
> > required checking swap cache states first, then allocating and falling
> > back separately. Now all these steps happen in the same compact loop.
> >
> > Order fallback and statistics are also unified, callers just need to
> > check and pass the acceptable order bitmask.
> >
> > There is basically no behavior change. This only makes things more
> > unified and prepares for later commits. Cgroup and zero map checks can
> > also be moved into the compact loop, further reducing race windows and
> > redundancy
> >
>
> You should spell out the rename from swapin_folio() to swapin_entry() [and why
> it is done].
>
> swapin_readahead() vs. swapin_entry() looks a bit odd, fiven that both consume
> an entry.
Yes, the current status is a bit odd, about two years ago I also
wanted to name it `swapin_direct()`.
https://lore.kernel.org/linux-mm/20240326185032.72159-3-ryncsn@gmail.com/
But actually ZRAM or shmem would also benefit from supporting unified
readahead like this:
https://lore.kernel.org/linux-mm/20240102175338.62012-6-ryncsn@gmail.com/
So calling it `swapin_entry` seems more future-proof. At some point in
the future we might remove `swapin_readahead`. All swapin operations
could have a unified or at least a per-device readahead policy like
the one in the link above, instead of the current policy where the
caller must decide whether to perform readahead.
But any suggestion on naming is welcome :)
> > #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> > /*
> > * Check if the PTEs within a range are contiguous swap entries
> > @@ -4642,8 +4622,6 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
> > */
> > if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))
> > return false;
> > - if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages))
> > - return false;
> >
>
> This should also be pointed out in the patch description. (and why it is ok)
Right, the check is now resolved by the swap cache layer, so the
caller no longer needs to check it. I'll describe that in the commit
message.
Thanks!
^ permalink raw reply
* Re: [PATCH v3 05/12] mm, swap: unify large folio allocation
From: David Hildenbrand (Arm) @ 2026-05-11 15:15 UTC (permalink / raw)
To: Kairui Song
Cc: linux-mm, Andrew Morton, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <CAMgjq7DegMz2ZEHOhHkAqDEWDuCSZ7Ktsxw1ibDY8axFzRRGnQ@mail.gmail.com>
On 5/11/26 16:37, Kairui Song wrote:
> On Mon, May 11, 2026 at 8:58 PM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
>>
>> On 4/21/26 08:16, Kairui Song via B4 Relay wrote:
>>> From: Kairui Song <kasong@tencent.com>
>>>
>>> Now that direct large order allocation is supported in the swap cache,
>>> both anon and shmem can use it instead of implementing their own methods.
>>> This unifies the fallback and swap cache check, which also reduces the
>>> TOCTOU race window of swap cache state: previously, high order swapin
>>> required checking swap cache states first, then allocating and falling
>>> back separately. Now all these steps happen in the same compact loop.
>>>
>>> Order fallback and statistics are also unified, callers just need to
>>> check and pass the acceptable order bitmask.
>>>
>>> There is basically no behavior change. This only makes things more
>>> unified and prepares for later commits. Cgroup and zero map checks can
>>> also be moved into the compact loop, further reducing race windows and
>>> redundancy
>>>
>>
>> You should spell out the rename from swapin_folio() to swapin_entry() [and why
>> it is done].
>>
>> swapin_readahead() vs. swapin_entry() looks a bit odd, fiven that both consume
>> an entry.
>
> Yes, the current status is a bit odd, about two years ago I also
> wanted to name it `swapin_direct()`.
> https://lore.kernel.org/linux-mm/20240326185032.72159-3-ryncsn@gmail.com/
>
> But actually ZRAM or shmem would also benefit from supporting unified
> readahead like this:
> https://lore.kernel.org/linux-mm/20240102175338.62012-6-ryncsn@gmail.com/
>
> So calling it `swapin_entry` seems more future-proof. At some point in
> the future we might remove `swapin_readahead`. All swapin operations
> could have a unified or at least a per-device readahead policy like
> the one in the link above, instead of the current policy where the
> caller must decide whether to perform readahead.
>
> But any suggestion on naming is welcome :)
The other proposal
https://lore.kernel.org/all/tencent_CD11FE9B4A0B362E95E776C5F679598FAA07@qq.com/
calls it
swapin_synchronous_folio
Maybe just swapin_sync_io()/swapin_sync() or sth like that?
--
Cheers,
David
^ permalink raw reply
* Re: [RFC PATCH 0/9 v2] mm/memcontrol: Make memory cgroup limits tier-aware
From: David Hildenbrand (Arm) @ 2026-05-11 15:56 UTC (permalink / raw)
To: Joshua Hahn, linux-mm
Cc: Tejun Heo, Johannes Weiner, Michal Koutny, Michal Hocko,
Roman Gushchin, Shakeel Butt, Andrew Morton, Chris Li,
Kairui Song, Muchun Song, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Kemeng Shi,
Nhat Pham, Baoquan He, Barry Song, Youngjun Park, Qi Zheng,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Kaiyang Zhao, David Rientjes,
Yiannis Nikolakopoulos, Rao, Bharata Bhasker, cgroups,
linux-kernel, kernel-team
In-Reply-To: <20260423203445.2914963-1-joshua.hahnjy@gmail.com>
On 4/23/26 22:34, Joshua Hahn wrote:
> INTRODUCTION
> ============
> Memory cgroups provide an interface that allow multiple works on a host to
> co-exist via weak and strong memory isolation guarantees. This works, because
> for the most part, all memory has equal utility. Isolating a cgroup’s memory
> footprint restricts how much it can hurt other workloads competing for memory,
> or protects it from other cgroups looking for more memory.
>
> However, on systems with tiered memory (e.g. CXL), memory utility is no longer
> homogeneous; toptier and lowtier memory provide different performance
> characteristics and have different scarcity, meaning memory footprint no longer
> serves as an accurate representation of a cgroup’s consumption of the system’s
> limited resources. As an extreme example, a cgroup with 10G of toptier
> (e.g. DRAM) memory and a cgroup with 10G of lowtier (e.g. CXL) memory both
> appear to be consuming the same amount of system resources from memcg’s
> perspective, despite the performance asymmetry between the two workloads.
>
> Therefore on tiered systems, memory isolation cannot currently happen, as
> workloads that are well-behaved within their memcg limits may still hurt the
> performance of other well-behaving workloads by hogging more than its
> “fair share” of toptier memory.
>
> Introduce tier-aware memcg limits, which establish independent toptier limits
> that scale with the memory limits and the ratio of toptier:total memory
> available on the system.
>
> INTERFACE
> =========
> This series introduces only one adjustable knob to userspace; a new cgroup mount
> option “memory_tiered_limits” which toggles whether the cgroup mount will scale
> toptier limits. It also introduces 4 new read-only sysfs entries per-cgroup:
> memory.toptier_{min, low, high, max}.
>
> The new toptier memory limits are scaled according to the amount of toptier
> memory and total memory available on the system as such:
>
> memory.toptier_high = (toptier_mem / total_mem) * memory.high
>
> For instance, on a host with 100GB memory, with 75G toptier and 25G CXL, the
> “toptier ratio” would be 75 / 100 = 0.75. A cgroup with the following memcg
> limits {min: 8G, low: 12G, high: 20G, max: 24G} might see toptier limits scaled
> at {min: 6G, low: 9G, high: 15G, max: 18G}.
Assume you have a bigger hierarchy (HBP, DRAM, CXL), or assume you have multiple
NUMA nodes with a hierarchy each.
Your proposal doesn't really seem to be very versatile, or am I wrong?
--
Cheers,
David
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: K Prateek Nayak @ 2026-05-11 16:21 UTC (permalink / raw)
To: Peter Zijlstra, mingo
Cc: longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, qyousef
In-Reply-To: <20260511120628.206700041@infradead.org>
Hello Peter,
On 5/11/2026 5:01 PM, Peter Zijlstra wrote:
> @@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
> + se = pick_next_entity(rq, true);
> + if (!se)
> + goto again;
>
> p = task_of(se);
> - if (unlikely(throttled))
> + if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
> task_throttle_setup_work(p);
I think this bit should also be replicated in set_next_task() after
account_cfs_rq_runtime() since any part of the hierarchy may get
throttled as a result of failing to grab runtime.
Also check_cfs_rq_runtime() only sees if the cfs_rq is throttled
but the task can fail to run if it is on a throttled_hierarchy() too
so that should be the correct check here.
Something like below (only build tested on queue/sched/flat):
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e54da4c6c945..950c072244b2 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9224,7 +9224,19 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
goto again;
p = task_of(se);
- if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
+ /*
+ * For cases where prev is picked again after
+ * being throttled, entity_tick() would have
+ * already marked its hierarchy as throttled.
+ *
+ * Add throttle work here since
+ * put_prev_set_next_task() is skipped on
+ * same task's selection.
+ *
+ * For other case, set_next_task_fair() will
+ * handle adding the throttle work.
+ */
+ if (throttled_hierarchy(cfs_rq_of(se)))
task_throttle_setup_work(p);
return p;
@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
if (on_rq)
weight = __calc_prop_weight(cfs_rq, se, weight);
}
+ /*
+ * Add throttle work if the bandwidth allocation above failed
+ * to grab any runtime and throttled the task's hierarchy.
+ */
+ if (throttled_hierarchy(task_cfs_rq(p)))
+ task_throttle_setup_work(p);
se = &p->se;
cfs_rq->curr = se;
---
> return p;
>
--
Thanks and Regards,
Prateek
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox