* [PATCH v4 5/8] mm: list_lru: introduce caller locking for additions and deletions
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
Locking is currently internal to the list_lru API. However, a caller
might want to keep auxiliary state synchronized with the LRU state.
For example, the THP shrinker uses the lock of its custom LRU to keep
PG_partially_mapped and vmstats consistent.
To allow the THP shrinker to switch to list_lru, provide normal and
irqsafe locking primitives as well as caller-locked variants of the
addition and deletion functions.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
include/linux/list_lru.h | 41 ++++++++++++
mm/list_lru.c | 131 ++++++++++++++++++++++++++++++---------
2 files changed, 141 insertions(+), 31 deletions(-)
diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h
index fe739d35a864..c79ed378311f 100644
--- a/include/linux/list_lru.h
+++ b/include/linux/list_lru.h
@@ -83,6 +83,44 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
gfp_t gfp);
void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+/**
+ * list_lru_lock: lock the sublist for the given node and memcg
+ * @lru: the lru pointer
+ * @nid: the node id of the sublist to lock.
+ * @memcg: the cgroup of the sublist to lock.
+ *
+ * Returns the locked list_lru_one sublist. The caller must call
+ * list_lru_unlock() when done.
+ *
+ * You must ensure that the memcg is not freed during this call (e.g., with
+ * rcu or by taking a css refcnt).
+ *
+ * Return: the locked list_lru_one, or NULL on failure
+ */
+struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg);
+
+/**
+ * list_lru_unlock: unlock a sublist locked by list_lru_lock()
+ * @l: the list_lru_one to unlock
+ */
+void list_lru_unlock(struct list_lru_one *l);
+
+struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg);
+void list_lru_unlock_irq(struct list_lru_one *l);
+
+struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg, unsigned long *irq_flags);
+void list_lru_unlock_irqrestore(struct list_lru_one *l,
+ unsigned long *irq_flags);
+
+/* Caller-locked variants, see list_lru_add() etc for documentation */
+bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid, struct mem_cgroup *memcg);
+bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid);
+
/**
* list_lru_add: add an element to the lru list's tail
* @lru: the lru pointer
@@ -115,6 +153,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg);
+bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
+ struct mem_cgroup *memcg);
+
/**
* list_lru_add_obj: add an element to the lru list's tail
* @lru: the lru pointer
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 65962dbf6dda..df58226eea8c 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -15,17 +15,23 @@
#include "slab.h"
#include "internal.h"
-static inline void lock_list_lru(struct list_lru_one *l, bool irq)
+static inline void lock_list_lru(struct list_lru_one *l, bool irq,
+ unsigned long *irq_flags)
{
- if (irq)
+ if (irq_flags)
+ spin_lock_irqsave(&l->lock, *irq_flags);
+ else if (irq)
spin_lock_irq(&l->lock);
else
spin_lock(&l->lock);
}
-static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
+static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off,
+ unsigned long *irq_flags)
{
- if (irq_off)
+ if (irq_flags)
+ spin_unlock_irqrestore(&l->lock, *irq_flags);
+ else if (irq_off)
spin_unlock_irq(&l->lock);
else
spin_unlock(&l->lock);
@@ -78,7 +84,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
- bool irq, bool skip_empty)
+ bool irq, unsigned long *irq_flags, bool skip_empty)
{
struct list_lru_one *l;
@@ -86,12 +92,12 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
again:
l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
if (likely(l)) {
- lock_list_lru(l, irq);
+ lock_list_lru(l, irq, irq_flags);
if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
rcu_read_unlock();
return l;
}
- unlock_list_lru(l, irq);
+ unlock_list_lru(l, irq, irq_flags);
}
/*
* Caller may simply bail out if raced with reparenting or
@@ -132,38 +138,106 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
- bool irq, bool skip_empty)
+ bool irq, unsigned long *irq_flags, bool skip_empty)
{
struct list_lru_one *l = &lru->node[nid].lru;
- lock_list_lru(l, irq);
+ lock_list_lru(l, irq, irq_flags);
return l;
}
#endif /* CONFIG_MEMCG */
-/* The caller must ensure the memcg lifetime. */
-bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
- struct mem_cgroup *memcg)
+struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg)
{
- struct list_lru_node *nlru = &lru->node[nid];
- struct list_lru_one *l;
+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/false,
+ /*irq_flags=*/NULL, /*skip_empty=*/false);
+}
+
+void list_lru_unlock(struct list_lru_one *l)
+{
+ unlock_list_lru(l, /*irq_off=*/false, /*irq_flags=*/NULL);
+}
+
+struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg)
+{
+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
+ /*irq_flags=*/NULL, /*skip_empty=*/false);
+}
+
+void list_lru_unlock_irq(struct list_lru_one *l)
+{
+ unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/NULL);
+}
- l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
+struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg,
+ unsigned long *flags)
+{
+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
+ /*irq_flags=*/flags, /*skip_empty=*/false);
+}
+
+void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *flags)
+{
+ unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/flags);
+}
+
+bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid,
+ struct mem_cgroup *memcg)
+{
if (list_empty(item)) {
list_add_tail(item, &l->list);
/* Set shrinker bit if the first element was added */
if (!l->nr_items++)
set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
- unlock_list_lru(l, false);
- atomic_long_inc(&nlru->nr_items);
+ atomic_long_inc(&lru->node[nid].nr_items);
return true;
}
- unlock_list_lru(l, false);
return false;
}
EXPORT_SYMBOL_GPL(list_lru_add);
+bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid)
+{
+ if (!list_empty(item)) {
+ list_del_init(item);
+ l->nr_items--;
+ atomic_long_dec(&lru->node[nid].nr_items);
+ return true;
+ }
+ return false;
+}
+
+/* The caller must ensure the memcg lifetime. */
+bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
+ struct mem_cgroup *memcg)
+{
+ struct list_lru_one *l;
+ bool ret;
+
+ l = list_lru_lock(lru, nid, memcg);
+ ret = __list_lru_add(lru, l, item, nid, memcg);
+ list_lru_unlock(l);
+ return ret;
+}
+
+bool list_lru_add_irq(struct list_lru *lru, struct list_head *item,
+ int nid, struct mem_cgroup *memcg)
+{
+ struct list_lru_one *l;
+ bool ret;
+
+ l = list_lru_lock_irq(lru, nid, memcg);
+ ret = __list_lru_add(lru, l, item, nid, memcg);
+ list_lru_unlock_irq(l);
+ return ret;
+}
+
bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)
{
bool ret;
@@ -185,19 +259,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj);
bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg)
{
- struct list_lru_node *nlru = &lru->node[nid];
struct list_lru_one *l;
+ bool ret;
- l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
- if (!list_empty(item)) {
- list_del_init(item);
- l->nr_items--;
- unlock_list_lru(l, false);
- atomic_long_dec(&nlru->nr_items);
- return true;
- }
- unlock_list_lru(l, false);
- return false;
+ l = list_lru_lock(lru, nid, memcg);
+ ret = __list_lru_del(lru, l, item, nid);
+ list_lru_unlock(l);
+ return ret;
}
bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)
@@ -270,7 +338,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
unsigned long isolated = 0;
restart:
- l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true);
+ l = lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/irq_off,
+ /*irq_flags=*/NULL, /*skip_empty=*/true);
if (!l)
return isolated;
list_for_each_safe(item, n, &l->list) {
@@ -311,7 +380,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
BUG();
}
}
- unlock_list_lru(l, irq_off);
+ unlock_list_lru(l, irq_off, NULL);
out:
return isolated;
}
--
2.54.0
^ permalink raw reply related
* [PATCH v4 4/8] mm: list_lru: deduplicate lock_list_lru()
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
The MEMCG and !MEMCG paths have the same pattern. Share the code.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
mm/list_lru.c | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 9da6fce19832..65962dbf6dda 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -15,6 +15,14 @@
#include "slab.h"
#include "internal.h"
+static inline void lock_list_lru(struct list_lru_one *l, bool irq)
+{
+ if (irq)
+ spin_lock_irq(&l->lock);
+ else
+ spin_lock(&l->lock);
+}
+
static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
{
if (irq_off)
@@ -68,14 +76,6 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
return &lru->node[nid].lru;
}
-static inline void lock_list_lru(struct list_lru_one *l, bool irq)
-{
- if (irq)
- spin_lock_irq(&l->lock);
- else
- spin_lock(&l->lock);
-}
-
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
bool irq, bool skip_empty)
@@ -136,10 +136,7 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
{
struct list_lru_one *l = &lru->node[nid].lru;
- if (irq)
- spin_lock_irq(&l->lock);
- else
- spin_lock(&l->lock);
+ lock_list_lru(l, irq);
return l;
}
--
2.54.0
^ permalink raw reply related
* [PATCH v4 3/8] mm: list_lru: move list dead check to lock_list_lru_of_memcg()
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
Only the MEMCG variant of lock_list_lru() needs to check if there is a
race with cgroup deletion and list reparenting. Move the check to the
caller, so that the next patch can unify the lock_list_lru() variants.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
mm/list_lru.c | 17 ++++++++---------
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 9a68177619bf..9da6fce19832 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -68,17 +68,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
return &lru->node[nid].lru;
}
-static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
+static inline void lock_list_lru(struct list_lru_one *l, bool irq)
{
if (irq)
spin_lock_irq(&l->lock);
else
spin_lock(&l->lock);
- if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
- unlock_list_lru(l, irq);
- return false;
- }
- return true;
}
static inline struct list_lru_one *
@@ -90,9 +85,13 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
rcu_read_lock();
again:
l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
- if (likely(l) && lock_list_lru(l, irq)) {
- rcu_read_unlock();
- return l;
+ if (likely(l)) {
+ lock_list_lru(l, irq);
+ if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
+ rcu_read_unlock();
+ return l;
+ }
+ unlock_list_lru(l, irq);
}
/*
* Caller may simply bail out if raced with reparenting or
--
2.54.0
^ permalink raw reply related
* [PATCH v4 2/8] mm: list_lru: deduplicate unlock_list_lru()
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
The MEMCG and !MEMCG variants are the same. lock_list_lru() has the
same pattern when bailing. Consolidate into a common implementation.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
mm/list_lru.c | 29 +++++++++--------------------
1 file changed, 9 insertions(+), 20 deletions(-)
diff --git a/mm/list_lru.c b/mm/list_lru.c
index d3619961a7ac..9a68177619bf 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -15,6 +15,14 @@
#include "slab.h"
#include "internal.h"
+static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
+{
+ if (irq_off)
+ spin_unlock_irq(&l->lock);
+ else
+ spin_unlock(&l->lock);
+}
+
#ifdef CONFIG_MEMCG
static LIST_HEAD(memcg_list_lrus);
static DEFINE_MUTEX(list_lrus_mutex);
@@ -67,10 +75,7 @@ static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
else
spin_lock(&l->lock);
if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
- if (irq)
- spin_unlock_irq(&l->lock);
- else
- spin_unlock(&l->lock);
+ unlock_list_lru(l, irq);
return false;
}
return true;
@@ -101,14 +106,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
memcg = parent_mem_cgroup(memcg);
goto again;
}
-
-static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
-{
- if (irq_off)
- spin_unlock_irq(&l->lock);
- else
- spin_unlock(&l->lock);
-}
#else
static void list_lru_register(struct list_lru *lru)
{
@@ -147,14 +144,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
return l;
}
-
-static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
-{
- if (irq_off)
- spin_unlock_irq(&l->lock);
- else
- spin_unlock(&l->lock);
-}
#endif /* CONFIG_MEMCG */
/* The caller must ensure the memcg lifetime. */
--
2.54.0
^ permalink raw reply related
* [PATCH v4 1/8] mm: list_lru: lock_list_lru_of_memcg() cannot return NULL if !skip_empty
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
skip_empty is only for the shrinker to abort and skip a list that's
empty or whose cgroup is being deleted.
For list additions and deletions, the cgroup hierarchy is walked
upwards until a valid list_lru head is found, or it will fall back to
the node list. Acquiring the lock won't fail. Remove the NULL checks
in those callers.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
mm/list_lru.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/mm/list_lru.c b/mm/list_lru.c
index dd29bcf8eb5f..d3619961a7ac 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -165,8 +165,6 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
struct list_lru_one *l;
l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
- if (!l)
- return false;
if (list_empty(item)) {
list_add_tail(item, &l->list);
/* Set shrinker bit if the first element was added */
@@ -204,9 +202,8 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
{
struct list_lru_node *nlru = &lru->node[nid];
struct list_lru_one *l;
+
l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
- if (!l)
- return false;
if (!list_empty(item)) {
list_del_init(item);
l->nr_items--;
--
2.54.0
^ permalink raw reply related
* [PATCH v4 0/8] mm: switch THP shrinker to list_lru
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
This is version 4 of switching the THP shrinker to list_lru.
Changes in v4:
- guard folio_memcg_alloc_deferred() with mem_cgroup_disabled() to fix
NULL deref in __memcg_list_lru_alloc() when booting with
cgroup_disable=memory (e.g., kdump capture kernel) -- reported and
tested by Mikhail Zaslonko on s390 and x86
- flatten if (folio) branches in alloc_swap_folio() and alloc_anon_folio()
in a prep patch so the list_lru allocation additions are a clean minimal
diff (Lorenzo)
- folio_memcg_alloc_deferred() moved out of alloc_charge_folio() into the
anon-only collapse_huge_page() path; collapse_file() shares that helper
but its pages don't go on the THP shrinker queue (David)
- guard folio_memcg_alloc_deferred() with order > 1; mTHPs below order-2
can't be queued on the deferred split list (David)
- make deferred_split_lru static, hide behind folio_memcg_alloc_deferred()
wrapper with GFP_KERNEL (Lorenzo)
- rename l -> lru throughout huge_memory.c (Lorenzo)
- kdoc for folio_memcg_list_lru_alloc() (Lorenzo)
- list_lru_lock_irq()/unlock_irq()/add_irq() irq-disabling variants;
use list_lru_add_irq() in deferred_split_scan() (Lorenzo)
- reorder shrinker_free() before list_lru_destroy() (Lorenzo)
Changes in v3:
- dedicated lockdep_key for irqsafe deferred_split_lru.lock (syzbot)
- conditional list_lru ops in __folio_freeze_and_split_unmapped() (syzbot)
- annotate runs of inscrutable false, NULL, false function arguments (David)
- rename to folio_memcg_list_lru_alloc() (David)
Changes in v2:
- explicit rcu_read_lock() in __folio_freeze_and_split_unmapped() (Usama)
- split out list_lru prep bits (Dave)
The open-coded deferred split queue has issues. It's not NUMA-aware
(when cgroup is enabled), and it's more complicated in the callsites
interacting with it. Switching to list_lru fixes the NUMA problem and
streamlines things. It also simplifies planned shrinker work.
Patches 1-4 are cleanups and small refactors in list_lru code. They're
basically independent, but make the THP shrinker conversion easier.
Patch 5 extends the list_lru API to allow the caller to control the
locking scope. The THP shrinker has private state it needs to keep
synchronized with the LRU state.
Patch 6 extends the list_lru API with a convenience helper to do
list_lru head allocation (memcg_list_lru_alloc) when coming from a
folio. Anon THPs are instantiated in several places, and with the
folio reparenting patches pending, folio_memcg() access is now a more
delicate dance. This avoids having to replicate that dance everywhere.
Patch 7 flattens the folio allocation retry loops in alloc_swap_folio()
and alloc_anon_folio() without functional change, in preparation for
patch 8.
Patch 8 finally switches the deferred_split_queue to list_lru.
Based on mm-unstable.
include/linux/huge_mm.h | 7 +-
include/linux/list_lru.h | 68 +++++++++
include/linux/memcontrol.h | 4 -
include/linux/mmzone.h | 12 --
mm/huge_memory.c | 355 ++++++++++++++-----------------------------
mm/internal.h | 2 +-
mm/khugepaged.c | 3 +
mm/list_lru.c | 220 ++++++++++++++++++---------
mm/memcontrol.c | 12 +-
mm/memory.c | 52 ++++---
mm/mm_init.c | 15 --
11 files changed, 374 insertions(+), 376 deletions(-)
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-21 14:01 UTC (permalink / raw)
To: Vincent Guittot
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260521132901.GJ3126523@noisy.programming.kicks-ass.net>
On Thu, May 21, 2026 at 03:29:01PM +0200, Peter Zijlstra wrote:
> On Thu, May 21, 2026 at 02:13:48PM +0200, Vincent Guittot wrote:
>
> > > Would it not be simpler to just move the update_entity_lag() call up a
> > > bit, like so?
> > >
> > > ---
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
> > >
> > > clear_buddies(cfs_rq, se);
> > >
> > > + update_curr(cfs_rq);
> >
> > I agree it's simpler although we will call update_curr twice for one
> > level, but the 2nd call should be nop because of delta_exec being null
> >
> > Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p)
> > will ensure that we keep the same ordering as for_each_sched_entity
>
> Given:
>
> R
> |
> G
> |
> t
>
> Then task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.
>
> Since all the actual running happens inside R, this is what is required
> by update_entity_lag().
>
> Doing update_curr(task_cfs_rq()) here doesn't make sense.
>
> I'm not sure I see a way in which running them out of order hurts
> anything.
Bah, I'm so full of fail. So update_curr() takes ->h_curr, which for R
would be G's se, not t. So yeah, Prateek is right and I should stop
trying to do more than one thing at a time :-(
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Vincent Guittot @ 2026-05-21 13:56 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260521133904.GR3102924@noisy.programming.kicks-ass.net>
On Thu, 21 May 2026 at 15:39, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Thu, May 21, 2026 at 12:31:17PM +0200, Peter Zijlstra wrote:
>
> > Would it not be simpler to just move the update_entity_lag() call up a
> > bit, like so?
> >
> > ---
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
> >
> > clear_buddies(cfs_rq, se);
> >
> > + update_curr(cfs_rq);
> > + update_entity_lag(cfs_rq, se);
> > +
> > if (flags & DEQUEUE_DELAYED) {
> > WARN_ON_ONCE(!se->sched_delayed);
> > } else {
> > @@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
> >
> > dequeue_hierarchy(p, flags);
> >
> > - update_entity_lag(cfs_rq, se);
> > if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> > se->deadline -= se->vruntime;
> > se->rel_deadline = 1;
>
> FWIW, I pushed out a new queue:sched/flat with this on. I had to rebase
> because of: 6d2051403d6c ("sched/fair: Update util_est after updating
> util_avg during dequeue"), hopefully I didn't wreck that :/
This looks good to me
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Vincent Guittot @ 2026-05-21 13:44 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260521132901.GJ3126523@noisy.programming.kicks-ass.net>
On Thu, 21 May 2026 at 15:29, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Thu, May 21, 2026 at 02:13:48PM +0200, Vincent Guittot wrote:
>
> > > Would it not be simpler to just move the update_entity_lag() call up a
> > > bit, like so?
> > >
> > > ---
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
> > >
> > > clear_buddies(cfs_rq, se);
> > >
> > > + update_curr(cfs_rq);
> >
> > I agree it's simpler although we will call update_curr twice for one
> > level, but the 2nd call should be nop because of delta_exec being null
> >
> > Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p)
> > will ensure that we keep the same ordering as for_each_sched_entity
>
> Given:
>
> R
> |
> G
> |
> t
>
> Then task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.
Yes but update_curr() moves to R's cfs anyway before updating
vruntime, deadline and dl_server
>
> Since all the actual running happens inside R, this is what is required
> by update_entity_lag().
In other places like task_tick_fair, we follow the G then R order and
vruntime and deadline are updated while updating G
>
> Doing update_curr(task_cfs_rq()) here doesn't make sense.
>
> I'm not sure I see a way in which running them out of order hurts
> anything.
I was thinking of use cases which involves throttling but I haven't
gone deeply in the analyses
>
> > > + update_entity_lag(cfs_rq, se);
> > > +
> > > if (flags & DEQUEUE_DELAYED) {
> > > WARN_ON_ONCE(!se->sched_delayed);
> > > } else {
> > > @@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
> > >
> > > dequeue_hierarchy(p, flags);
> > >
> > > - update_entity_lag(cfs_rq, se);
> > > if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> > > se->deadline -= se->vruntime;
> > > se->rel_deadline = 1;
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-21 13:39 UTC (permalink / raw)
To: Vincent Guittot
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260521103117.GC3102624@noisy.programming.kicks-ass.net>
On Thu, May 21, 2026 at 12:31:17PM +0200, Peter Zijlstra wrote:
> Would it not be simpler to just move the update_entity_lag() call up a
> bit, like so?
>
> ---
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
>
> clear_buddies(cfs_rq, se);
>
> + update_curr(cfs_rq);
> + update_entity_lag(cfs_rq, se);
> +
> if (flags & DEQUEUE_DELAYED) {
> WARN_ON_ONCE(!se->sched_delayed);
> } else {
> @@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
>
> dequeue_hierarchy(p, flags);
>
> - update_entity_lag(cfs_rq, se);
> if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> se->deadline -= se->vruntime;
> se->rel_deadline = 1;
FWIW, I pushed out a new queue:sched/flat with this on. I had to rebase
because of: 6d2051403d6c ("sched/fair: Update util_est after updating
util_avg during dequeue"), hopefully I didn't wreck that :/
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-21 13:29 UTC (permalink / raw)
To: Vincent Guittot
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <CAKfTPtCpt7jYSPF5-wE8jjVPMBJrp_SGUV4brpbF9tASaJFp5g@mail.gmail.com>
On Thu, May 21, 2026 at 02:13:48PM +0200, Vincent Guittot wrote:
> > Would it not be simpler to just move the update_entity_lag() call up a
> > bit, like so?
> >
> > ---
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
> >
> > clear_buddies(cfs_rq, se);
> >
> > + update_curr(cfs_rq);
>
> I agree it's simpler although we will call update_curr twice for one
> level, but the 2nd call should be nop because of delta_exec being null
>
> Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p)
> will ensure that we keep the same ordering as for_each_sched_entity
Given:
R
|
G
|
t
Then task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.
Since all the actual running happens inside R, this is what is required
by update_entity_lag().
Doing update_curr(task_cfs_rq()) here doesn't make sense.
I'm not sure I see a way in which running them out of order hurts
anything.
> > + update_entity_lag(cfs_rq, se);
> > +
> > if (flags & DEQUEUE_DELAYED) {
> > WARN_ON_ONCE(!se->sched_delayed);
> > } else {
> > @@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
> >
> > dequeue_hierarchy(p, flags);
> >
> > - update_entity_lag(cfs_rq, se);
> > if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> > se->deadline -= se->vruntime;
> > se->rel_deadline = 1;
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-21 13:21 UTC (permalink / raw)
To: Vincent Guittot
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260521103117.GC3102624@noisy.programming.kicks-ass.net>
On Thu, May 21, 2026 at 12:31:17PM +0200, Peter Zijlstra wrote:
> On Wed, May 20, 2026 at 06:32:11PM +0200, Vincent Guittot wrote:
>
> > I finally fount the root cause of regression: the update of entity lag happened
> > after the task has been dequeued which screwed update_entity_lag():
> >
> > update_entity_lag must be called after updating curr and cfs_rd and before
> > clearing on_rq
> >
> > With the fix below I'm back to original hackbench figures and maybe even a bit better.
> > I haven't checked shceduling latency yet
I see a very slight hackbench regression on the high end, but meh. The
latency-slice test seems to have slightly improved max values, but this
isn't the most stable of things.
^ permalink raw reply
* Re: [PATCH 0/8] per-memcg-per-node kmem accounting
From: Alexandre Ghiti @ 2026-05-21 13:00 UTC (permalink / raw)
To: Joshua Hahn
Cc: Andrew Morton, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Dennis Zhou, Tejun Heo, Christoph Lameter,
Vlastimil Babka, Yosry Ahmed, Nhat Pham, Sergey Senozhatsky,
Chengming Zhou, Suren Baghdasaryan, Qi Zheng, David Hildenbrand,
Lorenzo Stoakes, Minchan Kim, Mike Rapoport, Axel Rasmussen,
Barry Song, Kairui Song, Wei Xu, Yuanchu Xie, Liam R . Howlett,
linux-mm, linux-kernel, cgroups
In-Reply-To: <20260521034604.4126295-1-joshua.hahnjy@gmail.com>
On 5/21/26 05:46, Joshua Hahn wrote:
> On Wed, 20 May 2026 10:39:59 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:
>
>> Hi Joshua,
>>
>> On 5/18/26 16:57, Joshua Hahn wrote:
>>> On Mon, 11 May 2026 22:20:35 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:
>>>
>>>> This series pursues the work initiated by Joshua [1]. We need kernel
>>>> memory to be accounted on a per-node basis in order to be able to
>>>> know the memcg and physical memory association.
>>>>
>>>> This series takes advantage of the recent introduction of per-node
>>>> obj_cgroup [2] and makes those obj_cgroup tied to their numa node.
>>>>
>>>> The bulk of the series is percpu per-node accounting: percpu
>>>> "precharges" the memcg before we know the actual location of the pages
>>>> it uses, so charging and accounting had to be split. All other kmem
>>>> users (slab, zswap, __memcg_kmem_charge_page) are straightforward
>>>> conversions (zswap support is limited in this series because Joshua
>>>> is working on it in parallel [3]).
>>>>
>>>> Thanks Joshua for your early feedbacks!
>>> Hello Alex,
>>>
>>> Thank you for your work!
>>>
>>> Overall I think the direction makes sense to me. Pre-overcharging makes sense to
>>> me as an approach, we would much rather overaccount than underaccount and
>>> later have to breach limits.
>>>
>>> I do have some concerns on performance, though. Namely, I think there are
>>> some expensive operations that I think would benefit from some performane
>>> benchmarking with this patch added (maybe some simple microbenchmarks that
>>> demonstrates kernel allocation overhead could be useful).
>>>
>>> From what I can tell, there is some additional performance overhead that has
>>> to do with iterating over num_possible_cpus() x pages_per_alloc, which
>>> doesn't seem trivial to me.
>> Indeed, let me microbenchmark the overhead on a large system.
> Hi Alex,
>
> That sounds great with me : -) Looking forward to the numbers!
>
>>> Another concern that I see is the stock credit system. Maybe we could be
>>> bypassing the stock check leading to more time spent doing the atomic
>>> operations.
>> I'm not following on this one, which atomic operations do you see that
>> could be bypassed?
> So in my initial scan of the patch 7 I had a concern that if we have a nested
> stock system (obj_cgroup stock and local credit "stock"), then we could
> incur more work if these are out of sync; do extra work in the stock refill
> path in obj_cgroup_precharge, and then do extra work on top in the loop
> within the pcpu_memcg_post_alloc_hook (obj_cgroup_account_kmem does the
> charging atomically I think).
>
> So what I mean is, I'm not sure what the "size" is typically for
> pcpu_memcg_post_alloc_hook. But it might be a worthwhile optimization to
> do precharge all the pages, then for each cpu iterate over the pages to
> figure out how many pages are used per nid (doing just math, not actually
> doing the atomic adds), and then outside both of these loops just iterate
> over every nid_objcg once to perform the atomic operation.
>
> Maybe this is needed or not (depending on how big "size" typically is
> and whether we go from doing O(1000) atomic adds --> O(10) or some
> big reduction, but I just wanted to toss it out there as something that
> could potentially be expensive.
I get it, I'll trace the microbenchmarks to see what happens there,
thanks for the suggestion.
Thanks again,
Alex
>>> obj_stock caches a single obj_cgroup, which means that if we split the objcg
>>> to be per-node (in patch 6), then the obj_stock basically gets invalidated
>>> every operation since we iterate over more objcgs (even though we are in
>>> the same logical objcg). Maybe I'm missing something?
>>
>> The objcg split comes from commit 01b9da291c49 ("mm: memcontrol: convert
>> objcg to be per-memcg per-node type") and the problem you describe is
>> exactly what Shakeel is trying to fix [1].
> Whoops O_o I completely missed that one. Sorry for flagging it again!
>
>> But I remember trying a microbenchmark and noticed a +5% regression (on
>> top of the 67% then...), I'll rebase this series on top of Shakeel's and
>> re-run.
> Sounds like a great idea! Thanks again Alex, have a great day! : -)
> Joshua
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Vincent Guittot @ 2026-05-21 12:13 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260521103117.GC3102624@noisy.programming.kicks-ass.net>
On Thu, 21 May 2026 at 12:31, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Wed, May 20, 2026 at 06:32:11PM +0200, Vincent Guittot wrote:
>
> > I finally fount the root cause of regression: the update of entity lag happened
> > after the task has been dequeued which screwed update_entity_lag():
> >
> > update_entity_lag must be called after updating curr and cfs_rd and before
> > clearing on_rq
> >
> > With the fix below I'm back to original hackbench figures and maybe even a bit better.
> > I haven't checked shceduling latency yet
> >
> > ---
> > kernel/sched/fair.c | 5 ++++-
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 77d0e1937f2c..32fe57004f27 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >
> > update_stats_dequeue_fair(cfs_rq, se, flags);
> >
> > + if (entity_is_task(se))
> > + update_entity_lag(&rq_of(cfs_rq)->cfs, se);
> > +
> > se->on_rq = 0;
> > account_entity_dequeue(cfs_rq, se);
> >
> > @@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> > if (sched_feat(DELAY_DEQUEUE) && delay &&
> > !entity_eligible(cfs_rq, se)) {
> > update_load_avg(cfs_rq_of(se), se, 0);
> > + update_entity_lag(cfs_rq, se);
> > set_delayed(se);
> > return false;
> > }
> > @@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> >
> > dequeue_hierarchy(p, flags);
> >
> > - update_entity_lag(cfs_rq, se);
> > if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> > se->deadline -= se->vruntime;
> > se->rel_deadline = 1;
>
> Argh!!! Thank you! I've gone blind staring at all this :/
>
> Would it not be simpler to just move the update_entity_lag() call up a
> bit, like so?
>
> ---
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
>
> clear_buddies(cfs_rq, se);
>
> + update_curr(cfs_rq);
I agree it's simpler although we will call update_curr twice for one
level, but the 2nd call should be nop because of delta_exec being null
Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p)
will ensure that we keep the same ordering as for_each_sched_entity
> + update_entity_lag(cfs_rq, se);
> +
> if (flags & DEQUEUE_DELAYED) {
> WARN_ON_ONCE(!se->sched_delayed);
> } else {
> @@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
>
> dequeue_hierarchy(p, flags);
>
> - update_entity_lag(cfs_rq, se);
> if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> se->deadline -= se->vruntime;
> se->rel_deadline = 1;
^ permalink raw reply
* Re: [PATCH] cgroup/dmem: implement dmem.high soft limit and throttling
From: Qiliang Yuan @ 2026-05-21 11:28 UTC (permalink / raw)
To: tj
Cc: dev, mripard, natalie.vock, hannes, mkoutny, cgroups, dri-devel,
linux-kernel
In-Reply-To: <ag2EWbmlWhK2a3zz@slm.duckdns.org>
Hello Tejun,
On Wed, May 20, 2026 at 09:52 AM Tejun Heo wrote:
> I'm not sure about complicating dmem control model without implementing
> reclaim. What are we slowing them down for if the only recovery action is
> killing them?
Thank you for the feedback. Your point about the lack of a reclaim path
is well-taken. Simple throttling without a way to recover resources is
indeed incomplete and inconsistent with the cgroup v2 philosophy.
To address this from several perspectives in v2:
1. Recovery Path: As suggested by Maarten Lankhorst, we will pivot to a
reclaim-centric model. Exceeding `dmem.high` will trigger a prioritized
eviction process, where memory objects from over-limit cgroups are
targeted first for reclaim. This provides the meaningful "recovery action"
you mentioned.
2. Backpressure: Throttling will then serve as a secondary tool to
synchronize user-space demand with the kernel's reclaim speed, preventing
bursty workloads from overwhelming the system before reclaim can finish.
3. Graceful Degradation: For GPU compute jobs, this model provides a
managed "pressure point" that allows transient peaks to be handled via
rebalancing rather than immediate, fatal allocation failures (max/OOM).
The goal for v2 is to achieve convergence with the `memory.high` model,
pairing prioritized reclaim with backpressure.
Thanks,
Qiliang
^ permalink raw reply
* Re: [PATCH] cgroup/dmem: implement dmem.high soft limit and throttling
From: Qiliang Yuan @ 2026-05-21 11:28 UTC (permalink / raw)
To: dev
Cc: cgroups, dri-devel, hannes, linux-kernel, mkoutny, mripard,
natalie.vock, tj
In-Reply-To: <63878874-39d2-43d5-9fc3-68addf9ebbdd@lankhorst.se>
Hello Maarten,
On Thu, May 21, 2026 at 09:45 AM Maarten Lankhorst wrote:
> It's the approach I'm more worried about. I believe that it's
> better if we punish exceeding their high limit by preferentially
> evicting those.
>
> It would make eviction run in 3 passes on the affected cgroup tree:
> - Round 1: Clients above their 'high' limit
> - Round 2: Clients above their 'low/min' limits
> - Round 3: Clients at or below their 'low' limit
Thank you for this concrete suggestion. This 3-pass eviction model is
exactly what's needed to make the dmem soft limit effective.
It addresses the core problem of providing a viable "recovery action" when
the limit is reached. By integrating these thresholds directly into the
TTM/dmem eviction weight calculation, we can achieve a more natural
over-subscription model.
I will rework the series for v2 to incorporate this hierarchy-aware
eviction logic.
Kind regards,
~Qiliang
^ permalink raw reply
* Re: [PATCH] cgroup/dmem: implement dmem.high soft limit and throttling
From: Qiliang Yuan @ 2026-05-21 11:28 UTC (permalink / raw)
To: natalie.vock
Cc: dev, mripard, tj, hannes, mkoutny, cgroups, dri-devel,
linux-kernel
In-Reply-To: <c9eeee76-25a8-482e-9ef4-74971537457f@gmx.de>
Hi Natalie,
On Thu, May 21, 2026 at 10:52 AM Natalie Vock wrote:
> Interesting proposal, but inserting sleeps on allocation is never a good
> idea and doesn't work like you might think it does. In graphics driver
> land, lots of random things may result in buffer allocation functions
> being called.
[...]
> Your approach could lead to every single
> submission sleeping for at least 100ms, thus permanently destroying
> performance.
Thank you very much for the detailed explanation of the impact on TTM and
Submit IOCTLs. You are absolutely right—injecting sleeps into the charge
path, which is hit frequently during buffer validation and residency changes,
would indeed be catastrophic for GPU performance.
> Maarten's suggestion of preferentially evicting memory that is over the
> high limit sounds like a better approach.
I agree. Blocking the submission pipeline is not the right way to apply
backpressure. I will abandon the current sleep-on-allocation approach and
focus on implemented prioritized eviction as you and Maarten suggested.
This ensures that reaching the "high" limit triggers a meaningful reclaim
action rather than just stalling the GPU pipeline.
Best regards,
Qiliang
^ permalink raw reply
* Re: [PATCH] cgroup/dmem: implement dmem.high soft limit and throttling
From: Natalie Vock @ 2026-05-21 10:52 UTC (permalink / raw)
To: Qiliang Yuan, Maarten Lankhorst, Maxime Ripard, Tejun Heo,
Johannes Weiner, Michal Koutný
Cc: cgroups, dri-devel, linux-kernel
In-Reply-To: <20260520-feature-dmem-high-v1-1-97ca0cb7f95a@gmail.com>
On 5/20/26 08:07, Qiliang Yuan wrote:
> Introduce the "high" soft limit for the dmem cgroup v2 controller.
> When a cgroup's device memory usage exceeds its high limit, tasks
> belonging to that cgroup are throttled by being forced into a sleep
> before returning to user space, instead of being failed outright
> as with the "max" limit.
>
> Key changes:
> - Add high counter configuration to dmem_cgroup_pool.
> - Add over-high check in the try_charge path and set TIF_NOTIFY_RESUME.
> - Inject the dmem throttling handler into resume_user_mode_work.
> - Implement the handler to perform a 100ms interruptible sleep for
> over-limit tasks.
Interesting proposal, but inserting sleeps on allocation is never a good
idea and doesn't work like you might think it does. In graphics driver
land, lots of random things may result in buffer allocation functions
being called. Whenever TTM determines some buffer needs to be physically
moved (most often during VRAM contention, but also as a result of
pinning buffers for scanout, etc etc), dmem cgroup pools are
charged/uncharged in accordance with the change in buffer residency.
Sleeping in a charge/uncharge path means that in the worst case, a task
will be put to sleep over and over again for exceeding its high limit
just once.
Most critically, submit ioctls typically go over the task's entire
working set and call ttm_bo_validate() to make sure the buffer is
accessible by the GPU, since paging things in on fault is not available
in many consumer GPUs. Your approach could lead to every single
submission sleeping for at least 100ms, thus permanently destroying
performance.
Maarten's suggestion of preferentially evicting memory that is over the
high limit sounds like a better approach.
(Also, did you use AI for this? Please disclose your AI usage as per
kernel guidelines if so.)
Best,
Natalie
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-21 10:31 UTC (permalink / raw)
To: Vincent Guittot
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <ag3iC-jH6HPoWKGo@vingu-cube>
On Wed, May 20, 2026 at 06:32:11PM +0200, Vincent Guittot wrote:
> I finally fount the root cause of regression: the update of entity lag happened
> after the task has been dequeued which screwed update_entity_lag():
>
> update_entity_lag must be called after updating curr and cfs_rd and before
> clearing on_rq
>
> With the fix below I'm back to original hackbench figures and maybe even a bit better.
> I haven't checked shceduling latency yet
>
> ---
> kernel/sched/fair.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 77d0e1937f2c..32fe57004f27 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>
> update_stats_dequeue_fair(cfs_rq, se, flags);
>
> + if (entity_is_task(se))
> + update_entity_lag(&rq_of(cfs_rq)->cfs, se);
> +
> se->on_rq = 0;
> account_entity_dequeue(cfs_rq, se);
>
> @@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> if (sched_feat(DELAY_DEQUEUE) && delay &&
> !entity_eligible(cfs_rq, se)) {
> update_load_avg(cfs_rq_of(se), se, 0);
> + update_entity_lag(cfs_rq, se);
> set_delayed(se);
> return false;
> }
> @@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
>
> dequeue_hierarchy(p, flags);
>
> - update_entity_lag(cfs_rq, se);
> if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> se->deadline -= se->vruntime;
> se->rel_deadline = 1;
Argh!!! Thank you! I've gone blind staring at all this :/
Would it not be simpler to just move the update_entity_lag() call up a
bit, like so?
---
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
clear_buddies(cfs_rq, se);
+ update_curr(cfs_rq);
+ update_entity_lag(cfs_rq, se);
+
if (flags & DEQUEUE_DELAYED) {
WARN_ON_ONCE(!se->sched_delayed);
} else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
dequeue_hierarchy(p, flags);
- update_entity_lag(cfs_rq, se);
if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
se->deadline -= se->vruntime;
se->rel_deadline = 1;
^ permalink raw reply
* [tj-cgroup:for-next] BUILD SUCCESS 936f0880adaf8edab431e64503a5686a8d79e54e
From: kernel test robot @ 2026-05-21 10:07 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
branch HEAD: 936f0880adaf8edab431e64503a5686a8d79e54e Merge branch 'for-7.1-fixes' into for-next
elapsed time: 854m
configs tested: 218
configs skipped: 3
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allmodconfig gcc-15.2.0
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc allyesconfig gcc-15.2.0
arc defconfig gcc-15.2.0
arc randconfig-001-20260521 gcc-8.5.0
arc randconfig-002-20260521 gcc-8.5.0
arm allnoconfig clang-23
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm allyesconfig gcc-15.2.0
arm defconfig gcc-15.2.0
arm footbridge_defconfig clang-17
arm randconfig-001-20260521 gcc-8.5.0
arm randconfig-002-20260521 gcc-8.5.0
arm randconfig-003-20260521 gcc-8.5.0
arm randconfig-004-20260521 gcc-8.5.0
arm64 allmodconfig clang-19
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260521 gcc-8.5.0
arm64 randconfig-002-20260521 gcc-8.5.0
arm64 randconfig-003-20260521 gcc-8.5.0
arm64 randconfig-004-20260521 gcc-8.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260521 gcc-8.5.0
csky randconfig-002-20260521 gcc-8.5.0
hexagon allmodconfig clang-17
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig clang-23
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001-20260521 gcc-11.5.0
hexagon randconfig-002-20260521 gcc-11.5.0
i386 allmodconfig clang-20
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001-20260521 clang-20
i386 buildonly-randconfig-002-20260521 clang-20
i386 buildonly-randconfig-003-20260521 clang-20
i386 buildonly-randconfig-004-20260521 clang-20
i386 buildonly-randconfig-005-20260521 clang-20
i386 buildonly-randconfig-006-20260521 clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001-20260521 clang-20
i386 randconfig-002-20260521 clang-20
i386 randconfig-003-20260521 clang-20
i386 randconfig-004-20260521 clang-20
i386 randconfig-005-20260521 clang-20
i386 randconfig-006-20260521 clang-20
i386 randconfig-007-20260521 clang-20
i386 randconfig-011-20260521 gcc-14
i386 randconfig-012-20260521 gcc-14
i386 randconfig-013-20260521 gcc-14
i386 randconfig-014-20260521 gcc-14
i386 randconfig-015-20260521 gcc-14
i386 randconfig-016-20260521 gcc-14
i386 randconfig-017-20260521 gcc-14
loongarch allmodconfig clang-19
loongarch allmodconfig clang-23
loongarch allnoconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001-20260521 gcc-11.5.0
loongarch randconfig-002-20260521 gcc-11.5.0
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k allyesconfig gcc-15.2.0
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
mips rt305x_defconfig clang-23
nios2 allmodconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig clang-23
nios2 allnoconfig gcc-11.5.0
nios2 defconfig clang-19
nios2 randconfig-001-20260521 gcc-11.5.0
nios2 randconfig-002-20260521 gcc-11.5.0
openrisc allmodconfig clang-23
openrisc allmodconfig gcc-15.2.0
openrisc allnoconfig clang-23
openrisc allnoconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allnoconfig gcc-15.2.0
parisc allyesconfig clang-19
parisc allyesconfig gcc-15.2.0
parisc defconfig gcc-15.2.0
parisc randconfig-001-20260521 gcc-12.5.0
parisc randconfig-002-20260521 gcc-12.5.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc allnoconfig gcc-15.2.0
powerpc mgcoge_defconfig clang-23
powerpc mpc837x_rdb_defconfig gcc-15.2.0
powerpc randconfig-001-20260521 gcc-12.5.0
powerpc randconfig-002-20260521 gcc-12.5.0
powerpc sam440ep_defconfig gcc-15.2.0
powerpc64 randconfig-001-20260521 gcc-12.5.0
powerpc64 randconfig-002-20260521 gcc-12.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allnoconfig gcc-15.2.0
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001 gcc-15.2.0
riscv randconfig-001-20260521 gcc-15.2.0
riscv randconfig-002 gcc-15.2.0
riscv randconfig-002-20260521 gcc-15.2.0
s390 allmodconfig clang-18
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001 gcc-15.2.0
s390 randconfig-001-20260521 gcc-15.2.0
s390 randconfig-002 gcc-15.2.0
s390 randconfig-002-20260521 gcc-15.2.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allnoconfig gcc-15.2.0
sh allyesconfig clang-19
sh allyesconfig gcc-15.2.0
sh defconfig gcc-14
sh randconfig-001 gcc-15.2.0
sh randconfig-001-20260521 gcc-15.2.0
sh randconfig-002 gcc-15.2.0
sh randconfig-002-20260521 gcc-15.2.0
sparc allnoconfig clang-23
sparc allnoconfig gcc-15.2.0
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260521 gcc-8.5.0
sparc randconfig-002-20260521 gcc-8.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260521 gcc-8.5.0
sparc64 randconfig-002-20260521 gcc-8.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-14
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260521 gcc-8.5.0
um randconfig-002-20260521 gcc-8.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260521 clang-20
x86_64 buildonly-randconfig-002-20260521 clang-20
x86_64 buildonly-randconfig-003-20260521 clang-20
x86_64 buildonly-randconfig-004-20260521 clang-20
x86_64 buildonly-randconfig-005-20260521 clang-20
x86_64 buildonly-randconfig-006-20260521 clang-20
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260521 clang-20
x86_64 randconfig-002-20260521 clang-20
x86_64 randconfig-003-20260521 clang-20
x86_64 randconfig-004-20260521 clang-20
x86_64 randconfig-005-20260521 clang-20
x86_64 randconfig-006-20260521 clang-20
x86_64 randconfig-011 gcc-14
x86_64 randconfig-011-20260521 gcc-14
x86_64 randconfig-012 gcc-14
x86_64 randconfig-012-20260521 gcc-14
x86_64 randconfig-013 gcc-14
x86_64 randconfig-013-20260521 gcc-14
x86_64 randconfig-014 gcc-14
x86_64 randconfig-014-20260521 gcc-14
x86_64 randconfig-015 gcc-14
x86_64 randconfig-015-20260521 gcc-14
x86_64 randconfig-016 gcc-14
x86_64 randconfig-016-20260521 gcc-14
x86_64 randconfig-071 clang-20
x86_64 randconfig-071-20260521 clang-20
x86_64 randconfig-072 clang-20
x86_64 randconfig-072-20260521 clang-20
x86_64 randconfig-073 clang-20
x86_64 randconfig-073-20260521 clang-20
x86_64 randconfig-074 clang-20
x86_64 randconfig-074-20260521 clang-20
x86_64 randconfig-075 clang-20
x86_64 randconfig-075-20260521 clang-20
x86_64 randconfig-076 clang-20
x86_64 randconfig-076-20260521 clang-20
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allnoconfig gcc-15.2.0
xtensa allyesconfig clang-23
xtensa randconfig-001-20260521 gcc-8.5.0
xtensa randconfig-002-20260521 gcc-8.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH] cgroup/dmem: implement dmem.high soft limit and throttling
From: Maarten Lankhorst @ 2026-05-21 9:45 UTC (permalink / raw)
To: Qiliang Yuan, Maxime Ripard, Natalie Vock, Tejun Heo,
Johannes Weiner, Michal Koutný
Cc: cgroups, dri-devel, linux-kernel
In-Reply-To: <20260520-feature-dmem-high-v1-1-97ca0cb7f95a@gmail.com>
Hello Qiliang,
Den 2026-05-20 kl. 08:07, skrev Qiliang Yuan:
> Introduce the "high" soft limit for the dmem cgroup v2 controller.
> When a cgroup's device memory usage exceeds its high limit, tasks
> belonging to that cgroup are throttled by being forced into a sleep
> before returning to user space, instead of being failed outright
> as with the "max" limit.
>
> Key changes:
> - Add high counter configuration to dmem_cgroup_pool.
> - Add over-high check in the try_charge path and set TIF_NOTIFY_RESUME.
> - Inject the dmem throttling handler into resume_user_mode_work.
> - Implement the handler to perform a 100ms interruptible sleep for
> over-limit tasks.
>
> This mechanism provides smoother over-subscription support for device
> memory resources.
>
> Signed-off-by: Qiliang Yuan <realwujing@gmail.com>
> ---
> This series introduces the "high" soft limit and associated task
> throttling mechanism to the dmem cgroup v2 controller.
>
> The device memory (VRAM) management currently only supports hard limits
> (max), which leads to immediate allocation failures when reached. This
> can be disruptive for GPU-bound AI workloads. By introducing a soft
> limit, we allow cgroups to exceed their quota temporarily while
> applying backpressure via task throttling before the process returns
> to user space.
>
> The mechanism is inspired by the memory cgroup's high limit:
> - When usage > high, the task is marked with TIF_NOTIFY_RESUME.
> - Upon returning to user space, it triggers a 100ms sleep.
> - This provides a smoother over-subscription model for GPU resources.
>
> Qiliang Yuan (1):
>
> cgroup/dmem: implement dmem.high soft limit and throttling
> ---
> To: Maarten Lankhorst <dev@lankhorst.se>
> To: Maxime Ripard <mripard@kernel.org>
> To: Natalie Vock <natalie.vock@gmx.de>
> To: Tejun Heo <tj@kernel.org>
> To: Johannes Weiner <hannes@cmpxchg.org>
> To: Michal Koutný <mkoutny@suse.com>
> Cc: cgroups@vger.kernel.org
> Cc: dri-devel@lists.freedesktop.org
> Cc: linux-kernel@vger.kernel.org
> ---
I think the concept of allowing userspace to throttle on high
is interesting.
It's the approach I'm more worried about. I believe that it's
better if we punish exceeding their high limit by preferentially
evicting those.
It would make eviction run in 3 passes on the affected cgroup tree:
- Round 1: Clients above their 'high' limit
- Round 2: Clients above their 'low/min' limits
- Round 3: Clients at or below their 'low' limit
And the same client's cgroup, below 'min' limit as well.
I'm open for other ideas as well. Perhaps a flag that would allow
allocation or binding to an address space to fail if it would need
to evict, or a notification sent to the affected client that they
went over high.
Have you tried any other approaches before this one?
Kind regards,
~Maarten Lankhorst
^ permalink raw reply
* Re: [PATCH] blk-throttle: schedule parent dispatch in tg_flush_bios()
From: Tao Cui @ 2026-05-21 9:04 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: axboe, cgroups, josef, linux-block, tj
In-Reply-To: <ag6OXDuTc3JubfqV@shinmob>
Hello，
Following up on my analysis, I did some further investigation into
why I couldn't reproduce the failure locally.
The root cause is a timing-sensitive race between the dispatch and
SCSI state transition. Whether the race manifests depends on kernel
build configs that affect execution speed.
在 2026/5/21 12:54, Shin'ichiro Kawasaki 写道:
>> If the issue persists after running with LC_ALL=C ./check throtl/004, I'll investigate further.
>
I found the following key differences between my config and the
Fedora debug config:
Config Mine Fedora
CONFIG_PREEMPT n y
CONFIG_KASAN n y
CONFIG_PROVE_LOCKING n y
CONFIG_DEBUG_PREEMPT n y
CONFIG_DEBUG_SPINLOCK n y
CONFIG_LOCKDEP n y
CONFIG_DEBUG_KMEMLEAK n y
KASAN and lock debugging slow down the __del_gendisk() path
significantly. This gives dispatch_work enough time to run and
submit bios while the device is still in SDEV_CANCEL, triggering
the BLK_STS_OFFLINE (ENODEV) response.
After disabling the first four options (CONFIG_PREEMPT, CONFIG_KASAN,
CONFIG_PROVE_LOCKING, CONFIG_DEBUG_PREEMPT) on the Fedora config,
the test passes:
throtl/004 (nullb) (delete disk while IO is throttled) [passed]
runtime 0.763s
throtl/004 (sdebug) (delete disk while IO is throttled) [passed]
runtime 0.923s
This confirms the race is timing-sensitive and depends on the
kernel build config. The fix with bio_io_error() is still the
right approach -- it completes bios directly at the throttle
layer, avoiding the SCSI state check entirely.
Thanks,
Tao
^ permalink raw reply
* [tj-cgroup:for-7.1-fixes] BUILD SUCCESS 22572dbcd3486e6c4dced877125bbf50e4e24edf
From: kernel test robot @ 2026-05-21 8:02 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-7.1-fixes
branch HEAD: 22572dbcd3486e6c4dced877125bbf50e4e24edf cgroup: rstat: relax NMI guard after switch to try_cmpxchg
elapsed time: 729m
configs tested: 207
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allmodconfig gcc-15.2.0
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc allyesconfig gcc-15.2.0
arc defconfig gcc-15.2.0
arc randconfig-001-20260521 gcc-8.5.0
arc randconfig-002-20260521 gcc-8.5.0
arm allnoconfig clang-23
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm allyesconfig gcc-15.2.0
arm defconfig gcc-15.2.0
arm footbridge_defconfig clang-17
arm randconfig-001-20260521 gcc-8.5.0
arm randconfig-002-20260521 gcc-8.5.0
arm randconfig-003-20260521 gcc-8.5.0
arm randconfig-004-20260521 gcc-8.5.0
arm64 allmodconfig clang-19
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260521 gcc-8.5.0
arm64 randconfig-002-20260521 gcc-8.5.0
arm64 randconfig-003-20260521 gcc-8.5.0
arm64 randconfig-004-20260521 gcc-8.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260521 gcc-8.5.0
csky randconfig-002-20260521 gcc-8.5.0
hexagon allmodconfig clang-17
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig clang-23
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001-20260521 gcc-11.5.0
hexagon randconfig-002-20260521 gcc-11.5.0
i386 allmodconfig clang-20
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001-20260521 clang-20
i386 buildonly-randconfig-002-20260521 clang-20
i386 buildonly-randconfig-003-20260521 clang-20
i386 buildonly-randconfig-004-20260521 clang-20
i386 buildonly-randconfig-005-20260521 clang-20
i386 buildonly-randconfig-006-20260521 clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001-20260521 clang-20
i386 randconfig-002-20260521 clang-20
i386 randconfig-003-20260521 clang-20
i386 randconfig-004-20260521 clang-20
i386 randconfig-005-20260521 clang-20
i386 randconfig-006-20260521 clang-20
i386 randconfig-007-20260521 clang-20
i386 randconfig-011-20260521 gcc-14
i386 randconfig-012-20260521 gcc-14
i386 randconfig-013-20260521 gcc-14
i386 randconfig-014-20260521 gcc-14
i386 randconfig-015-20260521 gcc-14
i386 randconfig-016-20260521 gcc-14
i386 randconfig-017-20260521 gcc-14
loongarch allmodconfig clang-19
loongarch allmodconfig clang-23
loongarch allnoconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001-20260521 gcc-11.5.0
loongarch randconfig-002-20260521 gcc-11.5.0
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k allyesconfig gcc-15.2.0
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
mips rt305x_defconfig clang-23
nios2 10m50_defconfig gcc-11.5.0
nios2 allmodconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig clang-23
nios2 allnoconfig gcc-11.5.0
nios2 defconfig clang-19
nios2 randconfig-001-20260521 gcc-11.5.0
nios2 randconfig-002-20260521 gcc-11.5.0
openrisc allmodconfig clang-23
openrisc allmodconfig gcc-15.2.0
openrisc allnoconfig clang-23
openrisc allnoconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allnoconfig gcc-15.2.0
parisc allyesconfig clang-19
parisc allyesconfig gcc-15.2.0
parisc defconfig gcc-15.2.0
parisc randconfig-001-20260521 gcc-12.5.0
parisc randconfig-002-20260521 gcc-12.5.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc allnoconfig gcc-15.2.0
powerpc mgcoge_defconfig clang-23
powerpc randconfig-001-20260521 gcc-12.5.0
powerpc randconfig-002-20260521 gcc-12.5.0
powerpc sam440ep_defconfig gcc-15.2.0
powerpc64 randconfig-001-20260521 gcc-12.5.0
powerpc64 randconfig-002-20260521 gcc-12.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allnoconfig gcc-15.2.0
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260521 gcc-15.2.0
riscv randconfig-002-20260521 gcc-15.2.0
s390 allmodconfig clang-18
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260521 gcc-15.2.0
s390 randconfig-002-20260521 gcc-15.2.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allnoconfig gcc-15.2.0
sh allyesconfig clang-19
sh allyesconfig gcc-15.2.0
sh defconfig gcc-14
sh randconfig-001-20260521 gcc-15.2.0
sh randconfig-002-20260521 gcc-15.2.0
sparc allnoconfig clang-23
sparc allnoconfig gcc-15.2.0
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260521 gcc-8.5.0
sparc randconfig-002-20260521 gcc-8.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260521 gcc-8.5.0
sparc64 randconfig-002-20260521 gcc-8.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-14
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260521 gcc-8.5.0
um randconfig-002-20260521 gcc-8.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260521 clang-20
x86_64 buildonly-randconfig-002-20260521 clang-20
x86_64 buildonly-randconfig-003-20260521 clang-20
x86_64 buildonly-randconfig-004-20260521 clang-20
x86_64 buildonly-randconfig-005-20260521 clang-20
x86_64 buildonly-randconfig-006-20260521 clang-20
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260521 clang-20
x86_64 randconfig-002-20260521 clang-20
x86_64 randconfig-003-20260521 clang-20
x86_64 randconfig-004-20260521 clang-20
x86_64 randconfig-005-20260521 clang-20
x86_64 randconfig-006-20260521 clang-20
x86_64 randconfig-011-20260521 gcc-14
x86_64 randconfig-012-20260521 gcc-14
x86_64 randconfig-013-20260521 gcc-14
x86_64 randconfig-014-20260521 gcc-14
x86_64 randconfig-015-20260521 gcc-14
x86_64 randconfig-016-20260521 gcc-14
x86_64 randconfig-071 clang-20
x86_64 randconfig-071-20260521 clang-20
x86_64 randconfig-072 clang-20
x86_64 randconfig-072-20260521 clang-20
x86_64 randconfig-073 clang-20
x86_64 randconfig-073-20260521 clang-20
x86_64 randconfig-074 clang-20
x86_64 randconfig-074-20260521 clang-20
x86_64 randconfig-075 clang-20
x86_64 randconfig-075-20260521 clang-20
x86_64 randconfig-076 clang-20
x86_64 randconfig-076-20260521 clang-20
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allnoconfig gcc-15.2.0
xtensa allyesconfig clang-23
xtensa allyesconfig gcc-15.2.0
xtensa randconfig-001-20260521 gcc-8.5.0
xtensa randconfig-002-20260521 gcc-8.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Vincent Guittot @ 2026-05-21 7:56 UTC (permalink / raw)
To: K Prateek Nayak
Cc: Peter Zijlstra, mingo, longman, chenridong, juri.lelli,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, qyousef
In-Reply-To: <eb61103c-3dca-4032-90af-d472b26d2dbe@amd.com>
On Thu, 21 May 2026 at 04:57, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> On 5/20/2026 10:02 PM, Vincent Guittot wrote:
> > I finally fount the root cause of regression: the update of entity lag happened
> > after the task has been dequeued which screwed update_entity_lag():
>
> Great catch!
>
> >
> > update_entity_lag must be called after updating curr and cfs_rd and before
> > clearing on_rq
> >
> > With the fix below I'm back to original hackbench figures and maybe even a bit better.
> > I haven't checked shceduling latency yet
> >
> > ---
> > kernel/sched/fair.c | 5 ++++-
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 77d0e1937f2c..32fe57004f27 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >
> > update_stats_dequeue_fair(cfs_rq, se, flags);
> >
> > + if (entity_is_task(se))
> > + update_entity_lag(&rq_of(cfs_rq)->cfs, se);
> > +
> > se->on_rq = 0;
>
> Ah! The curr->on_rq indicator changes here and we'll start ignoring it
> for avg_vruntime() calculation afterwards! Makes sense.
>
> > account_entity_dequeue(cfs_rq, se);
> >
> > @@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> > if (sched_feat(DELAY_DEQUEUE) && delay &&
> > !entity_eligible(cfs_rq, se)) {
>
> Does this need a update_curr() before checking entity_eligible()?
Yes we need to update curr first
>
> Currently these bits reside in dequeue_entity() and is always done after
> a update_curr(cfs_rq) but here we may need a:
>
> update_curr(task_cfs_rq(p)); /* to catch up h_curr's vruntime */
>
> Just doing it for task_cfs_rq(p) should be fine since we only have to
> catch up curr's vruntime - sum_w_vruntime and sum_weight at root cfs_rq
> should be stable for all the tasks on rb-tree.
>
> > update_load_avg(cfs_rq_of(se), se, 0);
> > + update_entity_lag(cfs_rq, se);
> > set_delayed(se);
> > return false;
> > }
> > @@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> >
> > dequeue_hierarchy(p, flags);
> >
> > - update_entity_lag(cfs_rq, se);
>
> If we decide to do a update_curr(task_cfs_rq(p)) at the beginning of
> __dequeue_task(), we can just move this to above dequeue_hierarchy()
> before se->on_rq indicators are modified.
>
> Thoughts?
yes it's doable, we will have a spurious update_curr in
dequeue_hierarchy but that will be a nop because of a null delta_exec
With flat hierarchy, vruntime and deadline are no longer linked to the
cfs hierarchy. A possibility could be to move the update of vruntime
and deadline outside but this is more complex because of delta_exec
The same apply for dl_server
>
> > if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> > se->deadline -= se->vruntime;
> > se->rel_deadline = 1;
>
> --
> Thanks and Regards,
> Prateek
>
^ permalink raw reply
* Re: [PATCH] blk-throttle: schedule parent dispatch in tg_flush_bios()
From: Tao Cui @ 2026-05-21 7:54 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: axboe, cgroups, josef, linux-block, tj
In-Reply-To: <ag6OXDuTc3JubfqV@shinmob>
Hello,
I was able to reproduce the failure and have identified the root cause.
在 2026/5/21 12:54, Shin'ichiro Kawasaki 写道:
>> If the issue persists after running with LC_ALL=C ./check throtl/004, I'll investigate further.
>
> It still fails with LC_ALL=C.
>
> # LC_ALL=C ./check throtl/004
> throtl/004 (nullb) (delete disk while IO is throttled) [passed]
> runtime 1.250s ... 1.211s
> throtl/004 (sdebug) (delete disk while IO is throttled) [failed]
> runtime 2.518s ... 2.271s
> --- tests/throtl/004.out 2026-03-20 14:25:50.478000000 +0900
> +++ /home/shin/Blktests/blktests/results/nodev_sdebug/throtl/004.out.bad 2026-05-21 13:46:36.676000000 +0900
> @@ -1,3 +1,2 @@
> Running throtl/004
> -Input/output error
> Test complete
>
What the patch intends to fix
tg_flush_bios() schedules pending_timer on the child tg's own service_queue.
For leaf cgroups, the child's pending_tree is empty, so the timer fires but
dispatches nothing. The throttled bio remains stuck in the parent's
pending_tree, but the parent's timer is never rescheduled. This causes tg
residual in the parent's rb tree and uncontrolled dispatch latency (~500ms
before the fix vs ~30ms after, in my testing).
Why throtl/004 fails for scsi_debug
The patch changes the dispatch timing: bios are now dispatched immediately
through the parent instead of waiting for the parent timer to fire naturally.
I looked at the SCSI device deletion flow (please correct me if I'm wrong):
__scsi_remove_device()
-> scsi_device_set_state(SDEV_CANCEL) // step 1
-> device_del() -> sd_remove() -> del_gendisk() -> __del_gendisk()
-> __blk_mark_disk_dead() // sets GD_DEAD
-> blk_throtl_cancel_bios() // schedules dispatch
-> scsi_device_set_state(SDEV_DEL) // step 2
-> blk_mq_destroy_queue()
Without the patch:
bios remain in the parent's pending_tree and are
dispatched only after the SCSI device has transitioned to SDEV_DEL (step 2).
scsi_device_state_check(SDEV_DEL) returns BLK_STS_IOERR (EIO).
With the patch:
bios are dispatched immediately and reach the SCSI layer
while the device is still in SDEV_CANCEL (between steps 1 and 2).
scsi_device_state_check(SDEV_CANCEL) falls into the default case, which
returns BLK_STS_OFFLINE (ENODEV).
The FULL output confirms this:
# nullb (passes)
dd: error writing '/dev/dev_nullb': Input/output error
# sdebug (fails)
dd: error writing '/dev/sda': No such device
Possible fix directions
- Revert and redesign:
revert and fix the original latency/residual issue differently.
- Fix SCSI layer:
make scsi_device_state_check() return BLK_STS_IOERR for SDEV_CANCEL
to match SDEV_DEL behavior. This requires SCSI maintainer approval.
- Fix blk-throttle cancel path:
have blk_throtl_cancel_bios() directly complete bios via bio_io_error()
instead of going through the SCSI submission path. This requires distinguishing
device deletion (fail with EIO) from cgroup removal (dispatch normally since the device is still alive).
I prefer the third approach. The semantics of blk_throtl_cancel_bios are precisely "cancel the bios",
and completing them directly with -EIO is more logical than detouring through the SCSI layer.
We could either add a parameter to tg_flush_bios() or introduce a new dedicated function to handle the device removal scenario.
Does anyone have other ideas?
Thanks,
Tao
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox