* [PATCH RFC 10/15] mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
The last user of gfpflags_allow_spinning() in slab is
alloc_from_pcs_bulk(), which is only called from
kmem_cache_alloc_bulk().
It turns out that gfpflags_allow_spinning() is not necessary, because
kmem_cache_alloc_bulk() is only expected to be called from context that
does allow spinning, so simply replace it with 'true'.
With that, we can remove the "@flags must allow spinning" part of the
kernel doc, as there is no more connection to the gfp flags in the slab
implementation.
Also remove a comment in alloc_slab_obj_exts() because there should be
no more false positives possible due to gfp_allowed_mask during early
boot.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index b511d768e9b6..dee69e0b7780 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2171,12 +2171,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
sz = obj_exts_alloc_size(s, slab, gfp);
- /*
- * Note that allow_spin may be false during early boot and its
- * restricted GFP_BOOT_MASK. Due to kmalloc_nolock() only supporting
- * architectures with cmpxchg16b, early obj_exts will be missing for
- * very early allocations on those.
- */
if (unlikely(!allow_spin))
vec = kmalloc_nolock(sz, __GFP_ZERO | __GFP_NO_OBJ_EXT,
slab_nid(slab));
@@ -4851,7 +4845,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
}
full = barn_replace_empty_sheaf(barn, pcs->main,
- gfpflags_allow_spinning(gfp));
+ /* allow_spin = */ true);
if (full) {
stat(s, BARN_GET);
@@ -7317,8 +7311,7 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
* Allocate @size objects from @s and places them into @p. @size must be larger
* than 0.
*
- * Interrupts must be enabled when calling this function and @flags must allow
- * spinning.
+ * Interrupts must be enabled when calling this function.
*
* Unlike alloc_pages_bulk(), this function does not check for already allocated
* objects in @p, and thus the caller does not need to zero it.
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 09/15] mm/slab: replace slab_alloc_node() parameters with slab_alloc_context
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
The function takes all the parameters that exist as fields in
slab_alloc_context, except alloc_flags. Replace them with a single
pointer.
This moves slab_alloc_context initialization to a number of callers,
which is more verbose, but arguably also more clear than a long list of
parameters, and most do not use the 'lru' field.
This will also allow kmalloc_nolock() to call slab_alloc_node() and
reduce the special open-coding it currently has.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 75 ++++++++++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 53 insertions(+), 22 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 034f2cd1c1fd..b511d768e9b6 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4905,30 +4905,23 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
*
* Otherwise we can simply pick the next object from the lockless free list.
*/
-static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
- gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s,
+ gfp_t gfpflags, int node, struct slab_alloc_context *ac)
{
- const unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
void *object;
- struct slab_alloc_context ac = {
- .caller_addr = addr,
- .orig_size = orig_size,
- .alloc_flags = alloc_flags,
- .lru = lru,
- };
s = slab_pre_alloc_hook(s, gfpflags);
if (unlikely(!s))
return NULL;
- object = kfence_alloc(s, orig_size, gfpflags);
+ object = kfence_alloc(s, ac->orig_size, gfpflags);
if (unlikely(object))
goto out;
- object = alloc_from_pcs(s, gfpflags, alloc_flags, node);
+ object = alloc_from_pcs(s, gfpflags, ac->alloc_flags, node);
if (!object)
- object = __slab_alloc_node(s, gfpflags, node, &ac);
+ object = __slab_alloc_node(s, gfpflags, node, ac);
maybe_wipe_obj_freeptr(s, object);
@@ -4937,15 +4930,21 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
* In case this fails due to memcg_slab_post_alloc_hook(),
* object is set to NULL
*/
- slab_post_alloc_hook(s, gfpflags, 1, &object, &ac);
+ slab_post_alloc_hook(s, gfpflags, 1, &object, ac);
return object;
}
void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags)
{
- void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_,
- s->object_size);
+ void *ret;
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = s->object_size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
+
+ ret = slab_alloc_node(s, gfpflags, NUMA_NO_NODE, &ac);
trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
@@ -4956,8 +4955,15 @@ EXPORT_SYMBOL(kmem_cache_alloc_noprof);
void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru,
gfp_t gfpflags)
{
- void *ret = slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _RET_IP_,
- s->object_size);
+ void *ret;
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = s->object_size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ .lru = lru,
+ };
+
+ ret = slab_alloc_node(s, gfpflags, NUMA_NO_NODE, &ac);
trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
@@ -4989,7 +4995,14 @@ EXPORT_SYMBOL(kmem_cache_charge);
*/
void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node)
{
- void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size);
+ void *ret;
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = s->object_size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
+
+ ret = slab_alloc_node(s, gfpflags, node, &ac);
trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, node);
@@ -5319,6 +5332,11 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
{
struct kmem_cache *s;
void *ret;
+ struct slab_alloc_context ac = {
+ .caller_addr = caller,
+ .orig_size = size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
ret = __kmalloc_large_node_noprof(size, flags, node);
@@ -5332,7 +5350,7 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
s = kmalloc_slab(size, b, flags, token);
- ret = slab_alloc_node(s, NULL, flags, node, caller, size);
+ ret = slab_alloc_node(s, flags, node, &ac);
ret = kasan_kmalloc(s, ret, size, flags);
trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
@@ -5451,8 +5469,14 @@ EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size)
{
- void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE,
- _RET_IP_, size);
+ void *ret;
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
+
+ ret = slab_alloc_node(s, gfpflags, NUMA_NO_NODE, &ac);
trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, NUMA_NO_NODE);
@@ -5464,7 +5488,14 @@ EXPORT_SYMBOL(__kmalloc_cache_noprof);
void *__kmalloc_cache_node_noprof(struct kmem_cache *s, gfp_t gfpflags,
int node, size_t size)
{
- void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size);
+ void *ret;
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
+
+ ret = slab_alloc_node(s, gfpflags, node, &ac);
trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, node);
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 08/15] mm/slab: pass alloc_flags through slab_post_alloc_hook() chain
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
Convert the whole following call stack to pass either slab_alloc_context
(thus including alloc_flags) or just alloc_flags as necessary:
slab_post_alloc_hook()
alloc_tagging_slab_alloc_hook()
__alloc_tagging_slab_alloc_hook()
prepare_slab_obj_exts_hook()
alloc_slab_obj_exts()
memcg_slab_post_alloc_hook()
__memcg_slab_post_alloc_hook()
alloc_slab_obj_exts()
Converting all these at once avoids unnecessary churn and is mostly
mechanical.
This ultimately allows to decide if spinning is allowed using
alloc_flags in alloc_slab_obj_exts(), as well as slab_post_alloc_hook().
Aside from alloc_from_pcs_bulk() (to be handled next) there is nothing
else in slab itself relying on gfpflags_allow_spinning() which can
be false even if not called from kmalloc_nolock().
A followup change will also use the alloc_flags availability in the call
stack above to remove the __GFP_NO_OBJ_EXT flag.
For alloc_slab_obj_exts(), also replace the suboptimal "bool new_slab"
parameter with a SLAB_ALLOC_NEW_SLAB flag with identical functionality.
To further reduce the number of parameters of slab_post_alloc_hook(),
also make 'struct list_lru *lru' (which is NULL for most callers) a new
field of slab_alloc_context.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/memcontrol.c | 5 +--
mm/slab.h | 6 ++--
mm/slub.c | 94 +++++++++++++++++++++++++++++++++------------------------
3 files changed, 62 insertions(+), 43 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d466..29390ba13baa 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3424,7 +3424,8 @@ static inline size_t obj_full_size(struct kmem_cache *s)
}
bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
- gfp_t flags, size_t size, void **p)
+ gfp_t flags, unsigned int slab_alloc_flags,
+ size_t size, void **p)
{
size_t obj_size = obj_full_size(s);
struct obj_cgroup *objcg;
@@ -3472,7 +3473,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
slab = virt_to_slab(p[i]);
if (!slab_obj_exts(slab) &&
- alloc_slab_obj_exts(slab, s, flags, false)) {
+ alloc_slab_obj_exts(slab, s, flags, slab_alloc_flags)) {
continue;
}
diff --git a/mm/slab.h b/mm/slab.h
index 3e75182ee144..13517abcad21 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -19,6 +19,7 @@
/* slab's alloc_flags definitions */
#define SLAB_ALLOC_DEFAULT 0x00
#define SLAB_ALLOC_TRYLOCK 0x01
+#define SLAB_ALLOC_NEW_SLAB 0x02 /* a flag for alloc_slab_obj_exts() */
static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
{
@@ -612,7 +613,7 @@ static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
}
int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
- gfp_t gfp, bool new_slab);
+ gfp_t gfp, unsigned int alloc_flags);
#else /* CONFIG_SLAB_OBJ_EXT */
@@ -642,7 +643,8 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s)
#ifdef CONFIG_MEMCG
bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
- gfp_t flags, size_t size, void **p);
+ gfp_t flags, unsigned int slab_alloc_flags,
+ size_t size, void **p);
void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
void **p, int objects, unsigned long obj_exts);
#endif
diff --git a/mm/slub.c b/mm/slub.c
index 20df6b131f63..034f2cd1c1fd 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -218,6 +218,7 @@ struct slab_alloc_context {
unsigned long caller_addr;
unsigned long orig_size;
unsigned int alloc_flags;
+ struct list_lru *lru;
};
/* Structure holding parameters for get_partial_node_bulk() */
@@ -2155,9 +2156,9 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,
}
int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
- gfp_t gfp, bool new_slab)
+ gfp_t gfp, unsigned int alloc_flags)
{
- bool allow_spin = gfpflags_allow_spinning(gfp);
+ const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
unsigned int objects = objs_per_slab(s, slab);
unsigned long new_exts;
unsigned long old_exts;
@@ -2206,7 +2207,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
old_exts = READ_ONCE(slab->obj_exts);
handle_failed_objexts_alloc(old_exts, vec, objects);
- if (new_slab) {
+ if (alloc_flags & SLAB_ALLOC_NEW_SLAB) {
/*
* If the slab is brand new and nobody can yet access its
* obj_exts, no synchronization is required and obj_exts can
@@ -2331,7 +2332,7 @@ static inline void init_slab_obj_exts(struct slab *slab)
}
static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
- gfp_t gfp, bool new_slab)
+ gfp_t gfp, unsigned int alloc_flags)
{
return 0;
}
@@ -2351,10 +2352,10 @@ static inline void alloc_slab_obj_exts_early(struct kmem_cache *s,
static inline unsigned long
prepare_slab_obj_exts_hook(struct kmem_cache *s, struct slab *slab,
- gfp_t flags, void *p)
+ gfp_t flags, unsigned int alloc_flags, void *p)
{
if (!slab_obj_exts(slab) &&
- alloc_slab_obj_exts(slab, s, flags, false)) {
+ alloc_slab_obj_exts(slab, s, flags, alloc_flags)) {
pr_warn_once("%s, %s: Failed to create slab extension vector!\n",
__func__, s->name);
return 0;
@@ -2366,7 +2367,8 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, struct slab *slab,
/* Should be called only if mem_alloc_profiling_enabled() */
static noinline void
-__alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags)
+__alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
+ unsigned int alloc_flags)
{
unsigned long obj_exts;
struct slabobj_ext *obj_ext;
@@ -2382,7 +2384,7 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags)
return;
slab = virt_to_slab(object);
- obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, object);
+ obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
/*
* Currently obj_exts is used only for allocation profiling.
* If other users appear then mem_alloc_profiling_enabled()
@@ -2401,10 +2403,11 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags)
}
static inline void
-alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags)
+alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
+ unsigned int alloc_flags)
{
if (mem_alloc_profiling_enabled())
- __alloc_tagging_slab_alloc_hook(s, object, flags);
+ __alloc_tagging_slab_alloc_hook(s, object, flags, alloc_flags);
}
/* Should be called only if mem_alloc_profiling_enabled() */
@@ -2443,7 +2446,8 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
#else /* CONFIG_MEM_ALLOC_PROFILING */
static inline void
-alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags)
+alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
+ unsigned int alloc_flags)
{
}
@@ -2461,8 +2465,9 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
static void memcg_alloc_abort_single(struct kmem_cache *s, void *object);
static __fastpath_inline
-bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
- gfp_t flags, size_t size, void **p)
+bool memcg_slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags,
+ size_t size, void **p,
+ struct slab_alloc_context *ac)
{
if (likely(!memcg_kmem_online()))
return true;
@@ -2470,7 +2475,8 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
if (likely(!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT)))
return true;
- if (likely(__memcg_slab_post_alloc_hook(s, lru, flags, size, p)))
+ if (likely(__memcg_slab_post_alloc_hook(s, ac->lru, flags,
+ ac->alloc_flags, size, p)))
return true;
if (likely(size == 1)) {
@@ -2558,14 +2564,15 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
put_slab_obj_exts(obj_exts);
}
- return __memcg_slab_post_alloc_hook(s, NULL, flags, 1, &p);
+ return __memcg_slab_post_alloc_hook(s, NULL, flags, SLAB_ALLOC_DEFAULT,
+ 1, &p);
}
#else /* CONFIG_MEMCG */
static inline bool memcg_slab_post_alloc_hook(struct kmem_cache *s,
- struct list_lru *lru,
- gfp_t flags, size_t size,
- void **p)
+ gfp_t flags,
+ size_t size, void **p,
+ struct slab_alloc_context *ac)
{
return true;
}
@@ -3352,12 +3359,14 @@ static inline void init_freelist_randomization(void) { }
#endif /* CONFIG_SLAB_FREELIST_RANDOM */
static __always_inline void account_slab(struct slab *slab, int order,
- struct kmem_cache *s, gfp_t gfp)
+ struct kmem_cache *s, gfp_t gfp,
+ unsigned int alloc_flags)
{
if (memcg_kmem_online() &&
(s->flags & SLAB_ACCOUNT) &&
!slab_obj_exts(slab))
- alloc_slab_obj_exts(slab, s, gfp, true);
+ alloc_slab_obj_exts(slab, s, gfp,
+ alloc_flags | SLAB_ALLOC_NEW_SLAB);
mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s),
PAGE_SIZE << order);
@@ -3434,7 +3443,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
* to prevent the array from being overwritten.
*/
alloc_slab_obj_exts_early(s, slab);
- account_slab(slab, oo_order(oo), s, flags);
+ account_slab(slab, oo_order(oo), s, flags, alloc_flags);
return slab;
}
@@ -4568,9 +4577,8 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags)
}
static __fastpath_inline
-bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
- gfp_t flags, size_t size, void **p,
- unsigned int orig_size)
+bool slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, size_t size,
+ void **p, struct slab_alloc_context *ac)
{
bool init = slab_want_init_on_alloc(flags, s);
bool kasan_init = init;
@@ -4599,15 +4607,15 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
p[i] = kasan_slab_alloc(s, p[i], init_flags, kasan_init);
if (p[i] && init && (!kasan_init ||
!kasan_has_integrated_init()))
- memset(p[i], 0, orig_size);
- if (gfpflags_allow_spinning(flags))
+ memset(p[i], 0, ac->orig_size);
+ if (alloc_flags_allow_spinning(ac->alloc_flags))
kmemleak_alloc_recursive(p[i], s->object_size, 1,
s->flags, init_flags);
kmsan_slab_alloc(s, p[i], init_flags);
- alloc_tagging_slab_alloc_hook(s, p[i], flags);
+ alloc_tagging_slab_alloc_hook(s, p[i], flags, ac->alloc_flags);
}
- return memcg_slab_post_alloc_hook(s, lru, flags, size, p);
+ return memcg_slab_post_alloc_hook(s, flags, size, p, ac);
}
/*
@@ -4902,6 +4910,12 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
{
const unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
void *object;
+ struct slab_alloc_context ac = {
+ .caller_addr = addr,
+ .orig_size = orig_size,
+ .alloc_flags = alloc_flags,
+ .lru = lru,
+ };
s = slab_pre_alloc_hook(s, gfpflags);
if (unlikely(!s))
@@ -4913,14 +4927,8 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
object = alloc_from_pcs(s, gfpflags, alloc_flags, node);
- if (unlikely(!object)) {
- struct slab_alloc_context ac = {
- .caller_addr = addr,
- .orig_size = orig_size,
- .alloc_flags = alloc_flags,
- };
+ if (!object)
object = __slab_alloc_node(s, gfpflags, node, &ac);
- }
maybe_wipe_obj_freeptr(s, object);
@@ -4929,7 +4937,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
* In case this fails due to memcg_slab_post_alloc_hook(),
* object is set to NULL
*/
- slab_post_alloc_hook(s, lru, gfpflags, 1, &object, orig_size);
+ slab_post_alloc_hook(s, gfpflags, 1, &object, &ac);
return object;
}
@@ -5224,6 +5232,10 @@ kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
struct slab_sheaf *sheaf)
{
void *ret = NULL;
+ struct slab_alloc_context ac = {
+ .orig_size = s->object_size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
if (sheaf->size == 0)
goto out;
@@ -5234,7 +5246,7 @@ kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
ret = sheaf->objects[--sheaf->size];
/* add __GFP_NOFAIL to force successful memcg charging */
- slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, s->object_size);
+ slab_post_alloc_hook(s, gfp | __GFP_NOFAIL, 1, &ret, &ac);
out:
trace_kmem_cache_alloc(_RET_IP_, ret, s, gfp, NUMA_NO_NODE);
@@ -5421,7 +5433,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
success:
maybe_wipe_obj_freeptr(s, ret);
- slab_post_alloc_hook(s, NULL, alloc_gfp, 1, &ret, orig_size);
+ slab_post_alloc_hook(s, alloc_gfp, 1, &ret, &ac);
ret = kasan_kmalloc(s, ret, orig_size, alloc_gfp);
return ret;
@@ -7287,6 +7299,10 @@ bool kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags,
{
unsigned int i = 0;
void *kfence_obj;
+ struct slab_alloc_context ac = {
+ .orig_size = s->object_size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
+ };
if (!size)
return false;
@@ -7337,7 +7353,7 @@ bool kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags,
out:
/* memcg and kmem_cache debug support and memory initialization */
- return likely(slab_post_alloc_hook(s, NULL, flags, size, p, s->object_size));
+ return likely(slab_post_alloc_hook(s, flags, size, p, &ac));
}
EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof);
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 07/15] mm/slab: pass alloc_flags to new slab allocation
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
Add the alloc_flags parameter to allocate_slab() and new_slab()
so it can be used to determine if spinning is allowed, independently
from gfp flags.
refill_objects() passes SLAB_ALLOC_DEFAULT because it can only be
reached from contexts that allow spinning.
Also change how trynode_flags are constructed in ___slab_alloc() to
achieve the same "do not upgrade to GFP_NOWAIT" by using masking instead
of a branch. It will now also not upgrade in cases where gfp is weaker
than GFP_NOWAIT (i.e. lacks __GFP_KSWAPD_RECLAIM) but doesn't come from
kmalloc_nolock() - which is more correct anyway.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 0bde4f6d9126..20df6b131f63 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3378,9 +3378,10 @@ static __always_inline void unaccount_slab(struct slab *slab, int order,
}
/* Allocate and initialize a slab without building its freelist. */
-static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
+static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
+ unsigned int alloc_flags, int node)
{
- bool allow_spin = gfpflags_allow_spinning(flags);
+ bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
struct slab *slab;
struct kmem_cache_order_objects oo = s->oo;
gfp_t alloc_gfp;
@@ -3438,15 +3439,17 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
return slab;
}
-static struct slab *new_slab(struct kmem_cache *s, gfp_t flags, int node)
+static struct slab *new_slab(struct kmem_cache *s, gfp_t flags,
+ unsigned int alloc_flags, int node)
{
if (unlikely(flags & GFP_SLAB_BUG_MASK))
flags = kmalloc_fix_flags(flags);
WARN_ON_ONCE(s->ctor && (flags & __GFP_ZERO));
- return allocate_slab(s,
- flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node);
+ flags &= GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK;
+
+ return allocate_slab(s, flags, alloc_flags, node);
}
static void __free_slab(struct kmem_cache *s, struct slab *slab, bool allow_spin)
@@ -4467,25 +4470,22 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
* 1) try to get a partial slab from target node only by having
* __GFP_THISNODE in pc.flags for get_from_partial()
* 2) if 1) failed, try to allocate a new slab from target node with
- * GPF_NOWAIT | __GFP_THISNODE opportunistically
+ * (at most) GPF_NOWAIT | __GFP_THISNODE opportunistically
* 3) if 2) failed, retry with original gfpflags which will allow
* get_from_partial() try partial lists of other nodes before
* potentially allocating new page from other nodes
*/
if (unlikely(node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
&& try_thisnode)) {
- if (unlikely(!allow_spin))
- /* Do not upgrade gfp to NOWAIT from more restrictive mode */
- trynode_flags = gfpflags | __GFP_THISNODE;
- else
- trynode_flags = GFP_NOWAIT | __GFP_THISNODE;
+ trynode_flags &= GFP_NOWAIT;
+ trynode_flags |= __GFP_NOWARN | __GFP_THISNODE;
}
object = get_from_partial(s, node, trynode_flags, ac);
if (object)
goto success;
- slab = new_slab(s, trynode_flags, node);
+ slab = new_slab(s, trynode_flags, ac->alloc_flags, node);
if (unlikely(!slab)) {
if (node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
@@ -7215,7 +7215,7 @@ refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
new_slab:
- slab = new_slab(s, gfp, local_node);
+ slab = new_slab(s, gfp, SLAB_ALLOC_DEFAULT, local_node);
if (!slab)
goto out;
@@ -7563,7 +7563,7 @@ static void early_kmem_cache_node_alloc(int node)
BUG_ON(kmem_cache_node->size < sizeof(struct kmem_cache_node));
- slab = new_slab(kmem_cache_node, GFP_NOWAIT, node);
+ slab = new_slab(kmem_cache_node, GFP_NOWAIT, SLAB_ALLOC_DEFAULT, node);
BUG_ON(!slab);
if (slab_nid(slab) != node) {
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 06/15] mm/slab: replace struct partial_context with slab_alloc_context
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
Refactor get_from_partial_node(), get_from_any_partial(),
get_from_partial() and ___slab_alloc().
Remove struct partial_context, which used to be more substantial but
shrank as part of the sheaves conversion. Instead pass gfp_flags and
pointer to the new slab_alloc_context, which together is a superset of
partial_context.
This means alloc_flags are now available and we can use them to
determine if spinning is allowed, further reducing false positive "not
allowed" in the slow path due to gfp flags lacking __GFP_RECLAIM.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 52 ++++++++++++++++++++++++----------------------------
1 file changed, 24 insertions(+), 28 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index b2a452dd70fa..0bde4f6d9126 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -220,12 +220,6 @@ struct slab_alloc_context {
unsigned int alloc_flags;
};
-/* Structure holding parameters for get_from_partial() call chain */
-struct partial_context {
- gfp_t flags;
- unsigned int orig_size;
-};
-
/* Structure holding parameters for get_partial_node_bulk() */
struct partial_bulk_context {
gfp_t flags;
@@ -3826,7 +3820,8 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
*/
static void *get_from_partial_node(struct kmem_cache *s,
struct kmem_cache_node *n,
- struct partial_context *pc)
+ gfp_t gfp_flags,
+ struct slab_alloc_context *ac)
{
struct slab *slab, *slab2;
unsigned long flags;
@@ -3841,7 +3836,7 @@ static void *get_from_partial_node(struct kmem_cache *s,
if (!n || !n->nr_partial)
return NULL;
- if (gfpflags_allow_spinning(pc->flags))
+ if (alloc_flags_allow_spinning(ac->alloc_flags))
spin_lock_irqsave(&n->list_lock, flags);
else if (!spin_trylock_irqsave(&n->list_lock, flags))
return NULL;
@@ -3849,12 +3844,12 @@ static void *get_from_partial_node(struct kmem_cache *s,
struct freelist_counters old, new;
- if (!pfmemalloc_match(slab, pc->flags))
+ if (!pfmemalloc_match(slab, gfp_flags))
continue;
if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
object = alloc_single_from_partial(s, n, slab,
- pc->orig_size);
+ ac->orig_size);
if (object)
break;
continue;
@@ -3888,15 +3883,16 @@ static void *get_from_partial_node(struct kmem_cache *s,
/*
* Get an object from somewhere. Search in increasing NUMA distances.
*/
-static void *get_from_any_partial(struct kmem_cache *s, struct partial_context *pc)
+static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
+ struct slab_alloc_context *ac)
{
#ifdef CONFIG_NUMA
struct zonelist *zonelist;
struct zoneref *z;
struct zone *zone;
- enum zone_type highest_zoneidx = gfp_zone(pc->flags);
+ enum zone_type highest_zoneidx = gfp_zone(gfp_flags);
unsigned int cpuset_mems_cookie;
- bool allow_spin = gfpflags_allow_spinning(pc->flags);
+ bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
/*
* The defrag ratio allows a configuration of the tradeoffs between
@@ -3930,16 +3926,17 @@ static void *get_from_any_partial(struct kmem_cache *s, struct partial_context *
if (allow_spin)
cpuset_mems_cookie = read_mems_allowed_begin();
- zonelist = node_zonelist(mempolicy_slab_node(), pc->flags);
+ zonelist = node_zonelist(mempolicy_slab_node(), gfp_flags);
for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) {
struct kmem_cache_node *n;
n = get_node(s, zone_to_nid(zone));
- if (n && cpuset_zone_allowed(zone, pc->flags) &&
+ if (n && cpuset_zone_allowed(zone, gfp_flags) &&
n->nr_partial > s->min_partial) {
- void *object = get_from_partial_node(s, n, pc);
+ void *object = get_from_partial_node(s, n,
+ gfp_flags, ac);
if (object) {
/*
@@ -3961,8 +3958,8 @@ static void *get_from_any_partial(struct kmem_cache *s, struct partial_context *
/*
* Get an object from a partial slab
*/
-static void *get_from_partial(struct kmem_cache *s, int node,
- struct partial_context *pc)
+static void *get_from_partial(struct kmem_cache *s, int node, gfp_t flags,
+ struct slab_alloc_context *ac)
{
int searchnode = node;
void *object;
@@ -3970,11 +3967,11 @@ static void *get_from_partial(struct kmem_cache *s, int node,
if (node == NUMA_NO_NODE)
searchnode = numa_mem_id();
- object = get_from_partial_node(s, get_node(s, searchnode), pc);
- if (object || (node != NUMA_NO_NODE && (pc->flags & __GFP_THISNODE)))
+ object = get_from_partial_node(s, get_node(s, searchnode), flags, ac);
+ if (object || (node != NUMA_NO_NODE && (flags & __GFP_THISNODE)))
return object;
- return get_from_any_partial(s, pc);
+ return get_from_any_partial(s, flags, ac);
}
static bool has_pcs_used(int cpu, struct kmem_cache *s)
@@ -4454,16 +4451,16 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
struct slab_alloc_context *ac)
{
bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
+ gfp_t trynode_flags;
void *object;
struct slab *slab;
- struct partial_context pc;
bool try_thisnode = true;
stat(s, ALLOC_SLOWPATH);
new_objects:
- pc.flags = gfpflags;
+ trynode_flags = gfpflags;
/*
* When a preferred node is indicated but no __GFP_THISNODE
*
@@ -4479,17 +4476,16 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
&& try_thisnode)) {
if (unlikely(!allow_spin))
/* Do not upgrade gfp to NOWAIT from more restrictive mode */
- pc.flags = gfpflags | __GFP_THISNODE;
+ trynode_flags = gfpflags | __GFP_THISNODE;
else
- pc.flags = GFP_NOWAIT | __GFP_THISNODE;
+ trynode_flags = GFP_NOWAIT | __GFP_THISNODE;
}
- pc.orig_size = ac->orig_size;
- object = get_from_partial(s, node, &pc);
+ object = get_from_partial(s, node, trynode_flags, ac);
if (object)
goto success;
- slab = new_slab(s, pc.flags, node);
+ slab = new_slab(s, trynode_flags, node);
if (unlikely(!slab)) {
if (node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 05/15] mm/slab: add alloc_flags to slab_alloc_context
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
Add alloc_flags as a new field to the slab_alloc_context helper struct,
so we can pass it to more functions in the slab implementation without
adding another function parameter.
Start checking them via alloc_flags_allow_spinning() in
alloc_single_from_new_slab() (where we can drop the allow_spin
parameter) and ___slab_alloc(). This further reduces false-positive
spinning-not-allowed from allocations that are not kmalloc_nolock() but
lack __GFP_RECLAIM flags.
_kmalloc_nolock_noprof() initializes ac.alloc_flags using its flags that
are SLAB_ALLOC_TRYLOCK. slab_alloc_node() and __kmem_cache_alloc_bulk()
are not reachable from kmalloc_nolock() and all their callers expect
spinning to be allowed, so they can use SLAB_ALLOC_DEFAULT. This is
temporary as the scope of slab_alloc_context will further move to the
callers, making the alloc_flags usage more obvious.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 278d8cbcc7ee..b2a452dd70fa 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -217,6 +217,7 @@ static DEFINE_STATIC_KEY_FALSE(strict_numa);
struct slab_alloc_context {
unsigned long caller_addr;
unsigned long orig_size;
+ unsigned int alloc_flags;
};
/* Structure holding parameters for get_from_partial() call chain */
@@ -3693,9 +3694,9 @@ static inline void init_slab_obj_iter(struct kmem_cache *s, struct slab *slab,
* and put the slab to the partial (or full) list.
*/
static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
- struct slab_alloc_context *ac,
- bool allow_spin)
+ struct slab_alloc_context *ac)
{
+ bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
struct kmem_cache_node *n;
struct slab_obj_iter iter;
bool needs_add_partial;
@@ -4452,7 +4453,7 @@ static unsigned int alloc_from_new_slab(struct kmem_cache *s, struct slab *slab,
static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
struct slab_alloc_context *ac)
{
- bool allow_spin = gfpflags_allow_spinning(gfpflags);
+ bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
void *object;
struct slab *slab;
struct partial_context pc;
@@ -4503,7 +4504,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
stat(s, ALLOC_SLAB);
if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
- object = alloc_single_from_new_slab(s, slab, ac, allow_spin);
+ object = alloc_single_from_new_slab(s, slab, ac);
if (likely(object))
goto success;
@@ -4903,6 +4904,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
{
+ const unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
void *object;
s = slab_pre_alloc_hook(s, gfpflags);
@@ -4913,12 +4915,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
if (unlikely(object))
goto out;
- object = alloc_from_pcs(s, gfpflags, SLAB_ALLOC_DEFAULT, node);
+ object = alloc_from_pcs(s, gfpflags, alloc_flags, node);
if (unlikely(!object)) {
struct slab_alloc_context ac = {
.caller_addr = addr,
.orig_size = orig_size,
+ .alloc_flags = alloc_flags,
};
object = __slab_alloc_node(s, gfpflags, node, &ac);
}
@@ -5390,6 +5393,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
struct slab_alloc_context ac = {
.caller_addr = _RET_IP_,
.orig_size = orig_size,
+ .alloc_flags = alloc_flags,
};
/*
@@ -7240,6 +7244,7 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
struct slab_alloc_context ac = {
.caller_addr = _RET_IP_,
.orig_size = s->object_size,
+ .alloc_flags = SLAB_ALLOC_DEFAULT,
};
for (i = 0; i < size; i++) {
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 04/15] mm/slab: introduce alloc_flags and SLAB_ALLOC_TRYLOCK
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
Similarly to the page allocators, introduce slab-allocator specific
alloc flags that internally control allocation behavior in addition to
gfp_flags, without occupying the limited gfp flags space.
Introduce the first flag SLAB_ALLOC_TRYLOCK that behaves similarly to
page allocator's ALLOC_TRYLOCK and will be used to reimplement
kmalloc_nolock()'s "!allow_spin" behavior. That currently relies on
gfpflags_allow_spinning() and thus the lack of both __GFP_RECLAIM flags,
importantly __GFP_KSWAPD_RECLAIM. This can give false-positive results
e.g. in early boot with a restricted gfp_allowed_mask.
Also introduce alloc_flags_allow_spinning() to replace the usage of
gfpflags_allow_spinning().
Start using alloc_flags and the new check first in alloc_from_pcs() and
__pcs_replace_empty_main(). This means some slab allocations that were
falsely treated as kmalloc_nolock() due to their gfp flags will now have
higher chances of succeed, and this will further increase with followup
changes.
Remove a WARN_ON_ONCE() from refill_objects() as it's now legitimate to
reach it from a slab allocation that's not _nolock() and yet lacks
__GFP_KSWAPD_RECLAIM for other reasons.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slab.h | 9 +++++++++
mm/slub.c | 17 ++++++++---------
2 files changed, 17 insertions(+), 9 deletions(-)
diff --git a/mm/slab.h b/mm/slab.h
index 1bf9c3021ae3..3e75182ee144 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -16,6 +16,15 @@
* Internal slab definitions
*/
+/* slab's alloc_flags definitions */
+#define SLAB_ALLOC_DEFAULT 0x00
+#define SLAB_ALLOC_TRYLOCK 0x01
+
+static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
+{
+ return !(alloc_flags & SLAB_ALLOC_TRYLOCK);
+}
+
#ifdef CONFIG_64BIT
# ifdef system_has_cmpxchg128
# define system_has_freelist_aba() system_has_cmpxchg128()
diff --git a/mm/slub.c b/mm/slub.c
index 06fc1656080f..278d8cbcc7ee 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4622,7 +4622,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
* unlocked.
*/
static struct slub_percpu_sheaves *
-__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
+__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
+ gfp_t gfp, unsigned int alloc_flags)
{
struct slab_sheaf *empty = NULL;
struct slab_sheaf *full;
@@ -4648,7 +4649,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
return NULL;
}
- allow_spin = gfpflags_allow_spinning(gfp);
+ allow_spin = alloc_flags_allow_spinning(alloc_flags);
full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
@@ -4734,7 +4735,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
}
static __fastpath_inline
-void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
+void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags, int node)
{
struct slub_percpu_sheaves *pcs;
bool node_requested;
@@ -4779,7 +4780,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
pcs = this_cpu_ptr(s->cpu_sheaves);
if (unlikely(pcs->main->size == 0)) {
- pcs = __pcs_replace_empty_main(s, pcs, gfp);
+ pcs = __pcs_replace_empty_main(s, pcs, gfp, alloc_flags);
if (unlikely(!pcs))
return NULL;
}
@@ -4912,7 +4913,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
if (unlikely(object))
goto out;
- object = alloc_from_pcs(s, gfpflags, node);
+ object = alloc_from_pcs(s, gfpflags, SLAB_ALLOC_DEFAULT, node);
if (unlikely(!object)) {
struct slab_alloc_context ac = {
@@ -5343,6 +5344,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
size_t orig_size = size;
+ unsigned int alloc_flags = SLAB_ALLOC_TRYLOCK;
struct kmem_cache *s;
bool can_retry = true;
void *ret;
@@ -5381,7 +5383,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
*/
return NULL;
- ret = alloc_from_pcs(s, alloc_gfp, node);
+ ret = alloc_from_pcs(s, alloc_gfp, alloc_flags, node);
if (ret)
goto success;
@@ -7200,9 +7202,6 @@ refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
unsigned int refilled;
struct slab *slab;
- if (WARN_ON_ONCE(!gfpflags_allow_spinning(gfp)))
- return 0;
-
refilled = __refill_objects_node(s, p, gfp, min, max,
get_node(s, local_node),
/* allow_spin = */ true);
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 03/15] mm/slab: introduce slab_alloc_context
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
Similarly to page allocator's struct alloc_context, introduce a helper
struct to hold a part of the allocation arguments. This will allow
reducing the number of parameters in many functions of the
implementation, and extend them easily if needed.
For now, make it hold the caller address and the originally requested
allocation size.
Convert alloc_single_from_new_slab(), __slab_alloc_node() and
___slab_alloc(). No functional change intended.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 46 +++++++++++++++++++++++++++++++++-------------
1 file changed, 33 insertions(+), 13 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index af85f338db4f..06fc1656080f 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -213,6 +213,12 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled);
static DEFINE_STATIC_KEY_FALSE(strict_numa);
#endif
+/* Structure holding extra parameters for slab allocations */
+struct slab_alloc_context {
+ unsigned long caller_addr;
+ unsigned long orig_size;
+};
+
/* Structure holding parameters for get_from_partial() call chain */
struct partial_context {
gfp_t flags;
@@ -3687,7 +3693,8 @@ static inline void init_slab_obj_iter(struct kmem_cache *s, struct slab *slab,
* and put the slab to the partial (or full) list.
*/
static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
- int orig_size, bool allow_spin)
+ struct slab_alloc_context *ac,
+ bool allow_spin)
{
struct kmem_cache_node *n;
struct slab_obj_iter iter;
@@ -3705,7 +3712,7 @@ static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
/* alloc_debug_processing() always expects a valid freepointer */
set_freepointer(s, object, slab->freelist);
- if (!alloc_debug_processing(s, slab, object, orig_size)) {
+ if (!alloc_debug_processing(s, slab, object, ac->orig_size)) {
/*
* It's not really expected that this would fail on a
* freshly allocated slab, but a concurrent memory
@@ -4443,7 +4450,7 @@ static unsigned int alloc_from_new_slab(struct kmem_cache *s, struct slab *slab,
* slab.
*/
static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
- unsigned long addr, unsigned int orig_size)
+ struct slab_alloc_context *ac)
{
bool allow_spin = gfpflags_allow_spinning(gfpflags);
void *object;
@@ -4476,7 +4483,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
pc.flags = GFP_NOWAIT | __GFP_THISNODE;
}
- pc.orig_size = orig_size;
+ pc.orig_size = ac->orig_size;
object = get_from_partial(s, node, &pc);
if (object)
goto success;
@@ -4496,7 +4503,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
stat(s, ALLOC_SLAB);
if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
- object = alloc_single_from_new_slab(s, slab, orig_size, allow_spin);
+ object = alloc_single_from_new_slab(s, slab, ac, allow_spin);
if (likely(object))
goto success;
@@ -4514,13 +4521,13 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
success:
if (kmem_cache_debug_flags(s, SLAB_STORE_USER))
- set_track(s, object, TRACK_ALLOC, addr, gfpflags);
+ set_track(s, object, TRACK_ALLOC, ac->caller_addr, gfpflags);
return object;
}
static void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node,
- unsigned long addr, size_t orig_size)
+ struct slab_alloc_context *ac)
{
void *object;
@@ -4545,7 +4552,7 @@ static void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node,
}
#endif
- object = ___slab_alloc(s, gfpflags, node, addr, orig_size);
+ object = ___slab_alloc(s, gfpflags, node, ac);
return object;
}
@@ -4907,8 +4914,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
object = alloc_from_pcs(s, gfpflags, node);
- if (unlikely(!object))
- object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
+ if (unlikely(!object)) {
+ struct slab_alloc_context ac = {
+ .caller_addr = addr,
+ .orig_size = orig_size,
+ };
+ object = __slab_alloc_node(s, gfpflags, node, &ac);
+ }
maybe_wipe_obj_freeptr(s, object);
@@ -5373,13 +5385,18 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
if (ret)
goto success;
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = orig_size,
+ };
+
/*
* Do not call slab_alloc_node(), since trylock mode isn't
* compatible with slab_pre_alloc_hook/should_failslab and
* kfence_alloc. Hence call __slab_alloc_node() (at most twice)
* and slab_post_alloc_hook() directly.
*/
- ret = __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, orig_size);
+ ret = __slab_alloc_node(s, alloc_gfp, node, &ac);
/*
* It's possible we failed due to trylock as we preempted someone with
@@ -7221,10 +7238,13 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
int i;
if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
+ struct slab_alloc_context ac = {
+ .caller_addr = _RET_IP_,
+ .orig_size = s->object_size,
+ };
for (i = 0; i < size; i++) {
- p[i] = ___slab_alloc(s, flags, NUMA_NO_NODE, _RET_IP_,
- s->object_size);
+ p[i] = ___slab_alloc(s, flags, NUMA_NO_NODE, &ac);
if (unlikely(!p[i]))
goto error;
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 02/15] mm/slab: stop inlining __slab_alloc_node()
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
With sheaves, this is no longer part of the allocation fastpath. For
the same reason, also mark the call to it from slab_alloc_node() as
unlikely().
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index f787dc422d1b..af85f338db4f 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4519,8 +4519,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
return object;
}
-static __always_inline void *__slab_alloc_node(struct kmem_cache *s,
- gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+static void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node,
+ unsigned long addr, size_t orig_size)
{
void *object;
@@ -4907,7 +4907,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
object = alloc_from_pcs(s, gfpflags, node);
- if (!object)
+ if (unlikely(!object))
object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
maybe_wipe_obj_freeptr(s, object);
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 01/15] mm/slab: always zero only requested size on alloc
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
In-Reply-To: <20260609-slab_alloc_flags-v1-0-2bf4a4b9b526@kernel.org>
When zeroing on alloc is requested (by __GFP_ZERO or the init_on_alloc
parameter), we have been trying to zero the whole kmalloc bucket size
and not just requested size, if possible.
This probably comes from the past where ksize() could be used to
discover the bucket size and use it opportunistically beyond the
requested size. This is now forbidden and enabling debugging such as
KASAN or slab's red zoning would catch this misuse. Therefore, nobody
can be relying on __GFP_ZERO zeroing beyond requested size.
Theoretically it might still improve hardening in case of unintended
accesses beond requested size accessing some sensitive data from a
previous allocation. But then, init_on_free is probably used also for
hardening and would have cleared that.
So the usefullness of zeroing beyond requested size is practically none
nowadays. The disadvantages for doing it are:
- Interaction with KFENCE, which perfoms the zeroing on its own because
it has its own redzone beyond requested size. As a consequence
slab_post_alloc_hook() has an 'init' parameter which has to be
evaluated in all callers (via slab_want_init_on_alloc()).
For kfence allocations in slab_alloc_node() this evaluation is subtly
skipped over in order to do the right thing. Other callers (i.e.
kmem_cache_alloc_bulk_noprof()) evaluate it unconditionally even if
they do end up with a kfence allocation. This is only subtly not a
problem, as those are not kmalloc allocations and are using
s->object_size as requested size, so it doesn't interfere with kfence's
redzone. There's just a unnecessary double zeroing (in both kfence and
slab_post_alloc_hook()), but it's all very fragile and contradicts the
comment in kfence_guarded_alloc().
- Interaction with slab's redzoning where we have to limit the zeroing
to requested size.
We can make the code much more simple by always zeroing only up to the
requested size. Move slab_want_init_on_alloc() call to
slab_post_alloc_hook(), removing the parameter. Remove the red zone
handling.
For kfence's zeroing code, update the comment. We could remove it
completely, but due to possible interactions with KASAN, there are
configurations where neither slab or KASAN would zero the object,
so simply do it in kfence. At worst the zeroing will happen twice, but
kfence allocations are rare by design so the cost is negligible.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/kfence/core.c | 6 +++---
mm/slub.c | 35 +++++++----------------------------
2 files changed, 10 insertions(+), 31 deletions(-)
diff --git a/mm/kfence/core.c b/mm/kfence/core.c
index 655dc5ce3240..c765ba0a3a67 100644
--- a/mm/kfence/core.c
+++ b/mm/kfence/core.c
@@ -499,9 +499,9 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g
set_canary(meta);
/*
- * We check slab_want_init_on_alloc() ourselves, rather than letting
- * SL*B do the initialization, as otherwise we might overwrite KFENCE's
- * redzone.
+ * SLUB will generally init kfence objects, but due to possible
+ * interactions with KASAN, it might not happen, so do it ourselves.
+ * In the worst case the init just happens twice.
*/
if (unlikely(slab_want_init_on_alloc(gfp, cache)))
memzero_explicit(addr, size);
diff --git a/mm/slub.c b/mm/slub.c
index 63c1ef998dd3..f787dc422d1b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4565,26 +4565,14 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags)
static __fastpath_inline
bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
- gfp_t flags, size_t size, void **p, bool init,
+ gfp_t flags, size_t size, void **p,
unsigned int orig_size)
{
- unsigned int zero_size = s->object_size;
+ bool init = slab_want_init_on_alloc(flags, s);
bool kasan_init = init;
size_t i;
gfp_t init_flags = flags & gfp_allowed_mask;
- /*
- * For kmalloc object, the allocated memory size(object_size) is likely
- * larger than the requested size(orig_size). If redzone check is
- * enabled for the extra space, don't zero it, as it will be redzoned
- * soon. The redzone operation for this extra space could be seen as a
- * replacement of current poisoning under certain debug option, and
- * won't break other sanity checks.
- */
- if (kmem_cache_debug_flags(s, SLAB_STORE_USER | SLAB_RED_ZONE) &&
- (s->flags & SLAB_KMALLOC))
- zero_size = orig_size;
-
/*
* When slab_debug is enabled, avoid memory initialization integrated
* into KASAN and instead zero out the memory via the memset below with
@@ -4607,7 +4595,7 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
p[i] = kasan_slab_alloc(s, p[i], init_flags, kasan_init);
if (p[i] && init && (!kasan_init ||
!kasan_has_integrated_init()))
- memset(p[i], 0, zero_size);
+ memset(p[i], 0, orig_size);
if (gfpflags_allow_spinning(flags))
kmemleak_alloc_recursive(p[i], s->object_size, 1,
s->flags, init_flags);
@@ -4908,7 +4896,6 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
{
void *object;
- bool init = false;
s = slab_pre_alloc_hook(s, gfpflags);
if (unlikely(!s))
@@ -4924,16 +4911,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
maybe_wipe_obj_freeptr(s, object);
- init = slab_want_init_on_alloc(gfpflags, s);
out:
/*
- * When init equals 'true', like for kzalloc() family, only
- * @orig_size bytes might be zeroed instead of s->object_size
* In case this fails due to memcg_slab_post_alloc_hook(),
* object is set to NULL
*/
- slab_post_alloc_hook(s, lru, gfpflags, 1, &object, init, orig_size);
+ slab_post_alloc_hook(s, lru, gfpflags, 1, &object, orig_size);
return object;
}
@@ -5228,7 +5212,6 @@ kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
struct slab_sheaf *sheaf)
{
void *ret = NULL;
- bool init;
if (sheaf->size == 0)
goto out;
@@ -5238,10 +5221,8 @@ kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
if (likely(!ret))
ret = sheaf->objects[--sheaf->size];
- init = slab_want_init_on_alloc(gfp, s);
-
/* add __GFP_NOFAIL to force successful memcg charging */
- slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, init, s->object_size);
+ slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, s->object_size);
out:
trace_kmem_cache_alloc(_RET_IP_, ret, s, gfp, NUMA_NO_NODE);
@@ -5421,8 +5402,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
success:
maybe_wipe_obj_freeptr(s, ret);
- slab_post_alloc_hook(s, NULL, alloc_gfp, 1, &ret,
- slab_want_init_on_alloc(alloc_gfp, s), orig_size);
+ slab_post_alloc_hook(s, NULL, alloc_gfp, 1, &ret, orig_size);
ret = kasan_kmalloc(s, ret, orig_size, alloc_gfp);
return ret;
@@ -7337,8 +7317,7 @@ bool kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags,
out:
/* memcg and kmem_cache debug support and memory initialization */
- return likely(slab_post_alloc_hook(s, NULL, flags, size, p,
- slab_want_init_on_alloc(flags, s), s->object_size));
+ return likely(slab_post_alloc_hook(s, NULL, flags, size, p, s->object_size));
}
EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof);
--
2.54.0
^ permalink raw reply related
* [PATCH RFC 00/15] mm/slab: introduce alloc_flags and slab_alloc_context
From: Vlastimil Babka (SUSE) @ 2026-06-09 9:17 UTC (permalink / raw)
To: Harry Yoo
Cc: Hao Li, Christoph Lameter, David Rientjes, Roman Gushchin,
Suren Baghdasaryan, Alexei Starovoitov, Andrew Morton,
Johannes Weiner, Michal Hocko, Shakeel Butt, Alexander Potapenko,
Marco Elver, Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel,
cgroups, Vlastimil Babka (SUSE)
This series is based on slab/for-next. If all goes well, it would
hopefully go to slab/for-next soon after the 7.2 merge window, so any
other work can be based on it to avoid conflicts, as it touches a lot
parts of slab.
Git: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slab_alloc_flags
The slab implementation currently relies on gfp flags to convey
some context information internally:
- The absence of both __GFP_RECLAIM flags is interpreted as "cannot spin
on locks", and intended to be used by kmalloc_nolock(). But false
positives are possible e.g. during early boot where gfp_allowed_mask
clears __GFP_RECLAIM from all allocations. This leads to unnecessary
allocation failures and workarounds such as fd3634312a04 ("debugobject:
Make it work with deferred page initialization - again").
- __GFP_NO_OBJ_EXT exists and takes up valuable bit in the gfp flags
space, only to prevent recursive kmalloc() allocations for obj_ext
arrays and sheaves.
The page allocator uses its internal alloc_flags to convey various
context information, including ALLOC_TRYLOCK (meaning "cannot spin").
This series copies that concept for the slab allocator, with its own
slab-specific internal flags:
- SLAB_ALLOC_DEFAULT - no extra flags (the value is 0), but explicit
- SLAB_ALLOC_TRYLOCK - do not spin on locks (used by kmalloc_nolock())
- SLAB_ALLOC_NEW_SLAB - replacing existing 'bool new_slab' parameter
for allocating obj_ext arrays
- SLAB_ALLOC_NO_RECURSE - replacing usage of __GFP_NO_OBJ_EXT
To reduce the amount of parameters in various internal functions, we
additionally introduce slab_alloc_context (also inspired by page
allocator's alloc_context) for passing a number of existing arguments
and the new alloc_flags:
/* Structure holding extra parameters for slab allocations */
struct slab_alloc_context {
unsigned long caller_addr;
unsigned long orig_size;
unsigned int alloc_flags;
struct list_lru *lru;
};
This also replaces the existing struct partial_context.
The last necessary piece is kmalloc_flags() which can take the
alloc_flags in addition to gfp flags and is intended for the recursive
allocations of sheaves and obj_ext arrays, so that both
SLAB_ALLOC_TRYLOCK and SLAB_ALLOC_NO_RECURSE can be communicated.
Internally it decides between kmalloc_nolock() and normal kmalloc()
depending SLAB_ALLOC_TRYLOCK.
The rest of the series is gradually expanding the usage of both
alloc_flags and slab_alloc_context as necessary, with bits of
refactoring. Then, __GFP_NO_OBJ_EXT is removed completely.
Note that some usage of gfpflags_allow_spinning() relying on absence of
__GFP_RECLAIM remains outside of slab (and page allocator) in memcg,
page_owner and stackdepot code. These can thus yield false-positive
decisions that spinning is not allowed, but should not result in
important allocations failing anymore.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
Vlastimil Babka (SUSE) (15):
mm/slab: always zero only requested size on alloc
mm/slab: stop inlining __slab_alloc_node()
mm/slab: introduce slab_alloc_context
mm/slab: introduce alloc_flags and SLAB_ALLOC_TRYLOCK
mm/slab: add alloc_flags to slab_alloc_context
mm/slab: replace struct partial_context with slab_alloc_context
mm/slab: pass alloc_flags to new slab allocation
mm/slab: pass alloc_flags through slab_post_alloc_hook() chain
mm/slab: replace slab_alloc_node() parameters with slab_alloc_context
mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags
mm/slab: pass slab_alloc_context to __do_kmalloc_node()
mm/slab: introduce kmalloc_flags()
mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()
mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves
mm: remove the __GFP_NO_OBJ_EXT flag
include/linux/gfp_types.h | 7 -
include/linux/slab.h | 14 +-
include/trace/events/mmflags.h | 10 +-
lib/alloc_tag.c | 2 +-
mm/kfence/core.c | 6 +-
mm/memcontrol.c | 5 +-
mm/slab.h | 16 +-
mm/slub.c | 423 ++++++++++++++++++++++++----------------
tools/include/linux/gfp_types.h | 7 -
9 files changed, 288 insertions(+), 202 deletions(-)
---
base-commit: 500b2c9755301742bdbb61249511ac11a4665dae
change-id: 20260601-slab_alloc_flags-25c782b0c57c
Best regards,
--
Vlastimil Babka (SUSE) <vbabka@kernel.org>
^ permalink raw reply
* Re: [PATCH v5 9/9] mm: switch deferred split shrinker to list_lru
From: Lance Yang @ 2026-06-09 8:02 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: hannes, baolin.wang, akpm, david, shakeel.butt, mhocko, david,
roman.gushchin, muchun.song, qi.zheng, yosry.ahmed, ziy, liam,
usama.arif, kas, vbabka, ryncsn, zaslonko, gor, baohua, dev.jain,
npache, ryan.roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <aifB-YGdJAGzgprt@lucifer>
On 2026/6/9 15:35, Lorenzo Stoakes wrote:
> On Tue, Jun 09, 2026 at 11:20:58AM +0800, Lance Yang wrote:
>>
>> On Mon, Jun 01, 2026 at 02:17:28PM -0400, Johannes Weiner wrote:
>>> On Mon, Jun 01, 2026 at 09:21:35PM +0800, Lance Yang wrote:
>>>>
>>>> On Wed, May 27, 2026 at 04:45:16PM -0400, Johannes Weiner wrote:
>>>> [...]
>>>>> diff --git a/mm/swap_state.c b/mm/swap_state.c
>>>>> index 04f5ce992401..9c3a5cf99778 100644
>>>>> --- a/mm/swap_state.c
>>>>> +++ b/mm/swap_state.c
>>>>> @@ -465,6 +465,16 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
>>>>> return ERR_PTR(-ENOMEM);
>>>>> }
>>>>>
>>>>
>>>> Shouldn't this be limited to anon swapin?
>>>>
>>>> e.g. vmf && vma_is_anonymous(vmf->vma)
>>>>
>>>>> + if (order > 1 && folio_memcg_alloc_deferred(folio)) {
>>>>
>>>> __swap_cache_alloc() is also used by shmem direct swapin, so shmem can
>>>> get here too when handling a large swap entry:
>>>>
>>>> shmem_get_folio_gfp()
>>>> shmem_swapin_folio()
>>>> shmem_swap_alloc_folio()
>>>> swapin_sync()
>>>> swap_cache_alloc_folio()
>>>> __swap_cache_alloc()
>>>> folio_memcg_alloc_deferred()
>>>
>>> Good catch, I think you're right. I shouldn't have dismissed that
>>> branch due to "/* Direct swapin skipping swap cache & readahead */"
>>>
>>>> @Baolin please correct me if I got it wrong :)
>>>>
>>>> folio_memcg_alloc_deferred() itself doesn't filter shmem out either; it
>>>> only allocates the memcg list_lru metadata for deferred_split_lru:
>>>>
>>>> int folio_memcg_alloc_deferred(struct folio *folio)
>>>> {
>>>> if (mem_cgroup_disabled())
>>>> return 0;
>>>> return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
>>>> }
>>>>
>>>> Since deferred_split_lru only queues anon large folios, doing this for
>>>> shmem swapin doesn't buy us anything :)
>>>
>>> Yes, agreed. I don't think it's a big deal / show stopper in terms of
>>> user-visible effect, but of course still worth fixing.
>>>
>>> I'll send a follow-up patch.
>>
>> Thanks.
>>
>> Looks like this has already landed in mm-stable. If you're okay with it,
>> I can send the follow-up.
>>
>> From: Lance Yang <lance.yang@linux.dev>
>> Date: Tue, 9 Jun 2026 10:56:45 +0800
>> Subject: [PATCH] mm: prepare deferred split metadata only for anon swapin
>>
>> __swap_cache_alloc() prepares deferred split metadata for large swapcache
>> folios.
>>
>> That also covers shmem swapin, because shmem_swap_alloc_folio() can call
>> swapin_sync() with a large order[1]. But shmem folios are not queued on
>> the deferred split queue, so preparing the metadata doesn't buy us
>> anything there.
>>
>> So let's limit it to anon swapin.
>>
>> [1] https://lore.kernel.org/all/20260601132135.14272-1-lance.yang@linux.dev/
>>
>> Signed-off-by: Lance Yang <lance.yang@linux.dev>
>> ---
>> mm/swap_state.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/mm/swap_state.c b/mm/swap_state.c
>> index 9c3a5cf99778..7adac957c2b8 100644
>> --- a/mm/swap_state.c
>> +++ b/mm/swap_state.c
>> @@ -465,7 +465,8 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
>> return ERR_PTR(-ENOMEM);
>> }
>>
>> - if (order > 1 && folio_memcg_alloc_deferred(folio)) {
>> + if (order > 1 && vma && vma_is_anonymous(vma) &&
>
> A folio can be anon for a non-shmem file-backed VMA though?
> E.g. MAP_PRIVATE-mapped file-backed mappings?
Ah, good point! vma_is_anonymous() is too strong here ...
Maybe use !vma_is_shmem(vma) instead?
> Not sure if that's something that'd be a factor here/meaningful though.
>
>> + folio_memcg_alloc_deferred(folio)) {
>> spin_lock(&ci->lock);
>> __swap_cache_do_del_folio(ci, folio, entry, shadow);
>> spin_unlock(&ci->lock);
>> --
>> 2.39.3 (Apple Git-146)
>
> Cheers, Lorenzo
^ permalink raw reply
* Re: [PATCH] mm: constify oom_control, scan_control, and alloc_context nodemask
From: Barry Song @ 2026-06-09 7:48 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, linux-kernel, cgroups, kernel-team, longman, chenridong,
akpm, david, ljs, liam, vbabka, rppt, surenb, mhocko, kasong,
qi.zheng, shakeel.butt, axelrasmussen, yuanchu, weixugc, rientjes,
chrisl, shikemeng, nphamcs, baoquan.he, youngjun.park, tj, hannes,
mkoutny, jackmanb, ziy
In-Reply-To: <20260609002919.3967782-1-gourry@gourry.net>
On Tue, Jun 9, 2026 at 8:29 AM Gregory Price <gourry@gourry.net> wrote:
>
> The nodemasks in these structures may come from a variety of sources,
> including tasks and cpusets - and should never be modified by any code
> when being passed around inside another context.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
LGTM,
Reviewed-by: Barry Song <baohua@kernel.org>
^ permalink raw reply
* Re: [PATCH] mm: constify oom_control, scan_control, and alloc_context nodemask
From: Lorenzo Stoakes @ 2026-06-09 7:41 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, linux-kernel, cgroups, kernel-team, longman, chenridong,
akpm, david, liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, rientjes,
chrisl, shikemeng, nphamcs, baoquan.he, youngjun.park, tj, hannes,
mkoutny, jackmanb, ziy
In-Reply-To: <20260609002919.3967782-1-gourry@gourry.net>
On Mon, Jun 08, 2026 at 08:29:19PM -0400, Gregory Price wrote:
> The nodemasks in these structures may come from a variety of sources,
> including tasks and cpusets - and should never be modified by any code
> when being passed around inside another context.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
Thanks for doing this, it's nice to gradually up our const correctness game
(as much as C can ever be const correct :)
LGTM, builds locally too, so:
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
> ---
> include/linux/cpuset.h | 4 ++--
> include/linux/mm.h | 4 ++--
> include/linux/mmzone.h | 6 +++---
> include/linux/oom.h | 2 +-
> include/linux/swap.h | 2 +-
> kernel/cgroup/cpuset.c | 2 +-
> mm/internal.h | 2 +-
> mm/mmzone.c | 5 +++--
> mm/page_alloc.c | 6 +++---
> mm/show_mem.c | 9 ++++++---
> mm/vmscan.c | 6 +++---
> 11 files changed, 26 insertions(+), 22 deletions(-)
>
> diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
> index 65d76a38974b..a80d38e752d2 100644
> --- a/include/linux/cpuset.h
> +++ b/include/linux/cpuset.h
> @@ -83,7 +83,7 @@ extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
> extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
> #define cpuset_current_mems_allowed (current->mems_allowed)
> void cpuset_init_current_mems_allowed(void);
> -int cpuset_nodemask_valid_mems_allowed(nodemask_t *nodemask);
> +int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask);
>
> extern bool cpuset_current_node_allowed(int node, gfp_t gfp_mask);
>
> @@ -224,7 +224,7 @@ static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
> #define cpuset_current_mems_allowed (node_states[N_MEMORY])
> static inline void cpuset_init_current_mems_allowed(void) {}
>
> -static inline int cpuset_nodemask_valid_mems_allowed(nodemask_t *nodemask)
> +static inline int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
> {
> return 1;
> }
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 485df9c2dbdd..2101e5205fc0 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -4042,7 +4042,7 @@ extern int __meminit early_pfn_to_nid(unsigned long pfn);
> extern void mem_init(void);
> extern void __init mmap_init(void);
>
> -extern void __show_mem(unsigned int flags, nodemask_t *nodemask, int max_zone_idx);
> +extern void __show_mem(unsigned int flags, const nodemask_t *nodemask, int max_zone_idx);
> static inline void show_mem(void)
> {
> __show_mem(0, NULL, MAX_NR_ZONES - 1);
> @@ -4052,7 +4052,7 @@ extern void si_meminfo(struct sysinfo * val);
> extern void si_meminfo_node(struct sysinfo *val, int nid);
>
> extern __printf(3, 4)
> -void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...);
> +void warn_alloc(gfp_t gfp_mask, const nodemask_t *nodemask, const char *fmt, ...);
>
> extern void setup_per_cpu_pageset(void);
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index ca2712187147..919be0d9b4fa 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -1815,7 +1815,7 @@ static inline int zonelist_node_idx(const struct zoneref *zoneref)
>
> struct zoneref *__next_zones_zonelist(struct zoneref *z,
> enum zone_type highest_zoneidx,
> - nodemask_t *nodes);
> + const nodemask_t *nodes);
>
> /**
> * next_zones_zonelist - Returns the next zone at or below highest_zoneidx within the allowed nodemask using a cursor within a zonelist as a starting point
> @@ -1834,7 +1834,7 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,
> */
> static __always_inline struct zoneref *next_zones_zonelist(struct zoneref *z,
> enum zone_type highest_zoneidx,
> - nodemask_t *nodes)
> + const nodemask_t *nodes)
> {
> if (likely(!nodes && zonelist_zone_idx(z) <= highest_zoneidx))
> return z;
> @@ -1860,7 +1860,7 @@ static __always_inline struct zoneref *next_zones_zonelist(struct zoneref *z,
> */
> static inline struct zoneref *first_zones_zonelist(struct zonelist *zonelist,
> enum zone_type highest_zoneidx,
> - nodemask_t *nodes)
> + const nodemask_t *nodes)
> {
> return next_zones_zonelist(zonelist->_zonerefs,
> highest_zoneidx, nodes);
> diff --git a/include/linux/oom.h b/include/linux/oom.h
> index 7b02bc1d0a7e..00da05d227e6 100644
> --- a/include/linux/oom.h
> +++ b/include/linux/oom.h
> @@ -30,7 +30,7 @@ struct oom_control {
> struct zonelist *zonelist;
>
> /* Used to determine mempolicy */
> - nodemask_t *nodemask;
> + const nodemask_t *nodemask;
>
> /* Memory cgroup in which oom is invoked, or NULL for global oom */
> struct mem_cgroup *memcg;
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 8f0f68e245ba..bf76356d94fe 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -351,7 +351,7 @@ extern void swap_setup(void);
> /* linux/mm/vmscan.c */
> extern unsigned long zone_reclaimable_pages(struct zone *zone);
> extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
> - gfp_t gfp_mask, nodemask_t *mask);
> + gfp_t gfp_mask, const nodemask_t *mask);
> unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone_idx);
>
> #define MEMCG_RECLAIM_MAY_SWAP (1 << 1)
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 5c33ab20cc20..536ce591c7ba 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -4153,7 +4153,7 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
> *
> * Are any of the nodes in the nodemask allowed in current->mems_allowed?
> */
> -int cpuset_nodemask_valid_mems_allowed(nodemask_t *nodemask)
> +int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
> {
> return nodes_intersects(*nodemask, current->mems_allowed);
> }
> diff --git a/mm/internal.h b/mm/internal.h
> index 181e79f1d6a2..a2ef9512b5bc 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -668,7 +668,7 @@ void page_alloc_sysctl_init(void);
> */
> struct alloc_context {
> struct zonelist *zonelist;
> - nodemask_t *nodemask;
> + const nodemask_t *nodemask;
> struct zoneref *preferred_zoneref;
> int migratetype;
>
> diff --git a/mm/mmzone.c b/mm/mmzone.c
> index 0c8f181d9d50..59dc3f2076a6 100644
> --- a/mm/mmzone.c
> +++ b/mm/mmzone.c
> @@ -43,7 +43,8 @@ struct zone *next_zone(struct zone *zone)
> return zone;
> }
>
> -static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes)
> +static inline int zref_in_nodemask(struct zoneref *zref,
> + const nodemask_t *nodes)
> {
> #ifdef CONFIG_NUMA
> return node_isset(zonelist_node_idx(zref), *nodes);
> @@ -55,7 +56,7 @@ static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes)
> /* Returns the next zone at or below highest_zoneidx in a zonelist */
> struct zoneref *__next_zones_zonelist(struct zoneref *z,
> enum zone_type highest_zoneidx,
> - nodemask_t *nodes)
> + const nodemask_t *nodes)
> {
> /*
> * Find the next suitable zone to use for the allocation.
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 81a9d4d1e6c0..586524bbde9c 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3979,7 +3979,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
> return NULL;
> }
>
> -static void warn_alloc_show_mem(gfp_t gfp_mask, nodemask_t *nodemask)
> +static void warn_alloc_show_mem(gfp_t gfp_mask, const nodemask_t *nodemask)
> {
> unsigned int filter = SHOW_MEM_FILTER_NODES;
>
> @@ -3999,7 +3999,7 @@ static void warn_alloc_show_mem(gfp_t gfp_mask, nodemask_t *nodemask)
> mem_cgroup_show_protected_memory(NULL);
> }
>
> -void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...)
> +void warn_alloc(gfp_t gfp_mask, const nodemask_t *nodemask, const char *fmt, ...)
> {
> struct va_format vaf;
> va_list args;
> @@ -4687,7 +4687,7 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
> return false;
> }
>
> -static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
> +static void check_alloc_stall_warn(gfp_t gfp_mask, const nodemask_t *nodemask,
> unsigned int order, unsigned long alloc_start_time)
> {
> static DEFINE_SPINLOCK(alloc_stall_lock);
> diff --git a/mm/show_mem.c b/mm/show_mem.c
> index 43aca5a2ac99..1b721a8ade67 100644
> --- a/mm/show_mem.c
> +++ b/mm/show_mem.c
> @@ -116,7 +116,8 @@ void si_meminfo_node(struct sysinfo *val, int nid)
> * Determine whether the node should be displayed or not, depending on whether
> * SHOW_MEM_FILTER_NODES was passed to show_free_areas().
> */
> -static bool show_mem_node_skip(unsigned int flags, int nid, nodemask_t *nodemask)
> +static bool show_mem_node_skip(unsigned int flags, int nid,
> + const nodemask_t *nodemask)
> {
> if (!(flags & SHOW_MEM_FILTER_NODES))
> return false;
> @@ -177,7 +178,8 @@ static bool node_has_managed_zones(pg_data_t *pgdat, int max_zone_idx)
> * SHOW_MEM_FILTER_NODES: suppress nodes that are not allowed by current's
> * cpuset.
> */
> -static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
> +static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
> + int max_zone_idx)
> {
> unsigned long free_pcp = 0;
> int cpu, nid;
> @@ -402,7 +404,8 @@ static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_z
> show_swap_cache_info();
> }
>
> -void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
> +void __show_mem(unsigned int filter, const nodemask_t *nodemask,
> + int max_zone_idx)
> {
> unsigned long total = 0, reserved = 0, highmem = 0;
> struct zone *zone;
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index e8a90911bf88..47d3f3361fb9 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -79,7 +79,7 @@ struct scan_control {
> * Nodemask of nodes allowed by the caller. If NULL, all nodes
> * are scanned.
> */
> - nodemask_t *nodemask;
> + const nodemask_t *nodemask;
>
> /*
> * The memory cgroup that hit its limit and as a result is the
> @@ -6599,7 +6599,7 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)
> * happens, the page allocator should not consider triggering the OOM killer.
> */
> static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
> - nodemask_t *nodemask)
> + const nodemask_t *nodemask)
> {
> struct zoneref *z;
> struct zone *zone;
> @@ -6679,7 +6679,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
> }
>
> unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
> - gfp_t gfp_mask, nodemask_t *nodemask)
> + gfp_t gfp_mask, const nodemask_t *nodemask)
> {
> unsigned long nr_reclaimed;
> struct scan_control sc = {
> --
> 2.54.0
>
^ permalink raw reply
* Re: [PATCH] mm: constify oom_control, scan_control, and alloc_context nodemask
From: Lorenzo Stoakes @ 2026-06-09 7:39 UTC (permalink / raw)
To: Zi Yan
Cc: Gregory Price, linux-mm, linux-kernel, cgroups, kernel-team,
longman, chenridong, akpm, david, liam, vbabka, rppt, surenb,
mhocko, kasong, qi.zheng, shakeel.butt, baohua, axelrasmussen,
yuanchu, weixugc, rientjes, chrisl, shikemeng, nphamcs,
baoquan.he, youngjun.park, tj, hannes, mkoutny, jackmanb
In-Reply-To: <8C4E5377-F5CF-458E-BA49-3D962CB75477@nvidia.com>
On Mon, Jun 08, 2026 at 09:44:42PM -0400, Zi Yan wrote:
> On 8 Jun 2026, at 20:29, Gregory Price wrote:
>
> > The nodemasks in these structures may come from a variety of sources,
> > including tasks and cpusets - and should never be modified by any code
> > when being passed around inside another context.
> >
> > Signed-off-by: Gregory Price <gourry@gourry.net>
> > ---
> > include/linux/cpuset.h | 4 ++--
> > include/linux/mm.h | 4 ++--
> > include/linux/mmzone.h | 6 +++---
> > include/linux/oom.h | 2 +-
> > include/linux/swap.h | 2 +-
> > kernel/cgroup/cpuset.c | 2 +-
> > mm/internal.h | 2 +-
> > mm/mmzone.c | 5 +++--
> > mm/page_alloc.c | 6 +++---
> > mm/show_mem.c | 9 ++++++---
> > mm/vmscan.c | 6 +++---
> > 11 files changed, 26 insertions(+), 22 deletions(-)
> >
>
> LGTM and it compiles. As long as Sashiko does not complain, feel free to
> add:
I would add caveats of:
- Complains legitimately
- And it's about this actual patch not something unrelated
:P
(Not speaking for Zi of course, but I mean just in general I feel these caveats
should be implicit :))
>
> Acked-by: Zi Yan <ziy@nvidia.com>
>
> Best Regards,
> Yan, Zi
Cheers, Lorenzo
^ permalink raw reply
* Re: [PATCH v5 9/9] mm: switch deferred split shrinker to list_lru
From: Lorenzo Stoakes @ 2026-06-09 7:35 UTC (permalink / raw)
To: Lance Yang
Cc: hannes, baolin.wang, akpm, david, shakeel.butt, mhocko, david,
roman.gushchin, muchun.song, qi.zheng, yosry.ahmed, ziy, liam,
usama.arif, kas, vbabka, ryncsn, zaslonko, gor, baohua, dev.jain,
npache, ryan.roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260609032058.23770-1-lance.yang@linux.dev>
On Tue, Jun 09, 2026 at 11:20:58AM +0800, Lance Yang wrote:
>
> On Mon, Jun 01, 2026 at 02:17:28PM -0400, Johannes Weiner wrote:
> >On Mon, Jun 01, 2026 at 09:21:35PM +0800, Lance Yang wrote:
> >>
> >> On Wed, May 27, 2026 at 04:45:16PM -0400, Johannes Weiner wrote:
> >> [...]
> >> >diff --git a/mm/swap_state.c b/mm/swap_state.c
> >> >index 04f5ce992401..9c3a5cf99778 100644
> >> >--- a/mm/swap_state.c
> >> >+++ b/mm/swap_state.c
> >> >@@ -465,6 +465,16 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
> >> > return ERR_PTR(-ENOMEM);
> >> > }
> >> >
> >>
> >> Shouldn't this be limited to anon swapin?
> >>
> >> e.g. vmf && vma_is_anonymous(vmf->vma)
> >>
> >> >+ if (order > 1 && folio_memcg_alloc_deferred(folio)) {
> >>
> >> __swap_cache_alloc() is also used by shmem direct swapin, so shmem can
> >> get here too when handling a large swap entry:
> >>
> >> shmem_get_folio_gfp()
> >> shmem_swapin_folio()
> >> shmem_swap_alloc_folio()
> >> swapin_sync()
> >> swap_cache_alloc_folio()
> >> __swap_cache_alloc()
> >> folio_memcg_alloc_deferred()
> >
> >Good catch, I think you're right. I shouldn't have dismissed that
> >branch due to "/* Direct swapin skipping swap cache & readahead */"
> >
> >> @Baolin please correct me if I got it wrong :)
> >>
> >> folio_memcg_alloc_deferred() itself doesn't filter shmem out either; it
> >> only allocates the memcg list_lru metadata for deferred_split_lru:
> >>
> >> int folio_memcg_alloc_deferred(struct folio *folio)
> >> {
> >> if (mem_cgroup_disabled())
> >> return 0;
> >> return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
> >> }
> >>
> >> Since deferred_split_lru only queues anon large folios, doing this for
> >> shmem swapin doesn't buy us anything :)
> >
> >Yes, agreed. I don't think it's a big deal / show stopper in terms of
> >user-visible effect, but of course still worth fixing.
> >
> >I'll send a follow-up patch.
>
> Thanks.
>
> Looks like this has already landed in mm-stable. If you're okay with it,
> I can send the follow-up.
>
> From: Lance Yang <lance.yang@linux.dev>
> Date: Tue, 9 Jun 2026 10:56:45 +0800
> Subject: [PATCH] mm: prepare deferred split metadata only for anon swapin
>
> __swap_cache_alloc() prepares deferred split metadata for large swapcache
> folios.
>
> That also covers shmem swapin, because shmem_swap_alloc_folio() can call
> swapin_sync() with a large order[1]. But shmem folios are not queued on
> the deferred split queue, so preparing the metadata doesn't buy us
> anything there.
>
> So let's limit it to anon swapin.
>
> [1] https://lore.kernel.org/all/20260601132135.14272-1-lance.yang@linux.dev/
>
> Signed-off-by: Lance Yang <lance.yang@linux.dev>
> ---
> mm/swap_state.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index 9c3a5cf99778..7adac957c2b8 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -465,7 +465,8 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
> return ERR_PTR(-ENOMEM);
> }
>
> - if (order > 1 && folio_memcg_alloc_deferred(folio)) {
> + if (order > 1 && vma && vma_is_anonymous(vma) &&
A folio can be anon for a non-shmem file-backed VMA though?
E.g. MAP_PRIVATE-mapped file-backed mappings?
Not sure if that's something that'd be a factor here/meaningful though.
> + folio_memcg_alloc_deferred(folio)) {
> spin_lock(&ci->lock);
> __swap_cache_do_del_folio(ci, folio, entry, shadow);
> spin_unlock(&ci->lock);
> --
> 2.39.3 (Apple Git-146)
Cheers, Lorenzo
^ permalink raw reply
* Re: [PATCH v3 0/7] sched: Flatten the pick
From: K Prateek Nayak @ 2026-06-09 5:37 UTC (permalink / raw)
To: Peter Zijlstra, mingo
Cc: longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
Hello Peter,
On 6/5/2026 6:10 PM, Peter Zijlstra wrote:
> Can also be had:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
Here are early performance numbers at commit 7eb85c33dd20 ("sched/eevdf:
Move to a single runqueue").
tl;dr
Apart form a regression in the tbench and schbench at super high
utilization, all numbers for concur mode looks good so 7.2-rc1 target
should be good.
P.S. I won't be able to look into this until Thursday; Sorry in
advance.
Note: I haven't gotten around to analyzing anything in depth; These are
just early numbers on microbenchmarks and DeathStarBench. Please take
them with a grain of salt.
tip was at commit f666241e6bd5 ("sched/fair: Unify cfs_rq throttling via
account_cfs_rq_runtime()") and each label are the individual cgroup
mode togged on at queue:sched/flat:
Machine:
o 4th Generation EPYC system (Zen4c)
o 2 x 128C/256T (32LLCs)
o Boost enabled
o C2 disabled; MWAIT based C1 and POLL remained enabled
Benchmark numbers:
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
Case: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1-groups 1.00 [ -0.00](11.18) 1.01 [ -0.60](12.84) 1.12 [-12.08](12.50) 1.16 [-15.71]( 9.46) 1.07 [ -7.25](11.46) 0.99 [ 1.21](15.44)
2-groups 1.00 [ -0.00]( 4.73) 1.01 [ -1.00]( 9.29) 2.88 [-188.25](63.11) 0.97 [ 2.75](11.62) 0.95 [ 5.00](12.47) 0.96 [ 4.00](11.72)
4-groups 1.00 [ -0.00]( 2.90) 0.99 [ 0.96]( 2.48) 0.99 [ 0.72]( 2.38) 1.00 [ 0.48]( 1.78) 0.99 [ 0.96]( 6.61) 1.00 [ -0.00]( 5.23)
8-groups 1.00 [ -0.00]( 1.82) 1.03 [ -2.51]( 2.81) 0.99 [ 0.91]( 3.33) 0.99 [ 0.91]( 2.40) 1.01 [ -0.68]( 2.44) 1.02 [ -1.60]( 2.45)
16-groups 1.00 [ -0.00]( 3.05) 1.03 [ -2.96]( 1.97) 1.23 [-22.82](22.67) 1.01 [ -1.31]( 2.03) 0.99 [ 0.99]( 2.48) 1.01 [ -0.66]( 2.76)
Note: For smp variant runs, I think there was some system noise form
an unrelated job that stated by mistake. I have to go back and
rerun to confirm if the regression holds.
==================================================================
Test : tbench
Units : Normalized throughput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ 0.00]( 0.23) 1.02 [ 1.53]( 0.10) 1.02 [ 1.80]( 0.96) 1.00 [ 0.19]( 0.17) 0.99 [ -0.58]( 0.18) 1.01 [ 1.13]( 0.18)
2 1.00 [ 0.00]( 0.11) 1.01 [ 1.13]( 0.04) 1.01 [ 1.26]( 0.08) 1.00 [ -0.12]( 0.21) 0.99 [ -1.00]( 0.29) 1.01 [ 0.51]( 0.02)
4 1.00 [ 0.00]( 0.11) 1.01 [ 1.30]( 0.16) 1.02 [ 2.26]( 0.48) 1.00 [ 0.06]( 0.37) 0.99 [ -0.74]( 0.52) 1.01 [ 1.35]( 0.35)
8 1.00 [ 0.00]( 0.24) 1.01 [ 1.19]( 0.81) 1.02 [ 2.17]( 0.52) 1.00 [ 0.45]( 0.36) 0.99 [ -0.80]( 0.05) 1.01 [ 0.92]( 0.39)
16 1.00 [ 0.00]( 0.15) 1.01 [ 0.87]( 0.70) 1.03 [ 2.56]( 0.18) 1.00 [ -0.41]( 0.46) 0.99 [ -0.99]( 0.36) 1.01 [ 0.55]( 1.02)
32 1.00 [ 0.00]( 1.02) 1.02 [ 2.42]( 0.36) 1.04 [ 3.76]( 0.94) 1.01 [ 1.20]( 0.19) 1.00 [ -0.10]( 0.44) 1.02 [ 1.61]( 0.27)
64 1.00 [ 0.00]( 0.36) 1.02 [ 1.92]( 1.71) 1.03 [ 2.59]( 1.15) 0.99 [ -0.51]( 0.88) 1.01 [ 1.19]( 0.29) 1.02 [ 2.42]( 0.57)
128 1.00 [ 0.00]( 0.45) 1.01 [ 1.11]( 1.37) 1.05 [ 4.64]( 1.05) 1.01 [ 0.98]( 2.47) 1.01 [ 0.84]( 1.97) 1.03 [ 2.56]( 1.22)
256 1.00 [ 0.00]( 0.06) 1.02 [ 2.23]( 1.11) 1.02 [ 2.17]( 0.69) 1.03 [ 2.87]( 0.46) 1.03 [ 2.57]( 0.41) 1.03 [ 2.99]( 0.84)
512 1.00 [ 0.00]( 1.50) 0.92 [ -7.62]( 6.42) 1.02 [ 1.94]( 2.12) 1.01 [ 0.74]( 6.70) 0.94 [ -6.09]( 5.19) 0.98 [ -2.40]( 3.00)
1024 1.00 [ 0.00]( 0.07) 0.98 [ -1.51]( 0.30) 1.02 [ 1.66]( 0.13) 0.97 [ -2.97]( 0.76) 0.94 [ -6.33]( 0.26) 0.95 [ -4.63]( 0.47)
2048 1.00 [ 0.00]( 0.25) 0.98 [ -1.57]( 0.59) 1.02 [ 1.81]( 0.20) 0.98 [ -1.94]( 0.38) 0.94 [ -5.54]( 0.17) 0.95 [ -4.56]( 0.77)
==================================================================
Test : stream-10
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
Copy 1.00 [ 0.00]( 0.51) 0.92 [ -7.61](12.64) 0.99 [ -0.79]( 0.30) 0.85 [-15.33](21.73) 0.99 [ -0.61]( 0.35) 0.99 [ -0.90]( 0.38)
Scale 1.00 [ 0.00]( 0.35) 0.90 [ -9.96](14.86) 0.99 [ -1.45]( 0.76) 0.85 [-14.73](21.14) 0.99 [ -1.12]( 0.86) 0.99 [ -1.13]( 0.71)
Add 1.00 [ 0.00]( 0.21) 0.93 [ -6.95](10.26) 0.98 [ -1.51]( 0.80) 0.95 [ -4.91]( 9.84) 0.99 [ -0.98]( 0.69) 0.99 [ -1.37]( 0.67)
Triad 1.00 [ 0.00]( 0.24) 0.89 [-10.61](15.55) 0.99 [ -1.46]( 0.72) 0.95 [ -4.87]( 9.50) 0.99 [ -0.90]( 0.62) 0.99 [ -1.44]( 0.61)
==================================================================
Test : stream-100
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
Copy 1.00 [ 0.00]( 1.52) 1.00 [ -0.16]( 0.56) 0.97 [ -2.54]( 2.30) 1.00 [ 0.09]( 0.18) 1.00 [ 0.17]( 0.44) 0.99 [ -0.74]( 1.49)
Scale 1.00 [ 0.00]( 1.43) 0.99 [ -0.63]( 0.55) 0.97 [ -2.72]( 2.44) 1.00 [ 0.20]( 0.16) 0.99 [ -0.67]( 0.43) 0.99 [ -1.39]( 1.45)
Add 1.00 [ 0.00]( 1.06) 0.99 [ -1.27]( 0.42) 0.97 [ -3.08]( 1.95) 1.00 [ -0.13]( 0.19) 0.99 [ -1.19]( 0.34) 0.98 [ -1.76]( 1.07)
Triad 1.00 [ 0.00]( 1.10) 0.98 [ -1.55]( 0.41) 0.97 [ -3.40]( 1.95) 1.00 [ -0.42]( 0.16) 0.99 [ -1.49]( 0.35) 0.98 [ -2.08]( 1.11)
==================================================================
Test : netperf
Units : Normalized Througput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1-clients 1.00 [ 0.00]( 0.15) 1.02 [ 2.24]( 0.12) 1.02 [ 1.50]( 0.22) 1.07 [ 6.86]( 2.15) 1.00 [ 0.27]( 0.45) 1.02 [ 1.86]( 0.15)
2-clients 1.00 [ 0.00]( 0.37) 1.03 [ 2.50]( 0.80) 1.03 [ 3.11]( 0.78) 1.07 [ 7.15]( 0.96) 1.00 [ -0.43]( 0.54) 1.02 [ 1.67]( 0.69)
4-clients 1.00 [ 0.00]( 0.26) 1.02 [ 2.26]( 0.33) 1.02 [ 2.02]( 0.37) 1.07 [ 7.37]( 0.78) 1.00 [ -0.24]( 0.42) 1.02 [ 1.84]( 0.22)
8-clients 1.00 [ 0.00]( 0.21) 1.02 [ 2.47]( 0.55) 1.02 [ 2.13]( 0.48) 1.08 [ 7.60]( 0.57) 1.00 [ -0.15]( 0.37) 1.02 [ 1.87]( 0.29)
16-clients 1.00 [ 0.00]( 0.23) 1.02 [ 2.07]( 0.69) 1.02 [ 1.87]( 0.42) 1.07 [ 7.38]( 0.50) 0.99 [ -0.55]( 0.48) 1.02 [ 1.78]( 0.25)
32-clients 1.00 [ 0.00]( 0.47) 1.02 [ 2.14]( 0.63) 1.02 [ 1.81]( 0.75) 1.07 [ 7.44]( 0.94) 1.00 [ -0.38]( 0.53) 1.02 [ 1.76]( 0.43)
64-clients 1.00 [ 0.00]( 0.91) 1.02 [ 2.00]( 0.81) 1.02 [ 1.74]( 0.96) 1.07 [ 7.12]( 1.06) 1.00 [ -0.20]( 0.68) 1.02 [ 1.63]( 0.76)
128-clients 1.00 [ 0.00]( 1.19) 1.01 [ 1.36]( 1.28) 1.01 [ 1.34]( 1.18) 1.06 [ 6.37]( 1.47) 1.00 [ -0.49]( 1.15) 1.01 [ 1.06]( 1.09)
256-clients 1.00 [ 0.00]( 1.00) 1.02 [ 1.70]( 1.15) 1.02 [ 1.64]( 1.18) 1.07 [ 7.17]( 1.87) 1.00 [ -0.31]( 1.19) 1.01 [ 1.29]( 1.12)
512-clients 1.00 [ 0.00]( 5.16) 1.00 [ 0.02]( 6.48) 0.99 [ -0.60]( 4.31) 1.04 [ 4.08]( 3.83) 1.00 [ 0.02]( 2.52) 1.00 [ 0.48]( 2.86)
768-clients 1.00 [ 0.00](34.61) 1.03 [ 2.84](62.48) 1.00 [ 0.26](30.91) 0.98 [ -2.12](13.41) 0.94 [ -6.34](11.15) 0.95 [ -4.82](10.61)
1024-clients 1.00 [ 0.00](41.78) 1.04 [ 3.95](76.45) 1.01 [ 0.99](40.23) 0.97 [ -2.58](11.79) 0.95 [ -5.36](11.21) 0.96 [ -4.01](12.98)
==================================================================
Test : schbench
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ -0.00]( 5.88) 0.94 [ 5.88](19.25) 0.88 [ 11.76]( 7.37) 0.88 [ 11.76](34.02) 1.06 [ -5.88](11.11) 0.94 [ 5.88]( 3.69)
2 1.00 [ -0.00](36.56) 0.97 [ 3.03]( 9.52) 0.97 [ 3.03]( 4.82) 1.00 [ -0.00](26.93) 0.97 [ 3.03](14.53) 0.82 [ 18.18](13.87)
4 1.00 [ -0.00]( 9.35) 0.97 [ 3.12](18.25) 0.94 [ 6.25]( 9.12) 1.00 [ -0.00](20.82) 1.00 [ -0.00]( 4.82) 0.88 [ 12.50](15.93)
8 1.00 [ -0.00](23.38) 0.94 [ 6.45](23.26) 1.26 [-25.81](14.43) 1.26 [-25.81](26.13) 1.06 [ -6.45]( 4.68) 0.97 [ 3.23](10.00)
16 1.00 [ -0.00]( 2.71) 1.04 [ -3.57]( 2.62) 1.00 [ -0.00]( 2.04) 0.98 [ 1.79]( 2.76) 1.04 [ -3.57]( 0.00) 1.02 [ -1.79]( 2.70)
32 1.00 [ -0.00]( 0.72) 1.00 [ -0.00]( 0.72) 0.96 [ 3.75]( 1.30) 1.01 [ -1.25]( 0.71) 1.01 [ -1.25]( 1.88) 1.02 [ -2.50]( 1.86)
64 1.00 [ -0.00]( 1.52) 0.96 [ 4.41]( 1.18) 0.97 [ 2.94]( 4.49) 0.94 [ 5.88]( 1.19) 0.95 [ 5.15]( 0.45) 0.96 [ 4.41]( 0.45)
128 1.00 [ -0.00]( 3.24) 0.96 [ 3.80]( 0.91) 0.96 [ 3.80]( 1.53) 0.95 [ 4.64]( 0.26) 0.96 [ 3.80]( 0.44) 0.97 [ 2.53]( 1.32)
256 1.00 [ -0.00]( 0.90) 0.92 [ 7.60]( 0.75) 0.95 [ 5.40]( 0.61) 0.92 [ 7.80]( 0.55) 0.93 [ 6.60]( 6.23) 1.00 [ -0.00]( 0.61)
512 1.00 [ -0.00]( 1.76) 1.10 [-10.36]( 0.48) 1.04 [ -4.15]( 1.46) 0.97 [ 3.45]( 7.07) 1.07 [ -6.56]( 2.54) 0.93 [ 6.56]( 2.45)
768 1.00 [ -0.00]( 2.79) 0.95 [ 5.10]( 6.65) 1.71 [-70.87]( 4.67) 0.80 [ 20.13]( 1.81) 0.78 [ 21.80]( 1.19) 0.78 [ 22.36]( 1.46)
1024 1.00 [ -0.00]( 0.86) 0.35 [ 64.79](18.07) 1.16 [-15.56]( 1.17) 1.00 [ 0.32]( 3.65) 0.95 [ 4.54]( 3.10) 0.90 [ 9.72]( 2.33)
==================================================================
Test : new-schbench-requests-per-second
Units : Normalized Requests per second
Interpretation: Higher is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ 0.00]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ -0.30]( 0.00) 1.00 [ -0.30]( 0.31) 0.99 [ -0.59]( 0.15) 1.00 [ 0.00]( 0.00)
2 1.00 [ 0.00]( 0.00) 1.00 [ 0.30]( 0.00) 1.00 [ 0.00]( 0.61) 1.00 [ 0.30]( 0.15) 1.00 [ -0.30]( 0.70) 1.00 [ 0.00]( 0.61)
4 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.30) 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.00) 1.00 [ -0.29]( 0.15) 0.99 [ -0.59]( 0.31)
8 1.00 [ 0.00]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.00) 1.00 [ -0.29]( 0.15)
16 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.15) 1.00 [ 0.00]( 0.15) 1.00 [ 0.00]( 0.15) 1.00 [ -0.29]( 0.00) 1.00 [ -0.29]( 0.15)
32 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.15) 1.00 [ 0.29]( 0.15) 1.00 [ 0.29]( 0.00) 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.15)
64 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.15)
128 1.00 [ 0.00](17.05) 1.01 [ 0.59](14.45) 1.00 [ 0.30]( 0.00) 1.01 [ 0.59](10.49) 1.00 [ 0.00]( 9.79) 1.00 [ 0.30]( 0.31)
256 1.00 [ 0.00]( 0.59) 1.00 [ 0.28]( 0.59) 1.00 [ 0.28]( 0.44) 1.00 [ -0.28]( 0.82) 0.99 [ -0.57]( 0.39) 1.00 [ 0.00]( 0.67)
512 1.00 [ 0.00]( 0.69) 1.01 [ 1.50]( 1.66) 1.00 [ 0.00]( 0.33) 0.99 [ -1.50]( 0.39) 0.99 [ -1.12]( 0.58) 1.00 [ 0.00]( 0.78)
768 1.00 [ 0.00]( 1.13) 1.12 [ 11.51]( 9.06) 0.99 [ -1.15]( 0.21) 0.91 [ -8.52]( 0.47) 0.91 [ -8.98]( 0.47) 0.93 [ -6.67]( 0.84)
1024 1.00 [ 0.00]( 1.23) 1.12 [ 12.41]( 4.32) 1.01 [ 1.10]( 0.71) 0.87 [-13.24]( 0.59) 0.87 [-12.97]( 0.49) 0.86 [-14.07]( 0.00)
==================================================================
Test : new-schbench-wakeup-latency
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ -0.00](21.48) 2.75 [-175.00](14.06) 3.00 [-200.00](12.91) 3.62 [-262.50](29.64) 3.62 [-262.50](21.84) 2.25 [-125.00](12.45)
2 1.00 [ -0.00]( 5.96) 1.33 [-33.33](24.78) 1.33 [-33.33](18.25) 1.11 [-11.11](25.37) 1.11 [-11.11]( 9.68) 1.11 [-11.11]( 5.00)
4 1.00 [ -0.00]( 6.20) 1.38 [-37.50]( 4.84) 1.12 [-12.50]( 0.00) 1.25 [-25.00](21.51) 1.12 [-12.50](19.99) 1.38 [-37.50](12.81)
8 1.00 [ -0.00]( 5.34) 1.20 [-20.00]( 0.00) 1.00 [ -0.00]( 5.00) 1.20 [-20.00]( 7.45) 1.20 [-20.00]( 0.00) 1.20 [-20.00]( 4.19)
16 1.00 [ -0.00]( 5.53) 1.22 [-22.22]( 0.00) 1.11 [-11.11]( 0.00) 1.11 [-11.11]( 9.68) 1.33 [-33.33]( 0.00) 1.33 [-33.33]( 4.43)
32 1.00 [ -0.00]( 5.53) 1.11 [-11.11]( 5.00) 1.00 [ -0.00]( 0.00) 1.11 [-11.11]( 5.00) 1.11 [-11.11]( 0.00) 1.11 [-11.11]( 0.00)
64 1.00 [ -0.00](12.81) 1.00 [ -0.00]( 0.00) 0.91 [ 9.09]( 5.34) 0.91 [ 9.09]( 5.00) 0.91 [ 9.09]( 5.00) 0.82 [ 18.18](10.68)
128 1.00 [ -0.00](12.14) 1.12 [-12.50]( 4.97) 0.81 [ 18.75](13.62) 1.06 [ -6.25]( 5.26) 0.94 [ 6.25](14.68) 0.81 [ 18.75]( 6.88)
256 1.00 [ -0.00]( 4.83) 0.98 [ 1.90]( 7.97) 0.98 [ 2.37]( 2.41) 0.95 [ 4.74]( 5.31) 0.95 [ 5.21]( 3.25) 0.97 [ 2.84]( 2.84)
512 1.00 [ -0.00]( 0.00) 0.98 [ 2.00]( 5.86) 0.96 [ 3.99]( 1.61) 0.83 [ 16.83]( 1.08) 0.85 [ 14.84]( 7.53) 0.97 [ 3.42]( 7.91)
768 1.00 [ -0.00]( 0.71) 0.96 [ 3.63](14.27) 0.96 [ 3.73]( 1.19) 0.67 [ 32.98]( 0.00) 0.67 [ 32.98]( 0.00) 0.67 [ 32.98]( 0.15)
1024 1.00 [ -0.00]( 2.55) 0.85 [ 14.99](14.23) 0.98 [ 1.80]( 0.82) 0.79 [ 21.29]( 0.00) 0.79 [ 21.29]( 0.00) 0.79 [ 21.29]( 0.00)
==================================================================
Test : new-schbench-request-latency
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ -0.00]( 0.14) 1.00 [ -0.00]( 0.00) 1.00 [ -0.00]( 0.14) 1.00 [ -0.26]( 0.27) 1.01 [ -0.53]( 0.27) 1.00 [ -0.00]( 0.14)
2 1.00 [ -0.00]( 0.00) 1.00 [ 0.26]( 0.14) 1.02 [ -2.36]( 1.51) 0.99 [ 0.52]( 0.14) 1.00 [ -0.00]( 2.00) 1.00 [ -0.00]( 1.74)
4 1.00 [ -0.00]( 0.00) 1.00 [ 0.26]( 1.82) 1.03 [ -2.63]( 0.23) 0.99 [ 0.53]( 0.14) 1.00 [ -0.26]( 1.61) 1.03 [ -3.42]( 1.79)
8 1.00 [ -0.00]( 0.14) 0.99 [ 0.79]( 0.00) 1.00 [ -0.00]( 0.00) 0.99 [ 1.05]( 0.14) 1.00 [ -0.00]( 0.00) 1.00 [ -0.00]( 0.00)
16 1.00 [ -0.00]( 0.14) 1.00 [ -0.00]( 1.16) 1.00 [ -0.26]( 1.28) 0.99 [ 0.79]( 1.97) 1.00 [ -0.26]( 0.00) 1.02 [ -2.38]( 1.15)
32 1.00 [ -0.00]( 0.23) 0.99 [ 1.04]( 0.24) 1.00 [ -0.00]( 0.67) 0.99 [ 1.04]( 0.14) 0.99 [ 0.52]( 0.49) 0.99 [ 1.30]( 0.95)
64 1.00 [ -0.00](12.39) 1.72 [-71.69](26.18) 1.01 [ -1.29](28.90) 0.97 [ 3.09]( 0.27) 0.98 [ 2.06]( 0.41) 0.98 [ 2.32]( 1.22)
128 1.00 [ -0.00]( 4.75) 0.98 [ 2.07](12.01) 0.99 [ 0.59]( 0.41) 1.00 [ 0.30]( 3.02) 1.00 [ -0.00]( 2.72) 0.99 [ 0.59]( 0.77)
256 1.00 [ -0.00]( 0.13) 0.99 [ 0.76]( 0.13) 1.00 [ -0.00]( 0.13) 0.99 [ 0.76]( 0.00) 0.99 [ 0.51]( 0.13) 0.99 [ 0.51]( 0.13)
512 1.00 [ -0.00]( 7.49) 1.04 [ -4.36](28.15) 0.89 [ 10.50]( 4.15) 0.42 [ 57.93]( 9.03) 0.48 [ 52.23](25.18) 0.71 [ 28.83](25.82)
768 1.00 [ -0.00]( 3.13) 1.31 [-31.19](10.91) 1.10 [-10.13]( 1.12) 0.71 [ 28.69]( 3.69) 0.70 [ 29.59]( 3.58) 0.84 [ 15.65]( 4.74)
1024 1.00 [ -0.00]( 2.28) 1.44 [-44.48](16.05) 1.13 [-13.09]( 0.88) 0.85 [ 15.39]( 6.64) 0.85 [ 15.03]( 0.39) 0.81 [ 19.39]( 1.36)
---
DeathStarBench was run on a 3rd Generation EPYC system (2 x 64C/128T)
with boost enabled and C2 disabled:
==================================================================
Test : DeathStarBench
Units : %diff compared to tip
Interpretation: Higher is better
Statistic : Average throughput
==================================================================
Scaling\cg_mode up smp max concur tasks
2x 1.63% -0.32% -0.03% -0.65% 0.62%
4x -4.94% 1.38% -6.59% 1.09% -6.11%
6x 1.83% 0.16% -0.87% 2.24% -0.23%
---
I'll get to look any deeper into any of the regressions until Thursday
but overall the concur mode seems good in my testing for most part.
--
Thanks and Regards,
Prateek
^ permalink raw reply
* Re: [PATCH v3 2/4] mm/zswap: Implement proactive writeback
From: YoungJun Park @ 2026-06-09 4:19 UTC (permalink / raw)
To: Yosry Ahmed
Cc: Shakeel Butt, Hao Jia, Johannes Weiner, mhocko, tj, mkoutny,
roman.gushchin, Nhat Pham, akpm, chengming.zhou, muchun.song,
cgroups, linux-mm, linux-kernel, linux-doc, Hao Jia, chrisl,
kasong, baoquan.he
In-Reply-To: <CAO9r8zPvCaCqvoUhPdAN5Oi_Sj0mK-t7DJhOOz3Xf1DT-Wrgcw@mail.gmail.com>
On Mon, Jun 08, 2026 at 03:27:07PM -0700, Yosry Ahmed wrote:
+Chris +Kairui +Baoquan
Hello
Thanks for inviting me to the discussion, Shakeel.
> > > > Youngjun is working on swap tiers. At the moment he is more interested in
> > > > allowing a specific swap device to a memcg or not. I can imagine in future there
> > > > will be use-cases where there will be a need to demote data on higher tier swap
> > > > to lower tier swap. What would be the appropriate interface?
Speaking of my work on swap tiers, I recently submitted a patch and am
currently considering memcg integration:
https://lore.kernel.org/linux-mm/20260527062247.3440692-1-youngjun.park@lge.com/
The future use-cases imagined above seem to align with this
direction. (BTW, I am currently waiting for reviews/feedback from the memcg
folks on this patch. Any reviews would be highly appreciated!)
We could potentially assign a target tier
for writeback within the existing memory.zswap.writeback interface.
For instance, '0' could mean disabled, while non-zero values could represent
specific tiers, which would maintain backward compatibility with the current
version. Alternatively, if zswap is treated as the default top tier,
the `memory.swap.tiers` interface could potentially replace `memory.zswap.writeback`.
Furthermore, this could be expanded so that each swap tier can demote data
user-triggered demotion between swap tiers.
Based on the current patch's ideas combined with my swap tiers concept:
Assuming a hierarchy like:
zswap -> tier1 (SSD swap) -> tier2 (HDD swap) -> tier3 (Network swap)
We could configure the active tiers via a setting like `memory.swap.tiers`
(tier2 enabled, tier3 enabled).
For example, the concept of `echo "100M zswap_writeback_only > memory.reclaim"`
could be extended. A user could run `echo "100M tier2 > memory.reclaim"`
to explicitly trigger demotion from tier2 to tier3.
(BTW, if we combine these features, my personal preference for the keyword
format would be `<size> <demote_prefix><tier_name>`. I think it would be
better to explicitly indicate that it is a swap demotion by using a specific
prefix followed by the tier name.
Or make demote prefix another key is also possible)
So, the whole picture would look something like this:
- memory.swap.tiers : Interface for configuring the tier mask.
- memory.reclaim : Entry point for user-triggered demotion.
> > > Things will probably get more
> > > blurry with memory tiers and compressed memory nodes though.
> >
> > I think there will still be distinction between byte addressable and fault on
> > access devices.
>
> Yeah, I think it makes sense to define "swap" as fault on access
> (zswap, SSD, etc), and memory tiers as byte-addressable (even if you
> put an SSD behind CXL and make it byte-addressable). But I also
> remember seeing discussions about unifying memory tiers and swap in a
> way, and it makes sense from a reclaim perspective (swap or demote
> first?).
> > > > will be use-cases where there will be a need to demote data on higher tier swap
> > > > to lower tier swap. What would be the appropriate interface?
> > > >
> > > > BTW does zswap folks think of zswap as a top swap tier or something different?
> > >
> > > I haven't been following the swap tiers work closely, but personally I
> > > do think of zswap as a top swap tier.
Regarding zswap's position, I agree it needs to be defined as the default,
top-most tier in swap_tier. In my early RFC, I allocated a separate tier
specifically for zswap:
(https://lore.kernel.org/linux-mm/20251109124947.1101520-3-youngjun.park@lge.com/)
> > Same for me though I imagine swap tiers would introduce some duplication i.e.
> > different way (interface) to set limits for swap tiers for a given memcg.
> >
I also agree with the concern about interface duplication. We will eventually
need a mechanism to control swap amounts per tier, which requires thinking
about its relationship with swap.max. (I raised this as an open question in
my early RFC).
https://lore.kernel.org/linux-mm/20251109124947.1101520-1-youngjun.park@lge.com/
(Further Discussion and Open Questions Part)
However, since this feature is necessary anyway, wouldn't the proposed
interface be acceptable without causing conflicts at this early stage?
Additionally, `memory.zswap.writeback` seems redundant. Restricting a cgroup
to only use the zswap tier (assuming it's the first tier) is practically
identical to disabling `memory.zswap.writeback` (correct me if I'm wrong).
But there is no problem to integrate it as I think
e.g `memory.zswap.writeback` could internally act as an alias for setting `memory.swap.tier` to 'zswap only'.
BR,
Youngjun Park
^ permalink raw reply
* Re: [PATCH v5 9/9] mm: switch deferred split shrinker to list_lru
From: Lance Yang @ 2026-06-09 3:20 UTC (permalink / raw)
To: hannes
Cc: lance.yang, baolin.wang, akpm, david, ljs, shakeel.butt, mhocko,
david, roman.gushchin, muchun.song, qi.zheng, yosry.ahmed, ziy,
liam, usama.arif, kas, vbabka, ryncsn, zaslonko, gor, baohua,
dev.jain, npache, ryan.roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <ah3MuK3GuimKVORB@cmpxchg.org>
On Mon, Jun 01, 2026 at 02:17:28PM -0400, Johannes Weiner wrote:
>On Mon, Jun 01, 2026 at 09:21:35PM +0800, Lance Yang wrote:
>>
>> On Wed, May 27, 2026 at 04:45:16PM -0400, Johannes Weiner wrote:
>> [...]
>> >diff --git a/mm/swap_state.c b/mm/swap_state.c
>> >index 04f5ce992401..9c3a5cf99778 100644
>> >--- a/mm/swap_state.c
>> >+++ b/mm/swap_state.c
>> >@@ -465,6 +465,16 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
>> > return ERR_PTR(-ENOMEM);
>> > }
>> >
>>
>> Shouldn't this be limited to anon swapin?
>>
>> e.g. vmf && vma_is_anonymous(vmf->vma)
>>
>> >+ if (order > 1 && folio_memcg_alloc_deferred(folio)) {
>>
>> __swap_cache_alloc() is also used by shmem direct swapin, so shmem can
>> get here too when handling a large swap entry:
>>
>> shmem_get_folio_gfp()
>> shmem_swapin_folio()
>> shmem_swap_alloc_folio()
>> swapin_sync()
>> swap_cache_alloc_folio()
>> __swap_cache_alloc()
>> folio_memcg_alloc_deferred()
>
>Good catch, I think you're right. I shouldn't have dismissed that
>branch due to "/* Direct swapin skipping swap cache & readahead */"
>
>> @Baolin please correct me if I got it wrong :)
>>
>> folio_memcg_alloc_deferred() itself doesn't filter shmem out either; it
>> only allocates the memcg list_lru metadata for deferred_split_lru:
>>
>> int folio_memcg_alloc_deferred(struct folio *folio)
>> {
>> if (mem_cgroup_disabled())
>> return 0;
>> return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
>> }
>>
>> Since deferred_split_lru only queues anon large folios, doing this for
>> shmem swapin doesn't buy us anything :)
>
>Yes, agreed. I don't think it's a big deal / show stopper in terms of
>user-visible effect, but of course still worth fixing.
>
>I'll send a follow-up patch.
Thanks.
Looks like this has already landed in mm-stable. If you're okay with it,
I can send the follow-up.
From: Lance Yang <lance.yang@linux.dev>
Date: Tue, 9 Jun 2026 10:56:45 +0800
Subject: [PATCH] mm: prepare deferred split metadata only for anon swapin
__swap_cache_alloc() prepares deferred split metadata for large swapcache
folios.
That also covers shmem swapin, because shmem_swap_alloc_folio() can call
swapin_sync() with a large order[1]. But shmem folios are not queued on
the deferred split queue, so preparing the metadata doesn't buy us
anything there.
So let's limit it to anon swapin.
[1] https://lore.kernel.org/all/20260601132135.14272-1-lance.yang@linux.dev/
Signed-off-by: Lance Yang <lance.yang@linux.dev>
---
mm/swap_state.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 9c3a5cf99778..7adac957c2b8 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -465,7 +465,8 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
return ERR_PTR(-ENOMEM);
}
- if (order > 1 && folio_memcg_alloc_deferred(folio)) {
+ if (order > 1 && vma && vma_is_anonymous(vma) &&
+ folio_memcg_alloc_deferred(folio)) {
spin_lock(&ci->lock);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
spin_unlock(&ci->lock);
--
2.39.3 (Apple Git-146)
^ permalink raw reply related
* Re: [PATCH v3 1/4] mm/zswap: Make shrink_worker writeback cursor per-memcg
From: Hao Jia @ 2026-06-09 3:18 UTC (permalink / raw)
To: Nhat Pham, Yosry Ahmed, shakeel.butt
Cc: akpm, tj, hannes, mhocko, mkoutny, chengming.zhou, muchun.song,
roman.gushchin, cgroups, linux-mm, linux-kernel, linux-doc,
Hao Jia
In-Reply-To: <CAKEwX=MCFbsh9ndBtR0-bGRr_=v-6bBwTo=muzd9ZSD-LAK1nQ@mail.gmail.com>
On 2026/6/9 02:01, Nhat Pham wrote:
> On Mon, Jun 8, 2026 at 9:48 AM Yosry Ahmed <yosry@kernel.org> wrote:
>>
>>> But OTOH, this does seem like a recipe for inefficient reclaim. We
>>> might exhaust hotter memory of a cgroup while sparing colder memory of
>>> another cgroup... But maybe if they're all cold anyway, then who
>>> cares, and eventually you'll get to the cold stuff of other child?
>>
>> Forgot to respond to this part, the unfairness is limited to the batch
>> size per-invocation, so it should be fine as long as you don't divide
>> the amount over 100 iterations for some reason. Also yes, all memory
>> in zswap is cold, the relative coldness is not that important (e.g.
>> compared to relative coldness during reclaim).
>
> Ok then yeah, I think we should shelve per-memcg cursor for the next
> version. Down the line, if we have more data that unfairness is an
> issue, we can always fix it. One step at a time :)
Thanks a lot to Yosry, Nhat, and Shakeel for the great suggestions!
Let me summarize what I plan to do in the next version to make sure we
are on the same page:
- Drop the per-memcg cursor and keep the root cgroup cursor
(zswap_next_shrink) logic intact.
- Stick to using the zswap_writeback_only key, and change the
proactive writeback size to use the compressed size.
- Consolidate and reuse the logic between shrink_worker() and
shrink_memcg(). Enable batch writeback in the shrink_worker() path,
while keeping the writeback behavior in the zswap_store() path unchanged.
Please let me know if I missed or misunderstood anything. Thanks again
for clearing things up!
Thanks，
Hao
^ permalink raw reply
* [tj-cgroup:for-7.2] BUILD SUCCESS a99ce697ea5e27b867c9ba4ee55fa5ba3b8d1188
From: kernel test robot @ 2026-06-09 2:58 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-7.2
branch HEAD: a99ce697ea5e27b867c9ba4ee55fa5ba3b8d1188 cgroup: Migrate tasks to the root css when a controller is rebound
elapsed time: 8964m
configs tested: 86
configs skipped: 9
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-16.1.0
alpha allyesconfig gcc-16.1.0
arc allmodconfig gcc-16.1.0
arc allnoconfig gcc-16.1.0
arc allyesconfig gcc-16.1.0
arm allnoconfig clang-23
arm allyesconfig gcc-16.1.0
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-16.1.0
csky allmodconfig gcc-16.1.0
csky allnoconfig gcc-16.1.0
csky randconfig-001-20260609 gcc-9.5.0
csky randconfig-002-20260609 gcc-11.5.0
hexagon allmodconfig clang-23
hexagon allnoconfig clang-23
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001-20260609 gcc-14
i386 buildonly-randconfig-002-20260609 gcc-14
i386 buildonly-randconfig-003-20260609 clang-22
i386 buildonly-randconfig-004-20260609 clang-22
i386 buildonly-randconfig-005-20260609 gcc-14
i386 buildonly-randconfig-006-20260609 clang-22
i386 randconfig-011-20260609 clang-22
i386 randconfig-012-20260609 clang-22
i386 randconfig-013-20260609 clang-22
i386 randconfig-014-20260609 gcc-14
i386 randconfig-015-20260609 clang-22
i386 randconfig-016-20260609 clang-22
i386 randconfig-017-20260609 gcc-14
loongarch allmodconfig clang-19
loongarch allnoconfig clang-20
m68k allmodconfig gcc-16.1.0
m68k allnoconfig gcc-16.1.0
m68k allyesconfig gcc-16.1.0
microblaze allnoconfig gcc-16.1.0
microblaze allyesconfig gcc-16.1.0
mips allmodconfig gcc-16.1.0
mips allnoconfig gcc-16.1.0
mips allyesconfig gcc-16.1.0
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig gcc-11.5.0
openrisc allmodconfig gcc-16.1.0
openrisc allnoconfig gcc-16.1.0
openrisc defconfig gcc-16.1.0
parisc allmodconfig gcc-16.1.0
parisc allnoconfig gcc-16.1.0
parisc allyesconfig gcc-16.1.0
parisc defconfig gcc-16.1.0
powerpc allmodconfig gcc-16.1.0
powerpc allnoconfig gcc-16.1.0
powerpc mpc834x_itxgp_defconfig clang-23
riscv allmodconfig clang-23
riscv allnoconfig gcc-16.1.0
riscv allyesconfig clang-23
riscv defconfig clang-23
riscv randconfig-002-20260609 clang-23
s390 allmodconfig clang-23
s390 allnoconfig clang-23
s390 allyesconfig gcc-16.1.0
s390 defconfig clang-18
s390 randconfig-001-20260609 clang-23
s390 randconfig-002-20260609 gcc-16.1.0
sh allmodconfig gcc-16.1.0
sh allnoconfig gcc-16.1.0
sh allyesconfig gcc-16.1.0
sh randconfig-001-20260609 gcc-13.4.0
sh randconfig-002-20260609 gcc-11.5.0
sparc allnoconfig gcc-16.1.0
sparc defconfig gcc-16.1.0
sparc randconfig-001-20260609 gcc-16.1.0
sparc64 allmodconfig clang-20
sparc64 randconfig-001-20260609 clang-23
um allmodconfig clang-23
um allnoconfig clang-16
um allyesconfig gcc-14
um randconfig-002-20260609 gcc-12
x86_64 allmodconfig clang-22
x86_64 allnoconfig clang-22
x86_64 allyesconfig clang-22
x86_64 rhel-9.4-rust clang-22
xtensa allnoconfig gcc-16.1.0
xtensa allyesconfig gcc-16.1.0
xtensa randconfig-001-20260609 gcc-16.1.0
xtensa randconfig-002-20260609 gcc-8.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* [tj-cgroup:for-7.1-fixes] BUILD SUCCESS 57aff991119693e09b414aff3267c0eae5e81da0
From: kernel test robot @ 2026-06-09 2:58 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-7.1-fixes
branch HEAD: 57aff991119693e09b414aff3267c0eae5e81da0 cgroup/cpuset: Change Ridong's email
elapsed time: 8965m
configs tested: 66
configs skipped: 10
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-16.1.0
alpha allyesconfig gcc-16.1.0
arc allmodconfig gcc-16.1.0
arc allnoconfig gcc-16.1.0
arc allyesconfig gcc-16.1.0
arm allnoconfig clang-23
arm allyesconfig gcc-16.1.0
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-16.1.0
csky allmodconfig gcc-16.1.0
csky allnoconfig gcc-16.1.0
csky randconfig-001-20260609 gcc-9.5.0
csky randconfig-002-20260609 gcc-11.5.0
hexagon allmodconfig clang-23
hexagon allnoconfig clang-23
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allyesconfig gcc-14
loongarch allmodconfig clang-19
loongarch allnoconfig clang-20
m68k allmodconfig gcc-16.1.0
m68k allnoconfig gcc-16.1.0
m68k allyesconfig gcc-16.1.0
microblaze allnoconfig gcc-16.1.0
microblaze allyesconfig gcc-16.1.0
mips allmodconfig gcc-16.1.0
mips allnoconfig gcc-16.1.0
mips allyesconfig gcc-16.1.0
mips jazz_defconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig gcc-11.5.0
openrisc allmodconfig gcc-16.1.0
openrisc allnoconfig gcc-16.1.0
openrisc defconfig gcc-16.1.0
parisc allmodconfig gcc-16.1.0
parisc allnoconfig gcc-16.1.0
parisc allyesconfig gcc-16.1.0
parisc defconfig gcc-16.1.0
powerpc allmodconfig gcc-16.1.0
powerpc allnoconfig gcc-16.1.0
riscv allmodconfig clang-23
riscv allnoconfig gcc-16.1.0
riscv allyesconfig clang-23
riscv defconfig clang-23
s390 allmodconfig clang-23
s390 allnoconfig clang-23
s390 allyesconfig gcc-16.1.0
s390 defconfig clang-18
s390 randconfig-002-20260609 gcc-16.1.0
sh allmodconfig gcc-16.1.0
sh allnoconfig gcc-16.1.0
sh allyesconfig gcc-16.1.0
sh randconfig-001-20260609 gcc-13.4.0
sh randconfig-002-20260609 gcc-11.5.0
sparc allnoconfig gcc-16.1.0
sparc defconfig gcc-16.1.0
sparc64 allmodconfig clang-20
um allmodconfig clang-23
um allnoconfig clang-16
um allyesconfig gcc-14
x86_64 allmodconfig clang-22
x86_64 allnoconfig clang-22
x86_64 allyesconfig clang-22
x86_64 rhel-9.4-rust clang-22
xtensa allnoconfig gcc-16.1.0
xtensa allyesconfig gcc-16.1.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH] mm: constify oom_control, scan_control, and alloc_context nodemask
From: Zi Yan @ 2026-06-09 1:44 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, linux-kernel, cgroups, kernel-team, longman, chenridong,
akpm, david, ljs, liam, vbabka, rppt, surenb, mhocko, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
rientjes, chrisl, shikemeng, nphamcs, baoquan.he, youngjun.park,
tj, hannes, mkoutny, jackmanb
In-Reply-To: <20260609002919.3967782-1-gourry@gourry.net>
On 8 Jun 2026, at 20:29, Gregory Price wrote:
> The nodemasks in these structures may come from a variety of sources,
> including tasks and cpusets - and should never be modified by any code
> when being passed around inside another context.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> include/linux/cpuset.h | 4 ++--
> include/linux/mm.h | 4 ++--
> include/linux/mmzone.h | 6 +++---
> include/linux/oom.h | 2 +-
> include/linux/swap.h | 2 +-
> kernel/cgroup/cpuset.c | 2 +-
> mm/internal.h | 2 +-
> mm/mmzone.c | 5 +++--
> mm/page_alloc.c | 6 +++---
> mm/show_mem.c | 9 ++++++---
> mm/vmscan.c | 6 +++---
> 11 files changed, 26 insertions(+), 22 deletions(-)
>
LGTM and it compiles. As long as Sashiko does not complain, feel free to
add:
Acked-by: Zi Yan <ziy@nvidia.com>
Best Regards,
Yan, Zi
^ permalink raw reply
* [PATCH] mm: constify oom_control, scan_control, and alloc_context nodemask
From: Gregory Price @ 2026-06-09 0:29 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, cgroups, kernel-team, longman, chenridong, akpm,
david, ljs, liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, rientjes,
chrisl, shikemeng, nphamcs, baoquan.he, youngjun.park, tj, hannes,
mkoutny, jackmanb, ziy
The nodemasks in these structures may come from a variety of sources,
including tasks and cpusets - and should never be modified by any code
when being passed around inside another context.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/cpuset.h | 4 ++--
include/linux/mm.h | 4 ++--
include/linux/mmzone.h | 6 +++---
include/linux/oom.h | 2 +-
include/linux/swap.h | 2 +-
kernel/cgroup/cpuset.c | 2 +-
mm/internal.h | 2 +-
mm/mmzone.c | 5 +++--
mm/page_alloc.c | 6 +++---
mm/show_mem.c | 9 ++++++---
mm/vmscan.c | 6 +++---
11 files changed, 26 insertions(+), 22 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 65d76a38974b..a80d38e752d2 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -83,7 +83,7 @@ extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
#define cpuset_current_mems_allowed (current->mems_allowed)
void cpuset_init_current_mems_allowed(void);
-int cpuset_nodemask_valid_mems_allowed(nodemask_t *nodemask);
+int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask);
extern bool cpuset_current_node_allowed(int node, gfp_t gfp_mask);
@@ -224,7 +224,7 @@ static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
#define cpuset_current_mems_allowed (node_states[N_MEMORY])
static inline void cpuset_init_current_mems_allowed(void) {}
-static inline int cpuset_nodemask_valid_mems_allowed(nodemask_t *nodemask)
+static inline int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
{
return 1;
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbdd..2101e5205fc0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -4042,7 +4042,7 @@ extern int __meminit early_pfn_to_nid(unsigned long pfn);
extern void mem_init(void);
extern void __init mmap_init(void);
-extern void __show_mem(unsigned int flags, nodemask_t *nodemask, int max_zone_idx);
+extern void __show_mem(unsigned int flags, const nodemask_t *nodemask, int max_zone_idx);
static inline void show_mem(void)
{
__show_mem(0, NULL, MAX_NR_ZONES - 1);
@@ -4052,7 +4052,7 @@ extern void si_meminfo(struct sysinfo * val);
extern void si_meminfo_node(struct sysinfo *val, int nid);
extern __printf(3, 4)
-void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...);
+void warn_alloc(gfp_t gfp_mask, const nodemask_t *nodemask, const char *fmt, ...);
extern void setup_per_cpu_pageset(void);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ca2712187147..919be0d9b4fa 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1815,7 +1815,7 @@ static inline int zonelist_node_idx(const struct zoneref *zoneref)
struct zoneref *__next_zones_zonelist(struct zoneref *z,
enum zone_type highest_zoneidx,
- nodemask_t *nodes);
+ const nodemask_t *nodes);
/**
* next_zones_zonelist - Returns the next zone at or below highest_zoneidx within the allowed nodemask using a cursor within a zonelist as a starting point
@@ -1834,7 +1834,7 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,
*/
static __always_inline struct zoneref *next_zones_zonelist(struct zoneref *z,
enum zone_type highest_zoneidx,
- nodemask_t *nodes)
+ const nodemask_t *nodes)
{
if (likely(!nodes && zonelist_zone_idx(z) <= highest_zoneidx))
return z;
@@ -1860,7 +1860,7 @@ static __always_inline struct zoneref *next_zones_zonelist(struct zoneref *z,
*/
static inline struct zoneref *first_zones_zonelist(struct zonelist *zonelist,
enum zone_type highest_zoneidx,
- nodemask_t *nodes)
+ const nodemask_t *nodes)
{
return next_zones_zonelist(zonelist->_zonerefs,
highest_zoneidx, nodes);
diff --git a/include/linux/oom.h b/include/linux/oom.h
index 7b02bc1d0a7e..00da05d227e6 100644
--- a/include/linux/oom.h
+++ b/include/linux/oom.h
@@ -30,7 +30,7 @@ struct oom_control {
struct zonelist *zonelist;
/* Used to determine mempolicy */
- nodemask_t *nodemask;
+ const nodemask_t *nodemask;
/* Memory cgroup in which oom is invoked, or NULL for global oom */
struct mem_cgroup *memcg;
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8f0f68e245ba..bf76356d94fe 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -351,7 +351,7 @@ extern void swap_setup(void);
/* linux/mm/vmscan.c */
extern unsigned long zone_reclaimable_pages(struct zone *zone);
extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone_idx);
#define MEMCG_RECLAIM_MAY_SWAP (1 << 1)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 5c33ab20cc20..536ce591c7ba 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4153,7 +4153,7 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
*
* Are any of the nodes in the nodemask allowed in current->mems_allowed?
*/
-int cpuset_nodemask_valid_mems_allowed(nodemask_t *nodemask)
+int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
{
return nodes_intersects(*nodemask, current->mems_allowed);
}
diff --git a/mm/internal.h b/mm/internal.h
index 181e79f1d6a2..a2ef9512b5bc 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -668,7 +668,7 @@ void page_alloc_sysctl_init(void);
*/
struct alloc_context {
struct zonelist *zonelist;
- nodemask_t *nodemask;
+ const nodemask_t *nodemask;
struct zoneref *preferred_zoneref;
int migratetype;
diff --git a/mm/mmzone.c b/mm/mmzone.c
index 0c8f181d9d50..59dc3f2076a6 100644
--- a/mm/mmzone.c
+++ b/mm/mmzone.c
@@ -43,7 +43,8 @@ struct zone *next_zone(struct zone *zone)
return zone;
}
-static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes)
+static inline int zref_in_nodemask(struct zoneref *zref,
+ const nodemask_t *nodes)
{
#ifdef CONFIG_NUMA
return node_isset(zonelist_node_idx(zref), *nodes);
@@ -55,7 +56,7 @@ static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes)
/* Returns the next zone at or below highest_zoneidx in a zonelist */
struct zoneref *__next_zones_zonelist(struct zoneref *z,
enum zone_type highest_zoneidx,
- nodemask_t *nodes)
+ const nodemask_t *nodes)
{
/*
* Find the next suitable zone to use for the allocation.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 81a9d4d1e6c0..586524bbde9c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3979,7 +3979,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
return NULL;
}
-static void warn_alloc_show_mem(gfp_t gfp_mask, nodemask_t *nodemask)
+static void warn_alloc_show_mem(gfp_t gfp_mask, const nodemask_t *nodemask)
{
unsigned int filter = SHOW_MEM_FILTER_NODES;
@@ -3999,7 +3999,7 @@ static void warn_alloc_show_mem(gfp_t gfp_mask, nodemask_t *nodemask)
mem_cgroup_show_protected_memory(NULL);
}
-void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...)
+void warn_alloc(gfp_t gfp_mask, const nodemask_t *nodemask, const char *fmt, ...)
{
struct va_format vaf;
va_list args;
@@ -4687,7 +4687,7 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac)
return false;
}
-static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
+static void check_alloc_stall_warn(gfp_t gfp_mask, const nodemask_t *nodemask,
unsigned int order, unsigned long alloc_start_time)
{
static DEFINE_SPINLOCK(alloc_stall_lock);
diff --git a/mm/show_mem.c b/mm/show_mem.c
index 43aca5a2ac99..1b721a8ade67 100644
--- a/mm/show_mem.c
+++ b/mm/show_mem.c
@@ -116,7 +116,8 @@ void si_meminfo_node(struct sysinfo *val, int nid)
* Determine whether the node should be displayed or not, depending on whether
* SHOW_MEM_FILTER_NODES was passed to show_free_areas().
*/
-static bool show_mem_node_skip(unsigned int flags, int nid, nodemask_t *nodemask)
+static bool show_mem_node_skip(unsigned int flags, int nid,
+ const nodemask_t *nodemask)
{
if (!(flags & SHOW_MEM_FILTER_NODES))
return false;
@@ -177,7 +178,8 @@ static bool node_has_managed_zones(pg_data_t *pgdat, int max_zone_idx)
* SHOW_MEM_FILTER_NODES: suppress nodes that are not allowed by current's
* cpuset.
*/
-static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
+static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
+ int max_zone_idx)
{
unsigned long free_pcp = 0;
int cpu, nid;
@@ -402,7 +404,8 @@ static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_z
show_swap_cache_info();
}
-void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
+void __show_mem(unsigned int filter, const nodemask_t *nodemask,
+ int max_zone_idx)
{
unsigned long total = 0, reserved = 0, highmem = 0;
struct zone *zone;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index e8a90911bf88..47d3f3361fb9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -79,7 +79,7 @@ struct scan_control {
* Nodemask of nodes allowed by the caller. If NULL, all nodes
* are scanned.
*/
- nodemask_t *nodemask;
+ const nodemask_t *nodemask;
/*
* The memory cgroup that hit its limit and as a result is the
@@ -6599,7 +6599,7 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)
* happens, the page allocator should not consider triggering the OOM killer.
*/
static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
- nodemask_t *nodemask)
+ const nodemask_t *nodemask)
{
struct zoneref *z;
struct zone *zone;
@@ -6679,7 +6679,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
}
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
--
2.54.0
^ permalink raw reply related
* Re: [PATCH v3 2/4] mm/zswap: Implement proactive writeback
From: Yosry Ahmed @ 2026-06-08 22:27 UTC (permalink / raw)
To: Shakeel Butt
Cc: Hao Jia, Johannes Weiner, mhocko, tj, mkoutny, roman.gushchin,
Nhat Pham, akpm, chengming.zhou, muchun.song, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia, youngjun.park
In-Reply-To: <aictKA0XWMWbxFdN@linux.dev>
> > > Youngjun is working on swap tiers. At the moment he is more interested in
> > > allowing a specific swap device to a memcg or not. I can imagine in future there
> > > will be use-cases where there will be a need to demote data on higher tier swap
> > > to lower tier swap. What would be the appropriate interface?
> > >
> > > BTW does zswap folks think of zswap as a top swap tier or something different?
> >
> > I haven't been following the swap tiers work closely, but personally I
> > do think of zswap as a top swap tier.
>
> Same for me though I imagine swap tiers would introduce some duplication i.e.
> different way (interface) to set limits for swap tiers for a given memcg.
>
> > Things will probably get more
> > blurry with memory tiers and compressed memory nodes though.
>
> I think there will still be distinction between byte addressable and fault on
> access devices.
Yeah, I think it makes sense to define "swap" as fault on access
(zswap, SSD, etc), and memory tiers as byte-addressable (even if you
put an SSD behind CXL and make it byte-addressable). But I also
remember seeing discussions about unifying memory tiers and swap in a
way, and it makes sense from a reclaim perspective (swap or demote
first?).
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox