* [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste
@ 2026-07-15 10:10 Vlastimil Babka (SUSE)
2026-07-15 10:10 ` [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling Vlastimil Babka (SUSE)
` (12 more replies)
0 siblings, 13 replies; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
The recent fixes for objext array handling inspired me to look into this
finally. It's been bothering me that the memory usage of struct
slabobj_ext depend only on config options and not whether the fields are
actually used. So with both CONFIG_MEMCG=y and
CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref
field. And thus:
1) Having memory allocation profiling config-enabled but not
boot-enabled means wasted memory on unused codetag_refs. This makes
it less suitable for a general distro config and the page allocator
side doesn't suffer from this, only slab and percpu.
2) Complementary, with memory allocation profiling enabled, there are
caches/slabs that don't need the objcg field, so memory is wasted on
those.
This series should solve the point 1) fully for slab, pcpuobj_ext
handling can be perhaps improved similarly, haven't looked into that.
For 2) it avoids allocating objcg fields for KMALLOC_NORMAL caches where
we know they are not necessary because kmalloc() with __GFP_ACCOUNT will
pick a KMALLOC_CGROUP type.
The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT
and then we know objcg fields are always needed. But also they can be
created without SLAB_ACCOUNT and then some allocations have
__GFP_ACCOUNT and some not and we don't know that in advance.
A possible future solution is to introduce e.g. SLAB_MAYBE_ACCOUNT, add
it to caches where we know __GFP_ACCOUNT is used, and only honour
__GFP_ACCOUNT for those, while warning for an unexpected usage
elsewhere.
Only lightly tested, need to run at least some microbenchmarks to see if
the now somewhat more complicated access to objcg is visible or not.
Based on slab/for-next-fixes
Git branch: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/objext_split
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
Vlastimil Babka (SUSE) (12):
mm/slab: skip kfence objects in allocation profiling
mm/slab: remove objs_per_slab()
mm: move struct slabobj_ext to mm/slab.h
mm/slab: make slab_obj_ext() determine object index
mm/slab: abstract slabobj_ext.objcg access
mm/slab: abstract slabobj_ext.ref access
mm/slab: replace slab.stride with obj_exts_in_object
mm/slab: change struct slabobj_ext to a union
mm/slab: introduce slab_obj_ext_has_codetag()
mm/slab: reduce slabobj_ext memory with allocation profiling disabled
mm/slab: add slab_needs_objcg() helper
mm/slab: stop allocating objcg pointers when unnecessary
include/linux/memcontrol.h | 13 ----
mm/kfence/core.c | 5 +-
mm/kfence/kfence_test.c | 2 +-
mm/memcontrol.c | 41 ++++++-----
mm/slab.h | 169 ++++++++++++++++++++++++++++++++++++---------
mm/slub.c | 162 +++++++++++++++++++++++++++----------------
6 files changed, 267 insertions(+), 125 deletions(-)
---
base-commit: d9e6a7623938968e3752b67e37eaff097e559a54
change-id: 20260714-b4-objext_split-da82426257d5
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-15 16:02 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 02/12] mm/slab: remove objs_per_slab() Vlastimil Babka (SUSE)
` (11 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
struct kfence_metadata only has obj_exts with CONFIG_MEMCG. If it's
enabled, it does also work for allocation profiling, but there's little
value recording tags for KFENCE objects. Furthermore it would complicate
the upcoming changes, so just skip them in the slab hooks.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slub.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/mm/slub.c b/mm/slub.c
index 0337e60db5ac..a4be70d080fb 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
return;
+ if (is_kfence_address(object))
+ return;
+
slab = virt_to_slab(object);
obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
/*
@@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
for (i = 0; i < objects; i++) {
unsigned int off = obj_to_index(s, slab, p[i]);
+ if (is_kfence_address(p[i]))
+ continue;
+
alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
}
put_slab_obj_exts(obj_exts);
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 02/12] mm/slab: remove objs_per_slab()
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
2026-07-15 10:10 ` [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-15 16:13 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h Vlastimil Babka (SUSE)
` (10 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
The function has an unused kmem_cache argument and almost nothing uses
it anyway; doing slab->objects is simpler. Remove it with the last two
users. KUNIT_EXPECT_EQ() needs a cast to avoid "error: ‘typeof’ applied
to a bit-field" but we don't need to keep a wrapper just for that.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/kfence/kfence_test.c | 2 +-
mm/slab.h | 6 ------
mm/slub.c | 3 +--
3 files changed, 2 insertions(+), 9 deletions(-)
diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
index de2d0f7d62b1..9867c03ef0ae 100644
--- a/mm/kfence/kfence_test.c
+++ b/mm/kfence/kfence_test.c
@@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat
* memcg accounting works correctly.
*/
KUNIT_EXPECT_EQ(test, obj_to_index(s, slab, alloc), 0U);
- KUNIT_EXPECT_EQ(test, objs_per_slab(s, slab), 1);
+ KUNIT_EXPECT_EQ(test, ((unsigned int)slab->objects), 1);
if (policy == ALLOCATE_ANY)
return alloc;
diff --git a/mm/slab.h b/mm/slab.h
index f5e336b6b6b0..01535e1e2d3c 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,
return __obj_to_index(cache, slab_address(slab), obj);
}
-static inline int objs_per_slab(const struct kmem_cache *cache,
- const struct slab *slab)
-{
- return slab->objects;
-}
-
/*
* State of the slab allocator.
*
diff --git a/mm/slub.c b/mm/slub.c
index a4be70d080fb..9e25f2dce7a6 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2127,7 +2127,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
gfp_t gfp, unsigned int alloc_flags)
{
const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
- unsigned int objects = objs_per_slab(s, slab);
bool new_slab = alloc_flags & SLAB_ALLOC_NEW_SLAB;
unsigned long new_exts;
unsigned long old_exts;
@@ -2183,7 +2182,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
#endif
retry:
old_exts = READ_ONCE(slab->obj_exts);
- handle_failed_objexts_alloc(old_exts, vec, objects);
+ handle_failed_objexts_alloc(old_exts, vec, slab->objects);
if (new_slab) {
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
2026-07-15 10:10 ` [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling Vlastimil Babka (SUSE)
2026-07-15 10:10 ` [PATCH RFC 02/12] mm/slab: remove objs_per_slab() Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 0:56 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index Vlastimil Babka (SUSE)
` (9 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
Users of include/linux/memcontrol.h don't need to see this internal
structure. Further changes to the struct will reduce recompiling.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
include/linux/memcontrol.h | 13 -------------
mm/slab.h | 13 +++++++++++++
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e1f46a0016fc..93869cc35c25 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1440,19 +1440,6 @@ static inline void mem_cgroup_flush_workqueue(void) { }
static inline int mem_cgroup_init(void) { return 0; }
#endif /* CONFIG_MEMCG */
-/*
- * Extended information for slab objects stored as an array in page->memcg_data
- * if MEMCG_DATA_OBJEXTS is set.
- */
-struct slabobj_ext {
-#ifdef CONFIG_MEMCG
- struct obj_cgroup *objcg;
-#endif
-#ifdef CONFIG_MEM_ALLOC_PROFILING
- union codetag_ref ref;
-#endif
-} __aligned(8);
-
static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
{
struct mem_cgroup *memcg;
diff --git a/mm/slab.h b/mm/slab.h
index 01535e1e2d3c..7bd361447c54 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -549,6 +549,19 @@ static inline bool need_kmalloc_no_objext(void)
return false;
}
+/*
+ * Extended information for slab objects stored as an array in page->memcg_data
+ * if MEMCG_DATA_OBJEXTS is set.
+ */
+struct slabobj_ext {
+#ifdef CONFIG_MEMCG
+ struct obj_cgroup *objcg;
+#endif
+#ifdef CONFIG_MEM_ALLOC_PROFILING
+ union codetag_ref ref;
+#endif
+} __aligned(8);
+
#ifdef CONFIG_SLAB_OBJ_EXT
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (2 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 1:02 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access Vlastimil Babka (SUSE)
` (8 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
All callers perform the same obj_to_index() calculation to pass the
index. Simplify by passing object pointer instead and determining the
index by slab_obj_ext().
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/memcontrol.c | 12 +++---------
mm/slab.h | 14 ++++++++------
mm/slub.c | 22 +++++++---------------
3 files changed, 18 insertions(+), 30 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6dc4888a90f3..4e427286a88a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2865,15 +2865,13 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
*/
unsigned long obj_exts;
struct slabobj_ext *obj_ext;
- unsigned int off;
obj_exts = slab_obj_exts(slab);
if (!obj_exts)
return NULL;
get_slab_obj_exts(obj_exts);
- off = obj_to_index(slab->slab_cache, slab, p);
- obj_ext = slab_obj_ext(slab, obj_exts, off);
+ obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
if (obj_ext->objcg) {
struct obj_cgroup *objcg = obj_ext->objcg;
@@ -3541,7 +3539,6 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
size_t obj_size = obj_full_size(s);
struct obj_cgroup *objcg;
struct slab *slab;
- unsigned long off;
size_t i;
/*
@@ -3616,8 +3613,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
obj_exts = slab_obj_exts(slab);
get_slab_obj_exts(obj_exts);
- off = obj_to_index(s, slab, p[i]);
- obj_ext = slab_obj_ext(slab, obj_exts, off);
+ obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
obj_cgroup_get(objcg);
obj_ext->objcg = objcg;
put_slab_obj_exts(obj_exts);
@@ -3635,10 +3631,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
struct obj_cgroup *objcg;
struct slabobj_ext *obj_ext;
struct obj_stock_pcp *stock;
- unsigned int off;
- off = obj_to_index(s, slab, p[i]);
- obj_ext = slab_obj_ext(slab, obj_exts, off);
+ obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
objcg = obj_ext->objcg;
if (!objcg)
continue;
diff --git a/mm/slab.h b/mm/slab.h
index 7bd361447c54..36d067d6e7c0 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -646,14 +646,16 @@ static inline unsigned int slab_get_stride(struct slab *slab)
* Returns a pointer to the object extension associated with the object.
* Must be called within a section covered by get/put_slab_obj_exts().
*/
-static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
- unsigned long obj_exts,
- unsigned int index)
+static inline struct slabobj_ext *
+slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
+ const void *obj)
{
struct slabobj_ext *obj_ext;
+ unsigned int index;
VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
+ index = obj_to_index(s, slab, obj);
obj_ext = (struct slabobj_ext *)(obj_exts +
slab_get_stride(slab) * index);
return kasan_reset_tag(obj_ext);
@@ -669,9 +671,9 @@ static inline unsigned long slab_obj_exts(struct slab *slab)
return 0;
}
-static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
- unsigned long obj_exts,
- unsigned int index)
+static inline struct slabobj_ext *
+slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
+ const void *obj)
{
return NULL;
}
diff --git a/mm/slub.c b/mm/slub.c
index 9e25f2dce7a6..5e3f53bcd0d3 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2070,11 +2070,10 @@ static inline void mark_obj_codetag_empty(const void *obj)
obj_slab = virt_to_slab(obj);
slab_exts = slab_obj_exts(obj_slab);
if (slab_exts) {
+ struct slabobj_ext *ext;
+
get_slab_obj_exts(slab_exts);
- unsigned int offs = obj_to_index(obj_slab->slab_cache,
- obj_slab, obj);
- struct slabobj_ext *ext = slab_obj_ext(obj_slab,
- slab_exts, offs);
+ ext = slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj);
if (unlikely(is_codetag_empty(&ext->ref))) {
put_slab_obj_exts(slab_exts);
@@ -2362,10 +2361,8 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
* check should be added before alloc_tag_add().
*/
if (obj_exts) {
- unsigned int obj_idx = obj_to_index(s, slab, object);
-
get_slab_obj_exts(obj_exts);
- obj_ext = slab_obj_ext(slab, obj_exts, obj_idx);
+ obj_ext = slab_obj_ext(s, slab, obj_exts, object);
alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size);
put_slab_obj_exts(obj_exts);
} else {
@@ -2386,7 +2383,6 @@ static noinline void
__alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
int objects)
{
- int i;
unsigned long obj_exts;
/* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */
@@ -2398,13 +2394,11 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
return;
get_slab_obj_exts(obj_exts);
- for (i = 0; i < objects; i++) {
- unsigned int off = obj_to_index(s, slab, p[i]);
-
+ for (int i = 0; i < objects; i++) {
if (is_kfence_address(p[i]))
continue;
- alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
+ alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size);
}
put_slab_obj_exts(obj_exts);
}
@@ -2489,7 +2483,6 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
struct kmem_cache *s;
struct page *page;
struct slab *slab;
- unsigned long off;
page = virt_to_page(p);
if (PageLargeKmalloc(page)) {
@@ -2529,8 +2522,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
obj_exts = slab_obj_exts(slab);
if (obj_exts) {
get_slab_obj_exts(obj_exts);
- off = obj_to_index(s, slab, p);
- obj_ext = slab_obj_ext(slab, obj_exts, off);
+ obj_ext = slab_obj_ext(s, slab, obj_exts, p);
if (unlikely(obj_ext->objcg)) {
put_slab_obj_exts(obj_exts);
return true;
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (3 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 1:27 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access Vlastimil Babka (SUSE)
` (7 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
In preparation for changes to the structure, abstract access to the
objcg field with a slab_obj_ext_objcgp() function.
Rename the field to _objcg to make an unexpected direct access a compile
error.
No functional change intended.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/kfence/core.c | 2 +-
mm/memcontrol.c | 23 +++++++++++++++--------
mm/slab.h | 9 ++++++++-
mm/slub.c | 2 +-
4 files changed, 25 insertions(+), 11 deletions(-)
diff --git a/mm/kfence/core.c b/mm/kfence/core.c
index 6577bd76954e..717e8baf7e5d 100644
--- a/mm/kfence/core.c
+++ b/mm/kfence/core.c
@@ -1249,7 +1249,7 @@ void __kfence_free(void *addr)
struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
#ifdef CONFIG_MEMCG
- KFENCE_WARN_ON(meta->obj_exts.objcg);
+ KFENCE_WARN_ON(*slab_obj_ext_objcgp(&meta->obj_exts));
#endif
/*
* If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 4e427286a88a..6303a2b1a9d0 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
*/
unsigned long obj_exts;
struct slabobj_ext *obj_ext;
+ struct obj_cgroup *objcg;
obj_exts = slab_obj_exts(slab);
if (!obj_exts)
@@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
get_slab_obj_exts(obj_exts);
obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
- if (obj_ext->objcg) {
- struct obj_cgroup *objcg = obj_ext->objcg;
-
+ objcg = *slab_obj_ext_objcgp(obj_ext);
+ if (objcg) {
put_slab_obj_exts(obj_exts);
return obj_cgroup_memcg(objcg);
}
@@ -3577,6 +3577,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
unsigned long obj_exts;
struct slabobj_ext *obj_ext;
struct obj_stock_pcp *stock;
+ struct obj_cgroup **objcgp;
slab = virt_to_slab(p[i]);
@@ -3612,10 +3613,15 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
unlock_stock(stock);
obj_exts = slab_obj_exts(slab);
+
get_slab_obj_exts(obj_exts);
+
obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
+ objcgp = slab_obj_ext_objcgp(obj_ext);
+
obj_cgroup_get(objcg);
- obj_ext->objcg = objcg;
+ *objcgp = objcg;
+
put_slab_obj_exts(obj_exts);
}
@@ -3628,16 +3634,17 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
size_t obj_size = obj_full_size(s);
for (int i = 0; i < objects; i++) {
- struct obj_cgroup *objcg;
+ struct obj_cgroup **objcgp, *objcg;
struct slabobj_ext *obj_ext;
struct obj_stock_pcp *stock;
obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
- objcg = obj_ext->objcg;
- if (!objcg)
+ objcgp = slab_obj_ext_objcgp(obj_ext);
+ if (!*objcgp)
continue;
- obj_ext->objcg = NULL;
+ objcg = *objcgp;
+ *objcgp = NULL;
stock = trylock_stock();
__refill_obj_stock(objcg, stock, obj_size, true);
diff --git a/mm/slab.h b/mm/slab.h
index 36d067d6e7c0..789bd292075f 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
*/
struct slabobj_ext {
#ifdef CONFIG_MEMCG
- struct obj_cgroup *objcg;
+ struct obj_cgroup *_objcg;
#endif
#ifdef CONFIG_MEM_ALLOC_PROFILING
union codetag_ref ref;
@@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
return kasan_reset_tag(obj_ext);
}
+#ifdef CONFIG_MEMCG
+static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
+{
+ return &obj_ext->_objcg;
+}
+#endif
+
int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
gfp_t gfp, unsigned int alloc_flags);
diff --git a/mm/slub.c b/mm/slub.c
index 5e3f53bcd0d3..48e10198a3ce 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
if (obj_exts) {
get_slab_obj_exts(obj_exts);
obj_ext = slab_obj_ext(s, slab, obj_exts, p);
- if (unlikely(obj_ext->objcg)) {
+ if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
put_slab_obj_exts(obj_exts);
return true;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (4 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 1:28 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object Vlastimil Babka (SUSE)
` (6 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
In preparation for changes to the structure, abstract access to the ref
field with a slab_obj_ext_codetag_ref() function. Rename the field to
_ctref to make an unexpected direct access a compile error.
No functional change intended.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slab.h | 10 +++++++++-
mm/slub.c | 42 ++++++++++++++++++++++++++++--------------
2 files changed, 37 insertions(+), 15 deletions(-)
diff --git a/mm/slab.h b/mm/slab.h
index 789bd292075f..e3f8e42070f1 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -558,7 +558,7 @@ struct slabobj_ext {
struct obj_cgroup *_objcg;
#endif
#ifdef CONFIG_MEM_ALLOC_PROFILING
- union codetag_ref ref;
+ union codetag_ref _ctref;
#endif
} __aligned(8);
@@ -668,6 +668,14 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
}
#endif
+#ifdef CONFIG_MEM_ALLOC_PROFILING
+static inline union codetag_ref *
+slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
+{
+ return &obj_ext->_ctref;
+}
+#endif
+
int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
gfp_t gfp, unsigned int alloc_flags);
diff --git a/mm/slub.c b/mm/slub.c
index 48e10198a3ce..2bfcabc4c51a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2071,18 +2071,20 @@ static inline void mark_obj_codetag_empty(const void *obj)
slab_exts = slab_obj_exts(obj_slab);
if (slab_exts) {
struct slabobj_ext *ext;
+ union codetag_ref *ref;
get_slab_obj_exts(slab_exts);
ext = slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj);
+ ref = slab_obj_ext_codetag_ref(obj_slab, ext);
- if (unlikely(is_codetag_empty(&ext->ref))) {
+ if (unlikely(is_codetag_empty(ref))) {
put_slab_obj_exts(slab_exts);
return;
}
/* codetag should be NULL here */
- WARN_ON(ext->ref.ct);
- set_codetag_empty(&ext->ref);
+ WARN_ON(ref->ct);
+ set_codetag_empty(ref);
put_slab_obj_exts(slab_exts);
}
}
@@ -2092,19 +2094,22 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)
return cmpxchg(&slab->obj_exts, 0, OBJEXTS_ALLOC_FAIL) == 0;
}
-static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
- struct slabobj_ext *vec, unsigned int objects)
+static inline void handle_failed_objexts_alloc(struct slab *slab,
+ unsigned long obj_exts, struct slabobj_ext *vec)
{
/*
* If vector previously failed to allocate then we have live
* objects with no tag reference. Mark all references in this
* vector as empty to avoid warnings later on.
*/
- if (obj_exts == OBJEXTS_ALLOC_FAIL) {
- unsigned int i;
+ if (obj_exts != OBJEXTS_ALLOC_FAIL)
+ return;
+
+ for (unsigned int i = 0; i < slab->objects; i++) {
+ union codetag_ref *ref = slab_obj_ext_codetag_ref(slab, vec);
- for (i = 0; i < objects; i++)
- set_codetag_empty(&vec[i].ref);
+ set_codetag_empty(ref);
+ vec++;
}
}
@@ -2112,8 +2117,8 @@ static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
static inline void mark_obj_codetag_empty(const void *obj) {}
static inline bool mark_failed_objexts_alloc(struct slab *slab) { return false; }
-static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
- struct slabobj_ext *vec, unsigned int objects) {}
+static inline void handle_failed_objexts_alloc(struct slab *slab,
+ unsigned long obj_exts, struct slabobj_ext *vec) {}
#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */
@@ -2181,7 +2186,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
#endif
retry:
old_exts = READ_ONCE(slab->obj_exts);
- handle_failed_objexts_alloc(old_exts, vec, slab->objects);
+ handle_failed_objexts_alloc(slab, old_exts, vec);
if (new_slab) {
/*
@@ -2361,9 +2366,15 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
* check should be added before alloc_tag_add().
*/
if (obj_exts) {
+ union codetag_ref *ref;
+
get_slab_obj_exts(obj_exts);
+
obj_ext = slab_obj_ext(s, slab, obj_exts, object);
- alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size);
+ ref = slab_obj_ext_codetag_ref(slab, obj_ext);
+
+ alloc_tag_add(ref, current->alloc_tag, s->size);
+
put_slab_obj_exts(obj_exts);
} else {
alloc_tag_set_inaccurate(current->alloc_tag);
@@ -2395,10 +2406,13 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
get_slab_obj_exts(obj_exts);
for (int i = 0; i < objects; i++) {
+ struct slabobj_ext *ext;
+
if (is_kfence_address(p[i]))
continue;
- alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size);
+ ext = slab_obj_ext(s, slab, obj_exts, p[i]);
+ alloc_tag_sub(slab_obj_ext_codetag_ref(slab, ext), s->size);
}
put_slab_obj_exts(obj_exts);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (5 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 1:44 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union Vlastimil Babka (SUSE)
` (5 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
The stride field is used to convert object index to an slabobj_ext so
both compact arrays (kmalloc() or in-slab-leftover) and spread
in-object-padding obj_ext layouts are supported.
In practice thus the stride is always sizeof(slabobj_ext) or s->size.
This simplifies the calculations, but with the upcoming slabobj_ext
handling changes, it will be easier to stop storing the stride and
instead just have a flag whether obj_ext is in the object padding.
obj_exts_in_object() can then rely on this flag and slab_obj_ext()
can use that to determine the stride.
No functional change intended. Performance impact TBD, hopefully
in the noise.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slab.h | 43 ++++++++++++++++++++++++-------------------
mm/slub.c | 42 ++++++++++++++++--------------------------
2 files changed, 40 insertions(+), 45 deletions(-)
diff --git a/mm/slab.h b/mm/slab.h
index e3f8e42070f1..3ad9777ad600 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -81,10 +81,10 @@ struct freelist_counters {
#ifdef CONFIG_64BIT
/*
* Some optimizations use free bits in 'counters' field
- * to save memory. In case ->stride field is not available,
+ * to save memory. If these free bits are not available,
* such optimizations are disabled.
*/
- unsigned int stride;
+ unsigned obj_exts_in_object:1;
#endif
};
};
@@ -617,22 +617,20 @@ static inline void put_slab_obj_exts(unsigned long obj_exts)
}
#ifdef CONFIG_64BIT
-static inline void slab_set_stride(struct slab *slab, unsigned int stride)
+static inline bool obj_exts_in_object(struct slab *slab)
{
- slab->stride = stride;
-}
-static inline unsigned int slab_get_stride(struct slab *slab)
-{
- return slab->stride;
+ /*
+ * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
+ * check the per-slab bit. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
+ * allocations within_slab_leftover are preferred. And those may be
+ * possible or not depending on the particular slab's size.
+ */
+ return slab->obj_exts_in_object;
}
#else
-static inline void slab_set_stride(struct slab *slab, unsigned int stride)
+static inline bool obj_exts_in_object(struct slab *slab)
{
- VM_WARN_ON_ONCE(stride != sizeof(struct slabobj_ext));
-}
-static inline unsigned int slab_get_stride(struct slab *slab)
-{
- return sizeof(struct slabobj_ext);
+ return false;
}
#endif
@@ -656,8 +654,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
index = obj_to_index(s, slab, obj);
- obj_ext = (struct slabobj_ext *)(obj_exts +
- slab_get_stride(slab) * index);
+
+ if (!obj_exts_in_object(slab)) {
+ obj_ext = ((struct slabobj_ext *)obj_exts) + index;
+ } else {
+ unsigned int stride = s->size;
+ obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
+ }
+
return kasan_reset_tag(obj_ext);
}
@@ -693,9 +697,10 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
return NULL;
}
-static inline void slab_set_stride(struct slab *slab, unsigned int stride) { }
-static inline unsigned int slab_get_stride(struct slab *slab) { return 0; }
-
+static inline bool obj_exts_in_object(struct slab *slab)
+{
+ return false;
+}
#endif /* CONFIG_SLAB_OBJ_EXT */
diff --git a/mm/slub.c b/mm/slub.c
index 2bfcabc4c51a..98a14e5842a2 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -870,18 +870,6 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
#endif
#if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
-static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
-{
- /*
- * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
- * check the stride. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
- * allocations within_slab_leftover are preferred. And those may be
- * possible or not depending on the particular slab's size.
- */
- return obj_exts_in_slab(s, slab) &&
- (slab_get_stride(slab) == s->size);
-}
-
static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
{
unsigned int offset = get_info_end(s);
@@ -896,16 +884,20 @@ static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
return offset;
}
-#else
-static inline bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
+
+static inline void slab_set_obj_exts_in_object(struct slab *slab)
{
- return false;
+ slab->obj_exts_in_object = 1;
}
-
+#else
static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
{
return 0;
}
+
+static inline void slab_set_obj_exts_in_object(struct slab *slab)
+{
+}
#endif
#ifdef CONFIG_SLUB_DEBUG
@@ -1206,7 +1198,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p)
off += kasan_metadata_size(s, false);
- if (obj_exts_in_object(s, slab))
+ if (obj_exts_in_object(slab))
off += sizeof(struct slabobj_ext);
if (off != size_from_object(s))
@@ -1411,7 +1403,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p)
off += kasan_metadata_size(s, false);
- if (obj_exts_in_object(s, slab))
+ if (obj_exts_in_object(slab))
off += sizeof(struct slabobj_ext);
if (size_from_object(s) == off)
@@ -1439,7 +1431,7 @@ slab_pad_check(struct kmem_cache *s, struct slab *slab)
length = slab_size(slab);
end = start + length;
- if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(s, slab)) {
+ if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(slab)) {
remainder = length;
remainder -= obj_exts_offset_in_slab(s, slab);
remainder -= obj_exts_size_in_slab(slab);
@@ -2253,9 +2245,6 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
void *addr;
unsigned long obj_exts;
- /* Initialize stride early to avoid memory ordering issues */
- slab_set_stride(slab, sizeof(struct slabobj_ext));
-
if (!need_slab_obj_exts(s))
return;
@@ -2289,7 +2278,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
obj_exts |= MEMCG_DATA_OBJEXTS;
#endif
slab->obj_exts = obj_exts;
- slab_set_stride(slab, s->size);
+ slab_set_obj_exts_in_object(slab);
}
}
@@ -3402,9 +3391,10 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
stat(s, ORDER_FALLBACK);
}
+ /* Initializes frozen, inuse, and any extra 64bit-only flags */
+ slab->counters = 0;
+
slab->objects = oo_objects(oo);
- slab->inuse = 0;
- slab->frozen = 0;
slab->slab_cache = s;
@@ -6537,7 +6527,7 @@ static inline size_t slab_ksize(struct slab *slab)
*/
if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
return s->inuse;
- else if (obj_exts_in_object(s, slab))
+ else if (obj_exts_in_object(slab))
return s->inuse;
/*
* Else we can use all the padding etc for the allocation
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (6 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 4:11 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag() Vlastimil Babka (SUSE)
` (4 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
Currently, struct slabobj_ext can hold both objcg pointer and
codetag_ref (when both are compile-enabled) and there is an array of as
many slabobj_ext instances as there are objects in a slab.
This makes the layout fixed so even if codetag_ref is unused (because
memory allocation profiling is disabled), the space for them is
allocated and wasted. Similarly, some caches (currently kmalloc_normal)
do not ever need objcg pointers, leading to wasted memory with memory
allocation profiling enabled.
To make this more flexible, change the layout so that struct slabobj_ext
becomes an union of objcg pointer and codetag_ref (to ensure uniform
size; in practice both are the same size anyway). The slabobj_ext array
then can have twice as many elements as before. For cache locality
purposes, the effective memory layout is unchanged, so objcg and codetag
ref for a given object are still adjacent.
static_obj_ext_size() returns the effective size of (0-2) struct
slabobj_ext's, depending on the config. slab_obj_ext_size() is currently
static as well, but takes a slab pointer so it can be made dynamic
later. Replace all sizeof(slabobj_ext) usage with these.
No functional change intended, the layout is still effectively static.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slab.h | 41 +++++++++++++++++++++++++++++++++--------
mm/slub.c | 17 +++++++++--------
2 files changed, 42 insertions(+), 16 deletions(-)
diff --git a/mm/slab.h b/mm/slab.h
index 3ad9777ad600..359ab8caf61e 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void)
* if MEMCG_DATA_OBJEXTS is set.
*/
struct slabobj_ext {
+ union {
#ifdef CONFIG_MEMCG
- struct obj_cgroup *_objcg;
+ struct obj_cgroup *_objcg;
#endif
#ifdef CONFIG_MEM_ALLOC_PROFILING
- union codetag_ref _ctref;
+ union codetag_ref _ctref;
#endif
+ };
} __aligned(8);
+static inline size_t static_obj_ext_size(void)
+{
+ size_t sz = 0;
+
+ if (IS_ENABLED(CONFIG_MEMCG))
+ sz += 1;
+
+ if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING))
+ sz += 1;
+
+ return sizeof(struct slabobj_ext) * sz;
+}
+
+static inline size_t slab_obj_ext_size(struct slab *slab)
+{
+ return static_obj_ext_size();
+}
+
#ifdef CONFIG_SLAB_OBJ_EXT
/*
@@ -650,17 +670,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
{
struct slabobj_ext *obj_ext;
unsigned int index;
+ unsigned int stride;
VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
index = obj_to_index(s, slab, obj);
- if (!obj_exts_in_object(slab)) {
- obj_ext = ((struct slabobj_ext *)obj_exts) + index;
- } else {
- unsigned int stride = s->size;
- obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
- }
+ if (!obj_exts_in_object(slab))
+ stride = slab_obj_ext_size(slab);
+ else
+ stride = s->size;
+
+ obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
return kasan_reset_tag(obj_ext);
}
@@ -668,6 +689,7 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
#ifdef CONFIG_MEMCG
static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
{
+ /* if objcg exists, it's first, so we don't need to do anything */
return &obj_ext->_objcg;
}
#endif
@@ -676,6 +698,9 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
static inline union codetag_ref *
slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
{
+ if (IS_ENABLED(CONFIG_MEMCG))
+ obj_ext += 1;
+
return &obj_ext->_ctref;
}
#endif
diff --git a/mm/slub.c b/mm/slub.c
index 98a14e5842a2..dd15af8abd62 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cache *s)
static inline unsigned int obj_exts_size_in_slab(struct slab *slab)
{
- return sizeof(struct slabobj_ext) * slab->objects;
+ return slab_obj_ext_size(slab) * slab->objects;
}
static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s,
@@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p)
off += kasan_metadata_size(s, false);
if (obj_exts_in_object(slab))
- off += sizeof(struct slabobj_ext);
+ off += slab_obj_ext_size(slab);
if (off != size_from_object(s))
/* Beginning of the filler is the free pointer */
@@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p)
off += kasan_metadata_size(s, false);
if (obj_exts_in_object(slab))
- off += sizeof(struct slabobj_ext);
+ off += slab_obj_ext_size(slab);
if (size_from_object(s) == off)
return 1;
@@ -2089,6 +2089,8 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)
static inline void handle_failed_objexts_alloc(struct slab *slab,
unsigned long obj_exts, struct slabobj_ext *vec)
{
+ unsigned int stride = slab_obj_ext_size(slab) / sizeof(*vec);
+
/*
* If vector previously failed to allocate then we have live
* objects with no tag reference. Mark all references in this
@@ -2101,7 +2103,7 @@ static inline void handle_failed_objexts_alloc(struct slab *slab,
union codetag_ref *ref = slab_obj_ext_codetag_ref(slab, vec);
set_codetag_empty(ref);
- vec++;
+ vec += stride;
}
}
@@ -2127,7 +2129,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
unsigned long new_exts;
unsigned long old_exts;
struct slabobj_ext *vec;
- size_t sz = sizeof(struct slabobj_ext) * slab->objects;
+ size_t sz = slab_obj_ext_size(slab) * slab->objects;
gfp &= ~OBJCGS_CLEAR_MASK;
/*
@@ -2270,8 +2272,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
get_slab_obj_exts(obj_exts);
for_each_object(addr, s, slab_address(slab), slab->objects)
- memset(kasan_reset_tag(addr) + offset, 0,
- sizeof(struct slabobj_ext));
+ memset(kasan_reset_tag(addr) + offset, 0, slab_obj_ext_size(slab));
put_slab_obj_exts(obj_exts);
#ifdef CONFIG_MEMCG
@@ -7930,7 +7931,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
aligned_size = ALIGN(size, s->align);
#if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
if (slab_args_unmergeable(args, s->flags) &&
- (aligned_size - size >= sizeof(struct slabobj_ext)))
+ (aligned_size - size >= static_obj_ext_size()))
s->flags |= SLAB_OBJ_EXT_IN_OBJ;
#endif
size = aligned_size;
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag()
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (7 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 4:25 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled Vlastimil Babka (SUSE)
` (3 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
mem_alloc_profiling_enabled() allows evaluating (with a static key) if
memory profiling is currently enabled. mem_profiling_support is a
variable where false means it's not possible to enable it anymore,
because the system was booted with "never" or it was later shut down.
This is possible to query by mem_alloc_profiling_permanently_disabled().
To make slabobj_ext array size handling dynamic, we need a snapshot of
mem_alloc_profiling_permanently_disabled() early in boot, so that's not
affected by a later shutdown. We also need it to be static key based for
performance. Neither mem_alloc_profiling_enabled() nor
mem_alloc_profiling_permanently_disabled() satisfy this.
Therefore introduce slab_obj_ext_has_codetag() with an underlying static
key for that use case. Its state is made to reflect the result of
mem_alloc_profiling_permanently_disabled() during kmem_cache_init(),
which does happen after setup_early_mem_profiling().
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slab.h | 16 ++++++++++++++++
mm/slub.c | 31 +++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
diff --git a/mm/slab.h b/mm/slab.h
index 359ab8caf61e..dcca86799fc9 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -564,6 +564,22 @@ struct slabobj_ext {
};
} __aligned(8);
+#ifdef CONFIG_MEM_ALLOC_PROFILING
+DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT,
+ slab_obj_ext_has_codetag_key);
+
+static inline bool slab_obj_ext_has_codetag(void)
+{
+ return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT,
+ &slab_obj_ext_has_codetag_key);
+}
+#else
+static inline bool slab_obj_ext_has_codetag(void)
+{
+ return false;
+}
+#endif
+
static inline size_t static_obj_ext_size(void)
{
size_t sz = 0;
diff --git a/mm/slub.c b/mm/slub.c
index dd15af8abd62..4200e7105b30 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -213,6 +213,11 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled);
static DEFINE_STATIC_KEY_FALSE(strict_numa);
#endif
+#ifdef CONFIG_MEM_ALLOC_PROFILING
+DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT,
+ slab_obj_ext_has_codetag_key);
+#endif
+
/* Structure holding extra parameters for slab allocations */
struct slab_alloc_context {
unsigned long caller_addr;
@@ -2415,6 +2420,26 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
__alloc_tagging_slab_free_hook(s, slab, p, objects);
}
+/*
+ * Make sure the static key used by slab_obj_ext_has_codetag() reflects the
+ * value of !mem_alloc_profiling_permanently_disabled()
+ *
+ * Any later mem alloc profiling shutdown won't be reflected in the static key
+ * because obj_exts with codetags might already exist.
+ */
+static void __init slab_obj_ext_has_codetag_init(void)
+{
+ bool key_enabled = IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT);
+ bool need_codetag = !mem_alloc_profiling_permanently_disabled();
+
+ if (key_enabled != need_codetag) {
+ if (need_codetag)
+ static_branch_enable(&slab_obj_ext_has_codetag_key);
+ else
+ static_branch_disable(&slab_obj_ext_has_codetag_key);
+ }
+}
+
#else /* CONFIG_MEM_ALLOC_PROFILING */
static inline void
@@ -2429,6 +2454,10 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
{
}
+static inline void slab_obj_ext_has_codetag_init(void)
+{
+}
+
#endif /* CONFIG_MEM_ALLOC_PROFILING */
@@ -8541,6 +8570,8 @@ void __init kmem_cache_init(void)
boot_kmem_cache_node;
int node;
+ slab_obj_ext_has_codetag_init();
+
if (debug_guardpage_minorder())
slub_max_order = 0;
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (8 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag() Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 4:30 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper Vlastimil Babka (SUSE)
` (2 subsequent siblings)
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
When memory allocation profiling is compiled in but permanently disabled
on boot with (implicit or explicit) "never" parameter, stop allocating
(wasting) memory for the codetag_ref parts of slabobj_ext metadata.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/slab.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/slab.h b/mm/slab.h
index dcca86799fc9..a50347c9dbe3 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -587,7 +587,7 @@ static inline size_t static_obj_ext_size(void)
if (IS_ENABLED(CONFIG_MEMCG))
sz += 1;
- if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING))
+ if (slab_obj_ext_has_codetag())
sz += 1;
return sizeof(struct slabobj_ext) * sz;
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (9 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 4:36 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary Vlastimil Babka (SUSE)
2026-07-15 15:32 ` [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Suren Baghdasaryan
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
Slabs of some caches never need the objcg part of struct slabobj_ext.
Introduce a helper to query this for a slab.
Currently only is_kmalloc_normal() caches are considered as not needing
objcg's. We could also consider all kmem caches without SLAB_ACCOUNT,
however some might be used with and without __GFP_ACCOUNT concurrently
and we currently don't restrict that. This can be improved later.
To make the evaluation of slab_needs_objcg() faster in the allocation
and free fast paths, add a obj_exts_needs_objcg flag into slab itself.
This optimization is only available on 64bit architectures where free
bits are available for the flag.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/kfence/core.c | 3 +++
mm/slab.h | 31 +++++++++++++++++++++++++++++--
mm/slub.c | 4 ++++
3 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/mm/kfence/core.c b/mm/kfence/core.c
index 717e8baf7e5d..afeaf80484ad 100644
--- a/mm/kfence/core.c
+++ b/mm/kfence/core.c
@@ -640,6 +640,9 @@ static unsigned long kfence_init_pool(void)
struct slab *slab = page_slab(page);
slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts |
MEMCG_DATA_OBJEXTS;
+#ifdef CONFIG_64BIT
+ slab->obj_exts_needs_objcg = 1;
+#endif
#endif
}
diff --git a/mm/slab.h b/mm/slab.h
index a50347c9dbe3..948d075cdbef 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -81,10 +81,11 @@ struct freelist_counters {
#ifdef CONFIG_64BIT
/*
* Some optimizations use free bits in 'counters' field
- * to save memory. If these free bits are not available,
- * such optimizations are disabled.
+ * to save memory or CPU. If these free bits are not
+ * available, such optimizations are disabled.
*/
unsigned obj_exts_in_object:1;
+ unsigned obj_exts_needs_objcg:1;
#endif
};
};
@@ -580,6 +581,32 @@ static inline bool slab_obj_ext_has_codetag(void)
}
#endif
+#ifdef CONFIG_MEMCG
+static inline bool cache_needs_objcg(struct kmem_cache *cache)
+{
+ return !is_kmalloc_normal(cache);
+}
+
+static inline bool slab_needs_objcg(struct slab *slab)
+{
+#ifdef CONFIG_64BIT
+ return slab->obj_exts_needs_objcg;
+#else
+ return cache_needs_objcg(slab->slab_cache);
+#endif
+}
+#else
+static inline bool cache_needs_objcg(struct kmem_cache *cache)
+{
+ return false;
+}
+
+static inline bool slab_needs_objcg(struct slab *slab)
+{
+ return false;
+}
+#endif
+
static inline size_t static_obj_ext_size(void)
{
size_t sz = 0;
diff --git a/mm/slub.c b/mm/slub.c
index 4200e7105b30..771d73abacb6 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3426,6 +3426,10 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
slab->objects = oo_objects(oo);
+#ifdef CONFIG_64BIT
+ if (cache_needs_objcg(s))
+ slab->obj_exts_needs_objcg = 1;
+#endif
slab->slab_cache = s;
kasan_poison_slab(slab);
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (10 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper Vlastimil Babka (SUSE)
@ 2026-07-15 10:10 ` Vlastimil Babka (SUSE)
2026-07-16 4:46 ` Suren Baghdasaryan
2026-07-15 15:32 ` [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Suren Baghdasaryan
12 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-15 10:10 UTC (permalink / raw)
To: Harry Yoo, Suren Baghdasaryan
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups, Vlastimil Babka (SUSE)
Start using the slab_needs_objcg() helper to calculate slabobj_ext size.
Caches that we know to never need objcg pointers (currently
KMALLOC_NORMAL caches) will thus stop wasting memory on them when memory
allocation profiling is enabled.
For things to work properly, we need to also add slab_needs_objcg()
checks to mem_cgroup_from_obj_slab() and memcg_slab_free_hook(), because
when obj_exts array exists for a slab only due to mem_alloc profiling,
we would otherwise attempt to access a non-existing objcg pointer in
that slab.
The function __memcg_slab_post_alloc_hook() should not be possible to
call for a slab where slab_needs_objcg() is false, but add a DEBUG_VM
check there to prevent breaking this assumption accidentally.
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/memcontrol.c | 6 ++++++
mm/slab.h | 12 ++++++++++--
mm/slub.c | 3 +++
3 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6303a2b1a9d0..09659722ec85 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2871,6 +2871,9 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
if (!obj_exts)
return NULL;
+ if (!slab_needs_objcg(slab))
+ return NULL;
+
get_slab_obj_exts(obj_exts);
obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
objcg = *slab_obj_ext_objcgp(obj_ext);
@@ -3581,6 +3584,9 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
slab = virt_to_slab(p[i]);
+ if (IS_ENABLED(CONFIG_DEBUG_VM) && WARN_ON_ONCE(!slab_needs_objcg(slab)))
+ continue;
+
if (!slab_obj_exts(slab) &&
alloc_slab_obj_exts(slab, s, flags, slab_alloc_flags)) {
continue;
diff --git a/mm/slab.h b/mm/slab.h
index 948d075cdbef..072cc2506756 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -622,7 +622,15 @@ static inline size_t static_obj_ext_size(void)
static inline size_t slab_obj_ext_size(struct slab *slab)
{
- return static_obj_ext_size();
+ size_t sz = 0;
+
+ if (slab_needs_objcg(slab))
+ sz += 1;
+
+ if (slab_obj_ext_has_codetag())
+ sz += 1;
+
+ return sizeof(struct slabobj_ext) * sz;
}
#ifdef CONFIG_SLAB_OBJ_EXT
@@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
static inline union codetag_ref *
slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
{
- if (IS_ENABLED(CONFIG_MEMCG))
+ if (slab_needs_objcg(slab))
obj_ext += 1;
return &obj_ext->_ctref;
diff --git a/mm/slub.c b/mm/slub.c
index 771d73abacb6..09c4931e5435 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
if (likely(!obj_exts))
return;
+ if (!slab_needs_objcg(slab))
+ return;
+
get_slab_obj_exts(obj_exts);
__memcg_slab_free_hook(s, slab, p, objects, obj_exts);
put_slab_obj_exts(obj_exts);
--
2.55.0
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
` (11 preceding siblings ...)
2026-07-15 10:10 ` [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary Vlastimil Babka (SUSE)
@ 2026-07-15 15:32 ` Suren Baghdasaryan
2026-07-16 3:28 ` Harry Yoo
12 siblings, 1 reply; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-15 15:32 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> The recent fixes for objext array handling inspired me to look into this
> finally. It's been bothering me that the memory usage of struct
> slabobj_ext depend only on config options and not whether the fields are
> actually used. So with both CONFIG_MEMCG=y and
> CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref
> field. And thus:
>
> 1) Having memory allocation profiling config-enabled but not
> boot-enabled means wasted memory on unused codetag_refs. This makes
> it less suitable for a general distro config and the page allocator
> side doesn't suffer from this, only slab and percpu.
>
> 2) Complementary, with memory allocation profiling enabled, there are
> caches/slabs that don't need the objcg field, so memory is wasted on
> those.
It's funny because yesterday I started working on a prototype for the
same optimization. But your patchset is much more mature, so I'll
focus instead on reviewing it.
>
> This series should solve the point 1) fully for slab, pcpuobj_ext
> handling can be perhaps improved similarly, haven't looked into that.
>
> For 2) it avoids allocating objcg fields for KMALLOC_NORMAL caches where
> we know they are not necessary because kmalloc() with __GFP_ACCOUNT will
> pick a KMALLOC_CGROUP type.
>
> The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT
> and then we know objcg fields are always needed. But also they can be
> created without SLAB_ACCOUNT and then some allocations have
> __GFP_ACCOUNT and some not and we don't know that in advance.
Do you know how often this happens that a named cache with no
SLAB_ACCOUNT is used for __GFP_ACCOUNT allocation?
>
> A possible future solution is to introduce e.g. SLAB_MAYBE_ACCOUNT, add
> it to caches where we know __GFP_ACCOUNT is used, and only honour
> __GFP_ACCOUNT for those, while warning for an unexpected usage
> elsewhere.
I wonder if for such caches we could create two separate caches, one
serving __GFP_ACCOUNT and using extentions containing objcg and
another one for non-__GFP_ACCOUNT with optimized extentions?
>
> Only lightly tested, need to run at least some microbenchmarks to see if
> the now somewhat more complicated access to objcg is visible or not.
>
> Based on slab/for-next-fixes
>
> Git branch: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/objext_split
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
> Vlastimil Babka (SUSE) (12):
> mm/slab: skip kfence objects in allocation profiling
> mm/slab: remove objs_per_slab()
> mm: move struct slabobj_ext to mm/slab.h
> mm/slab: make slab_obj_ext() determine object index
> mm/slab: abstract slabobj_ext.objcg access
> mm/slab: abstract slabobj_ext.ref access
> mm/slab: replace slab.stride with obj_exts_in_object
> mm/slab: change struct slabobj_ext to a union
> mm/slab: introduce slab_obj_ext_has_codetag()
> mm/slab: reduce slabobj_ext memory with allocation profiling disabled
> mm/slab: add slab_needs_objcg() helper
> mm/slab: stop allocating objcg pointers when unnecessary
>
> include/linux/memcontrol.h | 13 ----
> mm/kfence/core.c | 5 +-
> mm/kfence/kfence_test.c | 2 +-
> mm/memcontrol.c | 41 ++++++-----
> mm/slab.h | 169 ++++++++++++++++++++++++++++++++++++---------
> mm/slub.c | 162 +++++++++++++++++++++++++++----------------
> 6 files changed, 267 insertions(+), 125 deletions(-)
> ---
> base-commit: d9e6a7623938968e3752b67e37eaff097e559a54
> change-id: 20260714-b4-objext_split-da82426257d5
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling
2026-07-15 10:10 ` [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling Vlastimil Babka (SUSE)
@ 2026-07-15 16:02 ` Suren Baghdasaryan
2026-07-16 9:11 ` Vlastimil Babka (SUSE)
0 siblings, 1 reply; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-15 16:02 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> struct kfence_metadata only has obj_exts with CONFIG_MEMCG.
I don't quite understand this statement. obj_exts are allocated when
either CONFIG_MEMCG or CONFIG_MEM_ALLOC_PROFILING is enabled.
> If it's
> enabled, it does also work for allocation profiling, but there's little
> value recording tags for KFENCE objects.
Unless we are leaking them, right?
> Furthermore it would complicate
> the upcoming changes, so just skip them in the slab hooks.
Ok, I can understand that. If we do this, we should document that
kfence objects are no longer tracked.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
> mm/slub.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 0337e60db5ac..a4be70d080fb 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
> if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
> return;
>
> + if (is_kfence_address(object))
> + return;
> +
> slab = virt_to_slab(object);
> obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
> /*
> @@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
> for (i = 0; i < objects; i++) {
> unsigned int off = obj_to_index(s, slab, p[i]);
>
> + if (is_kfence_address(p[i]))
> + continue;
> +
> alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
> }
> put_slab_obj_exts(obj_exts);
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 02/12] mm/slab: remove objs_per_slab()
2026-07-15 10:10 ` [PATCH RFC 02/12] mm/slab: remove objs_per_slab() Vlastimil Babka (SUSE)
@ 2026-07-15 16:13 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-15 16:13 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> The function has an unused kmem_cache argument and almost nothing uses
> it anyway; doing slab->objects is simpler. Remove it with the last two
> users. KUNIT_EXPECT_EQ() needs a cast to avoid "error: ‘typeof’ applied
> to a bit-field" but we don't need to keep a wrapper just for that.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasayan <surenb@google.com>
> ---
> mm/kfence/kfence_test.c | 2 +-
> mm/slab.h | 6 ------
> mm/slub.c | 3 +--
> 3 files changed, 2 insertions(+), 9 deletions(-)
>
> diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
> index de2d0f7d62b1..9867c03ef0ae 100644
> --- a/mm/kfence/kfence_test.c
> +++ b/mm/kfence/kfence_test.c
> @@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat
> * memcg accounting works correctly.
> */
> KUNIT_EXPECT_EQ(test, obj_to_index(s, slab, alloc), 0U);
> - KUNIT_EXPECT_EQ(test, objs_per_slab(s, slab), 1);
> + KUNIT_EXPECT_EQ(test, ((unsigned int)slab->objects), 1);
>
> if (policy == ALLOCATE_ANY)
> return alloc;
> diff --git a/mm/slab.h b/mm/slab.h
> index f5e336b6b6b0..01535e1e2d3c 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,
> return __obj_to_index(cache, slab_address(slab), obj);
> }
>
> -static inline int objs_per_slab(const struct kmem_cache *cache,
> - const struct slab *slab)
> -{
> - return slab->objects;
> -}
> -
> /*
> * State of the slab allocator.
> *
> diff --git a/mm/slub.c b/mm/slub.c
> index a4be70d080fb..9e25f2dce7a6 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2127,7 +2127,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> gfp_t gfp, unsigned int alloc_flags)
> {
> const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
> - unsigned int objects = objs_per_slab(s, slab);
> bool new_slab = alloc_flags & SLAB_ALLOC_NEW_SLAB;
> unsigned long new_exts;
> unsigned long old_exts;
> @@ -2183,7 +2182,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> #endif
> retry:
> old_exts = READ_ONCE(slab->obj_exts);
> - handle_failed_objexts_alloc(old_exts, vec, objects);
> + handle_failed_objexts_alloc(old_exts, vec, slab->objects);
>
> if (new_slab) {
> /*
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h
2026-07-15 10:10 ` [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h Vlastimil Babka (SUSE)
@ 2026-07-16 0:56 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 0:56 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> Users of include/linux/memcontrol.h don't need to see this internal
> structure. Further changes to the struct will reduce recompiling.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> include/linux/memcontrol.h | 13 -------------
> mm/slab.h | 13 +++++++++++++
> 2 files changed, 13 insertions(+), 13 deletions(-)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index e1f46a0016fc..93869cc35c25 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1440,19 +1440,6 @@ static inline void mem_cgroup_flush_workqueue(void) { }
> static inline int mem_cgroup_init(void) { return 0; }
> #endif /* CONFIG_MEMCG */
>
> -/*
> - * Extended information for slab objects stored as an array in page->memcg_data
> - * if MEMCG_DATA_OBJEXTS is set.
> - */
> -struct slabobj_ext {
> -#ifdef CONFIG_MEMCG
> - struct obj_cgroup *objcg;
> -#endif
> -#ifdef CONFIG_MEM_ALLOC_PROFILING
> - union codetag_ref ref;
> -#endif
> -} __aligned(8);
> -
> static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
> {
> struct mem_cgroup *memcg;
> diff --git a/mm/slab.h b/mm/slab.h
> index 01535e1e2d3c..7bd361447c54 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -549,6 +549,19 @@ static inline bool need_kmalloc_no_objext(void)
> return false;
> }
>
> +/*
> + * Extended information for slab objects stored as an array in page->memcg_data
> + * if MEMCG_DATA_OBJEXTS is set.
> + */
> +struct slabobj_ext {
> +#ifdef CONFIG_MEMCG
> + struct obj_cgroup *objcg;
> +#endif
> +#ifdef CONFIG_MEM_ALLOC_PROFILING
> + union codetag_ref ref;
> +#endif
> +} __aligned(8);
> +
> #ifdef CONFIG_SLAB_OBJ_EXT
>
> /*
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index
2026-07-15 10:10 ` [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index Vlastimil Babka (SUSE)
@ 2026-07-16 1:02 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 1:02 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> All callers perform the same obj_to_index() calculation to pass the
> index. Simplify by passing object pointer instead and determining the
> index by slab_obj_ext().
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/memcontrol.c | 12 +++---------
> mm/slab.h | 14 ++++++++------
> mm/slub.c | 22 +++++++---------------
> 3 files changed, 18 insertions(+), 30 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 6dc4888a90f3..4e427286a88a 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2865,15 +2865,13 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
> */
> unsigned long obj_exts;
> struct slabobj_ext *obj_ext;
> - unsigned int off;
>
> obj_exts = slab_obj_exts(slab);
> if (!obj_exts)
> return NULL;
>
> get_slab_obj_exts(obj_exts);
> - off = obj_to_index(slab->slab_cache, slab, p);
> - obj_ext = slab_obj_ext(slab, obj_exts, off);
> + obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
> if (obj_ext->objcg) {
> struct obj_cgroup *objcg = obj_ext->objcg;
>
> @@ -3541,7 +3539,6 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
> size_t obj_size = obj_full_size(s);
> struct obj_cgroup *objcg;
> struct slab *slab;
> - unsigned long off;
> size_t i;
>
> /*
> @@ -3616,8 +3613,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>
> obj_exts = slab_obj_exts(slab);
> get_slab_obj_exts(obj_exts);
> - off = obj_to_index(s, slab, p[i]);
> - obj_ext = slab_obj_ext(slab, obj_exts, off);
> + obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> obj_cgroup_get(objcg);
> obj_ext->objcg = objcg;
> put_slab_obj_exts(obj_exts);
> @@ -3635,10 +3631,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
> struct obj_cgroup *objcg;
> struct slabobj_ext *obj_ext;
> struct obj_stock_pcp *stock;
> - unsigned int off;
>
> - off = obj_to_index(s, slab, p[i]);
> - obj_ext = slab_obj_ext(slab, obj_exts, off);
> + obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> objcg = obj_ext->objcg;
> if (!objcg)
> continue;
> diff --git a/mm/slab.h b/mm/slab.h
> index 7bd361447c54..36d067d6e7c0 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -646,14 +646,16 @@ static inline unsigned int slab_get_stride(struct slab *slab)
> * Returns a pointer to the object extension associated with the object.
> * Must be called within a section covered by get/put_slab_obj_exts().
> */
> -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
> - unsigned long obj_exts,
> - unsigned int index)
> +static inline struct slabobj_ext *
> +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> + const void *obj)
> {
> struct slabobj_ext *obj_ext;
> + unsigned int index;
>
> VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
>
> + index = obj_to_index(s, slab, obj);
> obj_ext = (struct slabobj_ext *)(obj_exts +
> slab_get_stride(slab) * index);
> return kasan_reset_tag(obj_ext);
> @@ -669,9 +671,9 @@ static inline unsigned long slab_obj_exts(struct slab *slab)
> return 0;
> }
>
> -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
> - unsigned long obj_exts,
> - unsigned int index)
> +static inline struct slabobj_ext *
> +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> + const void *obj)
> {
> return NULL;
> }
> diff --git a/mm/slub.c b/mm/slub.c
> index 9e25f2dce7a6..5e3f53bcd0d3 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2070,11 +2070,10 @@ static inline void mark_obj_codetag_empty(const void *obj)
> obj_slab = virt_to_slab(obj);
> slab_exts = slab_obj_exts(obj_slab);
> if (slab_exts) {
> + struct slabobj_ext *ext;
> +
> get_slab_obj_exts(slab_exts);
> - unsigned int offs = obj_to_index(obj_slab->slab_cache,
> - obj_slab, obj);
> - struct slabobj_ext *ext = slab_obj_ext(obj_slab,
> - slab_exts, offs);
> + ext = slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj);
>
> if (unlikely(is_codetag_empty(&ext->ref))) {
> put_slab_obj_exts(slab_exts);
> @@ -2362,10 +2361,8 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
> * check should be added before alloc_tag_add().
> */
> if (obj_exts) {
> - unsigned int obj_idx = obj_to_index(s, slab, object);
> -
> get_slab_obj_exts(obj_exts);
> - obj_ext = slab_obj_ext(slab, obj_exts, obj_idx);
> + obj_ext = slab_obj_ext(s, slab, obj_exts, object);
> alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size);
> put_slab_obj_exts(obj_exts);
> } else {
> @@ -2386,7 +2383,6 @@ static noinline void
> __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
> int objects)
> {
> - int i;
> unsigned long obj_exts;
>
> /* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */
> @@ -2398,13 +2394,11 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
> return;
>
> get_slab_obj_exts(obj_exts);
> - for (i = 0; i < objects; i++) {
> - unsigned int off = obj_to_index(s, slab, p[i]);
> -
> + for (int i = 0; i < objects; i++) {
> if (is_kfence_address(p[i]))
> continue;
>
> - alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
> + alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size);
> }
> put_slab_obj_exts(obj_exts);
> }
> @@ -2489,7 +2483,6 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
> struct kmem_cache *s;
> struct page *page;
> struct slab *slab;
> - unsigned long off;
>
> page = virt_to_page(p);
> if (PageLargeKmalloc(page)) {
> @@ -2529,8 +2522,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
> obj_exts = slab_obj_exts(slab);
> if (obj_exts) {
> get_slab_obj_exts(obj_exts);
> - off = obj_to_index(s, slab, p);
> - obj_ext = slab_obj_ext(slab, obj_exts, off);
> + obj_ext = slab_obj_ext(s, slab, obj_exts, p);
> if (unlikely(obj_ext->objcg)) {
> put_slab_obj_exts(obj_exts);
> return true;
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access
2026-07-15 10:10 ` [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access Vlastimil Babka (SUSE)
@ 2026-07-16 1:27 ` Suren Baghdasaryan
2026-07-16 13:58 ` Vlastimil Babka (SUSE)
0 siblings, 1 reply; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 1:27 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> In preparation for changes to the structure, abstract access to the
> objcg field with a slab_obj_ext_objcgp() function.
> Rename the field to _objcg to make an unexpected direct access a compile
> error.
>
> No functional change intended.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
> mm/kfence/core.c | 2 +-
> mm/memcontrol.c | 23 +++++++++++++++--------
> mm/slab.h | 9 ++++++++-
> mm/slub.c | 2 +-
> 4 files changed, 25 insertions(+), 11 deletions(-)
>
> diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> index 6577bd76954e..717e8baf7e5d 100644
> --- a/mm/kfence/core.c
> +++ b/mm/kfence/core.c
> @@ -1249,7 +1249,7 @@ void __kfence_free(void *addr)
> struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
>
> #ifdef CONFIG_MEMCG
> - KFENCE_WARN_ON(meta->obj_exts.objcg);
> + KFENCE_WARN_ON(*slab_obj_ext_objcgp(&meta->obj_exts));
> #endif
> /*
> * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 4e427286a88a..6303a2b1a9d0 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
> */
> unsigned long obj_exts;
> struct slabobj_ext *obj_ext;
> + struct obj_cgroup *objcg;
>
> obj_exts = slab_obj_exts(slab);
> if (!obj_exts)
> @@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
>
> get_slab_obj_exts(obj_exts);
> obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
> - if (obj_ext->objcg) {
> - struct obj_cgroup *objcg = obj_ext->objcg;
> -
> + objcg = *slab_obj_ext_objcgp(obj_ext);
> + if (objcg) {
> put_slab_obj_exts(obj_exts);
> return obj_cgroup_memcg(objcg);
> }
> @@ -3577,6 +3577,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
> unsigned long obj_exts;
> struct slabobj_ext *obj_ext;
> struct obj_stock_pcp *stock;
> + struct obj_cgroup **objcgp;
>
> slab = virt_to_slab(p[i]);
>
> @@ -3612,10 +3613,15 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
> unlock_stock(stock);
>
> obj_exts = slab_obj_exts(slab);
> +
> get_slab_obj_exts(obj_exts);
> +
> obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> + objcgp = slab_obj_ext_objcgp(obj_ext);
> +
> obj_cgroup_get(objcg);
> - obj_ext->objcg = objcg;
> + *objcgp = objcg;
> +
> put_slab_obj_exts(obj_exts);
> }
>
> @@ -3628,16 +3634,17 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
> size_t obj_size = obj_full_size(s);
>
> for (int i = 0; i < objects; i++) {
> - struct obj_cgroup *objcg;
> + struct obj_cgroup **objcgp, *objcg;
> struct slabobj_ext *obj_ext;
> struct obj_stock_pcp *stock;
>
> obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> - objcg = obj_ext->objcg;
> - if (!objcg)
> + objcgp = slab_obj_ext_objcgp(obj_ext);
> + if (!*objcgp)
> continue;
>
> - obj_ext->objcg = NULL;
> + objcg = *objcgp;
> + *objcgp = NULL;
>
> stock = trylock_stock();
> __refill_obj_stock(objcg, stock, obj_size, true);
> diff --git a/mm/slab.h b/mm/slab.h
> index 36d067d6e7c0..789bd292075f 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
> */
> struct slabobj_ext {
> #ifdef CONFIG_MEMCG
> - struct obj_cgroup *objcg;
> + struct obj_cgroup *_objcg;
> #endif
> #ifdef CONFIG_MEM_ALLOC_PROFILING
> union codetag_ref ref;
> @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> return kasan_reset_tag(obj_ext);
> }
>
> +#ifdef CONFIG_MEMCG
> +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
I understand why you need to return obj_cgroup** but maybe instead we
can have a separate setter function and just return a pointer here?
That would also help catch cases when someone wants to set
obj_ext._objcg that was optimized away (for that the setter would need
an additional parameter to identify whether we allocated optimized
objext vector).
> +{
> + return &obj_ext->_objcg;
> +}
> +#endif
> +
> int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> gfp_t gfp, unsigned int alloc_flags);
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 5e3f53bcd0d3..48e10198a3ce 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
> if (obj_exts) {
> get_slab_obj_exts(obj_exts);
> obj_ext = slab_obj_ext(s, slab, obj_exts, p);
> - if (unlikely(obj_ext->objcg)) {
> + if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
> put_slab_obj_exts(obj_exts);
> return true;
> }
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access
2026-07-15 10:10 ` [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access Vlastimil Babka (SUSE)
@ 2026-07-16 1:28 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 1:28 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> In preparation for changes to the structure, abstract access to the ref
> field with a slab_obj_ext_codetag_ref() function. Rename the field to
> _ctref to make an unexpected direct access a compile error.
>
> No functional change intended.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/slab.h | 10 +++++++++-
> mm/slub.c | 42 ++++++++++++++++++++++++++++--------------
> 2 files changed, 37 insertions(+), 15 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index 789bd292075f..e3f8e42070f1 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -558,7 +558,7 @@ struct slabobj_ext {
> struct obj_cgroup *_objcg;
> #endif
> #ifdef CONFIG_MEM_ALLOC_PROFILING
> - union codetag_ref ref;
> + union codetag_ref _ctref;
> #endif
> } __aligned(8);
>
> @@ -668,6 +668,14 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
> }
> #endif
>
> +#ifdef CONFIG_MEM_ALLOC_PROFILING
> +static inline union codetag_ref *
> +slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
> +{
> + return &obj_ext->_ctref;
> +}
> +#endif
> +
> int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> gfp_t gfp, unsigned int alloc_flags);
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 48e10198a3ce..2bfcabc4c51a 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2071,18 +2071,20 @@ static inline void mark_obj_codetag_empty(const void *obj)
> slab_exts = slab_obj_exts(obj_slab);
> if (slab_exts) {
> struct slabobj_ext *ext;
> + union codetag_ref *ref;
>
> get_slab_obj_exts(slab_exts);
> ext = slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj);
> + ref = slab_obj_ext_codetag_ref(obj_slab, ext);
>
> - if (unlikely(is_codetag_empty(&ext->ref))) {
> + if (unlikely(is_codetag_empty(ref))) {
> put_slab_obj_exts(slab_exts);
> return;
> }
>
> /* codetag should be NULL here */
> - WARN_ON(ext->ref.ct);
> - set_codetag_empty(&ext->ref);
> + WARN_ON(ref->ct);
> + set_codetag_empty(ref);
> put_slab_obj_exts(slab_exts);
> }
> }
> @@ -2092,19 +2094,22 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)
> return cmpxchg(&slab->obj_exts, 0, OBJEXTS_ALLOC_FAIL) == 0;
> }
>
> -static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
> - struct slabobj_ext *vec, unsigned int objects)
> +static inline void handle_failed_objexts_alloc(struct slab *slab,
> + unsigned long obj_exts, struct slabobj_ext *vec)
> {
> /*
> * If vector previously failed to allocate then we have live
> * objects with no tag reference. Mark all references in this
> * vector as empty to avoid warnings later on.
> */
> - if (obj_exts == OBJEXTS_ALLOC_FAIL) {
> - unsigned int i;
> + if (obj_exts != OBJEXTS_ALLOC_FAIL)
> + return;
> +
> + for (unsigned int i = 0; i < slab->objects; i++) {
> + union codetag_ref *ref = slab_obj_ext_codetag_ref(slab, vec);
>
> - for (i = 0; i < objects; i++)
> - set_codetag_empty(&vec[i].ref);
> + set_codetag_empty(ref);
> + vec++;
> }
> }
>
> @@ -2112,8 +2117,8 @@ static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
>
> static inline void mark_obj_codetag_empty(const void *obj) {}
> static inline bool mark_failed_objexts_alloc(struct slab *slab) { return false; }
> -static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
> - struct slabobj_ext *vec, unsigned int objects) {}
> +static inline void handle_failed_objexts_alloc(struct slab *slab,
> + unsigned long obj_exts, struct slabobj_ext *vec) {}
>
> #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */
>
> @@ -2181,7 +2186,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> #endif
> retry:
> old_exts = READ_ONCE(slab->obj_exts);
> - handle_failed_objexts_alloc(old_exts, vec, slab->objects);
> + handle_failed_objexts_alloc(slab, old_exts, vec);
>
> if (new_slab) {
> /*
> @@ -2361,9 +2366,15 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
> * check should be added before alloc_tag_add().
> */
> if (obj_exts) {
> + union codetag_ref *ref;
> +
> get_slab_obj_exts(obj_exts);
> +
> obj_ext = slab_obj_ext(s, slab, obj_exts, object);
> - alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size);
> + ref = slab_obj_ext_codetag_ref(slab, obj_ext);
> +
> + alloc_tag_add(ref, current->alloc_tag, s->size);
> +
> put_slab_obj_exts(obj_exts);
> } else {
> alloc_tag_set_inaccurate(current->alloc_tag);
> @@ -2395,10 +2406,13 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
>
> get_slab_obj_exts(obj_exts);
> for (int i = 0; i < objects; i++) {
> + struct slabobj_ext *ext;
> +
> if (is_kfence_address(p[i]))
> continue;
>
> - alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size);
> + ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> + alloc_tag_sub(slab_obj_ext_codetag_ref(slab, ext), s->size);
> }
> put_slab_obj_exts(obj_exts);
> }
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object
2026-07-15 10:10 ` [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object Vlastimil Babka (SUSE)
@ 2026-07-16 1:44 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 1:44 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> The stride field is used to convert object index to an slabobj_ext so
> both compact arrays (kmalloc() or in-slab-leftover) and spread
> in-object-padding obj_ext layouts are supported.
>
> In practice thus the stride is always sizeof(slabobj_ext) or s->size.
>
> This simplifies the calculations, but with the upcoming slabobj_ext
> handling changes, it will be easier to stop storing the stride and
> instead just have a flag whether obj_ext is in the object padding.
> obj_exts_in_object() can then rely on this flag and slab_obj_ext()
> can use that to determine the stride.
>
> No functional change intended. Performance impact TBD, hopefully
> in the noise.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/slab.h | 43 ++++++++++++++++++++++++-------------------
> mm/slub.c | 42 ++++++++++++++++--------------------------
> 2 files changed, 40 insertions(+), 45 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index e3f8e42070f1..3ad9777ad600 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -81,10 +81,10 @@ struct freelist_counters {
> #ifdef CONFIG_64BIT
> /*
> * Some optimizations use free bits in 'counters' field
> - * to save memory. In case ->stride field is not available,
> + * to save memory. If these free bits are not available,
> * such optimizations are disabled.
> */
> - unsigned int stride;
> + unsigned obj_exts_in_object:1;
> #endif
> };
> };
> @@ -617,22 +617,20 @@ static inline void put_slab_obj_exts(unsigned long obj_exts)
> }
>
> #ifdef CONFIG_64BIT
> -static inline void slab_set_stride(struct slab *slab, unsigned int stride)
> +static inline bool obj_exts_in_object(struct slab *slab)
> {
> - slab->stride = stride;
> -}
> -static inline unsigned int slab_get_stride(struct slab *slab)
> -{
> - return slab->stride;
> + /*
> + * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
> + * check the per-slab bit. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
> + * allocations within_slab_leftover are preferred. And those may be
> + * possible or not depending on the particular slab's size.
> + */
> + return slab->obj_exts_in_object;
> }
> #else
> -static inline void slab_set_stride(struct slab *slab, unsigned int stride)
> +static inline bool obj_exts_in_object(struct slab *slab)
> {
> - VM_WARN_ON_ONCE(stride != sizeof(struct slabobj_ext));
> -}
> -static inline unsigned int slab_get_stride(struct slab *slab)
> -{
> - return sizeof(struct slabobj_ext);
> + return false;
> }
> #endif
>
> @@ -656,8 +654,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
>
> index = obj_to_index(s, slab, obj);
> - obj_ext = (struct slabobj_ext *)(obj_exts +
> - slab_get_stride(slab) * index);
> +
> + if (!obj_exts_in_object(slab)) {
> + obj_ext = ((struct slabobj_ext *)obj_exts) + index;
> + } else {
> + unsigned int stride = s->size;
> + obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
> + }
> +
> return kasan_reset_tag(obj_ext);
> }
>
> @@ -693,9 +697,10 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> return NULL;
> }
>
> -static inline void slab_set_stride(struct slab *slab, unsigned int stride) { }
> -static inline unsigned int slab_get_stride(struct slab *slab) { return 0; }
> -
> +static inline bool obj_exts_in_object(struct slab *slab)
> +{
> + return false;
> +}
>
> #endif /* CONFIG_SLAB_OBJ_EXT */
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 2bfcabc4c51a..98a14e5842a2 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -870,18 +870,6 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
> #endif
>
> #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
> -static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
> -{
> - /*
> - * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
> - * check the stride. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
> - * allocations within_slab_leftover are preferred. And those may be
> - * possible or not depending on the particular slab's size.
> - */
> - return obj_exts_in_slab(s, slab) &&
> - (slab_get_stride(slab) == s->size);
> -}
> -
> static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
> {
> unsigned int offset = get_info_end(s);
> @@ -896,16 +884,20 @@ static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
>
> return offset;
> }
> -#else
> -static inline bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
> +
> +static inline void slab_set_obj_exts_in_object(struct slab *slab)
> {
> - return false;
> + slab->obj_exts_in_object = 1;
> }
> -
> +#else
> static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
> {
> return 0;
> }
> +
> +static inline void slab_set_obj_exts_in_object(struct slab *slab)
> +{
> +}
> #endif
>
> #ifdef CONFIG_SLUB_DEBUG
> @@ -1206,7 +1198,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p)
>
> off += kasan_metadata_size(s, false);
>
> - if (obj_exts_in_object(s, slab))
> + if (obj_exts_in_object(slab))
> off += sizeof(struct slabobj_ext);
>
> if (off != size_from_object(s))
> @@ -1411,7 +1403,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p)
>
> off += kasan_metadata_size(s, false);
>
> - if (obj_exts_in_object(s, slab))
> + if (obj_exts_in_object(slab))
> off += sizeof(struct slabobj_ext);
>
> if (size_from_object(s) == off)
> @@ -1439,7 +1431,7 @@ slab_pad_check(struct kmem_cache *s, struct slab *slab)
> length = slab_size(slab);
> end = start + length;
>
> - if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(s, slab)) {
> + if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(slab)) {
> remainder = length;
> remainder -= obj_exts_offset_in_slab(s, slab);
> remainder -= obj_exts_size_in_slab(slab);
> @@ -2253,9 +2245,6 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
> void *addr;
> unsigned long obj_exts;
>
> - /* Initialize stride early to avoid memory ordering issues */
> - slab_set_stride(slab, sizeof(struct slabobj_ext));
> -
> if (!need_slab_obj_exts(s))
> return;
>
> @@ -2289,7 +2278,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
> obj_exts |= MEMCG_DATA_OBJEXTS;
> #endif
> slab->obj_exts = obj_exts;
> - slab_set_stride(slab, s->size);
> + slab_set_obj_exts_in_object(slab);
> }
> }
>
> @@ -3402,9 +3391,10 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
> stat(s, ORDER_FALLBACK);
> }
>
> + /* Initializes frozen, inuse, and any extra 64bit-only flags */
> + slab->counters = 0;
> +
> slab->objects = oo_objects(oo);
> - slab->inuse = 0;
> - slab->frozen = 0;
>
> slab->slab_cache = s;
>
> @@ -6537,7 +6527,7 @@ static inline size_t slab_ksize(struct slab *slab)
> */
> if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
> return s->inuse;
> - else if (obj_exts_in_object(s, slab))
> + else if (obj_exts_in_object(slab))
> return s->inuse;
> /*
> * Else we can use all the padding etc for the allocation
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste
2026-07-15 15:32 ` [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Suren Baghdasaryan
@ 2026-07-16 3:28 ` Harry Yoo
2026-07-16 4:55 ` Suren Baghdasaryan
0 siblings, 1 reply; 37+ messages in thread
From: Harry Yoo @ 2026-07-16 3:28 UTC (permalink / raw)
To: Suren Baghdasaryan, Vlastimil Babka (SUSE)
Cc: Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
[-- Attachment #1.1: Type: text/plain, Size: 2804 bytes --]
On 7/16/26 12:32 AM, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>>
>> The recent fixes for objext array handling inspired me to look into this
>> finally. It's been bothering me that the memory usage of struct
>> slabobj_ext depend only on config options and not whether the fields are
>> actually used.
+1 one more person bothered by this...
>> So with both CONFIG_MEMCG=y and
>> CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref
>> field. And thus:
>>
>> 1) Having memory allocation profiling config-enabled but not
>> boot-enabled means wasted memory on unused codetag_refs. This makes
>> it less suitable for a general distro config and the page allocator
>> side doesn't suffer from this, only slab and percpu.
>>
>> 2) Complementary, with memory allocation profiling enabled, there are
>> caches/slabs that don't need the objcg field, so memory is wasted on
>> those.
>
> It's funny because yesterday I started working on a prototype for the
> same optimization. But your patchset is much more mature, so I'll
> focus instead on reviewing it.
Ouch, a race condition!
>> This series should solve the point 1) fully for slab, pcpuobj_ext
>> handling can be perhaps improved similarly, haven't looked into that.
>>
>> For 2) it avoids allocating objcg fields for KMALLOC_NORMAL caches where
>> we know they are not necessary because kmalloc() with __GFP_ACCOUNT will
>> pick a KMALLOC_CGROUP type.
Unless KMALLOC_RECLAIM != KMALLOC_NORMAL! (yes, SLUB_TINY)
>> The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT
>> and then we know objcg fields are always needed. But also they can be
>> created without SLAB_ACCOUNT and then some allocations have
>> __GFP_ACCOUNT and some not and we don't know that in advance.
>
> Do you know how often this happens that a named cache with no
> SLAB_ACCOUNT is used for __GFP_ACCOUNT allocation?
Not sure about how often, but one thing I recall is xarray
(radix_tree_node cache), which decides to account the objects based on
xarray flags.
>> A possible future solution is to introduce e.g. SLAB_MAYBE_ACCOUNT, add
>> it to caches where we know __GFP_ACCOUNT is used, and only honour
>> __GFP_ACCOUNT for those, while warning for an unexpected usage
>> elsewhere.
>
> I wonder if for such caches we could create two separate caches, one
> serving __GFP_ACCOUNT and using extentions containing objcg and
> another one for non-__GFP_ACCOUNT with optimized extentions?
You mean transparently to users? (e.g., user thinks it has created
a single kmem_cache but actually there are two of them, multiplexed by
__GFP_ACCOUNT bit)
--
Cheers,
Harry / Hyeonggon
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union
2026-07-15 10:10 ` [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union Vlastimil Babka (SUSE)
@ 2026-07-16 4:11 ` Suren Baghdasaryan
2026-07-16 14:43 ` Vlastimil Babka (SUSE)
0 siblings, 1 reply; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 4:11 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> Currently, struct slabobj_ext can hold both objcg pointer and
> codetag_ref (when both are compile-enabled) and there is an array of as
> many slabobj_ext instances as there are objects in a slab.
>
> This makes the layout fixed so even if codetag_ref is unused (because
> memory allocation profiling is disabled), the space for them is
> allocated and wasted. Similarly, some caches (currently kmalloc_normal)
> do not ever need objcg pointers, leading to wasted memory with memory
> allocation profiling enabled.
>
> To make this more flexible, change the layout so that struct slabobj_ext
> becomes an union of objcg pointer and codetag_ref (to ensure uniform
> size; in practice both are the same size anyway). The slabobj_ext array
> then can have twice as many elements as before. For cache locality
> purposes, the effective memory layout is unchanged, so objcg and codetag
> ref for a given object are still adjacent.
>
> static_obj_ext_size() returns the effective size of (0-2) struct
> slabobj_ext's, depending on the config. slab_obj_ext_size() is currently
> static as well, but takes a slab pointer so it can be made dynamic
> later. Replace all sizeof(slabobj_ext) usage with these.
>
> No functional change intended, the layout is still effectively static.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
This is much cleaner than what I was preparing. Nicely done!
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/slab.h | 41 +++++++++++++++++++++++++++++++++--------
> mm/slub.c | 17 +++++++++--------
> 2 files changed, 42 insertions(+), 16 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index 3ad9777ad600..359ab8caf61e 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void)
> * if MEMCG_DATA_OBJEXTS is set.
> */
> struct slabobj_ext {
Perhaps we should add a comment here stating that every element of
this union should be pointer-sized?
> + union {
> #ifdef CONFIG_MEMCG
> - struct obj_cgroup *_objcg;
> + struct obj_cgroup *_objcg;
> #endif
> #ifdef CONFIG_MEM_ALLOC_PROFILING
> - union codetag_ref _ctref;
> + union codetag_ref _ctref;
> #endif
> + };
> } __aligned(8);
>
> +static inline size_t static_obj_ext_size(void)
> +{
> + size_t sz = 0;
> +
> + if (IS_ENABLED(CONFIG_MEMCG))
> + sz += 1;
> +
> + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING))
> + sz += 1;
> +
> + return sizeof(struct slabobj_ext) * sz;
> +}
> +
> +static inline size_t slab_obj_ext_size(struct slab *slab)
> +{
> + return static_obj_ext_size();
> +}
> +
> #ifdef CONFIG_SLAB_OBJ_EXT
>
> /*
> @@ -650,17 +670,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> {
> struct slabobj_ext *obj_ext;
> unsigned int index;
> + unsigned int stride;
>
> VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
>
> index = obj_to_index(s, slab, obj);
>
> - if (!obj_exts_in_object(slab)) {
> - obj_ext = ((struct slabobj_ext *)obj_exts) + index;
> - } else {
> - unsigned int stride = s->size;
> - obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
> - }
> + if (!obj_exts_in_object(slab))
> + stride = slab_obj_ext_size(slab);
> + else
> + stride = s->size;
> +
> + obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
>
> return kasan_reset_tag(obj_ext);
> }
> @@ -668,6 +689,7 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> #ifdef CONFIG_MEMCG
> static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
> {
> + /* if objcg exists, it's first, so we don't need to do anything */
> return &obj_ext->_objcg;
> }
> #endif
> @@ -676,6 +698,9 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
> static inline union codetag_ref *
> slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
> {
> + if (IS_ENABLED(CONFIG_MEMCG))
> + obj_ext += 1;
> +
> return &obj_ext->_ctref;
> }
> #endif
> diff --git a/mm/slub.c b/mm/slub.c
> index 98a14e5842a2..dd15af8abd62 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cache *s)
>
> static inline unsigned int obj_exts_size_in_slab(struct slab *slab)
> {
> - return sizeof(struct slabobj_ext) * slab->objects;
> + return slab_obj_ext_size(slab) * slab->objects;
> }
>
> static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s,
> @@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p)
> off += kasan_metadata_size(s, false);
>
> if (obj_exts_in_object(slab))
> - off += sizeof(struct slabobj_ext);
> + off += slab_obj_ext_size(slab);
>
> if (off != size_from_object(s))
> /* Beginning of the filler is the free pointer */
> @@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p)
> off += kasan_metadata_size(s, false);
>
> if (obj_exts_in_object(slab))
> - off += sizeof(struct slabobj_ext);
> + off += slab_obj_ext_size(slab);
>
> if (size_from_object(s) == off)
> return 1;
> @@ -2089,6 +2089,8 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)
> static inline void handle_failed_objexts_alloc(struct slab *slab,
> unsigned long obj_exts, struct slabobj_ext *vec)
> {
> + unsigned int stride = slab_obj_ext_size(slab) / sizeof(*vec);
> +
> /*
> * If vector previously failed to allocate then we have live
> * objects with no tag reference. Mark all references in this
> @@ -2101,7 +2103,7 @@ static inline void handle_failed_objexts_alloc(struct slab *slab,
> union codetag_ref *ref = slab_obj_ext_codetag_ref(slab, vec);
>
> set_codetag_empty(ref);
> - vec++;
> + vec += stride;
> }
> }
>
> @@ -2127,7 +2129,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> unsigned long new_exts;
> unsigned long old_exts;
> struct slabobj_ext *vec;
> - size_t sz = sizeof(struct slabobj_ext) * slab->objects;
> + size_t sz = slab_obj_ext_size(slab) * slab->objects;
>
> gfp &= ~OBJCGS_CLEAR_MASK;
> /*
> @@ -2270,8 +2272,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
>
> get_slab_obj_exts(obj_exts);
> for_each_object(addr, s, slab_address(slab), slab->objects)
> - memset(kasan_reset_tag(addr) + offset, 0,
> - sizeof(struct slabobj_ext));
> + memset(kasan_reset_tag(addr) + offset, 0, slab_obj_ext_size(slab));
> put_slab_obj_exts(obj_exts);
>
> #ifdef CONFIG_MEMCG
> @@ -7930,7 +7931,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
> aligned_size = ALIGN(size, s->align);
> #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
> if (slab_args_unmergeable(args, s->flags) &&
> - (aligned_size - size >= sizeof(struct slabobj_ext)))
> + (aligned_size - size >= static_obj_ext_size()))
> s->flags |= SLAB_OBJ_EXT_IN_OBJ;
> #endif
> size = aligned_size;
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag()
2026-07-15 10:10 ` [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag() Vlastimil Babka (SUSE)
@ 2026-07-16 4:25 ` Suren Baghdasaryan
2026-07-16 14:47 ` Vlastimil Babka (SUSE)
0 siblings, 1 reply; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 4:25 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> mem_alloc_profiling_enabled() allows evaluating (with a static key) if
> memory profiling is currently enabled. mem_profiling_support is a
> variable where false means it's not possible to enable it anymore,
> because the system was booted with "never" or it was later shut down.
> This is possible to query by mem_alloc_profiling_permanently_disabled().
>
> To make slabobj_ext array size handling dynamic, we need a snapshot of
> mem_alloc_profiling_permanently_disabled() early in boot, so that's not
> affected by a later shutdown. We also need it to be static key based for
> performance. Neither mem_alloc_profiling_enabled() nor
> mem_alloc_profiling_permanently_disabled() satisfy this.
>
> Therefore introduce slab_obj_ext_has_codetag() with an underlying static
> key for that use case. Its state is made to reflect the result of
> mem_alloc_profiling_permanently_disabled() during kmem_cache_init(),
> which does happen after setup_early_mem_profiling().
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/slab.h | 16 ++++++++++++++++
> mm/slub.c | 31 +++++++++++++++++++++++++++++++
> 2 files changed, 47 insertions(+)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index 359ab8caf61e..dcca86799fc9 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -564,6 +564,22 @@ struct slabobj_ext {
> };
> } __aligned(8);
>
> +#ifdef CONFIG_MEM_ALLOC_PROFILING
> +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT,
> + slab_obj_ext_has_codetag_key);
> +
> +static inline bool slab_obj_ext_has_codetag(void)
> +{
> + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT,
> + &slab_obj_ext_has_codetag_key);
> +}
> +#else
> +static inline bool slab_obj_ext_has_codetag(void)
> +{
> + return false;
> +}
> +#endif
> +
> static inline size_t static_obj_ext_size(void)
> {
> size_t sz = 0;
> diff --git a/mm/slub.c b/mm/slub.c
> index dd15af8abd62..4200e7105b30 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -213,6 +213,11 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled);
> static DEFINE_STATIC_KEY_FALSE(strict_numa);
> #endif
>
> +#ifdef CONFIG_MEM_ALLOC_PROFILING
> +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT,
> + slab_obj_ext_has_codetag_key);
> +#endif
> +
> /* Structure holding extra parameters for slab allocations */
> struct slab_alloc_context {
> unsigned long caller_addr;
> @@ -2415,6 +2420,26 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
> __alloc_tagging_slab_free_hook(s, slab, p, objects);
> }
>
> +/*
> + * Make sure the static key used by slab_obj_ext_has_codetag() reflects the
> + * value of !mem_alloc_profiling_permanently_disabled()
> + *
> + * Any later mem alloc profiling shutdown won't be reflected in the static key
> + * because obj_exts with codetags might already exist.
> + */
> +static void __init slab_obj_ext_has_codetag_init(void)
> +{
> + bool key_enabled = IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT);
nit: This seems a bit indirect. Reader needs to remember that
slab_obj_ext_has_codetag_key is initialized based on
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Wouldn't below code be
simpler?:
bool need_codetag = !mem_alloc_profiling_permanently_disabled();
if (need_codetag != static_key_enabled(&slab_obj_ext_has_codetag_key)) {
if (need_codetag)
static_branch_enable(&slab_obj_ext_has_codetag_key);
else
static_branch_disable(&slab_obj_ext_has_codetag_key);
}
It might be less performant but who cares, it's __init function used
only one time.
> + bool need_codetag = !mem_alloc_profiling_permanently_disabled();
> +
> + if (key_enabled != need_codetag) {
> + if (need_codetag)
> + static_branch_enable(&slab_obj_ext_has_codetag_key);
> + else
> + static_branch_disable(&slab_obj_ext_has_codetag_key);
> + }
> +}
> +
> #else /* CONFIG_MEM_ALLOC_PROFILING */
>
> static inline void
> @@ -2429,6 +2454,10 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
> {
> }
>
> +static inline void slab_obj_ext_has_codetag_init(void)
> +{
> +}
> +
> #endif /* CONFIG_MEM_ALLOC_PROFILING */
>
>
> @@ -8541,6 +8570,8 @@ void __init kmem_cache_init(void)
> boot_kmem_cache_node;
> int node;
>
> + slab_obj_ext_has_codetag_init();
> +
> if (debug_guardpage_minorder())
> slub_max_order = 0;
>
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled
2026-07-15 10:10 ` [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled Vlastimil Babka (SUSE)
@ 2026-07-16 4:30 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 4:30 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> When memory allocation profiling is compiled in but permanently disabled
> on boot with (implicit or explicit) "never" parameter, stop allocating
> (wasting) memory for the codetag_ref parts of slabobj_ext metadata.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Looks good but we should verify that slab allocation performance is
not regressed, especially if
CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=Y and then profiling was
disabled via "never" in the command line. I think that would be the
worst case scenario for performance.
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/slab.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index dcca86799fc9..a50347c9dbe3 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -587,7 +587,7 @@ static inline size_t static_obj_ext_size(void)
> if (IS_ENABLED(CONFIG_MEMCG))
> sz += 1;
>
> - if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING))
> + if (slab_obj_ext_has_codetag())
> sz += 1;
>
> return sizeof(struct slabobj_ext) * sz;
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper
2026-07-15 10:10 ` [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper Vlastimil Babka (SUSE)
@ 2026-07-16 4:36 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 4:36 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> Slabs of some caches never need the objcg part of struct slabobj_ext.
> Introduce a helper to query this for a slab.
>
> Currently only is_kmalloc_normal() caches are considered as not needing
> objcg's. We could also consider all kmem caches without SLAB_ACCOUNT,
> however some might be used with and without __GFP_ACCOUNT concurrently
> and we currently don't restrict that. This can be improved later.
>
> To make the evaluation of slab_needs_objcg() faster in the allocation
> and free fast paths, add a obj_exts_needs_objcg flag into slab itself.
> This optimization is only available on 64bit architectures where free
> bits are available for the flag.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> ---
> mm/kfence/core.c | 3 +++
> mm/slab.h | 31 +++++++++++++++++++++++++++++--
> mm/slub.c | 4 ++++
> 3 files changed, 36 insertions(+), 2 deletions(-)
>
> diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> index 717e8baf7e5d..afeaf80484ad 100644
> --- a/mm/kfence/core.c
> +++ b/mm/kfence/core.c
> @@ -640,6 +640,9 @@ static unsigned long kfence_init_pool(void)
> struct slab *slab = page_slab(page);
> slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts |
> MEMCG_DATA_OBJEXTS;
> +#ifdef CONFIG_64BIT
> + slab->obj_exts_needs_objcg = 1;
> +#endif
> #endif
> }
>
> diff --git a/mm/slab.h b/mm/slab.h
> index a50347c9dbe3..948d075cdbef 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -81,10 +81,11 @@ struct freelist_counters {
> #ifdef CONFIG_64BIT
> /*
> * Some optimizations use free bits in 'counters' field
> - * to save memory. If these free bits are not available,
> - * such optimizations are disabled.
> + * to save memory or CPU. If these free bits are not
> + * available, such optimizations are disabled.
> */
> unsigned obj_exts_in_object:1;
> + unsigned obj_exts_needs_objcg:1;
> #endif
> };
> };
> @@ -580,6 +581,32 @@ static inline bool slab_obj_ext_has_codetag(void)
> }
> #endif
>
> +#ifdef CONFIG_MEMCG
> +static inline bool cache_needs_objcg(struct kmem_cache *cache)
> +{
> + return !is_kmalloc_normal(cache);
> +}
> +
> +static inline bool slab_needs_objcg(struct slab *slab)
> +{
> +#ifdef CONFIG_64BIT
> + return slab->obj_exts_needs_objcg;
> +#else
> + return cache_needs_objcg(slab->slab_cache);
> +#endif
> +}
> +#else
> +static inline bool cache_needs_objcg(struct kmem_cache *cache)
> +{
> + return false;
> +}
> +
> +static inline bool slab_needs_objcg(struct slab *slab)
> +{
> + return false;
> +}
> +#endif
> +
> static inline size_t static_obj_ext_size(void)
> {
> size_t sz = 0;
> diff --git a/mm/slub.c b/mm/slub.c
> index 4200e7105b30..771d73abacb6 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3426,6 +3426,10 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
>
> slab->objects = oo_objects(oo);
>
> +#ifdef CONFIG_64BIT
> + if (cache_needs_objcg(s))
> + slab->obj_exts_needs_objcg = 1;
> +#endif
> slab->slab_cache = s;
>
> kasan_poison_slab(slab);
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary
2026-07-15 10:10 ` [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary Vlastimil Babka (SUSE)
@ 2026-07-16 4:46 ` Suren Baghdasaryan
2026-07-16 15:08 ` Vlastimil Babka (SUSE)
0 siblings, 1 reply; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 4:46 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> Start using the slab_needs_objcg() helper to calculate slabobj_ext size.
> Caches that we know to never need objcg pointers (currently
> KMALLOC_NORMAL caches) will thus stop wasting memory on them when memory
> allocation profiling is enabled.
>
> For things to work properly, we need to also add slab_needs_objcg()
> checks to mem_cgroup_from_obj_slab() and memcg_slab_free_hook(), because
> when obj_exts array exists for a slab only due to mem_alloc profiling,
> we would otherwise attempt to access a non-existing objcg pointer in
> that slab.
>
> The function __memcg_slab_post_alloc_hook() should not be possible to
> call for a slab where slab_needs_objcg() is false, but add a DEBUG_VM
> check there to prevent breaking this assumption accidentally.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
The change looks nice and simple. I'll take some more time to check if
we missed anything.
Only one nit below.
> ---
> mm/memcontrol.c | 6 ++++++
> mm/slab.h | 12 ++++++++++--
> mm/slub.c | 3 +++
> 3 files changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 6303a2b1a9d0..09659722ec85 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2871,6 +2871,9 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
> if (!obj_exts)
> return NULL;
>
> + if (!slab_needs_objcg(slab))
> + return NULL;
> +
> get_slab_obj_exts(obj_exts);
> obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
> objcg = *slab_obj_ext_objcgp(obj_ext);
> @@ -3581,6 +3584,9 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>
> slab = virt_to_slab(p[i]);
>
> + if (IS_ENABLED(CONFIG_DEBUG_VM) && WARN_ON_ONCE(!slab_needs_objcg(slab)))
> + continue;
> +
> if (!slab_obj_exts(slab) &&
> alloc_slab_obj_exts(slab, s, flags, slab_alloc_flags)) {
> continue;
> diff --git a/mm/slab.h b/mm/slab.h
> index 948d075cdbef..072cc2506756 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -622,7 +622,15 @@ static inline size_t static_obj_ext_size(void)
>
> static inline size_t slab_obj_ext_size(struct slab *slab)
> {
> - return static_obj_ext_size();
Maybe now we should rename static_obj_ext_size() to
static_obj_ext_max_size() as it reflects the max possible size of
slabobj_ext?
> + size_t sz = 0;
> +
> + if (slab_needs_objcg(slab))
> + sz += 1;
> +
> + if (slab_obj_ext_has_codetag())
> + sz += 1;
> +
> + return sizeof(struct slabobj_ext) * sz;
> }
>
> #ifdef CONFIG_SLAB_OBJ_EXT
> @@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
> static inline union codetag_ref *
> slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
> {
> - if (IS_ENABLED(CONFIG_MEMCG))
> + if (slab_needs_objcg(slab))
> obj_ext += 1;
>
> return &obj_ext->_ctref;
> diff --git a/mm/slub.c b/mm/slub.c
> index 771d73abacb6..09c4931e5435 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
> if (likely(!obj_exts))
> return;
>
> + if (!slab_needs_objcg(slab))
> + return;
> +
> get_slab_obj_exts(obj_exts);
> __memcg_slab_free_hook(s, slab, p, objects, obj_exts);
> put_slab_obj_exts(obj_exts);
>
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste
2026-07-16 3:28 ` Harry Yoo
@ 2026-07-16 4:55 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 4:55 UTC (permalink / raw)
To: Harry Yoo
Cc: Vlastimil Babka (SUSE), Hao Li, Shakeel Butt, Alexander Potapenko,
Marco Elver, Andrew Morton, Christoph Lameter, David Rientjes,
Roman Gushchin, linux-mm, linux-kernel, cgroups
On Wed, Jul 15, 2026 at 8:29 PM Harry Yoo <harry@kernel.org> wrote:
>
>
>
> On 7/16/26 12:32 AM, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >>
> >> The recent fixes for objext array handling inspired me to look into this
> >> finally. It's been bothering me that the memory usage of struct
> >> slabobj_ext depend only on config options and not whether the fields are
> >> actually used.
>
> +1 one more person bothered by this...
>
> >> So with both CONFIG_MEMCG=y and
> >> CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref
> >> field. And thus:
> >>
> >> 1) Having memory allocation profiling config-enabled but not
> >> boot-enabled means wasted memory on unused codetag_refs. This makes
> >> it less suitable for a general distro config and the page allocator
> >> side doesn't suffer from this, only slab and percpu.
> >>
> >> 2) Complementary, with memory allocation profiling enabled, there are
> >> caches/slabs that don't need the objcg field, so memory is wasted on
> >> those.
> >
> > It's funny because yesterday I started working on a prototype for the
> > same optimization. But your patchset is much more mature, so I'll
> > focus instead on reviewing it.
>
> Ouch, a race condition!
>
> >> This series should solve the point 1) fully for slab, pcpuobj_ext
> >> handling can be perhaps improved similarly, haven't looked into that.
> >>
> >> For 2) it avoids allocating objcg fields for KMALLOC_NORMAL caches where
> >> we know they are not necessary because kmalloc() with __GFP_ACCOUNT will
> >> pick a KMALLOC_CGROUP type.
>
> Unless KMALLOC_RECLAIM != KMALLOC_NORMAL! (yes, SLUB_TINY)
>
> >> The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT
> >> and then we know objcg fields are always needed. But also they can be
> >> created without SLAB_ACCOUNT and then some allocations have
> >> __GFP_ACCOUNT and some not and we don't know that in advance.
> >
> > Do you know how often this happens that a named cache with no
> > SLAB_ACCOUNT is used for __GFP_ACCOUNT allocation?
>
> Not sure about how often, but one thing I recall is xarray
> (radix_tree_node cache), which decides to account the objects based on
> xarray flags.
>
> >> A possible future solution is to introduce e.g. SLAB_MAYBE_ACCOUNT, add
> >> it to caches where we know __GFP_ACCOUNT is used, and only honour
> >> __GFP_ACCOUNT for those, while warning for an unexpected usage
> >> elsewhere.
> >
> > I wonder if for such caches we could create two separate caches, one
> > serving __GFP_ACCOUNT and using extentions containing objcg and
> > another one for non-__GFP_ACCOUNT with optimized extentions?
>
> You mean transparently to users? (e.g., user thinks it has created
> a single kmem_cache but actually there are two of them, multiplexed by
> __GFP_ACCOUNT bit)
Yeah and only when we detect that a cache that does not have
SLAB_ACCOUNT is used to allocate with __GFP_ACCOUNT set. Not sure
about the performance impact though...
>
> --
> Cheers,
> Harry / Hyeonggon
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling
2026-07-15 16:02 ` Suren Baghdasaryan
@ 2026-07-16 9:11 ` Vlastimil Babka (SUSE)
2026-07-16 15:24 ` Suren Baghdasaryan
0 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-16 9:11 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On 7/15/26 18:02, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>>
>> struct kfence_metadata only has obj_exts with CONFIG_MEMCG.
>
> I don't quite understand this statement. obj_exts are allocated when
> either CONFIG_MEMCG or CONFIG_MEM_ALLOC_PROFILING is enabled.
See in mm/kfence/kfence.h
struct kfence_metadata {
...
#ifdef CONFIG_MEMCG
struct slabobj_ext obj_exts;
#endif
...
then in mm/kfence/core.c kfence_init_pool()
#ifdef CONFIG_MEMCG
struct slab *slab = page_slab(page);
slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts |
MEMCG_DATA_OBJEXTS;
#endif
So the slab kfence fakes only has this pre-inited obj_exts
with CONFIG_MEMCG.
What happens if we run memalloc profiling without MEMCG and this
fake slab doesn't have a pre-assigned obj_exts? I guess
__alloc_tagging_slab_alloc_hook() will allocate it via
prepare_slab_obj_exts_hook().
But it's something that was done consciously and probably
just accidentally works.
>> If it's
>> enabled, it does also work for allocation profiling, but there's little
>> value recording tags for KFENCE objects.
>
> Unless we are leaking them, right?
Well if there are leaks in a particular callsite, we should see that from all
the allocations that don't end up in kfence (as kfence allocations are rare).
So we are very unlikely to miss a leak due to this.
>> Furthermore it would complicate
>> the upcoming changes, so just skip them in the slab hooks.
>
> Ok, I can understand that. If we do this, we should document that
> kfence objects are no longer tracked.
>
>>
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>> ---
>> mm/slub.c | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index 0337e60db5ac..a4be70d080fb 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
>> if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
>> return;
>>
>> + if (is_kfence_address(object))
>> + return;
>> +
>> slab = virt_to_slab(object);
>> obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
>> /*
>> @@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
>> for (i = 0; i < objects; i++) {
>> unsigned int off = obj_to_index(s, slab, p[i]);
>>
>> + if (is_kfence_address(p[i]))
>> + continue;
>> +
>> alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
>> }
>> put_slab_obj_exts(obj_exts);
>>
>> --
>> 2.55.0
>>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access
2026-07-16 1:27 ` Suren Baghdasaryan
@ 2026-07-16 13:58 ` Vlastimil Babka (SUSE)
2026-07-16 15:27 ` Suren Baghdasaryan
0 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-16 13:58 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On 7/16/26 03:27, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>> --- a/mm/slab.h
>> +++ b/mm/slab.h
>> @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
>> */
>> struct slabobj_ext {
>> #ifdef CONFIG_MEMCG
>> - struct obj_cgroup *objcg;
>> + struct obj_cgroup *_objcg;
>> #endif
>> #ifdef CONFIG_MEM_ALLOC_PROFILING
>> union codetag_ref ref;
>> @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
>> return kasan_reset_tag(obj_ext);
>> }
>>
>> +#ifdef CONFIG_MEMCG
>> +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
>
> I understand why you need to return obj_cgroup** but maybe instead we
> can have a separate setter function and just return a pointer here?
Yeah I rewrote it now and it looks better. Thanks! It ended up like this due
to some previous attempt I made.
> That would also help catch cases when someone wants to set
> obj_ext._objcg that was optimized away (for that the setter would need
> an additional parameter to identify whether we allocated optimized
> objext vector).
It would need passing the slab parameter to assert
slab_obj_ext_has_codetag() in debug mode. I don't think it's worth it given
the get/set is only called from few specialized places.
>
>> +{
>> + return &obj_ext->_objcg;
>> +}
>> +#endif
>> +
>> int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>> gfp_t gfp, unsigned int alloc_flags);
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index 5e3f53bcd0d3..48e10198a3ce 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
>> if (obj_exts) {
>> get_slab_obj_exts(obj_exts);
>> obj_ext = slab_obj_ext(s, slab, obj_exts, p);
>> - if (unlikely(obj_ext->objcg)) {
>> + if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
>> put_slab_obj_exts(obj_exts);
>> return true;
>> }
>>
>> --
>> 2.55.0
>>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union
2026-07-16 4:11 ` Suren Baghdasaryan
@ 2026-07-16 14:43 ` Vlastimil Babka (SUSE)
2026-07-16 15:28 ` Suren Baghdasaryan
0 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-16 14:43 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On 7/16/26 06:11, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>> No functional change intended, the layout is still effectively static.
>>
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>
> This is much cleaner than what I was preparing. Nicely done!
>
> Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Thanks!
>> ---
>> mm/slab.h | 41 +++++++++++++++++++++++++++++++++--------
>> mm/slub.c | 17 +++++++++--------
>> 2 files changed, 42 insertions(+), 16 deletions(-)
>>
>> diff --git a/mm/slab.h b/mm/slab.h
>> index 3ad9777ad600..359ab8caf61e 100644
>> --- a/mm/slab.h
>> +++ b/mm/slab.h
>> @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void)
>> * if MEMCG_DATA_OBJEXTS is set.
>> */
>> struct slabobj_ext {
>
> Perhaps we should add a comment here stating that every element of
> this union should be pointer-sized?
>
OK, how about this?
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -550,10 +550,14 @@ static inline bool need_kmalloc_no_objext(void)
}
/*
- * Extended information for slab objects stored as an array in page->memcg_data
- * if MEMCG_DATA_OBJEXTS is set.
+ * Extended information for slab objects stored as a pointer to an array in
+ * slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set.
*/
struct slabobj_ext {
+ /*
+ * All elements of the union should be pointer-sized to avoid memory
+ * waste
+ */
union {
#ifdef CONFIG_MEMCG
struct obj_cgroup *_objcg;
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag()
2026-07-16 4:25 ` Suren Baghdasaryan
@ 2026-07-16 14:47 ` Vlastimil Babka (SUSE)
0 siblings, 0 replies; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-16 14:47 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On 7/16/26 06:25, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>>
>> mem_alloc_profiling_enabled() allows evaluating (with a static key) if
>> memory profiling is currently enabled. mem_profiling_support is a
>> variable where false means it's not possible to enable it anymore,
>> because the system was booted with "never" or it was later shut down.
>> This is possible to query by mem_alloc_profiling_permanently_disabled().
>>
>> To make slabobj_ext array size handling dynamic, we need a snapshot of
>> mem_alloc_profiling_permanently_disabled() early in boot, so that's not
>> affected by a later shutdown. We also need it to be static key based for
>> performance. Neither mem_alloc_profiling_enabled() nor
>> mem_alloc_profiling_permanently_disabled() satisfy this.
>>
>> Therefore introduce slab_obj_ext_has_codetag() with an underlying static
>> key for that use case. Its state is made to reflect the result of
>> mem_alloc_profiling_permanently_disabled() during kmem_cache_init(),
>> which does happen after setup_early_mem_profiling().
>>
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>
> Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Thanks!
>> +/*
>> + * Make sure the static key used by slab_obj_ext_has_codetag() reflects the
>> + * value of !mem_alloc_profiling_permanently_disabled()
>> + *
>> + * Any later mem alloc profiling shutdown won't be reflected in the static key
>> + * because obj_exts with codetags might already exist.
>> + */
>> +static void __init slab_obj_ext_has_codetag_init(void)
>> +{
>> + bool key_enabled = IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT);
>
> nit: This seems a bit indirect. Reader needs to remember that
> slab_obj_ext_has_codetag_key is initialized based on
> CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Wouldn't below code be
> simpler?:
>
> bool need_codetag = !mem_alloc_profiling_permanently_disabled();
> if (need_codetag != static_key_enabled(&slab_obj_ext_has_codetag_key)) {
> if (need_codetag)
> static_branch_enable(&slab_obj_ext_has_codetag_key);
> else
> static_branch_disable(&slab_obj_ext_has_codetag_key);
> }
>
> It might be less performant but who cares, it's __init function used
> only one time.
Good idea, will do!
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary
2026-07-16 4:46 ` Suren Baghdasaryan
@ 2026-07-16 15:08 ` Vlastimil Babka (SUSE)
2026-07-16 15:32 ` Suren Baghdasaryan
0 siblings, 1 reply; 37+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-07-16 15:08 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On 7/16/26 06:46, Suren Baghdasaryan wrote:
> Maybe now we should rename static_obj_ext_size() to
> static_obj_ext_max_size() as it reflects the max possible size of
> slabobj_ext?
Hm with _max_size() it doesn't have to be called static_ anymore?
But maybe we can get rid of it completely.
The only caller is calculate_sizes():
#if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
if (slab_args_unmergeable(args, s->flags) &&
(aligned_size - size >= static_obj_ext_size()))
s->flags |= SLAB_OBJ_EXT_IN_OBJ;
#endif
We don't have slab pointer to pass to static_obj_ext_max_size().
But we should be able to figure it out from the cache via
cache_needs_objcg().
So we'd need cache_obj_ext_size()?
The static max size here works, but may prevent SLAB_OBJ_EXT_IN_OBJ
needlessly if padding can only fit codetag_ref and not objcg, but
we don't need objcg.
Which is probably only theoretical at this point as kmalloc_normal
caches (that have no objcg) have no padding, at least without
slab_debug. But still.
>> + size_t sz = 0;
>> +
>> + if (slab_needs_objcg(slab))
>> + sz += 1;
>> +
>> + if (slab_obj_ext_has_codetag())
>> + sz += 1;
>> +
>> + return sizeof(struct slabobj_ext) * sz;
>> }
>>
>> #ifdef CONFIG_SLAB_OBJ_EXT
>> @@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
>> static inline union codetag_ref *
>> slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
>> {
>> - if (IS_ENABLED(CONFIG_MEMCG))
>> + if (slab_needs_objcg(slab))
>> obj_ext += 1;
>>
>> return &obj_ext->_ctref;
>> diff --git a/mm/slub.c b/mm/slub.c
>> index 771d73abacb6..09c4931e5435 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
>> if (likely(!obj_exts))
>> return;
>>
>> + if (!slab_needs_objcg(slab))
>> + return;
>> +
>> get_slab_obj_exts(obj_exts);
>> __memcg_slab_free_hook(s, slab, p, objects, obj_exts);
>> put_slab_obj_exts(obj_exts);
>>
>> --
>> 2.55.0
>>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling
2026-07-16 9:11 ` Vlastimil Babka (SUSE)
@ 2026-07-16 15:24 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 15:24 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Thu, Jul 16, 2026 at 2:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/15/26 18:02, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >>
> >> struct kfence_metadata only has obj_exts with CONFIG_MEMCG.
> >
> > I don't quite understand this statement. obj_exts are allocated when
> > either CONFIG_MEMCG or CONFIG_MEM_ALLOC_PROFILING is enabled.
>
> See in mm/kfence/kfence.h
>
> struct kfence_metadata {
> ...
> #ifdef CONFIG_MEMCG
> struct slabobj_ext obj_exts;
> #endif
> ...
>
> then in mm/kfence/core.c kfence_init_pool()
>
> #ifdef CONFIG_MEMCG
> struct slab *slab = page_slab(page);
> slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts |
> MEMCG_DATA_OBJEXTS;
> #endif
>
> So the slab kfence fakes only has this pre-inited obj_exts
> with CONFIG_MEMCG.
>
> What happens if we run memalloc profiling without MEMCG and this
> fake slab doesn't have a pre-assigned obj_exts? I guess
> __alloc_tagging_slab_alloc_hook() will allocate it via
> prepare_slab_obj_exts_hook().
>
> But it's something that was done consciously and probably
> just accidentally works.
Ok, I see. Maybe you can expand this description to explain the
details more thoroughly? This whole kfence.obj_exts deal is not very
intuitive.
>
> >> If it's
> >> enabled, it does also work for allocation profiling, but there's little
> >> value recording tags for KFENCE objects.
> >
> > Unless we are leaking them, right?
>
> Well if there are leaks in a particular callsite, we should see that from all
> the allocations that don't end up in kfence (as kfence allocations are rare).
> So we are very unlikely to miss a leak due to this.
Ok, makes sense. Thanks for the explanation!
>
> >> Furthermore it would complicate
> >> the upcoming changes, so just skip them in the slab hooks.
> >
> > Ok, I can understand that. If we do this, we should document that
> > kfence objects are no longer tracked.
> >
> >>
> >> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> >> ---
> >> mm/slub.c | 6 ++++++
> >> 1 file changed, 6 insertions(+)
> >>
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 0337e60db5ac..a4be70d080fb 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
> >> if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
> >> return;
> >>
> >> + if (is_kfence_address(object))
> >> + return;
> >> +
> >> slab = virt_to_slab(object);
> >> obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
> >> /*
> >> @@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
> >> for (i = 0; i < objects; i++) {
> >> unsigned int off = obj_to_index(s, slab, p[i]);
> >>
> >> + if (is_kfence_address(p[i]))
> >> + continue;
> >> +
> >> alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
> >> }
> >> put_slab_obj_exts(obj_exts);
> >>
> >> --
> >> 2.55.0
> >>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access
2026-07-16 13:58 ` Vlastimil Babka (SUSE)
@ 2026-07-16 15:27 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 15:27 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Thu, Jul 16, 2026 at 6:58 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/16/26 03:27, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >> --- a/mm/slab.h
> >> +++ b/mm/slab.h
> >> @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
> >> */
> >> struct slabobj_ext {
> >> #ifdef CONFIG_MEMCG
> >> - struct obj_cgroup *objcg;
> >> + struct obj_cgroup *_objcg;
> >> #endif
> >> #ifdef CONFIG_MEM_ALLOC_PROFILING
> >> union codetag_ref ref;
> >> @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> >> return kasan_reset_tag(obj_ext);
> >> }
> >>
> >> +#ifdef CONFIG_MEMCG
> >> +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
> >
> > I understand why you need to return obj_cgroup** but maybe instead we
> > can have a separate setter function and just return a pointer here?
>
> Yeah I rewrote it now and it looks better. Thanks! It ended up like this due
> to some previous attempt I made.
>
> > That would also help catch cases when someone wants to set
> > obj_ext._objcg that was optimized away (for that the setter would need
> > an additional parameter to identify whether we allocated optimized
> > objext vector).
>
> It would need passing the slab parameter to assert
> slab_obj_ext_has_codetag() in debug mode. I don't think it's worth it given
> the get/set is only called from few specialized places.
Sounds good.
>
> >
> >> +{
> >> + return &obj_ext->_objcg;
> >> +}
> >> +#endif
> >> +
> >> int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> >> gfp_t gfp, unsigned int alloc_flags);
> >>
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 5e3f53bcd0d3..48e10198a3ce 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
> >> if (obj_exts) {
> >> get_slab_obj_exts(obj_exts);
> >> obj_ext = slab_obj_ext(s, slab, obj_exts, p);
> >> - if (unlikely(obj_ext->objcg)) {
> >> + if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
> >> put_slab_obj_exts(obj_exts);
> >> return true;
> >> }
> >>
> >> --
> >> 2.55.0
> >>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union
2026-07-16 14:43 ` Vlastimil Babka (SUSE)
@ 2026-07-16 15:28 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 15:28 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Thu, Jul 16, 2026 at 7:43 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/16/26 06:11, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >> No functional change intended, the layout is still effectively static.
> >>
> >> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> >
> > This is much cleaner than what I was preparing. Nicely done!
> >
> > Reviewed-by: Suren Baghdasaryan <surenb@google.com>
>
> Thanks!
>
> >> ---
> >> mm/slab.h | 41 +++++++++++++++++++++++++++++++++--------
> >> mm/slub.c | 17 +++++++++--------
> >> 2 files changed, 42 insertions(+), 16 deletions(-)
> >>
> >> diff --git a/mm/slab.h b/mm/slab.h
> >> index 3ad9777ad600..359ab8caf61e 100644
> >> --- a/mm/slab.h
> >> +++ b/mm/slab.h
> >> @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void)
> >> * if MEMCG_DATA_OBJEXTS is set.
> >> */
> >> struct slabobj_ext {
> >
> > Perhaps we should add a comment here stating that every element of
> > this union should be pointer-sized?
> >
>
> OK, how about this?
>
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -550,10 +550,14 @@ static inline bool need_kmalloc_no_objext(void)
> }
>
> /*
> - * Extended information for slab objects stored as an array in page->memcg_data
> - * if MEMCG_DATA_OBJEXTS is set.
> + * Extended information for slab objects stored as a pointer to an array in
> + * slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set.
> */
> struct slabobj_ext {
> + /*
> + * All elements of the union should be pointer-sized to avoid memory
> + * waste
> + */
Perfect. Thank you!
> union {
> #ifdef CONFIG_MEMCG
> struct obj_cgroup *_objcg;
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary
2026-07-16 15:08 ` Vlastimil Babka (SUSE)
@ 2026-07-16 15:32 ` Suren Baghdasaryan
0 siblings, 0 replies; 37+ messages in thread
From: Suren Baghdasaryan @ 2026-07-16 15:32 UTC (permalink / raw)
To: Vlastimil Babka (SUSE)
Cc: Harry Yoo, Hao Li, Shakeel Butt, Alexander Potapenko, Marco Elver,
Andrew Morton, Christoph Lameter, David Rientjes, Roman Gushchin,
linux-mm, linux-kernel, cgroups
On Thu, Jul 16, 2026 at 8:08 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/16/26 06:46, Suren Baghdasaryan wrote:
> > Maybe now we should rename static_obj_ext_size() to
> > static_obj_ext_max_size() as it reflects the max possible size of
> > slabobj_ext?
>
> Hm with _max_size() it doesn't have to be called static_ anymore?
Yeah, static_ is an implementation detail anyway.
>
> But maybe we can get rid of it completely.
>
> The only caller is calculate_sizes():
I was thinking the same but either way sounds fine to me. Keeping
obj_ext_max_size() would be a bit more self-documenting I think.
>
> #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
> if (slab_args_unmergeable(args, s->flags) &&
> (aligned_size - size >= static_obj_ext_size()))
> s->flags |= SLAB_OBJ_EXT_IN_OBJ;
> #endif
>
> We don't have slab pointer to pass to static_obj_ext_max_size().
> But we should be able to figure it out from the cache via
> cache_needs_objcg().
>
> So we'd need cache_obj_ext_size()?
>
> The static max size here works, but may prevent SLAB_OBJ_EXT_IN_OBJ
> needlessly if padding can only fit codetag_ref and not objcg, but
> we don't need objcg.
> Which is probably only theoretical at this point as kmalloc_normal
> caches (that have no objcg) have no padding, at least without
> slab_debug. But still.
Maybe keep it simple as is for now and then we can optimize it further?
>
> >> + size_t sz = 0;
> >> +
> >> + if (slab_needs_objcg(slab))
> >> + sz += 1;
> >> +
> >> + if (slab_obj_ext_has_codetag())
> >> + sz += 1;
> >> +
> >> + return sizeof(struct slabobj_ext) * sz;
> >> }
> >>
> >> #ifdef CONFIG_SLAB_OBJ_EXT
> >> @@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
> >> static inline union codetag_ref *
> >> slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
> >> {
> >> - if (IS_ENABLED(CONFIG_MEMCG))
> >> + if (slab_needs_objcg(slab))
> >> obj_ext += 1;
> >>
> >> return &obj_ext->_ctref;
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 771d73abacb6..09c4931e5435 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
> >> if (likely(!obj_exts))
> >> return;
> >>
> >> + if (!slab_needs_objcg(slab))
> >> + return;
> >> +
> >> get_slab_obj_exts(obj_exts);
> >> __memcg_slab_free_hook(s, slab, p, objects, obj_exts);
> >> put_slab_obj_exts(obj_exts);
> >>
> >> --
> >> 2.55.0
> >>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2026-07-16 15:33 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-15 10:10 [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Vlastimil Babka (SUSE)
2026-07-15 10:10 ` [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling Vlastimil Babka (SUSE)
2026-07-15 16:02 ` Suren Baghdasaryan
2026-07-16 9:11 ` Vlastimil Babka (SUSE)
2026-07-16 15:24 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 02/12] mm/slab: remove objs_per_slab() Vlastimil Babka (SUSE)
2026-07-15 16:13 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h Vlastimil Babka (SUSE)
2026-07-16 0:56 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index Vlastimil Babka (SUSE)
2026-07-16 1:02 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access Vlastimil Babka (SUSE)
2026-07-16 1:27 ` Suren Baghdasaryan
2026-07-16 13:58 ` Vlastimil Babka (SUSE)
2026-07-16 15:27 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access Vlastimil Babka (SUSE)
2026-07-16 1:28 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object Vlastimil Babka (SUSE)
2026-07-16 1:44 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union Vlastimil Babka (SUSE)
2026-07-16 4:11 ` Suren Baghdasaryan
2026-07-16 14:43 ` Vlastimil Babka (SUSE)
2026-07-16 15:28 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag() Vlastimil Babka (SUSE)
2026-07-16 4:25 ` Suren Baghdasaryan
2026-07-16 14:47 ` Vlastimil Babka (SUSE)
2026-07-15 10:10 ` [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled Vlastimil Babka (SUSE)
2026-07-16 4:30 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper Vlastimil Babka (SUSE)
2026-07-16 4:36 ` Suren Baghdasaryan
2026-07-15 10:10 ` [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary Vlastimil Babka (SUSE)
2026-07-16 4:46 ` Suren Baghdasaryan
2026-07-16 15:08 ` Vlastimil Babka (SUSE)
2026-07-16 15:32 ` Suren Baghdasaryan
2026-07-15 15:32 ` [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste Suren Baghdasaryan
2026-07-16 3:28 ` Harry Yoo
2026-07-16 4:55 ` Suren Baghdasaryan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.