Re: [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste

Re: [PATCH RFC 00/12] mm/slab, alloc_tag: reduce obj_ext memory waste

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> The recent fixes for objext array handling inspired me to look into this
> finally. It's been bothering me that the memory usage of struct
> slabobj_ext depend only on config options and not whether the fields are
> actually used. So with both CONFIG_MEMCG=y and
> CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref
> field. And thus:
>
> 1) Having memory allocation profiling config-enabled but not
>    boot-enabled means wasted memory on unused codetag_refs. This makes
>    it less suitable for a general distro config and the page allocator
>    side doesn't suffer from this, only slab and percpu.
>
> 2) Complementary, with memory allocation profiling enabled, there are
>    caches/slabs that don't need the objcg field, so memory is wasted on
>    those.

It's funny because yesterday I started working on a prototype for the
same optimization. But your patchset is much more mature, so I'll
focus instead on reviewing it.

>
> This series should solve the point 1) fully for slab, pcpuobj_ext
> handling can be perhaps improved similarly, haven't looked into that.
>
> For 2) it avoids allocating objcg fields for KMALLOC_NORMAL caches where
> we know they are not necessary because kmalloc() with __GFP_ACCOUNT will
> pick a KMALLOC_CGROUP type.
>
> The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT
> and then we know objcg fields are always needed. But also they can be
> created without SLAB_ACCOUNT and then some allocations have
> __GFP_ACCOUNT and some not and we don't know that in advance.

Do you know how often this happens that a named cache with no
SLAB_ACCOUNT is used for __GFP_ACCOUNT allocation?

>
> A possible future solution is to introduce e.g. SLAB_MAYBE_ACCOUNT, add
> it to caches where we know __GFP_ACCOUNT is used, and only honour
> __GFP_ACCOUNT for those, while warning for an unexpected usage
> elsewhere.

I wonder if for such caches we could create two separate caches, one
serving __GFP_ACCOUNT and using extentions containing objcg and
another one for non-__GFP_ACCOUNT with optimized extentions?

>
> Only lightly tested, need to run at least some microbenchmarks to see if
> the now somewhat more complicated access to objcg is visible or not.
>
> Based on slab/for-next-fixes
>
> Git branch: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/objext_split
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
> Vlastimil Babka (SUSE) (12):
>       mm/slab: skip kfence objects in allocation profiling
>       mm/slab: remove objs_per_slab()
>       mm: move struct slabobj_ext to mm/slab.h
>       mm/slab: make slab_obj_ext() determine object index
>       mm/slab: abstract slabobj_ext.objcg access
>       mm/slab: abstract slabobj_ext.ref access
>       mm/slab: replace slab.stride with obj_exts_in_object
>       mm/slab: change struct slabobj_ext to a union
>       mm/slab: introduce slab_obj_ext_has_codetag()
>       mm/slab: reduce slabobj_ext memory with allocation profiling disabled
>       mm/slab: add slab_needs_objcg() helper
>       mm/slab: stop allocating objcg pointers when unnecessary
>
>  include/linux/memcontrol.h |  13 ----
>  mm/kfence/core.c           |   5 +-
>  mm/kfence/kfence_test.c    |   2 +-
>  mm/memcontrol.c            |  41 ++++++-----
>  mm/slab.h                  | 169 ++++++++++++++++++++++++++++++++++++---------
>  mm/slub.c                  | 162 +++++++++++++++++++++++++++----------------
>  6 files changed, 267 insertions(+), 125 deletions(-)
> ---
> base-commit: d9e6a7623938968e3752b67e37eaff097e559a54
> change-id: 20260714-b4-objext_split-da82426257d5
>

Re: [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> struct kfence_metadata only has obj_exts with CONFIG_MEMCG.

I don't quite understand this statement. obj_exts are allocated when
either CONFIG_MEMCG or CONFIG_MEM_ALLOC_PROFILING is enabled.

> If it's
> enabled, it does also work for allocation profiling, but there's little
> value recording tags for KFENCE objects.

Unless we are leaking them, right?

> Furthermore it would complicate
> the upcoming changes, so just skip them in the slab hooks.

Ok, I can understand that. If we do this, we should document that
kfence objects are no longer tracked.

>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
>  mm/slub.c | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 0337e60db5ac..a4be70d080fb 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
>         if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
>                 return;
>
> +       if (is_kfence_address(object))
> +               return;
> +
>         slab = virt_to_slab(object);
>         obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
>         /*
> @@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
>         for (i = 0; i < objects; i++) {
>                 unsigned int off = obj_to_index(s, slab, p[i]);
>
> +               if (is_kfence_address(p[i]))
> +                       continue;
> +
>                 alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
>         }
>         put_slab_obj_exts(obj_exts);
>
> --
> 2.55.0
>

Re: [PATCH RFC 02/12] mm/slab: remove objs_per_slab()

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> The function has an unused kmem_cache argument and almost nothing uses
> it anyway; doing slab->objects is simpler. Remove it with the last two
> users. KUNIT_EXPECT_EQ() needs a cast to avoid "error: ‘typeof’ applied
> to a bit-field" but we don't need to keep a wrapper just for that.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

Reviewed-by: Suren Baghdasayan <surenb@google.com>

> ---
>  mm/kfence/kfence_test.c | 2 +-
>  mm/slab.h               | 6 ------
>  mm/slub.c               | 3 +--
>  3 files changed, 2 insertions(+), 9 deletions(-)
>
> diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
> index de2d0f7d62b1..9867c03ef0ae 100644
> --- a/mm/kfence/kfence_test.c
> +++ b/mm/kfence/kfence_test.c
> @@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat
>                          * memcg accounting works correctly.
>                          */
>                         KUNIT_EXPECT_EQ(test, obj_to_index(s, slab, alloc), 0U);
> -                       KUNIT_EXPECT_EQ(test, objs_per_slab(s, slab), 1);
> +                       KUNIT_EXPECT_EQ(test, ((unsigned int)slab->objects), 1);
>
>                         if (policy == ALLOCATE_ANY)
>                                 return alloc;
> diff --git a/mm/slab.h b/mm/slab.h
> index f5e336b6b6b0..01535e1e2d3c 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,
>         return __obj_to_index(cache, slab_address(slab), obj);
>  }
>
> -static inline int objs_per_slab(const struct kmem_cache *cache,
> -                               const struct slab *slab)
> -{
> -       return slab->objects;
> -}
> -
>  /*
>   * State of the slab allocator.
>   *
> diff --git a/mm/slub.c b/mm/slub.c
> index a4be70d080fb..9e25f2dce7a6 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2127,7 +2127,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>                         gfp_t gfp, unsigned int alloc_flags)
>  {
>         const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
> -       unsigned int objects = objs_per_slab(s, slab);
>         bool new_slab = alloc_flags & SLAB_ALLOC_NEW_SLAB;
>         unsigned long new_exts;
>         unsigned long old_exts;
> @@ -2183,7 +2182,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>  #endif
>  retry:
>         old_exts = READ_ONCE(slab->obj_exts);
> -       handle_failed_objexts_alloc(old_exts, vec, objects);
> +       handle_failed_objexts_alloc(old_exts, vec, slab->objects);
>
>         if (new_slab) {
>                 /*
>
> --
> 2.55.0
>

Re: [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> Users of include/linux/memcontrol.h don't need to see this internal
> structure. Further changes to the struct will reduce recompiling.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

Reviewed-by: Suren Baghdasaryan <surenb@google.com>

> ---
>  include/linux/memcontrol.h | 13 -------------
>  mm/slab.h                  | 13 +++++++++++++
>  2 files changed, 13 insertions(+), 13 deletions(-)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index e1f46a0016fc..93869cc35c25 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1440,19 +1440,6 @@ static inline void mem_cgroup_flush_workqueue(void) { }
>  static inline int mem_cgroup_init(void) { return 0; }
>  #endif /* CONFIG_MEMCG */
>
> -/*
> - * Extended information for slab objects stored as an array in page->memcg_data
> - * if MEMCG_DATA_OBJEXTS is set.
> - */
> -struct slabobj_ext {
> -#ifdef CONFIG_MEMCG
> -       struct obj_cgroup *objcg;
> -#endif
> -#ifdef CONFIG_MEM_ALLOC_PROFILING
> -       union codetag_ref ref;
> -#endif
> -} __aligned(8);
> -
>  static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
>  {
>         struct mem_cgroup *memcg;
> diff --git a/mm/slab.h b/mm/slab.h
> index 01535e1e2d3c..7bd361447c54 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -549,6 +549,19 @@ static inline bool need_kmalloc_no_objext(void)
>         return false;
>  }
>
> +/*
> + * Extended information for slab objects stored as an array in page->memcg_data
> + * if MEMCG_DATA_OBJEXTS is set.
> + */
> +struct slabobj_ext {
> +#ifdef CONFIG_MEMCG
> +       struct obj_cgroup *objcg;
> +#endif
> +#ifdef CONFIG_MEM_ALLOC_PROFILING
> +       union codetag_ref ref;
> +#endif
> +} __aligned(8);
> +
>  #ifdef CONFIG_SLAB_OBJ_EXT
>
>  /*
>
> --
> 2.55.0
>

Re: [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> All callers perform the same obj_to_index() calculation to pass the
> index. Simplify by passing object pointer instead and determining the
> index by slab_obj_ext().
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

Reviewed-by: Suren Baghdasaryan <surenb@google.com>

> ---
>  mm/memcontrol.c | 12 +++---------
>  mm/slab.h       | 14 ++++++++------
>  mm/slub.c       | 22 +++++++---------------
>  3 files changed, 18 insertions(+), 30 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 6dc4888a90f3..4e427286a88a 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2865,15 +2865,13 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
>          */
>         unsigned long obj_exts;
>         struct slabobj_ext *obj_ext;
> -       unsigned int off;
>
>         obj_exts = slab_obj_exts(slab);
>         if (!obj_exts)
>                 return NULL;
>
>         get_slab_obj_exts(obj_exts);
> -       off = obj_to_index(slab->slab_cache, slab, p);
> -       obj_ext = slab_obj_ext(slab, obj_exts, off);
> +       obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
>         if (obj_ext->objcg) {
>                 struct obj_cgroup *objcg = obj_ext->objcg;
>
> @@ -3541,7 +3539,6 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>         size_t obj_size = obj_full_size(s);
>         struct obj_cgroup *objcg;
>         struct slab *slab;
> -       unsigned long off;
>         size_t i;
>
>         /*
> @@ -3616,8 +3613,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>
>                 obj_exts = slab_obj_exts(slab);
>                 get_slab_obj_exts(obj_exts);
> -               off = obj_to_index(s, slab, p[i]);
> -               obj_ext = slab_obj_ext(slab, obj_exts, off);
> +               obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
>                 obj_cgroup_get(objcg);
>                 obj_ext->objcg = objcg;
>                 put_slab_obj_exts(obj_exts);
> @@ -3635,10 +3631,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
>                 struct obj_cgroup *objcg;
>                 struct slabobj_ext *obj_ext;
>                 struct obj_stock_pcp *stock;
> -               unsigned int off;
>
> -               off = obj_to_index(s, slab, p[i]);
> -               obj_ext = slab_obj_ext(slab, obj_exts, off);
> +               obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
>                 objcg = obj_ext->objcg;
>                 if (!objcg)
>                         continue;
> diff --git a/mm/slab.h b/mm/slab.h
> index 7bd361447c54..36d067d6e7c0 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -646,14 +646,16 @@ static inline unsigned int slab_get_stride(struct slab *slab)
>   * Returns a pointer to the object extension associated with the object.
>   * Must be called within a section covered by get/put_slab_obj_exts().
>   */
> -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
> -                                              unsigned long obj_exts,
> -                                              unsigned int index)
> +static inline struct slabobj_ext *
> +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> +            const void *obj)
>  {
>         struct slabobj_ext *obj_ext;
> +       unsigned int index;
>
>         VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
>
> +       index = obj_to_index(s, slab, obj);
>         obj_ext = (struct slabobj_ext *)(obj_exts +
>                                          slab_get_stride(slab) * index);
>         return kasan_reset_tag(obj_ext);
> @@ -669,9 +671,9 @@ static inline unsigned long slab_obj_exts(struct slab *slab)
>         return 0;
>  }
>
> -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
> -                                              unsigned long obj_exts,
> -                                              unsigned int index)
> +static inline struct slabobj_ext *
> +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> +            const void *obj)
>  {
>         return NULL;
>  }
> diff --git a/mm/slub.c b/mm/slub.c
> index 9e25f2dce7a6..5e3f53bcd0d3 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2070,11 +2070,10 @@ static inline void mark_obj_codetag_empty(const void *obj)
>         obj_slab = virt_to_slab(obj);
>         slab_exts = slab_obj_exts(obj_slab);
>         if (slab_exts) {
> +               struct slabobj_ext *ext;
> +
>                 get_slab_obj_exts(slab_exts);
> -               unsigned int offs = obj_to_index(obj_slab->slab_cache,
> -                                                obj_slab, obj);
> -               struct slabobj_ext *ext = slab_obj_ext(obj_slab,
> -                                                      slab_exts, offs);
> +               ext = slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj);
>
>                 if (unlikely(is_codetag_empty(&ext->ref))) {
>                         put_slab_obj_exts(slab_exts);
> @@ -2362,10 +2361,8 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
>          * check should be added before alloc_tag_add().
>          */
>         if (obj_exts) {
> -               unsigned int obj_idx = obj_to_index(s, slab, object);
> -
>                 get_slab_obj_exts(obj_exts);
> -               obj_ext = slab_obj_ext(slab, obj_exts, obj_idx);
> +               obj_ext = slab_obj_ext(s, slab, obj_exts, object);
>                 alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size);
>                 put_slab_obj_exts(obj_exts);
>         } else {
> @@ -2386,7 +2383,6 @@ static noinline void
>  __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
>                                int objects)
>  {
> -       int i;
>         unsigned long obj_exts;
>
>         /* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */
> @@ -2398,13 +2394,11 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
>                 return;
>
>         get_slab_obj_exts(obj_exts);
> -       for (i = 0; i < objects; i++) {
> -               unsigned int off = obj_to_index(s, slab, p[i]);
> -
> +       for (int i = 0; i < objects; i++) {
>                 if (is_kfence_address(p[i]))
>                         continue;
>
> -               alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
> +               alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size);
>         }
>         put_slab_obj_exts(obj_exts);
>  }
> @@ -2489,7 +2483,6 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
>         struct kmem_cache *s;
>         struct page *page;
>         struct slab *slab;
> -       unsigned long off;
>
>         page = virt_to_page(p);
>         if (PageLargeKmalloc(page)) {
> @@ -2529,8 +2522,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
>         obj_exts = slab_obj_exts(slab);
>         if (obj_exts) {
>                 get_slab_obj_exts(obj_exts);
> -               off = obj_to_index(s, slab, p);
> -               obj_ext = slab_obj_ext(slab, obj_exts, off);
> +               obj_ext = slab_obj_ext(s, slab, obj_exts, p);
>                 if (unlikely(obj_ext->objcg)) {
>                         put_slab_obj_exts(obj_exts);
>                         return true;
>
> --
> 2.55.0
>

Re: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> In preparation for changes to the structure, abstract access to the
> objcg field with a slab_obj_ext_objcgp() function.
> Rename the field to _objcg to make an unexpected direct access a compile
> error.
>
> No functional change intended.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
>  mm/kfence/core.c |  2 +-
>  mm/memcontrol.c  | 23 +++++++++++++++--------
>  mm/slab.h        |  9 ++++++++-
>  mm/slub.c        |  2 +-
>  4 files changed, 25 insertions(+), 11 deletions(-)
>
> diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> index 6577bd76954e..717e8baf7e5d 100644
> --- a/mm/kfence/core.c
> +++ b/mm/kfence/core.c
> @@ -1249,7 +1249,7 @@ void __kfence_free(void *addr)
>         struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
>
>  #ifdef CONFIG_MEMCG
> -       KFENCE_WARN_ON(meta->obj_exts.objcg);
> +       KFENCE_WARN_ON(*slab_obj_ext_objcgp(&meta->obj_exts));
>  #endif
>         /*
>          * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 4e427286a88a..6303a2b1a9d0 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
>          */
>         unsigned long obj_exts;
>         struct slabobj_ext *obj_ext;
> +       struct obj_cgroup *objcg;
>
>         obj_exts = slab_obj_exts(slab);
>         if (!obj_exts)
> @@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
>
>         get_slab_obj_exts(obj_exts);
>         obj_ext = slab_obj_ext(slab->slab_cache, slab, obj_exts, p);
> -       if (obj_ext->objcg) {
> -               struct obj_cgroup *objcg = obj_ext->objcg;
> -
> +       objcg = *slab_obj_ext_objcgp(obj_ext);
> +       if (objcg) {
>                 put_slab_obj_exts(obj_exts);
>                 return obj_cgroup_memcg(objcg);
>         }
> @@ -3577,6 +3577,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>                 unsigned long obj_exts;
>                 struct slabobj_ext *obj_ext;
>                 struct obj_stock_pcp *stock;
> +               struct obj_cgroup **objcgp;
>
>                 slab = virt_to_slab(p[i]);
>
> @@ -3612,10 +3613,15 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>                 unlock_stock(stock);
>
>                 obj_exts = slab_obj_exts(slab);
> +
>                 get_slab_obj_exts(obj_exts);
> +
>                 obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> +               objcgp = slab_obj_ext_objcgp(obj_ext);
> +
>                 obj_cgroup_get(objcg);
> -               obj_ext->objcg = objcg;
> +               *objcgp = objcg;
> +
>                 put_slab_obj_exts(obj_exts);
>         }
>
> @@ -3628,16 +3634,17 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
>         size_t obj_size = obj_full_size(s);
>
>         for (int i = 0; i < objects; i++) {
> -               struct obj_cgroup *objcg;
> +               struct obj_cgroup **objcgp, *objcg;
>                 struct slabobj_ext *obj_ext;
>                 struct obj_stock_pcp *stock;
>
>                 obj_ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> -               objcg = obj_ext->objcg;
> -               if (!objcg)
> +               objcgp = slab_obj_ext_objcgp(obj_ext);
> +               if (!*objcgp)
>                         continue;
>
> -               obj_ext->objcg = NULL;
> +               objcg = *objcgp;
> +               *objcgp = NULL;
>
>                 stock = trylock_stock();
>                 __refill_obj_stock(objcg, stock, obj_size, true);
> diff --git a/mm/slab.h b/mm/slab.h
> index 36d067d6e7c0..789bd292075f 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
>   */
>  struct slabobj_ext {
>  #ifdef CONFIG_MEMCG
> -       struct obj_cgroup *objcg;
> +       struct obj_cgroup *_objcg;
>  #endif
>  #ifdef CONFIG_MEM_ALLOC_PROFILING
>         union codetag_ref ref;
> @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
>         return kasan_reset_tag(obj_ext);
>  }
>
> +#ifdef CONFIG_MEMCG
> +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)

I understand why you need to return obj_cgroup** but maybe instead we
can have a separate setter function and just return a pointer here?
That would also help catch cases when someone wants to set
obj_ext._objcg that was optimized away (for that the setter would need
an additional parameter to identify whether we allocated optimized
objext vector).

> +{
> +       return &obj_ext->_objcg;
> +}
> +#endif
> +
>  int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>                         gfp_t gfp, unsigned int alloc_flags);
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 5e3f53bcd0d3..48e10198a3ce 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
>         if (obj_exts) {
>                 get_slab_obj_exts(obj_exts);
>                 obj_ext = slab_obj_ext(s, slab, obj_exts, p);
> -               if (unlikely(obj_ext->objcg)) {
> +               if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
>                         put_slab_obj_exts(obj_exts);
>                         return true;
>                 }
>
> --
> 2.55.0
>

Re: [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> In preparation for changes to the structure, abstract access to the ref
> field with a slab_obj_ext_codetag_ref() function. Rename the field to
> _ctref to make an unexpected direct access a compile error.
>
> No functional change intended.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

Reviewed-by: Suren Baghdasaryan <surenb@google.com>

> ---
>  mm/slab.h | 10 +++++++++-
>  mm/slub.c | 42 ++++++++++++++++++++++++++++--------------
>  2 files changed, 37 insertions(+), 15 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index 789bd292075f..e3f8e42070f1 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -558,7 +558,7 @@ struct slabobj_ext {
>         struct obj_cgroup *_objcg;
>  #endif
>  #ifdef CONFIG_MEM_ALLOC_PROFILING
> -       union codetag_ref ref;
> +       union codetag_ref _ctref;
>  #endif
>  } __aligned(8);
>
> @@ -668,6 +668,14 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
>  }
>  #endif
>
> +#ifdef CONFIG_MEM_ALLOC_PROFILING
> +static inline union codetag_ref *
> +slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
> +{
> +       return &obj_ext->_ctref;
> +}
> +#endif
> +
>  int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>                         gfp_t gfp, unsigned int alloc_flags);
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 48e10198a3ce..2bfcabc4c51a 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2071,18 +2071,20 @@ static inline void mark_obj_codetag_empty(const void *obj)
>         slab_exts = slab_obj_exts(obj_slab);
>         if (slab_exts) {
>                 struct slabobj_ext *ext;
> +               union codetag_ref *ref;
>
>                 get_slab_obj_exts(slab_exts);
>                 ext = slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj);
> +               ref = slab_obj_ext_codetag_ref(obj_slab, ext);
>
> -               if (unlikely(is_codetag_empty(&ext->ref))) {
> +               if (unlikely(is_codetag_empty(ref))) {
>                         put_slab_obj_exts(slab_exts);
>                         return;
>                 }
>
>                 /* codetag should be NULL here */
> -               WARN_ON(ext->ref.ct);
> -               set_codetag_empty(&ext->ref);
> +               WARN_ON(ref->ct);
> +               set_codetag_empty(ref);
>                 put_slab_obj_exts(slab_exts);
>         }
>  }
> @@ -2092,19 +2094,22 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)
>         return cmpxchg(&slab->obj_exts, 0, OBJEXTS_ALLOC_FAIL) == 0;
>  }
>
> -static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
> -                       struct slabobj_ext *vec, unsigned int objects)
> +static inline void handle_failed_objexts_alloc(struct slab *slab,
> +               unsigned long obj_exts, struct slabobj_ext *vec)
>  {
>         /*
>          * If vector previously failed to allocate then we have live
>          * objects with no tag reference. Mark all references in this
>          * vector as empty to avoid warnings later on.
>          */
> -       if (obj_exts == OBJEXTS_ALLOC_FAIL) {
> -               unsigned int i;
> +       if (obj_exts != OBJEXTS_ALLOC_FAIL)
> +               return;
> +
> +       for (unsigned int i = 0; i < slab->objects; i++) {
> +               union codetag_ref *ref = slab_obj_ext_codetag_ref(slab, vec);
>
> -               for (i = 0; i < objects; i++)
> -                       set_codetag_empty(&vec[i].ref);
> +               set_codetag_empty(ref);
> +               vec++;
>         }
>  }
>
> @@ -2112,8 +2117,8 @@ static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
>
>  static inline void mark_obj_codetag_empty(const void *obj) {}
>  static inline bool mark_failed_objexts_alloc(struct slab *slab) { return false; }
> -static inline void handle_failed_objexts_alloc(unsigned long obj_exts,
> -                       struct slabobj_ext *vec, unsigned int objects) {}
> +static inline void handle_failed_objexts_alloc(struct slab *slab,
> +               unsigned long obj_exts, struct slabobj_ext *vec) {}
>
>  #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */
>
> @@ -2181,7 +2186,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>  #endif
>  retry:
>         old_exts = READ_ONCE(slab->obj_exts);
> -       handle_failed_objexts_alloc(old_exts, vec, slab->objects);
> +       handle_failed_objexts_alloc(slab, old_exts, vec);
>
>         if (new_slab) {
>                 /*
> @@ -2361,9 +2366,15 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
>          * check should be added before alloc_tag_add().
>          */
>         if (obj_exts) {
> +               union codetag_ref *ref;
> +
>                 get_slab_obj_exts(obj_exts);
> +
>                 obj_ext = slab_obj_ext(s, slab, obj_exts, object);
> -               alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size);
> +               ref = slab_obj_ext_codetag_ref(slab, obj_ext);
> +
> +               alloc_tag_add(ref, current->alloc_tag, s->size);
> +
>                 put_slab_obj_exts(obj_exts);
>         } else {
>                 alloc_tag_set_inaccurate(current->alloc_tag);
> @@ -2395,10 +2406,13 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
>
>         get_slab_obj_exts(obj_exts);
>         for (int i = 0; i < objects; i++) {
> +               struct slabobj_ext *ext;
> +
>                 if (is_kfence_address(p[i]))
>                         continue;
>
> -               alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size);
> +               ext = slab_obj_ext(s, slab, obj_exts, p[i]);
> +               alloc_tag_sub(slab_obj_ext_codetag_ref(slab, ext), s->size);
>         }
>         put_slab_obj_exts(obj_exts);
>  }
>
> --
> 2.55.0
>

Re: [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object

[Suren Baghdasaryan]

On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> The stride field is used to convert object index to an slabobj_ext so
> both compact arrays (kmalloc() or in-slab-leftover) and spread
> in-object-padding obj_ext layouts are supported.
>
> In practice thus the stride is always sizeof(slabobj_ext) or s->size.
>
> This simplifies the calculations, but with the upcoming slabobj_ext
> handling changes, it will be easier to stop storing the stride and
> instead just have a flag whether obj_ext is in the object padding.
> obj_exts_in_object() can then rely on this flag and slab_obj_ext()
> can use that to determine the stride.
>
> No functional change intended. Performance impact TBD, hopefully
> in the noise.
>
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

Reviewed-by: Suren Baghdasaryan <surenb@google.com>

> ---
>  mm/slab.h | 43 ++++++++++++++++++++++++-------------------
>  mm/slub.c | 42 ++++++++++++++++--------------------------
>  2 files changed, 40 insertions(+), 45 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index e3f8e42070f1..3ad9777ad600 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -81,10 +81,10 @@ struct freelist_counters {
>  #ifdef CONFIG_64BIT
>                                         /*
>                                          * Some optimizations use free bits in 'counters' field
> -                                        * to save memory. In case ->stride field is not available,
> +                                        * to save memory. If these free bits are not available,
>                                          * such optimizations are disabled.
>                                          */
> -                                       unsigned int stride;
> +                                       unsigned obj_exts_in_object:1;
>  #endif
>                                 };
>                         };
> @@ -617,22 +617,20 @@ static inline void put_slab_obj_exts(unsigned long obj_exts)
>  }
>
>  #ifdef CONFIG_64BIT
> -static inline void slab_set_stride(struct slab *slab, unsigned int stride)
> +static inline bool obj_exts_in_object(struct slab *slab)
>  {
> -       slab->stride = stride;
> -}
> -static inline unsigned int slab_get_stride(struct slab *slab)
> -{
> -       return slab->stride;
> +       /*
> +        * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
> +        * check the per-slab bit. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
> +        * allocations within_slab_leftover are preferred. And those may be
> +        * possible or not depending on the particular slab's size.
> +        */
> +       return slab->obj_exts_in_object;
>  }
>  #else
> -static inline void slab_set_stride(struct slab *slab, unsigned int stride)
> +static inline bool obj_exts_in_object(struct slab *slab)
>  {
> -       VM_WARN_ON_ONCE(stride != sizeof(struct slabobj_ext));
> -}
> -static inline unsigned int slab_get_stride(struct slab *slab)
> -{
> -       return sizeof(struct slabobj_ext);
> +       return false;
>  }
>  #endif
>
> @@ -656,8 +654,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
>         VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
>
>         index = obj_to_index(s, slab, obj);
> -       obj_ext = (struct slabobj_ext *)(obj_exts +
> -                                        slab_get_stride(slab) * index);
> +
> +       if (!obj_exts_in_object(slab)) {
> +               obj_ext = ((struct slabobj_ext *)obj_exts) + index;
> +       } else {
> +               unsigned int stride = s->size;
> +               obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);
> +       }
> +
>         return kasan_reset_tag(obj_ext);
>  }
>
> @@ -693,9 +697,10 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
>         return NULL;
>  }
>
> -static inline void slab_set_stride(struct slab *slab, unsigned int stride) { }
> -static inline unsigned int slab_get_stride(struct slab *slab) { return 0; }
> -
> +static inline bool obj_exts_in_object(struct slab *slab)
> +{
> +       return false;
> +}
>
>  #endif /* CONFIG_SLAB_OBJ_EXT */
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 2bfcabc4c51a..98a14e5842a2 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -870,18 +870,6 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
>  #endif
>
>  #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
> -static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
> -{
> -       /*
> -        * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
> -        * check the stride. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
> -        * allocations within_slab_leftover are preferred. And those may be
> -        * possible or not depending on the particular slab's size.
> -        */
> -       return obj_exts_in_slab(s, slab) &&
> -              (slab_get_stride(slab) == s->size);
> -}
> -
>  static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
>  {
>         unsigned int offset = get_info_end(s);
> @@ -896,16 +884,20 @@ static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
>
>         return offset;
>  }
> -#else
> -static inline bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
> +
> +static inline void slab_set_obj_exts_in_object(struct slab *slab)
>  {
> -       return false;
> +       slab->obj_exts_in_object = 1;
>  }
> -
> +#else
>  static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
>  {
>         return 0;
>  }
> +
> +static inline void slab_set_obj_exts_in_object(struct slab *slab)
> +{
> +}
>  #endif
>
>  #ifdef CONFIG_SLUB_DEBUG
> @@ -1206,7 +1198,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p)
>
>         off += kasan_metadata_size(s, false);
>
> -       if (obj_exts_in_object(s, slab))
> +       if (obj_exts_in_object(slab))
>                 off += sizeof(struct slabobj_ext);
>
>         if (off != size_from_object(s))
> @@ -1411,7 +1403,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p)
>
>         off += kasan_metadata_size(s, false);
>
> -       if (obj_exts_in_object(s, slab))
> +       if (obj_exts_in_object(slab))
>                 off += sizeof(struct slabobj_ext);
>
>         if (size_from_object(s) == off)
> @@ -1439,7 +1431,7 @@ slab_pad_check(struct kmem_cache *s, struct slab *slab)
>         length = slab_size(slab);
>         end = start + length;
>
> -       if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(s, slab)) {
> +       if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(slab)) {
>                 remainder = length;
>                 remainder -= obj_exts_offset_in_slab(s, slab);
>                 remainder -= obj_exts_size_in_slab(slab);
> @@ -2253,9 +2245,6 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
>         void *addr;
>         unsigned long obj_exts;
>
> -       /* Initialize stride early to avoid memory ordering issues */
> -       slab_set_stride(slab, sizeof(struct slabobj_ext));
> -
>         if (!need_slab_obj_exts(s))
>                 return;
>
> @@ -2289,7 +2278,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
>                 obj_exts |= MEMCG_DATA_OBJEXTS;
>  #endif
>                 slab->obj_exts = obj_exts;
> -               slab_set_stride(slab, s->size);
> +               slab_set_obj_exts_in_object(slab);
>         }
>  }
>
> @@ -3402,9 +3391,10 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
>                 stat(s, ORDER_FALLBACK);
>         }
>
> +       /* Initializes frozen, inuse, and any extra 64bit-only flags */
> +       slab->counters = 0;
> +
>         slab->objects = oo_objects(oo);
> -       slab->inuse = 0;
> -       slab->frozen = 0;
>
>         slab->slab_cache = s;
>
> @@ -6537,7 +6527,7 @@ static inline size_t slab_ksize(struct slab *slab)
>          */
>         if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
>                 return s->inuse;
> -       else if (obj_exts_in_object(s, slab))
> +       else if (obj_exts_in_object(slab))
>                 return s->inuse;
>         /*
>          * Else we can use all the padding etc for the allocation
>
> --
> 2.55.0
>

Re: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access

[Vlastimil Babka (SUSE)]

On 7/16/26 03:27, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>> --- a/mm/slab.h
>> +++ b/mm/slab.h
>> @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
>>   */
>>  struct slabobj_ext {
>>  #ifdef CONFIG_MEMCG
>> -       struct obj_cgroup *objcg;
>> +       struct obj_cgroup *_objcg;
>>  #endif
>>  #ifdef CONFIG_MEM_ALLOC_PROFILING
>>         union codetag_ref ref;
>> @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
>>         return kasan_reset_tag(obj_ext);
>>  }
>>
>> +#ifdef CONFIG_MEMCG
>> +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
> 
> I understand why you need to return obj_cgroup** but maybe instead we
> can have a separate setter function and just return a pointer here?

Yeah I rewrote it now and it looks better. Thanks! It ended up like this due
to some previous attempt I made.

> That would also help catch cases when someone wants to set
> obj_ext._objcg that was optimized away (for that the setter would need
> an additional parameter to identify whether we allocated optimized
> objext vector).

It would need passing the slab parameter to assert
slab_obj_ext_has_codetag() in debug mode. I don't think it's worth it given
the get/set is only called from few specialized places.

> 
>> +{
>> +       return &obj_ext->_objcg;
>> +}
>> +#endif
>> +
>>  int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
>>                         gfp_t gfp, unsigned int alloc_flags);
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index 5e3f53bcd0d3..48e10198a3ce 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
>>         if (obj_exts) {
>>                 get_slab_obj_exts(obj_exts);
>>                 obj_ext = slab_obj_ext(s, slab, obj_exts, p);
>> -               if (unlikely(obj_ext->objcg)) {
>> +               if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
>>                         put_slab_obj_exts(obj_exts);
>>                         return true;
>>                 }
>>
>> --
>> 2.55.0
>>

Re: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union

[Vlastimil Babka (SUSE)]

On 7/16/26 06:11, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>> No functional change intended, the layout is still effectively static.
>>
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> 
> This is much cleaner than what I was preparing. Nicely done!
> 
> Reviewed-by: Suren Baghdasaryan <surenb@google.com>

Thanks!

>> ---
>>  mm/slab.h | 41 +++++++++++++++++++++++++++++++++--------
>>  mm/slub.c | 17 +++++++++--------
>>  2 files changed, 42 insertions(+), 16 deletions(-)
>>
>> diff --git a/mm/slab.h b/mm/slab.h
>> index 3ad9777ad600..359ab8caf61e 100644
>> --- a/mm/slab.h
>> +++ b/mm/slab.h
>> @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void)
>>   * if MEMCG_DATA_OBJEXTS is set.
>>   */
>>  struct slabobj_ext {
> 
> Perhaps we should add a comment here stating that every element of
> this union should be pointer-sized?
> 

OK, how about this?

--- a/mm/slab.h
+++ b/mm/slab.h
@@ -550,10 +550,14 @@ static inline bool need_kmalloc_no_objext(void)
 }
 
 /*
- * Extended information for slab objects stored as an array in page->memcg_data
- * if MEMCG_DATA_OBJEXTS is set.
+ * Extended information for slab objects stored as a pointer to an array in
+ * slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set.
  */
 struct slabobj_ext {
+       /*
+        * All elements of the union should be pointer-sized to avoid memory
+        * waste
+        */
        union {
 #ifdef CONFIG_MEMCG
                struct obj_cgroup *_objcg;

Re: [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag()

[Vlastimil Babka (SUSE)]

On 7/16/26 06:25, Suren Baghdasaryan wrote:
> On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> <vbabka@kernel.org> wrote:
>>
>> mem_alloc_profiling_enabled() allows evaluating (with a static key) if
>> memory profiling is currently enabled. mem_profiling_support is a
>> variable where false means it's not possible to enable it anymore,
>> because the system was booted with "never" or it was later shut down.
>> This is possible to query by mem_alloc_profiling_permanently_disabled().
>>
>> To make slabobj_ext array size handling dynamic, we need a snapshot of
>> mem_alloc_profiling_permanently_disabled() early in boot, so that's not
>> affected by a later shutdown. We also need it to be static key based for
>> performance. Neither mem_alloc_profiling_enabled() nor
>> mem_alloc_profiling_permanently_disabled() satisfy this.
>>
>> Therefore introduce slab_obj_ext_has_codetag() with an underlying static
>> key for that use case. Its state is made to reflect the result of
>> mem_alloc_profiling_permanently_disabled() during kmem_cache_init(),
>> which does happen after setup_early_mem_profiling().
>>
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> 
> Reviewed-by: Suren Baghdasaryan <surenb@google.com>

Thanks!

>> +/*
>> + * Make sure the static key used by slab_obj_ext_has_codetag() reflects the
>> + * value of !mem_alloc_profiling_permanently_disabled()
>> + *
>> + * Any later mem alloc profiling shutdown won't be reflected in the static key
>> + * because obj_exts with codetags might already exist.
>> + */
>> +static void __init slab_obj_ext_has_codetag_init(void)
>> +{
>> +       bool key_enabled = IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT);
> 
> nit: This seems a bit indirect. Reader needs to remember that
> slab_obj_ext_has_codetag_key is initialized based on
> CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Wouldn't below code be
> simpler?:
> 
> bool need_codetag = !mem_alloc_profiling_permanently_disabled();
> if (need_codetag != static_key_enabled(&slab_obj_ext_has_codetag_key)) {
>         if (need_codetag)
>                 static_branch_enable(&slab_obj_ext_has_codetag_key);
>         else
>                 static_branch_disable(&slab_obj_ext_has_codetag_key);
> }
> 
> It might be less performant but who cares, it's __init function used
> only one time.

Good idea, will do!

Re: [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary

[Vlastimil Babka (SUSE)]

On 7/16/26 06:46, Suren Baghdasaryan wrote:
> Maybe now we should rename static_obj_ext_size() to
> static_obj_ext_max_size() as it reflects the max possible size of
> slabobj_ext?

Hm with _max_size() it doesn't have to be called static_ anymore?

But maybe we can get rid of it completely.

The only caller is calculate_sizes():

#if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
        if (slab_args_unmergeable(args, s->flags) &&
                        (aligned_size - size >= static_obj_ext_size()))
                s->flags |= SLAB_OBJ_EXT_IN_OBJ;
#endif

We don't have slab pointer to pass to static_obj_ext_max_size().
But we should be able to figure it out from the cache via
cache_needs_objcg().

So we'd need cache_obj_ext_size()?

The static max size here works, but may prevent SLAB_OBJ_EXT_IN_OBJ
needlessly if padding can only fit codetag_ref and not objcg, but
we don't need objcg.
Which is probably only theoretical at this point as kmalloc_normal
caches (that have no objcg) have no padding, at least without
slab_debug. But still.

>> +       size_t sz = 0;
>> +
>> +       if (slab_needs_objcg(slab))
>> +               sz += 1;
>> +
>> +       if (slab_obj_ext_has_codetag())
>> +               sz += 1;
>> +
>> +       return sizeof(struct slabobj_ext) * sz;
>>  }
>>
>>  #ifdef CONFIG_SLAB_OBJ_EXT
>> @@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
>>  static inline union codetag_ref *
>>  slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
>>  {
>> -       if (IS_ENABLED(CONFIG_MEMCG))
>> +       if (slab_needs_objcg(slab))
>>                 obj_ext += 1;
>>
>>         return &obj_ext->_ctref;
>> diff --git a/mm/slub.c b/mm/slub.c
>> index 771d73abacb6..09c4931e5435 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
>>         if (likely(!obj_exts))
>>                 return;
>>
>> +       if (!slab_needs_objcg(slab))
>> +               return;
>> +
>>         get_slab_obj_exts(obj_exts);
>>         __memcg_slab_free_hook(s, slab, p, objects, obj_exts);
>>         put_slab_obj_exts(obj_exts);
>>
>> --
>> 2.55.0
>>

Re: [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling

[Suren Baghdasaryan]

On Thu, Jul 16, 2026 at 2:11 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/15/26 18:02, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:10 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >>
> >> struct kfence_metadata only has obj_exts with CONFIG_MEMCG.
> >
> > I don't quite understand this statement. obj_exts are allocated when
> > either CONFIG_MEMCG or CONFIG_MEM_ALLOC_PROFILING is enabled.
>
> See in mm/kfence/kfence.h
>
> struct kfence_metadata {
> ...
> #ifdef CONFIG_MEMCG
>         struct slabobj_ext obj_exts;
> #endif
> ...
>
> then in mm/kfence/core.c kfence_init_pool()
>
> #ifdef CONFIG_MEMCG
>                 struct slab *slab = page_slab(page);
>                 slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts |
>                                  MEMCG_DATA_OBJEXTS;
> #endif
>
> So the slab kfence fakes only has this pre-inited obj_exts
> with CONFIG_MEMCG.
>
> What happens if we run memalloc profiling without MEMCG and this
> fake slab doesn't have a pre-assigned obj_exts? I guess
> __alloc_tagging_slab_alloc_hook() will allocate it via
> prepare_slab_obj_exts_hook().
>
> But it's something that was done consciously and probably
> just accidentally works.

Ok, I see. Maybe you can expand this description to explain the
details more thoroughly? This whole kfence.obj_exts deal is not very
intuitive.

>
> >> If it's
> >> enabled, it does also work for allocation profiling, but there's little
> >> value recording tags for KFENCE objects.
> >
> > Unless we are leaking them, right?
>
> Well if there are leaks in a particular callsite, we should see that from all
> the allocations that don't end up in kfence (as kfence allocations are rare).
> So we are very unlikely to miss a leak due to this.

Ok, makes sense. Thanks for the explanation!

>
> >> Furthermore it would complicate
> >> the upcoming changes, so just skip them in the slab hooks.
> >
> > Ok, I can understand that. If we do this, we should document that
> > kfence objects are no longer tracked.
> >
> >>
> >> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> >> ---
> >>  mm/slub.c | 6 ++++++
> >>  1 file changed, 6 insertions(+)
> >>
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 0337e60db5ac..a4be70d080fb 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
> >>         if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
> >>                 return;
> >>
> >> +       if (is_kfence_address(object))
> >> +               return;
> >> +
> >>         slab = virt_to_slab(object);
> >>         obj_exts = prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, object);
> >>         /*
> >> @@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p
> >>         for (i = 0; i < objects; i++) {
> >>                 unsigned int off = obj_to_index(s, slab, p[i]);
> >>
> >> +               if (is_kfence_address(p[i]))
> >> +                       continue;
> >> +
> >>                 alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size);
> >>         }
> >>         put_slab_obj_exts(obj_exts);
> >>
> >> --
> >> 2.55.0
> >>
>

Re: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access

[Suren Baghdasaryan]

On Thu, Jul 16, 2026 at 6:58 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/16/26 03:27, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >> --- a/mm/slab.h
> >> +++ b/mm/slab.h
> >> @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)
> >>   */
> >>  struct slabobj_ext {
> >>  #ifdef CONFIG_MEMCG
> >> -       struct obj_cgroup *objcg;
> >> +       struct obj_cgroup *_objcg;
> >>  #endif
> >>  #ifdef CONFIG_MEM_ALLOC_PROFILING
> >>         union codetag_ref ref;
> >> @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,
> >>         return kasan_reset_tag(obj_ext);
> >>  }
> >>
> >> +#ifdef CONFIG_MEMCG
> >> +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ext)
> >
> > I understand why you need to return obj_cgroup** but maybe instead we
> > can have a separate setter function and just return a pointer here?
>
> Yeah I rewrote it now and it looks better. Thanks! It ended up like this due
> to some previous attempt I made.
>
> > That would also help catch cases when someone wants to set
> > obj_ext._objcg that was optimized away (for that the setter would need
> > an additional parameter to identify whether we allocated optimized
> > objext vector).
>
> It would need passing the slab parameter to assert
> slab_obj_ext_has_codetag() in debug mode. I don't think it's worth it given
> the get/set is only called from few specialized places.

Sounds good.

>
> >
> >> +{
> >> +       return &obj_ext->_objcg;
> >> +}
> >> +#endif
> >> +
> >>  int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
> >>                         gfp_t gfp, unsigned int alloc_flags);
> >>
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 5e3f53bcd0d3..48e10198a3ce 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)
> >>         if (obj_exts) {
> >>                 get_slab_obj_exts(obj_exts);
> >>                 obj_ext = slab_obj_ext(s, slab, obj_exts, p);
> >> -               if (unlikely(obj_ext->objcg)) {
> >> +               if (unlikely(*slab_obj_ext_objcgp(obj_ext))) {
> >>                         put_slab_obj_exts(obj_exts);
> >>                         return true;
> >>                 }
> >>
> >> --
> >> 2.55.0
> >>
>

Re: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union

[Suren Baghdasaryan]

On Thu, Jul 16, 2026 at 7:43 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/16/26 06:11, Suren Baghdasaryan wrote:
> > On Wed, Jul 15, 2026 at 3:11 AM Vlastimil Babka (SUSE)
> > <vbabka@kernel.org> wrote:
> >> No functional change intended, the layout is still effectively static.
> >>
> >> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> >
> > This is much cleaner than what I was preparing. Nicely done!
> >
> > Reviewed-by: Suren Baghdasaryan <surenb@google.com>
>
> Thanks!
>
> >> ---
> >>  mm/slab.h | 41 +++++++++++++++++++++++++++++++++--------
> >>  mm/slub.c | 17 +++++++++--------
> >>  2 files changed, 42 insertions(+), 16 deletions(-)
> >>
> >> diff --git a/mm/slab.h b/mm/slab.h
> >> index 3ad9777ad600..359ab8caf61e 100644
> >> --- a/mm/slab.h
> >> +++ b/mm/slab.h
> >> @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void)
> >>   * if MEMCG_DATA_OBJEXTS is set.
> >>   */
> >>  struct slabobj_ext {
> >
> > Perhaps we should add a comment here stating that every element of
> > this union should be pointer-sized?
> >
>
> OK, how about this?
>
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -550,10 +550,14 @@ static inline bool need_kmalloc_no_objext(void)
>  }
>
>  /*
> - * Extended information for slab objects stored as an array in page->memcg_data
> - * if MEMCG_DATA_OBJEXTS is set.
> + * Extended information for slab objects stored as a pointer to an array in
> + * slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set.
>   */
>  struct slabobj_ext {
> +       /*
> +        * All elements of the union should be pointer-sized to avoid memory
> +        * waste
> +        */

Perfect. Thank you!

>         union {
>  #ifdef CONFIG_MEMCG
>                 struct obj_cgroup *_objcg;
>

Re: [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary

[Suren Baghdasaryan]

On Thu, Jul 16, 2026 at 8:08 AM Vlastimil Babka (SUSE)
<vbabka@kernel.org> wrote:
>
> On 7/16/26 06:46, Suren Baghdasaryan wrote:
> > Maybe now we should rename static_obj_ext_size() to
> > static_obj_ext_max_size() as it reflects the max possible size of
> > slabobj_ext?
>
> Hm with _max_size() it doesn't have to be called static_ anymore?

Yeah, static_ is an implementation detail anyway.

>
> But maybe we can get rid of it completely.
>
> The only caller is calculate_sizes():

I was thinking the same but either way sounds fine to me. Keeping
obj_ext_max_size() would be a bit more self-documenting I think.

>
> #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
>         if (slab_args_unmergeable(args, s->flags) &&
>                         (aligned_size - size >= static_obj_ext_size()))
>                 s->flags |= SLAB_OBJ_EXT_IN_OBJ;
> #endif
>
> We don't have slab pointer to pass to static_obj_ext_max_size().
> But we should be able to figure it out from the cache via
> cache_needs_objcg().
>
> So we'd need cache_obj_ext_size()?
>
> The static max size here works, but may prevent SLAB_OBJ_EXT_IN_OBJ
> needlessly if padding can only fit codetag_ref and not objcg, but
> we don't need objcg.
> Which is probably only theoretical at this point as kmalloc_normal
> caches (that have no objcg) have no padding, at least without
> slab_debug. But still.

Maybe keep it simple as is for now and then we can optimize it further?

>
> >> +       size_t sz = 0;
> >> +
> >> +       if (slab_needs_objcg(slab))
> >> +               sz += 1;
> >> +
> >> +       if (slab_obj_ext_has_codetag())
> >> +               sz += 1;
> >> +
> >> +       return sizeof(struct slabobj_ext) * sz;
> >>  }
> >>
> >>  #ifdef CONFIG_SLAB_OBJ_EXT
> >> @@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *obj_ex
> >>  static inline union codetag_ref *
> >>  slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext)
> >>  {
> >> -       if (IS_ENABLED(CONFIG_MEMCG))
> >> +       if (slab_needs_objcg(slab))
> >>                 obj_ext += 1;
> >>
> >>         return &obj_ext->_ctref;
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 771d73abacb6..09c4931e5435 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,
> >>         if (likely(!obj_exts))
> >>                 return;
> >>
> >> +       if (!slab_needs_objcg(slab))
> >> +               return;
> >> +
> >>         get_slab_obj_exts(obj_exts);
> >>         __memcg_slab_free_hook(s, slab, p, objects, obj_exts);
> >>         put_slab_obj_exts(obj_exts);
> >>
> >> --
> >> 2.55.0
> >>
>