[PATCH] mm/percpu: account full object size for percpu memory allocations

[PATCH] mm/percpu: account full object size for percpu memory allocations

[Harry Yoo (Meta)]

When memory allocation profiling is enabled, the pages backing percpu
memory chunks are attributed to func:pcpu_alloc_pages. However, the
accounting of individual percpu memory allocations via alloc_percpu()
and friends record only the size of a single object rather than its
full size across all CPUs.

As a result, it is often not straightforward to identify which kernel
code is actually responsible for percpu memory, because the size is
under-accounted by a factor of num_possible_cpus().

Account full object size with pcpu_obj_full_size(size) instead, which
multiplies the object size by num_possible_cpus() and adds the objcg
pointer overhead. This is consistent with what memcg already does for
charging percpu allocations.

Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
measuring mm_init()'s rss_stat counters.

Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:

                  bytes   calls   per allocation
    unpatched      6320     395         16
    patched      418176     396       1056

'percpu' field in memory.stat: 422400 (1056 bytes per process)

Memory allocation profiling now accounts the same amount of memory as
what memcg charges.

Cc: stable@vger.kernel.org
Fixes: 60fa4a9e2323 ("mm: percpu: add codetag reference into pcpuobj_ext")
Assisted-by: LLM # Verification
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
---
 mm/percpu.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index 3eff382e565a..d7883507dc15 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1696,14 +1696,15 @@ static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off,
 {
 	if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) {
 		alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag,
-			      current->alloc_tag, size);
+			      current->alloc_tag, pcpu_obj_full_size(size));
 	}
 }
 
 static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size)
 {
 	if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts))
-		alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size);
+		alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag,
+			      pcpu_obj_full_size(size));
 }
 #else
 static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off,

---
base-commit: 0e199ae6e803e76ebf128e009d336e5917221e63
change-id: 20261013-fix-alloc-percpu-accounting-48bf24919caf

Best regards,
--  
Cheers,
Harry / Hyeonggon

Re: [PATCH] mm/percpu: account full object size for percpu memory allocations

[Andrew Morton]

On Tue, 13 Oct 2026 20:34:39 +0100 "Harry Yoo (Meta)" <harry@kernel.org> wrote:

> When memory allocation profiling is enabled, the pages backing percpu
> memory chunks are attributed to func:pcpu_alloc_pages. However, the
> accounting of individual percpu memory allocations via alloc_percpu()
> and friends record only the size of a single object rather than its
> full size across all CPUs.
> 
> As a result, it is often not straightforward to identify which kernel
> code is actually responsible for percpu memory, because the size is
> under-accounted by a factor of num_possible_cpus().
> 
> Account full object size with pcpu_obj_full_size(size) instead, which
> multiplies the object size by num_possible_cpus() and adds the objcg
> pointer overhead. This is consistent with what memcg already does for
> charging percpu allocations.
> 
> Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
> measuring mm_init()'s rss_stat counters.
> 
> Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:

Please make it use mm_init() rather than func:mm_init.

> 
>                   bytes   calls   per allocation
>     unpatched      6320     395         16
>     patched      418176     396       1056
> 
> 'percpu' field in memory.stat: 422400 (1056 bytes per process)
> 
> Memory allocation profiling now accounts the same amount of memory as
> what memcg charges.

Fair enough.

> Cc: stable@vger.kernel.org

Why cc:stable?

Re: [PATCH] mm/percpu: account full object size for percpu memory allocations

[Harry Yoo]

On Tue, Oct 13, 2026 at 02:54:24PM -0700, Andrew Morton wrote:
> On Tue, 13 Oct 2026 20:34:39 +0100 "Harry Yoo (Meta)" <harry@kernel.org> wrote:
> 
> > When memory allocation profiling is enabled, the pages backing percpu
> > memory chunks are attributed to func:pcpu_alloc_pages. However, the
> > accounting of individual percpu memory allocations via alloc_percpu()
> > and friends record only the size of a single object rather than its
> > full size across all CPUs.
> > 
> > As a result, it is often not straightforward to identify which kernel
> > code is actually responsible for percpu memory, because the size is
> > under-accounted by a factor of num_possible_cpus().
> > 
> > Account full object size with pcpu_obj_full_size(size) instead, which
> > multiplies the object size by num_possible_cpus() and adds the objcg
> > pointer overhead. This is consistent with what memcg already does for
> > charging percpu allocations.
> > 
> > Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
> > measuring mm_init()'s rss_stat counters.
> > 
> > Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:
> 
> Please make it use mm_init() rather than func:mm_init.

This is the output format of /proc/allocinfo:
https://docs.kernel.org/filesystems/proc.html#allocinfo
> Example output.
> 
>  > tail -n +3 /proc/allocinfo | sort -rn
> 127664128    31168 mm/page_ext.c:270 func:alloc_page_ext
>  56373248     4737 mm/slub.c:2259 func:alloc_slab_page
>  14880768     3633 mm/readahead.c:247 func:page_cache_ra_unbounded

> > 
> >                   bytes   calls   per allocation
> >     unpatched      6320     395         16
> >     patched      418176     396       1056
> > 
> > 'percpu' field in memory.stat: 422400 (1056 bytes per process)
> > 
> > Memory allocation profiling now accounts the same amount of memory as
> > what memcg charges.
> 
> Fair enough.
> 
> > Cc: stable@vger.kernel.org
> 
> Why cc:stable?

For anyone who uses memory allocation profiling to understand which
code locations allocated how much memory, it gives a false impression
that certain perpcu memory allocations are not as big as they actually
are.

On some workloads percpu memory (func:pcpu_alloc_pages) shows up as one
of the biggest memory allocations, but the actual users of percpu
memory allocator don't show up as they are under-reported.

-- 
Cheers,
Harry / Hyeonggon

Re: [PATCH] mm/percpu: account full object size for percpu memory allocations

[Andrew Morton]

On Tue, 13 Oct 2026 23:23:17 +0100 Harry Yoo <harry@kernel.org> wrote:

> On Tue, Oct 13, 2026 at 02:54:24PM -0700, Andrew Morton wrote:
> > On Tue, 13 Oct 2026 20:34:39 +0100 "Harry Yoo (Meta)" <harry@kernel.org> wrote:
> > 
> > > When memory allocation profiling is enabled, the pages backing percpu
> > > memory chunks are attributed to func:pcpu_alloc_pages. However, the
> > > accounting of individual percpu memory allocations via alloc_percpu()
> > > and friends record only the size of a single object rather than its
> > > full size across all CPUs.
> > > 
> > > As a result, it is often not straightforward to identify which kernel
> > > code is actually responsible for percpu memory, because the size is
> > > under-accounted by a factor of num_possible_cpus().
> > > 
> > > Account full object size with pcpu_obj_full_size(size) instead, which
> > > multiplies the object size by num_possible_cpus() and adds the objcg
> > > pointer overhead. This is consistent with what memcg already does for
> > > charging percpu allocations.
> > > 
> > > Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
> > > measuring mm_init()'s rss_stat counters.
> > > 
> > > Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:
> > 
> > Please make it use mm_init() rather than func:mm_init.
> 
> This is the output format of /proc/allocinfo:

oh, OK.

> https://docs.kernel.org/filesystems/proc.html#allocinfo
> > Example output.
> > 
> >  > tail -n +3 /proc/allocinfo | sort -rn
> > 127664128    31168 mm/page_ext.c:270 func:alloc_page_ext
> >  56373248     4737 mm/slub.c:2259 func:alloc_slab_page
> >  14880768     3633 mm/readahead.c:247 func:page_cache_ra_unbounded

Well that's odd.  What can a caller be other than a function?

> > 
> > Why cc:stable?
> 
> For anyone who uses memory allocation profiling to understand which
> code locations allocated how much memory, it gives a false impression
> that certain perpcu memory allocations are not as big as they actually
> are.
> 
> On some workloads percpu memory (func:pcpu_alloc_pages) shows up as one
> of the biggest memory allocations, but the actual users of percpu
> memory allocator don't show up as they are under-reported.

OK.  I guess it's a small deal, but a larger deal for allocinfo users!

Re: [PATCH] mm/percpu: account full object size for percpu memory allocations

[Joshua Hahn]

On Tue, 13 Oct 2026 20:34:39 +0100 "Harry Yoo (Meta)" <harry@kernel.org> wrote:

> When memory allocation profiling is enabled, the pages backing percpu
> memory chunks are attributed to func:pcpu_alloc_pages. However, the
> accounting of individual percpu memory allocations via alloc_percpu()
> and friends record only the size of a single object rather than its
> full size across all CPUs.
> 
> As a result, it is often not straightforward to identify which kernel
> code is actually responsible for percpu memory, because the size is
> under-accounted by a factor of num_possible_cpus().
> 
> Account full object size with pcpu_obj_full_size(size) instead, which
> multiplies the object size by num_possible_cpus() and adds the objcg
> pointer overhead. This is consistent with what memcg already does for
> charging percpu allocations.

Hi Harry,

This looks quite good to me. Thanks for finding this!
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

Some comments below that shouldn't be blocking:

> Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
> measuring mm_init()'s rss_stat counters.
> 
> Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:
> 
>                   bytes   calls   per allocation
>     unpatched      6320     395         16
>     patched      418176     396       1056
> 
> 'percpu' field in memory.stat: 422400 (1056 bytes per process)
> 
> Memory allocation profiling now accounts the same amount of memory as
> what memcg charges.
> 
> Cc: stable@vger.kernel.org
> Fixes: 60fa4a9e2323 ("mm: percpu: add codetag reference into pcpuobj_ext")
> Assisted-by: LLM # Verification
> Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
> ---
>  mm/percpu.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/percpu.c b/mm/percpu.c
> index 3eff382e565a..d7883507dc15 100644
> --- a/mm/percpu.c
> +++ b/mm/percpu.c
> @@ -1696,14 +1696,15 @@ static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off,
>  {
>  	if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) {
>  		alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag,
> -			      current->alloc_tag, size);
> +			      current->alloc_tag, pcpu_obj_full_size(size));
>  	}
>  }

So @size by itself isn't used, so I was looking to see if we could just
pass pcpu_obj_full_size(size) to the function instead. Looking at the
call site, pcpu_memcg_post_alloc_hook() also has the same oddity where
pcpu_obj_full_size(size) is used for all calcualtions.

So I'm wondering if we can send a follow up to just unify all of these
and pass pcpu_obj_full_size(size) where the caller never uses @size
by itself, I think it would also help in demonstrating the intent to
account the full size. Same idea for the free hook below as well,
paired with pcpu_memcg_free_hook().

Happy to follow up on that myself too : -)

Anyways, the change looks good. Just wanted to bring this up as a
follow-up cleanup patch.

Have a great day Harry!

>  static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size)
>  {
>  	if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts))
> -		alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size);
> +		alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag,
> +			      pcpu_obj_full_size(size));
>  }
>  #else
>  static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off,
> 
> ---
> base-commit: 0e199ae6e803e76ebf128e009d336e5917221e63
> change-id: 20261013-fix-alloc-percpu-accounting-48bf24919caf
> 
> Best regards,
> --  
> Cheers,
> Harry / Hyeonggon

Re: [PATCH] mm/percpu: account full object size for percpu memory allocations

[Harry Yoo]

On Wed, Oct 14, 2026 at 09:43:48AM -0700, Joshua Hahn wrote:
> On Tue, 13 Oct 2026 20:34:39 +0100 "Harry Yoo (Meta)" <harry@kernel.org> wrote:
> Hi Harry,

Hi Joshua,

> This looks quite good to me. Thanks for finding this!
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

Thanks for reviewing!

> > diff --git a/mm/percpu.c b/mm/percpu.c
> > index 3eff382e565a..d7883507dc15 100644
> > --- a/mm/percpu.c
> > +++ b/mm/percpu.c
> > @@ -1696,14 +1696,15 @@ static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off,
> >  {
> >  	if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) {
> >  		alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag,
> > -			      current->alloc_tag, size);
> > +			      current->alloc_tag, pcpu_obj_full_size(size));
> >  	}
> >  }
> 
> So @size by itself isn't used, so I was looking to see if we could just
> pass pcpu_obj_full_size(size) to the function instead. Looking at the
> call site, pcpu_memcg_post_alloc_hook() also has the same oddity where
> pcpu_obj_full_size(size) is used for all calcualtions.

Agreed.

At first look I actually tried to clean up things, then realized
I need to backport this and so cleanup should be a follow-up :) 

> So I'm wondering if we can send a follow up to just unify all of these
> and pass pcpu_obj_full_size(size) where the caller never uses @size
> by itself, I think it would also help in demonstrating the intent to
> account the full size. Same idea for the free hook below as well,
> paired with pcpu_memcg_free_hook().

Makes perfect sense to me.

Perhaps mm/percpu.c should still distinguish 'full_size' (uh, naming
sucks) and 'size'.

> Happy to follow up on that myself too : -)

Sure, thanks!
Happy to review that from my side.

> Anyways, the change looks good. Just wanted to bring this up as a
> follow-up cleanup patch.

Thanks!

> Have a great day Harry!

Likewise, have a nice day, Joshua.

-- 
Cheers,
Harry / Hyeonggon