Re: [PATCH 13/13] percpu: queue the balance work on an unbound workqueue

["Lorenzo Stoakes (ARM)" <ljs@kernel.org>]

On Thu, Nov 12, 2026 at 01:41:15PM -0500, Gregory Price (Meta) wrote:
> On 2026-11-11 18:13 +0000, Lorenzo Stoakes (ARM) wrote:
> > From: Chris Mason <mason@kernel.org>
> >
> > A burst of small atomic per-CPU allocations can fail with -ENOMEM while the
> > machine has free memory and idle CPUs, as atomic per-CPU allocations are
> > only served from pages that are already populated, and those can be used up
> > before a worker gets to populate more.
> >
> > Consider:
> >
> >     caller on CPU 1                     kworker bound to CPU 1
> >     ===============                     ======================
> >     allocates 8 bytes, which leaves
> >     one empty page and 4088 bytes
> >       schedule_work()                   woken, waits for CPU 1
> >     allocates 8 bytes, 1023 times
> >     next allocation: "no space left"
> >     gives up CPU 1                      populates pages
> >
> > The allocator keeps a few empty populated pages in reserve, and the work
> > item pcpu_balance_work refills them.
> >
> > pcpu_alloc_noprof() queues the work item using schedule_work(), so only a
> > worker bound to the CPU of the caller can run the work item.
> >
> > The issue is that the caller keeps that CPU - a task keeps it for the rest
> > of its time slice, and a softirq keeps it until the softirq ends.
> >
> > In that time the caller can use up the reserve, resulting in an atomic
> > per-CPU allocation failing.
> >
> > Fix the issue by queueing the work item on an unbound workqueue so that
> > another CPU can populate pages while the caller keeps allocating.
> >
> > Fixes: 1a4d76076cda ("percpu: implement asynchronous chunk population")
> > Signed-off-by: Chris Mason <mason@kernel.org>
> > [ ljs: tweaked commit message, comment ]
> > Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
>
> Reproduced with a cpu hammering atomic percpu allocations with preempt
> disabled. Results are pretty cool.
>
> In about 1 second of hammering allocations with the full 13 patch series:
>
>                         Base       Patch
> Successful Allocs      1,188      131,072
> Failed allocs          4,097         0
> balance worker runs      0          169
> successful retries       0         4,096
>
> Micro-bench obviously, but pretty stark difference.
>
> Tested-by: Gregory Price (Meta) <gourry@gourry.net>
> Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>

Thanks, that's great!

I will include that in commit msg on respin.

>
> --
> ~Gregory
>

--
Cheers, Lorenzo