Re: [PATCH] memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done

Re: [PATCH] memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done

[Johannes Weiner]

On Tue, Jul 28, 2026 at 07:46:12PM -0700, Shakeel Butt wrote:
> At Meta, we are seeing instances where an OOM killed job is stuck in the
> exit path for several hours. In one particular case, the job was stuck
> for more than 8 hours and I had to manually remove the memory.max limits
> to allow the process to exit.
> 
> The job was a single process job and had ~55 GiB memory.max and zswap
> enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed
> to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap).
> Nothing was left on the LRUs to reclaim.
> 
> On further inspection, I observed ~20k threads of that process stuck
> with the following stack:
> 
> [<0>] mem_cgroup_out_of_memory+0x4e/0xa0
> [<0>] charge_memcg+0x8bf/0x990
> [<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80
> [<0>] __read_swap_cache_async+0x10c/0x260
> [<0>] swapin_readahead+0x116/0x3f0
> [<0>] do_swap_page+0x13c/0x1ce0
> [<0>] handle_mm_fault+0x61d/0x11f0
> [<0>] do_user_addr_fault+0x3e7/0x6d0
> [<0>] exc_page_fault+0x8f/0x110
> [<0>] asm_exc_page_fault+0x22/0x30
> [<0>] __get_user_8+0x14/0x20
> [<0>] futex_cleanup+0x27/0x1c0
> [<0>] futex_exit_release+0x47/0x60
> [<0>] do_exit+0x107/0x940
> [<0>] do_group_exit+0x81/0xa0
> [<0>] get_signal+0x2b1/0x6e0
> [<0>] arch_do_signal_or_restart+0x1a/0x1c0
> [<0>] exit_to_user_mode_loop+0xa8/0x1c0
> [<0>] do_syscall_64+0x152/0x250
> [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
> 
> In addition the dmesg was filled with "Out of memory and no killable
> processes..." messages.
> 
> I have no idea why oom reaper was not able to reap/unmap the process. My
> guess is that since oom reaper tries to acquire mmap_lock in read mode
> limited number of times and then gives up, there might be a thread of
> that process which had mmap_lock in write mode at that time.
> 
> My initial suspicion was the futex_cleanup and kernel page fault causing
> infinite fault and charge retries but that was put to rest in previous
> discussions happened on similar problem [1].
> 
> My current theory is that it is just a simple slow serialization behind
> the oom_lock. Unlike page allocator, memcg charge code takes the
> oom_lock without the "try". Though memcg oom code uses
> mutex_lock_killable(), note that in the call stack get_signal() consumes
> SIGKILL (or sigdelset(SIGKILL)) before calling do_group_exit(). So this
> mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of
> thousands of threads are waiting on oom_lock and one by one they get
> -EFAULT from get_user() in the futex cleanup code and bails out.
> 
> Discussion from [1] lead to the commit a75ffa26122b ("memcg, oom: do not
> bypass oom killer for dying tasks") which routes dying tasks into the OOM
> path precisely so the oom_reaper can reap their mm and free the memory
> asynchronously. But the reaper is best-effort and one-shot: if it cannot
> take mmap_lock for read (e.g. a sibling thread holds it for write) it
> sets MMF_OOM_SKIP and never retries, leaving only the glacial
> oom_lock-serialized synchronous drain.
> 
> Once MMF_OOM_SKIP is set there is no more asynchronous reclaim coming for
> the mm, so a dying task charging against it has nothing left to wait for:
> it frees its memory only once it finishes exiting. Running reclaim and the
> (no-victim) OOM killer for it is then pointless, and doing it for 10s of
> thousands of exiting threads is what serializes them behind oom_lock. So
> before reclaim, if current is an OOM victim whose reaper is done, fail the
> charge.
> 
> Reproduced with 20k threads, each parking a robust futex head on
> its own zswapped page, OOM-group-killed while a sibling holds mmap_lock
> for write so the reaper gives up and sets MMF_OOM_SKIP. Tested on
> next-20260728 and baseline show ~90 seconds exit time while with the
> patch the exit time reduced to ~3 seconds.
> 
> Link: https://lore.kernel.org/7a4e5591f45df455e6a485fc5400989569d3d22d.camel@surriel.com/ [1]
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

Re: [PATCH] memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done

[Andrew Morton]

On Tue, 28 Jul 2026 19:46:12 -0700 Shakeel Butt <shakeel.butt@linux.dev> wrote:

> At Meta, we are seeing instances where an OOM killed job is stuck in the
> exit path for several hours. In one particular case, the job was stuck
> for more than 8 hours and I had to manually remove the memory.max limits
> to allow the process to exit.

Not seeing the problem here - that sounds like normal kernel behavior ;)

> Reproduced with 20k threads, each parking a robust futex head on
> its own zswapped page, OOM-group-killed while a sibling holds mmap_lock
> for write so the reaper gives up and sets MMF_OOM_SKIP. Tested on
> next-20260728 and baseline show ~90 seconds exit time while with the
> patch the exit time reduced to ~3 seconds.

Very positive.

But it doesn't sound like you're fully confident that the problem is
fully solved?

> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2653,6 +2653,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
>  	if (!gfpflags_allow_blocking(gfp_mask))
>  		goto nomem;
>  
> +	/*
> +	 * OOM victim still needs to charge memory to exit. OOM reaper should
> +	 * help but it might fail on mmap_lock contention. If the victim is a
> +	 * large thread group then all exiting threads might compete on oom_lock
> +	 * just to learn that there is nothing really killable anymore. Bail
> +	 * out early and fail the charge to expedite their exit. They are
> +	 * considered fully reclaimed by the oom reaper and they shouldn't
> +	 * contribute further charges.
> +	 */
> +	if (tsk_is_oom_victim(current) &&
> +	    mm_flags_test(MMF_OOM_SKIP, current->signal->oom_mm))
> +		goto nomem;
> +
>  	__memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);
>  	raised_max_event = true;

Should we backport this?

Re: [PATCH] memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done

[Shakeel Butt]

On Wed, Jul 29, 2026 at 06:13:49PM -0700, Andrew Morton wrote:
> On Tue, 28 Jul 2026 19:46:12 -0700 Shakeel Butt <shakeel.butt@linux.dev> wrote:
> 
> > At Meta, we are seeing instances where an OOM killed job is stuck in the
> > exit path for several hours. In one particular case, the job was stuck
> > for more than 8 hours and I had to manually remove the memory.max limits
> > to allow the process to exit.
> 
> Not seeing the problem here - that sounds like normal kernel behavior ;)
> 
> > Reproduced with 20k threads, each parking a robust futex head on
> > its own zswapped page, OOM-group-killed while a sibling holds mmap_lock
> > for write so the reaper gives up and sets MMF_OOM_SKIP. Tested on
> > next-20260728 and baseline show ~90 seconds exit time while with the
> > patch the exit time reduced to ~3 seconds.
> 
> Very positive.
> 
> But it doesn't sound like you're fully confident that the problem is
> fully solved?

Yes as I am not able to reproduce multi hour stall, so I might be missing
something else. The patch at least fixes the issue which I am able to reproduce.

> 
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -2653,6 +2653,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
> >  	if (!gfpflags_allow_blocking(gfp_mask))
> >  		goto nomem;
> >  
> > +	/*
> > +	 * OOM victim still needs to charge memory to exit. OOM reaper should
> > +	 * help but it might fail on mmap_lock contention. If the victim is a
> > +	 * large thread group then all exiting threads might compete on oom_lock
> > +	 * just to learn that there is nothing really killable anymore. Bail
> > +	 * out early and fail the charge to expedite their exit. They are
> > +	 * considered fully reclaimed by the oom reaper and they shouldn't
> > +	 * contribute further charges.
> > +	 */
> > +	if (tsk_is_oom_victim(current) &&
> > +	    mm_flags_test(MMF_OOM_SKIP, current->signal->oom_mm))
> > +		goto nomem;
> > +
> >  	__memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);
> >  	raised_max_event = true;
> 
> Should we backport this?

I don't have any strong opinion. I wanted to deploy this patch in Meta fleet to
get some exposure. Maybe backporting will give more exposure to it.