Re: [RFC] Robust futex causing memcg OOM storm on exit

[Michal Hocko <mhocko@suse.com>]

On Tue 28-07-26 09:55:21, Shakeel Butt wrote:
> On Tue, Jul 28, 2026 at 09:27:39AM +0200, Michal Hocko wrote:
> > On Mon 27-07-26 13:48:58, Shakeel Butt wrote:
> > > On Mon, Jul 27, 2026 at 08:38:37AM -0700, Shakeel Butt wrote:
> > > > On Mon, Jul 27, 2026 at 04:36:23PM +0200, Michal Hocko wrote:
> > > > > On Wed 22-07-26 17:19:07, Shakeel Butt wrote:
> > > 
> > > [...]
> > > 
> > > > > 
> > > > > TBH I am not entirely happy about this approach. It effectivelly reverts
> > > > > a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks").
> > > > > It just makes it lockless. Assumption that a dying task will do so
> > > > > quickly and with bounded resources has turned wrong on several
> > > > > occasions.
> > > > 
> > > > That's why I kept it as RFC :)
> > > > 
> > > > > 
> > > > > On the other hand I do undestand the contention issues and I can imagine
> > > > > that the existing solution doesn't really work well for huge thread
> > > > > groups that all end up lining up on the oom_lock just to learn there is
> > > > > nothing really killable anymore because they are the oom victim...
> > > > > 
> > > > > Would it be just safer to bail out only for oom victim threads. This
> > > > > would narrow down potential runaways for oom victims which should be
> > > > > more limited than any killed/exiting task. It would also give the oom
> > > > > killer/reaper chance to work. WDYT?
> > > > 
> > > > I will give the following patch a try with the reproducer I have. I am still
> > > > improving the reproducer as I am still not able to recreate multi hour slowdown
> > > > yet. I will report back once I have some results. 
> > > > 
> > > 
> > > I have some results with the reproducer I (AI) wrote.
> > > 
> > > Setup: 20k threads, each with its robust_list head on its own page in a
> > > large anon region pushed into a zswap pool (writeback off, so it's
> > > unreclaimable and pins the memcg at the limit). On the OOM-group-kill
> > > every thread faults that page in futex_cleanup() -> do_swap_page() ->
> > > swapin charge -> memcg OOM. A separate thread holds mmap_lock for write
> > > in a loop so the oom_reaper can't reap the pool -- matching the host,
> > > where the reaper was blocked and the memcg never fell below the limit.
> > > 
> > > I tested on next-20260726 as baseline and then tested your suggestion over it.
> > > For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to
> > > drain the 20k threads in a VM (hours on the host, where each invocation
> > > was a dump_header over netconsole). With the patch the dying tasks stop
> > > entering out_of_memory and the storm is gone.
> > > 
> > > With the following suggestion, the threads of dying tasks stop entering
> > > out_of_memory but the exit time doubled as each thread has to decompress the
> > > page from zswap.
> > > 
> > > 	if (tsk_is_oom_victim(current))
> > > 		goto force;
> > > 
> > > I also tested by replacing goto force with goto nomem and the exit time halved
> > > i.e. ~20 seconds. Looking deeper it seems like each thread is still going
> > > through the reclaim loops.
> > > 
> > > Just for testing purpose I moved this check along with goto nomem just before
> > > the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls
> > > below 1 second. I think what we need is to detect early that the cgroup is
> > > unreclaimable and the threads of exiting/reaped process should just return with
> > > enomem.
> > 
> > Do you mean, something like this?
> > --- 
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 6dc4888a90f3..66f1eebf8c22 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -2648,6 +2648,18 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
> >  	if (!gfpflags_allow_blocking(gfp_mask))
> >  		goto nomem;
> >  
> > +	/*
> > +	 * OOM victim still needs to charge memory to exit. OOM reaper should
> > +	 * help but it might fail on mmap_lock contention. If the victim is a
> > +	 * large thread group then all exiting threads might compete on oom_lock
> > +	 * just to learn that there is nothing really killable anymore. Bail
> > +	 * out early and fail the charge to expedite their exit. They are
> > +	 * considered fully reclaimed by the oom reaper and they shouldn't contribute
> > +	 * further charges.
> > +	 */
> > +	if (tsk_is_oom_victim(current) && mm_flags_test(current->signal->oom_mm))
> 
> I simply had tsk_is_oom_victim(current) check here. 

I was hoping for a more specific treatment. 

> I assume you want
> MMF_OOM_REAP_QUEUED in the mm_flags_test() above. Do you want me to test with
> this additional mm_flags_test() as well?

Sorry I meant MMF_OOM_SKIP there.

-- 
Michal Hocko
SUSE Labs