Re: [RFC] Robust futex causing memcg OOM storm on exit

[Shakeel Butt <shakeel.butt@linux.dev>]

On Mon, Jul 27, 2026 at 08:38:37AM -0700, Shakeel Butt wrote:
> On Mon, Jul 27, 2026 at 04:36:23PM +0200, Michal Hocko wrote:
> > On Wed 22-07-26 17:19:07, Shakeel Butt wrote:

[...]

> > 
> > TBH I am not entirely happy about this approach. It effectivelly reverts
> > a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks").
> > It just makes it lockless. Assumption that a dying task will do so
> > quickly and with bounded resources has turned wrong on several
> > occasions.
> 
> That's why I kept it as RFC :)
> 
> > 
> > On the other hand I do undestand the contention issues and I can imagine
> > that the existing solution doesn't really work well for huge thread
> > groups that all end up lining up on the oom_lock just to learn there is
> > nothing really killable anymore because they are the oom victim...
> > 
> > Would it be just safer to bail out only for oom victim threads. This
> > would narrow down potential runaways for oom victims which should be
> > more limited than any killed/exiting task. It would also give the oom
> > killer/reaper chance to work. WDYT?
> 
> I will give the following patch a try with the reproducer I have. I am still
> improving the reproducer as I am still not able to recreate multi hour slowdown
> yet. I will report back once I have some results. 
> 

I have some results with the reproducer I (AI) wrote.

Setup: 20k threads, each with its robust_list head on its own page in a
large anon region pushed into a zswap pool (writeback off, so it's
unreclaimable and pins the memcg at the limit). On the OOM-group-kill
every thread faults that page in futex_cleanup() -> do_swap_page() ->
swapin charge -> memcg OOM. A separate thread holds mmap_lock for write
in a loop so the oom_reaper can't reap the pool -- matching the host,
where the reaper was blocked and the memcg never fell below the limit.

I tested on next-20260726 as baseline and then tested your suggestion over it.
For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to
drain the 20k threads in a VM (hours on the host, where each invocation
was a dump_header over netconsole). With the patch the dying tasks stop
entering out_of_memory and the storm is gone.

With the following suggestion, the threads of dying tasks stop entering
out_of_memory but the exit time doubled as each thread has to decompress the
page from zswap.

	if (tsk_is_oom_victim(current))
		goto force;

I also tested by replacing goto force with goto nomem and the exit time halved
i.e. ~20 seconds. Looking deeper it seems like each thread is still going
through the reclaim loops.

Just for testing purpose I moved this check along with goto nomem just before
the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls
below 1 second. I think what we need is to detect early that the cgroup is
unreclaimable and the threads of exiting/reaped process should just return with
enomem.