Re: [RFC] Robust futex causing memcg OOM storm on exit

[Michal Hocko <mhocko@suse.com>]

On Mon 27-07-26 13:48:58, Shakeel Butt wrote:
> On Mon, Jul 27, 2026 at 08:38:37AM -0700, Shakeel Butt wrote:
> > On Mon, Jul 27, 2026 at 04:36:23PM +0200, Michal Hocko wrote:
> > > On Wed 22-07-26 17:19:07, Shakeel Butt wrote:
> 
> [...]
> 
> > > 
> > > TBH I am not entirely happy about this approach. It effectivelly reverts
> > > a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks").
> > > It just makes it lockless. Assumption that a dying task will do so
> > > quickly and with bounded resources has turned wrong on several
> > > occasions.
> > 
> > That's why I kept it as RFC :)
> > 
> > > 
> > > On the other hand I do undestand the contention issues and I can imagine
> > > that the existing solution doesn't really work well for huge thread
> > > groups that all end up lining up on the oom_lock just to learn there is
> > > nothing really killable anymore because they are the oom victim...
> > > 
> > > Would it be just safer to bail out only for oom victim threads. This
> > > would narrow down potential runaways for oom victims which should be
> > > more limited than any killed/exiting task. It would also give the oom
> > > killer/reaper chance to work. WDYT?
> > 
> > I will give the following patch a try with the reproducer I have. I am still
> > improving the reproducer as I am still not able to recreate multi hour slowdown
> > yet. I will report back once I have some results. 
> > 
> 
> I have some results with the reproducer I (AI) wrote.
> 
> Setup: 20k threads, each with its robust_list head on its own page in a
> large anon region pushed into a zswap pool (writeback off, so it's
> unreclaimable and pins the memcg at the limit). On the OOM-group-kill
> every thread faults that page in futex_cleanup() -> do_swap_page() ->
> swapin charge -> memcg OOM. A separate thread holds mmap_lock for write
> in a loop so the oom_reaper can't reap the pool -- matching the host,
> where the reaper was blocked and the memcg never fell below the limit.
> 
> I tested on next-20260726 as baseline and then tested your suggestion over it.
> For baseline ~48k no-victim OOM invocations serialized on oom_lock; ~40s to
> drain the 20k threads in a VM (hours on the host, where each invocation
> was a dump_header over netconsole). With the patch the dying tasks stop
> entering out_of_memory and the storm is gone.
> 
> With the following suggestion, the threads of dying tasks stop entering
> out_of_memory but the exit time doubled as each thread has to decompress the
> page from zswap.
> 
> 	if (tsk_is_oom_victim(current))
> 		goto force;
> 
> I also tested by replacing goto force with goto nomem and the exit time halved
> i.e. ~20 seconds. Looking deeper it seems like each thread is still going
> through the reclaim loops.
> 
> Just for testing purpose I moved this check along with goto nomem just before
> the reclaim fuction i.e. try_to_free_mem_cgroup_pages() and the exit time falls
> below 1 second. I think what we need is to detect early that the cgroup is
> unreclaimable and the threads of exiting/reaped process should just return with
> enomem.

Do you mean, something like this?
--- 
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6dc4888a90f3..66f1eebf8c22 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2648,6 +2648,18 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (!gfpflags_allow_blocking(gfp_mask))
 		goto nomem;
 
+	/*
+	 * OOM victim still needs to charge memory to exit. OOM reaper should
+	 * help but it might fail on mmap_lock contention. If the victim is a
+	 * large thread group then all exiting threads might compete on oom_lock
+	 * just to learn that there is nothing really killable anymore. Bail
+	 * out early and fail the charge to expedite their exit. They are
+	 * considered fully reclaimed by the oom reaper and they shouldn't contribute
+	 * further charges.
+	 */
+	if (tsk_is_oom_victim(current) && mm_flags_test(current->signal->oom_mm))
+		goto nomem;
+
 	__memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);
 	raised_max_event = true;
 

-- 
Michal Hocko
SUSE Labs