Re: [PATCH 01/13] mm/mmap: free gathered pages before MMF_OOM_SKIP

["Lorenzo Stoakes (ARM)" <ljs@kernel.org>]

On Thu, Nov 12, 2026 at 12:43:57PM +0100, Michal Hocko wrote:
> On Wed 11-11-26 18:13:15, Lorenzo Stoakes (ARM) wrote:
> > The OOM killer can quickly trigger again after having selected a
> > victim. That victim might not have freed its memory yet, but its mm will
> > be marked MMF_OOM_SKIP by this point, so the OOM killer will ignore it.
> >
> > This can cause the OOM killer to kill an innocent process when the first
> > might have freed enough memory to relieve the memory pressure.
> >
> > Fix the issue by freeing the memory before marking the mm MMF_OOM_SKIP.
> >
> > Note that the task which calls mmput() might not be the victim - if it
> > took a reference on the mm (e.g. ptrace), it may hold the last reference
> > and release it when it calls mmput().
>
> I do not follow the reasoning here. MMF_OOM_SKIP is called after the
> address space has been unmapped either by the last mmput or the
> oom_reaper (or both when they are racing). Now all your patch does is to
> call tlb_flush_mmu which I suspect is to release batched pages. But why
> is that needed only for the oom victim and why there was no final flush
> in unmap_vmas?

Sorry maybe commit message isn't clear enough. I did have to pare these down
from really massive ones.

And yeah tlb_flush_mmu() is about freeing the batches.

unmap_vmas() doesn't have a final flush, it relies on the caller to figure that
out, and just batches up.

And currently tlb_finish_mmu() comes after MMF_OOM_SKIP is set (in
exit_mmap()), so at that point the memory is unmapped but whatever the last
batch is isn't freed yet, and the batch can be large (PMDs are single
entries if I recall correctly for e.g.).

This only matters for the OOM victim, hence reinstating MMF_OOM_VICTIM, which
made the patch _way_ smaller than the one proposed :)

>
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index 7618e72766a5..4cb18b217048 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -1313,11 +1313,18 @@ void exit_mmap(struct mm_struct *mm)
> >  	/* update_hiwater_rss(mm) here? but nobody should be looking */
> >  	/* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */
> >  	unmap_vmas(&tlb, &unmap);
> > +
> > +	/*
> > +	 * Free the memory before marking MMF_OOM_SKIP so a re-triggered OOM
> > +	 * killer will only skip this mm if it's already freed all it can.
> > +	 */
> > +	if (mm_is_oom_victim(mm))
> > +		tlb_flush_mmu(&tlb);
> >  	mmap_read_unlock(mm);
> >
> >  	/*
> >  	 * Set MMF_OOM_SKIP to hide this task from the oom killer/reaper
> > -	 * because the memory has been already freed.
> > +	 * because the memory has been already unmapped.
> >  	 */
> >  	mm_flags_set(MMF_OOM_SKIP, mm);
> >  	mmap_write_lock(mm);
>
> --
> Michal Hocko
> SUSE Labs

--
Cheers, Lorenzo