Re: Tools for explaining memory mappings/usage/pressure

[David Rientjes <rientjes@google.com>]

On Mon, 22 Jul 2024, Vlastimil Babka (SUSE) wrote:

> On 7/6/24 10:55 PM, David Rientjes wrote:
> > Hi all,
> > 
> > I'm trying to crowdsource information on open source tools that can be 
> > used directly by customers to explain memory mappings, usage, pressure, 
> > etc.
> > 
> > We encounter both internal and external users that are looking for this 
> > insight and it often requires significant engineering time to collect data 
> > to make any conclusions.
> > 
> > A recent example is an external customer that recently upgraded their 
> > userspace and started to run into memcg constrained memory pressure that 
> > wasn't previously observed.  After handing off a hacky script to run in 
> > the background, it was immediately obvious that the source of the direct 
> > reclaim was all of the MADV_FREE memory that was sitting around.  
> > Converting to MADV_DONTNEED solved their issue.
> 
> BTW, was this reported/fixed upstream? Sounds like a bug to me that would
> better be fixed than suggesting the MADV_DONTNEED workaround to everyone
> from now on.
> 

From the kernel perspective, it was working as intended: nearly half of 
the customer's memory (32GB VM instance) was lazily freeable under memory 
pressure and they were not expecting any reclaim stalls for future page 
faults.  A recent upgrade of their userspace had switched heap freeing in 
a library from MADV_DONTNEED -> MADV_FREE implicitly without the user's 
direct knowledge :/