Re: What kind of memory is DAMON RECLAIM able to free?

[SeongJae Park <sj@kernel.org>]

Hi Grzegorz,

On Fri, 28 Apr 2023 16:15:12 +0200 Grzegorz Uriasz <gorbak25@gmail.com> wrote:

> Hi!
> 
> I'm running some experiments using DAMON RECLAIM on the 6.2 kernel. I've 
> set up an VM with free page reporting enabled with 16 vcores and 16GB of 
> ram with very aggressive memory reclamation settings, my kernel boot 
> line includes:
> - transparent_hugepage=never
> - page_reporting.page_reporting_order=0
> - damon_reclaim.enabled=Y
> - damon_reclaim.min_age=10000000
> - damon_reclaim.wmarks_low=0
> - damon_reclaim.wmarks_mid=999
> - damon_reclaim.wmarks_high=1000
> - damon_reclaim.quota_sz=1073741824
> - damon_reclaim.quota_reset_interval_ms=1000
> 
> The memory usage of the VM starts at 800 MB, after running some 
> workloads and ballooning the VM to 16 GB DAMON RECLAIM was able to 
> quickly bring the memory usage back down to 3GB, after which it just 
> stopped doing anything. What concerns me is that 20%(3.2GB for that VM) 
> is the default low watermark in the DAMON RECLAIM module. I've verified 
> that the watermarks were properly set in sysfs to my custom values, but 
> it doesn't seem to affect anything as free -mh shows 400Mb for apps but 
> 2.6GB for caches/buffers. The VM besides idling for a very long time 
> isn't able to free the buffers. When dropping the caches manually using 
> /proc/sys/vm/drop_caches the memory usage returns back to the starting 
> one. The cache/buffers don't increase at all after dropping them 
> indicating that this memory was indeed idling.

Thank you for sharing your great experience and questions!

> 
> My questions:
> 1. Are there types of freeable memory which DAMON is not allowed to touch?

Basically there is no such limitation.  We implemented page type of cgroups
based DAMOS filtering feature in v6.3, but as you're using v6.2, it shouldn't
be related with your use case.

One possible limitation for this case might be the monitoring region.  You can
specify the region to monitor and reclaim using `monitor_region_{start,end}`
parameters.  By default, it's set to biggest System RAM.  If your system is
having non-countinuous System RAMs and the biggest one is not covering the 3GiB
region, the 3GiB regions will not be moitored and therefore not reclaimed.

Can you check if it is excluding the 3GiB region?  You may be able to get it
using `proc/iomem` like files.  You could also refer to DAMON user-space tool
to show its usage of the file[1].

Also, you could get DAMON_RECLAIM internal statistics[2].  Checking those could
also provide some hints, or help excluding unnecessary suspects.

> 2. What prevents DAMON from getting back the memory?

Other than quotas, watermarks and access pattern, there should be nothing
preventing DAMON_RECLAIM reclaiming memory on v6.2 kernel.  DAMOS filters could
also make some effect, but as mentioned-above, it's available from v6.3.

> 2. /sys/kernel/debug/damon/* seems separate from DAMON RECLAIM, 
> /sys/module/damon_reclaim/parameters/kdamond_pid shows DAMON RECLAIM is 
> running but the DAMON debugfs doesn't show it nor exposes any registered 
> reclamation schemes.

You're correct.  DAMON provides two main user interfaces, via debugfs
(/sys/kernel/debug/damon/) and sysfs (/sys/kernel/mm/damon/).  Those are for
fine-controlled use of all DAMON capabilities.  Btw, the debugfs interface is
deprecated now, so please use the sysfs interface.

DAMON modules like DAMON_RECLAIM and DAMON_LRU_SORT are for simpler control of
DAMON for only special purpose system-wide utilization, like proactive reclaim
and LRU lists manipulation.  Those hence provide simpler module parameters
interface.


[1] https://github.com/awslabs/damo/blob/next/_damo_paddr_layout.py
[2] https://docs.kernel.org/admin-guide/mm/damon/reclaim.html#nr-reclaim-tried-regions


Thanks,
SJ

> 
> Best Regards,
> Grzegorz Uriasz