[PATCH 0/4] mm/damon: introduce DAMON_STAT for simple and practical access monitoring

[PATCH 0/4] mm/damon: introduce DAMON_STAT for simple and practical access monitoring

[SeongJae Park]

DAMON-based access monitoring is not simple due to required DAMON
control and results visualizations.  Introduce a static kernel module
for making it simple.  The module can be enabled without manual setup
and provides access pattern metrics that easy to fetch and understand
the practical access pattern information, namely estimated memory
bandwidth and memory idle time percentiles.

Background and Problems
=======================

DAMON can be used for monitoring data access patterns of the system and
workloads.  Specifically, users can start DAMON to monitor access events
on specific address space with fine controls including address ranges to
monitor and time intervals between samplings and aggregations.  The
resulting access information snapshot contains access frequency
(nr_accesses) and how long the frequency was kept (age) for each byte.

The monitoring usage is not simple and practical enough for production
usage.  Users should first start DAMON with a number of parameters, and
wait until DAMON's monitoring results capture a reasonable amount of the
time data (age).  In production, such manual start and wait is
impractical to capture useful information from a high number of machines
in a timely manner.

The monitoring result is also too detailed to be used on production
environments.  The raw results are hard to be aggregated and/or compared
for production environments having a large scale of time, space and
machines fleet.

Users have to implement and use their own automation of DAMON control
and results processing.  It is repetitive and challenging since there is
no good reference or guideline for such automation.

Solution: DAMON_STAT
====================

Implement such automation in kernel space as a static kernel module,
namely DAMON_STAT.  It can be enabled at build, boot, or run time via
its build configuration or module parameter.  It monitors the entire
physical address space with monitoring intervals that auto-tuned for a
reasonable amount of access observations and minimum overhead.  It
converts the raw monitoring results into simpler metrics that can easily
be aggregated and compared, namely estimated memory bandwidth and idle
time percentiles.  Refer to the commit messages of the second and the
third patches of this patch series for more details about the metrics.

Discussions
===========

The module aims to be useful on production environments constructed with
a large number of machines that run a long time.  The auto-tuned
monitoring intervals ensure a reasonable quality of the outputs.  The
auto-tuning also ensures its overhead be reasonable and low enough to be
enabled always on the production.  The simplified monitoring results
metrics can be useful for showing both coldness (idle time percentiles)
and hotness (memory bandwidth) of the system's access pattern.  We
expect the information can be useful for assessing system memory
utilization and inspiring optimizations or investigations on both kernel
and user space memory management logics for large scale fleets.

We hence expect the module is good enough to be just used in most
environments.  For special cases that require a custom access monitoring
automation, users will still benefit by using DAMON_STAT as a reference
or a guideline for their specialized automation.

Revision History
================

Changes from RFC
(https://lore.kernel.org/20250519164415.43935-1-sj@kernel.org)
- Add an admin-guide documentation
- Wordsmith commit messages
- Rebase to latest mm-new

SeongJae Park (4):
  mm/damon: introduce DAMON_STAT module
  mm/damon/stat: calculate and expose estimated memory bandwidth
  mm/damon/stat: calculate and expose idle time percentiles
  Docs/admin-guide/mm/damon: add DAMON_STAT usage document

 Documentation/admin-guide/mm/damon/index.rst |   1 +
 Documentation/admin-guide/mm/damon/stat.rst  |  69 ++++++
 mm/damon/Kconfig                             |  16 ++
 mm/damon/Makefile                            |   1 +
 mm/damon/stat.c                              | 245 +++++++++++++++++++
 5 files changed, 332 insertions(+)
 create mode 100644 Documentation/admin-guide/mm/damon/stat.rst
 create mode 100644 mm/damon/stat.c


base-commit: 90887f57d7a67917136e7c70d26fb3f2fcdc6f53
-- 
2.39.5

[PATCH 4/4] Docs/admin-guide/mm/damon: add DAMON_STAT usage document

[SeongJae Park]

Document DAMON_STAT usage and add a link to it on DAMON admin-guide
page.

Signed-off-by: SeongJae Park <sj@kernel.org>
---
 Documentation/admin-guide/mm/damon/index.rst |  1 +
 Documentation/admin-guide/mm/damon/stat.rst  | 69 ++++++++++++++++++++
 2 files changed, 70 insertions(+)
 create mode 100644 Documentation/admin-guide/mm/damon/stat.rst

diff --git a/Documentation/admin-guide/mm/damon/index.rst b/Documentation/admin-guide/mm/damon/index.rst
index bc7e976120e0..3ce3164480c7 100644
--- a/Documentation/admin-guide/mm/damon/index.rst
+++ b/Documentation/admin-guide/mm/damon/index.rst
@@ -14,3 +14,4 @@ access monitoring and access-aware system operations.
    usage
    reclaim
    lru_sort
+   stat
diff --git a/Documentation/admin-guide/mm/damon/stat.rst b/Documentation/admin-guide/mm/damon/stat.rst
new file mode 100644
index 000000000000..4c517c2c219a
--- /dev/null
+++ b/Documentation/admin-guide/mm/damon/stat.rst
@@ -0,0 +1,69 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================
+Data Access Monitoring Results Stat
+===================================
+
+Data Access Monitoring Results Stat (DAMON_STAT) is a static kernel module that
+is aimed to be used for simple access pattern monitoring.  It monitors accesses
+on the system's entire physical memory using DAMON, and provides simplified
+access monitoring results statistics, namely idle time percentiles and
+estimated memory bandwidth.
+
+Monitoring Accuracy and Overhead
+================================
+
+DAMON_STAT uses monitoring intervals :ref:`auto-tuning
+<damon_design_monitoring_intervals_autotuning>` to make its accuracy high and
+overhead minimum.  It auto-tunes the intervals aiming 4 % of observable access
+events to be captured in each snapshot, while limiting the resulting sampling
+events to be 5 milliseconds in minimum and 10 seconds in maximum.  On a few
+production server systems, it resulted in consuming only 0.x % single CPU time,
+while capturing reasonable quality of access patterns.
+
+Interface: Module Parameters
+============================
+
+To use this feature, you should first ensure your system is running on a kernel
+that is built with ``CONFIG_DAMON_STAT=y``.  The feature can be enabled by
+default at build time, by setting ``CONFIG_DAMON_STAT_ENABLED_DEFAULT`` true.
+
+To let sysadmins enable or disable it at boot and/or runtime, and read the
+monitoring results, DAMON_STAT provides module parameters.  Following
+sections are descriptions of the parameters.
+
+enabled
+-------
+
+Enable or disable DAMON_STAT.
+
+You can enable DAMON_STAT by setting the value of this parameter as ``Y``.
+Setting it as ``N`` disables DAMON_STAT.  The default value is set by
+``CONFIG_DAMON_STAT_ENABLED_DEFAULT`` build config option.
+
+estimated_memory_bandwidth
+--------------------------
+
+Estimated memory bandwidth consumption (bytes per second) of the system.
+
+DAMON_STAT reads observed access events on the current DAMON results snapshot
+and converts it to memory bandwidth consumption estimation in bytes per second.
+The resulting metric is exposed to user via this read-only parameter.  Because
+DAMON uses sampling, this is only an estimation of the access intensity rather
+than accurate memory bandwidth.
+
+memory_idle_ms_percentiles
+--------------------------
+
+Per-byte idle time (milliseconds) percentiles of the system.
+
+DAMON_STAT calculates how long each byte of the memory was not accessed until
+now (idle time), based on the current DAMON results snapshot.  If DAMON found a
+region of access frequency (nr_accesses) larger than zero, every byte of the
+region gets zero idle time.  If a region has zero access frequency
+(nr_accesses), how long the region was keeping the zero access frequency (age)
+becomes the idle time of every byte of the region.  Then, DAMON_STAT exposes
+the percentiles of the idle time values via this read-only parameter.  Reading
+the parameter returns 101 idle time values in milliseconds, separated by comma.
+Each value represents 0-th, 1st, 2nd, 3rd, ..., 99th and 100th percentile idle
+times.
-- 
2.39.5

Re: [PATCH 0/4] mm/damon: introduce DAMON_STAT for simple and practical access monitoring

[Honggyu Kim]

Hi SeongJae,

Thanks for your work.

On 5/27/2025 6:09 AM, SeongJae Park wrote:
> DAMON-based access monitoring is not simple due to required DAMON
> control and results visualizations.  Introduce a static kernel module
> for making it simple.  The module can be enabled without manual setup
> and provides access pattern metrics that easy to fetch and understand
> the practical access pattern information, namely estimated memory
> bandwidth and memory idle time percentiles.
> 
> Background and Problems
> =======================
> 
> DAMON can be used for monitoring data access patterns of the system and
> workloads.  Specifically, users can start DAMON to monitor access events
> on specific address space with fine controls including address ranges to
> monitor and time intervals between samplings and aggregations.  The
> resulting access information snapshot contains access frequency
> (nr_accesses) and how long the frequency was kept (age) for each byte.
> 
> The monitoring usage is not simple and practical enough for production
> usage.  Users should first start DAMON with a number of parameters, and
> wait until DAMON's monitoring results capture a reasonable amount of the
> time data (age).  In production, such manual start and wait is
> impractical to capture useful information from a high number of machines
> in a timely manner.
> 
> The monitoring result is also too detailed to be used on production
> environments.  The raw results are hard to be aggregated and/or compared
> for production environments having a large scale of time, space and
> machines fleet.
> 
> Users have to implement and use their own automation of DAMON control
> and results processing.  It is repetitive and challenging since there is
> no good reference or guideline for such automation.
> 
> Solution: DAMON_STAT
> ====================
> 
> Implement such automation in kernel space as a static kernel module,
> namely DAMON_STAT.  It can be enabled at build, boot, or run time via
> its build configuration or module parameter.  It monitors the entire
> physical address space with monitoring intervals that auto-tuned for a
> reasonable amount of access observations and minimum overhead.  It
> converts the raw monitoring results into simpler metrics that can easily
> be aggregated and compared, namely estimated memory bandwidth and idle
> time percentiles.  Refer to the commit messages of the second and the
> third patches of this patch series for more details about the metrics.

I see the description looks good but it'd be useful if you could share some
execution commands and expected output examples that some newbies can get better
ideas.

I honestly do not have a clear idea how I can use this kind of static kernel
modules as general users although I have developed some features of DAMON.

So could you please help?

Thanks,
Honggyu

> 
> Discussions
> ===========
> 
> The module aims to be useful on production environments constructed with
> a large number of machines that run a long time.  The auto-tuned
> monitoring intervals ensure a reasonable quality of the outputs.  The
> auto-tuning also ensures its overhead be reasonable and low enough to be
> enabled always on the production.  The simplified monitoring results
> metrics can be useful for showing both coldness (idle time percentiles)
> and hotness (memory bandwidth) of the system's access pattern.  We
> expect the information can be useful for assessing system memory
> utilization and inspiring optimizations or investigations on both kernel
> and user space memory management logics for large scale fleets.
> 
> We hence expect the module is good enough to be just used in most
> environments.  For special cases that require a custom access monitoring
> automation, users will still benefit by using DAMON_STAT as a reference
> or a guideline for their specialized automation.
> 
> Revision History
> ================
> 
> Changes from RFC
> (https://lore.kernel.org/20250519164415.43935-1-sj@kernel.org)
> - Add an admin-guide documentation
> - Wordsmith commit messages
> - Rebase to latest mm-new
> 
> SeongJae Park (4):
>    mm/damon: introduce DAMON_STAT module
>    mm/damon/stat: calculate and expose estimated memory bandwidth
>    mm/damon/stat: calculate and expose idle time percentiles
>    Docs/admin-guide/mm/damon: add DAMON_STAT usage document
> 
>   Documentation/admin-guide/mm/damon/index.rst |   1 +
>   Documentation/admin-guide/mm/damon/stat.rst  |  69 ++++++
>   mm/damon/Kconfig                             |  16 ++
>   mm/damon/Makefile                            |   1 +
>   mm/damon/stat.c                              | 245 +++++++++++++++++++
>   5 files changed, 332 insertions(+)
>   create mode 100644 Documentation/admin-guide/mm/damon/stat.rst
>   create mode 100644 mm/damon/stat.c
> 
> 
> base-commit: 90887f57d7a67917136e7c70d26fb3f2fcdc6f53

Re: [PATCH 0/4] mm/damon: introduce DAMON_STAT for simple and practical access monitoring

[SeongJae Park]

Hi Honggyu,

On Thu, 29 May 2025 16:21:39 +0900 Honggyu Kim <honggyu.kim@sk.com> wrote:

> Hi SeongJae,
> 
> Thanks for your work.
> 
> On 5/27/2025 6:09 AM, SeongJae Park wrote:
> > DAMON-based access monitoring is not simple due to required DAMON
> > control and results visualizations.  Introduce a static kernel module
> > for making it simple.  The module can be enabled without manual setup
> > and provides access pattern metrics that easy to fetch and understand
> > the practical access pattern information, namely estimated memory
> > bandwidth and memory idle time percentiles.
> > 
> > Background and Problems
> > =======================
> > 
> > DAMON can be used for monitoring data access patterns of the system and
> > workloads.  Specifically, users can start DAMON to monitor access events
> > on specific address space with fine controls including address ranges to
> > monitor and time intervals between samplings and aggregations.  The
> > resulting access information snapshot contains access frequency
> > (nr_accesses) and how long the frequency was kept (age) for each byte.
> > 
> > The monitoring usage is not simple and practical enough for production
> > usage.  Users should first start DAMON with a number of parameters, and
> > wait until DAMON's monitoring results capture a reasonable amount of the
> > time data (age).  In production, such manual start and wait is
> > impractical to capture useful information from a high number of machines
> > in a timely manner.
> > 
> > The monitoring result is also too detailed to be used on production
> > environments.  The raw results are hard to be aggregated and/or compared
> > for production environments having a large scale of time, space and
> > machines fleet.
> > 
> > Users have to implement and use their own automation of DAMON control
> > and results processing.  It is repetitive and challenging since there is
> > no good reference or guideline for such automation.
> > 
> > Solution: DAMON_STAT
> > ====================
> > 
> > Implement such automation in kernel space as a static kernel module,
> > namely DAMON_STAT.  It can be enabled at build, boot, or run time via
> > its build configuration or module parameter.  It monitors the entire
> > physical address space with monitoring intervals that auto-tuned for a
> > reasonable amount of access observations and minimum overhead.  It
> > converts the raw monitoring results into simpler metrics that can easily
> > be aggregated and compared, namely estimated memory bandwidth and idle
> > time percentiles.  Refer to the commit messages of the second and the
> > third patches of this patch series for more details about the metrics.
> 
> I see the description looks good but it'd be useful if you could share some
> execution commands and expected output examples that some newbies can get better
> ideas.

Thank you for the feedback.  In my humble opinion, nevertheless, this is what
the fourth patch of this series is aiming to provide.  And thanks to your
comment, I now realize I forgot mentioning this on the above comment.

Of course we can add execution commands and output example here, but without
understanding of the metrics, it would just look like mysterious numbers.
Meanhile, user interface of the module is simply module parameters that should
be familiar to most kernel users, so I don't think it necessarily deserves
example commands.

> 
> I honestly do not have a clear idea how I can use this kind of static kernel
> modules as general users although I have developed some features of DAMON.
> 
> So could you please help?

I agree this cover letter is not enough for getting clear idea of how readers
can use this feature.  Thank you for letting me realize this.  But I think this
cover letter might be better to keep the brevity, as long as it points where
readers can get necessary details.  What do you think about revising this cover
letter to

1. point the fourth patch for usage of DAMON_STAT, and
2. elaborate the fact that readers are recommended to read the metric
   definitions and usages to get more clear idea of DAMON_STAT?


Thanks,
SJ

[...]