Re: [RFC PATCH 00/19] mm/damon: introduce data attributes monitoring

[Gutierrez Asier <gutierrez.asier@huawei-partners.com>]

Hi SeonJae,

On 4/26/2026 11:52 PM, SeongJae Park wrote:
> TL; DR
> ======
> 
> Extend DAMON for monitoring general data attributes other than accesses.
> This is for enabling light-weight page type (e.g., belonging cgroup)
> aware monitoring in short term.  In long term, this will help extending
> DAMON for multiple access events capture primitives (e.g., page faults
> and PMU) and eventually pivotting DAMON to a "Data Attributes Monitoring
> and Operations eNgine" in long term.

Very interesting. Looking forward to seeing this in upstream.

> 
> Background: High Cost of Page Level Properties Monitoring
> =========================================================
> 
> DAMON is initially introduced as a Data Access MONitor.  It has been
> extended for not only access monitoring but also data access-aware
> system operations (DAMOS).  But still the monitoring part is only for
> data accesses.
> 
> Data access patterns is good information, but some users need more
> holistic views.  Particularly, users want to show the access pattern
> information together with the types of the memory.  For example, users
> who work for making huge pages efficiently want to know how much of
> DAMON-found hot/cold regions are backed by huge pages.  Users who run
> multiple workloads with different cgroups want to know how much of
> DAMON-found hot/cold regions belong to specific cgroups.
> 
> For the user demand, we developed a DAMOS extension for page level
> properties based monitoring [1], which has landed on 6.14.  Using the
> feature, users can inform the page level data properties that they are
> interested in, in a flexible format that uses DAMOS filters.  Then,
> DAMON applies the filters to each folio of the entire DAMON region and
> lets users know how many bytes of memory in each DAMON region passed the
> given filters.
> 
> This gives page level detailed and deterministic information to users.
> But, because the operation is done at page level, the overhead is
> proportional to the memory size.  It was useful for test or debugging
> purposes on a small number of machines.  But it was obviously too heavy
> to be enabled always on all machines running the real user workloads.
> For real world workloads, it was recommended to use the feature with
> user-space controlled sampling approaches.  For example, users could do
> the page level monitoring only once per hour, on randomly selected one
> percent of machines of their fleet.  If the runtime and the  size of the
> fleet is long and big enough, it should provide statistically meaningful
> data.
> 
> But users are too busy to implement such controls on their own.
> 
> Data Attributes Monitoring
> ==========================
> 
> Extend DAMON to monitor not only data accesses, but also general data
> attributes.  Do the extension while keeping the main promise of DAMON,
> the bounded and best-effort minimum overhead.
> 
> Allow users to specify what data attributes in addition to the data
> access they want to monitor.  Users can install one 'data probe' per
> data attribute of their interest for this purpose.  The 'data probe'
> should be able to be applied to any memory, and determine if the given
> memory has the appropriate data attribute.  E.g., if memory of physical
> address 42 belongs to cgroup A.  Each 'data probe' is configured with
> filters that are very similar to the DAMOS filters.
> 
> When DAMON checks if each sampling address memory of each region is
> accessed since the last check, it applies data probes if registered.
> Same to the number of access check-positive samples accounting
> (nr_accesses), it accounts the number of each data probe-positive
> samples in another per-region counters array, namely 'probe_hits'. When
> DAMON resets nr_accesses every aggregation interval, it resets
> 'probe_hits' together.
> 
> Users can read 'probe_hits' just before the values are reset.  In this
> way, users can know how many hot/cold memory regions have data
> attributes of their interest.  E.g., 30 percent of this system's hot
> memory is belonging to cgroup A and 80 percent of the hot cgroup A
> memory is backed by huge pages.
> 
> Patches Sequence
> ================
> 
> First eight patches implement the core feature, interface and the
> working support.  Patch 1 introduces data probe data structure, namely
> damon_probe.  Patch 2 extends damon_ctx for installing data probes.
> Patch 3 introduces another data structure for filters of each data
> probe, namely damon_filter.  Patch 4 updates damon_ctx commit function
> to handle the probes.  Patch 5 extends damon_region for the per-region
> per-probe positive samples counter, namely probe_hits.  Patch 6 extends
> damon_operations for applying probes on the underlying DAMON operations
> implementation.  Patch 7 updates kdamond_fn() to invoke the probes
> applying callback.  Patch 8 finally implements the probes support on
> paddr ops.
> 
> Eight changes for user interface (patches 9-16) come next.  Patches 9-13
> implements sysfs directories and files for setting data probes, namely
> probes directory, probe directory, filters directory, filter directory
> and filter directory internal files, respectively.  Patch 14 connects
> the user inputs that are made via the sysfs files to DAMON core.
> Patch 15 implements sysfs files for showing the per-region per-probe
> positive samples count, namely probe_hits.  Patch 16 introduces a new
> tracepoint for showing the counts via tracefs.
> 
> Patch 14 adds a selftest for the sysfs files.
> 
> Patches 15 and 16 documents the design and usage of the new feature,
> respectively.
> 
> Discussions
> ===========
> 
> This allows the page properties monitoring with overhead that is low
> enough to be enabled always on real world workloads.  Because the
> sampling time for access check is reused for data attributes check,  the
> upper-bounded and best-effort minimum overhead of DAMON is kept.
> Because the sampling memory for access check is reused for data
> attributes check, additional overhead is minimum.
> 
> Still DAMOS-based page level properties monitoring should be useful,
> because it provides a deterministic page level information.  When in
> doubt of the sampling based information, running DAMOS-based one
> together and comparing the results would be useful, for debugging and
> tuning.
> 
> Plan for Dropping RFC tag
> =========================
> 
> The user ABI for reading probe_hits is not yet convincing.  It is
> exposed to users by a tracepoint and new sysfs file.  For the
> tracepoint, a new one namely damon:damon_aggregated_v2 is introduced.
> The name is not convincing, and its internal mechanism seems to have
> room to be improved before dropping RFC.  For the sysfs, a file under
> the DAMOS-tried region directory namely 'probe_hits' is added.  Reading
> it returns four probe_hits values with ',' as a separator.  With the
> maximum number of data probes, this should work.  This can make future
> changes of the limit difficult.  I will try to find a better way before
> dropping the RFC tag.  Maybe 'probe_hits/' directory having files of
> name '0' to 'N-1' for each of user-registered 'N' data probes.
> 
> I'm currently hoping to drop the RFC tag by 7.2-rc1.
> 
> Future Works: Short Term
> ========================
> 
> This series is introducing only a single type of data attribute:
> anonymous page.  Once this is landed, I will extend it for
> cgroup-belonging, so that we can do cgroup-level monitoring with low
> overhead.  After that, I may further work on supporting all DAMOS filter
> types.  And as demands are found, we could extend the types.
> 
> This version of implementation is limiting the maximum number of data
> probes to four.  I will try to find a way to remove the limit in future,
> if it is easy to do.  I personally think it should be enough for common
> use cases, though, and therefore not giving high priority at the moment.
> 
> Future Works: Long Term
> =======================
> 
> There are user requests for extending DAMON with detailed access
> information, for example, per-CPUs/threads/read/writes monitoring.  For
> that, I was working [2] on extending DAMON to use page fault events as
> another access check primitives, and making the infrastructure flexible
> for future use of yet another access check primitive.  Actually there is
> another ongoing work [3] for extending DAMON with PMU events.  The
> motivation of the work is reducing the overhead, though.
> 
> In my work [2], I was introducing a new interface for access sampling
> primitives control.  Now I think this data probe interface can be used
> for that, too.  That is, data access becomes just one type of data
> attribute.  Also, pg_idle-confirmed access, page fault-confirmed access,
> and PMU event-confirmed access will be different types of data
> attributes.
> 
> The regions adjustment mechanism is currently working based on the
> access information.  That's because DAMON is designed for data access
> monitoring.  That is, data access information is the primary interest,
> and therefore DAMON adjusts regions in a way that can best-present the
> information.
> 
> Once data access becomes just one of data attributes, there is no reason
> to think data access that special.  There might be some users not
> interested in access at all but want to know the location of memory of
> specific type.  Data probes interface will allow doing that.  Further,
> we could extend the interface to let users set any data attribute as the
> 'primary' attribute.  Then, DAMON will split and merge regions in a way
> that can best-present the 'primary' attributes.
> 
> DAMOS will also be extended, to specify targets based on not only the
> data access pattern, but all user-registered data attributes.  From this
> stage, we may be able to call DAMON as a "Data Attributes Monitoring and
> Operations eNgine".
> 
> [1] https://lore.kernel.org/20250106193401.109161-1-sj@kernel.org
> [2] https://lore.kernel.org/20251208062943.68824-1-sj@kernel.org/
> [3] https://lore.kernel.org/20260423004211.7037-1-akinobu.mita@gmail.com
> 
> SeongJae Park (19):
>   mm/damon/core: introduce struct damon_probe
>   mm/damon/core: embed damon_probe objects in damon_ctx
>   mm/damon/core: introduce damon_filter
>   mm/damon/core: commit probes
>   mm/damon/core: introduce damon_region->probe_hits
>   mm/damon/core: introduce damon_ops->apply_probes
>   mm/damon/core: do data attributes monitoring
>   mm/damon/paddr: support data attributes monitoring
>   mm/damon/sysfs: implement probes dir
>   mm/damon/sysfs: implement probe dir
>   mm/damon/sysfs: implement filters directory
>   mm/damon/sysfs: implement filter dir
>   mm/damon/sysfs: implement filter dir files
>   mm/damon/sysfs: setup probes on DAMON core API parameters
>   mm/damon/sysfs-schemes: implement tried_region/probe_hits file
>   mm/damon: trace probe_hits
>   selftests/damon/sysfs.sh: test probes dir
>   Docs/mm/damon/design: document data attributes monitoring
>   Docs/admin-guide/mm/damon/usage: document data attributes monitoring
> 
>  Documentation/admin-guide/mm/damon/usage.rst |  44 +-
>  Documentation/mm/damon/design.rst            |  37 ++
>  include/linux/damon.h                        |  60 +++
>  include/trace/events/damon.h                 |  41 ++
>  mm/damon/core.c                              | 182 +++++++
>  mm/damon/paddr.c                             |  45 ++
>  mm/damon/sysfs-schemes.c                     |  30 ++
>  mm/damon/sysfs.c                             | 502 +++++++++++++++++++
>  tools/testing/selftests/damon/sysfs.sh       |  48 ++
>  9 files changed, 982 insertions(+), 7 deletions(-)
> 
> 
> base-commit: 8f22aa2e28454419ed2031119ad32ea4a6c9f1f1

My main concern is about potential pollution of sysfs. DAMON is already
complex to set up, with a lot of knobs. Adding more configuration options
may make admin's job more complex.

Do you plan to support this extension in damo user space?

-- 
Asier Gutierrez
Huawei