Roadmap for extending DAMON beyond pte-accessed bit

[SeongJae Park <sj@kernel.org>]

Hello,


TLDR: Let's extend DAMON for data attributes monitoring, andd then further
extend that for multiple access check primitives including page faults, perf
memory-access events and optimized AMD IBS-like h/w feature drivers.

Ongoing Projects
================

At the moment, DAMON is utilizing page table accessed bits as its major access
check primitive.  Because of the limitations in the primitive, interests in
extending DAMON to use different access check primitives, including page fault
events and h/w features such as AMD IBS, were expressed multiple times.

I started working [1] on the page fault events based extension for
per-CPUs/threads/reads/writes monitoring.  Akinobu is working [2] on perf
events based extension for lower overhead fixed granularity monitoring.  Ravi
is working [3] on AMD IBS based extension for memory tiering.

Xin Hao proposed [4] extending DAMON for NUMA systems in 2022.  Pedro Demarchi
Gomes proposed [5,6] extending DAMON for write-only monitoring in 2022 and
2025.  My page faults based monitoring extension is partly for their projects.

Types of Required Works
=======================

Each work commonly requires three types of changes.  Startign from the lowest
layer, the required changes are as below.

Firstly, we need to extend existing DAMON operation set (paddr and vaddr) or
implement a new one for controlling the new access check primitives.  This
part, at least for page fault events and perf events are not overlapping and we
could work in parallel.

Secondly, we need DAMON core layer change for the reporting-based access check.
Page table accessed bits are set by h/w and DAMON does nothing about it.  DAMON
wait until h/w sets the bits, and later harvest the information by reading the
bits.  In other words, DAMON only "pulls" the information.  For page faults and
perf events like primitives, we need to hook the access events and "push" the
information to DAMON.  For the page fault events based extension [1], I
implemented damon_report_access() and related infrastructure for this purpose.
The implementation is quite unoptimized.  But I believe the design is good
enough for long term maintainability.  I want us to use this framework.  We can
make it works first, and later optimize.

Finally, we need DAMON user interface.  For the page fault events based
extension [1], I introduced access sampling control interface.  The idea is
letting users control the access sampling in a more detailed way.   At that
time, I was thinking it is long term maintainable and Akinobu and I could reuse
it.  I still think it would work.  But, I recently got another idea.

The new idea is extending DAMON for general data attributes monitoring.  The
first change [7] for making it able to monitor not only data access but also
general dasta attributes including anonymity of the pages and belonging memory
cgroup is recently merged into mm.git.  As the cover letter of the patch series
is also explaining, the work has started for light-weight per-cgroup access
monitoring.  But, in long term, we want to make data access information as just
one of the supported data attributes.  Then, we can further extend access
confirmation coming from different primitives as different data attributes.
For example, in addition to "PTE Accessed-bit" attribute, we can add "perf
event mem-access reported" attribute and "AMD IBS access reported" attribute.
I think this is long term maintainable and therefore want us to use this
interface.

Roadmap
=======

Assuming you agree to use the data attributes monitoring interface, I suggest
us to do the work in below roadmap.

Milestone 1: PTE Accessed bit as one of the data attributes
-----------------------------------------------------------

I will work on stabilizing and further extending the data attributes interface
and internal framework to be able to support PTE Accessed bit.  This may take
no small amount of efforts, but hopefully doable by the end of this year.

Milestone 2: First damon_report_access()-based data attribute monitoring
------------------------------------------------------------------------

Once the first milestone is completed, we will add the damon_report_access()
and related infrastructure changes into the DAMON core.  On top of it, we will
further implement the first data attribute that utilizing the reporting
infrastructure.  I personally think the perf event based one is a good target
at the momentt.  I think so mainly because the perf maintainers didn't show
special concern yet.  We can discuss the target again after the first milestone
is completed, though.  Hopefully this is doable by the LSFMMBPF 2027.

The timeline is just a gut feeling.  It could be done much earlier or later.

Milestone 3: Parallel extenstion for other primitives
-----------------------------------------------------

After milestone 2 is done, we have the user interface and the infrastructure.
We will be able to further implement our favorite access check primitives on
top of it in individual schedule, without blocked by each other.

Collaboration
=============

Milestone 1 and damon_report_access() part of milestone 2 would need to
primarly done by myself.  If you are interested in this project, reviewing the
patches that I will post for the milestones and doing some testing would be
very helpful.

For the second half of milestone 2, I may need more help from Akinobu, Ravi or
someone else who may experienced with the first target primitive.  Maybe I
could develop the damon_report_access() part and post it on the mailing list,
and the other collaborator could further develop the first target primitive
extension on top of the patches, like we did for vaddr page interleaving.
Let's discuss the details after milestone 1, though.

Request For Comments
====================

I would appreciate any inputs about this roadmap and plan.  I'm primarily
curious if the plan and the timeline makes sense to Akinobu and Ravi, who are
actively working on, and might need to wait for my milestone 1.  Also if there
are people who interested in this, please feel free to add your inputs publicly
or privately, in your preferred way.

[1] https://lore.kernel.org/20251208062943.68824-1-sj@kernel.org
[2] https://lore.kernel.org/20260423004211.7037-1-akinobu.mita@gmail.com/
[3] https://lore.kernel.org/20260516223439.4033-1-ravis.opensrc@gmail.com
[4] https://lore.kernel.org/linux-mm/cover.1645024354.git.xhao@linux.alibaba.com/
[5] https://lore.kernel.org/lkml/20220203131237.298090-1-pedrodemargomes@gmail.com/
[6] https://lore.kernel.org/damon/20250129044041.25884-1-pedrodemargomes@gmail.com/
[7] https://lore.kernel.org/20260518234119.97569-1-sj@kernel.org


Thanks,
SJ