[PATCH 0/6] mm/damon: support access monitoring of hugetlb-backed memory

[PATCH 0/6] mm/damon: support access monitoring of hugetlb-backed memory

[Krishna Iyer]

On virtualization hosts, most system memory is often backed by
hugetlbfs.  On our production hosts, for example, ~95% of RAM is 1 GiB
hugetlb pages backing guest memory.  DAMON's physical address space
monitoring cannot produce a useful access signal for such memory, for
two layered reasons.

First, it is blind to hugetlb folios.  Every access check starts at
damon_get_folio(), which rejects folios that are not on the LRU lists,
and hugetlb folios are managed outside of the LRU by design.  As a
result, all hugetlb-backed memory is silently reported as never
accessed.

Second, even once hugetlb folios are visible, access-bit-based sampling
under-reports hot memory.  DAMON deliberately clears accessed bits
without TLB flushes to keep the overhead low, so accesses served from
cached translations do not walk the page tables and never re-set the
bit.  With 1 GiB mappings, translations effectively never leave the
TLBs, pinning the observed access rate to the TLB-refill rate: a
saturating guest is indistinguishable from a nearly idle one.

Patches 1-3 address the visibility problem.  The first patch moves
damon_hugetlb_mkold() from vaddr to ops-common as a preparation.  The
second patch teaches the folio mkold/young rmap walkers to handle
hugetlb folios, aging the huge PTE and notifying secondary MMUs across
the whole huge page size; the secondary MMU notification is what
surfaces guest-side (e.g., KVM/EPT) accessed bits.  The third patch
adds damon_get_folio_incl_hugetlb() and uses it from the paddr
monitoring primitives only.  DAMOS action appliers such as
DAMON_RECLAIM and DAMON_LRU_SORT keep the LRU-only lookup and are
behaviorally unchanged.

Patches 4-6 address the intensity problem with a new default-off
monitoring context option, 'aging_flush'.  When set, access bit
clearing for monitoring uses the flushing primitives
(mmu_notifier_clear_flush_young() and TLB-flushing PTE/PMD/hugetlb
aging) so that the next access is guaranteed to re-set the accessed
bit.  The option is exposed via the DAMON sysfs interface (patch 5) and
as a DAMON_STAT module parameter (patch 6).  With the option unset, the
behavior is identical to before this series.

Evaluation on an Intel EPT host running a 944 GiB guest with an 842 GiB
in-guest memory workload (masim):

  configuration          estimated access rate    idle -> load
                         idle -> load (GB/s)      separation
  stock                  0.08 -> 0.21             none (guest memory
                                                  invisible)
  patches 1-3            1.70 -> 1.96             1.2x
  full series, flush on  1.21 -> 5.34             4.4x

With aging_flush set, the estimated hot footprint covers ~82% of the
actual working set size, compared to ~40% without flushing, where the
signal mostly reflects the ambient TLB-refill rate rather than guest
accesses.  kdamond CPU consumption stays approximately 0%, and the
worst-case flush overhead measured is below 0.1%, which is acceptable
for hosts that opt in.

Per Documentation/process/generated-content.rst, this series was
developed with the assistance of an AI coding assistant (Anthropic
Claude, via Claude Code).  The assistant helped draft the code and
changelogs, and performed the rebase of the series from its original
6.17-based development tree onto mm-new, including merge conflict
resolution against recent DAMON changes.  All changes were reviewed by
the human submitter, who takes full responsibility for the
contribution.

The series as posted here was regression-tested on its base commit
with a full x86_64 kernel build (no new W=1 warnings in mm/damon), the
DAMON kunit suite (41/41 passing) and the DAMON selftests (15/15
passing) on a kernel booted with virtme-ng.  Functional validation
(the evaluation above, ftrace verification that the flag switches the
notifier variants, and enable/disable stress testing) was performed by
the submitter on production-like hosts running the 6.17-based backport
of this series.

Krishna Iyer (6):
  mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common
  mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap
    walkers
  mm/damon/paddr: support hugetlb folios in access monitoring
  mm/damon: support flush-assisted access bit clearing for monitoring
  mm/damon/sysfs: support aging_flush
  mm/damon/stat: support aging_flush

 include/linux/damon.h |   2 +
 mm/damon/core.c       |   2 +
 mm/damon/ops-common.c | 154 ++++++++++++++++++++++++++++++++++++------
 mm/damon/ops-common.h |  18 ++++-
 mm/damon/paddr.c      |  11 +--
 mm/damon/stat.c       |   6 ++
 mm/damon/sysfs.c      |  29 ++++++++
 mm/damon/vaddr.c      |  46 +++----------
 8 files changed, 203 insertions(+), 65 deletions(-)


base-commit: d2aad7fdcda7ae8a726926f2d6de7fe9e8ee7563
-- 
2.54.0

[PATCH 1/6] mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common

[Krishna Iyer]

damon_hugetlb_mkold() clears the accessed bit of a hugetlb-mapping huge
PTE and propagates the aging to secondary MMUs via
mmu_notifier_clear_young(), spanning the whole huge page size.  It
currently lives in vaddr.c, and is thus usable only by the virtual
address space monitoring operations set.

The physical address space monitoring operations set will need the same
logic, to support access monitoring of hugetlb-backed memory.  Move the
function to ops-common as-is, with no behavioral change.  A follow-up
change will use it from the folio-granular rmap walkers.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
---
 mm/damon/ops-common.c | 30 ++++++++++++++++++++++++++++++
 mm/damon/ops-common.h |  9 +++++++++
 mm/damon/vaddr.c      | 27 ---------------------------
 3 files changed, 39 insertions(+), 27 deletions(-)

diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index fbda70d8ea4d..f5fe92b825bb 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -3,6 +3,7 @@
  * Common Code for Data Access Monitoring
  */
 
+#include <linux/hugetlb.h>
 #include <linux/migrate.h>
 #include <linux/mmu_notifier.h>
 #include <linux/page_idle.h>
@@ -98,6 +99,35 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
+void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
+		struct vm_area_struct *vma, unsigned long addr)
+{
+	bool referenced = false;
+	pte_t entry = huge_ptep_get(mm, addr, pte);
+	struct folio *folio = pfn_folio(pte_pfn(entry));
+	unsigned long psize = huge_page_size(hstate_vma(vma));
+
+	folio_get(folio);
+
+	if (pte_young(entry)) {
+		referenced = true;
+		entry = pte_mkold(entry);
+		set_huge_pte_at(mm, addr, pte, entry, psize);
+	}
+
+	if (mmu_notifier_clear_young(mm, addr,
+				     addr + huge_page_size(hstate_vma(vma))))
+		referenced = true;
+
+	if (referenced)
+		folio_set_young(folio);
+
+	folio_set_idle(folio);
+	folio_put(folio);
+}
+#endif	/* CONFIG_HUGETLB_PAGE */
+
 #define DAMON_MAX_SUBSCORE	(100)
 #define DAMON_MAX_AGE_IN_LOG	(32)
 
diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h
index 38d295488fa1..f7811c9c7a02 100644
--- a/mm/damon/ops-common.h
+++ b/mm/damon/ops-common.h
@@ -9,6 +9,15 @@ struct folio *damon_get_folio(unsigned long pfn);
 
 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr);
 void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr);
+#ifdef CONFIG_HUGETLB_PAGE
+void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
+		struct vm_area_struct *vma, unsigned long addr);
+#else
+static inline void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
+		struct vm_area_struct *vma, unsigned long addr)
+{
+}
+#endif	/* CONFIG_HUGETLB_PAGE */
 void damon_folio_mkold(struct folio *folio);
 bool damon_folio_young(struct folio *folio);
 
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index 0648400b2d65..15379d984689 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -293,33 +293,6 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
 }
 
 #ifdef CONFIG_HUGETLB_PAGE
-static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
-				struct vm_area_struct *vma, unsigned long addr)
-{
-	bool referenced = false;
-	pte_t entry = huge_ptep_get(mm, addr, pte);
-	struct folio *folio = pfn_folio(pte_pfn(entry));
-	unsigned long psize = huge_page_size(hstate_vma(vma));
-
-	folio_get(folio);
-
-	if (pte_young(entry)) {
-		referenced = true;
-		entry = pte_mkold(entry);
-		set_huge_pte_at(mm, addr, pte, entry, psize);
-	}
-
-	if (mmu_notifier_clear_young(mm, addr,
-				     addr + huge_page_size(hstate_vma(vma))))
-		referenced = true;
-
-	if (referenced)
-		folio_set_young(folio);
-
-	folio_set_idle(folio);
-	folio_put(folio);
-}
-
 static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask,
 				     unsigned long addr, unsigned long end,
 				     struct mm_walk *walk)
-- 
2.54.0

[PATCH 2/6] mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers

[Krishna Iyer]

damon_folio_mkold_one() and damon_folio_young_one() assume the folios
they walk are mapped by normal PTEs or THP PMDs.  When the folio is a
hugetlb folio, page_vma_mapped_walk() returns the huge PTE in pvmw.pte
with its page table lock held, but the walkers treat it as a normal
PTE: they read and age it with PAGE_SIZE-granularity helpers, which is
wrong for huge PTEs (up to PUD level), and notify secondary MMUs for
only PAGE_SIZE of the mapping.

Add hugetlb branches to both walkers.  The mkold walker reuses
damon_hugetlb_mkold(), which the virtual address space operations set
has been using for hugetlb aging: it clears the young bit of the huge
PTE via set_huge_pte_at() and calls mmu_notifier_clear_young() spanning
the whole huge page size.  The young walker gets an equivalent new
helper, damon_hugetlb_young(), which reads the huge PTE with
huge_ptep_get() and consults the page idle flag and
mmu_notifier_test_young() like the existing PTE branch.

Locking mirrors what page_vma_mapped_walk() provides: the huge PTE's
page table lock is held inside the walk, and for shared hugetlb
mappings (the only ones subject to huge PMD sharing), rmap_walk_file()
already holds i_mmap_rwsem, satisfying hugetlb_walk()'s locking
requirements.

This is currently dead code: both rmap walkers are only reachable
through damon_get_folio(), which rejects hugetlb folios since they are
not on the LRU lists.  A following commit will let the physical address
space monitoring primitives opt in to hugetlb folios.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
---
 mm/damon/ops-common.c | 63 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 52 insertions(+), 11 deletions(-)

diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index f5fe92b825bb..62004206ca31 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -193,10 +193,20 @@ static bool damon_folio_mkold_one(struct folio *folio,
 
 	while (page_vma_mapped_walk(&pvmw)) {
 		addr = pvmw.address;
-		if (pvmw.pte)
-			damon_ptep_mkold(pvmw.pte, vma, addr);
-		else
+		if (pvmw.pte) {
+			/*
+			 * For hugetlb folios, page_vma_mapped_walk() sets
+			 * pvmw.pte to the huge PTE with its page table lock
+			 * held.
+			 */
+			if (folio_test_hugetlb(folio))
+				damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma,
+						addr);
+			else
+				damon_ptep_mkold(pvmw.pte, vma, addr);
+		} else {
 			damon_pmdp_mkold(pvmw.pmd, vma, addr);
+		}
 	}
 	return true;
 }
@@ -221,6 +231,24 @@ void damon_folio_mkold(struct folio *folio)
 
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
+static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma,
+		unsigned long addr, struct folio *folio)
+{
+	pte_t entry = huge_ptep_get(vma->vm_mm, addr, pte);
+
+	return (pte_present(entry) && pte_young(entry)) ||
+		!folio_test_idle(folio) ||
+		mmu_notifier_test_young(vma->vm_mm, addr);
+}
+#else
+static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma,
+		unsigned long addr, struct folio *folio)
+{
+	return false;
+}
+#endif	/* CONFIG_HUGETLB_PAGE */
+
 static bool damon_folio_young_one(struct folio *folio,
 		struct vm_area_struct *vma, unsigned long addr, void *arg)
 {
@@ -232,16 +260,29 @@ static bool damon_folio_young_one(struct folio *folio,
 	while (page_vma_mapped_walk(&pvmw)) {
 		addr = pvmw.address;
 		if (pvmw.pte) {
-			pte = ptep_get(pvmw.pte);
-
 			/*
-			 * PFN swap PTEs, such as device-exclusive ones, that
-			 * actually map pages are "old" from a CPU perspective.
-			 * The MMU notifier takes care of any device aspects.
+			 * For hugetlb folios, page_vma_mapped_walk() sets
+			 * pvmw.pte to the huge PTE with its page table lock
+			 * held.
 			 */
-			*accessed = (pte_present(pte) && pte_young(pte)) ||
-				!folio_test_idle(folio) ||
-				mmu_notifier_test_young(vma->vm_mm, addr);
+			if (folio_test_hugetlb(folio)) {
+				*accessed = damon_hugetlb_young(pvmw.pte, vma,
+						addr, folio);
+			} else {
+				pte = ptep_get(pvmw.pte);
+
+				/*
+				 * PFN swap PTEs, such as device-exclusive
+				 * ones, that actually map pages are "old"
+				 * from a CPU perspective. The MMU notifier
+				 * takes care of any device aspects.
+				 */
+				*accessed = (pte_present(pte) &&
+						pte_young(pte)) ||
+					!folio_test_idle(folio) ||
+					mmu_notifier_test_young(vma->vm_mm,
+							addr);
+			}
 		} else {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 			pmd_t pmd = pmdp_get(pvmw.pmd);
-- 
2.54.0

[PATCH 3/6] mm/damon/paddr: support hugetlb folios in access monitoring

[Krishna Iyer]

DAMON's physical address space monitoring is blind to hugetlb-backed
memory.  Every access check starts at damon_get_folio(), which rejects
folios that are not on the LRU lists.  Hugetlb folios are managed
outside of the LRU by design, so every sampling attempt on
hugetlb-backed memory silently fails and the pages are reported as
never accessed.

This is a significant blind spot on virtualization hosts.  Cloud
hypervisor hosts commonly back guest memory with 1 GiB hugetlbfs pages,
covering the vast majority of the machine's memory.  On such hosts,
modules like DAMON_STAT observe only the host-side remainder (kernel,
page cache, daemons) and report all guest working sets as permanently
idle, defeating the purpose of host-level access monitoring.  In
testing on a 1 TiB host, an hour of 4-thread random access over 842 GiB
inside a guest was statistically indistinguishable from an idle host,
while a 40x smaller host-side workload produced a quantitatively
correct response.

Add damon_get_folio_incl_hugetlb(), which additionally accepts hugetlb
folios, and use it in the two paddr access monitoring primitives,
damon_pa_mkold() and damon_pa_young().  With the previous commit
teaching the folio-granular rmap walkers to age huge PTEs and to call
the mmu notifiers spanning the whole huge page, this makes guest
accesses visible through secondary MMU (e.g. KVM/EPT) young bits.

Free hugetlb pool folios have a zero refcount, so folio_try_get()
naturally keeps rejecting them.

The DAMOS action appliers (damon_pa_pageout(),
damon_pa_mark_accessed_or_deactivate(), damon_pa_migrate(),
damon_pa_stat()) keep using damon_get_folio(): reclaim, LRU
manipulation and migration cannot act on hugetlb folios, so their
behavior is unchanged.

Note that the access check granularity for hugetlb-backed memory is the
huge page size: one touched byte reports the whole (up to 1 GiB) page
as accessed.  Also, DAMON now consumes secondary MMU young bits that
KVM's own aging uses; at DAMON's sampling rate (one page per region per
sampling interval) the interference is negligible.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
---
 mm/damon/ops-common.c | 31 +++++++++++++++++++++++++++----
 mm/damon/ops-common.h |  1 +
 mm/damon/paddr.c      |  4 ++--
 3 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index 62004206ca31..ece101d34684 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -15,14 +15,20 @@
 #include "../internal.h"
 #include "ops-common.h"
 
+static bool damon_folio_observable(struct folio *folio, bool incl_hugetlb)
+{
+	return folio_test_lru(folio) ||
+		(incl_hugetlb && folio_test_hugetlb(folio));
+}
+
 /*
- * Get an online page for a pfn if it's in the LRU list.  Otherwise, returns
- * NULL.
+ * Get an online page for a pfn if it's in the LRU list, or a hugetlb folio if
+ * @incl_hugetlb is set.  Otherwise, returns NULL.
  *
  * The body of this function is stolen from the 'page_idle_get_folio()'.  We
  * steal rather than reuse it because the code is quite simple.
  */
-struct folio *damon_get_folio(unsigned long pfn)
+static struct folio *__damon_get_folio(unsigned long pfn, bool incl_hugetlb)
 {
 	struct page *page = pfn_to_online_page(pfn);
 	struct folio *folio;
@@ -33,13 +39,30 @@ struct folio *damon_get_folio(unsigned long pfn)
 	folio = page_folio(page);
 	if (!folio_try_get(folio))
 		return NULL;
-	if (unlikely(page_folio(page) != folio) || !folio_test_lru(folio)) {
+	if (unlikely(page_folio(page) != folio) ||
+			!damon_folio_observable(folio, incl_hugetlb)) {
 		folio_put(folio);
 		folio = NULL;
 	}
 	return folio;
 }
 
+struct folio *damon_get_folio(unsigned long pfn)
+{
+	return __damon_get_folio(pfn, false);
+}
+
+/*
+ * Same to damon_get_folio(), but also accepts hugetlb folios, which are
+ * managed outside of the LRU lists.  Aimed to be used by access monitoring
+ * primitives.  DAMOS actions that assume LRU-managed folios should keep using
+ * damon_get_folio().
+ */
+struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn)
+{
+	return __damon_get_folio(pfn, true);
+}
+
 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr)
 {
 	pte_t pteval = ptep_get(pte);
diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h
index f7811c9c7a02..68d7de49c87a 100644
--- a/mm/damon/ops-common.h
+++ b/mm/damon/ops-common.h
@@ -6,6 +6,7 @@
 #include <linux/damon.h>
 
 struct folio *damon_get_folio(unsigned long pfn);
+struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn);
 
 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr);
 void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr);
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 5c6c3a597fd0..09d418b2874b 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -37,7 +37,7 @@ static unsigned long damon_pa_core_addr(
 
 static void damon_pa_mkold(phys_addr_t paddr)
 {
-	struct folio *folio = damon_get_folio(PHYS_PFN(paddr));
+	struct folio *folio = damon_get_folio_incl_hugetlb(PHYS_PFN(paddr));
 
 	if (!folio)
 		return;
@@ -67,7 +67,7 @@ static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
 
 static bool damon_pa_young(phys_addr_t paddr)
 {
-	struct folio *folio = damon_get_folio(PHYS_PFN(paddr));
+	struct folio *folio = damon_get_folio_incl_hugetlb(PHYS_PFN(paddr));
 	bool accessed;
 
 	if (!folio)
-- 
2.54.0

[PATCH 4/6] mm/damon: support flush-assisted access bit clearing for monitoring

[Krishna Iyer]

DAMON's access sampling clears the accessed bit of the sampled page and
re-reads it one sampling interval later.  The clearing uses the
non-flushing primitives, deliberately trading accuracy for low
overhead: cached TLB translations survive the clearing, and accesses
through such cached translations do not perform page table walks, hence
do not set the accessed bit again.  The bit therefore re-arms at the
TLB-refill rate rather than the actual access rate.

For most setups this is a good trade-off, since TLB pressure re-walks
hot pages frequently anyway.  It breaks down when translations are
long-lived, most prominently on virtualization hosts backing guest
memory with 1 GiB hugetlb pages: covering hundreds of GiB takes only
hundreds of TLB entries, translations essentially never get evicted,
and secondary MMU (e.g. KVM/EPT) accessed bits cleared via
mmu_notifier_clear_young() are rarely set again.  Measurements on a
1 TiB host showed the reported access fraction pinned at the TLB-refill
rate: a guest workload continuously accessing 842 GiB was
indistinguishable from an idle guest.

Add an 'aging_flush' monitoring context option, default off.  When set,
the sampling primitives pair the accessed bit clearing with
mmu_notifier_clear_flush_young(), so secondary MMUs invalidate their
cached translations and the next guest access must re-walk and re-set
the accessed bit, making the sampled young state reflect the actual
access rate.

The flush covers secondary MMU TLBs only, uniformly across the PTE,
PMD and hugetlb paths.  Primary MMU TLB young-flushing is deliberately
not added: x86 already implements ptep_clear_flush_young() without a
flush on the grounds that primary-TLB accessed-bit staleness is not
worth the flush cost, and the observability target here is secondary
MMU (guest) access state.

The cost is secondary TLB invalidations during the sampling prepare
pass.  KVM coalesces flush requests issued in a burst (pending-request
deduping), so a prepare pass over all regions results in roughly one
guest TLB flush per pass, and hosts whose guests are backed by huge
pages have small TLB working sets that are cheap to re-fill -- exactly
the hosts that need this option.  Measured on a 1 TiB host with a
176-vCPU guest: ~240 flush calls/sec at auto-tuned intervals,
bounding worst-case aggregate overhead below 0.1%.

DAMOS filter paths keep the non-flushing behavior unconditionally.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
---
 include/linux/damon.h |  2 ++
 mm/damon/core.c       |  2 ++
 mm/damon/ops-common.c | 40 ++++++++++++++++++++++++++++------------
 mm/damon/ops-common.h | 12 +++++++-----
 mm/damon/paddr.c      |  7 ++++---
 mm/damon/vaddr.c      | 19 +++++++++++--------
 6 files changed, 54 insertions(+), 28 deletions(-)

diff --git a/include/linux/damon.h b/include/linux/damon.h
index 0c8b7ddef9ab..7fc438677dd8 100644
--- a/include/linux/damon.h
+++ b/include/linux/damon.h
@@ -848,6 +848,7 @@ struct damon_attrs {
  * including damon_call() and damos_walk().
  *
  * @addr_unit:	Scale factor for core to ops address conversion.
+ * @aging_flush:	Flush TLBs when clearing access bits for sampling.
  * @min_region_sz:	Minimum region size.
  * @pause:	Pause kdamond main loop.
  */
@@ -899,6 +900,7 @@ struct damon_ctx {
 
 /* public: */
 	unsigned long addr_unit;
+	bool aging_flush;
 	unsigned long min_region_sz;
 	bool pause;
 
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 644daf5a1656..46d6ec3dbffa 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -859,6 +859,7 @@ struct damon_ctx *damon_new_ctx(void)
 	INIT_LIST_HEAD(&ctx->probes);
 
 	ctx->addr_unit = 1;
+	ctx->aging_flush = false;
 	ctx->min_region_sz = DAMON_MIN_REGION_SZ;
 
 	INIT_LIST_HEAD(&ctx->adaptive_targets);
@@ -1800,6 +1801,7 @@ static int __damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
 	if (err)
 		return err;
 	dst->addr_unit = src->addr_unit;
+	dst->aging_flush = src->aging_flush;
 	dst->min_region_sz = src->min_region_sz;
 
 	dst->maybe_corrupted = false;
diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
index ece101d34684..5c2cd13b9b05 100644
--- a/mm/damon/ops-common.c
+++ b/mm/damon/ops-common.c
@@ -63,7 +63,8 @@ struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn)
 	return __damon_get_folio(pfn, true);
 }
 
-void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr)
+void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr,
+		bool flush)
 {
 	pte_t pteval = ptep_get(pte);
 	struct folio *folio;
@@ -86,7 +87,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr
 	 */
 	if (likely(pte_present(pteval)))
 		young |= ptep_test_and_clear_young(vma, addr, pte);
-	young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE);
+	if (flush)
+		young |= mmu_notifier_clear_flush_young(vma->vm_mm, addr,
+				addr + PAGE_SIZE);
+	else
+		young |= mmu_notifier_clear_young(vma->vm_mm, addr,
+				addr + PAGE_SIZE);
 	if (young)
 		folio_set_young(folio);
 
@@ -94,7 +100,8 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr
 	folio_put(folio);
 }
 
-void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr)
+void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr,
+		bool flush)
 {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	pmd_t pmdval = pmdp_get(pmd);
@@ -113,7 +120,12 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr
 
 	if (likely(pmd_present(pmdval)))
 		young |= pmdp_test_and_clear_young(vma, addr, pmd);
-	young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SIZE);
+	if (flush)
+		young |= mmu_notifier_clear_flush_young(vma->vm_mm, addr,
+				addr + HPAGE_PMD_SIZE);
+	else
+		young |= mmu_notifier_clear_young(vma->vm_mm, addr,
+				addr + HPAGE_PMD_SIZE);
 	if (young)
 		folio_set_young(folio);
 
@@ -124,7 +136,7 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr
 
 #ifdef CONFIG_HUGETLB_PAGE
 void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
-		struct vm_area_struct *vma, unsigned long addr)
+		struct vm_area_struct *vma, unsigned long addr, bool flush)
 {
 	bool referenced = false;
 	pte_t entry = huge_ptep_get(mm, addr, pte);
@@ -139,8 +151,10 @@ void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
 		set_huge_pte_at(mm, addr, pte, entry, psize);
 	}
 
-	if (mmu_notifier_clear_young(mm, addr,
-				     addr + huge_page_size(hstate_vma(vma))))
+	if (flush ? mmu_notifier_clear_flush_young(mm, addr,
+				addr + huge_page_size(hstate_vma(vma))) :
+			mmu_notifier_clear_young(mm, addr,
+				addr + huge_page_size(hstate_vma(vma))))
 		referenced = true;
 
 	if (referenced)
@@ -212,6 +226,7 @@ int damon_cold_score(struct damon_ctx *c, struct damon_region *r,
 static bool damon_folio_mkold_one(struct folio *folio,
 		struct vm_area_struct *vma, unsigned long addr, void *arg)
 {
+	bool flush = *(bool *)arg;
 	DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, 0);
 
 	while (page_vma_mapped_walk(&pvmw)) {
@@ -224,19 +239,20 @@ static bool damon_folio_mkold_one(struct folio *folio,
 			 */
 			if (folio_test_hugetlb(folio))
 				damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma,
-						addr);
+						addr, flush);
 			else
-				damon_ptep_mkold(pvmw.pte, vma, addr);
+				damon_ptep_mkold(pvmw.pte, vma, addr, flush);
 		} else {
-			damon_pmdp_mkold(pvmw.pmd, vma, addr);
+			damon_pmdp_mkold(pvmw.pmd, vma, addr, flush);
 		}
 	}
 	return true;
 }
 
-void damon_folio_mkold(struct folio *folio)
+void damon_folio_mkold(struct folio *folio, bool flush)
 {
 	struct rmap_walk_control rwc = {
+		.arg = &flush,
 		.rmap_one = damon_folio_mkold_one,
 		.anon_lock = folio_lock_anon_vma_read,
 	};
@@ -377,7 +393,7 @@ bool damos_folio_filter_match(struct damos_filter *filter, struct folio *folio)
 	case DAMOS_FILTER_TYPE_YOUNG:
 		matched = damon_folio_young(folio);
 		if (matched)
-			damon_folio_mkold(folio);
+			damon_folio_mkold(folio, false);
 		break;
 	case DAMOS_FILTER_TYPE_HUGEPAGE_SIZE:
 		folio_sz = folio_size(folio);
diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h
index 68d7de49c87a..df7ef025bc1a 100644
--- a/mm/damon/ops-common.h
+++ b/mm/damon/ops-common.h
@@ -8,18 +8,20 @@
 struct folio *damon_get_folio(unsigned long pfn);
 struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn);
 
-void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr);
-void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr);
+void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr,
+		bool flush);
+void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr,
+		bool flush);
 #ifdef CONFIG_HUGETLB_PAGE
 void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
-		struct vm_area_struct *vma, unsigned long addr);
+		struct vm_area_struct *vma, unsigned long addr, bool flush);
 #else
 static inline void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
-		struct vm_area_struct *vma, unsigned long addr)
+		struct vm_area_struct *vma, unsigned long addr, bool flush)
 {
 }
 #endif	/* CONFIG_HUGETLB_PAGE */
-void damon_folio_mkold(struct folio *folio);
+void damon_folio_mkold(struct folio *folio, bool flush);
 bool damon_folio_young(struct folio *folio);
 
 int damon_cold_score(struct damon_ctx *c, struct damon_region *r,
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 09d418b2874b..f2aaa82325d1 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -35,14 +35,14 @@ static unsigned long damon_pa_core_addr(
 	return pa / addr_unit;
 }
 
-static void damon_pa_mkold(phys_addr_t paddr)
+static void damon_pa_mkold(phys_addr_t paddr, bool flush)
 {
 	struct folio *folio = damon_get_folio_incl_hugetlb(PHYS_PFN(paddr));
 
 	if (!folio)
 		return;
 
-	damon_folio_mkold(folio);
+	damon_folio_mkold(folio, flush);
 	folio_put(folio);
 }
 
@@ -51,7 +51,8 @@ static void __damon_pa_prepare_access_check(struct damon_region *r,
 {
 	r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
 
-	damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit));
+	damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit),
+			ctx->aging_flush);
 }
 
 static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index 15379d984689..7c11022bb664 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -276,7 +276,8 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
 		pmd_t pmde = pmdp_get(pmd);
 
 		if (pmd_present(pmde))
-			damon_pmdp_mkold(pmd, walk->vma, addr);
+			damon_pmdp_mkold(pmd, walk->vma, addr,
+					*(bool *)walk->private);
 		spin_unlock(ptl);
 		return 0;
 	}
@@ -286,7 +287,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
 		return 0;
 	if (!pte_present(ptep_get(pte)))
 		goto out;
-	damon_ptep_mkold(pte, walk->vma, addr);
+	damon_ptep_mkold(pte, walk->vma, addr, *(bool *)walk->private);
 out:
 	pte_unmap_unlock(pte, ptl);
 	return 0;
@@ -306,7 +307,8 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask,
 	if (!pte_present(entry))
 		goto out;
 
-	damon_hugetlb_mkold(pte, walk->mm, walk->vma, addr);
+	damon_hugetlb_mkold(pte, walk->mm, walk->vma, addr,
+			    *(bool *)walk->private);
 
 out:
 	spin_unlock(ptl);
@@ -316,14 +318,15 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask,
 #define damon_mkold_hugetlb_entry NULL
 #endif /* CONFIG_HUGETLB_PAGE */
 
-static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)
+static void damon_va_mkold(struct mm_struct *mm, unsigned long addr,
+		bool flush)
 {
 	struct mm_walk_ops damon_mkold_ops = {
 		.pmd_entry = damon_mkold_pmd_entry,
 		.hugetlb_entry = damon_mkold_hugetlb_entry,
 	};
 
-	damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);
+	damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, &flush);
 }
 
 /*
@@ -336,7 +339,7 @@ static void __damon_va_prepare_access_check(struct mm_struct *mm,
 {
 	r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
 
-	damon_va_mkold(mm, r->sampling_addr);
+	damon_va_mkold(mm, r->sampling_addr, ctx->aging_flush);
 }
 
 static void damon_va_prepare_access_checks(struct damon_ctx *ctx)
@@ -508,9 +511,9 @@ static bool damos_va_filter_young_match(struct damos_filter *filter,
 		mmu_notifier_test_young(vma->vm_mm, addr);
 
 	if (young && ptep)
-		damon_ptep_mkold(ptep, vma, addr);
+		damon_ptep_mkold(ptep, vma, addr, false);
 	else if (young && pmdp)
-		damon_pmdp_mkold(pmdp, vma, addr);
+		damon_pmdp_mkold(pmdp, vma, addr, false);
 
 	return young == filter->matching;
 }
-- 
2.54.0

[PATCH 5/6] mm/damon/sysfs: support aging_flush

[Krishna Iyer]

Expose the aging_flush monitoring context option via a DAMON sysfs
context file of the same name, next to addr_unit.  Accepts boolean
strings (kstrtobool); defaults to N (current behavior).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
---
 mm/damon/sysfs.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c
index e3858ffab4b2..c8bc9cfbe4cf 100644
--- a/mm/damon/sysfs.c
+++ b/mm/damon/sysfs.c
@@ -1403,6 +1403,7 @@ struct damon_sysfs_context {
 	struct kobject kobj;
 	enum damon_ops_id ops_id;
 	unsigned long addr_unit;
+	bool aging_flush;
 	struct damon_sysfs_attrs *attrs;
 	struct damon_sysfs_targets *targets;
 	struct damon_sysfs_schemes *schemes;
@@ -1618,6 +1619,29 @@ static ssize_t pause_store(struct kobject *kobj, struct kobj_attribute *attr,
 	return count;
 }
 
+static ssize_t aging_flush_show(struct kobject *kobj,
+		struct kobj_attribute *attr, char *buf)
+{
+	struct damon_sysfs_context *context = container_of(kobj,
+			struct damon_sysfs_context, kobj);
+
+	return sysfs_emit(buf, "%c\n", context->aging_flush ? 'Y' : 'N');
+}
+
+static ssize_t aging_flush_store(struct kobject *kobj,
+		struct kobj_attribute *attr, const char *buf, size_t count)
+{
+	struct damon_sysfs_context *context = container_of(kobj,
+			struct damon_sysfs_context, kobj);
+	bool input_aging_flush;
+	int err = kstrtobool(buf, &input_aging_flush);
+
+	if (err)
+		return err;
+
+	context->aging_flush = input_aging_flush;
+	return count;
+}
 
 static void damon_sysfs_context_release(struct kobject *kobj)
 {
@@ -1636,11 +1660,15 @@ static struct kobj_attribute damon_sysfs_context_addr_unit_attr =
 static struct kobj_attribute damon_sysfs_context_pause_attr =
 		__ATTR_RW_MODE(pause, 0600);
 
+static struct kobj_attribute damon_sysfs_context_aging_flush_attr =
+		__ATTR_RW_MODE(aging_flush, 0600);
+
 static struct attribute *damon_sysfs_context_attrs[] = {
 	&damon_sysfs_context_avail_operations_attr.attr,
 	&damon_sysfs_context_operations_attr.attr,
 	&damon_sysfs_context_addr_unit_attr.attr,
 	&damon_sysfs_context_pause_attr.attr,
+	&damon_sysfs_context_aging_flush_attr.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(damon_sysfs_context);
@@ -2102,6 +2130,7 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx,
 	if (err)
 		return err;
 	ctx->addr_unit = READ_ONCE(sys_ctx->addr_unit);
+	ctx->aging_flush = READ_ONCE(sys_ctx->aging_flush);
 	/* addr_unit is respected by only DAMON_OPS_PADDR */
 	if (ops_id == DAMON_OPS_PADDR)
 		ctx->min_region_sz = max(
-- 
2.54.0

[PATCH 6/6] mm/damon/stat: support aging_flush

[Krishna Iyer]

Let DAMON_STAT users opt in to flush-assisted access bit clearing via a
module parameter of the same name as the monitoring context option.
The parameter is applied when the monitoring context is built, i.e., at
enable time.  Defaults to off (current behavior).

Intended for virtualization hosts backing guest memory with hugetlb
pages, where non-flushing aging pins the observed access rate to the
TLB-refill rate (see the aging_flush core patch for details).  Such
deployments can set damon_stat.aging_flush=1 on the kernel command
line.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
---
 mm/damon/stat.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/damon/stat.c b/mm/damon/stat.c
index b05b68f73e10..e093383ba406 100644
--- a/mm/damon/stat.c
+++ b/mm/damon/stat.c
@@ -47,6 +47,11 @@ module_param(aggr_interval_us, ulong, 0400);
 MODULE_PARM_DESC(aggr_interval_us,
 		"Current tuned aggregation interval in microseconds");
 
+static bool aging_flush __read_mostly;
+module_param(aging_flush, bool, 0600);
+MODULE_PARM_DESC(aging_flush,
+		"Flush TLBs when clearing access bits (applied at enable time)");
+
 static struct damon_ctx *damon_stat_context;
 
 static unsigned long damon_stat_last_refresh_jiffies;
@@ -178,6 +183,7 @@ static struct damon_ctx *damon_stat_build_ctx(void)
 
 	if (damon_select_ops(ctx, DAMON_OPS_PADDR))
 		goto free_out;
+	ctx->aging_flush = aging_flush;
 
 	target = damon_new_target();
 	if (!target)
-- 
2.54.0

Re: [PATCH 6/6] mm/damon/stat: support aging_flush

[sashiko-bot]

> Let DAMON_STAT users opt in to flush-assisted access bit clearing via a
> module parameter of the same name as the monitoring context option.
> The parameter is applied when the monitoring context is built, i.e., at
> enable time.  Defaults to off (current behavior).
> 
> Intended for virtualization hosts backing guest memory with hugetlb
> pages, where non-flushing aging pins the observed access rate to the
> TLB-refill rate (see the aging_flush core patch for details).  Such
> deployments can set damon_stat.aging_flush=1 on the kernel command
> line.
> 
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=6

Re: [PATCH 5/6] mm/damon/sysfs: support aging_flush

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly added `aging_flush` field in `struct damon_sysfs_context` is not initialized during allocation, leading to undefined default behavior and a minor information leak.
--

commit d37fc04bb82ae96572d3c46e4e494980a145a1c1
Author: Krishna Iyer <kiyer@crusoe.ai>

mm/damon/sysfs: support aging_flush

This commit exposes the aging_flush monitoring context option via a new DAMON
sysfs context file. It accepts boolean strings to configure the value, which
the commit message states defaults to N.

> diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c
> index e3858ffab4b22..c8bc9cfbe4cf5 100644
> --- a/mm/damon/sysfs.c
> +++ b/mm/damon/sysfs.c
> @@ -1403,6 +1403,7 @@ struct damon_sysfs_context {
>  	struct kobject kobj;
>  	enum damon_ops_id ops_id;
>  	unsigned long addr_unit;
> +	bool aging_flush;

[Severity: Medium]
Since this structure is allocated using kmalloc_obj() in
damon_sysfs_context_alloc(), does this leave the new aging_flush field
uninitialized?

The commit message mentions it "defaults to N", but without an explicit
initialization in damon_sysfs_context_alloc(), it appears it could inherit a
random value from the heap:

mm/damon/sysfs.c:damon_sysfs_context_alloc() {
    ...
	struct damon_sysfs_context *context = kmalloc_obj(*context);
    ...
	context->pause = false;
	return context;
}

>  	struct damon_sysfs_attrs *attrs;
>  	struct damon_sysfs_targets *targets;
>  	struct damon_sysfs_schemes *schemes;

[ ... ]

> @@ -1618,6 +1619,29 @@ static ssize_t pause_store(struct kobject *kobj, struct kobj_attribute *attr,
>  	return count;
>  }
>  
> +static ssize_t aging_flush_show(struct kobject *kobj,
> +		struct kobj_attribute *attr, char *buf)
> +{
> +	struct damon_sysfs_context *context = container_of(kobj,
> +			struct damon_sysfs_context, kobj);
> +
> +	return sysfs_emit(buf, "%c\n", context->aging_flush ? 'Y' : 'N');

[Severity: Medium]
If aging_flush is left uninitialized during allocation, could reading it here
expose uninitialized kernel memory to user-space?

> +}
> +
> +static ssize_t aging_flush_store(struct kobject *kobj,

[ ... ]

> @@ -2102,6 +2130,7 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx,
>  	if (err)
>  		return err;
>  	ctx->addr_unit = READ_ONCE(sys_ctx->addr_unit);
> +	ctx->aging_flush = READ_ONCE(sys_ctx->aging_flush);

[Severity: Medium]
Could reading an uninitialized sys_ctx->aging_flush here cause unpredictable
behavior by randomly applying the uninitialized value to the core DAMON
context?

>  	/* addr_unit is respected by only DAMON_OPS_PADDR */
>  	if (ops_id == DAMON_OPS_PADDR)
>  		ctx->min_region_sz = max(

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=5

Re: [PATCH 4/6] mm/damon: support flush-assisted access bit clearing for monitoring

[sashiko-bot]

> DAMON's access sampling clears the accessed bit of the sampled page and
> re-reads it one sampling interval later.  The clearing uses the
> non-flushing primitives, deliberately trading accuracy for low
> overhead: cached TLB translations survive the clearing, and accesses
> through such cached translations do not perform page table walks, hence
> do not set the accessed bit again.  The bit therefore re-arms at the
> TLB-refill rate rather than the actual access rate.
> 
> For most setups this is a good trade-off, since TLB pressure re-walks
> hot pages frequently anyway.  It breaks down when translations are
> long-lived, most prominently on virtualization hosts backing guest
> memory with 1 GiB hugetlb pages: covering hundreds of GiB takes only
> hundreds of TLB entries, translations essentially never get evicted,
> and secondary MMU (e.g. KVM/EPT) accessed bits cleared via
> mmu_notifier_clear_young() are rarely set again.  Measurements on a
> [ ... ]
> 
> DAMOS filter paths keep the non-flushing behavior unconditionally.
> 
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=4

Re: [PATCH 1/6] mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Non-atomic update of HugeTLB PTE in damon_hugetlb_mkold() overwrites hardware-set dirty bits and triggers unintended BBM sequences.
--

commit a960b7ee57eac1d4fccc44af4918b4b8331a38a3
Author: Krishna Iyer <kiyer@crusoe.ai>

    mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common
    
    damon_hugetlb_mkold() clears the accessed bit of a hugetlb-mapping huge
    PTE and propagates the aging to secondary MMUs via
    mmu_notifier_clear_young(), spanning the whole huge page size.  It
    currently lives in vaddr.c, and is thus usable only by the virtual
    address space monitoring operations set.

> diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> index fbda70d8ea4d0..f5fe92b825bbc 100644
> --- a/mm/damon/ops-common.c
> +++ b/mm/damon/ops-common.c

[ ... ]

> @@ -98,6 +99,35 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr
>  #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
>  }
>  
> +#ifdef CONFIG_HUGETLB_PAGE
> +void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
> +		struct vm_area_struct *vma, unsigned long addr)
> +{
> +	bool referenced = false;
> +	pte_t entry = huge_ptep_get(mm, addr, pte);
> +	struct folio *folio = pfn_folio(pte_pfn(entry));
> +	unsigned long psize = huge_page_size(hstate_vma(vma));
> +
> +	folio_get(folio);
> +
> +	if (pte_young(entry)) {
> +		referenced = true;
> +		entry = pte_mkold(entry);
> +		set_huge_pte_at(mm, addr, pte, entry, psize);

[Severity: Critical]
This is a pre-existing issue, but does this non-atomic update of the hugetlb
PTE overwrite hardware-set dirty bits?

The hardware MMU can set the dirty bit (on x86, or arm64 with hardware managed
dirty bits) between the huge_ptep_get() read and this set_huge_pte_at()
write. This non-atomic read-modify-write drops the hardware dirty bit, leading
to data loss upon reclaim or sync.

Additionally, on arm64, using set_huge_pte_at() on an already valid contiguous
PTE triggers an unintended break-before-make sequence with a broadcast TLB
flush via clear_flush(), which causes severe performance regressions.

> +	}
> +
> +	if (mmu_notifier_clear_young(mm, addr,
> +				     addr + huge_page_size(hstate_vma(vma))))
> +		referenced = true;
> +
> +	if (referenced)
> +		folio_set_young(folio);
> +
> +	folio_set_idle(folio);
> +	folio_put(folio);
> +}
> +#endif	/* CONFIG_HUGETLB_PAGE */

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=1

Re: [PATCH 2/6] mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Medium] mmu_notifier_test_young() evaluates only a single base page, masking accesses within shattered huge pages.

Pre-existing issues:
- [High] Non-atomic read-modify-write of huge PTE in damon_hugetlb_mkold() silently drops hardware-managed dirty bits.
--

commit 69bb3f48e79ddddcf3eaf4001d7704e62df0da69
Author: Krishna Iyer <kiyer@crusoe.ai>

mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers

This commit introduces hugetlb branches to DAMON's rmap walkers by reusing
damon_hugetlb_mkold() for clearing the young bit and introducing
damon_hugetlb_young() for checking it. It ensures huge PTEs are read and
aged correctly rather than using base page granularity helpers.

> diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> index f5fe92b825bbc..62004206ca310 100644
> --- a/mm/damon/ops-common.c
> +++ b/mm/damon/ops-common.c
> @@ -193,10 +193,20 @@ static bool damon_folio_mkold_one(struct folio *folio,
>  
>  	while (page_vma_mapped_walk(&pvmw)) {
>  		addr = pvmw.address;
> -		if (pvmw.pte)
> -			damon_ptep_mkold(pvmw.pte, vma, addr);
> -		else
> +		if (pvmw.pte) {
> +			/*
> +			 * For hugetlb folios, page_vma_mapped_walk() sets
> +			 * pvmw.pte to the huge PTE with its page table lock
> +			 * held.
> +			 */
> +			if (folio_test_hugetlb(folio))
> +				damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma,
> +						addr);

[Severity: High]
This is a pre-existing issue, but does damon_hugetlb_mkold() perform a
non-atomic read-modify-write that could silently drop hardware-managed
dirty bits?

Looking at damon_hugetlb_mkold(), the code reads the PTE, clears the young
bit, and writes it back:

	pte_t entry = huge_ptep_get(mm, addr, pte);
...
	if (pte_young(entry)) {
		referenced = true;
		entry = pte_mkold(entry);
		set_huge_pte_at(mm, addr, pte, entry, psize);
	}

If a hardware write access occurs and sets the hardware A/D bit after
huge_ptep_get() but before set_huge_pte_at(), won't the dirty bit be
overwritten with the stale, clean value?

This could potentially lead to data loss if hugetlb pages are reclaimed
or migrated without writeback. Hooking this up for DAMON folio monitoring
means this path can be triggered frequently in the background.

> +			else
> +				damon_ptep_mkold(pvmw.pte, vma, addr);
> +		} else {
>  			damon_pmdp_mkold(pvmw.pmd, vma, addr);
> +		}
>  	}

[ ... ]

> +#ifdef CONFIG_HUGETLB_PAGE
> +static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma,
> +		unsigned long addr, struct folio *folio)
> +{
> +	pte_t entry = huge_ptep_get(vma->vm_mm, addr, pte);
> +
> +	return (pte_present(entry) && pte_young(entry)) ||
> +		!folio_test_idle(folio) ||
> +		mmu_notifier_test_young(vma->vm_mm, addr);

[Severity: Medium]
Will mmu_notifier_test_young() miss accesses within shattered huge pages here?

Since the kernel's mmu_notifier_test_young() API does not take a size
parameter, passing only the base address of the huge page means it will
only evaluate the first base page.

For secondary MMUs like KVM EPT that map huge pages as multiple smaller
pages, wouldn't accesses to other subpages within the huge page be completely
missed? This could cause DAMON to classify the huge page as colder than it
actually is, which might lead to inappropriate reclaim or migration decisions.

> +}
> +#else

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=2

Re: [PATCH 3/6] mm/damon/paddr: support hugetlb folios in access monitoring

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Non-atomic huge PTE modification in damon_hugetlb_mkold() causes a TOCTOU race with hardware page table walkers, silently dropping dirty bits.
- [High] Incorrect physical address advancement when starting from a tail page of a large folio.
--

commit fa7cc985c2891991281afce141ed40264af354fe
Author: Krishna Iyer <kiyer@crusoe.ai>

    mm/damon/paddr: support hugetlb folios in access monitoring
    
    DAMON's physical address space monitoring is blind to hugetlb-backed
    memory. Every access check starts at damon_get_folio(), which rejects
    folios that are not on the LRU lists. This commit adds
    damon_get_folio_incl_hugetlb() to additionally accept hugetlb folios.

[ ... ]

> diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> index 62004206ca310..ece101d346844 100644
> --- a/mm/damon/ops-common.c
> +++ b/mm/damon/ops-common.c

[ ... ]

[Severity: Critical]
This is a pre-existing issue, but does this non-atomic huge PTE modification
drop hardware dirty bits?

In damon_hugetlb_mkold() in mm/damon/ops-common.c:

	pte_t entry = huge_ptep_get(mm, addr, pte);
	struct folio *folio = pfn_folio(pte_pfn(entry));
	unsigned long psize = huge_page_size(hstate_vma(vma));

	folio_get(folio);

	if (pte_young(entry)) {
		referenced = true;
		entry = pte_mkold(entry);
		set_huge_pte_at(mm, addr, pte, entry, psize);
	}

If a hardware page table walker sets the dirty bit concurrently between the
read and the write, does the non-atomic writeback via set_huge_pte_at()
silently overwrite and clear the dirty bit?

[Severity: High]
This isn't a bug introduced by this patch, but does this physical address
advancement skip memory pages?

In damon_pa_pageout() in mm/damon/paddr.c:

	addr = damon_pa_phys_addr(r->ar.start, addr_unit);
	while (addr < damon_pa_phys_addr(r->ar.end, addr_unit)) {
		folio = damon_get_folio(PHYS_PFN(addr));
...
put_folio:
		addr += folio_size(folio);
		folio_put(folio);
	}

If the region's start address is not naturally aligned to the folio boundary
and lands on a tail page of a large folio, does adding the full folio size
overshoot the start of the next folio?

The same pattern appears in damon_pa_stat() where it also over-accounts the
filter size:

		if (damos_pa_filter_out(s, folio))
			goto put_folio;
		else
			*sz_filter_passed += folio_size(folio) / addr_unit;
...
put_folio:
		addr += folio_size(folio);
		folio_put(folio);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=3

Re: [PATCH 2/6] mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers

[SJ Park]

On Sun, 30 Aug 2026 05:28:55 +0000 sashiko-bot@kernel.org wrote:

> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> 
> New issues:
> - [Medium] mmu_notifier_test_young() evaluates only a single base page, masking accesses within shattered huge pages.
> 
> Pre-existing issues:
> - [High] Non-atomic read-modify-write of huge PTE in damon_hugetlb_mkold() silently drops hardware-managed dirty bits.
> --
> 
> commit 69bb3f48e79ddddcf3eaf4001d7704e62df0da69
> Author: Krishna Iyer <kiyer@crusoe.ai>
> 
> mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers
> 
> This commit introduces hugetlb branches to DAMON's rmap walkers by reusing
> damon_hugetlb_mkold() for clearing the young bit and introducing
> damon_hugetlb_young() for checking it. It ensures huge PTEs are read and
> aged correctly rather than using base page granularity helpers.
> 
> > diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> > index f5fe92b825bbc..62004206ca310 100644
> > --- a/mm/damon/ops-common.c
> > +++ b/mm/damon/ops-common.c
> > @@ -193,10 +193,20 @@ static bool damon_folio_mkold_one(struct folio *folio,
> >  
> >  	while (page_vma_mapped_walk(&pvmw)) {
> >  		addr = pvmw.address;
> > -		if (pvmw.pte)
> > -			damon_ptep_mkold(pvmw.pte, vma, addr);
> > -		else
> > +		if (pvmw.pte) {
> > +			/*
> > +			 * For hugetlb folios, page_vma_mapped_walk() sets
> > +			 * pvmw.pte to the huge PTE with its page table lock
> > +			 * held.
> > +			 */
> > +			if (folio_test_hugetlb(folio))
> > +				damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma,
> > +						addr);
> 
> [Severity: High]
> This is a pre-existing issue, but does damon_hugetlb_mkold() perform a
> non-atomic read-modify-write that could silently drop hardware-managed
> dirty bits?
> 
> Looking at damon_hugetlb_mkold(), the code reads the PTE, clears the young
> bit, and writes it back:
> 
> 	pte_t entry = huge_ptep_get(mm, addr, pte);
> ...
> 	if (pte_young(entry)) {
> 		referenced = true;
> 		entry = pte_mkold(entry);
> 		set_huge_pte_at(mm, addr, pte, entry, psize);
> 	}
> 
> If a hardware write access occurs and sets the hardware A/D bit after
> huge_ptep_get() but before set_huge_pte_at(), won't the dirty bit be
> overwritten with the stale, clean value?
> 
> This could potentially lead to data loss if hugetlb pages are reclaimed
> or migrated without writeback. Hooking this up for DAMON folio monitoring
> means this path can be triggered frequently in the background.

I'm planning to fix this.

> 
> > +			else
> > +				damon_ptep_mkold(pvmw.pte, vma, addr);
> > +		} else {
> >  			damon_pmdp_mkold(pvmw.pmd, vma, addr);
> > +		}
> >  	}
> 
> [ ... ]
> 
> > +#ifdef CONFIG_HUGETLB_PAGE
> > +static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma,
> > +		unsigned long addr, struct folio *folio)
> > +{
> > +	pte_t entry = huge_ptep_get(vma->vm_mm, addr, pte);
> > +
> > +	return (pte_present(entry) && pte_young(entry)) ||
> > +		!folio_test_idle(folio) ||
> > +		mmu_notifier_test_young(vma->vm_mm, addr);
> 
> [Severity: Medium]
> Will mmu_notifier_test_young() miss accesses within shattered huge pages here?
> 
> Since the kernel's mmu_notifier_test_young() API does not take a size
> parameter, passing only the base address of the huge page means it will
> only evaluate the first base page.
> 
> For secondary MMUs like KVM EPT that map huge pages as multiple smaller
> pages, wouldn't accesses to other subpages within the huge page be completely
> missed? This could cause DAMON to classify the huge page as colder than it
> actually is, which might lead to inappropriate reclaim or migration decisions.

This is also a pre-existing issue.  I will work on fixing this.


Thanks,
SJ

[...]

Re: [PATCH 3/6] mm/damon/paddr: support hugetlb folios in access monitoring

[SJ Park]

On Sun, 30 Aug 2026 05:29:35 +0000 sashiko-bot@kernel.org wrote:

> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> 
> Pre-existing issues:
> - [Critical] Non-atomic huge PTE modification in damon_hugetlb_mkold() causes a TOCTOU race with hardware page table walkers, silently dropping dirty bits.
> - [High] Incorrect physical address advancement when starting from a tail page of a large folio.
> --
> 
> commit fa7cc985c2891991281afce141ed40264af354fe
> Author: Krishna Iyer <kiyer@crusoe.ai>
> 
>     mm/damon/paddr: support hugetlb folios in access monitoring
>     
>     DAMON's physical address space monitoring is blind to hugetlb-backed
>     memory. Every access check starts at damon_get_folio(), which rejects
>     folios that are not on the LRU lists. This commit adds
>     damon_get_folio_incl_hugetlb() to additionally accept hugetlb folios.
> 
> [ ... ]
> 
> > diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> > index 62004206ca310..ece101d346844 100644
> > --- a/mm/damon/ops-common.c
> > +++ b/mm/damon/ops-common.c
> 
> [ ... ]
> 
> [Severity: Critical]
> This is a pre-existing issue, but does this non-atomic huge PTE modification
> drop hardware dirty bits?
> 
> In damon_hugetlb_mkold() in mm/damon/ops-common.c:
> 
> 	pte_t entry = huge_ptep_get(mm, addr, pte);
> 	struct folio *folio = pfn_folio(pte_pfn(entry));
> 	unsigned long psize = huge_page_size(hstate_vma(vma));
> 
> 	folio_get(folio);
> 
> 	if (pte_young(entry)) {
> 		referenced = true;
> 		entry = pte_mkold(entry);
> 		set_huge_pte_at(mm, addr, pte, entry, psize);
> 	}
> 
> If a hardware page table walker sets the dirty bit concurrently between the
> read and the write, does the non-atomic writeback via set_huge_pte_at()
> silently overwrite and clear the dirty bit?

I'm planning to work on fixing this.

> 
> [Severity: High]
> This isn't a bug introduced by this patch, but does this physical address
> advancement skip memory pages?
> 
> In damon_pa_pageout() in mm/damon/paddr.c:
> 
> 	addr = damon_pa_phys_addr(r->ar.start, addr_unit);
> 	while (addr < damon_pa_phys_addr(r->ar.end, addr_unit)) {
> 		folio = damon_get_folio(PHYS_PFN(addr));
> ...
> put_folio:
> 		addr += folio_size(folio);
> 		folio_put(folio);
> 	}
> 
> If the region's start address is not naturally aligned to the folio boundary
> and lands on a tail page of a large folio, does adding the full folio size
> overshoot the start of the next folio?
> 
> The same pattern appears in damon_pa_stat() where it also over-accounts the
> filter size:
> 
> 		if (damos_pa_filter_out(s, folio))
> 			goto put_folio;
> 		else
> 			*sz_filter_passed += folio_size(folio) / addr_unit;
> ...
> put_folio:
> 		addr += folio_size(folio);
> 		folio_put(folio);

Known issue.  Fix is cooking:
https://lore.kernel.org/20260802162631.90304-1-sj@kernel.org

> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20260830051407.50008-1-kiyer@crusoe.ai?part=3


Thanks,
SJ

Re: [PATCH 1/6] mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common

[SJ Park]

On Sat, 29 Aug 2026 22:14:02 -0700 Krishna Iyer <kiyer@crusoe.ai> wrote:

> damon_hugetlb_mkold() clears the accessed bit of a hugetlb-mapping huge
> PTE and propagates the aging to secondary MMUs via
> mmu_notifier_clear_young(), spanning the whole huge page size.  It
> currently lives in vaddr.c, and is thus usable only by the virtual
> address space monitoring operations set.
> 
> The physical address space monitoring operations set will need the same
> logic, to support access monitoring of hugetlb-backed memory.  Move the
> function to ops-common as-is, with no behavioral change.  A follow-up
> change will use it from the folio-granular rmap walkers.

Looks good to me, thank you for this patch!

> 
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>

Reviewed-by: SJ Park <sj@kernel.org>


Thanks,
SJ

[...]

Re: [PATCH 2/6] mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers

[SJ Park]

On Sat, 29 Aug 2026 22:14:03 -0700 Krishna Iyer <kiyer@crusoe.ai> wrote:

> damon_folio_mkold_one() and damon_folio_young_one() assume the folios
> they walk are mapped by normal PTEs or THP PMDs.  When the folio is a
> hugetlb folio, page_vma_mapped_walk() returns the huge PTE in pvmw.pte
> with its page table lock held, but the walkers treat it as a normal
> PTE: they read and age it with PAGE_SIZE-granularity helpers, which is
> wrong for huge PTEs (up to PUD level), and notify secondary MMUs for
> only PAGE_SIZE of the mapping.
> 
> Add hugetlb branches to both walkers.  The mkold walker reuses
> damon_hugetlb_mkold(), which the virtual address space operations set
> has been using for hugetlb aging: it clears the young bit of the huge
> PTE via set_huge_pte_at() and calls mmu_notifier_clear_young() spanning
> the whole huge page size.  The young walker gets an equivalent new
> helper, damon_hugetlb_young(), which reads the huge PTE with
> huge_ptep_get() and consults the page idle flag and
> mmu_notifier_test_young() like the existing PTE branch.
> 
> Locking mirrors what page_vma_mapped_walk() provides: the huge PTE's
> page table lock is held inside the walk, and for shared hugetlb
> mappings (the only ones subject to huge PMD sharing), rmap_walk_file()
> already holds i_mmap_rwsem, satisfying hugetlb_walk()'s locking
> requirements.
> 
> This is currently dead code: both rmap walkers are only reachable
> through damon_get_folio(), which rejects hugetlb folios since they are
> not on the LRU lists.  A following commit will let the physical address
> space monitoring primitives opt in to hugetlb folios.
> 
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
> ---
>  mm/damon/ops-common.c | 63 +++++++++++++++++++++++++++++++++++--------
>  1 file changed, 52 insertions(+), 11 deletions(-)
> 
> diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> index f5fe92b825bb..62004206ca31 100644
> --- a/mm/damon/ops-common.c
> +++ b/mm/damon/ops-common.c
> @@ -193,10 +193,20 @@ static bool damon_folio_mkold_one(struct folio *folio,
>  
>  	while (page_vma_mapped_walk(&pvmw)) {
>  		addr = pvmw.address;
> -		if (pvmw.pte)
> -			damon_ptep_mkold(pvmw.pte, vma, addr);
> -		else
> +		if (pvmw.pte) {
> +			/*
> +			 * For hugetlb folios, page_vma_mapped_walk() sets
> +			 * pvmw.pte to the huge PTE with its page table lock
> +			 * held.
> +			 */

This comment looks too verbose.  Let's drop.

> +			if (folio_test_hugetlb(folio))
> +				damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma,
> +						addr);
> +			else
> +				damon_ptep_mkold(pvmw.pte, vma, addr);
> +		} else {
>  			damon_pmdp_mkold(pvmw.pmd, vma, addr);
> +		}
>  	}
>  	return true;
>  }
> @@ -221,6 +231,24 @@ void damon_folio_mkold(struct folio *folio)
>  
>  }
>  
> +#ifdef CONFIG_HUGETLB_PAGE
> +static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma,
> +		unsigned long addr, struct folio *folio)
> +{
> +	pte_t entry = huge_ptep_get(vma->vm_mm, addr, pte);
> +
> +	return (pte_present(entry) && pte_young(entry)) ||
> +		!folio_test_idle(folio) ||
> +		mmu_notifier_test_young(vma->vm_mm, addr);
> +}
> +#else
> +static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma,
> +		unsigned long addr, struct folio *folio)
> +{
> +	return false;
> +}
> +#endif	/* CONFIG_HUGETLB_PAGE */
> +
>  static bool damon_folio_young_one(struct folio *folio,
>  		struct vm_area_struct *vma, unsigned long addr, void *arg)
>  {
> @@ -232,16 +260,29 @@ static bool damon_folio_young_one(struct folio *folio,
>  	while (page_vma_mapped_walk(&pvmw)) {
>  		addr = pvmw.address;
>  		if (pvmw.pte) {
> -			pte = ptep_get(pvmw.pte);
> -
>  			/*
> -			 * PFN swap PTEs, such as device-exclusive ones, that
> -			 * actually map pages are "old" from a CPU perspective.
> -			 * The MMU notifier takes care of any device aspects.
> +			 * For hugetlb folios, page_vma_mapped_walk() sets
> +			 * pvmw.pte to the huge PTE with its page table lock
> +			 * held.
>  			 */

Again, this new comment looks unnecessary.  Let's drop.

> -			*accessed = (pte_present(pte) && pte_young(pte)) ||
> -				!folio_test_idle(folio) ||
> -				mmu_notifier_test_young(vma->vm_mm, addr);
> +			if (folio_test_hugetlb(folio)) {
> +				*accessed = damon_hugetlb_young(pvmw.pte, vma,
> +						addr, folio);
> +			} else {
> +				pte = ptep_get(pvmw.pte);
> +
> +				/*
> +				 * PFN swap PTEs, such as device-exclusive
> +				 * ones, that actually map pages are "old"
> +				 * from a CPU perspective. The MMU notifier
> +				 * takes care of any device aspects.
> +				 */
> +				*accessed = (pte_present(pte) &&
> +						pte_young(pte)) ||
> +					!folio_test_idle(folio) ||
> +					mmu_notifier_test_young(vma->vm_mm,
> +							addr);
> +			}

I feel like the indentation becomes too deep.  Could we split out this into
another static function, say, damon_pte_young()?

>  		} else {
>  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
>  			pmd_t pmd = pmdp_get(pvmw.pmd);
> -- 
> 2.54.0

Other than the above two simple things, this patch looks good to me.


Thanks,
SJ

Re: [PATCH 3/6] mm/damon/paddr: support hugetlb folios in access monitoring

[SJ Park]

On Sat, 29 Aug 2026 22:14:04 -0700 Krishna Iyer <kiyer@crusoe.ai> wrote:

> DAMON's physical address space monitoring is blind to hugetlb-backed
> memory.  Every access check starts at damon_get_folio(), which rejects
> folios that are not on the LRU lists.  Hugetlb folios are managed
> outside of the LRU by design, so every sampling attempt on
> hugetlb-backed memory silently fails and the pages are reported as
> never accessed.
> 
> This is a significant blind spot on virtualization hosts.  Cloud
> hypervisor hosts commonly back guest memory with 1 GiB hugetlbfs pages,
> covering the vast majority of the machine's memory.  On such hosts,
> modules like DAMON_STAT observe only the host-side remainder (kernel,
> page cache, daemons)

DAMON can monitor accessses on page cache and user-space daemon memory.  But it
cannot monitor normal kernel memory, which is not LRU-managed.  Let's drop
'kernel' from the example.

> and report all guest working sets as permanently
> idle, defeating the purpose of host-level access monitoring.  In
> testing on a 1 TiB host, an hour of 4-thread random access over 842 GiB
> inside a guest was statistically indistinguishable from an idle host,
> while a 40x smaller host-side workload produced a quantitatively
> correct response.
> 
> Add damon_get_folio_incl_hugetlb(), which additionally accepts hugetlb
> folios, and use it in the two paddr access monitoring primitives,
> damon_pa_mkold() and damon_pa_young().  With the previous commit
> teaching the folio-granular rmap walkers to age huge PTEs and to call
> the mmu notifiers spanning the whole huge page, this makes guest
> accesses visible through secondary MMU (e.g. KVM/EPT) young bits.
> 
> Free hugetlb pool folios have a zero refcount, so folio_try_get()
> naturally keeps rejecting them.
> 
> The DAMOS action appliers (damon_pa_pageout(),
> damon_pa_mark_accessed_or_deactivate(), damon_pa_migrate(),
> damon_pa_stat()) keep using damon_get_folio(): reclaim, LRU
> manipulation and migration cannot act on hugetlb folios, so their
> behavior is unchanged.

Thank you for keeping the DAMOS-side behavior unchanged.

However, incl_hugetlb() sounds too specific to me.  It will keep working for
monitoring purpose folios.  What about something more geeneric, say,
damon_get_monitor_folio()?

> 
> Note that the access check granularity for hugetlb-backed memory is the
> huge page size: one touched byte reports the whole (up to 1 GiB) page
> as accessed.  Also, DAMON now consumes secondary MMU young bits that
> KVM's own aging uses; at DAMON's sampling rate (one page per region per
> sampling interval) the interference is negligible.
> 
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Krishna Iyer <kiyer@crusoe.ai>
> ---
>  mm/damon/ops-common.c | 31 +++++++++++++++++++++++++++----
>  mm/damon/ops-common.h |  1 +
>  mm/damon/paddr.c      |  4 ++--
>  3 files changed, 30 insertions(+), 6 deletions(-)
> 
> diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c
> index 62004206ca31..ece101d34684 100644
> --- a/mm/damon/ops-common.c
> +++ b/mm/damon/ops-common.c
> @@ -15,14 +15,20 @@
>  #include "../internal.h"
>  #include "ops-common.h"
>  
> +static bool damon_folio_observable(struct folio *folio, bool incl_hugetlb)
> +{
> +	return folio_test_lru(folio) ||
> +		(incl_hugetlb && folio_test_hugetlb(folio));
> +}

This is for not only monitoring but also for DAMOS.  What about changing the
name to be more generic, say, damon_folio_acceptable()?

And 'incl_hugetlb' could be more specific to the usage purpose.  E.g.,
'monitor'?

> +
>  /*
> - * Get an online page for a pfn if it's in the LRU list.  Otherwise, returns
> - * NULL.
> + * Get an online page for a pfn if it's in the LRU list, or a hugetlb folio if
> + * @incl_hugetlb is set.  Otherwise, returns NULL.
>   *
>   * The body of this function is stolen from the 'page_idle_get_folio()'.  We
>   * steal rather than reuse it because the code is quite simple.
>   */
> -struct folio *damon_get_folio(unsigned long pfn)
> +static struct folio *__damon_get_folio(unsigned long pfn, bool incl_hugetlb)

As I also mentioned above, I feel like incl_hugetlb is too case specific.
Could we rename it to somewhat like 'monitor' or 'damos'?

>  {
>  	struct page *page = pfn_to_online_page(pfn);
>  	struct folio *folio;
> @@ -33,13 +39,30 @@ struct folio *damon_get_folio(unsigned long pfn)
>  	folio = page_folio(page);
>  	if (!folio_try_get(folio))
>  		return NULL;
> -	if (unlikely(page_folio(page) != folio) || !folio_test_lru(folio)) {
> +	if (unlikely(page_folio(page) != folio) ||
> +			!damon_folio_observable(folio, incl_hugetlb)) {
>  		folio_put(folio);
>  		folio = NULL;
>  	}
>  	return folio;
>  }
>  
> +struct folio *damon_get_folio(unsigned long pfn)
> +{
> +	return __damon_get_folio(pfn, false);
> +}
> +
> +/*
> + * Same to damon_get_folio(), but also accepts hugetlb folios, which are
> + * managed outside of the LRU lists.  Aimed to be used by access monitoring
> + * primitives.  DAMOS actions that assume LRU-managed folios should keep using
> + * damon_get_folio().
> + */
> +struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn)
> +{
> +	return __damon_get_folio(pfn, true);
> +}

As commented above, 'incl_hugetlb()' feels too case specific to me.  What about
damon_get_monitor_folio()?

> +
>  void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr)
>  {
>  	pte_t pteval = ptep_get(pte);
> diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h
> index f7811c9c7a02..68d7de49c87a 100644
> --- a/mm/damon/ops-common.h
> +++ b/mm/damon/ops-common.h
> @@ -6,6 +6,7 @@
>  #include <linux/damon.h>
>  
>  struct folio *damon_get_folio(unsigned long pfn);
> +struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn);
>  
>  void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr);
>  void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr);
> diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
> index 5c6c3a597fd0..09d418b2874b 100644
> --- a/mm/damon/paddr.c
> +++ b/mm/damon/paddr.c
> @@ -37,7 +37,7 @@ static unsigned long damon_pa_core_addr(
>  
>  static void damon_pa_mkold(phys_addr_t paddr)
>  {
> -	struct folio *folio = damon_get_folio(PHYS_PFN(paddr));
> +	struct folio *folio = damon_get_folio_incl_hugetlb(PHYS_PFN(paddr));
>  
>  	if (!folio)
>  		return;
> @@ -67,7 +67,7 @@ static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
>  
>  static bool damon_pa_young(phys_addr_t paddr)
>  {
> -	struct folio *folio = damon_get_folio(PHYS_PFN(paddr));
> +	struct folio *folio = damon_get_folio_incl_hugetlb(PHYS_PFN(paddr));
>  	bool accessed;
>  
>  	if (!folio)
> -- 
> 2.54.0

Other parts look good to me.


Thanks,
SJ

Re: [PATCH 0/6] mm/damon: support access monitoring of hugetlb-backed memory

[SJ Park]

Hi Krishna,


Thanks you for this great patch series.

On Sat, 29 Aug 2026 22:14:01 -0700 Krishna Iyer <kiyer@crusoe.ai> wrote:

> On virtualization hosts, most system memory is often backed by
> hugetlbfs.  On our production hosts, for example, ~95% of RAM is 1 GiB
> hugetlb pages backing guest memory.  DAMON's physical address space
> monitoring cannot produce a useful access signal for such memory, for
> two layered reasons.
> 
> First, it is blind to hugetlb folios.  Every access check starts at
> damon_get_folio(), which rejects folios that are not on the LRU lists,
> and hugetlb folios are managed outside of the LRU by design.  As a
> result, all hugetlb-backed memory is silently reported as never
> accessed.

Nice catch.  I admit DAMON is completely broken for your use case, and this
will effectively and completely fix it.  I want to get this fix as soon as
possible.

> 
> Second, even once hugetlb folios are visible, access-bit-based sampling
> under-reports hot memory.  DAMON deliberately clears accessed bits
> without TLB flushes to keep the overhead low, so accesses served from
> cached translations do not walk the page tables and never re-set the
> bit.  With 1 GiB mappings, translations effectively never leave the
> TLBs, pinning the observed access rate to the TLB-refill rate: a
> saturating guest is indistinguishable from a nearly idle one.

There were similar issue reports.  In most cases, it was because they were
running small test workloads on large systems.  It was unclear if the same
issue can happen on real production workloads.  That's because we expect real
workloads to intentionally access most of system memory, and therefore
naturally flush TLB entries.  On 1 TiB memory system utilizing 1 GiB huge tlb
pages, I can agree the issue can happen more frequently, even on real
workloads.  But, I'm still curious what is the real impact.

> 
> Patches 1-3 address the visibility problem.  The first patch moves
> damon_hugetlb_mkold() from vaddr to ops-common as a preparation.  The
> second patch teaches the folio mkold/young rmap walkers to handle
> hugetlb folios, aging the huge PTE and notifying secondary MMUs across
> the whole huge page size; the secondary MMU notification is what
> surfaces guest-side (e.g., KVM/EPT) accessed bits.  The third patch
> adds damon_get_folio_incl_hugetlb() and uses it from the paddr
> monitoring primitives only.  DAMOS action appliers such as
> DAMON_RECLAIM and DAMON_LRU_SORT keep the LRU-only lookup and are
> behaviorally unchanged.

I left a few trivial comments for names and comments to patches 2 and 3.  After
resolving the trivial things, these will be ready to be merged in my opinion.

Meanwhile, I have a few concerns and want to have sufficient discussions for
rest of this series.  Could you pleae split the first patches out of this
series, revision for the given comments and post as another series?  After
finishing it, we could take more time on the second part of this series.

> 
> Patches 4-6 address the intensity problem with a new default-off
> monitoring context option, 'aging_flush'.  When set, access bit
> clearing for monitoring uses the flushing primitives
> (mmu_notifier_clear_flush_young() and TLB-flushing PTE/PMD/hugetlb
> aging) so that the next access is guaranteed to re-set the accessed
> bit.  The option is exposed via the DAMON sysfs interface (patch 5) and
> as a DAMON_STAT module parameter (patch 6).  With the option unset, the
> behavior is identical to before this series.

I understand this is really needed for your use case.  But, I'd like to take
sufficient time here.  Two concerns stand out to me.  The maintenance of the
new interface, and the performance impact.

Firstly, the new interface.  We are changing DAMON for general data attributes
monitoring.  Memory access will be only one of the attributes that DAMON can
monitor.  For this, we introduced probes interface.  As soon as the new
interface is matured, we would discourage use of the old interface for access
monitoring.  We might even deprecate it over long time.  To make it easier, I
don't want to add more features for access monitoring only.  I'd like to make
this TLB flush interface as part of the new interface.

More specifically, I'm working [1] on introduce monitoring preparation actions
concept.  We could add TLB flushing as one of preparation actions and implement
interface on it.  For example,

    cd /sys/kernel/mm/damon/...
    echo tlb_flush > preps/0/prep_action

It could also later be extended for custom periodic TLB flush (we might not
want to flush TLB for every sampling but once per N samplings), like below.

    # do the given prep action once per 5 sampling intervals.
    echo 5 > preps/0/action_interval

I'm giving above as an example brain-storming ideas.  I'm not suggesting to do
exactly that.  But my point is, I think we need more thought about the
long-term sustainable and maintainable interface.

> 
> Evaluation on an Intel EPT host running a 944 GiB guest with an 842 GiB
> in-guest memory workload (masim):
> 
>   configuration          estimated access rate    idle -> load
>                          idle -> load (GB/s)      separation
>   stock                  0.08 -> 0.21             none (guest memory
>                                                   invisible)
>   patches 1-3            1.70 -> 1.96             1.2x
>   full series, flush on  1.21 -> 5.34             4.4x

So, you can show some of the guest access after patches 1-3, because TLB is
naturally flushed even in your setup.  But with explicit and periodic TLB
flush, you show definitely more and real access.  This makes sense to me, and
align with the theory.

> 
> With aging_flush set, the estimated hot footprint covers ~82% of the
> actual working set size, compared to ~40% without flushing, where the
> signal mostly reflects the ambient TLB-refill rate rather than guest
> accesses.

I think that makes sense, but having more details would be even nicer.  For
example, below questions are raised on my head.

How do you define and detect the actual working set size, and DAMON-found
working set size?

What is the size of TLB in your setup?

Could you share more details about your masim [2] run setup?

What is the estimated memory bandwidth if you run DAMON inside guest?

> kdamond CPU consumption stays approximately 0%, and the
> worst-case flush overhead measured is below 0.1%, which is acceptable
> for hosts that opt in.

Flushing TLB makes overhead in kdamond.  But it also makes performance
degradation to the running workloads.  Actually that's bigger concern.  Have
you had a chance to measure that?  If so, could you share that too?

> 
> Per Documentation/process/generated-content.rst, this series was
> developed with the assistance of an AI coding assistant (Anthropic
> Claude, via Claude Code).  The assistant helped draft the code and
> changelogs, and performed the rebase of the series from its original
> 6.17-based development tree onto mm-new, including merge conflict
> resolution against recent DAMON changes.  All changes were reviewed by
> the human submitter, who takes full responsibility for the
> contribution.
> 
> The series as posted here was regression-tested on its base commit
> with a full x86_64 kernel build (no new W=1 warnings in mm/damon), the
> DAMON kunit suite (41/41 passing) and the DAMON selftests (15/15
> passing) on a kernel booted with virtme-ng.  Functional validation
> (the evaluation above, ftrace verification that the flag switches the
> notifier variants, and enable/disable stress testing) was performed by
> the submitter on production-like hosts running the 6.17-based backport
> of this series.

This kind of transparent and detailed disclosures help reviewers.  Thank you
for doing this.

So, to summarize, I'd suggest reposting the first three patches as another
series and focus on it for now.  In parallel or after it is done, we could
further discuss about TLB flush change.

> 
> Krishna Iyer (6):
>   mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common
>   mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap
>     walkers
>   mm/damon/paddr: support hugetlb folios in access monitoring
>   mm/damon: support flush-assisted access bit clearing for monitoring
>   mm/damon/sysfs: support aging_flush
>   mm/damon/stat: support aging_flush

Because there are high level things that I'd like to discuss first, I will hold
reviewing patches 4-6 for now.

> 
>  include/linux/damon.h |   2 +
>  mm/damon/core.c       |   2 +
>  mm/damon/ops-common.c | 154 ++++++++++++++++++++++++++++++++++++------
>  mm/damon/ops-common.h |  18 ++++-
>  mm/damon/paddr.c      |  11 +--
>  mm/damon/stat.c       |   6 ++
>  mm/damon/sysfs.c      |  29 ++++++++
>  mm/damon/vaddr.c      |  46 +++----------
>  8 files changed, 203 insertions(+), 65 deletions(-)
> 
> 
> base-commit: d2aad7fdcda7ae8a726926f2d6de7fe9e8ee7563
> -- 
> 2.54.0

[1] https://lore.kernel.org/20260728041242.187380-1-sj@kernel.org
[2] https://github.com/sjp38/masim


Thanks,
SJ

Re: [PATCH 2/6] mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers

[SJ Park]

On Sun, 30 Aug 2026 09:13:33 -0700 SJ Park <sj@kernel.org> wrote:

> On Sun, 30 Aug 2026 05:28:55 +0000 sashiko-bot@kernel.org wrote:
[...]
> > [Severity: Medium]
> > Will mmu_notifier_test_young() miss accesses within shattered huge pages here?
> > 
> > Since the kernel's mmu_notifier_test_young() API does not take a size
> > parameter, passing only the base address of the huge page means it will
> > only evaluate the first base page.
> > 
> > For secondary MMUs like KVM EPT that map huge pages as multiple smaller
> > pages, wouldn't accesses to other subpages within the huge page be completely
> > missed? This could cause DAMON to classify the huge page as colder than it
> > actually is, which might lead to inappropriate reclaim or migration decisions.
> 
> This is also a pre-existing issue.  I will work on fixing this.

Actually, the curernt implementation is correct.  The goal here is to know if
the single byte of the address is accessed.  It works in page granularity for
most cases due to limitations.  That's not the intention.  So I think this can
just be as-is.


Thanks,
SJ

[...]