[PATCH v7 3/7] mm: Hot page tracking and promotion - pghot

Bharata B Rao
2026-05-04T06:10:33Z

pghot is a subsystem that collects memory access information from
multiple sources, classifies hot pages resident in lower-tier memory,
and promotes them to faster tiers. It stores per-PFN hotness metadata
and performs asynchronous, batched promotion via a per-lower-tier-node
kernel thread (kmigrated).

This change introduces the default (compact) mode of pghot:

- Per-PFN hotness record (phi_t = u8) embedded via mem_section:
- 2 bits: access frequency (4 levels)
- 5 bits: time bucket (≈4s window with HZ=1000, bucketed jiffies)
- 1 bit : migration-ready flag (MSB)
The LSB of mem_section->hot_map pointer is used as a per-section
"hot" flag to gate scanning.

- Event recording API:
int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
- Sources (e.g., NUMA hint faults, HW hints) call this to report
accesses.
- In default mode, the nid is not stored/used for targeting;
promotion goes to a configurable toptier node (pghot_target_nid).

- Promotion engine:
- One kmigrated thread per lower-tier node.
- Scans only sections whose "hot" flag was raised, iterates PFNs,
and batches candidates by destination node.
- Uses migrate_misplaced_folios_batch() to move batched folios.

- Tunables & stats:
- debugfs: enabled_sources, target_nid, freq_threshold,
kmigrated_sleep_ms, kmigrated_batch_nr
- sysctl : vm.pghot_promote_freq_window_ms
- vmstat : pghot_recorded_accesses, pghot_recorded_hintfaults,
pghot_recorded_hwhints

Memory overhead
---------------
Default mode uses 1 byte of hotness metadata per PFN on lower-tier
nodes.

Behavior & policy
-----------------
- Default mode promotion target:
The nid passed by sources is not stored; hot pages promote to
pghot_target_nid (toptier). Precision mode (added later in the
series) changes this.

- Record consumption:
kmigrated consumes (clears) the "migration-ready" bit before
attempting isolation. Additionally the hotness record is reset.
If isolation/migration fails, the folio is not re-queued automatically;
subsequent accesses will re-arm it. This avoids retry storms and
keeps batching stable.

- Wakeups:
kmigrated wakeups are intentionally timeout-driven. We set
the per-pgdat "activate" flag on access, and kmigrated checks this
flag on its next sleep interval. This keeps the first cut simple
and avoids potential wake storms; active wakeups can be considered
in a follow-up.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
Documentation/admin-guide/mm/index.rst | 1 +
Documentation/admin-guide/mm/pghot.rst | 80 ++++
include/linux/migrate.h | 4 +-
include/linux/mmzone.h | 20 +
include/linux/pghot.h | 82 ++++
include/linux/vm_event_item.h | 5 +
mm/Kconfig | 14 +
mm/Makefile | 1 +
mm/migrate.c | 16 +-
mm/mm_init.c | 10 +
mm/pghot-default.c | 79 ++++
mm/pghot-tunables.c | 182 +++++++++
mm/pghot.c | 494 +++++++++++++++++++++++++
mm/vmstat.c | 5 +
14 files changed, 986 insertions(+), 7 deletions(-)
create mode 100644 Documentation/admin-guide/mm/pghot.rst
create mode 100644 include/linux/pghot.h
create mode 100644 mm/pghot-default.c
create mode 100644 mm/pghot-tunables.c
create mode 100644 mm/pghot.c

diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst
index bbb563cba5d2..4d6810b02365 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -43,3 +43,4 @@ the Linux memory management.
userfaultfd
zswap
kho
+ pghot
diff --git a/Documentation/admin-guide/mm/pghot.rst b/Documentation/admin-guide/mm/pghot.rst
new file mode 100644
index 000000000000..5f51dd1d4d45
--- /dev/null
+++ b/Documentation/admin-guide/mm/pghot.rst
@@ -0,0 +1,80 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================================
+PGHOT: Hot Page Tracking Tunables
+=================================
+
+Overview
+========
+The PGHOT subsystem tracks frequently accessed pages in lower-tier memory and
+promotes them to faster tiers. It uses per-PFN hotness metadata and asynchronous
+migration via per-node kernel threads (kmigrated).
+
+This document describes tunables available via **debugfs** and **sysctl** for
+PGHOT.
+
+Debugfs Interface
+=================
+Path: /sys/kernel/debug/pghot/
+
+1. **enabled_sources**
+ - Bitmask to enable/disable hotness sources.
+ - Bits:
+ - 0: Hint faults (value 0x1)
+ - 1: Hardware hints (value 0x2)
+ - Default: 0 (disabled)
+ - Example:
+ # echo 0x3 > /sys/kernel/debug/pghot/enabled_sources
+ Enables all sources.
+
+2. **target_nid**
+ - Toptier NUMA node ID to which hot pages should be promoted when source
+ does not provide nid. Used when hotness source can't provide accessing
+ NID or when the tracking mode is default.
+ - Default: 0
+ - Example:
+ # echo 1 > /sys/kernel/debug/pghot/target_nid
+
+3. **freq_threshold**
+ - Minimum access frequency before a page is marked ready for promotion.
+ - Range: 1 to 3
+ - Default: 2
+ - Example:
+ # echo 3 > /sys/kernel/debug/pghot/freq_threshold
+
+4. **kmigrated_sleep_ms**
+ - Sleep interval (ms) for kmigrated thread between scans.
+ - Default: 100
+
+5. **kmigrated_batch_nr**
+ - Maximum number of folios migrated in one batch.
+ - Default: 512
+
+Sysctl Interface
+================
+1. pghot_promote_freq_window_ms
+
+Path: /proc/sys/vm/pghot_promote_freq_window_ms
+
+- Controls the time window (in ms) for counting access frequency. A page is
+ considered hot only when **freq_threshold** number of accesses occur with
+ this time period.
+- Default: 3000 (3 seconds)
+- Example:
+ # sysctl vm.pghot_promote_freq_window_ms=3000
+
+Vmstat Counters
+===============
+Following vmstat counters provide some stats about pghot subsystem.
+
+Path: /proc/vmstat
+
+1. **pghot_recorded_accesses**
+ - Number of total hot page accesses recorded by pghot.
+
+2. **pghot_recorded_hintfaults**
+ - Number of recorded accesses reported by NUMA Balancing based
+ hotness source.
+
+3. **pghot_recorded_hwhints**
+ - Number of recorded accesses reported by hwhints source.
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index d136612eef9d..53bae80d11ae 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -107,7 +107,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p

#endif /* CONFIG_MIGRATION */

-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
int migrate_misplaced_folio_prepare(struct folio *folio,
struct vm_area_struct *vma, int node);
int migrate_misplaced_folio(struct folio *folio, int node);
@@ -126,7 +126,7 @@ static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, i
{
return -EAGAIN; /* can't migrate now */
}
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */

#ifdef CONFIG_MIGRATION

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 9adb2ad21da5..eb08431dc9fb 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1155,6 +1155,7 @@ enum pgdat_flags {
* many pages under writeback
*/
PGDAT_RECLAIM_LOCKED, /* prevents concurrent reclaim */
+ PGDAT_KMIGRATED_ACTIVATE, /* activates kmigrated */
};

enum zone_flags {
@@ -1609,6 +1610,10 @@ typedef struct pglist_data {
#ifdef CONFIG_MEMORY_FAILURE
struct memory_failure_stats mf_stats;
#endif
+#ifdef CONFIG_PGHOT
+ struct task_struct *kmigrated;
+ wait_queue_head_t kmigrated_wait;
+#endif
} pg_data_t;

#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
@@ -2019,12 +2024,27 @@ struct mem_section {
unsigned long section_mem_map;

struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+ /*
+ * Per-PFN hotness data for this section.
+ * Array of phi_t (u8 in default mode).
+ * LSB is used as PGHOT_SECTION_HOT_BIT flag.
+ */
+ void *hot_map;
+#endif
#ifdef CONFIG_PAGE_EXTENSION
/*
* If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
* section. (see page_ext.h about this.)
*/
struct page_ext *page_ext;
+#endif
+ /*
+ * Padding to maintain consistent mem_section size when exactly
+ * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+ * optimal alignment regardless of configuration.
+ */
+#if (defined(CONFIG_PGHOT) ^ defined(CONFIG_PAGE_EXTENSION))
unsigned long pad;
#endif
/*
diff --git a/include/linux/pghot.h b/include/linux/pghot.h
new file mode 100644
index 000000000000..525d4dd28fc1
--- /dev/null
+++ b/include/linux/pghot.h
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_PGHOT_H
+#define _LINUX_PGHOT_H
+
+/* Page hotness temperature sources */
+enum pghot_src {
+ PGHOT_HINTFAULTS = 0,
+ PGHOT_HWHINTS,
+ PGHOT_SRC_MAX
+};
+
+#ifdef CONFIG_PGHOT
+#include <linux/static_key.h>
+
+extern unsigned int pghot_target_nid;
+extern unsigned int pghot_src_enabled;
+extern unsigned int pghot_freq_threshold;
+extern unsigned int kmigrated_sleep_ms;
+extern unsigned int kmigrated_batch_nr;
+extern unsigned int sysctl_pghot_freq_window;
+
+void pghot_debug_init(void);
+
+DECLARE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
+
+#define PGHOT_HINTFAULTS_ENABLED BIT(PGHOT_HINTFAULTS)
+#define PGHOT_HWHINTS_ENABLED BIT(PGHOT_HWHINTS)
+#define PGHOT_SRC_ENABLED_MASK GENMASK(PGHOT_SRC_MAX - 1, 0)
+
+#define PGHOT_DEFAULT_FREQ_THRESHOLD 2
+
+#define KMIGRATED_DEFAULT_SLEEP_MS 100
+#define KMIGRATED_DEFAULT_BATCH_NR 512
+
+#define PGHOT_DEFAULT_NODE 0
+
+#define PGHOT_DEFAULT_FREQ_WINDOW (3 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-6 are used to store frequency and time.
+ * Bit 7 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY 7
+
+#define PGHOT_FREQ_WIDTH 2
+/* Bucketed time is stored in 5 bits which can represent up to 3.9s with HZ=1000 */
+#define PGHOT_TIME_BUCKETS_SHIFT 7
+#define PGHOT_TIME_WIDTH 5
+#define PGHOT_NID_WIDTH 10
+
+#define PGHOT_FREQ_SHIFT 0
+#define PGHOT_TIME_SHIFT (PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_FREQ_MASK GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+#define PGHOT_TIME_BUCKETS_MASK (PGHOT_TIME_MASK << PGHOT_TIME_BUCKETS_SHIFT)
+
+#define PGHOT_NID_MAX ((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX ((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX ((1 << PGHOT_TIME_WIDTH) - 1)
+
+typedef u8 phi_t;
+
+#define PGHOT_RECORD_SIZE sizeof(phi_t)
+
+#define PGHOT_SECTION_HOT_BIT 0
+#define PGHOT_SECTION_HOT_MASK BIT(PGHOT_SECTION_HOT_BIT)
+
+bool pghot_nid_valid(int nid);
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time);
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now);
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time);
+
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now);
+#else
+static inline int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+ return 0;
+}
+#endif /* CONFIG_PGHOT */
+#endif /* _LINUX_PGHOT_H */
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 03fe95f5a020..58d510711bd4 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -175,6 +175,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
KSTACK_REST,
#endif
#endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+ PGHOT_RECORDED_ACCESSES,
+ PGHOT_RECORDED_HINTFAULTS,
+ PGHOT_RECORDED_HWHINTS,
+#endif /* CONFIG_PGHOT */
NR_VM_EVENT_ITEMS
};

diff --git a/mm/Kconfig b/mm/Kconfig
index 0a43bb80df4f..ebfa149d8123 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1469,6 +1469,20 @@ config LAZY_MMU_MODE_KUNIT_TEST

If unsure, say N.

+config PGHOT
+ bool "Hot page tracking and promotion"
+ default n
+ depends on NUMA_MIGRATION && SPARSEMEM
+ help
+ A sub-system to track page accesses in lower tier memory and
+ maintain hot page information. Promotes hot pages from lower
+ tiers to top tier by using the memory access information provided
+ by various sources. Asynchronous promotion is done by per-node
+ kernel threads.
+
+ This adds 1 byte of metadata overhead per page in lower-tier
+ memory nodes.
+
source "mm/damon/Kconfig"

endmenu
diff --git a/mm/Makefile b/mm/Makefile
index 8ad2ab08244e..33014de43acc 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
obj-$(CONFIG_EXECMEM) += execmem.o
obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
diff --git a/mm/migrate.c b/mm/migrate.c
index 747277aadf19..726d27b61a46 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2625,7 +2625,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
}
#endif /* CONFIG_NUMA_MIGRATION */

-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
/*
* Returns true if this is a safe migration target node for misplaced NUMA
* pages. Currently it only checks the watermarks which is crude.
@@ -2745,12 +2745,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
*/
int migrate_misplaced_folio(struct folio *folio, int node)
{
- pg_data_t *pgdat = NODE_DATA(node);
int nr_remaining;
unsigned int nr_succeeded;
LIST_HEAD(migratepages);
struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
- struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);

list_add(&folio->lru, &migratepages);
nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2759,12 +2757,18 @@ int migrate_misplaced_folio(struct folio *folio, int node)
if (nr_remaining && !list_empty(&migratepages))
putback_movable_pages(&migratepages);
if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
&& !node_is_toptier(folio_nid(folio))
- && node_is_toptier(node))
+ && node_is_toptier(node)) {
+ pg_data_t *pgdat = NODE_DATA(node);
+ struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
+ }
+#endif
}
mem_cgroup_put(memcg);
BUG_ON(!list_empty(&migratepages));
@@ -2817,14 +2821,16 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
putback_movable_pages(folio_list);

if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
PGPROMOTE_SUCCESS, nr_succeeded);
+#endif
}

mem_cgroup_put(memcg);
WARN_ON(!list_empty(folio_list));
return nr_remaining ? -EAGAIN : 0;
}
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
diff --git a/mm/mm_init.c b/mm/mm_init.c
index f9f8e1af921c..2396c42028ae 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1384,6 +1384,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
#endif

+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+ init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
{
int i;
@@ -1393,6 +1402,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)

pgdat_init_split_queue(pgdat);
pgdat_init_kcompactd(pgdat);
+ pgdat_init_kmigrated(pgdat);

init_waitqueue_head(&pgdat->kswapd_wait);
init_waitqueue_head(&pgdat->pfmemalloc_wait);
diff --git a/mm/pghot-default.c b/mm/pghot-default.c
new file mode 100644
index 000000000000..e610062345e4
--- /dev/null
+++ b/mm/pghot-default.c
@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Default mode
+ *
+ * 1 byte hotness record per PFN.
+ * Bucketed time and frequency tracked as part of the record.
+ * Promotion to @pghot_target_nid by default.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+
+/* pghot-default doesn't store and hence no NID validation is required */
+bool pghot_nid_valid(int nid)
+{
+ return true;
+}
+
+/*
+ * @time is regular time, @old_time is bucketed time.
+ */
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+ time &= PGHOT_TIME_BUCKETS_MASK;
+ old_time <<= PGHOT_TIME_BUCKETS_SHIFT;
+
+ return jiffies_to_msecs((time - old_time) & PGHOT_TIME_BUCKETS_MASK);
+}
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
+ phi_t freq, old_freq, hotness, old_hotness, old_time;
+ phi_t time = now >> PGHOT_TIME_BUCKETS_SHIFT;
+
+ old_hotness = READ_ONCE(*phi);
+ do {
+ bool new_window = false;
+
+ hotness = old_hotness;
+ old_freq = (hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+ old_time = (hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+
+ if (pghot_access_latency(old_time, now) > sysctl_pghot_freq_window)
+ new_window = true;
+
+ if (new_window)
+ freq = 1;
+ else if (old_freq < PGHOT_FREQ_MAX)
+ freq = old_freq + 1;
+ else
+ freq = old_freq;
+
+ hotness &= ~(PGHOT_FREQ_MASK << PGHOT_FREQ_SHIFT);
+ hotness &= ~(PGHOT_TIME_MASK << PGHOT_TIME_SHIFT);
+
+ hotness |= (freq & PGHOT_FREQ_MASK) << PGHOT_FREQ_SHIFT;
+ hotness |= (time & PGHOT_TIME_MASK) << PGHOT_TIME_SHIFT;
+
+ if (freq >= pghot_freq_threshold)
+ hotness |= BIT(PGHOT_MIGRATE_READY);
+ } while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+ return !!(hotness & BIT(PGHOT_MIGRATE_READY));
+}
+
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time)
+{
+ phi_t old_hotness, hotness = 0;
+
+ old_hotness = READ_ONCE(*phi);
+ do {
+ if (!(old_hotness & BIT(PGHOT_MIGRATE_READY)))
+ return -EINVAL;
+ } while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+
+ *nid = pghot_target_nid;
+ *freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+ *time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+ return 0;
+}
diff --git a/mm/pghot-tunables.c b/mm/pghot-tunables.c
new file mode 100644
index 000000000000..f04e2137309e
--- /dev/null
+++ b/mm/pghot-tunables.c
@@ -0,0 +1,182 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot tunables in debugfs
+ */
+#include <linux/pghot.h>
+#include <linux/memory-tiers.h>
+#include <linux/debugfs.h>
+
+static struct dentry *debugfs_pghot;
+static DEFINE_MUTEX(pghot_tunables_lock);
+
+static ssize_t pghot_freq_th_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[16];
+ unsigned int freq;
+
+ if (cnt > 15)
+ cnt = 15;
+
+ if (copy_from_user(&buf, ubuf, cnt))
+ return -EFAULT;
+ buf[cnt] = '\0';
+
+ if (kstrtouint(buf, 10, &freq))
+ return -EINVAL;
+
+ if (!freq || freq > PGHOT_FREQ_MAX)
+ return -EINVAL;
+
+ mutex_lock(&pghot_tunables_lock);
+ pghot_freq_threshold = freq;
+ mutex_unlock(&pghot_tunables_lock);
+
+ *ppos += cnt;
+ return cnt;
+}
+
+static int pghot_freq_th_show(struct seq_file *m, void *v)
+{
+ seq_printf(m, "%d\n", pghot_freq_threshold);
+ return 0;
+}
+
+static int pghot_freq_th_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, pghot_freq_th_show, NULL);
+}
+
+static const struct file_operations pghot_freq_th_fops = {
+ .open = pghot_freq_th_open,
+ .write = pghot_freq_th_write,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static ssize_t pghot_target_nid_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[16];
+ unsigned int nid;
+
+ if (cnt > 15)
+ cnt = 15;
+
+ if (copy_from_user(&buf, ubuf, cnt))
+ return -EFAULT;
+ buf[cnt] = '\0';
+
+ if (kstrtouint(buf, 10, &nid))
+ return -EINVAL;
+
+ if (nid > PGHOT_NID_MAX || !node_online(nid) || !node_is_toptier(nid))
+ return -EINVAL;
+ mutex_lock(&pghot_tunables_lock);
+ pghot_target_nid = nid;
+ mutex_unlock(&pghot_tunables_lock);
+
+ *ppos += cnt;
+ return cnt;
+}
+
+static int pghot_target_nid_show(struct seq_file *m, void *v)
+{
+ seq_printf(m, "%d\n", pghot_target_nid);
+ return 0;
+}
+
+static int pghot_target_nid_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, pghot_target_nid_show, NULL);
+}
+
+static const struct file_operations pghot_target_nid_fops = {
+ .open = pghot_target_nid_open,
+ .write = pghot_target_nid_write,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static void pghot_src_enabled_update(unsigned int enabled)
+{
+ unsigned int changed = pghot_src_enabled ^ enabled;
+
+ if (changed & PGHOT_HINTFAULTS_ENABLED) {
+ if (enabled & PGHOT_HINTFAULTS_ENABLED)
+ static_branch_enable(&pghot_src_hintfaults);
+ else
+ static_branch_disable(&pghot_src_hintfaults);
+ }
+
+ if (changed & PGHOT_HWHINTS_ENABLED) {
+ if (enabled & PGHOT_HWHINTS_ENABLED)
+ static_branch_enable(&pghot_src_hwhints);
+ else
+ static_branch_disable(&pghot_src_hwhints);
+ }
+}
+
+static ssize_t pghot_src_enabled_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[16];
+ unsigned int enabled;
+
+ if (cnt > 15)
+ cnt = 15;
+
+ if (copy_from_user(&buf, ubuf, cnt))
+ return -EFAULT;
+ buf[cnt] = '\0';
+
+ if (kstrtouint(buf, 0, &enabled))
+ return -EINVAL;
+
+ if (enabled & ~PGHOT_SRC_ENABLED_MASK)
+ return -EINVAL;
+
+ mutex_lock(&pghot_tunables_lock);
+ pghot_src_enabled_update(enabled);
+ pghot_src_enabled = enabled;
+ mutex_unlock(&pghot_tunables_lock);
+
+ *ppos += cnt;
+ return cnt;
+}
+
+static int pghot_src_enabled_show(struct seq_file *m, void *v)
+{
+ seq_printf(m, "%u\n", pghot_src_enabled);
+ return 0;
+}
+
+static int pghot_src_enabled_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, pghot_src_enabled_show, NULL);
+}
+
+static const struct file_operations pghot_src_enabled_fops = {
+ .open = pghot_src_enabled_open,
+ .write = pghot_src_enabled_write,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+void pghot_debug_init(void)
+{
+ debugfs_pghot = debugfs_create_dir("pghot", NULL);
+ debugfs_create_file("enabled_sources", 0644, debugfs_pghot, NULL,
+ &pghot_src_enabled_fops);
+ debugfs_create_file("target_nid", 0644, debugfs_pghot, NULL,
+ &pghot_target_nid_fops);
+ debugfs_create_file("freq_threshold", 0644, debugfs_pghot, NULL,
+ &pghot_freq_th_fops);
+ debugfs_create_u32("kmigrated_sleep_ms", 0644, debugfs_pghot,
+ &kmigrated_sleep_ms);
+ debugfs_create_u32("kmigrated_batch_nr", 0644, debugfs_pghot,
+ &kmigrated_batch_nr);
+}
diff --git a/mm/pghot.c b/mm/pghot.c
new file mode 100644
index 000000000000..02e6959b647a
--- /dev/null
+++ b/mm/pghot.c
@@ -0,0 +1,494 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Maintains information about hot pages from slower tier nodes and
+ * promotes them.
+ *
+ * Per-PFN hotness information is stored for lower tier nodes in
+ * mem_section.
+ *
+ * In the default mode, a single byte (u8) is used to store
+ * the frequency of access and last access time. Promotions are done
+ * to a default toptier NID.
+ *
+ * A kernel thread named kmigrated is provided to migrate or promote
+ * the hot pages. kmigrated runs for each lower tier node. It iterates
+ * over the node's PFNs and migrates pages marked for migration into
+ * their targeted nodes.
+ */
+#include <linux/mm.h>
+#include <linux/migrate.h>
+#include <linux/memory.h>
+#include <linux/memory-tiers.h>
+#include <linux/pghot.h>
+
+unsigned int pghot_target_nid = PGHOT_DEFAULT_NODE;
+unsigned int pghot_src_enabled;
+unsigned int pghot_freq_threshold = PGHOT_DEFAULT_FREQ_THRESHOLD;
+unsigned int kmigrated_sleep_ms = KMIGRATED_DEFAULT_SLEEP_MS;
+unsigned int kmigrated_batch_nr = KMIGRATED_DEFAULT_BATCH_NR;
+
+unsigned int sysctl_pghot_freq_window = PGHOT_DEFAULT_FREQ_WINDOW;
+
+DEFINE_STATIC_KEY_FALSE(pghot_src_hwhints);
+DEFINE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+
+#ifdef CONFIG_SYSCTL
+static const struct ctl_table pghot_sysctls[] = {
+ {
+ .procname = "pghot_promote_freq_window_ms",
+ .data = &sysctl_pghot_freq_window,
+ .maxlen = sizeof(unsigned int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ },
+};
+#endif
+
+static bool kmigrated_started __ro_after_init;
+
+/**
+ * pghot_record_access() - Record page accesses from lower tier memory
+ * for the purpose of tracking page hotness and subsequent promotion.
+ *
+ * @pfn: PFN of the page
+ * @nid: Unused
+ * @src: The identifier of the sub-system that reports the access
+ * @now: Access time in jiffies
+ *
+ * Updates the frequency and time of access and marks the page as
+ * ready for migration if the frequency crosses a threshold. The pages
+ * marked for migration are migrated by kmigrated kernel thread.
+ *
+ * Return: 0 on success and -EINVAL on failure to record the access.
+ */
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+ struct mem_section *ms;
+ struct folio *folio;
+ phi_t *phi, *hot_map;
+ struct page *page;
+ int src_nid;
+
+ if (!kmigrated_started)
+ return 0;
+
+ if (!pghot_nid_valid(nid))
+ return -EINVAL;
+
+ switch (src) {
+ case PGHOT_HINTFAULTS:
+ if (!static_branch_unlikely(&pghot_src_hintfaults))
+ return 0;
+ count_vm_event(PGHOT_RECORDED_HINTFAULTS);
+ break;
+ case PGHOT_HWHINTS:
+ if (!static_branch_unlikely(&pghot_src_hwhints))
+ return 0;
+ count_vm_event(PGHOT_RECORDED_HWHINTS);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ src_nid = pfn_to_nid(pfn);
+ if (src_nid == nid)
+ return 0;
+
+ /*
+ * Record only accesses from lower tiers.
+ */
+ if (node_is_toptier(src_nid))
+ return 0;
+
+ /*
+ * Reject the non-migratable pages right away.
+ */
+ page = pfn_to_online_page(pfn);
+ if (!page || is_zone_device_page(page))
+ return 0;
+
+ folio = page_folio(page);
+ if (!folio_try_get(folio))
+ return 0;
+
+ if (unlikely(page_folio(page) != folio))
+ goto out;
+
+ if (!folio_test_lru(folio))
+ goto out;
+
+ /* Get the hotness slot corresponding to the 1st PFN of the folio */
+ pfn = folio_pfn(folio);
+ ms = __pfn_to_section(pfn);
+ if (!ms || !ms->hot_map)
+ goto out;
+
+ hot_map = (phi_t *)(((unsigned long)(ms->hot_map)) & ~PGHOT_SECTION_HOT_MASK);
+ phi = &hot_map[pfn % PAGES_PER_SECTION];
+
+ count_vm_event(PGHOT_RECORDED_ACCESSES);
+
+ /*
+ * Update the hotness parameters.
+ */
+ if (pghot_update_record(phi, nid, now)) {
+ set_bit(PGHOT_SECTION_HOT_BIT, (unsigned long *)&ms->hot_map);
+ set_bit(PGDAT_KMIGRATED_ACTIVATE, &page_pgdat(page)->flags);
+ }
+out:
+ folio_put(folio);
+ return 0;
+}
+
+static int pghot_get_hotness(unsigned long pfn, int *nid, int *freq,
+ unsigned long *time)
+{
+ phi_t *phi, *hot_map;
+ struct mem_section *ms;
+
+ ms = __pfn_to_section(pfn);
+ if (!ms || !ms->hot_map)
+ return -EINVAL;
+
+ hot_map = (phi_t *)(((unsigned long)(ms->hot_map)) & ~PGHOT_SECTION_HOT_MASK);
+ phi = &hot_map[pfn % PAGES_PER_SECTION];
+
+ return pghot_get_record(phi, nid, freq, time);
+}
+
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+ int src_nid)
+{
+ struct mem_cgroup *cur_memcg = NULL;
+ int cur_nid = NUMA_NO_NODE;
+ LIST_HEAD(migrate_list);
+ int batch_count = 0;
+ struct folio *folio;
+ struct page *page;
+ unsigned long pfn;
+
+ pfn = start_pfn;
+ do {
+ int nid = NUMA_NO_NODE, nr = 1;
+ struct mem_cgroup *memcg;
+ unsigned long time = 0;
+ int freq = 0;
+
+ if (!pfn_valid(pfn))
+ goto out_next;
+
+ page = pfn_to_online_page(pfn);
+ if (!page)
+ goto out_next;
+
+ folio = page_folio(page);
+ if (!folio_try_get(folio))
+ goto out_next;
+
+ if (unlikely(page_folio(page) != folio)) {
+ folio_put(folio);
+ goto out_next;
+ }
+
+ nr = folio_nr_pages(folio);
+ if (folio_nid(folio) != src_nid) {
+ folio_put(folio);
+ goto out_next;
+ }
+
+ if (!folio_test_lru(folio)) {
+ folio_put(folio);
+ goto out_next;
+ }
+
+ if (pghot_get_hotness(pfn, &nid, &freq, &time)) {
+ folio_put(folio);
+ goto out_next;
+ }
+
+ if (nid == NUMA_NO_NODE)
+ nid = pghot_target_nid;
+
+ if (folio_nid(folio) == nid) {
+ folio_put(folio);
+ goto out_next;
+ }
+
+ if (migrate_misplaced_folio_prepare(folio, NULL, nid)) {
+ folio_put(folio);
+ goto out_next;
+ }
+
+ memcg = folio_memcg(folio);
+ if (cur_nid == NUMA_NO_NODE) {
+ cur_nid = nid;
+ cur_memcg = memcg;
+ }
+
+ /* If NID or memcg changed, flush the previous batch first */
+ if (cur_nid != nid || cur_memcg != memcg) {
+ if (!list_empty(&migrate_list))
+ promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+ cur_nid = nid;
+ cur_memcg = memcg;
+ batch_count = 0;
+ cond_resched();
+ }
+
+ list_add(&folio->lru, &migrate_list);
+ folio_put(folio);
+
+ if (++batch_count > kmigrated_batch_nr) {
+ promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+ batch_count = 0;
+ cond_resched();
+ }
+out_next:
+ pfn += nr;
+ } while (pfn < end_pfn);
+ if (!list_empty(&migrate_list))
+ promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+}
+
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+ unsigned long section_nr, s_begin, start_pfn;
+ struct mem_section *ms;
+ int nid;
+
+ clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+ s_begin = next_present_section_nr(-1);
+ for_each_present_section_nr(s_begin, section_nr) {
+ start_pfn = section_nr_to_pfn(section_nr);
+ ms = __nr_to_section(section_nr);
+
+ if (!pfn_valid(start_pfn))
+ continue;
+
+ nid = pfn_to_nid(start_pfn);
+ if (node_is_toptier(nid) || nid != pgdat->node_id)
+ continue;
+
+ if (!test_and_clear_bit(PGHOT_SECTION_HOT_BIT, (unsigned long *)&ms->hot_map))
+ continue;
+
+ kmigrated_walk_zone(start_pfn, start_pfn + PAGES_PER_SECTION,
+ pgdat->node_id);
+ }
+}
+
+static inline bool kmigrated_work_requested(pg_data_t *pgdat)
+{
+ return test_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+}
+
+/*
+ * Per-node kthread that iterates over its PFNs and migrates the
+ * pages that have been marked for migration.
+ */
+static int kmigrated(void *p)
+{
+ pg_data_t *pgdat = p;
+
+ while (!kthread_should_stop()) {
+ long timeout = msecs_to_jiffies(READ_ONCE(kmigrated_sleep_ms));
+
+ if (wait_event_timeout(pgdat->kmigrated_wait, kmigrated_work_requested(pgdat),
+ timeout))
+ kmigrated_do_work(pgdat);
+ }
+ return 0;
+}
+
+static int kmigrated_run(int nid)
+{
+ pg_data_t *pgdat = NODE_DATA(nid);
+ int ret;
+
+ if (!pgdat->kmigrated) {
+ pgdat->kmigrated = kthread_create_on_node(kmigrated, pgdat, nid,
+ "kmigrated%d", nid);
+ if (IS_ERR(pgdat->kmigrated)) {
+ ret = PTR_ERR(pgdat->kmigrated);
+ pgdat->kmigrated = NULL;
+ pr_err("Failed to start kmigrated%d, ret %d\n", nid, ret);
+ return ret;
+ }
+ pr_info("pghot: Started kmigrated thread for node %d\n", nid);
+ }
+ wake_up_process(pgdat->kmigrated);
+ return 0;
+}
+
+static void pghot_free_hot_map(struct mem_section *ms)
+{
+ kfree((void *)((unsigned long)ms->hot_map & ~PGHOT_SECTION_HOT_MASK));
+ ms->hot_map = NULL;
+}
+
+static int pghot_alloc_hot_map(struct mem_section *ms, int nid)
+{
+ ms->hot_map = kcalloc_node(PAGES_PER_SECTION, PGHOT_RECORD_SIZE, GFP_KERNEL,
+ nid);
+ if (!ms->hot_map)
+ return -ENOMEM;
+ return 0;
+}
+
+static void pghot_offline_sec_hotmap(unsigned long start_pfn,
+ unsigned long nr_pages)
+{
+ unsigned long start, end, pfn;
+ struct mem_section *ms;
+
+ start = SECTION_ALIGN_DOWN(start_pfn);
+ end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+ for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+ ms = __pfn_to_section(pfn);
+ if (!ms || !ms->hot_map)
+ continue;
+
+ pghot_free_hot_map(ms);
+ }
+}
+
+static int pghot_online_sec_hotmap(unsigned long start_pfn,
+ unsigned long nr_pages)
+{
+ int nid = pfn_to_nid(start_pfn);
+ unsigned long start, end, pfn;
+ struct mem_section *ms;
+ int fail = 0;
+
+ start = SECTION_ALIGN_DOWN(start_pfn);
+ end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+ for (pfn = start; !fail && pfn < end; pfn += PAGES_PER_SECTION) {
+ ms = __pfn_to_section(pfn);
+ if (!ms || ms->hot_map)
+ continue;
+
+ fail = pghot_alloc_hot_map(ms, nid);
+ }
+
+ if (!fail)
+ return 0;
+
+ /* rollback */
+ end = pfn - PAGES_PER_SECTION;
+ for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+ ms = __pfn_to_section(pfn);
+ if (ms && ms->hot_map)
+ pghot_free_hot_map(ms);
+ }
+ return -ENOMEM;
+}
+
+static int pghot_memhp_callback(struct notifier_block *self,
+ unsigned long action, void *arg)
+{
+ struct memory_notify *mn = arg;
+ int ret = 0;
+
+ switch (action) {
+ case MEM_GOING_ONLINE:
+ ret = pghot_online_sec_hotmap(mn->start_pfn, mn->nr_pages);
+ break;
+ case MEM_OFFLINE:
+ case MEM_CANCEL_ONLINE:
+ pghot_offline_sec_hotmap(mn->start_pfn, mn->nr_pages);
+ break;
+ }
+
+ return notifier_from_errno(ret);
+}
+
+static struct notifier_block pghot_mem_notifier = {
+ .notifier_call = pghot_memhp_callback,
+ .priority = DEFAULT_CALLBACK_PRI,
+};
+
+static void pghot_destroy_hot_map(void)
+{
+ unsigned long section_nr, s_begin;
+ struct mem_section *ms;
+
+ s_begin = next_present_section_nr(-1);
+ for_each_present_section_nr(s_begin, section_nr) {
+ ms = __nr_to_section(section_nr);
+ pghot_free_hot_map(ms);
+ }
+
+ unregister_memory_notifier(&pghot_mem_notifier);
+}
+
+static int pghot_setup_hot_map(void)
+{
+ unsigned long section_nr, s_begin, start_pfn;
+ struct mem_section *ms;
+ int nid, ret;
+
+ ret = register_memory_notifier(&pghot_mem_notifier);
+ if (ret)
+ return ret;
+
+ s_begin = next_present_section_nr(-1);
+ for_each_present_section_nr(s_begin, section_nr) {
+ ms = __nr_to_section(section_nr);
+ start_pfn = section_nr_to_pfn(section_nr);
+ nid = pfn_to_nid(start_pfn);
+
+ if (node_is_toptier(nid) || !pfn_valid(start_pfn))
+ continue;
+
+ if (pghot_alloc_hot_map(ms, nid))
+ goto out_free_hot_map;
+ }
+ return 0;
+
+out_free_hot_map:
+ pghot_destroy_hot_map();
+ return -ENOMEM;
+}
+
+static int __init pghot_init(void)
+{
+ pg_data_t *pgdat;
+ int nid, ret;
+
+ ret = pghot_setup_hot_map();
+ if (ret)
+ return ret;
+
+ for_each_node_state(nid, N_MEMORY) {
+ if (node_is_toptier(nid))
+ continue;
+
+ ret = kmigrated_run(nid);
+ if (ret)
+ goto out_stop_kthread;
+ }
+ register_sysctl_init("vm", pghot_sysctls);
+ pghot_debug_init();
+
+ kmigrated_started = true;
+ return 0;
+
+out_stop_kthread:
+ for_each_node_state(nid, N_MEMORY) {
+ pgdat = NODE_DATA(nid);
+ if (pgdat->kmigrated) {
+ kthread_stop(pgdat->kmigrated);
+ pgdat->kmigrated = NULL;
+ }
+ }
+ pghot_destroy_hot_map();
+ return ret;
+}
+
+late_initcall_sync(pghot_init)
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..4064ead568cc 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1489,6 +1489,11 @@ const char * const vmstat_text[] = {
[I(KSTACK_REST)] = "kstack_rest",
#endif
#endif
+#ifdef CONFIG_PGHOT
+ [I(PGHOT_RECORDED_ACCESSES)] = "pghot_recorded_accesses",
+ [I(PGHOT_RECORDED_HINTFAULTS)] = "pghot_recorded_hintfaults",
+ [I(PGHOT_RECORDED_HWHINTS)] = "pghot_recorded_hwhints",
+#endif /* CONFIG_PGHOT */
#undef I
#endif /* CONFIG_VM_EVENT_COUNTERS */
};
--
2.34.1