[PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure

[PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure

[Bharata B Rao]

Hi,

This patchset introduces pghot, a subsystem for hot page tracking and
promotion. It collects memory access information from multiple sources,
classifies hot pages resident in lower-tier memory, and promotes them to
faster tiers via a per-lower-tier-node kernel thread (kmigrated).

The main changes in v8 are a rework of the per-section hotness map to be
RCU-protected, a move of the pghot tunables from debugfs to sysctl, and a
rework of the AMD IBS Memory Profiler driver that now uses a dedicated system
interrupt vector instead of an NMI handler. The IBS driver is also split
into core infrastructure and a separate patch that adds the runtime
controls and arms the profiler. Also there are a few fixes to the hint
faults source.

Goals of this patchset:

- Unify hot page detection from multiple sources like hint faults,
  page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel
  thread.
- Move promotion rate-limiting and related logic used by numa_balancing=2
  (NUMAB2, the current NUMA balancing-based promotion) from the scheduler
  to pghot for broader reuse.

Currently, multiple kernel subsystems detect page accesses independently.
This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Performance summary
===================
All results are on 3-node tiered systems (DRAM top tier + CPU-less CXL
lower tier). Speedups below are normalized to the base kernel with no
tiering (NUMAB=0). Columns compare mainline hint-fault tiering
(base NUMAB=2) against pghot in two modes: hint-fault driven (pghot-hf,
NUMAB=2) and HW-hint driven via the AMD IBS memory profiler (pghot-hw,
NUMAB=0, no NUMA scanning). Single run per config (abench is avg of 3).

Benchmark      Metric (higher=better)   base-NUMAB2  pghot-hf  pghot-hw
-------------------------------------------------------------------------
NAS BT (MPI)   Mop/s total                  2.26x     2.26x     2.13x
Graph500 BFS   harmonic-mean TEPS           2.64x     2.49x     2.68x
llama.cpp      decode tok/s (tg128)         1.09x     1.41x     1.21x
Redis+memtier  ops/sec                      1.05x     1.05x     1.00x
Microbench     completion time (1/t)        2.41x     2.40x     2.35x
-------------------------------------------------------------------------
(baseline = base kernel, no tiering = 1.00x)

Summary: pghot hint-faults reproduces mainline NUMAB=2 with no
regression on every workload; pghot HW-hints/IBS matches it mostly.

- NAS BT (MPI, Class F modified, 16 ranks): full footprint pre-staged
  on the CXL lower tier, then measured cold; isolates hot-page promotion.
- Graph500 (reference BFS, SCALE=28, edgefactor=16, 128 ranks):
  working set on the CXL node; figure of merit is harmonic-mean TEPS.
- llama.cpp (llama-bench, Mixtral-8x22B Q4_K_M): ~1/3 of the model
  demoted to CXL under memory pressure; decode (tg128) is tier-sensitive.
- Redis + memtier (~64 GB dataset, 62.2M x 1 KB keys): dataset
  migrated to CXL, GET-heavy traffic over 50% of the keyspace.
- Microbenchmark: 64 threads, random 4K access over 8G on
  the CXL node; latency-bound.

Detailed per-benchmark tables (throughput/latency + vmstat and pghot
promotion counters) will be sent as replies to this patchset post.

How pghot works
===============
- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded
  access is tracked in precision mode.
- These hotness parameters are maintained in a per-PFN hotness record
  within a per-section map hanging off the existing mem_section data
  structure. The map is RCU-protected and also carries a section-level
  "hot" flag used to gate scanning by kmigrated.
  - In default mode, one byte (u8) is used for the hotness record. 5 bits
    store time using a bucketing scheme representing up to 4s with
    HZ=1000. The default toptier NID (0) is used as the promotion target
    and can be changed via the vm.pghot_target_nid sysctl.
  - In precision mode, 4 bytes (u32) are used for each hotness record.
    14 bits store time, representing around 16s with HZ=1000, and the
    accessing NID is stored per-PFN as the promotion target.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked ready for migration using the ready
  bit (MSB of the hotness record in both modes).
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of
  lower-tier nodes, checking the migration-ready bit to perform batched
  migrations. The scan interval and batch size are configurable via
  debugfs tunables.

Memory overhead
===============
Default mode: 1 byte per lower-tier PFN. For 1TB of lower-tier memory
this amounts to 256MB overhead (assuming 4K pages).

Precision mode: 4 bytes per lower-tier PFN. For 1TB of lower-tier memory
this amounts to 1GB overhead.

Bit layout of hotness record
============================
Default mode
- Bits 0-1: Frequency (2 bits, 4 access samples)
- Bits 2-6: Bucketed time (5 bits, up to 4s with HZ=1000)
- Bit 7:    Migration ready bit

Precision mode
- Bits 0-9:   Target NID (10 bits)
- Bits 10-12: Frequency (3 bits, 8 access samples)
- Bits 13-26: Time (14 bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31:     Migration ready bit

Tunables
========
sysctl (/proc/sys/vm/):
- pghot_enabled_sources      - bitmask of hotness sources (0x1 hint
                               faults, 0x2 hardware hints)
- pghot_target_nid           - toptier target NID (default mode)
- pghot_freq_threshold       - accesses before a page is promotable
- pghot_promote_freq_window_ms - access-frequency counting window
- pghot_promote_rate_limit_MBps - per-target promotion rate limit
  (numa_balancing_promote_rate_limit_MBps is retained as a compat alias)

debugfs (/sys/kernel/debug/pghot/):
- kmigrated_sleep_ms, kmigrated_batch_nr

Potential hotness sources
=========================
1. NUMA Balancing (NUMAB2, Tiering mode) - included in this patchset.
2. AMD IBS Memory Profiler: HW based access profiler - included in this
   patchset.
3. klruscand - PTE-A bit scanning built on MGLRU's walk helpers - was
   showcased in previous versions but not part of this patchset.
4. folio_mark_accessed() - Page cache access tracking (unmapped page
   cache pages) - was showcased in previous versions but not part of
   this patchset.

Changes in v8
=============
pghot-core
- Augmented documentation with per-source information
- Reworked the per-section hotness map (mem_section->hot_map) into an
  RCU-protected structure (struct pghot_hot_map) that holds the per-PFN
  records and section-level flags, for safe concurrent access and
  teardown. The section "hot" flag now lives in this structure instead of
  the pointer's LSB.
- Cleanup and fixes in pghot_update_record().
- Moved pghot tunables enabled_sources, target_nid and freq_threshold
  from debugfs to sysctl (/proc/sys/vm/). Only kmigrated_sleep_ms and
  kmigrated_batch_nr remain in debugfs.
- Kmigrated walk: Protect folio_memcg() under RCU readlock

pghot-hintfaults
- Retained kernel.numa_balancing_promote_rate_limit_MBps for compatibility
- Filter shared executable pages at NUMA hint fault time in
  do_numa_page()/do_huge_pmd_numa_page(), since the VMA is not available
  to kmigrated's batched migration from non-process context.
- Fixed NUMAB2 to continue feeding to task_numa_fault() so that scan
  period updates happens for pghot-NUMAB2.
- Now hint faults source is enabled by default in pghot so that setting
  kernel.numa_balancing=2 is enough for tiering mode to get activated
  as before.
- To avoid a regression for existing configs, default PGHOT to y when
  NUMA_BALANCING is enabled and default NUMA_BALANCING_TIERING to y.
  Since the tiering promotion engine now lives in pghot, an existing
  CONFIG_NUMA_BALANCING=y config would otherwise silently lose the ability
  to use kernel.numa_balancing=2 (writes returning -EOPNOTSUPP) after an
  oldconfig upgrade.
  Note: PGHOT depends on SPARSEMEM, so a NUMA_BALANCING=y && !SPARSEMEM
  (e.g. FLATMEM) config can no longer use kernel.numa_balancing=2.

pghot-hwhints
- Split the IBS Memory Profiler driver into core infrastructure and a
  separate patch that adds runtime controls (debugfs/sysfs and the
  ibs-memprof= cmdline parameter) and arms the profiler.
- Changed IBS Memory Profiler source to use regular interrupts rather
  than NMI. Unlike the primary IBS which needs to obtain samples from
  kernel and interrupt contexts for performance profiling, with memory access
  profiling only user space samples are obtained. Hence NMI isn't required.
  Also this simplifies the code by getting rid of irq_work hander.
- Fixed lost wakeup issue in work handler that drains the samples by
  draining samples in a loop until the ring buffer is observably empty.
- Drain and discard the samples of the CPU that is going offline.
- Fix the issue where the work item queued for a dying CPU can execute
  on a different online CPU leading to draining of that CPU's ring.
- Added ibs-memprof kernel cmdline option to enable hwhint source at boot
  time itself.
- Added vmstat counter hwhint_dropped_events that provides the count of
  those samples which were dropped due to sample buffer overrun.
- Added a sysfs tunable /sys/devices/system/cpu/ibs-mprof/enabled to
  dynamically enable or disable IBS Memory Profiler.
- Added debugfs tunables (l3miss-only, period, lat-filter, lat-thresh)
  to support configuration changes to IBS Memory profiler.

Compile test summary
====================
The series was compile-tested across the full range of relevant Kconfig
combinations on x86_64 (defconfig-based) and cross-compiled on arm64.
Combinations of PGHOT, PGHOT_PRECISE, NUMA_BALANCING,
NUMA_BALANCING_TIERING, NUMA_MIGRATION, HWMEM_PROFILER/AMD_IBS_MEMPROF,
DEBUG_FS and SYSCTL were verified, including pghot disabled, pghot
standalone (no NUMA balancing), NUMA balancing mode 1 and mode 2 (tiering),
precise mode, the AMD IBS memory profiler, NUMA_MIGRATION=n, and pghot with
DEBUG_FS and/or SYSCTL disabled. Kconfig guards were confirmed
(NUMA_BALANCING_TIERING without PGHOT, and HWMEM_PROFILER without a
selecting driver, are dropped). Since PGHOT defaults to y when
NUMA_BALANCING is set and NUMA_BALANCING_TIERING defaults to y, an existing
config with NUMA_BALANCING=y but no pghot symbols resolves, via
olddefconfig, to PGHOT=y and NUMA_BALANCING_TIERING=y and builds cleanly,
while NUMA_BALANCING=n leaves PGHOT off. allnoconfig, tinyconfig,
allyesconfig and allmodconfig were also built. All configurations build with
no new warnings.

This v8 patchset applies on top of upstream commit 8cdeaa50eae8 and can be
fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-v8

v7: https://lore.kernel.org/linux-mm/20260504060924.344313-1-bharata@amd.com/
v6: https://lore.kernel.org/linux-mm/20260323095104.238982-1-bharata@amd.com/
v5: https://lore.kernel.org/linux-mm/20260129144043.231636-1-bharata@amd.com/
v4: https://lore.kernel.org/linux-mm/20251206101423.5004-1-bharata@amd.com/
v3: https://lore.kernel.org/linux-mm/20251110052343.208768-1-bharata@amd.com/
v2: https://lore.kernel.org/linux-mm/20250910144653.212066-1-bharata@amd.com/
v1: https://lore.kernel.org/linux-mm/20250814134826.154003-1-bharata@amd.com/
v0: https://lore.kernel.org/linux-mm/20250306054532.221138-1-bharata@amd.com/

base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda

IBS Memory Profiler part of this patchset depends on the patchset that
increases the number of APIC EILVT registers -
https://lore.kernel.org/lkml/cover.1775019269.git.naveen@kernel.org/

AI assistance was taken to run and summarize the benchmark results.

Bharata B Rao (7):
  mm: migrate: Allow misplaced migration without VMA
  mm: Hot page tracking and promotion - pghot
  mm: pghot: Precision mode for pghot
  mm: sched: move NUMA balancing tiering promotion to pghot
  x86/ibs: Move IBS caps definitions into its own header
  x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler
  x86/mm/ibs: Add runtime controls for IBS memprofiler

Gregory Price (1):
  mm: migrate: Add promote_misplaced_memcg_folios()

 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/admin-guide/mm/index.rst        |   1 +
 Documentation/admin-guide/mm/pghot.rst        | 196 ++++
 arch/x86/Kconfig                              |  16 +
 arch/x86/entry/entry_fred.c                   |   1 +
 arch/x86/include/asm/hardirq.h                |   3 +
 arch/x86/include/asm/ibs-caps.h               |  93 ++
 arch/x86/include/asm/ibs-mprof.h              |  61 ++
 arch/x86/include/asm/idtentry.h               |   6 +
 arch/x86/include/asm/irq_vectors.h            |   4 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/perf_event.h             |  81 +-
 arch/x86/kernel/idt.c                         |   3 +
 arch/x86/kernel/irq.c                         |   3 +
 arch/x86/mm/Makefile                          |   1 +
 arch/x86/mm/ibs-mprof.c                       | 704 ++++++++++++++
 include/linux/cpuhotplug.h                    |   1 +
 include/linux/migrate.h                       |   9 +-
 include/linux/mm.h                            |  35 +-
 include/linux/mmzone.h                        |  25 +-
 include/linux/pghot.h                         | 140 +++
 include/linux/vm_event_item.h                 |  12 +
 init/Kconfig                                  |  14 +
 kernel/sched/core.c                           |   7 +
 kernel/sched/debug.c                          |   1 -
 kernel/sched/fair.c                           | 171 +---
 kernel/sched/sched.h                          |   1 -
 mm/Kconfig                                    |  34 +
 mm/Makefile                                   |   6 +
 mm/huge_memory.c                              |  32 +-
 mm/memcontrol.c                               |   6 +-
 mm/memory-tiers.c                             |  15 +-
 mm/memory.c                                   |  36 +-
 mm/mempolicy.c                                |   3 -
 mm/migrate.c                                  |  99 +-
 mm/mm_init.c                                  |  10 +
 mm/pghot-default.c                            |  76 ++
 mm/pghot-precise.c                            |  77 ++
 mm/pghot.c                                    | 891 ++++++++++++++++++
 mm/vmstat.c                                   |  14 +-
 40 files changed, 2593 insertions(+), 308 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 arch/x86/include/asm/ibs-caps.h
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot.c

-- 
2.34.1

[PATCH v8 1/8] mm: migrate: Allow misplaced migration without VMA

[Bharata B Rao]

We want isolation of misplaced folios to work in contexts
where VMA isn't available, typically when performing migrations
from a kernel thread context. In order to prepare for that,
allow migrate_misplaced_folio_prepare() to be called with
a NULL VMA.

When migrate_misplaced_folio_prepare() is called with non-NULL
VMA, it will check if the folio is mapped shared and that requires
holding PTL lock. This path isn't taken when the function is
invoked with NULL VMA (migration outside of process context).
Therefore, when VMA == NULL, migrate_misplaced_folio_prepare()
does not require the caller to hold the PTL.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 mm/migrate.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index d9b23909d716..ab7227376757 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2674,7 +2674,12 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. Must be called with the PTL still held if called with a non-NULL
+ * vma.
+ *
+ * When called with a NULL vma (e.g., kernel thread initiated migration),
+ * migrate_misplaced_folio_prepare() will allow shared executable folios
+ * to be migrated.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2691,7 +2696,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if (vma && (vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
 			return -EACCES;
 
 		/*
-- 
2.34.1

[PATCH v8 2/8] mm: migrate: Add promote_misplaced_memcg_folios()

[Bharata B Rao]

From: Gregory Price <gourry@gourry.net>

Tiered memory systems often require migrating multiple folios at once.
Currently, migrate_misplaced_folio() handles only one folio per call,
which is inefficient for batch operations. This patch introduces
promote_misplaced_memcg_folios(), a batch variant that leverages
migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using
migrate_misplaced_folio_prepare(). Additionally all the folios in the
isolated list must belong to the same memcg. On return, the folio list
will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

Signed-off-by: Gregory Price <gourry@gourry.net>
[Rewrote commit description, memcg awareness]
Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 include/linux/migrate.h |  5 ++++
 mm/migrate.c            | 58 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index d5af2b7f577b..d136612eef9d 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -111,6 +111,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -121,6 +122,10 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
diff --git a/mm/migrate.c b/mm/migrate.c
index ab7227376757..58a8a0cf6fa3 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2773,4 +2773,62 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/**
+ * promote_misplaced_memcg_folios() - Batch variant of migrate_misplaced_folio
+ * Attempts to promote a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-promoted.
+ * @node: The NUMA node ID to where the folios should be promoted.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folios. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.
+ *
+ * This function will un-isolate the folios, drop the elevated reference
+ * and remove them from the list before returning. This should be called
+ * only for batched promotion of hot pages from lower tier nodes.
+ *
+ * Return: 0 on success and -EAGAIN on failure or partial promotion.
+ *         On return, @folio_list will be empty regardless of success/failure.
+ */
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+	struct mem_cgroup *memcg = NULL;
+	unsigned int nr_succeeded = 0;
+	struct folio *first;
+	int nr_remaining;
+
+	if (list_empty(folio_list))
+		return 0;
+
+	first = list_first_entry(folio_list, struct folio, lru);
+#ifdef CONFIG_DEBUG_VM
+	{
+		struct folio *f;
+
+		list_for_each_entry(f, folio_list, lru)
+			VM_WARN_ON_ONCE(folio_memcg(f) != folio_memcg(first));
+	}
+#endif
+	memcg = get_mem_cgroup_from_folio(first);
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
+				 PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+
+	mem_cgroup_put(memcg);
+	WARN_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
-- 
2.34.1

[PATCH v8 3/8] mm: Hot page tracking and promotion - pghot

[Bharata B Rao]

pghot is a subsystem that collects memory access information from
multiple sources, classifies hot pages resident in lower-tier memory,
and promotes them to faster tiers. It stores per-PFN hotness metadata
and performs asynchronous, batched promotion via a per-lower-tier-node
kernel thread (kmigrated).

This change introduces the default (compact) mode of pghot:

- Per-PFN hotness record (phi_t = u8) embedded via mem_section:
  - 2 bits: access frequency (4 levels)
  - 5 bits: time bucket (≈4s window with HZ=1000, bucketed jiffies)
  - 1 bit : migration-ready flag (MSB)

- Event recording API:
  int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
  @pfn: The PFN of the memory accessed
  @nid: The accessing NUMA node ID
  @src: The temperature source (subsystem) that generated the
        access info
  @time: The access time in jiffies
  - Sources (e.g., NUMA hint faults, HW hints) call this to report
    accesses.
  - In default mode, the nid is not stored/used for targeting;
    promotion goes to a configurable toptier node (pghot_target_nid).

- Promotion engine:
  - One kmigrated thread per lower-tier node.
  - Scans only sections whose "hot" flag was raised, iterates PFNs,
    and batches candidates by destination node.
  - Uses promote_misplaced_memcg_folios() to move batched folios.

- Tunables & stats:
  - debugfs: kmigrated_sleep_ms, kmigrated_batch_nr
  - sysctl : vm.pghot_promote_freq_window_ms, vm.pghot_freq_threshold,
	     vm.pghot_enabled_sources, vm.pghot_target_nid
  - vmstat : pghot_recorded_accesses, pghot_recorded_hintfaults,
             pghot_recorded_hwhints

Memory overhead
---------------
Default mode uses 1 byte of hotness metadata per PFN on lower-tier
nodes.

Behavior & policy
-----------------
- Default mode promotion target:
  The nid passed by sources is not stored; hot pages promote to
  pghot_target_nid (toptier). Precision mode (added later in the
  series) changes this.

- Record consumption:
  kmigrated consumes (clears) the "migration-ready" bit before
  attempting isolation. Additionally the hotness record is reset.
  If isolation/migration fails, the folio is not re-queued automatically;
  subsequent accesses will re-arm it. This avoids retry storms and
  keeps batching stable.

- Wakeups:
  kmigrated wakeups are intentionally timeout-driven. We set
  the per-pgdat "activate" flag on access, and kmigrated checks this
  flag on its next sleep interval. This keeps the first cut simple
  and avoids potential wake storms; active wakeups can be considered
  in a follow-up.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 Documentation/admin-guide/mm/index.rst |   1 +
 Documentation/admin-guide/mm/pghot.rst |  89 +++
 include/linux/migrate.h                |   4 +-
 include/linux/mmzone.h                 |  21 +
 include/linux/pghot.h                  | 103 ++++
 include/linux/vm_event_item.h          |   5 +
 mm/Kconfig                             |  14 +
 mm/Makefile                            |   1 +
 mm/migrate.c                           |  16 +-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  76 +++
 mm/pghot.c                             | 725 +++++++++++++++++++++++++
 mm/vmstat.c                            |   5 +
 13 files changed, 1063 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot.c

diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst
index bbb563cba5d2..4d6810b02365 100644
--- a/Documentation/admin-guide/mm/index.rst
+++ b/Documentation/admin-guide/mm/index.rst
@@ -43,3 +43,4 @@ the Linux memory management.
    userfaultfd
    zswap
    kho
+   pghot
diff --git a/Documentation/admin-guide/mm/pghot.rst b/Documentation/admin-guide/mm/pghot.rst
new file mode 100644
index 000000000000..0edbe0082816
--- /dev/null
+++ b/Documentation/admin-guide/mm/pghot.rst
@@ -0,0 +1,89 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+================================================
+pghot: Hot Page Tracking and Promotion Subsystem
+================================================
+
+Overview
+========
+The pghot subsystem tracks frequently accessed pages in lower-tier memory and
+promotes them to faster tiers. It uses per-PFN hotness metadata and asynchronous
+migration via per-node kernel threads (kmigrated).
+
+This document describes tunables available via **debugfs** and **sysctl** for
+pghot.
+
+Debugfs Interface
+=================
+Path: /sys/kernel/debug/pghot/
+
+1. **kmigrated_sleep_ms**
+   - Sleep interval (ms) for kmigrated thread between scans.
+   - Default: 100
+
+2. **kmigrated_batch_nr**
+   - Maximum number of folios migrated in one batch.
+   - Default: 512
+
+Sysctl Interface
+================
+1. pghot_enabled_sources
+
+Path: /proc/sys/vm/pghot_enabled_sources
+
+- Bitmask to enable/disable hotness sources.
+- Bits:
+  - 0: Hint faults (value 0x1)
+  - 1: Hardware hints (value 0x2)
+- Default: 0 (disabled)
+- Example:
+  # echo 0x3 > /proc/sys/vm/pghot_enabled_sources
+  Enables both hint faults and hwhints sources
+
+2. pghot_target_nid
+
+Path: /proc/sys/vm/pghot_target_nid
+
+- Toptier NUMA node ID to which hot pages should be promoted when source
+  does not provide nid. Used when hotness source can't provide accessing
+  NID or when the tracking mode is default.
+- Default: 0
+- Example:
+  # echo 1 > /proc/sys/vm/pghot_target_nid
+
+3. pghot_freq_threshold
+
+Path: /proc/sys/vm/pghot_freq_threshold
+
+- Minimum access frequency before a page is marked ready for promotion.
+  Range is 1 to 3 in default mode.
+- Default: 2
+- Example:
+  # sysctl vm.pghot_freq_threshold=1
+
+4. pghot_promote_freq_window_ms
+
+Path: /proc/sys/vm/pghot_promote_freq_window_ms
+
+- Controls the time window (in ms) for counting access frequency. A page is
+  considered hot only when **pghot_freq_threshold** number of accesses occur
+  with this time period.
+- Default: 3000 (3 seconds)
+- Example:
+  # sysctl vm.pghot_promote_freq_window_ms=3000
+
+pghot Vmstat Counters
+=====================
+Following vmstat counters provide some stats about pghot subsystem.
+
+Path: /proc/vmstat
+
+1. **pghot_recorded_accesses**
+   - Number of total hot page accesses recorded by pghot.
+
+2. **pghot_recorded_hintfaults**
+   - Number of recorded accesses reported by NUMA Balancing based
+     hotness source.
+
+3. **pghot_recorded_hwhints**
+   - Number of recorded accesses reported by hwhints source.
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index d136612eef9d..53bae80d11ae 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -107,7 +107,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 
 #endif /* CONFIG_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
@@ -126,7 +126,7 @@ static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, i
 {
 	return -EAGAIN; /* can't migrate now */
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
 
 #ifdef CONFIG_MIGRATION
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ca2712187147..11c547a51f43 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1156,6 +1156,7 @@ enum pgdat_flags {
 					 * many pages under writeback
 					 */
 	PGDAT_RECLAIM_LOCKED,		/* prevents concurrent reclaim */
+	PGDAT_KMIGRATED_ACTIVATE,	/* activates kmigrated */
 };
 
 enum zone_flags {
@@ -1598,6 +1599,10 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kmigrated;
+	wait_queue_head_t kmigrated_wait;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
@@ -1992,6 +1997,7 @@ struct mem_section_usage {
 
 struct page;
 struct page_ext;
+struct pghot_hot_map;
 struct mem_section {
 	/*
 	 * This is, logically, a pointer to an array of struct
@@ -2008,12 +2014,27 @@ struct mem_section {
 	unsigned long section_mem_map;
 
 	struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+	/*
+	 * RCU-protected pointer to this section's struct pghot_hot_map,
+	 * which holds the per-PFN hotness records and the section-level
+	 * flags.
+	 */
+	struct pghot_hot_map __rcu *hot_map;
+#endif
 #ifdef CONFIG_PAGE_EXTENSION
 	/*
 	 * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
 	 * section. (see page_ext.h about this.)
 	 */
 	struct page_ext *page_ext;
+#endif
+	/*
+	 * Padding to maintain consistent mem_section size when exactly
+	 * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+	 * optimal alignment regardless of configuration.
+	 */
+#if (defined(CONFIG_PGHOT) ^ defined(CONFIG_PAGE_EXTENSION))
 	unsigned long pad;
 #endif
 	/*
diff --git a/include/linux/pghot.h b/include/linux/pghot.h
new file mode 100644
index 000000000000..7b85d717f410
--- /dev/null
+++ b/include/linux/pghot.h
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_PGHOT_H
+#define _LINUX_PGHOT_H
+
+/* Page hotness temperature sources */
+enum pghot_src {
+	PGHOT_HINTFAULTS = 0,
+	PGHOT_HWHINTS,
+	PGHOT_SRC_MAX
+};
+
+#ifdef CONFIG_PGHOT
+#include <linux/static_key.h>
+#include <linux/mutex.h>
+
+extern unsigned int sysctl_pghot_target_nid;
+extern unsigned int sysctl_pghot_freq_window;
+extern unsigned int sysctl_pghot_freq_threshold;
+extern struct mutex pghot_tunables_lock;
+
+DECLARE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
+
+#define PGHOT_HINTFAULTS_ENABLED	BIT(PGHOT_HINTFAULTS)
+#define PGHOT_HWHINTS_ENABLED		BIT(PGHOT_HWHINTS)
+#define PGHOT_SRC_ENABLED_MASK		GENMASK(PGHOT_SRC_MAX - 1, 0)
+
+#define PGHOT_FREQ_THRESHOLD_DEFAULT	2
+
+#define KMIGRATED_SLEEP_MS_MIN		50
+#define KMIGRATED_SLEEP_MS_DEFAULT	100
+#define KMIGRATED_SLEEP_MS_MAX		1000
+
+#define KMIGRATED_BATCH_NR_MIN		64
+#define KMIGRATED_BATCH_NR_DEFAULT	512
+#define KMIGRATED_BATCH_NR_MAX		1024
+
+#define PGHOT_DEFAULT_NODE		0
+
+#define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT	(3 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-6 are used to store frequency and time.
+ * Bit 7 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		7
+
+#define PGHOT_FREQ_WIDTH		2
+/* Bucketed time is stored in 5 bits which can represent up to 3.9s with HZ=1000 */
+#define PGHOT_TIME_BUCKETS_SHIFT	7
+#define PGHOT_TIME_WIDTH		5
+#define PGHOT_NID_WIDTH			10
+
+#define PGHOT_FREQ_SHIFT		0
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+#define PGHOT_TIME_BUCKETS_MASK		(PGHOT_TIME_MASK << PGHOT_TIME_BUCKETS_SHIFT)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+#define PGHOT_FREQ_WINDOW_MAX		PGHOT_TIME_BUCKETS_MASK
+
+typedef u8 phi_t;
+
+#define PGHOT_RECORD_SIZE		sizeof(phi_t)
+
+#define PGHOT_SECTION_HOT_BIT		0
+
+/**
+ * struct pghot_hot_map - per-section hotness map
+ * @rcu:   used to free the map after an RCU grace period via kfree_rcu().
+ * @flags: section-level flags; currently only PGHOT_SECTION_HOT_BIT, set
+ *         when the section contains at least one page classified as hot
+ *         and consumed by the kmigrated thread.
+ * @phi:   per-PFN hotness records, PAGES_PER_SECTION entries.
+ *
+ * Referenced from struct mem_section through an RCU-protected pointer
+ * (mem_section->hot_map). Readers must dereference ->hot_map under
+ * rcu_read_lock() and updaters must use the RCU pointer accessors.
+ */
+struct pghot_hot_map {
+	struct rcu_head rcu;
+	unsigned long flags;
+	phi_t phi[];
+};
+
+bool pghot_nid_valid(int nid);
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time);
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now);
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time);
+
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now);
+#else
+static inline int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+	return 0;
+}
+#endif /* CONFIG_PGHOT */
+#endif /* _LINUX_PGHOT_H */
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 03fe95f5a020..58d510711bd4 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -175,6 +175,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSTACK_REST,
 #endif
 #endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORDED_HINTFAULTS,
+		PGHOT_RECORDED_HWHINTS,
+#endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
 
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca4824905..0a5bcd5d45ed 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1509,6 +1509,20 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 	  If unsure, say N.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	default n
+	depends on NUMA_MIGRATION && SPARSEMEM
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
+
+	  This adds 1 byte of metadata overhead per page in lower-tier
+	  memory nodes.
+
 source "mm/damon/Kconfig"
 
 endmenu
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061..4939a1a74c1d 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-default.o
diff --git a/mm/migrate.c b/mm/migrate.c
index 58a8a0cf6fa3..7ab15780226e 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2628,7 +2628,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
 }
 #endif /* CONFIG_NUMA_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 /*
  * Returns true if this is a safe migration target node for misplaced NUMA
  * pages. Currently it only checks the watermarks which is crude.
@@ -2748,12 +2748,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
  */
 int migrate_misplaced_folio(struct folio *folio, int node)
 {
-	pg_data_t *pgdat = NODE_DATA(node);
 	int nr_remaining;
 	unsigned int nr_succeeded;
 	LIST_HEAD(migratepages);
 	struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
-	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
 	list_add(&folio->lru, &migratepages);
 	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2762,12 +2760,18 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	if (nr_remaining && !list_empty(&migratepages))
 		putback_movable_pages(&migratepages);
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
-		    && node_is_toptier(node))
+		    && node_is_toptier(node)) {
+			pg_data_t *pgdat = NODE_DATA(node);
+			struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
 			mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
+		}
+#endif
 	}
 	mem_cgroup_put(memcg);
 	BUG_ON(!list_empty(&migratepages));
@@ -2821,14 +2825,16 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
 		putback_movable_pages(folio_list);
 
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
 				 PGPROMOTE_SUCCESS, nr_succeeded);
+#endif
 	}
 
 	mem_cgroup_put(memcg);
 	WARN_ON(!list_empty(folio_list));
 	return nr_remaining ? -EAGAIN : 0;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 0f64909e8d20..13ef3cf1e48c 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1386,6 +1386,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
 static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
 #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
 static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 {
 	int i;
@@ -1393,6 +1402,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 	pgdat_resize_init(pgdat);
 	pgdat_kswapd_lock_init(pgdat);
 	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kmigrated(pgdat);
 
 	init_waitqueue_head(&pgdat->kswapd_wait);
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
diff --git a/mm/pghot-default.c b/mm/pghot-default.c
new file mode 100644
index 000000000000..ed66a867a644
--- /dev/null
+++ b/mm/pghot-default.c
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Default mode
+ *
+ * 1 byte hotness record per PFN.
+ * Bucketed time and frequency tracked as part of the record.
+ * Promotion to @sysctl_pghot_target_nid by default.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+
+/* pghot-default doesn't store and hence no NID validation is required */
+bool pghot_nid_valid(int nid)
+{
+	return true;
+}
+
+/*
+ * @time is regular time, @old_time is bucketed time.
+ */
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+	time &= PGHOT_TIME_BUCKETS_MASK;
+	old_time <<= PGHOT_TIME_BUCKETS_SHIFT;
+
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_BUCKETS_MASK);
+}
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
+	phi_t freq, old_freq, hotness, old_hotness, old_time;
+	phi_t time = now >> PGHOT_TIME_BUCKETS_SHIFT;
+
+	old_hotness = READ_ONCE(*phi);
+	do {
+		bool new_window = false;
+
+		old_freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+		old_time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+
+		if (pghot_access_latency(old_time, now) > sysctl_pghot_freq_window)
+			new_window = true;
+
+		if (new_window)
+			freq = 1;
+		else if (old_freq < PGHOT_FREQ_MAX)
+			freq = old_freq + 1;
+		else
+			freq = old_freq;
+
+		hotness = 0;
+		hotness |= (freq & PGHOT_FREQ_MASK) << PGHOT_FREQ_SHIFT;
+		hotness |= (time & PGHOT_TIME_MASK) << PGHOT_TIME_SHIFT;
+
+		if (freq >= sysctl_pghot_freq_threshold)
+			hotness |= BIT(PGHOT_MIGRATE_READY);
+	} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+	return !!(hotness & BIT(PGHOT_MIGRATE_READY));
+}
+
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time)
+{
+	phi_t old_hotness, hotness = 0;
+
+	old_hotness = READ_ONCE(*phi);
+	do {
+		if (!(old_hotness & BIT(PGHOT_MIGRATE_READY)))
+			return -EINVAL;
+	} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+
+	*nid = sysctl_pghot_target_nid;
+	*freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+	*time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+	return 0;
+}
diff --git a/mm/pghot.c b/mm/pghot.c
new file mode 100644
index 000000000000..19cc023091f4
--- /dev/null
+++ b/mm/pghot.c
@@ -0,0 +1,725 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Maintains information about hot pages from slower tier nodes and
+ * promotes them.
+ *
+ * Per-PFN hotness information is stored for lower tier nodes in
+ * mem_section.
+ *
+ * In the default mode, a single byte (u8) is used to store
+ * the frequency of access and last access time. Promotions are done
+ * to a default toptier NID.
+ *
+ * A kernel thread named kmigrated is provided to migrate or promote
+ * the hot pages. kmigrated runs for each lower tier node. It iterates
+ * over the node's PFNs and  migrates pages marked for migration into
+ * their targeted nodes.
+ */
+#include <linux/mm.h>
+#include <linux/migrate.h>
+#include <linux/memory.h>
+#include <linux/memory-tiers.h>
+#include <linux/debugfs.h>
+#include <linux/rcupdate.h>
+#include <linux/pghot.h>
+
+unsigned int pghot_freq_window_min = PGHOT_FREQ_WINDOW_MIN;
+unsigned int pghot_freq_window_max = PGHOT_FREQ_WINDOW_MAX;
+unsigned int pghot_freq_threshold_max = PGHOT_FREQ_MAX;
+unsigned int kmigrated_sleep_ms = KMIGRATED_SLEEP_MS_DEFAULT;
+unsigned int kmigrated_batch_nr = KMIGRATED_BATCH_NR_DEFAULT;
+
+unsigned int sysctl_pghot_src_enabled;
+unsigned int sysctl_pghot_target_nid = PGHOT_DEFAULT_NODE;
+unsigned int sysctl_pghot_freq_window = PGHOT_FREQ_WINDOW_DEFAULT;
+unsigned int sysctl_pghot_freq_threshold = PGHOT_FREQ_THRESHOLD_DEFAULT;
+
+DEFINE_STATIC_KEY_FALSE(pghot_src_hwhints);
+DEFINE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+DEFINE_MUTEX(pghot_tunables_lock);
+
+#ifdef CONFIG_SYSCTL
+static void pghot_src_enabled_update(unsigned int enabled)
+{
+	unsigned int changed = sysctl_pghot_src_enabled ^ enabled;
+
+	if (changed & PGHOT_HINTFAULTS_ENABLED) {
+		if (enabled & PGHOT_HINTFAULTS_ENABLED)
+			static_branch_enable(&pghot_src_hintfaults);
+		else
+			static_branch_disable(&pghot_src_hintfaults);
+	}
+
+	if (changed & PGHOT_HWHINTS_ENABLED) {
+		if (enabled & PGHOT_HWHINTS_ENABLED)
+			static_branch_enable(&pghot_src_hwhints);
+		else
+			static_branch_disable(&pghot_src_hwhints);
+	}
+}
+
+static int sysctl_src_enabled_handler(const struct ctl_table *table, int write,
+				      void *buffer, size_t *lenp, loff_t *ppos)
+{
+	struct ctl_table t;
+	int err;
+	unsigned int enabled;
+
+	guard(mutex)(&pghot_tunables_lock);
+
+	enabled = sysctl_pghot_src_enabled;
+	t = *table;
+	t.data = &enabled;
+	err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos);
+	if (err < 0)
+		return err;
+
+	if (write) {
+		if (enabled & ~PGHOT_SRC_ENABLED_MASK)
+			return -EINVAL;
+		pghot_src_enabled_update(enabled);
+		sysctl_pghot_src_enabled = enabled;
+	}
+	return err;
+}
+
+static int sysctl_target_nid_handler(const struct ctl_table *table, int write,
+				     void *buffer, size_t *lenp, loff_t *ppos)
+{
+	struct ctl_table t;
+	int err;
+	unsigned int nid;
+
+	guard(mutex)(&pghot_tunables_lock);
+
+	nid = sysctl_pghot_target_nid;
+	t = *table;
+	t.data = &nid;
+	err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos);
+	if (err < 0)
+		return err;
+
+	if (write) {
+		if (!numa_valid_node(nid) || nid > PGHOT_NID_MAX ||
+		    !node_online(nid) || !node_is_toptier(nid))
+			return -EINVAL;
+		sysctl_pghot_target_nid = nid;
+	}
+	return err;
+}
+
+static const struct ctl_table pghot_sysctls[] = {
+	{
+		.procname	= "pghot_enabled_sources",
+		.data		= &sysctl_pghot_src_enabled,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sysctl_src_enabled_handler,
+	},
+	{
+		.procname       = "pghot_promote_freq_window_ms",
+		.data           = &sysctl_pghot_freq_window,
+		.maxlen         = sizeof(unsigned int),
+		.mode           = 0644,
+		.proc_handler   = proc_dointvec_minmax,
+		.extra1         = &pghot_freq_window_min,
+		.extra2         = &pghot_freq_window_max,
+	},
+	{
+		.procname	= "pghot_target_nid",
+		.data		= &sysctl_pghot_target_nid,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sysctl_target_nid_handler,
+	},
+	{
+		.procname	= "pghot_freq_threshold",
+		.data		= &sysctl_pghot_freq_threshold,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ONE,
+		.extra2		= &pghot_freq_threshold_max,
+	},
+};
+
+static void __init pghot_sysctl_init(void)
+{
+	register_sysctl_init("vm", pghot_sysctls);
+}
+#else
+static inline void pghot_sysctl_init(void) { }
+#endif
+
+static bool kmigrated_started __ro_after_init;
+
+/**
+ * pghot_record_access() - Record page accesses from lower tier memory
+ * for the purpose of tracking page hotness and subsequent promotion.
+ *
+ * @pfn: PFN of the page
+ * @nid: Unused
+ * @src: The identifier of the sub-system that reports the access
+ * @now: Access time in jiffies
+ *
+ * Updates the frequency and time of access and marks the page as
+ * ready for migration if the frequency crosses a threshold. The pages
+ * marked for migration are migrated by kmigrated kernel thread.
+ *
+ * Return: 0 on success and -EINVAL on failure to record the access.
+ */
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+	struct mem_section *ms;
+	struct folio *folio;
+	struct pghot_hot_map *hot_map;
+	phi_t *phi;
+	struct page *page;
+	int src_nid;
+
+	if (!kmigrated_started)
+		return 0;
+
+	if (!pghot_nid_valid(nid))
+		return -EINVAL;
+
+	switch (src) {
+	case PGHOT_HINTFAULTS:
+		if (!static_branch_unlikely(&pghot_src_hintfaults))
+			return 0;
+		count_vm_event(PGHOT_RECORDED_HINTFAULTS);
+		break;
+	case PGHOT_HWHINTS:
+		if (!static_branch_unlikely(&pghot_src_hwhints))
+			return 0;
+		count_vm_event(PGHOT_RECORDED_HWHINTS);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	src_nid = pfn_to_nid(pfn);
+	if (src_nid == nid)
+		return 0;
+
+	/*
+	 * Record only accesses from lower tiers.
+	 */
+	if (node_is_toptier(src_nid))
+		return 0;
+
+	/*
+	 * Reject the non-migratable pages right away.
+	 */
+	page = pfn_to_online_page(pfn);
+	if (!page || is_zone_device_page(page))
+		return 0;
+
+	folio = page_folio(page);
+	if (!folio_try_get(folio))
+		return 0;
+
+	if (unlikely(page_folio(page) != folio))
+		goto out;
+
+	if (!folio_test_lru(folio))
+		goto out;
+
+	/* Get the hotness slot corresponding to the 1st PFN of the folio */
+	pfn = folio_pfn(folio);
+	ms = __pfn_to_section(pfn);
+
+	rcu_read_lock();
+	hot_map = ms ? rcu_dereference(ms->hot_map) : NULL;
+	if (!hot_map) {
+		rcu_read_unlock();
+		goto out;
+	}
+
+	phi = &hot_map->phi[pfn % PAGES_PER_SECTION];
+
+	count_vm_event(PGHOT_RECORDED_ACCESSES);
+
+	/*
+	 * Update the hotness parameters.
+	 */
+	if (pghot_update_record(phi, nid, now)) {
+		set_bit(PGHOT_SECTION_HOT_BIT, &hot_map->flags);
+		set_bit(PGDAT_KMIGRATED_ACTIVATE, &page_pgdat(page)->flags);
+	}
+	rcu_read_unlock();
+out:
+	folio_put(folio);
+	return 0;
+}
+
+static int pghot_get_hotness(unsigned long pfn, int *nid, int *freq,
+			     unsigned long *time)
+{
+	struct pghot_hot_map *hot_map;
+	struct mem_section *ms;
+	phi_t *phi;
+	int ret;
+
+	ms = __pfn_to_section(pfn);
+
+	rcu_read_lock();
+	hot_map = ms ? rcu_dereference(ms->hot_map) : NULL;
+	if (!hot_map) {
+		rcu_read_unlock();
+		return -EINVAL;
+	}
+
+	phi = &hot_map->phi[pfn % PAGES_PER_SECTION];
+	ret = pghot_get_record(phi, nid, freq, time);
+	rcu_read_unlock();
+
+	return ret;
+}
+
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+				int src_nid)
+{
+	struct mem_cgroup *cur_memcg = NULL;
+	int cur_nid = NUMA_NO_NODE;
+	LIST_HEAD(migrate_list);
+	int batch_count = 0;
+	struct folio *folio;
+	struct page *page;
+	unsigned long pfn;
+
+	pfn = start_pfn;
+	do {
+		int nid = NUMA_NO_NODE, nr = 1;
+		struct mem_cgroup *memcg;
+		unsigned long time = 0;
+		int freq = 0;
+
+		if (!pfn_valid(pfn))
+			goto out_next;
+
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto out_next;
+
+		folio = page_folio(page);
+		if (!folio_try_get(folio))
+			goto out_next;
+
+		if (unlikely(page_folio(page) != folio)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		nr = folio_nr_pages(folio);
+		if (folio_nid(folio) != src_nid) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (!folio_test_lru(folio)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (pghot_get_hotness(pfn, &nid, &freq, &time)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (folio_nid(folio) == nid) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (migrate_misplaced_folio_prepare(folio, NULL, nid)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		rcu_read_lock();
+		memcg = folio_memcg(folio);
+		rcu_read_unlock();
+		if (cur_nid == NUMA_NO_NODE) {
+			cur_nid = nid;
+			cur_memcg = memcg;
+		}
+
+		/* If NID or memcg changed, flush the previous batch first */
+		if (cur_nid != nid || cur_memcg != memcg) {
+			if (!list_empty(&migrate_list))
+				promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+			cur_nid = nid;
+			cur_memcg = memcg;
+			batch_count = 0;
+			cond_resched();
+		}
+
+		list_add(&folio->lru, &migrate_list);
+		folio_put(folio);
+
+		if (++batch_count > READ_ONCE(kmigrated_batch_nr)) {
+			promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+			batch_count = 0;
+			cond_resched();
+		}
+out_next:
+		pfn += nr;
+	} while (pfn < end_pfn);
+	if (!list_empty(&migrate_list))
+		promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+}
+
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct pghot_hot_map *hot_map;
+	struct mem_section *ms;
+	bool hot;
+	int nid;
+
+	clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		start_pfn = section_nr_to_pfn(section_nr);
+		ms = __nr_to_section(section_nr);
+
+		if (!pfn_valid(start_pfn))
+			continue;
+
+		nid = pfn_to_nid(start_pfn);
+		if (node_is_toptier(nid) || nid != pgdat->node_id)
+			continue;
+
+		rcu_read_lock();
+		hot_map = rcu_dereference(ms->hot_map);
+		hot = hot_map &&
+		      test_and_clear_bit(PGHOT_SECTION_HOT_BIT, &hot_map->flags);
+		rcu_read_unlock();
+
+		if (!hot)
+			continue;
+
+		kmigrated_walk_zone(start_pfn, start_pfn + PAGES_PER_SECTION,
+				    pgdat->node_id);
+	}
+}
+
+static inline bool kmigrated_work_requested(pg_data_t *pgdat)
+{
+	return test_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+}
+
+/*
+ * Per-node kthread that iterates over its PFNs and migrates the
+ * pages that have been marked for migration.
+ */
+static int kmigrated(void *p)
+{
+	pg_data_t *pgdat = p;
+
+	while (!kthread_should_stop()) {
+		long timeout = msecs_to_jiffies(READ_ONCE(kmigrated_sleep_ms));
+
+		if (wait_event_timeout(pgdat->kmigrated_wait, kmigrated_work_requested(pgdat),
+				       timeout))
+			kmigrated_do_work(pgdat);
+	}
+	return 0;
+}
+
+static int kmigrated_run(int nid)
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
+	int ret;
+
+	if (!pgdat->kmigrated) {
+		pgdat->kmigrated = kthread_create_on_node(kmigrated, pgdat, nid,
+							  "kmigrated%d", nid);
+		if (IS_ERR(pgdat->kmigrated)) {
+			ret = PTR_ERR(pgdat->kmigrated);
+			pgdat->kmigrated = NULL;
+			pr_err("Failed to start kmigrated%d, ret %d\n", nid, ret);
+			return ret;
+		}
+		pr_info("pghot: Started kmigrated thread for node %d\n", nid);
+	}
+	wake_up_process(pgdat->kmigrated);
+	return 0;
+}
+
+static void pghot_free_hot_map(struct mem_section *ms)
+{
+	struct pghot_hot_map *hot_map;
+
+	hot_map = rcu_dereference_protected(ms->hot_map, 1);
+	if (!hot_map)
+		return;
+
+	RCU_INIT_POINTER(ms->hot_map, NULL);
+	kfree_rcu(hot_map, rcu);
+}
+
+static int pghot_alloc_hot_map(struct mem_section *ms, int nid)
+{
+	struct pghot_hot_map *hot_map;
+
+	hot_map = kzalloc_node(struct_size(hot_map, phi, PAGES_PER_SECTION),
+			       GFP_KERNEL, nid);
+	if (!hot_map)
+		return -ENOMEM;
+
+	rcu_assign_pointer(ms->hot_map, hot_map);
+	return 0;
+}
+
+static void pghot_offline_sec_hotmap(unsigned long start_pfn,
+				     unsigned long nr_pages)
+{
+	unsigned long start, end, pfn;
+	struct mem_section *ms;
+
+	start = SECTION_ALIGN_DOWN(start_pfn);
+	end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+	for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (!ms)
+			continue;
+
+		pghot_free_hot_map(ms);
+	}
+}
+
+static int pghot_online_sec_hotmap(unsigned long start_pfn,
+				   unsigned long nr_pages)
+{
+	int nid = pfn_to_nid(start_pfn);
+	unsigned long start, end, pfn;
+	struct mem_section *ms;
+	int fail = 0;
+
+	/*
+	 * No section hot maps for toptier sections.
+	 */
+	if (node_is_toptier(nid))
+		return 0;
+
+	start = SECTION_ALIGN_DOWN(start_pfn);
+	end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+	for (pfn = start; !fail && pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (!ms || rcu_access_pointer(ms->hot_map))
+			continue;
+
+		fail = pghot_alloc_hot_map(ms, nid);
+	}
+
+	if (!fail)
+		return 0;
+
+	/* rollback: free the sections allocated before the failure */
+	end = pfn - PAGES_PER_SECTION;
+	for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (ms)
+			pghot_free_hot_map(ms);
+	}
+	return -ENOMEM;
+}
+
+static int pghot_memhp_callback(struct notifier_block *self,
+				unsigned long action, void *arg)
+{
+	struct memory_notify *mn = arg;
+	int ret = 0;
+
+	switch (action) {
+	case MEM_GOING_ONLINE:
+		ret = pghot_online_sec_hotmap(mn->start_pfn, mn->nr_pages);
+		break;
+	case MEM_OFFLINE:
+	case MEM_CANCEL_ONLINE:
+		pghot_offline_sec_hotmap(mn->start_pfn, mn->nr_pages);
+		break;
+	}
+
+	return notifier_from_errno(ret);
+}
+
+static struct notifier_block pghot_mem_notifier = {
+	.notifier_call = pghot_memhp_callback,
+	.priority = DEFAULT_CALLBACK_PRI,
+};
+
+static void pghot_destroy_hot_map(void)
+{
+	unsigned long section_nr, s_begin;
+	struct mem_section *ms;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		pghot_free_hot_map(ms);
+	}
+
+	unregister_memory_notifier(&pghot_mem_notifier);
+}
+
+static int pghot_setup_hot_map(void)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	int nid, ret;
+
+	ret = register_memory_notifier(&pghot_mem_notifier);
+	if (ret)
+		return ret;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		start_pfn = section_nr_to_pfn(section_nr);
+		nid = pfn_to_nid(start_pfn);
+
+		if (node_is_toptier(nid) || !pfn_valid(start_pfn))
+			continue;
+
+		if (pghot_alloc_hot_map(ms, nid))
+			goto out_free_hot_map;
+	}
+	return 0;
+
+out_free_hot_map:
+	pghot_destroy_hot_map();
+	return -ENOMEM;
+}
+
+static int pghot_parse_uint(const char __user *ubuf, size_t cnt, unsigned int *val)
+{
+	char buf[16];
+
+	if (cnt > sizeof(buf) - 1)
+		cnt = sizeof(buf) - 1;
+	if (copy_from_user(buf, ubuf, cnt))
+		return -EFAULT;
+	buf[cnt] = '\0';
+	if (kstrtouint(buf, 0, val))
+		return -EINVAL;
+	return 0;
+}
+
+static ssize_t pghot_kmigrated_sleep_ms_write(struct file *filp, const char __user *ubuf,
+					      size_t cnt, loff_t *ppos)
+{
+	unsigned int val;
+	int ret;
+
+	guard(mutex)(&pghot_tunables_lock);
+
+	ret = pghot_parse_uint(ubuf, cnt, &val);
+	if (ret)
+		return ret;
+	if (val < KMIGRATED_SLEEP_MS_MIN  || val > KMIGRATED_SLEEP_MS_MAX)
+		return -EINVAL;
+
+	WRITE_ONCE(kmigrated_sleep_ms, val);
+	*ppos += cnt;
+	return cnt;
+}
+
+static ssize_t pghot_kmigrated_batch_nr_write(struct file *filp, const char __user *ubuf,
+					      size_t cnt, loff_t *ppos)
+{
+	unsigned int val;
+	int ret;
+
+	guard(mutex)(&pghot_tunables_lock);
+
+	ret = pghot_parse_uint(ubuf, cnt, &val);
+	if (ret)
+		return ret;
+	if (val < KMIGRATED_BATCH_NR_MIN  || val > KMIGRATED_BATCH_NR_MAX)
+		return -EINVAL;
+
+	WRITE_ONCE(kmigrated_batch_nr, val);
+	*ppos += cnt;
+	return cnt;
+}
+
+#define PGHOT_SHOW(name)						\
+static int pghot_##name##_show(struct seq_file *m, void *v)		\
+{									\
+	seq_printf(m, "%u\n", name);				\
+	return 0;							\
+}									\
+static int pghot_##name##_open(struct inode *inode, struct file *filp)	\
+{									\
+	return single_open(filp, pghot_##name##_show, NULL);		\
+}
+
+PGHOT_SHOW(kmigrated_sleep_ms)
+PGHOT_SHOW(kmigrated_batch_nr)
+
+#define PGHOT_FOPS(name)						\
+static const struct file_operations pghot_##name##_fops = {		\
+	.open		= pghot_##name##_open,				\
+	.read		= seq_read,					\
+	.write		= pghot_##name##_write,				\
+	.llseek		= seq_lseek,					\
+	.release	= single_release,				\
+}
+
+PGHOT_FOPS(kmigrated_sleep_ms);
+PGHOT_FOPS(kmigrated_batch_nr);
+
+static void pghot_debug_init(void)
+{
+	struct dentry *dir = debugfs_create_dir("pghot", NULL);
+
+	debugfs_create_file("kmigrated_sleep_ms", 0644, dir, NULL,
+			    &pghot_kmigrated_sleep_ms_fops);
+	debugfs_create_file("kmigrated_batch_nr", 0644, dir, NULL,
+			    &pghot_kmigrated_batch_nr_fops);
+}
+
+static int __init pghot_init(void)
+{
+	pg_data_t *pgdat;
+	int nid, ret;
+
+	ret = pghot_setup_hot_map();
+	if (ret)
+		return ret;
+
+	for_each_node_state(nid, N_MEMORY) {
+		if (node_is_toptier(nid))
+			continue;
+
+		ret = kmigrated_run(nid);
+		if (ret)
+			goto out_stop_kthread;
+	}
+	pghot_sysctl_init();
+	pghot_debug_init();
+
+	kmigrated_started = true;
+	return 0;
+
+out_stop_kthread:
+	for_each_node_state(nid, N_MEMORY) {
+		pgdat = NODE_DATA(nid);
+		if (pgdat->kmigrated) {
+			kthread_stop(pgdat->kmigrated);
+			pgdat->kmigrated = NULL;
+		}
+	}
+	pghot_destroy_hot_map();
+	return ret;
+}
+
+late_initcall_sync(pghot_init)
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..4064ead568cc 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1489,6 +1489,11 @@ const char * const vmstat_text[] = {
 	[I(KSTACK_REST)]			= "kstack_rest",
 #endif
 #endif
+#ifdef CONFIG_PGHOT
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
-- 
2.34.1

[PATCH v8 4/8] mm: pghot: Precision mode for pghot

[Bharata B Rao]

Default pghot stores hotness in a 1‑byte record per PFN, limiting
frequency to 2 bits, time to a 5‑bit bucket, and preventing storage
of per‑PFN toptier NID. This restricts time granularity and forces
all promotions to use the global pghot_target_nid.

This patch adds an optional precision mode (CONFIG_PGHOT_PRECISE)
that expands the hotness record to 4 bytes (u32) and provides:

- 10‑bit NID field for per‑PFN promotion target,
- 3‑bit frequency field (freq_threshold range 1–7),
- 14‑bit time field offering finer recency tracking,
- MSB migrate‑ready bit.

Precision mode improves placement accuracy on systems with multiple
toptier nodes and provides higher‑resolution hotness tracking, at
the cost of increasing metadata to 4 bytes per PFN.

Documentation, tunables, and the record layout are updated accordingly.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 Documentation/admin-guide/mm/pghot.rst |  4 +-
 include/linux/pghot.h                  | 39 ++++++++++++-
 mm/Kconfig                             | 11 ++++
 mm/Makefile                            |  7 ++-
 mm/pghot-precise.c                     | 77 ++++++++++++++++++++++++++
 mm/pghot.c                             | 13 +++--
 6 files changed, 143 insertions(+), 8 deletions(-)
 create mode 100644 mm/pghot-precise.c

diff --git a/Documentation/admin-guide/mm/pghot.rst b/Documentation/admin-guide/mm/pghot.rst
index 0edbe0082816..6d9b3d9522d1 100644
--- a/Documentation/admin-guide/mm/pghot.rst
+++ b/Documentation/admin-guide/mm/pghot.rst
@@ -56,7 +56,7 @@ Path: /proc/sys/vm/pghot_target_nid
 Path: /proc/sys/vm/pghot_freq_threshold
 
 - Minimum access frequency before a page is marked ready for promotion.
-  Range is 1 to 3 in default mode.
+  Range is 1 to 3 in default mode and 1 to 7 in precision mode.
 - Default: 2
 - Example:
   # sysctl vm.pghot_freq_threshold=1
@@ -68,7 +68,7 @@ Path: /proc/sys/vm/pghot_promote_freq_window_ms
 - Controls the time window (in ms) for counting access frequency. A page is
   considered hot only when **pghot_freq_threshold** number of accesses occur
   with this time period.
-- Default: 3000 (3 seconds)
+- Default: 3000 (3 seconds) in default mode and 5000 (5s) in precision mode.
 - Example:
   # sysctl vm.pghot_promote_freq_window_ms=3000
 
diff --git a/include/linux/pghot.h b/include/linux/pghot.h
index 7b85d717f410..313e5a6973e4 100644
--- a/include/linux/pghot.h
+++ b/include/linux/pghot.h
@@ -37,8 +37,40 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 #define PGHOT_DEFAULT_NODE		0
 
+#if defined(CONFIG_PGHOT_PRECISE)
 #define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
-#define PGHOT_FREQ_WINDOW_DEFAULT	(3 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT	(5 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-26 are used to store nid, frequency and time.
+ * Bits 27-30 are unused now.
+ * Bit 31 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		31
+
+#define PGHOT_NID_WIDTH			10
+#define PGHOT_FREQ_WIDTH		3
+/* time is stored in 14 bits which can represent up to 16s with HZ=1000 */
+#define PGHOT_TIME_WIDTH		14
+
+#define PGHOT_NID_SHIFT			0
+#define PGHOT_FREQ_SHIFT		(PGHOT_NID_SHIFT + PGHOT_NID_WIDTH)
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_NID_MASK			GENMASK(PGHOT_NID_WIDTH - 1, 0)
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+#define PGHOT_FREQ_WINDOW_MAX		PGHOT_TIME_MAX
+
+typedef u32 phi_t;
+
+#else	/* !CONFIG_PGHOT_PRECISE */
+#define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT       (3 * MSEC_PER_SEC)
 
 /*
  * Bits 0-6 are used to store frequency and time.
@@ -66,6 +98,11 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 typedef u8 phi_t;
 
+static_assert(MAX_NUMNODES <= (1 << PGHOT_NID_WIDTH),
+	      "pghot precise nid field too narrow for MAX_NUMNODES");
+
+#endif /* CONFIG_PGHOT_PRECISE */
+
 #define PGHOT_RECORD_SIZE		sizeof(phi_t)
 
 #define PGHOT_SECTION_HOT_BIT		0
diff --git a/mm/Kconfig b/mm/Kconfig
index 0a5bcd5d45ed..955a826ecfe9 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1523,6 +1523,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	default n
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.
+	  Adds fine-grained access time tracking and explicit toptier target
+	  NID tracking. Precise hot page tracking comes at the cost of using
+	  4 bytes per page against the default one byte per page. Preferable
+	  to enable this on systems with multiple nodes in toptier.
+
 source "mm/damon/Kconfig"
 
 endmenu
diff --git a/mm/Makefile b/mm/Makefile
index 4939a1a74c1d..83bac64cab26 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -147,4 +147,9 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
-obj-$(CONFIG_PGHOT) += pghot.o pghot-default.o
+obj-$(CONFIG_PGHOT) += pghot.o
+ifdef CONFIG_PGHOT_PRECISE
+obj-$(CONFIG_PGHOT) += pghot-precise.o
+else
+obj-$(CONFIG_PGHOT) += pghot-default.o
+endif
diff --git a/mm/pghot-precise.c b/mm/pghot-precise.c
new file mode 100644
index 000000000000..df82f2068a99
--- /dev/null
+++ b/mm/pghot-precise.c
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Precision mode
+ *
+ * 4 byte hotness record per PFN (u32)
+ * NID, time and frequency tracked as part of the record.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+#include <linux/memory-tiers.h>
+
+bool pghot_nid_valid(int nid)
+{
+	if (nid != NUMA_NO_NODE &&
+	    (!numa_valid_node(nid) || nid > PGHOT_NID_MAX ||
+	     !node_online(nid) || !node_is_toptier(nid)))
+		return false;
+
+	return true;
+}
+
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_MASK);
+}
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
+	phi_t freq, old_freq, hotness, old_hotness, old_time;
+	phi_t time = now & PGHOT_TIME_MASK;
+
+	nid = (nid == NUMA_NO_NODE) ? sysctl_pghot_target_nid : nid;
+	old_hotness = READ_ONCE(*phi);
+
+	do {
+		bool new_window = false;
+
+		old_freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+		old_time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+
+		if (pghot_access_latency(old_time, time) > sysctl_pghot_freq_window)
+			new_window = true;
+
+		if (new_window)
+			freq = 1;
+		else if (old_freq < PGHOT_FREQ_MAX)
+			freq = old_freq + 1;
+		else
+			freq = old_freq;
+
+		hotness = 0;
+		hotness |= (nid & PGHOT_NID_MASK) << PGHOT_NID_SHIFT;
+		hotness |= (freq & PGHOT_FREQ_MASK) << PGHOT_FREQ_SHIFT;
+		hotness |= (time & PGHOT_TIME_MASK) << PGHOT_TIME_SHIFT;
+
+		if (freq >= sysctl_pghot_freq_threshold)
+			hotness |= BIT(PGHOT_MIGRATE_READY);
+	} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+	return !!(hotness & BIT(PGHOT_MIGRATE_READY));
+}
+
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time)
+{
+	phi_t old_hotness, hotness = 0;
+
+	old_hotness = READ_ONCE(*phi);
+	do {
+		if (!(old_hotness & BIT(PGHOT_MIGRATE_READY)))
+			return -EINVAL;
+	} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+
+	*nid = (old_hotness >> PGHOT_NID_SHIFT) & PGHOT_NID_MASK;
+	*freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+	*time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+	return 0;
+}
diff --git a/mm/pghot.c b/mm/pghot.c
index 19cc023091f4..f66baacb2862 100644
--- a/mm/pghot.c
+++ b/mm/pghot.c
@@ -10,6 +10,9 @@
  * the frequency of access and last access time. Promotions are done
  * to a default toptier NID.
  *
+ * In the precision mode, 4 bytes are used to store the frequency
+ * of access, last access time and the accessing NID.
+ *
  * A kernel thread named kmigrated is provided to migrate or promote
  * the hot pages. kmigrated runs for each lower tier node. It iterates
  * over the node's PFNs and  migrates pages marked for migration into
@@ -158,13 +161,15 @@ static bool kmigrated_started __ro_after_init;
  * for the purpose of tracking page hotness and subsequent promotion.
  *
  * @pfn: PFN of the page
- * @nid: Unused
+ * @nid: Target NID to where the page needs to be migrated in precision
+ *       mode but unused in default mode
  * @src: The identifier of the sub-system that reports the access
  * @now: Access time in jiffies
  *
- * Updates the frequency and time of access and marks the page as
- * ready for migration if the frequency crosses a threshold. The pages
- * marked for migration are migrated by kmigrated kernel thread.
+ * Updates the NID (in precision mode only), frequency and time of access
+ * and marks the page as ready for migration if the frequency crosses a
+ * threshold. The pages marked for migration are migrated by kmigrated
+ * kernel thread.
  *
  * Return: 0 on success and -EINVAL on failure to record the access.
  */
-- 
2.34.1

[PATCH v8 5/8] mm: sched: move NUMA balancing tiering promotion to pghot

[Bharata B Rao]

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING
mode of NUMA Balancing) does hot page detection (via hint faults),
hot page classification and eventual promotion, all by itself and
sits within the scheduler.

With pghot, the new hot page tracking and promotion mechanism
being available, NUMA Balancing can limit itself to detection
of hot pages (via hint faults) and off-load rest of the
functionality to pghot.

To achieve this, pghot_record_access(PGHOT_HINTFAULTS) API
is used to feed the hot page info to pghot. In addition, the
migration rate limiting and dynamic threshold logic are moved to
kmigrated so that the same can be used for hot pages reported by
other sources too. Hence it becomes necessary to introduce a
new config option CONFIG_NUMA_BALANCING_TIERING to control
the hint faults source for hot page promotion. This option
controls the NUMA_BALANCING_MEMORY_TIERING mode of
kernel.numa_balancing

This movement of hot page promotion to pghot results in the following
changes to the behaviour of hint faults based hot page promotion:

1. Promotion is no longer done in the fault path but instead is
   deferred to kmigrated and happens in batches.
2. NUMA_BALANCING_MEMORY_TIERING mode used to promote on first
   access. Pghot by default, promotes on second access though this
   can be changed by setting /proc/sys/vm/pghot_freq_threshold tunable.
3. hot_threshold_ms debugfs tunable now gets replaced by
   /proc/sys/vm/pghot_promote_freq_window_ms with default value of
   1000ms changed to 3000ms in pghot-default and 5000ms pghot-precise.
4. In NUMA_BALANCING_MEMORY_TIERING mode, hint fault latency is the
   difference between the PTE update time (during scanning) and the
   access time (hint fault). However with pghot, a single latency
   threshold is used for two purposes:
   a) If the time difference between successive accesses are within
      the threshold, the page is marked as hot.
   b) Later when kmigrated picks up the page for migration, it will
      migrate only if the difference between the current time and
      the time when the page was marked hot is with the threshold.
5. The max scan period which is used in dynamic threshold logic
   was a debugfs tunable (sysctl_numa_balancing_scan_period_max).
   However this has been converted to a scalar metric in pghot.
6. In the uncommon case of using NUMA_BALANCING_NORMAL mode
   to balance between lower and higher tier nodes, we end up
   waking the kswapd when there is no headroom in the toptier.

Key code changes due to this movement are detailed below to help
easy understanding of the restructuring.

1. Scanning and access times are no longer tracked in last_cpupid
   field of folio flags. Hence all code related to this (like
   folio_xchg_access_time(), cpupid_valid()) are removed.
2. The misplaced migration routines become conditional to
   CONFIG_PGHOT in addition to CONFIG_NUMA_BALANCING.
3. The promotion related stats (like PGPROMOTE_SUCCESS etc) are
   now moved to under CONFIG_PGHOT as these stats are part of
   promotion engine which will be used for other hotness sources
   as well.
4. Routines that are responsibile for migration rate limiting
   dynamic thresholding, pgdat balancing during promotion etc
   are moved to pghot with appropriate renaming.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 Documentation/admin-guide/mm/pghot.rst |  23 +++-
 include/linux/mm.h                     |  35 +----
 include/linux/mmzone.h                 |   4 +-
 init/Kconfig                           |  14 ++
 kernel/sched/core.c                    |   7 +
 kernel/sched/debug.c                   |   1 -
 kernel/sched/fair.c                    | 171 +------------------------
 kernel/sched/sched.h                   |   1 -
 mm/Kconfig                             |   2 +-
 mm/huge_memory.c                       |  32 ++++-
 mm/memcontrol.c                        |   6 +-
 mm/memory-tiers.c                      |  15 ++-
 mm/memory.c                            |  36 +++++-
 mm/mempolicy.c                         |   3 -
 mm/migrate.c                           |  16 ++-
 mm/pghot.c                             | 163 ++++++++++++++++++++++-
 mm/vmstat.c                            |   2 +-
 17 files changed, 310 insertions(+), 221 deletions(-)

diff --git a/Documentation/admin-guide/mm/pghot.rst b/Documentation/admin-guide/mm/pghot.rst
index 6d9b3d9522d1..accf78d95d02 100644
--- a/Documentation/admin-guide/mm/pghot.rst
+++ b/Documentation/admin-guide/mm/pghot.rst
@@ -35,7 +35,11 @@ Path: /proc/sys/vm/pghot_enabled_sources
 - Bits:
   - 0: Hint faults (value 0x1)
   - 1: Hardware hints (value 0x2)
-- Default: 0 (disabled)
+- Default: 0x1 (hint faults enabled, hardware hints disabled). Hint faults
+  are enabled by default so that selecting the NUMA Balancing memory
+  tiering mode (kernel.numa_balancing=2) promotes hot pages without any
+  additional opt-in. Hint faults are fed to pghot only while tiering mode
+  is selected, so this default has no effect otherwise.
 - Example:
   # echo 0x3 > /proc/sys/vm/pghot_enabled_sources
   Enables both hint faults and hwhints sources
@@ -87,3 +91,20 @@ Path: /proc/vmstat
 
 3. **pghot_recorded_hwhints**
    - Number of recorded accesses reported by hwhints source.
+
+NUMA Hint Faults Source
+=======================
+The "hint faults" source is the tiering mode of NUMA Balancing
+which acts as a source of page hotness to pghot.
+
+It is controlled by the NUMA_BALANCING_TIERING config option, which
+gates the memory tiering mode (NUMA_BALANCING_MEMORY_TIERING) of NUMA
+Balancing. At runtime that mode must additionally be selected through
+the kernel.numa_balancing sysctl.
+
+This source is enabled by default through the hint faults bit (0x1) of
+**pghot_enabled_sources**, so selecting the tiering mode alone is enough
+to promote hot pages. It can be disabled at runtime by clearing that bit:
+
+# echo 0x0 > /proc/sys/vm/pghot_enabled_sources
+
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbdd..1fdb0e143c65 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2305,17 +2305,6 @@ static inline int folio_nid(const struct folio *folio)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
-/* page access time bits needs to hold at least 4 seconds */
-#define PAGE_ACCESS_TIME_MIN_BITS	12
-#if LAST_CPUPID_SHIFT < PAGE_ACCESS_TIME_MIN_BITS
-#define PAGE_ACCESS_TIME_BUCKETS				\
-	(PAGE_ACCESS_TIME_MIN_BITS - LAST_CPUPID_SHIFT)
-#else
-#define PAGE_ACCESS_TIME_BUCKETS	0
-#endif
-
-#define PAGE_ACCESS_TIME_MASK				\
-	(LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS)
 
 static inline int cpu_pid_to_cpupid(int cpu, int pid)
 {
@@ -2381,15 +2370,6 @@ static inline void page_cpupid_reset_last(struct page *page)
 }
 #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	int last_time;
-
-	last_time = folio_xchg_last_cpupid(folio,
-					   time >> PAGE_ACCESS_TIME_BUCKETS);
-	return last_time << PAGE_ACCESS_TIME_BUCKETS;
-}
-
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 	unsigned int pid_bit;
@@ -2400,18 +2380,12 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 	}
 }
 
-bool folio_use_access_time(struct folio *folio);
 #else /* !CONFIG_NUMA_BALANCING */
 static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
 {
 	return folio_nid(folio); /* XXX */
 }
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	return 0;
-}
-
 static inline int folio_last_cpupid(struct folio *folio)
 {
 	return folio_nid(folio); /* XXX */
@@ -2454,11 +2428,16 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 }
-static inline bool folio_use_access_time(struct folio *folio)
+#endif /* CONFIG_NUMA_BALANCING */
+
+#ifdef CONFIG_NUMA_BALANCING_TIERING
+bool folio_is_promo_candidate(struct folio *folio);
+#else
+static inline bool folio_is_promo_candidate(struct folio *folio)
 {
 	return false;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING_TIERING */
 
 #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 11c547a51f43..c5a9237eedd8 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -288,7 +288,7 @@ enum node_stat_item {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,	/* promote successfully */
 	/**
 	 * Candidate pages for promotion based on hint fault latency.  This
@@ -1555,7 +1555,7 @@ typedef struct pglist_data {
 	unsigned long first_deferred_pfn;
 #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	/* start time in ms of current promote rate limit period */
 	unsigned int nbp_rl_start;
 	/* number of promote candidate pages at start time of current rate limit period */
diff --git a/init/Kconfig b/init/Kconfig
index 5230d4879b1c..75f5a0cb9065 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1038,6 +1038,20 @@ config NUMA_BALANCING_DEFAULT_ENABLED
 	  If set, automatic NUMA balancing will be enabled if running on a NUMA
 	  machine.
 
+config NUMA_BALANCING_TIERING
+	bool "NUMA balancing memory tiering promotion"
+	depends on NUMA_BALANCING && PGHOT
+	default y
+	help
+	  Enable NUMA balancing mode 2 (memory tiering). This allows
+	  automatic promotion of hot pages from slower memory tiers to
+	  faster tiers using the pghot subsystem.
+
+	  This requires CONFIG_PGHOT for the hot page tracking engine.
+	  This option is required for kernel.numa_balancing=2.
+
+	  If unsure, say N.
+
 config SLAB_OBJ_EXT
 	bool
 
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 96226707c2f6..10863460fed2 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4639,6 +4639,7 @@ void set_numabalancing_state(bool enabled)
 }
 
 #ifdef CONFIG_PROC_SYSCTL
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 static void reset_memory_tiering(void)
 {
 	struct pglist_data *pgdat;
@@ -4649,6 +4650,7 @@ static void reset_memory_tiering(void)
 		pgdat->nbp_th_start = jiffies_to_msecs(jiffies);
 	}
 }
+#endif
 
 static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 			  void *buffer, size_t *lenp, loff_t *ppos)
@@ -4666,9 +4668,14 @@ static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 	if (err < 0)
 		return err;
 	if (write) {
+		if ((state & NUMA_BALANCING_MEMORY_TIERING) &&
+		    !IS_ENABLED(CONFIG_NUMA_BALANCING_TIERING))
+			return -EOPNOTSUPP;
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 		    (state & NUMA_BALANCING_MEMORY_TIERING))
 			reset_memory_tiering();
+#endif
 		sysctl_numa_balancing_mode = state;
 		__set_numabalancing_state(state);
 	}
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 40584b27ea0c..7918de84abcf 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -665,7 +665,6 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
-	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_SCHED_CACHE
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..f865b93c99b5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -125,11 +125,6 @@ int __weak arch_asym_cpu_priority(int cpu)
 static unsigned int sysctl_sched_cfs_bandwidth_slice		= 5000UL;
 #endif
 
-#ifdef CONFIG_NUMA_BALANCING
-/* Restrict the NUMA promotion throughput (MB/s) for each target node. */
-static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
-#endif
-
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table sched_fair_sysctls[] = {
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -142,16 +137,6 @@ static const struct ctl_table sched_fair_sysctls[] = {
 		.extra1         = SYSCTL_ONE,
 	},
 #endif
-#ifdef CONFIG_NUMA_BALANCING
-	{
-		.procname	= "numa_balancing_promote_rate_limit_MBps",
-		.data		= &sysctl_numa_balancing_promote_rate_limit,
-		.maxlen		= sizeof(unsigned int),
-		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
-		.extra1		= SYSCTL_ZERO,
-	},
-#endif /* CONFIG_NUMA_BALANCING */
 };
 
 static int __init sched_fair_sysctl_init(void)
@@ -2214,9 +2199,6 @@ unsigned int sysctl_numa_balancing_scan_size = 256;
 /* Scan @scan_size MB every @scan_period after an initial @scan_delay in ms */
 unsigned int sysctl_numa_balancing_scan_delay = 1000;
 
-/* The page with hint page fault latency < threshold in ms is considered hot */
-unsigned int sysctl_numa_balancing_hot_threshold = MSEC_PER_SEC;
-
 struct numa_group {
 	refcount_t refcount;
 
@@ -2559,120 +2541,6 @@ static inline unsigned long group_weight(struct task_struct *p, int nid,
 	return 1000 * faults / total_faults;
 }
 
-/*
- * If memory tiering mode is enabled, cpupid of slow memory page is
- * used to record scan time instead of CPU and PID.  When tiering mode
- * is disabled at run time, the scan time (in cpupid) will be
- * interpreted as CPU and PID.  So CPU needs to be checked to avoid to
- * access out of array bound.
- */
-static inline bool cpupid_valid(int cpupid)
-{
-	return cpupid_to_cpu(cpupid) < nr_cpu_ids;
-}
-
-/*
- * For memory tiering mode, if there are enough free pages (more than
- * enough watermark defined here) in fast memory node, to take full
- * advantage of fast memory capacity, all recently accessed slow
- * memory pages will be migrated to fast memory node without
- * considering hot threshold.
- */
-static bool pgdat_free_space_enough(struct pglist_data *pgdat)
-{
-	int z;
-	unsigned long enough_wmark;
-
-	enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,
-			   pgdat->node_present_pages >> 4);
-	for (z = pgdat->nr_zones - 1; z >= 0; z--) {
-		struct zone *zone = pgdat->node_zones + z;
-
-		if (!populated_zone(zone))
-			continue;
-
-		if (zone_watermark_ok(zone, 0,
-				      promo_wmark_pages(zone) + enough_wmark,
-				      ZONE_MOVABLE, 0))
-			return true;
-	}
-	return false;
-}
-
-/*
- * For memory tiering mode, when page tables are scanned, the scan
- * time will be recorded in struct page in addition to make page
- * PROT_NONE for slow memory page.  So when the page is accessed, in
- * hint page fault handler, the hint page fault latency is calculated
- * via,
- *
- *	hint page fault latency = hint page fault time - scan time
- *
- * The smaller the hint page fault latency, the higher the possibility
- * for the page to be hot.
- */
-static int numa_hint_fault_latency(struct folio *folio)
-{
-	int last_time, time;
-
-	time = jiffies_to_msecs(jiffies);
-	last_time = folio_xchg_access_time(folio, time);
-
-	return (time - last_time) & PAGE_ACCESS_TIME_MASK;
-}
-
-/*
- * For memory tiering mode, too high promotion/demotion throughput may
- * hurt application latency.  So we provide a mechanism to rate limit
- * the number of pages that are tried to be promoted.
- */
-static bool numa_promotion_rate_limit(struct pglist_data *pgdat,
-				      unsigned long rate_limit, int nr)
-{
-	unsigned long nr_cand;
-	unsigned int now, start;
-
-	now = jiffies_to_msecs(jiffies);
-	mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE, nr);
-	nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
-	start = pgdat->nbp_rl_start;
-	if (now - start > MSEC_PER_SEC &&
-	    cmpxchg(&pgdat->nbp_rl_start, start, now) == start)
-		pgdat->nbp_rl_nr_cand = nr_cand;
-	if (nr_cand - pgdat->nbp_rl_nr_cand >= rate_limit)
-		return true;
-	return false;
-}
-
-#define NUMA_MIGRATION_ADJUST_STEPS	16
-
-static void numa_promotion_adjust_threshold(struct pglist_data *pgdat,
-					    unsigned long rate_limit,
-					    unsigned int ref_th)
-{
-	unsigned int now, start, th_period, unit_th, th;
-	unsigned long nr_cand, ref_cand, diff_cand;
-
-	now = jiffies_to_msecs(jiffies);
-	th_period = sysctl_numa_balancing_scan_period_max;
-	start = pgdat->nbp_th_start;
-	if (now - start > th_period &&
-	    cmpxchg(&pgdat->nbp_th_start, start, now) == start) {
-		ref_cand = rate_limit *
-			sysctl_numa_balancing_scan_period_max / MSEC_PER_SEC;
-		nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
-		diff_cand = nr_cand - pgdat->nbp_th_nr_cand;
-		unit_th = ref_th * 2 / NUMA_MIGRATION_ADJUST_STEPS;
-		th = pgdat->nbp_threshold ? : ref_th;
-		if (diff_cand > ref_cand * 11 / 10)
-			th = max(th - unit_th, unit_th);
-		else if (diff_cand < ref_cand * 9 / 10)
-			th = min(th + unit_th, ref_th * 2);
-		pgdat->nbp_th_nr_cand = nr_cand;
-		pgdat->nbp_threshold = th;
-	}
-}
-
 bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 				int src_nid, int dst_cpu)
 {
@@ -2688,41 +2556,15 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 
 	/*
 	 * The pages in slow memory node should be migrated according
-	 * to hot/cold instead of private/shared.
-	 */
-	if (folio_use_access_time(folio)) {
-		struct pglist_data *pgdat;
-		unsigned long rate_limit;
-		unsigned int latency, th, def_th;
-		long nr = folio_nr_pages(folio);
-
-		pgdat = NODE_DATA(dst_nid);
-		if (pgdat_free_space_enough(pgdat)) {
-			/* workload changed, reset hot threshold */
-			pgdat->nbp_threshold = 0;
-			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
-			return true;
-		}
-
-		def_th = sysctl_numa_balancing_hot_threshold;
-		rate_limit = MB_TO_PAGES(sysctl_numa_balancing_promote_rate_limit);
-		numa_promotion_adjust_threshold(pgdat, rate_limit, def_th);
-
-		th = pgdat->nbp_threshold ? : def_th;
-		latency = numa_hint_fault_latency(folio);
-		if (latency >= th)
-			return false;
-
-		return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
-	}
+	 * to hot/cold instead of private/shared. Also the migration
+	 * of such pages are handled by kmigrated.
+	 */
+	if (folio_is_promo_candidate(folio))
+		return true;
 
 	this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
 	last_cpupid = folio_xchg_last_cpupid(folio, this_cpupid);
 
-	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
-	    !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid))
-		return false;
-
 	/*
 	 * Allow first faults or private faults to migrate immediately early in
 	 * the lifetime of a task. The magic number 4 is based on waiting for
@@ -3956,8 +3798,7 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	 * node for memory tiering mode.
 	 */
 	if (!node_is_toptier(mem_node) &&
-	    (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ||
-	     !cpupid_valid(last_cpupid)))
+	    (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
 		return;
 
 	/* Allocate buffer to track faults on a per-node basis */
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 56acf502ba26..84b94f143a23 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3120,7 +3120,6 @@ extern unsigned int sysctl_numa_balancing_scan_delay;
 extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
-extern unsigned int sysctl_numa_balancing_hot_threshold;
 
 #ifdef CONFIG_SCHED_HRTICK
 
diff --git a/mm/Kconfig b/mm/Kconfig
index 955a826ecfe9..61ab18f4aebb 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1511,7 +1511,7 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 config PGHOT
 	bool "Hot page tracking and promotion"
-	default n
+	default y if NUMA_BALANCING
 	depends on NUMA_MIGRATION && SPARSEMEM
 	help
 	  A sub-system to track page accesses in lower tier memory and
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 2bccb0a53a0a..c36092ebca42 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -41,6 +41,7 @@
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/pagewalk.h>
+#include <linux/pghot.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2205,7 +2206,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	int target_nid, last_cpupid;
 	pmd_t pmd, old_pmd;
-	bool writable = false;
+	bool writable = false, needs_promotion = false;
 	int flags = 0;
 
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -2232,11 +2233,29 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 *
+		 * Isolation and migration are handled by pghot. Since VMA
+		 * won't be available to kmigrated which does batched migration
+		 * from non-process context, filter shared EXEC pages here itself.
+		 */
+		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+			nid = NUMA_NO_NODE;
+		else
+			nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -2268,8 +2287,17 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 	spin_unlock(vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		/*
+		 * Even when the hint fault is handed off to pghot for
+		 * promotion, keep feeding NUMA Balancing per-task fault
+		 * statistics for scan-period adjustment.
+		 */
 		task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	}
 	return 0;
 }
 
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6dc4888a90f3..295f41f1ee16 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -400,7 +400,7 @@ static const unsigned int memcg_node_stat_items[] = {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,
 #endif
 	PGDEMOTE_KSWAPD,
@@ -1603,7 +1603,7 @@ static const struct memory_stat memory_stats[] = {
 	{ "pgscan_khugepaged",		PGSCAN_KHUGEPAGED	},
 	{ "pgscan_proactive",		PGSCAN_PROACTIVE	},
 	{ "pgrefill",			PGREFILL		},
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	{ "pgpromote_success",		PGPROMOTE_SUCCESS	},
 #endif
 };
@@ -1655,7 +1655,7 @@ static int memcg_page_state_output_unit(int item)
 	case PGSCAN_KHUGEPAGED:
 	case PGSCAN_PROACTIVE:
 	case PGREFILL:
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	case PGPROMOTE_SUCCESS:
 #endif
 		return 1;
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b..be134a32f5bf 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -51,18 +51,19 @@ static const struct bus_type memory_tier_subsys = {
 	.dev_name = "memory_tier",
 };
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 /**
- * folio_use_access_time - check if a folio reuses cpupid for page access time
+ * folio_is_promo_candidate - check if the folio qualifies for promotion
+ *
  * @folio: folio to check
  *
- * folio's _last_cpupid field is repurposed by memory tiering. In memory
- * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
- * record page access time.
+ * Checks if NUMA Balancing tiering mode is set and the folio belongs
+ * to lower tier. If so, it qualifies for promotion to toptier when
+ * it is categorized as hot.
  *
- * Return: the folio _last_cpupid is used to record page access time
+ * Return: True if the above condition is met, else False.
  */
-bool folio_use_access_time(struct folio *folio)
+bool folio_is_promo_candidate(struct folio *folio)
 {
 	return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 	       !node_is_toptier(folio_nid(folio));
diff --git a/mm/memory.c b/mm/memory.c
index ff338c2abe92..91daba4c2bc9 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -75,6 +75,7 @@
 #include <linux/perf_event.h>
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
+#include <linux/pghot.h>
 #include <linux/sched/sysctl.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
@@ -6029,10 +6030,9 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED))
 		*flags |= TNF_SHARED;
 	/*
-	 * For memory tiering mode, cpupid of slow memory page is used
-	 * to record page access time.  So use default value.
+	 * For memory tiering mode, last_cpupid is unused. So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_is_promo_candidate(folio))
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		*last_cpupid = folio_last_cpupid(folio);
@@ -6113,6 +6113,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	bool writable = false, ignore_writable = false;
 	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
+	bool needs_promotion = false;
 	int last_cpupid;
 	int target_nid;
 	pte_t pte, old_pte;
@@ -6147,12 +6148,30 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 	nr_pages = folio_nr_pages(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 *
+		 * Isolation and migration are handled by pghot. Since VMA
+		 * won't be available to kmigrated which does batched migration
+		 * from non-process context, filter shared EXEC pages here itself.
+		 */
+		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+			nid = NUMA_NO_NODE;
+		else
+			nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -6192,8 +6211,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 					    writable);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		/*
+		 * Even when the hint fault is handed off to pghot for
+		 * promotion, keep feeding NUMA Balancing per-task fault
+		 * statistics for scan-period adjustment.
+		 */
 		task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	}
 	return 0;
 }
 
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 36699fabd3c2..111ac78c2b23 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -872,9 +872,6 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	    node_is_toptier(nid))
 		return false;
 
-	if (folio_use_access_time(folio))
-		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
-
 	return true;
 }
 
diff --git a/mm/migrate.c b/mm/migrate.c
index 7ab15780226e..4b97d6997f4b 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2712,8 +2712,18 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
 		int z;
 
-		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
+		/*
+		 * Kswapd wakeup for creating headroom in toptier is done only
+		 * for hot page promotion case and not for misplaced migrations
+		 * between toptier nodes.
+		 *
+		 * In the uncommon case of using NUMA_BALANCING_NORMAL mode
+		 * to balance between lower and higher tier nodes, we end up
+		 * waking the kswapd.
+		 */
+		if (node_is_toptier(folio_nid(folio)))
 			return -EAGAIN;
+
 		for (z = pgdat->nr_zones - 1; z >= 0; z--) {
 			if (managed_zone(pgdat->node_zones + z))
 				break;
@@ -2763,6 +2773,8 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
 		    && node_is_toptier(node)) {
@@ -2828,6 +2840,8 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
 				 PGPROMOTE_SUCCESS, nr_succeeded);
 #endif
diff --git a/mm/pghot.c b/mm/pghot.c
index f66baacb2862..9babaff7c0e6 100644
--- a/mm/pghot.c
+++ b/mm/pghot.c
@@ -17,6 +17,9 @@
  * the hot pages. kmigrated runs for each lower tier node. It iterates
  * over the node's PFNs and  migrates pages marked for migration into
  * their targeted nodes.
+ *
+ * Migration rate-limiting and dynamic threshold logic implementations
+ * were moved from NUMA Balancing mode 2.
  */
 #include <linux/mm.h>
 #include <linux/migrate.h>
@@ -32,11 +35,17 @@ unsigned int pghot_freq_threshold_max = PGHOT_FREQ_MAX;
 unsigned int kmigrated_sleep_ms = KMIGRATED_SLEEP_MS_DEFAULT;
 unsigned int kmigrated_batch_nr = KMIGRATED_BATCH_NR_DEFAULT;
 
-unsigned int sysctl_pghot_src_enabled;
+unsigned int sysctl_pghot_src_enabled = PGHOT_HINTFAULTS_ENABLED;
 unsigned int sysctl_pghot_target_nid = PGHOT_DEFAULT_NODE;
 unsigned int sysctl_pghot_freq_window = PGHOT_FREQ_WINDOW_DEFAULT;
 unsigned int sysctl_pghot_freq_threshold = PGHOT_FREQ_THRESHOLD_DEFAULT;
 
+/* Restrict the NUMA promotion throughput (MB/s) for each target node. */
+static unsigned int sysctl_pghot_promote_rate_limit = 65536;
+
+#define KMIGRATED_MIGRATION_ADJUST_STEPS	16
+#define KMIGRATED_PROMOTION_THRESHOLD_WINDOW	60000
+
 DEFINE_STATIC_KEY_FALSE(pghot_src_hwhints);
 DEFINE_STATIC_KEY_FALSE(pghot_src_hintfaults);
 DEFINE_MUTEX(pghot_tunables_lock);
@@ -144,11 +153,35 @@ static const struct ctl_table pghot_sysctls[] = {
 		.extra1		= SYSCTL_ONE,
 		.extra2		= &pghot_freq_threshold_max,
 	},
+	{
+		.procname	= "pghot_promote_rate_limit_MBps",
+		.data		= &sysctl_pghot_promote_rate_limit,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+	},
+};
+
+/*
+ * numa_balancing_promote_rate_limit_MBps has been replaced by
+ * pghot_promote_rate_limit_MBps but retained here for compatibility.
+ */
+static const struct ctl_table compat_pghot_sysctls[] = {
+	{
+		.procname	= "numa_balancing_promote_rate_limit_MBps",
+		.data		= &sysctl_pghot_promote_rate_limit,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+	},
 };
 
 static void __init pghot_sysctl_init(void)
 {
 	register_sysctl_init("vm", pghot_sysctls);
+	register_sysctl_init("kernel", compat_pghot_sysctls);
 }
 #else
 static inline void pghot_sysctl_init(void) { }
@@ -258,6 +291,110 @@ int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
 	return 0;
 }
 
+/*
+ * For memory tiering mode, if there are enough free pages (more than
+ * enough watermark defined here) in fast memory node, to take full
+ * advantage of fast memory capacity, all recently accessed slow
+ * memory pages will be migrated to fast memory node without
+ * considering hot threshold.
+ */
+static bool pgdat_free_space_enough(struct pglist_data *pgdat)
+{
+	int z;
+	unsigned long enough_wmark;
+
+	enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,
+			   pgdat->node_present_pages >> 4);
+	for (z = pgdat->nr_zones - 1; z >= 0; z--) {
+		struct zone *zone = pgdat->node_zones + z;
+
+		if (!populated_zone(zone))
+			continue;
+
+		if (zone_watermark_ok(zone, 0,
+				      promo_wmark_pages(zone) + enough_wmark,
+				      ZONE_MOVABLE, 0))
+			return true;
+	}
+	return false;
+}
+
+/*
+ * For memory tiering mode, too high promotion/demotion throughput may
+ * hurt application latency.  So we provide a mechanism to rate limit
+ * the number of pages that are tried to be promoted.
+ */
+static bool kmigrated_promotion_rate_limit(struct pglist_data *pgdat, unsigned long rate_limit,
+					   int nr, unsigned int now_ms)
+{
+	unsigned long nr_cand;
+	unsigned int start;
+
+	mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE, nr);
+	nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
+	start = pgdat->nbp_rl_start;
+	if (now_ms - start > MSEC_PER_SEC &&
+	    cmpxchg(&pgdat->nbp_rl_start, start, now_ms) == start)
+		pgdat->nbp_rl_nr_cand = nr_cand;
+	if (nr_cand - pgdat->nbp_rl_nr_cand >= rate_limit)
+		return true;
+	return false;
+}
+
+static void kmigrated_promotion_adjust_threshold(struct pglist_data *pgdat,
+						 unsigned long rate_limit, unsigned int ref_th,
+						 unsigned int now_ms)
+{
+	unsigned int start, th_period, unit_th, th;
+	unsigned long nr_cand, ref_cand, diff_cand;
+
+	th_period = KMIGRATED_PROMOTION_THRESHOLD_WINDOW;
+	start = pgdat->nbp_th_start;
+	if (now_ms - start > th_period &&
+	    cmpxchg(&pgdat->nbp_th_start, start, now_ms) == start) {
+		ref_cand = rate_limit *
+			KMIGRATED_PROMOTION_THRESHOLD_WINDOW / MSEC_PER_SEC;
+		nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
+		diff_cand = nr_cand - pgdat->nbp_th_nr_cand;
+		unit_th = ref_th * 2 / KMIGRATED_MIGRATION_ADJUST_STEPS;
+		th = pgdat->nbp_threshold ? : ref_th;
+		if (diff_cand > ref_cand * 11 / 10)
+			th = max(th - unit_th, unit_th);
+		else if (diff_cand < ref_cand * 9 / 10)
+			th = min(th + unit_th, ref_th * 2);
+		pgdat->nbp_th_nr_cand = nr_cand;
+		pgdat->nbp_threshold = th;
+	}
+}
+
+static bool kmigrated_should_migrate_memory(unsigned long nr_pages, int nid,
+					    unsigned long time)
+{
+	struct pglist_data *pgdat;
+	unsigned long rate_limit;
+	unsigned int th, def_th;
+	unsigned int now_ms = jiffies_to_msecs(jiffies); /* Based on full-width jiffies */
+	unsigned long now = jiffies;
+
+	pgdat = NODE_DATA(nid);
+	if (pgdat_free_space_enough(pgdat)) {
+		/* workload changed, reset hot threshold */
+		pgdat->nbp_threshold = 0;
+		mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr_pages);
+		return true;
+	}
+
+	def_th = sysctl_pghot_freq_window;
+	rate_limit = MB_TO_PAGES(sysctl_pghot_promote_rate_limit);
+	kmigrated_promotion_adjust_threshold(pgdat, rate_limit, def_th, now_ms);
+
+	th = pgdat->nbp_threshold ? : def_th;
+	if (pghot_access_latency(time, now) >= th)
+		return false;
+
+	return !kmigrated_promotion_rate_limit(pgdat, rate_limit, nr_pages, now_ms);
+}
+
 static int pghot_get_hotness(unsigned long pfn, int *nid, int *freq,
 			     unsigned long *time)
 {
@@ -340,6 +477,11 @@ static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
 			goto out_next;
 		}
 
+		if (!kmigrated_should_migrate_memory(nr, nid, time)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
 		if (migrate_misplaced_folio_prepare(folio, NULL, nid)) {
 			folio_put(folio);
 			goto out_next;
@@ -692,6 +834,24 @@ static void pghot_debug_init(void)
 			    &pghot_kmigrated_batch_nr_fops);
 }
 
+/*
+ * Sync the hotness-source static keys with the initial value of
+ * sysctl_pghot_src_enabled. Hint faults are enabled by default so that
+ * selecting the NUMA Balancing memory tiering mode (numa_balancing=2)
+ * promotes hot pages without any additional opt-in, matching the
+ * pre-pghot behaviour.
+ *
+ * This runs unconditionally (not only under CONFIG_SYSCTL) because the
+ * static key, not the sysctl variable, gates pghot_record_access().
+ */
+static void __init pghot_src_enabled_init(void)
+{
+	if (sysctl_pghot_src_enabled & PGHOT_HINTFAULTS_ENABLED)
+		static_branch_enable(&pghot_src_hintfaults);
+	if (sysctl_pghot_src_enabled & PGHOT_HWHINTS_ENABLED)
+		static_branch_enable(&pghot_src_hwhints);
+}
+
 static int __init pghot_init(void)
 {
 	pg_data_t *pgdat;
@@ -711,6 +871,7 @@ static int __init pghot_init(void)
 	}
 	pghot_sysctl_init();
 	pghot_debug_init();
+	pghot_src_enabled_init();
 
 	kmigrated_started = true;
 	return 0;
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 4064ead568cc..da668ff05032 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1268,7 +1268,7 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_SWAP
 	[I(NR_SWAPCACHE)]			= "nr_swapcached",
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	[I(PGPROMOTE_SUCCESS)]			= "pgpromote_success",
 	[I(PGPROMOTE_CANDIDATE)]		= "pgpromote_candidate",
 	[I(PGPROMOTE_CANDIDATE_NRL)]		= "pgpromote_candidate_nrl",
-- 
2.34.1

[PATCH v8 6/8] x86/ibs: Move IBS caps definitions into its own header

[Bharata B Rao]

Subsequent patch adds IBS Memory Profiler driver that is
independent of the perf subsystem but needs the CPUID
0x8000001B capability bits. Hence move those bit definitions
out of asm/perf_event.h into a dedicated header so the new
driver can consume them without pulling in perf.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 arch/x86/include/asm/ibs-caps.h   | 85 +++++++++++++++++++++++++++++++
 arch/x86/include/asm/perf_event.h | 81 +----------------------------
 2 files changed, 86 insertions(+), 80 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs-caps.h

diff --git a/arch/x86/include/asm/ibs-caps.h b/arch/x86/include/asm/ibs-caps.h
new file mode 100644
index 000000000000..ddf6c512c8f9
--- /dev/null
+++ b/arch/x86/include/asm/ibs-caps.h
@@ -0,0 +1,85 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_IBS_CAPS_H
+#define _ASM_X86_IBS_CAPS_H
+
+/*
+ * IBS cpuid feature detection
+ */
+
+#define IBS_CPUID_FEATURES		0x8000001b
+
+/*
+ * Same bit mask as for IBS cpuid feature flags (Fn8000_001B_EAX), but
+ * bit 0 is used to indicate the existence of IBS.
+ */
+#define IBS_CAPS_AVAIL			(1U<<0)
+#define IBS_CAPS_FETCHSAM		(1U<<1)
+#define IBS_CAPS_OPSAM			(1U<<2)
+#define IBS_CAPS_RDWROPCNT		(1U<<3)
+#define IBS_CAPS_OPCNT			(1U<<4)
+#define IBS_CAPS_BRNTRGT		(1U<<5)
+#define IBS_CAPS_OPCNTEXT		(1U<<6)
+#define IBS_CAPS_RIPINVALIDCHK		(1U<<7)
+#define IBS_CAPS_OPBRNFUSE		(1U<<8)
+#define IBS_CAPS_FETCHCTLEXTD		(1U<<9)
+#define IBS_CAPS_OPDATA4		(1U<<10)
+#define IBS_CAPS_ZEN4			(1U<<11)
+#define IBS_CAPS_OPLDLAT		(1U<<12)
+#define IBS_CAPS_DIS			(1U<<13)
+#define IBS_CAPS_FETCHLAT		(1U<<14)
+#define IBS_CAPS_BIT63_FILTER		(1U<<15)
+#define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
+#define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
+
+#define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
+					 | IBS_CAPS_FETCHSAM	\
+					 | IBS_CAPS_OPSAM)
+
+/*
+ * IBS APIC setup
+ */
+#define IBSCTL				0x1cc
+#define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
+#define IBSCTL_LVT_OFFSET_MASK		0x0F
+
+/* IBS fetch bits/masks */
+#define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
+#define IBS_FETCH_RAND_EN		      (1ULL << 57)
+#define IBS_FETCH_VAL			      (1ULL << 49)
+#define IBS_FETCH_ENABLE		      (1ULL << 48)
+#define IBS_FETCH_CNT			     0xFFFF0000ULL
+#define IBS_FETCH_MAX_CNT		     0x0000FFFFULL
+
+#define IBS_FETCH_2_DIS			      (1ULL <<  0)
+#define IBS_FETCH_2_FETCHLAT_FILTER	    (0xFULL <<  1)
+#define IBS_FETCH_2_FETCHLAT_FILTER_SHIFT	       (1)
+#define IBS_FETCH_2_EXCL_RIP_63_EQ_1	      (1ULL <<  5)
+#define IBS_FETCH_2_EXCL_RIP_63_EQ_0	      (1ULL <<  6)
+
+/*
+ * IBS op bits/masks
+ * The lower 7 bits of the current count are random bits
+ * preloaded by hardware and ignored in software
+ */
+#define IBS_OP_LDLAT_EN			      (1ULL << 63)
+#define IBS_OP_LDLAT_THRSH		    (0xFULL << 59)
+#define IBS_OP_LDLAT_THRSH_SHIFT		      (59)
+#define IBS_OP_CUR_CNT			(0xFFF80ULL << 32)
+#define IBS_OP_CUR_CNT_RAND		(0x0007FULL << 32)
+#define IBS_OP_CUR_CNT_EXT_MASK		   (0x7FULL << 52)
+#define IBS_OP_CNT_CTL			      (1ULL << 19)
+#define IBS_OP_VAL			      (1ULL << 18)
+#define IBS_OP_ENABLE			      (1ULL << 17)
+#define IBS_OP_L3MISSONLY		      (1ULL << 16)
+#define IBS_OP_MAX_CNT			     0x0000FFFFULL
+#define IBS_OP_MAX_CNT_EXT		     0x007FFFFFULL	/* not a register bit mask */
+#define IBS_OP_MAX_CNT_EXT_MASK		   (0x7FULL << 20)	/* separate upper 7 bits */
+#define IBS_RIP_INVALID			      (1ULL << 38)
+
+#define IBS_OP_2_DIS			      (1ULL <<  0)
+#define IBS_OP_2_EXCL_RIP_63_EQ_0	      (1ULL <<  1)
+#define IBS_OP_2_EXCL_RIP_63_EQ_1	      (1ULL <<  2)
+#define IBS_OP_2_STRM_ST_FILTER		      (1ULL <<  3)
+#define IBS_OP_2_STRM_ST_FILTER_SHIFT		       (3)
+
+#endif /*  _ASM_X86_IBS_CAPS_H */
diff --git a/arch/x86/include/asm/perf_event.h b/arch/x86/include/asm/perf_event.h
index 1eb13673e889..de18e63c5c3e 100644
--- a/arch/x86/include/asm/perf_event.h
+++ b/arch/x86/include/asm/perf_event.h
@@ -3,6 +3,7 @@
 #define _ASM_X86_PERF_EVENT_H
 
 #include <linux/static_call.h>
+#include <asm/ibs-caps.h>
 
 /*
  * Performance event hw details:
@@ -622,86 +623,6 @@ struct arch_pebs_cntr_header {
  */
 #define EXT_PERFMON_DEBUG_FEATURES		0x80000022
 
-/*
- * IBS cpuid feature detection
- */
-
-#define IBS_CPUID_FEATURES		0x8000001b
-
-/*
- * Same bit mask as for IBS cpuid feature flags (Fn8000_001B_EAX), but
- * bit 0 is used to indicate the existence of IBS.
- */
-#define IBS_CAPS_AVAIL			(1U<<0)
-#define IBS_CAPS_FETCHSAM		(1U<<1)
-#define IBS_CAPS_OPSAM			(1U<<2)
-#define IBS_CAPS_RDWROPCNT		(1U<<3)
-#define IBS_CAPS_OPCNT			(1U<<4)
-#define IBS_CAPS_BRNTRGT		(1U<<5)
-#define IBS_CAPS_OPCNTEXT		(1U<<6)
-#define IBS_CAPS_RIPINVALIDCHK		(1U<<7)
-#define IBS_CAPS_OPBRNFUSE		(1U<<8)
-#define IBS_CAPS_FETCHCTLEXTD		(1U<<9)
-#define IBS_CAPS_OPDATA4		(1U<<10)
-#define IBS_CAPS_ZEN4			(1U<<11)
-#define IBS_CAPS_OPLDLAT		(1U<<12)
-#define IBS_CAPS_DIS			(1U<<13)
-#define IBS_CAPS_FETCHLAT		(1U<<14)
-#define IBS_CAPS_BIT63_FILTER		(1U<<15)
-#define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
-#define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
-
-#define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
-					 | IBS_CAPS_FETCHSAM	\
-					 | IBS_CAPS_OPSAM)
-
-/*
- * IBS APIC setup
- */
-#define IBSCTL				0x1cc
-#define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
-#define IBSCTL_LVT_OFFSET_MASK		0x0F
-
-/* IBS fetch bits/masks */
-#define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
-#define IBS_FETCH_RAND_EN		      (1ULL << 57)
-#define IBS_FETCH_VAL			      (1ULL << 49)
-#define IBS_FETCH_ENABLE		      (1ULL << 48)
-#define IBS_FETCH_CNT			     0xFFFF0000ULL
-#define IBS_FETCH_MAX_CNT		     0x0000FFFFULL
-
-#define IBS_FETCH_2_DIS			      (1ULL <<  0)
-#define IBS_FETCH_2_FETCHLAT_FILTER	    (0xFULL <<  1)
-#define IBS_FETCH_2_FETCHLAT_FILTER_SHIFT	       (1)
-#define IBS_FETCH_2_EXCL_RIP_63_EQ_1	      (1ULL <<  5)
-#define IBS_FETCH_2_EXCL_RIP_63_EQ_0	      (1ULL <<  6)
-
-/*
- * IBS op bits/masks
- * The lower 7 bits of the current count are random bits
- * preloaded by hardware and ignored in software
- */
-#define IBS_OP_LDLAT_EN			      (1ULL << 63)
-#define IBS_OP_LDLAT_THRSH		    (0xFULL << 59)
-#define IBS_OP_LDLAT_THRSH_SHIFT		      (59)
-#define IBS_OP_CUR_CNT			(0xFFF80ULL << 32)
-#define IBS_OP_CUR_CNT_RAND		(0x0007FULL << 32)
-#define IBS_OP_CUR_CNT_EXT_MASK		   (0x7FULL << 52)
-#define IBS_OP_CNT_CTL			      (1ULL << 19)
-#define IBS_OP_VAL			      (1ULL << 18)
-#define IBS_OP_ENABLE			      (1ULL << 17)
-#define IBS_OP_L3MISSONLY		      (1ULL << 16)
-#define IBS_OP_MAX_CNT			     0x0000FFFFULL
-#define IBS_OP_MAX_CNT_EXT		     0x007FFFFFULL	/* not a register bit mask */
-#define IBS_OP_MAX_CNT_EXT_MASK		   (0x7FULL << 20)	/* separate upper 7 bits */
-#define IBS_RIP_INVALID			      (1ULL << 38)
-
-#define IBS_OP_2_DIS			      (1ULL <<  0)
-#define IBS_OP_2_EXCL_RIP_63_EQ_0	      (1ULL <<  1)
-#define IBS_OP_2_EXCL_RIP_63_EQ_1	      (1ULL <<  2)
-#define IBS_OP_2_STRM_ST_FILTER		      (1ULL <<  3)
-#define IBS_OP_2_STRM_ST_FILTER_SHIFT		       (3)
-
 #ifdef CONFIG_X86_LOCAL_APIC
 extern u32 get_ibs_caps(void);
 extern int forward_event_to_ibs(struct perf_event *event);
-- 
2.34.1

[PATCH v8 7/8] x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler

[Bharata B Rao]

Use IBS (Instruction Based Sampling) Memory Profiler feature
present in AMD Zen6 processors for memory access tracking. The
access information obtained from IBS Memory Profiler is fed to
pghot sub-system for further action using
pghot_record_access(PGHOT_HWHINTS, ...) API.

IBS Memory Profiler as page hotness source is enabled by the
new config option HWMEM_PROFILER and is also gated by the
existing pghot_src_hwhints static key set via sysctl tunable
vm.pghot_enabled_sources.

New system vector 0xea is used for memprofiler interrupt.

This adds the driver infrastructure (interrupt setup, per-CPU
ring, workqueue, cpuhp state, pghot call site etc) but the
profiler isn't armed yet and that is done in the next commit.

More details about IBS Memory Profiler can be obtained from
the AMD document titled "AMD64 Zen6 Instruction Based Sampling (IBS)
Extensions and Features".

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 Documentation/admin-guide/mm/pghot.rst |  44 ++-
 arch/x86/Kconfig                       |  16 ++
 arch/x86/entry/entry_fred.c            |   1 +
 arch/x86/include/asm/hardirq.h         |   3 +
 arch/x86/include/asm/ibs-caps.h        |   8 +
 arch/x86/include/asm/ibs-mprof.h       |  47 ++++
 arch/x86/include/asm/idtentry.h        |   6 +
 arch/x86/include/asm/irq_vectors.h     |   4 +-
 arch/x86/include/asm/msr-index.h       |   8 +
 arch/x86/kernel/idt.c                  |   3 +
 arch/x86/kernel/irq.c                  |   3 +
 arch/x86/mm/Makefile                   |   1 +
 arch/x86/mm/ibs-mprof.c                | 363 +++++++++++++++++++++++++
 include/linux/cpuhotplug.h             |   1 +
 include/linux/vm_event_item.h          |   7 +
 mm/Kconfig                             |   9 +
 mm/vmstat.c                            |   7 +
 17 files changed, 529 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c

diff --git a/Documentation/admin-guide/mm/pghot.rst b/Documentation/admin-guide/mm/pghot.rst
index accf78d95d02..b802b80e5c24 100644
--- a/Documentation/admin-guide/mm/pghot.rst
+++ b/Documentation/admin-guide/mm/pghot.rst
@@ -87,7 +87,7 @@ Path: /proc/vmstat
 
 2. **pghot_recorded_hintfaults**
    - Number of recorded accesses reported by NUMA Balancing based
-     hotness source.
+     hint faults source.
 
 3. **pghot_recorded_hwhints**
    - Number of recorded accesses reported by hwhints source.
@@ -108,3 +108,45 @@ to promote hot pages. It can be disabled at runtime by clearing that bit:
 
 # echo 0x0 > /proc/sys/vm/pghot_enabled_sources
 
+Hardware Hints Source
+=====================
+pghot can consume memory access samples reported by hardware profilers.
+This "hardware hints" (hwhints) source feeds hardware-observed accesses
+into pghot for hot page detection and promotion.
+
+Generic in-kernel support for such profilers is controlled by the
+HWMEM_PROFILER config option. It is not enabled directly by the user;
+an in-kernel driver that forwards hardware-observed accesses to pghot
+selects it.
+
+The AMD IBS Memory Profiler (config AMD_IBS_MEMPROF, available on Zen6
+and later AMD CPUs) is one such driver. It uses the AMD Instruction
+Based Sampling (IBS) Memory Profiler facility to sample user memory
+accesses and record them with pghot.
+
+This source can be activated at runtime through the hardware
+hints bit (0x2) of **pghot_enabled_sources**.
+
+# echo 0x2 > /proc/sys/vm/pghot_enabled_sources
+
+HWHINTS Vmstat Counters
+=======================
+Following vmstat counters provide some stats about hardware hints source.
+
+Path: /proc/vmstat
+
+1. **hwhint_total_events**
+   - Number of total hwhint events recorded by hwhints source.
+
+2. **hwhint_useful_events**
+   - Number of actionable events from hwhints source.
+
+3. **hwhint_dropped_events**
+   - Number of events that were dropped due to buffer overrun.
+
+4. **hwhint_dram_accesses**
+   - Number of DRAM accesses reported by hwhints source.
+
+5. **hwhint_extmem_accesses**
+   - Number of external memory (like CXL) accesses reported by hwhints source.
+
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index bdad90f210e4..c6c5ed83506a 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1501,6 +1501,22 @@ config AMD_MEM_ENCRYPT
 	  This requires an AMD processor that supports Secure Memory
 	  Encryption (SME).
 
+config AMD_IBS_MEMPROF
+	bool "AMD IBS Memory Profiler"
+	depends on X86_64 && CPU_SUP_AMD
+	depends on PGHOT
+	select HWMEM_PROFILER
+	help
+	  Use the AMD Instruction Based Sampling (IBS) Memory Profiler
+	  facility (present on Zen6 and later AMD CPUs) to feed
+	  hardware-observed memory accesses into the pghot subsystem
+	  for hot-page detection and promotion.
+
+	  When disabled, no IBS Memory Profiler MSRs are programmed and
+	  the corresponding interrupt handler is not installed.
+
+	  If unsure, say N.
+
 # Common NUMA Features
 config NUMA
 	bool "NUMA Memory Allocation and Scheduler Support"
diff --git a/arch/x86/entry/entry_fred.c b/arch/x86/entry/entry_fred.c
index fb3594ddf731..f52099e3fa85 100644
--- a/arch/x86/entry/entry_fred.c
+++ b/arch/x86/entry/entry_fred.c
@@ -120,6 +120,7 @@ static idtentry_t sysvec_table[NR_SYSTEM_VECTORS] __ro_after_init = {
 	SYSVEC(POSTED_INTR_NESTED_VECTOR,	kvm_posted_intr_nested_ipi),
 
 	SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR,	posted_msi_notification),
+	SYSVEC(IBS_MEMPROF_VECTOR,		ibs_memprof),
 };
 
 static bool fred_setup_done __initdata;
diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h
index dea60d66d976..2da4930ea399 100644
--- a/arch/x86/include/asm/hardirq.h
+++ b/arch/x86/include/asm/hardirq.h
@@ -49,6 +49,9 @@ enum irq_stat_counts {
 #endif
 #ifdef CONFIG_X86_POSTED_MSI
 	IRQ_COUNT_POSTED_MSI_NOTIFICATION,
+#endif
+#ifdef CONFIG_AMD_IBS_MEMPROF
+	IRQ_COUNT_IBS_MEMPROF,
 #endif
 	IRQ_COUNT_PIC_APIC_ERROR,
 #ifdef CONFIG_X86_IO_APIC
diff --git a/arch/x86/include/asm/ibs-caps.h b/arch/x86/include/asm/ibs-caps.h
index ddf6c512c8f9..1f6c4058a0e3 100644
--- a/arch/x86/include/asm/ibs-caps.h
+++ b/arch/x86/include/asm/ibs-caps.h
@@ -29,6 +29,7 @@
 #define IBS_CAPS_FETCHLAT		(1U<<14)
 #define IBS_CAPS_BIT63_FILTER		(1U<<15)
 #define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
+#define IBS_CAPS_MEM_PROFILER		(1U<<18)
 #define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
 
 #define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
@@ -42,6 +43,13 @@
 #define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
 #define IBSCTL_LVT_OFFSET_MASK		0x0F
 
+/*
+ * IBS Memprofiler setup
+ */
+#define IBSCTL_MPROF_LVT_OFFSET_VALID	(1ULL << 24)
+#define IBSCTL_MPROF_LVT_OFFSET_SHIFT	16
+#define IBSCTL_MPROF_LVT_OFFSET_MASK	(0xFULL << IBSCTL_MPROF_LVT_OFFSET_SHIFT)
+
 /* IBS fetch bits/masks */
 #define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
 #define IBS_FETCH_RAND_EN		      (1ULL << 57)
diff --git a/arch/x86/include/asm/ibs-mprof.h b/arch/x86/include/asm/ibs-mprof.h
new file mode 100644
index 000000000000..3de1880c4126
--- /dev/null
+++ b/arch/x86/include/asm/ibs-mprof.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_IBS_MPROF_H
+#define _ASM_X86_IBS_MPROF_H
+
+/*
+ * All bits are documented here for clarity even if the current
+ * driver doesn't use all of them.
+ */
+
+/* MSR_AMD64_IBS_MPROF_DATA2 bits */
+#define IBS_MPROF_DATA2_DATASRC_MASK		0x7
+#define IBS_MPROF_DATA2_DATASRC_MASK_HIGH	0xC0
+#define IBS_MPROF_DATA2_DATASRC_MASK_HIGH_SHIFT	0x3
+#define IBS_MPROF_DATA2_DATASRC_LCL_CCX		0x1
+#define IBS_MPROF_DATA2_DATASRC_PEER_CCX_NEAR	0x2
+#define IBS_MPROF_DATA2_DATASRC_DRAM		0x3
+#define IBS_MPROF_DATA2_DATASRC_CCX_FAR		0x5
+#define IBS_MPROF_DATA2_DATASRC_DRAM_FAR	0x7
+#define IBS_MPROF_DATA2_DATASRC_EXT_MEM		0x8
+#define IBS_MPROF_DATA2_RMT_NODE		BIT_ULL(4)
+#define IBS_MPROF_DATA2_RMT_SOCKET		BIT_ULL(9)
+
+/* MSR_AMD64_IBS_MPROF_DATA3 bits */
+#define IBS_MPROF_DATA3_LDOP		BIT_ULL(0)
+#define IBS_MPROF_DATA3_STOP		BIT_ULL(1)
+#define IBS_MPROF_DATA3_DCMISS		BIT_ULL(7)
+#define IBS_MPROF_DATA3_LADDR_VALID	BIT_ULL(17)
+#define IBS_MPROF_DATA3_PADDR_VALID	BIT_ULL(18)
+#define IBS_MPROF_DATA3_L2MISS		BIT_ULL(20)
+#define IBS_MPROF_DATA3_SW_PREFETCH	BIT_ULL(21)
+
+/* MSR_AMD64_IBS_MPROF_CTL bits */
+#define IBS_MPROF_CTL_CNT_CTL		BIT_ULL(19)
+#define IBS_MPROF_CTL_VAL		BIT_ULL(18)
+#define IBS_MPROF_CTL_ENABLE		BIT_ULL(17)
+#define IBS_MPROF_CTL_L3MISSONLY	BIT_ULL(16)
+#define IBS_MPROF_CTL_MAXCNT_MASK	0x0000FFFFULL
+#define IBS_MPROF_CTL_MAXCNT_EXT_MASK	(0x7FULL << 20)	/* separate upper 7 bits */
+
+/* MSR_AMD64_IBS_MPROF_CTL2 bits */
+#define IBS_MPROF_CTL2_DISABLE		BIT_ULL(0)
+#define IBS_MPROF_CTL2_EXCLUDE_USER	BIT_ULL(1)
+#define IBS_MPROF_CTL2_EXCLUDE_KERNEL	BIT_ULL(2)
+
+#define IBS_MPROF_SAMPLE_PERIOD	10000
+
+#endif /* _ASM_X86_IBS_MPROF_H */
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 20f548702404..4c82e5c6d759 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -735,6 +735,12 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR,	sysvec_posted_msi_notifi
 # define fred_sysvec_posted_msi_notification		NULL
 # endif
 
+# ifdef CONFIG_AMD_IBS_MEMPROF
+DECLARE_IDTENTRY_SYSVEC(IBS_MEMPROF_VECTOR,		sysvec_ibs_memprof);
+#else
+# define fred_sysvec_ibs_memprof			NULL
+#endif
+
 #if IS_ENABLED(CONFIG_HYPERV)
 DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_CALLBACK_VECTOR,	sysvec_hyperv_callback);
 DECLARE_IDTENTRY_SYSVEC(HYPERV_REENLIGHTENMENT_VECTOR,	sysvec_hyperv_reenlightenment);
diff --git a/arch/x86/include/asm/irq_vectors.h b/arch/x86/include/asm/irq_vectors.h
index 85253fc8e384..575f880cb147 100644
--- a/arch/x86/include/asm/irq_vectors.h
+++ b/arch/x86/include/asm/irq_vectors.h
@@ -105,10 +105,12 @@
  */
 #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
 
+#define IBS_MEMPROF_VECTOR		0xea
+
 #define NR_VECTORS			 256
 
 #ifdef CONFIG_X86_LOCAL_APIC
-#define FIRST_SYSTEM_VECTOR		POSTED_MSI_NOTIFICATION_VECTOR
+#define FIRST_SYSTEM_VECTOR		IBS_MEMPROF_VECTOR
 #else
 #define FIRST_SYSTEM_VECTOR		NR_VECTORS
 #endif
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 18c4be75e927..58d1bd7ebe72 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -1317,4 +1317,12 @@
 						* a #GP
 						*/
 
+/* AMD IBS Memory Profiler MSRs */
+#define MSR_AMD64_IBS_MPROF_CTL		0xc0010380
+#define MSR_AMD64_IBS_MPROF_CTL2	0xc0010381
+#define MSR_AMD64_IBS_MPROF_DATA2	0xc0010382
+#define MSR_AMD64_IBS_MPROF_DATA3	0xc0010383
+#define MSR_AMD64_IBS_MPROF_LINADDR	0xc0010384
+#define MSR_AMD64_IBS_MPROF_PHYADDR	0xc0010385
+
 #endif /* _ASM_X86_MSR_INDEX_H */
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 90a22e24a9eb..50ec5fc82619 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -169,6 +169,9 @@ static const __initconst struct idt_data apic_idts[] = {
 # ifdef CONFIG_X86_POSTED_MSI
 	INTG(POSTED_MSI_NOTIFICATION_VECTOR,	asm_sysvec_posted_msi_notification),
 # endif
+#ifdef CONFIG_AMD_IBS_MEMPROF
+	INTG(IBS_MEMPROF_VECTOR,		asm_sysvec_ibs_memprof),
+#endif
 #endif
 };
 
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index 30122f0b3af9..c59a474d9202 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -121,6 +121,9 @@ static const struct irq_stat_info irq_stat_info[IRQ_COUNT_MAX] = {
 #endif
 #ifdef CONFIG_X86_POSTED_MSI
 	ISS(POSTED_MSI_NOTIFICATION,	"PMN",	"  Posted MSI notification event\n"),
+#endif
+#ifdef CONFIG_AMD_IBS_MEMPROF
+	ISS(IBS_MEMPROF,		"IMP",	"  IBS Memory Profiler interrupts\n"),
 #endif
 	IDS(PIC_APIC_ERROR,		"ERR",	"  PIC/APIC error interrupts\n"),
 #ifdef CONFIG_X86_IO_APIC
diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
index 3a5364853eab..050a7379d9f7 100644
--- a/arch/x86/mm/Makefile
+++ b/arch/x86/mm/Makefile
@@ -59,3 +59,4 @@ obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
 
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_boot.o
+obj-$(CONFIG_AMD_IBS_MEMPROF)	+= ibs-mprof.o
diff --git a/arch/x86/mm/ibs-mprof.c b/arch/x86/mm/ibs-mprof.c
new file mode 100644
index 000000000000..923fb8f99552
--- /dev/null
+++ b/arch/x86/mm/ibs-mprof.c
@@ -0,0 +1,363 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define pr_fmt(fmt)	"amd_ibs_memprof: " fmt
+
+#include <linux/init.h>
+#include <linux/pghot.h>
+#include <linux/percpu.h>
+#include <linux/workqueue.h>
+#include <linux/mm.h>
+#include <linux/vm_event_item.h>
+#include <linux/vmstat.h>
+#include <linux/cpuhotplug.h>
+
+#include <asm/ibs-mprof.h>
+#include <asm/ibs-caps.h>
+#include <asm/irq_vectors.h>
+#include <asm/idtentry.h>
+#include <asm/apic.h>
+#include <asm/cpuid/api.h>
+
+#define IBS_NR_SAMPLES		150	/* Percpu sample buffer size */
+
+static DEFINE_PER_CPU(bool, mprof_work_pending);
+
+/*
+ * Basic access info captured for each memory access.
+ */
+struct mprof_sample {
+	unsigned long pfn;
+	unsigned long time;	/* jiffies when accessed */
+	int nid;		/* Accessing node ID, if known */
+};
+
+/*
+ * Percpu buffer of access samples. Samples are accumulated here
+ * before pushing them to pghot sub-system for further action.
+ */
+struct mprof_sample_pcpu {
+	struct mprof_sample samples[IBS_NR_SAMPLES];
+	int head, tail;
+};
+
+static struct mprof_sample_pcpu __percpu *mprof_s;
+
+/*
+ * Per-CPU work for pushing the percpu access samples to pghot sub-system.
+ *
+ * @cpu records which CPU's sample ring this work item is responsible for
+ * draining.
+ */
+struct mprof_worker {
+	struct work_struct work;
+	unsigned int cpu;
+};
+static DEFINE_PER_CPU(struct mprof_worker, mprof_work);
+
+/*
+ * Record the IBS-reported access sample in percpu buffer.
+ * Called from IBS interrupt handler.
+ */
+static bool mprof_push_sample(unsigned long pfn, int nid, unsigned long time)
+{
+	struct mprof_sample_pcpu *pcpu = raw_cpu_ptr(mprof_s);
+	int head = READ_ONCE(pcpu->head);
+	int tail = READ_ONCE(pcpu->tail);
+	int next = head + 1;
+
+	if (next >= IBS_NR_SAMPLES)
+		next = 0;
+
+	if (next == tail) {
+		count_vm_event(HWHINT_DROPPED_EVENTS);
+		return false;
+	}
+
+	pcpu->samples[head].pfn = pfn;
+	pcpu->samples[head].time = time;
+	pcpu->samples[head].nid = nid;
+
+	/*
+	 * Publish the sample slot stores before advancing head; pairs
+	 * with the smp_load_acquire() of head in mprof_pop_sample().
+	 */
+	smp_store_release(&pcpu->head, next);
+	return true;
+}
+
+static bool mprof_pop_sample(struct mprof_sample_pcpu *pcpu, struct mprof_sample *s)
+{
+	int tail = READ_ONCE(pcpu->tail);
+	/*
+	 * Pairs with the smp_store_release() of head in mprof_push_sample();
+	 * ensures the sample slot stores are visible before we read the slot.
+	 */
+	int head = smp_load_acquire(&pcpu->head);
+	int next = tail + 1;
+
+	if (head == tail)
+		return false;
+
+	if (next >= IBS_NR_SAMPLES)
+		next = 0;
+
+	*s = pcpu->samples[tail];
+
+	WRITE_ONCE(pcpu->tail, next);
+	return true;
+}
+
+/*
+ * Remove access samples from percpu buffer and send them
+ * to pghot sub-system for further action.
+ */
+static void mprof_work_handler(struct work_struct *work)
+{
+	struct mprof_worker *mw = container_of(work, struct mprof_worker, work);
+	unsigned int cpu = mw->cpu;
+	struct mprof_sample_pcpu *pcpu = per_cpu_ptr(mprof_s, cpu);
+	struct mprof_sample s;
+
+	for (;;) {
+		while (mprof_pop_sample(pcpu, &s))
+			pghot_record_access(s.pfn, s.nid, PGHOT_HWHINTS,
+					    s.time);
+
+		per_cpu(mprof_work_pending, cpu) = false;
+
+		/*
+		 * Publish the cleared pending flag before re-checking the
+		 * ring.
+		 */
+		smp_mb();
+
+		if (READ_ONCE(pcpu->head) == READ_ONCE(pcpu->tail))
+			return;
+
+		/*
+		 * A sample was pushed after we exited the drain loop; the
+		 * producer may have skipped re-queuing because pending was
+		 * still set. Keep draining until the ring is observably empty
+		 * with pending cleared.
+		 */
+	}
+}
+
+/*
+ * Empty a dying CPU's sample ring.
+ *
+ * This is called from hotplug teardown callback and IBS Memory Profiler
+ * has been disabled at this point. There is no concurrent producer or
+ * consumer for this CPU's ring here, hence the plain accesses.
+ */
+static inline void mprof_drain_cpu(unsigned int cpu)
+{
+	struct mprof_sample_pcpu *pcpu = per_cpu_ptr(mprof_s, cpu);
+
+	pcpu->head = 0;
+	pcpu->tail = 0;
+}
+
+/*
+ * L3MissOnly + Exclude kernel RIP
+ */
+static void mprof_enable_profiling(void)
+{
+	u64 mprof_config = IBS_MPROF_CTL_CNT_CTL | IBS_MPROF_CTL_L3MISSONLY;
+	unsigned int period = IBS_MPROF_SAMPLE_PERIOD;
+	u64 ctl, ctl2;
+
+	/*
+	 * Assemble bits 26:20 and 19:4 of periodic op counter in ctl.
+	 * The lower 4 bits are always 0000b.
+	 */
+	ctl = (period >> 4) & IBS_MPROF_CTL_MAXCNT_MASK;
+	ctl |= (period & IBS_MPROF_CTL_MAXCNT_EXT_MASK);
+	ctl |= mprof_config;
+	wrmsrq(MSR_AMD64_IBS_MPROF_CTL, ctl);
+
+	/*
+	 * Exclude samples that have bit 63 of their RIP set.
+	 */
+	ctl2 = IBS_MPROF_CTL2_EXCLUDE_KERNEL;
+	wrmsrq(MSR_AMD64_IBS_MPROF_CTL2, ctl2);
+}
+
+static void mprof_disable_profiling(u64 mem_ctl)
+{
+	mem_ctl &= ~IBS_MPROF_CTL_ENABLE;
+	mem_ctl &= ~IBS_MPROF_CTL_VAL;
+	wrmsrq(MSR_AMD64_IBS_MPROF_CTL, mem_ctl);
+
+	wrmsrq(MSR_AMD64_IBS_MPROF_CTL2, IBS_MPROF_CTL2_DISABLE);
+}
+
+/*
+ * IBS interrupt handler: Process the memory access info reported by IBS.
+ *
+ * Reads the MSRs to collect all the information about the reported
+ * memory access, validates the access, stores the valid sample and
+ * schedules the work on this CPU to further process the sample.
+ */
+static void mprof_overflow_handler(void)
+{
+	u64 mem_ctl, mem_data3, mem_data2, paddr, data_src;
+	struct work_struct *w = &this_cpu_ptr(&mprof_work)->work;
+	unsigned long pfn;
+	struct page *page;
+
+	rdmsrq(MSR_AMD64_IBS_MPROF_CTL, mem_ctl);
+	if (!(mem_ctl & IBS_MPROF_CTL_VAL))
+		return;
+
+	mprof_disable_profiling(mem_ctl);
+	count_vm_event(HWHINT_TOTAL_EVENTS);
+
+	rdmsrq(MSR_AMD64_IBS_MPROF_DATA3, mem_data3);
+	rdmsrq(MSR_AMD64_IBS_MPROF_DATA2, mem_data2);
+
+	data_src = mem_data2 & IBS_MPROF_DATA2_DATASRC_MASK;
+	data_src |= ((mem_data2 & IBS_MPROF_DATA2_DATASRC_MASK_HIGH) >>
+			IBS_MPROF_DATA2_DATASRC_MASK_HIGH_SHIFT);
+
+	switch (data_src) {
+	case IBS_MPROF_DATA2_DATASRC_DRAM:
+		count_vm_event(HWHINT_DRAM_ACCESSES);
+		break;
+	case IBS_MPROF_DATA2_DATASRC_EXT_MEM:
+		count_vm_event(HWHINT_EXTMEM_ACCESSES);
+		break;
+	}
+
+	/* Is linear addr valid? */
+	if (!(mem_data3 & IBS_MPROF_DATA3_LADDR_VALID))
+		goto handled;
+
+	/* Is phys addr valid? */
+	if (!(mem_data3 & IBS_MPROF_DATA3_PADDR_VALID))
+		goto handled;
+	rdmsrq(MSR_AMD64_IBS_MPROF_PHYADDR, paddr);
+
+	pfn = PHYS_PFN(paddr);
+	page = pfn_to_online_page(pfn);
+	if (!page)
+		goto handled;
+
+	/*
+	 * Use the accessing CPU's node as the migration target. On
+	 * topologies where all CPUs reside on toptier nodes (the common
+	 * case), this is the desired behaviour. Topologies that place
+	 * CPUs on lower-tier nodes are rejected later by
+	 * pghot_record_access() via the src_nid == nid early return.
+	 */
+	if (!mprof_push_sample(pfn, numa_node_id(), jiffies))
+		goto handled;
+
+	if (!this_cpu_read(mprof_work_pending)) {
+		this_cpu_write(mprof_work_pending, true);
+		schedule_work_on(smp_processor_id(), w);
+	}
+	count_vm_event(HWHINT_USEFUL_EVENTS);
+
+handled:
+	mprof_enable_profiling();
+}
+
+DEFINE_IDTENTRY_SYSVEC(sysvec_ibs_memprof)
+{
+	inc_irq_stat(IBS_MEMPROF);
+	mprof_overflow_handler();
+	apic_eoi();
+}
+
+static int get_mprof_lvt_offset(void)
+{
+	u64 val;
+
+	rdmsrq(MSR_AMD64_IBSCTL, val);
+	if (!(val & IBSCTL_MPROF_LVT_OFFSET_VALID))
+		return -EINVAL;
+
+	return (val & IBSCTL_MPROF_LVT_OFFSET_MASK) >>
+		IBSCTL_MPROF_LVT_OFFSET_SHIFT;
+}
+
+static int x86_amd_ibs_mprof_startup(unsigned int cpu)
+{
+	int offset = get_mprof_lvt_offset();
+
+	if (offset < 0) {
+		pr_warn("offset not valid on cpu #%d\n", cpu);
+		return 0;
+	}
+
+	if (setup_APIC_eilvt(offset, IBS_MEMPROF_VECTOR, APIC_DELIVERY_MODE_FIXED, 0)) {
+		pr_warn("APIC setup failed on cpu #%d\n", cpu);
+		return 0;
+	}
+
+	mprof_enable_profiling();
+	return 0;
+}
+
+static int x86_amd_ibs_mprof_teardown(unsigned int cpu)
+{
+	int offset = get_mprof_lvt_offset();
+	u64 mem_ctl;
+
+	if (offset >= 0)
+		setup_APIC_eilvt(offset, IBS_MEMPROF_VECTOR, APIC_DELIVERY_MODE_FIXED, 1);
+
+	rdmsrq(MSR_AMD64_IBS_MPROF_CTL, mem_ctl);
+	mprof_disable_profiling(mem_ctl);
+
+	/*
+	 * The producer is now silenced and this CPU's worker is gone. Drop
+	 * any unconsumed samples (see mprof_drain_cpu) and clear the pending
+	 * flag so a subsequent re-online of this CPU starts from a clean
+	 * state.
+	 */
+	mprof_drain_cpu(cpu);
+	per_cpu(mprof_work_pending, cpu) = false;
+
+	return 0;
+}
+
+static int __init mprof_access_profiling_init(void)
+{
+	u32 mprof_caps = cpuid_eax(IBS_CPUID_FEATURES);
+	int cpu, ret;
+
+	if (!(mprof_caps & IBS_CAPS_MEM_PROFILER)) {
+		pr_info("capability is unavailable for access profiling\n");
+		return 0;
+	}
+
+	mprof_s = alloc_percpu_gfp(struct mprof_sample_pcpu, GFP_KERNEL | __GFP_ZERO);
+	if (!mprof_s) {
+		pr_err("alloc_percpu_gfp failed\n");
+		return 0;
+	}
+
+	for_each_possible_cpu(cpu) {
+		struct mprof_worker *mw = per_cpu_ptr(&mprof_work, cpu);
+
+		INIT_WORK(&mw->work, mprof_work_handler);
+		mw->cpu = cpu;
+	}
+
+	ret = cpuhp_setup_state(CPUHP_AP_MM_AMD_IBS_MEMPROF_STARTING,
+				"x86/amd/ibs_mprof:starting",
+				x86_amd_ibs_mprof_startup,
+				x86_amd_ibs_mprof_teardown);
+
+	if (ret) {
+		free_percpu(mprof_s);
+		pr_err("cpuhp_setup_state failed: %d\n", ret);
+	} else {
+		pr_info("IBS Memory Profiler is available for memory access profiling\n");
+	}
+	return 0;
+}
+
+device_initcall(mprof_access_profiling_init);
diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 0fb3a2a62eb0..2cd232b3bfcc 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -150,6 +150,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING,
 	CPUHP_AP_PERF_X86_STARTING,
 	CPUHP_AP_PERF_X86_AMD_IBS_STARTING,
+	CPUHP_AP_MM_AMD_IBS_MEMPROF_STARTING,
 	CPUHP_AP_PERF_XTENSA_STARTING,
 	CPUHP_AP_ARM_VFP_STARTING,
 	CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING,
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 58d510711bd4..39409aa95482 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -179,6 +179,13 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		PGHOT_RECORDED_ACCESSES,
 		PGHOT_RECORDED_HINTFAULTS,
 		PGHOT_RECORDED_HWHINTS,
+#ifdef CONFIG_HWMEM_PROFILER
+		HWHINT_TOTAL_EVENTS,
+		HWHINT_DRAM_ACCESSES,
+		HWHINT_EXTMEM_ACCESSES,
+		HWHINT_USEFUL_EVENTS,
+		HWHINT_DROPPED_EVENTS,
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
diff --git a/mm/Kconfig b/mm/Kconfig
index 61ab18f4aebb..6a70e81b371e 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1534,6 +1534,15 @@ config PGHOT_PRECISE
 	  4 bytes per page against the default one byte per page. Preferable
 	  to enable this on systems with multiple nodes in toptier.
 
+config HWMEM_PROFILER
+	bool
+	depends on PGHOT
+	help
+	  Umbrella symbol enabled by any in-kernel driver that forwards
+	  hardware-observed memory accesses to the pghot subsystem (for
+	  example AMD_IBS_MEMPROF on x86_64). Drivers select this; users
+	  do not enable it directly.
+
 source "mm/damon/Kconfig"
 
 endmenu
diff --git a/mm/vmstat.c b/mm/vmstat.c
index da668ff05032..0d0ea5e664e3 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1493,6 +1493,13 @@ const char * const vmstat_text[] = {
 	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
 	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
 	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#ifdef CONFIG_HWMEM_PROFILER
+	[I(HWHINT_TOTAL_EVENTS)]		= "hwhint_total_events",
+	[I(HWHINT_DRAM_ACCESSES)]		= "hwhint_dram_accesses",
+	[I(HWHINT_EXTMEM_ACCESSES)]		= "hwhint_extmem_accesses",
+	[I(HWHINT_USEFUL_EVENTS)]		= "hwhint_useful_events",
+	[I(HWHINT_DROPPED_EVENTS)]		= "hwhint_dropped_events",
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
-- 
2.34.1

[PATCH v8 8/8] x86/mm/ibs: Add runtime controls for IBS memprofiler

[Bharata B Rao]

Expose runtime tunables for IBS Memory profiler and arm the profiler.

<debugfs>/ibs-mprof/l3miss-only  - IbsMemL3MissOnly filter
<debugfs>/ibs-mprof/period       - op sample period (IbsMemMaxCnt)
<debugfs>/ibs-mprof/lat-filter   - IbsMemLatFltEn load-latency filter
<debugfs>/ibs-mprof/lat-thresh   - IbsMemLatThrsh (0x0..0xf)

In addition, a kernel cmdline parameter (ibs-memprof) is introduced
to enable profiling right from boot time.

/sys/devices/system/cpu/ibs-mprof/enabled can be used to turn the
profiling on or off.

Enabling the profiler will result in IBS Memory Profiler samples to
be reported to pghot subsystem. pghot will act on these samples only
if hwhints source is enabled explicilty using pghot/enabled_sources
debugfs tunable.

Signed-off-by: Bharata B Rao <bharata@amd.com>
---
 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/admin-guide/mm/pghot.rst        |  44 +++
 arch/x86/include/asm/ibs-mprof.h              |  14 +
 arch/x86/mm/ibs-mprof.c                       | 371 +++++++++++++++++-
 4 files changed, 419 insertions(+), 15 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index b5493a7f8f22..c25bf35767eb 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2216,6 +2216,11 @@ Kernel parameters
 			syscalls, essentially overriding IA32_EMULATION_DEFAULT_DISABLED at
 			boot time. When false, unconditionally disables IA32 emulation.
 
+	ibs-memprof=	[X86-64]
+			Format: <bool>
+			When true, IBS Memory Profiler will be enabled at boottime
+			for memory access profiling. It can be changed at runtime too
+			using /sys/devices/system/cpu/ibs-mprof/enabled.
 
 	idle=		[X86,EARLY]
 			Format: idle=poll, idle=halt, idle=nomwait
diff --git a/Documentation/admin-guide/mm/pghot.rst b/Documentation/admin-guide/mm/pghot.rst
index b802b80e5c24..69210e9f8e93 100644
--- a/Documentation/admin-guide/mm/pghot.rst
+++ b/Documentation/admin-guide/mm/pghot.rst
@@ -150,3 +150,47 @@ Path: /proc/vmstat
 5. **hwhint_extmem_accesses**
    - Number of external memory (like CXL) accesses reported by hwhints source.
 
+AMD IBS Memory Profiler Tunables
+================================
+sysfs tunables
+--------------
+1. enabled
+
+Path: /sys/devices/system/cpu/ibs-mprof/enabled
+
+- Enable (1) or disable (0) memory access profiling. Enabling here
+  would only arm the memory profiler and results in generation of IBS
+  samples. pghot will act on the reported samples only if hwhints source
+  is enabled in **pghot_enabled_sources**.
+- Default: 0 (Disabled)
+
+Debugfs tunables
+----------------
+Path: /sys/kernel/debug/ibs-mprof/
+
+These tune the AMD IBS Memory Profiler hardware hints source and are
+present only when AMD_IBS_MEMPROF is enabled. A write takes effect on
+all CPUs immediately.
+
+1. **l3miss-only**
+   - When 1, only accesses that miss the L3 cache are sampled. It is
+     recommended to run the memory profiler with L3 miss filtering
+     enabled.
+   - Default: 1
+
+2. **period**
+   - Sample period as the number of ops between samples (IbsMemMaxCnt).
+   - Range: 5000 to 134217727.
+   - Default: 10000
+
+3. **lat-filter**
+   - When 1, enable load latency filtering: only loads whose latency
+     exceeds the **lat-thresh** threshold are reported.
+   - Default: 0
+
+4. **lat-thresh**
+   - Load latency threshold, effective only when **lat-filter** is 1. A
+     sample is reported when the load latency exceeds
+     (lat-thresh + 1) * 128 core cycles.
+   - Range: 0x0 to 0xf
+   - Default: 0
diff --git a/arch/x86/include/asm/ibs-mprof.h b/arch/x86/include/asm/ibs-mprof.h
index 3de1880c4126..8b91c0ff1c33 100644
--- a/arch/x86/include/asm/ibs-mprof.h
+++ b/arch/x86/include/asm/ibs-mprof.h
@@ -30,6 +30,9 @@
 #define IBS_MPROF_DATA3_SW_PREFETCH	BIT_ULL(21)
 
 /* MSR_AMD64_IBS_MPROF_CTL bits */
+#define IBS_MPROF_CTL_LATFLTEN		BIT_ULL(63)	/* IbsMemLatFltEn */
+#define IBS_MPROF_CTL_LATTHRSH_SHIFT	59
+#define IBS_MPROF_CTL_LATTHRSH_MASK	(0xFULL << IBS_MPROF_CTL_LATTHRSH_SHIFT)
 #define IBS_MPROF_CTL_CNT_CTL		BIT_ULL(19)
 #define IBS_MPROF_CTL_VAL		BIT_ULL(18)
 #define IBS_MPROF_CTL_ENABLE		BIT_ULL(17)
@@ -37,6 +40,17 @@
 #define IBS_MPROF_CTL_MAXCNT_MASK	0x0000FFFFULL
 #define IBS_MPROF_CTL_MAXCNT_EXT_MASK	(0x7FULL << 20)	/* separate upper 7 bits */
 
+/*
+ * IbsMemMaxCnt is a 27-bit op count; the low 4 bits are always zero.
+ * The hardware minimum is 16, but such a short interval is impractical
+ * (interrupt storm), so restrict the sample period to a sensible floor.
+ */
+#define IBS_MPROF_MAXCNT_MIN		5000
+#define IBS_MPROF_MAXCNT_MAX		((1U << 27) - 1)
+
+/* IbsMemLatThrsh is a 4-bit field. */
+#define IBS_MPROF_LATTHRSH_MAX		0xF
+
 /* MSR_AMD64_IBS_MPROF_CTL2 bits */
 #define IBS_MPROF_CTL2_DISABLE		BIT_ULL(0)
 #define IBS_MPROF_CTL2_EXCLUDE_USER	BIT_ULL(1)
diff --git a/arch/x86/mm/ibs-mprof.c b/arch/x86/mm/ibs-mprof.c
index 923fb8f99552..07e0516db2ee 100644
--- a/arch/x86/mm/ibs-mprof.c
+++ b/arch/x86/mm/ibs-mprof.c
@@ -10,6 +10,11 @@
 #include <linux/vm_event_item.h>
 #include <linux/vmstat.h>
 #include <linux/cpuhotplug.h>
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+#include <linux/mutex.h>
+#include <linux/smp.h>
+#include <linux/uaccess.h>
 
 #include <asm/ibs-mprof.h>
 #include <asm/ibs-caps.h>
@@ -54,6 +59,41 @@ struct mprof_worker {
 };
 static DEFINE_PER_CPU(struct mprof_worker, mprof_work);
 
+/*
+ * Runtime-configurable profiler parameters, exposed via debugfs. The
+ * interrupt handler consumes the published snapshot on every re-arm, so
+ * configuration changes need no locking on the fast path. Writers are
+ * serialized by mprof_cfg_lock and publish an immutable snapshot; the hot
+ * path only does an smp_load_acquire() of the pointer.
+ */
+struct mprof_config {
+	bool enabled;		/* profiling enabled (IbsMemDis inverted) */
+	bool l3miss_only;	/* IbsMemL3MissOnly */
+	bool lat_filter;	/* IbsMemLatFltEn */
+	u8   lat_thresh;	/* IbsMemLatThrsh, 0x0 .. 0xf */
+	u32  period;		/* op sample period (IbsMemMaxCnt) */
+	/* Precomputed register values for the interrupt fast path. */
+	u64  ctl;
+	u64  ctl2;
+};
+
+static struct mprof_config mprof_cfg_slots[2];
+static struct mprof_config *mprof_cfg;
+static DEFINE_MUTEX(mprof_cfg_lock);
+
+/*
+ * ibs-memprof: kernel cmdline parameter to arm IBS Memory
+ * Profiler right from boot time.
+ */
+bool ibs_mprof_enabled __read_mostly;
+
+static int __init setup_ibs_mprof(char *str)
+{
+	return (kstrtobool(str, &ibs_mprof_enabled) == 0);
+}
+
+__setup("ibs-memprof=", setup_ibs_mprof);
+
 /*
  * Record the IBS-reported access sample in percpu buffer.
  * Called from IBS interrupt handler.
@@ -159,28 +199,55 @@ static inline void mprof_drain_cpu(unsigned int cpu)
 }
 
 /*
- * L3MissOnly + Exclude kernel RIP
+ * Translate the software config into IbsMemCtl / IbsMemCtl2 register values.
+ * Kernel RIP is always excluded: the profiler only samples user memory
+ * accesses.
  */
-static void mprof_enable_profiling(void)
+static void mprof_compose(struct mprof_config *cfg)
 {
-	u64 mprof_config = IBS_MPROF_CTL_CNT_CTL | IBS_MPROF_CTL_L3MISSONLY;
-	unsigned int period = IBS_MPROF_SAMPLE_PERIOD;
-	u64 ctl, ctl2;
+	u64 ctl;
+
+	if (!cfg->enabled) {
+		ctl = 0;
+		cfg->ctl = ctl;
+		cfg->ctl2 = IBS_MPROF_CTL2_DISABLE;
+		return;
+	}
 
 	/*
-	 * Assemble bits 26:20 and 19:4 of periodic op counter in ctl.
+	 * Assemble bits 26:20 and 19:4 of the periodic op counter in ctl.
 	 * The lower 4 bits are always 0000b.
 	 */
-	ctl = (period >> 4) & IBS_MPROF_CTL_MAXCNT_MASK;
-	ctl |= (period & IBS_MPROF_CTL_MAXCNT_EXT_MASK);
-	ctl |= mprof_config;
-	wrmsrq(MSR_AMD64_IBS_MPROF_CTL, ctl);
+	ctl = (cfg->period >> 4) & IBS_MPROF_CTL_MAXCNT_MASK;
+	ctl |= cfg->period & IBS_MPROF_CTL_MAXCNT_EXT_MASK;
+	ctl |= IBS_MPROF_CTL_CNT_CTL | IBS_MPROF_CTL_ENABLE;
 
-	/*
-	 * Exclude samples that have bit 63 of their RIP set.
-	 */
-	ctl2 = IBS_MPROF_CTL2_EXCLUDE_KERNEL;
-	wrmsrq(MSR_AMD64_IBS_MPROF_CTL2, ctl2);
+	if (cfg->l3miss_only)
+		ctl |= IBS_MPROF_CTL_L3MISSONLY;
+
+	if (cfg->lat_filter) {
+		ctl |= IBS_MPROF_CTL_LATFLTEN;
+		ctl |= ((u64)cfg->lat_thresh << IBS_MPROF_CTL_LATTHRSH_SHIFT) &
+			IBS_MPROF_CTL_LATTHRSH_MASK;
+	}
+
+	cfg->ctl = ctl;
+	/* Exclude samples that have bit 63 of their RIP set (kernel). */
+	cfg->ctl2 = IBS_MPROF_CTL2_EXCLUDE_KERNEL;
+}
+
+/*
+ * Program the profiler from the currently published config snapshot. Called
+ * at CPU startup, from the interrupt handler on every re-arm, and via
+ * on_each_cpu() when the config changes.
+ */
+static void mprof_enable_profiling(void)
+{
+	/* Acquire the snapshot; pairs with smp_store_release() in the writers. */
+	struct mprof_config *cfg = smp_load_acquire(&mprof_cfg);
+
+	wrmsrq(MSR_AMD64_IBS_MPROF_CTL, cfg->ctl);
+	wrmsrq(MSR_AMD64_IBS_MPROF_CTL2, cfg->ctl2);
 }
 
 static void mprof_disable_profiling(u64 mem_ctl)
@@ -192,6 +259,52 @@ static void mprof_disable_profiling(u64 mem_ctl)
 	wrmsrq(MSR_AMD64_IBS_MPROF_CTL2, IBS_MPROF_CTL2_DISABLE);
 }
 
+static void mprof_reprogram_this_cpu(void *info)
+{
+	mprof_enable_profiling();
+}
+
+/*
+ * Publish a new config snapshot and push it to every online CPU
+ * immediately. Must be called with mprof_cfg_lock held.
+ */
+static void mprof_publish(const struct mprof_config *newcfg)
+{
+	struct mprof_config *slot;
+
+	lockdep_assert_held(&mprof_cfg_lock);
+
+	/* Fill the slot that is not currently published, then flip to it. */
+	slot = (mprof_cfg == &mprof_cfg_slots[0]) ?
+		&mprof_cfg_slots[1] : &mprof_cfg_slots[0];
+	*slot = *newcfg;
+	mprof_compose(slot);
+	/* Publish the fully composed slot; pairs with smp_load_acquire() in readers. */
+	smp_store_release(&mprof_cfg, slot);
+
+	/*
+	 * on_each_cpu() with wait serializes against any in-flight interrupt
+	 * handler on each CPU, so the previously published slot has no readers
+	 * once this returns and can be safely reused by the next writer.
+	 */
+	on_each_cpu(mprof_reprogram_this_cpu, NULL, 1);
+}
+
+static void mprof_config_init(void)
+{
+	struct mprof_config *cfg = &mprof_cfg_slots[0];
+
+	if (ibs_mprof_enabled)
+		cfg->enabled = true;
+	cfg->l3miss_only = true;
+	cfg->lat_filter = false;
+	cfg->lat_thresh = 0;
+	cfg->period = IBS_MPROF_SAMPLE_PERIOD;
+	mprof_compose(cfg);
+	/* Publish the initial snapshot; pairs with smp_load_acquire() in readers. */
+	smp_store_release(&mprof_cfg, cfg);
+}
+
 /*
  * IBS interrupt handler: Process the memory access info reported by IBS.
  *
@@ -216,6 +329,13 @@ static void mprof_overflow_handler(void)
 	rdmsrq(MSR_AMD64_IBS_MPROF_DATA3, mem_data3);
 	rdmsrq(MSR_AMD64_IBS_MPROF_DATA2, mem_data2);
 
+	/*
+	 * If L3 miss filtering is turned off, non load/store
+	 * samples may get reported. Ignore them.
+	 */
+	if (!(mem_data3 & (IBS_MPROF_DATA3_LDOP | IBS_MPROF_DATA3_STOP)))
+		goto handled;
+
 	data_src = mem_data2 & IBS_MPROF_DATA2_DATASRC_MASK;
 	data_src |= ((mem_data2 & IBS_MPROF_DATA2_DATASRC_MASK_HIGH) >>
 			IBS_MPROF_DATA2_DATASRC_MASK_HIGH_SHIFT);
@@ -323,6 +443,223 @@ static int x86_amd_ibs_mprof_teardown(unsigned int cpu)
 	return 0;
 }
 
+/*
+ * debugfs interface. Each parameter is a separate file under
+ * <debugfs>/ibs-mprof/. A write validates the value, updates the current
+ * config and actively propagates it to all CPUs via mprof_publish().
+ */
+enum mprof_field {
+	MPROF_L3MISS_ONLY,
+	MPROF_LAT_FILTER,
+	MPROF_LAT_THRESH,
+	MPROF_PERIOD,
+};
+
+static int mprof_parse_uint(const char __user *ubuf, size_t cnt, unsigned int *val)
+{
+	char buf[16];
+
+	if (cnt > sizeof(buf) - 1)
+		cnt = sizeof(buf) - 1;
+	if (copy_from_user(buf, ubuf, cnt))
+		return -EFAULT;
+	buf[cnt] = '\0';
+	if (kstrtouint(buf, 0, val))
+		return -EINVAL;
+	return 0;
+}
+
+static void mprof_store(enum mprof_field field, unsigned int val)
+{
+	struct mprof_config new;
+
+	mutex_lock(&mprof_cfg_lock);
+	new = *mprof_cfg;
+	switch (field) {
+	case MPROF_L3MISS_ONLY:
+		new.l3miss_only = val;
+		break;
+	case MPROF_LAT_FILTER:
+		new.lat_filter = val;
+		break;
+	case MPROF_LAT_THRESH:
+		new.lat_thresh = val;
+		break;
+	case MPROF_PERIOD:
+		new.period = val;
+		break;
+	}
+	mprof_publish(&new);
+	mutex_unlock(&mprof_cfg_lock);
+}
+
+#define MPROF_SHOW(name, field)						\
+static int mprof_##name##_show(struct seq_file *m, void *v)		\
+{									\
+	/* Acquire the snapshot; pairs with smp_store_release() in writers. */	\
+	struct mprof_config *cfg = smp_load_acquire(&mprof_cfg);	\
+									\
+	seq_printf(m, "%u\n", cfg->field);				\
+	return 0;							\
+}									\
+static int mprof_##name##_open(struct inode *inode, struct file *filp)	\
+{									\
+	return single_open(filp, mprof_##name##_show, NULL);		\
+}
+
+MPROF_SHOW(l3miss_only, l3miss_only)
+MPROF_SHOW(lat_filter, lat_filter)
+MPROF_SHOW(lat_thresh, lat_thresh)
+MPROF_SHOW(period, period)
+
+static ssize_t mprof_l3miss_only_write(struct file *filp, const char __user *ubuf,
+				       size_t cnt, loff_t *ppos)
+{
+	unsigned int val;
+	int ret;
+
+	ret = mprof_parse_uint(ubuf, cnt, &val);
+	if (ret)
+		return ret;
+	if (val > 1)
+		return -EINVAL;
+
+	mprof_store(MPROF_L3MISS_ONLY, val);
+	*ppos += cnt;
+	return cnt;
+}
+
+static ssize_t mprof_lat_filter_write(struct file *filp, const char __user *ubuf,
+				      size_t cnt, loff_t *ppos)
+{
+	unsigned int val;
+	int ret;
+
+	ret = mprof_parse_uint(ubuf, cnt, &val);
+	if (ret)
+		return ret;
+	if (val > 1)
+		return -EINVAL;
+
+	mprof_store(MPROF_LAT_FILTER, val);
+	*ppos += cnt;
+	return cnt;
+}
+
+static ssize_t mprof_lat_thresh_write(struct file *filp, const char __user *ubuf,
+				      size_t cnt, loff_t *ppos)
+{
+	unsigned int val;
+	int ret;
+
+	ret = mprof_parse_uint(ubuf, cnt, &val);
+	if (ret)
+		return ret;
+	if (val > IBS_MPROF_LATTHRSH_MAX)
+		return -EINVAL;
+
+	mprof_store(MPROF_LAT_THRESH, val);
+	*ppos += cnt;
+	return cnt;
+}
+
+static ssize_t mprof_period_write(struct file *filp, const char __user *ubuf,
+				  size_t cnt, loff_t *ppos)
+{
+	unsigned int val;
+	int ret;
+
+	ret = mprof_parse_uint(ubuf, cnt, &val);
+	if (ret)
+		return ret;
+	if (val < IBS_MPROF_MAXCNT_MIN || val > IBS_MPROF_MAXCNT_MAX)
+		return -EINVAL;
+
+	mprof_store(MPROF_PERIOD, val);
+	*ppos += cnt;
+	return cnt;
+}
+
+#define MPROF_FOPS(name)						\
+static const struct file_operations mprof_##name##_fops = {		\
+	.open		= mprof_##name##_open,				\
+	.read		= seq_read,					\
+	.write		= mprof_##name##_write,				\
+	.llseek		= seq_lseek,					\
+	.release	= single_release,				\
+}
+
+MPROF_FOPS(l3miss_only);
+MPROF_FOPS(lat_filter);
+MPROF_FOPS(lat_thresh);
+MPROF_FOPS(period);
+
+static void mprof_debugfs_init(void)
+{
+	struct dentry *dir = debugfs_create_dir("ibs-mprof", NULL);
+
+	debugfs_create_file("l3miss-only", 0644, dir, NULL, &mprof_l3miss_only_fops);
+	debugfs_create_file("lat-filter", 0644, dir, NULL, &mprof_lat_filter_fops);
+	debugfs_create_file("lat-thresh", 0644, dir, NULL, &mprof_lat_thresh_fops);
+	debugfs_create_file("period", 0644, dir, NULL, &mprof_period_fops);
+}
+
+static ssize_t enabled_show(struct device *dev,
+			    struct device_attribute *attr, char *buf)
+{
+	/* Acquire the snapshot; pairs with smp_store_release() in the writers. */
+	struct mprof_config *cfg = smp_load_acquire(&mprof_cfg);
+
+	return sysfs_emit(buf, "%s\n", str_enabled_disabled(cfg->enabled));
+}
+
+static ssize_t enabled_store(struct device *dev, struct device_attribute *attr,
+			     const char *buf, size_t count)
+{
+	struct mprof_config new;
+	bool enabled;
+	int ret;
+
+	ret = kstrtobool(buf, &enabled);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&mprof_cfg_lock);
+	new = *mprof_cfg;
+	new.enabled = enabled;
+	mprof_publish(&new);
+
+	return count;
+}
+
+static DEVICE_ATTR_RW(enabled);
+
+static struct attribute *ibs_mprof_attributes[] = {
+	&dev_attr_enabled.attr,
+	NULL
+};
+
+static const struct attribute_group ibs_mprof_attr_group = {
+	.name = "ibs-mprof",
+	.attrs = ibs_mprof_attributes,
+};
+
+static void mprof_tunables_init(void)
+{
+	struct device *dev_root;
+	int ret;
+
+	dev_root = bus_get_dev_root(&cpu_subsys);
+	if (dev_root) {
+		ret = sysfs_create_group(&dev_root->kobj, &ibs_mprof_attr_group);
+		put_device(dev_root);
+		if (ret)
+			return;
+	}
+
+	mprof_debugfs_init();
+}
+
 static int __init mprof_access_profiling_init(void)
 {
 	u32 mprof_caps = cpuid_eax(IBS_CPUID_FEATURES);
@@ -346,6 +683,9 @@ static int __init mprof_access_profiling_init(void)
 		mw->cpu = cpu;
 	}
 
+	/* Publish the default config before the startup callback consumes it. */
+	mprof_config_init();
+
 	ret = cpuhp_setup_state(CPUHP_AP_MM_AMD_IBS_MEMPROF_STARTING,
 				"x86/amd/ibs_mprof:starting",
 				x86_amd_ibs_mprof_startup,
@@ -356,6 +696,7 @@ static int __init mprof_access_profiling_init(void)
 		pr_err("cpuhp_setup_state failed: %d\n", ret);
 	} else {
 		pr_info("IBS Memory Profiler is available for memory access profiling\n");
+		mprof_tunables_init();
 	}
 	return 0;
 }
-- 
2.34.1

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - microbenchmark numbers

[Bharata B Rao]

Microbenchmark
==============
Multi-threaded application with 64 threads that access memory(8G) at
4K granularity repetitively and randomly. The number of accesses per
thread and the randomness pattern for each thread are fixed beforehand.
The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on
CXL node 2 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated
detecting and migrating them. The benchmark score is the time taken to
finish the accesses in microseconds. The sooner it finishes the better it is.
All the numbers shown below are average of 3 runs.

Table 1: Benchmark completion time (lower is better), avg of 3 runs
64 threads, random 4K access over 8G, mem on CXL node 2, cpu node 0

Config    Time (us)   Time (s)  Speedup
---------------------------------------
C1       97,165,273       97.2    1.00x
C2       40,378,842       40.4    2.41x
C3       95,431,823       95.4    1.02x
C4       40,455,945       40.5    2.40x
C5       41,289,646       41.3    2.35x
C6       64,260,194       64.3    1.51x

Speedup = C1 (base-NUMAB0) time / config time (>1.00x is faster)

Legend:
  C1 = base kernel, NUMAB0 (hot page promotion disabled)
  C2 = base kernel, NUMAB2 (hot page promotion enabled)
  C3 = pghot kernel, NUMAB0 (hint faults source disabled)
  C4 = pghot kernel, NUMAB2 (hint faults source enabled)
  C5 = pghot kernel, IBS Memory profiler, pghot_freq_threshold=1 (NUMAB0)
  C6 = pghot kernel, IBS Memory profiler, pghot_freq_threshold=2 (NUMAB0)


Table 2: Page migration / hotness vmstat counters (avg of 3 runs)
'-' = counter not present for that kernel/config

ctr           C1         C2         C3         C4         C5         C6
-----------------------------------------------------------------------
M1             0  2,097,152          0  2,097,152  1,966,188  1,444,783
M2             0  2,283,787          0  2,097,152  1,966,188  1,444,783
M3             0  2,097,152          0  2,097,152  1,966,188  1,444,783
M4             0  2,097,152          0  2,097,152          0          0
M5             0  2,283,787          0  2,097,152          0          0
M6             -          -          0  2,097,152  1,968,079  3,508,998
M7             -          -          0  2,097,152          0          0
M8             -          -          0          0  5,654,097  5,422,763
M9             -          -          0          0  5,654,097  5,422,763
M10            -          -          0          0  3,439,001  1,701,889
M11            -          -          0          0  1,879,453  3,105,182
M12            -          -          0          0  5,654,097  5,422,763
M13            -          -          0          0          0          0

Counter legend:
  M1 = pgpromote_success
  M2 = pgpromote_candidate_nrl
  M3 = numa_pages_migrated
  M4 = numa_pte_updates
  M5 = numa_hint_faults
  M6 = pghot_recorded_accesses
  M7 = pghot_recorded_hintfaults
  M8 = pghot_recorded_hwhints
  M9 = hwhint_total_events
  M10 = hwhint_dram_accesses
  M11 = hwhint_extmem_accesses
  M12 = hwhint_useful_events
  M13 = hwhint_dropped_events

Key observations
================
- The base NUMAB0 vs NUMAB2 numbers clearly show the benefit of promotion.
- pghot-hintfault matches the base tiering case both in speedup and number
  of promottions.
- pghot-hwhint also matches the base tiering case both in speedup and
  number of promotions.

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - NAS BT

[Bharata B Rao]

NAS Parallel Benchmark BT (Block Tri-diagonal solver), MPI version,
Class F modified, run with 16 ranks. Compute threads are bound to node 1 via:

  /usr/bin/mpirun -np 16 /usr/bin/numactl --cpunodebind=1 \
      NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

While class D uses around 24G of memory (which is too less to show the
benefit of promition), class E results in around 368G of memory which
overflows my toptier. Hence I wanted something in between these classes.
So I have  modified class F to the problem size of 768 which results in
around 160GB of memory.

System is a 3-node tiered setup: node 0 and node 1 are DRAM (CPUs),
node 2 is a CPU-less CXL lower tier (~252 GiB per node). Before the
measurement phase all provisioned memory (~161.4 GiB across ranks)
is migrated to the lower-tier CXL node 2. The ranks are then resumed
and measurement begins, so the run starts fully cold on slow memory.
This isolates the effect of hot-page promotion (node 2 -> node 1)
on a bandwidth/latency-sensitive workload.

Legend (kernel / promotion source)
----------------------------------------------------------------------
  A = base-numab0       Base kernel, NO tiering (NUMAB=0); reference
  B = base-numab2       Base kernel, tiering via NUMA Balancing
                        mode 2 (hint-fault driven promotion)
  C = pghot-hintfaults  pghot kernel, hint-fault source (NUMAB=2)
  D = pghot-hwhints     pghot kernel, HW-hint source (IBS memory
                        profiler); does NOT need NUMA Balancing

Table 1a - Runtime (lower is better)
----------------------------------------------------------------------
Metric                          A          B          C          D
----------------------------------------------------------------------
NPB Time (s)              7482.25    3306.81    3313.82    3506.55
Speedup vs A (time)          1.00       2.26       2.26       2.13

Table 1b - Throughput (higher is better)
----------------------------------------------------------------------
Metric                          A          B          C          D
----------------------------------------------------------------------
NPB Mop/s total          52305.50  118350.48  118100.13  111608.96
Mop/s ratio vs A             1.00       2.26       2.26       2.13
Mop/s ratio vs B             0.44       1.00       1.00       0.94
Peak prov. mem (GiB)       161.38     161.38     161.38     161.38

Table 2 - vmstat counter deltas (after - before measurement window)
----------------------------------------------------------------------
Counter                             A         B         C         D
----------------------------------------------------------------------
pgpromote_success                   0  42180536  42169996  39898316
pgpromote_candidate                 0         0         0         0
pgpromote_candidate_nrl             0  42180536  42169996  39899553
pgdemote_kswapd                     0         0         0         0
pgdemote_direct                     0         0         0         0
numa_pte_updates                    0  42250308  45027035         0
numa_hint_faults                    0  42182545  44119239         0
numa_pages_migrated                 0  42182543  42169996  39898316
pgmigrate_success                   0  42182543  42169996  39898316
pghot_recorded_accesses             0         0  44119239  40004334
pghot_recorded_hintfaults           0         0  44119239         0
pghot_recorded_hwhints              0         0         0 584209652
hwhint_total_events                 0         0         0 584209975
hwhint_dram_accesses                0         0         0 544195074
hwhint_extmem_accesses              0         0         0  40005074
hwhint_useful_events                0         0         0 584209652

Key observations
----------------------------------------------------------------------
Bottom line: hint-fault pghot reproduces mainline NUMAB=2 exactly,
while HW-hint pghot delivers ~94% of that performance with zero
NUMA-balancing overhead - a good trade-off where IBS is available.

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - Graph500

[Bharata B Rao]

Benchmark : Graph500 reference BFS, SCALE=28, edgefactor=16, 128 ranks
Topology  : top-tier NUMA node=1 (CPUs+DRAM), lower-tier node=2 (mem-only/CXL)

Config legend
-------------
R1 : base  kernel, no tiering        (numa_balancing=0)
R2 : base  kernel, tiering           (numa_balancing=2 / NUMAB2)
R3 : pghot kernel, hintfaults source (pghot_enabled_sources=1,
     pghot_freq_threshold=1, numa_balancing=2, promotion=on)
R4 : pghot kernel, hwhints  source   (pghot_enabled_sources=2,
     pghot_freq_threshold=1, IBS mem-profiler on, numa_balancing=0,
     promotion=off)

Column legend
-------------
hmean   : harmonic_mean_TEPS   (primary Graph500 metric)
hstddev : harmonic_stddev_TEPS
median  : median_TEPS
bfs_t   : mean BFS time (seconds)
spdup   : speedup of hmean vs R1 baseline

Table 1: Performance
--------------------
+--------+------------+----------+------------+---------+--------+
| Config |   hmean    | hstddev  |   median   |  bfs_t  | spdup  |
|        |   (TEPS)   |  (TEPS)  |   (TEPS)   |  (sec)  |        |
+--------+------------+----------+------------+---------+--------+
| R1     | 5.469e+08  | 4.55e+05 | 5.468e+08  |  7.853  | 1.00x  |
| R2     | 1.442e+09  | 8.04e+07 | 1.491e+09  |  2.979  | 2.64x  |
| R3     | 1.362e+09  | 7.92e+07 | 1.515e+09  |  3.153  | 2.49x  |
| R4     | 1.463e+09  | 1.35e+07 | 1.487e+09  |  2.935  | 2.68x  |
+--------+------------+----------+------------+---------+--------+

Table 2: Relevant kernel counters (/proc/vmstat deltas over the run)
--------------------------------------------------------------------
Values are accumulated deltas (before -> after) for the whole run.

+----------------------------+-----------+-----------+-----------+-----------+
| Counter                    |    R1     |    R2     |    R3     |    R4     |
+----------------------------+-----------+-----------+-----------+-----------+
| numa_pte_updates           |         0 |  26675234 |  28872628 |         0 |
| numa_hint_faults           |         0 |  13747383 |  15501854 |         0 |
| numa_pages_migrated        |         0 |  13747376 |  13726437 |   3683579 |
| pgpromote_success          |         0 |  13747254 |  13726437 |   3683579 |
| pghot_recorded_accesses    |         0 |         0 |  15501778 |   3682112 |
| pghot_recorded_hintfaults  |         0 |         0 |  15501854 |         0 |
| pghot_recorded_hwhints     |         0 |         0 |         0 |  24291151 |
| hwhint_total_events        |         0 |         0 |         0 |  24291198 |
| hwhint_dram_accesses       |         0 |         0 |         0 |  19588681 |
| hwhint_extmem_accesses     |         0 |         0 |         0 |   3663926 |
| pgmigrate_success          |  26840188 |  40586709 |  40570940 |  30521808 |
+----------------------------+-----------+-----------+-----------+-----------+

Key findings
------------
1. Tiering helps Graph500.
2. pghot-hintfaults is on par with base NUMAB2 tiering.
3. pghot-hwhints (R4) matches NUMAB=2 performance with zero NUMA hint
   faults.

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - redis-memtier

[Bharata B Rao]

Redis-memtier

In-memory Redis server is loaded with a ~64 GB dataset
(62.2M keys x 1 KB) whose pages are then explicitly migrated to the CXL
node (lower tier), while the Redis server and the memtier client run on a
top-tier DRAM node.  The measurement phase drives GET traffic with
memtier (16 threads x 100 conns, PASSES=12) over 50% of the keyspace so
that repeatedly-accessed lower-tier pages become promotion candidates.
We compare throughput/latency and the kernel promotion activity across
the base kernel and the pghot kernel, and across hotness sources (NUMA-
balancing hint faults vs. the IBS hardware memory profiler).

System: AMD Zen6, Nodes 0,1 = DRAM (top tier); node 2 = CXL (lower tier).

Cases:
  C1 base/NUMAB0     base kernel, promotion OFF (baseline)
  C2 base/NUMAB2     base kernel, NUMAB tiering promotion ON
  C3 pghot/NUMAB2    pghot kernel, promotion via pghot, source =
                     NUMA hint faults (kmigrated batched migration)
  C4 pghot/IBS-10000 pghot, source = IBS profiler (period 10000), NUMAB=0
  C5 pghot/IBS-5000  same as C4 but IBS period=5000 (2x sampling)

=========================================================================
Table 1: Benchmark metrics (memtier)
=========================================================================
Case               NUMAB   Ops/sec  vs base    Avg    p50    p99   p99.9
                                                          latency (ms) ->
------------------------------------------------------------------------
C1 base/NUMAB0         0   281,424      ref 181.62 180.22 348.16  362.50
C2 base/NUMAB2         2   295,551    +5.0% 173.18 168.96 307.20  354.30
C3 pghot/NUMAB2        2   296,269    +5.3% 172.42 169.98 327.68  364.54
C4 pghot/IBS-10000     0   281,645    +0.1% 181.67 179.20 350.21  362.50
C5 pghot/IBS-5000      0   281,665    +0.1% 181.72 180.22 348.16  362.50
------------------------------------------------------------------------

=========================================================================
Table 2: Page-migration metrics (vmstat delta over measurement)
=========================================================================
Legend: C1=base/NUMAB0  C2=base/NUMAB2  C3=pghot/NUMAB2
        C4=pghot/IBS-10000  C5=pghot/IBS-5000  ('-'=counter absent)

metric                       C1         C2         C3        C4        C5
-------------------------------------------------------------------------
pgpromote_success             0 10,433,576 10,433,484   594,762 1,128,469
pgpromote_candidate_nrl       0 10,433,576 10,433,484   594,762 1,128,469
numa_pte_updates              0 20,333,714 20,344,211         0         0
numa_hint_faults              0 10,433,576 10,433,533         0         0
numa_pages_migrated           0 10,433,576 10,433,484   594,762 1,128,463
pgmigrate_success             0 10,433,576 10,433,484   594,762 1,128,463
pghot_recorded_accesses       -          - 10,433,533   595,426 1,130,753
pghot_recorded_hintfaults     -          - 10,433,533         0         0
pghot_recorded_hwhints        -          -          0 1,000,187 2,026,558
pgdemote_kswapd               0          0          0         0         0
-------------------------------------------------------------------------

=========================================================================
Key observations
=========================================================================
1. Promotion helps this workload: enabling NUMAB tiering promotion lifts
   throughput from 281,424 to 295,551 ops/sec (+5.0%) and cuts average
   latency 181.6 -> 173.2 ms.
2. pghot-hintfault has no regression vs base as it matches the base tiering
   case.
3. pghot-hwhint sources promotes far fewer pages than hint faults and it
   barely shows any improvement over base notier case.
4. Halving the period (10000->5000) ~doubles hw hints (1.00M->2.03M) and
   promotions (0.59M->1.13M) as expected, but is still far short of the
   hot set, so throughput does not yet move.

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - llama-bench

[Bharata B Rao]

llama-bench is the micro-benchmark shipped with llama.cpp. It loads a GGUF
model through the production inference path and reports two throughput numbers
per run: pp512 (one batched 512-token prefill, compute-bound) and tg128 (128
sequential autoregressive decode steps, bandwidth-bound).

Workload: llama-bench (llama.cpp), Mixtral-8x22B-Instruct Q4_K_M (140.6B
params, 79.7 GiB), -t 64 -p 512 -n 128 -r 5 --mmap 0.  pp512 = prefill
(compute-bound); tg128 = decode (memory-bandwidth-bound).

Setup: 3-node box, N0/N1 DRAM (256 GiB each), N2 CXL (distances 10/12/50).
Bench pinned to N1 CPUs with MPOL_PREFERRED_MANY({1}); a 200 GiB hot hog on
N1 forces kswapd to naturally demote ~1/3 of the model to CXL; each mode is
then measured while the hog keeps N1 under pressure.  pghot target_nid=1,
freq_threshold=1, freq_window=3000ms, rate_limit=65536 MBps, kmigrated
100ms/512.  hwhints runs arm AMD IBS (l3miss-only=1, period=10000).

Legend (columns)
----------------
  r1 = base / notier        7.2.0-rc2-base   nb=0 src=-
  r2 = base / tier (NUMAB2) 7.2.0-rc2-base   nb=2 src=-
  r3 = pghot / hintfaults   7.2.0-rc2-pghot  nb=2 src=0x1
  r4 = pghot / hwhints(IBS) 7.2.0-rc2-pghot  nb=0 src=0x2
  r5 = pghot / both         7.2.0-rc2-pghot  nb=2 src=0x3
  (nb = kernel.numa_balancing; src = pghot_enabled_sources)

Table 1 - Throughput (llama-bench, tokens/s over 5 reps)
-------------------------------------------------------
metric                       r1       r2       r3       r4       r5
-------------------------------------------------------------------
pp512 t/s                 69.31    62.46    54.53    53.95    51.49
  stddev                   2.70     6.65    10.41     8.12     4.79
tg128 t/s                 4.262    4.625    5.993    5.161    6.851
  stddev                  0.004    0.313    1.255    0.535    1.807
tg128 vs r1               1.00x    1.09x    1.41x    1.21x    1.61x
pp512 vs r1               1.00x    0.90x    0.79x    0.78x    0.74x

Table 1b - tg128 per-iteration (t/s), shows convergence
-------------------------------------------------------
iter                         r1       r2       r3       r4       r5
-------------------------------------------------------------------
iter1                     4.268    4.183    4.393    4.462    4.415
iter2                     4.261    4.457    5.336    4.820    5.602
iter3                     4.260    4.678    5.995    5.218    7.322
iter4                     4.261    4.841    6.501    5.494    8.152
iter5                     4.259    4.969    7.740    5.813    8.763

Table 2 - Key vmstat counters, run-phase deltas (millions of pages/events)
--------------------------------------------------------------------------
metric                          r1      r2      r3      r4      r5
------------------------------------------------------------------
pgpromote_success             0.00    3.05    6.95    2.17    6.53
pgpromote_candidate           0.00   14.34   24.12   13.83   21.09
pgdemote_kswapd               0.69    5.02   10.84    3.89    8.05
pgmigrate_success             0.69    8.07   17.80    6.06   14.59
numa_pte_updates              0.00   28.77   24.56    0.00   12.04
numa_hint_faults              0.00   27.77   24.12    0.00   11.39
pghot_recorded_accesses       0.00    0.00   24.12   13.83   21.28
pghot_recorded_hintfaults     0.00    0.00   24.12    0.00   11.39
pghot_recorded_hwhints        0.00    0.00    0.00   34.03   32.42
hwhint_total_events           0.00    0.00    0.00   34.03   32.42
hwhint_dram_accesses          0.00    0.00    0.00   20.03   22.34
hwhint_extmem_accesses        0.00    0.00    0.00   13.83    9.89

Key observations
----------------
1. pghot-both (both hintfaults and hwhints enabled) improves decode over both
   base baselines.
2. Prefill (pp512) suffers from tiering promotion (0.79x hintfaults, 0.78x hwhints,
   0.74x both vs notier); pghot/hf and /hw pp512 are noisy (stddev ~8-10).