[PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure

[Bharata B Rao <bharata@amd.com>]

Hi,

This patchset introduces pghot, a subsystem for hot page tracking and
promotion. It collects memory access information from multiple sources,
classifies hot pages resident in lower-tier memory, and promotes them to
faster tiers via a per-lower-tier-node kernel thread (kmigrated).

The main changes in v8 are a rework of the per-section hotness map to be
RCU-protected, a move of the pghot tunables from debugfs to sysctl, and a
rework of the AMD IBS Memory Profiler driver that now uses a dedicated system
interrupt vector instead of an NMI handler. The IBS driver is also split
into core infrastructure and a separate patch that adds the runtime
controls and arms the profiler. Also there are a few fixes to the hint
faults source.

Goals of this patchset:

- Unify hot page detection from multiple sources like hint faults,
  page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel
  thread.
- Move promotion rate-limiting and related logic used by numa_balancing=2
  (NUMAB2, the current NUMA balancing-based promotion) from the scheduler
  to pghot for broader reuse.

Currently, multiple kernel subsystems detect page accesses independently.
This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Performance summary
===================
All results are on 3-node tiered systems (DRAM top tier + CPU-less CXL
lower tier). Speedups below are normalized to the base kernel with no
tiering (NUMAB=0). Columns compare mainline hint-fault tiering
(base NUMAB=2) against pghot in two modes: hint-fault driven (pghot-hf,
NUMAB=2) and HW-hint driven via the AMD IBS memory profiler (pghot-hw,
NUMAB=0, no NUMA scanning). Single run per config (abench is avg of 3).

Benchmark      Metric (higher=better)   base-NUMAB2  pghot-hf  pghot-hw
-------------------------------------------------------------------------
NAS BT (MPI)   Mop/s total                  2.26x     2.26x     2.13x
Graph500 BFS   harmonic-mean TEPS           2.64x     2.49x     2.68x
llama.cpp      decode tok/s (tg128)         1.09x     1.41x     1.21x
Redis+memtier  ops/sec                      1.05x     1.05x     1.00x
Microbench     completion time (1/t)        2.41x     2.40x     2.35x
-------------------------------------------------------------------------
(baseline = base kernel, no tiering = 1.00x)

Summary: pghot hint-faults reproduces mainline NUMAB=2 with no
regression on every workload; pghot HW-hints/IBS matches it mostly.

- NAS BT (MPI, Class F modified, 16 ranks): full footprint pre-staged
  on the CXL lower tier, then measured cold; isolates hot-page promotion.
- Graph500 (reference BFS, SCALE=28, edgefactor=16, 128 ranks):
  working set on the CXL node; figure of merit is harmonic-mean TEPS.
- llama.cpp (llama-bench, Mixtral-8x22B Q4_K_M): ~1/3 of the model
  demoted to CXL under memory pressure; decode (tg128) is tier-sensitive.
- Redis + memtier (~64 GB dataset, 62.2M x 1 KB keys): dataset
  migrated to CXL, GET-heavy traffic over 50% of the keyspace.
- Microbenchmark: 64 threads, random 4K access over 8G on
  the CXL node; latency-bound.

Detailed per-benchmark tables (throughput/latency + vmstat and pghot
promotion counters) will be sent as replies to this patchset post.

How pghot works
===============
- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded
  access is tracked in precision mode.
- These hotness parameters are maintained in a per-PFN hotness record
  within a per-section map hanging off the existing mem_section data
  structure. The map is RCU-protected and also carries a section-level
  "hot" flag used to gate scanning by kmigrated.
  - In default mode, one byte (u8) is used for the hotness record. 5 bits
    store time using a bucketing scheme representing up to 4s with
    HZ=1000. The default toptier NID (0) is used as the promotion target
    and can be changed via the vm.pghot_target_nid sysctl.
  - In precision mode, 4 bytes (u32) are used for each hotness record.
    14 bits store time, representing around 16s with HZ=1000, and the
    accessing NID is stored per-PFN as the promotion target.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked ready for migration using the ready
  bit (MSB of the hotness record in both modes).
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of
  lower-tier nodes, checking the migration-ready bit to perform batched
  migrations. The scan interval and batch size are configurable via
  debugfs tunables.

Memory overhead
===============
Default mode: 1 byte per lower-tier PFN. For 1TB of lower-tier memory
this amounts to 256MB overhead (assuming 4K pages).

Precision mode: 4 bytes per lower-tier PFN. For 1TB of lower-tier memory
this amounts to 1GB overhead.

Bit layout of hotness record
============================
Default mode
- Bits 0-1: Frequency (2 bits, 4 access samples)
- Bits 2-6: Bucketed time (5 bits, up to 4s with HZ=1000)
- Bit 7:    Migration ready bit

Precision mode
- Bits 0-9:   Target NID (10 bits)
- Bits 10-12: Frequency (3 bits, 8 access samples)
- Bits 13-26: Time (14 bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31:     Migration ready bit

Tunables
========
sysctl (/proc/sys/vm/):
- pghot_enabled_sources      - bitmask of hotness sources (0x1 hint
                               faults, 0x2 hardware hints)
- pghot_target_nid           - toptier target NID (default mode)
- pghot_freq_threshold       - accesses before a page is promotable
- pghot_promote_freq_window_ms - access-frequency counting window
- pghot_promote_rate_limit_MBps - per-target promotion rate limit
  (numa_balancing_promote_rate_limit_MBps is retained as a compat alias)

debugfs (/sys/kernel/debug/pghot/):
- kmigrated_sleep_ms, kmigrated_batch_nr

Potential hotness sources
=========================
1. NUMA Balancing (NUMAB2, Tiering mode) - included in this patchset.
2. AMD IBS Memory Profiler: HW based access profiler - included in this
   patchset.
3. klruscand - PTE-A bit scanning built on MGLRU's walk helpers - was
   showcased in previous versions but not part of this patchset.
4. folio_mark_accessed() - Page cache access tracking (unmapped page
   cache pages) - was showcased in previous versions but not part of
   this patchset.

Changes in v8
=============
pghot-core
- Augmented documentation with per-source information
- Reworked the per-section hotness map (mem_section->hot_map) into an
  RCU-protected structure (struct pghot_hot_map) that holds the per-PFN
  records and section-level flags, for safe concurrent access and
  teardown. The section "hot" flag now lives in this structure instead of
  the pointer's LSB.
- Cleanup and fixes in pghot_update_record().
- Moved pghot tunables enabled_sources, target_nid and freq_threshold
  from debugfs to sysctl (/proc/sys/vm/). Only kmigrated_sleep_ms and
  kmigrated_batch_nr remain in debugfs.
- Kmigrated walk: Protect folio_memcg() under RCU readlock

pghot-hintfaults
- Retained kernel.numa_balancing_promote_rate_limit_MBps for compatibility
- Filter shared executable pages at NUMA hint fault time in
  do_numa_page()/do_huge_pmd_numa_page(), since the VMA is not available
  to kmigrated's batched migration from non-process context.
- Fixed NUMAB2 to continue feeding to task_numa_fault() so that scan
  period updates happens for pghot-NUMAB2.
- Now hint faults source is enabled by default in pghot so that setting
  kernel.numa_balancing=2 is enough for tiering mode to get activated
  as before.
- To avoid a regression for existing configs, default PGHOT to y when
  NUMA_BALANCING is enabled and default NUMA_BALANCING_TIERING to y.
  Since the tiering promotion engine now lives in pghot, an existing
  CONFIG_NUMA_BALANCING=y config would otherwise silently lose the ability
  to use kernel.numa_balancing=2 (writes returning -EOPNOTSUPP) after an
  oldconfig upgrade.
  Note: PGHOT depends on SPARSEMEM, so a NUMA_BALANCING=y && !SPARSEMEM
  (e.g. FLATMEM) config can no longer use kernel.numa_balancing=2.

pghot-hwhints
- Split the IBS Memory Profiler driver into core infrastructure and a
  separate patch that adds runtime controls (debugfs/sysfs and the
  ibs-memprof= cmdline parameter) and arms the profiler.
- Changed IBS Memory Profiler source to use regular interrupts rather
  than NMI. Unlike the primary IBS which needs to obtain samples from
  kernel and interrupt contexts for performance profiling, with memory access
  profiling only user space samples are obtained. Hence NMI isn't required.
  Also this simplifies the code by getting rid of irq_work hander.
- Fixed lost wakeup issue in work handler that drains the samples by
  draining samples in a loop until the ring buffer is observably empty.
- Drain and discard the samples of the CPU that is going offline.
- Fix the issue where the work item queued for a dying CPU can execute
  on a different online CPU leading to draining of that CPU's ring.
- Added ibs-memprof kernel cmdline option to enable hwhint source at boot
  time itself.
- Added vmstat counter hwhint_dropped_events that provides the count of
  those samples which were dropped due to sample buffer overrun.
- Added a sysfs tunable /sys/devices/system/cpu/ibs-mprof/enabled to
  dynamically enable or disable IBS Memory Profiler.
- Added debugfs tunables (l3miss-only, period, lat-filter, lat-thresh)
  to support configuration changes to IBS Memory profiler.

Compile test summary
====================
The series was compile-tested across the full range of relevant Kconfig
combinations on x86_64 (defconfig-based) and cross-compiled on arm64.
Combinations of PGHOT, PGHOT_PRECISE, NUMA_BALANCING,
NUMA_BALANCING_TIERING, NUMA_MIGRATION, HWMEM_PROFILER/AMD_IBS_MEMPROF,
DEBUG_FS and SYSCTL were verified, including pghot disabled, pghot
standalone (no NUMA balancing), NUMA balancing mode 1 and mode 2 (tiering),
precise mode, the AMD IBS memory profiler, NUMA_MIGRATION=n, and pghot with
DEBUG_FS and/or SYSCTL disabled. Kconfig guards were confirmed
(NUMA_BALANCING_TIERING without PGHOT, and HWMEM_PROFILER without a
selecting driver, are dropped). Since PGHOT defaults to y when
NUMA_BALANCING is set and NUMA_BALANCING_TIERING defaults to y, an existing
config with NUMA_BALANCING=y but no pghot symbols resolves, via
olddefconfig, to PGHOT=y and NUMA_BALANCING_TIERING=y and builds cleanly,
while NUMA_BALANCING=n leaves PGHOT off. allnoconfig, tinyconfig,
allyesconfig and allmodconfig were also built. All configurations build with
no new warnings.

This v8 patchset applies on top of upstream commit 8cdeaa50eae8 and can be
fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-v8

v7: https://lore.kernel.org/linux-mm/20260504060924.344313-1-bharata@amd.com/
v6: https://lore.kernel.org/linux-mm/20260323095104.238982-1-bharata@amd.com/
v5: https://lore.kernel.org/linux-mm/20260129144043.231636-1-bharata@amd.com/
v4: https://lore.kernel.org/linux-mm/20251206101423.5004-1-bharata@amd.com/
v3: https://lore.kernel.org/linux-mm/20251110052343.208768-1-bharata@amd.com/
v2: https://lore.kernel.org/linux-mm/20250910144653.212066-1-bharata@amd.com/
v1: https://lore.kernel.org/linux-mm/20250814134826.154003-1-bharata@amd.com/
v0: https://lore.kernel.org/linux-mm/20250306054532.221138-1-bharata@amd.com/

base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda

IBS Memory Profiler part of this patchset depends on the patchset that
increases the number of APIC EILVT registers -
https://lore.kernel.org/lkml/cover.1775019269.git.naveen@kernel.org/

AI assistance was taken to run and summarize the benchmark results.

Bharata B Rao (7):
  mm: migrate: Allow misplaced migration without VMA
  mm: Hot page tracking and promotion - pghot
  mm: pghot: Precision mode for pghot
  mm: sched: move NUMA balancing tiering promotion to pghot
  x86/ibs: Move IBS caps definitions into its own header
  x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler
  x86/mm/ibs: Add runtime controls for IBS memprofiler

Gregory Price (1):
  mm: migrate: Add promote_misplaced_memcg_folios()

 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/admin-guide/mm/index.rst        |   1 +
 Documentation/admin-guide/mm/pghot.rst        | 196 ++++
 arch/x86/Kconfig                              |  16 +
 arch/x86/entry/entry_fred.c                   |   1 +
 arch/x86/include/asm/hardirq.h                |   3 +
 arch/x86/include/asm/ibs-caps.h               |  93 ++
 arch/x86/include/asm/ibs-mprof.h              |  61 ++
 arch/x86/include/asm/idtentry.h               |   6 +
 arch/x86/include/asm/irq_vectors.h            |   4 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/perf_event.h             |  81 +-
 arch/x86/kernel/idt.c                         |   3 +
 arch/x86/kernel/irq.c                         |   3 +
 arch/x86/mm/Makefile                          |   1 +
 arch/x86/mm/ibs-mprof.c                       | 704 ++++++++++++++
 include/linux/cpuhotplug.h                    |   1 +
 include/linux/migrate.h                       |   9 +-
 include/linux/mm.h                            |  35 +-
 include/linux/mmzone.h                        |  25 +-
 include/linux/pghot.h                         | 140 +++
 include/linux/vm_event_item.h                 |  12 +
 init/Kconfig                                  |  14 +
 kernel/sched/core.c                           |   7 +
 kernel/sched/debug.c                          |   1 -
 kernel/sched/fair.c                           | 171 +---
 kernel/sched/sched.h                          |   1 -
 mm/Kconfig                                    |  34 +
 mm/Makefile                                   |   6 +
 mm/huge_memory.c                              |  32 +-
 mm/memcontrol.c                               |   6 +-
 mm/memory-tiers.c                             |  15 +-
 mm/memory.c                                   |  36 +-
 mm/mempolicy.c                                |   3 -
 mm/migrate.c                                  |  99 +-
 mm/mm_init.c                                  |  10 +
 mm/pghot-default.c                            |  76 ++
 mm/pghot-precise.c                            |  77 ++
 mm/pghot.c                                    | 891 ++++++++++++++++++
 mm/vmstat.c                                   |  14 +-
 40 files changed, 2593 insertions(+), 308 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 arch/x86/include/asm/ibs-caps.h
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot.c

-- 
2.34.1