[RFC PATCH 0/4] Kernel daemon for detecting and promoting hot pages

Bharata B Rao
2025-03-06T05:46:58Z

Hi,

This is an attempt towards having a single subsystem that accumulates
hot page information from lower memory tiers and does hot page
promotion.

At the heart of this subsystem is a kernel daemon named kpromoted that
does the following:

1. Exposes an API that other subsystems which detect/generate memory
access information can use to inform the daemon about memory
accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to
toptiers.

Currently I have added AMD IBS driver as one source that provides
page access information as an example. This driver feeds info to
kpromoted in this RFC patchset. More sources were discussed in a
similar context here at [1].

This is just an early attempt to check what it takes to maintain
a single source of page hotness info and also separate hot page
detection mechanisms from the promotion mechanism. There are too
many open ends right now and I have listed a few of them below.

- The API that is provided to register memory access expects
the PFN, NID and time of access at the minimum. This is
described more in patch 2/4. This API currently can be called
only from contexts that allow sleeping and hence this rules
out using it from PTE scanning paths. The API needs to be
more flexible with respect to this.
- Some sources like PTE A bit scanning can't provide the precise
time of access or the NID that is accessing the page. The latter
has been an open problem to which I haven't come across a good
and acceptable solution.
- The way the hot page information is maintained is pretty
primitive right now. Ideally we would like to store hotness info
in such a way that it should be easily possible to lookup say N
most hot pages.
- If PTE A bit scanners are considered as hotness sources, we will
be bombarded with accesses. Do we want to accomodate all those
accesses or just go with hotness info for fixed number of pages
(possibly as a ratio of lower tier memory capacity)?
- Undoubtedly the mechanism to classify a page as hot and subsequent
promotion needs to be more sophisticated than what I have right now.

This is just an early RFC posted now to ignite some discussion
in the context of LSFMM [2].

I am also working with Raghu to integrate his kmmdscan [3] as the
hotness source and use kpromoted for migration.

Also, I had posted the IBS driver ealier as an alternative to
hint faults based NUMA Balancing [4]. However here I am using
it as generic page hotness source.

[1] https://lore.kernel.org/linux-mm/de31971e-98fc-4baf-8f4f-09d153902e2e@amd.com/
[2] https://lore.kernel.org/linux-mm/20250123105721.424117-1-raghavendra.kt@amd.com/
[3] https://lore.kernel.org/all/20241201153818.2633616-1-raghavendra.kt@amd.com/
[3] https://lore.kernel.org/lkml/20230208073533.715-2-bharata@amd.com/

Regards,
Bharata.

Bharata B Rao (4):
mm: migrate: Allow misplaced migration without VMA too
mm: kpromoted: Hot page info collection and promotion daemon
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accesses

arch/x86/events/amd/ibs.c | 11 +
arch/x86/include/asm/entry-common.h | 3 +
arch/x86/include/asm/hardirq.h | 2 +
arch/x86/include/asm/ibs.h | 9 +
arch/x86/include/asm/msr-index.h | 16 ++
arch/x86/mm/Makefile | 3 +-
arch/x86/mm/ibs.c | 344 ++++++++++++++++++++++++++++
include/linux/kpromoted.h | 54 +++++
include/linux/mmzone.h | 4 +
include/linux/vm_event_item.h | 30 +++
mm/Kconfig | 7 +
mm/Makefile | 1 +
mm/kpromoted.c | 305 ++++++++++++++++++++++++
mm/migrate.c | 5 +-
mm/mm_init.c | 10 +
mm/vmstat.c | 30 +++
16 files changed, 831 insertions(+), 3 deletions(-)
create mode 100644 arch/x86/include/asm/ibs.h
create mode 100644 arch/x86/mm/ibs.c
create mode 100644 include/linux/kpromoted.h
create mode 100644 mm/kpromoted.c

--
2.34.1