[PATCH] mm/memory_hotplug: cache zone stats for "auto-movable" online policy

[Zhijian Han <hanzhijian1991@gmail.com>]

When onlining memory using the "auto-movable" online policy with
nid == NUMA_NO_NODE, auto_movable_can_online_movable() walks all
populated zones across all nodes on each invocation to collect the
MOVABLE vs. KERNEL_EARLY stats.  This walk happens twice for every
memory block onlining decision (global check and, with NUMA awareness
enabled, per-node check) and is repeated for every single memory block
that gets onlined, although the underlying zone counters rarely change.

Cache the stats for nid == NUMA_NO_NODE and only recalculate them when
a zone counter actually changes.  All modifications of the relevant
zone counters (present_pages, present_early_pages) funnel through
adjust_present_page_count(), which is only called while holding the
mem_hotplug_lock in write mode and the device_lock() of the memory
block device.  Invalidate the cache there, such that the next
auto-movable onlining decision recalculates the stats from scratch.

CMA adjustments (zone->cma_pages) only happen during boot (via
init_cma_reserved_pageblock()/init_cma_pageblock(), both __init) and
cannot race with memory onlining.  Boot-time zone initialization
happens before any memory block can be onlined.

The per-node path (nid != NUMA_NO_NODE) is left uncached; it only
walks a single node's zones.

Resolve a TODO that was left when the "auto-movable" online policy was
introduced.

Tested on QEMU (x86_64, 512M boot + 256M hotplugged pc-dimm,
online_policy=auto-movable): hotplugged memory blocks get onlined to
ZONE_MOVABLE as expected, and an offline/online cycle of a hotplugged
block shows the correct zone counters after cache invalidation, with
no kernel warnings.

Link: https://lore.kernel.org/r/20210806124715.17090-3-david@redhat.com
Signed-off-by: Zhijian Han <hanzhijian1991@gmail.com>
---
 mm/memory_hotplug.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index d7a59167bec4..dfac59739a6d 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -789,6 +789,19 @@ struct auto_movable_stats {
 	unsigned long movable_pages;
 };
 
+/*
+ * Cached stats for all populated zones across all nodes. Modified (incl.
+ * invalidation) only while holding the mem_hotplug_lock in write mode and
+ * the device_lock() of the memory block device.
+ */
+static struct auto_movable_stats auto_movable_zone_stats;
+static bool auto_movable_zone_stats_valid;
+
+static void auto_movable_zone_stats_invalidate(void)
+{
+	auto_movable_zone_stats_valid = false;
+}
+
 static void auto_movable_stats_account_zone(struct auto_movable_stats *stats,
 					    struct zone *zone)
 {
@@ -849,9 +862,14 @@ static bool auto_movable_can_online_movable(int nid, struct memory_group *group,
 
 	/* Walk all relevant zones and collect MOVABLE vs. KERNEL stats. */
 	if (nid == NUMA_NO_NODE) {
-		/* TODO: cache values */
-		for_each_populated_zone(zone)
-			auto_movable_stats_account_zone(&stats, zone);
+		if (auto_movable_zone_stats_valid) {
+			stats = auto_movable_zone_stats;
+		} else {
+			for_each_populated_zone(zone)
+				auto_movable_stats_account_zone(&stats, zone);
+			auto_movable_zone_stats = stats;
+			auto_movable_zone_stats_valid = true;
+		}
 	} else {
 		for (i = 0; i < MAX_NR_ZONES; i++) {
 			pg_data_t *pgdat = NODE_DATA(nid);
@@ -1079,6 +1097,8 @@ void adjust_present_page_count(struct page *page, struct memory_group *group,
 	zone->present_pages += nr_pages;
 	zone->zone_pgdat->node_present_pages += nr_pages;
 
+	auto_movable_zone_stats_invalidate();
+
 	if (group && movable)
 		group->present_movable_pages += nr_pages;
 	else if (group && !movable)
-- 
2.43.0