[PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[Gregory Price]

A per-node vmstat counter is pgdat->vm_stat[] plus per-cpu deltas.
A balanced counter can sit split as global=+N / per-cpu=-N.

The folds reconciling the split only walk online nodes, so when
try_offline_node() marks a node offline - per-cpu deltas are stranded.

A subsequent online zeroes the per-cpu area but not pgdat->vm_stat[],
orphaning the +N permanently. All NR_VM_NODE_STAT_ITEMS are affected.

Flush the deltas before the node leaves the online set.  A remote
fold races the periodic per-cpu fold, so do it as per-cpu work.

Discovered when a node/compact call hung for a nearly empty node, as
the math to determine throttling broke. Reproduced by repeated memory
hotplug/unplug cycles on a node under pressure. NR_ISOLATED_ANON
ratchets up and never returns to zero.

Fixes: 75ef71840539 ("mm, vmstat: add infrastructure for per-node vmstats")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price <gourry@gourry.net>
---
 include/linux/vmstat.h |  2 ++
 mm/memory_hotplug.c    |  5 ++++-
 mm/vmstat.c            | 10 ++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 3c9c266cf782..ea1017427811 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -293,6 +293,7 @@ extern void __dec_node_state(struct pglist_data *, enum node_stat_item);
 
 void quiet_vmstat(void);
 void cpu_vm_stats_fold(int cpu);
+void sync_vm_stats(void);
 void refresh_zone_stat_thresholds(void);
 
 void drain_zonestat(struct zone *zone, struct per_cpu_zonestat *);
@@ -397,6 +398,7 @@ static inline void __dec_node_page_state(struct page *page,
 
 static inline void refresh_zone_stat_thresholds(void) { }
 static inline void cpu_vm_stats_fold(int cpu) { }
+static inline void sync_vm_stats(void) { }
 static inline void quiet_vmstat(void) { }
 static inline void vmstat_flush_workqueue(void) { }
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 7d60a7dd1e7b..10f676566f56 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -2338,8 +2338,11 @@ void try_offline_node(int nid)
 
 	/*
 	 * all memory/cpu of this node are removed, we can offline this
-	 * node now.
+	 * node now.  Fold any pending per-cpu vmstat diffs into the global
+	 * counters first: once the node leaves the online set the periodic
+	 * fold skips it, orphaning the residual on a later online.
 	 */
+	sync_vm_stats();
 	node_set_offline(nid);
 	unregister_node(nid);
 }
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..ad77343212d3 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -941,6 +941,16 @@ void cpu_vm_stats_fold(int cpu)
 	fold_diff(global_zone_diff, global_node_diff);
 }
 
+static void vmstat_fold_work(struct work_struct *w)
+{
+	refresh_cpu_vm_stats(false);
+}
+
+void sync_vm_stats(void)
+{
+	schedule_on_each_cpu(vmstat_fold_work);
+}
+
 /*
  * this is only called if !populated_zone(zone), which implies no other users of
  * pset->vm_stat_diff[] exist.
-- 
2.53.0-Meta

Re: [PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Recursive acquisition of `cpus_read_lock()` causes a wait-die (AB-BA) reader-writer deadlock with CPU hotplug operations.
- [Medium] Ignoring the `-ENOMEM` return value of `schedule_on_each_cpu()` causes a silent failure, leaving stats permanently orphaned under memory pressure.
--

commit 195f8756c6e31f6d21c338743702f407e4ff93d1
Author: Gregory Price <gourry@gourry.net>

mm/vmstat: flush per-cpu node stats when a node goes offline

This patch flushes per-cpu vmstat deltas to global counters before a node
is offlined. It aims to prevent orphaned statistics that cause throttling
issues by using per-cpu work to synchronize the stats.

> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index f534972f517de..ad77343212d37 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -941,6 +941,16 @@ void cpu_vm_stats_fold(int cpu)
>  	fold_diff(global_zone_diff, global_node_diff);
>  }
>  
> +static void vmstat_fold_work(struct work_struct *w)
> +{
> +	refresh_cpu_vm_stats(false);
> +}
> +
> +void sync_vm_stats(void)
> +{
> +	schedule_on_each_cpu(vmstat_fold_work);

[Severity: High]
Can this cause a reader-writer deadlock with CPU hotplug operations?

In the memory hot-remove path, try_remove_memory() acquires the
cpu_hotplug_lock for read via mem_hotplug_begin() before calling
try_offline_node():

mm/memory_hotplug.c:try_remove_memory() {
        mem_hotplug_begin();
        ...
        try_offline_node(nid);
}

Then try_offline_node() calls sync_vm_stats(), which bridges into
schedule_on_each_cpu(). The issue is that schedule_on_each_cpu()
unconditionally attempts to acquire the cpu_hotplug_lock for read again:

kernel/workqueue.c:schedule_on_each_cpu() {
        ...
        cpus_read_lock();
}

If a concurrent CPU hotplug operation takes the lock for write between the
first and second read lock acquisitions, it sets the writer-pending flag.
The second read lock attempt will then block waiting for the writer, while
the writer is blocked indefinitely waiting for the first read lock to be
released.

[Severity: Medium]
Does this silently ignore allocation failures?

Under memory pressure, schedule_on_each_cpu() can fail to allocate
the per-cpu work items and return -ENOMEM:

kernel/workqueue.c:schedule_on_each_cpu() {
        works = alloc_percpu(struct work_struct);
        if (!works)
                return -ENOMEM;
        ...
}

Since sync_vm_stats() returns void, this failure is swallowed. If this
allocation fails during try_offline_node(), the stats will not be flushed
and the per-CPU deltas will be permanently orphaned on a later online,
which is the original bug this patch aims to fix.

> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260627073107.523499-1-gourry@gourry.net?part=1

Re: [PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[Gregory Price]

On Sat, Jun 27, 2026 at 03:31:07AM -0400, Gregory Price wrote:
>  
>  	/*
>  	 * all memory/cpu of this node are removed, we can offline this
> -	 * node now.
> +	 * node now.  Fold any pending per-cpu vmstat diffs into the global
> +	 * counters first: once the node leaves the online set the periodic
> +	 * fold skips it, orphaning the residual on a later online.
>  	 */
> +	sync_vm_stats();

Sashiko points out this can deadlock on concurrent cpu hotplug, which
after a quick look seems accurate.

Has also made me realize the sync code is also racy with cpu hotplug
as well as it iterates per-online-cpu.

There are probably more races like this out there.

Will need to give this a little more thought.

>  	node_set_offline(nid);
>  	unregister_node(nid);
>  }

Re: [PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[Gregory Price]

On Sat, Jun 27, 2026 at 03:31:07AM -0400, Gregory Price wrote:
> A per-node vmstat counter is pgdat->vm_stat[] plus per-cpu deltas.
> A balanced counter can sit split as global=+N / per-cpu=-N.
> 
> The folds reconciling the split only walk online nodes, so when
> try_offline_node() marks a node offline - per-cpu deltas are stranded.
> 
> A subsequent online zeroes the per-cpu area but not pgdat->vm_stat[],
> orphaning the +N permanently. All NR_VM_NODE_STAT_ITEMS are affected.
> 
> Flush the deltas before the node leaves the online set.  A remote
> fold races the periodic per-cpu fold, so do it as per-cpu work.
> 
> Discovered when a node/compact call hung for a nearly empty node, as
> the math to determine throttling broke. Reproduced by repeated memory
> hotplug/unplug cycles on a node under pressure. NR_ISOLATED_ANON
> ratchets up and never returns to zero.
> 
> Fixes: 75ef71840539 ("mm, vmstat: add infrastructure for per-node vmstats")
> Cc: stable@vger.kernel.org
> Signed-off-by: Gregory Price <gourry@gourry.net>

Realized I changed the title on v2:
https://lore.kernel.org/linux-mm/20260627073107.523499-1-gourry@gourry.net/

Core issue was the zeroing at online time causing the skew, just have to
do the fold there instead.  disregard this version please