[RFC PATCH 0/3] mm/vmscan: adaptive multi-threaded kswapd for NUMA-aware reclaim

[Ehab Ababneh <ehab.ababneh@intel.com>]

This series revives Buddy Lumpkin's earlier multi-kswapd proposal:
https://lkml.iu.edu/hypermail/linux/kernel/1804.0/00342.html

The motivation is stronger now than when the patch was first discussed.
Many current systems have hundreds of cores per NUMA node, not the
single-digit or low-tens core counts that were more common at the time.
When reclaim does not keep up, direct reclaim can still push allocation
latency into application paths and leave substantial CPU capacity waiting
for memory to be freed.

This patchset adds adaptive multi-threaded kswapd. The wakeup policy uses
node load to decide how many kswapd workers to run, so reclaim can scale
when it helps and stay conservative on already busy nodes.

Series summary:

1. Allow multiple kswapd threads per node and add control plumbing.
2. Wake an appropriate number of kswapd threads from per-node
   runnable load.

Concerns from the original discussion and how this series addresses some
of them:

- Concern: Direct reclaim is intended to slow a memory-hogging thread.
  Response: That can be acceptable on lower-core systems. On high-core
  systems, idling many cores while reclaim catches up can cost more than
  allowing reclaim parallelism to scale. It can also block higher-priority
  tasks in direct reclaim while they perform reclaim work on behalf of
  lower-priority memory-hogging tasks.

- Concern: More kswapd threads may hide deeper reclaim issues.
  Response: This series is additive to ongoing reclaim improvements. In
  our testing, multi-threaded kswapd was able to improve performance on
  top of what multi-gen LRU already provides.

- Concern: Existing knobs (such as swappiness and watermarks) should be
  preferred.
  Response: In our testing, those knobs alone did not reliably hit
  performance targets and could increase CPU cost for the same workload
  objective.

- Concern: Need evidence from real workloads.
  Response: This cover letter includes Cassandra results showing higher
  throughput and lower response latency.

- Concern: More reclaim threads may increase pressure on well-behaved
  tasks.
  Response: Adaptive wakeup addresses this by choosing thread count from
  node load.

- Concern: Additional configuration can increase operational complexity.
  Response: The user-facing interface is intentionally minimal:
  max_kswapds_per_node.

- Concern: Lock contention may serialize workers.
  Response: The Cassandra runs below still show net gains, indicating
  contention did not erase the benefit for this workload. The wakeup path
  now uses wake_up_nr() against the existing kswapd_wait queue, avoiding
  pgdat->kswapd_lock (a sleeping mutex) entirely on the allocation hot
  path.

Real-world workload results (Cassandra):

Tests were performed on 7.0.0-rc1.

- max_kswapds_per_node=1
  - throughput sample: 171146
  - reference latency value: 6.375
  - op rates: 43256, 42731, 42341, 42818 ops/s
  - p99 latency: 6.3, 6.4, 6.4, 6.4 ms

- max_kswapds_per_node=8
  - throughput sample: 183639
  - reference latency value: 6.0
  - op rates: 45791, 45253, 46534, 46061 ops/s
  - p99 latency: 6.0, 6.1, 5.9, 6.0 ms

Observed improvement in these runs was about +7.3% throughput and about
-5.9% response latency, which shows practical benefit for production-style
database workloads.

In our runs, performance numbers were essentially unchanged with and
without the adaptive multi-threaded kswapd wakeup policy. In both cases,
they outperformed the single-kswapd-thread baseline. This indicates the
adaptive method preserved the multi-threaded performance improvement.

Addendum: alternative approaches evaluated

- PSI per NUMA node.
  I prototyped PSI-based node pressure ranges to drive wakeup count.
  This became cumbersome because robust PSI-to-thread mappings were not
  straightforward across workload types.

- CPU mask snapshot policy.
  I also tested a simple CPU mask snapshot approach.
  While functional, it reflects a moment-in-time view and does not capture
  pressure trends over a broader sampling window.

Buddy Lumpkin (1):
  vmscan: Support multiple kswapd threads per node

Ehab Ababneh (2):
  mm/vmscan: handle racing max_seq advancement
  mm/vmscan: make kswapd wakeups NUMA load-aware

 include/linux/mmzone.h        |   5 +-
 include/trace/events/vmscan.h |  28 +++
 mm/compaction.c               |   8 +-
 mm/internal.h                 |   3 +
 mm/page_alloc.c               |  26 +++
 mm/vmscan.c                   | 419 +++++++++++++++++++++++++++++++++++++++---
 6 files changed, 465 insertions(+), 24 deletions(-)

-- 
2.43.0