[RFC PATCH v3 00/14] Introduce tiered memcg limits

[Joshua Hahn <joshua.hahnjy@gmail.com>]

INTRODUCTION
============
On machines serving multiple workloads whose memory is isolated with the
memory cgroup controller, it is currently impossible to enforce a fair
distribution of memory placement. memory.{max, high} bounds a cgroup's
total footprint but says nothing about where that memory resides, so a
cgroup that starts first can occupy all of a fast tier, while latecomers
are pushed out to slower tiers.

On systems with tiered memory (e.g. HBM, DRAM, CXL, PMEM, etc.),
memory placement directly maps to performance, meaning memory and
performance isolation breaks down on tiered systems; well-behaved
workloads using less memory than their memcg limits may still hurt other
workloads' performance by hogging more than their "fair share" of memory
on fast tiers. Performance then depends on workload-external factors,
like which other workloads are present on the system and what order they
were launched in.

However, ensuring fairness in memory placement is not free; ensuring
fair memory placement can mean underutilizing fast tiers, entering
reclaim earlier to proactively prevent overconsuming on fast tiers,
and host-level hotness inversion. On lightly-loaded systems, it means
that reclaim happens on fast tiers, even if no other workloads compete
for that memory. For some workloads, this tradeoff can be worth taking
to ensure higher run-to-run consistency and isolation from
noisy neighbors (see USECASES).

Introduce tiered memcg limits, which establish memory limits per-tier
that scale with memcg limits and the system's per-tier capacity.

MECHANISM
=========
memory.{min, low, high, max} are partitioned across the system's memory
tiers in proportion to the tiers' share of the total memory.
For instance, on a two-tier system with 75% of its memory in tier 0:
  - a cgroup with memory.max = 100G has a 75G tier 0 max limit
  - a cgroup with memory.low = 40G has 30G of tier 0 protection

Enforcement limits (high / max) are enforced in five ways:
  - At allocation time, a cgroup's allocation will try to serve a page
    from a tier where it still has headroom.
  - At try_charge_memcg time, a folio that lands on an exhausted tier
    anyways (because of __GFP_THISNODE, mempolicy, or memory pressure)
    triggers targeted reclaim restricted to that tier's nodes.
  - In the background, a tier pushed over its tiered memcg limit by page
    migration or folio replacement gets reclaimed by the memory.high
    worker.
  - Migration (promotion / demotion) now performs a try_charge on
    cross-tier migrations, and will attempt to reclaim the destination
    tier if it breaches the limits.
  - In addition, promotions are ratelimited when they are attempted on
    exhausted memcg tiers.

A few details about the enforcement:
  - Allocation time steering is best-effort and yields to explicit
    allocation requests, like mempolicy, cpusets, and allocations that
    request a specific node (promotion / demotion). If that allocation
    were to push the memcg tier over its limit, enforcement happens at
    charge time.
  - Tier max can be exceeded by demotions, which may force charge
    since they can't (and should not) trigger recursive reclaim.
  - As of this time, the mechanism is completely transparent to the
    user. Booting the system with the "cgroup.memory=tiered_limits"
    boot parameter will enable the feature, with no other interfaces.
  - There are no behavioral or performance effects on systems without
    the boot parameter, since all of the hooks are gated behind a static
    branch that compiles to a no-op. However, there are allocations and
    static variables that we must define unconditionally
    (even though they are not used).
  - Tiered memcg limits are a v2-only feature. A warning is emitted if a
    cgroup v1 user attempts to use tiered memcg limits.

USECASES
========
As mentioned in the introduction, some users may prefer to trade off
total throughput for reduced run-to-run variance:
  - VM hosting services that must provide the maximal performance
    guarantee for any workload present on a host.
  - Database workloads that want to minimize the maximum latency for
    queries hosted on the host.
  - Hosts running memory-isolated sharded workloads that block progress
    until the last shard terminates.
  - Any workload that wants to minimize variance, as a means to gather
    measurable gains in performance over time.

RFC QUESTIONS
=============
  - Should we OOM when we fail to allocate on a given tier?
  - What kinds of observability (if any) do we want?
  - What kinds of user interfaces (if any) do we want?

TESTING
=======
All the tests were run on a 1TB 2-tier 316 CPU machine, with
750G DRAM and 250G CXL. The first benchmark runs 12 containers on the
host, where each container runs an identical pointer-chasing task in a
loop. Each container is 80GB, so DRAM is overcommitted:
    (80GB * 12 = 960GB > 750GB)
I ran 8 trials and aggregated the data. When 50% of the pointers
are kept hot, these are the results:

Mean DRAM usage across the 12 workloads:
+---------+----------+--------+
|         | Untiered | Tiered |
+---------+----------+--------+
| min     | 43.09    | 51.25  |
| max     | 68.54    | 53.03  |
| max/min | 1.59x    | 1.03x  |
+---------+----------+--------+

Mean CXL usage across the 12 workloads:
+---------+----------+--------+
|         | Untiered | Tiered |
+---------+----------+--------+
| min     | 3.05     | 13.98  |
| max     | 21.65    | 14.87  |
| max/min | 7.10x    | 1.06x  |
+---------+----------+--------+

For the tiered system, the deviance from max to min DRAM and CXL usage
is quite low across workloads. What is interesting is that for untiered
systems, there is up to a 7x difference between the cgroup using the
most CXL and the least CXL; there's a clear inequality in how the
resource gets allocated.

This gets reflected in the distribution of the job completion times of
the 12 workloads as well.

Job completion time (s)
+-----------+----------+--------+-------+
|           | Untiered | Tiered | Delta |
+-----------+----------+--------+-------+
| min       |    260.1 |  276.5 | +6.3% |
| mean      |    274.4 |  290.0 | +5.7% |
| max       |    315.0 |  302.6 | -3.9% |
| max - min |     54.9 |   26.1 | -52%  |
+-----------+----------+--------+-------+

SERIES OVERVIEW
===============
Commits 1-3 are preparatory patches. We introduce the new boot
parameter, refactor try_charge_memcg() to make the following charges
easier to follow, and cache frequently-used values like the
node <-> tier mapping.

Commits 4-5 allocate the new per-memcg-tier page_counters for tracking
and set their limits according to the standard memcg limits.

Commits 6-10 track per-tier charges, and introduce the protection and
enforcement mechanisms for min/low/high/max. 

Commits 11-13 further enforce the migration (promotion / demotion) paths
and ensure that we do not introduce unnecessary churn by reclaiming
earlier. 

Commit 14 steers allocations so that we do not accidentally introduce
zone_reclaim_mode-like behavior and immediately reclaim after allocating
on exhausted memcg tiers.

FUTURE WORK
===========
I am currently working on another series [1], which pushes the memcg
cached charges (stock) to the page_counter level, meaning each tier can
manage its own independent stock. This should increase the performance
and simplicity of the code much more.

CHANGELOG
=========
v2 --> v3
- N tier support, instead of toptier vs. rest enforcement
- Max enforcement
- Migration enforcement
- Page allocator steering for exhausted tiers
- Promotion fastpath restriction & throttling

[1] https://lore.kernel.org/all/20260623180124.868655-1-joshua.hahnjy@gmail.com/

Joshua Hahn (14):
  mm/memcontrol: Introduce cgroup.memory=tiered_limits boot parameter
  mm/memcontrol: Refactor page_counter charging in try_charge_memcg
  mm/memory-tiers: Introduce a mapping from nid to tier_slot
  mm/memcontrol: Allocate per-tier page_counters
  mm/memcontrol: Set tier limits proportional to memory limits
  mm/vmscan, memcontrol: Add nodemask to try_to_free_mem_cgroup_pages
  mm/memcontrol: Charge/uncharge tiered memory to mem_cgroup
  mm/memcontrol: Make memory.low and memory.min tier-aware
  mm/memcontrol: Make memory.high tier-aware
  mm/memcontrol: Make memory.max tier-aware
  mm/memcontrol, migrate: Transfer tier charge on migration
  mm/memcontrol: Kick async reclaim on migration and folio replacement
  mm/memcontrol, sched/numa: Gate NUMA promotions into memcg tiers
  mm/page_alloc: steer allocations away from exhausted memory tiers

 include/linux/memcontrol.h   |  75 ++++-
 include/linux/memory-tiers.h |  24 ++
 kernel/sched/fair.c          |   3 +-
 mm/internal.h                |   3 +-
 mm/memcontrol-v1.c           |   5 +-
 mm/memcontrol.c              | 528 ++++++++++++++++++++++++++++++++---
 mm/memory-tiers.c            | 110 +++++++-
 mm/migrate.c                 |  30 +-
 mm/page_alloc.c              |  19 +-
 mm/vmscan.c                  |  25 +-
 10 files changed, 753 insertions(+), 69 deletions(-)


base-commit: 7b25c83e4711038989b5b08a8977fb68468c854e
-- 
2.53.0-Meta