[RFC PATCH v3 00/14] Introduce tiered memcg limits

[RFC PATCH v3 00/14] Introduce tiered memcg limits

[Joshua Hahn]

INTRODUCTION
============
On machines serving multiple workloads whose memory is isolated with the
memory cgroup controller, it is currently impossible to enforce a fair
distribution of memory placement. memory.{max, high} bounds a cgroup's
total footprint but says nothing about where that memory resides, so a
cgroup that starts first can occupy all of a fast tier, while latecomers
are pushed out to slower tiers.

On systems with tiered memory (e.g. HBM, DRAM, CXL, PMEM, etc.),
memory placement directly maps to performance, meaning memory and
performance isolation breaks down on tiered systems; well-behaved
workloads using less memory than their memcg limits may still hurt other
workloads' performance by hogging more than their "fair share" of memory
on fast tiers. Performance then depends on workload-external factors,
like which other workloads are present on the system and what order they
were launched in.

However, ensuring fairness in memory placement is not free; ensuring
fair memory placement can mean underutilizing fast tiers, entering
reclaim earlier to proactively prevent overconsuming on fast tiers,
and host-level hotness inversion. On lightly-loaded systems, it means
that reclaim happens on fast tiers, even if no other workloads compete
for that memory. For some workloads, this tradeoff can be worth taking
to ensure higher run-to-run consistency and isolation from
noisy neighbors (see USECASES).

Introduce tiered memcg limits, which establish memory limits per-tier
that scale with memcg limits and the system's per-tier capacity.

MECHANISM
=========
memory.{min, low, high, max} are partitioned across the system's memory
tiers in proportion to the tiers' share of the total memory.
For instance, on a two-tier system with 75% of its memory in tier 0:
  - a cgroup with memory.max = 100G has a 75G tier 0 max limit
  - a cgroup with memory.low = 40G has 30G of tier 0 protection

Enforcement limits (high / max) are enforced in five ways:
  - At allocation time, a cgroup's allocation will try to serve a page
    from a tier where it still has headroom.
  - At try_charge_memcg time, a folio that lands on an exhausted tier
    anyways (because of __GFP_THISNODE, mempolicy, or memory pressure)
    triggers targeted reclaim restricted to that tier's nodes.
  - In the background, a tier pushed over its tiered memcg limit by page
    migration or folio replacement gets reclaimed by the memory.high
    worker.
  - Migration (promotion / demotion) now performs a try_charge on
    cross-tier migrations, and will attempt to reclaim the destination
    tier if it breaches the limits.
  - In addition, promotions are ratelimited when they are attempted on
    exhausted memcg tiers.

A few details about the enforcement:
  - Allocation time steering is best-effort and yields to explicit
    allocation requests, like mempolicy, cpusets, and allocations that
    request a specific node (promotion / demotion). If that allocation
    were to push the memcg tier over its limit, enforcement happens at
    charge time.
  - Tier max can be exceeded by demotions, which may force charge
    since they can't (and should not) trigger recursive reclaim.
  - As of this time, the mechanism is completely transparent to the
    user. Booting the system with the "cgroup.memory=tiered_limits"
    boot parameter will enable the feature, with no other interfaces.
  - There are no behavioral or performance effects on systems without
    the boot parameter, since all of the hooks are gated behind a static
    branch that compiles to a no-op. However, there are allocations and
    static variables that we must define unconditionally
    (even though they are not used).
  - Tiered memcg limits are a v2-only feature. A warning is emitted if a
    cgroup v1 user attempts to use tiered memcg limits.

USECASES
========
As mentioned in the introduction, some users may prefer to trade off
total throughput for reduced run-to-run variance:
  - VM hosting services that must provide the maximal performance
    guarantee for any workload present on a host.
  - Database workloads that want to minimize the maximum latency for
    queries hosted on the host.
  - Hosts running memory-isolated sharded workloads that block progress
    until the last shard terminates.
  - Any workload that wants to minimize variance, as a means to gather
    measurable gains in performance over time.

RFC QUESTIONS
=============
  - Should we OOM when we fail to allocate on a given tier?
  - What kinds of observability (if any) do we want?
  - What kinds of user interfaces (if any) do we want?

TESTING
=======
All the tests were run on a 1TB 2-tier 316 CPU machine, with
750G DRAM and 250G CXL. The first benchmark runs 12 containers on the
host, where each container runs an identical pointer-chasing task in a
loop. Each container is 80GB, so DRAM is overcommitted:
    (80GB * 12 = 960GB > 750GB)
I ran 8 trials and aggregated the data. When 50% of the pointers
are kept hot, these are the results:

Mean DRAM usage across the 12 workloads:
+---------+----------+--------+
|         | Untiered | Tiered |
+---------+----------+--------+
| min     | 43.09    | 51.25  |
| max     | 68.54    | 53.03  |
| max/min | 1.59x    | 1.03x  |
+---------+----------+--------+

Mean CXL usage across the 12 workloads:
+---------+----------+--------+
|         | Untiered | Tiered |
+---------+----------+--------+
| min     | 3.05     | 13.98  |
| max     | 21.65    | 14.87  |
| max/min | 7.10x    | 1.06x  |
+---------+----------+--------+

For the tiered system, the deviance from max to min DRAM and CXL usage
is quite low across workloads. What is interesting is that for untiered
systems, there is up to a 7x difference between the cgroup using the
most CXL and the least CXL; there's a clear inequality in how the
resource gets allocated.

This gets reflected in the distribution of the job completion times of
the 12 workloads as well.

Job completion time (s)
+-----------+----------+--------+-------+
|           | Untiered | Tiered | Delta |
+-----------+----------+--------+-------+
| min       |    260.1 |  276.5 | +6.3% |
| mean      |    274.4 |  290.0 | +5.7% |
| max       |    315.0 |  302.6 | -3.9% |
| max - min |     54.9 |   26.1 | -52%  |
+-----------+----------+--------+-------+

SERIES OVERVIEW
===============
Commits 1-3 are preparatory patches. We introduce the new boot
parameter, refactor try_charge_memcg() to make the following charges
easier to follow, and cache frequently-used values like the
node <-> tier mapping.

Commits 4-5 allocate the new per-memcg-tier page_counters for tracking
and set their limits according to the standard memcg limits.

Commits 6-10 track per-tier charges, and introduce the protection and
enforcement mechanisms for min/low/high/max. 

Commits 11-13 further enforce the migration (promotion / demotion) paths
and ensure that we do not introduce unnecessary churn by reclaiming
earlier. 

Commit 14 steers allocations so that we do not accidentally introduce
zone_reclaim_mode-like behavior and immediately reclaim after allocating
on exhausted memcg tiers.

FUTURE WORK
===========
I am currently working on another series [1], which pushes the memcg
cached charges (stock) to the page_counter level, meaning each tier can
manage its own independent stock. This should increase the performance
and simplicity of the code much more.

CHANGELOG
=========
v2 --> v3
- N tier support, instead of toptier vs. rest enforcement
- Max enforcement
- Migration enforcement
- Page allocator steering for exhausted tiers
- Promotion fastpath restriction & throttling

[1] https://lore.kernel.org/all/20260623180124.868655-1-joshua.hahnjy@gmail.com/

Joshua Hahn (14):
  mm/memcontrol: Introduce cgroup.memory=tiered_limits boot parameter
  mm/memcontrol: Refactor page_counter charging in try_charge_memcg
  mm/memory-tiers: Introduce a mapping from nid to tier_slot
  mm/memcontrol: Allocate per-tier page_counters
  mm/memcontrol: Set tier limits proportional to memory limits
  mm/vmscan, memcontrol: Add nodemask to try_to_free_mem_cgroup_pages
  mm/memcontrol: Charge/uncharge tiered memory to mem_cgroup
  mm/memcontrol: Make memory.low and memory.min tier-aware
  mm/memcontrol: Make memory.high tier-aware
  mm/memcontrol: Make memory.max tier-aware
  mm/memcontrol, migrate: Transfer tier charge on migration
  mm/memcontrol: Kick async reclaim on migration and folio replacement
  mm/memcontrol, sched/numa: Gate NUMA promotions into memcg tiers
  mm/page_alloc: steer allocations away from exhausted memory tiers

 include/linux/memcontrol.h   |  75 ++++-
 include/linux/memory-tiers.h |  24 ++
 kernel/sched/fair.c          |   3 +-
 mm/internal.h                |   3 +-
 mm/memcontrol-v1.c           |   5 +-
 mm/memcontrol.c              | 528 ++++++++++++++++++++++++++++++++---
 mm/memory-tiers.c            | 110 +++++++-
 mm/migrate.c                 |  30 +-
 mm/page_alloc.c              |  19 +-
 mm/vmscan.c                  |  25 +-
 10 files changed, 753 insertions(+), 69 deletions(-)


base-commit: 7b25c83e4711038989b5b08a8977fb68468c854e
-- 
2.53.0-Meta

[RFC PATCH v3 01/14] mm/memcontrol: Introduce cgroup.memory=tiered_limits boot parameter

[Joshua Hahn]

Introduce a "tiered_limits" option for the cgroup.memory= kernel
commandline parameter to enable tier-proportional scaling and
enforcing of the memory cgroup controller limits
memory.{min, low, high}. Since mem_cgroup_tiered_limits() will
become a hotpath in the later commits to gate charging, demotion,
and promotion decisions, use a static key so that cgroups not using
tier-aware-memcg limits has minimal overhead.

Enable it by adding to the kernel command line:
	cgroup.memory=tiered_limits

The option is boot-time only, since flipping the bit at runtime could
leave charges uncharged in the future, or uncharges for folios that
were never charged.

This feature is incompatible with cgroup v1, and wil raise a single
warning statement if a system booted with tiered limits mounts a
legacy cgroup:

[XXX] cgroup.memory=tiered_limits should not be enabled with cgroupv1

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h | 18 ++++++++++++++++++
 mm/memcontrol.c            | 19 +++++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 2118d5b33d051..dce03df7eae05 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -530,6 +530,19 @@ static inline bool mem_cgroup_disabled(void)
 	return !cgroup_subsys_enabled(memory_cgrp_subsys);
 }
 
+#ifdef CONFIG_NUMA
+DECLARE_STATIC_KEY_FALSE(memcg_tiered_limits_key);
+static inline bool mem_cgroup_tiered_limits(void)
+{
+	return static_branch_unlikely(&memcg_tiered_limits_key);
+}
+#else
+static inline bool mem_cgroup_tiered_limits(void)
+{
+	return false;
+}
+#endif
+
 static inline void mem_cgroup_protection(struct mem_cgroup *root,
 					 struct mem_cgroup *memcg,
 					 unsigned long *min,
@@ -1083,6 +1096,11 @@ static inline bool mem_cgroup_disabled(void)
 	return true;
 }
 
+static inline bool mem_cgroup_tiered_limits(void)
+{
+	return false;
+}
+
 static inline void memcg_memory_event(struct mem_cgroup *memcg,
 				      enum memcg_memory_event event)
 {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 29330f5f9d4eb..cefe33b5fd285 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -320,6 +320,13 @@ EXPORT_SYMBOL(memcg_kmem_online_key);
 DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key);
 EXPORT_SYMBOL(memcg_bpf_enabled_key);
 
+#ifdef CONFIG_NUMA
+DEFINE_STATIC_KEY_FALSE(memcg_tiered_limits_key);
+
+/* Tier-proportional scaling of memory controller limits enabled? */
+static bool cgroup_memory_tiered_limits __ro_after_init;
+#endif
+
 /**
  * get_mem_cgroup_css_from_folio - acquire a css of the memcg associated with a folio
  * @folio: folio of interest
@@ -4202,6 +4209,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 	struct mem_cgroup *memcg, *old_memcg;
 	bool memcg_on_dfl = cgroup_subsys_on_dfl(memory_cgrp_subsys);
 
+	if (mem_cgroup_tiered_limits() && !memcg_on_dfl)
+		pr_warn_once("cgroup.memory=tiered_limits should not be enabled with cgroupv1\n");
+
 	old_memcg = set_active_memcg(parent);
 	memcg = mem_cgroup_alloc(parent);
 	set_active_memcg(old_memcg);
@@ -5584,6 +5594,10 @@ static int __init cgroup_memory(char *s)
 			cgroup_memory_nokmem = true;
 		if (!strcmp(token, "nobpf"))
 			cgroup_memory_nobpf = true;
+#ifdef CONFIG_NUMA
+		if (!strcmp(token, "tiered_limits"))
+			cgroup_memory_tiered_limits = true;
+#endif
 	}
 	return 1;
 }
@@ -5630,6 +5644,11 @@ int __init mem_cgroup_init(void)
 	memcg_pn_cachep = KMEM_CACHE(mem_cgroup_per_node,
 				     SLAB_PANIC | SLAB_HWCACHE_ALIGN);
 
+#ifdef CONFIG_NUMA
+	if (cgroup_memory_tiered_limits)
+		static_branch_enable(&memcg_tiered_limits_key);
+#endif
+
 	return 0;
 }
 
-- 
2.53.0-Meta

[RFC PATCH v3 02/14] mm/memcontrol: Refactor page_counter charging in try_charge_memcg

[Joshua Hahn]

In preparation for adding charging and uncharging of a new page_counter
toptier to try_charge_memcg, refactor the code so that it is easier to
distinguish between the memcg v1/v2 cases.

No functional changes intended.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/memcontrol.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index cefe33b5fd285..ec28512de6a23 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2668,18 +2668,22 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		batch = nr_pages;
 
 	reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
-	if (!do_memsw_account() ||
-	    page_counter_try_charge(&memcg->memsw, batch, &counter)) {
-		if (page_counter_try_charge(&memcg->memory, batch, &counter))
-			goto done_restock;
-		if (do_memsw_account())
-			page_counter_uncharge(&memcg->memsw, batch);
-		mem_over_limit = mem_cgroup_from_counter(counter, memory);
-	} else {
+
+	if (do_memsw_account() &&
+	    !page_counter_try_charge(&memcg->memsw, batch, &counter)) {
 		mem_over_limit = mem_cgroup_from_counter(counter, memsw);
 		reclaim_options &= ~MEMCG_RECLAIM_MAY_SWAP;
+		goto reclaim;
 	}
 
+	if (page_counter_try_charge(&memcg->memory, batch, &counter))
+		goto done_restock;
+
+	if (do_memsw_account())
+		page_counter_uncharge(&memcg->memsw, batch);
+	mem_over_limit = mem_cgroup_from_counter(counter, memory);
+
+reclaim:
 	if (batch > nr_pages) {
 		batch = nr_pages;
 		goto retry;
-- 
2.53.0-Meta

[RFC PATCH v3 03/14] mm/memory-tiers: Introduce a mapping from nid to tier_slot

[Joshua Hahn]

Establishing tiered memcg limits will require an ordering of tiers, as
well as a way to account how much memory is present in each tier.
This will need to be done starting at boot, so that all memory becomes
properly accounted.

However, tiers can come online and offline at runtime due to DAX memory
whose nodes can be hotplugged / hot-unplugged, and these nodes' tiers
are not available at boot.

Therefore, to establish a fixed mapping from nid to tier that isn't
sparse like the tier_ids, introduce a new "tier_slot" which is a dense
index that does not change once a tier comes online.

Also introduce a helper to retrieve the nodemask associated with a tier.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memory-tiers.h | 18 ++++++++
 mm/memory-tiers.c            | 86 +++++++++++++++++++++++++++++++++++-
 2 files changed, 102 insertions(+), 2 deletions(-)

diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 7999c58629eeb..0e49645cdd1a9 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -41,6 +41,8 @@ extern struct memory_dev_type *default_dram_type;
 extern nodemask_t default_dram_nodes;
 struct memory_dev_type *alloc_memory_type(int adistance);
 void put_memory_type(struct memory_dev_type *memtype);
+int mt_nr_tier_slots(void);
+int nid_tier_slot(int nid);
 void init_node_memory_type(int node, struct memory_dev_type *default_type);
 void clear_node_memory_type(int node, struct memory_dev_type *memtype);
 int register_mt_adistance_algorithm(struct notifier_block *nb);
@@ -52,6 +54,7 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist);
 struct memory_dev_type *mt_find_alloc_memory_type(int adist,
 						  struct list_head *memory_types);
 void mt_put_memory_types(struct list_head *memory_types);
+const nodemask_t *mt_tier_nodes(int slot);
 #ifdef CONFIG_NUMA_MIGRATION
 int next_demotion_node(int node, const nodemask_t *allowed_mask);
 void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
@@ -151,5 +154,20 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist,
 static inline void mt_put_memory_types(struct list_head *memory_types)
 {
 }
+
+static inline int mt_nr_tier_slots(void)
+{
+	return 0;
+}
+
+static inline int nid_tier_slot(int nid)
+{
+	return -1;
+}
+
+static inline const nodemask_t *mt_tier_nodes(int slot)
+{
+	return NULL;
+}
 #endif	/* CONFIG_NUMA */
 #endif  /* _LINUX_MEMORY_TIERS_H */
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b0..36187c0ea9ded 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -43,6 +43,16 @@ static LIST_HEAD(memory_tiers);
  */
 static LIST_HEAD(default_memory_types);
 static struct node_memory_type_map node_memory_types[MAX_NUMNODES];
+
+/*
+ * nr_tier_slots and tier_slot_ids are written with memory_tier_lock and
+ * read locklessly. nr_tier_slots is monotonically increasing.
+ */
+static int nr_tier_slots;
+static int tier_slot_ids[MAX_NUMNODES]   = {[0 ... MAX_NUMNODES - 1] = -1,};
+static int node_tier_slots[MAX_NUMNODES] = {[0 ... MAX_NUMNODES - 1] = -1,};
+static nodemask_t tier_nodemasks[MAX_NUMNODES];
+
 struct memory_dev_type *default_dram_type;
 nodemask_t default_dram_nodes __initdata = NODE_MASK_NONE;
 
@@ -273,6 +283,65 @@ static struct memory_tier *__node_get_memory_tier(int node)
 				     lockdep_is_held(&memory_tier_lock));
 }
 
+/* Caller must hold memory_tier_lock */
+static int tier_id_slot(int tier_id)
+{
+	int slot, free_slot = -1;
+
+	for (slot = 0; slot < nr_node_ids; slot++) {
+		if (tier_slot_ids[slot] == tier_id)
+			return slot;
+		if (tier_slot_ids[slot] == -1 && free_slot == -1) {
+			free_slot = slot;
+			tier_slot_ids[slot] = tier_id;
+		}
+	}
+
+	return free_slot;
+}
+
+static void establish_tier_slots(void)
+{
+	int old_nr_tier_slots = mt_nr_tier_slots();
+	int highest_slot = old_nr_tier_slots;
+
+	lockdep_assert_held_once(&memory_tier_lock);
+
+	for (int slot = 0; slot < old_nr_tier_slots; slot++)
+		nodes_clear(tier_nodemasks[slot]);
+
+	for (int nid = 0; nid < nr_node_ids; nid++) {
+		struct memory_tier *memtier = NULL;
+		int slot = -1;
+
+		if (node_state(nid, N_MEMORY))
+			memtier = __node_get_memory_tier(nid);
+		if (memtier) {
+			slot = tier_id_slot(memtier->dev.id);
+			highest_slot = max(highest_slot, slot + 1);
+		}
+
+		WRITE_ONCE(node_tier_slots[nid], slot);
+
+		if (slot != -1)
+			node_set(nid, tier_nodemasks[slot]);
+	}
+	WRITE_ONCE(nr_tier_slots, highest_slot);
+}
+
+int mt_nr_tier_slots(void)
+{
+	return READ_ONCE(nr_tier_slots);
+}
+
+int nid_tier_slot(int nid)
+{
+	if (nid < 0 || nid >= MAX_NUMNODES)
+		return -1;
+
+	return READ_ONCE(node_tier_slots[nid]);
+}
+
 #ifdef CONFIG_NUMA_MIGRATION
 bool node_is_toptier(int node)
 {
@@ -729,6 +798,7 @@ static int __init memory_tier_late_init(void)
 	}
 
 	establish_demotion_targets();
+	establish_tier_slots();
 	put_online_mems();
 
 	return 0;
@@ -878,6 +948,14 @@ int mt_calc_adistance(int node, int *adist)
 }
 EXPORT_SYMBOL_GPL(mt_calc_adistance);
 
+const nodemask_t *mt_tier_nodes(int slot)
+{
+	if (slot < 0)
+		return NULL;
+
+	return &tier_nodemasks[slot];
+}
+
 static int __meminit memtier_hotplug_callback(struct notifier_block *self,
 					      unsigned long action, void *_arg)
 {
@@ -887,15 +965,19 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self,
 	switch (action) {
 	case NODE_REMOVED_LAST_MEMORY:
 		mutex_lock(&memory_tier_lock);
-		if (clear_node_memory_tier(nn->nid))
+		if (clear_node_memory_tier(nn->nid)) {
 			establish_demotion_targets();
+			establish_tier_slots();
+		}
 		mutex_unlock(&memory_tier_lock);
 		break;
 	case NODE_ADDED_FIRST_MEMORY:
 		mutex_lock(&memory_tier_lock);
 		memtier = set_node_memory_tier(nn->nid);
-		if (!IS_ERR(memtier))
+		if (!IS_ERR(memtier)) {
 			establish_demotion_targets();
+			establish_tier_slots();
+		}
 		mutex_unlock(&memory_tier_lock);
 		break;
 	}
-- 
2.53.0-Meta

[RFC PATCH v3 04/14] mm/memcontrol: Allocate per-tier page_counters

[Joshua Hahn]

Tier-aware limits need one page_counter per memory tier.  Add a
page_counter array (memcg->tier) to struct mem_cgroup, sized to
nr_node_ids (an upper bound on the number of tiers) and allocated only
when tiered limits are enabled, so memcgs pay nothing when the feature is
off.  Initialise and parent-link every slot in mem_cgroup_css_alloc(),
and free the array in __mem_cgroup_free().

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h |  1 +
 mm/memcontrol.c            | 23 +++++++++++++++++++++++
 mm/memory-tiers.c          | 17 ++++++++++++++---
 3 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dce03df7eae05..bb5bde87ac85a 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -207,6 +207,7 @@ struct mem_cgroup {
 
 	/* Accounted resources */
 	struct page_counter memory;		/* Both v1 & v2 */
+	struct page_counter *tier;		/* v2 only */
 
 	union {
 		struct page_counter swap;	/* v2 only */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index ec28512de6a23..d096010366515 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -53,6 +53,7 @@
 #include <linux/seq_file.h>
 #include <linux/vmpressure.h>
 #include <linux/memremap.h>
+#include <linux/memory-tiers.h>
 #include <linux/mm_inline.h>
 #include <linux/cpu.h>
 #include <linux/oom.h>
@@ -4126,6 +4127,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)
 	memcg1_free_events(memcg);
 	kfree(memcg->vmstats);
 	free_percpu(memcg->vmstats_percpu);
+	kfree(memcg->tier);
 	kfree(memcg);
 }
 
@@ -4178,6 +4180,13 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)
 		if (!alloc_mem_cgroup_per_node_info(memcg, node))
 			goto fail;
 
+	if (mem_cgroup_tiered_limits()) {
+		memcg->tier = kcalloc(nr_node_ids, sizeof(*memcg->tier),
+				      GFP_KERNEL);
+		if (!memcg->tier)
+			goto fail;
+	}
+
 	if (memcg_wb_domain_init(memcg, GFP_KERNEL))
 		goto fail;
 
@@ -4206,6 +4215,16 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)
 	return ERR_PTR(error);
 }
 
+static void memcg_init_tier_counters(struct mem_cgroup *memcg,
+				     struct mem_cgroup *parent, bool protection)
+{
+	for (int i = 0; i < nr_node_ids; i++) {
+		page_counter_init(&memcg->tier[i],
+				  parent ? &parent->tier[i] : NULL, protection);
+		page_counter_set_high(&memcg->tier[i], PAGE_COUNTER_MAX);
+	}
+}
+
 static struct cgroup_subsys_state * __ref
 mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 {
@@ -4231,6 +4250,8 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
 	if (parent) {
 		page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl);
+		if (mem_cgroup_tiered_limits())
+			memcg_init_tier_counters(memcg, parent, memcg_on_dfl);
 		page_counter_init(&memcg->swap, &parent->swap, false);
 #ifdef CONFIG_MEMCG_V1
 		WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));
@@ -4243,6 +4264,8 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 		init_memcg_stats();
 		init_memcg_events();
 		page_counter_init(&memcg->memory, NULL, true);
+		if (mem_cgroup_tiered_limits())
+			memcg_init_tier_counters(memcg, NULL, true);
 		page_counter_init(&memcg->swap, NULL, false);
 #ifdef CONFIG_MEMCG_V1
 		page_counter_init(&memcg->kmem, NULL, false);
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 36187c0ea9ded..bd5c78cd26ec4 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -52,6 +52,7 @@ static int nr_tier_slots;
 static int tier_slot_ids[MAX_NUMNODES]   = {[0 ... MAX_NUMNODES - 1] = -1,};
 static int node_tier_slots[MAX_NUMNODES] = {[0 ... MAX_NUMNODES - 1] = -1,};
 static nodemask_t tier_nodemasks[MAX_NUMNODES];
+static unsigned long tier_capacity[MAX_NUMNODES];
 
 struct memory_dev_type *default_dram_type;
 nodemask_t default_dram_nodes __initdata = NODE_MASK_NONE;
@@ -307,8 +308,10 @@ static void establish_tier_slots(void)
 
 	lockdep_assert_held_once(&memory_tier_lock);
 
-	for (int slot = 0; slot < old_nr_tier_slots; slot++)
+	for (int slot = 0; slot < old_nr_tier_slots; slot++) {
 		nodes_clear(tier_nodemasks[slot]);
+		WRITE_ONCE(tier_capacity[slot], 0);
+	}
 
 	for (int nid = 0; nid < nr_node_ids; nid++) {
 		struct memory_tier *memtier = NULL;
@@ -323,8 +326,16 @@ static void establish_tier_slots(void)
 
 		WRITE_ONCE(node_tier_slots[nid], slot);
 
-		if (slot != -1)
-			node_set(nid, tier_nodemasks[slot]);
+		if (slot < 0)
+			continue;
+
+		node_set(nid, tier_nodemasks[slot]);
+		for (int i = 0; i < MAX_NR_ZONES; i++) {
+			struct zone *zone = &NODE_DATA(nid)->node_zones[i];
+
+			WRITE_ONCE(tier_capacity[slot],
+				tier_capacity[slot] + zone_managed_pages(zone));
+		}
 	}
 	WRITE_ONCE(nr_tier_slots, highest_slot);
 }
-- 
2.53.0-Meta

[RFC PATCH v3 05/14] mm/memcontrol: Set tier limits proportional to memory limits

[Joshua Hahn]

Compute proportional per-tier limits based on memory limits when
users write to memory limit sysfs files, or when memory hotplug causes
tier proportions to be shifted.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h   | 10 +++++++
 include/linux/memory-tiers.h |  6 ++++
 mm/memcontrol.c              | 54 ++++++++++++++++++++++++++++++++++++
 mm/memory-tiers.c            | 13 +++++++++
 4 files changed, 83 insertions(+)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index bb5bde87ac85a..f7a92b66330ec 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -537,11 +537,17 @@ static inline bool mem_cgroup_tiered_limits(void)
 {
 	return static_branch_unlikely(&memcg_tiered_limits_key);
 }
+
+void establish_memcg_tier_limits(void);
 #else
 static inline bool mem_cgroup_tiered_limits(void)
 {
 	return false;
 }
+
+static inline void establish_memcg_tier_limits(void)
+{
+}
 #endif
 
 static inline void mem_cgroup_protection(struct mem_cgroup *root,
@@ -1102,6 +1108,10 @@ static inline bool mem_cgroup_tiered_limits(void)
 	return false;
 }
 
+static inline void establish_memcg_tier_limits(void)
+{
+}
+
 static inline void memcg_memory_event(struct mem_cgroup *memcg,
 				      enum memcg_memory_event event)
 {
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 0e49645cdd1a9..04b396f60b457 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -55,6 +55,7 @@ struct memory_dev_type *mt_find_alloc_memory_type(int adist,
 						  struct list_head *memory_types);
 void mt_put_memory_types(struct list_head *memory_types);
 const nodemask_t *mt_tier_nodes(int slot);
+unsigned long mt_scale_by_tier(unsigned long val, int slot);
 #ifdef CONFIG_NUMA_MIGRATION
 int next_demotion_node(int node, const nodemask_t *allowed_mask);
 void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
@@ -169,5 +170,10 @@ static inline const nodemask_t *mt_tier_nodes(int slot)
 {
 	return NULL;
 }
+
+static inline unsigned long mt_scale_by_tier(unsigned long val, int slot)
+{
+	return val;
+}
 #endif	/* CONFIG_NUMA */
 #endif  /* _LINUX_MEMORY_TIERS_H */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d096010366515..defd04acfb3fd 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4421,6 +4421,35 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css)
 	mem_cgroup_free(memcg);
 }
 
+static inline unsigned long page_counter_max_or_scale(unsigned long val,
+						      int slot)
+{
+	return val == PAGE_COUNTER_MAX ? PAGE_COUNTER_MAX :
+					 mt_scale_by_tier(val, slot);
+}
+
+static void memcg_scale_tier_limits(struct mem_cgroup *memcg)
+{
+	unsigned long min = READ_ONCE(memcg->memory.min);
+	unsigned long low = READ_ONCE(memcg->memory.low);
+	unsigned long high = READ_ONCE(memcg->memory.high);
+	unsigned long max = READ_ONCE(memcg->memory.max);
+	int nr_tier_slots = mt_nr_tier_slots();
+
+	for (int slot = 0; slot < nr_tier_slots; slot++) {
+		unsigned long new_min = page_counter_max_or_scale(min, slot);
+		unsigned long new_low = page_counter_max_or_scale(low, slot);
+		unsigned long new_high = page_counter_max_or_scale(high, slot);
+		unsigned long new_max = page_counter_max_or_scale(max, slot);
+		struct page_counter *tier = &memcg->tier[slot];
+
+		page_counter_set_min(tier, new_min);
+		page_counter_set_low(tier, new_low);
+		page_counter_set_high(tier, new_high);
+		xchg(&tier->max, new_max);
+	}
+}
+
 /**
  * mem_cgroup_css_reset - reset the states of a mem_cgroup
  * @css: the target css
@@ -4454,6 +4483,8 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css)
 	page_counter_set_high(&memcg->memory, PAGE_COUNTER_MAX);
 	memcg1_soft_limit_reset(memcg);
 	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
+	if (mem_cgroup_tiered_limits())
+		memcg_scale_tier_limits(memcg);
 	memcg_wb_domain_size_changed(memcg);
 }
 
@@ -4797,6 +4828,21 @@ static ssize_t memory_peak_write(struct kernfs_open_file *of, char *buf,
 			  &memcg->memory_peaks);
 }
 
+#ifdef CONFIG_NUMA
+void establish_memcg_tier_limits(void)
+{
+	struct mem_cgroup *memcg;
+
+	if (!mem_cgroup_tiered_limits())
+		return;
+
+	for_each_mem_cgroup_tree(memcg, NULL) {
+		if (memcg != root_mem_cgroup)
+			memcg_scale_tier_limits(memcg);
+	}
+}
+#endif
+
 #undef OFP_PEAK_UNSET
 
 static int memory_min_show(struct seq_file *m, void *v)
@@ -4818,6 +4864,8 @@ static ssize_t memory_min_write(struct kernfs_open_file *of,
 		return err;
 
 	page_counter_set_min(&memcg->memory, min);
+	if (mem_cgroup_tiered_limits())
+		memcg_scale_tier_limits(memcg);
 
 	return nbytes;
 }
@@ -4841,6 +4889,8 @@ static ssize_t memory_low_write(struct kernfs_open_file *of,
 		return err;
 
 	page_counter_set_low(&memcg->memory, low);
+	if (mem_cgroup_tiered_limits())
+		memcg_scale_tier_limits(memcg);
 
 	return nbytes;
 }
@@ -4866,6 +4916,8 @@ static ssize_t memory_high_write(struct kernfs_open_file *of,
 		return err;
 
 	page_counter_set_high(&memcg->memory, high);
+	if (mem_cgroup_tiered_limits())
+		memcg_scale_tier_limits(memcg);
 
 	if (of->file->f_flags & O_NONBLOCK)
 		goto out;
@@ -4925,6 +4977,8 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
 		return err;
 
 	xchg(&memcg->memory.max, max);
+	if (mem_cgroup_tiered_limits())
+		memcg_scale_tier_limits(memcg);
 
 	if (of->file->f_flags & O_NONBLOCK)
 		goto out;
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index bd5c78cd26ec4..e2c99f51c36d1 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -810,6 +810,7 @@ static int __init memory_tier_late_init(void)
 
 	establish_demotion_targets();
 	establish_tier_slots();
+	establish_memcg_tier_limits();
 	put_online_mems();
 
 	return 0;
@@ -967,6 +968,16 @@ const nodemask_t *mt_tier_nodes(int slot)
 	return &tier_nodemasks[slot];
 }
 
+unsigned long mt_scale_by_tier(unsigned long val, int slot)
+{
+	unsigned long total_capacity = totalram_pages();
+
+	if (slot < 0 || !total_capacity)
+		return 0;
+
+	return mult_frac(val, READ_ONCE(tier_capacity[slot]), total_capacity);
+}
+
 static int __meminit memtier_hotplug_callback(struct notifier_block *self,
 					      unsigned long action, void *_arg)
 {
@@ -979,6 +990,7 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self,
 		if (clear_node_memory_tier(nn->nid)) {
 			establish_demotion_targets();
 			establish_tier_slots();
+			establish_memcg_tier_limits();
 		}
 		mutex_unlock(&memory_tier_lock);
 		break;
@@ -988,6 +1000,7 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self,
 		if (!IS_ERR(memtier)) {
 			establish_demotion_targets();
 			establish_tier_slots();
+			establish_memcg_tier_limits();
 		}
 		mutex_unlock(&memory_tier_lock);
 		break;
-- 
2.53.0-Meta

[RFC PATCH v3 06/14] mm/vmscan, memcontrol: Add nodemask to try_to_free_mem_cgroup_pages

[Joshua Hahn]

Add a new nodemask parameter to try_to_free_mem_cgroup_pages to allow
selective reclaim on certain nodes. This new function signature can be
used in future patches to selectively perform reclaim on toptier and
place downward pressure when toptier limits are breached but memcg-wide
limits are not yet breached.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/internal.h      |  3 ++-
 mm/memcontrol-v1.c |  5 +++--
 mm/memcontrol.c    | 11 +++++++----
 mm/vmscan.c        | 10 +++++++---
 4 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index f47f06c555481..5f2f6a1757bd9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -84,7 +84,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 					   unsigned long nr_pages,
 					   gfp_t gfp_mask,
 					   unsigned int reclaim_options,
-					   int *swappiness);
+					   int *swappiness,
+					   const nodemask_t *allowed);
 unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg,
 				     gfp_t gfp_mask, bool noswap,
 				     pg_data_t *pgdat,
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 835fc8e511844..688896b59efe4 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -1815,7 +1815,8 @@ static int mem_cgroup_resize_max(struct mem_cgroup *memcg,
 		}
 
 		if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL,
-				memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP, NULL)) {
+				memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP,
+				NULL, NULL)) {
 			ret = -EBUSY;
 			break;
 		}
@@ -1851,7 +1852,7 @@ static int mem_cgroup_force_empty(struct mem_cgroup *memcg)
 			break;
 
 		if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL,
-						  MEMCG_RECLAIM_MAY_SWAP, NULL))
+					MEMCG_RECLAIM_MAY_SWAP, NULL, NULL))
 			nr_retries--;
 	}
 
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index defd04acfb3fd..6c67d9d2c9ac7 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2397,7 +2397,7 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg,
 		nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages,
 							gfp_mask,
 							MEMCG_RECLAIM_MAY_SWAP,
-							NULL);
+							NULL, NULL);
 		psi_memstall_leave(&pflags);
 	} while ((memcg = parent_mem_cgroup(memcg)) &&
 		 !mem_cgroup_is_root(memcg));
@@ -2710,7 +2710,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 
 	psi_memstall_enter(&pflags);
 	nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
-						    gfp_mask, reclaim_options, NULL);
+						    gfp_mask, reclaim_options,
+						    NULL, NULL);
 	psi_memstall_leave(&pflags);
 
 	if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
@@ -4946,7 +4947,8 @@ static ssize_t memory_high_write(struct kernfs_open_file *of,
 		}
 
 		reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages - high,
-					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP, NULL);
+					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP,
+					NULL, NULL);
 
 		if (!reclaimed && !nr_retries--)
 			break;
@@ -5007,7 +5009,8 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
 
 		if (nr_reclaims) {
 			if (!try_to_free_mem_cgroup_pages(memcg, nr_pages - max,
-					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP, NULL))
+					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP,
+					NULL, NULL))
 				nr_reclaims--;
 			continue;
 		}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 17d2b793cbfc4..ffe7ea3c5aff6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6859,7 +6859,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 					   unsigned long nr_pages,
 					   gfp_t gfp_mask,
 					   unsigned int reclaim_options,
-					   int *swappiness)
+					   int *swappiness,
+					   const nodemask_t *allowed)
 {
 	unsigned long nr_reclaimed;
 	unsigned int noreclaim_flag;
@@ -6875,6 +6876,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 		.may_unmap = 1,
 		.may_swap = !!(reclaim_options & MEMCG_RECLAIM_MAY_SWAP),
 		.proactive = !!(reclaim_options & MEMCG_RECLAIM_PROACTIVE),
+		.nodemask = allowed,
 	};
 	/*
 	 * Traverse the ZONELIST_FALLBACK zonelist of the current node to put
@@ -6900,7 +6902,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 					   unsigned long nr_pages,
 					   gfp_t gfp_mask,
 					   unsigned int reclaim_options,
-					   int *swappiness)
+					   int *swappiness,
+					   const nodemask_t *allowed)
 {
 	return 0;
 }
@@ -8033,7 +8036,8 @@ int user_proactive_reclaim(char *buf,
 			reclaimed = try_to_free_mem_cgroup_pages(memcg,
 						 batch_size, gfp_mask,
 						 reclaim_options,
-						 swappiness == -1 ? NULL : &swappiness);
+						 swappiness == -1 ? NULL : &swappiness,
+						 NULL);
 		} else {
 			struct scan_control sc = {
 				.gfp_mask = current_gfp_context(gfp_mask),
-- 
2.53.0-Meta

[RFC PATCH v3 07/14] mm/memcontrol: Charge/uncharge tiered memory to mem_cgroup

[Joshua Hahn]

Memory cgroup limits isolate memory as a resource, but treat all
memory as equally valuable regardless of which memory tier it
resides in.

Account tiered memory usage in parallel with existing memory
accounting. Add an nid parameter to try_charge_memcg(); callers
resolve it to a tier slot with nid_tier_slot() and charge
memcg->tier[slot] alongside memcg->memory.

Uncharging reuses uncharge_gather to batch. Because the high-volume
free path reclaims per-node, a batch is normally single-tier; if a
folio in a different tier appears mid-batch, flush the accumulated
uncharge and start accumulating for the new tier. Folios on
different nodes within the same tier do not force a flush.

Also, mem_cgroup_migrate() and mem_cgroup_replace_folio() now move the
tier charge when the folio changes tier.

Currently this only tracks LRU folios (try_charge_memcg() callers
from charge_memcg()). The other two sites, obj_cgroup_charge_pages()
and mem_cgroup_sk_charge(), will be handled by a future series that
transitions enum memcg_stat_item to a per-lruvec counter
(enum node_stat_item).

The per-tier limits computed in the previous patch are not consulted
yet, this patch only acounts the memory. Enforcement will come in the
following patches.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/memcontrol.c | 116 ++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 108 insertions(+), 8 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6c67d9d2c9ac7..f3714dfd85aa0 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1561,6 +1561,15 @@ void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
 		*lru_size += nr_pages;
 }
 
+static struct page_counter *mem_cgroup_tier_counter(struct mem_cgroup *memcg,
+						    int slot)
+{
+	if (slot < 0)
+		return NULL;
+
+	return &memcg->tier[slot];
+}
+
 /**
  * mem_cgroup_margin - calculate chargeable space of a memory cgroup
  * @memcg: the memory cgroup
@@ -2645,13 +2654,32 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
 	css_put(&memcg->css);
 }
 
+static void mem_cgroup_uncharge_tier(struct mem_cgroup *memcg,
+				     int slot, unsigned int nr_pages)
+{
+	struct page_counter *tier_counter = mem_cgroup_tier_counter(memcg, slot);
+
+	if (tier_counter)
+		page_counter_uncharge(tier_counter, nr_pages);
+}
+
+static void mem_cgroup_charge_tier(struct mem_cgroup *memcg,
+				   int slot, unsigned int nr_pages)
+{
+	struct page_counter *tier_counter = mem_cgroup_tier_counter(memcg, slot);
+
+	if (tier_counter)
+		page_counter_charge(tier_counter, nr_pages);
+}
+
 static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
-			    unsigned int nr_pages)
+			    unsigned int nr_pages, int nid)
 {
 	unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
 	int nr_retries = MAX_RECLAIM_RETRIES;
 	struct mem_cgroup *mem_over_limit;
 	struct page_counter *counter;
+	struct page_counter *tier_counter = NULL;
 	unsigned long nr_reclaimed;
 	bool passed_oom = false;
 	unsigned int reclaim_options;
@@ -2659,10 +2687,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	bool raised_max_event = false;
 	unsigned long pflags;
 	bool allow_spinning = gfpflags_allow_spinning(gfp_mask);
+	int slot = -1;
+
+	if (mem_cgroup_tiered_limits()) {
+		slot = nid_tier_slot(nid);
+		tier_counter = mem_cgroup_tier_counter(memcg, slot);
+	}
 
 retry:
-	if (consume_stock(memcg, nr_pages))
+	if (consume_stock(memcg, nr_pages)) {
+		if (tier_counter)
+			page_counter_charge(tier_counter, nr_pages);
 		return 0;
+	}
 
 	if (!allow_spinning)
 		/* Avoid the refill and flush of the older stock */
@@ -2677,8 +2714,11 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		goto reclaim;
 	}
 
-	if (page_counter_try_charge(&memcg->memory, batch, &counter))
+	if (page_counter_try_charge(&memcg->memory, batch, &counter)) {
+		if (tier_counter)
+			page_counter_charge(tier_counter, nr_pages);
 		goto done_restock;
+	}
 
 	if (do_memsw_account())
 		page_counter_uncharge(&memcg->memsw, batch);
@@ -2781,6 +2821,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 * temporarily by force charging it.
 	 */
 	page_counter_charge(&memcg->memory, nr_pages);
+	if (tier_counter)
+		page_counter_charge(tier_counter, nr_pages);
 	if (do_memsw_account())
 		page_counter_charge(&memcg->memsw, nr_pages);
 
@@ -2852,7 +2894,7 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (mem_cgroup_is_root(memcg))
 		return 0;
 
-	return try_charge_memcg(memcg, gfp_mask, nr_pages);
+	return try_charge_memcg(memcg, gfp_mask, nr_pages, NUMA_NO_NODE);
 }
 
 static void commit_charge(struct folio *folio, struct obj_cgroup *objcg)
@@ -3152,7 +3194,7 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,
 
 	memcg = get_mem_cgroup_from_objcg(objcg);
 
-	ret = try_charge_memcg(memcg, gfp, nr_pages);
+	ret = try_charge_memcg(memcg, gfp, nr_pages, NUMA_NO_NODE);
 	if (ret)
 		goto out;
 
@@ -5280,7 +5322,8 @@ static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,
 	objcg = get_obj_cgroup_from_memcg(memcg);
 	/* Do not account at the root objcg level. */
 	if (!obj_cgroup_is_root(objcg))
-		ret = try_charge_memcg(memcg, gfp, folio_nr_pages(folio));
+		ret = try_charge_memcg(memcg, gfp, folio_nr_pages(folio),
+				       folio_nid(folio));
 	if (ret) {
 		obj_cgroup_put(objcg);
 		return ret;
@@ -5376,6 +5419,8 @@ struct uncharge_gather {
 	unsigned long pgpgout;
 	unsigned long nr_kmem;
 	int nid;
+	int tier_slot;
+	unsigned long tier_nr;
 };
 
 static inline void uncharge_gather_clear(struct uncharge_gather *ug)
@@ -5383,6 +5428,34 @@ static inline void uncharge_gather_clear(struct uncharge_gather *ug)
 	memset(ug, 0, sizeof(*ug));
 }
 
+static void flush_tier_charge(struct mem_cgroup *memcg,
+			      const struct uncharge_gather *ug)
+{
+	struct page_counter *tier_counter;
+
+	tier_counter = &memcg->tier[ug->tier_slot];
+	page_counter_uncharge(tier_counter, ug->tier_nr);
+}
+
+static void gather_tier_charge(struct uncharge_gather *ug, struct folio *folio,
+			       unsigned long nr_pages)
+{
+	int slot = nid_tier_slot(folio_nid(folio));
+
+	if (slot < 0)
+		return;
+
+	if (ug->tier_nr && slot != ug->tier_slot) {
+		rcu_read_lock();
+		flush_tier_charge(obj_cgroup_memcg(ug->objcg), ug);
+		rcu_read_unlock();
+		ug->tier_nr = 0;
+	}
+
+	ug->tier_slot = slot;
+	ug->tier_nr += nr_pages;
+}
+
 static void uncharge_batch(const struct uncharge_gather *ug)
 {
 	struct mem_cgroup *memcg;
@@ -5395,6 +5468,8 @@ static void uncharge_batch(const struct uncharge_gather *ug)
 			mod_memcg_state(memcg, MEMCG_KMEM, -ug->nr_kmem);
 			memcg1_account_kmem(memcg, -ug->nr_kmem);
 		}
+		if (ug->tier_nr)
+			flush_tier_charge(memcg, ug);
 		memcg1_oom_recover(memcg);
 	}
 
@@ -5440,8 +5515,11 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug)
 		ug->nr_kmem += nr_pages;
 	} else {
 		/* LRU pages aren't accounted at the root level */
-		if (!obj_cgroup_is_root(objcg))
+		if (!obj_cgroup_is_root(objcg)) {
 			ug->nr_memory += nr_pages;
+			if (mem_cgroup_tiered_limits())
+				gather_tier_charge(ug, folio, nr_pages);
+		}
 		ug->pgpgout++;
 
 		WARN_ON_ONCE(folio_unqueue_deferred_split(folio));
@@ -5514,6 +5592,11 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
 	/* Force-charge the new page. The old one will be freed soon */
 	if (!obj_cgroup_is_root(objcg)) {
 		page_counter_charge(&memcg->memory, nr_pages);
+		if (mem_cgroup_tiered_limits()) {
+			int slot = nid_tier_slot(folio_nid(new));
+
+			mem_cgroup_charge_tier(memcg, slot, nr_pages);
+		}
 		if (do_memsw_account())
 			page_counter_charge(&memcg->memsw, nr_pages);
 	}
@@ -5558,6 +5641,23 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new)
 	if (!objcg)
 		return;
 
+	if (!obj_cgroup_is_root(objcg) && mem_cgroup_tiered_limits()) {
+		struct mem_cgroup *memcg;
+		unsigned long nr_pages = folio_nr_pages(old);
+		int old_slot, new_slot;
+
+		rcu_read_lock();
+		memcg = obj_cgroup_memcg(objcg);
+		old_slot = nid_tier_slot(folio_nid(old));
+		new_slot = nid_tier_slot(folio_nid(new));
+
+		if (old_slot != new_slot) {
+			mem_cgroup_uncharge_tier(memcg, old_slot, nr_pages);
+			mem_cgroup_charge_tier(memcg, new_slot, nr_pages);
+		}
+		rcu_read_unlock();
+	}
+
 	/* Transfer the charge and the objcg ref */
 	commit_charge(new, objcg);
 
@@ -5633,7 +5733,7 @@ bool mem_cgroup_sk_charge(const struct sock *sk, unsigned int nr_pages,
 	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		return memcg1_charge_skmem(memcg, nr_pages, gfp_mask);
 
-	if (try_charge_memcg(memcg, gfp_mask, nr_pages) == 0) {
+	if (try_charge_memcg(memcg, gfp_mask, nr_pages, NUMA_NO_NODE) == 0) {
 		mod_memcg_state(memcg, MEMCG_SOCK, nr_pages);
 		return true;
 	}
-- 
2.53.0-Meta

[RFC PATCH v3 08/14] mm/memcontrol: Make memory.low and memory.min tier-aware

[Joshua Hahn]

On machines serving multiple workloads whose memory is isolated via
the memory cgroup controller, it is currently impossible to enforce a
fair distribution of tiered memory among the workloads, as the only
enforceable limits have to do with total memory footprint, but not where
that memory resides.

This makes ensuring a consistent and baseline performance difficult, as
each workload's performance is heavily impacted by workload-external
factors such as which other workloads are co-located in the same host,
and the order at which different workloads are started.

Extend the existing memory.{low, min} protection to be tier-aware in
order to enforce proportional best-effort and guaranteed memory
protection of higher-tier memory.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h | 25 +++++++++++++++++++++----
 mm/memcontrol.c            | 11 ++++++++++-
 mm/vmscan.c                | 15 +++++++++------
 3 files changed, 40 insertions(+), 11 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index f7a92b66330ec..ceba0fd6de184 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -18,6 +18,7 @@
 #include <linux/page_counter.h>
 #include <linux/vmpressure.h>
 #include <linux/eventfd.h>
+#include <linux/memory-tiers.h>
 #include <linux/mm.h>
 #include <linux/vmstat.h>
 #include <linux/writeback.h>
@@ -618,21 +619,37 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target,
 }
 
 static inline bool mem_cgroup_below_low(struct mem_cgroup *target,
-					struct mem_cgroup *memcg)
+					struct mem_cgroup *memcg, int nid)
 {
 	if (mem_cgroup_unprotected(target, memcg))
 		return false;
 
+	if (mem_cgroup_tiered_limits()) {
+		int slot = nid_tier_slot(nid);
+
+		if (slot >= 0)
+			return READ_ONCE(memcg->tier[slot].elow) >=
+				page_counter_read(&memcg->tier[slot]);
+	}
+
 	return READ_ONCE(memcg->memory.elow) >=
 		page_counter_read(&memcg->memory);
 }
 
 static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
-					struct mem_cgroup *memcg)
+					struct mem_cgroup *memcg, int nid)
 {
 	if (mem_cgroup_unprotected(target, memcg))
 		return false;
 
+	if (mem_cgroup_tiered_limits()) {
+		int slot = nid_tier_slot(nid);
+
+		if (slot >= 0)
+			return READ_ONCE(memcg->tier[slot].emin) >=
+				page_counter_read(&memcg->tier[slot]);
+	}
+
 	return READ_ONCE(memcg->memory.emin) >=
 		page_counter_read(&memcg->memory);
 }
@@ -1142,13 +1159,13 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target,
 	return true;
 }
 static inline bool mem_cgroup_below_low(struct mem_cgroup *target,
-					struct mem_cgroup *memcg)
+					struct mem_cgroup *memcg, int nid)
 {
 	return false;
 }
 
 static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
-					struct mem_cgroup *memcg)
+					struct mem_cgroup *memcg, int nid)
 {
 	return false;
 }
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f3714dfd85aa0..025496794cb91 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5310,7 +5310,16 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
 	if (!root)
 		root = root_mem_cgroup;
 
-	page_counter_calculate_protection(&root->memory, &memcg->memory, recursive_protection);
+	page_counter_calculate_protection(&root->memory, &memcg->memory,
+					  recursive_protection);
+
+	if (mem_cgroup_tiered_limits()) {
+		int nr_tier_slots = mt_nr_tier_slots();
+
+		for (int slot = 0; slot < nr_tier_slots; slot++)
+			page_counter_calculate_protection(&root->tier[slot],
+				&memcg->tier[slot], recursive_protection);
+	}
 }
 
 static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,
diff --git a/mm/vmscan.c b/mm/vmscan.c
index ffe7ea3c5aff6..29f3f12042650 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4190,7 +4190,7 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc
 	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
 	DEFINE_MIN_SEQ(lruvec);
 
-	if (mem_cgroup_below_min(NULL, memcg))
+	if (mem_cgroup_below_min(NULL, memcg, lruvec_pgdat(lruvec)->node_id))
 		return false;
 
 	if (!lruvec_is_sizable(lruvec, sc))
@@ -5057,6 +5057,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
 	bool need_rotate = false, should_age = false;
 	long nr_batch, nr_to_scan;
 	int swappiness = get_swappiness(lruvec, sc);
+	int nid = lruvec_pgdat(lruvec)->node_id;
 	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
 
 	nr_to_scan = get_nr_to_scan(lruvec, sc, memcg, swappiness);
@@ -5064,7 +5065,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
 		int delta;
 		DEFINE_MAX_SEQ(lruvec);
 
-		if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) {
+		if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg, nid)) {
 			need_rotate = true;
 			break;
 		}
@@ -5104,12 +5105,13 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc)
 	unsigned long reclaimed = sc->nr_reclaimed;
 	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(lruvec);
+	int nid = pgdat->node_id;
 
 	/* lru_gen_age_node() called mem_cgroup_calculate_protection() */
-	if (mem_cgroup_below_min(NULL, memcg))
+	if (mem_cgroup_below_min(NULL, memcg, nid))
 		return MEMCG_LRU_YOUNG;
 
-	if (mem_cgroup_below_low(NULL, memcg)) {
+	if (mem_cgroup_below_low(NULL, memcg, nid)) {
 		/* see the comment on MEMCG_NR_GENS */
 		if (READ_ONCE(lruvec->lrugen.seg) != MEMCG_LRU_TAIL)
 			return MEMCG_LRU_TAIL;
@@ -6168,6 +6170,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
 	};
 	struct mem_cgroup_reclaim_cookie *partial = &reclaim;
 	struct mem_cgroup *memcg;
+	int nid = pgdat->node_id;
 
 	/*
 	 * In most cases, direct reclaimers can do partial walks
@@ -6197,13 +6200,13 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
 
 		mem_cgroup_calculate_protection(target_memcg, memcg);
 
-		if (mem_cgroup_below_min(target_memcg, memcg)) {
+		if (mem_cgroup_below_min(target_memcg, memcg, nid)) {
 			/*
 			 * Hard protection.
 			 * If there is no reclaimable memory, OOM.
 			 */
 			continue;
-		} else if (mem_cgroup_below_low(target_memcg, memcg)) {
+		} else if (mem_cgroup_below_low(target_memcg, memcg, nid)) {
 			/*
 			 * Soft protection.
 			 * Respect the protection only as long as
-- 
2.53.0-Meta

[RFC PATCH v3 09/14] mm/memcontrol: Make memory.high tier-aware

[Joshua Hahn]

On machines serving multiple workloads whose memory is isolated via
the memory cgroup controller, it is currently impossible to enforce a
fair distribution of tiered memory among the workloads, as the only
enforceable limits have to do with total memory footprint, but not where
that memory resides.

This makes ensuring consistent baseline performance difficult, as each
workload's performance is heavily impacted by workload-external factors
such as which other workloads are co-located in the same host, and the
order in which the workloads are started.

Extend the existing memory.high protection to be tier-aware.

Depending on the combination of limit breaches, selectively reclaim on
tiers: when memory.high is breached, perform reclaim on all tiers.
When memory.high is safe but individual tier limits are breached,
perform targeted reclaim on those tiers only.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/memcontrol.c | 66 ++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 55 insertions(+), 11 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 025496794cb91..44ea465b2005d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2387,6 +2387,28 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu)
 	return 0;
 }
 
+static bool memcg_tier_over_limit(struct mem_cgroup *memcg,
+				  unsigned long *overage, int *breached_slot)
+{
+	int nr_tier_slots = mt_nr_tier_slots();
+
+	for (int slot = 0; slot < nr_tier_slots; slot++) {
+		unsigned long usage = page_counter_read(&memcg->tier[slot]);
+		unsigned long limit = READ_ONCE(memcg->tier[slot].high);
+
+		if (usage <= limit)
+			continue;
+
+		if (overage)
+			*overage = usage - limit;
+		if (breached_slot)
+			*breached_slot = slot;
+		return true;
+	}
+
+	return false;
+}
+
 static unsigned long reclaim_high(struct mem_cgroup *memcg,
 				  unsigned int nr_pages,
 				  gfp_t gfp_mask)
@@ -2395,10 +2417,19 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg,
 
 	do {
 		unsigned long pflags;
+		const nodemask_t *reclaim_nodes = NULL;
 
 		if (page_counter_read(&memcg->memory) <=
-		    READ_ONCE(memcg->memory.high))
-			continue;
+		    READ_ONCE(memcg->memory.high)) {
+			int slot;
+
+			if (!mem_cgroup_tiered_limits())
+				continue;
+			if (!memcg_tier_over_limit(memcg, NULL, &slot))
+				continue;
+
+			reclaim_nodes = mt_tier_nodes(slot);
+		}
 
 		memcg_memory_event(memcg, MEMCG_HIGH);
 
@@ -2406,7 +2437,7 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg,
 		nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages,
 							gfp_mask,
 							MEMCG_RECLAIM_MAY_SWAP,
-							NULL, NULL);
+							NULL, reclaim_nodes);
 		psi_memstall_leave(&pflags);
 	} while ((memcg = parent_mem_cgroup(memcg)) &&
 		 !mem_cgroup_is_root(memcg));
@@ -2842,23 +2873,25 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 * reclaim, the cost of mismatch is negligible.
 	 */
 	do {
-		bool mem_high, swap_high;
+		bool mem_high, swap_high, tier_high;
 
 		mem_high = page_counter_read(&memcg->memory) >
 			READ_ONCE(memcg->memory.high);
 		swap_high = page_counter_read(&memcg->swap) >
 			READ_ONCE(memcg->swap.high);
+		tier_high = mem_cgroup_tiered_limits() &&
+			memcg_tier_over_limit(memcg, NULL, NULL);
 
 		/* Don't bother a random interrupted task */
 		if (!in_task()) {
-			if (mem_high) {
+			if (mem_high || tier_high) {
 				schedule_work(&memcg->high_work);
 				break;
 			}
 			continue;
 		}
 
-		if (mem_high || swap_high) {
+		if (mem_high || swap_high || tier_high) {
 			/*
 			 * The allocating tasks in this cgroup will need to do
 			 * reclaim or be throttled to prevent further growth
@@ -4967,13 +5000,24 @@ static ssize_t memory_high_write(struct kernfs_open_file *of,
 
 	for (;;) {
 		unsigned long nr_pages = page_counter_read(&memcg->memory);
-		unsigned long reclaimed;
+		unsigned long reclaimed, charge;
+		const nodemask_t *reclaim_nodes = NULL;
 
 		if (high != READ_ONCE(memcg->memory.high))
 			break;
 
-		if (nr_pages <= high)
-			break;
+		if (nr_pages <= high) {
+			int slot;
+
+			if (!mem_cgroup_tiered_limits())
+				break;
+			if (!memcg_tier_over_limit(memcg, &charge, &slot))
+				break;
+
+			reclaim_nodes = mt_tier_nodes(slot);
+		} else {
+			charge = nr_pages - high;
+		}
 
 		if (signal_pending(current))
 			break;
@@ -4988,9 +5032,9 @@ static ssize_t memory_high_write(struct kernfs_open_file *of,
 			continue;
 		}
 
-		reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages - high,
+		reclaimed = try_to_free_mem_cgroup_pages(memcg, charge,
 					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP,
-					NULL, NULL);
+					NULL, reclaim_nodes);
 
 		if (!reclaimed && !nr_retries--)
 			break;
-- 
2.53.0-Meta

[RFC PATCH v3 10/14] mm/memcontrol: Make memory.max tier-aware

[Joshua Hahn]

On machines serving multiple workloads whose memory is isolated via
the memory cgroup controller, it is currently impossible to enforce a
fair distribution of tiered memory among the workloads, as the only
enforceable limits have to do with total memory footprint, but not
where that memory resides.

Extend the existing memory.max limit to be tier-aware. A folio charge
is now attempted against the page_counter of the tier the folio was
allocated on, in addition to memory and memsw. When the tier counter
is over its limit, reclaim is targeted at that tier's nodes only.

Tier counters are parent-linked index-for-index with the memcg
hierarchy, so the counter reported by page_counter_try_charge() belongs
to an ancestor of the charging memcg.  memcg->tier is a separate
allocation, so container_of() cannot recover the owner;
tier_counter_mem_cgroup() walks the chain instead. This is only done
on the charge failure path.

mem_cgroup_margin() takes the tier slot so that the retry check and the
OOM re-check under oom_lock consider the tier that actually failed.
Taking the minimum across all tiers would report no headroom whenever
any tier is full, which would disable the pile-on guard in
mem_cgroup_out_of_memory() for plain memory.max breaches as well.

Note that stock is currently a per-memcg resource and does not
distinguish between tiers. There is ongoing work to change this,
however. Until then, tiered usage may transiently breach the max limit.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/memcontrol.c | 112 ++++++++++++++++++++++++++++++++++++------------
 1 file changed, 84 insertions(+), 28 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 44ea465b2005d..de6762520f475 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1570,14 +1570,32 @@ static struct page_counter *mem_cgroup_tier_counter(struct mem_cgroup *memcg,
 	return &memcg->tier[slot];
 }
 
+static struct mem_cgroup *tier_counter_mem_cgroup(struct mem_cgroup *memcg,
+						  struct page_counter *counter,
+						  int slot)
+{
+	struct mem_cgroup *iter;
+
+	for (iter = memcg; iter; iter = parent_mem_cgroup(iter)) {
+		if (&iter->tier[slot] == counter)
+			return iter;
+	}
+
+	/* the failing counter is always an ancestor of the given memcg */
+	VM_WARN_ON_ONCE(1);
+	return memcg;
+}
+
 /**
  * mem_cgroup_margin - calculate chargeable space of a memory cgroup
  * @memcg: the memory cgroup
+ * @slot: the memory tier slot
  *
- * Returns the maximum amount of memory @mem can be charged with, in
- * pages.
+ * Returns the maximum amount of memory @mem can be charged with, in pages.
+ * If the system has tiered memcg limits, then it returns the minimum of the
+ * tiered margin and the memcg margin.
  */
-static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg)
+static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg, int slot)
 {
 	unsigned long margin = 0;
 	unsigned long count;
@@ -1597,6 +1615,21 @@ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg)
 			margin = 0;
 	}
 
+	if (mem_cgroup_tiered_limits()) {
+		struct page_counter *tier_counter;
+
+		tier_counter = mem_cgroup_tier_counter(memcg, slot);
+		if (!tier_counter)
+			return margin;
+
+		count = page_counter_read(tier_counter);
+		limit = READ_ONCE(tier_counter->max);
+		if (count < limit)
+			margin = min(margin, limit - count);
+		else
+			margin = 0;
+	}
+
 	return margin;
 }
 
@@ -1945,7 +1978,7 @@ void __memcg_memory_event(struct mem_cgroup *memcg,
 EXPORT_SYMBOL_GPL(__memcg_memory_event);
 
 static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
-				     int order)
+				     int order, int slot)
 {
 	struct oom_control oc = {
 		.zonelist = NULL,
@@ -1959,7 +1992,7 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	if (mutex_lock_killable(&oom_lock))
 		return true;
 
-	if (mem_cgroup_margin(memcg) >= (1 << order))
+	if (mem_cgroup_margin(memcg, slot) >= (1 << order))
 		goto unlock;
 
 	/*
@@ -1977,7 +2010,8 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
  * Returns true if successfully killed one or more processes. Though in some
  * corner cases it can return true even without killing any process.
  */
-static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
+static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order,
+			   int slot)
 {
 	bool locked, ret;
 
@@ -1989,7 +2023,7 @@ static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
 	if (!memcg1_oom_prepare(memcg, &locked))
 		return false;
 
-	ret = mem_cgroup_out_of_memory(memcg, mask, order);
+	ret = mem_cgroup_out_of_memory(memcg, mask, order, slot);
 
 	memcg1_oom_finish(memcg, locked);
 
@@ -2388,13 +2422,15 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu)
 }
 
 static bool memcg_tier_over_limit(struct mem_cgroup *memcg,
-				  unsigned long *overage, int *breached_slot)
+				  unsigned long *overage, int *breached_slot,
+				  bool high)
 {
 	int nr_tier_slots = mt_nr_tier_slots();
 
 	for (int slot = 0; slot < nr_tier_slots; slot++) {
 		unsigned long usage = page_counter_read(&memcg->tier[slot]);
-		unsigned long limit = READ_ONCE(memcg->tier[slot].high);
+		unsigned long limit = high ? READ_ONCE(memcg->tier[slot].high) :
+					     READ_ONCE(memcg->tier[slot].max);
 
 		if (usage <= limit)
 			continue;
@@ -2425,7 +2461,7 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg,
 
 			if (!mem_cgroup_tiered_limits())
 				continue;
-			if (!memcg_tier_over_limit(memcg, NULL, &slot))
+			if (!memcg_tier_over_limit(memcg, NULL, &slot, true))
 				continue;
 
 			reclaim_nodes = mt_tier_nodes(slot);
@@ -2719,6 +2755,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	unsigned long pflags;
 	bool allow_spinning = gfpflags_allow_spinning(gfp_mask);
 	int slot = -1;
+	const nodemask_t *reclaim_nodes;
 
 	if (mem_cgroup_tiered_limits()) {
 		slot = nid_tier_slot(nid);
@@ -2737,6 +2774,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		batch = nr_pages;
 
 	reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
+	reclaim_nodes = NULL;
 
 	if (do_memsw_account() &&
 	    !page_counter_try_charge(&memcg->memsw, batch, &counter)) {
@@ -2745,15 +2783,23 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		goto reclaim;
 	}
 
-	if (page_counter_try_charge(&memcg->memory, batch, &counter)) {
+	if (tier_counter &&
+	    !page_counter_try_charge(tier_counter, nr_pages, &counter)) {
+		mem_over_limit = tier_counter_mem_cgroup(memcg, counter, slot);
+		reclaim_nodes = mt_tier_nodes(slot);
+		goto reclaim;
+	}
+
+	if (!page_counter_try_charge(&memcg->memory, batch, &counter)) {
+		mem_over_limit = mem_cgroup_from_counter(counter, memory);
+		if (do_memsw_account())
+			page_counter_uncharge(&memcg->memsw, batch);
 		if (tier_counter)
-			page_counter_charge(tier_counter, nr_pages);
-		goto done_restock;
+			page_counter_uncharge(tier_counter, nr_pages);
+		goto reclaim;
 	}
 
-	if (do_memsw_account())
-		page_counter_uncharge(&memcg->memsw, batch);
-	mem_over_limit = mem_cgroup_from_counter(counter, memory);
+	goto done_restock;
 
 reclaim:
 	if (batch > nr_pages) {
@@ -2782,13 +2828,13 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	psi_memstall_enter(&pflags);
 	nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
 						    gfp_mask, reclaim_options,
-						    NULL, NULL);
+						    NULL, reclaim_nodes);
 	psi_memstall_leave(&pflags);
 
-	if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
+	if (mem_cgroup_margin(mem_over_limit, slot) >= nr_pages)
 		goto retry;
 
-	if (!drained) {
+	if (!drained && !reclaim_nodes) {
 		drain_all_stock(mem_over_limit);
 		drained = true;
 		goto retry;
@@ -2824,7 +2870,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 * couldn't make any progress.
 	 */
 	if (mem_cgroup_oom(mem_over_limit, gfp_mask,
-			   get_order(nr_pages * PAGE_SIZE))) {
+			   get_order(nr_pages * PAGE_SIZE), slot)) {
 		passed_oom = true;
 		nr_retries = MAX_RECLAIM_RETRIES;
 		goto retry;
@@ -2880,7 +2926,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		swap_high = page_counter_read(&memcg->swap) >
 			READ_ONCE(memcg->swap.high);
 		tier_high = mem_cgroup_tiered_limits() &&
-			memcg_tier_over_limit(memcg, NULL, NULL);
+			memcg_tier_over_limit(memcg, NULL, NULL, true);
 
 		/* Don't bother a random interrupted task */
 		if (!in_task()) {
@@ -5011,7 +5057,7 @@ static ssize_t memory_high_write(struct kernfs_open_file *of,
 
 			if (!mem_cgroup_tiered_limits())
 				break;
-			if (!memcg_tier_over_limit(memcg, &charge, &slot))
+			if (!memcg_tier_over_limit(memcg, &charge, &slot, true))
 				break;
 
 			reclaim_nodes = mt_tier_nodes(slot);
@@ -5073,12 +5119,22 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
 
 	for (;;) {
 		unsigned long nr_pages = page_counter_read(&memcg->memory);
+		unsigned long charge;
+		const nodemask_t *reclaim_nodes = NULL;
+		int slot = -1;
 
 		if (max != READ_ONCE(memcg->memory.max))
 			break;
 
-		if (nr_pages <= max)
-			break;
+		if (nr_pages <= max) {
+			if (!mem_cgroup_tiered_limits())
+				break;
+			if (!memcg_tier_over_limit(memcg, &charge, &slot, false))
+				break;
+			reclaim_nodes = mt_tier_nodes(slot);
+		} else {
+			charge = nr_pages - max;
+		}
 
 		if (signal_pending(current))
 			break;
@@ -5087,22 +5143,22 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
 		if (memcg_is_dying(memcg))
 			break;
 
-		if (!drained) {
+		if (!drained && !reclaim_nodes) {
 			drain_all_stock(memcg);
 			drained = true;
 			continue;
 		}
 
 		if (nr_reclaims) {
-			if (!try_to_free_mem_cgroup_pages(memcg, nr_pages - max,
+			if (!try_to_free_mem_cgroup_pages(memcg, charge,
 					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP,
-					NULL, NULL))
+					NULL, reclaim_nodes))
 				nr_reclaims--;
 			continue;
 		}
 
 		memcg_memory_event(memcg, MEMCG_OOM);
-		if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0))
+		if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0, slot))
 			break;
 		cond_resched();
 	}
-- 
2.53.0-Meta

[RFC PATCH v3 11/14] mm/memcontrol, migrate: Transfer tier charge on migration

[Joshua Hahn]

Folio migration is charge-neutral, and existing migration paths take
advantage of this fact to simply force destination folio charges or
just transfer memcg data across folios.

Per-tier memcg limits break this assumption. A migration across tiers
(i.e. promotion or demotion) keeps the memcg-level charge neutral,
but the per-memcg tier charges change. As a result, the destination tier
may go over the limit.

Charge the destination separately instead, from migrate_folio_unmap
where the destination folio has just been allocated but can still be
rolled back. This charge attempts a single pass at reclaim if it goes
over the hard limit, and fails the migration if not enough headroom is
created on the destination memcg tier.

Note that this source of migration failure returns -EBUSY and not
-ENOMEM since -ENOMEM will attempt the migration again by splitting the
folio and aborting the batch, which both do nothing to reduce the
memory usage of the memcg tier. We also don't try too hard to reclaim
here (__GFP_NORETRY) since failing migrations is cheap, and we don't
want to OOM kill because of a promotion attempt.

One side effect is that cross-tier migrations now hold both folios'
charges until the source is freed, the same way mem_cgroup_replace_folio
temporarily holds a duplicate charge.

No-op unless the system has tiered memcg limits enabled.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h |  8 ++++++++
 mm/memcontrol.c            | 35 +++++++++++++++++++++++++++++++++++
 mm/migrate.c               | 30 +++++++++++++++++++++++++++---
 3 files changed, 70 insertions(+), 3 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index ceba0fd6de184..9c2f11191a499 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -708,6 +708,8 @@ static inline void mem_cgroup_uncharge_folios(struct folio_batch *folios)
 
 void mem_cgroup_replace_folio(struct folio *old, struct folio *new);
 void mem_cgroup_migrate(struct folio *old, struct folio *new);
+int mem_cgroup_migrate_charge(struct folio *src, struct folio *dst,
+			      bool force);
 
 /**
  * mem_cgroup_lruvec - get the lru list vector for a memcg & node
@@ -1204,6 +1206,12 @@ static inline void mem_cgroup_migrate(struct folio *old, struct folio *new)
 {
 }
 
+static inline int mem_cgroup_migrate_charge(struct folio *src,
+					    struct folio *dst, bool force)
+{
+	return 0;
+}
+
 static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,
 					       struct pglist_data *pgdat)
 {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index de6762520f475..1161934e81380 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5716,6 +5716,41 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
 	rcu_read_unlock();
 }
 
+/**
+ * mem_cgroup_migrate_charge - Charge a migration destination up front.
+ * @src: Folio being migrated away from.
+ * @dst: Folio being migrated to.
+ * @force: Charge even if the destination tier is at its limit.
+ *
+ * Folio migrations result in a net 0 memcg charge, but the node location of the
+ * charge may change during promotions or demotions. When this happens, charge
+ * @dst in its own right instead of inheriting @src's charge.
+ *
+ * Return: 0, or -ENOMEM if @dst could not be charged.
+ */
+int mem_cgroup_migrate_charge(struct folio *src, struct folio *dst, bool force)
+{
+	struct mem_cgroup *memcg;
+	gfp_t gfp = GFP_KERNEL;
+	int ret;
+
+	if (mem_cgroup_disabled() || !folio_memcg_charged(src))
+		return 0;
+
+	if (!mem_cgroup_tiered_limits() ||
+	    nid_tier_slot(folio_nid(src)) == nid_tier_slot(folio_nid(dst)))
+		return 0;
+
+	/* Refuse the migration if the first reclaim round fails */
+	gfp |= force ? __GFP_NOFAIL : __GFP_NORETRY;
+
+	memcg = get_mem_cgroup_from_folio(src);
+	ret = charge_memcg(dst, memcg, gfp);
+	mem_cgroup_put(memcg);
+
+	return ret;
+}
+
 /**
  * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
  * @old: Currently circulating folio.
diff --git a/mm/migrate.c b/mm/migrate.c
index ab15a4dddd047..45d6d23d53859 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -862,7 +862,13 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 	folio_copy_owner(newfolio, folio);
 	pgalloc_tag_swap(newfolio, folio);
 
-	mem_cgroup_migrate(folio, newfolio);
+	/*
+	 * For failable memcg charge transfers (demotion / promotion) the charge
+	 * has already been transferred at this point. For everyone else simply
+	 * transfer the charge here, where it can no longer fail.
+	 */
+	if (!folio_memcg_charged(newfolio))
+		mem_cgroup_migrate(folio, newfolio);
 }
 EXPORT_SYMBOL(folio_migrate_flags);
 
@@ -1216,7 +1222,7 @@ static void migrate_folio_done(struct folio *src,
 static int migrate_folio_unmap(new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
 		struct folio *src, struct folio **dstp, enum migrate_mode mode,
-		struct list_head *ret)
+		bool force_charge, struct list_head *ret)
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
@@ -1228,6 +1234,17 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst = get_new_folio(src, private);
 	if (!dst)
 		return -ENOMEM;
+
+	if (mem_cgroup_migrate_charge(src, dst, force_charge)) {
+		if (put_new_folio)
+			put_new_folio(dst, private);
+		else
+			folio_put(dst);
+		if (ret)
+			list_move_tail(&src->lru, ret);
+		return -EBUSY;
+	}
+
 	*dstp = dst;
 
 	dst->migrate_info = 0;
@@ -1918,8 +1935,15 @@ static int migrate_pages_batch(struct list_head *from,
 				continue;
 			}
 
+			/*
+			 * Hotplug must not be refused: offline_pages() retries
+			 * indefinitely and ignores migration failures, so a
+			 * refusal would hang it rather than fail it.
+			 */
 			rc = migrate_folio_unmap(get_new_folio, put_new_folio,
-					private, folio, &dst, mode, ret_folios);
+					private, folio, &dst, mode,
+					reason == MR_MEMORY_HOTPLUG,
+					ret_folios);
 			/*
 			 * The rules are:
 			 *	0: folio will be put on unmap_folios list,
-- 
2.53.0-Meta

[RFC PATCH v3 12/14] mm/memcontrol: Kick async reclaim on migration and folio replacement

[Joshua Hahn]

Memcg tier charges are now transferred during folio migrations, but
this still leaves forced charges and unenforced paths like folio
replacement, hugetlb migration, and memory hotplug.

Forced charges can and should not be enforced by synchronous reclaim,
but if soft limits are set, we can kick async reclaimers to try and
bring the usage below the high memcg tier limit.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/memcontrol.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1161934e81380..4dce7c6fefd98 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5663,6 +5663,16 @@ void __mem_cgroup_uncharge_folios(struct folio_batch *folios)
 		uncharge_batch(&ug);
 }
 
+static void tier_kick_high(struct mem_cgroup *memcg, int slot)
+{
+	struct page_counter *tier_counter;
+
+	tier_counter = mem_cgroup_tier_counter(memcg, slot);
+	if (tier_counter && page_counter_read(tier_counter) >
+			    READ_ONCE(tier_counter->high))
+		schedule_work(&memcg->high_work);
+}
+
 /**
  * mem_cgroup_replace_folio - Charge a folio's replacement.
  * @old: Currently circulating folio.
@@ -5705,6 +5715,7 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
 			int slot = nid_tier_slot(folio_nid(new));
 
 			mem_cgroup_charge_tier(memcg, slot, nr_pages);
+			tier_kick_high(memcg, slot);
 		}
 		if (do_memsw_account())
 			page_counter_charge(&memcg->memsw, nr_pages);
@@ -5798,6 +5809,7 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new)
 		if (old_slot != new_slot) {
 			mem_cgroup_uncharge_tier(memcg, old_slot, nr_pages);
 			mem_cgroup_charge_tier(memcg, new_slot, nr_pages);
+			tier_kick_high(memcg, new_slot);
 		}
 		rcu_read_unlock();
 	}
-- 
2.53.0-Meta

[RFC PATCH v3 13/14] mm/memcontrol, sched/numa: Gate NUMA promotions into memcg tiers

[Joshua Hahn]

Memory promotions that go through should_numa_migrate_memory determine
if a promotion should be ratelimited / throttled by checking how much
headroom there is in the destination node. If there is enough headroom,
there is no reason to be throttling promotions.

On tiered systems, however, a promotion may trigger reclaim on a node
that has plenty of promotion headroom since the memcg tier may be at
the limit. For these allocations, we should make sure that memcg tier
fullness is also considered when determining whether a promotion should
be able to go through without getting limited.

Add an additional condition to check before letting a promotion
candidate go through un-ratelimited, by checking if the memcg tier is
already at its limit.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h |  7 +++++++
 kernel/sched/fair.c        |  3 ++-
 mm/memcontrol.c            | 35 +++++++++++++++++++++++++++++++++++
 3 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 9c2f11191a499..a7c366b431a0e 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -654,6 +654,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
 		page_counter_read(&memcg->memory);
 }
 
+bool mem_cgroup_tier_over_limit(struct folio *folio, int dst_nid);
+
 int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp);
 
 /**
@@ -1172,6 +1174,11 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
 	return false;
 }
 
+static inline bool mem_cgroup_tier_over_limit(struct folio *folio, int dst_nid)
+{
+	return false;
+}
+
 static inline int mem_cgroup_charge(struct folio *folio,
 		struct mm_struct *mm, gfp_t gfp)
 {
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee13..397b0f3e67f5f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -2697,7 +2697,8 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		long nr = folio_nr_pages(folio);
 
 		pgdat = NODE_DATA(dst_nid);
-		if (pgdat_free_space_enough(pgdat)) {
+		if (pgdat_free_space_enough(pgdat) &&
+		    !mem_cgroup_tier_over_limit(folio, dst_nid)) {
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 4dce7c6fefd98..05611a01aa082 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2590,6 +2590,41 @@ static u64 swap_find_max_overage(struct mem_cgroup *memcg)
 	return max_overage;
 }
 
+bool mem_cgroup_tier_over_limit(struct folio *folio, int dst_nid)
+{
+	struct mem_cgroup *memcg;
+	int dst_slot;
+
+	if (!mem_cgroup_tiered_limits())
+		return false;
+
+	dst_slot = nid_tier_slot(dst_nid);
+	if (nid_tier_slot(folio_nid(folio)) == dst_slot)
+		return false;
+
+	guard(rcu)();
+	memcg = folio_memcg(folio);
+	if (!memcg || mem_cgroup_is_root(memcg))
+		return false;
+
+	do {
+		struct page_counter *tier_counter;
+		unsigned long limit;
+
+		tier_counter = mem_cgroup_tier_counter(memcg, dst_slot);
+		if (!tier_counter)
+			continue;
+
+		limit = min(READ_ONCE(tier_counter->max),
+			    READ_ONCE(tier_counter->high));
+		if (page_counter_read(tier_counter) > limit)
+			return true;
+	} while ((memcg = parent_mem_cgroup(memcg)) &&
+		  !mem_cgroup_is_root(memcg));
+
+	return false;
+}
+
 /*
  * Get the number of jiffies that we should penalise a mischievous cgroup which
  * is exceeding its memory.high by checking both it and its ancestors.
-- 
2.53.0-Meta

[RFC PATCH v3 14/14] mm/page_alloc: steer allocations away from exhausted memory tiers

[Joshua Hahn]

A memcg only finds out that it is over a tier's limit at
try_charge_memcg time, after the page has already been allocated on the
full tier's node. This triggers reclaim to push memory to lower tiers or
to swap, much like how zone_reclaim_mode triggers reclaim when a node is
full.

This causes a lot of unnecessary churn. Instead of allocating a page on
a full tier only to immediately reclaim, make the page allocator
aware of tier limits at allocation time and steer the first allocation
attempt to nodes belonging to tiers with tiered limit headroom.

This only affects the ac->nodemask for the fastpath, meaning if no
node can satisfy this allocation, it falls back to the caller's nodemask
and places a page on a full tier, triggering reclaim. Moreover, if it
turns out that the intersection of the allocation context nodemask and
the under-limit nodemask is empty, skip the steering.

This steering only affects MIGRATE_MOVABLE allocations. We try not to
steer kernel allocations using NUMA_NO_NODE which would prefer to remain
on higher tiers, even if they are full.

This steering also does not affect alloc_pages_bulk_noprof, since
its callers do not charge their memory allocations to a tier.

No-op unless the system has tiered memcg limits enabled.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 include/linux/memcontrol.h |  6 +++++
 mm/memcontrol.c            | 46 ++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c            | 19 +++++++++++++---
 3 files changed, 68 insertions(+), 3 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index a7c366b431a0e..b700f3224cbaa 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -655,6 +655,7 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
 }
 
 bool mem_cgroup_tier_over_limit(struct folio *folio, int dst_nid);
+bool mem_cgroup_tier_allowed_nodemask(nodemask_t *mask);
 
 int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp);
 
@@ -1179,6 +1180,11 @@ static inline bool mem_cgroup_tier_over_limit(struct folio *folio, int dst_nid)
 	return false;
 }
 
+static inline bool mem_cgroup_tier_allowed_nodemask(nodemask_t *mask)
+{
+	return false;
+}
+
 static inline int mem_cgroup_charge(struct folio *folio,
 		struct mm_struct *mm, gfp_t gfp)
 {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 05611a01aa082..e499e58baa287 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2625,6 +2625,52 @@ bool mem_cgroup_tier_over_limit(struct folio *folio, int dst_nid)
 	return false;
 }
 
+/*
+ * Returns whether a mem_cgroup is above a tier's limit.
+ * The nodemask becomes populated with nodes that are under their limits.
+ */
+bool mem_cgroup_tier_allowed_nodemask(nodemask_t *mask)
+{
+	struct mem_cgroup *memcg;
+	int nr_tier_slots;
+	bool restricted = false;
+
+	if (!mem_cgroup_tiered_limits())
+		return false;
+
+	nr_tier_slots = mt_nr_tier_slots();
+	*mask = node_states[N_MEMORY];
+
+	rcu_read_lock();
+	memcg = active_memcg();
+	if (!memcg && in_task() && current->mm)
+		memcg = mem_cgroup_from_task(rcu_dereference(current->mm->owner));
+
+	for (; memcg && !mem_cgroup_is_root(memcg);
+	     memcg = parent_mem_cgroup(memcg)) {
+		if (READ_ONCE(memcg->memory.max) == PAGE_COUNTER_MAX &&
+		    READ_ONCE(memcg->memory.high) == PAGE_COUNTER_MAX)
+			continue;
+
+		for (int slot = 0; slot < nr_tier_slots; slot++) {
+			struct page_counter *tier = &memcg->tier[slot];
+			unsigned long limit;
+
+			limit = min(READ_ONCE(tier->high),
+				    READ_ONCE(tier->max));
+
+			if (page_counter_read(tier) <= limit)
+				continue;
+
+			restricted = true;
+			nodes_andnot(*mask, *mask, *mt_tier_nodes(slot));
+		}
+	}
+	rcu_read_unlock();
+
+	return restricted;
+}
+
 /*
  * Get the number of jiffies that we should penalise a mischievous cgroup which
  * is exceeding its memory.high by checking both it and its ancestors.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0aeca106a4fde..7ca0acc37d2af 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5153,7 +5153,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
 		int preferred_nid, nodemask_t *nodemask,
 		struct alloc_context *ac, gfp_t *alloc_gfp,
-		unsigned int *alloc_flags)
+		unsigned int *alloc_flags, nodemask_t *tier_nodes)
 {
 	ac->highest_zoneidx = gfp_zone(gfp_mask);
 	ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
@@ -5187,6 +5187,17 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
 	/* Dirty zone balancing only done in the fast path */
 	ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE);
 
+	if (mem_cgroup_tiered_limits() && tier_nodes &&
+	    !(gfp_mask & __GFP_THISNODE) &&
+	    ac->migratetype == MIGRATE_MOVABLE &&
+	    mem_cgroup_tier_allowed_nodemask(tier_nodes)) {
+		if (ac->nodemask)
+			nodes_and(*tier_nodes, *tier_nodes, *ac->nodemask);
+		/* tier_nodes must outlive the function call since ac uses it */
+		if (!nodes_empty(*tier_nodes))
+			ac->nodemask = tier_nodes;
+	}
+
 	/*
 	 * The preferred zone is used for statistics but crucially it is
 	 * also used as the starting point for the zonelist iterator. It
@@ -5269,7 +5280,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 
 	/* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
 	gfp &= gfp_allowed_mask;
-	if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
+	if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp,
+				 &alloc_flags, NULL))
 		goto out;
 
 	/* Find an allowed local zone that meets the low watermark. */
@@ -5451,6 +5463,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 		.alloc_flags = alloc_flags,
 	};
 	unsigned int fastpath_alloc_flags = alloc_flags;
+	nodemask_t tier_nodes;
 
 	/* Other flags could be supported later if needed. */
 	if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
@@ -5481,7 +5494,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 	gfp = current_gfp_context(gfp);
 	alloc_gfp = gfp;
 	if (!prepare_alloc_pages(gfp, order, preferred_nid, nodemask, &ac,
-			&alloc_gfp, &fastpath_alloc_flags))
+			&alloc_gfp, &fastpath_alloc_flags, &tier_nodes))
 		return NULL;
 
 	if (!(alloc_flags & ALLOC_NOLOCK)) {
-- 
2.53.0-Meta