[PATCH v2 0/1] mm: memcg: don't hand out large folios above memory.high

[PATCH v2 0/1] mm: memcg: don't hand out large folios above memory.high

[Qinyun Tan]

memory.high is enforced at two points after a charge succeeds: on
return to userspace, and synchronously in try_charge_memcg() for
large overcharges, the latter gated on gfpflags_allow_blocking().

The THP fault paths pass the physical allocation gfp from
vma_thp_gfp_mask() to the memcg charge.  With the default
defrag=madvise (and no MADV_HUGEPAGE), as well as with defrag=defer,
that gfp does not allow blocking, so inside a single-syscall populate
loop - mlock(), MADV_POPULATE_*, any GUP-driven population - neither
enforcement point runs: usage grows from memory.high straight up to
memory.max with no reclaim and no penalty sleep, consuming the
reaction window that userspace OOM handlers (oomd, Kubernetes) depend
on.

v1 [1] settled the accrued over-high debt from the fault paths after
each non-blocking large folio charge.  As Zi Yan pointed out in his
review [2], falling back to order-0 above memory.high is the better
answer: above high the cgroup is meant to be under heavy reclaim
pressure, and the order-0 fallback's charge blocks, so it is throttled
synchronously as is.  v2 does that instead, and is a single patch now.

Qinyun Tan (1):
  mm: memcg: don't hand out large folios above memory.high

 include/linux/memcontrol.h | 33 ++++++++++++++++++++++
 mm/huge_memory.c           | 12 ++++++++
 mm/memcontrol.c            | 56 ++++++++++++++++++++++++++++++++++++++
 mm/memory.c                |  7 ++++++
 mm/swap_state.c            | 16 ++++++++++-
 5 files changed, 123 insertions(+), 1 deletion(-)

[1] https://lore.kernel.org/lkml/20260904035407.4098627-1-qinyuntan@linux.alibaba.com/
[2] https://lore.kernel.org/lkml/6B46E8DF-679B-4254-A4AF-7B992B0F6460@nvidia.com/

--
2.43.7

[PATCH v2 1/1] mm: memcg: don't hand out large folios above memory.high

[Qinyun Tan]

memory.high is enforced on return to userspace, and synchronously in
try_charge_memcg() for large overcharges, but only when the charge gfp
allows blocking.  A populate loop - mlock(), MADV_POPULATE_*, any
GUP-driven population - never returns to userspace, and large folios are
charged with the THP allocation gfp, which does not allow blocking under
the default defrag=madvise without MADV_HUGEPAGE, nor under defrag=defer.
So neither runs: usage grows from memory.high straight up to memory.max
with no reclaim and no penalty sleep.

mlock(200M) in a cgroup with memory.high=30M and memory.max=140M.  Of the
110M between high and max, the burst consumed:

  4K pages                       3M in 5s, then still throttled
  THP, defrag=always             6M in 5s, then still throttled
  THP, defrag=madvise          110M in 13ms, then OOM killed at 16ms
  THP, defrag=madvise, patched   3M in 5s, then still throttled

The defrag=madvise run raised no memory.high event at all - the throttling
machinery never ran once - and nothing in userspace, oomd or a Kubernetes
eviction manager, reacts inside 16ms.  The gfp it was charged with is not a
statement about sleeping either: it describes allocation policy, a THP is
not worth direct compaction, while the order-0 fallback of the very same
fault charges with GFP_KERNEL and sleeps just fine.  memory.high should not
stop working because of the order of the folios a workload happens to fault
in.

Fix this where the order is chosen: above memory.high, refuse a large folio
whose charge cannot be throttled - in the anon THP fault paths and for
large folio swapin - and let the fault fall back to order-0.  That charge
blocks, so try_charge_memcg() reclaims and throttles it as before, and the
refusal happens before the allocation, so the fault path only gets shorter.
mlock() above is the worst case since the pages are unevictable; where
reclaim can keep up, usage simply stays at memory.high.

A charge that may block is already throttled, so the check is gated on
gfpflags_allow_blocking(): defrag=always and MADV_HUGEPAGE VMAs are
unaffected.  Other VMAs get large folios again once usage is back below
memory.high, which is the intent - above it the cgroup is meant to be under
reclaim pressure.

Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Qinyun Tan <qinyuntan@linux.alibaba.com>
---
 include/linux/memcontrol.h | 33 ++++++++++++++++++++++
 mm/huge_memory.c           | 12 ++++++++
 mm/memcontrol.c            | 56 ++++++++++++++++++++++++++++++++++++++
 mm/memory.c                |  7 +++++
 mm/swap_state.c            | 16 ++++++++++-
 5 files changed, 123 insertions(+), 1 deletion(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 46bf724cae7a..defef0d3d710 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -941,6 +941,33 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
 		__mem_cgroup_handle_over_high(gfp_mask);
 }
 
+bool __mem_cgroup_large_folio_over_high(struct mm_struct *mm, gfp_t gfp);
+
+/**
+ * mem_cgroup_large_folio_over_high - would a large folio escape memory.high?
+ * @mm: mm the folio would be charged against, may be NULL
+ * @gfp: gfp mask the folio would be allocated and charged with
+ *
+ * See __mem_cgroup_large_folio_over_high().
+ *
+ * The task's over-high debt is more than a fast path here, it also scopes
+ * the check to the case that is broken.  The debt is settled and cleared on
+ * every return to userspace, so a task faulting large folios from a
+ * userspace loop always finds it zero and keeps getting them - memory.high
+ * is enforced for it on the way out.  Only a loop that stays in the kernel,
+ * where nothing throttles at all, accrues debt and reaches the counters.
+ *
+ * Return: %true if the caller should fall back to a smaller order.
+ */
+static inline bool mem_cgroup_large_folio_over_high(struct mm_struct *mm,
+						    gfp_t gfp)
+{
+	if (likely(!current->memcg_nr_pages_over_high))
+		return false;
+
+	return __mem_cgroup_large_folio_over_high(mm, gfp);
+}
+
 unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg);
 
 void mem_cgroup_print_oom_context(struct mem_cgroup *memcg,
@@ -1390,6 +1417,12 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
 {
 }
 
+static inline bool mem_cgroup_large_folio_over_high(struct mm_struct *mm,
+						    gfp_t gfp)
+{
+	return false;
+}
+
 static inline struct mem_cgroup *mem_cgroup_get_oom_group(
 	struct task_struct *victim, struct mem_cgroup *oom_domain)
 {
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 30b7c63b0e35..9247c36da64a 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1355,6 +1355,18 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,
 	const int order = HPAGE_PMD_ORDER;
 	struct folio *folio;
 
+	/*
+	 * A THP charged with a gfp that cannot block escapes the memory.high
+	 * throttling in try_charge_memcg().  Don't hand out one while the
+	 * cgroup is already above memory.high: the order-0 fallback is charged
+	 * with a blocking gfp and throttles as documented.
+	 */
+	if (mem_cgroup_large_folio_over_high(vma->vm_mm, gfp)) {
+		count_vm_event(THP_FAULT_FALLBACK);
+		count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
+		return NULL;
+	}
+
 	folio = vma_alloc_folio(gfp, order, vma, addr & HPAGE_PMD_MASK);
 
 	if (unlikely(!folio)) {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1460cba53588..dd7a62c3dacb 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2587,6 +2587,62 @@ static unsigned long calculate_high_delay(unsigned int nr_pages,
 	return penalty_jiffies * nr_pages / MEMCG_CHARGE_BATCH;
 }
 
+/**
+ * __mem_cgroup_large_folio_over_high - would a large folio escape memory.high?
+ * @mm: mm the folio would be charged against, may be NULL
+ * @gfp: gfp mask the folio would be allocated and charged with
+ *
+ * memory.high is enforced on return to userspace, or synchronously in
+ * try_charge_memcg() - but the synchronous path is gated on the charge gfp
+ * allowing blocking.  Large folios are charged with the THP allocation gfp,
+ * which does not allow blocking unless the allocation policy asks for direct
+ * compaction, so those charges escape throttling entirely: a fault loop that
+ * does not return to userspace inbetween - the populate loop of mlock() or
+ * MADV_POPULATE_*, any GUP-driven population - can grow usage from
+ * memory.high all the way up to memory.max with no reclaim and no delay.
+ *
+ * Above memory.high the cgroup is supposed to be under reclaim pressure, so
+ * refuse the large folio instead.  Callers fall back to order-0, which is
+ * charged with a blocking gfp and throttled as documented.
+ *
+ * This is a lockless snapshot of the counters; a stale result only costs one
+ * large folio either way.
+ *
+ * Callers should use mem_cgroup_large_folio_over_high(), which keeps the
+ * counter lookup off the fault path unless the task has actually charged
+ * above memory.high before.
+ *
+ * Return: %true if the caller should fall back to a smaller order.
+ */
+bool __mem_cgroup_large_folio_over_high(struct mm_struct *mm, gfp_t gfp)
+{
+	struct mem_cgroup *memcg, *iter;
+	bool over_high = false;
+
+	/*
+	 * A charge that can block is throttled by try_charge_memcg() itself,
+	 * there is no reason to give up the large folio for it.
+	 */
+	if (gfpflags_allow_blocking(gfp))
+		return false;
+
+	memcg = get_mem_cgroup_from_mm(mm);
+	if (!memcg)
+		return false;
+
+	for (iter = memcg; iter; iter = parent_mem_cgroup(iter)) {
+		if (page_counter_read(&iter->memory) >
+		    READ_ONCE(iter->memory.high)) {
+			over_high = true;
+			break;
+		}
+	}
+
+	mem_cgroup_put(memcg);
+
+	return over_high;
+}
+
 /*
  * Reclaims memory over the high limit. Called directly from
  * try_charge() (context permitting), as well as from the userland
diff --git a/mm/memory.c b/mm/memory.c
index 926276d41920..b28fccc86216 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5321,6 +5321,13 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
 
 	/* Try allocating the highest of the remaining orders. */
 	gfp = vma_thp_gfp_mask(vma);
+
+	/* Same reasoning as in vma_alloc_anon_folio_pmd(). */
+	if (mem_cgroup_large_folio_over_high(vma->vm_mm, gfp)) {
+		count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
+		goto fallback;
+	}
+
 	while (orders) {
 		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
 		folio = vma_alloc_folio(gfp, order, vma, addr);
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 625c185a1ca4..a235c9aecaf1 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -440,9 +440,23 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
 	 * Limit THP gfp. The limitation is a no-op for typical
 	 * GFP_HIGHUSER_MOVABLE but matters for shmem.
 	 */
-	if (order)
+	if (order) {
 		gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
 
+		/*
+		 * This gfp may not allow blocking, in which case the charge
+		 * below escapes the memory.high throttling in
+		 * try_charge_memcg().  Above memory.high, have the caller
+		 * retry with a smaller order, down to order-0, which is
+		 * charged with the caller's blocking gfp and throttled as
+		 * documented.  Only fault context is throttled here; the
+		 * readahead and zswap writeback callers have no @vmf and are
+		 * not the ones to hold back.
+		 */
+		if (vmf && mem_cgroup_large_folio_over_high(vma->vm_mm, gfp))
+			return ERR_PTR(-ENOMEM);
+	}
+
 	if (mpol || !vmf) {
 		folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
 	} else {
-- 
2.43.7

Re: [PATCH v2 1/1] mm: memcg: don't hand out large folios above memory.high

[Shakeel Butt]

On Tue, Sep 15, 2026 at 12:25:46PM +0800, Qinyun Tan wrote:
> memory.high is enforced on return to userspace, and synchronously in

Over time I feel like this sync throttle for memory.high was a bad decision but
that is an orthogonal discussion.

> try_charge_memcg() for large overcharges, but only when the charge gfp
> allows blocking.  A populate loop - mlock(), MADV_POPULATE_*, any
> GUP-driven population - never returns to userspace, and large folios are
> charged with the THP allocation gfp, which does not allow blocking under
> the default defrag=madvise without MADV_HUGEPAGE, nor under defrag=defer.
> So neither runs: usage grows from memory.high straight up to memory.max
> with no reclaim and no penalty sleep.
> 
> mlock(200M) in a cgroup with memory.high=30M and memory.max=140M.  Of the
> 110M between high and max, the burst consumed:
> 
>   4K pages                       3M in 5s, then still throttled
>   THP, defrag=always             6M in 5s, then still throttled
>   THP, defrag=madvise          110M in 13ms, then OOM killed at 16ms
>   THP, defrag=madvise, patched   3M in 5s, then still throttled

I don't really like polluting non-memcg MM code with memcg internal details.
Let's first discuss the semantics we want here. Please tell why getting
oom-killed by kernel in the scenario you have described (mlock() larger than
memory.max) is wrong. How does throttling it will help? What can the external
observer (userspace oom-killer) do other than killing it? Or you are thinking
that the workload itself is observing itself and change the behavior (though
the big mlock one can't do anything).

What I am looking for is the real use-case you have for this case. Not
mlock()ing big chunk or creating a lot of unreclaimable memory and going over
memory.high. For example the reason I added the sync throttle in memory.high was
to implement a feature Google has in their internal kernel where a workload hits
its memory.max limit, kernel delays the kill for couple of seconds and during
that time, the node controller may decide to increase the limit based on the
memory situation of the memory. However even with sync throttle, we couldn't
reliably implement the alternative because applications there have thousands of
threads and there is threading library which keeps cloning more threads if it
observes threads getting throttled.

So, let's talk about some real use-case and then we can decide if throttling
makes sense here. If we decide to allow throttling in such cases, I would rather
do it in memcg code i.e. somehow inform memcg code that this is THP allocation
and can be throttled.

Re: [PATCH v2 1/1] mm: memcg: don't hand out large folios above memory.high

[Qinyun Tan]

On 9/18/26 9:26 AM, Shakeel Butt wrote:
> On Tue, Sep 15, 2026 at 12:25:46PM +0800, Qinyun Tan wrote:
>> memory.high is enforced on return to userspace, and synchronously in
> 
> Over time I feel like this sync throttle for memory.high was a bad decision but
> that is an orthogonal discussion.
> 
>> try_charge_memcg() for large overcharges, but only when the charge gfp
>> allows blocking.  A populate loop - mlock(), MADV_POPULATE_*, any
>> GUP-driven population - never returns to userspace, and large folios are
>> charged with the THP allocation gfp, which does not allow blocking under
>> the default defrag=madvise without MADV_HUGEPAGE, nor under defrag=defer.
>> So neither runs: usage grows from memory.high straight up to memory.max
>> with no reclaim and no penalty sleep.
>>
>> mlock(200M) in a cgroup with memory.high=30M and memory.max=140M.  Of the
>> 110M between high and max, the burst consumed:
>>
>>   4K pages                       3M in 5s, then still throttled
>>   THP, defrag=always             6M in 5s, then still throttled
>>   THP, defrag=madvise          110M in 13ms, then OOM killed at 16ms
>>   THP, defrag=madvise, patched   3M in 5s, then still throttled
> 
> I don't really like polluting non-memcg MM code with memcg internal details.
> Let's first discuss the semantics we want here. Please tell why getting
> oom-killed by kernel in the scenario you have described (mlock() larger than
> memory.max) is wrong. How does throttling it will help? What can the external
> observer (userspace oom-killer) do other than killing it? Or you are thinking
> that the workload itself is observing itself and change the behavior (though
> the big mlock one can't do anything).
> 
> What I am looking for is the real use-case you have for this case. Not
> mlock()ing big chunk or creating a lot of unreclaimable memory and going over
> memory.high. For example the reason I added the sync throttle in memory.high was
> to implement a feature Google has in their internal kernel where a workload hits
> its memory.max limit, kernel delays the kill for couple of seconds and during
> that time, the node controller may decide to increase the limit based on the
> memory situation of the memory. However even with sync throttle, we couldn't
> reliably implement the alternative because applications there have thousands of
> threads and there is threading library which keeps cloning more threads if it
> observes threads getting throttled.
> 
> So, let's talk about some real use-case and then we can decide if throttling
> makes sense here. If we decide to allow throttling in such cases, I would rather
> do it in memcg code i.e. somehow inform memcg code that this is THP allocation
> and can be throttled.

Hi Shakeel,

Thanks for the detailed review.

Honestly I don't have a real use-case. I ran into this while testing THP
under memory.high, so it's a semantic inconsistency I noticed rather than
a workload I'm fixing. 

The thing that seemed off is just that memory.high enforcement depends on
the folio order the workload happens to fault in - order-0 gets
throttled, THP under the default THP settings doesn't - which the user
has no control over. 

I don't have a firm view on the right semantics here either, and I think
you're right that that's the thing to pin down first. Whether the
order-dependence above is worth doing something about, or whether it's
acceptable for large folios to skip memory.high, I don't have a strong
opinion on - I'm happy to go with whatever the discussion settles on, and
if it's working as intended, that's just as useful to know.

And if it does turn out to be worth throttling, I'd be happy to rework it.

Thanks,
Qinyun Tan

Re: [PATCH v2 1/1] mm: memcg: don't hand out large folios above memory.high

[David Hildenbrand (Arm)]

On 9/18/26 03:26, Shakeel Butt wrote:
> On Tue, Sep 15, 2026 at 12:25:46PM +0800, Qinyun Tan wrote:
>> memory.high is enforced on return to userspace, and synchronously in
> 
> Over time I feel like this sync throttle for memory.high was a bad decision but
> that is an orthogonal discussion.
> 
>> try_charge_memcg() for large overcharges, but only when the charge gfp
>> allows blocking.  A populate loop - mlock(), MADV_POPULATE_*, any
>> GUP-driven population - never returns to userspace, and large folios are
>> charged with the THP allocation gfp, which does not allow blocking under
>> the default defrag=madvise without MADV_HUGEPAGE, nor under defrag=defer.
>> So neither runs: usage grows from memory.high straight up to memory.max
>> with no reclaim and no penalty sleep.
>>
>> mlock(200M) in a cgroup with memory.high=30M and memory.max=140M.  Of the
>> 110M between high and max, the burst consumed:
>>
>>   4K pages                       3M in 5s, then still throttled
>>   THP, defrag=always             6M in 5s, then still throttled
>>   THP, defrag=madvise          110M in 13ms, then OOM killed at 16ms
>>   THP, defrag=madvise, patched   3M in 5s, then still throttled
> 
> I don't really like polluting non-memcg MM code with memcg internal details.

That's the same thought I had when skimming the code.

-- 
Cheers,

David

Re: [PATCH v2 0/1] mm: memcg: don't hand out large folios above memory.high

[Lorenzo Stoakes (ARM)]

This is the friendly patch-bot of Lorenzo Stoakes.

You have sent him a patch/series that has triggered this response.

He used to manually respond to these common problems, but in order to save
his sanity (he kept writing the same thing over and over, yet to different
people), I was created.

Hopefully you will not take offence and will fix the problem in your patch
and resubmit it so that it can be accepted into the Linux kernel tree.

When sending emails to mm:

1. Patch series with 1 patch sent with cover letter

Please send single patches without a cover letter.

The easiest way of accomplishing this (+ our preference) is to use
b4 [0], otherwise format patches like this:

$ git format-patch HEAD~1

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

2. Missing cc's

You are missing cc's. Fixing this is easy with b4 [0] (the recommended way
of sending patches to mm):

$ b4 prep --auto-to-cc

Alternatively, you can use scripts/get_maintainer.pl:

$ scripts/get_maintainer.pl --nogit-fallback <files-or-patches>

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

Specifically, the following appear to be missing:

  chrisl@kernel.org

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Lorenzo will reply once he has dug out from the pending patches received
from other developers.

thanks,

Lorenzo's patch email bot

[ Idea shamelessly stolen from greg-kh ]

--
Cheers, Lorenzo