* [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating Shakeel Butt
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
Since commit c9afe31ec443 ("memcg: synchronously enforce memory.high
for large overcharges"), a charge that can block handles memory.high
overage right away once more than a batch is pending. It reclaims and
sleeps with whatever locks the caller holds, so tasks in other cgroups
that need those locks wait too.
A later patch removes that step. Operations that can charge a lot in
one call then need another place to handle the overage: a point in
their loop where they hold no locks, or only the locks of their file.
Add two helpers for them. mem_cgroup_over_high_pending() says whether
more than a batch is pending, the threshold the synchronous step uses.
mem_cgroup_over_high_throttle() handles the overage if so; its comment
says where to call it and which gfp mask to pass. Export
__mem_cgroup_handle_over_high() for filesystems built as modules.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
include/linux/memcontrol.h | 34 ++++++++++++++++++++++++++++++++++
mm/memcontrol.c | 1 +
2 files changed, 35 insertions(+)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 980358bc6f49..77115fe1873e 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -942,6 +942,31 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
__mem_cgroup_handle_over_high(gfp_mask);
}
+/*
+ * True if the current task has more than a batch of memory.high or
+ * memory.swap.high overage pending. Each charge that finds overage
+ * counts at least a batch.
+ */
+static inline bool mem_cgroup_over_high_pending(void)
+{
+ return unlikely(current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH);
+}
+
+/*
+ * Handle the current task's memory.high overage if more than a batch is
+ * pending, for operations that can charge a lot before they return to
+ * userspace. The task may reclaim and sleep here, so call it only where
+ * other tasks won't wait for it: with no locks held anywhere up the call
+ * chain, or only the locks of the file being worked on. Pass GFP_KERNEL
+ * if no filesystem locks are held, and mapping_gfp_constraint() in file
+ * read and write paths.
+ */
+static inline void mem_cgroup_over_high_throttle(gfp_t gfp_mask)
+{
+ if (mem_cgroup_over_high_pending())
+ __mem_cgroup_handle_over_high(gfp_mask);
+}
+
unsigned long mem_cgroup_get_max(const struct mem_cgroup *memcg);
void mem_cgroup_print_oom_context(const struct mem_cgroup *memcg,
@@ -1376,6 +1401,15 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
{
}
+static inline bool mem_cgroup_over_high_pending(void)
+{
+ return false;
+}
+
+static inline void mem_cgroup_over_high_throttle(gfp_t gfp_mask)
+{
+}
+
static inline struct mem_cgroup *mem_cgroup_get_oom_group(
struct task_struct *victim, struct mem_cgroup *oom_domain)
{
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 091401cecf39..6d147153bd06 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2707,6 +2707,7 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
out:
css_put(&memcg->css);
}
+EXPORT_SYMBOL_GPL(__mem_cgroup_handle_over_high);
/*
* Called after @nr_pages were charged to @memcg. If @memcg or one of its
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes Shakeel Butt
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
mlock(), mlockall(), mmap() with MAP_POPULATE or MAP_LOCKED, and
MADV_POPULATE_READ/WRITE can fault in a lot of memory in one call,
with mmap_lock held. When their charges can block, the charge path
throttles them for memory.high with mmap_lock still held. In a VM with
THP on, mlock() of an 80M MADV_HUGEPAGE range under memory.high=64M
made another thread wait up to 17.2s to mmap(), and a task in another
cgroup wait up to 15.3s to read the process's /proc/<pid>/cmdline.
THP charges in VMAs without MADV_HUGEPAGE can't block under the
default defrag setting, so they are not throttled at all: mlock(512M)
reached 513M.
Throttle these calls with mmap_lock dropped instead. The populate
loops, __mm_populate() and madvise_populate(), now fault in at most
2MiB per GUP call. Between calls, if more than a batch of overage is
pending, they drop mmap_lock, handle it, and go on. GUP itself doesn't
change. One fault can still charge a whole folio.
With this patch, mlock(512M) stays at 84M, and populating 2G with no
limit takes as long as before. The mmap_lock stall goes away when a
later patch removes the synchronous step.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/gup.c | 9 +++++++--
mm/internal.h | 27 +++++++++++++++++++++++++++
mm/madvise.c | 11 +++++++++--
3 files changed, 43 insertions(+), 4 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 7605da0e06d8..d93d89f6aad0 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1951,6 +1951,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
end = start + len;
for (nstart = start; nstart < end; nstart = nend) {
+ populate_handle_over_high(mm, &locked);
/*
* We want to fault in pages for [nstart; end) address range.
* Find first corresponding VMA.
@@ -1976,9 +1977,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
/*
* Now fault in a range of pages. populate_vma_page_range()
* double checks the vma flags, so that it won't mlock pages
- * if the vma was already munlocked.
+ * if the vma was already munlocked. Do one chunk at a time,
+ * see POPULATE_CHUNK_SIZE. On error, nend skips the rest of
+ * the VMA.
*/
- ret = populate_vma_page_range(vma, nstart, nend, &locked);
+ ret = populate_vma_page_range(vma, nstart,
+ populate_chunk_end(nstart, nend),
+ &locked);
if (ret < 0) {
if (ignore_errors) {
ret = 0;
diff --git a/mm/internal.h b/mm/internal.h
index de4593b3e3b6..12f7b4b1254f 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -948,6 +948,33 @@ extern long populate_vma_page_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end, int *locked);
extern long faultin_page_range(struct mm_struct *mm, unsigned long start,
unsigned long end, bool write, int *locked);
+
+/*
+ * __mm_populate() and madvise_populate() fault in at most
+ * POPULATE_CHUNK_SIZE at a time, and handle memory.high overage between
+ * chunks with mmap_lock dropped, so other tasks don't wait for them.
+ */
+#define POPULATE_CHUNK_SIZE SZ_2M
+
+static inline unsigned long populate_chunk_end(unsigned long start,
+ unsigned long end)
+{
+ return min(end, round_down(start, POPULATE_CHUNK_SIZE) +
+ POPULATE_CHUNK_SIZE);
+}
+
+static inline void populate_handle_over_high(struct mm_struct *mm,
+ int *locked)
+{
+ if (!mem_cgroup_over_high_pending())
+ return;
+ if (*locked) {
+ mmap_read_unlock(mm);
+ *locked = 0;
+ }
+ mem_cgroup_handle_over_high(GFP_KERNEL);
+}
+
bool mlock_future_ok(const struct mm_struct *mm, bool is_vma_locked,
unsigned long bytes);
diff --git a/mm/madvise.c b/mm/madvise.c
index 2d54cb9863ea..c77924e52d79 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -992,8 +992,15 @@ static long madvise_populate(struct madvise_behavior *madv_behavior)
long pages;
while (start < end) {
- /* Populate (prefault) page tables readable/writable. */
- pages = faultin_page_range(mm, start, end, write, &locked);
+ /*
+ * Populate (prefault) page tables readable/writable, at most a
+ * chunk at a time, see POPULATE_CHUNK_SIZE.
+ */
+ pages = faultin_page_range(mm, start,
+ populate_chunk_end(start, end),
+ write, &locked);
+ if (pages >= 0)
+ populate_handle_over_high(mm, &locked);
if (!locked) {
mmap_read_lock(mm);
locked = 1;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads Shakeel Butt
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
A buffered read or write can copy almost 2GiB in one call. A write
charges the page cache it fills. Dirty throttling limits only the dirty
part, and does nothing on tmpfs and memfd, whose bdi has no
BDI_CAP_WRITEBACK. A read charges the page cache it reads, and the
anonymous pages it faults in when the user buffer is untouched, which
can't be reclaimed without swap. Today the charge path throttles these
calls for memory.high, except huge tmpfs pages and THP faults on the
user buffer, whose charges can't block under the default defrag
setting.
A later patch stops enforcing memory.high in the charge path, where
the task may hold locks other tasks need. Handle the overage in the
copy loops instead:
- generic_perform_write() and iomap_write_iter(): next to the dirty
throttling call.
- btrfs_buffered_write(): between ranges. btrfs has its own loop, which
the other two hooks don't cover.
- filemap_read(): after each folio is copied. Once per batch isn't
enough, as a batch of large folios can hold tens of MiB.
Writers hold the inode's i_rwsem there, and dirty throttling already
sleeps under it. Two cases hold more: splice() from a pipe also holds
the pipe's lock, and gfs2 holds the resource group and quota glocks in
writes that allocate, which other writers to that resource group wait
for, on any node. The charge path sleeps with all of these locks today,
plus those taken in write_begin(). In btrfs, copy_one_range() has
already released its space reservation or handed it to delalloc, and
dropped the snapshot lock of a NOCOW write.
Most readers hold no locks. XFS, NFS and netfs hold the file's lock
shared, and gfs2 and ocfs2 a cluster lock, so a writer to that file
waits while a reader is throttled, as it does today. filemap_read()
keeps its folio batch referenced while the task is throttled. If it
dropped the batch, reclaim could free the folios not copied yet, and
they would be read again: in a VM, a cold 256M read under
memory.high=64M then read five to nine times as much from disk for the
same progress. After the sleep, it reads i_size and
mapping_writably_mapped() again.
IOCB_NOWAIT writes and IOCB_NOWAIT or IOCB_WAITQ reads skip the hook:
io_uring issues them with ctx->uring_lock held. Reads into kernel
buffers skip it too: kernel_read() callers may hold other locks, like
cred_guard_mutex in execve(). Reclaim uses the gfp mask of the
allocation that follows, so filesystems that clear __GFP_FS get
GFP_NOFS reclaim.
Without these hooks, once the synchronous step is gone, a 256M write
under memory.high=64M in a VM reaches 257-264M on ext4, XFS and btrfs
instead of 64-67M.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/btrfs/file.c | 9 +++++++++
fs/iomap/buffered-io.c | 9 +++++++++
mm/filemap.c | 31 +++++++++++++++++++++++++++++++
3 files changed, 49 insertions(+)
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index f978c6524aa0..d5a00a26fba6 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -18,6 +18,7 @@
#include <linux/uio.h>
#include <linux/iversion.h>
#include <linux/fsverity.h>
+#include <linux/memcontrol.h>
#include "ctree.h"
#include "direct-io.h"
#include "disk-io.h"
@@ -1320,6 +1321,7 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)
struct file *file = iocb->ki_filp;
loff_t pos;
struct inode *inode = file_inode(file);
+ const gfp_t high_gfp = btrfs_alloc_write_mask(inode->i_mapping);
struct extent_changeset *data_reserved = NULL;
size_t num_written = 0;
ssize_t ret;
@@ -1356,6 +1358,13 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)
break;
pos += ret;
num_written += ret;
+ /*
+ * A large write can charge a lot before the task returns to
+ * userspace. Handle memory.high overage between ranges, where
+ * btrfs holds only the inode lock.
+ */
+ if (!nowait)
+ mem_cgroup_over_high_throttle(high_gfp);
cond_resched();
}
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 6d843b168d5f..861aa600b3af 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -7,6 +7,7 @@
#include <linux/buffer_head.h>
#include <linux/writeback.h>
#include <linux/swap.h>
+#include <linux/memcontrol.h>
#include <linux/migrate.h>
#include <linux/fserror.h>
#include <linux/fsverity.h>
@@ -1193,6 +1194,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
struct address_space *mapping = iter->inode->i_mapping;
size_t chunk = mapping_max_folio_size(mapping);
unsigned int bdp_flags = (iter->flags & IOMAP_NOWAIT) ? BDP_ASYNC : 0;
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
do {
struct folio *folio;
@@ -1212,6 +1214,13 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
if (unlikely(status))
break;
+ /*
+ * A large write can charge a lot before the task returns to
+ * userspace. Handle memory.high overage here too.
+ */
+ if (!(iter->flags & IOMAP_NOWAIT))
+ mem_cgroup_over_high_throttle(high_gfp);
+
if (bytes > iomap_length(iter))
bytes = iomap_length(iter);
diff --git a/mm/filemap.c b/mm/filemap.c
index dc0ceaa58a0f..b0cefe71c1fd 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2822,6 +2822,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
struct file_ra_state *ra = &filp->f_ra;
struct address_space *mapping = filp->f_mapping;
struct inode *inode = mapping->host;
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
struct folio_batch fbatch;
int i, error = 0;
bool writably_mapped;
@@ -2913,6 +2914,28 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
error = -EFAULT;
break;
}
+
+ /*
+ * A large read can charge a lot, for the page cache and
+ * for faulting in the buffer. Handle memory.high
+ * overage after each folio, as a batch of large folios
+ * can be big. Keep the batch, or reclaim could free
+ * folios not copied yet and they would be read again.
+ * The file may change meanwhile, so check its size and
+ * mappings again. Reads into kernel buffers skip this:
+ * their callers may hold other locks.
+ */
+ if (user_backed_iter(iter) &&
+ !(iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) &&
+ mem_cgroup_over_high_pending()) {
+ mem_cgroup_handle_over_high(high_gfp);
+ isize = i_size_read(inode);
+ if (iocb->ki_pos >= isize)
+ break;
+ end_offset = min_t(loff_t, isize,
+ iocb->ki_pos + iter->count);
+ writably_mapped = mapping_writably_mapped(mapping);
+ }
}
put_folios:
for (i = 0; i < folio_batch_count(&fbatch); i++) {
@@ -4385,6 +4408,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
struct address_space *mapping = file->f_mapping;
const struct address_space_operations *a_ops = mapping->a_ops;
size_t chunk = mapping_max_folio_size(mapping);
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
long status = 0;
ssize_t written = 0;
@@ -4401,6 +4425,13 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
bytes = min(chunk - offset, bytes);
balance_dirty_pages_ratelimited(mapping);
+ /*
+ * A large write can charge a lot before the task returns to
+ * userspace. Handle memory.high overage here too.
+ */
+ if (!(iocb->ki_flags & IOCB_NOWAIT))
+ mem_cgroup_over_high_throttle(high_gfp);
+
if (fatal_signal_pending(current)) {
status = -EINTR;
break;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (2 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE Shakeel Butt
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
fallocate() on tmpfs or memfd allocates the whole range in one call,
and a read can copy almost 2GiB, faulting in the user buffer and
swapping the file back in. Without swap, this memory can't be
reclaimed, and the internal mount behind memfd_create(),
MAP_SHARED|MAP_ANONYMOUS and SysV shm has no size limit. Today the
charge path throttles both for memory.high, except huge page charges,
which can't block under the default defrag setting. Writes go through
generic_perform_write(), which the previous patch covers.
A later patch stops enforcing memory.high in the charge path, where
the task may hold locks other tasks need. Handle the overage after each
fallocate iteration, where only inode_lock is held, and after each
folio a read copies, where no locks are held. This covers huge pages
too. Reclaim doesn't need inode_lock: shmem_writeout() reads i_private
under i_lock. As in filemap_read(), reads into kernel buffers and
IOCB_NOWAIT or IOCB_WAITQ reads skip it. Reclaim uses the mapping's gfp
constraint.
With swap, this reclaim finds the folios the fallocate added.
shmem_writeout() won't swap them out and counts each try in
nr_unswapped, and once there are more tries than new folios, the
fallocate fails with -ENOMEM. memory.high should throttle the call, not
fail it, so tries made by the fallocating task's own memory.high
reclaim don't count. Tries by other reclaim still count. In a VM with
swap, a 512M tmpfs fallocate under memory.high=64M fails with -ENOMEM 3
times out of 3 today with small pages, and would with huge pages too if
this reclaim's tries counted. With this patch it is throttled instead.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/shmem.c | 37 ++++++++++++++++++++++++++++++++++---
1 file changed, 34 insertions(+), 3 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 951bb3c8f837..fad86c96170f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -110,6 +110,8 @@ struct shmem_falloc {
pgoff_t next; /* the next page offset to be fallocated */
pgoff_t nr_falloced; /* how many new pages have been fallocated */
pgoff_t nr_unswapped; /* how often writeout refused to swap out */
+ /* set while the fallocating task handles memory.high overage */
+ struct task_struct *high_reclaimer;
};
struct shmem_options {
@@ -1918,10 +1920,13 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
if (shmem_falloc &&
!shmem_falloc->waitq &&
index >= shmem_falloc->start &&
- index < shmem_falloc->next)
- shmem_falloc->nr_unswapped += nr_pages;
- else
+ index < shmem_falloc->next) {
+ /* See the memory.high comment in shmem_fallocate() */
+ if (READ_ONCE(shmem_falloc->high_reclaimer) != current)
+ shmem_falloc->nr_unswapped += nr_pages;
+ } else {
shmem_falloc = NULL;
+ }
spin_unlock(&inode->i_lock);
if (shmem_falloc)
goto redirty;
@@ -3567,6 +3572,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
struct file *file = iocb->ki_filp;
struct inode *inode = file_inode(file);
struct address_space *mapping = inode->i_mapping;
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
pgoff_t index;
unsigned long offset;
int error = 0;
@@ -3675,6 +3681,15 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
error = -EFAULT;
break;
}
+ /*
+ * A large read can charge a lot, for faulting in the buffer
+ * and swapping in the file. Handle memory.high overage here,
+ * where no locks are held. As in filemap_read(), reads into
+ * kernel buffers and reads that must not block skip this.
+ */
+ if (user_backed_iter(to) &&
+ !(iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)))
+ mem_cgroup_over_high_throttle(high_gfp);
cond_resched();
}
@@ -3883,6 +3898,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
struct inode *inode = file_inode(file);
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
struct shmem_inode_info *info = SHMEM_I(inode);
+ gfp_t high_gfp = mapping_gfp_constraint(inode->i_mapping, GFP_KERNEL);
struct shmem_falloc shmem_falloc;
pgoff_t start, index, end, undo_fallocend;
loff_t aligned_end;
@@ -3962,6 +3978,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
shmem_falloc.next = start;
shmem_falloc.nr_falloced = 0;
shmem_falloc.nr_unswapped = 0;
+ shmem_falloc.high_reclaimer = NULL;
spin_lock(&inode->i_lock);
WRITE_ONCE(inode->i_private, &shmem_falloc);
spin_unlock(&inode->i_lock);
@@ -4031,6 +4048,20 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
folio_mark_dirty(folio);
folio_unlock(folio);
folio_put(folio);
+
+ /*
+ * Handle memory.high overage here, where only inode_lock is
+ * held; reclaim doesn't take it. shmem_writeout() won't swap
+ * out the folios added so far, and counts each try toward the
+ * -ENOMEM check above. memory.high should only throttle the
+ * fallocate, so tell shmem_writeout() not to count tries made
+ * by this reclaim.
+ */
+ if (mem_cgroup_over_high_pending()) {
+ WRITE_ONCE(shmem_falloc.high_reclaimer, current);
+ mem_cgroup_handle_over_high(high_gfp);
+ WRITE_ONCE(shmem_falloc.high_reclaimer, NULL);
+ }
cond_resched();
}
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (3 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds Shakeel Butt
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
MADV_COLLAPSE doesn't limit empty PTEs, so it can turn a PMD range with
one resident page into a 2MiB THP. Over a large sparse range, one
madvise() call can charge far more than the range used before. Today
the charge path throttles it for memory.high.
A later patch stops enforcing memory.high in the charge path, where
the task may hold locks other tasks need. Handle the overage right
after each collapse that dropped mmap_lock instead:
collapse_single_pmd() drops it to allocate and charge the THP. This
also covers the last PMD of each call, so madvise() over many small
VMAs, or a process_madvise() vector of small ranges, is throttled too.
process_madvise() on another process charges that process's cgroup.
If the caller is in that cgroup or below it, the caller handles the
overage here. Otherwise high_work reclaims it and no task is
throttled, as before.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/khugepaged.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 2dc4245814df..1ede7d30853d 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -3306,6 +3306,14 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
result = collapse_single_pmd(addr, vma, &mmap_unlocked, cc);
+ /*
+ * collapse_single_pmd() drops mmap_lock to allocate and charge
+ * a THP. Handle memory.high overage before taking it again,
+ * including after the last PMD.
+ */
+ if (mmap_unlocked)
+ mem_cgroup_over_high_throttle(GFP_KERNEL);
+
switch (result) {
case SCAN_SUCCEED:
case SCAN_PMD_MAPPED:
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (4 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds Shakeel Butt
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
A task stuck in memcg limit enforcement in the charge path keeps its
locks, and tasks in other cgroups that need them wait for a limit that
has nothing to do with them.
memory.max is a hard limit, so the charge path has to enforce it.
memory.high doesn't: since commit b23afb93d317 ("memcg: punt high
overage reclaim to return-to-userland path"), the charge only records
the overage, and the task reclaims and is throttled on return to
userspace, where it holds no locks. Avoiding this priority inversion
was one of the reasons for that commit.
But return to userspace is too late when one system call charges a
lot, like mlock() of a large range. So commit c9afe31ec443 ("memcg:
synchronously enforce memory.high for large overcharges") made a
charge that can block handle the overage right away once more than a
batch is pending. That step reclaims and sleeps, up to 2s per call,
under whatever locks the caller holds: mmap_lock while mlock() faults
in a MADV_HUGEPAGE range, or invalidate_lock and locked folios in
readahead. It also misses THP charges in other ranges, which can't
block under the default defrag setting.
The previous patches handle the overage where the task holds no locks,
or only the locks of its file, in the operations that charge the most
in one call: populating memory, shmem fallocate, buffered writes,
reads into user memory and MADV_COLLAPSE. Remove the synchronous step.
Other paths now handle the overage only on return to userspace, and
memory.max still limits them. Many can get a hook later. For example,
reading /dev/urandom into an untouched buffer reaches 513M under
memory.high=64M. With THP on, it did before too, but with small pages
the synchronous step used to throttle it. Another is mprotect() making
an mlocked private range writable, which copies its pages with
mmap_lock held for writing.
Update the memory.high documentation to say when tasks are throttled.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
Documentation/admin-guide/cgroup-v2.rst | 12 ++++++++++--
mm/memcontrol.c | 20 ++++++--------------
2 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 9e3c67b02ae2..215710a44a88 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1386,6 +1386,14 @@ write "max"; to set it to zero explicitly, write "0".
over the high boundary, the processes of the cgroup are
throttled and put under heavy reclaim pressure.
+ A task is throttled when it returns to userspace after
+ charging, and at some points in long operations, such as
+ populating memory or large buffered reads and writes. A
+ system call that allocates a lot without reaching such a
+ point can take the usage over the high boundary by up to
+ the amount it allocates. Kernel threads are not throttled:
+ a worker reclaims for them instead.
+
Going over the high limit never invokes the OOM killer and
under extreme conditions the limit may be breached. The high
limit should be used in scenarios where an external process
@@ -1396,8 +1404,8 @@ write "max"; to set it to zero explicitly, write "0".
reclaim is bypassed. This is useful for admin processes that
need to dynamically adjust the job's memory limits without
expending their own CPU resources on memory reclamation. The
- job will trigger the reclaim and/or get throttled on its
- next charge request.
+ job will reclaim and/or get throttled after it charges
+ memory again.
Please note that with O_NONBLOCK, there is a chance that the
target memory cgroup may take indefinite amount of time to
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6d147153bd06..4771730e017a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2615,8 +2615,9 @@ static unsigned long calculate_high_delay(unsigned int nr_pages,
}
/*
- * Handle pending high overage during a blocking charge or on return
- * to userspace.
+ * Handle pending high overage on return to userspace, or at a point in a
+ * long-running operation where the task can sleep (see
+ * mem_cgroup_over_high_throttle()).
*/
void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
{
@@ -2725,9 +2726,9 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,
!current->mm || !mm_match_cgroup(current->mm, memcg);
/*
- * Check the charged cgroup and its parents. Local charges normally defer
- * reclaim until return to userspace so it can use GFP_KERNEL. Large
- * pending overages may be handled synchronously below. The handler looks
+ * Check the charged cgroup and its parents. Local charges defer reclaim
+ * until return to userspace, or until a long-running operation reaches
+ * a point where it can sleep and calls the handler. The handler looks
* up current->mm and rechecks the limits. If the task moved to another
* cgroup in the meantime, the handler works on the new cgroup instead.
* That is rare and costs little.
@@ -2765,15 +2766,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,
current->memcg_nr_pages_over_high += nr_pages;
set_notify_resume(current);
}
-
- /*
- * Earlier charges may have left high overage pending even if this charge
- * found no new overage. Handle it now if this charge can block, to limit
- * overrun while the task remains in the kernel.
- */
- if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
- gfpflags_allow_blocking(gfp_mask))
- __mem_cgroup_handle_over_high(gfp_mask);
}
static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (5 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
6 siblings, 0 replies; 8+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
test_memcg_high_sync mlock()s 200M with memory.high=30M and
memory.max=140M, and expects memory.high to throttle the call before it
reaches memory.max. But it checked memory.events right after the first
high event, so it passed as soon as the handler ran once, even if the
rest of the call was not throttled.
That hid a real gap: with THP=always and the default defrag=madvise,
THP charges skipped the synchronous handler, and only a few page table
charges ran it. In a VM, usage reached 85M within 115ms and memory.max
about 2s later.
After the first event, check for 5 seconds that no max event shows up
and the child is still running, then that memory.current went up.
Counting high events isn't enough: one handler call can log many while
it retries reclaim. Also kill and reap the child, which may still be
throttled.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
.../selftests/cgroup/test_memcontrol.c | 39 +++++++++++++++----
1 file changed, 32 insertions(+), 7 deletions(-)
diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c
index bb673012dcf0..1af984761a02 100644
--- a/tools/testing/selftests/cgroup/test_memcontrol.c
+++ b/tools/testing/selftests/cgroup/test_memcontrol.c
@@ -771,14 +771,16 @@ static int alloc_anon_mlock(const char *cgroup, void *arg)
/*
* This test checks that memory.high is able to throttle big single shot
- * allocation i.e. large allocation within one kernel entry.
+ * allocation i.e. large allocation within one kernel entry. The name is
+ * from when this was done synchronously in the charge path.
*/
static int test_memcg_high_sync(const char *root)
{
- int ret = KSFT_FAIL, pid, fd = -1;
+ int ret = KSFT_FAIL, pid = -1, fd = -1, i;
char *memcg;
long pre_high, pre_max;
- long post_high, post_max;
+ long first_high, first_current;
+ long post_max, post_current;
memcg = cg_name(root, "memcg_test");
if (!memcg)
@@ -811,17 +813,40 @@ static int test_memcg_high_sync(const char *root)
cg_wait_for(fd);
- post_high = cg_read_key_long(memcg, "memory.events", "high ");
- post_max = cg_read_key_long(memcg, "memory.events", "max ");
- if (post_high < 0 || post_max < 0)
+ /*
+ * The first event only shows that throttling started. For 5 seconds,
+ * check that the child is still running and doesn't reach memory.max,
+ * then that memory.current went up, so it is throttled, not stuck.
+ * High events don't show progress: one handler call can log many.
+ */
+ first_high = cg_read_key_long(memcg, "memory.events", "high ");
+ first_current = cg_read_long(memcg, "memory.current");
+ if (first_high < 0 || first_high == pre_high || first_current < 0)
goto cleanup;
- if (pre_high == post_high || pre_max != post_max)
+ for (i = 0; i < 50; i++) {
+ usleep(100000);
+ post_max = cg_read_key_long(memcg, "memory.events", "max ");
+ if (post_max < 0 || post_max != pre_max)
+ goto cleanup;
+ if (waitpid(pid, NULL, WNOHANG) == pid) {
+ pid = -1;
+ goto cleanup;
+ }
+ }
+
+ post_current = cg_read_long(memcg, "memory.current");
+ if (post_current <= first_current)
goto cleanup;
ret = KSFT_PASS;
cleanup:
+ if (pid > 0) {
+ kill(pid, SIGKILL);
+ while (waitpid(pid, NULL, 0) < 0 && errno == EINTR)
+ ;
+ }
if (fd >= 0)
close(fd);
cg_destroy(memcg);
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread