* [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues
@ 2026-11-22 2:47 Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations Shakeel Butt
` (6 more replies)
0 siblings, 7 replies; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
(My initial plan [1, 2] was to track kernel lock holders, let them
bypass memcg limits and throttle them on unlock or on return to
userspace. I was asked to look for alternative solutions, and this
series is one.)
memcg limits can cause isolation problems. A task stuck in limit
enforcement in the charge path keeps holding its locks, and tasks in
other cgroups that need them wait for a limit that has nothing to do
with them. btrfs, for example, charges its tree blocks to the root
cgroup for this reason [3].
We see this in Meta's fleet. Over 28 days, on several tens of
thousands of unique hosts, a bpftrace script [4] recorded sleeping
locks held in memcg reclaim while another task waited for them. Most
belong to the reclaiming workload, like its mmap_lock (680,579
times, up to 4.7s), but some are shared by the whole machine or a
whole filesystem:
- cgroup_mutex, used by every cgroup change. mkdir charges the new
memcg to the parent while holding it, and reclaims if the parent is
at its limit: 350 times, up to 2.5s.
- kernfs_rwsem, used by cgroupfs lookups. The kernfs notification
worker held it while queueing inotify events charged to the
watcher's cgroup: 2,324 times, up to 3.3s.
- btrfs tree locks, used by every b-tree search and change. On kernels
with [3], the XArray nodes that index tree blocks are still charged
to the task, so tasks reclaimed with tree locks held 2,853 times,
850 of them with a root node locked, up to 2.6s.
Hung-task reports show worse cases. A container's systemd created a
cgroup while its own cgroup was over memory.high, and mkdir ran
memory.high reclaim with cgroup_mutex and the parent directory's lock
held for over 30 minutes. The host's systemd, tasks forking into
cgroups, oomd and a monitoring agent all waited, so the userspace OOM
killer could not even read the cgroup tree.
memory.max is a hard limit, so this series does not change it.
memory.high doesn't have to be enforced in the charge path: since
commit b23afb93d317 ("memcg: punt high overage reclaim to
return-to-userland path"), the charge only records the overage (usage
above the limit), and the task reclaims and is throttled on return to
userspace, where it holds no locks. Avoiding this priority inversion
was one of the reasons for that commit.
But return to userspace is too late when one system call charges a
lot, like mlock() of a large range. So commit c9afe31ec443 ("memcg:
synchronously enforce memory.high for large overcharges") handles the
overage in the charge itself when the charge can block and more than
a batch is pending. That brought the problem back: the task reclaims
and sleeps, up to 2s per call, under whatever locks it holds. It also
misses THP charges outside MADV_HUGEPAGE ranges, which can't block
under the default defrag setting. RESULTS shows both problems.
This series is a first step in rethinking the memcg high limit around
isolation. It makes memory.high enforcement asynchronous (to
allocation and charging) again and keeps it effective for large calls.
The task handles the overage on return to userspace, and also inside
the operations that can charge a lot in one call, where it holds no
locks or only the locks of the file it works on:
- Populating memory (mlock, mlockall, MAP_POPULATE, MAP_LOCKED,
MADV_POPULATE_*): 2M at a time, with mmap_lock dropped in between.
- tmpfs fallocate, buffered writes (generic_perform_write(), iomap,
btrfs) and reads into user memory (filemap_read(), shmem): once per
folio or range.
- MADV_COLLAPSE: after each collapse, with mmap_lock dropped.
These points also throttle THP charges, which the synchronous step
skips. A file's lock only blocks tasks using that file. Two cases hold
more: splice() from a pipe also holds the pipe's lock, and gfs2 holds
its allocation glocks in writes that allocate. The synchronous step
sleeps with these locks today too.
Why not keep the synchronous step, with a higher threshold, without
its sleep, or only outside these operations? It would still reclaim or
sleep under locks, like mmap_lock in THP faults and invalidate_lock in
readahead, and it would not close the THP gap.
This also lets users drop memory.max, which still reclaims in the
charge path and can stall lock holders as above. They can set only
memory.high and let a userspace OOM killer like oomd [5], or a future
BPF OOM policy [6], decide when to kill. Meta plans to do this.
The last two patches remove the synchronous step, update the
memory.high documentation and fix test_memcg_high_sync, which missed
the THP gap.
RESULTS
Tests run in a VM with 4 CPUs and THP on for anonymous memory and
shmem (enabled=always, shmem_enabled=always, tmpfs huge=always, the
default defrag=madvise), in a cgroup with memory.high=64M. Reads go
into an untouched buffer. "No hooks" is the base plus only the patch
that removes the synchronous step. Usage 18s into a 512M operation,
without swap:
test base series no hooks
mlock, mlockall, MAP_POPULATE, 513-514M 84M 513-514M
MADV_POPULATE_WRITE
tmpfs fallocate and write 512-513M 84M 512-513M
reads of a tmpfs file 513M 84-85M 513M
MADV_COLLAPSE of a sparse range 96M 98M 514M
(one VMA, or one VMA per PMD)
reads of /dev/zero or /dev/urandom 513M 513M 513M
(known gaps)
Usage grows by one 2M folio per throttle, so it is 84-85M after 18s.
On ext4, XFS and btrfs, 256M writes stay at 64-67M on both kernels
(257-264M with no hooks), and reads with a cold page cache at 70-73M.
Reads with a warm page cache, where only the buffer is charged, reach
257M on base and stay at 84-85M with the series.
With swap, the MAP_POPULATE, MADV_POPULATE_WRITE and tmpfs write tests
finish in under 2s at 65-66M with the series (base: 157-169M, no hooks:
513M). tmpfs fallocate is throttled with or without swap. With swap,
base fails it with -ENOMEM for small pages.
During mlock() of an 80M MADV_HUGEPAGE range, a task in another cgroup
that reads the process's /proc/<pid>/cmdline every 10ms waits up to
15.3s on base and under 1ms with the series. Populating 2G with no
limit takes the same time on both kernels.
All patches build with gcc and clang (W=1), with CONFIG_MEMCG=n and
with btrfs as a module. A lockdep kernel showed no warnings. The
memcontrol selftests pass, and the fixed test_memcg_high_sync fails on
base.
KNOWN GAPS
Other paths that charge a lot in one call are now throttled only on
return to userspace, and memory.max still limits them. Most can get a
throttle point later, some after dropping a lock:
- /dev/zero and /dev/urandom reads, getrandom(), sendfile() to a
socket, io_uring submissions, netfs and FUSE write-through writes
- MADV_WILLNEED, UFFDIO_COPY, KSM unmerge, mprotect() of mlocked
mappings
- FIDEDUPERANGE, FS_IOC_ENABLE_VERITY, KVM_PRE_FAULT_MEMORY,
guest_memfd fallocate, KVM page splitting
Rarer ones, like sendmmsg(), btrfs send and FITRIM, can wait. Memory
pinning (VFIO, iommufd, RDMA, io_uring buffers, direct I/O and
others), fork() and single large allocations (BPF maps and others)
hold a lock for the whole call or can't be split, so they stay as they
are.
[1] https://lore.kernel.org/20260924155025.949998-1-shakeel.butt@linux.dev/
[2] https://lore.kernel.org/20260921192559.2619635-1-shakeel.butt@linux.dev/
[3] commit cf1dec76ba8a ("mm/filemap: add AS_KERNEL_FILE")
[4] https://github.com/bpftrace/user-tools/tree/master/reclaimlockmonitor
[5] https://github.com/facebookincubator/oomd
[6] https://lore.kernel.org/20260127024421.494929-1-roman.gushchin@linux.dev
Based on mm-everything-2026-11-20-06-34.
Shakeel Butt (7):
memcg: add helpers to handle memory.high overage in long operations
mm/gup: handle memory.high overage while populating
mm, fs: handle memory.high overage in buffered reads and writes
mm/shmem: handle memory.high overage in fallocate and reads
mm/khugepaged: handle memory.high overage in MADV_COLLAPSE
memcg: remove synchronous memory.high enforcement
selftests/cgroup: make test_memcg_high_sync check that throttling
holds
Documentation/admin-guide/cgroup-v2.rst | 12 +++++-
fs/btrfs/file.c | 9 +++++
fs/iomap/buffered-io.c | 9 +++++
include/linux/memcontrol.h | 34 ++++++++++++++++
mm/filemap.c | 31 +++++++++++++++
mm/gup.c | 9 ++++-
mm/internal.h | 27 +++++++++++++
mm/khugepaged.c | 8 ++++
mm/madvise.c | 11 +++++-
mm/memcontrol.c | 21 ++++------
mm/shmem.c | 37 ++++++++++++++++--
.../selftests/cgroup/test_memcontrol.c | 39 +++++++++++++++----
12 files changed, 217 insertions(+), 30 deletions(-)
base-commit: b298839c2f77b60eea645a68d2245aa088463955
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 11+ messages in thread
* [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating Shakeel Butt
` (5 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
Since commit c9afe31ec443 ("memcg: synchronously enforce memory.high
for large overcharges"), a charge that can block handles memory.high
overage right away once more than a batch is pending. It reclaims and
sleeps with whatever locks the caller holds, so tasks in other cgroups
that need those locks wait too.
A later patch removes that step. Operations that can charge a lot in
one call then need another place to handle the overage: a point in
their loop where they hold no locks, or only the locks of their file.
Add two helpers for them. mem_cgroup_over_high_pending() says whether
more than a batch is pending, the threshold the synchronous step uses.
mem_cgroup_over_high_throttle() handles the overage if so; its comment
says where to call it and which gfp mask to pass. Export
__mem_cgroup_handle_over_high() for filesystems built as modules.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
include/linux/memcontrol.h | 34 ++++++++++++++++++++++++++++++++++
mm/memcontrol.c | 1 +
2 files changed, 35 insertions(+)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 980358bc6f49..77115fe1873e 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -942,6 +942,31 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
__mem_cgroup_handle_over_high(gfp_mask);
}
+/*
+ * True if the current task has more than a batch of memory.high or
+ * memory.swap.high overage pending. Each charge that finds overage
+ * counts at least a batch.
+ */
+static inline bool mem_cgroup_over_high_pending(void)
+{
+ return unlikely(current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH);
+}
+
+/*
+ * Handle the current task's memory.high overage if more than a batch is
+ * pending, for operations that can charge a lot before they return to
+ * userspace. The task may reclaim and sleep here, so call it only where
+ * other tasks won't wait for it: with no locks held anywhere up the call
+ * chain, or only the locks of the file being worked on. Pass GFP_KERNEL
+ * if no filesystem locks are held, and mapping_gfp_constraint() in file
+ * read and write paths.
+ */
+static inline void mem_cgroup_over_high_throttle(gfp_t gfp_mask)
+{
+ if (mem_cgroup_over_high_pending())
+ __mem_cgroup_handle_over_high(gfp_mask);
+}
+
unsigned long mem_cgroup_get_max(const struct mem_cgroup *memcg);
void mem_cgroup_print_oom_context(const struct mem_cgroup *memcg,
@@ -1376,6 +1401,15 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
{
}
+static inline bool mem_cgroup_over_high_pending(void)
+{
+ return false;
+}
+
+static inline void mem_cgroup_over_high_throttle(gfp_t gfp_mask)
+{
+}
+
static inline struct mem_cgroup *mem_cgroup_get_oom_group(
struct task_struct *victim, struct mem_cgroup *oom_domain)
{
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 091401cecf39..6d147153bd06 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2707,6 +2707,7 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
out:
css_put(&memcg->css);
}
+EXPORT_SYMBOL_GPL(__mem_cgroup_handle_over_high);
/*
* Called after @nr_pages were charged to @memcg. If @memcg or one of its
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 3:06 ` sashiko-bot
2026-11-22 2:47 ` [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes Shakeel Butt
` (4 subsequent siblings)
6 siblings, 1 reply; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
mlock(), mlockall(), mmap() with MAP_POPULATE or MAP_LOCKED, and
MADV_POPULATE_READ/WRITE can fault in a lot of memory in one call,
with mmap_lock held. When their charges can block, the charge path
throttles them for memory.high with mmap_lock still held. In a VM with
THP on, mlock() of an 80M MADV_HUGEPAGE range under memory.high=64M
made another thread wait up to 17.2s to mmap(), and a task in another
cgroup wait up to 15.3s to read the process's /proc/<pid>/cmdline.
THP charges in VMAs without MADV_HUGEPAGE can't block under the
default defrag setting, so they are not throttled at all: mlock(512M)
reached 513M.
Throttle these calls with mmap_lock dropped instead. The populate
loops, __mm_populate() and madvise_populate(), now fault in at most
2MiB per GUP call. Between calls, if more than a batch of overage is
pending, they drop mmap_lock, handle it, and go on. GUP itself doesn't
change. One fault can still charge a whole folio.
With this patch, mlock(512M) stays at 84M, and populating 2G with no
limit takes as long as before. The mmap_lock stall goes away when a
later patch removes the synchronous step.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/gup.c | 9 +++++++--
mm/internal.h | 27 +++++++++++++++++++++++++++
mm/madvise.c | 11 +++++++++--
3 files changed, 43 insertions(+), 4 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 7605da0e06d8..d93d89f6aad0 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1951,6 +1951,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
end = start + len;
for (nstart = start; nstart < end; nstart = nend) {
+ populate_handle_over_high(mm, &locked);
/*
* We want to fault in pages for [nstart; end) address range.
* Find first corresponding VMA.
@@ -1976,9 +1977,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
/*
* Now fault in a range of pages. populate_vma_page_range()
* double checks the vma flags, so that it won't mlock pages
- * if the vma was already munlocked.
+ * if the vma was already munlocked. Do one chunk at a time,
+ * see POPULATE_CHUNK_SIZE. On error, nend skips the rest of
+ * the VMA.
*/
- ret = populate_vma_page_range(vma, nstart, nend, &locked);
+ ret = populate_vma_page_range(vma, nstart,
+ populate_chunk_end(nstart, nend),
+ &locked);
if (ret < 0) {
if (ignore_errors) {
ret = 0;
diff --git a/mm/internal.h b/mm/internal.h
index de4593b3e3b6..12f7b4b1254f 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -948,6 +948,33 @@ extern long populate_vma_page_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end, int *locked);
extern long faultin_page_range(struct mm_struct *mm, unsigned long start,
unsigned long end, bool write, int *locked);
+
+/*
+ * __mm_populate() and madvise_populate() fault in at most
+ * POPULATE_CHUNK_SIZE at a time, and handle memory.high overage between
+ * chunks with mmap_lock dropped, so other tasks don't wait for them.
+ */
+#define POPULATE_CHUNK_SIZE SZ_2M
+
+static inline unsigned long populate_chunk_end(unsigned long start,
+ unsigned long end)
+{
+ return min(end, round_down(start, POPULATE_CHUNK_SIZE) +
+ POPULATE_CHUNK_SIZE);
+}
+
+static inline void populate_handle_over_high(struct mm_struct *mm,
+ int *locked)
+{
+ if (!mem_cgroup_over_high_pending())
+ return;
+ if (*locked) {
+ mmap_read_unlock(mm);
+ *locked = 0;
+ }
+ mem_cgroup_handle_over_high(GFP_KERNEL);
+}
+
bool mlock_future_ok(const struct mm_struct *mm, bool is_vma_locked,
unsigned long bytes);
diff --git a/mm/madvise.c b/mm/madvise.c
index 2d54cb9863ea..c77924e52d79 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -992,8 +992,15 @@ static long madvise_populate(struct madvise_behavior *madv_behavior)
long pages;
while (start < end) {
- /* Populate (prefault) page tables readable/writable. */
- pages = faultin_page_range(mm, start, end, write, &locked);
+ /*
+ * Populate (prefault) page tables readable/writable, at most a
+ * chunk at a time, see POPULATE_CHUNK_SIZE.
+ */
+ pages = faultin_page_range(mm, start,
+ populate_chunk_end(start, end),
+ write, &locked);
+ if (pages >= 0)
+ populate_handle_over_high(mm, &locked);
if (!locked) {
mmap_read_lock(mm);
locked = 1;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 3:06 ` sashiko-bot
2026-11-22 2:47 ` [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads Shakeel Butt
` (3 subsequent siblings)
6 siblings, 1 reply; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
A buffered read or write can copy almost 2GiB in one call. A write
charges the page cache it fills. Dirty throttling limits only the dirty
part, and does nothing on tmpfs and memfd, whose bdi has no
BDI_CAP_WRITEBACK. A read charges the page cache it reads, and the
anonymous pages it faults in when the user buffer is untouched, which
can't be reclaimed without swap. Today the charge path throttles these
calls for memory.high, except huge tmpfs pages and THP faults on the
user buffer, whose charges can't block under the default defrag
setting.
A later patch stops enforcing memory.high in the charge path, where
the task may hold locks other tasks need. Handle the overage in the
copy loops instead:
- generic_perform_write() and iomap_write_iter(): next to the dirty
throttling call.
- btrfs_buffered_write(): between ranges. btrfs has its own loop, which
the other two hooks don't cover.
- filemap_read(): after each folio is copied. Once per batch isn't
enough, as a batch of large folios can hold tens of MiB.
Writers hold the inode's i_rwsem there, and dirty throttling already
sleeps under it. Two cases hold more: splice() from a pipe also holds
the pipe's lock, and gfs2 holds the resource group and quota glocks in
writes that allocate, which other writers to that resource group wait
for, on any node. The charge path sleeps with all of these locks today,
plus those taken in write_begin(). In btrfs, copy_one_range() has
already released its space reservation or handed it to delalloc, and
dropped the snapshot lock of a NOCOW write.
Most readers hold no locks. XFS, NFS and netfs hold the file's lock
shared, and gfs2 and ocfs2 a cluster lock, so a writer to that file
waits while a reader is throttled, as it does today. filemap_read()
keeps its folio batch referenced while the task is throttled. If it
dropped the batch, reclaim could free the folios not copied yet, and
they would be read again: in a VM, a cold 256M read under
memory.high=64M then read five to nine times as much from disk for the
same progress. After the sleep, it reads i_size and
mapping_writably_mapped() again.
IOCB_NOWAIT writes and IOCB_NOWAIT or IOCB_WAITQ reads skip the hook:
io_uring issues them with ctx->uring_lock held. Reads into kernel
buffers skip it too: kernel_read() callers may hold other locks, like
cred_guard_mutex in execve(). Reclaim uses the gfp mask of the
allocation that follows, so filesystems that clear __GFP_FS get
GFP_NOFS reclaim.
Without these hooks, once the synchronous step is gone, a 256M write
under memory.high=64M in a VM reaches 257-264M on ext4, XFS and btrfs
instead of 64-67M.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/btrfs/file.c | 9 +++++++++
fs/iomap/buffered-io.c | 9 +++++++++
mm/filemap.c | 31 +++++++++++++++++++++++++++++++
3 files changed, 49 insertions(+)
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index f978c6524aa0..d5a00a26fba6 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -18,6 +18,7 @@
#include <linux/uio.h>
#include <linux/iversion.h>
#include <linux/fsverity.h>
+#include <linux/memcontrol.h>
#include "ctree.h"
#include "direct-io.h"
#include "disk-io.h"
@@ -1320,6 +1321,7 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)
struct file *file = iocb->ki_filp;
loff_t pos;
struct inode *inode = file_inode(file);
+ const gfp_t high_gfp = btrfs_alloc_write_mask(inode->i_mapping);
struct extent_changeset *data_reserved = NULL;
size_t num_written = 0;
ssize_t ret;
@@ -1356,6 +1358,13 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)
break;
pos += ret;
num_written += ret;
+ /*
+ * A large write can charge a lot before the task returns to
+ * userspace. Handle memory.high overage between ranges, where
+ * btrfs holds only the inode lock.
+ */
+ if (!nowait)
+ mem_cgroup_over_high_throttle(high_gfp);
cond_resched();
}
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 6d843b168d5f..861aa600b3af 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -7,6 +7,7 @@
#include <linux/buffer_head.h>
#include <linux/writeback.h>
#include <linux/swap.h>
+#include <linux/memcontrol.h>
#include <linux/migrate.h>
#include <linux/fserror.h>
#include <linux/fsverity.h>
@@ -1193,6 +1194,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
struct address_space *mapping = iter->inode->i_mapping;
size_t chunk = mapping_max_folio_size(mapping);
unsigned int bdp_flags = (iter->flags & IOMAP_NOWAIT) ? BDP_ASYNC : 0;
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
do {
struct folio *folio;
@@ -1212,6 +1214,13 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
if (unlikely(status))
break;
+ /*
+ * A large write can charge a lot before the task returns to
+ * userspace. Handle memory.high overage here too.
+ */
+ if (!(iter->flags & IOMAP_NOWAIT))
+ mem_cgroup_over_high_throttle(high_gfp);
+
if (bytes > iomap_length(iter))
bytes = iomap_length(iter);
diff --git a/mm/filemap.c b/mm/filemap.c
index dc0ceaa58a0f..b0cefe71c1fd 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2822,6 +2822,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
struct file_ra_state *ra = &filp->f_ra;
struct address_space *mapping = filp->f_mapping;
struct inode *inode = mapping->host;
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
struct folio_batch fbatch;
int i, error = 0;
bool writably_mapped;
@@ -2913,6 +2914,28 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
error = -EFAULT;
break;
}
+
+ /*
+ * A large read can charge a lot, for the page cache and
+ * for faulting in the buffer. Handle memory.high
+ * overage after each folio, as a batch of large folios
+ * can be big. Keep the batch, or reclaim could free
+ * folios not copied yet and they would be read again.
+ * The file may change meanwhile, so check its size and
+ * mappings again. Reads into kernel buffers skip this:
+ * their callers may hold other locks.
+ */
+ if (user_backed_iter(iter) &&
+ !(iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) &&
+ mem_cgroup_over_high_pending()) {
+ mem_cgroup_handle_over_high(high_gfp);
+ isize = i_size_read(inode);
+ if (iocb->ki_pos >= isize)
+ break;
+ end_offset = min_t(loff_t, isize,
+ iocb->ki_pos + iter->count);
+ writably_mapped = mapping_writably_mapped(mapping);
+ }
}
put_folios:
for (i = 0; i < folio_batch_count(&fbatch); i++) {
@@ -4385,6 +4408,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
struct address_space *mapping = file->f_mapping;
const struct address_space_operations *a_ops = mapping->a_ops;
size_t chunk = mapping_max_folio_size(mapping);
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
long status = 0;
ssize_t written = 0;
@@ -4401,6 +4425,13 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
bytes = min(chunk - offset, bytes);
balance_dirty_pages_ratelimited(mapping);
+ /*
+ * A large write can charge a lot before the task returns to
+ * userspace. Handle memory.high overage here too.
+ */
+ if (!(iocb->ki_flags & IOCB_NOWAIT))
+ mem_cgroup_over_high_throttle(high_gfp);
+
if (fatal_signal_pending(current)) {
status = -EINTR;
break;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (2 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE Shakeel Butt
` (2 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
fallocate() on tmpfs or memfd allocates the whole range in one call,
and a read can copy almost 2GiB, faulting in the user buffer and
swapping the file back in. Without swap, this memory can't be
reclaimed, and the internal mount behind memfd_create(),
MAP_SHARED|MAP_ANONYMOUS and SysV shm has no size limit. Today the
charge path throttles both for memory.high, except huge page charges,
which can't block under the default defrag setting. Writes go through
generic_perform_write(), which the previous patch covers.
A later patch stops enforcing memory.high in the charge path, where
the task may hold locks other tasks need. Handle the overage after each
fallocate iteration, where only inode_lock is held, and after each
folio a read copies, where no locks are held. This covers huge pages
too. Reclaim doesn't need inode_lock: shmem_writeout() reads i_private
under i_lock. As in filemap_read(), reads into kernel buffers and
IOCB_NOWAIT or IOCB_WAITQ reads skip it. Reclaim uses the mapping's gfp
constraint.
With swap, this reclaim finds the folios the fallocate added.
shmem_writeout() won't swap them out and counts each try in
nr_unswapped, and once there are more tries than new folios, the
fallocate fails with -ENOMEM. memory.high should throttle the call, not
fail it, so tries made by the fallocating task's own memory.high
reclaim don't count. Tries by other reclaim still count. In a VM with
swap, a 512M tmpfs fallocate under memory.high=64M fails with -ENOMEM 3
times out of 3 today with small pages, and would with huge pages too if
this reclaim's tries counted. With this patch it is throttled instead.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/shmem.c | 37 ++++++++++++++++++++++++++++++++++---
1 file changed, 34 insertions(+), 3 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 951bb3c8f837..fad86c96170f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -110,6 +110,8 @@ struct shmem_falloc {
pgoff_t next; /* the next page offset to be fallocated */
pgoff_t nr_falloced; /* how many new pages have been fallocated */
pgoff_t nr_unswapped; /* how often writeout refused to swap out */
+ /* set while the fallocating task handles memory.high overage */
+ struct task_struct *high_reclaimer;
};
struct shmem_options {
@@ -1918,10 +1920,13 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
if (shmem_falloc &&
!shmem_falloc->waitq &&
index >= shmem_falloc->start &&
- index < shmem_falloc->next)
- shmem_falloc->nr_unswapped += nr_pages;
- else
+ index < shmem_falloc->next) {
+ /* See the memory.high comment in shmem_fallocate() */
+ if (READ_ONCE(shmem_falloc->high_reclaimer) != current)
+ shmem_falloc->nr_unswapped += nr_pages;
+ } else {
shmem_falloc = NULL;
+ }
spin_unlock(&inode->i_lock);
if (shmem_falloc)
goto redirty;
@@ -3567,6 +3572,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
struct file *file = iocb->ki_filp;
struct inode *inode = file_inode(file);
struct address_space *mapping = inode->i_mapping;
+ gfp_t high_gfp = mapping_gfp_constraint(mapping, GFP_KERNEL);
pgoff_t index;
unsigned long offset;
int error = 0;
@@ -3675,6 +3681,15 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
error = -EFAULT;
break;
}
+ /*
+ * A large read can charge a lot, for faulting in the buffer
+ * and swapping in the file. Handle memory.high overage here,
+ * where no locks are held. As in filemap_read(), reads into
+ * kernel buffers and reads that must not block skip this.
+ */
+ if (user_backed_iter(to) &&
+ !(iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)))
+ mem_cgroup_over_high_throttle(high_gfp);
cond_resched();
}
@@ -3883,6 +3898,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
struct inode *inode = file_inode(file);
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
struct shmem_inode_info *info = SHMEM_I(inode);
+ gfp_t high_gfp = mapping_gfp_constraint(inode->i_mapping, GFP_KERNEL);
struct shmem_falloc shmem_falloc;
pgoff_t start, index, end, undo_fallocend;
loff_t aligned_end;
@@ -3962,6 +3978,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
shmem_falloc.next = start;
shmem_falloc.nr_falloced = 0;
shmem_falloc.nr_unswapped = 0;
+ shmem_falloc.high_reclaimer = NULL;
spin_lock(&inode->i_lock);
WRITE_ONCE(inode->i_private, &shmem_falloc);
spin_unlock(&inode->i_lock);
@@ -4031,6 +4048,20 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
folio_mark_dirty(folio);
folio_unlock(folio);
folio_put(folio);
+
+ /*
+ * Handle memory.high overage here, where only inode_lock is
+ * held; reclaim doesn't take it. shmem_writeout() won't swap
+ * out the folios added so far, and counts each try toward the
+ * -ENOMEM check above. memory.high should only throttle the
+ * fallocate, so tell shmem_writeout() not to count tries made
+ * by this reclaim.
+ */
+ if (mem_cgroup_over_high_pending()) {
+ WRITE_ONCE(shmem_falloc.high_reclaimer, current);
+ mem_cgroup_handle_over_high(high_gfp);
+ WRITE_ONCE(shmem_falloc.high_reclaimer, NULL);
+ }
cond_resched();
}
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (3 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds Shakeel Butt
6 siblings, 0 replies; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
MADV_COLLAPSE doesn't limit empty PTEs, so it can turn a PMD range with
one resident page into a 2MiB THP. Over a large sparse range, one
madvise() call can charge far more than the range used before. Today
the charge path throttles it for memory.high.
A later patch stops enforcing memory.high in the charge path, where
the task may hold locks other tasks need. Handle the overage right
after each collapse that dropped mmap_lock instead:
collapse_single_pmd() drops it to allocate and charge the THP. This
also covers the last PMD of each call, so madvise() over many small
VMAs, or a process_madvise() vector of small ranges, is throttled too.
process_madvise() on another process charges that process's cgroup.
If the caller is in that cgroup or below it, the caller handles the
overage here. Otherwise high_work reclaims it and no task is
throttled, as before.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/khugepaged.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 2dc4245814df..1ede7d30853d 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -3306,6 +3306,14 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
result = collapse_single_pmd(addr, vma, &mmap_unlocked, cc);
+ /*
+ * collapse_single_pmd() drops mmap_lock to allocate and charge
+ * a THP. Handle memory.high overage before taking it again,
+ * including after the last PMD.
+ */
+ if (mmap_unlocked)
+ mem_cgroup_over_high_throttle(GFP_KERNEL);
+
switch (result) {
case SCAN_SUCCEED:
case SCAN_PMD_MAPPED:
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (4 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
2026-11-22 3:06 ` sashiko-bot
2026-11-22 2:47 ` [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds Shakeel Butt
6 siblings, 1 reply; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
A task stuck in memcg limit enforcement in the charge path keeps its
locks, and tasks in other cgroups that need them wait for a limit that
has nothing to do with them.
memory.max is a hard limit, so the charge path has to enforce it.
memory.high doesn't: since commit b23afb93d317 ("memcg: punt high
overage reclaim to return-to-userland path"), the charge only records
the overage, and the task reclaims and is throttled on return to
userspace, where it holds no locks. Avoiding this priority inversion
was one of the reasons for that commit.
But return to userspace is too late when one system call charges a
lot, like mlock() of a large range. So commit c9afe31ec443 ("memcg:
synchronously enforce memory.high for large overcharges") made a
charge that can block handle the overage right away once more than a
batch is pending. That step reclaims and sleeps, up to 2s per call,
under whatever locks the caller holds: mmap_lock while mlock() faults
in a MADV_HUGEPAGE range, or invalidate_lock and locked folios in
readahead. It also misses THP charges in other ranges, which can't
block under the default defrag setting.
The previous patches handle the overage where the task holds no locks,
or only the locks of its file, in the operations that charge the most
in one call: populating memory, shmem fallocate, buffered writes,
reads into user memory and MADV_COLLAPSE. Remove the synchronous step.
Other paths now handle the overage only on return to userspace, and
memory.max still limits them. Many can get a hook later. For example,
reading /dev/urandom into an untouched buffer reaches 513M under
memory.high=64M. With THP on, it did before too, but with small pages
the synchronous step used to throttle it. Another is mprotect() making
an mlocked private range writable, which copies its pages with
mmap_lock held for writing.
Update the memory.high documentation to say when tasks are throttled.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
Documentation/admin-guide/cgroup-v2.rst | 12 ++++++++++--
mm/memcontrol.c | 20 ++++++--------------
2 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 9e3c67b02ae2..215710a44a88 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1386,6 +1386,14 @@ write "max"; to set it to zero explicitly, write "0".
over the high boundary, the processes of the cgroup are
throttled and put under heavy reclaim pressure.
+ A task is throttled when it returns to userspace after
+ charging, and at some points in long operations, such as
+ populating memory or large buffered reads and writes. A
+ system call that allocates a lot without reaching such a
+ point can take the usage over the high boundary by up to
+ the amount it allocates. Kernel threads are not throttled:
+ a worker reclaims for them instead.
+
Going over the high limit never invokes the OOM killer and
under extreme conditions the limit may be breached. The high
limit should be used in scenarios where an external process
@@ -1396,8 +1404,8 @@ write "max"; to set it to zero explicitly, write "0".
reclaim is bypassed. This is useful for admin processes that
need to dynamically adjust the job's memory limits without
expending their own CPU resources on memory reclamation. The
- job will trigger the reclaim and/or get throttled on its
- next charge request.
+ job will reclaim and/or get throttled after it charges
+ memory again.
Please note that with O_NONBLOCK, there is a chance that the
target memory cgroup may take indefinite amount of time to
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6d147153bd06..4771730e017a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2615,8 +2615,9 @@ static unsigned long calculate_high_delay(unsigned int nr_pages,
}
/*
- * Handle pending high overage during a blocking charge or on return
- * to userspace.
+ * Handle pending high overage on return to userspace, or at a point in a
+ * long-running operation where the task can sleep (see
+ * mem_cgroup_over_high_throttle()).
*/
void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
{
@@ -2725,9 +2726,9 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,
!current->mm || !mm_match_cgroup(current->mm, memcg);
/*
- * Check the charged cgroup and its parents. Local charges normally defer
- * reclaim until return to userspace so it can use GFP_KERNEL. Large
- * pending overages may be handled synchronously below. The handler looks
+ * Check the charged cgroup and its parents. Local charges defer reclaim
+ * until return to userspace, or until a long-running operation reaches
+ * a point where it can sleep and calls the handler. The handler looks
* up current->mm and rechecks the limits. If the task moved to another
* cgroup in the meantime, the handler works on the new cgroup instead.
* That is rare and costs little.
@@ -2765,15 +2766,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,
current->memcg_nr_pages_over_high += nr_pages;
set_notify_resume(current);
}
-
- /*
- * Earlier charges may have left high overage pending even if this charge
- * found no new overage. Handle it now if this charge can block, to limit
- * overrun while the task remains in the kernel.
- */
- if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
- gfpflags_allow_blocking(gfp_mask))
- __mem_cgroup_handle_over_high(gfp_mask);
}
static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
` (5 preceding siblings ...)
2026-11-22 2:47 ` [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement Shakeel Butt
@ 2026-11-22 2:47 ` Shakeel Butt
6 siblings, 0 replies; 11+ messages in thread
From: Shakeel Butt @ 2026-11-22 2:47 UTC (permalink / raw)
To: Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, cgroups, linux-mm,
linux-fsdevel, linux-btrfs, linux-kernel
test_memcg_high_sync mlock()s 200M with memory.high=30M and
memory.max=140M, and expects memory.high to throttle the call before it
reaches memory.max. But it checked memory.events right after the first
high event, so it passed as soon as the handler ran once, even if the
rest of the call was not throttled.
That hid a real gap: with THP=always and the default defrag=madvise,
THP charges skipped the synchronous handler, and only a few page table
charges ran it. In a VM, usage reached 85M within 115ms and memory.max
about 2s later.
After the first event, check for 5 seconds that no max event shows up
and the child is still running, then that memory.current went up.
Counting high events isn't enough: one handler call can log many while
it retries reclaim. Also kill and reap the child, which may still be
throttled.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
.../selftests/cgroup/test_memcontrol.c | 39 +++++++++++++++----
1 file changed, 32 insertions(+), 7 deletions(-)
diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c
index bb673012dcf0..1af984761a02 100644
--- a/tools/testing/selftests/cgroup/test_memcontrol.c
+++ b/tools/testing/selftests/cgroup/test_memcontrol.c
@@ -771,14 +771,16 @@ static int alloc_anon_mlock(const char *cgroup, void *arg)
/*
* This test checks that memory.high is able to throttle big single shot
- * allocation i.e. large allocation within one kernel entry.
+ * allocation i.e. large allocation within one kernel entry. The name is
+ * from when this was done synchronously in the charge path.
*/
static int test_memcg_high_sync(const char *root)
{
- int ret = KSFT_FAIL, pid, fd = -1;
+ int ret = KSFT_FAIL, pid = -1, fd = -1, i;
char *memcg;
long pre_high, pre_max;
- long post_high, post_max;
+ long first_high, first_current;
+ long post_max, post_current;
memcg = cg_name(root, "memcg_test");
if (!memcg)
@@ -811,17 +813,40 @@ static int test_memcg_high_sync(const char *root)
cg_wait_for(fd);
- post_high = cg_read_key_long(memcg, "memory.events", "high ");
- post_max = cg_read_key_long(memcg, "memory.events", "max ");
- if (post_high < 0 || post_max < 0)
+ /*
+ * The first event only shows that throttling started. For 5 seconds,
+ * check that the child is still running and doesn't reach memory.max,
+ * then that memory.current went up, so it is throttled, not stuck.
+ * High events don't show progress: one handler call can log many.
+ */
+ first_high = cg_read_key_long(memcg, "memory.events", "high ");
+ first_current = cg_read_long(memcg, "memory.current");
+ if (first_high < 0 || first_high == pre_high || first_current < 0)
goto cleanup;
- if (pre_high == post_high || pre_max != post_max)
+ for (i = 0; i < 50; i++) {
+ usleep(100000);
+ post_max = cg_read_key_long(memcg, "memory.events", "max ");
+ if (post_max < 0 || post_max != pre_max)
+ goto cleanup;
+ if (waitpid(pid, NULL, WNOHANG) == pid) {
+ pid = -1;
+ goto cleanup;
+ }
+ }
+
+ post_current = cg_read_long(memcg, "memory.current");
+ if (post_current <= first_current)
goto cleanup;
ret = KSFT_PASS;
cleanup:
+ if (pid > 0) {
+ kill(pid, SIGKILL);
+ while (waitpid(pid, NULL, 0) < 0 && errno == EINTR)
+ ;
+ }
if (fd >= 0)
close(fd);
cg_destroy(memcg);
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating
2026-11-22 2:47 ` [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating Shakeel Butt
@ 2026-11-22 3:06 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-11-22 3:06 UTC (permalink / raw)
To: Shakeel Butt, Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, linux-mm, dsterba
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] mm: unsigned long overflow to 0 in populate_chunk_end() near
top of address space
--
commit ec48ff5e44f8914e2b82f0cee190a74feb7645ec
Author: Shakeel Butt <shakeel.butt@linux.dev>
mm/gup: handle memory.high overage while populating
This commit chunks page fault-in loops in __mm_populate() and
madvise_populate() into 2 MiB ranges and drops mmap_lock between chunks to
handle pending memory.high overage.
> diff --git a/mm/internal.h b/mm/internal.h
> index de4593b3e3b60..12f7b4b1254fe 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -948,6 +948,33 @@ extern long populate_vma_page_range(struct vm_area_struct *vma,
> unsigned long start, unsigned long end, int *locked);
> extern long faultin_page_range(struct mm_struct *mm, unsigned long start,
> unsigned long end, bool write, int *locked);
> +
> +/*
> + * __mm_populate() and madvise_populate() fault in at most
> + * POPULATE_CHUNK_SIZE at a time, and handle memory.high overage between
> + * chunks with mmap_lock dropped, so other tasks don't wait for them.
> + */
> +#define POPULATE_CHUNK_SIZE SZ_2M
> +
> +static inline unsigned long populate_chunk_end(unsigned long start,
> + unsigned long end)
> +{
> + return min(end, round_down(start, POPULATE_CHUNK_SIZE) +
> + POPULATE_CHUNK_SIZE);
> +}
[Severity: High]
Can populate_chunk_end() overflow to 0 when start is within the top
POPULATE_CHUNK_SIZE (2 MiB) of the unsigned long address space?
When start >= ULONG_MAX - POPULATE_CHUNK_SIZE + 1,
round_down(start, POPULATE_CHUNK_SIZE) + POPULATE_CHUNK_SIZE wraps to 0, so
min(end, 0) returns 0. Passing 0 as end to populate_vma_page_range() or
faultin_page_range() underflows nr_pages = (end - start) / PAGE_SIZE to
(0 - start) / PAGE_SIZE, faulting pages past the requested end boundary. If a
VMA in that range has VM_LOCKONFAULT set (for example, on 32-bit parisc or
64-bit s390 where TASK_SIZE reaches the top 2 MiB), populate_vma_page_range()
returns that wrapped nr_pages and __mm_populate() computes
nend = nstart + ret * PAGE_SIZE == 0, looping forever from address 0 with
mmap_read_lock(mm) held.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261122024737.2741569-1-shakeel.butt@linux.dev?part=2
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes
2026-11-22 2:47 ` [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes Shakeel Butt
@ 2026-11-22 3:06 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-11-22 3:06 UTC (permalink / raw)
To: Shakeel Butt, Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, linux-mm, dsterba
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] mm/filemap: read corruption in filemap_read() if i_size grows
during throttling
--
commit 0c9715c2d32c8a6aa3086d8505b1f86c7ebcf8cb
Author: Shakeel Butt <shakeel.butt@linux.dev>
mm, fs: handle memory.high overage in buffered reads and writes
This commit adds memory.high throttling hooks to buffered read and write
copy loops so overage is handled outside the memcg charge path.
> diff --git a/mm/filemap.c b/mm/filemap.c
> index dc0ceaa58a0f7..b0cefe71c1fdd 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
[ ... ]
> @@ -2913,6 +2914,28 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
> error = -EFAULT;
> break;
> }
> +
> + /*
> + * A large read can charge a lot, for the page cache and
> + * for faulting in the buffer. Handle memory.high
> + * overage after each folio, as a batch of large folios
> + * can be big. Keep the batch, or reclaim could free
> + * folios not copied yet and they would be read again.
> + * The file may change meanwhile, so check its size and
> + * mappings again. Reads into kernel buffers skip this:
> + * their callers may hold other locks.
> + */
> + if (user_backed_iter(iter) &&
> + !(iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) &&
> + mem_cgroup_over_high_pending()) {
> + mem_cgroup_handle_over_high(high_gfp);
> + isize = i_size_read(inode);
> + if (iocb->ki_pos >= isize)
> + break;
> + end_offset = min_t(loff_t, isize,
> + iocb->ki_pos + iter->count);
> + writably_mapped = mapping_writably_mapped(mapping);
> + }
> }
[Severity: High]
Can refreshing isize and end_offset here corrupt the read buffer if
i_size grows while the reader is throttled?
filemap_get_pages() does not clamp the batch to i_size, so fbatch can
already hold uptodate folios past the initial end_offset (for example,
from a concurrent append that marked fbatch.folios[i + 1] uptodate
before calling i_size_write(), or after fallocate(FALLOC_FL_KEEP_SIZE)):
mm/filemap.c:filemap_read() {
...
for (i = 0; i < folio_batch_count(&fbatch); i++) {
struct folio *folio = fbatch.folios[i];
size_t fsize = folio_size(folio);
size_t offset = iocb->ki_pos & (fsize - 1);
size_t bytes = min_t(loff_t, end_offset - iocb->ki_pos,
fsize - offset);
size_t copied;
if (end_offset < folio_pos(folio))
break;
...
}
When end_offset lies inside fbatch.folios[i], iteration i copies only up
to end_offset (offset + copied < fsize), leaving iocb->ki_pos inside
fbatch.folios[i]. Normally the next iteration immediately breaks out of
the loop because end_offset < folio_pos(fbatch.folios[i + 1]).
If mem_cgroup_handle_over_high() runs at the end of iteration i and a
concurrent writer extends i_size past folio_pos(fbatch.folios[i + 1]),
end_offset is raised past folio_pos(fbatch.folios[i + 1]). The loop
then advances to fbatch.folios[i + 1] while iocb->ki_pos still points
into fbatch.folios[i], skipping the rest of fbatch.folios[i] and copying
from fbatch.folios[i + 1] at the wrong file offset.
Breaking out of the fbatch loop when offset + copied < fsize (or before
re-evaluating a larger end_offset) would keep iocb->ki_pos aligned with
the current folio.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261122024737.2741569-1-shakeel.butt@linux.dev?part=3
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement
2026-11-22 2:47 ` [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement Shakeel Butt
@ 2026-11-22 3:06 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-11-22 3:06 UTC (permalink / raw)
To: Shakeel Butt, Tejun Heo, Andrew Morton
Cc: Johannes Weiner, Michal Koutny, Christian Brauner, Michal Hocko,
Roman Gushchin, Muchun Song, Jan Kara, Matthew Wilcox,
David Hildenbrand, Vlastimil Babka, Zi Yan, Baolin Wang,
Nico Pache, Usama Arif, Kiryl Shutsemau, Lorenzo Stoakes,
Suren Baghdasaryan, Meta kernel team, linux-mm, dsterba
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] memcg: memcg_enforce_high() leaves vhost_task PF_USER_WORKER
threads unthrottled
--
commit dfb4aaa1098cc7070f618d5995322b418d647d3c
Author: Shakeel Butt <shakeel.butt@linux.dev>
memcg: remove synchronous memory.high enforcement
This commit removes synchronous memory.high enforcement from
memcg_enforce_high() so that pending high overages are handled on return
to userspace or at explicit throttling points in long-running operations,
and updates the cgroup v2 documentation accordingly.
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 6d147153bd06b..4771730e017a5 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
[ ... ]
> @@ -2765,15 +2766,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages,
> current->memcg_nr_pages_over_high += nr_pages;
> set_notify_resume(current);
> }
> -
> - /*
> - * Earlier charges may have left high overage pending even if this charge
> - * found no new overage. Handle it now if this charge can block, to limit
> - * overrun while the task remains in the kernel.
> - */
> - if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
> - gfpflags_allow_blocking(gfp_mask))
> - __mem_cgroup_handle_over_high(gfp_mask);
> }
[Severity: High]
Does removing the synchronous __mem_cgroup_handle_over_high() call leave
vhost_task worker threads completely unthrottled by memory.high and
memory.swap.high?
vhost_task_create() clones workers with CLONE_VM and args.user_worker = 1,
setting PF_USER_WORKER rather than PF_KTHREAD while sharing the owner's mm.
In memcg_enforce_high(), use_high_work checks current->flags & PF_KTHREAD,
so it evaluates to false for vhost workers and only increments
current->memcg_nr_pages_over_high and calls set_notify_resume(current).
Unlike io_uring PF_USER_WORKER threads, which check TIF_NOTIFY_RESUME and
call resume_user_mode_work(), vhost_task_fn() loops in kernel space until
exit without ever returning to userspace or handling notify-resume work:
kernel/vhost_task.c:vhost_task_fn() {
for (;;) {
...
did_work = vtsk->fn(vtsk->data);
if (!did_work)
schedule();
}
do_exit(0);
}
Previously, blocking charges from vhost workers (such as guest memory
faults or skb allocations) were caught by the synchronous
__mem_cgroup_handle_over_high() check here once memcg_nr_pages_over_high
exceeded MEMCG_CHARGE_BATCH. Without it, vhost workers neither schedule
high_work nor run return-to-userspace reclaim, bypassing memory.high and
memory.swap.high for the lifetime of the worker while
memcg_nr_pages_over_high overflows.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261122024737.2741569-1-shakeel.butt@linux.dev?part=6
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2026-11-22 3:06 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-22 2:47 [RFC PATCH 0/7] memcg: effective high limit that avoids isolation issues Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 1/7] memcg: add helpers to handle memory.high overage in long operations Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating Shakeel Butt
2026-11-22 3:06 ` sashiko-bot
2026-11-22 2:47 ` [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes Shakeel Butt
2026-11-22 3:06 ` sashiko-bot
2026-11-22 2:47 ` [RFC PATCH 4/7] mm/shmem: handle memory.high overage in fallocate and reads Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 5/7] mm/khugepaged: handle memory.high overage in MADV_COLLAPSE Shakeel Butt
2026-11-22 2:47 ` [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement Shakeel Butt
2026-11-22 3:06 ` sashiko-bot
2026-11-22 2:47 ` [RFC PATCH 7/7] selftests/cgroup: make test_memcg_high_sync check that throttling holds Shakeel Butt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox