[RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[Aditya Sharma]

Address-space teardown on process exit runs synchronously in the dying
task's context: exit_mm() -> mmput() -> __mmput() -> exit_mmap() walks
page tables, updates rmap, frees the RSS and drops file references, all
on the exiting CPU. For a multi-GB process that is hundreds of
milliseconds of exit-path latency, paid by whoever is waiting on the
death: a supervisor's kill-and-respawn cycle, a shell's waitpid(), an
orchestrator reaping a fleet of workers. On the test box below, killing
a 16GB process costs ~280ms before the parent's waitpid() returns.

This series adds CONFIG_ASYNC_MM_TEARDOWN: an opt-in, default-off path
that defers __mmput() of large exiting processes to a dedicated kernel
thread (mm_reaper), so the exiting CPU is released as soon as the task
is reaped and the teardown runs off to the side.

Design
======

 - A single freezable kthread, mm_reaper, modeled on oom_reaper, fed
   through a lock-free llist + waitqueue. llist rather than oom_reaper's
   spinlock+list so many-core parallel exits do not serialize on a
   global lock at enqueue. The kthread runs at nice 19; its affinity is
   a preference (kthread_affine_preferred()) for the HK_TYPE_KTHREAD
   housekeeping mask, so it keeps off isolcpus= and cpuset-isolated
   CPUs without the hard bind of kthread_bind_mask() --
   PF_NO_SETAFFINITY would leave admins unable to re-affine it.

 - mmput_exit(), called only from exit_mm()'s final reference drop
   (patch 7). Every other mmput() caller (get_task_mm() users such as
   ptrace and /proc, kthread_unuse_mm(), ...) is unchanged and still
   tears down inline if it happens to hold the last reference.

 - Deferral only when ALL of the following hold, else inline __mmput():
     * vm.async_mm_teardown=1 (static key, default off)
     * RSS >= vm.async_mm_teardown_thresh_pages (default 64MB)
     * the mm is not an OOM-kill target (MMF_OOM_TARGETED, patch 3) and
       not already reaped (MMF_OOM_SKIP)
     * charging the RSS to a pending-pages budget stays under
       vm.async_mm_teardown_max_pending_pages (add-then-check: can
       over-reject under concurrency, never over-admit)

 - exit_aio() is pulled forward: mmput_exit() runs it eagerly on the
   exiting task before reserving/queuing. exit_aio() can block
   indefinitely waiting for in-flight AIO to drain; run eagerly, a
   stuck AIO context stalls only that task -- exactly what a
   synchronous mmput() would do -- instead of blocking the single
   reaper and stranding every teardown queued behind it. It is
   idempotent (a no-op once ioctx_table is gone), so __mmput() calling
   it again later is harmless.

 - OOM exclusion: an OOM victim's memory must come back promptly, not
   sit behind a nice-19 kthread working against the oom_reaper.
   MMF_OOM_TARGETED is mm-scoped (not per-task) because the final
   mm_users drop can come from a CLONE_VM sharer in another thread
   group that mark_oom_victim() never sees; it is set before any
   SIGKILL for the kill event is dispatched, and the marking and the
   eligibility check cannot overlap in time (see patch 3 for the
   ordering argument). That argument is belt-and-suspenders, not a
   correctness dependency: a missed flag would only defer the
   victim's teardown (a latency/priority-inversion regression), never
   corrupt anything, and the oom_reaper reaps the victim
   independently either way. An LKMM litmus test for the
   CLONE_VM-sharer case is available on request.

 - No extra mm refcounting: deferring __mmput() keeps the mm_count
   reference that mm_users > 0 stands for, exactly as mmput_async()
   does; the mm and its embedded llist node stay alive until the
   kthread runs.

 - Observability: three vmstat counters (queued / sync / rejected) and
   two tracepoints. queue carries pid/comm/rss/node of the exiting
   task; reap carries the charged RSS vs a fresh RSS read plus the
   reaper's node, so the gap measures how much reclaim ate out of the
   queue, and queue-vs-reap node mismatches expose NUMA-remote
   teardown.


The work is moved, not eliminated. Measured checks (same box
as below):

 - Freeing latency: with async on, a killed 16GB region is freed on
   the same timescale as inline teardown -- nr_free_pages recovers
   within ~300ms of the kill in both configurations (single-rep
   watcher readings of 240ms async vs 283ms inline, within run-to-run
   variance; not a claim that async frees faster). Deferral does not
   delay the return of memory; it makes the exit path stop waiting
   for it.

 - CPU conservation: 5 consecutive 4GB inline teardowns cost 372ms of
   exit-path CPU; mm_reaper spends 440ms of kthread CPU draining the
   same five, read from its schedstat over a fully drained window
   (ratio 1.18, nice-19 kthread, cache-cold on another CPU).

 - Pages queued for teardown are invisible to reclaim until the reaper
   frees them. The pending cap bounds that exposure coarsely; it is not
   a reclaim mechanism. Reclaim integration (unmap-first draining, a
   shrinker) is future work, and is also an answer to the cap
   sizing question below.

Scope of the claims: what this series claims today is the common-case
exit-latency win under a modest threshold and a conservative cap. It
deliberately does not claim the multi-GB-teardown-under-memory-pressure
regime: until the queue is visible to reclaim, that regime is exactly
where the cap must stay conservative, and the 16GB+ numbers below are
capability demonstrations, run with the cap deliberately raised on an
otherwise idle machine -- not a recommended production configuration.
Open question 1 is where this either gets fixed (reclaim integration)
or stays scoped.

Numbers
=======

Bare metal, 32 CPUs / 32GB RAM (31 GiB usable), single NUMA node,
performance governor, medians, THP off unless noted. reap = SIGKILL -> waitpid()
returns. For all async-on runs vm.async_mm_teardown_max_pending_pages
was raised to 3/4 of RAM so the measurements exercise the reaper, not
the backpressure fallback: with the RAM/4 default, a single 16GB mm on
this box (8GB cap) -- or the 64GB guest case -- exceeds the cap and
falls back to sync teardown by design. See open question 1.

  kill, async off -> on:
    1GB      18.01 ms   ->  0.14 ms
    4GB      73.71 ms   ->  0.14 ms
    16GB    282.16 ms   ->  0.15 ms
    16GB     27.68 ms   ->  0.13 ms    (THP on)
  voluntary exit (_exit() -> reap), 16GB:
            271.27 ms   ->  0.12 ms
  M simultaneous 0.5GB exits, tail until ALL reaped:
    M=8      40.64 ms   ->  0.15 ms
    M=16     79.99 ms   ->  0.17 ms
    M=32    157.42 ms   ->  0.32 ms
  below-threshold 30MB process, feature ON vs OFF:
              0.56 ms   ->  0.60 ms    (sync path, within noise)
  file-backed (tmpfs) 4GB mapping, kill:
             35.34 ms   ->  0.12 ms
  real workload -- redis-server with a 16GB populated dataset
  (DEBUG POPULATE, VmRSS ~16.2GB), kill:
            292.97 ms   ->  0.24 ms

The redis point (5 reps/pass, sync spread 288.5-309.2ms, async
0.12-0.32ms) lands within ~3% of the synthetic 16GB per-GB rate, so
the benchmark's memset region is a fair proxy for a real heap, and
the async side carries over unchanged.

CONFIG=n control: the same driver on a =n build of the same tree
(interface verified fully absent: no sysctls, no vmstat counters, no
kthread, no tracepoints) reproduces the =y async-off column within
noise -- 18.01 vs 18.01 ms at 1GB kill, 282.14 vs 282.16 ms at 16GB
kill -- so with the key off, the config costs nothing measurable.

Single-reaper drain throughput: 16.4GB backlog (32 x 0.5GB) returned in
650ms, ~25GB/s on this box. For scale, a 32-vCPU/88GB QEMU guest,
pinned to one NUMA node of its host (vCPUs and preallocated guest RAM
both bound node-local; guest numbers, same kernel/config): 16GB kill
426.15ms -> 0.13ms, a 64GB backlog (32 x 2GB) drained in ~2.1s
(~31GB/s), a 16GB shmem inode evicted in reaper context, and the
pm_test freezer cycle passed with the full 64GB backlog queued.

Robustness testing (driver script, all sections pass, dmesg clean)
==================================================================

 - Backpressure: with the cap forced to 1GB, 8 parallel 0.5GB exits
   produce queued += 4, sync fallbacks += 12; the cap is respected and
   fallback is graceful.
 - OOM killer firing while 8GB of teardowns are queued: 3/3 reps clean,
   backlog drains in ~650ms alongside the kill.
 - Targeted MMF_OOM_TARGETED probes, verified per-pid against the
   tracepoints (a positive control first proves a plain large exit of
   the same binary IS seen as queued): a ~512MB memcg OOM victim is
   never queued; a CLONE_VM pair sharing one mm, swept by the same
   kill, produces no queue event from either thread group. Two claims
   are documented as untestable from userspace rather than tested:
   task_will_free_mem() fast-path marking, and flag stickiness/fork
   non-inheritance (no userspace process can survive owning a targeted
   mm: oom_score_adj writes propagate across mm sharers, and the
   __oom_kill_process() sweep kills every sharer unconditionally).
 - unlink-while-mapped tmpfs file: the VMA's fput at teardown is the
   last reference, so the final iput -> shmem_evict_inode of a 4GB
   shmem inode runs in reaper context; tmpfs is empty 212ms after the
   reap. Note: a kthread's fput cannot use task_work, so the evict
   actually executes in the (bound) delayed_fput workqueue -- see open
   question 4.
 - Toggling: disabling mid-drain still drains the backlog (8.0GB in
   670ms) while new exits take the sync path; 50 rapid static-key
   flip cycles under concurrent exits, clean.
 - pm_test freezer cycle with a 16GB backlog queued: completes, and
   the async path works after thaw. Caveat in open question 6.

NOT tested: NUMA. I have no bare-metal access to a multi-node machine;
all bare-metal numbers above are single-node, and the QEMU guests
present a single node too. The driver has a ready-to-run NUMA section
(reaper pinned to node 0, victim on node 0 vs node 1, kthread_tail
compared, tracepoints capture queue-vs-reap node) that skips on
single-node systems. Numbers from a multi-node box would be very
welcome and directly feed open question 2.

Open questions, input wanted
============================

1. The default for vm.async_mm_teardown_max_pending_pages. It is
   currently totalram_pages()/4, an explicitly marked placeholder.
   A static fraction of RAM is arbitrary; but making it a dynamic
   function of available RAM is wrong in a different way: high
   occupancy is not load. A box full of page cache or running a
   legitimately large in-memory workload is not necessarily under
   pressure, and those are exactly the boxes that want async teardown.
   Under real pressure the cap is the wrong tool anyway, because the
   queued pages are invisible to reclaim regardless of the cap value;
   the correct fix there is reclaim integration (above).

   The default also cuts against the feature: RAM/4 rejects any
   single mm larger than a quarter of RAM from deferral entirely --
   the exact processes with the worst exit latency are the first ones
   sent back to the sync path. Every async number above needed the
   cap raised to 3/4 of RAM to be measurable at all.

   And the loose end of the range bites in practice: in a large
   guest with the cap at that same 3/4 of RAM, a stress run that
   started a new round of large exits before the previous round had
   drained ran the box into OOM kills. That is what the design
   predicts: queued mms are invisible to reclaim and unreachable by
   the OOM killer -- their tasks are already dead, so they appear in
   no task dump and the oom_reaper has no task to reap; only
   mm_reaper can free them. The kills do resolve the pressure (OOM
   victims are MMF_OOM_TARGETED, so their teardown is sync), but
   tasks die on a box that is not really out of memory -- the memory
   is in the queue. With the RAM/4 default the same pattern
   self-limits: reservation fails and exits fall back to sync
   teardown, which is its own backpressure.
   Note the motivating workload (kill-and-respawn) is exactly the
   pattern that re-faults immediately after reap, so transient demand
   approaches old+new RSS; the cap bounds the old half. So: is RAM/4
   acceptable as a coarse safety bound for now? Should the cap react
   to watermarks or PSI instead? Should out_of_memory() drain or
   expedite the pending queue before picking a victim, the way it
   already coordinates with the oom_reaper? That last one is
   attractive because every queued mm is already dead: draining is
   guaranteed-progress reclaim with no victim to choose. But an
   unbounded synchronous drain inside the allocation path is its own
   stall (seconds for a large backlog), so it would need a budget.
   Or stay dumb until the shrinker exists?

   The shape I have in mind for the real fix: a shrinker whose
   count_objects() reports the pending pages and whose scan path
   kicks the reaper and switches it to unmap-first draining, so
   memory returns ahead of the slow metadata teardown. Running full
   __mmput()s inside direct reclaim would trade one latency cliff for
   another, so the shrinker's job is to accelerate and prioritize the
   drain, not to execute it in the caller's context. If the consensus
   is that reclaim visibility must land before exit_mm() is routed
   (patch 7), I would make that the centerpiece of v2.

2. One reaper vs per-node reapers vs an unbound workqueue. One thread
   sustains ~25GB/s on this box and a 64GB backlog still drains in
   ~2.1s in the big guest, but all of that is node-local. The tracepoint
   node fields exist precisely to evaluate this once multi-node
   numbers exist.

3. CPU accounting. Teardown CPU moves from the exiting task (charged
   to its cgroup) to a root-cgroup kthread. That is the same class of
   escape as kswapd or the oom_reaper, but this extends it to routine
   process exit. Patch 5's admin docs state the escape explicitly so
   nobody enables this blind; the open question is whether that
   suffices for an opt-in feature or whether cgroup-aware charging of
   the reaper's CPU is a prerequisite.

4. The delayed_fput hop. Because mm_reaper is a kthread, the final
   fput() it drops cannot use task_work and is punted to the bound
   delayed_fput workqueue -- so a multi-GB shmem inode evict runs on a
   bound system_wq worker (the workqueue watchdog already complains:
   "delayed_fput hogged CPU for >13333us, consider switching to
   WQ_UNBOUND"). Should the reaper drain its own fput backlog inline,
   or is this an argument for making delayed_fput unbound generally?

5. memcg zombies. Deferring teardown also defers the final uncharge,
   so a dying memcg can be pinned a little longer by a queued mm
   (bounded by the pending cap and measured drain times, i.e.
   normally milliseconds). I believe this is acceptable.

6. Freezing. The drain loop calls try_to_freeze() after every entry,
   so a freeze request is honoured between mms mid-batch, and the
   pm_test freezer cycle passes with a 16GB backlog queued. What
   remains is that a single in-flight __mmput() cannot freeze
   mid-teardown, so one sufficiently large mm could still exceed the
   freeze timeout -- but that is equally true of the same exit_mmap()
   running synchronously in the exiting task today. Is per-mm
   granularity enough?

Related work
============
 - "Two alternatives for mm async teardown" (Claudio Imbrenda, 2021,
   https://lore.kernel.org/all/20211111095008.264412-1-imbrenda@linux.ibm.com/
   ): an earlier RFC with the same
   goal, motivated by huge s390 KVM guest address spaces. It offered
   either an arch hook in exit_mm() or a process_mmput_async syscall
   that runs the teardown in another process's context, plus an OOM
   notifier to shield the teardown from the OOM killer. Neither
   variant was merged; the s390 protected-guest case was later
   addressed with KVM-specific asynchronous destruction. This series
   is arch-independent, needs no syscall or userspace agent (kernel
   kthread plus an eligibility gate), and integrates with the OOM
   killer by exclusion (MMF_OOM_TARGETED keeps victims synchronous)
   rather than by blocking it.
 - oom_reaper: same kthread pattern, but it strips memory from a
   victim that cannot exit; this series is about not making healthy
   exits pay the teardown latency.
 - process_mrelease(2): userspace-driven reaping of a killed process.
   It needs a daemon to issue it per-kill, only helps the kill case
   (not voluntary exit), and does not remove the exit-path work from
   the dying task -- it races it. This series needs no userspace
   agent and covers both exit flavors.
 - mmput_async(): existing deferral of __mmput() to a workqueue for
   contexts that cannot sleep; this series reuses its lifetime
   argument but adds gating, backpressure, OOM exclusion and a
   dedicated low-priority thread, none of which mmput_async() has.

Patches
=======

 1  mm: add CONFIG_ASYNC_MM_TEARDOWN scaffolding for off-CPU exit
    teardown -- config, mm_struct::async_reap_node, mmput_exit()
    fallback; no functional change.
 2  mm: dispatch __mmput() to an mm_reaper kthread via llist -- the
    kthread and queue; built but unreferenced.
 3  mm/oom_kill: mark an OOM victim's mm with MMF_OOM_TARGETED -- the
    mm-scoped victim flag and its visibility/ordering argument;
    nothing reads it yet.
 4  mm: gate async teardown on RSS with pending-pages backpressure --
    eligibility predicate + budget reservation; charge memoized at
    enqueue because RSS keeps shrinking after mm_users hits zero;
    exit_aio() pulled forward onto the exiting task.
 5  mm: add runtime toggle and sysctls for async teardown -- static
    key, three vm.* sysctls, admin-guide docs.
 6  mm: add tracepoints and vmstat counters for async teardown.
 7  exit: route exit_mm()'s final mmput() through mmput_exit() -- the
    switch that makes 1-6 operative.

Patches 1-6 are individually inert (each is either scaffolding or
dead code until patch 7), so the series bisects cleanly with the
feature dark until the last commit, and the runtime default is off
even then.

The benchmark (exit_latency.c) and the driver that produced every
number above (sanity/headline/wait-free/CPU-conservation/reuse/
parallel/backpressure/numa/smallproc/oom/oom-targeted/file-backed/
toggle/freezer sections, ~2k lines total), plus the standalone redis
driver behind the real-workload number, are available on request; if
there is interest I am happy to rework them into
tools/testing/selftests/mm for a later revision.

Based on mm-unstable, commit 890f8c4e827c
("mm/secretmem: don't allow highmem folios").

Aditya Sharma (7):
  mm: add CONFIG_ASYNC_MM_TEARDOWN scaffolding for off-CPU exit teardown
  mm: dispatch __mmput() to an mm_reaper kthread via llist
  mm/oom_kill: mark an OOM victim's mm with MMF_OOM_TARGETED
  mm: gate async teardown on RSS with pending-pages backpressure
  mm: add runtime toggle and sysctls for async teardown
  mm: add tracepoints and vmstat counters for async teardown
  exit: route exit_mm()'s final mmput() through mmput_exit()

 Documentation/admin-guide/sysctl/vm.rst |  38 +++++
 include/linux/mm_types.h                |  15 ++
 include/linux/sched/mm.h                |   6 +
 include/linux/vm_event_item.h           |   5 +
 include/trace/events/mm_reaper.h        |  73 +++++++++
 kernel/exit.c                           |   2 +-
 kernel/fork.c                           | 199 ++++++++++++++++++++++++
 mm/Kconfig                              |  12 ++
 mm/oom_kill.c                           |  61 ++++++++
 mm/vmstat.c                             |   5 +
 10 files changed, 415 insertions(+), 1 deletion(-)
 create mode 100644 include/trace/events/mm_reaper.h


base-commit: 890f8c4e827c918dac668a12eaf63180ba8a9e6d
-- 
2.34.1

[RFC PATCH 1/7] mm: add CONFIG_ASYNC_MM_TEARDOWN scaffolding for off-CPU exit teardown

[Aditya Sharma]

Address-space teardown on process exit runs synchronously in the dying
task's context: exit_mm() -> mmput() -> __mmput() -> exit_mmap() walks
page tables, updates rmap, frees the RSS and drops file refs, all on the
exiting CPU.

Add the scaffolding to later move __mmput() off-CPU for large exiting
address spaces, with no functional change in this patch:
  - new config ASYNC_MM_TEARDOWN (bool, depends on MMU, default n)
  - struct mm_struct::async_reap_node (llist_node), CONFIG-gated, with
    an explicit <linux/llist.h> include (it was only ever available
    transitively via spinlock.h)
  - mmput_exit() declaration, with a static inline fallback that is a
    plain mmput

When the config is disabled mmput_exit() compiles to mmput() with zero
delta, so nothing changes until the kthread and dispatch land.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 include/linux/mm_types.h |  4 ++++
 include/linux/sched/mm.h |  6 ++++++
 mm/Kconfig               | 11 +++++++++++
 3 files changed, 21 insertions(+)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 939b5ea8c..fffa285ef 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -7,6 +7,7 @@
 #include <linux/auxvec.h>
 #include <linux/kref.h>
 #include <linux/list.h>
+#include <linux/llist.h>
 #include <linux/spinlock.h>
 #include <linux/rbtree.h>
 #include <linux/maple_tree.h>
@@ -1367,6 +1368,9 @@ struct mm_struct {
 		atomic_long_t hugetlb_usage;
 #endif
 		struct work_struct async_put_work;
+#ifdef CONFIG_ASYNC_MM_TEARDOWN
+		struct llist_node async_reap_node;
+#endif /* CONFIG_ASYNC_MM_TEARDOWN */
 
 #ifdef CONFIG_IOMMU_MM_DATA
 		struct iommu_mm_data *iommu_mm;
diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h
index 10be8a54b..20aeb1d7e 100644
--- a/include/linux/sched/mm.h
+++ b/include/linux/sched/mm.h
@@ -147,6 +147,12 @@ extern void mmput(struct mm_struct *);
 void mmput_async(struct mm_struct *);
 #endif
 
+#ifdef CONFIG_ASYNC_MM_TEARDOWN
+void mmput_exit(struct mm_struct *);
+#else
+static inline void mmput_exit(struct mm_struct *mm) { mmput(mm); }
+#endif
+
 /* Grab a reference to a task's mm, if it is not already going away */
 extern struct mm_struct *get_task_mm(struct task_struct *task);
 /*
diff --git a/mm/Kconfig b/mm/Kconfig
index c52ab6afc..296b16ad4 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1500,6 +1500,17 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 	  If unsure, say N.
 
+config ASYNC_MM_TEARDOWN
+	bool "Async off-CPU address space teardown on exit"
+	depends on MMU
+	default n
+	help
+	  Defer the address space teardown (exit_mmap()) of large exiting processes
+	  to a kernel thread instead of running it on the exiting CPU. The feature
+	  is off by default.
+
+	  If unsure, say N.
+
 source "mm/damon/Kconfig"
 
 endmenu
-- 
2.34.1

[RFC PATCH 2/7] mm: dispatch __mmput() to an mm_reaper kthread via llist

[Aditya Sharma]

Introduce a single dedicated, freezable, sleepable kthread (mm_reaper)
modeled on oom_reaper, and the machinery to route the deferrable unit of
teardown -- __mmput() -- to it through a lock-free llist plus waitqueue.
mmput_exit() does llist_add() and wakes the kthread, which drains with
llist_del_all() and runs __mmput() per entry with cond_resched(). The
drain loop also calls try_to_freeze() after each entry, so a freeze
request is honoured mid-batch rather than only at the wait point.

Nothing calls mmput_exit() yet: exit_mm() is not routed to it, and there
is no eligibility check or gate, so it would queue every final-drop mm
unconditionally if invoked. This changes no behavior -- the function is
built but unreferenced, the kthread receives no work, and __mmput() runs
inline on the exiting CPU exactly as today. The exit_mm() hook and the
eligibility/gating policy follow in later patches.

llist (rather than oom_reaper's spinlock+list) avoids a global lock on
the exit path under many-core parallel exits. llist_for_each_entry_safe()
caches the next pointer before __mmput(), so iteration is safe even when
the final mmdrop() inside __mmput() frees mm.

Lifetime: the mm_count reference that mm_users>0 stands for is released
only by that final mmdrop() inside __mmput(). Deferring __mmput() keeps
the reference held, so the mm_struct and its embedded async_reap_node
stay alive until the kthread runs. No extra mmgrab() is needed, for the
same reason mmput_async() needs none.

The kthread is started at subsys_initcall and set to nice 19 to minimize
competition with other EEVDF tasks. Its affinity is expressed as a
preference via kthread_affine_preferred() over the HK_TYPE_KTHREAD
housekeeping mask, so it keeps off isolcpus= and cpuset-isolated CPUs
without the hard binding kthread_bind_mask() would impose --
PF_NO_SETAFFINITY would otherwise leave admins unable to re-affine it.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 kernel/fork.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/kernel/fork.c b/kernel/fork.c
index f0e2e131a..884e4e767 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -112,6 +112,7 @@
 #include <linux/unwind_deferred.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
+#include <linux/sched/isolation.h>
 
 #include <asm/mmu_context.h>
 #include <asm/cacheflush.h>
@@ -3407,3 +3408,56 @@ static int __init init_fork_sysctl(void)
 }
 
 subsys_initcall(init_fork_sysctl);
+
+#ifdef CONFIG_ASYNC_MM_TEARDOWN
+static LLIST_HEAD(mm_reaper_list);
+static DECLARE_WAIT_QUEUE_HEAD(mm_reaper_wait);
+
+static void async_mm_teardown_queue(struct mm_struct *mm)
+{
+	if (llist_add(&mm->async_reap_node, &mm_reaper_list))
+		wake_up(&mm_reaper_wait);
+}
+
+static int mm_reaper(void *unused)
+{
+	set_freezable();
+	while (true) {
+		struct llist_node *batch;
+		struct mm_struct *mm, *n;
+
+		wait_event_freezable(mm_reaper_wait, !llist_empty(&mm_reaper_list));
+		batch = llist_del_all(&mm_reaper_list);
+		llist_for_each_entry_safe(mm, n, batch, async_reap_node) {
+			__mmput(mm); /* may free mm via mmdrop */
+			cond_resched();
+			try_to_freeze();
+		}
+	}
+	return 0;
+}
+
+void mmput_exit(struct mm_struct *mm)
+{
+	might_sleep();
+	if (!atomic_dec_and_test(&mm->mm_users))
+		return;
+	async_mm_teardown_queue(mm);
+}
+
+static int __init mm_reaper_init(void)
+{
+	struct task_struct *th;
+
+	th = kthread_create(mm_reaper, NULL, "mm_reaper");
+	if (IS_ERR(th)) {
+		pr_err("mm_reaper: failed to start kthread: %ld\n", PTR_ERR(th));
+		return PTR_ERR(th);
+	}
+	set_user_nice(th, 19); /* minimize competition with other fair class tasks */
+	kthread_affine_preferred(th, housekeeping_cpumask(HK_TYPE_KTHREAD));
+	wake_up_process(th);
+	return 0;
+}
+subsys_initcall(mm_reaper_init);
+#endif
-- 
2.34.1

[RFC PATCH 3/7] mm/oom_kill: mark an OOM victim's mm with MMF_OOM_TARGETED

[Aditya Sharma]

A later part of this series defers the address-space teardown of large
exiting processes to a kernel thread (mm_reaper) instead of running it
inline on the exiting CPU. That deferral must never be applied to an
OOM victim: its memory has to be released promptly to relieve the
pressure that triggered the kill, and queuing it behind the
low-priority, housekeeping-CPU-confined mm_reaper would work directly
against the oom_reaper. The teardown path therefore needs a cheap
"is this mm an OOM victim?" test.

The eventual consumer runs from mmput_exit() on the exit path, after
exit_mm() has already cleared current->mm, so the only handle it has
is the mm pointer itself. Existing OOM state does not answer the
question well from an mm alone:

  - signal->oom_mm and TIF_MEMDIE are per-task / per-signal and are
    set only on the single task passed to mark_oom_victim(). A sibling
    thread, or a CLONE_VM-sharing process in another thread group
    (which __oom_kill_process() SIGKILLs but never marks), can be the
    task that drops the last mm_users reference and runs the teardown.
    A per-task check misses those.

  - MMF_OOM_SKIP is mm-scoped but does not mean "OOM victim".
    exit_mmap() sets it on every exiting mm once the memory is freed,
    and dup_mmap() sets it on the fork-failure cleanup path, so it
    produces false positives on ordinary exits. It is also not reliably
    early: on a real victim it is normally set only once exit_mmap() or
    oom_reap_task_mm() has already released the memory -- the exception
    being the mm-pinned-by-init case in __oom_kill_process(), which sets
    it at kill time with RSS fully intact. Wrong in both directions, so
    it cannot serve as the predicate this path needs.

Add MMF_OOM_TARGETED, an mm-scoped flag, and set it in two places:

  - in mark_oom_victim(), covering every caller uniformly (this is
    the only marking for the two task_will_free_mem() fast paths in
    oom_kill_process() and out_of_memory());

  - early in __oom_kill_process(), before any SIGKILL for the kill
    event is dispatched, to victim's own thread group or to other
    thread groups sharing the mm.

Observing this flag is not load-bearing for correctness. Should it
ever be missed, the consequence is bounded: the victim's mm is queued
to mm_reaper instead of being torn down inline, i.e. a latency and
priority-inversion regression against the oom_reaper -- not
corruption, use-after-free or a leak. The mm is still fully torn down
and the oom_reaper still reaps the victim independently. The
visibility argument below is therefore belt-and-suspenders.

Ordering and coverage: this flag is reliably visible to whichever
thread ends up dropping the last reference in mmput_exit(), because
marking and queuing for async teardown can never overlap in time.
Every mark site runs while some task's ->mm still points at this mm --
either tsk is current, marking itself in program order (the
task_will_free_mem() fast paths), or tsk is task_lock()ed by the
caller (oom_kill_process()'s fast path, and __oom_kill_process() via
find_lock_task_mm()) -- while mmput_exit(), the only path that queues
for teardown, is reached only after mm_users hits 0, which requires
that same task's own exit_mm() (or exec_mmap(), the other path that
sheds ->mm) to have already cleared ->mm under that same task_lock().
Program order, or task_lock() release/acquire, therefore orders the
flag store before that task's own eventual mmput_exit(). This is stated
for userspace tasks: any userspace task's ->mm holds an mm_users
reference, whereas kthread_use_mm() borrows an mm under mmgrab() alone
-- kthreads are not signal-killable and are never OOM victims, so the
borrowed-mm case does not arise here.

If a different CLONE_VM sharer in another thread group performs the
actual final decrement instead -- a task mark_oom_victim() never marks
directly -- the ordering still carries, though not from the flag store
itself. mm_flags_set() is set_bit(), a non-value-returning RMW, so it
is unordered and contributes nothing on its own. The ordering comes
from the marking task's own atomic_dec_and_test(), a value-returning
RMW and therefore fully ordered (Documentation/atomic_t.txt: an
smp_mb() before and after). Decrements are totally ordered in
mm_users' modification order with the zeroing one last, so the general
barriers on both sides make the flag store visible to whichever thread
performs it.

The redundant set in mark_oom_victim() on the __oom_kill_process()
path is harmless and keeps the invariant "mark_oom_victim() implies
MMF_OOM_TARGETED" independent of future code motion.

The flag is intentionally sticky: it is never cleared. That is fine
because a marked mm is dying: __oom_kill_process() SIGKILLs every
process sharing the victim's mm -- only global init (whose mm also
gets MMF_OOM_SKIP set on the spot, keeping it off the async path
anyway) and kthreads are exempt from the sweep, and a kthread's
kthread_unuse_mm() drop is a plain mmput(), never mmput_exit(). So
no task that could reach the async path survives owning a marked mm.
Should some future path leave a survivor, the consequence is only
conservative: that mm keeps tearing down synchronously. The flag is
structurally outside legacy-mask inheritance: mm_init() clears the
whole flags bitmap and re-seeds only word 0 from
MMF_INIT_LEGACY_MASK, so flags above bit 31 are never copied on
fork(). NUM_MM_FLAG_BITS is a fixed 64 on all architectures, so bit 32
exists on 32-bit builds as well.

This commit only defines and sets the flag; nothing reads it yet. The
write is gated by IS_ENABLED(CONFIG_ASYNC_MM_TEARDOWN) and compiles
out entirely on kernels that will not consume it. The reader
(async_mm_teardown_eligible()) is added in the next patch, and the
deferred teardown is not routed onto the exit path until later in the
series, so this change is inert on its own.

Note: a bit named MMF_OOM_VICTIM existed until 2022 and was removed
as dead state once exit_mmap()'s special-case reap-before-teardown
was replaced by mmap_lock/MMF_OOM_SKIP-based concurrency between
exit_mmap() and oom_reap_task_mm(). MMF_OOM_TARGETED is a
deliberately distinct name for a new consumer -- deferred exit-path
teardown -- and does not revive the old exit_mmap() behaviour.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 include/linux/mm_types.h | 10 +++++++
 mm/oom_kill.c            | 61 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index fffa285ef..3240c029f 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1989,6 +1989,16 @@ enum {
 #define MMF_TOPDOWN		31	/* mm searches top down by default */
 #define MMF_TOPDOWN_MASK	BIT(MMF_TOPDOWN)
 
+/*
+ * mm is the target of an in-flight OOM kill. Only written under
+ * CONFIG_ASYNC_MM_TEARDOWN today. Sticky: never cleared -- the mm is
+ * dying (__oom_kill_process() kills every process sharing it), and a
+ * hypothetical surviving sharer would merely keep tearing down
+ * synchronously. Above bit 31, so it is outside MMF_INIT_LEGACY_MASK's
+ * domain entirely and is never inherited on fork.
+ */
+#define MMF_OOM_TARGETED	32
+
 #define MMF_INIT_LEGACY_MASK	(MMF_DUMP_FILTER_MASK |\
 				 MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\
 				 MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 5f372f6e2..b5490d1b1 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -754,6 +754,40 @@ static void mark_oom_victim(struct task_struct *tsk)
 	struct mm_struct *mm = tsk->mm;
 
 	WARN_ON(oom_killer_disabled);
+
+	/*
+	 * Mark the mm itself, not just this task/signal, as an OOM target,
+	 * so the deferred-teardown path can test it from the mm alone. Set
+	 * ahead of the TIF_MEMDIE test-and-set below so that every call
+	 * marks the mm, including a repeat mark on a task that is already
+	 * TIF_MEMDIE.
+	 *
+	 * This is reliably visible to whichever thread ends up dropping the
+	 * last reference in mmput_exit(): marking and queuing for async
+	 * teardown can never overlap in time. Every caller reaches this with
+	 * tsk->mm still set -- either tsk is current, marking itself in
+	 * program order, or tsk is task_lock()ed by the caller (the
+	 * task_will_free_mem() fast path in oom_kill_process()) -- while
+	 * mmput_exit(), the only path that queues for teardown, is reached
+	 * only after mm_users hits 0, which requires tsk's own exit_mm() (or
+	 * exec_mmap(), the other path that sheds ->mm) to have already
+	 * cleared ->mm under that same task_lock(). Program order or
+	 * task_lock() release/acquire therefore orders this store before
+	 * tsk's own eventual mmput_exit().
+	 *
+	 * If a different CLONE_VM sharer performs the actual final decrement
+	 * instead, the ordering does not come from this store:
+	 * mm_flags_set() is set_bit(), a non-value-returning RMW, so it is
+	 * unordered and contributes nothing on its own. It comes from the
+	 * marking task's own atomic_dec_and_test(), a value-returning RMW
+	 * and therefore fully ordered (an smp_mb() before and after).
+	 * Decrements are totally ordered in mm_users' modification order
+	 * with the zeroing one last, so the general barriers on both sides
+	 * make this store visible to whichever thread performs it.
+	 */
+	if (IS_ENABLED(CONFIG_ASYNC_MM_TEARDOWN))
+		mm_flags_set(MMF_OOM_TARGETED, mm);
+
 	/* OOM killer might race with memcg OOM */
 	if (test_and_set_tsk_thread_flag(tsk, TIF_MEMDIE))
 		return;
@@ -931,6 +965,33 @@ static void __oom_kill_process(struct task_struct *victim, const char *message)
 	mm = victim->mm;
 	mmgrab(mm);
 
+	/*
+	 * Set this before any SIGKILL for this kill event goes out below,
+	 * while task_lock(victim) (held since find_lock_task_mm() above) is
+	 * still held. This is reliably visible to whichever thread ends up
+	 * running mmput_exit() and dropping the last reference to this mm --
+	 * victim itself, a sibling, or a CLONE_VM sharer in another thread
+	 * group (which mark_oom_victim() never marks, but which still
+	 * observes this store on the shared mm). Marking and queuing for
+	 * async teardown can never overlap: mmput_exit() is reached only
+	 * after mm_users hits 0, and that requires victim's own exit_mm()
+	 * (or exec_mmap(), the other path that sheds ->mm) to have cleared
+	 * ->mm under this same task_lock() first, which release/acquire
+	 * orders after the store above.
+	 *
+	 * If some other sharer performs the actual final decrement instead,
+	 * the ordering does not come from this store: mm_flags_set() is
+	 * set_bit(), a non-value-returning RMW, so it is unordered and
+	 * contributes nothing on its own. It comes from the marking task's
+	 * own atomic_dec_and_test(), a value-returning RMW and therefore
+	 * fully ordered (an smp_mb() before and after). Decrements are
+	 * totally ordered in mm_users' modification order with the zeroing
+	 * one last, so the general barriers on both sides make this store
+	 * visible to whichever thread performs it.
+	 */
+	if (IS_ENABLED(CONFIG_ASYNC_MM_TEARDOWN))
+		mm_flags_set(MMF_OOM_TARGETED, mm);
+
 	/* Raise event before sending signal: task reaper must see this */
 	count_vm_event(OOM_KILL);
 	memcg_memory_event_mm(mm, MEMCG_OOM_KILL);
-- 
2.34.1

[RFC PATCH 4/7] mm: gate async teardown on RSS with pending-pages backpressure

[Aditya Sharma]

Deferring every exit would churn small processes for no benefit and let
unreaped memory grow without bound. Split the defer decision into a
pure predicate and a budget reservation:

  - async_mm_teardown_eligible() (side-effect free): defer only when
    RSS >= async_mm_teardown_thresh_pages and both MMF_OOM_SKIP and
    MMF_OOM_TARGETED are clear:
      * MMF_OOM_SKIP -- the mm is hidden from the OOM killer and
        reaper. Usually that means it has already been reaped or run
        through exit_mmap() and has no RSS left worth deferring; in the
        mm-pinned-by-init case (__oom_kill_process()) the flag is set at
        kill time with RSS intact, and keeping such an mm synchronous is
        equally what we want;
      * MMF_OOM_TARGETED -- the mm is the target of an in-flight OOM
        kill (added in the preceding patch). Such a victim is torn
        down synchronously so its memory is released promptly rather
        than queued behind the low-priority, housekeeping-confined
        mm_reaper, which would work against the oom_reaper during the
        exact pressure that triggered the kill. The flag is set while
        some task still holds a live reference to the mm, and this
        check is reached only after mm_users has dropped to 0 -- which
        requires that task's own exit_mm() (or exec_mmap(), the other
        path that sheds ->mm) to have already cleared ->mm under that
        same task_lock() -- so marking and checking can never overlap in
        time. Whichever thread performs the final decrement is therefore
        guaranteed to observe the flag already set, per the preceding
        patch.

  - async_mm_teardown_reserve(): charge RSS against a pending-pages
    budget with atomic_long_add_return(); if the result exceeds
    async_mm_teardown_max_pending_pages it subtracts back and returns
    false, so the caller falls back to synchronous __mmput().

mmput_exit() reads RSS once and checks eligibility before reservation,
so the budget is only touched for processes that pass the cheap
checks. On success it stores the charged amount in mm->async_reap_node's
companion field async_reap_rss before enqueueing, and the reaper
releases exactly that stored charge after __mmput(). The charge cannot
be re-derived at reap time: RSS keeps shrinking after mm_users reaches
zero -- rmap-based reclaim can still swap out anon pages and truncation
can still unmap file pages, neither of which requires mm_users -- and
get_mm_rss() is an approximate per-CPU counter read besides. Two
independent reads at enqueue and reap would systematically
under-release, drifting the pending counter upward until the cap
permanently forces synchronous fallback.

An eligible mm would otherwise have its entire __mmput() -- including
exit_aio() -- deferred onto the single mm_reaper kthread.
exit_aio() can block indefinitely waiting for in-flight AIO to drain,
so mmput_exit() calls it eagerly, on the exiting task's own context,
before reservation: a stuck AIO context then only stalls this task,
same as a synchronous mmput() would, instead of blocking mm_reaper and
stranding every other mm already queued behind it. exit_aio() is
idempotent -- it is a no-op once mm->ioctx_table has been torn down --
so __mmput() calling it again unconditionally is harmless, whether that
second call lands on the reaper or -- when the reservation fails --
back on the exiting task itself.

Because reservation is add-then-check rather than read-then-add,
committed pending never stays above the cap: an exit that would exceed
it tears down synchronously instead. Under concurrent exits a failing
reservation can transiently inflate the counter between its add and
roll-back, which may cause another exiter to fall back to sync
conservatively -- it can over-reject but never over-admit.

The cap is a coarse safety bound on unreaped memory, not a reclaim
mechanism: pages queued here are invisible to reclaim until the reaper
frees them. Integrating the queue with reclaim (unmap-first draining,
a shrinker) is future work; see the cover letter.

The RSS threshold defaults to a fixed 64MB at init; the pending cap
defaults to totalram_pages() / 4 as a placeholder pending tuning. Both
are static module variables here and not yet user-tunable. The static
key that gates the whole path, and sysctl registration, follow in the
next patch.

This patch is dormant: nothing calls mmput_exit() yet (exit_mm() is
routed to it later in the series), so no teardown is deferred.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 include/linux/mm_types.h |  1 +
 kernel/fork.c            | 79 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 79 insertions(+), 1 deletion(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 3240c029f..ee6c0ac84 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1370,6 +1370,7 @@ struct mm_struct {
 		struct work_struct async_put_work;
 #ifdef CONFIG_ASYNC_MM_TEARDOWN
 		struct llist_node async_reap_node;
+		unsigned long async_reap_rss;
 #endif /* CONFIG_ASYNC_MM_TEARDOWN */
 
 #ifdef CONFIG_IOMMU_MM_DATA
diff --git a/kernel/fork.c b/kernel/fork.c
index 884e4e767..d45418e66 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -113,6 +113,7 @@
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
 #include <linux/sched/isolation.h>
+#include <linux/sizes.h>
 
 #include <asm/mmu_context.h>
 #include <asm/cacheflush.h>
@@ -3412,6 +3413,48 @@ subsys_initcall(init_fork_sysctl);
 #ifdef CONFIG_ASYNC_MM_TEARDOWN
 static LLIST_HEAD(mm_reaper_list);
 static DECLARE_WAIT_QUEUE_HEAD(mm_reaper_wait);
+static atomic_long_t mm_reaper_pending_pages;
+static unsigned long sysctl_async_mm_teardown_thresh_pages;
+static unsigned long sysctl_async_mm_teardown_max_pending_pages;
+
+static bool async_mm_teardown_eligible(struct mm_struct *mm, unsigned long rss)
+{
+	if (rss < READ_ONCE(sysctl_async_mm_teardown_thresh_pages))
+		return false;
+	if (mm_flags_test(MMF_OOM_SKIP, mm))	/* reaped, or hidden from the reaper */
+		return false;
+	/*
+	 * MMF_OOM_TARGETED is set by the OOM killer while some task still
+	 * holds a live reference to this mm (see mark_oom_victim() and
+	 * __oom_kill_process()), and this eligibility check is reached only
+	 * from mmput_exit(), after mm_users has dropped to 0 -- so marking
+	 * and this check can never overlap in time. Whichever thread
+	 * performs the final decrement is therefore guaranteed to observe
+	 * the flag already set, regardless of whether that thread was
+	 * itself ever passed to mark_oom_victim() (a CLONE_VM-sharing
+	 * process in another thread group never is, but still observes
+	 * this flag on the shared mm).
+	 */
+	if (mm_flags_test(MMF_OOM_TARGETED, mm))
+		return false;
+	return true;
+}
+
+/*
+ * Charge rss against the pending-teardown budget.  Returns true if it fits
+ * under the cap, in which case the caller enqueues and the reaper releases
+ * the same rss after __mmput().  On overshoot nothing stays charged and the
+ * caller must tear down synchronously.
+ */
+static bool async_mm_teardown_reserve(unsigned long rss)
+{
+	if (atomic_long_add_return(rss, &mm_reaper_pending_pages) >
+	    READ_ONCE(sysctl_async_mm_teardown_max_pending_pages)) {
+		atomic_long_sub(rss, &mm_reaper_pending_pages);
+		return false;
+	}
+	return true;
+}
 
 static void async_mm_teardown_queue(struct mm_struct *mm)
 {
@@ -3429,7 +3472,10 @@ static int mm_reaper(void *unused)
 		wait_event_freezable(mm_reaper_wait, !llist_empty(&mm_reaper_list));
 		batch = llist_del_all(&mm_reaper_list);
 		llist_for_each_entry_safe(mm, n, batch, async_reap_node) {
+			unsigned long pages = mm->async_reap_rss;
+
 			__mmput(mm); /* may free mm via mmdrop */
+			atomic_long_sub(pages, &mm_reaper_pending_pages);
 			cond_resched();
 			try_to_freeze();
 		}
@@ -3439,10 +3485,38 @@ static int mm_reaper(void *unused)
 
 void mmput_exit(struct mm_struct *mm)
 {
+	unsigned long rss;
+
 	might_sleep();
+	/*
+	 * Fully ordered RMW: combined with exit_mm() (or exec_mmap(), the
+	 * other path that sheds ->mm) clearing ->mm under task_lock() before
+	 * this call, it guarantees MMF_OOM_TARGETED set at any mark site is
+	 * visible here -- see mark_oom_victim(). Any new caller of
+	 * mmput_exit() outside exit_mm() must re-verify that argument.
+	 */
 	if (!atomic_dec_and_test(&mm->mm_users))
 		return;
-	async_mm_teardown_queue(mm);
+
+	rss = get_mm_rss(mm);
+
+	if (async_mm_teardown_eligible(mm, rss)) {
+		/*
+		 * exit_aio() can block indefinitely. Run it here so a stuck
+		 * AIO only hangs this task (same as how it would happen in
+		 * case of synchronous mmput()) instead of stranding every
+		 * teardown queued behind this mm
+		 */
+		exit_aio(mm);
+
+		if (async_mm_teardown_reserve(rss)) {
+			mm->async_reap_rss = rss;
+			async_mm_teardown_queue(mm);
+			return;
+		}
+	}
+
+	__mmput(mm);
 }
 
 static int __init mm_reaper_init(void)
@@ -3456,7 +3530,10 @@ static int __init mm_reaper_init(void)
 	}
 	set_user_nice(th, 19); /* minimize competition with other fair class tasks */
 	kthread_affine_preferred(th, housekeeping_cpumask(HK_TYPE_KTHREAD));
+	sysctl_async_mm_teardown_thresh_pages      = SZ_64M >> PAGE_SHIFT;
+	sysctl_async_mm_teardown_max_pending_pages = totalram_pages() / 4;  /* TODO: placeholder */
 	wake_up_process(th);
+
 	return 0;
 }
 subsys_initcall(mm_reaper_init);
-- 
2.34.1

[RFC PATCH 5/7] mm: add runtime toggle and sysctls for async teardown

[Aditya Sharma]

Add the userspace interface that arms and tunes off-CPU exit teardown,
and the static key that gates it.

Introduce async_mm_teardown_key (static key, default false) and gate the
deferral path in mmput_exit() on it via static_branch_unlikely(); when
off, mmput_exit() runs __mmput() inline. The get_mm_rss() read moves
inside the static_branch_unlikely() block, alongside eligibility and
exit_aio(), so the default (key off) path costs nothing beyond the
branch itself -- no RSS scan for every exiting process on kernels or
boots that never enable the feature. Under CONFIG_SYSCTL, register
three knobs under vm. with register_sysctl_init():

  - async_mm_teardown: write 1/0 to enable/disable. A custom handler
    flips the static key, serialized by a mutex so the key state always
    matches the last value written.
  - async_mm_teardown_thresh_pages: RSS threshold (default ~64MB).
  - async_mm_teardown_max_pending_pages: backpressure cap.

The threshold defaults are set in the initcall outside any CONFIG_SYSCTL
guard, since eligible()/reserve() read them whenever the key is on; only
the toggle variable, its lock, the handler and the table depend on
CONFIG_SYSCTL. With CONFIG_SYSCTL=n there is no path to enable the key,
so the feature stays off and the thresholds are inert.

This patch is still dormant: nothing calls mmput_exit() yet, so flipping
async_mm_teardown on has no observable effect. exit_mm() is routed
through mmput_exit() later in the series; only then does enabling the key
defer teardown. The toggle and gate are introduced here so that, once
routing lands, the feature defaults off and an explicit opt-in is
required.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 Documentation/admin-guide/sysctl/vm.rst | 38 +++++++++++
 kernel/fork.c                           | 90 ++++++++++++++++++++-----
 mm/Kconfig                              |  3 +-
 3 files changed, 114 insertions(+), 17 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
index 5b318d17a..ccda6c75f 100644
--- a/Documentation/admin-guide/sysctl/vm.rst
+++ b/Documentation/admin-guide/sysctl/vm.rst
@@ -22,6 +22,9 @@ the writeout of dirty data to disk.
 Currently, these files are in /proc/sys/vm:
 
 - admin_reserve_kbytes
+- async_mm_teardown
+- async_mm_teardown_max_pending_pages
+- async_mm_teardown_thresh_pages
 - compact_memory
 - compaction_proactiveness
 - compact_unevictable_allowed
@@ -108,6 +111,41 @@ On x86_64 this is about 128MB.
 Changing this takes effect whenever an application requests memory.
 
 
+async_mm_teardown
+=================
+
+Available only when CONFIG_ASYNC_MM_TEARDOWN is set. Controls whether the
+address-space teardown (exit_mmap()) of large exiting processes is deferred
+to the mm_reaper kernel thread instead of running on the exiting CPU.
+
+Writing 1 enables deferral, 0 disables it. Disabling makes mmput_exit() tear
+down inline again; it does not drain mms already queued. Defaults to 0.
+
+Deferred teardown consumes CPU in the mm_reaper kernel thread, which runs in
+the root cgroup: the teardown work of a task in a CPU-limited cgroup is not
+charged to that cgroup, similar to kswapd and the oom_reaper. Consider this
+before enabling the feature on systems that rely on strict per-cgroup CPU
+accounting.
+
+
+async_mm_teardown_max_pending_pages
+===================================
+
+Available only when CONFIG_ASYNC_MM_TEARDOWN is set. Backpressure cap, in
+pages, on the total RSS of mms queued for the mm_reaper but not yet torn
+down. An exit that would push the total over this cap tears down
+synchronously instead of queuing. Defaults to totalram_pages() / 4.
+
+
+async_mm_teardown_thresh_pages
+==============================
+
+Available only when CONFIG_ASYNC_MM_TEARDOWN is set. Minimum RSS, in pages,
+for an exiting mm to be eligible for deferral to the mm_reaper. Exiting mms
+below this threshold are always torn down synchronously. Defaults to 64MB
+worth of pages.
+
+
 compact_memory
 ==============
 
diff --git a/kernel/fork.c b/kernel/fork.c
index d45418e66..0976770db 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -3414,8 +3414,13 @@ subsys_initcall(init_fork_sysctl);
 static LLIST_HEAD(mm_reaper_list);
 static DECLARE_WAIT_QUEUE_HEAD(mm_reaper_wait);
 static atomic_long_t mm_reaper_pending_pages;
+static DEFINE_STATIC_KEY_FALSE(async_mm_teardown_key);
 static unsigned long sysctl_async_mm_teardown_thresh_pages;
 static unsigned long sysctl_async_mm_teardown_max_pending_pages;
+#ifdef CONFIG_SYSCTL
+static u8 sysctl_async_mm_teardown_enabled;
+static DEFINE_MUTEX(async_mm_teardown_lock);
+#endif
 
 static bool async_mm_teardown_eligible(struct mm_struct *mm, unsigned long rss)
 {
@@ -3485,8 +3490,6 @@ static int mm_reaper(void *unused)
 
 void mmput_exit(struct mm_struct *mm)
 {
-	unsigned long rss;
-
 	might_sleep();
 	/*
 	 * Fully ordered RMW: combined with exit_mm() (or exec_mmap(), the
@@ -3498,27 +3501,80 @@ void mmput_exit(struct mm_struct *mm)
 	if (!atomic_dec_and_test(&mm->mm_users))
 		return;
 
-	rss = get_mm_rss(mm);
+	if (static_branch_unlikely(&async_mm_teardown_key)) {
+		unsigned long rss = get_mm_rss(mm);
 
-	if (async_mm_teardown_eligible(mm, rss)) {
-		/*
-		 * exit_aio() can block indefinitely. Run it here so a stuck
-		 * AIO only hangs this task (same as how it would happen in
-		 * case of synchronous mmput()) instead of stranding every
-		 * teardown queued behind this mm
-		 */
-		exit_aio(mm);
+		if (async_mm_teardown_eligible(mm, rss)) {
+			/*
+			 * exit_aio() can block indefinitely. Run it here so a stuck
+			 * AIO only hangs this task (same as how it would happen in
+			 * case of synchronous mmput()) instead of stranding every
+			 * teardown queued behind this mm
+			 */
+			exit_aio(mm);
 
-		if (async_mm_teardown_reserve(rss)) {
-			mm->async_reap_rss = rss;
-			async_mm_teardown_queue(mm);
-			return;
+			if (async_mm_teardown_reserve(rss)) {
+				mm->async_reap_rss = rss;
+				async_mm_teardown_queue(mm);
+				return;
+			}
 		}
 	}
 
 	__mmput(mm);
 }
 
+#ifdef CONFIG_SYSCTL
+static int async_mm_teardown_enabled_handler(const struct ctl_table *table,
+					     int write, void *buffer,
+					     size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	/*
+	 * Serialize concurrent writers so the static key state always matches
+	 * the last value written to the variable.
+	 */
+	guard(mutex)(&async_mm_teardown_lock);
+
+	ret = proc_dou8vec_minmax(table, write, buffer, lenp, ppos);
+	if (ret || !write)
+		return ret;
+
+	if (sysctl_async_mm_teardown_enabled)
+		static_branch_enable(&async_mm_teardown_key);
+	else
+		static_branch_disable(&async_mm_teardown_key);
+	return 0;
+}
+
+static const struct ctl_table async_mm_teardown_table[] = {
+	{
+		.procname	= "async_mm_teardown",
+		.data		= &sysctl_async_mm_teardown_enabled,
+		.maxlen		= sizeof(u8),
+		.mode		= 0644,
+		.proc_handler	= async_mm_teardown_enabled_handler,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
+	},
+	{
+		.procname	= "async_mm_teardown_thresh_pages",
+		.data		= &sysctl_async_mm_teardown_thresh_pages,
+		.maxlen		= sizeof(unsigned long),
+		.mode		= 0644,
+		.proc_handler	= proc_doulongvec_minmax,
+	},
+	{
+		.procname	= "async_mm_teardown_max_pending_pages",
+		.data		= &sysctl_async_mm_teardown_max_pending_pages,
+		.maxlen		= sizeof(unsigned long),
+		.mode		= 0644,
+		.proc_handler	= proc_doulongvec_minmax,
+	},
+};
+#endif /* CONFIG_SYSCTL */
+
 static int __init mm_reaper_init(void)
 {
 	struct task_struct *th;
@@ -3533,7 +3589,9 @@ static int __init mm_reaper_init(void)
 	sysctl_async_mm_teardown_thresh_pages      = SZ_64M >> PAGE_SHIFT;
 	sysctl_async_mm_teardown_max_pending_pages = totalram_pages() / 4;  /* TODO: placeholder */
 	wake_up_process(th);
-
+#ifdef CONFIG_SYSCTL
+	register_sysctl_init("vm", async_mm_teardown_table);
+#endif
 	return 0;
 }
 subsys_initcall(mm_reaper_init);
diff --git a/mm/Kconfig b/mm/Kconfig
index 296b16ad4..a9d925406 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1507,7 +1507,8 @@ config ASYNC_MM_TEARDOWN
 	help
 	  Defer the address space teardown (exit_mmap()) of large exiting processes
 	  to a kernel thread instead of running it on the exiting CPU. The feature
-	  is off by default.
+	  is off by default and is enabled at runtime via the vm.async_mm_teardown
+	  sysctl.
 
 	  If unsure, say N.
 
-- 
2.34.1

[RFC PATCH 6/7] mm: add tracepoints and vmstat counters for async teardown

[Aditya Sharma]

Add observability for the deferred teardown path:

  - vmstat (CONFIG_ASYNC_MM_TEARDOWN, in /proc/vmstat):
    async_mm_teardown_queued   - deferred to the mm_reaper kthread
    async_mm_teardown_sync     - torn down inline on the exit path:
                                 feature off, mm below the RSS
                                 threshold, MMF_OOM_SKIP or
                                 MMF_OOM_TARGETED set, or rejected by
                                 backpressure. Counts only teardowns
                                 reaching mmput_exit(); final drops via
                                 mmput()/mmput_async() elsewhere in the
                                 kernel are inline but not counted here.
    async_mm_teardown_rejected - subset of the above that was eligible
                                 but lost to async_mm_teardown_reserve()
                                 (counted for both this and sync, so
                                 sync stays the total for the exit path)
  - tracepoints in a new include/trace/events/mm_reaper.h:
    mm_async_teardown_queue  at enqueue, and
    mm_async_teardown_reap   when the reaper picks an mm up, just before
                             __mmput(). queue carries the mm, its RSS,
                             and the exiting task's pid/comm, captured
                             while current is still that task, plus the
                             node of the exiting task. reap carries the
                             mm, the charged RSS (mm->async_reap_rss, the
                             amount mmput_exit() reserved and what the
                             pending-pages accounting releases), a fresh
                             live RSS read taken just before __mmput(),
                             and the node of the reaper. The gap between
                             charged and live measures how much reclaim
                             ate out of the queue before the reaper got
                             to it.

reap is emitted before __mmput() because the final mmdrop() inside
__mmput() may free the mm.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 include/linux/vm_event_item.h    |  5 +++
 include/trace/events/mm_reaper.h | 73 ++++++++++++++++++++++++++++++++
 kernel/fork.c                    | 14 +++++-
 mm/vmstat.c                      |  5 +++
 4 files changed, 95 insertions(+), 2 deletions(-)
 create mode 100644 include/trace/events/mm_reaper.h

diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 2628ccda0..de4dc20b7 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -179,6 +179,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		NRSWPIN,
 		NRSWPOUT,
 #endif /* CONFIG_SWAP */
+#ifdef CONFIG_ASYNC_MM_TEARDOWN
+		ASYNC_MM_TEARDOWN_QUEUED,
+		ASYNC_MM_TEARDOWN_SYNC,
+		ASYNC_MM_TEARDOWN_REJECTED,
+#endif
 		NR_VM_EVENT_ITEMS
 };
 
diff --git a/include/trace/events/mm_reaper.h b/include/trace/events/mm_reaper.h
new file mode 100644
index 000000000..315bced7c
--- /dev/null
+++ b/include/trace/events/mm_reaper.h
@@ -0,0 +1,73 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM mm_reaper
+
+#if !defined(_TRACE_MM_REAPER_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_MM_REAPER_H
+
+#include <linux/tracepoint.h>
+#include <linux/sched.h>
+#include <linux/topology.h>
+
+TRACE_EVENT(mm_async_teardown_queue,
+
+	TP_PROTO(struct mm_struct *mm, unsigned long rss),
+
+	TP_ARGS(mm, rss),
+
+	TP_STRUCT__entry(
+		__field(struct mm_struct *, mm)
+		__field(int, pid)
+		__array(char, comm, TASK_COMM_LEN)
+		__field(unsigned long, rss)
+		__field(int, node)
+	),
+
+	TP_fast_assign(
+		__entry->mm = mm;
+		__entry->pid = current->pid;
+		memcpy(__entry->comm, current->comm, TASK_COMM_LEN);
+		__entry->rss = rss;
+		__entry->node = numa_node_id();
+	),
+
+	TP_printk("mm=%p pid=%d comm=%s rss=%lukB node=%d",
+		__entry->mm,
+		__entry->pid,
+		__entry->comm,
+		__entry->rss << (PAGE_SHIFT - 10),
+		__entry->node
+	)
+);
+
+TRACE_EVENT(mm_async_teardown_reap,
+
+	TP_PROTO(struct mm_struct *mm, unsigned long charged_rss, unsigned long live_rss),
+
+	TP_ARGS(mm, charged_rss, live_rss),
+
+	TP_STRUCT__entry(
+		__field(struct mm_struct *, mm)
+		__field(unsigned long, charged_rss)
+		__field(unsigned long, live_rss)
+		__field(int, node)
+	),
+
+	TP_fast_assign(
+		__entry->mm = mm;
+		__entry->charged_rss = charged_rss;
+		__entry->live_rss = live_rss;
+		__entry->node = numa_node_id();
+	),
+
+	TP_printk("mm=%p charged_rss=%lukB live_rss=%lukB node=%d",
+		__entry->mm,
+		__entry->charged_rss << (PAGE_SHIFT - 10),
+		__entry->live_rss << (PAGE_SHIFT - 10),
+		__entry->node
+	)
+);
+
+#endif /* _TRACE_MM_REAPER_H */
+
+#include <trace/define_trace.h>
diff --git a/kernel/fork.c b/kernel/fork.c
index 0976770db..4bdffa188 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -126,6 +126,9 @@
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/task.h>
+#ifdef CONFIG_ASYNC_MM_TEARDOWN
+#include <trace/events/mm_reaper.h>
+#endif
 
 #include <kunit/visibility.h>
 
@@ -3461,8 +3464,10 @@ static bool async_mm_teardown_reserve(unsigned long rss)
 	return true;
 }
 
-static void async_mm_teardown_queue(struct mm_struct *mm)
+static void async_mm_teardown_queue(struct mm_struct *mm, unsigned long rss)
 {
+	count_vm_event(ASYNC_MM_TEARDOWN_QUEUED);
+	trace_mm_async_teardown_queue(mm, rss);
 	if (llist_add(&mm->async_reap_node, &mm_reaper_list))
 		wake_up(&mm_reaper_wait);
 }
@@ -3478,6 +3483,9 @@ static int mm_reaper(void *unused)
 		batch = llist_del_all(&mm_reaper_list);
 		llist_for_each_entry_safe(mm, n, batch, async_reap_node) {
 			unsigned long pages = mm->async_reap_rss;
+			unsigned long live = get_mm_rss(mm);
+
+			trace_mm_async_teardown_reap(mm, pages, live);
 
 			__mmput(mm); /* may free mm via mmdrop */
 			atomic_long_sub(pages, &mm_reaper_pending_pages);
@@ -3515,12 +3523,14 @@ void mmput_exit(struct mm_struct *mm)
 
 			if (async_mm_teardown_reserve(rss)) {
 				mm->async_reap_rss = rss;
-				async_mm_teardown_queue(mm);
+				async_mm_teardown_queue(mm, rss);
 				return;
 			}
+			count_vm_event(ASYNC_MM_TEARDOWN_REJECTED);
 		}
 	}
 
+	count_vm_event(ASYNC_MM_TEARDOWN_SYNC);
 	__mmput(mm);
 }
 
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 4e26e5fd6..e3ff30027 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1494,6 +1494,11 @@ const char * const vmstat_text[] = {
 	[I(NRSWPIN)]				= "nrswpin",
 	[I(NRSWPOUT)]				= "nrswpout",
 #endif /* CONFIG_SWAP */
+#ifdef CONFIG_ASYNC_MM_TEARDOWN
+	[I(ASYNC_MM_TEARDOWN_QUEUED)]	= "async_mm_teardown_queued",
+	[I(ASYNC_MM_TEARDOWN_SYNC)]		= "async_mm_teardown_sync",
+	[I(ASYNC_MM_TEARDOWN_REJECTED)]	= "async_mm_teardown_rejected",
+#endif
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
-- 
2.34.1

[RFC PATCH 7/7] exit: route exit_mm()'s final mmput() through mmput_exit()

[Aditya Sharma]

This makes the series operative. So far, nothing called mmput_exit().
Switch exit_mm() from mmput() to mmput_exit(), so the exiting task's final
reference drop can defer the address-space teardown to mm_reaper
instead of running exit_mmap() on the exiting CPU.

Only the exit path is routed. Every other mmput() caller (get_task_mm()
users like ptrace and /proc, kthread_unuse_mm(), etc.) still tears down
inline, and if one of those ends up holding the actual last reference,
behavior is unchanged -- the deferral applies only when the exiting
task's own drop is the final one.

The deferral triggers only when all of the following hold, and falls
back to inline __mmput() otherwise:

- vm.async_mm_teardown=1 (static key, default off)
- RSS >= vm.async_mm_teardown_thresh_pages (default 64MB)
- the mm is not an OOM target (MMF_OOM_TARGETED) and has not already
been reaped (MMF_OOM_SKIP)
- charging the RSS to the pending budget stays under
vm.async_mm_teardown_max_pending_pages

Results, bare metal, 32 CPUs / 32GB RAM (31 GiB usable), single
NUMA node, performance governor, medians of 10 reps, THP off unless noted,
vm.async_mm_teardown_max_pending_pages raised to 3/4 of RAM for the
async runs (the RAM/4 default would reject a single 16GB mm on this
box; see the cover letter's open question on the cap default):

reap latency (SIGKILL -> waitpid() returns), async off -> on:
1GB    18.01 ms  ->  0.14 ms
4GB    73.71 ms  ->  0.14 ms
16GB  282.16 ms  ->  0.15 ms
16GB   27.68 ms  ->  0.13 ms   (THP on)

voluntary exit (_exit() -> reap) matches within noise:
16GB  271.27 ms  ->  0.12 ms

redis-server with a 16GB populated dataset (real heap, VmRSS ~16.2GB):
      292.97 ms  ->  0.24 ms

32 simultaneous 0.5GB exits, tail until all reaped:
      157.42 ms  ->  0.32 ms

below-threshold 30MB process with the feature enabled:
        0.56 ms  ->  0.60 ms   (sync path, within noise)

A CONFIG_ASYNC_MM_TEARDOWN=n build of the same tree reproduces the
async-off column within noise (18.01 vs 18.01 ms at 1GB, 282.14 vs
282.16 ms at 16GB), so the config itself costs nothing when dark.

The work is moved, not eliminated: with async on, the 16GB region is
fully back in the buddy allocator ~240 ms after the kill (vs ~283 ms
when torn down inline), and the reaper spends ~1.18x the inline CPU on
the same teardowns (nice-19 kthread, cache-cold on another CPU). The
cover letter carries the full matrix: freeing-latency checks,
backpressure fallback, OOM-killer interaction (an OOM victim is never
queued, tracepoint-verified, including the CLONE_VM-sharer case),
tmpfs inode eviction in reaper context, sysctl toggling under load,
and a freezer cycle with a queued backlog.

Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in>
---
 kernel/exit.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/exit.c b/kernel/exit.c
index 1056422bc..6f97d204f 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -607,7 +607,7 @@ static void exit_mm(void)
 	task_unlock(current);
 	mmap_read_unlock(mm);
 	mm_update_next_owner(mm);
-	mmput(mm);
+	mmput_exit(mm);
 	if (test_thread_flag(TIF_MEMDIE))
 		exit_oom_victim();
 }
-- 
2.34.1

Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[Liam R . Howlett]

On 26/07/20 12:24AM, Aditya Sharma wrote:
> Address-space teardown on process exit runs synchronously in the dying
> task's context: exit_mm() -> mmput() -> __mmput() -> exit_mmap() walks
> page tables, updates rmap, frees the RSS and drops file references, all
> on the exiting CPU. For a multi-GB process that is hundreds of
> milliseconds of exit-path latency, paid by whoever is waiting on the
> death: a supervisor's kill-and-respawn cycle, a shell's waitpid(), an
> orchestrator reaping a fleet of workers. On the test box below, killing
> a 16GB process costs ~280ms before the parent's waitpid() returns.
> 
> This series adds CONFIG_ASYNC_MM_TEARDOWN: an opt-in, default-off path
> that defers __mmput() of large exiting processes to a dedicated kernel
> thread (mm_reaper), so the exiting CPU is released as soon as the task
> is reaped and the teardown runs off to the side.

This reads like it was written by an AI, but there is no assisted-by
tags.  Which AI did you use to generate these patches and/or cover
letter?

I do not see a benefit to running the task cleanup 'off the side'.  In
modern computers, that may mean switching core types that run at
different speeds.

Regardless of speed, the CPU that will be doing the work is remote to
the workload by design, (or may be, depending on the scheduling?).  That
is, you are tasking another CPU to do cleanup of local CPU-aware
information.. The ideal CPU to do a task exit is the one that's doing it
already.

There are hidden costs to your approach and I don't really understand
how this could possibly speed things up - even parallelized tasks will
take more CPU time in total, even if locking issues are avoided.

Sure, the task waiting for cleanup to complete may think it's done and
continue to do the Next Thing - but if it really needs the cleanup to be
completed then you've just made it impossible to know when it has
happened.

Also, you've just made this extremely hard to debug if something is
missed.

The work that is done in teardown is necessary.  Your time (and tokens?)
would be better spent trying to improve what we have to do instead of
shifting the work around.

Thanks,
Liam

Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[David Hildenbrand (Arm)]

On 7/19/26 19:54, Aditya Sharma wrote:
> Address-space teardown on process exit runs synchronously in the dying
> task's context: exit_mm() -> mmput() -> __mmput() -> exit_mmap() walks
> page tables, updates rmap, frees the RSS and drops file references, all
> on the exiting CPU. For a multi-GB process that is hundreds of
> milliseconds of exit-path latency, paid by whoever is waiting on the
> death: a supervisor's kill-and-respawn cycle, a shell's waitpid(), an
> orchestrator reaping a fleet of workers. On the test box below, killing
> a 16GB process costs ~280ms before the parent's waitpid() returns.
> 
> This series adds CONFIG_ASYNC_MM_TEARDOWN: an opt-in, default-off path
> that defers __mmput() of large exiting processes to a dedicated kernel
> thread (mm_reaper), so the exiting CPU is released as soon as the task
> is reaped and the teardown runs off to the side.

QEMU has an rather short implementation for async teardown using
clone(CLONE_VM), which is IIRC essentially the result of Claudios previous
kernel work you note below.

So nothing got merged because the problem was solvable in userspace.

Without something like CONFIG_ASYNC_MM_TEARDOWN in the kernel.

[1] https://github.com/qemu/qemu/blob/master/system/async-teardown.c

-- 
Cheers,

David

Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[Aditya Sharma]

Sent using Zoho Mail


From: David Hildenbrand (Arm) <david@kernel.org>
To: "Aditya Sharma"<adi.sharma@zohomail.in>, "Andrew Morton"<akpm@linux-foundation.org>, "Lorenzo Stoakes"<ljs@kernel.org>, "Liam R . Howlett"<liam@infradead.org>, "Vlastimil Babka"<vbabka@kernel.org>, "Mike Rapoport"<rppt@kernel.org>, "Suren Baghdasaryan"<surenb@google.com>, "Michal Hocko"<mhocko@suse.com>
Cc: "David Rientjes"<rientjes@google.com>, "Shakeel Butt"<shakeel.butt@linux.dev>, "Jonathan Corbet"<corbet@lwn.net>, "Shuah Khan"<skhan@linuxfoundation.org>, "Steven Rostedt"<rostedt@goodmis.org>, "Masami Hiramatsu"<mhiramat@kernel.org>, "Mathieu Desnoyers"<mathieu.desnoyers@efficios.com>, "Kees Cook"<kees@kernel.org>, "Ingo Molnar"<mingo@redhat.com>, "Peter Zijlstra"<peterz@infradead.org>, "Juri Lelli"<juri.lelli@redhat.com>, "Vincent Guittot"<vincent.guittot@linaro.org>, "Dietmar Eggemann"<dietmar.eggemann@arm.com>, "Ben Segall"<bsegall@google.com>, "Mel Gorman"<mgorman@suse.de>, "Valentin Schneider"<vschneid@redhat.com>, "K Prateek Nayak"<kprateek.nayak@amd.com>, <linux-mm@kvack.org>, <linux-doc@vger.kernel.org>, <linux-trace-kernel@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <imbrenda@linux.ibm.com>
Date: Mon, 20 Jul 2026 13:43:34 +0530
Subject: Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread


 > QEMU has an rather short implementation for async teardown using
 > clone(CLONE_VM), which is IIRC essentially the result of Claudios previous
 > kernel work you note below.
 > 
 > So nothing got merged because the problem was solvable in userspace.

Thanks for this. My related works paragraph is wrong on this then. It
says that the s390 case was addressed with KVM specific async destruction,
but what you point to is QEMU's shadow process. I will correct that in v2.

Yeah, that CLONE_VM trick works, so I would measure and compare them
and get back.

I plan to measure:

- Memory return latency
- N concurrent shadow teardowns on N cpus vs the reaper.

One difference between CLONE_VM approach and CONFIG_ASYNC_MM_TEARDOWN
is that the shadow process has to be created by the process that would
be torn down itself. If that same behaviour is wanted on a workload
where we dont own the source, CONFIG_ASYNC_MM_TEARDOWN enables it here
as well. Also, in the QEMU case, it also means that the deferred teardown
decision was made without knowing how large the process might become.
CONFIG_ASYNC_MM_TEARDOWN adds a gate where for the exiting process,
RSS is evaluated againsts the current backlog before queuing it for
async teardown.

I will drop a follow up with numbers

Aditya

Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[Aditya Sharma]

Sent using Zoho Mail


From: Liam R . Howlett <liam@infradead.org>
To: "Aditya Sharma"<adi.sharma@zohomail.in>
Cc: "Andrew Morton"<akpm@linux-foundation.org>, "David Hildenbrand"<david@kernel.org>, "Lorenzo Stoakes"<ljs@kernel.org>, "Vlastimil Babka"<vbabka@kernel.org>, "Mike Rapoport"<rppt@kernel.org>, "Suren Baghdasaryan"<surenb@google.com>, "Michal Hocko"<mhocko@suse.com>, "David Rientjes"<rientjes@google.com>, "Shakeel Butt"<shakeel.butt@linux.dev>, "Jonathan Corbet"<corbet@lwn.net>, "Shuah Khan"<skhan@linuxfoundation.org>, "Steven Rostedt"<rostedt@goodmis.org>, "Masami Hiramatsu"<mhiramat@kernel.org>, "Mathieu Desnoyers"<mathieu.desnoyers@efficios.com>, "Kees Cook"<kees@kernel.org>, "Ingo Molnar"<mingo@redhat.com>, "Peter Zijlstra"<peterz@infradead.org>, "Juri Lelli"<juri.lelli@redhat.com>, "Vincent Guittot"<vincent.guittot@linaro.org>, "Dietmar Eggemann"<dietmar.eggemann@arm.com>, "Ben Segall"<bsegall@google.com>, "Mel Gorman"<mgorman@suse.de>, "Valentin Schneider"<vschneid@redhat.com>, "K Prateek Nayak"<kprateek.nayak@amd.com>, <linux-mm@kvack.org>, <linux-doc@vger.kernel.org>, <linux-trace-kernel@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <imbrenda@linux.ibm.com>
Date: Mon, 20 Jul 2026 03:58:47 +0530
Subject: Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread


 > This reads like it was written by an AI, but there is no assisted-by
 > tags.  Which AI did you use to generate these patches and/or cover
 > letter?

I used claude opus 4.8 to refine commit messages (the design is mine, 
the model was used to convey the intentions concretely). Further, I used
claude fable 5 for drafting the cover letter and reviewing the series, where
it flagged issues (I fixed some of those by hand and some were fixed by fable itself,
which I then reviewed). It also assisted in writing the benchmark harness
behind the numbers and did the litmus test the cover letter mentions. Every numbers
was measured by me, on my hardware. I take full responsibily of the entire 
series.

My bad that I did not add an assisted-by tag. Should have added it to v1
v2 will add it to the patches

 > I do not see a benefit to running the task cleanup 'off the side'.  In
 > modern computers, that may mean switching core types that run at
 > different speeds.

That's why the reaper's affinity is a preference, not a bind (patch 2).
It uses kthread_affine_preferred() with housekeeping mask, so it can be
re-affined. kthread_bind_mask() would have taken that away. On a hybrid cpu,
it can be affined(by the (super)user) to whichever cores are appropriate. 

I do not have numbers for P core and E core teardown costs. Will measure and
send them.

 > Regardless of speed, the CPU that will be doing the work is remote to
 > the workload by design, (or may be, depending on the scheduling?).  That
 > is, you are tasking another CPU to do cleanup of local CPU-aware
 > information.. The ideal CPU to do a task exit is the one that's doing it
 > already.

Yes, it does cost something. Five consecutive 4GB inline teardowns in my testing
costed 372ms in exit path, while 440 ms in the case of kthread drain for the same
(five teardowns on another cpu). This cost became 1.18x. For multi gb mms,
the memory to be reaped would be much larger than the cpu cache can hold.
For small mms (< 64 MB, configurable by sysctl), the teardowns happen inline. 

The oom_reaper already reaps the victim's address space from a remote context,
as returning memory matters more than locality of teardown.

 > There are hidden costs to your approach and I don't really understand
 > how this could possibly speed things up - even parallelized tasks will
 > take more CPU time in total, even if locking issues are avoided.

This is not meant to speed up teardown. Rather the latency for someone waiting
on death is reduced. Killing a 16 GB redis cost 292.97ms before returning to
waitpid(). It costs 0.24 with async teardown. The memory itself was returned
in about 300ms.

"parallelized" - This series does not parallelize the teardown. The design is serial.
N procesess exiting together today would tear down concurrently today. The reaper here,
uses one thread (nice 19, affined to housekeeping mask).

The numbers in cover letter support that this work is memory bound rather than CPU bound.
In sync case: teardown is around 56 GB/s on my machine (1GB in 18.01 ms and 16GB in 282.16ms).
Thirty two concurrent 0.5GB exits reap in 157.42ms (around 101 GB/s). So, a concurrency
of 32 gets about 1.8x. The single reaper thread takes 650ms to clear the same (about 25 GB/s).
Thus, this serialization took 4x the drain time, but takes peak concurrency down to
one cpu. Memory actully comes back later. Whether that's a good tradeoff, depends on 
what else we wish to run on the box (open questions #1 in cover letter)

 > Sure, the task waiting for cleanup to complete may think it's done and
 > continue to do the Next Thing - but if it really needs the cleanup to be
 > completed

It is off by default and opt in (sysctl), so anything that depends on the
current semantics, continues to work exactly the same by doing nothing.

Also, mmput_async() already defers the teardown to a workqueue. So, in the current
kernel, there already exists a way where continuing to the next thing
does not necessarily mean that the memory is reclaimed.

 > completed then you've just made it impossible to know when it has
 > happened.

There are three vmstat counters and two tracepoint (patch 6).
Someone who opts in to the mechanism, and does need to make sure that the
cleanup is completed, can use the counters and tracepoints.

 > Also, you've just made this extremely hard to debug if something is
 > missed.

There are reap tracepoints that carry the pid, so we can get per pid completion status.

However, there is no number indicating the current depth of the queue. The series addds queued,
sync and rejected counters. Will add a reaped counter in v2 so that current queue depth
can be estimated. Please mention what else tracability can be added to this,
and what senairo do you have in mind that these counters and tracepoints might not
cover. I can add those to v2 as well.

 > The work that is done in teardown is necessary.  Your time (and tokens?)
 > would be better spent trying to improve what we have to do instead of
 > shifting the work around.

This does not compete with with current teardown path. it goes through the
same exit_mm(). Any future improvements there, also improve the reaper's drain
times.

Thanks

Aditya

Re: [RFC PATCH 0/7] mm: defer address-space teardown of large exiting processes to a kthread

[David Hildenbrand (Arm)]

[...]

> says that the s390 case was addressed with KVM specific async destruction,
> but what you point to is QEMU's shadow process. I will correct that in v2.

CLONE_VM seems to work for the use cases you had in mind.

There would have to be a *pretty* convincing story why we would want more
complexity in the kernel to handle this.

Some smaller performance improvements are not worth it.
-- 
Cheers,

David

Re: [RFC PATCH 6/7] mm: add tracepoints and vmstat counters for async teardown

[Steven Rostedt]

On Mon, 20 Jul 2026 00:24:08 +0530
Aditya Sharma <adi.sharma@zohomail.in> wrote:

> diff --git a/include/trace/events/mm_reaper.h b/include/trace/events/mm_reaper.h
> new file mode 100644
> index 000000000..315bced7c
> --- /dev/null
> +++ b/include/trace/events/mm_reaper.h
> @@ -0,0 +1,73 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#undef TRACE_SYSTEM
> +#define TRACE_SYSTEM mm_reaper
> +
> +#if !defined(_TRACE_MM_REAPER_H) || defined(TRACE_HEADER_MULTI_READ)
> +#define _TRACE_MM_REAPER_H
> +
> +#include <linux/tracepoint.h>
> +#include <linux/sched.h>
> +#include <linux/topology.h>
> +
> +TRACE_EVENT(mm_async_teardown_queue,
> +
> +	TP_PROTO(struct mm_struct *mm, unsigned long rss),
> +
> +	TP_ARGS(mm, rss),
> +
> +	TP_STRUCT__entry(
> +		__field(struct mm_struct *, mm)
> +		__field(int, pid)
> +		__array(char, comm, TASK_COMM_LEN)

There's an effort to make comm size more dynamic and we want to prevent more
memcpy of the comm based on TASK_COMM_LEN. Please change the above to:

		__string(comm, comm);


> +		__field(unsigned long, rss)
> +		__field(int, node)
> +	),
> +
> +	TP_fast_assign(
> +		__entry->mm = mm;
> +		__entry->pid = current->pid;
> +		memcpy(__entry->comm, current->comm, TASK_COMM_LEN);

And this to:

		__assign_str(comm);

> +		__entry->rss = rss;
> +		__entry->node = numa_node_id();
> +	),
> +
> +	TP_printk("mm=%p pid=%d comm=%s rss=%lukB node=%d",
> +		__entry->mm,
> +		__entry->pid,
> +		__entry->comm,

and this to:

		__get_str(comm),

Thanks,

-- Steve

> +		__entry->rss << (PAGE_SHIFT - 10),
> +		__entry->node
> +	)
> +);
> +