[RFC PATCH v5] mm: retry page faults once under the per-VMA lock

[RFC PATCH v5] mm: retry page faults once under the per-VMA lock

[Hongru Zhang]

From: Hongru Zhang <zhanghongru@xiaomi.com>

The per-VMA lock fault path falls back to mmap_lock on VM_FAULT_RETRY.
When mmap_lock is write-contended, the fallback can reduce page-fault
throughput.

Add a single retry under the per-VMA lock in the arch fault handler
instead of adding a new VM_FAULT_* flag. This avoids overloading
vm_fault_t with retry policy and leaves the lower-level fault handlers
unchanged. It lets faults that can make progress on an immediate retry
stay on the per-VMA lock path, avoiding waits on mmap_lock when it is
write-contended and thereby improving page-fault throughput. Some faults
may retry unnecessarily, for example in the __vmf_anon_prepare() path or
device-private fault handling, but these cases are expected to be
infrequent and only add one cheap per-VMA lock attempt. If the second
attempt still returns VM_FAULT_RETRY, the fault continues through the
existing mmap_lock path.

Based on the stress model from Kunwu Chan and Wang Lian in RFC v2, we
adapted a benchmark [1] to a 20-core Intel i7-12700 desktop by reducing
the thread count and adjusting the memcg limits. The benchmark uses
concurrent page faults under memcg pressure with parallel munmap to
amplify mmap_lock read-write contention.

Filemap Throughput (higher is better):
+---------+------------+---------------------+
| Threads |  Vanilla   |       Patched       |
+---------+------------+---------------------+
|   40    | 1069.34 /s | 1400.13 /s (+30.9%) |
+---------+------------+---------------------+
|   60    | 1038.12 /s | 1683.37 /s (+62.2%) |
+---------+------------+---------------------+
|   80    | 1042.62 /s | 1767.83 /s (+69.6%) |
+---------+------------+---------------------+

mmap_lock contention count (lower is better):
+---------+-----------+---------+-----------+
| Threads |  Vanilla  | Patched | Reduction |
+---------+-----------+---------+-----------+
|   40    | 3,187,336 |  52,086 |  -98.4%   |
+---------+-----------+---------+-----------+
|   60    | 4,385,154 |  65,079 |  -98.5%   |
+---------+-----------+---------+-----------+
|   80    | 5,337,890 |  69,708 |  -98.7%   |
+---------+-----------+---------+-----------+

These results show that retrying once under the per-VMA lock keeps more
file-backed faults on the fast path, improving throughput and reducing
mmap_lock contention.

Using benchmark [2], we tested this on a 20-core Intel i7-12700 desktop
with a 2GB swapfile. The benchmark uses one pressure thread under memcg
limits to keep a 128MB non-zero anonymous mapping under swap pressure,
12 reader threads to fault it back in, and optional mmap writer threads
to amplify mmap_lock read-write contention. Each test ran for 60 seconds
and reported completed reader rounds per second under swap pressure.

Swap Throughput (higher is better):
+--------------+-------------+---------------------------+
| mmap writers |   Vanilla   |          Patched          |
+--------------+-------------+---------------------------+
|      0       | 17303.09 /s | 17899.48 /s       (+3.4%) |
+--------------+-------------+---------------------------+
|      4       | 12596.23 /s | 16095.20 /s      (+27.8%) |
+--------------+-------------+---------------------------+
|      8       |     0.58 /s | 15420.57 /s (+2658619.0%) |
+--------------+-------------+---------------------------+

With increasing mmap_lock write pressure, Vanilla degrades sharply and
drops to near zero at eight writers. Patched kernel holds up much better.

[1] https://gist.github.com/zhr250/c36c2c54d9351df37e12fd072d4926ef
[2] https://gist.github.com/zhr250/218ffe693f842346b56434483127422c

Signed-off-by: Hongru Zhang <zhanghongru@xiaomi.com>
Suggested-by: Barry Song <baohua@kernel.org>
Suggested-by: Suren Baghdasaryan <surenb@google.com>
Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
Changes since RFC v4:
- Drop `VM_FAULT_MAY_USE_VMA_LOCK` and always retry once under the
  per-VMA lock, based on feedback from Lorenzo and Barry. Thanks!

Changes since RFC v3:
- Keep VM_FAULT_RETRY unchanged and add VM_FAULT_MAY_USE_VMA_LOCK as an advisory bit
- Bound VMA-lock retries with FAULT_FLAG_TRIED
- Opt in filemap_fault() and do_swap_page() to VM_FAULT_MAY_USE_VMA_LOCK
- Rebased on mm-unstable

Changes since RFC v2:
- Redesigned as a single blacklist-based patch (v2 was 5 per-path patches)
- Added retry_vma loop to all architectures (not just x86)
- Rebased on mm-unstable

Changes since RFC v1:
- collect tags from Pedro, Kunwu and Lian, thanks!
- handle case (2), for uptodate folios, don't retry PF

Link to RFC v4:
https://lore.kernel.org/lkml/20260804095135.45897-1-zhanghongru@xiaomi.com/

Link to RFC v3:
https://lore.kernel.org/all/20260626075019.1833065-1-zhanghongru@xiaomi.com/

Link to RFC v2:
https://lore.kernel.org/all/20260430040427.4672-1-baohua@kernel.org/

Link to RFC v1:
https://lore.kernel.org/all/20251127011438.6918-1-21cnbao@gmail.com/

 arch/arm/mm/fault.c       | 8 ++++++++
 arch/arm64/mm/fault.c     | 8 ++++++++
 arch/loongarch/mm/fault.c | 8 ++++++++
 arch/powerpc/mm/fault.c   | 7 +++++++
 arch/riscv/mm/fault.c     | 8 ++++++++
 arch/s390/mm/fault.c      | 6 ++++++
 arch/x86/mm/fault.c       | 8 ++++++++
 7 files changed, 53 insertions(+)

diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index e62cc4be5adf..319fd89c4263 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -340,6 +340,7 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 	vm_fault_t fault;
 	unsigned int flags = FAULT_FLAG_DEFAULT;
 	vm_flags_t vm_flags = VM_ACCESS_FLAGS;
+	bool vma_lock_retried = false;
 
 	if (kprobe_page_fault(regs, fsr))
 		return 0;
@@ -391,6 +392,7 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 	if (!(flags & FAULT_FLAG_USER))
 		goto lock_mmap;
 
+lock_vma:
 	vma = lock_vma_under_rcu(mm, addr);
 	if (!vma)
 		goto lock_mmap;
@@ -420,6 +422,12 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 			goto no_context;
 		return 0;
 	}
+
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
+
 lock_mmap:
 
 retry:
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0b52557652be..b8633863380e 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -610,6 +610,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 	struct vm_area_struct *vma;
 	int si_code;
 	int pkey = -1;
+	bool vma_lock_retried = false;
 
 	if (kprobe_page_fault(regs, esr))
 		return 0;
@@ -678,6 +679,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 	if (!(mm_flags & FAULT_FLAG_USER))
 		goto lock_mmap;
 
+lock_vma:
 	vma = lock_vma_under_rcu(mm, addr);
 	if (!vma)
 		goto lock_mmap;
@@ -724,6 +726,12 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 			goto no_context;
 		return 0;
 	}
+
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
+
 lock_mmap:
 
 retry:
diff --git a/arch/loongarch/mm/fault.c b/arch/loongarch/mm/fault.c
index 2c93d33356e5..ef6ea847b1e0 100644
--- a/arch/loongarch/mm/fault.c
+++ b/arch/loongarch/mm/fault.c
@@ -181,6 +181,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs,
 	struct mm_struct *mm = tsk->mm;
 	struct vm_area_struct *vma = NULL;
 	vm_fault_t fault;
+	bool vma_lock_retried = false;
 
 	if (kprobe_page_fault(regs, current->thread.trap_nr))
 		return;
@@ -219,6 +220,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs,
 	if (!(flags & FAULT_FLAG_USER))
 		goto lock_mmap;
 
+lock_vma:
 	vma = lock_vma_under_rcu(mm, address);
 	if (!vma)
 		goto lock_mmap;
@@ -265,6 +267,12 @@ static void __kprobes __do_page_fault(struct pt_regs *regs,
 			no_context(regs, write, address);
 		return;
 	}
+
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
+
 lock_mmap:
 
 retry:
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index 806c74e0d5ab..06018b6d7086 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -422,6 +422,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
 	int is_write = page_fault_is_write(error_code);
 	vm_fault_t fault, major = 0;
 	bool kprobe_fault = kprobe_page_fault(regs, 11);
+	bool vma_lock_retried = false;
 
 	if (unlikely(debugger_fault_handler(regs) || kprobe_fault))
 		return 0;
@@ -487,6 +488,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
 	if (!(flags & FAULT_FLAG_USER))
 		goto lock_mmap;
 
+lock_vma:
 	vma = lock_vma_under_rcu(mm, address);
 	if (!vma)
 		goto lock_mmap;
@@ -517,6 +519,11 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
 	if (fault_signal_pending(fault, regs))
 		return user_mode(regs) ? 0 : SIGBUS;
 
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
+
 lock_mmap:
 
 	/* When running in the kernel we expect faults to occur only to
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4..ff861793dba9 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -284,6 +284,7 @@ void handle_page_fault(struct pt_regs *regs)
 	unsigned int flags = FAULT_FLAG_DEFAULT;
 	int code = SEGV_MAPERR;
 	vm_fault_t fault;
+	bool vma_lock_retried = false;
 
 	cause = regs->cause;
 	addr = regs->badaddr;
@@ -347,6 +348,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (!(flags & FAULT_FLAG_USER))
 		goto lock_mmap;
 
+lock_vma:
 	vma = lock_vma_under_rcu(mm, addr);
 	if (!vma)
 		goto lock_mmap;
@@ -376,6 +378,12 @@ void handle_page_fault(struct pt_regs *regs)
 			no_context(regs, addr);
 		return;
 	}
+
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
+
 lock_mmap:
 
 retry:
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index 028aeb9c48d6..faae0e91b90b 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -271,6 +271,7 @@ static void do_exception(struct pt_regs *regs, int access)
 	unsigned int flags;
 	vm_fault_t fault;
 	bool is_write;
+	bool vma_lock_retried = false;
 
 	/*
 	 * The instruction that caused the program check has
@@ -294,6 +295,7 @@ static void do_exception(struct pt_regs *regs, int access)
 		flags |= FAULT_FLAG_WRITE;
 	if (!(flags & FAULT_FLAG_USER))
 		goto lock_mmap;
+lock_vma:
 	vma = lock_vma_under_rcu(mm, address);
 	if (!vma)
 		goto lock_mmap;
@@ -318,6 +320,10 @@ static void do_exception(struct pt_regs *regs, int access)
 			handle_fault_error_nolock(regs, 0);
 		return;
 	}
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
 lock_mmap:
 retry:
 	vma = lock_mm_and_find_vma(mm, address, regs);
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 45b99c3b1442..c3ab30d32a15 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1222,6 +1222,7 @@ void do_user_addr_fault(struct pt_regs *regs,
 	struct mm_struct *mm;
 	vm_fault_t fault;
 	unsigned int flags = FAULT_FLAG_DEFAULT;
+	bool vma_lock_retried = false;
 
 	tsk = current;
 	mm = tsk->mm;
@@ -1331,6 +1332,7 @@ void do_user_addr_fault(struct pt_regs *regs,
 	if (!(flags & FAULT_FLAG_USER))
 		goto lock_mmap;
 
+lock_vma:
 	vma = lock_vma_under_rcu(mm, address);
 	if (!vma)
 		goto lock_mmap;
@@ -1360,6 +1362,12 @@ void do_user_addr_fault(struct pt_regs *regs,
 						 ARCH_DEFAULT_PKEY);
 		return;
 	}
+
+	if (!vma_lock_retried) {
+		vma_lock_retried = true;
+		goto lock_vma;
+	}
+
 lock_mmap:
 
 retry:

base-commit: e737cebb8de0d38e8f64584a8bbfbcf9176c7537
-- 
2.43.0

Re: [RFC PATCH v5] mm: retry page faults once under the per-VMA lock

[Andrew Morton]

On Fri, 14 Aug 2026 16:53:00 +0800 Hongru Zhang <zhanghongru06@gmail.com> wrote:

> From: Hongru Zhang <zhanghongru@xiaomi.com>
> 
> The per-VMA lock fault path falls back to mmap_lock on VM_FAULT_RETRY.
> When mmap_lock is write-contended, the fallback can reduce page-fault
> throughput.
> 
> Add a single retry under the per-VMA lock in the arch fault handler
> instead of adding a new VM_FAULT_* flag.
>
> ...
> 
> Swap Throughput (higher is better):
> +--------------+-------------+---------------------------+
> | mmap writers |   Vanilla   |          Patched          |
> +--------------+-------------+---------------------------+
> |      0       | 17303.09 /s | 17899.48 /s       (+3.4%) |
> +--------------+-------------+---------------------------+
> |      4       | 12596.23 /s | 16095.20 /s      (+27.8%) |
> +--------------+-------------+---------------------------+
> |      8       |     0.58 /s | 15420.57 /s (+2658619.0%) |
> +--------------+-------------+---------------------------+

Well.

> With increasing mmap_lock write pressure, Vanilla degrades sharply and
> drops to near zero at eight writers. Patched kernel holds up much better.

Thanks for working on this.

Are you able to help us understand how much effect this change will
have upon any real-world workloads?

Re: [RFC PATCH v5] mm: retry page faults once under the per-VMA lock

[Hongru Zhang]

On Sat, Aug 15, 2026 at 2:51 AM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Fri, 14 Aug 2026 16:53:00 +0800 Hongru Zhang <zhanghongru06@gmail.com> wrote:
>
> > From: Hongru Zhang <zhanghongru@xiaomi.com>
> >
> > The per-VMA lock fault path falls back to mmap_lock on VM_FAULT_RETRY.
> > When mmap_lock is write-contended, the fallback can reduce page-fault
> > throughput.
> >
> > Add a single retry under the per-VMA lock in the arch fault handler
> > instead of adding a new VM_FAULT_* flag.
> >
> > ...
> >
> > Swap Throughput (higher is better):
> > +--------------+-------------+---------------------------+
> > | mmap writers |   Vanilla   |          Patched          |
> > +--------------+-------------+---------------------------+
> > |      0       | 17303.09 /s | 17899.48 /s       (+3.4%) |
> > +--------------+-------------+---------------------------+
> > |      4       | 12596.23 /s | 16095.20 /s      (+27.8%) |
> > +--------------+-------------+---------------------------+
> > |      8       |     0.58 /s | 15420.57 /s (+2658619.0%) |
> > +--------------+-------------+---------------------------+
>
> Well.
>
> > With increasing mmap_lock write pressure, Vanilla degrades sharply and
> > drops to near zero at eight writers. Patched kernel holds up much better.
>
> Thanks for working on this.
>
> Are you able to help us understand how much effect this change will
> have upon any real-world workloads?

Thanks for raising this.

I'll run some real-world workload tests and follow up with the results.

Re: [RFC PATCH v5] mm: retry page faults once under the per-VMA lock

[Barry Song]

On Mon, Aug 17, 2026 at 11:02 AM Hongru Zhang <zhanghongru06@gmail.com> wrote:
>
> On Sat, Aug 15, 2026 at 2:51 AM Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Fri, 14 Aug 2026 16:53:00 +0800 Hongru Zhang <zhanghongru06@gmail.com> wrote:
> >
> > > From: Hongru Zhang <zhanghongru@xiaomi.com>
> > >
> > > The per-VMA lock fault path falls back to mmap_lock on VM_FAULT_RETRY.
> > > When mmap_lock is write-contended, the fallback can reduce page-fault
> > > throughput.
> > >
> > > Add a single retry under the per-VMA lock in the arch fault handler
> > > instead of adding a new VM_FAULT_* flag.
> > >
> > > ...
> > >
> > > Swap Throughput (higher is better):
> > > +--------------+-------------+---------------------------+
> > > | mmap writers |   Vanilla   |          Patched          |
> > > +--------------+-------------+---------------------------+
> > > |      0       | 17303.09 /s | 17899.48 /s       (+3.4%) |
> > > +--------------+-------------+---------------------------+
> > > |      4       | 12596.23 /s | 16095.20 /s      (+27.8%) |
> > > +--------------+-------------+---------------------------+
> > > |      8       |     0.58 /s | 15420.57 /s (+2658619.0%) |
> > > +--------------+-------------+---------------------------+
> >
> > Well.
> >
> > > With increasing mmap_lock write pressure, Vanilla degrades sharply and
> > > drops to near zero at eight writers. Patched kernel holds up much better.
> >
> > Thanks for working on this.
> >
> > Are you able to help us understand how much effect this change will
> > have upon any real-world workloads?
>
> Thanks for raising this.
>
> I'll run some real-world workload tests and follow up with the results.

I guess you just mean double-checking the data with the latest code.

Note that this is a real issue observed with a real workload. In an
earlier version, we had the following data from Douyin (the Chinese
version of TikTok). Oven’s data from the earlier version is shown below
[1].

"The dramatically reduced mmap_lock contention leads to a
significant improvement in Douyin performance.

Douyin (the Chinese version of TikTok) warm start on a smartphone with
8GB RAM."

== mmap_lock Acquisitions And Wait Time ==

Metric                    Before (Avg)    After (Avg)    Change
------------------------------------------------------------------------
Read Lock Count           20,010          5,719          -71.42%
Read Total Wait (us)      10,695,877     408,436        -96.18%
Read Avg Wait (us)        534.00         71.00           -86.70%
Write Lock Count          838             909            +8.47%
Write Total Wait (us)     501,293        97,633          -80.52%
Write Avg Wait (us)       598.00         107.00          -82.11%


== Read Lock Waiting Time Distribution of mmap_lock ==

Range (us)                 Before (Avg)    After (Avg)    Change
------------------------------------------------------------------------
[0, 1)                     9,927           4,286          -56.82%
[1, 10)                    9,179           1,327          -85.54%
[10, 100)                  191             88             -53.93%
[100, 1000)                57              6              -89.47%
[1000, 10000)              328             9              -97.26%
[10000, 100000)            328             6              -98.17%
[100000, 1000000)          0               0              N/A
[1000000, +)               0               0              N/A

== Write Lock Waiting Time Distribution of mmap_lock ==

Range (us)                 Before (Avg)    After (Avg)    Change
------------------------------------------------------------------------
[0, 1)                     250             300            +20.00%
[1, 10)                    483             556            +15.11%
[10, 100)                  52              41             -21.15%
[100, 1000)                12              5              -58.33%
[1000, 10000)              22              4              -81.82%
[10000, 100000)            16              1              -93.75%
[100000, 1000000)          0               0              N/A
[1000000, +)               0               0              N/A


You also have the Pixel 6 data showing that `mmap_lock` is a major
source of frame drops.
I guess you may just want to re-test those Android cases with the latest
code from this version.

[1] https://lore.kernel.org/all/20260430040427.4672-1-baohua@kernel.org/

Thanks
Barry