* [RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache
2026-07-09 8:15 [RFC PATCH v2 0/2] mm: enable lru cache for smaller large folios Barry Song (Xiaomi)
@ 2026-07-09 8:15 ` Barry Song (Xiaomi)
2026-07-09 8:15 ` [RFC PATCH v2 2/2] mm: improve large folio reuse for LRU-cached folios Barry Song (Xiaomi)
1 sibling, 0 replies; 3+ messages in thread
From: Barry Song (Xiaomi) @ 2026-07-09 8:15 UTC (permalink / raw)
To: akpm, david, linux-mm, ljs
Cc: baolin.wang, dev.jain, lance.yang, liam, linux-kernel, npache,
ryan.roberts, ziy, vbabka, rppt, surenb, mhocko,
Barry Song (Xiaomi)
For a system which primarily uses smaller orders for large folios,
it could be beneficial to enable lru_cache to avoid lock contention.
For large folios with higher orders, the number of folios involved
would be small, so the contention is less significant.
The following small program runs on a system with 16 KiB
mTHP enabled.
#include <pthread.h>
#include <sys/mman.h>
#include <string.h>
#define NUM_THREADS 20
#define MEM_SIZE (16 * 1024 * 1024)
#define LOOP_COUNT 1000
void* thread_worker(void* arg) {
void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
for (int i = 0; i < LOOP_COUNT; i++) {
memset(addr, 0x55, MEM_SIZE);
madvise(addr, MEM_SIZE, MADV_DONTNEED);
}
munmap(addr, MEM_SIZE);
pthread_exit(NULL);
}
int main() {
pthread_t threads[NUM_THREADS];
for (long t = 0; t < NUM_THREADS; t++) {
pthread_create(&threads[t], NULL, thread_worker, (void*)t);
}
for (int t = 0; t < NUM_THREADS; t++) {
pthread_join(threads[t], NULL);
}
return 0;
}
Before patch:
root@barry-desktop:/home/barry# time ./a.out
real 0m20.233s
user 0m21.739s
sys 6m21.143s
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
21158662 21158662 29.20 us 10.30 m 67.83 us 884 ns
13547 13547 14.68 us 198.93 ms 135.39 us 982 ns
rcu_state 41 41 1.83 us 75.21 us 3.46 us 1.27 us
rcu_state 34 34 1.83 us 62.09 us 2.60 us 1.16 us
5 5 21.31 us 106.57 us 25.77 us 18.17 us
5 5 3.01 us 15.07 us 5.19 us 1.53 us
3 3 6.42 us 19.26 us 9.63 us 4.22 us
3 3 4.20 us 12.61 us 7.45 us 2.10 us
2 2 5.03 us 10.06 us 7.88 us 2.18 us
2 2 2.79 us 5.59 us 3.34 us 2.25 us
1 1 4.70 us 4.70 us 4.70 us 4.70 us
1 1 9.77 us 9.77 us 9.77 us 9.77 us
After patch:
root@barry-desktop:/home/barry# time ./a.out
real 0m17.796s
user 0m24.962s
sys 5m28.774s
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
1407013 1407013 25.44 us 35.80 s 127.83 us 972 ns
845113 845113 47.96 us 40.54 s 152.86 us 1.21 us
rcu_state 1920 1920 5.87 us 11.27 ms 25.02 us 1.76 us
rcu_state 1889 1889 5.80 us 10.96 ms 30.35 us 1.61 us
140 140 3.97 us 555.52 us 12.51 us 1.20 us
10 10 61.34 us 613.42 us 96.90 us 44.48 us
8 8 24.03 us 192.26 us 37.64 us 6.95 us
8 8 12.32 us 98.56 us 25.55 us 4.16 us
Both system time and lock contention are noticeably reduced.
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
include/linux/swap.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8d19be675baf..9c5f7a11c7b0 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
/*
* Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
* Holding small numbers of low-order mTHP folios in per-CPU LRU cache
- * will be sensible, but nobody has implemented and tested that yet.
+ * will be sensible.
*/
- return !folio_test_large(folio);
+ return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
}
extern atomic_t lru_disable_count;
--
2.39.3 (Apple Git-146)
^ permalink raw reply related [flat|nested] 3+ messages in thread* [RFC PATCH v2 2/2] mm: improve large folio reuse for LRU-cached folios
2026-07-09 8:15 [RFC PATCH v2 0/2] mm: enable lru cache for smaller large folios Barry Song (Xiaomi)
2026-07-09 8:15 ` [RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache Barry Song (Xiaomi)
@ 2026-07-09 8:15 ` Barry Song (Xiaomi)
1 sibling, 0 replies; 3+ messages in thread
From: Barry Song (Xiaomi) @ 2026-07-09 8:15 UTC (permalink / raw)
To: akpm, david, linux-mm, ljs
Cc: baolin.wang, dev.jain, lance.yang, liam, linux-kernel, npache,
ryan.roberts, ziy, vbabka, rppt, surenb, mhocko,
Barry Song (Xiaomi)
Large folios may now reside in the per-CPU LRU cache. Before
attempting to reuse them, drain the local LRU cache, which
can still be beneficial in cases where the folios are likely
to remain in this CPU's LRU cache:
int main(int argc, char *argv[])
{
int i;
while (1) {
volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS,
-1, 0);
for (int i = 0; i < SIZE / sizeof(int); i++)
p[i] = i;
madvise((void *)p, SIZE, MADV_PAGEOUT);
if (!fork())
_exit(0);
for (int i = 0; i < SIZE / sizeof(int); i++)
p[i] = i;
munmap((void *)p, SIZE);
}
return 0;
}
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
mm/memory.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/mm/memory.c b/mm/memory.c
index 5689b7cff76c..1d08ed5ba99b 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4136,6 +4136,19 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
folio_unlock(folio);
}
+ if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
+ if (folio_ref_count(folio) > folio_large_mapcount(folio) + 1)
+ return false;
+ /*
+ * A global LRU drain is too expensive, but a local drain
+ * can still be beneficial. For example, large folios that
+ * have just been swapped in and fall back from
+ * do_swap_page() to do_wp_page() are likely still in this
+ * CPU's LRU cache.
+ */
+ lru_add_drain();
+ }
+
if (folio_large_mapcount(folio) != folio_ref_count(folio))
return false;
--
2.39.3 (Apple Git-146)
^ permalink raw reply related [flat|nested] 3+ messages in thread