* [RFC PATCH] mm: allow smaller large folios to use lru_cache
@ 2026-07-02 12:23 Barry Song (Xiaomi)
2026-07-02 13:00 ` David Hildenbrand (Arm)
0 siblings, 1 reply; 3+ messages in thread
From: Barry Song (Xiaomi) @ 2026-07-02 12:23 UTC (permalink / raw)
To: akpm, linux-mm
Cc: linux-kernel, Barry Song (Xiaomi), David Hildenbrand,
Lorenzo Stoakes, Zi Yan, Baolin Wang, Liam R . Howlett,
Nico Pache, Ryan Roberts, Dev Jain, Lance Yang
For a system which primarily uses smaller orders for large folios,
it could be beneficial to enable lru_cache to avoid lock contention.
For large folios with higher orders, the number of folios involved
would be small, so the contention is less significant.
The following small program runs on a system with 16 KiB
mTHP enabled.
#include <pthread.h>
#include <sys/mman.h>
#include <string.h>
#define NUM_THREADS 20
#define MEM_SIZE (16 * 1024 * 1024)
#define LOOP_COUNT 1000
void* thread_worker(void* arg) {
void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
for (int i = 0; i < LOOP_COUNT; i++) {
memset(addr, 0x55, MEM_SIZE);
madvise(addr, MEM_SIZE, MADV_DONTNEED);
}
munmap(addr, MEM_SIZE);
pthread_exit(NULL);
}
int main() {
pthread_t threads[NUM_THREADS];
for (long t = 0; t < NUM_THREADS; t++) {
pthread_create(&threads[t], NULL, thread_worker, (void*)t);
}
for (int t = 0; t < NUM_THREADS; t++) {
pthread_join(threads[t], NULL);
}
return 0;
}
Before patch:
root@barry-desktop:/home/barry# time ./a.out
real 0m20.233s
user 0m21.739s
sys 6m21.143s
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
21158662 21158662 29.20 us 10.30 m 67.83 us 884 ns
13547 13547 14.68 us 198.93 ms 135.39 us 982 ns
rcu_state 41 41 1.83 us 75.21 us 3.46 us 1.27 us
rcu_state 34 34 1.83 us 62.09 us 2.60 us 1.16 us
5 5 21.31 us 106.57 us 25.77 us 18.17 us
5 5 3.01 us 15.07 us 5.19 us 1.53 us
3 3 6.42 us 19.26 us 9.63 us 4.22 us
3 3 4.20 us 12.61 us 7.45 us 2.10 us
2 2 5.03 us 10.06 us 7.88 us 2.18 us
2 2 2.79 us 5.59 us 3.34 us 2.25 us
1 1 4.70 us 4.70 us 4.70 us 4.70 us
1 1 9.77 us 9.77 us 9.77 us 9.77 us
After patch:
root@barry-desktop:/home/barry# time ./a.out
real 0m17.796s
user 0m24.962s
sys 5m28.774s
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
1407013 1407013 25.44 us 35.80 s 127.83 us 972 ns
845113 845113 47.96 us 40.54 s 152.86 us 1.21 us
rcu_state 1920 1920 5.87 us 11.27 ms 25.02 us 1.76 us
rcu_state 1889 1889 5.80 us 10.96 ms 30.35 us 1.61 us
140 140 3.97 us 555.52 us 12.51 us 1.20 us
10 10 61.34 us 613.42 us 96.90 us 44.48 us
8 8 24.03 us 192.26 us 37.64 us 6.95 us
8 8 12.32 us 98.56 us 25.55 us 4.16 us
Both system time and lock contention are noticeably reduced.
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
include/linux/swap.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8f0f68e245ba..fce1d84aa2c8 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
/*
* Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
* Holding small numbers of low-order mTHP folios in per-CPU LRU cache
- * will be sensible, but nobody has implemented and tested that yet.
+ * will be sensible.
*/
- return !folio_test_large(folio);
+ return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
}
extern atomic_t lru_disable_count;
--
2.39.3 (Apple Git-146)
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [RFC PATCH] mm: allow smaller large folios to use lru_cache
2026-07-02 12:23 [RFC PATCH] mm: allow smaller large folios to use lru_cache Barry Song (Xiaomi)
@ 2026-07-02 13:00 ` David Hildenbrand (Arm)
2026-07-06 9:51 ` Barry Song
0 siblings, 1 reply; 3+ messages in thread
From: David Hildenbrand (Arm) @ 2026-07-02 13:00 UTC (permalink / raw)
To: Barry Song (Xiaomi), akpm, linux-mm
Cc: linux-kernel, Lorenzo Stoakes, Zi Yan, Baolin Wang,
Liam R . Howlett, Nico Pache, Ryan Roberts, Dev Jain, Lance Yang
On 7/2/26 14:23, Barry Song (Xiaomi) wrote:
> For a system which primarily uses smaller orders for large folios,
> it could be beneficial to enable lru_cache to avoid lock contention.
>
> For large folios with higher orders, the number of folios involved
> would be small, so the contention is less significant.
>
> The following small program runs on a system with 16 KiB
> mTHP enabled.
>
> #include <pthread.h>
> #include <sys/mman.h>
> #include <string.h>
>
> #define NUM_THREADS 20
> #define MEM_SIZE (16 * 1024 * 1024)
> #define LOOP_COUNT 1000
>
> void* thread_worker(void* arg) {
> void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>
> for (int i = 0; i < LOOP_COUNT; i++) {
> memset(addr, 0x55, MEM_SIZE);
> madvise(addr, MEM_SIZE, MADV_DONTNEED);
> }
>
> munmap(addr, MEM_SIZE);
> pthread_exit(NULL);
> }
>
> int main() {
> pthread_t threads[NUM_THREADS];
>
> for (long t = 0; t < NUM_THREADS; t++) {
> pthread_create(&threads[t], NULL, thread_worker, (void*)t);
> }
>
> for (int t = 0; t < NUM_THREADS; t++) {
> pthread_join(threads[t], NULL);
> }
>
> return 0;
> }
>
> Before patch:
> root@barry-desktop:/home/barry# time ./a.out
>
> real 0m20.233s
> user 0m21.739s
> sys 6m21.143s
>
> Perf lock report:
> Name acquired contended avg wait total wait max wait min wait
>
> 21158662 21158662 29.20 us 10.30 m 67.83 us 884 ns
> 13547 13547 14.68 us 198.93 ms 135.39 us 982 ns
> rcu_state 41 41 1.83 us 75.21 us 3.46 us 1.27 us
> rcu_state 34 34 1.83 us 62.09 us 2.60 us 1.16 us
> 5 5 21.31 us 106.57 us 25.77 us 18.17 us
> 5 5 3.01 us 15.07 us 5.19 us 1.53 us
> 3 3 6.42 us 19.26 us 9.63 us 4.22 us
> 3 3 4.20 us 12.61 us 7.45 us 2.10 us
> 2 2 5.03 us 10.06 us 7.88 us 2.18 us
> 2 2 2.79 us 5.59 us 3.34 us 2.25 us
> 1 1 4.70 us 4.70 us 4.70 us 4.70 us
> 1 1 9.77 us 9.77 us 9.77 us 9.77 us
>
> After patch:
> root@barry-desktop:/home/barry# time ./a.out
>
> real 0m17.796s
> user 0m24.962s
> sys 5m28.774s
>
> Perf lock report:
> Name acquired contended avg wait total wait max wait min wait
>
> 1407013 1407013 25.44 us 35.80 s 127.83 us 972 ns
> 845113 845113 47.96 us 40.54 s 152.86 us 1.21 us
> rcu_state 1920 1920 5.87 us 11.27 ms 25.02 us 1.76 us
> rcu_state 1889 1889 5.80 us 10.96 ms 30.35 us 1.61 us
> 140 140 3.97 us 555.52 us 12.51 us 1.20 us
> 10 10 61.34 us 613.42 us 96.90 us 44.48 us
> 8 8 24.03 us 192.26 us 37.64 us 6.95 us
> 8 8 12.32 us 98.56 us 25.55 us 4.16 us
>
> Both system time and lock contention are noticeably reduced.
>
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Lorenzo Stoakes <ljs@kernel.org>
> Cc: Zi Yan <ziy@nvidia.com>
> Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
> Cc: Liam R. Howlett <liam@infradead.org>
> Cc: Nico Pache <npache@redhat.com>
> Cc: Ryan Roberts <ryan.roberts@arm.com>
> Cc: Dev Jain <dev.jain@arm.com>
> Cc: Lance Yang <lance.yang@linux.dev>
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
> include/linux/swap.h | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 8f0f68e245ba..fce1d84aa2c8 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
> /*
> * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
> * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
> - * will be sensible, but nobody has implemented and tested that yet.
> + * will be sensible.
> */
> - return !folio_test_large(folio);
> + return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
> }
>
> extern atomic_t lru_disable_count;
Not that __wp_can_reuse_large_anon_folio() currently assumes that large folios
are not in the LRU cache.
--
Cheers,
David
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [RFC PATCH] mm: allow smaller large folios to use lru_cache
2026-07-02 13:00 ` David Hildenbrand (Arm)
@ 2026-07-06 9:51 ` Barry Song
0 siblings, 0 replies; 3+ messages in thread
From: Barry Song @ 2026-07-06 9:51 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: akpm, linux-mm, linux-kernel, Lorenzo Stoakes, Zi Yan,
Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts, Dev Jain,
Lance Yang
On Thu, Jul 2, 2026 at 9:00 PM David Hildenbrand (Arm) <david@kernel.org> wrote:
[...]
> > diff --git a/include/linux/swap.h b/include/linux/swap.h
> > index 8f0f68e245ba..fce1d84aa2c8 100644
> > --- a/include/linux/swap.h
> > +++ b/include/linux/swap.h
> > @@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
> > /*
> > * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
> > * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
> > - * will be sensible, but nobody has implemented and tested that yet.
> > + * will be sensible.
> > */
> > - return !folio_test_large(folio);
> > + return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
> > }
> >
> > extern atomic_t lru_disable_count;
>
> Not that __wp_can_reuse_large_anon_folio() currently assumes that large folios
> are not in the LRU cache.
Thanks for pointing this out. I assume we could do something
similar to wp_can_reuse_anon_folio(). It won't always help
drain lru_cache, but it could help in some cases?
diff --git a/mm/memory.c b/mm/memory.c
index a29cd38ad547..db82ac8057ac 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4143,6 +4143,12 @@ static bool
__wp_can_reuse_large_anon_folio(struct folio *folio,
folio_unlock(folio);
}
+ if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
+ if (folio_ref_count(folio) > folio_large_mapcount(folio) + 1)
+ return false;
+ lru_add_drain();
+ }
+
if (folio_large_mapcount(folio) != folio_ref_count(folio))
return false;
Best Regards
Barry
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-07-06 9:51 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-02 12:23 [RFC PATCH] mm: allow smaller large folios to use lru_cache Barry Song (Xiaomi)
2026-07-02 13:00 ` David Hildenbrand (Arm)
2026-07-06 9:51 ` Barry Song
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.