[RFC PATCH v2 0/2] mm: enable lru cache for smaller large folios

[RFC PATCH v2 0/2] mm: enable lru cache for smaller large folios

[Barry Song (Xiaomi)]

This patchset enables the per-CPU LRU cache for large folios smaller
than `COSTLY_ORDER`. It is particularly beneficial on systems that use
relatively small large folios. For larger folios, the benefit is likely
to be smaller because far fewer folios are expected to contend for the
LRU cache.

Patch 1/2 enables the per-CPU LRU cache for eligible large folios.
Patch 2/2 updates `__wp_can_reuse_large_anon_folio()` to account for
the fact that large folios may now also reside in the per-CPU LRU cache
before they are reused.

-RFC v2:
 * Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large
   folios. As David pointed out, it currently does not account for
   large folios residing in the per-CPU LRU cache.

Barry Song (Xiaomi) (2):
  mm: allow smaller large folios to use lru_cache
  mm: improve large folio reuse for LRU-cached folios

 include/linux/swap.h |  4 ++--
 mm/memory.c          | 13 +++++++++++++
 2 files changed, 15 insertions(+), 2 deletions(-)

-- 
2.39.3 (Apple Git-146)

[RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache

[Barry Song (Xiaomi)]

For a system which primarily uses smaller orders for large folios,
it could be beneficial to enable lru_cache to avoid lock contention.

For large folios with higher orders, the number of folios involved
would be small, so the contention is less significant.

The following small program runs on a system with 16 KiB
mTHP enabled.

 #include <pthread.h>
 #include <sys/mman.h>
 #include <string.h>

 #define NUM_THREADS     20
 #define MEM_SIZE        (16 * 1024 * 1024)
 #define LOOP_COUNT      1000

 void* thread_worker(void* arg) {
     void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

     for (int i = 0; i < LOOP_COUNT; i++) {
         memset(addr, 0x55, MEM_SIZE);
         madvise(addr, MEM_SIZE, MADV_DONTNEED);
     }

     munmap(addr, MEM_SIZE);
     pthread_exit(NULL);
 }

 int main() {
     pthread_t threads[NUM_THREADS];

     for (long t = 0; t < NUM_THREADS; t++) {
         pthread_create(&threads[t], NULL, thread_worker, (void*)t);
     }

     for (int t = 0; t < NUM_THREADS; t++) {
         pthread_join(threads[t], NULL);
     }

     return 0;
 }

Before patch:
root@barry-desktop:/home/barry# time ./a.out

real	0m20.233s
user	0m21.739s
sys	6m21.143s

Perf lock report:
      Name   acquired  contended     avg wait   total wait     max wait     min wait

             21158662   21158662     29.20 us     10.30 m      67.83 us       884 ns
                13547      13547     14.68 us    198.93 ms    135.39 us       982 ns
 rcu_state         41         41      1.83 us     75.21 us      3.46 us      1.27 us
 rcu_state         34         34      1.83 us     62.09 us      2.60 us      1.16 us
                    5          5     21.31 us    106.57 us     25.77 us     18.17 us
                    5          5      3.01 us     15.07 us      5.19 us      1.53 us
                    3          3      6.42 us     19.26 us      9.63 us      4.22 us
                    3          3      4.20 us     12.61 us      7.45 us      2.10 us
                    2          2      5.03 us     10.06 us      7.88 us      2.18 us
                    2          2      2.79 us      5.59 us      3.34 us      2.25 us
                    1          1      4.70 us      4.70 us      4.70 us      4.70 us
                    1          1      9.77 us      9.77 us      9.77 us      9.77 us

After patch:
root@barry-desktop:/home/barry# time ./a.out

real	0m17.796s
user	0m24.962s
sys	5m28.774s

Perf lock report:
      Name   acquired  contended     avg wait   total wait     max wait     min wait

              1407013    1407013     25.44 us     35.80 s     127.83 us       972 ns
               845113     845113     47.96 us     40.54 s     152.86 us      1.21 us
 rcu_state       1920       1920      5.87 us     11.27 ms     25.02 us      1.76 us
 rcu_state       1889       1889      5.80 us     10.96 ms     30.35 us      1.61 us
                  140        140      3.97 us    555.52 us     12.51 us      1.20 us
                   10         10     61.34 us    613.42 us     96.90 us     44.48 us
                    8          8     24.03 us    192.26 us     37.64 us      6.95 us
                    8          8     12.32 us     98.56 us     25.55 us      4.16 us

Both system time and lock contention are noticeably reduced.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 include/linux/swap.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8d19be675baf..9c5f7a11c7b0 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
 	/*
 	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
 	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
-	 * will be sensible, but nobody has implemented and tested that yet.
+	 * will be sensible.
 	 */
-	return !folio_test_large(folio);
+	return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
 }
 
 extern atomic_t lru_disable_count;
-- 
2.39.3 (Apple Git-146)

[RFC PATCH v2 2/2] mm: improve large folio reuse for LRU-cached folios

[Barry Song (Xiaomi)]

Large folios may now reside in the per-CPU LRU cache. Before
attempting to reuse them, drain the local LRU cache, which
can still be beneficial in cases where the folios are likely
to remain in this CPU's LRU cache:

int main(int argc, char *argv[])
{
    int i;
    while (1) {
        volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
                               MAP_PRIVATE | MAP_ANONYMOUS,
                               -1, 0);
        for (int i = 0; i < SIZE / sizeof(int); i++)
            p[i] = i;
        madvise((void *)p, SIZE, MADV_PAGEOUT);
        if (!fork())
            _exit(0);
        for (int i = 0; i < SIZE / sizeof(int); i++)
            p[i] = i;
        munmap((void *)p, SIZE);
    }
    return 0;
}

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/memory.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/mm/memory.c b/mm/memory.c
index 5689b7cff76c..1d08ed5ba99b 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4136,6 +4136,19 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
 		folio_unlock(folio);
 	}
 
+	if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
+		if (folio_ref_count(folio) > folio_large_mapcount(folio) + 1)
+			return false;
+		/*
+		 * A global LRU drain is too expensive, but a local drain
+		 * can still be beneficial. For example, large folios that
+		 * have just been swapped in and fall back from
+		 * do_swap_page() to do_wp_page() are likely still in this
+		 * CPU's LRU cache.
+		 */
+		lru_add_drain();
+	}
+
 	if (folio_large_mapcount(folio) != folio_ref_count(folio))
 		return false;
 
-- 
2.39.3 (Apple Git-146)

Re: [RFC PATCH v2 2/2] mm: improve large folio reuse for LRU-cached folios

[David Hildenbrand (Arm)]

On 7/9/26 10:15, Barry Song (Xiaomi) wrote:
> Large folios may now reside in the per-CPU LRU cache. Before
> attempting to reuse them, drain the local LRU cache, which
> can still be beneficial in cases where the folios are likely
> to remain in this CPU's LRU cache:
> 
> int main(int argc, char *argv[])
> {
>     int i;
>     while (1) {
>         volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
>                                MAP_PRIVATE | MAP_ANONYMOUS,
>                                -1, 0);
>         for (int i = 0; i < SIZE / sizeof(int); i++)
>             p[i] = i;
>         madvise((void *)p, SIZE, MADV_PAGEOUT);
>         if (!fork())
>             _exit(0);
>         for (int i = 0; i < SIZE / sizeof(int); i++)
>             p[i] = i;
>         munmap((void *)p, SIZE);
>     }
>     return 0;
> }
> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  mm/memory.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/mm/memory.c b/mm/memory.c
> index 5689b7cff76c..1d08ed5ba99b 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -4136,6 +4136,19 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
>  		folio_unlock(folio);
>  	}
>  
> +	if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
> +		if (folio_ref_count(folio) > folio_large_mapcount(folio) + 1)

I assume that can just be a "!=".

> +			return false;
> +		/*
> +		 * A global LRU drain is too expensive, but a local drain
> +		 * can still be beneficial. For example, large folios that
> +		 * have just been swapped in and fall back from
> +		 * do_swap_page() to do_wp_page() are likely still in this
> +		 * CPU's LRU cache.
> +		 */

Do we really need that comment? It's just the same as we have in
wp_can_reuse_anon_folio(). So either drop it or use the same comment.

I was wondering whether we could unify some logic with
wp_can_reuse_anon_folio(), but the checks are just to different.


Apart from that LGTM. I'm sure there are some micro-optimizations to be had, but
this here should suffice.

-- 
Cheers,

David

Re: [RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache

[David Hildenbrand (Arm)]

On 7/9/26 10:15, Barry Song (Xiaomi) wrote:
> For a system which primarily uses smaller orders for large folios,
> it could be beneficial to enable lru_cache to avoid lock contention.
> 
> For large folios with higher orders, the number of folios involved
> would be small, so the contention is less significant.
> 
> The following small program runs on a system with 16 KiB
> mTHP enabled.
> 
>  #include <pthread.h>
>  #include <sys/mman.h>
>  #include <string.h>
> 
>  #define NUM_THREADS     20
>  #define MEM_SIZE        (16 * 1024 * 1024)
>  #define LOOP_COUNT      1000
> 
>  void* thread_worker(void* arg) {
>      void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
>                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> 
>      for (int i = 0; i < LOOP_COUNT; i++) {
>          memset(addr, 0x55, MEM_SIZE);
>          madvise(addr, MEM_SIZE, MADV_DONTNEED);
>      }
> 
>      munmap(addr, MEM_SIZE);
>      pthread_exit(NULL);
>  }
> 
>  int main() {
>      pthread_t threads[NUM_THREADS];
> 
>      for (long t = 0; t < NUM_THREADS; t++) {
>          pthread_create(&threads[t], NULL, thread_worker, (void*)t);
>      }
> 
>      for (int t = 0; t < NUM_THREADS; t++) {
>          pthread_join(threads[t], NULL);
>      }
> 
>      return 0;
>  }
> 
> Before patch:
> root@barry-desktop:/home/barry# time ./a.out
> 
> real	0m20.233s
> user	0m21.739s
> sys	6m21.143s
> 
> Perf lock report:
>       Name   acquired  contended     avg wait   total wait     max wait     min wait
> 
>              21158662   21158662     29.20 us     10.30 m      67.83 us       884 ns
>                 13547      13547     14.68 us    198.93 ms    135.39 us       982 ns
>  rcu_state         41         41      1.83 us     75.21 us      3.46 us      1.27 us
>  rcu_state         34         34      1.83 us     62.09 us      2.60 us      1.16 us
>                     5          5     21.31 us    106.57 us     25.77 us     18.17 us
>                     5          5      3.01 us     15.07 us      5.19 us      1.53 us
>                     3          3      6.42 us     19.26 us      9.63 us      4.22 us
>                     3          3      4.20 us     12.61 us      7.45 us      2.10 us
>                     2          2      5.03 us     10.06 us      7.88 us      2.18 us
>                     2          2      2.79 us      5.59 us      3.34 us      2.25 us
>                     1          1      4.70 us      4.70 us      4.70 us      4.70 us
>                     1          1      9.77 us      9.77 us      9.77 us      9.77 us
> 
> After patch:
> root@barry-desktop:/home/barry# time ./a.out
> 
> real	0m17.796s
> user	0m24.962s
> sys	5m28.774s
> 
> Perf lock report:
>       Name   acquired  contended     avg wait   total wait     max wait     min wait
> 
>               1407013    1407013     25.44 us     35.80 s     127.83 us       972 ns
>                845113     845113     47.96 us     40.54 s     152.86 us      1.21 us
>  rcu_state       1920       1920      5.87 us     11.27 ms     25.02 us      1.76 us
>  rcu_state       1889       1889      5.80 us     10.96 ms     30.35 us      1.61 us
>                   140        140      3.97 us    555.52 us     12.51 us      1.20 us
>                    10         10     61.34 us    613.42 us     96.90 us     44.48 us
>                     8          8     24.03 us    192.26 us     37.64 us      6.95 us
>                     8          8     12.32 us     98.56 us     25.55 us      4.16 us
> 
> Both system time and lock contention are noticeably reduced.
> 
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  include/linux/swap.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 8d19be675baf..9c5f7a11c7b0 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
>  	/*
>  	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
>  	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
> -	 * will be sensible, but nobody has implemented and tested that yet.
> +	 * will be sensible.
>  	 */
> -	return !folio_test_large(folio);
> +	return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
>  }
>  
>  extern atomic_t lru_disable_count;

Sashiko rightfully raises that split_folio() can now fail more easily.

So we might want to proactively drain (earlier?) on some more of the split paths.

-- 
Cheers,

David

Re: [RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache

[Barry Song (Xiaomi)]

On Wed, Jul 29, 2026 at 8:13 PM David Hildenbrand (Arm) <david@kernel.org> wrote:
[...]
> > diff --git a/include/linux/swap.h b/include/linux/swap.h
> > index 8d19be675baf..9c5f7a11c7b0 100644
> > --- a/include/linux/swap.h
> > +++ b/include/linux/swap.h
> > @@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
> >       /*
> >        * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
> >        * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
> > -      * will be sensible, but nobody has implemented and tested that yet.
> > +      * will be sensible.
> >        */
> > -     return !folio_test_large(folio);
> > +     return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
> >  }
> >
> >  extern atomic_t lru_disable_count;
>
> Sashiko rightfully raises that split_folio() can now fail more easily.
>
> So we might want to proactively drain (earlier?) on some more of the split paths.

Yes, this is a pain. I haven't tested it, but perhaps a
conceptual model could be something like this?

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 04e8a6b55343..fcd449dec96b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4069,6 +4069,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 	struct folio *new_folio, *next;
 	int nr_shmem_dropped = 0;
 	enum ttu_flags ttu_flags = 0;
+	bool maybe_in_lru_cache;
+	int expected_ref_count;
 	int ret;
 	pgoff_t end = 0;
 
@@ -4154,7 +4156,15 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 	 * Racy check if we can split the page, before unmap_folio() will
 	 * split PMDs
 	 */
-	if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
+	maybe_in_lru_cache = folio_may_be_lru_cached(folio) && !folio_test_lru(folio);
+	expected_ref_count = folio_expected_ref_count(folio);
+	if (expected_ref_count + maybe_in_lru_cache < folio_ref_count(folio) - 1) {
+		ret = -EAGAIN;
+		goto out_unlock;
+	}
+	if (maybe_in_lru_cache)
+		lru_add_drain_all();
+	if (expected_ref_count != folio_ref_count(folio) - 1) {
 		ret = -EAGAIN;
 		goto out_unlock;
 	}

Best Regards
Barry

Re: [RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache

[David Hildenbrand (Arm)]

On 7/30/26 00:26, Barry Song (Xiaomi) wrote:
> On Wed, Jul 29, 2026 at 8:13 PM David Hildenbrand (Arm) <david@kernel.org> wrote:
> [...]
>>> diff --git a/include/linux/swap.h b/include/linux/swap.h
>>> index 8d19be675baf..9c5f7a11c7b0 100644
>>> --- a/include/linux/swap.h
>>> +++ b/include/linux/swap.h
>>> @@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
>>>       /*
>>>        * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
>>>        * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
>>> -      * will be sensible, but nobody has implemented and tested that yet.
>>> +      * will be sensible.
>>>        */
>>> -     return !folio_test_large(folio);
>>> +     return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
>>>  }
>>>
>>>  extern atomic_t lru_disable_count;
>>
>> Sashiko rightfully raises that split_folio() can now fail more easily.
>>
>> So we might want to proactively drain (earlier?) on some more of the split paths.
> 
> Yes, this is a pain. I haven't tested it, but perhaps a
> conceptual model could be something like this?
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 04e8a6b55343..fcd449dec96b 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -4069,6 +4069,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
>  	struct folio *new_folio, *next;
>  	int nr_shmem_dropped = 0;
>  	enum ttu_flags ttu_flags = 0;
> +	bool maybe_in_lru_cache;
> +	int expected_ref_count;
>  	int ret;
>  	pgoff_t end = 0;
>  
> @@ -4154,7 +4156,15 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
>  	 * Racy check if we can split the page, before unmap_folio() will
>  	 * split PMDs
>  	 */
> -	if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
> +	maybe_in_lru_cache = folio_may_be_lru_cached(folio) && !folio_test_lru(folio);
> +	expected_ref_count = folio_expected_ref_count(folio);
> +	if (expected_ref_count + maybe_in_lru_cache < folio_ref_count(folio) - 1) {
> +		ret = -EAGAIN;
> +		goto out_unlock;
> +	}
> +	if (maybe_in_lru_cache)
> +		lru_add_drain_all();

The drain-all is nasty. In  collect_longterm_unpinnable_folios() we escalate the
drainining.

I think we recently stumbled into a similar problem in guest_memfd.

Ah, yes, there it is:

https://lore.kernel.org/r/20260728-gmem-inplace-conversion-v9-14-35f9aec2aed2@google.com


-- 
Cheers,

David

Re: [RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache

[Zi Yan]

On Thu Jul 30, 2026 at 5:28 AM EDT, David Hildenbrand (Arm) wrote:
> On 7/30/26 00:26, Barry Song (Xiaomi) wrote:
>> On Wed, Jul 29, 2026 at 8:13 PM David Hildenbrand (Arm) <david@kernel.org> wrote:
>> [...]
>>>> diff --git a/include/linux/swap.h b/include/linux/swap.h
>>>> index 8d19be675baf..9c5f7a11c7b0 100644
>>>> --- a/include/linux/swap.h
>>>> +++ b/include/linux/swap.h
>>>> @@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
>>>>       /*
>>>>        * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
>>>>        * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
>>>> -      * will be sensible, but nobody has implemented and tested that yet.
>>>> +      * will be sensible.
>>>>        */
>>>> -     return !folio_test_large(folio);
>>>> +     return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
>>>>  }
>>>>
>>>>  extern atomic_t lru_disable_count;
>>>
>>> Sashiko rightfully raises that split_folio() can now fail more easily.
>>>
>>> So we might want to proactively drain (earlier?) on some more of the split paths.
>> 
>> Yes, this is a pain. I haven't tested it, but perhaps a
>> conceptual model could be something like this?
>> 
>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
>> index 04e8a6b55343..fcd449dec96b 100644
>> --- a/mm/huge_memory.c
>> +++ b/mm/huge_memory.c
>> @@ -4069,6 +4069,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
>>  	struct folio *new_folio, *next;
>>  	int nr_shmem_dropped = 0;
>>  	enum ttu_flags ttu_flags = 0;
>> +	bool maybe_in_lru_cache;
>> +	int expected_ref_count;
>>  	int ret;
>>  	pgoff_t end = 0;
>>  
>> @@ -4154,7 +4156,15 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
>>  	 * Racy check if we can split the page, before unmap_folio() will
>>  	 * split PMDs
>>  	 */
>> -	if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
>> +	maybe_in_lru_cache = folio_may_be_lru_cached(folio) && !folio_test_lru(folio);
>> +	expected_ref_count = folio_expected_ref_count(folio);
>> +	if (expected_ref_count + maybe_in_lru_cache < folio_ref_count(folio) - 1) {
>> +		ret = -EAGAIN;
>> +		goto out_unlock;
>> +	}
>> +	if (maybe_in_lru_cache)
>> +		lru_add_drain_all();
>
> The drain-all is nasty. In  collect_longterm_unpinnable_folios() we escalate the
> drainining.
>
> I think we recently stumbled into a similar problem in guest_memfd.
>
> Ah, yes, there it is:
>
> https://lore.kernel.org/r/20260728-gmem-inplace-conversion-v9-14-35f9aec2aed2@google.com

Page migration has this issue for a long time.

I wonder if we can re-design per-CPU LRU cache so that we can remove a
folio from per-CPU LRU cache without drainining any to LRU lists.

Something like:
1. have a folio on per-CPU LRU cache to store a pointer to its
fb_batch's folio array position;

2. zero that position when we want to drain that specific folio.

But I have not looked into all the details to make sure that is feasible
yet.


-- 
Best Regards,
Yan, Zi