[RFC PATCH v3 0/4] mm: enable lru cache for smaller large folios

[RFC PATCH v3 0/4] mm: enable lru cache for smaller large folios

[Barry Song (Xiaomi)]

This patchset enables the per-CPU LRU cache for large folios with fewer
than `FOLIO_BATCH_SIZE` (31) pages. It also limits each per-CPU LRU cache
to at most `FOLIO_BATCH_SIZE` pages to avoid negatively affecting
accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small
large folios. For larger folios, the benefit is likely to be smaller
because far fewer folios are expected to contend for the LRU cache.

* Use the following microbenchmark:

 #include <pthread.h>
 #include <sys/mman.h>
 #include <string.h>

 #define NUM_THREADS     20
 #define MEM_SIZE        (16 * 1024 * 1024)
 #define LOOP_COUNT      1000

 void* thread_worker(void* arg) {
        void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

        for (int i = 0; i < LOOP_COUNT; i++) {
                for(int j = 0; j < MEM_SIZE; j += 4096)
                        *(unsigned char *)(addr + j) = 0x55;
                madvise(addr, MEM_SIZE, MADV_DONTNEED);
        }

        munmap(addr, MEM_SIZE);
        pthread_exit(NULL);
 }

 int main() {
        pthread_t threads[NUM_THREADS];

        for (long t = 0; t < NUM_THREADS; t++) {
                pthread_create(&threads[t], NULL, thread_worker, (void*)t);
        }

        for (int t = 0; t < NUM_THREADS; t++) {
                pthread_join(threads[t], NULL);
        }

        return 0;
 }

w/o patch:

$ time ./a.out 

real	0m17.760s
user	0m8.207s
sys	5m46.067s

perf lock report:
                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                       21414857   21414857     30.07 us     10.73 m     101.94 us       880 ns 
                          42890      42890     11.29 us    484.21 ms    123.08 us       923 ns 
                           1678       1678      1.57 us      2.63 ms      5.77 us       985 ns 
                             52         52      1.46 us     75.98 us      2.69 us      1.07 us 
                             18         18     17.82 us    320.69 us     58.92 us      1.62 us 
                             18         18      2.36 ms     42.52 ms      5.31 ms      1.47 us 
           rcu_state         12         12      1.98 us     23.78 us      2.62 us      1.47 us 
           rcu_state          9          9      1.72 us     15.49 us      2.27 us      1.34 us 
                              2          2      2.42 us      4.83 us      2.48 us      2.36 us 

w/ patch:

$ time ./a.out 

real	0m16.292s
user	0m8.587s
sys	5m13.787s

perf lock report

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                        2641286    2641286     46.87 us      2.06 m     107.92 us      1.01 us 
                         235275     235275     10.57 us      2.49 s     115.32 us      1.02 us 
           rcu_state       1982       1982      7.58 us     15.02 ms     31.72 us      1.26 us 
           rcu_state       1929       1929      7.42 us     14.32 ms     30.50 us      1.43 us 
                             86         86      3.61 us    310.19 us     13.89 us      1.15 us 
                             20         20     19.70 us    394.03 us    108.07 us      2.15 us 
       tasklist_lock          1          1      2.13 us      2.13 us      2.13 us      2.13 us 


* Build the kernel in a 1 GiB memcg by -j20 with zRAM configured as swap:

w/o patch:

Perf lock report:

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                         782337     782337     17.61 us     13.78 s     402.22 us       896 ns 
                          55459      55459     19.48 us      1.08 s     117.59 us      1.01 us 
                           7826       7826      8.01 us     62.68 ms     19.03 us       887 ns 
                           5324       5324      7.83 us     41.68 ms     37.59 us      1.11 us 
           rcu_state       5144       5144      6.27 us     32.23 ms     25.17 us      1.58 us 
           rcu_state       5142       5142      6.35 us     32.67 ms     30.55 us      1.48 us 
                           3855       3855      8.74 us     33.68 ms     42.32 us       996 ns 
                           2770       2770      9.22 us     25.55 ms     27.83 us       914 ns 
                           2342       2342      5.88 us     13.77 ms    318.75 us      1.00 us 
time:

*** Executing round 0 ***

real    1m46.847s
user    25m10.848s
sys     2m57.282s

*** Executing round 1 ***

real    1m46.423s
user    25m10.072s
sys     2m54.348s

*** Executing round 2 ***

real    1m46.308s
user    25m13.800s
sys     2m58.963s

*** Executing round 3 ***

real    1m46.155s
user    25m18.079s
sys     2m59.721s

*** Executing round 4 ***

real    1m45.980s
user    25m15.493s
sys     2m56.959s

w/ patch:

perf lock report
                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                         202647     202647     34.27 us      6.94 s     467.82 us      1.18 us 
                          51819      51819     16.26 us    842.46 ms    245.55 us       885 ns 
     inode_hash_lock      15169      15169      8.58 us    130.21 ms     23.23 us      1.04 us 
                           5306       5306      7.54 us     40.03 ms     31.53 us      1.17 us 
           rcu_state       4945       4945      6.97 us     34.47 ms     30.00 us      1.76 us 
           rcu_state       4899       4899      7.03 us     34.42 ms     24.65 us      1.42 us 
                           3923       3923      8.56 us     33.57 ms     27.51 us      1.08 us 
                           2212       2212      5.23 us     11.56 ms    222.79 us       965 ns 
                           1412       1412      6.35 us      8.97 ms     23.09 us      1.77 us 

time:

*** Executing round 0 ***

real	1m46.463s
user	25m17.448s
sys	2m49.524s

*** Executing round 1 ***

real	1m46.274s
user	25m12.178s
sys	2m53.522s

*** Executing round 2 ***

real	1m46.362s
user	25m13.115s
sys	2m53.005s

*** Executing round 3 ***

real	1m46.036s
user	25m17.627s
sys	2m53.477s

*** Executing round 4 ***

real	1m46.329s
user	25m15.130s
sys	2m51.508s

-RFC v3:
  * Rather than hard-coding `< COSTLY_ORDER` to enable the lru_cache,
    allow the lru_cache for larger orders as long as the folio contains
    fewer than `FOLIO_BATCH_LRU` pages. Also limit the total number of
    pages held in the lru_cache. Hugh may prefer this approach as well.
  * Clean up the comments and `if` conditions in "mm: improve large folio
    reuse for LRU-cached folios" based on David's feedback. Thanks!
  * Properly drain the lru_cache when splitting folios. Thanks to Sashiko
    and David!

-RFC v2:
 * Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large
   folios. As David pointed out, it currently does not account for
   large folios residing in the per-CPU LRU cache.
   https://lore.kernel.org/all/20260709081536.82768-1-baohua@kernel.org/

Barry Song (Xiaomi) (4):
  mm: allow smaller large folios to use lru_cache
  mm: improve large folio reuse for LRU-cached folios
  mm: drain LRU cache if necessary for splitting large folios
  mm: batch lru_cache draining in deferred_split_scan

 include/linux/folio_batch.h | 25 +++++++++++++++++++++++++
 mm/folio.c                  | 10 +++++++++-
 mm/huge_memory.c            | 23 +++++++++++++++++++----
 mm/internal.h               |  4 ++--
 mm/memory.c                 |  6 ++++++
 5 files changed, 61 insertions(+), 7 deletions(-)

-- 
2.34.1

[RFC PATCH v3 1/4] mm: allow smaller large folios to use lru_cache

[Barry Song (Xiaomi)]

For systems that primarily use smaller-order large folios, enabling the
lru_cache can help reduce lock contention.

For higher-order large folios, the number of folios involved is likely
to be smaller, making lock contention less significant.

This patch enables the lru_cache for large folios whose `nr_pages` is
smaller than `FOLIO_BATCH_SIZE`. To avoid holding too many pages in the
lru_cache, which could affect accounting and reclamation, we also limit
the total number of pages in the cache to `FOLIO_BATCH_SIZE`.

To track the number of pages, this patch adds an `unsigned short
nr_pages` field to `struct folio_batch`. It cannot overflow because the
batch contains at most `FOLIO_BATCH_SIZE` folios, each of which has fewer
than `FOLIO_BATCH_SIZE` pages.

For non-LRU caches, `folio_batch` only needs to track the number of
folios, so `nr_pages` is left at zero.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 include/linux/folio_batch.h | 25 +++++++++++++++++++++++++
 mm/folio.c                  | 10 +++++++++-
 mm/internal.h               |  4 ++--
 3 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h
index b45946adc50b..ffc7de091fa3 100644
--- a/include/linux/folio_batch.h
+++ b/include/linux/folio_batch.h
@@ -10,6 +10,7 @@
 #define _LINUX_FOLIO_BATCH_H
 
 #include <linux/types.h>
+#include <linux/mm.h>
 
 /* 31 pointers + header align the folio_batch structure to a power of two */
 #define FOLIO_BATCH_SIZE	31
@@ -28,6 +29,7 @@ struct folio;
 struct folio_batch {
 	unsigned char nr;
 	unsigned char i;
+	unsigned short nr_pages;
 	bool percpu_pvec_drained;
 	struct folio *folios[FOLIO_BATCH_SIZE];
 };
@@ -42,6 +44,7 @@ static inline void folio_batch_init(struct folio_batch *fbatch)
 {
 	fbatch->nr = 0;
 	fbatch->i = 0;
+	fbatch->nr_pages = 0;
 	fbatch->percpu_pvec_drained = false;
 }
 
@@ -49,6 +52,7 @@ static inline void folio_batch_reinit(struct folio_batch *fbatch)
 {
 	fbatch->nr = 0;
 	fbatch->i = 0;
+	fbatch->nr_pages = 0;
 }
 
 static inline unsigned int folio_batch_count(const struct folio_batch *fbatch)
@@ -78,6 +82,27 @@ static inline unsigned folio_batch_add(struct folio_batch *fbatch,
 	return folio_batch_space(fbatch);
 }
 
+/**
+ * folio_batch_add_lru_cache() - Add a folio to a batch of lru_cache
+ * @fbatch: The folio batch.
+ * @folio: The folio to add.
+ *
+ * The folio is added to the end of the batch.
+ * The batch must have previously been initialised using folio_batch_init().
+ *
+ * Return: 0 if the lru_cache is filled with more than FOLIO_BATCH_SIZE
+ *         pages; otherwise, the number of available slots.
+ */
+static inline unsigned folio_batch_add_lru_cache(struct folio_batch *fbatch,
+		struct folio *folio)
+{
+	fbatch->folios[fbatch->nr++] = folio;
+	fbatch->nr_pages += (unsigned short)folio_nr_pages(folio);
+	if (fbatch->nr_pages > FOLIO_BATCH_SIZE)
+		return 0;
+	return folio_batch_space(fbatch);
+}
+
 /**
  * folio_batch_next - Return the next folio to process.
  * @fbatch: The folio batch being processed.
diff --git a/mm/folio.c b/mm/folio.c
index 59c477120b9a..e5820d7263e8 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -219,7 +219,7 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
 	else
 		local_lock(&cpu_fbatches.lock);
 
-	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
+	if (!folio_batch_add_lru_cache(this_cpu_ptr(fbatch), folio) ||
 			!folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
 
@@ -981,6 +981,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 	int i, j;
 	struct lruvec *lruvec = NULL;
 	unsigned long flags = 0;
+	unsigned long nr_pages = 0;
 
 	for (i = 0, j = 0; i < folios->nr; i++) {
 		struct folio *folio = folios->folios[i];
@@ -1020,6 +1021,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 
 		if (j != i)
 			folios->folios[j] = folio;
+		nr_pages += folio_nr_pages(folio);
 		j++;
 	}
 	if (lruvec)
@@ -1030,6 +1032,12 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 	}
 
 	folios->nr = j;
+	/*
+	 * For lru_cache, track the number of pages; for non-LRU caches,
+	 * folio_batch->nr_pages is always 0.
+	 */
+	if (folios->nr_pages > 0)
+		folios->nr_pages = nr_pages;
 	mem_cgroup_uncharge_folios(folios);
 	free_unref_folios(folios);
 }
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c9..06adf78e13a2 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -48,9 +48,9 @@ static inline bool folio_may_be_lru_cached(const struct folio *folio)
 	/*
 	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
 	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
-	 * will be sensible, but nobody has implemented and tested that yet.
+	 * will be sensible.
 	 */
-	return !folio_test_large(folio);
+	return folio_nr_pages(folio) < FOLIO_BATCH_SIZE;
 }
 
 static inline void lru_cache_enable(void)
-- 
2.34.1

[RFC PATCH v3 2/4] mm: improve large folio reuse for LRU-cached folios

[Barry Song (Xiaomi)]

Large folios may now reside in the per-CPU LRU cache. Before
attempting to reuse them, drain the local LRU cache, which
can still be beneficial in cases where the folios are likely
to remain in this CPU's LRU cache:

int main(int argc, char *argv[])
{
    int i;
    while (1) {
        volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
                               MAP_PRIVATE | MAP_ANONYMOUS,
                               -1, 0);
        for (int i = 0; i < SIZE / sizeof(int); i++)
            p[i] = i;
        madvise((void *)p, SIZE, MADV_PAGEOUT);
        if (!fork())
            _exit(0);
        for (int i = 0; i < SIZE / sizeof(int); i++)
            p[i] = i;
        munmap((void *)p, SIZE);
    }
    return 0;
}

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/memory.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/memory.c b/mm/memory.c
index 4134ac607ee0..efdf82b3c418 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4275,6 +4275,12 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
 		folio_unlock(folio);
 	}
 
+	if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
+		if (folio_ref_count(folio) != folio_large_mapcount(folio) + 1)
+			return false;
+		lru_add_drain();
+	}
+
 	if (folio_large_mapcount(folio) != folio_ref_count(folio))
 		return false;
 
-- 
2.34.1

[RFC PATCH v3 3/4] mm: drain LRU cache if necessary for splitting large folios

[Barry Song (Xiaomi)]

Smaller large folios might now be present in the LRU cache. Use David's
new lru_cache_drain_for_folio() helper to drain the LRU cache before
splitting a folio, ensuring that the folio can be split successfully.

Also, we only perform the drain when it may actually help, assuming
that the lru_cache holds an extra reference.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/huge_memory.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ced400f72d43..263ef9b6949d 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4201,6 +4201,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 		if (shmem_mapping(mapping))
 			end = shmem_fallocend(mapping->host, end);
 	}
+	if (folio_ref_count(folio) == folio_expected_ref_count(folio) + 1 +
+	    folio_may_be_lru_cached(folio))
+		lru_cache_drain_for_folio(folio, 1, NULL);
 
 	/*
 	 * Racy check if we can split the page, before unmap_folio() will
@@ -4325,6 +4328,9 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
 
+	if (folio_ref_count(folio) == folio_expected_ref_count(folio) + 1 +
+	    folio_may_be_lru_cached(folio))
+		lru_cache_drain_for_folio(folio, 1, NULL);
 	if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
 		return -EAGAIN;
 
@@ -4805,6 +4811,10 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start,
 			goto next;
 
 		total++;
+
+		if (folio_ref_count(folio) == folio_expected_ref_count(folio) +
+		    folio_may_be_lru_cached(folio))
+			lru_cache_drain_for_folio(folio, 0, NULL);
 		/*
 		 * For folios with private, split_huge_page_to_list_to_order()
 		 * will try to drop it before split and then check if the folio
-- 
2.34.1

[RFC PATCH v3 4/4] mm: batch lru_cache draining in deferred_split_scan

[Barry Song (Xiaomi)]

deferred_split_scan() splits a batch of folios, so we only need to
drain the lru_cache once for the entire batch.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/huge_memory.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 263ef9b6949d..e797f6d1837e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4085,6 +4085,8 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
  * @lock_at: a page within @folio to be left locked to caller
  * @list: after-split folios will be put on it if non NULL
  * @split_type: perform uniform split or not (non-uniform split)
+ * @lru_cache_drained: whether lru_cache has been drained locally
+ *  or on all CPUs for a batch of folios
  *
  * It calls __split_unmapped_folio() to perform uniform and non-uniform split.
  * It is in charge of checking whether the split is supported or not and
@@ -4100,7 +4102,8 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
  */
 static int __folio_split(struct folio *folio, unsigned int new_order,
 		struct page *split_at, struct page *lock_at,
-		struct list_head *list, enum split_type split_type)
+		struct list_head *list, enum split_type split_type,
+		enum lru_cache_drained *lru_cache_drained)
 {
 	XA_STATE(xas, &folio->mapping->i_pages, folio->index);
 	struct folio *end_folio = folio_next(folio);
@@ -4203,7 +4206,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 	}
 	if (folio_ref_count(folio) == folio_expected_ref_count(folio) + 1 +
 	    folio_may_be_lru_cached(folio))
-		lru_cache_drain_for_folio(folio, 1, NULL);
+		lru_cache_drain_for_folio(folio, 1, lru_cache_drained);
 
 	/*
 	 * Racy check if we can split the page, before unmap_folio() will
@@ -4395,7 +4398,7 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
 	struct folio *folio = page_folio(page);
 
 	return __folio_split(folio, new_order, &folio->page, page, list,
-			     SPLIT_TYPE_UNIFORM);
+			     SPLIT_TYPE_UNIFORM, NULL);
 }
 
 /**
@@ -4426,7 +4429,7 @@ int folio_split(struct folio *folio, unsigned int new_order,
 		struct page *split_at, struct list_head *list)
 {
 	return __folio_split(folio, new_order, split_at, &folio->page, list,
-			     SPLIT_TYPE_NON_UNIFORM);
+			     SPLIT_TYPE_NON_UNIFORM, NULL);
 }
 
 /**
@@ -4622,6 +4625,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 	struct folio *folio, *next;
 	int split = 0;
 	unsigned long isolated;
+	enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
 
 	isolated = list_lru_shrink_walk_irq(&deferred_split_lru, sc,
 					    deferred_split_isolate, &dispose);
@@ -4646,7 +4650,8 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 		}
 		if (!folio_trylock(folio))
 			goto requeue;
-		if (!split_folio(folio)) {
+		if (!__folio_split(folio, 0, &folio->page, &folio->page, NULL,
+		    SPLIT_TYPE_UNIFORM, &drained)) {
 			did_split = true;
 			if (underused)
 				count_vm_event(THP_UNDERUSED_SPLIT_PAGE);
-- 
2.34.1

Re: [RFC PATCH v3 0/4] mm: enable lru cache for smaller large folios

[Lance Yang]

On Wed, Aug 19, 2026 at 06:59:00AM +0800, Barry Song (Xiaomi) wrote:
[...]
>
>-RFC v3:
>  * Rather than hard-coding `< COSTLY_ORDER` to enable the lru_cache,
>    allow the lru_cache for larger orders as long as the folio contains
>    fewer than `FOLIO_BATCH_LRU` pages. Also limit the total number of

Tiny typo ... should FOLIO_BATCH_LRU be FOLIO_BATCH_SIZE here?