[RFC PATCH v3 0/4] mm: enable lru cache for smaller large folios

["Barry Song (Xiaomi)" <baohua@kernel.org>]

This patchset enables the per-CPU LRU cache for large folios with fewer
than `FOLIO_BATCH_SIZE` (31) pages. It also limits each per-CPU LRU cache
to at most `FOLIO_BATCH_SIZE` pages to avoid negatively affecting
accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small
large folios. For larger folios, the benefit is likely to be smaller
because far fewer folios are expected to contend for the LRU cache.

* Use the following microbenchmark:

 #include <pthread.h>
 #include <sys/mman.h>
 #include <string.h>

 #define NUM_THREADS     20
 #define MEM_SIZE        (16 * 1024 * 1024)
 #define LOOP_COUNT      1000

 void* thread_worker(void* arg) {
        void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

        for (int i = 0; i < LOOP_COUNT; i++) {
                for(int j = 0; j < MEM_SIZE; j += 4096)
                        *(unsigned char *)(addr + j) = 0x55;
                madvise(addr, MEM_SIZE, MADV_DONTNEED);
        }

        munmap(addr, MEM_SIZE);
        pthread_exit(NULL);
 }

 int main() {
        pthread_t threads[NUM_THREADS];

        for (long t = 0; t < NUM_THREADS; t++) {
                pthread_create(&threads[t], NULL, thread_worker, (void*)t);
        }

        for (int t = 0; t < NUM_THREADS; t++) {
                pthread_join(threads[t], NULL);
        }

        return 0;
 }

w/o patch:

$ time ./a.out 

real	0m17.760s
user	0m8.207s
sys	5m46.067s

perf lock report:
                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                       21414857   21414857     30.07 us     10.73 m     101.94 us       880 ns 
                          42890      42890     11.29 us    484.21 ms    123.08 us       923 ns 
                           1678       1678      1.57 us      2.63 ms      5.77 us       985 ns 
                             52         52      1.46 us     75.98 us      2.69 us      1.07 us 
                             18         18     17.82 us    320.69 us     58.92 us      1.62 us 
                             18         18      2.36 ms     42.52 ms      5.31 ms      1.47 us 
           rcu_state         12         12      1.98 us     23.78 us      2.62 us      1.47 us 
           rcu_state          9          9      1.72 us     15.49 us      2.27 us      1.34 us 
                              2          2      2.42 us      4.83 us      2.48 us      2.36 us 

w/ patch:

$ time ./a.out 

real	0m16.292s
user	0m8.587s
sys	5m13.787s

perf lock report

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                        2641286    2641286     46.87 us      2.06 m     107.92 us      1.01 us 
                         235275     235275     10.57 us      2.49 s     115.32 us      1.02 us 
           rcu_state       1982       1982      7.58 us     15.02 ms     31.72 us      1.26 us 
           rcu_state       1929       1929      7.42 us     14.32 ms     30.50 us      1.43 us 
                             86         86      3.61 us    310.19 us     13.89 us      1.15 us 
                             20         20     19.70 us    394.03 us    108.07 us      2.15 us 
       tasklist_lock          1          1      2.13 us      2.13 us      2.13 us      2.13 us 


* Build the kernel in a 1 GiB memcg by -j20 with zRAM configured as swap:

w/o patch:

Perf lock report:

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                         782337     782337     17.61 us     13.78 s     402.22 us       896 ns 
                          55459      55459     19.48 us      1.08 s     117.59 us      1.01 us 
                           7826       7826      8.01 us     62.68 ms     19.03 us       887 ns 
                           5324       5324      7.83 us     41.68 ms     37.59 us      1.11 us 
           rcu_state       5144       5144      6.27 us     32.23 ms     25.17 us      1.58 us 
           rcu_state       5142       5142      6.35 us     32.67 ms     30.55 us      1.48 us 
                           3855       3855      8.74 us     33.68 ms     42.32 us       996 ns 
                           2770       2770      9.22 us     25.55 ms     27.83 us       914 ns 
                           2342       2342      5.88 us     13.77 ms    318.75 us      1.00 us 
time:

*** Executing round 0 ***

real    1m46.847s
user    25m10.848s
sys     2m57.282s

*** Executing round 1 ***

real    1m46.423s
user    25m10.072s
sys     2m54.348s

*** Executing round 2 ***

real    1m46.308s
user    25m13.800s
sys     2m58.963s

*** Executing round 3 ***

real    1m46.155s
user    25m18.079s
sys     2m59.721s

*** Executing round 4 ***

real    1m45.980s
user    25m15.493s
sys     2m56.959s

w/ patch:

perf lock report
                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                         202647     202647     34.27 us      6.94 s     467.82 us      1.18 us 
                          51819      51819     16.26 us    842.46 ms    245.55 us       885 ns 
     inode_hash_lock      15169      15169      8.58 us    130.21 ms     23.23 us      1.04 us 
                           5306       5306      7.54 us     40.03 ms     31.53 us      1.17 us 
           rcu_state       4945       4945      6.97 us     34.47 ms     30.00 us      1.76 us 
           rcu_state       4899       4899      7.03 us     34.42 ms     24.65 us      1.42 us 
                           3923       3923      8.56 us     33.57 ms     27.51 us      1.08 us 
                           2212       2212      5.23 us     11.56 ms    222.79 us       965 ns 
                           1412       1412      6.35 us      8.97 ms     23.09 us      1.77 us 

time:

*** Executing round 0 ***

real	1m46.463s
user	25m17.448s
sys	2m49.524s

*** Executing round 1 ***

real	1m46.274s
user	25m12.178s
sys	2m53.522s

*** Executing round 2 ***

real	1m46.362s
user	25m13.115s
sys	2m53.005s

*** Executing round 3 ***

real	1m46.036s
user	25m17.627s
sys	2m53.477s

*** Executing round 4 ***

real	1m46.329s
user	25m15.130s
sys	2m51.508s

-RFC v3:
  * Rather than hard-coding `< COSTLY_ORDER` to enable the lru_cache,
    allow the lru_cache for larger orders as long as the folio contains
    fewer than `FOLIO_BATCH_LRU` pages. Also limit the total number of
    pages held in the lru_cache. Hugh may prefer this approach as well.
  * Clean up the comments and `if` conditions in "mm: improve large folio
    reuse for LRU-cached folios" based on David's feedback. Thanks!
  * Properly drain the lru_cache when splitting folios. Thanks to Sashiko
    and David!

-RFC v2:
 * Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large
   folios. As David pointed out, it currently does not account for
   large folios residing in the per-CPU LRU cache.
   https://lore.kernel.org/all/20260709081536.82768-1-baohua@kernel.org/

Barry Song (Xiaomi) (4):
  mm: allow smaller large folios to use lru_cache
  mm: improve large folio reuse for LRU-cached folios
  mm: drain LRU cache if necessary for splitting large folios
  mm: batch lru_cache draining in deferred_split_scan

 include/linux/folio_batch.h | 25 +++++++++++++++++++++++++
 mm/folio.c                  | 10 +++++++++-
 mm/huge_memory.c            | 23 +++++++++++++++++++----
 mm/internal.h               |  4 ++--
 mm/memory.c                 |  6 ++++++
 5 files changed, 61 insertions(+), 7 deletions(-)

-- 
2.34.1