Re: [RFC PATCH] mm: allow smaller large folios to use lru_cache

["David Hildenbrand (Arm)" <david@kernel.org>]

On 7/2/26 14:23, Barry Song (Xiaomi) wrote:
> For a system which primarily uses smaller orders for large folios,
> it could be beneficial to enable lru_cache to avoid lock contention.
> 
> For large folios with higher orders, the number of folios involved
> would be small, so the contention is less significant.
> 
> The following small program runs on a system with 16 KiB
> mTHP enabled.
> 
>  #include <pthread.h>
>  #include <sys/mman.h>
>  #include <string.h>
> 
>  #define NUM_THREADS     20
>  #define MEM_SIZE        (16 * 1024 * 1024)
>  #define LOOP_COUNT      1000
> 
>  void* thread_worker(void* arg) {
>      void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
>                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> 
>      for (int i = 0; i < LOOP_COUNT; i++) {
>          memset(addr, 0x55, MEM_SIZE);
>          madvise(addr, MEM_SIZE, MADV_DONTNEED);
>      }
> 
>      munmap(addr, MEM_SIZE);
>      pthread_exit(NULL);
>  }
> 
>  int main() {
>      pthread_t threads[NUM_THREADS];
> 
>      for (long t = 0; t < NUM_THREADS; t++) {
>          pthread_create(&threads[t], NULL, thread_worker, (void*)t);
>      }
> 
>      for (int t = 0; t < NUM_THREADS; t++) {
>          pthread_join(threads[t], NULL);
>      }
> 
>      return 0;
>  }
> 
> Before patch:
> root@barry-desktop:/home/barry# time ./a.out
> 
> real	0m20.233s
> user	0m21.739s
> sys	6m21.143s
> 
> Perf lock report:
>       Name   acquired  contended     avg wait   total wait     max wait     min wait
> 
>              21158662   21158662     29.20 us     10.30 m      67.83 us       884 ns
>                 13547      13547     14.68 us    198.93 ms    135.39 us       982 ns
>  rcu_state         41         41      1.83 us     75.21 us      3.46 us      1.27 us
>  rcu_state         34         34      1.83 us     62.09 us      2.60 us      1.16 us
>                     5          5     21.31 us    106.57 us     25.77 us     18.17 us
>                     5          5      3.01 us     15.07 us      5.19 us      1.53 us
>                     3          3      6.42 us     19.26 us      9.63 us      4.22 us
>                     3          3      4.20 us     12.61 us      7.45 us      2.10 us
>                     2          2      5.03 us     10.06 us      7.88 us      2.18 us
>                     2          2      2.79 us      5.59 us      3.34 us      2.25 us
>                     1          1      4.70 us      4.70 us      4.70 us      4.70 us
>                     1          1      9.77 us      9.77 us      9.77 us      9.77 us
> 
> After patch:
> root@barry-desktop:/home/barry# time ./a.out
> 
> real	0m17.796s
> user	0m24.962s
> sys	5m28.774s
> 
> Perf lock report:
>       Name   acquired  contended     avg wait   total wait     max wait     min wait
> 
>               1407013    1407013     25.44 us     35.80 s     127.83 us       972 ns
>                845113     845113     47.96 us     40.54 s     152.86 us      1.21 us
>  rcu_state       1920       1920      5.87 us     11.27 ms     25.02 us      1.76 us
>  rcu_state       1889       1889      5.80 us     10.96 ms     30.35 us      1.61 us
>                   140        140      3.97 us    555.52 us     12.51 us      1.20 us
>                    10         10     61.34 us    613.42 us     96.90 us     44.48 us
>                     8          8     24.03 us    192.26 us     37.64 us      6.95 us
>                     8          8     12.32 us     98.56 us     25.55 us      4.16 us
> 
> Both system time and lock contention are noticeably reduced.
> 
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Lorenzo Stoakes <ljs@kernel.org>
> Cc: Zi Yan <ziy@nvidia.com>
> Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
> Cc: Liam R. Howlett <liam@infradead.org>
> Cc: Nico Pache <npache@redhat.com>
> Cc: Ryan Roberts <ryan.roberts@arm.com>
> Cc: Dev Jain <dev.jain@arm.com>
> Cc: Lance Yang <lance.yang@linux.dev>
> Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
> ---
>  include/linux/swap.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 8f0f68e245ba..fce1d84aa2c8 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
>  	/*
>  	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
>  	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
> -	 * will be sensible, but nobody has implemented and tested that yet.
> +	 * will be sensible.
>  	 */
> -	return !folio_test_large(folio);
> +	return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
>  }
>  
>  extern atomic_t lru_disable_count;

Not that __wp_can_reuse_large_anon_folio() currently assumes that large folios
are not in the LRU cache.

-- 
Cheers,

David