[RFC PATCH] mm: allow smaller large folios to use lru_cache

["Barry Song (Xiaomi)" <baohua@kernel.org>]

For a system which primarily uses smaller orders for large folios,
it could be beneficial to enable lru_cache to avoid lock contention.

For large folios with higher orders, the number of folios involved
would be small, so the contention is less significant.

The following small program runs on a system with 16 KiB
mTHP enabled.

 #include <pthread.h>
 #include <sys/mman.h>
 #include <string.h>

 #define NUM_THREADS     20
 #define MEM_SIZE        (16 * 1024 * 1024)
 #define LOOP_COUNT      1000

 void* thread_worker(void* arg) {
     void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

     for (int i = 0; i < LOOP_COUNT; i++) {
         memset(addr, 0x55, MEM_SIZE);
         madvise(addr, MEM_SIZE, MADV_DONTNEED);
     }

     munmap(addr, MEM_SIZE);
     pthread_exit(NULL);
 }

 int main() {
     pthread_t threads[NUM_THREADS];

     for (long t = 0; t < NUM_THREADS; t++) {
         pthread_create(&threads[t], NULL, thread_worker, (void*)t);
     }

     for (int t = 0; t < NUM_THREADS; t++) {
         pthread_join(threads[t], NULL);
     }

     return 0;
 }

Before patch:
root@barry-desktop:/home/barry# time ./a.out

real	0m20.233s
user	0m21.739s
sys	6m21.143s

Perf lock report:
      Name   acquired  contended     avg wait   total wait     max wait     min wait

             21158662   21158662     29.20 us     10.30 m      67.83 us       884 ns
                13547      13547     14.68 us    198.93 ms    135.39 us       982 ns
 rcu_state         41         41      1.83 us     75.21 us      3.46 us      1.27 us
 rcu_state         34         34      1.83 us     62.09 us      2.60 us      1.16 us
                    5          5     21.31 us    106.57 us     25.77 us     18.17 us
                    5          5      3.01 us     15.07 us      5.19 us      1.53 us
                    3          3      6.42 us     19.26 us      9.63 us      4.22 us
                    3          3      4.20 us     12.61 us      7.45 us      2.10 us
                    2          2      5.03 us     10.06 us      7.88 us      2.18 us
                    2          2      2.79 us      5.59 us      3.34 us      2.25 us
                    1          1      4.70 us      4.70 us      4.70 us      4.70 us
                    1          1      9.77 us      9.77 us      9.77 us      9.77 us

After patch:
root@barry-desktop:/home/barry# time ./a.out

real	0m17.796s
user	0m24.962s
sys	5m28.774s

Perf lock report:
      Name   acquired  contended     avg wait   total wait     max wait     min wait

              1407013    1407013     25.44 us     35.80 s     127.83 us       972 ns
               845113     845113     47.96 us     40.54 s     152.86 us      1.21 us
 rcu_state       1920       1920      5.87 us     11.27 ms     25.02 us      1.76 us
 rcu_state       1889       1889      5.80 us     10.96 ms     30.35 us      1.61 us
                  140        140      3.97 us    555.52 us     12.51 us      1.20 us
                   10         10     61.34 us    613.42 us     96.90 us     44.48 us
                    8          8     24.03 us    192.26 us     37.64 us      6.95 us
                    8          8     12.32 us     98.56 us     25.55 us      4.16 us

Both system time and lock contention are noticeably reduced.

Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 include/linux/swap.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8f0f68e245ba..fce1d84aa2c8 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
 	/*
 	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
 	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
-	 * will be sensible, but nobody has implemented and tested that yet.
+	 * will be sensible.
 	 */
-	return !folio_test_large(folio);
+	return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
 }
 
 extern atomic_t lru_disable_count;
-- 
2.39.3 (Apple Git-146)