[RFC PATCH v2 0/2] mm: enable lru cache for smaller large folios

[RFC PATCH v2 0/2] mm: enable lru cache for smaller large folios

[Barry Song (Xiaomi)]

This patchset enables the per-CPU LRU cache for large folios smaller
than `COSTLY_ORDER`. It is particularly beneficial on systems that use
relatively small large folios. For larger folios, the benefit is likely
to be smaller because far fewer folios are expected to contend for the
LRU cache.

Patch 1/2 enables the per-CPU LRU cache for eligible large folios.
Patch 2/2 updates `__wp_can_reuse_large_anon_folio()` to account for
the fact that large folios may now also reside in the per-CPU LRU cache
before they are reused.

-RFC v2:
 * Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large
   folios. As David pointed out, it currently does not account for
   large folios residing in the per-CPU LRU cache.

Barry Song (Xiaomi) (2):
  mm: allow smaller large folios to use lru_cache
  mm: improve large folio reuse for LRU-cached folios

 include/linux/swap.h |  4 ++--
 mm/memory.c          | 13 +++++++++++++
 2 files changed, 15 insertions(+), 2 deletions(-)

-- 
2.39.3 (Apple Git-146)

[RFC PATCH v2 1/2] mm: allow smaller large folios to use lru_cache

[Barry Song (Xiaomi)]

For a system which primarily uses smaller orders for large folios,
it could be beneficial to enable lru_cache to avoid lock contention.

For large folios with higher orders, the number of folios involved
would be small, so the contention is less significant.

The following small program runs on a system with 16 KiB
mTHP enabled.

 #include <pthread.h>
 #include <sys/mman.h>
 #include <string.h>

 #define NUM_THREADS     20
 #define MEM_SIZE        (16 * 1024 * 1024)
 #define LOOP_COUNT      1000

 void* thread_worker(void* arg) {
     void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

     for (int i = 0; i < LOOP_COUNT; i++) {
         memset(addr, 0x55, MEM_SIZE);
         madvise(addr, MEM_SIZE, MADV_DONTNEED);
     }

     munmap(addr, MEM_SIZE);
     pthread_exit(NULL);
 }

 int main() {
     pthread_t threads[NUM_THREADS];

     for (long t = 0; t < NUM_THREADS; t++) {
         pthread_create(&threads[t], NULL, thread_worker, (void*)t);
     }

     for (int t = 0; t < NUM_THREADS; t++) {
         pthread_join(threads[t], NULL);
     }

     return 0;
 }

Before patch:
root@barry-desktop:/home/barry# time ./a.out

real	0m20.233s
user	0m21.739s
sys	6m21.143s

Perf lock report:
      Name   acquired  contended     avg wait   total wait     max wait     min wait

             21158662   21158662     29.20 us     10.30 m      67.83 us       884 ns
                13547      13547     14.68 us    198.93 ms    135.39 us       982 ns
 rcu_state         41         41      1.83 us     75.21 us      3.46 us      1.27 us
 rcu_state         34         34      1.83 us     62.09 us      2.60 us      1.16 us
                    5          5     21.31 us    106.57 us     25.77 us     18.17 us
                    5          5      3.01 us     15.07 us      5.19 us      1.53 us
                    3          3      6.42 us     19.26 us      9.63 us      4.22 us
                    3          3      4.20 us     12.61 us      7.45 us      2.10 us
                    2          2      5.03 us     10.06 us      7.88 us      2.18 us
                    2          2      2.79 us      5.59 us      3.34 us      2.25 us
                    1          1      4.70 us      4.70 us      4.70 us      4.70 us
                    1          1      9.77 us      9.77 us      9.77 us      9.77 us

After patch:
root@barry-desktop:/home/barry# time ./a.out

real	0m17.796s
user	0m24.962s
sys	5m28.774s

Perf lock report:
      Name   acquired  contended     avg wait   total wait     max wait     min wait

              1407013    1407013     25.44 us     35.80 s     127.83 us       972 ns
               845113     845113     47.96 us     40.54 s     152.86 us      1.21 us
 rcu_state       1920       1920      5.87 us     11.27 ms     25.02 us      1.76 us
 rcu_state       1889       1889      5.80 us     10.96 ms     30.35 us      1.61 us
                  140        140      3.97 us    555.52 us     12.51 us      1.20 us
                   10         10     61.34 us    613.42 us     96.90 us     44.48 us
                    8          8     24.03 us    192.26 us     37.64 us      6.95 us
                    8          8     12.32 us     98.56 us     25.55 us      4.16 us

Both system time and lock contention are noticeably reduced.

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 include/linux/swap.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8d19be675baf..9c5f7a11c7b0 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -316,9 +316,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)
 	/*
 	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
 	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
-	 * will be sensible, but nobody has implemented and tested that yet.
+	 * will be sensible.
 	 */
-	return !folio_test_large(folio);
+	return folio_order(folio) < PAGE_ALLOC_COSTLY_ORDER;
 }
 
 extern atomic_t lru_disable_count;
-- 
2.39.3 (Apple Git-146)

[RFC PATCH v2 2/2] mm: improve large folio reuse for LRU-cached folios

[Barry Song (Xiaomi)]

Large folios may now reside in the per-CPU LRU cache. Before
attempting to reuse them, drain the local LRU cache, which
can still be beneficial in cases where the folios are likely
to remain in this CPU's LRU cache:

int main(int argc, char *argv[])
{
    int i;
    while (1) {
        volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
                               MAP_PRIVATE | MAP_ANONYMOUS,
                               -1, 0);
        for (int i = 0; i < SIZE / sizeof(int); i++)
            p[i] = i;
        madvise((void *)p, SIZE, MADV_PAGEOUT);
        if (!fork())
            _exit(0);
        for (int i = 0; i < SIZE / sizeof(int); i++)
            p[i] = i;
        munmap((void *)p, SIZE);
    }
    return 0;
}

Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
 mm/memory.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/mm/memory.c b/mm/memory.c
index 5689b7cff76c..1d08ed5ba99b 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4136,6 +4136,19 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
 		folio_unlock(folio);
 	}
 
+	if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
+		if (folio_ref_count(folio) > folio_large_mapcount(folio) + 1)
+			return false;
+		/*
+		 * A global LRU drain is too expensive, but a local drain
+		 * can still be beneficial. For example, large folios that
+		 * have just been swapped in and fall back from
+		 * do_swap_page() to do_wp_page() are likely still in this
+		 * CPU's LRU cache.
+		 */
+		lru_add_drain();
+	}
+
 	if (folio_large_mapcount(folio) != folio_ref_count(folio))
 		return false;
 
-- 
2.39.3 (Apple Git-146)