From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68C3F328243 for ; Mon, 27 Apr 2026 18:11:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777313486; cv=none; b=t/mSZX0Ma6EPZZVrpCIlbcfjAFy473x1R6GSskYg/po9DjQ/ObmJSBFat+XAtt4c6B5EvG7uuKIRuiyt30/Mbxm8KprMQYyNGz5GdhvfknqfIdCTsd+kK46NIcmdWwzVaXCXv/U+e/yogLM71KQVKEnY/JAbFbNYJfD42w5xZEg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777313486; c=relaxed/simple; bh=w70abfFI29VnCZKP0rWHj80dttrSkD9WMcDZARSFD04=; h=Date:To:From:Subject:Message-Id; b=RkREijkvN9yn6Mhh2+yMLAVY+A41n251XsZ8ZB2jMJEeOLQN0THH/3DS69bhKLjCMGqdUFBMO4FIw3hSfiqW7hLWriCsj5YcBxxkREaB8q7MrgdLhmh9xK4p0WSHZMAnhCthkx+Y0ZLDBDT7ZfX+x4z2Ny9BLywR6joVNhgc5nY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=buJTlTiS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="buJTlTiS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EEA62C19425; Mon, 27 Apr 2026 18:11:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1777313486; bh=w70abfFI29VnCZKP0rWHj80dttrSkD9WMcDZARSFD04=; h=Date:To:From:Subject:From; b=buJTlTiS7QoaPbph6V9lb4lH/45YIAOSvkNAQwgaEdFP1ujGoaqPOMz5AthOjngiH LtvRVkfJ1XkoJhXqLMPepThmojQfY1aMVqd4liC4llBOvCYMVrGdYGBJwrWZVbvF/j bGKNVeljr5fYy6MRrHuI13DXH57RHvDO7SmHXqA4= Date: Mon, 27 Apr 2026 11:11:25 -0700 To: mm-commits@vger.kernel.org,kasong@tencent.com,akpm@linux-foundation.org From: Andrew Morton Subject: [to-be-updated] mm-mglru-consolidate-common-code-for-retrieving-evictable-size.patch removed from -mm tree Message-Id: <20260427181125.EEA62C19425@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The quilt patch titled Subject: mm/mglru: consolidate common code for retrieving evictable size has been removed from the -mm tree. Its filename was mm-mglru-consolidate-common-code-for-retrieving-evictable-size.patch This patch was dropped because an updated version will be issued ------------------------------------------------------ From: Kairui Song Subject: mm/mglru: consolidate common code for retrieving evictable size Date: Fri, 24 Apr 2026 01:43:12 +0800 Patch series "mm/mglru: improve reclaim loop and dirty folio", v6. This series cleans up and slightly improves MGLRU's reclaim loop and dirty writeback handling. As a result, we can see an up to ~30% increase in some workloads like MongoDB with YCSB and a huge decrease in file refault, no swap involved. Other common benchmarks have no regression, and LOC is reduced, with less unexpected OOM, too. Some of the problems were found in our production environment, and others were mostly exposed while stress testing during the development of the LSM/MM/BPF topic on improving MGLRU [1]. This series cleans up the code base and fixes several performance issues, preparing for further work. MGLRU's reclaim loop is a bit complex, and hence these problems are somehow related to each other. The aging, scan number calculation, and reclaim loop are coupled together, and the dirty folio handling logic is quite different, making the reclaim loop hard to follow and the dirty flush ineffective. This series slightly cleans up and improves these issues using a scan budget by calculating the number of folios to scan at the beginning of the loop, and decouples aging from the reclaim calculation helpers. Then, move the dirty flush logic inside the reclaim loop so it can kick in more effectively. These issues are somehow related, and this series handles them and improves MGLRU reclaim in many ways. Test results: All tests are done on a 48c96t NUMA machine with 2 nodes and a 128G memory machine using NVME as storage. MongoDB ======= Running YCSB workloadb [2] (recordcount:20000000 operationcount:6000000, threads:32), which does 95% read and 5% update to generate mixed read and dirty writeback. MongoDB is set up in a 10G cgroup using Docker, and the WiredTiger cache size is set to 4.5G, using NVME as storage. Not using SWAP. Before: Throughput(ops/sec): 62485.02962831822 AverageLatency(us): 500.9746963330107 pgpgin 159347462 pgpgout 5413332 workingset_refault_anon 0 workingset_refault_file 34522071 After: Throughput(ops/sec): 79760.71784646061 (+27.6%, higher is better) AverageLatency(us): 391.25169970043726 (-21.9%, lower is better) pgpgin 111093923 (-30.3%, lower is better) pgpgout 5437456 workingset_refault_anon 0 workingset_refault_file 19566366 (-43.3%, lower is better) We can see a significant performance improvement after this series. The test is done on NVME and the performance gap would be even larger for slow devices, such as HDD or network storage. We observed over 100% gain for some workloads with slow IO. Chrome & Node.js [3] ==================== Using Yu Zhao's test script [3], testing on a x86_64 NUMA machine with 2 nodes and 128G memory, using 256G ZRAM as swap and spawn 32 memcg 64 workers: Before: Total requests: 79915 Per-worker 95% CI (mean): [1233.9, 1263.5] Per-worker stdev: 59.2 Jain's fairness: 0.997795 (1.0 = perfectly fair) Latency: Bucket Count Pct Cumul [0,1)s 26859 33.61% 33.61% [1,2)s 7818 9.78% 43.39% [2,4)s 5532 6.92% 50.31% [4,8)s 39706 49.69% 100.00% After: Total requests: 81382 Per-worker 95% CI (mean): [1241.9, 1301.3] Per-worker stdev: 118.8 Jain's fairness: 0.991480 (1.0 = perfectly fair) Latency: Bucket Count Pct Cumul [0,1)s 26696 32.80% 32.80% [1,2)s 8745 10.75% 43.55% [2,4)s 6865 8.44% 51.98% [4,8)s 39076 48.02% 100.00% Reclaim is still fair and effective, total requests number seems slightly better. OOM issue with aging and throttling =================================== For the throttling OOM issue, it can be easily reproduced using dd and cgroup limit as demonstrated in patch 14, and fixed by this series. The aging OOM is a bit tricky, a specific reproducer can be used to simulate what we encountered in production environment [4]: Spawns multiple workers that keep reading the given file using mmap, and pauses for 120ms after one file read batch. It also spawns another set of workers that keep allocating and freeing a given size of anonymous memory. The total memory size exceeds the memory limit (eg. 14G anon + 8G file, which is 22G vs a 16G memcg limit). - MGLRU disabled: Finished 128 iterations. - MGLRU enabled: OOM with following info after about ~10-20 iterations: [ 62.624130] file_anon_mix_p invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 [ 62.624999] memory: usage 16777216kB, limit 16777216kB, failcnt 24460 [ 62.640200] swap: usage 0kB, limit 9007199254740988kB, failcnt 0 [ 62.640823] Memory cgroup stats for /demo: [ 62.641017] anon 10604879872 [ 62.641941] file 6574858240 OOM occurs despite there being still evictable file folios. - MGLRU enabled after this series: Finished 128 iterations. Worth noting there is another OOM related issue reported in V1 of this series, which is tested and looking OK now [5]. MySQL: ====== Testing with innodb_buffer_pool_size=26106127360, in a 2G memcg, using ZRAM as swap and test command: sysbench /usr/share/sysbench/oltp_read_only.lua --mysql-db=sb \ --tables=48 --table-size=2000000 --threads=48 --time=600 run Before: 17260.781429 tps After this series: 17266.842857 tps MySQL is anon folios heavy, involves writeback and file and still looking good. Seems only noise level changes, no regression. FIO: ==== Testing with the following command, where /mnt/ramdisk is a 64G EXT4 ramdisk, each test file is 3G, in a 10G memcg, 6 test run each: fio --directory=/mnt/ramdisk --filename_format='test.$jobnum.img' \ --name=cached --numjobs=16 --size=3072M --buffered=1 --ioengine=mmap \ --rw=randread --norandommap --time_based \ --ramp_time=1m --runtime=5m --group_reporting Before: 9196.481429 MB/s After this series: 9256.105000 MB/s Also seem only noise level changes and no regression or slightly better. Build kernel: ============= Build kernel test using ZRAM as swap, on top of tmpfs, in a 3G memcg using make -j96 and defconfig, measuring system time, 12 test run each. Before: 2589.63s After this series: 2543.58s Also seem only noise level changes, no regression or very slightly better. Android: ======== Xinyu reported a performance gain on Android, too, with this series. The test consisted of cold-starting multiple applications sequentially under moderate system load. [6] Before: Launch Time Summary (all apps, all runs) Mean 868.0ms P50 888.0ms P90 1274.2ms P95 1399.0ms After: Launch Time Summary (all apps, all runs) Mean 850.5ms (-2.07%) P50 861.5ms (-3.04%) P90 1179.0ms (-8.05%) P95 1228.0ms (-12.2%) This patch (of 14): Merge commonly used code for counting evictable folios in a lruvec. No behavior change. Link: https://lore.kernel.org/20260424-mglru-reclaim-v6-0-a57622d770c3@tencent.com Link: https://lore.kernel.org/20260424-mglru-reclaim-v6-1-a57622d770c3@tencent.com Link: https://lore.kernel.org/linux-mm/CAMgjq7BoekNjg-Ra3C8M7=8=75su38w=HD782T5E_cxyeCeH_g@mail.gmail.com/ [1] Link: https://github.com/brianfrankcooper/YCSB/blob/master/workloads/workloadb [2] Link: https://lore.kernel.org/all/20221220214923.1229538-1-yuzhao@google.com/ [3] Link: https://github.com/ryncsn/emm-test-project/tree/master/file-anon-mix-pressure [4] Link: https://lore.kernel.org/linux-mm/acgNCzRDVmSbXrOE@KASONG-MC4/ [5] Link: https://lore.kernel.org/linux-mm/20260417025123.2971253-1-wxy2009nrrr@163.com/ [6] Signed-off-by: Kairui Song Acked-by: Yuanchu Xie Reviewed-by: Barry Song Reviewed-by: Chen Ridong Reviewed-by: Axel Rasmussen Reviewed-by: Baolin Wang Cc: Chris Li Cc: David Hildenbrand Cc: David Stevens Cc: Johannes Weiner Cc: Kalesh Singh Cc: Lorenzo Stoakes Cc: Lorenzo Stoakes (Oracle) Cc: Michal Hocko Cc: Qi Zheng Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vernon Yang Cc: Wei Xu Cc: Yafang Cc: Yu Zhao Cc: Leno Hou Signed-off-by: Andrew Morton --- mm/vmscan.c | 36 ++++++++++++++---------------------- 1 file changed, 14 insertions(+), 22 deletions(-) --- a/mm/vmscan.c~mm-mglru-consolidate-common-code-for-retrieving-evictable-size +++ a/mm/vmscan.c @@ -4084,27 +4084,33 @@ static void set_initial_priority(struct sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY); } -static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) +static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness) { int gen, type, zone; - unsigned long total = 0; - int swappiness = get_swappiness(lruvec, sc); + unsigned long seq, total = 0; struct lru_gen_folio *lrugen = &lruvec->lrugen; - struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); for_each_evictable_type(type, swappiness) { - unsigned long seq; - for (seq = min_seq[type]; seq <= max_seq; seq++) { gen = lru_gen_from_seq(seq); - for (zone = 0; zone < MAX_NR_ZONES; zone++) total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); } } + return total; +} + +static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) +{ + unsigned long total; + int swappiness = get_swappiness(lruvec, sc); + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + + total = lruvec_evictable_size(lruvec, swappiness); + /* whether the size is big enough to be helpful */ return mem_cgroup_online(memcg) ? (total >> sc->priority) : total; } @@ -4909,9 +4915,6 @@ retry: static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, int swappiness, unsigned long *nr_to_scan) { - int gen, type, zone; - unsigned long size = 0; - struct lru_gen_folio *lrugen = &lruvec->lrugen; DEFINE_MIN_SEQ(lruvec); *nr_to_scan = 0; @@ -4919,18 +4922,7 @@ static bool should_run_aging(struct lruv if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; - for_each_evictable_type(type, swappiness) { - unsigned long seq; - - for (seq = min_seq[type]; seq <= max_seq; seq++) { - gen = lru_gen_from_seq(seq); - - for (zone = 0; zone < MAX_NR_ZONES; zone++) - size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - } - } - - *nr_to_scan = size; + *nr_to_scan = lruvec_evictable_size(lruvec, swappiness); /* better to run aging even though eviction is still possible */ return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq; } _ Patches currently in -mm which might be from kasong@tencent.com are mm-mglru-rename-variables-related-to-aging-and-rotation.patch mm-mglru-relocate-the-lru-scan-batch-limit-to-callers.patch mm-mglru-restructure-the-reclaim-loop.patch mm-mglru-scan-and-count-the-exact-number-of-folios.patch mm-mglru-use-a-smaller-batch-for-reclaim.patch mm-mglru-dont-abort-scan-immediately-right-after-aging.patch mm-mglru-remove-redundant-swap-constrained-check-upon-isolation.patch mm-mglru-use-the-common-routine-for-dirty-writeback-reactivation.patch mm-mglru-simplify-and-improve-dirty-writeback-handling.patch mm-mglru-remove-no-longer-used-reclaim-argument-for-folio-protection.patch mm-vmscan-remove-sc-file_taken.patch mm-vmscan-remove-sc-unqueued_dirty.patch mm-vmscan-unify-writeback-reclaim-statistic-and-throttling.patch