From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5321C278161 for ; Thu, 23 Jul 2026 20:38:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839102; cv=none; b=obDrRpXEsVSX+3k0ItqiQU4imGTH6hfm2anEaDlaYGcEdpK7CSzS2wNgXRnAf0mLrNMXflS4fX7KHdaSNaAT85vqTJ0ffuFyvGW87jbDmwjCFpin075UCW4ytcIYft2SOTsR3tzgCH51R90m3OxPJLdnfKoSAh5AqjefaNhagJ0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839102; c=relaxed/simple; bh=fCHka9HqK7RXfO3uRt49tWt63iTjieLx7SBoH2hJhyQ=; h=Date:To:From:Subject:Message-Id; b=HK912Tck1uxk4UTblfqIznjNEtWxH1eNt8vg2s1kjUFYejq7IK8AniKr9Bl0DyZU3kta6+YLvffOWnsptARIStm9q2NZsRdWC1LY9gOxG1vawgKDklfctrBAHxFfUmxXQ3YyuHdNl2cOg6n4ghbVsG/cYQEciE6xPKXLO4wJROY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=H93Cb7A6; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="H93Cb7A6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 95B1A1F000E9; Thu, 23 Jul 2026 20:38:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1784839095; bh=6VQGtE6scjqoNBmyS5uevGtd1pp+A7iOGzd+8nSWiw8=; h=Date:To:From:Subject; b=H93Cb7A6mCqXy0JjQfg9XCDilIXL49SVLUjDCVwHq7szhWukSgkTPO8ENn5Inwn/D JvhfZTZOKAJCLMAzwF62pBIxN1fl2Jo3ybqiBFopbXvTRB/CI2kqocC+aE5N6D126s 3gVIFOmqv4F7bN76x52P44uuqGbFRAW1fR44UccA= Date: Thu, 23 Jul 2026 13:38:15 -0700 To: mm-commits@vger.kernel.org,yuanchu@google.com,weixugc@google.com,vbabka@kernel.org,surenb@google.com,shakeel.butt@linux.dev,rppt@kernel.org,roman.gushchin@linux.dev,rientjes@google.com,nphamcs@gmail.com,muchun.song@linux.dev,mhocko@suse.com,ljs@kernel.org,liam@infradead.org,kasong@tencent.com,hannes@cmpxchg.org,david@kernel.org,chrisl@kernel.org,baoquan.he@linux.dev,axelrasmussen@google.com,usama.arif@linux.dev,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost.patch added to mm-new branch Message-Id: <20260723203815.95B1A1F000E9@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost has been added to the -mm mm-new branch. Its filename is mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next If a few days of testing in mm-new is successful, the patch will me moved into mm.git's mm-unstable branch, which is included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Usama Arif Subject: mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost Date: Mon, 20 Jul 2026 09:41:23 -0700 The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are: - shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink. - shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink. - workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault. - prepare_scan_control() takes lru_lock just to snapshot the two scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call. This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth. Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was: cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy. Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill. Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node. Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics. NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed. memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged. Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations. Link: https://lore.kernel.org/20260720164207.450685-3-usama.arif@linux.dev Signed-off-by: Usama Arif Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Johannes Weiner Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 15 +++++- include/linux/vmstat.h | 1 mm/folio.c | 69 ---------------------------- mm/internal.h | 3 - mm/memcontrol-v1.c | 4 - mm/memcontrol.c | 5 +- mm/mmzone.c | 1 mm/vmscan.c | 93 +++++++++++++++++++++++++++++++++------ mm/vmstat.c | 2 mm/workingset.c | 5 -- 10 files changed, 101 insertions(+), 97 deletions(-) --- a/include/linux/mmzone.h~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/include/linux/mmzone.h @@ -323,6 +323,8 @@ enum node_stat_item { PGSCAN_PROACTIVE, PGSCAN_ANON, PGSCAN_FILE, + PGROTATE_ANON, + PGROTATE_FILE, PGREFILL, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, @@ -755,6 +757,12 @@ void lru_gen_reparent_memcg(struct mem_c #endif /* CONFIG_LRU_GEN */ +struct lru_cost { + unsigned long count; + unsigned long last_rotated; + unsigned long last_io; +}; + struct lruvec { struct list_head lists[NR_LRU_LISTS]; /* per lruvec lru_lock for memcg */ @@ -763,9 +771,12 @@ struct lruvec { * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU * increases, the reclaim scan balance tips toward the other. + * Updated and decayed at prepare_scan_control() time; cost_lock + * serialises that update. */ - unsigned long anon_cost; - unsigned long file_cost; + struct lru_cost cost[ANON_AND_FILE]; + /* Protects cost[]. */ + spinlock_t cost_lock; /* Non-resident age, driven by LRU movement */ atomic_long_t nonresident_age; /* Refaults at the time of last reclaim cycle */ --- a/include/linux/vmstat.h~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/include/linux/vmstat.h @@ -20,7 +20,6 @@ struct reclaim_stat { unsigned nr_congested; unsigned nr_writeback; unsigned nr_immediate; - unsigned nr_pageout; unsigned nr_activate[ANON_AND_FILE]; unsigned nr_ref_keep; unsigned nr_unmap_fail; --- a/mm/folio.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/folio.c @@ -265,73 +265,6 @@ void folio_rotate_reclaimable(struct fol folio_batch_add_and_move(folio, lru_move_tail); } -void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, - unsigned int nr_io, unsigned int nr_rotated) - __releases(lruvec->lru_lock) - __releases(rcu) -{ - unsigned long cost; - - /* - * Reflect the relative cost of incurring IO and spending CPU - * time on rotations. This doesn't attempt to make a precise - * comparison, it just says: if reloads are about comparable - * between the LRU lists, or rotations are overwhelmingly - * different between them, adjust scan balance for CPU work. - */ - cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated; - if (!cost) { - spin_unlock_irq(&lruvec->lru_lock); - rcu_read_unlock(); - return; - } - - for (;;) { - unsigned long lrusize; - - /* Record cost event */ - if (file) - lruvec->file_cost += cost; - else - lruvec->anon_cost += cost; - - /* - * Decay previous events - * - * Because workloads change over time (and to avoid - * overflow) we keep these statistics as a floating - * average, which ends up weighing recent refaults - * more than old ones. - */ - lrusize = lruvec_page_state(lruvec, NR_INACTIVE_ANON) + - lruvec_page_state(lruvec, NR_ACTIVE_ANON) + - lruvec_page_state(lruvec, NR_INACTIVE_FILE) + - lruvec_page_state(lruvec, NR_ACTIVE_FILE); - - if (lruvec->file_cost + lruvec->anon_cost > lrusize / 4) { - lruvec->file_cost /= 2; - lruvec->anon_cost /= 2; - } - - spin_unlock_irq(&lruvec->lru_lock); - lruvec = parent_lruvec(lruvec); - if (!lruvec) { - rcu_read_unlock(); - break; - } - spin_lock_irq(&lruvec->lru_lock); - } -} - -void lru_note_cost_refault(struct folio *folio) -{ - struct lruvec *lruvec; - - lruvec = folio_lruvec_lock_irq(folio); - lru_note_cost_unlock_irq(lruvec, folio_is_file_lru(folio), - folio_nr_pages(folio), 0); -} - static void lru_activate(struct lruvec *lruvec, struct folio *folio) { long nr_pages = folio_nr_pages(folio); @@ -1162,8 +1095,6 @@ void lru_reparent_memcg(struct mem_cgrou child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid)); - parent_lruvec->anon_cost += child_lruvec->anon_cost; - parent_lruvec->file_cost += child_lruvec->file_cost; for_each_lru(lru) lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid); --- a/mm/internal.h~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/internal.h @@ -41,9 +41,6 @@ void workingset_refault(struct folio *fo void workingset_activation(struct folio *folio); /* mm/folio.c */ -void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, - unsigned int nr_io, unsigned int nr_rotated); -void lru_note_cost_refault(struct folio *folio); void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma); static inline bool folio_may_be_lru_cached(struct folio *folio) --- a/mm/memcontrol.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/memcontrol.c @@ -399,6 +399,7 @@ static const unsigned int memcg_node_sta NR_SHMEM_THPS, NR_FILE_THPS, NR_ANON_THPS, + NR_VMSCAN_WRITE, NR_VMALLOC, NR_KERNEL_STACK_KB, NR_PAGETABLE, @@ -425,6 +426,8 @@ static const unsigned int memcg_node_sta PGSCAN_PROACTIVE, PGSCAN_ANON, PGSCAN_FILE, + PGROTATE_ANON, + PGROTATE_FILE, PGREFILL, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, @@ -524,7 +527,7 @@ unsigned long lruvec_page_state(struct l * reading from per-CPU delta skew must present as zero. * * XXX: This helper (and its node/global peers) exists because we place - * monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*) + * monotonically-incremented event counters (NR_VMSCAN_WRITE and PGROTATE_*) * into enum node_stat_item. */ unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, --- a/mm/memcontrol-v1.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/memcontrol-v1.c @@ -2287,8 +2287,8 @@ void memcg1_stat_format(struct mem_cgrou for_each_online_pgdat(pgdat) { mz = memcg->nodeinfo[pgdat->node_id]; - anon_cost += mz->lruvec.anon_cost; - file_cost += mz->lruvec.file_cost; + anon_cost += mz->lruvec.cost[WORKINGSET_ANON].count; + file_cost += mz->lruvec.cost[WORKINGSET_FILE].count; } seq_buf_printf(s, "anon_cost %lu\n", anon_cost); seq_buf_printf(s, "file_cost %lu\n", file_cost); --- a/mm/mmzone.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/mmzone.c @@ -79,6 +79,7 @@ void lruvec_init(struct lruvec *lruvec) memset(lruvec, 0, sizeof(struct lruvec)); spin_lock_init(&lruvec->lru_lock); + spin_lock_init(&lruvec->cost_lock); zswap_lruvec_state_init(lruvec); for_each_lru(lru) --- a/mm/vmscan.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/vmscan.c @@ -670,7 +670,7 @@ static pageout_t pageout(struct swap_io_ folio_clear_reclaim(folio); trace_mm_vmscan_write_folio(folio); - node_stat_add_folio(folio, NR_VMSCAN_WRITE); + lruvec_stat_mod_folio(folio, NR_VMSCAN_WRITE, folio_nr_pages(folio)); return PAGE_SUCCESS; } @@ -1412,8 +1412,6 @@ retry: sc->nr_scanned -= (nr_pages - 1); nr_pages = 1; } - stat->nr_pageout += nr_pages; - if (folio_test_writeback(folio)) goto keep; if (folio_test_dirty(folio)) @@ -2036,10 +2034,10 @@ static unsigned long shrink_inactive_lis item = PGSTEAL_KSWAPD + reclaimer_offset(sc); mod_lruvec_state(lruvec, item, nr_reclaimed); mod_lruvec_state(lruvec, PGSTEAL_ANON + file, nr_reclaimed); + if (nr_scanned > nr_reclaimed) + mod_lruvec_state(lruvec, PGROTATE_ANON + file, + nr_scanned - nr_reclaimed); - lruvec_lock_irq(lruvec); - lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, - nr_scanned - nr_reclaimed); handle_reclaim_writeback(nr_taken, pgdat, sc, &stat); trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_scanned, nr_reclaimed, &stat, sc->priority, file); @@ -2145,9 +2143,9 @@ static void shrink_active_list(unsigned count_vm_events(PGDEACTIVATE, nr_deactivate); count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); + if (nr_rotated) + mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated); - lruvec_lock_irq(lruvec); - lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -2280,8 +2278,10 @@ enum scan_balance { static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) { - unsigned long file; + struct lru_cost *anon_cost, *file_cost; struct lruvec *target_lruvec; + unsigned long lrusize; + unsigned long file; if (lru_gen_enabled() && !lru_gen_switching()) return; @@ -2297,11 +2297,69 @@ static void prepare_scan_control(pg_data /* * Determine the scan balance between anon and file LRUs. + * + * The cost model is based on rotations, refaults and + * reclaim-driven writes (anon only) on each side. + * + * These event counters are monotonic, so each reclaim cycle + * the delta since the last scan is extracted and incorporated + * into a decaying average. This ensures currency, as workloads + * change over time, and avoids overflow in the calculations. + * + * Use lruvec_page_state_monotonic() so unsigned subtraction + * yields the correct delta across a signed-long wraparound of + * the underlying counter (a real hazard on 32-bit that the + * clamp in lruvec_page_state() would otherwise turn into a huge + * spurious delta). */ - spin_lock_irq(&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; - spin_unlock_irq(&target_lruvec->lru_lock); + spin_lock(&target_lruvec->cost_lock); + + for (int f = 0; f <= 1; f++) { + struct lru_cost *cost = &target_lruvec->cost[f]; + unsigned long rotated, io, nr_rotated, nr_io; + + rotated = lruvec_page_state_monotonic(target_lruvec, + PGROTATE_ANON + f); + io = lruvec_page_state_monotonic(target_lruvec, + WORKINGSET_RESTORE_BASE + f); + if (f == WORKINGSET_ANON) + io += lruvec_page_state_monotonic(target_lruvec, + NR_VMSCAN_WRITE); + + nr_rotated = rotated - cost->last_rotated; + nr_io = io - cost->last_io; + + /* + * Reflect the relative cost of incurring IO and spending + * CPU time on rotations. This doesn't attempt to make a + * precise comparison, it just says: if reloads are about + * comparable between the LRU lists, or rotations are + * overwhelmingly different between them, adjust scan + * balance for CPU work. + */ + cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated; + + cost->last_rotated = rotated; + cost->last_io = io; + } + + anon_cost = &target_lruvec->cost[WORKINGSET_ANON]; + file_cost = &target_lruvec->cost[WORKINGSET_FILE]; + + lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) + + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) + + lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) + + lruvec_page_state(target_lruvec, NR_ACTIVE_FILE); + + while (anon_cost->count + file_cost->count > lrusize / 4) { + anon_cost->count /= 2; + file_cost->count /= 2; + } + + sc->anon_cost = anon_cost->count; + sc->file_cost = file_cost->count; + + spin_unlock(&target_lruvec->cost_lock); /* * Target desirable inactive:active list ratios for the anon @@ -4816,7 +4874,8 @@ static int evict_folios(unsigned long nr struct reclaim_stat stat; struct lru_gen_mm_walk *walk; int scanned, reclaimed; - int isolated = 0, type, type_scanned; + int isolated = 0, nr_isolated = 0, type, type_scanned; + unsigned long total_reclaimed = 0; bool skip_retry = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -4828,6 +4887,7 @@ static int evict_folios(unsigned long nr scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &list, &isolated, &type, &type_scanned); + nr_isolated = isolated; /* Scanning may have emptied the oldest gen, flush it */ if (scanned) @@ -4840,6 +4900,7 @@ static int evict_folios(unsigned long nr retry: reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false, memcg); sc->nr_reclaimed += reclaimed; + total_reclaimed += reclaimed; /* Retry pass is only meant for clean folios without new isolation */ if (isolated) handle_reclaim_writeback(isolated, pgdat, sc, &stat); @@ -4891,6 +4952,10 @@ retry: goto retry; } + if (nr_isolated > total_reclaimed) + mod_lruvec_state(lruvec, PGROTATE_ANON + type, + nr_isolated - total_reclaimed); + return scanned; } --- a/mm/vmstat.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/vmstat.c @@ -1301,6 +1301,8 @@ const char * const vmstat_text[] = { [I(PGSCAN_PROACTIVE)] = "pgscan_proactive", [I(PGSCAN_ANON)] = "pgscan_anon", [I(PGSCAN_FILE)] = "pgscan_file", + [I(PGROTATE_ANON)] = "pgrotate_anon", + [I(PGROTATE_FILE)] = "pgrotate_file", [I(PGREFILL)] = "pgrefill", #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] = "nr_hugetlb", --- a/mm/workingset.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +++ a/mm/workingset.c @@ -584,11 +584,6 @@ void workingset_refault(struct folio *fo /* Folio was active prior to eviction */ if (workingset) { folio_set_workingset(folio); - /* - * XXX: Move to folio_add_lru() when it supports new vs - * putback - */ - lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } out: _ Patches currently in -mm which might be from usama.arif@linux.dev are mm-swap_state-remove-unnecessary-lru_add_drain-from-readahead.patch mm-add-softleaf_to_pmd-and-convert-existing-callers.patch mm-extract-mm_prepare_for_swap_entries-helper.patch fs-proc-use-softleaf_has_pfn-in-pagemap-pmd-walker.patch mm-huge_memory-move-softleaf_to_folio-inside-migration-branch.patch mm-migrate_device-move-softleaf_to_folio-inside-device-private-branch.patch mm-rename-arch_enable_thp_migration-to-arch_has_pmd_softleaves.patch mm-vmpressure-skip-tree=true-accounting-on-cgroup-v2.patch mm-vmpressure-skip-tree=true-accounting-on-cgroup-v2-fix.patch mm-vmpressure-move-v1-userspace-eventfd-code-into-memcontrol-v1c.patch mm-migrate_device-pin-large-folios-before-splitting.patch mm-migrate_device-pin-large-folios-before-splitting-fix.patch mm-mempolicy-skip-non-present-pmds-when-queueing-folios.patch mm-madvise-skip-device-private-pmds-in-cold-and-pageout-walks.patch mm-huge_memory-skip-device-private-pmds-in-madvise_free_huge_pmd.patch mm-vmstat-mm-memcontrol-add-_monotonic-vmstat-readers.patch mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost.patch