From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 741C52E2663 for ; Thu, 3 Sep 2026 18:40:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788460834; cv=none; b=KIAN+7RWsc7rszyAEgp8rIMkhZzN5iLF/rIA0pnRx6JKEkXKMwf+K2FWfrsiYtpwbKwkxfelU/qO2r0P4cmICTMBFiSvyvJAXbLYtllvi1aLa+p+t9hB44P4p/0NFhSYqP+yVyrhrwYxIH3o3lWIT9C47t4E65J+5aSd9RZ2wHc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788460834; c=relaxed/simple; bh=JN3cWO8OGnu1zX1JB/47aOTsL2MFQgqWltBArCoqnM8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oYZrPF2w3l3PXx18G7SoTei4FL9+hcdzzOvpgLheuuliV+ayc/4J4FmI3XbOoes6umQzpb5oAS62ucSdH0GJ2CGSeWzWbUYnunygX5EqndYcveOx85GfSTVYNQm9i370OLbReieSw5WKf4bEzgTvOlUQnbbNbseAG8OAAiqG+Bo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PrtohTWf; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PrtohTWf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60CA81F00A3D; Thu, 3 Sep 2026 18:40:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788460831; bh=Vvg4o+C1wafZjo7bAKbK5wAcwZMYcZLicq267Lp0HUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=PrtohTWfQzlP0TIZS587IbalX4WWXF9L9KQdXrKuS0sGCL6/capXGEYcnlw7liVa5 lMZl7UAAZas2CC5xqf/eQhtI/ZXMPc5unk9qgtLslQoejOnIQxBg4XOvzpshxMOWjK /qsZ6ndg27gTUb2NbuWfDQP1Cy3lYm54N1A/as3+hAOPXYwkuzouoIiU8XuRTgYgju kcWJ3oRAqCW0r/lLkFXwot/zkjF/riI7cpcoZbcixSfmyumhgZhRgRhsAmtFNijef/ PWXdMP4PalEgEjSSrS+UZWiiy7Ur2gS8E8r/fJIsFeWIF+l0LNg0aXMK/qIdeEB8Ei 0p/tSh6vbez6w== From: Sasha Levin To: stable@vger.kernel.org Cc: Yu Zhao , Bharata B Rao , Kalesh Singh , Barry Song , David Stevens , Kairui Song , Andrew Morton , Sasha Levin Subject: [PATCH 6.12.y 2/4] mm/mglru: optimize deactivation Date: Thu, 3 Sep 2026 14:40:25 -0400 Message-ID: <20260903184027.1474890-2-sashal@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260903184027.1474890-1-sashal@kernel.org> References: <2026090303-freezable-flatness-e858@gregkh> <20260903184027.1474890-1-sashal@kernel.org> Precedence: bulk X-Mailing-List: stable@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Yu Zhao [ Upstream commit cc8ec7be78ffaa29f6e0781fd12602974970eb53 ] Do not shuffle a folio in the deactivation paths if it is already in the oldest generation. This reduces the LRU lock contention. Before this patch, the contention is reproducible by FIO, e.g., fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \ -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync \ -bs=4k -numjobs=400 -runtime=25000 --time_based \ -group_reporting -name=mglru 98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl After this patch, deactivate_file_folio() bails out early without taking the LRU lock. A side effect is that a folio can be left at the head of the oldest generation, rather than the tail. If reclaim happens at the same time, it cannot reclaim this folio immediately. Since there is no known correlation between truncation and reclaim, this side effect is considered insignificant. Link: https://lkml.kernel.org/r/20241231043538.4075764-3-yuzhao@google.com Reported-by: Bharata B Rao Closes: https://lore.kernel.org/CAOUHufawNerxqLm7L9Yywp3HJFiYVrYO26ePUb1jH-qxNGWzyA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh Cc: Barry Song Cc: David Stevens Cc: Kairui Song Signed-off-by: Andrew Morton Stable-dep-of: f7e698e326b2 ("mm/mglru: fix and remove redundant unevictable folio handling") Signed-off-by: Sasha Levin --- mm/swap.c | 49 +++++++++++++++++++++++++++++++++++++++++-------- mm/vmscan.c | 38 ++++++++++++++++++++++++++++++-------- 2 files changed, 71 insertions(+), 16 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index ff846915db454..94fa59d230b18 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -411,7 +411,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } #ifdef CONFIG_LRU_GEN -static void folio_inc_refs(struct folio *folio) + +static void lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); @@ -438,10 +439,34 @@ static void folio_inc_refs(struct folio *folio) new_flags |= old_flags & ~LRU_REFS_MASK; } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } -#else -static void folio_inc_refs(struct folio *folio) + +static bool lru_gen_clear_refs(struct folio *folio) +{ + struct lru_gen_folio *lrugen; + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + + if (gen < 0) + return true; + + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + + lrugen = &folio_lruvec(folio)->lrugen; + /* whether can do without shuffling under the LRU lock */ + return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); +} + +#else /* !CONFIG_LRU_GEN */ + +static void lru_gen_inc_refs(struct folio *folio) +{ +} + +static bool lru_gen_clear_refs(struct folio *folio) { + return false; } + #endif /* CONFIG_LRU_GEN */ /** @@ -460,7 +485,7 @@ static void folio_inc_refs(struct folio *folio) void folio_mark_accessed(struct folio *folio) { if (lru_gen_enabled()) { - folio_inc_refs(folio); + lru_gen_inc_refs(folio); return; } @@ -556,7 +581,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) */ static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio) { - bool active = folio_test_active(folio); + bool active = folio_test_active(folio) || lru_gen_enabled(); long nr_pages = folio_nr_pages(folio); if (folio_test_unevictable(folio)) @@ -621,7 +646,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) lruvec_del_folio(lruvec, folio); folio_clear_active(folio); - folio_clear_referenced(folio); + if (lru_gen_enabled()) + lru_gen_clear_refs(folio); + else + folio_clear_referenced(folio); /* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal @@ -689,6 +717,9 @@ void deactivate_file_folio(struct folio *folio) if (folio_test_unevictable(folio) || !folio_test_lru(folio)) return; + if (lru_gen_enabled() && lru_gen_clear_refs(folio)) + return; + folio_batch_add_and_move(folio, lru_deactivate_file); } @@ -702,8 +733,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio) || !folio_test_lru(folio) || - !(folio_test_active(folio) || lru_gen_enabled())) + if (folio_test_unevictable(folio) || !folio_test_lru(folio)) + return; + + if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio)) return; folio_batch_add_and_move(folio, lru_deactivate); diff --git a/mm/vmscan.c b/mm/vmscan.c index e5cdf114c3ba5..7a112f93a681a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4313,7 +4313,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c int tier_idx) { bool success; - bool dirty, writeback; int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); @@ -4359,8 +4358,9 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c return true; } - dirty = folio_test_dirty(folio); - writeback = folio_test_writeback(folio); + bool dirty = folio_test_dirty(folio); + bool writeback = folio_test_writeback(folio); + if (type == LRU_GEN_FILE && dirty) { sc->nr.file_taken += delta; if (!writeback) @@ -4627,12 +4627,23 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); list_for_each_entry_safe_reverse(folio, next, &list, lru) { + DEFINE_MIN_SEQ(lruvec); + if (!folio_evictable(folio)) { list_del(&folio->lru); folio_putback_lru(folio); continue; } + /* retry folios that may have missed folio_rotate_reclaimable() */ + if (!skip_retry && !folio_test_active(folio) && !folio_mapped(folio) && + !folio_test_dirty(folio) && !folio_test_writeback(folio)) { + if (!folio_test_referenced(folio) && !folio_test_locked(folio)) { + list_move(&folio->lru, &clean); + continue; + } + } + if (folio_test_reclaim(folio) && (folio_test_dirty(folio) || folio_test_writeback(folio))) { /* restore LRU_REFS_FLAGS cleared by isolate_folio() */ @@ -4644,14 +4655,25 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap if (skip_retry || folio_test_active(folio) || folio_test_referenced(folio) || folio_mapped(folio) || folio_test_locked(folio) || folio_test_dirty(folio) || folio_test_writeback(folio)) { + unsigned long seq = READ_ONCE(lruvec->lrugen.max_seq); + /* don't add rejected folios to the oldest generation */ - set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, - BIT(PG_active)); + if (!folio_test_active(folio)) { + if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) || + (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio)))) + seq--; + else if (min_seq[type] + MIN_NR_GENS < seq) + seq = min_seq[type] + 1; + else + seq = min_seq[type]; + } + + if (seq == min_seq[type]) + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, + BIT(PG_active)); continue; } - - /* retry folios that may have missed folio_rotate_reclaimable() */ - list_move(&folio->lru, &clean); } spin_lock_irq(&lruvec->lru_lock); -- 2.53.0