From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B92E0CD8CAC for ; Tue, 10 Oct 2023 18:19:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233117AbjJJST0 (ORCPT ); Tue, 10 Oct 2023 14:19:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234160AbjJJSTZ (ORCPT ); Tue, 10 Oct 2023 14:19:25 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE3E8B0 for ; Tue, 10 Oct 2023 11:19:23 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 044AFC433C8; Tue, 10 Oct 2023 18:19:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1696961963; bh=RPM38n6CXfjNKxEZQ0wrsWJN3xxo3xkTEYHq5tqQ+Zw=; h=Date:To:From:Subject:From; b=FaG0PhwSKrwXG+ADbNxNfg6tTCn2knAVy0gw1Vl1SP8/oicNh+MESGjXvI0lVB6A8 5ZHdPnPnELZsiJEdwUYXdOzeAUBg/S2w0h+z1oJM3JEbKUAlUnGFNxg3MvXfscAhB4 Y2d3dUur9OpydjeokKSUWeH7iuwYxA8UN/FVr8CY= Date: Tue, 10 Oct 2023 11:19:09 -0700 To: mm-commits@vger.kernel.org, weixugc@google.com, tj@kernel.org, shakeelb@google.com, roman.gushchin@linux.dev, muchun.song@linux.dev, mkoutny@suse.com, mhocko@kernel.org, longman@redhat.com, ivan@cloudflare.com, hannes@cmpxchg.org, gthelen@google.com, cerasuolodomenico@gmail.com, yosryahmed@google.com, akpm@linux-foundation.org From: Andrew Morton Subject: + mm-workingset-move-the-stats-flush-into-workingset_test_recent.patch added to mm-unstable branch Message-Id: <20231010181919.044AFC433C8@smtp.kernel.org> Precedence: bulk Reply-To: linux-kernel@vger.kernel.org List-ID: X-Mailing-List: mm-commits@vger.kernel.org The patch titled Subject: mm: workingset: move the stats flush into workingset_test_recent() has been added to the -mm mm-unstable branch. Its filename is mm-workingset-move-the-stats-flush-into-workingset_test_recent.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-workingset-move-the-stats-flush-into-workingset_test_recent.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Yosry Ahmed Subject: mm: workingset: move the stats flush into workingset_test_recent() Date: Tue, 10 Oct 2023 03:21:15 +0000 The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Link: https://lkml.kernel.org/r/20231010032117.1577496-5-yosryahmed@google.com Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo Cc: Greg Thelen Cc: Ivan Babrou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Tejun heo Cc: Waiman Long Cc: Wei Xu Signed-off-by: Andrew Morton --- mm/workingset.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) --- a/mm/workingset.c~mm-workingset-move-the-stats-flush-into-workingset_test_recent +++ a/mm/workingset.c @@ -425,8 +425,16 @@ bool workingset_test_recent(void *shadow struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, workingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent = lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, workingset); + + rcu_read_unlock(); + return recent; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -448,8 +456,16 @@ bool workingset_test_recent(void *shadow * configurations instead. */ eviction_memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !eviction_memcg) + if (!mem_cgroup_disabled() && + (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { + rcu_read_unlock(); return false; + } + + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +509,7 @@ bool workingset_test_recent(void *shadow } } + mem_cgroup_put(eviction_memcg); return refault_distance <= workingset_size; } @@ -519,19 +536,16 @@ void workingset_refault(struct folio *fo return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); @@ -540,7 +554,7 @@ void workingset_refault(struct folio *fo mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +570,6 @@ void workingset_refault(struct folio *fo lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /** _ Patches currently in -mm which might be from yosryahmed@google.com are mm-memcg-refactor-page-state-unit-helpers.patch mm-memcg-normalize-the-value-passed-into-memcg_rstat_updated.patch mm-memcg-change-flush_next_time-to-flush_last_time.patch mm-memcg-move-vmstats-structs-definition-above-flushing-code.patch mm-memcg-make-stats-flushing-threshold-per-memcg.patch mm-workingset-move-the-stats-flush-into-workingset_test_recent.patch mm-memcg-restore-subtree-stats-flushing.patch