From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A532EB64D9 for ; Thu, 6 Jul 2023 07:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 945448D0002; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D9118D0001; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 722C18D0002; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 59D4B8D0001 for ; Thu, 6 Jul 2023 03:45:12 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2438AA0213 for ; Thu, 6 Jul 2023 07:45:12 +0000 (UTC) X-FDA: 80980401264.24.205F83D Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf02.hostedemail.com (Postfix) with ESMTP id 8C5DC80013 for ; Thu, 6 Jul 2023 07:45:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=E7zR4qBb; spf=pass (imf02.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688629509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F8ycTdDdQYcTUA9W4ZMinDPt78Egmnbx9cZ0zDL/uk0=; b=TKPnNnZFSw5sO2/AdK3qPeio86KVuvQblvxDroE5RjaFlZCO+daHfZ3eLbH/5kTg3thlr8 oslVGHk8lNfikLv1R/fGKURb3+naZnSWaLnQTCR/TsQiL4K/WqKi0fIldJ4sBlcfHDjxXR cTVpdmN2Y3kk/n6xcwGaCKRMP1VpBJs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=E7zR4qBb; spf=pass (imf02.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688629509; a=rsa-sha256; cv=none; b=O088Gfd6F/yvg0qLofq/iTBMa7/InwXD3zSG2/+5mqfuPS7NYI/hK6ZYtpRhQs0V+ozfvb P3WEtTSWBVegfbJQfdghA6Y347ZuGMQW9+lSj7gxgpTnIiuTLhEXMrTTABeA9Fkvg6ljpB eWO8dq9rlYSp5ge9vVt4ceqSp6RKndY= Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3667gOn1008074; Thu, 6 Jul 2023 07:45:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=F8ycTdDdQYcTUA9W4ZMinDPt78Egmnbx9cZ0zDL/uk0=; b=E7zR4qBbKQQ1CTStQ5KFZ/4gsQNyJ7bLOGEaeuG9GA2dmhvrLhQbCvB2hX1Eko/UNDWh AOWeD9wD+4gHWRut7zT02NfO20k3uhFZChc1Yib1OaeWrHeZhEGL60pvIygO9YPWAVWt ygH2tvcRbHzUE+XDVubCjmkKWqN/T9flBe2KXKThvhzGpa1TpwYq/6nNNNUtrsjfxHDy 5HtZO2NudY3hfbkSQzgv+6JzaxKkP+N3fTyhAKe2uo29De1zDbdDdjK3fJE5gYAJomq0 UUMtN4Za487Dr0rpfen5np95Kn7yg6dLRjvh8RZcONz0qIUFPl9MutkFKn7d3JqaXZjj /Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnsev826q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 07:45:07 +0000 Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3667gega009373; Thu, 6 Jul 2023 07:45:06 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnsev8244-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 07:45:06 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 365IoHeP001861; Thu, 6 Jul 2023 06:21:04 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([9.208.130.101]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3rjbs57x1h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 06:21:04 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3666L3mR459354 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 06:21:03 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5F60158057; Thu, 6 Jul 2023 06:21:03 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A39FF58059; Thu, 6 Jul 2023 06:21:01 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 06:21:01 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Yu Zhao , "Aneesh Kumar K.V" Subject: [PATCH v2 2/5] mm/mglru: Move Bloom filter code around Date: Thu, 6 Jul 2023 11:50:41 +0530 Message-ID: <20230706062044.816068-3-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> References: <20230706062044.816068-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: J1aRR6Txd7H886ubl0dAyxFM-GyDAsR4 X-Proofpoint-ORIG-GUID: oFYcoHYmPZ5LnT0Cj4vcr7YSyuE9S5P3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_04,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 suspectscore=0 mlxlogscore=999 impostorscore=0 adultscore=0 spamscore=0 lowpriorityscore=0 phishscore=0 priorityscore=1501 bulkscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060065 X-Rspamd-Queue-Id: 8C5DC80013 X-Rspam-User: X-Stat-Signature: r8jdwdpi5ab3sbbyactbx33e4mtmr5t5 X-Rspamd-Server: rspam01 X-HE-Tag: 1688629509-148181 X-HE-Meta: U2FsdGVkX1+D3NXonYhM/0Y0qB/ZnPqosFBi9tOroljVWRaUQkEyNXwdFF3SBA0nBWZ3ve3TNzFZQOMiZKqvEfbsgYr5JZPlMqz2OfBKYpsjDnVa+qu7ZApfqCTjRy/1L4PdA8124hfRGGgbrQGWuHK2TIHiCuyKjDLVPLvNmHEGlbH3DQOI6pWGqAknu6wt2C4wAL2CFStJLuKOufZrIhxaAXKRj1V4BQ4RYNJ0aXG8vbGlSC4TTdgwW5GDyr1WGdB1YstG3buFIvTZr8nZ+HzltPIgvvZoWllaobYmZhyQ2XTXs2agneUEQR0ZdZyC8mMOKDVTmEpzFCd7OWiANz2VtgbWlC4BOPQ2oUbzm0T6cC1mxD4T9Cw5eBf1xY39liiMU4goRS3s9M24Qn83bKEK+XKWvT8jdIS5wSSCjFOl1QCU0DcI4ZBg2TOPIHEC6IuApUEBS5cxcLdMdf3cEhHAG3bSZ/EB0rQ9LFL4K9k+AJ9N2mBCqVyu1F2pkKIUE/7TOCzlYSnIYc3UdE69ToHE24rKRJdK445OXJERzH1DraJQpA/juVnfQUfAeD2Pf5lvO/GnDPz4zl/8/ARLafjaWBnVzyG4uSln3SX205ehVULSxp40VOuSY/u7ij43KpxMNqKQuKUQJTKoJ2ce+tcsulMSIcGY8vklE2R4SBB0tt6FaFxUjEqOCnL956vqOMr9jiN/mLRNeiEPRxDIhsrLYpUTblma6/K/YTNcB4JsYNxaweJhfmGFgkUu+AKESOBENHtF0THXiWy+qxjE8Zw9FTNfq6lLzhpHKjYQYnPopTNgi32+kcNL5gswzyBgp0bxlvF8b0fDetcgKR+QP1ZNxiUxgOMLUob9DdDOgYx4V9iFeJ37Quo+aPtr/9DVwy+fmfM8QNYKcxzS8CCxe5Gvgf4AyROOlCRVXQNR6bdyJC686Ss9cnJViFRcs3g2SQX8i/fHRFbYwQ6vO/S PjGYVz9j yonsOriYAlky98XCfQdcWAV4Gz0C0Ieq3ZFMSeg/VSaNHOs8V/qHBF+FioR6hGsnbacLVkDeZKQVMzjKAfDTxuo4xytWd7Rwn4HCXOcU7ODCfM0tiHf+UhAjVc/T1/U2zyZHhjOnx9swiCXuTdwRMcNpMmm91Fw209McbkoqIBRxQ1XTpgVxCcjzCthvDWWh+blMkMnaomqV0RzwjXrwyZGlNgqb1I/tumSyPG9XHKqTljeE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows to avoid building this code on powerpc architecture. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- mm/vmscan.c | 462 ++++++++++++++++++++++++++-------------------------- 1 file changed, 231 insertions(+), 231 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b183f704d5d..c5fbc3babcd8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3325,6 +3325,237 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) get_nr_gens(lruvec, LRU_GEN_ANON) <= MAX_NR_GENS; } +/****************************************************************************** + * PID controller + ******************************************************************************/ + +/* + * A feedback loop based on Proportional-Integral-Derivative (PID) controller. + * + * The P term is refaulted/(evicted+protected) from a tier in the generation + * currently being evicted; the I term is the exponential moving average of the + * P term over the generations previously evicted, using the smoothing factor + * 1/2; the D term isn't supported. + * + * The setpoint (SP) is always the first tier of one type; the process variable + * (PV) is either any tier of the other type or any other tier of the same + * type. + * + * The error is the difference between the SP and the PV; the correction is to + * turn off protection when SP>PV or turn on protection when SPlrugen; + int hist = lru_hist_from_seq(lrugen->min_seq[type]); + + pos->refaulted = lrugen->avg_refaulted[type][tier] + + atomic_long_read(&lrugen->refaulted[hist][type][tier]); + pos->total = lrugen->avg_total[type][tier] + + atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + pos->total += lrugen->protected[hist][type][tier - 1]; + pos->gain = gain; +} + +static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) +{ + int hist, tier; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + bool clear = carryover ? NR_HIST_GENS == 1 : NR_HIST_GENS > 1; + unsigned long seq = carryover ? lrugen->min_seq[type] : lrugen->max_seq + 1; + + lockdep_assert_held(&lruvec->lru_lock); + + if (!carryover && !clear) + return; + + hist = lru_hist_from_seq(seq); + + for (tier = 0; tier < MAX_NR_TIERS; tier++) { + if (carryover) { + unsigned long sum; + + sum = lrugen->avg_refaulted[type][tier] + + atomic_long_read(&lrugen->refaulted[hist][type][tier]); + WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); + + sum = lrugen->avg_total[type][tier] + + atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + sum += lrugen->protected[hist][type][tier - 1]; + WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); + } + + if (clear) { + atomic_long_set(&lrugen->refaulted[hist][type][tier], 0); + atomic_long_set(&lrugen->evicted[hist][type][tier], 0); + if (tier) + WRITE_ONCE(lrugen->protected[hist][type][tier - 1], 0); + } + } +} + +static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) +{ + /* + * Return true if the PV has a limited number of refaults or a lower + * refaulted/total than the SP. + */ + return pv->refaulted < MIN_LRU_BATCH || + pv->refaulted * (sp->total + MIN_LRU_BATCH) * sp->gain <= + (sp->refaulted + 1) * pv->total * pv->gain; +} + +/****************************************************************************** + * the aging + ******************************************************************************/ + +/* protect pages accessed multiple times through file descriptors */ +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +{ + int type = folio_is_file_lru(folio); + struct lru_gen_folio *lrugen = &lruvec->lrugen; + int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); + + do { + new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + /* folio_update_gen() has promoted this page? */ + if (new_gen >= 0 && new_gen != old_gen) + return new_gen; + + new_gen = (old_gen + 1) % MAX_NR_GENS; + + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); + new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF; + /* for folio_end_writeback() */ + if (reclaiming) + new_flags |= BIT(PG_reclaim); + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + lru_gen_update_size(lruvec, folio, old_gen, new_gen); + + return new_gen; +} + +static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long pfn = pte_pfn(pte); + + VM_WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end); + + if (!pte_present(pte) || is_zero_pfn(pfn)) + return -1; + + if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) + return -1; + + if (WARN_ON_ONCE(!pfn_valid(pfn))) + return -1; + + return pfn; +} + +static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, + struct pglist_data *pgdat, bool can_swap) +{ + struct folio *folio; + + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return NULL; + + folio = pfn_folio(pfn); + if (folio_nid(folio) != pgdat->node_id) + return NULL; + + if (folio_memcg_rcu(folio) != memcg) + return NULL; + + /* file VMAs can contain anon pages from COW */ + if (!folio_is_file_lru(folio) && !can_swap) + return NULL; + + return folio; +} + +/* promote pages accessed through page tables */ +static int folio_update_gen(struct folio *folio, int gen) +{ + unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); + VM_WARN_ON_ONCE(!rcu_read_lock_held()); + + do { + /* lru_gen_del_folio() has isolated this page? */ + if (!(old_flags & LRU_GEN_MASK)) { + /* for shrink_folio_list() */ + new_flags = old_flags | BIT(PG_referenced); + continue; + } + + new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); + new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; +} + +static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio, + int old_gen, int new_gen) +{ + int type = folio_is_file_lru(folio); + int zone = folio_zonenum(folio); + int delta = folio_nr_pages(folio); + + VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS); + VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS); + + walk->batched++; + + walk->nr_pages[old_gen][type][zone] -= delta; + walk->nr_pages[new_gen][type][zone] += delta; +} + +static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk) +{ + int gen, type, zone; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + + walk->batched = 0; + + for_each_gen_type_zone(gen, type, zone) { + enum lru_list lru = type * LRU_INACTIVE_FILE; + int delta = walk->nr_pages[gen][type][zone]; + + if (!delta) + continue; + + walk->nr_pages[gen][type][zone] = 0; + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + lru += LRU_ACTIVE; + __update_lru_size(lruvec, lru, zone, delta); + } +} + /****************************************************************************** * Bloom filters ******************************************************************************/ @@ -3672,237 +3903,6 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) return success; } -/****************************************************************************** - * PID controller - ******************************************************************************/ - -/* - * A feedback loop based on Proportional-Integral-Derivative (PID) controller. - * - * The P term is refaulted/(evicted+protected) from a tier in the generation - * currently being evicted; the I term is the exponential moving average of the - * P term over the generations previously evicted, using the smoothing factor - * 1/2; the D term isn't supported. - * - * The setpoint (SP) is always the first tier of one type; the process variable - * (PV) is either any tier of the other type or any other tier of the same - * type. - * - * The error is the difference between the SP and the PV; the correction is to - * turn off protection when SP>PV or turn on protection when SPlrugen; - int hist = lru_hist_from_seq(lrugen->min_seq[type]); - - pos->refaulted = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - pos->total += lrugen->protected[hist][type][tier - 1]; - pos->gain = gain; -} - -static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) -{ - int hist, tier; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - bool clear = carryover ? NR_HIST_GENS == 1 : NR_HIST_GENS > 1; - unsigned long seq = carryover ? lrugen->min_seq[type] : lrugen->max_seq + 1; - - lockdep_assert_held(&lruvec->lru_lock); - - if (!carryover && !clear) - return; - - hist = lru_hist_from_seq(seq); - - for (tier = 0; tier < MAX_NR_TIERS; tier++) { - if (carryover) { - unsigned long sum; - - sum = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); - - sum = lrugen->avg_total[type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); - if (tier) - sum += lrugen->protected[hist][type][tier - 1]; - WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); - } - - if (clear) { - atomic_long_set(&lrugen->refaulted[hist][type][tier], 0); - atomic_long_set(&lrugen->evicted[hist][type][tier], 0); - if (tier) - WRITE_ONCE(lrugen->protected[hist][type][tier - 1], 0); - } - } -} - -static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) -{ - /* - * Return true if the PV has a limited number of refaults or a lower - * refaulted/total than the SP. - */ - return pv->refaulted < MIN_LRU_BATCH || - pv->refaulted * (sp->total + MIN_LRU_BATCH) * sp->gain <= - (sp->refaulted + 1) * pv->total * pv->gain; -} - -/****************************************************************************** - * the aging - ******************************************************************************/ - -/* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) -{ - int type = folio_is_file_lru(folio); - struct lru_gen_folio *lrugen = &lruvec->lrugen; - int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); - unsigned long new_flags, old_flags = READ_ONCE(folio->flags); - - VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); - - do { - new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; - /* folio_update_gen() has promoted this page? */ - if (new_gen >= 0 && new_gen != old_gen) - return new_gen; - - new_gen = (old_gen + 1) % MAX_NR_GENS; - - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); - new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF; - /* for folio_end_writeback() */ - if (reclaiming) - new_flags |= BIT(PG_reclaim); - } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - - lru_gen_update_size(lruvec, folio, old_gen, new_gen); - - return new_gen; -} - -static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) -{ - unsigned long pfn = pte_pfn(pte); - - VM_WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end); - - if (!pte_present(pte) || is_zero_pfn(pfn)) - return -1; - - if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) - return -1; - - if (WARN_ON_ONCE(!pfn_valid(pfn))) - return -1; - - return pfn; -} - -static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, - struct pglist_data *pgdat, bool can_swap) -{ - struct folio *folio; - - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) - return NULL; - - folio = pfn_folio(pfn); - if (folio_nid(folio) != pgdat->node_id) - return NULL; - - if (folio_memcg_rcu(folio) != memcg) - return NULL; - - /* file VMAs can contain anon pages from COW */ - if (!folio_is_file_lru(folio) && !can_swap) - return NULL; - - return folio; -} - -/* promote pages accessed through page tables */ -static int folio_update_gen(struct folio *folio, int gen) -{ - unsigned long new_flags, old_flags = READ_ONCE(folio->flags); - - VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); - VM_WARN_ON_ONCE(!rcu_read_lock_held()); - - do { - /* lru_gen_del_folio() has isolated this page? */ - if (!(old_flags & LRU_GEN_MASK)) { - /* for shrink_folio_list() */ - new_flags = old_flags | BIT(PG_referenced); - continue; - } - - new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAGS); - new_flags |= (gen + 1UL) << LRU_GEN_PGOFF; - } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - - return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; -} - -static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio, - int old_gen, int new_gen) -{ - int type = folio_is_file_lru(folio); - int zone = folio_zonenum(folio); - int delta = folio_nr_pages(folio); - - VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS); - VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS); - - walk->batched++; - - walk->nr_pages[old_gen][type][zone] -= delta; - walk->nr_pages[new_gen][type][zone] += delta; -} - -static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk) -{ - int gen, type, zone; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - - walk->batched = 0; - - for_each_gen_type_zone(gen, type, zone) { - enum lru_list lru = type * LRU_INACTIVE_FILE; - int delta = walk->nr_pages[gen][type][zone]; - - if (!delta) - continue; - - walk->nr_pages[gen][type][zone] = 0; - WRITE_ONCE(lrugen->nr_pages[gen][type][zone], - lrugen->nr_pages[gen][type][zone] + delta); - - if (lru_gen_is_active(lruvec, gen)) - lru += LRU_ACTIVE; - __update_lru_size(lruvec, lru, zone, delta); - } -} - static int should_skip_vma(unsigned long start, unsigned long end, struct mm_walk *args) { struct address_space *mapping; -- 2.41.0