From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84107C71136 for ; Thu, 12 Jun 2025 18:15:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 226646B00A0; Thu, 12 Jun 2025 14:15:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B0796B00A1; Thu, 12 Jun 2025 14:15:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09FC96B00A2; Thu, 12 Jun 2025 14:15:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DA6DA6B00A0 for ; Thu, 12 Jun 2025 14:15:25 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7C3E7141CB9 for ; Thu, 12 Jun 2025 18:15:25 +0000 (UTC) X-FDA: 83547551010.24.F0F9767 Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) by imf28.hostedemail.com (Postfix) with ESMTP id 946C2C000D for ; Thu, 12 Jun 2025 18:15:23 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ou8xnJCD; spf=pass (imf28.hostedemail.com: domain of bijan311@gmail.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749752123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LArn2/NqwWy0wVHKXQau9kT4miDpoQMz65raO0syifs=; b=E+DwWZhtBG/1VREL9gpLEUcjPGFW366r3ytHy2GlRddfsOAKiA3VY9AOCE6UQkac1A71tX b7H7wSoW5OjFuAtC4agP8QcGaFvwdIJY088WC4Fnp6mYttqFxIPSG2AT+oaMDGynrdZjYc +CjHc8gOCk3yOhPt0Qf7QIBBNuA5ph4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749752123; a=rsa-sha256; cv=none; b=h685yHt8JuCm+qAV8WjFQuE/HeNp775eF2Oqt1ZAv2EEfXACvDmPkFDzQ5AWZU27tw6IRU kok5xXghge6rF9AUONNCDWVYQxLo6E0BdHwARIp2/7mUia1C5uCVqTLW2Ub0GOsmFEUV+3 FDKhuYVsq0kde9+JMRK+r+G8GqbR+g4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ou8xnJCD; spf=pass (imf28.hostedemail.com: domain of bijan311@gmail.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-e81ec95d944so2120458276.1 for ; Thu, 12 Jun 2025 11:15:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749752122; x=1750356922; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LArn2/NqwWy0wVHKXQau9kT4miDpoQMz65raO0syifs=; b=Ou8xnJCDGBg93YswP47WJcFG1aPPPk3LB/Xw2kWxkyyfcklMovrrKJWAZxhab+TOwT ff1SuJk5RZSdO0oTVb13glISt6DhqX54YAuQ74SkV1yb4YY1fOyekd0rZV5Q52zApD4a 98STVuFtX+MJUWngBcdmpDdUIT3YoVtyXfQoG7JqDPkS2d2zQrOGKEkrHNvlmXEvzxLS T7QoeIAlv9oMcQe0wpW3p6tcRRkEIVMYgh6KaKy4IsfXUlzf7Kp8O4HNzRtBJ+/qmmnA q1hMX9ZYeGkjRPWVpO39PmsENQf/GAmwmXR8FPldx+ii54R+5zEYdB7/flT8k86c2eXG FxSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749752122; x=1750356922; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LArn2/NqwWy0wVHKXQau9kT4miDpoQMz65raO0syifs=; b=QTEKba5v1ErVhZXNAIW2mFwJ4yxyer0B7or2s2S6GM/YRxiysKiunmGZvqPMU2oz4q 0/jiZw1u9EvnfAxvly9baWVH4zqXuk3czsHS7iJUv4bOk+LVJxY/X+T0OsJTILYxS/lW xYxJ/MGFJjC9XASsSEixTnKQBXTaOEYkkLlJX8UoU5gL9mMw/z4YjVHLvVvOgTS5ARp5 /LoIc+n7a+/w27U5ywxLTyvXwbjf9/+934FyZA65hoUgwet1XHBjUo0pLJX6mgU8SBjD gsQG5nft5kKNlTTmf5w90zhC5BWUG+vOjwGFQlYD2WmEh0rIr2cO+adR3NzFgaY3YHku SROg== X-Forwarded-Encrypted: i=1; AJvYcCVgX5ekIzK/MKmo7NSSsfmL0tprjOGDKNShBu/9jBZ/S0QkUxZmzFN3ViiTX0+Gjbz5XjQGBMev/g==@kvack.org X-Gm-Message-State: AOJu0Yz65uk3RWf7hqT1ijeX8HMlqyntPxJcF30J4UVTRap7oF+WDzFh RG05cmoae+XgaGUuyMF8fBKCj68pNIMYm26dIm3j4MdgvjCJcD5sQNAd X-Gm-Gg: ASbGncvnqrZ49luDrIbhG9vOtomhpqqwyS79f97VgVq+PQwIrjFsznpw7jXOldA3oLw XrvqrODZvJ0l30pb+aiJRdxKuU9MyP9ySy9od1ODj/52pSn3kL8upyKsX7/+Wo0kstqzITKu9oB M+lukmkuwEM9MrDXCi1Rf+kVK71BEghwwCA+Jk95OzIbQyOIVX0FYHI8sIM8+eyysJE13QTcbr1 T3udgtg+M/fb5GpbTwrapoK6E6iT57ircnSOnAYuNsuAjVSFKmk4okFb2fkR62DNuAKny4GNC1/ VjNVAXJqWnjlN/jwKs7eqgPQY5WOpZjcb/0ybCw/rQLOllxfimHMbqVpxjbsosP0cD/0bmhPA1Q cDU7AXBI= X-Google-Smtp-Source: AGHT+IH8StR2oqDE+bGgZZWyYluYInSYI54x04NKJ85FqXoRUgJbr/xsBUEHZcj6UiB3fcMUvqy1wA== X-Received: by 2002:a05:6902:220c:b0:e81:b5c8:3d71 with SMTP id 3f1490d57ef6-e820baf86edmr5215637276.17.1749752122004; Thu, 12 Jun 2025 11:15:22 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:dd1b:d4ae:15de:11db]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e312452sm592480276.40.2025.06.12.11.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 11:15:21 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: [RFC PATCH 3/4] mm/damon: Move damon_pa_migrate_pages to ops-common Date: Thu, 12 Jun 2025 13:13:29 -0500 Message-ID: <20250612181330.31236-4-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: <20250612181330.31236-1-bijan311@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: 946C2C000D X-Stat-Signature: o6nj7upwyz1fh3qyedt7fuh5dx6em68h X-Rspamd-Server: rspam04 X-HE-Tag: 1749752123-163971 X-HE-Meta: U2FsdGVkX18wSJZbIUNwcghPHDJet30BAMVOalrrhMer5Bq7S9XhObOjp5VtwZFNH4+mgG1UJiQ/fEw0igHacdJ06NsD4kHZ0d2R2ejSKB+jm/VjujEtuWvYF1n9mZF4votlGUWTiuACpGbn0hXz/d29+yoH2RtSL21Bf7XFMag7648c7KC/rWmcQ/R2y4cNww9J0LCzOuybhAxQx8n6x2wRJd57OS0YWTJld/GQIxricNkNECBFoW8Mygn6fUT2QCZQLm2+yfgnAJC08yUw5zKsA8MS3hjOcMOVyON6S4JtUpgUpIq3YiIce/laHa5lSpIXv6hJms73aie/5d5wzhc5+wcFMzi83eeDwNDcafjVb1dXT17rxw3uzvpZAYuOK45LbT0BkdnENqHDzVfzpuRh5XpZysvAR9gDd3TirkK/Jku1bCrk/Xh4TH6VjcNk9QiaQ12EItdHqI0Yx/lW1aFRci+K5+p2N6Toj728ccBYV1im2bSTQBi230ih8fQRpgGbu7oUgh1jRWvABTkfoxLhcPZZsHBotsYY4ZK9DzUY8b81GSsH2AZmJSp83YYr2vINm8NXnP3/zacfsfIEKKyW9Tt/BwM9R1FbNXEaAvqqFAsiPaiGBGGbh4n4w5nO3NLlH7givF6E1UJQ+ZV5vqA2bQBUEnx/R1ZpxQoOHtZqE3yTUht6IZXMeDwEiGr7cYFPBT9ikzqYOLy55l6zLcyH/KE43e9cKrYIfgO8ILuwY52ftRVWbm3sPo//UR2XBm2McLRjAD67NWgMsroBGtEXjBJne9EDLlK8gtMpbfZR/+G7wp7VaKDT3PoOGsPOyaoUfkMMnW8Zn8JGrNBiPiwL2PF5jzOFWAPVpxU+0uX4CuFJmvbp16VwJVyN0Q+V2uVO2G2lNegXAPYKxGGwDSd2zK/2xY5YSeWwTxPFuGY6QlxKhfsb0w8sd3jmMOyQTT7SpmBui0rFsS+3Slw g7FAmm68 fj3/CpQOxLWeCFzcbXoAVVMnEyEXZZnFKI8PnRGDoi9nsQse6Y5YhQsNY2cUNH76N/dZXHqeqNcYcOl8b43RdfW25t59iU++4gAIp2zbxDTf1kKjqJufvDjGGXadcqzbq05aVN1gRzz3y0eZUThgIY8Lz7pQrqPbvk/ZbMVw+eNWmRkKeXSFG6e/rGaRk24vYMO0VbbFEXKS/HbLfaja5Viq6VVtooYeSpx8cBg5v5xV/dcOxn2/DFzRYHHyT7wvTi03OCHM/6k+b4FCzqliPCVHon3YT655c3oTalG+cLdis3+UtFqAVEQ45altHh4t9MINlH90Al4yqbmQXge5T8p87MMTQEPeZospJj+CpdysAhuTPIf8r1AmaY2kZR3EGWP/kcsrHPO3JP8H9bpWf0sLZo6pJzI9Z5EvxfmMjeOgxCaBVAF6NrkPu/Zmiv4UkegZcGTzbtKnx/j3RDX4liB9MA1fmyqJYvXQSweXcmLpfIB/OZ2w09Lwuhp5kOcgl5H/jA6Eqpd/VN8cc3lGJCAwFll4iBCaRNq05JHowglduQFwrjdW8W4dkF/bXDg7q+EDiq0K2RNxzEuJaXwv6ZmaD9isSmn/zlAg0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Bijan Tabatabai This patch moves damon_pa_migrate_pages from mm/damon/paddr.c to mm/damon/ops-common.c and renames it damon_migrate_pages. This gives vaddr based actions the ability to migrate pages as well. Signed-off-by: Bijan Tabatabai --- mm/damon/ops-common.c | 123 +++++++++++++++++++++++++++++++++++++++++ mm/damon/ops-common.h | 2 + mm/damon/paddr.c | 125 +----------------------------------------- 3 files changed, 127 insertions(+), 123 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index b43620fee6bb..2c4fb274b7f6 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -5,6 +5,7 @@ * Author: SeongJae Park */ +#include #include #include #include @@ -12,6 +13,7 @@ #include #include +#include "../internal.h" #include "ops-common.h" /* @@ -138,3 +140,124 @@ int damon_cold_score(struct damon_ctx *c, struct damon_region *r, /* Return coldness of the region */ return DAMOS_MAX_SCORE - hotness; } + +static unsigned int __damon_migrate_folio_list( + struct list_head *migrate_folios, struct pglist_data *pgdat, + int target_nid) +{ + unsigned int nr_succeeded = 0; + nodemask_t allowed_mask = NODE_MASK_NONE; + struct migration_target_control mtc = { + /* + * Allocate from 'node', or fail quickly and quietly. + * When this happens, 'page' will likely just be discarded + * instead of migrated. + */ + .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | + __GFP_NOWARN | __GFP_NOMEMALLOC | GFP_NOWAIT, + .nid = target_nid, + .nmask = &allowed_mask + }; + + if (pgdat->node_id == target_nid || target_nid == NUMA_NO_NODE) + return 0; + + if (list_empty(migrate_folios)) + return 0; + + /* Migration ignores all cpuset and mempolicy settings */ + migrate_pages(migrate_folios, alloc_migrate_folio, NULL, + (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON, + &nr_succeeded); + + return nr_succeeded; +} + +static unsigned int damon_migrate_folio_list(struct list_head *folio_list, + struct pglist_data *pgdat, + int target_nid) +{ + unsigned int nr_migrated = 0; + struct folio *folio; + LIST_HEAD(ret_folios); + LIST_HEAD(migrate_folios); + + while (!list_empty(folio_list)) { + struct folio *folio; + + cond_resched(); + + folio = lru_to_folio(folio_list); + list_del(&folio->lru); + + if (!folio_trylock(folio)) + goto keep; + + /* Relocate its contents to another node. */ + list_add(&folio->lru, &migrate_folios); + folio_unlock(folio); + continue; +keep: + list_add(&folio->lru, &ret_folios); + } + /* 'folio_list' is always empty here */ + + /* Migrate folios selected for migration */ + nr_migrated += __damon_migrate_folio_list( + &migrate_folios, pgdat, target_nid); + /* + * Folios that could not be migrated are still in @migrate_folios. Add + * those back on @folio_list + */ + if (!list_empty(&migrate_folios)) + list_splice_init(&migrate_folios, folio_list); + + try_to_unmap_flush(); + + list_splice(&ret_folios, folio_list); + + while (!list_empty(folio_list)) { + folio = lru_to_folio(folio_list); + list_del(&folio->lru); + folio_putback_lru(folio); + } + + return nr_migrated; +} + +unsigned long damon_migrate_pages(struct list_head *folio_list, + int target_nid) +{ + int nid; + unsigned long nr_migrated = 0; + LIST_HEAD(node_folio_list); + unsigned int noreclaim_flag; + + if (list_empty(folio_list)) + return nr_migrated; + + noreclaim_flag = memalloc_noreclaim_save(); + + nid = folio_nid(lru_to_folio(folio_list)); + do { + struct folio *folio = lru_to_folio(folio_list); + + if (nid == folio_nid(folio)) { + list_move(&folio->lru, &node_folio_list); + continue; + } + + nr_migrated += damon_migrate_folio_list(&node_folio_list, + NODE_DATA(nid), + target_nid); + nid = folio_nid(lru_to_folio(folio_list)); + } while (!list_empty(folio_list)); + + nr_migrated += damon_migrate_folio_list(&node_folio_list, + NODE_DATA(nid), + target_nid); + + memalloc_noreclaim_restore(noreclaim_flag); + + return nr_migrated; +} diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index cc9f5da9c012..54209a7e70e6 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -16,3 +16,5 @@ int damon_cold_score(struct damon_ctx *c, struct damon_region *r, struct damos *s); int damon_hot_score(struct damon_ctx *c, struct damon_region *r, struct damos *s); + +unsigned long damon_migrate_pages(struct list_head *folio_list, int target_nid); diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index e989464635cd..722d69f26e37 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -381,127 +381,6 @@ static unsigned long damon_pa_deactivate_pages(struct damon_region *r, sz_filter_passed); } -static unsigned int __damon_pa_migrate_folio_list( - struct list_head *migrate_folios, struct pglist_data *pgdat, - int target_nid) -{ - unsigned int nr_succeeded = 0; - nodemask_t allowed_mask = NODE_MASK_NONE; - struct migration_target_control mtc = { - /* - * Allocate from 'node', or fail quickly and quietly. - * When this happens, 'page' will likely just be discarded - * instead of migrated. - */ - .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | - __GFP_NOWARN | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, - .nmask = &allowed_mask - }; - - if (pgdat->node_id == target_nid || target_nid == NUMA_NO_NODE) - return 0; - - if (list_empty(migrate_folios)) - return 0; - - /* Migration ignores all cpuset and mempolicy settings */ - migrate_pages(migrate_folios, alloc_migrate_folio, NULL, - (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON, - &nr_succeeded); - - return nr_succeeded; -} - -static unsigned int damon_pa_migrate_folio_list(struct list_head *folio_list, - struct pglist_data *pgdat, - int target_nid) -{ - unsigned int nr_migrated = 0; - struct folio *folio; - LIST_HEAD(ret_folios); - LIST_HEAD(migrate_folios); - - while (!list_empty(folio_list)) { - struct folio *folio; - - cond_resched(); - - folio = lru_to_folio(folio_list); - list_del(&folio->lru); - - if (!folio_trylock(folio)) - goto keep; - - /* Relocate its contents to another node. */ - list_add(&folio->lru, &migrate_folios); - folio_unlock(folio); - continue; -keep: - list_add(&folio->lru, &ret_folios); - } - /* 'folio_list' is always empty here */ - - /* Migrate folios selected for migration */ - nr_migrated += __damon_pa_migrate_folio_list( - &migrate_folios, pgdat, target_nid); - /* - * Folios that could not be migrated are still in @migrate_folios. Add - * those back on @folio_list - */ - if (!list_empty(&migrate_folios)) - list_splice_init(&migrate_folios, folio_list); - - try_to_unmap_flush(); - - list_splice(&ret_folios, folio_list); - - while (!list_empty(folio_list)) { - folio = lru_to_folio(folio_list); - list_del(&folio->lru); - folio_putback_lru(folio); - } - - return nr_migrated; -} - -static unsigned long damon_pa_migrate_pages(struct list_head *folio_list, - int target_nid) -{ - int nid; - unsigned long nr_migrated = 0; - LIST_HEAD(node_folio_list); - unsigned int noreclaim_flag; - - if (list_empty(folio_list)) - return nr_migrated; - - noreclaim_flag = memalloc_noreclaim_save(); - - nid = folio_nid(lru_to_folio(folio_list)); - do { - struct folio *folio = lru_to_folio(folio_list); - - if (nid == folio_nid(folio)) { - list_move(&folio->lru, &node_folio_list); - continue; - } - - nr_migrated += damon_pa_migrate_folio_list(&node_folio_list, - NODE_DATA(nid), - target_nid); - nid = folio_nid(lru_to_folio(folio_list)); - } while (!list_empty(folio_list)); - - nr_migrated += damon_pa_migrate_folio_list(&node_folio_list, - NODE_DATA(nid), - target_nid); - - memalloc_noreclaim_restore(noreclaim_flag); - - return nr_migrated; -} - static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s, unsigned long *sz_filter_passed) { @@ -529,7 +408,7 @@ static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s, addr += folio_size(folio); folio_put(folio); } - applied = damon_pa_migrate_pages(&folio_list, s->target_nid); + applied = damon_migrate_pages(&folio_list, s->target_nid); cond_resched(); s->last_applied = folio; return applied * PAGE_SIZE; @@ -627,7 +506,7 @@ static unsigned long damon_pa_interleave(struct damon_region *r, struct damos *s applied = 0; for (int i = 0; i < nr_node_ids; i++) { - applied += damon_pa_migrate_pages(&priv.folio_migration_list[i], i); + applied += damon_migrate_pages(&priv.folio_migration_list[i], i); cond_resched(); } -- 2.43.5