From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3081B4446E2 for ; Tue, 25 Aug 2026 12:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787661974; cv=none; b=fexS5JoIvURTwxxN+TX1CzVAPKyrcdcI6jZM6jVNoNDV28G7JB1l/TOFBEINv8y3h3QlT5ZYyirMj4sDUr9LU2UhZ3Q5yF6sLX3Xsg9tccsgYX6iFyPw24ya/L4cmZclfcjpdFKGRBKrnILpSl30MbzyymwTQBxaLQWPfNy+ly8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787661974; c=relaxed/simple; bh=rtRN37WYI8riUWI7m4LZmoEcuADBH39oC6zAOHRif0E=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=D89y/XMDNO13UkxLQXYoutuuqa5OrU4VHd2BBKHtjeDsQADIewJupKHtk44c3yeMTHK6o6KyaLcAq7ETaylY8A9vr+tAyFvSxArmuSK7XFaKFML8gpqG2eV1SKtfgWdIUCd/1SKU4OqfuRkB0zGtCYTJ9/VcyrlZSuJjm8primc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=giHu3rN2; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="giHu3rN2" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-384930ca5e2so4692322a91.3 for ; Tue, 25 Aug 2026 05:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787661972; x=1788266772; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=a2Asa4hLurNMrCEVCTFHSXC6wwWf0Qto8+IsnkijKbg=; b=giHu3rN2ZOKIQm7di7L4BX0fkpmTXrIU4My39YDE9XU2PKLrQnO/atMLo4X0fgLiNT knTu7PBbnsW5DYPUEt908iM24Dc5/oee02s6V+wYe39wzPxiCA4/BXvkaGM9cPHfmn/w fwSx2kBE7/S65puhGKVdiHsUcbNRvzGPGQ7Fe5E/lyG1uLEEenVo1eVpj9CHFhMmd5Sv 5zMuWoEkmi7BdpU4XyUwrUGP/URAMbXH73ruTYJBkQFPKPvBX8ybrsCjnOOU/CPA/s7A hNSzV5Yuc/b+lRn6ekfAPGOSBQCOvw9W0V/eEyxd25hyc8DrPKE4KYMS8iwupdIUHEON W1YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787661972; x=1788266772; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=a2Asa4hLurNMrCEVCTFHSXC6wwWf0Qto8+IsnkijKbg=; b=QFKjrPcHhoOXmCndAIkhHnXCGzsZrB3s0GGK2VwPHMKIma/BgxzSmCUpKcAPV5PVlU BC8O5Hkg2quUZao75cfBG3uwJvdX6NWQ0yptqLTgsUrv04Mljs2SVfkvFSCo87KJBE58 uyD3Wqwyn0ZGKhN7IkzGcZYmUycZFW8N4hKbKdTCd3iYhEWwUnglskKnK1XjEAaCltKV d7INE0izB1wzBYJL43L2+4xG6ExgDhEAsEaiSr3dOszPu7xY71awhUHl+6eMQPyYiS/6 QaLe6/+dOYC5uIHA0xpJ1YU/CvOiNrt/GU7X+V6LYUmvIcXW7rxD34cjamINEia/haZ7 Sz1w== X-Forwarded-Encrypted: i=1; AHgh+RrvlaELuudJuPH+GFvSdhSjKZwU8nWhDgHmhQVG4H18yYN/y4cFPdJJroZyuk86AkxJDfxpKLLUK1QWdIc=@vger.kernel.org X-Gm-Message-State: AFuF++nzw5fUTSqld7LVEZIziOa66shD9hFVrHkNjGMkIrph1O6QDcu8 rnrx1z6di4aXJk2o00Oa8eADxoskIMq9ZttJST/1w3U56xG43vuYLeQ8 X-Gm-Gg: AR+sD10oRmTEg8tlzZr/AaG1tiu6zshOI7/cLMgP98hTTZCEr7kVN8VO8cZqUk2LjcT jCwhYVYDcfL8fmpMc4l0751ZmrQBYn72f2jeNNHSjtaRij5/5H/RvXwt4YAzr65WWq9u/2kKlv5 BKB+klIaXgfRAmH0bMFC2AJWe/TRySXZcxVwLa5tJd57qhC/tC45VqLuStp80alb9H2HweBZ5a7 ktwHrQMAD45k+plb4RxVc27jI9gNCWlZMV72ehm2i+LfkbSPlQNLj9zFaz3jd3ZqtsZaUuamH1Q o6jA7R88yFmfOJlLm7300WTMUiX6cID4LNv46s/MpuBG7w/6mXlh84LXCjBFgop2+Xi+3FNLuvu 6G1DopU9Hm3RK9IYkLJh3WpfL5q6BFTot2lemhPZCkyFGdUQ46HaPhoC9XUsHaDUvcKe+GR+xWE xDsEh1gRWJUqf2IJnU5kscGVk/Cwz1wWJKj6qWErIlJs3DBzYnx2ocyxAivvOmyV5eXmstQNSOm arIXwvkK3IXEsDArxV2wc/ldiY= X-Received: by 2002:a17:90b:3811:b0:393:288:29e3 with SMTP id 98e67ed59e1d1-395df24f20cmr45613753a91.10.1787661972204; Tue, 25 Aug 2026 05:46:12 -0700 (PDT) Received: from celestia.taila51cc2.ts.net ([2402:1980:c23:7fa2:94a9:2164:3642:a5a8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645c06e02sm3738983a91.17.2026.08.25.05.46.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 05:46:11 -0700 (PDT) From: Liew Rui Yan To: SJ Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Liew Rui Yan Subject: [RFC PATCH] mm/damon: fix damos quota walk-position tracking Date: Tue, 25 Aug 2026 20:46:16 +0800 Message-ID: <20260825124616.5129-1-aethernet65535@gmail.com> X-Mailer: git-send-email 2.55.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit DAMOS uses charge_target_from/charge_addr_from to remember how far a quota-limited walk has progressed. The current implementation has two problems: 1. Once set, the cursor unconditionally skips and resets at the last region of the tracked target, so the last region can be skipped even when it has not been processed. 2. The cursor only advances when the quota becomes full. Regions that are filtered out do not move the cursor, and the scheme can remain stuck on the same regions. Fix this by treating the fields as a rotating walk cursor: - Advance the cursor for every region walked while a quota is set, regardless of whether the region is applied or filtered out. - Remove the unconditional last-region reset. - Skip regions before the cursor, and rotate to the next valid target only after the current target has been fully walked. - Rename charge_{target,addr}_from to walk_{target,addr}_from and damos_skip_charged_region() to damos_skip_walked_region(). Signed-off-by: Liew Rui Yan --- include/linux/damon.h | 6 +- mm/damon/core.c | 156 +++++++++++++++++++++++++++++------------- 2 files changed, 111 insertions(+), 51 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 0c8b7ddef9ab..5e0755ebad75 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -296,8 +296,10 @@ struct damos_quota { /* For charging the quota */ unsigned long charged_sz; unsigned long charged_from; - struct damon_target *charge_target_from; - unsigned long charge_addr_from; + + /* For tracking the DAMOS walk position (rotating cursor) */ + struct damon_target *walk_target_from; + unsigned long walk_addr_from; /* For prioritization */ unsigned int min_score; diff --git a/mm/damon/core.c b/mm/damon/core.c index 644daf5a1656..bbdb530513a1 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -404,6 +404,52 @@ static bool damon_is_last_region(struct damon_region *r, return list_is_last(&r->list, &t->regions_list); } +static bool damos_completely_walked(struct damos *s, struct damon_target *t) +{ + /* + * '>=' instead of '==', because the region set of @t could have been + * shrunken by damon_set_regions() after the cursor was saved. + */ + return s->quota.walk_target_from == t && + s->quota.walk_addr_from >= damon_last_region(t)->ar.end; +} + +static struct damon_target *damon_next_valid_target(struct damon_target *t, + struct damon_ctx *c) +{ + struct list_head *head = &c->adaptive_targets; + struct damon_target *start = t; + struct damon_target *next = t; + + while (true) { + if (list_is_last(&next->list, head)) + next = list_first_entry(head, struct damon_target, list); + else + next = list_next_entry(next, list); + + if (!c->ops.target_valid || c->ops.target_valid(next)) + return next; + + if (next == start) + break; + } + return NULL; +} + +static void damos_replace_invalid_target(struct damon_ctx *c, + struct damon_target *t) +{ + struct damos *s; + + damon_for_each_scheme(s, c) { + if (s->quota.walk_target_from != t) + continue; + + s->quota.walk_target_from = damon_next_valid_target(t, c); + s->quota.walk_addr_from = 0; + } +} + /** * damon_probe_hits_wsum() - Returns probe hits weighted sum of a region. * @r: region to get the weighted sum of. @@ -673,8 +719,8 @@ static struct damos_quota *damos_quota_init(struct damos_quota *quota) quota->total_charged_ns = 0; quota->charged_sz = 0; quota->charged_from = 0; - quota->charge_target_from = NULL; - quota->charge_addr_from = 0; + quota->walk_target_from = NULL; + quota->walk_addr_from = 0; quota->esz_bp = 0; return quota; } @@ -1628,9 +1674,9 @@ static int damon_commit_targets( damon_destroy_target(dst_target, dst); damon_for_each_scheme(s, dst) { - if (s->quota.charge_target_from == dst_target) { - s->quota.charge_target_from = NULL; - s->quota.charge_addr_from = 0; + if (s->quota.walk_target_from == dst_target) { + s->quota.walk_target_from = NULL; + s->quota.walk_addr_from = 0; } } } @@ -2312,64 +2358,54 @@ static bool damos_valid_target(struct damon_ctx *c, struct damon_region *r, } /* - * damos_skip_charged_region() - Check if the given region or starting part of - * it is already charged for the DAMOS quota. + * damos_skip_walked_region() - Check if the given region or starting part of + * it is already walked for the DAMOS quota. * @t: The target of the region. - * @rp: The pointer to the region. + * @r: The pointer to the region. * @s: The scheme to be applied. * @min_region_sz: minimum region size. * - * If a quota of a scheme has exceeded in a quota charge window, the scheme's - * action would applied to only a part of the target access pattern fulfilling - * regions. To avoid applying the scheme action to only already applied - * regions, DAMON skips applying the scheme action to the regions that charged - * in the previous charge window. - * - * This function checks if a given region should be skipped or not for the - * reason. If only the starting part of the region has previously charged, - * this function splits the region into two so that the second one covers the - * area that not charged in the previous charge widnow, and return true. The - * caller can see the second one on the next iteration of the region walk. - * Note that this means the caller should use damon_for_each_region() instead - * of damon_for_each_region_safe(). If damon_for_each_region_safe() is used, - * the second region will just be ignored. + * When a quota is configured, DAMON records how far it has walked so that + * subsequent quota windows continue from that point instead of re-applying the + * same regions. This function returns true for regions that are before the + * recorded cursor and therefore should be skipped. If only the starting part + * has been walked, the region is split so that the remaining part can be + * visited. * * Return: true if the region should be skipped, false otherwise. */ -static bool damos_skip_charged_region(struct damon_target *t, +static bool damos_skip_walked_region(struct damon_target *t, struct damon_region *r, struct damos *s, unsigned long min_region_sz) { struct damos_quota *quota = &s->quota; unsigned long sz_to_skip; - /* Skip previously charged regions */ - if (quota->charge_target_from) { - if (t != quota->charge_target_from) - return true; - if (r == damon_last_region(t)) { - quota->charge_target_from = NULL; - quota->charge_addr_from = 0; + if (!damos_quota_is_set(quota)) + return false; + + /* Skip previously walked regions */ + if (quota->walk_target_from) { + if (t != quota->walk_target_from) return true; - } - if (quota->charge_addr_from && - r->ar.end <= quota->charge_addr_from) + if (quota->walk_addr_from && + r->ar.end <= quota->walk_addr_from) return true; - if (quota->charge_addr_from && r->ar.start < - quota->charge_addr_from) { - sz_to_skip = ALIGN_DOWN(quota->charge_addr_from - + if (quota->walk_addr_from && r->ar.start < + quota->walk_addr_from) { + sz_to_skip = ALIGN_DOWN(quota->walk_addr_from - r->ar.start, min_region_sz); if (!sz_to_skip) { - if (damon_sz_region(r) <= min_region_sz) + if (damon_sz_region(r) <= min_region_sz) { + quota->walk_addr_from = r->ar.end; return true; + } sz_to_skip = min_region_sz; } damon_split_region_at(t, r, sz_to_skip); return true; } - quota->charge_target_from = NULL; - quota->charge_addr_from = 0; } return false; } @@ -2627,10 +2663,6 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, quota->total_charged_ns += timespec64_to_ns(&end) - timespec64_to_ns(&begin); damos_charge_quota(quota, sz, sz_applied); - if (damos_quota_is_full(quota, c->min_region_sz)) { - quota->charge_target_from = t; - quota->charge_addr_from = r->ar.end; - } } if (s->action != DAMOS_STAT) r->age = 0; @@ -2639,8 +2671,19 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, damos_update_stat(s, sz, sz_applied, sz_ops_filter_passed); } -static void damon_do_apply_schemes(struct damon_ctx *c, - struct damon_target *t, +static void damos_walk_maybe_rotate(struct damon_ctx *c, + struct damon_target *t, + struct damos *s) +{ + struct damos_quota *quota = &s->quota; + + if (damos_completely_walked(s, t)) { + quota->walk_target_from = damon_next_valid_target(t, c); + quota->walk_addr_from = 0; + } +} + +static void damon_do_apply_schemes(struct damon_ctx *c, struct damon_target *t, struct damon_region *r) { struct damos *s; @@ -2658,18 +2701,31 @@ static void damon_do_apply_schemes(struct damon_ctx *c, if (damos_quota_is_full(quota, c->min_region_sz)) continue; - if (damos_skip_charged_region(t, r, s, c->min_region_sz)) - continue; - if (s->max_nr_snapshots && s->max_nr_snapshots <= s->stat.nr_snapshots) continue; + /* + * Skip regions before the cursor. If the skip makes the cursor + * reach the end of this target, rotate to the next valid target + * before continuing with the next region/scheme. + */ + if (damos_skip_walked_region(t, r, s, c->min_region_sz)) { + damos_walk_maybe_rotate(c, t, s); + continue; + } + if (damos_valid_target(c, r, s)) damos_apply_scheme(c, t, r, s); if (damon_is_last_region(r, t)) s->stat.nr_snapshots++; + + if (damos_quota_is_set(quota)) { + quota->walk_target_from = t; + quota->walk_addr_from = r->ar.end; + damos_walk_maybe_rotate(c, t, s); + } } } @@ -3241,8 +3297,10 @@ static void kdamond_apply_schemes(struct damon_ctx *c) max_region_sz = damon_region_sz_limit(c); mutex_lock(&c->walk_control_lock); damon_for_each_target(t, c) { - if (c->ops.target_valid && c->ops.target_valid(t) == false) + if (c->ops.target_valid && c->ops.target_valid(t) == false) { + damos_replace_invalid_target(c, t); continue; + } damos_apply_target(c, t, max_region_sz); } -- 2.55.0