From: Joonwon Kang <joonwonkang@google.com>
To: dennis@kernel.org, tj@kernel.org, cl@gentwo.org
Cc: akpm@linux-foundation.org, linux-mm@kvack.org,
linux-kernel@vger.kernel.org,
Joonwon Kang <joonwonkang@google.com>
Subject: [PATCH] percpu: Fix hint invariant breakage
Date: Fri, 20 Mar 2026 11:52:14 +0000 [thread overview]
Message-ID: <20260320115214.143933-1-joonwonkang@google.com> (raw)
The invariant "scan_hint_start > contig_hint_start if and only if
scan_hint == contig_hint" should be kept for hint management. However,
it could be broken in some cases:
- if (new contig == contig_hint == scan_hint) && (contig_hint_start <
scan_hint_start < new contig start) && the new contig is to become a
new contig_hint due to its better alignment, then scan_hint should
be invalidated instead of keeping it.
- if (new contig == contig_hint > scan_hint) && (start <
contig_hint_start) && the new contig is not to become a new
contig_hint, then scan_hint should be invalidated instead of being
updated to the new contig.
This commit fixes this invariant breakage and also optimizes scan_hint
by keeping it or updating it when acceptable:
- if (new contig > contig_hint > scan_hint) && (scan_hint_start < new
contig start < contig_hint_start), then keep scan_hint instead of
invalidating it.
- if (new contig > contig_hint == scan_hint) && (contig_hint_start <
new contig start < scan_hint_start), then update scan_hint to the
old contig_hint instead of invalidating it.
- if (new contig == contig_hint > scan_hint) && (new contig start <
contig_hint_start) && the new contig is to become a new contig_hint
due to its better alignment, then update scan_hint to the old
contig_hint instead of invalidating or keeping it.
Signed-off-by: Joonwon Kang <joonwonkang@google.com>
---
mm/percpu.c | 60 ++++++++++++++++++++++++++++++++++-------------------
1 file changed, 39 insertions(+), 21 deletions(-)
diff --git a/mm/percpu.c b/mm/percpu.c
index 81462ce5866e..a0e4f8acb7c2 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -641,19 +641,13 @@ static void pcpu_block_update(struct pcpu_block_md *block, int start, int end)
if (contig > block->contig_hint) {
/* promote the old contig_hint to be the new scan_hint */
if (start > block->contig_hint_start) {
- if (block->contig_hint > block->scan_hint) {
+ if (block->contig_hint > block->scan_hint ||
+ start < block->scan_hint_start) {
block->scan_hint_start =
block->contig_hint_start;
block->scan_hint = block->contig_hint;
- } else if (start < block->scan_hint_start) {
- /*
- * The old contig_hint == scan_hint. But, the
- * new contig is larger so hold the invariant
- * scan_hint_start < contig_hint_start.
- */
- block->scan_hint = 0;
}
- } else {
+ } else if (start < block->scan_hint_start) {
block->scan_hint = 0;
}
block->contig_hint_start = start;
@@ -662,20 +656,44 @@ static void pcpu_block_update(struct pcpu_block_md *block, int start, int end)
if (block->contig_hint_start &&
(!start ||
__ffs(start) > __ffs(block->contig_hint_start))) {
+ if (block->contig_hint > block->scan_hint) {
+ if (start < block->contig_hint_start) {
+ block->scan_hint = block->contig_hint;
+ block->scan_hint_start = block->contig_hint_start;
+ }
+ } else if (start > block->scan_hint_start) {
+ /*
+ * old contig_hint == old scan_hint == contig.
+ * But, the new contig is farther than the old
+ * scan_hint so hold the invariant
+ * scan_hint_start > contig_hint_start iff
+ * scan_hint == contig_hint.
+ */
+ block->scan_hint = 0;
+ }
+
/* start has a better alignment so use it */
block->contig_hint_start = start;
- if (start < block->scan_hint_start &&
- block->contig_hint > block->scan_hint)
- block->scan_hint = 0;
- } else if (start > block->scan_hint_start ||
- block->contig_hint > block->scan_hint) {
- /*
- * Knowing contig == contig_hint, update the scan_hint
- * if it is farther than or larger than the current
- * scan_hint.
- */
- block->scan_hint_start = start;
- block->scan_hint = contig;
+ } else {
+ if (block->contig_hint > block->scan_hint) {
+ if (start < block->contig_hint_start) {
+ /*
+ * old scan_hint < contig == old
+ * contig_hint. But, the new contig is
+ * before the old contig_hint so hold
+ * the invariant
+ * scan_hint_start > contig_hint_start
+ * iff scan_hint == contig_hint.
+ */
+ block->scan_hint = 0;
+ } else {
+ block->scan_hint_start = start;
+ block->scan_hint = contig;
+ }
+ } else if (start > block->scan_hint_start) {
+ block->scan_hint_start = start;
+ block->scan_hint = contig;
+ }
}
} else {
/*
--
2.53.0.1018.g2bb0e51243-goog
next reply other threads:[~2026-03-20 11:52 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-20 11:52 Joonwon Kang [this message]
2026-03-20 19:08 ` [PATCH] percpu: Fix hint invariant breakage Andrew Morton
2026-03-23 12:02 ` Joonwon Kang
2026-03-21 17:09 ` Dennis Zhou
2026-03-23 14:05 ` Joonwon Kang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260320115214.143933-1-joonwonkang@google.com \
--to=joonwonkang@google.com \
--cc=akpm@linux-foundation.org \
--cc=cl@gentwo.org \
--cc=dennis@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox