From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 867D23612ED for ; Sun, 10 May 2026 07:22:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778397722; cv=none; b=mqMzUzi6w+QANkTLRLIgqms4nfN46JfVlvbeDNbr+hbEdfA3AflTVCqJ6grqdM+cdjeM29KnwWPMDexJAfN7xyi7JZhb4YX7fIOzEyxEzUEjsNQUvXBR3rwya15j2PsiBIn16zf0/sIuKuD1hxMVQSGDYMntqASIqw90XUKkwQI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778397722; c=relaxed/simple; bh=R15AqCaqkXwXuCC1/m6wMn/5W6CvsTb7P+cJRVHmHy4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aXBLy4QG0Z5CUNskjXebE+hXJuF84cFsP7K73ugzlP0lnwyq8dcR/cPRYgbidP0gn7Upd/q7pBfYzXe+EZAQOPw3l5mc9ASpHK/FfV/0nwhIUPbChvUOAj1YBQXOBzjYTzt93E2OC7lDv6DnRjo8Xld7qkN7A7hq2jT2XJVUvpc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--joonwonkang.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AHZbTkSK; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--joonwonkang.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AHZbTkSK" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-c828b1b7fddso39917a12.3 for ; Sun, 10 May 2026 00:22:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1778397720; x=1779002520; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o2dqdJSqZ2YzUGjO9Q4/Pde2a4UxV3zxrYkCuhRqOOM=; b=AHZbTkSK+YmE/ft/GOcJFz2T8YwsllLftiA2m8oNobDLFhMY4OtL3evfADMe2iQkWk OC172DOWKrl4jz38BdRoBZLIu8ZB872l1M8XjfS9RNZ7H/vJaEPYc3eKQM6hdVFOUx8e w1ZlXw1AswkH6dkKbQ/r55AhA9/3S+E9NHjEo40lHbI4WlV0SNGUbJEcyU9bVGRryedT uGLSplWRoCYa5Q6JbuifxdSb16W/xdNeZxrS/HziCV5DBHY8pMmkWimFtX0Fa4huH/Pw VRFaPmutcZkS0MyNbQmypGdpYNMftpH9WuFACQxqf8hwpvPaaTG47nazS62vGsL2QL5n poHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778397720; x=1779002520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o2dqdJSqZ2YzUGjO9Q4/Pde2a4UxV3zxrYkCuhRqOOM=; b=AcvlJ3FuHfLUnRTlQLtgXnv/a4X0mtVajnOb1djPyDNXbGaGcA4SQjGvPORkL/Bn/A dhVnsaXSLgqD8u9XE5p6bomlGXlwbQQQPF1mtGn6+gpGW1RsyYsBPoHPNnsDyjfgVvCs LiAiR7ZC8YQWCgPJYgMp88nrRE0K8cjGBzaJr4kRi05r9aQ/4URy63cbWEo+NPy4MSIe zra6tzJTeGgJD3q9Vlq0YXpcqMpu9tnVA8oiudstfOo0MzmllhGmVGo0ngNH156vkEhQ cTDyCXy6vZfbgUhsdu7pyRaFNP/K4tOsoEA5LQ0Qf4Re+n9jaXModvTzl+4WH2LqU+6n qv7g== X-Forwarded-Encrypted: i=1; AFNElJ++v+DZ5bIKplmXGQXUMtwPH2iyW9Pdo1ifQian/2/+8r+iVvn6oSyGLofK0sE3G3V3kdow63TdPdo4uEA=@vger.kernel.org X-Gm-Message-State: AOJu0Yxbspw7XMem5oWrztQOMCLkEM4bZ1Pr4R7zAKFUI1M6e7slrhaO X3hRkChlHS7ODfVrz6vOtzPAGE/yIfydF5cEMACBI9rArW+8G2TWvHq/5bFiDJH1J1He7jzU57w jjs9WqUqCk8xbNMtPNYlbLAdKXA== X-Received: from pgjw1.prod.google.com ([2002:a63:fb41:0:b0:c80:15fa:8bd5]) (user=joonwonkang job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:3c89:b0:38d:fad1:cf2a with SMTP id adf61e73a8af0-3aa5a95239dmr18835785637.13.1778397719609; Sun, 10 May 2026 00:21:59 -0700 (PDT) Date: Sun, 10 May 2026 07:21:49 +0000 In-Reply-To: <20260510072149.1279887-1-joonwonkang@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260510072149.1279887-1-joonwonkang@google.com> X-Mailer: git-send-email 2.54.0.563.g4f69b47b94-goog Message-ID: <20260510072149.1279887-4-joonwonkang@google.com> Subject: [PATCH v5 4/4] percpu: Fix hint invariant breakage From: Joonwon Kang To: dennis@kernel.org, tj@kernel.org, cl@gentwo.org Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dodam@google.com, joonwonkang@google.com Content-Type: text/plain; charset="UTF-8" The invariant "scan_hint_start > contig_hint_start if and only if scan_hint == contig_hint" should be kept for hint management. However, it could be broken in some cases: - if (new contig == contig_hint == scan_hint) && (contig_hint_start < scan_hint_start < new contig start) && the new contig is to become a new contig_hint due to its better alignment, then scan_hint should be invalidated instead of keeping the old value. - if (new contig == contig_hint > scan_hint) && (new contig start < contig_hint_start) && the new contig is not to become a new contig_hint, then scan_hint should be not updated to the new contig. This commit mainly fixes this invariant breakage and includes more: - Handle the cases where the new contig overlaps with the contig_hint or with scan_hint. - Merge the new contig with other hints when it overlaps with them and treat it as a whole free region instead of a separate small region. - Fix the invariant breakage and also optimizes scan_hint further. Some of the optimization cases when no overlap occurs are: - if (new contig > contig_hint > scan_hint) && (scan_hint_start < new contig start < contig_hint_start), then keep scan_hint instead of invalidating it. - if (new contig > contig_hint == scan_hint) && (contig_hint_start < new contig start < scan_hint_start), then update scan_hint to the old contig_hint instead of invalidating it. - if (new contig == contig_hint > scan_hint) && (new contig start < contig_hint_start) && the new contig is to become a new contig_hint due to its better alignment, then update scan_hint to the old contig_hint instead of invalidating or keeping it. Signed-off-by: Joonwon Kang --- v5: No change. v4: Refactor code by removing the scan_hint candidates and handle the overlapping cases where the new contig meets with the hints on the border. v3: Merge the new contig with other hints when it overlaps with them and treat it as a whole free region instead of a separate small region. v2: Consider the cases where the new contig overlaps with the existing contig_hint or scan_hint. Introduce the scan_hint candidates to calculate new scan_hint. v1: Initial version. mm/percpu.c | 119 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 94 insertions(+), 25 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 89b7f33500cf..abc36391e749 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -616,6 +616,20 @@ static inline bool pcpu_region_overlap(struct pcpu_region a, return (a.start < b.start + b.size) && (b.start < a.start + a.size); } +/* + * pcpu_region_concat - determines if two regions meet on the border + * @a: first region + * @b: second region + * + * This is used to determine if the hint region [a.start, a.start + a.size) + * meets with the allocated region [b.start, b.start + b.size) on the border. + */ +static inline bool pcpu_region_concat(struct pcpu_region a, + struct pcpu_region b) +{ + return (a.start == b.start + b.size) || (b.start == a.start + a.size); +} + /** * pcpu_block_update - updates a block given a free area * @block: block of interest @@ -629,6 +643,40 @@ static inline bool pcpu_region_overlap(struct pcpu_region a, static void pcpu_block_update(struct pcpu_block_md *block, int start, int end) { struct pcpu_region free = { .start = start, .size = end - start }; + bool overlap_with_contig_hint = + block->contig_hint.size && + (pcpu_region_overlap(block->contig_hint, free) || + pcpu_region_concat(block->contig_hint, free)); + + if (block->scan_hint.size && + (pcpu_region_overlap(block->scan_hint, free) || + pcpu_region_concat(block->scan_hint, free))) { + start = min(start, block->scan_hint.start); + end = max(end, block->scan_hint.start + block->scan_hint.size); + free = (struct pcpu_region){ + .start = start, + .size = end - start, + }; + + block->scan_hint.size = 0; + } + + if (overlap_with_contig_hint) { + start = min(start, block->contig_hint.start); + end = max(end, + block->contig_hint.start + block->contig_hint.size); + free = (struct pcpu_region){ + .start = start, + .size = end - start, + }; + + if (block->scan_hint.size && + free.size > block->scan_hint.size && + block->scan_hint.start > free.start) + block->scan_hint.size = 0; + + block->contig_hint = free; + } block->first_free = min(block->first_free, free.start); if (free.start == 0) @@ -637,23 +685,24 @@ static void pcpu_block_update(struct pcpu_block_md *block, int start, int end) if (free.start + free.size == block->nr_bits) block->right_free = free.size; + if (overlap_with_contig_hint) + return; + + /* + * At this point, it is guaranteed that the new contig does neither + * overlap with contig_hint nor with scan_hint. + */ + if (free.size > block->contig_hint.size) { /* promote the old contig_hint to be the new scan_hint */ if (block->contig_hint.size && free.start > block->contig_hint.start) { - if (block->contig_hint.size > block->scan_hint.size) { + if (block->contig_hint.size > block->scan_hint.size || + free.start < block->scan_hint.start) block->scan_hint = block->contig_hint; - } else if (block->scan_hint.size && - free.start < block->scan_hint.start) { - /* - * The old contig_hint.size == scan_hint.size. - * But, the new contig is larger so hold the - * invariant scan_hint.start < - * contig_hint.start. - */ - block->scan_hint.size = 0; - } - } else { + } else if (!block->contig_hint.size || + (block->scan_hint.size && + free.start < block->scan_hint.start)) { block->scan_hint.size = 0; } block->contig_hint = free; @@ -661,21 +710,41 @@ static void pcpu_block_update(struct pcpu_block_md *block, int start, int end) if (block->contig_hint.start && (!free.start || __ffs(free.start) > __ffs(block->contig_hint.start))) { + if (block->contig_hint.size > block->scan_hint.size) { + if (free.start < block->contig_hint.start) + block->scan_hint = block->contig_hint; + } else if (free.start > block->scan_hint.start) { + /* + * old contig_hint.size == old scan_hint.size + * == new contig size. But, the new contig is + * farther than the old scan_hint so hold the + * invariant scan_hint.start > contig_hint.start + * iff scan_hint.size == contig_hint.size. + */ + block->scan_hint.size = 0; + } + /* new start has a better alignment so use it */ block->contig_hint.start = free.start; - if (block->scan_hint.size && - free.start < block->scan_hint.start && - block->contig_hint.size > block->scan_hint.size) - block->scan_hint.size = 0; - } else if ((block->scan_hint.size && - free.start > block->scan_hint.start) || - block->contig_hint.size > block->scan_hint.size) { - /* - * Knowing new contig size == contig_hint.size, update - * the scan_hint if it is farther than or larger than - * the current scan_hint. - */ - block->scan_hint = free; + } else { + if (block->contig_hint.size > block->scan_hint.size) { + if (free.start < block->contig_hint.start) { + /* + * old scan_hint.size < new contig size + * == old contig_hint.size. But, the new + * contig is before the old contig_hint + * so hold the invariant + * scan_hint.start > contig_hint.start + * iff scan_hint.size == + * contig_hint.size. + */ + block->scan_hint.size = 0; + } else { + block->scan_hint = free; + } + } else if (free.start > block->scan_hint.start) { + block->scan_hint = free; + } } } else { /* -- 2.54.0.563.g4f69b47b94-goog