From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B2AF3A783C for ; Thu, 30 Apr 2026 20:22:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777580582; cv=none; b=VvY5gVlFDqeyuajOiOEyWWolBco7UQ96/cv9BUaajpNJXdMJJSplYK+gezlWCe/fj+09qOv/4f5N67ichggxv5zukefjkQ6xsN4mSrpqfN5unJZeXHLL5LgujWry8dYwIRzgwEL6p4uu+qFty/Z/Q+4oMB3akvC9wrH5Z33tTjc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777580582; c=relaxed/simple; bh=K1EhICStYc3felwejYX0BiH4DPQPTCI7MtoFZWhHCdo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XokNYqIh7+5IX+xfbacCoQyYY0fsolV3f+iC5DpzjKqpDBp3kiEp+IAqQe+DdAaM5auAUSHQoFVVixeDxjk3mT2npZmm8ZbBqDx1zkWJeZUwjXVq3gutITts9MTr+79J6UACDfbS9Gw8Tv6ndEtmlqoMpqpQ2wJxm7GGWgY5d6U= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=cUQ1szzO; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="cUQ1szzO" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=g4Zye7STkkMo2HD39nW35YeTAwT+eEQUHaKLcLVr+Lo=; b=cUQ1szzOqb+QTeaExA6M9yn2gT RRRzI/geg9es5l6ZpHo/AlPEIEut0FqTBD8dzmJCLg9LCYtEnWj9muHbUOY9eycfU16B2x2AZQiXF 3eKW8+bfZXcgC+psAc5SQp4kE41t2HdiTOPoLmQO10Ptmv7mT5oMgttl/+Djj5XULxfoEK5i2CWfP o84BGYZBipH4Cw8VBwxbWCVUba2YReTdnKQBsS1ba5jru6c2v2zcjTnVNv5IRro/e3gGrDKThM0bt Djtzx6zhPeUSNauFg0Y8hfpy5MuB0eTd8O+I1JsOuZQLDwDLCDaCRoytuAcX2zyV31ETRx87ZrW/x FuTb2r3Q==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wIXuC-000000001R0-40L2; Thu, 30 Apr 2026 16:22:40 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: kernel-team@meta.com, linux-mm@kvack.org, david@kernel.org, willy@infradead.org, surenb@google.com, hannes@cmpxchg.org, ljs@kernel.org, ziy@nvidia.com, usama.arif@linux.dev, Rik van Riel , Rik van Riel Subject: [RFC PATCH 22/45] mm: page_alloc: adopt partial pageblocks from tainted superpageblocks Date: Thu, 30 Apr 2026 16:20:51 -0400 Message-ID: <20260430202233.111010-23-riel@surriel.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260430202233.111010-1-riel@surriel.com> References: <20260430202233.111010-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: Rik van Riel Add Phase 2 to rmqueue_bulk: when refilling PCP for unmovable or reclaimable allocations, search tainted superpageblocks for partially-free pageblocks with sub-pageblock buddy entries of the requested migratetype. Claim ownership of the pageblock and move the found entry to PCP with PCPBuddy marking. Pass 0 (the existing owned-block recovery phase) picks up remaining buddy entries on subsequent refills, so there is no need to sweep the entire pageblock eagerly. This concentrates non-movable allocations into already-tainted superpageblocks, reducing fragmentation spread to clean superpageblocks. Before claiming ownership, verify the pageblock is not already owned by another CPU (pbd->cpu == 0). Without this check, two CPUs could have PCPBuddy pages from the same pageblock on separate PCP lists protected by different locks, and the PCP merge pass could corrupt the other CPU's list. Signed-off-by: Rik van Riel Assisted-by: Claude:claude-opus-4.7 syzkaller --- mm/page_alloc.c | 114 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 13 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8f925b5a2e5f..4f8105b89e47 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1130,7 +1130,7 @@ static inline void set_buddy_order(struct page *page, unsigned int order) * - Set when Phase 0/1 restore or acquire whole pageblocks. * - Propagated to split remainders in pcp_rmqueue_smallest(). * - Set on freed pages from owned blocks routed to the owner PCP. - * - NOT set for Phase 2/3 fragments or zone-owned frees. + * - NOT set for Phase 3 fragments or zone-owned frees. * - The merge pass in free_pcppages_bulk() only processes * PagePCPBuddy pages, ensuring it never touches pages on * another CPU's PCP list. @@ -3840,15 +3840,15 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype, * under a single hold of the lock, for efficiency. Add them to the * freelist of @pcp. * - * When @pcp is non-NULL and @count > 1 (normal pageset), uses a four-phase + * When @pcp is non-NULL and @count > 1 (normal pageset), uses a multi-phase * approach: - * Phase 0: Recover previously owned, partially drained blocks. - * Phase 1: Acquire whole pageblocks, claim ownership, set PagePCPBuddy. - * These pages are eligible for PCP-level buddy merging. - * Phase 2: Grab sub-pageblock fragments of the same migratetype. - * Phase 3: Fall back to __rmqueue() with migratetype fallback. - * Phase 2/3 pages are cached for batching only -- no ownership claim, - * no PagePCPBuddy, no PCP-level merging. + * Phase 0: Recover previously owned, partially drained blocks. + * Phase 1: Acquire whole pageblocks, claim ownership, set PagePCPBuddy. + * These pages are eligible for PCP-level buddy merging. + * Phase 2: Adopt partial pageblocks from tainted SPBs (non-movable only). + * Claims ownership so Pass 0 can recover buddy entries later. + * Phase 3: Fall back to __rmqueue() with migratetype fallback. + * No ownership claim, no PagePCPBuddy, no PCP-level merging. * * When @pcp is NULL or @count <= 1 (boot pageset), acquires individual * pages of the requested order directly. @@ -3976,11 +3976,99 @@ static bool rmqueue_bulk(struct zone *zone, unsigned int order, goto out; /* - * Phase 2 was removed: it swept zone free lists for sub-pageblock - * fragments, which are always empty when superpageblocks are enabled. - * Phase 3's __rmqueue() -> __rmqueue_smallest() properly searches - * per-superpageblock free lists at all orders. + * Phase 2: Adopt partial pageblocks from tainted SPBs. + * + * Phase 1 only grabs whole free pageblocks. When a tainted SPB + * has partially-used pageblocks with free sub-pageblock buddy + * entries, Phase 1 can't use them. Phase 3 can find them via + * __rmqueue_smallest, but without ownership or PCPBuddy marking, + * so they fragment further on drain. + * + * This phase bridges the gap: find a sub-pageblock free entry + * in a tainted SPB and claim ownership of its pageblock. Pass 0 + * will pick up remaining buddy entries on subsequent refills. + * + * Only for unmovable/reclaimable — movable should use clean SPBs. */ + if (migratetype != MIGRATE_MOVABLE && + !is_migrate_cma(migratetype)) { + enum sb_fullness full; + + for (full = SB_FULL; full < __NR_SB_FULLNESS; full++) { + struct superpageblock *sb; + + list_for_each_entry(sb, + &zone->spb_lists[SB_TAINTED][full], list) { + struct page *page; + int found_order = -1; + + if (sb->nr_free_pages < pageblock_nr_pages / 4) + continue; + + /* + * Find a sub-pageblock free entry for our + * migratetype, starting from the largest order. + */ + for (o = pageblock_order - 1; o >= order; o--) { + struct free_area *area; + + area = &sb->free_area[o]; + page = get_page_from_free_area( + area, migratetype); + if (page) { + found_order = o; + break; + } + } + if (found_order < 0) + continue; + + /* + * Check that this pageblock isn't already + * owned by another CPU. If it is, two CPUs + * would have PCPBuddy pages from the same + * pageblock, and the PCP merge pass could + * corrupt the other CPU's PCP list. + */ + pbd = pfn_to_pageblock(page, + page_to_pfn(page)); + if (pbd->cpu != 0) + continue; + + /* + * Found a free chunk in an unowned pageblock. + * Take it from buddy, claim ownership, and + * set PCPBuddy. Pass 0 will grab remaining + * buddy entries on future refills. + * + * Set PB_has_ since we bypass + * page_del_and_expand (which normally does + * PB_has tracking). + */ + del_page_from_free_list(page, zone, + found_order, + migratetype); + __spb_set_has_type(page, migratetype); + set_pcpblock_owner(page, cpu); + __SetPagePCPBuddy(page); + pcp_enqueue_tail(pcp, page, migratetype, + found_order); + refilled += 1 << found_order; + + /* + * Register for Phase 0 recovery so future + * drains from this pageblock can be swept + * back efficiently. + */ + if (list_empty(&pbd->cpu_node)) + list_add(&pbd->cpu_node, + &pcp->owned_blocks); + + if (refilled >= pages_needed) + goto out; + } + } + } /* * Phase 3: Last resort. Use __rmqueue() which does -- 2.52.0