[RFC PATCH 31/45] mm: page_alloc: cross-non-movable buddy borrow within tainted SPBs

Rik van Riel
2026-04-30T20:22:55Z

When pages get freed via __free_one_page, they're placed on the per-SPB
free_list determined by their pageblock's migratetype, not the original
allocation's migratetype. Slab-heavy workloads expose a structural
mismatch:

- RECLAIMABLE pageblocks fill up densely with live slab objects (e.g.
btrfs_inode caches), leaving very few sub-pageblock free fragments
on the RECL free list.

- UNMOVABLE pageblocks accumulate sparse free space from vmalloc and
raw-alloc churn — tens of thousands of free pages, all on the UNMOV
free list.

Net effect: a tainted SPB can show 87,000+ free pages in metadata while
having ZERO free buddies on the RECL list. A new RECL allocation walking
__rmqueue_smallest's preferred-SB Pass 1 finds nothing, falls through
Pass 2 (claim_whole_block on MOVABLE — but mov=0 in tainted SBs),
Pass 2b (sub-PB MOVABLE — same), and reaches Pass 3, which taints a
fresh clean SPB. Repeat per RECL burst.

Add a Pass 2c between 2b and 3: for non-movable allocations that
couldn't find their own migratetype, try borrowing a sub-pageblock buddy
from the *opposite* non-movable migratetype's free list within tainted
SPBs. UNMOV alloc → check RECL free list; RECL alloc → check UNMOV
free list. The pageblock tag is NOT changed — page_del_and_expand uses
the source migratetype for both delete and re-list, so the splits stay
on the source list, and when our borrowed page is later freed
__free_one_page returns it to the source list (based on pageblock tag).
The "borrow" is purely transient: physical page goes to a foreign-type
caller, returns to its native list on free.

PB_has_<requested_type> is set via __spb_set_has_type so spb_defrag
accounting reflects that the pageblock now hosts our type's content.
PB_has_<source_type> stays set since other buddies of that type remain.

Restricted to UNMOV ↔ RECL within SB_TAINTED — movable allocations have
their own Pass 4 fallback, and clean SPBs must not be polluted with
cross-type mixing (that's what the existing migratetype-isolation
machinery exists to prevent).

Live measurement on a 247 GB devvm with btrfs root, kernel 397 (Stage 1
+ simplified Stage 2a) at boot+7min: 12 tainted Normal-zone SPBs grew
from 4 baseline despite the existing 11 having between 825 and 87,062
free pages each, ALL on the UNMOV list while the workload kept
allocating RECL btrfs_inode slab pages. Pass 2c lets those allocs
absorb into the existing UNMOV-listed free pool rather than creating
fresh tainted SPBs.

Signed-off-by: Rik van Riel <riel@surriel.com>
Assisted-by: Claude:claude-opus-4.7 syzkaller
---
mm/page_alloc.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 85 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a72cb2da606d..f2db3dd86a84 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2806,6 +2806,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
struct page *page;
int full;
struct superpageblock *sb;
+ int opposite_mt;
/*
* Category search order: 2 passes.
* Movable: clean first, then tainted (pack into clean SBs).
@@ -2985,6 +2986,90 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
}
}
}
+
+ /*
+ * Pass 2c: cross-non-movable borrow within tainted SPBs.
+ *
+ * If we're a non-movable alloc and Pass 1/2/2b couldn't find a
+ * buddy on our migratetype's free list anywhere, but tainted
+ * SPBs have free buddies on the *opposite* non-movable type's
+ * free list, take one of those.
+ *
+ * Why this happens: when pages are freed, __free_one_page puts
+ * them on the free_list determined by their pageblock's tag,
+ * not the original allocation's migratetype. Slab caches tend
+ * to be dense (RECL pageblocks fill up; few sub-PB fragments),
+ * while UNMOV pageblocks accumulate sparse free space from
+ * vmalloc/raw alloc churn. Net effect: tainted SPBs frequently
+ * have tens of thousands of free pages all on the UNMOV list,
+ * invisible to RECL allocs (or vice versa). Without this pass,
+ * the alloc falls through to Pass 3 and taints a fresh clean
+ * SPB even though the existing tainted ones have plenty of
+ * unused space.
+ *
+ * We do NOT relabel the source pageblock. The buddy is taken
+ * from @opposite_mt's free list and the splits go back on
+ * @opposite_mt's list (page_del_and_expand uses the same mt
+ * for delete and expand). The pageblock tag is unchanged, so
+ * the page returns to @opposite_mt's list when freed via
+ * __free_one_page. Effectively a borrow: the alloc takes a
+ * physical page from a UNMOV-tagged pageblock for a RECL
+ * use, and the page cycles back to UNMOV's list on free.
+ *
+ * We do set PB_has_<migratetype> via __spb_set_has_type so
+ * spb_defrag accounting reflects that this pageblock now hosts
+ * our migratetype's content too. PB_has_<opposite_mt> stays
+ * set since other buddies of that type remain.
+ *
+ * Restricted to UNMOV ↔ RECL. Movable allocations don't
+ * participate (they have their own Pass 4 fallback path).
+ *
+ * Restricted to SB_TAINTED to avoid spreading mixing into
+ * clean SPBs.
+ */
+ opposite_mt = -1;
+ if (migratetype == MIGRATE_UNMOVABLE)
+ opposite_mt = MIGRATE_RECLAIMABLE;
+ else if (migratetype == MIGRATE_RECLAIMABLE)
+ opposite_mt = MIGRATE_UNMOVABLE;
+
+ if (opposite_mt >= 0) {
+ for (full = SB_FULL; full < __NR_SB_FULLNESS; full++) {
+ list_for_each_entry(sb,
+ &zone->spb_lists[SB_TAINTED][full], list) {
+ int co;
+
+ if (!sb->nr_free_pages)
+ continue;
+ for (co = min_t(int, pageblock_order - 1,
+ NR_PAGE_ORDERS - 1);
+ co >= (int)order;
+ --co) {
+ current_order = co;
+ area = &sb->free_area[current_order];
+ page = get_page_from_free_area(
+ area, opposite_mt);
+ if (!page)
+ continue;
+ if (get_pageblock_isolate(page))
+ continue;
+ if (is_migrate_cma(
+ get_pageblock_migratetype(page)))
+ continue;
+ page_del_and_expand(zone, page,
+ order, current_order,
+ opposite_mt);
+ __spb_set_has_type(page,
+ migratetype);
+ trace_mm_page_alloc_zone_locked(
+ page, order, migratetype,
+ pcp_allowed_order(order) &&
+ migratetype < MIGRATE_PCPTYPES);
+ return page;
+ }
+ }
+ }
+ }
}

/*
--
2.52.0