From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B40AE3C0A01 for ; Wed, 30 Sep 2026 19:01:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790794896; cv=none; b=uLnTEI17ggDGMluPt+HOvSnkAvg8pXo+2KTnRWvVxwLaP+GQ424fEA2Y7pVQ8rOo34BjOS4OG4VbzFw9i+5WO7/AcuANFCymOEf8Zimh3LuZRMt6rW0ev88l/IFGJnNhxQD6bkAi5z/9CENUTn4yBsFx3FAhKhf3MBrZHdrvVxU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790794896; c=relaxed/simple; bh=QLzisP55SE9Ec3QYPBrTnO4Hr7a5kr2gKCH1rl/dA8Y=; h=Date:To:From:Subject:Message-Id; b=P22I4sYoNHXVGZp0yoIsH8V0NQ/olLWZfuAIujxui9kjkD4zZj99ySPoyYEKL3d8CVVC94RFVwuQs7vKogSvzmDc4kFoKwCuBKhXPiHICxmd0P0VaizPABuF/u3TPiOgPLwk1qqSPq1y8AC6T9w2PB/zcDwML29LoFnZ1ULFEDg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=RV5E/+3a; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="RV5E/+3a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3AF0E1F000FF; Wed, 30 Sep 2026 19:01:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1790794894; bh=5iJnOBATP0I0qwnvIW9BMUstkwBJBBWdtwrD59wE7CM=; h=Date:To:From:Subject; b=RV5E/+3atvXjdi4JONyVHRyE+BK9A9MXAR7O9M0/ic5uylcaQloBpdSV3kXqW5Zc5 w4+5fNvTpcM+aqtyCqByit3azjXUoOMBlQuDBQK2TZqY5pqbQSqk4ZbMecQE2q5sv/ BfbwaDfbOLk+d06s0DaCqKK38WBZAMVZNDehldf0= Date: Wed, 30 Sep 2026 12:01:33 -0700 To: mm-commits@vger.kernel.org,muchun.song@linux.dev,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation.patch added to mm-unstable branch Message-Id: <20260930190134.3AF0E1F000FF@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/sparse-vmemmap: factor out shared vmemmap tail page allocation has been added to the -mm mm-unstable branch. Its filename is mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Muchun Song Subject: mm/sparse-vmemmap: factor out shared vmemmap tail page allocation Date: Wed, 30 Sep 2026 22:06:16 +0800 Patch series "mm: Switch device DAX to section-based vmemmap optimization", v6. This series is split out from the earlier, larger series "mm: Generalize HVO for HugeTLB and device DAX" [1]. While the parent series generalizes vmemmap optimization across HugeTLB and device DAX, this subset addresses a single, self-contained step: switching device DAX to the section-based sparse-vmemmap optimization infrastructure introduced for HugeTLB. After the HugeTLB conversion, optimized vmemmap state is described by the memory section and the sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages based on that metadata. Device DAX still uses the older DAX-specific population model, including a separate tail vmemmap page reservation and architecture-specific logic to locate or populate reusable tail pages. This series makes device DAX use the same section-based model. Device DAX records the compound page order from pgmap->vmemmap_shift in section metadata before vmemmap population, uses the common per-zone shared tail vmemmap page, and drops the extra reserved tail page. The powerpc radix path is updated to use the same shared tail-page helper, so the generic and powerpc DAX paths follow the same reservation model. The first patches prepare the shared infrastructure by factoring out shared tail-page allocation, allocating the per-zone shared tail-page array dynamically, and introducing a generic CONFIG_VMEMMAP_OPTIMIZATION symbol. The middle patches move device DAX onto that infrastructure by recording the device DAX compound page order in memory-section metadata, using that metadata to back generic device DAX mappings with the common per-zone shared tail page, exposing the shared helpers so the powerpc radix path can use the same model, and dropping the extra DAX-only tail page reservation and the now-unused section accounting arguments. The final patch updates the documentation for the new DAX layout. This is intended to be the third smaller step toward the broader HVO generalization. The wider HVO consolidation between HugeTLB and device DAX is left for follow-up series. This patch (of 12): HugeTLB and sparse-vmemmap each have their own helper to allocate the shared vmemmap tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper. It allocates the page through vmemmap_alloc_block(), initializes the tail struct pages, and uses cmpxchg() to install the per-zone shared page. This removes duplicate allocation logic while handling both early boot and runtime allocation through the same helper. Link: https://lore.kernel.org/20260930140627.57431-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20260930140627.57431-2-songmuchun@bytedance.com Link: https://lore.kernel.org/20260513130542.35604-1-songmuchun@bytedance.com/ [1] Signed-off-by: Muchun Song Signed-off-by: Andrew Morton Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Cc: Oscar Salvador Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Jonathan Corbet Cc: Lorenzo Stoakes Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Ritesh Harjani (IBM) Cc: Shrikanth Hegde Cc: Randy Dunlap Cc: Lance Yang --- mm/hugetlb_vmemmap.c | 29 ---------------- mm/sparse-vmemmap.c | 70 ++++++++++++++++++----------------------- mm/sparse.h | 3 + 3 files changed, 36 insertions(+), 66 deletions(-) --- a/mm/hugetlb_vmemmap.c~mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation +++ a/mm/hugetlb_vmemmap.c @@ -19,7 +19,6 @@ #include #include "hugetlb_vmemmap.h" #include "sparse.h" -#include "internal.h" /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -493,32 +492,6 @@ static bool vmemmap_should_optimize_foli return true; } -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - struct page *tail, *p; - int node = zone_to_nid(zone); - - tail = READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p = page_to_virt(tail); - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail = READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -535,7 +508,7 @@ static int __hugetlb_vmemmap_optimize_fo return ret; nid = folio_nid(folio); - vmemmap_tail = vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail = vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; --- a/mm/sparse.h~mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation +++ a/mm/sparse.h @@ -142,6 +142,9 @@ static inline void sparse_sections_init( * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zone); +#endif void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --- a/mm/sparse-vmemmap.c~mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation +++ a/mm/sparse-vmemmap.c @@ -42,27 +42,13 @@ #include "mm_init.h" #include "sparse.h" -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask = GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order = get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; page = alloc_pages_node(node, gfp_mask, order); @@ -76,8 +62,7 @@ void * __meminit vmemmap_alloc_block(uns } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -185,34 +170,43 @@ static void * __meminit vmemmap_alloc_bl } #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone) { - struct page *p, *tail; - unsigned int idx; - int node = zone_to_nid(zone); + const unsigned int idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER; + struct page *page; + void *addr; - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >= VMEMMAP_OPTIMIZATION_NR_ORDERS)) return NULL; - idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail = zone->vmemmap_tails[idx]; - if (tail) - return tail; - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + page = READ_ONCE(zone->vmemmap_tails[idx]); + if (page) + return page; + + addr = vmemmap_alloc_block(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - tail = virt_to_page(p); - zone->vmemmap_tails[idx] = tail; + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page = (struct page *)addr + i; + mm_zero_struct_page(page); + init_compound_tail(page, NULL, order, zone); + } + + page = virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(addr, PAGE_SIZE); + page = READ_ONCE(zone->vmemmap_tails[idx]); + } - return tail; + return page; } #else -static inline struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +static inline struct page *vmemmap_shared_tail_page(unsigned int order, + struct zone *zone) { return NULL; } @@ -229,7 +223,7 @@ static __meminit void *vmemmap_alloc_pte return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); zone = pfn_to_zone(pfn, node); - page = vmemmap_get_tail(order, zone); + page = vmemmap_shared_tail_page(order, zone); if (!page) return NULL; _ Patches currently in -mm which might be from muchun.song@linux.dev are mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation.patch mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch mm-sparse-vmemmap-introduce-config_vmemmap_optimization.patch mm-sparse-vmemmap-open-code-init_compound_tail.patch mm-sparse-vmemmap-prepare-dax-vmemmap-population-for-compound-page-orders.patch mm-sparse-vmemmap-set-compound-page-order-for-device-dax.patch mm-sparse-vmemmap-switch-device-dax-to-shared-tail-vmemmap-pages.patch mm-sparse-vmemmap-switch-device-dax-to-shared-tail-vmemmap-pages-fix.patch mm-sparse-vmemmap-move-vmemmap-optimization-helpers-to-a-public-header.patch powerpc-mm-switch-device-dax-to-shared-tail-vmemmap-pages.patch powerpc-mm-switch-device-dax-to-shared-tail-vmemmap-pages-fix.patch mm-sparse-vmemmap-drop-the-extra-tail-page-from-device-dax-reservation.patch mm-sparse-vmemmap-drop-unused-section_nr_vmemmap_pages-arguments.patch documentation-mm-update-dax-vmemmap-deduplication-docs.patch mm-sparse-vmemmap-drop-vmemmap_populate_dax.patch mm-sparse-vmemmap-support-device-dax-in-common-vmemmap-path.patch mm-sparse-vmemmap-drop-device-dax-specific-population-path.patch mm-sparse-vmemmap-remove-the-unused-ptpfn-argument.patch mm-sparse-vmemmap-open-code-vmemmap_populate_address.patch mm-mm_init-add-zone-mismatch-warning-during-page-init.patch