From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4838C88E5C for ; Wed, 16 Sep 2026 06:44:18 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4hl8Vq0z3hz30RT; Wed, 16 Sep 2026 16:44:03 +1000 (AEST) Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:39::10" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1789541043; cv=none; b=FkV2Mj7z0vheGoPru425oDLiSEWawwQGmFDMKM7WNUXTYBTpF8O6sMP/KimluPtLRDTIgAh9hTt9tByX5k4oQy1YGlSgCBUww8/uv3iLS4FaczG/ma90cqLB9xi/qgX4WwQeZyctDMJM0jKzxk7p+6xuMPKCllQ743IOfUZQi+1wwVLB4FXHUFY7dhsT71PJQmmvjqpqJgEmra2nDQoMuneE2x0HgTDLNfvTDawITMtRfk7aLEexAmL/zSbDNZPGaoNraO1cF+oYFAaIlFZXTecPp690ww2Z/dgj9DcdtMeA77l+6VrmTLCkDA4cBjWmxsUycTQM4pVciL+IEnit0A== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1789541043; c=relaxed/relaxed; bh=uBBuOi4IklhpEQjilPGFjs1/1RKNgj5LsNpTgitcyy0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c5N4nPbqA1OSzbtBfzqjUaISAShYw1YOR8bTz7szvsLVvIrY8/JgTB5WNVPefdfkNMOtRMfEfbMedZopD8b2n45MYkxzCQSyT5g21ut/imwuqYhsZCyVdDsH9QvoFBE52ho3IbN7G9LaLeIw7LjlXv2EKwc+mVMtQA2Hnw2jejMovE/ito+/Fws6fUQ+WDIN8waL5pQ5IYNehG0YrMn/fPo/Ty9dmUmX1OUwC4EMoKAvZcAjdv4y8bxPs1tX8eEVZetk9NV3ht1T3ICGU0dGs7GpbkgCPICYXzpty4hP20KNAWQ2ytDYlwjsZniYkVjJis3wR392WquUnq5TmFQOgA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HuLDP+0K; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:39::10; helo=mail-pj2-x10.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HuLDP+0K; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:39::10; helo=mail-pj2-x10.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj2-x10.google.com (mail-pj2-x10.google.com [IPv6:2607:f8b0:4864:39::10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4hl8Vp2FFCz2y2Q for ; Wed, 16 Sep 2026 16:44:02 +1000 (AEST) Received: by mail-pj2-x10.google.com with SMTP id d9443c01a7336-2d747ec6188so4080145ad.3 for ; Tue, 15 Sep 2026 23:44:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541040; x=1790145840; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uBBuOi4IklhpEQjilPGFjs1/1RKNgj5LsNpTgitcyy0=; b=HuLDP+0KCZ/FThVLU1TZ2h1ClxK316ReO9Q4zuzmDpEOfldfp6Z5LX9sVx3wOJdO/U 2R+0X0TzMEBwjFcVxy/krObe6pg2lNJdIEsGdoilcg0cDWgiqT6MqxaEA+pXraterCFG pjohc3JIpm16BeZjELHu14MGq6kCH1W14RzbzYB+BsRxBKNXVzGZD7g+pmbHyq6RSoT1 8a8kmMm8jBf8Qkqu/Re1hEjPHbTfjkFKtYkWJhi8zq286HqPjNMSx7mB6q4Q/PnDZem3 cFbt2J6l4FsKrY2OMUXqNtbU46edfRpo4g3Yy+0TyByf/jLRX5whJNwA7y//VTX4vur7 dO1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541040; x=1790145840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=uBBuOi4IklhpEQjilPGFjs1/1RKNgj5LsNpTgitcyy0=; b=Y8M+lLoCEeScZNQSXFKn8RI5v1kVzlRgGr48VhcAPzrJf8Kq6wOyZ2OwT0vUdYciTD f+x9EV+HHz7GWkYTl3uyOaIwr1H4rMV0aVrvCKeGumRSkkCVjxg9PLO9ghvq4WchytHx JvlmrnqZZa7UCXjO9cihjzyhyn6r7IUDlTJnd2TOx9+36ymfSthQ1KmHqYGiYDDucIL9 IbUnLNoxWLRlB+HXDugV/JJJ7rG4z993C83AAT2fk7Hp/oWcMbsax8z3RVQYFSyX8O6P A6/s3hvnv5WfBtiWzdwaog5beipSkXAkRxpM0ZE4PnOhbc4SLmc7DIE9qaI0p35N91wg 9OIg== X-Forwarded-Encrypted: i=1; AKwUvBzWGsweT7CbtPPnkGdy1MSztGnIOqrISDxwt0YzzlwSWAnD4RKBhUUwR+/cXn/JZH998ukQgA7byKiJvMg=@lists.ozlabs.org X-Gm-Message-State: AFuF++kisCz9JgyNqGT9fuFMTIsjJ6VdzpUg45l6RvXf/w22fKrCLTBf Uh6JKmZcVnAyY9ftz/ms41/C2HWClKA1fWq6IRxZmTCsmUKoticrECG/oRwK95WvRdw= X-Gm-Gg: AYBFou0DJmypIhHy5PolcEutGuge070BQi0dTk/mP8CXndYqjbUFOqWbD0gK5iJC9Xa L0pua6FOAIYAoH3D4zEySxRcQnkHHc6+nQFFpqeneleZwqPGM0LbAqUwdWPre2VhOnIsxZ87bF3 hk+BZ1dZP5U2e0q9wQJF2SsNRJ/tR79r3Boria+GBFlV/PNav3NCcHq1cNDnwgJ8BW2IaaJqKVX 5rOf/glFfV21GqgkHoX3yeRJ1llOKek5xRjjhW5tpVeLBsJfQJTcmOJDJrKd0d30Wq0ba/2eknt GFK8T8lLCS3NRJ3jXj8XLSMu3VYNYWnOogW4EK3A1c33Mls062v1I45BY5tIZfnS20F4QVTT8mO vQEkX9/DnDnJqBPR0e4/Sbk2dCGZWr/rTxsXwARx2KM89FtDmutvcJpopjniye8qn8fSPRMsBOI v/RlYtX3kkmaqUN5gzm2PWU0bps3areU1xQhy1jDxtgrgEhqq6wcpiub0Oe5XRKSVr/eqBuEcPE n32Zt47wa0ILgUuNzJg X-Received: by 2002:a17:902:ffcb:b0:2bf:27b2:4b80 with SMTP id d9443c01a7336-2dd8e40607cmr31031175ad.14.1789541040200; Tue, 15 Sep 2026 23:44:00 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.43.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:43:59 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 02/11] mm/sparse-vmemmap: factor out shared vmemmap tail page allocation Date: Wed, 16 Sep 2026 14:43:32 +0800 Message-ID: <20260916064341.1825793-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit HugeTLB and sparse-vmemmap each have their own helper to allocate the shared vmemmap tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper. It allocates the page through vmemmap_alloc_block(), and uses cmpxchg() to install the per-zone shared page. Expose zone->vmemmap_tails under CONFIG_SPARSEMEM_VMEMMAP to match the shared helper's build condition. This avoids a !CONFIG_VMEMMAP_OPTIMIZATION stub; when optimization is disabled, the array has no entries and the compiler folds away the unused paths, so no storage or runtime overhead is added. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v4: - Update the commit message for the renamed VMEMMAP_OPTIMIZATION config - Collect Acked-by from Mike Rapoport v2: - Collect Acked-by from Qi Zheng --- include/linux/mmzone.h | 2 +- mm/hugetlb_vmemmap.c | 28 +--------------- mm/sparse-vmemmap.c | 74 +++++++++++++++++------------------------- mm/sparse.h | 1 + 4 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 56403841e887..97147c764658 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1156,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_VMEMMAP_OPTIMIZATION +#ifdef CONFIG_SPARSEMEM_VMEMMAP struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index f977d0a7e002..5ddf06b83c96 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -493,32 +493,6 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio * return true; } -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - struct page *tail, *p; - int node = zone_to_nid(zone); - - tail = READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p = page_to_virt(tail); - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail = READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -535,7 +509,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, return ret; nid = folio_nid(folio); - vmemmap_tail = vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail = vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index f22d815d7af0..7388a5b5cce3 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -42,27 +42,13 @@ #include "mm_init.h" #include "sparse.h" -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask = GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order = get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; page = alloc_pages_node(node, gfp_mask, order); @@ -76,8 +62,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -184,39 +169,40 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) return p; } -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone) { - struct page *p, *tail; - unsigned int idx; - int node = zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >= ARRAY_SIZE(zone->vmemmap_tails))) return NULL; - idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail = zone->vmemmap_tails[idx]; - if (tail) - return tail; - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + page = READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; + + addr = vmemmap_alloc_block(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - tail = virt_to_page(p); - zone->vmemmap_tails[idx] = tail; + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page = (struct page *)addr + i; + mm_zero_struct_page(page); + init_compound_tail(page, NULL, order, zone); + } - return tail; -} -#else -static inline struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - return NULL; + page = virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(addr, PAGE_SIZE); + page = READ_ONCE(zone->vmemmap_tails[idx]); + } + + return page; } -#endif static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct vmem_altmap *altmap) @@ -229,7 +215,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); zone = pfn_to_zone(pfn, node); - page = vmemmap_get_tail(order, zone); + page = vmemmap_shared_tail_page(order, zone); if (!page) return NULL; diff --git a/mm/sparse.h b/mm/sparse.h index d8c08a388e4e..8f6ef0b50b4f 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -142,6 +142,7 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zone); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); -- 2.54.0