From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9341A36B076 for ; Wed, 8 Jul 2026 03:11:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783480307; cv=none; b=H2rlHOsmRTfv98rI9NuayluBNN33OAccbEJ+JUF4RfgvU6Gv2YNgke4fv+z2irn3R+oQu+iNHfgMvXxhJyd3/rhuCSu6he5l1aptiLuM4k/6rVrxahThHRLolirioZHMV4mtXQ52oUp3UJ6ldBoErZ/tvmFzGOlio8+EvvyFZQY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783480307; c=relaxed/simple; bh=3YGzHn2LgiibTEI2OLgL70eDAtVR9AeJAhmQXwn1QJ4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nJqTx1zaLNmoKOoyXu+kOle+yeIZ5OmWE3j2hNtWM45/GgeuRWVloCiIqo+voq5Wsm1U6p8AYweyOb2/U2xdtwH18+5Z8gIsoOeaUoGCIpZdAgFp12jKm+xVeZ8IpuCqU14DeY3vF0DvbV0IXRfmyHrVJ9n/dzVZ2oHEr/Vo7hk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ppbak6G0; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ppbak6G0" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2cca3673560so4408835ad.1 for ; Tue, 07 Jul 2026 20:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1783480303; x=1784085103; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=nyOZS/IOogCiOtSe1oniQEutV7yDNXcShcSAXMB8Vac=; b=ppbak6G0k/VCU+cfv/eWxJoT0GjkTMBgjDVCNVpsPOg5N8pBg+dpdzJnQMf4Mb6PTF ndZaS6bbuWlB0PWlRibTzhbor3lO8izt7cVxh3ME3tpHbvhD4scGY9wUfA9da+DXuZeD mns5cS9JZ7hOykRhMqhFmFk+4tR6htm4FiOIv0cRJgpd4NTkz9nVD+XOzWkxiib8FYPe IFojvOFjtARRcLS3f4we9dQzHcxnK3I2jKFuRo4SkCDSVHTi93ceZBSSbNuhOp0GFj+y myQ/VJpX42aQBlJoHvUbHSzsHZ89amYXSiK/OpVXyk4jjRB6R+FNlpxlarHO1JGiFMIu Gk+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783480303; x=1784085103; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=nyOZS/IOogCiOtSe1oniQEutV7yDNXcShcSAXMB8Vac=; b=fIyISMZUwz768iipc7aBPGGYo5AxeJZjEPqHrMNoaoyEoDFy7OljN6mtLL1gFkijCU 82irMQbrXRRgSV+JaaAlhDCEg/jXOVUQEtLcabPkaLsMSM4iHlr7KyTWA/vXrP3cgHOn jzYIPwbg8K/Y6AebsHtWe3adNDlcxifKG2chHISbvYhRbtsoEysCqs/7XOMzk5Y1CVVO U8UBs6jJMDiPNm+k/39yLybRK9gEz/8YuoOxkB6gCBGA2YUGVJnGvhFKWPG9uQmSxNBu qSS2AR6NXJaRxLTjNzctdinb8uLeCjY4S218dlWgtNPqPX6uXSyP4Fnnf9o5RwsXWgZe wLnA== X-Forwarded-Encrypted: i=1; AHgh+RqP0tMwEdYG/975CGRCXKcVbzTPkJ+VChEjyoFakUXjzbqsVRsE/0j+EvUToEtCHv2szNL3jrsjKpRT9BM=@vger.kernel.org X-Gm-Message-State: AOJu0YyIvAyRXjdLdcYMNgYa/+HZGQrZqn3aujtMHZraZ8OpOWUqJw2Y k3adjGrhnisACTKKCz/cxRN+7JIxBoOVNtdZggZHrGxMqyWMz44U2/zJYnUil+f6HHg2YoQawzR fNnYDjaBzyIqkHmLGlgLtKQ== X-Received: from plbmk12.prod.google.com ([2002:a17:903:2bcc:b0:2cc:e106:f652]) (user=jthoughton job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:4684:b0:2ca:9ab:e723 with SMTP id d9443c01a7336-2ccea481a57mr6652835ad.33.1783480302952; Tue, 07 Jul 2026 20:11:42 -0700 (PDT) Date: Wed, 8 Jul 2026 03:11:21 +0000 In-Reply-To: <20260708031129.3503195-1-jthoughton@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260708031129.3503195-1-jthoughton@google.com> X-Mailer: git-send-email 2.55.0.795.g602f6c329a-goog Message-ID: <20260708031129.3503195-12-jthoughton@google.com> Subject: [PATCH 11/18] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs From: James Houghton To: Will Deacon , Catalin Marinas , Muchun Song , Oscar Salvador Cc: Nikos Nikoleris , Linu Cherian , Mark Rutland , David Hildenbrand , Andrew Morton , Ryan Roberts , Nanyong Sun , Yu Zhao , Frank van der Linden , David Rientjes , James Houghton , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" This routine is to be used for updating in-use, leaf-level PMDs in the vmemmap without introducing a window where vmemmap accesses might fault. Because try_populate_vmemmap_pmd() can fail, the split_page() call needs to be moved to after the PMD update. Because both split_page() and the PMD update are done under the page table lock, this rearrangement is safe. Only leaf-level PMD to non-leaf PMD (mapping the same physical pages) is the only transition that needs to be supported. Collapsing a non-leaf PMD to a leaf-level PMD is not used by HVO, so try_populate_vmemmap_pmd() need not support it. This patch allows arm64's implementation to replaced with one that does not require BBML2_NOABORT. Signed-off-by: James Houghton --- arch/arm64/include/asm/pgalloc.h | 9 +++++++++ arch/loongarch/include/asm/pgalloc.h | 8 ++++++++ arch/riscv/include/asm/pgalloc.h | 8 ++++++++ arch/x86/include/asm/pgalloc.h | 8 ++++++++ include/linux/pgalloc.h | 20 ++++++++++++++++++++ mm/hugetlb_vmemmap.c | 28 ++++++++++++++++------------ 6 files changed, 69 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 1b4509d3382c..c8946250d431 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -121,4 +121,13 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_PXN); } +#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD +static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable, + unsigned long addr) +{ + /* BBML2_NOABORT is required. Its presence has been checked. */ + pmd_populate_kernel(&init_mm, pmdp, pgtable); + return 0; +} + #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 248f62d0b590..9322d962c9cc 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -24,6 +24,14 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte) set_pmd(pmd, __pmd((unsigned long)page_address(pte))); } +#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD +static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable, + unsigned long addr) +{ + pmd_populate_kernel(&init_mm, pmdp, pgtable); + return 0; +} + #ifndef __PAGETABLE_PMD_FOLDED static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 770ce18a7328..8dbbe62b2574 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -31,6 +31,14 @@ static inline void pmd_populate(struct mm_struct *mm, set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); } +#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD +static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable, + unsigned long addr) +{ + pmd_populate_kernel(&init_mm, pmdp, pgtable); + return 0; +} + #ifndef __PAGETABLE_PMD_FOLDED static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index c88691b15f3c..47b56bbd6236 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -82,6 +82,14 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, set_pmd(pmd, __pmd(((pteval_t)pfn << PAGE_SHIFT) | _PAGE_TABLE)); } +#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD +static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable, + unsigned long addr) +{ + pmd_populate_kernel(&init_mm, pmdp, pgtable); + return 0; +} + #if CONFIG_PGTABLE_LEVELS > 2 extern void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd); diff --git a/include/linux/pgalloc.h b/include/linux/pgalloc.h index 9174fa59bbc5..ed446d95ca37 100644 --- a/include/linux/pgalloc.h +++ b/include/linux/pgalloc.h @@ -26,4 +26,24 @@ arch_sync_kernel_mappings(addr, addr); \ } while (0) +#ifndef __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD +/* + * try_populate_vmemmap_pmd - Populate a PMD that is in use by the vmemmap. + * @addr: Base address of the remapped PMD. + * @pmdp: Page table pointer to be overwritten. + * @pgtable: Pointer to the page table that the new PMD will point to. + * + * This function is only to be used to update PMDs that map the vmemmap to + * point to a page of already-populated PTEs that map the same pages. + * + * Implementations of this function must ensure that, while the update is taking + * place, CPUs will not fault on the remapped virtual address range. + */ +static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable, + unsigned long addr) +{ + return -EOPNOTSUPP; +} +#endif + #endif /* _LINUX_PGALLOC_H */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 977249e22ed0..b445febac0d2 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -55,6 +55,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, struct vmemmap_remap_walk *walk) { pmd_t __pmd; + int ret; int i; unsigned long addr = start; pte_t *pgtable; @@ -74,8 +75,15 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, set_pte_at(&init_mm, addr, pte, entry); } + ret = 0; spin_lock(&init_mm.page_table_lock); if (likely(pmd_leaf(*pmd))) { + /* Make pte visible before pmd. See comment in pmd_install(). */ + smp_wmb(); + ret = try_populate_vmemmap_pmd(pmd, pgtable, start); + if (ret) + goto free; + /* * Higher order allocations from buddy allocator must be able to * be treated as independent small pages (as they can be freed @@ -84,21 +92,17 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, if (!PageReserved(head)) split_page(head, get_order(PMD_SIZE)); - /* Make pte visible before pmd. See comment in pmd_install(). */ - smp_wmb(); - /* - * On arm64, this requires BBML2_NOABORT. Its support has - * already been checked. - */ - pmd_populate_kernel(&init_mm, pmd, pgtable); if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, start + PMD_SIZE); - } else { - pte_free_kernel(&init_mm, pgtable); - } - spin_unlock(&init_mm.page_table_lock); + } else + goto free; - return 0; +out: + spin_unlock(&init_mm.page_table_lock); + return ret; +free: + pte_free_kernel(&init_mm, pgtable); + goto out; } static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr, -- 2.55.0.795.g602f6c329a-goog