From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1EECC53200 for ; Wed, 29 Jul 2026 12:25:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MHIF/aT1O3EUVM+gwRc5yndgIL250Z4yq+9OlldutD4=; b=WXX4ZkuaKIpzgaRw86SNC2sgAq MkDQn+0UNnRp8RSSFKpFcdWnmvh5MEG6CbPRuomO5wL89zot3hdW0v73EqjXHsBsJQtcne7g6Y534 97eiWDaO4sGpr+BcoknRg3PnwQZsWCVxRoFrQVpVROTUR+YWeqdukBUKoa+jnMoSgqAeQQ6mXwXZ8 bGqN8VXQNzEhMhGp4R6yWSqgSuFGrPJAeOY3KaxY7NftqCrnKCVeqj7jcJRVnkGTF0VpgCgol6Dk+ lJR6k1mUVaxTbG+gud+ArlxBL/HcaI7LOgkq5lAjbIr+AKP1PHtrvn7vaIH5soBgK2bGF9Pyvr6S5 y9PZhCmA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wp3Lt-00000007tH4-2LmV; Wed, 29 Jul 2026 12:25:37 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wp3Ld-00000007t8Y-30mj for linux-arm-kernel@lists.infradead.org; Wed, 29 Jul 2026 12:25:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1393E169C; Wed, 29 Jul 2026 05:25:17 -0700 (PDT) Received: from a085714.blr.arm.com (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 519D83F66F; Wed, 29 Jul 2026 05:25:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1785327921; bh=C1DQWx7VYKRGPi4zlsQhkY9GTpNb1XhVXT0s7ZGILy8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eM0OXzRVsZRl28AEv9J7yGGxyMutjplqRdkVX42NdjtAGm/nUOIEwIE9P0bRTQQyi ViKCtdN6sgKO9di0A0OvX/UZjeQistd+PQfT2ANH7hnWowGhn6vOrMByd2X+smkYjf TAIjRoKjpWuvaRMiuTPwUm2SdLl56CFH/MVsf97M= From: Anshuman Khandual To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: [PATCH 03/17] arm64/mm: Convert READ_ONCE() as pudp_get() while accessing PUD Date: Wed, 29 Jul 2026 17:54:38 +0530 Message-ID: <20260729122452.3797443-4-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260729122452.3797443-1-anshuman.khandual@arm.com> References: <20260729122452.3797443-1-anshuman.khandual@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260729_052521_834392_E8D096DE X-CRM114-Status: GOOD ( 23.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Convert all READ_ONCE() based PUD accesses as pudp_get() instead which will support both D64 and D128 translation regime going forward. That is because READ_ONCE() would need 128 bit single copy atomic guarantees, while reading 128 bit page table entries which is currently not supported on arm64. Build fails for READ_ONCE() while accessing beyond 64 bits. Load Pair/Store Pair (ldp/stp) are only single copy atomic if FEAT_LSE128 is supported (which is required when FEAT_D128 is supported). Currently 128 bit pgtables is a compile time decision - so we could have chosen to extend READ_ONCE()/WRITE_ONCE() to allow 128 bit for this configuration. But then it's a general purpose API and we were concerned that other users might eventually creep in that expect 128 and then fail to compile in the other configs. But worse, we are considering eventually making D128 a boot time option, at which point we'd have to make READ_ONCE() always allow 128 bit at compile time but then it might silently tear at runtime. So our preference is to standardize on these existing helpers, which we can override in arm64 to give the 128 bit single copy guarantee when required. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgtable.h | 3 ++- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 4 ++-- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 20 ++++++++++---------- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 8 files changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 5c0cc7c676fa..c9eef353455f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -917,7 +917,8 @@ static inline pmd_t *pud_pgtable(pud_t pud) } /* Find an entry in the second-level page table. */ -#define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t)) +#define pmd_offset_phys(dir, addr) (pud_page_paddr(pudp_get(dir)) + \ + pmd_index(addr) * sizeof(pmd_t)) #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) #define pmd_set_fixmap_offset(pud, addr) pmd_set_fixmap(pmd_offset_phys(pud, addr)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 30f8b3872331..ccfd4ff76f88 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -174,7 +174,7 @@ static void show_pte(unsigned long addr) break; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); pr_cont(", pud=%016llx", pud_val(pud)); if (pud_none(pud) || pud_bad(pud)) break; diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 3cdac8021d4f..d9a870836faf 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -56,7 +56,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr, unsigned long end) { unsigned long next; - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get(pudp); pmd_t *pmdp; if (pud_none(pud)) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index cdaa4500faf9..c9ad5e75b073 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -262,7 +262,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, WARN_ON(addr & (sz - 1)); ptep = pte_alloc_huge(mm, pmdp, addr); } else if (sz == PMD_SIZE) { - if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp))) + if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp))) ptep = huge_pmd_share(mm, vma, addr, pudp); else ptep = (pte_t *)pmd_alloc(mm, pudp, addr); @@ -292,7 +292,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (sz != PUD_SIZE && pud_none(pud)) return NULL; /* hugepage or swap? */ diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 8cb440379f17..b1926b661efe 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -76,7 +76,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, bool early) { - if (pud_none(READ_ONCE(*pudp))) { + if (pud_none(pudp_get(pudp))) { phys_addr_t pmd_phys = early ? __pa_symbol(kasan_early_shadow_pmd) : kasan_alloc_zeroed_page(node); @@ -150,7 +150,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); kasan_pmd_populate(pudp, addr, next, node, early); - } while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp))); + } while (pudp++, addr = next, addr != end && pud_none(pudp_get(pudp))); } static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 85b2e2f9ea3e..fd1ce8649686 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -310,7 +310,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, { int ret; unsigned long next; - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get(pudp); pmd_t *pmdp; /* @@ -391,7 +391,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, } do { - pud_t old_pud = READ_ONCE(*pudp); + pud_t old_pud = pudp_get(pudp); next = pud_addr_end(addr, end); @@ -409,7 +409,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); + pud_val(pudp_get(pudp)))); } else { ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); @@ -417,7 +417,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, goto out; VM_WARN_ON_ONCE(pud_val(old_pud) != 0 && - pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); + pud_val(old_pud) != pud_val(pudp_get(pudp))); } phys += next - addr; } while (pudp++, addr = next, addr != end); @@ -1543,7 +1543,7 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) continue; @@ -1700,7 +1700,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) continue; @@ -1721,7 +1721,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, */ pudp = pud_offset(p4dp, 0UL); for (i = 0; i < PTRS_PER_PUD; i++) { - if (!pud_none(READ_ONCE(pudp[i]))) + if (!pud_none(pudp_get(pudp + i))) return; } @@ -1819,7 +1819,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); /* Only allow permission changes for now */ - if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)), + if (!pgattr_change_is_safe(pud_val(pudp_get(pudp)), pud_val(new_pud))) return 0; @@ -1850,7 +1850,7 @@ void p4d_clear_huge(p4d_t *p4dp) int pud_clear_huge(pud_t *pudp) { - if (!pud_leaf(READ_ONCE(*pudp))) + if (!pud_leaf(pudp_get(pudp))) return 0; pud_clear(pudp); return 1; @@ -1903,7 +1903,7 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) pud_t pud; unsigned long next, end; - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (!pud_table(pud)) { VM_WARN_ON(1); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 0ca07bd5ded9..07b2fa4de57f 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -407,7 +407,7 @@ bool kernel_page_present(struct page *page) return false; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) return false; if (pud_leaf(pud)) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index b27b2d2c20c3..d119119455f1 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -64,7 +64,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp, unsigned long next; unsigned long addr = start; - if (pud_none(READ_ONCE(*dst_pudp))) { + if (pud_none(pudp_get(dst_pudp))) { dst_pmdp = trans_alloc(info); if (!dst_pmdp) return -ENOMEM; @@ -109,7 +109,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp, src_pudp = pud_offset(src_p4dp, start); do { - pud_t pud = READ_ONCE(*src_pudp); + pud_t pud = pudp_get(src_pudp); next = pud_addr_end(addr, end); if (pud_none(pud)) -- 2.43.0