From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 922E0CD37B6 for ; Wed, 13 May 2026 04:46:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06CB66B0005; Wed, 13 May 2026 00:46:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 044976B0093; Wed, 13 May 2026 00:46:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC38E6B0095; Wed, 13 May 2026 00:46:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DF35E6B0005 for ; Wed, 13 May 2026 00:46:42 -0400 (EDT) Received: from smtpin02.hostedemail.com (lb01a-stub [10.200.18.249]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9880A4071A for ; Wed, 13 May 2026 04:46:42 +0000 (UTC) X-FDA: 84761161044.02.7397874 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf27.hostedemail.com (Postfix) with ESMTP id DBD6940002 for ; Wed, 13 May 2026 04:46:40 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=arm.com header.s=foss header.b=liIK+4SL; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1778647601; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ya1Tq6dViZ4/2yJfUr9NCElQL/oWotTyEpDtSxDDoH4=; b=YxWM7803ffKGi/OH9fPV9Al1dbw2zzRG2LdjeGJh5BQJNprg/GEbb4kSxrM6Q1iP3RHcVe euDP2rC22aS4KaUweeEBF6P56VVZCbZsNb/wnYke/Euhw/2F/UGG6Wkhwr206+WeohzGv0 Ojl/fJ38RPcE8TV1j2yslHe76zrcmNU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1778647601; a=rsa-sha256; cv=none; b=8FxDEfl+9k/IU9K9/YW4lolzK6704j8fVosXHuTKE6ZfYdcaPhOS262BJQyxIoHh2QR/7W ZrSqi/9zWEk6BNG5dQuTnMATGh2EzdGBpCBmfJ15VHRPu5KImKgG78BXj7YA9K/JYavGCB 0LmRqqaQPfyDKO8Oyo7C2l7vrgKJ3oM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=arm.com header.s=foss header.b=liIK+4SL; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C10B81CE0; Tue, 12 May 2026 21:46:34 -0700 (PDT) Received: from a085714.blr.arm.com (a085714.arm.com [10.164.18.87]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D321D3F7B4; Tue, 12 May 2026 21:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1778647600; bh=eDTcdRCah3F4lcFSBavG1sJnRBj9WTFAMr1s/sKs6Fk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=liIK+4SLsPCEGc29ncBoSl2yLqjPrzsopfZGE1Lz2iJ1GLUrn/dmCNsFRPpmgge2Z 07M54qPhkT5RgXYdYRogVyM7W+b4qqI5JbHWGikZ9fSI1hlCDv2W7LitmKct/w0vlC MWreDt9Xw3XqrwJEgCFkBRwyspVv1+UYjGQI1J64= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , Usama Arif , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [RFC V2 04/14] arm64/mm: Convert READ_ONCE() as pudp_get() while accessing PUD Date: Wed, 13 May 2026 10:15:37 +0530 Message-ID: <20260513044547.4128549-5-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260513044547.4128549-1-anshuman.khandual@arm.com> References: <20260513044547.4128549-1-anshuman.khandual@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: DBD6940002 X-Stat-Signature: 7wngwbo4qgeommf5ptfh1g6rm8whcrtq X-Rspam-User: X-HE-Tag: 1778647600-800144 X-HE-Meta: U2FsdGVkX1/3Dy9ezaRYgeTBoQDAfjHivDcOpV73FPecOGK6MGW0DICsd1I6obOG2CMLiS84x6wERhNuwCuJygefC8SLga3vlom0mdQhRBXRG7+zjvu8IQH8+8MDAQFM6vyxnpPy6pUE45MjfrhAuD9/vPj/pinTmqx8dwU9LBfNdB/AdocNx4S8jmB+7vZZ3BLQrqba/kkJwA8Wl6VMF6ZfezqtFAShAP8qyEY3//37cc9qposDTaB39eta/i6eN/dmFk5Dm5yJMi/vIt3sSh9QniCekiPAql+MgjNx8m/JvUPpk/+AlPXb/Y4KBU4mLTlUuTtyBryJ3LexhPSwPmD3QEvsWp4QdzaVifW7KdrBm4ds9/KM+7dXN45qi3tx0EpEWresrUrtae3tbx6yjs26johhA+zqSuN89HAEYrdAalZsfbQ0IePj84iNJY+eGYGAoNXVEDEhz5g/i7p2YyiMcDjG9jNL6XrXo/ou+7w0yx7wo4h7mrCMPr69L6hbWN9RXWhHq+bGkZvM3iSlbDFdmDq04t4Hbnbpcd9T3FwSb/MSobiuKIRfgWWNP2txjHMBQ5CkC0iI2TYYRMfVDtw9q8+oX1og0Mz4YkXiAcOw20rWzTibhgko10Exk42ZFZQ5RIfxsjwAHNPBl4LxRysH2Sf9vMdcQW8fiGhYlPm224RqZ7MMibN7vNkKwrtDdCnvv2Dc3WyN/KcGq2bmFjRx8ttDtxdPSS91Uxy98QhjRu7c2em2pmeX9ujr/ChL5B21aPAfCytgUKIOoYb9df3qWRNt7LJBC4sNeeYT14qdB1sIu1gxdnr2PRliJU5QLTtD2xELA3syCPX/4RN5/v6D1vpV7zANFRpMf+YkVBBmgi26HQlaPDJISy5nzf08aZDl2tkOOsdTPv7miEY5DT9Ay7r3+MBJnDmMLIaOKyOU90j4IjEKTsD1wrYBj/s9PYq8h3R6Kq1bvgitWwM 26psRtQ9 K7842hmn88sE27YOcv27lWf1W9W0jVQG7DADqbobp2MMPd7p5m/ZWf244tTaloEWtwo9OUtP8ELveOH49d4kjAwjjYoGusJ0eEdKzdUaf+cKRuffM4YYQi9FoPYHZ3sTQCPYs0wqQkI8xG2PiDs69jU78cSO4lMfqQyqyJ0uNBNrLYZFghZdh4eyG8jieSTBtjLcLY2ZXnKKFGfFZ/oQRPxrhhmd3ocGmX74sttBgEdDamKSq4UkKD08nBBCYBSHgeQKGFL12GbqEy4DyFkh2ZxYpi6WUbQm+tSBq4StmnLgdEyPrsOtZCXGXe4rxHxdqKDtJDbHvCn8be+CYnPoq8fu4aGgMbG7Ghw8NIvN6Lcqg8qjtHOM42GKeghebMHk3lWxr4y9lSQzG0N0RVHtAsi/KqSEXF926U0rb5Cy+a3leQFZ6msHMtyxLuA== Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert all READ_ONCE() based PUD accesses as pudp_get() instead which will support both D64 and D128 translation regime going forward. That is because READ_ONCE() would need 128 bit single copy atomic guarantees, while reading 128 bit page table entries which is currently not supported on arm64. Build fails for READ_ONCE() while accessing beyond 64 bits. Load Pair/Store Pair (ldp/stp) are only single copy atomic if FEAT_LSE128 is supported (which is required when FEAT_D128 is supported). Currently 128 bit pgtables is a compile time decision - so we could have chosen to extend READ_ONCE()/WRITE_ONCE() to allow 128 bit for this configuration. But then it's a general purpose API and we were concerned that other users might eventually creep in that expect 128 and then fail to compile in the other configs. But worse, we are considering eventually making D128 a boot time option, at which point we'd have to make READ_ONCE() always allow 128 bit at compile time but then it might silently tear at runtime. So our preference is to standardize on these existing helpers, which we can override in arm64 to give the 128 bit single copy guarantee when required. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Signed-off-by: Anshuman Khandual --- Changes in RFC V2 - Moved back helpers back from arch/arm64/mm/mmu.c into the header arch/arm64/include/asm/pgtable.h | 3 ++- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 4 ++-- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 20 ++++++++++---------- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 8 files changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2100ead01750..cefe8ab86acd 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -917,7 +917,8 @@ static inline pmd_t *pud_pgtable(pud_t pud) } /* Find an entry in the second-level page table. */ -#define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t)) +#define pmd_offset_phys(dir, addr) (pud_page_paddr(pudp_get(dir)) + \ + pmd_index(addr) * sizeof(pmd_t)) #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) #define pmd_set_fixmap_offset(pud, addr) pmd_set_fixmap(pmd_offset_phys(pud, addr)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 330eb314d956..63979f05d52f 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -172,7 +172,7 @@ static void show_pte(unsigned long addr) break; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); pr_cont(", pud=%016llx", pud_val(pud)); if (pud_none(pud) || pud_bad(pud)) break; diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 7a4bbcb39094..dd58af6561e0 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -56,7 +56,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr, unsigned long end) { unsigned long next; - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get(pudp); pmd_t *pmdp; if (pud_none(pud)) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index ffaa65ff55b4..012558a80002 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -262,7 +262,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, WARN_ON(addr & (sz - 1)); ptep = pte_alloc_huge(mm, pmdp, addr); } else if (sz == PMD_SIZE) { - if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp))) + if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp))) ptep = huge_pmd_share(mm, vma, addr, pudp); else ptep = (pte_t *)pmd_alloc(mm, pudp, addr); @@ -292,7 +292,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (sz != PUD_SIZE && pud_none(pud)) return NULL; /* hugepage or swap? */ diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 709e8ad15603..19492ef5940a 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -76,7 +76,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, bool early) { - if (pud_none(READ_ONCE(*pudp))) { + if (pud_none(pudp_get(pudp))) { phys_addr_t pmd_phys = early ? __pa_symbol(kasan_early_shadow_pmd) : kasan_alloc_zeroed_page(node); @@ -150,7 +150,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); kasan_pmd_populate(pudp, addr, next, node, early); - } while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp))); + } while (pudp++, addr = next, addr != end && pud_none(pudp_get(pudp))); } static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index c6300a1dc36a..ff677505c4d4 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -290,7 +290,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, { int ret; unsigned long next; - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get(pudp); pmd_t *pmdp; /* @@ -370,7 +370,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, } do { - pud_t old_pud = READ_ONCE(*pudp); + pud_t old_pud = pudp_get(pudp); next = pud_addr_end(addr, end); @@ -387,7 +387,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); + pud_val(pudp_get(pudp)))); } else { ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); @@ -395,7 +395,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, goto out; BUG_ON(pud_val(old_pud) != 0 && - pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); + pud_val(old_pud) != pud_val(pudp_get(pudp))); } phys += next - addr; } while (pudp++, addr = next, addr != end); @@ -1530,7 +1530,7 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) continue; @@ -1686,7 +1686,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) continue; @@ -1707,7 +1707,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, */ pudp = pud_offset(p4dp, 0UL); for (i = 0; i < PTRS_PER_PUD; i++) { - if (!pud_none(READ_ONCE(pudp[i]))) + if (!pud_none(pudp_get(pudp + i))) return; } @@ -1819,7 +1819,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); /* Only allow permission changes for now */ - if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)), + if (!pgattr_change_is_safe(pud_val(pudp_get(pudp)), pud_val(new_pud))) return 0; @@ -1850,7 +1850,7 @@ void p4d_clear_huge(p4d_t *p4dp) int pud_clear_huge(pud_t *pudp) { - if (!pud_leaf(READ_ONCE(*pudp))) + if (!pud_leaf(pudp_get(pudp))) return 0; pud_clear(pudp); return 1; @@ -1903,7 +1903,7 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) pud_t pud; unsigned long next, end; - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (!pud_table(pud)) { VM_WARN_ON(1); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index c0d7404c687a..1898e07595cf 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -407,7 +407,7 @@ bool kernel_page_present(struct page *page) return false; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) return false; if (pud_leaf(pud)) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index b27b2d2c20c3..d119119455f1 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -64,7 +64,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp, unsigned long next; unsigned long addr = start; - if (pud_none(READ_ONCE(*dst_pudp))) { + if (pud_none(pudp_get(dst_pudp))) { dst_pmdp = trans_alloc(info); if (!dst_pmdp) return -ENOMEM; @@ -109,7 +109,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp, src_pudp = pud_offset(src_p4dp, start); do { - pud_t pud = READ_ONCE(*src_pudp); + pud_t pud = pudp_get(src_pudp); next = pud_addr_end(addr, end); if (pud_none(pud)) -- 2.43.0