From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D52ED2D7DE9; Mon, 13 Jul 2026 14:02:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783951380; cv=none; b=IYC0QF2T7XmCzGJFgEgGGuO0TK3oigwkd9z7HpozPMcCOZbsthlTOLHxN6WQ4ByFbhGJPJl7OrDAhtea23g9HpHqJfWUZCfJxNo/0zn7F6qYxvzxsXqZUT/TNNVyKiS6ZN15CMD8igOpAox2BRHKUwmM6IK9FNaOGLmZ8owbKo0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783951380; c=relaxed/simple; bh=Wb3EDgPFs5yteOunBQrcCkQDf3F9lsz4u6v0rE/RQe8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ne2IfOYBw6RshuRJk6ltOM4DvWeOGLzCe+EX+7L4c1CF+y5AJO9wvZJwKRsDzndPc+0kUgee+lZnUe/FyCByk9VKvi1Un34TEp4iCqOH689OJfJeNS/lF0Q1PGJEDHN5HFKz8ndKWIe4GcRw37/xMxVJWvvDdwk+TUGIhSWpICM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=tWYyPQT/; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="tWYyPQT/" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F40312309; Mon, 13 Jul 2026 07:02:53 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.2.213.3]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0A2613F7B4; Mon, 13 Jul 2026 07:02:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1783951378; bh=Wb3EDgPFs5yteOunBQrcCkQDf3F9lsz4u6v0rE/RQe8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tWYyPQT/4oiojuix1Q3/fWMtFxNunqtzs/ViWRtTeS8smwkUIPeAKI7T5PLwKwHNU p/T/N5uvmTUBna5JDgL/6rvh2pM3O8oZ1iX8Z2vBH1HHz5yRfhmDcE0pm/IMf9wibZ XCsJzbBdBmYW60FV78EgDKmG1puDET54+GkAgMO8= From: Yeoreum Yun To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-arch@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, x86@kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org Cc: david@kernel.org, linux@armlinux.org.uk, akpm@linux-foundation.org, ankur.a.arora@oracle.com, rppt@kernel.org, linmag7@gmail.com, chleroy@kernel.org, klarasmodin@gmail.com, chenhuacai@kernel.org, kernel@xen0n.name, kas@kernel.org, zhangtianyang@loongson.cn, wangyuli@aosc.io, tsbogend@alpha.franken.de, ljs@kernel.org, jgg@ziepe.ca, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, ryan.roberts@arm.com, pasha.tatashin@soleen.com, rmclure@linux.ibm.com, baolin.wang@linux.alibaba.com, tj@kernel.org, kevin.brodsky@arm.com, anup@brainfault.org, atish.patra@linux.dev, pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@kernel.org, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, hannes@cmpxchg.org, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, kasong@tencent.com, baohua@kernel.org, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, anshuman.khandual@arm.com, yang@os.amperecomputing.com, chaitanyas.prakash@arm.com, ardb@kernel.org, guoren@kernel.org, yang.li85200@gmail.com, viro@zeniv.linux.org.uk, dinguyen@kernel.org, schuster.simon@siemens-energy.com, wangruikang@iscas.ac.cn, junhui.liu@pigmoral.tech, muchun.song@linux.dev, vishal.moola@gmail.com, namcao@linutronix.de, pavel@kernel.org, djbw@kernel.org, yu-cheng.yu@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, coxu@redhat.com, andreas@gaisler.com, liam@infradead.org, vbabka@kernel.org, surenb@google.com, mhocko@suse.com, geert@linux-m68k.org, shorne@gmail.com, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, Yeoreum Yun Subject: [RFC PATCH 31/34] mm/pgtable: optimize pmdp_get() and friends for folded pagetable levels Date: Mon, 13 Jul 2026 14:56:10 +0100 Message-ID: <20260713135614.1618183-32-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260713135614.1618183-1-yeoreum.yun@arm.com> References: <20260713135614.1618183-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-arch@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "David Hildenbrand (Arm)" Using pmdp_get() and friends in common code on a kernel config with folded page tables is suboptimal: they default to a READ_ONCE(), forcing the compiler to actually read that value even though it will not actually be used afterwards. This was recently reported by Christophe Leroy [1] and block conversion of more common code to pmdp_get() and friends. (using pgdp_get() as one example) Most of the code ignores the result from pgdp_get() on configs with folded page tables entirely, as we hardcode: pgd_present()==1 && pgd_leaf()==false Common code will just treat it as a "this is a page table" and call p4d_offset() or p4d_offset_lockless() for the next lower level, where we just ignore the obtained pgdp_get() result entirely. So we can just return a dummy value and avoid any memory reads. There is a catch, though: 1) If code calls pgd_val() and somehow relies on the data, it would now see dummy values. The code really must be aware of folded page table levels. Fortunately, code usually ignores pgd_val() completely for page tables (with ptdump being one exception when calculating effective permissions). We checked + fixed the x86 ptdump mechanism. 2) If code passes the pgd_t to a function that would work on the result, it would now see dummy values. The only concern is really passing the pgd_t on the stack as a pointer to p4d_offset(). Most code that would do that, should actually use p4d_offset_lockless(), which handles this properly. We checked + fixed problematic instances. As an example, this is the generated code for perf_get_page_size() with PGTABLE_LEVELS=3 on arm64: Before: 00000000000052a0 : ... 52dc: d53b4234 mrs x20, DAIF 52e0: d50343df msr DAIFSet, #0x3 ... 52fc: d35e9a69 ubfx x9, x19, #30, #9 /* pud_offset_lockless() */ 5300: f9403508 ldr x8, [x8, #0x68] 5304: f869790a ldr x10, [x8, x9, lsl #3] /* pudp_get() */ 5308: f90007ea str x10, [sp, #0x8] 530c: f8697908 ldr x8, [x8, x9, lsl #3] /* pudp_get() */ ... 5360: 90000009 adrp x9, 0x5000 5364: 92746908 and x8, x8, #0x7ffffff000 5368: d3557675 ubfx x21, x19, #21, #9 /* pmd_offset_lockless() */ ... 5394: f8757ac8 ldr x8, [x22, x21, lsl #3] /* pmdp_get() */ After: 0000000000052a0 : ... 52dc: d53b4234 mrs x20, DAIF 52e0: d50343df msr DAIFSet, #0x3 ... /* no pud_offset_lockless() and pudp_get() */ 5318: 90000009 adrp x9, 0x5000 531c: 92746908 and x8, x8, #0x7ffffff000 5320: d3557675 ubfx x21, x19, #21, #9 /* pmd_offset_lockless() */ ... 5334: f8757ac8 ldr x8, [x22, x21, lsl #3] /* pmdp_get() */ [1] https://lore.kernel.org/all/0019d675-ce3d-4a5c-89ed-f126c45145c9@kernel.org/ Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Yeoreum Yun --- include/asm-generic/pgtable-nop4d.h | 8 ++++++++ include/asm-generic/pgtable-nopmd.h | 8 ++++++++ include/asm-generic/pgtable-nopud.h | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h index 25c26a587b512..84a529df27ee3 100644 --- a/include/asm-generic/pgtable-nop4d.h +++ b/include/asm-generic/pgtable-nop4d.h @@ -31,6 +31,14 @@ static inline bool pgd_leaf(pgd_t pgd) { return false; } #define set_pgd(pgdptr, pgdval) BUILD_BUG() +static inline pgd_t pgdp_get(pgd_t *p4dp) +{ + pgd_t dummy = { 0 }; + + return dummy; +} +#define pgdp_get pgdp_get + static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) { return (p4d_t *)pgd; diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h index 75650fa9111cd..96f3fccd22fda 100644 --- a/include/asm-generic/pgtable-nopmd.h +++ b/include/asm-generic/pgtable-nopmd.h @@ -40,6 +40,14 @@ static inline void pud_clear(pud_t *pud) { } #define set_pud(pudptr, pudval) BUILD_BUG() +static inline pud_t pudp_get(pud_t *pudp) +{ + pud_t dummy = { 0 }; + + return dummy; +} +#define pudp_get pudp_get + static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address) { return (pmd_t *)pud; diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h index b592fff18887a..faa0233bcde80 100644 --- a/include/asm-generic/pgtable-nopud.h +++ b/include/asm-generic/pgtable-nopud.h @@ -38,6 +38,14 @@ static inline bool p4d_leaf(p4d_t p4d) { return false; } #define set_p4d(p4dptr, p4dval) BUILD_BUG() +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + p4d_t dummy = { 0 }; + + return dummy; +} +#define p4dp_get p4dp_get + static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) { return (pud_t *)p4d; -- LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}