[RFC PATCH 00/34] mm: optimize unnecessary loads due to ptep_get() and friends out

[RFC PATCH 00/34] mm: optimize unnecessary loads due to ptep_get() and friends out

[Yeoreum Yun]

Using ptep_get() and its counterparts in common code is suboptimal on
kernel configurations with generic compile-time folded page tables.
By default, ptep_get() and its friends expands to READ_ONCE(),
forcing the compiler to emit a load even when the value is not used afterwards.

This issue was recently reported by Christophe Leroy [1] for ppc32
preventing futher code conversion to ptep_get()/pmdp_get()/... helper
and the same behavior can also be observed on arm64 when built with
2- or 3-level page tables

e.g) perf_get_page_size() in arm64 with CONFIG_PGTABLE_LEVEL=3:

00000000000052a0 <perf_get_page_size>:
    ...
    52dc: d53b4234     	mrs	x20, DAIF
    52e0: d50343df     	msr	DAIFSet, #0x3
    ...
    52fc: d35e9a69     	ubfx	x9, x19, #30, #9        /* pud_offset_lockless() */
    5300: f9403508     	ldr	x8, [x8, #0x68]
    5304: f869790a     	ldr	x10, [x8, x9, lsl #3]   /* pudp_get() */
    5308: f90007ea     	str	x10, [sp, #0x8]
    530c: f8697908     	ldr	x8, [x8, x9, lsl #3]    /* pudp_get() */
    ...
    5360: 90000009     	adrp	x9, 0x5000 <perf_prepare_sample+0x548>
    5364: 92746908     	and	x8, x8, #0x7ffffff000
    5368: d3557675     	ubfx	x21, x19, #21, #9       /* pmd_offset_lockless() */
    ...
    5394: f8757ac8     	ldr	x8, [x22, x21, lsl #3]  /* pmdp_get() */

Though PGTABLE_LEVEL=3, since the pudp_get() still remain with
READ_ONCE(), there's redundant load for the pud which is folded.

To prevent generating suboptimal code, define dummy pXdp_get() and
pXd_offset_lockless() in the pgtable-nopXd.h to handle folded page tables
properly.

As the pXdp_get() can return *dummy* entry, some of code using
the stack value where saves the pXdp_get() could be a problematic:

  1. Passing address of stack value where saves the pXdp_get() result
     to pXd_offset() for example:

       pud_t *pudp, pud;
       pmd_t *pmdp;

       pud = pudp_get(pudp, address);
       pmdp = pmd_offset(&pud, pud, address);

     (e.g. host_pfn_mapping_level() in loongarch).

  2. Using the pXdp_get() result to use as argument of pXd_val() and
     to check prot without checking pgtable is folded.
     for example, x86's effective_prot().

  3. Using set_pXd() with pXdp_get() will set problematic dummy entry
     in folded page table like:

       set_pXd(pxdp, pXdp_get(pxdp_k));

     To prevent this, Let set_pXd() triggers the compile error
     to catch the wrong usage of set_pXd() in the generic compile-time
     folded page table and change the set_pgd() usage to set the
     first-level page table with the proper set_pXd().

  4. Using pgd_page_vaddr() to get the first-level pgtable.
     passing dummy pxdp_get() for pgd_page_vaddr() will return wrong
     address. Therefore, make pgd_page_vaddr() and pXd_pgtable() to
     trigger the error for improper usage in the generic compile-time
     folded pgtable.

Thanksfully, above cases are rare since (1) most of usage using
pXd_offset() with result of upper pXd_offset(), (2) it's extreamely
rare to use pXd_val() for non-leaf entry in the kernel,
(3) is to handle the vmalloc_fault or set the first level of page table
and (4) to setup early page table and etc.

Therefore, convert this kind of problematic rare pattern properly.

Furthermore, passing the ptep_get() (or its counterparts) as argument
directly to pte_present() and related helpers can generate suboptimal code,
in arm64 as the current macro implementation may evaluate its argument
more than once like:

  !pte_val(READ_ONCE(*pte) || pte_present_invalid(READ_ONCE(*pte))

resulting in redundant loads.

A typical example is pud_free_pmd_page(), where the expansion of
pmd_present() generates:
    ...
    /* pmd_present() (x20 = pmdp) */
    1b88: f9400288     ldr	x8, [x20]        // read pmdp.
    1b8c: f9000fa8     str	x8, [x29, #0x18]
    1b90: 3707fec8     tbnz	w8, #0x0, 0x1b68 <pud_free_pmd_page+0xd0>
    1b94: f9400288     ldr	x8, [x20]        // redundant read of pmdp.
    1b98: 8a170109     and	x9, x8, x23
    1b9c: f9000fa8     str	x8, [x29, #0x18]
    1ba0: f120013f     cmp	x9, #0x800
    1ba4: 54fffe20     b.eq	0x1b68 <pud_free_pmd_page+0xd0>
    1ba8: 17fffff4     b	0x1b78 <pud_free_pmd_page+0xe0>
    ...

To address this in arm64, convert pte_present() macro and its friends
to static inline function.

Future work
===========
 - print_bad_page_map() and show_pte() still prints dummy values
   instead of printing the same content for all generic compile-time
   folded page tables. We might want to skip printing dummy values later.

 - We currently catch abuse of dummy values on the stack at compile-time by
   relying on constant propagation by the compiler. Usama's work [3] on using
   distinct types for sw vs. hw PTEs could help here as well."

Link: [1] https://lore.kernel.org/all/0019d675-ce3d-4a5c-89ed-f126c45145c9@kernel.org/
Link: [2] https://lore.kernel.org/all/20251113014656.2605447-1-samuel.holland@sifive.com/
Link: [3] https://lore.kernel.org/r/74182e50-b54f-4d2d-a27f-3a59a538d6bc@arm.com

David Hildenbrand (Arm) (13):
  ARM: mm: make nommu pgd_t a scalar
  ARM: mm: make 2-level pgd_t a scalar
  ARM: mm: remove custom pgdp_get()
  LoongArch: mm: define pud_leaf() only when PUD exists
  MIPS: mm: define pud_leaf() only when PUD exists
  mm/pgtable: define (pgd|p4d|pud)_leaf() for folded page tables
  mm/pgtable: define (pgd|p4d|pud)_offset_lockless() for folded page
    tables
  x86: mm: carve out the generic compile-time folded pgtable case in
    effective_prot()
  mm/pgtable: disallow calling folded set_pgd/set_p4d/set_pud
  mm/pgtable: disallow calling folded (pgd|p4d|pud)_page,
    pgd_page_vaddr() and (p4d|pud)_pgtable
  mm/pgtable: optimize pmdp_get() and friends for folded pagetable
    levels
  openrisc/pgtable: drop __pmd_offset()
  mm/pgtable: catch abuse of folded dummy pgd_t/p4d_t/pud_t

Yeoreum Yun (21):
  loongarch: kvm: remove stack copy address of pXd in pXd_offset()
  riscv: kvm: remove stack copy address of pXd in pXd_offset()
  mm: vmscan: remove stack copy address of pud pass in wallk_pud_range()
  arm64: mm: use proper set_pXd() for generic compile-time folded
    patable in kasan_early_init()
  arm64: mm: define pud_set_huge() when __PGTABLE_PMD_FOLDED not defined
  csky: mm: use proper set_pXd() for generic compile-time folded patable
    in vmalloc_fault()
  mips: mm: use proper set_pXd() for generic compile-time folded patable
    in vmalloc_fault path
  nios2: mm: use proper set_pXd() for generic compile-time folded
    patable in vmalloc_fault path
  riscv: mm: use proper set_pXd() for generic compile-time folded
    patable in vmalloc_fault()
  riscv: mm: use proper set_pXd() for generic compile-time folded
    patable in setup_vm_final()
  x86: power: use proper set_pXd() for generic compile-time folded
    patable in resume_one_md_table_init()
  x86: kexec: use proper set_pXd() for generic compile-time folded
    patable in machine_kexec_page_table_set_one()
  x86: platform: use proper set_pXd() for generic compile-time folded
    patable in setup_olpc_ofw_pgd()
  x86: mm: use proper set_pXd() for generic compile-time folded patable
    in one_md_table_init()
  x86: mm: skip pud setup when using generic compile-time folded
    pagetable
  x86: mm: call try_to_free_pmd_page() when CONFIG_PGTABLE_LEVELS > 2
  x86: mm: remove usage of pgd_page_vaddr() for CONFIG_x86_PAE
  x86: mm: define pudp_set_access_flags() when
    CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD is enabled only.
  m68k: mm: remove usage of pgd_page_vaddr() for CONFIG_PGTABLE_LEVELS=3
  arm: mm: use proper pgtable APIs for generic compile-time folded
    patable in kasan_init()
  arm64: pgtable: convert pte_present() from macro to static inline

 arch/arm/include/asm/page-nommu.h           |  4 +-
 arch/arm/include/asm/pgtable-2level-types.h | 13 ++++--
 arch/arm/include/asm/pgtable.h              |  2 -
 arch/arm/mm/kasan_init.c                    |  8 ++--
 arch/arm64/include/asm/pgtable.h            | 35 ++++----------
 arch/arm64/mm/kasan_init.c                  | 26 ++++++++++-
 arch/arm64/mm/mmu.c                         |  2 +
 arch/csky/mm/fault.c                        | 35 +++++---------
 arch/loongarch/include/asm/pgtable.h        |  2 +
 arch/loongarch/kvm/mmu.c                    | 20 ++++----
 arch/m68k/mm/init.c                         |  2 +-
 arch/m68k/mm/motorola.c                     |  2 +-
 arch/mips/include/asm/pgtable.h             |  2 +
 arch/mips/mm/fault.c                        | 13 ++++--
 arch/nios2/mm/fault.c                       | 36 +++++---------
 arch/openrisc/include/asm/pgtable.h         |  3 --
 arch/riscv/kvm/mmu.c                        | 20 ++++----
 arch/riscv/mm/fault.c                       | 52 +++++++++++++--------
 arch/riscv/mm/init.c                        |  7 ++-
 arch/x86/kernel/machine_kexec_32.c          |  8 ++--
 arch/x86/mm/dump_pagetables.c               | 18 ++++++-
 arch/x86/mm/init_32.c                       | 10 ++--
 arch/x86/mm/pat/set_memory.c                |  9 ++--
 arch/x86/mm/pgtable.c                       | 43 ++++++++++-------
 arch/x86/platform/olpc/olpc_ofw.c           |  5 +-
 arch/x86/power/hibernate_32.c               | 14 ++----
 include/asm-generic/pgtable-nop4d.h         | 45 ++++++++++++++----
 include/asm-generic/pgtable-nopmd.h         | 46 +++++++++++++-----
 include/asm-generic/pgtable-nopud.h         | 46 ++++++++++++++----
 include/linux/pgtable.h                     |  5 ++
 mm/vmscan.c                                 |  2 +-
 31 files changed, 329 insertions(+), 206 deletions(-)


base-commit: bdc38bfc1262e3d1432afadd2aa2ffd83d139dbb
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 01/34] ARM: mm: make nommu pgd_t a scalar

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

We don't want pgd_t to be an array, as it prohibits returning it from a
function, like ptep_get().

There is no need to match the actual arm page tables, because with nommu
there are no page tables. It's all just in place to make the compiler
happy. Making it a scalar will make the compiler happy.

So let's just use an u32.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm/include/asm/page-nommu.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm/include/asm/page-nommu.h b/arch/arm/include/asm/page-nommu.h
index e74415c959bea..88659b38e2364 100644
--- a/arch/arm/include/asm/page-nommu.h
+++ b/arch/arm/include/asm/page-nommu.h
@@ -18,12 +18,12 @@
  */
 typedef unsigned long pte_t;
 typedef unsigned long pmd_t;
-typedef unsigned long pgd_t[2];
+typedef unsigned long pgd_t;
 typedef unsigned long pgprot_t;
 
 #define pte_val(x)      (x)
 #define pmd_val(x)      (x)
-#define pgd_val(x)	((x)[0])
+#define pgd_val(x)      (x)
 #define pgprot_val(x)   (x)
 
 #define __pte(x)        (x)
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 02/34] ARM: mm: make 2-level pgd_t a scalar

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

We don't want pgd_t to be an array, as it prohibits returning it from a
function, like pgdp_get().

So let's just use an u64, and extract the right 32bit value in
pgd_val().

Leave the STRICT_MM_TYPECHECKS case alone for now.

As an alternative, we could use the STRICT_MM_TYPECHECKS approach here
as well, but using an u64 looks conceptually cleaner, even though
pgd_val() gets a bit more involved.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm/include/asm/pgtable-2level-types.h | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/arch/arm/include/asm/pgtable-2level-types.h b/arch/arm/include/asm/pgtable-2level-types.h
index 650e793f41429..02052cef9437a 100644
--- a/arch/arm/include/asm/pgtable-2level-types.h
+++ b/arch/arm/include/asm/pgtable-2level-types.h
@@ -25,7 +25,7 @@ typedef struct { pteval_t pgprot; } pgprot_t;
 
 #define pte_val(x)      ((x).pte)
 #define pmd_val(x)      ((x).pmd)
-#define pgd_val(x)	((x).pgd[0])
+#define pgd_val(x)      ((x).pgd[0])
 #define pgprot_val(x)   ((x).pgprot)
 
 #define __pte(x)        ((pte_t) { (x) } )
@@ -36,14 +36,21 @@ typedef struct { pteval_t pgprot; } pgprot_t;
 /*
  * .. while these make it easier on the compiler
  */
+typedef u64 pgdval_t;
+
 typedef pteval_t pte_t;
 typedef pmdval_t pmd_t;
-typedef pmdval_t pgd_t[2];
+typedef pgdval_t pgd_t;
 typedef pteval_t pgprot_t;
 
 #define pte_val(x)      (x)
 #define pmd_val(x)      (x)
-#define pgd_val(x)	((x)[0])
+
+static inline pmdval_t pgd_val(pgd_t pgd)
+{
+	return (*(pmdval_t (*)[2])&pgd)[0];
+}
+
 #define pgprot_val(x)   (x)
 
 #define __pte(x)        (x)
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 03/34] ARM: mm: remove custom pgdp_get()

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

Now that pgd_t is no longer an array, we can just rely on the common
code pgdp_get(), which is a static inline function that returns pgd_t.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm/include/asm/pgtable.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index 982795cf45637..eadd027fe0ee4 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -141,8 +141,6 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
 
 extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
 
-#define pgdp_get(pgpd)		READ_ONCE(*pgdp)
-
 #define pud_page(pud)		pmd_page(__pmd(pud_val(pud)))
 #define pud_write(pud)		pmd_write(__pmd(pud_val(pud)))
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 04/34] LoongArch: mm: define pud_leaf() only when PUD exists

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

In include/asm-generic/pgtable-nopmd.h, we do have a hard-coded
pud_leaf()=false inline function, but we miss the "define pud_leaf
pud_leaf" part.

To prepare for fixing that, define pud_leaf() only if we don't have
__PAGETABLE_PMD_FOLDED.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/loongarch/include/asm/pgtable.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h
index 1952e34bc8ee0..434e5b4834664 100644
--- a/arch/loongarch/include/asm/pgtable.h
+++ b/arch/loongarch/include/asm/pgtable.h
@@ -636,7 +636,9 @@ static inline long pmd_protnone(pmd_t pmd)
 #endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
 
 #define pmd_leaf(pmd)		((pmd_val(pmd) & _PAGE_HUGE) != 0)
+#ifndef __PAGETABLE_PMD_FOLDED
 #define pud_leaf(pud)		((pud_val(pud) & _PAGE_HUGE) != 0)
+#endif
 
 /*
  * We provide our own get_unmapped area to cope with the virtual aliasing
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 05/34] MIPS: mm: define pud_leaf() only when PUD exists

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

In include/asm-generic/pgtable-nopmd.h, we do have a hard-coded
pud_leaf()=false inline function, but we miss the "define pud_leaf
pud_leaf" part.

 To prepare for fixing that, define pud_leaf() only if we don't have
 __PAGETABLE_PMD_FOLDED.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/mips/include/asm/pgtable.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h
index fa7b935f947ca..151f4d15bd12b 100644
--- a/arch/mips/include/asm/pgtable.h
+++ b/arch/mips/include/asm/pgtable.h
@@ -745,8 +745,10 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
 
 #ifdef _PAGE_HUGE
 #define pmd_leaf(pmd)	((pmd_val(pmd) & _PAGE_HUGE) != 0)
+#ifndef __PAGETABLE_PMD_FOLDED
 #define pud_leaf(pud)	((pud_val(pud) & _PAGE_HUGE) != 0)
 #endif
+#endif
 
 #define gup_fast_permitted(start, end)	(!cpu_has_dc_aliases)
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 06/34] mm/pgtable: define (pgd|p4d|pud)_leaf() for folded page tables

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

Let's define (pgd|p4d|pud)_leaf(), hard-coding it to "false". Note
that we missed to define pud_leaf() before, allowing architectures to
unknowingly overwrite it.

Still use static inline functions (type checking), and while at it, just
use "bool" as a return value.

We can now drop the arm64 custom variant that did that. As it documents:

	Note: reusing the original pointer means that we may
	dereference the same (live) page-table entry multiple times.
	This is safe because it is still only loaded once in the
	context of each level and the CPU guarantees same-address
	read-after-read ordering.

We now have the same situation. Don't document that, as we will be
removing the double read next.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm64/include/asm/pgtable.h    | 23 -----------------------
 include/asm-generic/pgtable-nop4d.h |  2 ++
 include/asm-generic/pgtable-nopmd.h |  3 ++-
 include/asm-generic/pgtable-nopud.h |  2 ++
 4 files changed, 6 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a2681d7553584..6185fc291fd7d 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1188,29 +1188,6 @@ static inline bool pgtable_l5_enabled(void) { return false; }
 #define p4d_clear_fixmap()
 
 #define p4d_offset_kimg(dir,addr)	((p4d_t *)dir)
-
-static inline
-p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t pgd, unsigned long addr)
-{
-	/*
-	 * With runtime folding of the pud, pud_offset_lockless() passes
-	 * the 'pgd_t *' we return here to p4d_to_folded_pud(), which
-	 * will offset the pointer assuming that it points into
-	 * a page-table page. However, the fast GUP path passes us a
-	 * pgd_t allocated on the stack and so we must use the original
-	 * pointer in 'pgdp' to construct the p4d pointer instead of
-	 * using the generic p4d_offset_lockless() implementation.
-	 *
-	 * Note: reusing the original pointer means that we may
-	 * dereference the same (live) page-table entry multiple times.
-	 * This is safe because it is still only loaded once in the
-	 * context of each level and the CPU guarantees same-address
-	 * read-after-read ordering.
-	 */
-	return p4d_offset(pgdp, addr);
-}
-#define p4d_offset_lockless p4d_offset_lockless_folded
-
 #endif  /* CONFIG_PGTABLE_LEVELS > 4 */
 
 #define pgd_ERROR(e)	\
diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index 89c21f84cffbe..c6a5a43899b50 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -22,6 +22,8 @@ static inline int pgd_none(pgd_t pgd)		{ return 0; }
 static inline int pgd_bad(pgd_t pgd)		{ return 0; }
 static inline int pgd_present(pgd_t pgd)	{ return 1; }
 static inline void pgd_clear(pgd_t *pgd)	{ }
+static inline bool pgd_leaf(pgd_t pgd)		{ return false; }
+#define pgd_leaf pgd_leaf
 #define p4d_ERROR(p4d)				(pgd_ERROR((p4d).pgd))
 
 #define pgd_populate(mm, pgd, p4d)		do { } while (0)
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 36b6490ed1808..dbd38b4c3a056 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -31,7 +31,8 @@ static inline int pud_none(pud_t pud)		{ return 0; }
 static inline int pud_bad(pud_t pud)		{ return 0; }
 static inline int pud_present(pud_t pud)	{ return 1; }
 static inline int pud_user(pud_t pud)		{ return 0; }
-static inline int pud_leaf(pud_t pud)		{ return 0; }
+static inline bool pud_leaf(pud_t pud)		{ return false; }
+#define pud_leaf pud_leaf
 static inline void pud_clear(pud_t *pud)	{ }
 #define pmd_ERROR(pmd)				(pud_ERROR((pmd).pud))
 
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index 356cbfbaab247..6c9bca78047c4 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -29,6 +29,8 @@ static inline int p4d_none(p4d_t p4d)		{ return 0; }
 static inline int p4d_bad(p4d_t p4d)		{ return 0; }
 static inline int p4d_present(p4d_t p4d)	{ return 1; }
 static inline void p4d_clear(p4d_t *p4d)	{ }
+static inline bool p4d_leaf(p4d_t p4d)		{ return false; }
+#define p4d_leaf p4d_leaf
 #define pud_ERROR(pud)				(p4d_ERROR((pud).p4d))
 
 #define p4d_populate(mm, p4d, pud)		do { } while (0)
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 07/34] mm/pgtable: define (pgd|p4d|pud)_offset_lockless() for folded page tables

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

The default helpers fallback to pud_offset(), passing a pointer to the
stack value.

Let's provide variants that do exactly what (pgd|p4d|pud)_offset() do,
but ignore the passed pgd_t/p4d_t/pud_t, consequently not working on a
pointer to the stack value like the default handlers would.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 include/asm-generic/pgtable-nop4d.h | 7 +++++++
 include/asm-generic/pgtable-nopmd.h | 7 +++++++
 include/asm-generic/pgtable-nopud.h | 7 +++++++
 3 files changed, 21 insertions(+)

diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index c6a5a43899b50..019c3f074b771 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -39,6 +39,13 @@ static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
 	return (p4d_t *)pgd;
 }
 
+static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd,
+		unsigned long address)
+{
+	return (p4d_t *)pgdp;
+}
+#define p4d_offset_lockless p4d_offset_lockless
+
 #define p4d_val(x)				(pgd_val((x).pgd))
 #define __p4d(x)				((p4d_t) { __pgd(x) })
 
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index dbd38b4c3a056..ae2eff44889ac 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -50,6 +50,13 @@ static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address)
 }
 #define pmd_offset pmd_offset
 
+static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud,
+		unsigned long address)
+{
+	return (pmd_t *)pudp;
+}
+#define pmd_offset_lockless pmd_offset_lockless
+
 #define pmd_val(x)				(pud_val((x).pud))
 #define __pmd(x)				((pmd_t) { __pud(x) } )
 
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index 6c9bca78047c4..5a2b0a81ae197 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -47,6 +47,13 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
 }
 #define pud_offset pud_offset
 
+static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d,
+		unsigned long address)
+{
+	return (pud_t *)p4dp;
+}
+#define pud_offset_lockless pud_offset_lockless
+
 #define pud_val(x)				(p4d_val((x).p4d))
 #define __pud(x)				((pud_t) { __p4d(x) })
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 08/34] loongarch: kvm: remove stack copy address of pXd in pXd_offset()

[Yeoreum Yun]

We want to change how pXdp_get() works with generic compile-time folded
page tables. To prepare for that, rework host_pfn_mapping_level() to avoid
use of a stack copy of a pXd and pass it as argument to pXd_offset() so that
the folded entries value is ignored.

Replace direct uses of stack-based pXd values with pXd_offset_lockless()
and there should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/loongarch/kvm/mmu.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/arch/loongarch/kvm/mmu.c b/arch/loongarch/kvm/mmu.c
index e104897aa5328..837b40bb694f4 100644
--- a/arch/loongarch/kvm/mmu.c
+++ b/arch/loongarch/kvm/mmu.c
@@ -669,10 +669,10 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 	int level = 0;
 	unsigned long hva;
 	unsigned long flags;
-	pgd_t pgd;
-	p4d_t p4d;
-	pud_t pud;
-	pmd_t pmd;
+	pgd_t *pgdp, pgd;
+	p4d_t *p4dp, p4d;
+	pud_t *pudp, pud;
+	pmd_t *pmdp, pmd;
 
 	/*
 	 * Note, using the already-retrieved memslot and __gfn_to_hva_memslot()
@@ -698,19 +698,23 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 	 * value) and then p*d_offset() walks into the target huge page instead
 	 * of the old page table (sees the new value).
 	 */
-	pgd = pgdp_get(pgd_offset(kvm->mm, hva));
+	pgdp = pgd_offset(kvm->mm, hva);
+	pgd = pgdp_get(pgdp);
 	if (pgd_none(pgd))
 		goto out;
 
-	p4d = p4dp_get(p4d_offset(&pgd, hva));
+	p4dp = p4d_offset_lockless(pgdp, pgd, hva);
+	p4d = p4dp_get(p4dp);
 	if (p4d_none(p4d) || !p4d_present(p4d))
 		goto out;
 
-	pud = pudp_get(pud_offset(&p4d, hva));
+	pudp = pud_offset_lockless(p4dp, p4d, hva);
+	pud = pudp_get(pudp);
 	if (pud_none(pud) || !pud_present(pud))
 		goto out;
 
-	pmd = pmdp_get(pmd_offset(&pud, hva));
+	pmdp = pmd_offset_lockless(pudp, pud, hva);
+	pmd = pmdp_get(pmdp);
 	if (pmd_none(pmd) || !pmd_present(pmd))
 		goto out;
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 09/34] riscv: kvm: remove stack copy address of pXd in pXd_offset()

[Yeoreum Yun]

We want to change how pXdp_get() works with generic compile-time folded
page tables. To prepare for that, rework get_hva_mapping_size() to avoid
use of a stack copy of a pXd and pass it as argument to pXd_offset() so that
the folded entries value is ignored.

Replace direct uses of stack-based pXd values with pXd_offset_lockless()
and there should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/riscv/kvm/mmu.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 082f9b2617338..6d6042b81d790 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -362,10 +362,10 @@ static int get_hva_mapping_size(struct kvm *kvm,
 {
 	int size = PAGE_SIZE;
 	unsigned long flags;
-	pgd_t pgd;
-	p4d_t p4d;
-	pud_t pud;
-	pmd_t pmd;
+	pgd_t *pgdp, pgd;
+	p4d_t *p4dp, p4d;
+	pud_t *pudp, pud;
+	pmd_t *pmdp, pmd;
 
 	/*
 	 * Disable IRQs to prevent concurrent tear down of host page tables,
@@ -381,15 +381,18 @@ static int get_hva_mapping_size(struct kvm *kvm,
 	 * value) and then p*d_offset() walks into the target huge page instead
 	 * of the old page table (sees the new value).
 	 */
-	pgd = pgdp_get(pgd_offset(kvm->mm, hva));
+	pgdp = pgd_offset(kvm->mm, hva);
+	pgd = pgdp_get(pgdp);
 	if (pgd_none(pgd))
 		goto out;
 
-	p4d = p4dp_get(p4d_offset(&pgd, hva));
+	p4dp = p4d_offset_lockless(pgdp, pgd, hva);
+	p4d = p4dp_get(p4dp);
 	if (p4d_none(p4d) || !p4d_present(p4d))
 		goto out;
 
-	pud = pudp_get(pud_offset(&p4d, hva));
+	pudp = pud_offset_lockless(p4dp, p4d, hva);
+	pud = pudp_get(pudp);
 	if (pud_none(pud) || !pud_present(pud))
 		goto out;
 
@@ -398,7 +401,8 @@ static int get_hva_mapping_size(struct kvm *kvm,
 		goto out;
 	}
 
-	pmd = pmdp_get(pmd_offset(&pud, hva));
+	pmdp = pmd_offset_lockless(pudp, pud, hva);
+	pmd = pmdp_get(pmdp);
 	if (pmd_none(pmd) || !pmd_present(pmd))
 		goto out;
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

We want to change how pXdp_get() works with generic compile-time folded
page tables. To prepare for that, rework effective_prot() to ignore any
folded page tables entries, as it will unconditionally get called by
ptdump core with pXd_val(pXdp_get()), and we really should be ignoring
that value for folded entries.

For this, identify the first real page-table level and
update effective_prot only when the current entry is not folded.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/x86/mm/dump_pagetables.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/arch/x86/mm/dump_pagetables.c b/arch/x86/mm/dump_pagetables.c
index 2afa7a23340e9..e32848c7f26d4 100644
--- a/arch/x86/mm/dump_pagetables.c
+++ b/arch/x86/mm/dump_pagetables.c
@@ -27,6 +27,7 @@
  */
 struct pg_state {
 	struct ptdump_state ptdump;
+	struct mm_struct *mm;
 	int level;
 	pgprotval_t current_prot;
 	pgprotval_t effective_prot;
@@ -253,8 +254,22 @@ static void effective_prot(struct ptdump_state *pt_st, int level, u64 val)
 	struct pg_state *st = container_of(pt_st, struct pg_state, ptdump);
 	pgprotval_t prot = val & PTE_FLAGS_MASK;
 	pgprotval_t effective;
+	bool first_level = false;
 
-	if (level > 0) {
+	/* Ignore folded levels ... */
+	if (((level == 0) && mm_p4d_folded(st->mm)) ||
+	    ((level == 1) && mm_pud_folded(st->mm)) ||
+	    ((level == 2) && mm_pmd_folded(st->mm)))
+		return;
+
+	/* ... and make the actual first level remember the protection. */
+	if (((level == 0)) ||
+	    ((level == 1) && mm_p4d_folded(st->mm)) ||
+	    ((level == 2) && mm_pud_folded(st->mm)) ||
+	    ((level == 3) && mm_pmd_folded(st->mm)))
+		first_level = true;
+
+	if (!first_level) {
 		pgprotval_t higher_prot = st->prot_levels[level - 1];
 
 		effective = (higher_prot & prot & (_PAGE_USER | _PAGE_RW)) |
@@ -449,6 +464,7 @@ bool ptdump_walk_pgd_level_core(struct seq_file *m,
 			.effective_prot_pgd = effective_prot_pgd,
 			.range		= ptdump_ranges
 		},
+		.mm		= mm,
 		.level = -1,
 		.to_dmesg	= dmesg,
 		.check_wx	= checkwx,
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 11/34] mm: vmscan: remove stack copy address of pud pass in wallk_pud_range()

[Yeoreum Yun]

We want to change how pXdp_get() works with generic compile-time folded
page tables. To prepare for that, replace passing the address of a stack
copy of a pXd to walk_pmd_range() with pud pointer directly so that
the ignoring the value of folded entries.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 mm/vmscan.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 986dde8e7429f..61470223d8213 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3725,7 +3725,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
 		if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val)))
 			continue;
 
-		walk_pmd_range(&val, addr, next, args);
+		walk_pmd_range(pud + i, addr, next, args);
 
 		if (need_resched() || walk->batched >= MAX_LRU_BATCH) {
 			end = (addr | ~PUD_MASK) + 1;
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 13/34] arm64: mm: define pud_set_huge() when __PGTABLE_PMD_FOLDED not defined

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level set_pXd()
function is used instead.

As using set_pud() is invalid when __PGTABLE_PMD_FOLDED,
pud_set_huge() is also invalid. Therefore, define it when
__PGTABLE_PMD_FOLDED is not defined.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm64/mm/mmu.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index f2be501468ce5..194b62c528de9 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1807,6 +1807,7 @@ void vmemmap_free(unsigned long start, unsigned long end,
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
 
+#ifndef __PAGETABLE_PMD_FOLDED
 int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot)
 {
 	pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot));
@@ -1820,6 +1821,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot)
 	set_pud(pudp, new_pud);
 	return 1;
 }
+#endif
 
 int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
 {
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 14/34] csky: mm: use proper set_pXd() for generic compile-time folded patable in vmalloc_fault()

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time
folded page tables by disallowing its use and triggering a
compile-time error when it is used improperly, ensuring that the actual
first-level set_pXd() function is used instead.

Since csky's PGTABLE_LEVELS always is 2, use set_pmd() instead of
set_pgd() to handle vmalloc_fault().

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/csky/mm/fault.c | 35 +++++++++++------------------------
 1 file changed, 11 insertions(+), 24 deletions(-)

diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7ff4011089850..1467ea963806c 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -114,12 +114,12 @@ static inline void bad_area_nosemaphore(struct pt_regs *regs, struct mm_struct *
 
 static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
 {
-	pgd_t *pgd, *pgd_k;
-	pud_t *pud, *pud_k;
-	pmd_t *pmd, *pmd_k;
-	pte_t *pte_k;
+	pmd_t *pmdp, *pmdp_k, pmd_k;
+	pte_t *ptep_k;
 	int offset;
 
+	BUILD_BUG_ON(CONFIG_PGTABLE_LEVELS != 2);
+
 	/* User mode accesses just cause a SIGSEGV */
 	if (user_mode(regs)) {
 		do_trap(regs, SIGSEGV, code, addr);
@@ -135,32 +135,19 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
 	 */
 	offset = pgd_index(addr);
 
-	pgd = get_pgd() + offset;
-	pgd_k = init_mm.pgd + offset;
-
-	if (!pgd_present(*pgd_k)) {
-		no_context(regs, addr);
-		return;
-	}
-	set_pgd(pgd, *pgd_k);
+	pmdp = (pmd_t *)(get_pgd() + offset);
+	pmdp_k = (pmd_t *)(init_mm.pgd + offset);
 
-	pud = (pud_t *)pgd;
-	pud_k = (pud_t *)pgd_k;
-	if (!pud_present(*pud_k)) {
+	pmd_k = *pmdp_k;
+	if (!pmd_present(pmd_k)) {
 		no_context(regs, addr);
 		return;
 	}
 
-	pmd = pmd_offset(pud, addr);
-	pmd_k = pmd_offset(pud_k, addr);
-	if (!pmd_present(*pmd_k)) {
-		no_context(regs, addr);
-		return;
-	}
-	set_pmd(pmd, *pmd_k);
+	set_pmd(pmdp, pmd_k);
 
-	pte_k = pte_offset_kernel(pmd_k, addr);
-	if (!pte_present(*pte_k)) {
+	ptep_k = pte_offset_kernel(pmdp_k, addr);
+	if (!pte_present(*ptep_k)) {
 		no_context(regs, addr);
 		return;
 	}
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 15/34] mips: mm: use proper set_pXd() for generic compile-time folded patable in vmalloc_fault path

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic comiple-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level
set_pXd() function is used instead.

Therefore, call the proper set_pXd() in case of generic compile-time
folded pgtable to handle vmalloc_fault case with pXd_present() and
drop pgd_present() and set_pgd() usage since mips doesn't support
5 page-table levels.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/mips/mm/fault.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 37fedeaca2e9a..4ad281a3bf97a 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -289,28 +289,31 @@ static void __do_page_fault(struct pt_regs *regs, unsigned long write,
 		pmd_t *pmd, *pmd_k;
 		pte_t *pte_k;
 
+		BUILD_BUG_ON(CONFIG_PGTABLE_LEVELS > 4);
+
 		pgd = (pgd_t *) pgd_current[raw_smp_processor_id()] + offset;
 		pgd_k = init_mm.pgd + offset;
 
-		if (!pgd_present(*pgd_k))
-			goto no_context;
-		set_pgd(pgd, *pgd_k);
-
 		p4d = p4d_offset(pgd, address);
 		p4d_k = p4d_offset(pgd_k, address);
 		if (!p4d_present(*p4d_k))
 			goto no_context;
+		if (CONFIG_PGTABLE_LEVELS == 4)
+			set_p4d(p4d, *p4d_k);
 
 		pud = pud_offset(p4d, address);
 		pud_k = pud_offset(p4d_k, address);
 		if (!pud_present(*pud_k))
 			goto no_context;
+		if (CONFIG_PGTABLE_LEVELS == 3)
+			set_pud(pud, *pud_k);
 
 		pmd = pmd_offset(pud, address);
 		pmd_k = pmd_offset(pud_k, address);
 		if (!pmd_present(*pmd_k))
 			goto no_context;
-		set_pmd(pmd, *pmd_k);
+		if (CONFIG_PGTABLE_LEVELS == 2)
+			set_pmd(pmd, *pmd_k);
 
 		pte_k = pte_offset_kernel(pmd_k, address);
 		if (!pte_present(*pte_k))
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 16/34] nios2: mm: use proper set_pXd() for generic compile-time folded patable in vmalloc_fault path

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level set_pXd()
function is used instead.

Since nios2's PGTABLE_LEVELS is 2 always, usage set_pmd() instead of
set_pgd() to handle vmalloc_fault case.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/nios2/mm/fault.c | 36 ++++++++++++------------------------
 1 file changed, 12 insertions(+), 24 deletions(-)

diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index e3fa9c15181df..f67d599115e9e 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -227,35 +227,23 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
 		 * an interrupt in the middle of a task switch..
 		 */
 		int offset = pgd_index(address);
-		pgd_t *pgd, *pgd_k;
-		p4d_t *p4d, *p4d_k;
-		pud_t *pud, *pud_k;
-		pmd_t *pmd, *pmd_k;
-		pte_t *pte_k;
 
-		pgd = pgd_current + offset;
-		pgd_k = init_mm.pgd + offset;
+		BUILD_BUG_ON(CONFIG_PGTABLE_LEVELS != 2);
 
-		if (!pgd_present(*pgd_k))
-			goto no_context;
-		set_pgd(pgd, *pgd_k);
+		pmd_t *pmdp, *pmdp_k, pmd_k;
+		pte_t *ptep_k;
 
-		p4d = p4d_offset(pgd, address);
-		p4d_k = p4d_offset(pgd_k, address);
-		if (!p4d_present(*p4d_k))
-			goto no_context;
-		pud = pud_offset(p4d, address);
-		pud_k = pud_offset(p4d_k, address);
-		if (!pud_present(*pud_k))
-			goto no_context;
-		pmd = pmd_offset(pud, address);
-		pmd_k = pmd_offset(pud_k, address);
-		if (!pmd_present(*pmd_k))
+		pmdp = (pmd_t *)(pgd_current + offset);
+		pmdp_k = (pmd_t *)(init_mm.pgd + offset);
+		pmd_k = *pmdp_k;
+
+		if (!pmd_present(pmd_k))
 			goto no_context;
-		set_pmd(pmd, *pmd_k);
 
-		pte_k = pte_offset_kernel(pmd_k, address);
-		if (!pte_present(*pte_k))
+		set_pmd(pmdp, pmd_k);
+
+		ptep_k = pte_offset_kernel(pmdp_k, address);
+		if (!pte_present(*ptep_k))
 			goto no_context;
 
 		flush_tlb_kernel_page(address);
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 17/34] riscv: mm: use proper set_pXd() for generic compile-time folded patable in vmalloc_fault()

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time
folded page tables by disallowing its use and triggering a compile-time
error when it is used improperly, ensuring that the actual first-level
set_pXd() function is used instead.

Therefore, call the proper set_pXd() in case of generic compile-time
folded pgtable to handle vmalloc_fault() with pXd_present().

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/riscv/mm/fault.c | 52 +++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 19 deletions(-)

diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4f..b2bcaf10d13f1 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -167,14 +167,16 @@ bad_area(struct pt_regs *regs, struct mm_struct *mm, int code,
 
 static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
 {
-	pgd_t *pgd, *pgd_k;
-	pud_t *pud_k;
-	p4d_t *p4d_k;
-	pmd_t *pmd_k;
-	pte_t *pte_k;
+	pgd_t *pgdp_k, *pgdp, pgd_k;
+	p4d_t *p4dp_k, *p4dp;
+	pud_t *pudp_k, *pudp, pud_k;
+	pmd_t *pmdp_k, *pmdp, pmd_k;
+	pte_t *ptep_k;
 	int index;
 	unsigned long pfn;
 
+	BUILD_BUG_ON(CONFIG_PGTABLE_LEVELS != 5 && CONFIG_PGTABLE_LEVELS != 2);
+
 	/* User mode accesses just cause a SIGSEGV */
 	if (user_mode(regs))
 		return do_trap(regs, SIGSEGV, code, addr);
@@ -189,39 +191,51 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
 	 */
 	index = pgd_index(addr);
 	pfn = csr_read(CSR_SATP) & SATP_PPN;
-	pgd = (pgd_t *)pfn_to_virt(pfn) + index;
-	pgd_k = init_mm.pgd + index;
 
-	if (!pgd_present(pgdp_get(pgd_k))) {
+	pgdp = (pgd_t *)pfn_to_virt(pfn) + index;
+	pgdp_k = init_mm.pgd + index;
+
+	pgd_k = pgdp_get(pgdp_k);
+	if (!pgd_present(pgd_k)) {
 		no_context(regs, addr);
 		return;
 	}
-	set_pgd(pgd, pgdp_get(pgd_k));
+	if (CONFIG_PGTABLE_LEVELS == 5)
+		set_pgd(pgdp, pgd_k);
 
-	p4d_k = p4d_offset(pgd_k, addr);
-	if (!p4d_present(p4dp_get(p4d_k))) {
+	p4dp = p4d_offset(pgdp, addr);
+	p4dp_k = p4d_offset(pgdp_k, addr);
+	if (!p4d_present(p4dp_get(p4dp_k))) {
 		no_context(regs, addr);
 		return;
 	}
 
-	pud_k = pud_offset(p4d_k, addr);
-	if (!pud_present(pudp_get(pud_k))) {
+	pudp = pud_offset(p4dp, addr);
+	pudp_k = pud_offset(p4dp_k, addr);
+
+	pud_k = pudp_get(pudp_k);
+	if (!pud_present(pud_k)) {
 		no_context(regs, addr);
 		return;
 	}
-	if (pud_leaf(pudp_get(pud_k)))
+	if (pud_leaf(pud_k))
 		goto flush_tlb;
 
 	/*
 	 * Since the vmalloc area is global, it is unnecessary
 	 * to copy individual PTEs
 	 */
-	pmd_k = pmd_offset(pud_k, addr);
-	if (!pmd_present(pmdp_get(pmd_k))) {
+	pmdp = pmd_offset(pudp, addr);
+	pmdp_k = pmd_offset(pudp_k, addr);
+
+	pmd_k = pmdp_get(pmdp_k);
+	if (!pmd_present(pmd_k)) {
 		no_context(regs, addr);
 		return;
 	}
-	if (pmd_leaf(pmdp_get(pmd_k)))
+	if (CONFIG_PGTABLE_LEVELS == 2)
+		set_pmd(pmdp, pmd_k);
+	if (pmd_leaf(pmd_k))
 		goto flush_tlb;
 
 	/*
@@ -230,8 +244,8 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
 	 * addresses. If we don't do this, this will just
 	 * silently loop forever.
 	 */
-	pte_k = pte_offset_kernel(pmd_k, addr);
-	if (!pte_present(ptep_get(pte_k))) {
+	ptep_k = pte_offset_kernel(pmdp_k, addr);
+	if (!pte_present(ptep_get(ptep_k))) {
 		no_context(regs, addr);
 		return;
 	}
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 18/34] riscv: mm: use proper set_pXd() for generic compile-time folded patable in setup_vm_final()

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level
set_pXd() function is used instead.

Therefore, replace set_pgd() with set_pmd() to setup swapper_pg_dir
in setup_vm_final() since PGTABLE_LEVELS is always 2 in CONFIG_32BIT.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/riscv/mm/init.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index 5b1b3c88b4d13..78017edf7c343 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -1264,7 +1264,12 @@ static void __init setup_vm_final(void)
 	 */
 	unsigned long idx = pgd_index(__fix_to_virt(FIX_FDT));
 
-	set_pgd(&swapper_pg_dir[idx], early_pg_dir[idx]);
+	BUILD_BUG_ON (CONFIG_PGTABLE_LEVELS != 2);
+
+	pmd_t *pmdp_s = (void *)&swapper_pg_dir[idx];
+	pmd_t *pmdp = (void *)&early_pg_dir[idx];
+
+	set_pmd(pmdp_s, pmdp_get(pmdp));
 #endif
 	create_pgd_mapping(swapper_pg_dir, FIXADDR_START,
 			   __pa_symbol(fixmap_pgd_next),
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 19/34] x86: power: use proper set_pXd() for generic compile-time folded patable in resume_one_md_table_init()

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level set_pXd()
function is used instead.

Therefore, replace set_pgd() with set_pud() to setup first-level pgtable
in case of X86_PAE.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/x86/power/hibernate_32.c | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/arch/x86/power/hibernate_32.c b/arch/x86/power/hibernate_32.c
index 223d5bca29b86..ab5a203b45cb3 100644
--- a/arch/x86/power/hibernate_32.c
+++ b/arch/x86/power/hibernate_32.c
@@ -29,8 +29,7 @@ pgd_t *resume_pg_dir;
  */
 static pmd_t *resume_one_md_table_init(pgd_t *pgd)
 {
-	p4d_t *p4d;
-	pud_t *pud;
+	pud_t *pud = pud_offset(p4d_offset(pgd, 0), 0);
 	pmd_t *pmd_table;
 
 #ifdef CONFIG_X86_PAE
@@ -38,14 +37,10 @@ static pmd_t *resume_one_md_table_init(pgd_t *pgd)
 	if (!pmd_table)
 		return NULL;
 
-	set_pgd(pgd, __pgd(__pa(pmd_table) | _PAGE_PRESENT));
-	p4d = p4d_offset(pgd, 0);
-	pud = pud_offset(p4d, 0);
+	set_pud(pud, __pud(__pa(pmd_table) | _PAGE_PRESENT));
 
 	BUG_ON(pmd_table != pmd_offset(pud, 0));
 #else
-	p4d = p4d_offset(pgd, 0);
-	pud = pud_offset(p4d, 0);
 	pmd_table = pmd_offset(pud, 0);
 #endif
 
@@ -134,11 +129,12 @@ static inline void resume_init_first_level_page_table(pgd_t *pg_dir)
 {
 #ifdef CONFIG_X86_PAE
 	int i;
+	pud_t *pud = pud_offset(p4d_offset(pg_dir, 0), 0);
 
 	/* Init entries of the first-level page table to the zero page */
 	for (i = 0; i < PTRS_PER_PGD; i++)
-		set_pgd(pg_dir + i,
-			__pgd(__pa(empty_zero_page) | _PAGE_PRESENT));
+		set_pud(pud + i,
+			__pud(__pa(empty_zero_page) | _PAGE_PRESENT));
 #endif
 }
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 21/34] x86: platform: use proper set_pXd() for generic compile-time folded patable in setup_olpc_ofw_pgd()

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level
set_pXd() function is used instead.

Therefore, replace set_pgd() with set_pmd() when CONFIG_PGTABLE_LEVELS=2.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/x86/platform/olpc/olpc_ofw.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/x86/platform/olpc/olpc_ofw.c b/arch/x86/platform/olpc/olpc_ofw.c
index 6bab0f0aa8f3e..d709534d48373 100644
--- a/arch/x86/platform/olpc/olpc_ofw.c
+++ b/arch/x86/platform/olpc/olpc_ofw.c
@@ -25,6 +25,8 @@ void __init setup_olpc_ofw_pgd(void)
 {
 	pgd_t *base, *ofw_pde;
 
+	BUILD_BUG_ON(CONFIG_PGTABLE_LEVELS != 2);
+
 	if (!olpc_ofw_cif)
 		return;
 
@@ -38,7 +40,8 @@ void __init setup_olpc_ofw_pgd(void)
 	ofw_pde = &base[OLPC_OFW_PDE_NR];
 
 	/* install OFW's PDE permanently into the kernel's pgtable */
-	set_pgd(&swapper_pg_dir[OLPC_OFW_PDE_NR], *ofw_pde);
+	set_pmd((pmd_t *)&swapper_pg_dir[OLPC_OFW_PDE_NR],
+		*(pmd_t *)ofw_pde);
 	/* implicit optimization barrier here due to uninline function return */
 
 	early_iounmap(base, sizeof(olpc_ofw_pgd) * PTRS_PER_PGD);
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 22/34] x86: mm: use proper set_pXd() for generic compile-time folded patable in one_md_table_init()

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level
set_pXd() function is used instead.

Therefore, replace set_pgd() with set_pud() to setup first-level pgtable
in case of X86_PAE.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/x86/mm/init_32.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 0908c44d51e6f..690afa27a631b 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -70,19 +70,17 @@ static pmd_t * __init one_md_table_init(pgd_t *pgd)
 	pud_t *pud;
 	pmd_t *pmd_table;
 
+	p4d = p4d_offset(pgd, 0);
+	pud = pud_offset(p4d, 0);
 #ifdef CONFIG_X86_PAE
-	if (!(pgd_val(*pgd) & _PAGE_PRESENT)) {
+	if (!(pud_val(*pud) & _PAGE_PRESENT)) {
 		pmd_table = (pmd_t *)alloc_low_page();
-		set_pgd(pgd, __pgd(__pa(pmd_table) | _PAGE_PRESENT));
-		p4d = p4d_offset(pgd, 0);
-		pud = pud_offset(p4d, 0);
+		set_pud(pud, __pud(__pa(pmd_table) | _PAGE_PRESENT));
 		BUG_ON(pmd_table != pmd_offset(pud, 0));
 
 		return pmd_table;
 	}
 #endif
-	p4d = p4d_offset(pgd, 0);
-	pud = pud_offset(p4d, 0);
 	pmd_table = pmd_offset(pud, 0);
 
 	return pmd_table;
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 23/34] x86: mm: skip pud setup when using generic compile-time folded pagetable

[Yeoreum Yun]

We want to rework how set_pXd() behaves for generic compile-time folded
page tables by disallowing its use and triggering a compile-time error
when it is used improperly, ensuring that the actual first-level set_pXd()
function is used instead.

Therefore, skip collapse_pud_page() and populate_pud() according to
CONFIG_PGTABLE_LEVELS.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/x86/mm/pat/set_memory.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
index d023a40a1e034..cffb1cef869cc 100644
--- a/arch/x86/mm/pat/set_memory.c
+++ b/arch/x86/mm/pat/set_memory.c
@@ -1326,7 +1326,7 @@ static int collapse_pud_page(pud_t *pud, unsigned long addr,
 	pmd_t *pmd, first;
 	int i;
 
-	if (!direct_gbpages)
+	if (CONFIG_PGTABLE_LEVELS == 2 || !direct_gbpages)
 		return 0;
 
 	addr &= PUD_MASK;
@@ -1710,7 +1710,8 @@ static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d,
 	/*
 	 * Map everything starting from the Gb boundary, possibly with 1G pages
 	 */
-	while (boot_cpu_has(X86_FEATURE_GBPAGES) && end - start >= PUD_SIZE) {
+	while (CONFIG_PGTABLE_LEVELS > 3 && boot_cpu_has(X86_FEATURE_GBPAGES) &&
+	       end - start >= PUD_SIZE) {
 		set_pud(pud, pud_mkhuge(pfn_pud(cpa->pfn,
 				   canon_pgprot(pud_pgprot))));
 
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 24/34] x86: mm: call try_to_free_pmd_page() when CONFIG_PGTABLE_LEVELS > 2

[Yeoreum Yun]

We want to rework how pgd_page_vaddr() behaves for generic compile-time
folded page tables by disallowing its use and triggering a compile-time
error when it is used improperly, ensuring that the actual
first-level pXd_pgtable() function is used instead.

The try_to_free_pmd_page() wouldn't need to call when
CONFIG_PGTABLE_LEVELS=2. therefore call it when CONFIG_PGTABLE_LEVELS > 2.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/x86/mm/pat/set_memory.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
index cffb1cef869cc..2137174b733ae 100644
--- a/arch/x86/mm/pat/set_memory.c
+++ b/arch/x86/mm/pat/set_memory.c
@@ -1452,7 +1452,7 @@ static bool unmap_pte_range(pmd_t *pmd, unsigned long start, unsigned long end)
 static void __unmap_pmd_range(pud_t *pud, pmd_t *pmd,
 			      unsigned long start, unsigned long end)
 {
-	if (unmap_pte_range(pmd, start, end))
+	if (unmap_pte_range(pmd, start, end) && CONFIG_PGTABLE_LEVELS > 2)
 		if (try_to_free_pmd_page(pud_pgtable(*pud)))
 			pud_clear(pud);
 }
@@ -1496,7 +1496,7 @@ static void unmap_pmd_range(pud_t *pud, unsigned long start, unsigned long end)
 	/*
 	 * Try again to free the PMD page if haven't succeeded above.
 	 */
-	if (!pud_none(*pud))
+	if (!pud_none(*pud) && CONFIG_PGTABLE_LEVELS > 2)
 		if (try_to_free_pmd_page(pud_pgtable(*pud)))
 			pud_clear(pud);
 }
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 27/34] m68k: mm: remove usage of pgd_page_vaddr() for CONFIG_PGTABLE_LEVELS=3

[Yeoreum Yun]

We want to rework how pgd_page_vaddr() behaves for generic compile-time
folded page tables by disallowing its use and triggering a compile-time
error when it is used improperly, ensuring that the actual first-level
pXd_pgtable() function is used instead.

For this, change usage of pgd_page_vaddr() to pud_pgtable() for
the functions used when PGTABLE_LEVELS=3.

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/m68k/mm/init.c     | 2 +-
 arch/m68k/mm/motorola.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/m68k/mm/init.c b/arch/m68k/mm/init.c
index 3b88c0dd1616d..d8351c210f58d 100644
--- a/arch/m68k/mm/init.c
+++ b/arch/m68k/mm/init.c
@@ -95,7 +95,7 @@ static inline void init_pointer_tables(void)
 		if (!pud_present(*pud))
 			continue;
 
-		pmd_dir = (pmd_t *)pgd_page_vaddr(kernel_pg_dir[i]);
+		pmd_dir = pud_pgtable(*pud);
 		init_pointer_table(pmd_dir, TABLE_PMD);
 
 		for (j = 0; j < PTRS_PER_PMD; j++) {
diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c
index b30aa69a73a6a..03235d0618ef0 100644
--- a/arch/m68k/mm/motorola.c
+++ b/arch/m68k/mm/motorola.c
@@ -273,7 +273,7 @@ static pmd_t * __init kernel_ptr_table(void)
 
 			if (!pud_present(*pud))
 				continue;
-			pmd = pgd_page_vaddr(kernel_pg_dir[i]);
+			pmd = (unsigned long)pud_pgtable(*pud);
 			if (pmd > last)
 				last = pmd;
 		}
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 28/34] arm: mm: use proper pgtable APIs for generic compile-time folded patable in kasan_init()

[Yeoreum Yun]

We want to rework how set_pXd() and pgd_page_vaddr() behaves for
generic compile-time folded page tables by disallowing its use and
triggering a compile-time error when they're used improperly,
ensuring that the actual first-level pgtable APIs are used instead.

For this, Replace set_pgd() with set_pud() and pgd_page_vaddr() with pud_pgtable()
to setup kasan early patable since it used with LPAE and PGTABLE_LEVEL as 3

Since the first page-level is 3 to get a first page-level properly
introduce new helper pud_off_k() and replace direct dereference of
first page-table entry with pudp_get().

There should be no functional change.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm/mm/kasan_init.c | 8 +++++---
 include/linux/pgtable.h  | 5 +++++
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c
index c6625e808bf85..31fa1b11816c7 100644
--- a/arch/arm/mm/kasan_init.c
+++ b/arch/arm/mm/kasan_init.c
@@ -211,6 +211,7 @@ void __init kasan_init(void)
 {
 	phys_addr_t pa_start, pa_end;
 	u64 i;
+	pud_t *pudp __maybe_unused;
 
 	/*
 	 * We are going to perform proper setup of shadow memory.
@@ -230,11 +231,12 @@ void __init kasan_init(void)
 	/* We need to be in the same PGD or this won't work */
 	BUILD_BUG_ON(pgd_index(KASAN_SHADOW_START) !=
 		     pgd_index(KASAN_SHADOW_END));
+	pudp = pud_off_k(KASAN_SHADOW_START);
 	memcpy(tmp_pmd_table,
-	       (void*)pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_START)),
+	       (void*)pud_pgtable(pudp_get(pudp)),
 	       sizeof(tmp_pmd_table));
-	set_pgd(&tmp_pgd_table[pgd_index(KASAN_SHADOW_START)],
-		__pgd(__pa(tmp_pmd_table) | PMD_TYPE_TABLE | L_PGD_SWAPPER));
+	set_pud(&tmp_pgd_table[pgd_index(KASAN_SHADOW_START)],
+		__pmd(__pa(tmp_pmd_table) | PMD_TYPE_TABLE | L_PGD_SWAPPER));
 #endif
 	cpu_switch_mm(tmp_pgd_table, &init_mm);
 	local_flush_tlb_all();
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a8..4962d9764b487 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -155,6 +155,11 @@ static inline pgd_t *pgd_offset_pgd(pgd_t *pgd, unsigned long address)
  */
 #define pgd_offset_k(address)		pgd_offset(&init_mm, (address))
 
+static inline pud_t *pud_off_k(unsigned long va)
+{
+	return pud_offset(p4d_offset(pgd_offset_k(va), va), va);
+}
+
 /*
  * In many cases it is known that a virtual address is mapped at PMD or PTE
  * level, so instead of traversing all the page table levels, we can get a
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 29/34] mm/pgtable: disallow calling folded set_pgd/set_p4d/set_pud

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

The stubs are documented to "doesn't get actually called", which was
false for a couple of instances.

The helpers are dangerous, especially with some upcoming changes.

We made all code compile-out any calls to set_pgd/set_p4d/set_pud for
folded page tables. So let's make the compiler complain if these helpers
are abused in wrong context.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 include/asm-generic/pgtable-nop4d.h | 7 ++-----
 include/asm-generic/pgtable-nopmd.h | 6 +-----
 include/asm-generic/pgtable-nopud.h | 7 ++-----
 3 files changed, 5 insertions(+), 15 deletions(-)

diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index 019c3f074b771..0f3b56deaa165 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -28,11 +28,8 @@ static inline bool pgd_leaf(pgd_t pgd)		{ return false; }
 
 #define pgd_populate(mm, pgd, p4d)		do { } while (0)
 #define pgd_populate_safe(mm, pgd, p4d)		do { } while (0)
-/*
- * (p4ds are folded into pgds so this doesn't get actually called,
- * but the define is needed for a generic inline function.)
- */
-#define set_pgd(pgdptr, pgdval)	set_p4d((p4d_t *)(pgdptr), (p4d_t) { pgdval })
+
+#define set_pgd(pgdptr, pgdval)			BUILD_BUG()
 
 static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
 {
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index ae2eff44889ac..9d854211a55a5 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -38,11 +38,7 @@ static inline void pud_clear(pud_t *pud)	{ }
 
 #define pud_populate(mm, pmd, pte)		do { } while (0)
 
-/*
- * (pmds are folded into puds so this doesn't get actually called,
- * but the define is needed for a generic inline function.)
- */
-#define set_pud(pudptr, pudval)			set_pmd((pmd_t *)(pudptr), (pmd_t) { pudval })
+#define set_pud(pudptr, pudval)			BUILD_BUG()
 
 static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address)
 {
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index 5a2b0a81ae197..aa8a6ce139e47 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -35,11 +35,8 @@ static inline bool p4d_leaf(p4d_t p4d)		{ return false; }
 
 #define p4d_populate(mm, p4d, pud)		do { } while (0)
 #define p4d_populate_safe(mm, p4d, pud)		do { } while (0)
-/*
- * (puds are folded into p4ds so this doesn't get actually called,
- * but the define is needed for a generic inline function.)
- */
-#define set_p4d(p4dptr, p4dval)	set_pud((pud_t *)(p4dptr), (pud_t) { p4dval })
+
+#define set_p4d(p4dptr, p4dval)			BUILD_BUG()
 
 static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
 {
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 30/34] mm/pgtable: disallow calling folded (pgd|p4d|pud)_page, pgd_page_vaddr() and (p4d|pud)_pgtable

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

These helpers that silently fallback to the folded level
are dangerous, especially with some upcoming changes.

We made all code compile-out any calls to these helpers. So let's make the
compiler complain if these helpers are abused in wrong context.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 include/asm-generic/pgtable-nop4d.h | 4 ++--
 include/asm-generic/pgtable-nopmd.h | 4 ++--
 include/asm-generic/pgtable-nopud.h | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index 0f3b56deaa165..25c26a587b512 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -46,8 +46,8 @@ static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd,
 #define p4d_val(x)				(pgd_val((x).pgd))
 #define __p4d(x)				((p4d_t) { __pgd(x) })
 
-#define pgd_page(pgd)				(p4d_page((p4d_t){ pgd }))
-#define pgd_page_vaddr(pgd)			((unsigned long)(p4d_pgtable((p4d_t){ pgd })))
+#define pgd_page(pgd)				({ BUILD_BUG(); (struct page *)NULL; })
+#define pgd_page_vaddr(pgd)			({ BUILD_BUG(); 0UL; })
 
 /*
  * allocating and freeing a p4d is trivial: the 1-entry p4d is
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 9d854211a55a5..75650fa9111cd 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -56,8 +56,8 @@ static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud,
 #define pmd_val(x)				(pud_val((x).pud))
 #define __pmd(x)				((pmd_t) { __pud(x) } )
 
-#define pud_page(pud)				(pmd_page((pmd_t){ pud }))
-#define pud_pgtable(pud)			((pmd_t *)(pmd_page_vaddr((pmd_t){ pud })))
+#define pud_page(pud)				({ BUILD_BUG(); (struct page *)NULL; })
+#define pud_pgtable(pud)			({ BUILD_BUG(); NULL; })
 
 /*
  * allocating and freeing a pmd is trivial: the 1-entry pmd is
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index aa8a6ce139e47..b592fff18887a 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -54,8 +54,8 @@ static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d,
 #define pud_val(x)				(p4d_val((x).p4d))
 #define __pud(x)				((pud_t) { __p4d(x) })
 
-#define p4d_page(p4d)				(pud_page((pud_t){ p4d }))
-#define p4d_pgtable(p4d)			((pud_t *)(pud_pgtable((pud_t){ p4d })))
+#define p4d_page(p4d)				({ BUILD_BUG(); (struct page *)NULL; })
+#define p4d_pgtable(p4d)			({ BUILD_BUG(); NULL; })
 
 /*
  * allocating and freeing a pud is trivial: the 1-entry pud is
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 31/34] mm/pgtable: optimize pmdp_get() and friends for folded pagetable levels

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

Using pmdp_get() and friends in common code on a kernel config with
folded page tables is suboptimal: they default to a READ_ONCE(), forcing
the compiler to actually read that value even though it will not actually
be used afterwards.

This was recently reported by Christophe Leroy [1] and block conversion
of more common code to pmdp_get() and friends.

(using pgdp_get() as one example)

Most of the code ignores the result from pgdp_get() on configs with
folded page tables entirely, as we hardcode:

        pgd_present()==1 && pgd_leaf()==false

Common code will just treat it as a "this is a page table" and call
p4d_offset() or p4d_offset_lockless() for the next lower level, where
we just ignore the obtained pgdp_get() result entirely.

So we can just return a dummy value and avoid any memory reads.

There is a catch, though:

1) If code calls pgd_val() and somehow relies on the data, it would now
   see dummy values. The code really must be aware of folded page table
   levels. Fortunately, code usually ignores pgd_val() completely for
   page tables (with ptdump being one exception when calculating
   effective permissions). We checked + fixed the x86 ptdump mechanism.

2) If code passes the pgd_t to a function that would work on the result,
   it would now see dummy values. The only concern is really passing
   the pgd_t on the stack as a pointer to p4d_offset(). Most code that
   would do that, should actually use p4d_offset_lockless(), which
   handles this properly. We checked + fixed problematic instances.

As an example, this is the generated code for perf_get_page_size() with
PGTABLE_LEVELS=3 on arm64:

Before:
00000000000052a0 <perf_get_page_size>:
    ...
    52dc: d53b4234      mrs     x20, DAIF
    52e0: d50343df      msr     DAIFSet, #0x3
    ...
    52fc: d35e9a69      ubfx    x9, x19, #30, #9        /* pud_offset_lockless() */
    5300: f9403508      ldr     x8, [x8, #0x68]
    5304: f869790a      ldr     x10, [x8, x9, lsl #3]   /* pudp_get() */
    5308: f90007ea      str     x10, [sp, #0x8]
    530c: f8697908      ldr     x8, [x8, x9, lsl #3]    /* pudp_get() */
    ...
    5360: 90000009      adrp    x9, 0x5000 <perf_prepare_sample+0x548>
    5364: 92746908      and     x8, x8, #0x7ffffff000
    5368: d3557675      ubfx    x21, x19, #21, #9       /* pmd_offset_lockless() */
    ...
    5394: f8757ac8      ldr     x8, [x22, x21, lsl #3]  /* pmdp_get() */

After:
0000000000052a0 <perf_get_page_size>:
    ...
    52dc: d53b4234      mrs     x20, DAIF
    52e0: d50343df      msr     DAIFSet, #0x3
    ...                                                 /* no pud_offset_lockless() and pudp_get() */
    5318: 90000009      adrp    x9, 0x5000 <perf_prepare_sample+0x548>
    531c: 92746908      and     x8, x8, #0x7ffffff000
    5320: d3557675      ubfx    x21, x19, #21, #9       /* pmd_offset_lockless() */
    ...
    5334: f8757ac8      ldr     x8, [x22, x21, lsl #3]  /* pmdp_get() */

[1] https://lore.kernel.org/all/0019d675-ce3d-4a5c-89ed-f126c45145c9@kernel.org/

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 include/asm-generic/pgtable-nop4d.h | 8 ++++++++
 include/asm-generic/pgtable-nopmd.h | 8 ++++++++
 include/asm-generic/pgtable-nopud.h | 8 ++++++++
 3 files changed, 24 insertions(+)

diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index 25c26a587b512..84a529df27ee3 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -31,6 +31,14 @@ static inline bool pgd_leaf(pgd_t pgd)		{ return false; }
 
 #define set_pgd(pgdptr, pgdval)			BUILD_BUG()
 
+static inline pgd_t pgdp_get(pgd_t *p4dp)
+{
+	pgd_t dummy = { 0 };
+
+	return dummy;
+}
+#define pgdp_get pgdp_get
+
 static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
 {
 	return (p4d_t *)pgd;
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 75650fa9111cd..96f3fccd22fda 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -40,6 +40,14 @@ static inline void pud_clear(pud_t *pud)	{ }
 
 #define set_pud(pudptr, pudval)			BUILD_BUG()
 
+static inline pud_t pudp_get(pud_t *pudp)
+{
+	pud_t dummy = { 0 };
+
+	return dummy;
+}
+#define pudp_get pudp_get
+
 static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address)
 {
 	return (pmd_t *)pud;
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index b592fff18887a..faa0233bcde80 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -38,6 +38,14 @@ static inline bool p4d_leaf(p4d_t p4d)		{ return false; }
 
 #define set_p4d(p4dptr, p4dval)			BUILD_BUG()
 
+static inline p4d_t p4dp_get(p4d_t *p4dp)
+{
+	p4d_t dummy = { 0 };
+
+	return dummy;
+}
+#define p4dp_get p4dp_get
+
 static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
 {
 	return (pud_t *)p4d;
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 32/34] openrisc/pgtable: drop __pmd_offset()

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

The macro is unused, and will collide with a folded page table helper.
Let's just remove it.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/openrisc/include/asm/pgtable.h | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/openrisc/include/asm/pgtable.h b/arch/openrisc/include/asm/pgtable.h
index 6b89996d0b628..5a491cd0f0086 100644
--- a/arch/openrisc/include/asm/pgtable.h
+++ b/arch/openrisc/include/asm/pgtable.h
@@ -331,9 +331,6 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
 	return ((unsigned long) __va(pmd_val(pmd) & PAGE_MASK));
 }
 
-#define __pmd_offset(address) \
-	(((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1))
-
 #define PFN_PTE_SHIFT		PAGE_SHIFT
 #define pte_pfn(x)		((unsigned long)(((x).pte)) >> PAGE_SHIFT)
 #define pfn_pte(pfn, prot)  __pte((((pfn) << PAGE_SHIFT)) | pgprot_val(prot))
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 33/34] mm/pgtable: catch abuse of folded dummy pgd_t/p4d_t/pud_t

[Yeoreum Yun]

From: "David Hildenbrand (Arm)" <david@kernel.org>

Let's catch and prevent all abuse with dummy values on the stack
similar to:

	pud_t pud = pudp_get(pudp);
	pmd_t *pmdp = pud_offset(*pud, addr);

While this approach relies on the compiler propagating constants, it
should catch most of the issues in practice. It would have caught all
the issues we found through manual inspection.

To avoid build issues particularly on x86, where pgd_val() might not be
around in some inclusion paths, perform the new checks from wrapper
macros.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 include/asm-generic/pgtable-nop4d.h | 23 ++++++++++++++++++-----
 include/asm-generic/pgtable-nopmd.h | 24 ++++++++++++++++++------
 include/asm-generic/pgtable-nopud.h | 24 ++++++++++++++++++------
 3 files changed, 54 insertions(+), 17 deletions(-)

diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index 84a529df27ee3..fcd86aabc592c 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -31,7 +31,7 @@ static inline bool pgd_leaf(pgd_t pgd)		{ return false; }
 
 #define set_pgd(pgdptr, pgdval)			BUILD_BUG()
 
-static inline pgd_t pgdp_get(pgd_t *p4dp)
+static __always_inline pgd_t pgdp_get(pgd_t *p4dp)
 {
 	pgd_t dummy = { 0 };
 
@@ -39,17 +39,30 @@ static inline pgd_t pgdp_get(pgd_t *p4dp)
 }
 #define pgdp_get pgdp_get
 
-static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
+#define pgd_check_dummy(pgd) BUILD_BUG_ON(__builtin_constant_p(pgd_val(pgd)))
+
+static __always_inline p4d_t *__p4d_offset(pgd_t *pgdp, unsigned long address)
 {
-	return (p4d_t *)pgd;
+	return (p4d_t *)pgdp;
 }
 
-static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd,
+#define p4d_offset(pgdp, address)					\
+({									\
+	pgd_check_dummy(*(pgdp));					\
+	__p4d_offset(pgdp, address);					\
+})
+
+static __always_inline p4d_t *__p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd,
 		unsigned long address)
 {
 	return (p4d_t *)pgdp;
 }
-#define p4d_offset_lockless p4d_offset_lockless
+
+#define p4d_offset_lockless(pgdp, pgd, address)				\
+({									\
+	pgd_check_dummy(*(pgdp));					\
+	__p4d_offset_lockless(pgdp, pgd, address);			\
+})
 
 #define p4d_val(x)				(pgd_val((x).pgd))
 #define __p4d(x)				((p4d_t) { __pgd(x) })
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 96f3fccd22fda..ef6dd5895e7ba 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -40,7 +40,7 @@ static inline void pud_clear(pud_t *pud)	{ }
 
 #define set_pud(pudptr, pudval)			BUILD_BUG()
 
-static inline pud_t pudp_get(pud_t *pudp)
+static __always_inline pud_t pudp_get(pud_t *pudp)
 {
 	pud_t dummy = { 0 };
 
@@ -48,18 +48,30 @@ static inline pud_t pudp_get(pud_t *pudp)
 }
 #define pudp_get pudp_get
 
-static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address)
+#define pud_check_dummy(pud) BUILD_BUG_ON(__builtin_constant_p(pud_val(pud)))
+
+static __always_inline pmd_t *__pmd_offset(pud_t *pudp, unsigned long address)
 {
-	return (pmd_t *)pud;
+	return (pmd_t *)pudp;
 }
-#define pmd_offset pmd_offset
 
-static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud,
+#define pmd_offset(pudp, address)					\
+({									\
+	pud_check_dummy(*(pudp));					\
+	__pmd_offset(pudp, address);					\
+})
+
+static __always_inline pmd_t *__pmd_offset_lockless(pud_t *pudp, pud_t pud,
 		unsigned long address)
 {
 	return (pmd_t *)pudp;
 }
-#define pmd_offset_lockless pmd_offset_lockless
+
+#define pmd_offset_lockless(pudp, pud, address)				\
+({									\
+	pud_check_dummy(*(pudp));					\
+	__pmd_offset_lockless(pudp, pud, address);			\
+})
 
 #define pmd_val(x)				(pud_val((x).pud))
 #define __pmd(x)				((pmd_t) { __pud(x) } )
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index faa0233bcde80..873940f9d3f9b 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -38,7 +38,7 @@ static inline bool p4d_leaf(p4d_t p4d)		{ return false; }
 
 #define set_p4d(p4dptr, p4dval)			BUILD_BUG()
 
-static inline p4d_t p4dp_get(p4d_t *p4dp)
+static __always_inline p4d_t p4dp_get(p4d_t *p4dp)
 {
 	p4d_t dummy = { 0 };
 
@@ -46,18 +46,30 @@ static inline p4d_t p4dp_get(p4d_t *p4dp)
 }
 #define p4dp_get p4dp_get
 
-static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
+#define p4d_check_dummy(p4d) BUILD_BUG_ON(__builtin_constant_p(p4d_val(p4d)))
+
+static __always_inline pud_t *__pud_offset(p4d_t *p4dp, unsigned long address)
 {
-	return (pud_t *)p4d;
+	return (pud_t *)p4dp;
 }
-#define pud_offset pud_offset
 
-static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d,
+#define pud_offset(p4dp, address)					\
+({									\
+	p4d_check_dummy(*(p4dp));					\
+	__pud_offset(p4dp, address);					\
+})
+
+static __always_inline pud_t *__pud_offset_lockless(p4d_t *p4dp, p4d_t p4d,
 		unsigned long address)
 {
 	return (pud_t *)p4dp;
 }
-#define pud_offset_lockless pud_offset_lockless
+
+#define pud_offset_lockless(p4dp, p4d, address)				\
+({									\
+	p4d_check_dummy(*(p4dp));					\
+	__pud_offset_lockless(p4dp, p4d, address);			\
+})
 
 #define pud_val(x)				(p4d_val((x).p4d))
 #define __pud(x)				((pud_t) { __p4d(x) })
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

[RFC PATCH 34/34] arm64: pgtable: convert pte_present() from macro to static inline

[Yeoreum Yun]

pte_present() is used as the basis for both pmd_present() and pud_present().
It is currently implemented as a macro composed of pte_val() and
pte_present_invalid().

When pte_present() or its higher-level variants are used directly with
ptep_get() or pXdp_get(), for example:

  pte_present(ptep_get(pte));
  pmd_present(pmdp_get(pmd));
  pud_present(pudp_get(pud));

the macro expansion causes the compiler to evaluate the argument twice,
resulting in redundant loads. For example, pte_present() expands to:

  !pte_val(READ_ONCE(*pte) || pte_present_invalid(READ_ONCE(*pte))

A typical example is pud_free_pmd_page(), where the expansion of
pmd_present() generates:
    ...
    /* pmd_present() (x20 = pmdp) */
    1b88: f9400288     ldr	x8, [x20]        // read pmdp.
    1b8c: f9000fa8     str	x8, [x29, #0x18]
    1b90: 3707fec8     tbnz	w8, #0x0, 0x1b68 <pud_free_pmd_page+0xd0>
    1b94: f9400288     ldr	x8, [x20]        // redundant read of pmdp.
    1b98: 8a170109     and	x9, x8, x23
    1b9c: f9000fa8     str	x8, [x29, #0x18]
    1ba0: f120013f     cmp	x9, #0x800
    1ba4: 54fffe20     b.eq	0x1b68 <pud_free_pmd_page+0xd0>
    1ba8: 17fffff4     b	0x1b78 <pud_free_pmd_page+0xe0>
    ...

Convert pte_present() to static inline function so that prevent the
generation of redundant code and move pte_valid() and
pte_present_invalid() further up so the inline function can use them.

After this change, the generated code becomes:
    ...
    /* pmd_present() (x20 = pmdp) */
    1a30: f9400288     ldr	x8, [x20]
    1a34: 8a170109     and	x9, x8, x23
    1a38: f9000fa8     str	x8, [x29, #0x18]
    1a3c: f120013f     cmp	x9, #0x800
    1a40: 54fffe80     b.eq	0x1a10 <pud_free_pmd_page+0xd0>
    1a44: 3607fee8     tbz	w8, #0x0, 0x1a20 <pud_free_pmd_page+0xe0>
    1a48: 17fffff2     b	0x1a10 <pud_free_pmd_page+0xd0>
    ...

This eliminates the redundant load and also reduces code size at
call sites using this pattern. For example, pud_free_pmd_page() shrinks
from 7,500 bytes to 7,148 bytes, a reduction of approximately 4.7%.

Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
 arch/arm64/include/asm/pgtable.h | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 6185fc291fd7d..aaae3d14895f6 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -140,10 +140,17 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
 #define pte_none(pte)		(!pte_val(pte))
 #define pte_page(pte)		(pfn_to_page(pte_pfn(pte)))
 
+#define pte_valid(pte)		(!!(pte_val(pte) & PTE_VALID))
+#define pte_present_invalid(pte) \
+	((pte_val(pte) & (PTE_VALID | PTE_PRESENT_INVALID)) == PTE_PRESENT_INVALID)
+
 /*
  * The following only work if pte_present(). Undefined behaviour otherwise.
  */
-#define pte_present(pte)	(pte_valid(pte) || pte_present_invalid(pte))
+static __always_inline bool pte_present(pte_t pte)
+{
+	return pte_valid(pte) || pte_present_invalid(pte);
+}
 #define pte_young(pte)		(!!(pte_val(pte) & PTE_AF))
 #define pte_special(pte)	(!!(pte_val(pte) & PTE_SPECIAL))
 #define pte_write(pte)		(!!(pte_val(pte) & PTE_WRITE))
@@ -168,9 +175,6 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
 #define pte_sw_dirty(pte)	(!!(pte_val(pte) & PTE_DIRTY))
 #define pte_dirty(pte)		(pte_sw_dirty(pte) || pte_hw_dirty(pte))
 
-#define pte_valid(pte)		(!!(pte_val(pte) & PTE_VALID))
-#define pte_present_invalid(pte) \
-	((pte_val(pte) & (PTE_VALID | PTE_PRESENT_INVALID)) == PTE_PRESENT_INVALID)
 /*
  * Execute-only user mappings do not have the PTE_USER bit set. All valid
  * kernel mappings have the PTE_UXN bit set.
-- 
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}

Re: [RFC PATCH 00/34] mm: optimize unnecessary loads due to ptep_get() and friends out

[David Hildenbrand (Arm)]

> Future work
> ===========
>  - print_bad_page_map() and show_pte() still prints dummy values
>    instead of printing the same content for all generic compile-time
>    folded page tables. We might want to skip printing dummy values later.
> 
>  - We currently catch abuse of dummy values on the stack at compile-time by
>    relying on constant propagation by the compiler. Usama's work [3] on using
>    distinct types for sw vs. hw PTEs could help here as well."

Just adding here that this series was excessively build tested (both by me +
Yeoreum and the friendly build bots), and primarily tested on arm64 only so far
with differing page table level configurations.

There is the chance that there might still be some odd configs that reveal
set_pgd() etc places to rework: so the build bots better should chew some more
on this.

I want to do some more testing, especially on x86 (32bit), but did not get to it
yet.

-- 
Cheers,

David

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[Dave Hansen]

On 7/13/26 06:55, Yeoreum Yun wrote:
> We want to change how pXdp_get() works with generic compile-time folded
> page tables. To prepare for that, rework effective_prot() to ignore any
> folded page tables entries, as it will unconditionally get called by
> ptdump core with pXd_val(pXdp_get()), and we really should be ignoring
> that value for folded entries.
> 
> For this, identify the first real page-table level and
> update effective_prot only when the current entry is not folded.

Nit: could you please clean up the "we's" and move over to imperative
voice for the series? We're picky about it on the x86 side, but I do
think it's generally a good preferred practice across the kernel.

Also I really do think this made the code worse. effective_prot() is a
pretty tidy function. This change more than doubles the size and is also
basically the same block copied and pasted twice.

Further, this does seem like it will stop filling out some of the
st->prot_levels[] entries. That seems like it might break things. It
definitely needs to be covered in the changelog.

So, in the end, I'm not 100% sure what this is doing. Is it "optimizing"
the ptdump code? Or is it preemptively fixing code that will soon be
throwing a compile error?

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[David Hildenbrand (Arm)]

On 7/13/26 16:19, Dave Hansen wrote:
> On 7/13/26 06:55, Yeoreum Yun wrote:
>> We want to change how pXdp_get() works with generic compile-time folded
>> page tables. To prepare for that, rework effective_prot() to ignore any
>> folded page tables entries, as it will unconditionally get called by
>> ptdump core with pXd_val(pXdp_get()), and we really should be ignoring
>> that value for folded entries.
>>
>> For this, identify the first real page-table level and
>> update effective_prot only when the current entry is not folded.
> 

I'll let Yeoreum comment on most of it, just one thought below.

> Nit: could you please clean up the "we's" and move over to imperative
> voice for the series? We're picky about it on the x86 side, but I do
> think it's generally a good preferred practice across the kernel.
> 
> Also I really do think this made the code worse. effective_prot() is a
> pretty tidy function. This change more than doubles the size and is also
> basically the same block copied and pasted twice.
> 
> Further, this does seem like it will stop filling out some of the
> st->prot_levels[] entries. That seems like it might break things. It
> definitely needs to be covered in the changelog.
> 
> So, in the end, I'm not 100% sure what this is doing. Is it "optimizing"
> the ptdump code? Or is it preemptively fixing code that will soon be
> throwing a compile error?

It's a preparation for getting called with folded entries that no longer
duplicate the information.

Otherwise, with later changes in this series would break it.

Essentially, we have to teach x86 code to ignore levels that are folded (e.g.,
pgd, p4d) and start collecting effective prots from the actual first non-folded
level.

I was thinking about handling the "don't call effective_prot() with folded
entries" in ptdump code, which would likely be cleaner.

We'd still have to teach x86's effective_prot() about "this is your actual first
non-folded level, everything below is not initialized (or we decide to just
initialize it)"

-- 
Cheers,

David

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[Dave Hansen]

On 7/13/26 07:41, David Hildenbrand (Arm) wrote:
> Essentially, we have to teach x86 code to ignore levels that are folded (e.g.,
> pgd, p4d) and start collecting effective prots from the actual first non-folded
> level.

I guess that's what the series really comes down to.

The model is moving from (using 3-level PAE as an example):

 PGD => ...folding => PMD => PTE

to

 ...folding => PUD => PMD => PTE

Right?

That's actually not the worst thing in the world, but it is a pretty big
change. It moves the software model away from how the hardware works.
It's a bit more grand than I thought from glancing at the cover letter.

Re: [RFC PATCH 19/34] x86: power: use proper set_pXd() for generic compile-time folded patable in resume_one_md_table_init()

[David Hildenbrand (Arm)]

On 7/13/26 16:38, Dave Hansen wrote:
> On 7/13/26 06:55, Yeoreum Yun wrote:
>> @@ -134,11 +129,12 @@ static inline void resume_init_first_level_page_table(pgd_t *pg_dir)
>>  {
>>  #ifdef CONFIG_X86_PAE
>>  	int i;
>> +	pud_t *pud = pud_offset(p4d_offset(pg_dir, 0), 0);
>>  
>>  	/* Init entries of the first-level page table to the zero page */
>>  	for (i = 0; i < PTRS_PER_PGD; i++)
>> -		set_pgd(pg_dir + i,
>> -			__pgd(__pa(empty_zero_page) | _PAGE_PRESENT));
>> +		set_pud(pud + i,
>> +			__pud(__pa(empty_zero_page) | _PAGE_PRESENT));
>>  #endif
>>  }
> 
> If this is the way forward, it's going to require the retraining of some
> awfully old brain cells.
> 
> It also doesn't really read correctly. Loop over each PGD entry:
> 
> 	for (i = 0; i < PTRS_PER_PGD; i++)
> 
> ... and set a pud?
> 
> 		set_pud(...)

The PTRS_PER_PGD is indeed a nasty aspect. Maybe we could clean that up to
exclusively work on PUDs here.

It's already nasty today if you take a look at e.g., patch #22 where we do:

	set_pgd(pgd, __pgd(__pa(pmd_table) | _PAGE_PRESENT));

Or patch #25:

	if (pgd_val(pgd) !=·0) {
		pmd_t *pmd = (pmd_t *)pgd_page_vaddr(pgd);


So yes, we know that we want to set a PMD already in these cases. And using a
PUD is the natural choice :)


> 
> I'm not sure how I'd ever learn the rules to write this code from
> scratch. Right now, the code says: "The pgd_t is the top level of the
> page tables. I know there are PTRS_PER_PGD of those top level entries
> entries I need to 'zero'."
> 
> You don't have to know what is folded, just that you're dealing with the
> top level.
> 
> But *this* code says:
> 
> 1. The pgd_t is the top level of the page tables. Start there.
> 2. "Walk" down *two* (folded) levels
> 3. Set the third-level (pud) entries to 'zero'
> 
> So the code now *has* to know how the folding occurs.
> 
> This seems really hard to work with to me.

It all goes back to the design choice people made to say for folded p4d that

	1) PGD entries are always treated as present page table entries.

static inline int pgd_none(pgd_t pgd)           { return 0; }
static inline int pgd_bad(pgd_t pgd)            { return 0; }
static inline int pgd_present(pgd_t pgd)        { return 1; }
static inline void pgd_clear(pgd_t *pgd)        { }
static inline bool pgd_leaf(pgd_t pgd)          { return false; }

	2) P4D entries are actually what we interpret later.


It kind-of makes sense, because then you have all of the folding happening "on
the top", and not somewhere down the chain.

But yes, ideally we wouldn't have

#define PTRS_PER_P4D            1

I wonder if we could clean that up ...


The good thing is, that the compiler will complain to you if you use the wrong
setter now.

Note the comments we remove in patch #29:

/*
 * (p4ds are folded into pgds so this doesn't get actually called,
 * but the define is needed for a generic inline function.)
 */
#define set_pgd(pgdptr, pgdval) set_p4d((p4d_t *)(pgdptr), (p4d_t) { pgdval })


Doesn't actually get called, right? ;)

-- 
Cheers,

David

Re: [RFC PATCH 19/34] x86: power: use proper set_pXd() for generic compile-time folded patable in resume_one_md_table_init()

[Yeoreum Yun]

> On 7/13/26 06:55, Yeoreum Yun wrote:
> > @@ -134,11 +129,12 @@ static inline void resume_init_first_level_page_table(pgd_t *pg_dir)
> >  {
> >  #ifdef CONFIG_X86_PAE
> >  	int i;
> > +	pud_t *pud = pud_offset(p4d_offset(pg_dir, 0), 0);
> >  
> >  	/* Init entries of the first-level page table to the zero page */
> >  	for (i = 0; i < PTRS_PER_PGD; i++)
> > -		set_pgd(pg_dir + i,
> > -			__pgd(__pa(empty_zero_page) | _PAGE_PRESENT));
> > +		set_pud(pud + i,
> > +			__pud(__pa(empty_zero_page) | _PAGE_PRESENT));
> >  #endif
> >  }
> 
> If this is the way forward, it's going to require the retraining of some
> awfully old brain cells.
> 
> It also doesn't really read correctly. Loop over each PGD entry:
> 
> 	for (i = 0; i < PTRS_PER_PGD; i++)
> 
> ... and set a pud?
> 
> 		set_pud(...)
> 
> I'm not sure how I'd ever learn the rules to write this code from
> scratch. Right now, the code says: "The pgd_t is the top level of the
> page tables. I know there are PTRS_PER_PGD of those top level entries
> entries I need to 'zero'."
> 
> You don't have to know what is folded, just that you're dealing with the
> top level.
> 
> But *this* code says:
> 
> 1. The pgd_t is the top level of the page tables. Start there.
> 2. "Walk" down *two* (folded) levels
> 3. Set the third-level (pud) entries to 'zero'
> 
> So the code now *has* to know how the folding occurs.
> 
> This seems really hard to work with to me.

Actually, This is what the patch series want.
IOW, the "code" need to consider the folded pgtable level and
use a proper APIs.

Might for the clarification, change a PTRS_PER_PGD to PTRS_PER_PUD
seems better. but the using set_pgd() seems much more worse since
second argument type is p4d_t which doesn't exist in the p4d folded case.

IOW, this patch wants to use proper API for the real first page table
level. for example, X86_PAE's pgtable level is 3 so not use pgd friends function
but use a pud to setup first page table entries.

-- 
Sincerely,
Yeoreum Yun

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[David Hildenbrand (Arm)]

On 7/13/26 16:55, Dave Hansen wrote:
> On 7/13/26 07:41, David Hildenbrand (Arm) wrote:
>> Essentially, we have to teach x86 code to ignore levels that are folded (e.g.,
>> pgd, p4d) and start collecting effective prots from the actual first non-folded
>> level.
> 
> I guess that's what the series really comes down to.
> 
> The model is moving from (using 3-level PAE as an example):
> 
>  PGD => ...folding => PMD => PTE
> 
> to
> 
>  ...folding => PUD => PMD => PTE
> 
> Right?

Not quite, because it's complicated:

static inline int pgd_none(pgd_t pgd)           { return 0; }
static inline int pgd_bad(pgd_t pgd)            { return 0; }
static inline int pgd_present(pgd_t pgd)        { return 1; }
static inline void pgd_clear(pgd_t *pgd)        { }
#define p4d_ERROR(p4d)                          (pgd_ERROR((p4d).pgd))

#define pgd_populate(mm, pgd, p4d)              do { } while (0)
#define pgd_populate_safe(mm, pgd, p4d)         do { } while (0)


It's always been the latter:

	...folding => PUD => PMD => PTE


This series rather enforces that the latter is used consistently. (e.g., not
working on pgds when they are actually always look like present page tables)

We decided to go for strict BUILD_BUG() on set_pgd() etc. because otherwise it's
just waiting for trouble once we change what pgpd_get() returns.

-- 
Cheers,

David

Re: [RFC PATCH 02/34] ARM: mm: make 2-level pgd_t a scalar

[Arnd Bergmann]

On Mon, Jul 13, 2026, at 15:55, Yeoreum Yun wrote:
> From: "David Hildenbrand (Arm)" <david@kernel.org>
>
> We don't want pgd_t to be an array, as it prohibits returning it from a
> function, like pgdp_get().

What should pgdp_get() return on ARM 2-stage page tables then?
Does it return just the first entry or concatenate the two?

> +typedef u64 pgdval_t;

> +static inline pmdval_t pgd_val(pgd_t pgd)
> +{
> +	return (*(pmdval_t (*)[2])&pgd)[0];
> +}

This could use a comment to explain what this actually
returns and why the second entry is discarded.

      Arnd

Re: [RFC PATCH 02/34] ARM: mm: make 2-level pgd_t a scalar

[David Hildenbrand (Arm)]

On 7/13/26 17:38, Arnd Bergmann wrote:
> On Mon, Jul 13, 2026, at 15:55, Yeoreum Yun wrote:
>> From: "David Hildenbrand (Arm)" <david@kernel.org>
>>
>> We don't want pgd_t to be an array, as it prohibits returning it from a
>> function, like pgdp_get().
> 
> What should pgdp_get() return on ARM 2-stage page tables then?
> Does it return just the first entry or concatenate the two?

It is rather obscure what we do (below).

This patch tries to keep the existing behavior.

> 
>> +typedef u64 pgdval_t;
> 
>> +static inline pmdval_t pgd_val(pgd_t pgd)
>> +{
>> +	return (*(pmdval_t (*)[2])&pgd)[0];
>> +}
> 
> This could use a comment to explain what this actually
> returns and why the second entry is discarded.

It's rather crazy. We have two page table levels, but fake that we have 3.


There is some documentation in arch/arm/include/asm/pgtable-2level.h

/*
 * Hardware-wise, we have a two level page table structure, where the first
 * level has 4096 entries, and the second level has 256 entries.  Each entry
 * is one 32-bit word.  Most of the bits in the second level entry are used
 * by hardware, and there aren't any "accessed" and "dirty" bits.
 *
 * Linux on the other hand has a three level page table structure, which can
 * be wrapped to fit a two level page table structure easily - using the PGD
 * and PTE only.  However, Linux also expects one "PTE" table per page, and
 * at least a "dirty" bit.
 *
 * Therefore, we tweak the implementation slightly - we tell Linux that we
 * have 2048 entries in the first level, each of which is 8 bytes (iow, two
 * hardware pointers to the second level.)  The second level contains two
 * hardware PTE tables arranged contiguously, preceded by Linux versions
 * which contain the state information Linux needs.  We, therefore, end up
 * with 512 entries in the "PTE" level.
 *
 * This leads to the page tables having the following layout:
 *
 *    pgd             pte
 * |        |
 * +--------+
 * |        |       +------------+ +0
 * +- - - - +       | Linux pt 0 |
 * |        |       +------------+ +1024
 * +--------+ +0    | Linux pt 1 |
 * |        |-----> +------------+ +2048
 * +- - - - + +4    |  h/w pt 0  |
 * |        |-----> +------------+ +3072
 * +--------+ +8    |  h/w pt 1  |
 * |        |       +------------+ +4096
 *

So we interpret two 32bit entries as a pair (64bit value). Together they span
2x256 values.

Then we make PTRS_PER_PTE = 512, to iterate all of them (both pairs).

Let me do some more digging to come up with something I can confidentially
document here ... maybe I'll just refer to the comment in pgtable-2level.h.

-- 
Cheers,

David

Re: [RFC PATCH 02/34] ARM: mm: make 2-level pgd_t a scalar

[David Hildenbrand (Arm)]

>  * |        |
>  * +--------+
>  * |        |       +------------+ +0
>  * +- - - - +       | Linux pt 0 |
>  * |        |       +------------+ +1024
>  * +--------+ +0    | Linux pt 1 |
>  * |        |-----> +------------+ +2048
>  * +- - - - + +4    |  h/w pt 0  |
>  * |        |-----> +------------+ +3072
>  * +--------+ +8    |  h/w pt 1  |
>  * |        |       +------------+ +4096
>  *
> 
> So we interpret two 32bit entries as a pair (64bit value). Together they span
> 2x256 values.
> 
> Then we make PTRS_PER_PTE = 512, to iterate all of them (both pairs).
> 
> Let me do some more digging to come up with something I can confidentially

"with confidence", not confidentially :)

-- 
Cheers,

David

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[Dave Hansen]

On 7/13/26 08:02, David Hildenbrand (Arm) wrote:
> It's always been the latter:
> 
> 	...folding => PUD => PMD => PTE

This is kinda arguing semantics, and there are different ways to look at
it. But I'd argue it couldn't *possibly* have always been this way.
Linux hasn't always had a PUD or a PMD for that matter. The i386 only
had two levels. PAE support was added later.

I think the pmd_t came with Alpha support (before my time) and pud_t
came later IIRC. I'm pretty sure Linux didn't have a pud_t when I first
started hacking on it circa 2001.

Second (and again this is my subjective opinion) a pgd_t* has always
been the "top" or "base" of the page tables. This is why there's an
mm->pgd and not an mm->pud. We've also talked *forever* about things
like a "special pgd" for the LDT remap. Or having two pgd pages for KPTI.

My point is that the PGD hasn't itself ever been folded. It's the top
hardware level of the page tables and always has been.

I really do think this series is changing things, not keeping them the same.

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[David Hildenbrand (Arm)]

On 7/13/26 19:57, Dave Hansen wrote:
> On 7/13/26 08:02, David Hildenbrand (Arm) wrote:
>> It's always been the latter:
>>
>> 	...folding => PUD => PMD => PTE
> 
> This is kinda arguing semantics, and there are different ways to look at
> it. But I'd argue it couldn't *possibly* have always been this way.
> Linux hasn't always had a PUD or a PMD for that matter. The i386 only
> had two levels. PAE support was added later.
> 
> I think the pmd_t came with Alpha support (before my time) and pud_t
> came later IIRC. I'm pretty sure Linux didn't have a pud_t when I first
> started hacking on it circa 2001.

Yes, I'm sure there is some horrible, horrible history to how we came to what we
have today :)

> 
> Second (and again this is my subjective opinion) a pgd_t* has always
> been the "top" or "base" of the page tables. This is why there's an
> mm->pgd and not an mm->pud. We've also talked *forever* about things
> like a "special pgd" for the LDT remap. Or having two pgd pages for KPTI.

Agreed. We always called it pgd, which is just an alias to p4d/pud etc really.

> 
> My point is that the PGD hasn't itself ever been folded. It's the top
> hardware level of the page tables and always has been.

With the history in mind, yes.

Looking at the current state it's more complicated.

Because pgd_present() are really just dummy functions. And this series proposes
to let also pgdp_get() be a dummy function and disallow something like

	set_pgd(pgdp_get())

Which is just asking for trouble. Similar with using pgd_page(pgdp_get()) on

We could get away with catching that case similar to how we catch pgd_offset()
abuse, but I'd rather try get rid of the wrapper functions set_pgd() /
pgd_page() entirely.

But if we feel like that is too much, we can look into that direction.

> 
> I really do think this series is changing things, not keeping them the same.

Looking at it from that angle, I tend do agree: it further moves into the
direction that actually the p4d_present() carries value, and the previous
pgd_present() is just a dummy.

I can see that some more work is required to untangle the mess we already have.

I don't know why pgd_present() was made a dummy function and not p4d_present(),
but I'm sure there is a good reason to it. Changing that now ... omg.

-- 
Cheers,

David

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[Dave Hansen]

On 7/13/26 12:49, David Hildenbrand (Arm) wrote:
>> I really do think this series is changing things, not keeping them the same.
> Looking at it from that angle, I tend do agree: it further moves into the
> direction that actually the p4d_present() carries value, and the previous
> pgd_present() is just a dummy.
> 
> I can see that some more work is required to untangle the mess we already have.
> 
> I don't know why pgd_present() was made a dummy function and not p4d_present(),
> but I'm sure there is a good reason to it. Changing that now ... omg.

Yeah, the p4d addition added even more wrinkles because it was the first
time we tried runtime folding in addition to the historical compile-time
folding.

There were certainly some ugly corners of it that we never found a nice
way to fix. I know Kiryl tried pretty hard.

But, zooming back out... I don't think this diffstat:

 31 files changed, 329 insertions(+), 206 deletions(-)

is nearly justified to eliminate one virtually guaranteed cached memory
read instruction. "Unnecessary", sure. But, worth the diffstat? I'm not
quite there.

Also, I think a series that adds code like this:

> -	if (level > 0) {
> +	/* Ignore folded levels ... */
> +	if (((level == 0) && mm_p4d_folded(st->mm)) ||
> +	    ((level == 1) && mm_pud_folded(st->mm)) ||
> +	    ((level == 2) && mm_pmd_folded(st->mm)))
> +		return;
> +
> +	/* ... and make the actual first level remember the protection. */
> +	if (((level == 0)) ||
> +	    ((level == 1) && mm_p4d_folded(st->mm)) ||
> +	    ((level == 2) && mm_pud_folded(st->mm)) ||
> +	    ((level == 3) && mm_pmd_folded(st->mm)))
> +		first_level = true;

fails the sniff test of being an effective refactor. It unquestionably
makes _that_ code worse, not better.

If the series was:

	Subject: Refactor page table folding in all architectures

and we had a discussion about what the best overall thing to do for all
architectures is, I think we might have a better path forward. It would
also be a smaller series.

To me, the double READ_ONCE() is a non-issue. Having a nice, coherent
page table API in the kernel is much more important to talk about.

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[David Hildenbrand (Arm)]

On 7/13/26 22:27, Dave Hansen wrote:
> On 7/13/26 12:49, David Hildenbrand (Arm) wrote:
>>> I really do think this series is changing things, not keeping them the same.
>> Looking at it from that angle, I tend do agree: it further moves into the
>> direction that actually the p4d_present() carries value, and the previous
>> pgd_present() is just a dummy.
>>
>> I can see that some more work is required to untangle the mess we already have.
>>
>> I don't know why pgd_present() was made a dummy function and not p4d_present(),
>> but I'm sure there is a good reason to it. Changing that now ... omg.
> 
> Yeah, the p4d addition added even more wrinkles because it was the first
> time we tried runtime folding in addition to the historical compile-time
> folding.
> 
> There were certainly some ugly corners of it that we never found a nice
> way to fix. I know Kiryl tried pretty hard.
> 
> But, zooming back out... I don't think this diffstat:
> 
>  31 files changed, 329 insertions(+), 206 deletions(-)
> 
> is nearly justified to eliminate one virtually guaranteed cached memory
> read instruction. "Unnecessary", sure. But, worth the diffstat? I'm not
> quite there.

Just to emphasize, this prevents common code from being converted to pgdp_get()
and friends, as it would make folded page tables more expensive in other page
table walkers, and other arch maintainers rightfully complained about that.

I don't want to be stuck with this inconsistent mess of some common code page
walkers using pgp_get(), while others are stopped from being converted due to
inefficiency concerns.

*that* makes our common MM code worse, and it grinds my gears.

And I really care more about common MM code being clean and consistent than some
rusty old 32bit code being slightly worse.

As discussed, we can keep the set_pgd() monstrosity from working in all places
that don't use something like set_pgd(pgp_get()), bust as I raised off-list,
setting something that is unconditionally pgd_present() -- a dummy -- is just
extremely ugly. But ugly seems to be the theme in folded page table land.

That would certainly be less churn, while leaving this questionable code in
place, I'm fine with that as long as we don't add silent bugs when some arch
start doing e.g., set_pgd(pgp_get()).

> 
> Also, I think a series that adds code like this:
> 
>> -	if (level > 0) {
>> +	/* Ignore folded levels ... */
>> +	if (((level == 0) && mm_p4d_folded(st->mm)) ||
>> +	    ((level == 1) && mm_pud_folded(st->mm)) ||
>> +	    ((level == 2) && mm_pmd_folded(st->mm)))
>> +		return;
>> +
>> +	/* ... and make the actual first level remember the protection. */
>> +	if (((level == 0)) ||
>> +	    ((level == 1) && mm_p4d_folded(st->mm)) ||
>> +	    ((level == 2) && mm_pud_folded(st->mm)) ||
>> +	    ((level == 3) && mm_pmd_folded(st->mm)))
>> +		first_level = true;
> 
> fails the sniff test of being an effective refactor. It unquestionably
> makes _that_ code worse, not better.

As I said, we can improve that part. But we'll have to stop faking that each
level exist, when it's really folded.

That's all this code does, and the first part of this logic could be had in
common code easily (just don't call on folded page tables).

> 
> If the series was:
> 
> 	Subject: Refactor page table folding in all architectures
> 
> and we had a discussion about what the best overall thing to do for all
> architectures is, I think we might have a better path forward. It would
> also be a smaller series.

It took me too long to understand why a "nop4d.h" file defines PGD functions to
be dummies and assumes an arch to implement p4d functions.

So yes, in a perfect world we'd rewrite folded page table handling entirely.

BUT

Let's focus on the real problem:

Common code cannot consistently use pgdp_get() etc. as it harms configs with
folded page tables.

And that problem shouldn't require refactoring/rewriting all of page table
folding at this point.

> 
> To me, the double READ_ONCE() is a non-issue.

To other arch maintainers, it is!

> Having a nice, coherent
> page table API in the kernel is much more important to talk about.

While I agree, we are talking about avoiding degrading folded page table configs
as we make common code more consistent.

I hate our folded page table code when looking it from an arch perspective
(common code walking just works), and I agree that it should be improved. But I
disagree that it should be blocking of core-mm using page table getters
consistently.

-- 
Cheers,

David

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[Dave Hansen]

On 7/13/26 14:17, David Hildenbrand (Arm) wrote:
>> To me, the double READ_ONCE() is a non-issue.
> To other arch maintainers, it is!

Ahh, so it seems like Christophe in ppc32 land was looking at this more
like a regression that needed to get fixed.

Christophe, just out of curiosity, was this something that was causing
you practical problems like measurable performance regressions, or was
it really just insane code generation that seems unacceptably suboptimal?

Re: [RFC PATCH 02/34] ARM: mm: make 2-level pgd_t a scalar

[Pedro Falcato]

On Mon, Jul 13, 2026 at 02:55:41PM +0100, Yeoreum Yun wrote:
> From: "David Hildenbrand (Arm)" <david@kernel.org>
> 
> We don't want pgd_t to be an array, as it prohibits returning it from a
> function, like pgdp_get().
> 
> So let's just use an u64, and extract the right 32bit value in
> pgd_val().
> 
> Leave the STRICT_MM_TYPECHECKS case alone for now.

I have to ask: is there a good reason for the STRICT_MM_TYPECHECKS ifdef?

I see the compiler has an awkward time returning a u64 struct (see
https://godbolt.org/z/qejbv6j9a), but if this doesn't work maybe we should
get rid of the STRICT_MM_TYPECHECKS stuff? I seriously doubt anyone is
purposefully toggling it on for testing from time to time.

If STRICT_MM_TYPEDEFS's worse codegen doesn't matter then maybe we should
permanently toggle it on.
> 
> As an alternative, we could use the STRICT_MM_TYPECHECKS approach here
> as well, but using an u64 looks conceptually cleaner, even though
> pgd_val() gets a bit more involved.
> 
> Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> ---
>  arch/arm/include/asm/pgtable-2level-types.h | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/arm/include/asm/pgtable-2level-types.h b/arch/arm/include/asm/pgtable-2level-types.h
> index 650e793f41429..02052cef9437a 100644
> --- a/arch/arm/include/asm/pgtable-2level-types.h
> +++ b/arch/arm/include/asm/pgtable-2level-types.h
> @@ -25,7 +25,7 @@ typedef struct { pteval_t pgprot; } pgprot_t;
>  
>  #define pte_val(x)      ((x).pte)
>  #define pmd_val(x)      ((x).pmd)
> -#define pgd_val(x)	((x).pgd[0])
> +#define pgd_val(x)      ((x).pgd[0])
>  #define pgprot_val(x)   ((x).pgprot)
>  
>  #define __pte(x)        ((pte_t) { (x) } )
> @@ -36,14 +36,21 @@ typedef struct { pteval_t pgprot; } pgprot_t;
>  /*
>   * .. while these make it easier on the compiler
>   */
> +typedef u64 pgdval_t;
> +
>  typedef pteval_t pte_t;
>  typedef pmdval_t pmd_t;
> -typedef pmdval_t pgd_t[2];
> +typedef pgdval_t pgd_t;
>  typedef pteval_t pgprot_t;
>  
>  #define pte_val(x)      (x)
>  #define pmd_val(x)      (x)
> -#define pgd_val(x)	((x)[0])
> +
> +static inline pmdval_t pgd_val(pgd_t pgd)
> +{
> +	return (*(pmdval_t (*)[2])&pgd)[0];

Ugh. This isn't correct C code. It only works because the kernel passes
-fno-strict-aliasing. I would recommend either forcing a struct here, or
using a u64 with bitmasks/shifts.

-- 
Pedro

Re: [RFC PATCH 10/34] x86: mm: carve out the generic compile-time folded pgtable case in effective_prot()

[David Hildenbrand (Arm)]

On 7/13/26 23:40, Dave Hansen wrote:
> On 7/13/26 14:17, David Hildenbrand (Arm) wrote:
>>> To me, the double READ_ONCE() is a non-issue.
>> To other arch maintainers, it is!
> 
> Ahh, so it seems like Christophe in ppc32 land was looking at this more
> like a regression that needed to get fixed.

Yes, exactly. :)

And I agree that it is something we should be optimizing.

> 
> Christophe, just out of curiosity, was this something that was causing
> you practical problems like measurable performance regressions, or was
> it really just insane code generation that seems unacceptably suboptimal?

I'd be curious about that as well. I mean, looking at the generated code
it's clear that it is suboptimal.


The solution space I see:


(1) Pass the result from e.g., pgdp_get() into p4d_get(), so it can just return 
    the value with folded p4ds.

That requires extreme amounts of churn in core-mm AFAIKS, so I don't see that as feasible.


(2) Rewrite folding code to make p4d_present() be a dummy instead of 
    pgd_present(), and make p4dp_get() return a dummy value.

... a lot of churn across architectures. I'm sure we'll learn soon why it was done 
ike the way it is today in the first place. Something interesting to look at,
certainly, but a bit of a stretch just to optimize reads.


(3) Make folded pgdp_get() use an ordinary read instead of a READ_ONCE / dummy.

I don't like that, because we couldn't catch easily if the value is then
actually used some old/new code.

If pgd_present() etc are supposed to ignore the value, I'd rather have a mechanism
that enforces that the values are actually ignore in pgd_offset() etc as well.


(4) What we do in this series, but instead of forbidding set_pgd(), make it only 
    complain when we are passing in a dummy value.

The expectation is that it would avoid touching most architecture code in patch
12 -> 27 and still prevent mistakes in the future.

@Yeoreum can you play with that and see what the end result would be?


The problem of how to handle ptdump effective_prot remains. Which is unfortunately an
x86 32-bit only problem ;) ... and effective_prot is essentially an x86-only thing.

There are various ways we could handle that, my preferred one would be

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index fb298e2191792..3d56e48fe4151 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -96,7 +96,7 @@ config X86
        select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
        select ARCH_HAS_PMEM_API                if X86_64
        select ARCH_HAS_PREEMPT_LAZY
-       select ARCH_HAS_PTDUMP
+       select ARCH_HAS_PTDUMP                  if X86_64
        select ARCH_HAS_PTE_SPECIAL
        select ARCH_HAS_HW_PTE_YOUNG
        select ARCH_HAS_NONLEAF_PMD_YOUNG       if PGTABLE_LEVELS > 2


Another option would be to keep the behavior unchanged:


diff --git a/mm/ptdump.c b/mm/ptdump.c
index 5851096e6f656..ce2cf5e07ac0a 100644
--- a/mm/ptdump.c
+++ b/mm/ptdump.c
@@ -39,6 +39,9 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long addr,
                return note_kasan_page_table(walk, addr);
 #endif
 
+       if (mm_p4d_folded(walk->mm))
+               return 0;
+
        if (st->effective_prot_pgd)
                st->effective_prot_pgd(st, val);
 
@@ -62,6 +65,13 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long addr,
                return note_kasan_page_table(walk, addr);
 #endif
 
+       if (mm_pud_folded(walk->mm))
+               return 0;
+
+       /* Simulate old behavior: all page table levels exist with folded values. */
+       if (mm_p4d_folded(walk->mm) && (st->effective_prot_pgd))
+               st->effective_prot_pgd(st, __pgd(p4d_val(val)));
+
        if (st->effective_prot_p4d)
                st->effective_prot_p4d(st, val);
 
@@ -85,6 +95,15 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long addr,
                return note_kasan_page_table(walk, addr);
 #endif
 
+       if (mm_pmd_folded(walk->mm))
+               return 0;
+
+       /* Simulate old behavior: all page table levels exist with folded values. */
+       if (mm_pud_folded(walk->mm) && st->effective_prot_pgd)
+               st->effective_prot_pgd(st, __pgd(pud_val(val)));
+       if (mm_p4d_folded(walk->mm) && st->effective_prot_p4d)
+               st->effective_prot_p4d(st, __p4d(pud_val(val)));
+
        if (st->effective_prot_pud)
                st->effective_prot_pud(st, val);
 
@@ -107,6 +126,14 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long addr,
                return note_kasan_page_table(walk, addr);
 #endif
 
+       /* Simulate old behavior: all page table levels exist with folded values. */
+       if (mm_pud_folded(walk->mm) && st->effective_prot_pgd)
+               st->effective_prot_pgd(st, __pgd(pmd_val(val)));
+       if (mm_p4d_folded(walk->mm) && st->effective_prot_p4d)
+               st->effective_prot_p4d(st, __p4d(pmd_val(val)));
+       if (mm_pmd_folded(walk->mm) && st->effective_prot_pud)
+               st->effective_prot_pud(st, __pud(pmd_val(val)));
+
        if (st->effective_prot_pmd)
                st->effective_prot_pmd(st, val);
        if (pmd_leaf(val)) {


Then there would be another option of making x86 simply remembering the first level it got called for.


-- 
Cheers,

David

Re: [RFC PATCH 02/34] ARM: mm: make 2-level pgd_t a scalar

[David Hildenbrand (Arm)]

On 7/14/26 12:26, Pedro Falcato wrote:
> On Mon, Jul 13, 2026 at 02:55:41PM +0100, Yeoreum Yun wrote:
>> From: "David Hildenbrand (Arm)" <david@kernel.org>
>>
>> We don't want pgd_t to be an array, as it prohibits returning it from a
>> function, like pgdp_get().
>>
>> So let's just use an u64, and extract the right 32bit value in
>> pgd_val().
>>
>> Leave the STRICT_MM_TYPECHECKS case alone for now.
> 
> I have to ask: is there a good reason for the STRICT_MM_TYPECHECKS ifdef?
> 
> I see the compiler has an awkward time returning a u64 struct (see
> https://godbolt.org/z/qejbv6j9a), but if this doesn't work maybe we should
> get rid of the STRICT_MM_TYPECHECKS stuff? I seriously doubt anyone is
> purposefully toggling it on for testing from time to time.
> 
> If STRICT_MM_TYPEDEFS's worse codegen doesn't matter then maybe we should
> permanently toggle it on.
>>
>> As an alternative, we could use the STRICT_MM_TYPECHECKS approach here
>> as well, but using an u64 looks conceptually cleaner, even though
>> pgd_val() gets a bit more involved.
>>
>> Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
>> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
>> ---
>>  arch/arm/include/asm/pgtable-2level-types.h | 13 ++++++++++---
>>  1 file changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/arm/include/asm/pgtable-2level-types.h b/arch/arm/include/asm/pgtable-2level-types.h
>> index 650e793f41429..02052cef9437a 100644
>> --- a/arch/arm/include/asm/pgtable-2level-types.h
>> +++ b/arch/arm/include/asm/pgtable-2level-types.h
>> @@ -25,7 +25,7 @@ typedef struct { pteval_t pgprot; } pgprot_t;
>>  
>>  #define pte_val(x)      ((x).pte)
>>  #define pmd_val(x)      ((x).pmd)
>> -#define pgd_val(x)	((x).pgd[0])
>> +#define pgd_val(x)      ((x).pgd[0])
>>  #define pgprot_val(x)   ((x).pgprot)
>>  
>>  #define __pte(x)        ((pte_t) { (x) } )
>> @@ -36,14 +36,21 @@ typedef struct { pteval_t pgprot; } pgprot_t;
>>  /*
>>   * .. while these make it easier on the compiler
>>   */
>> +typedef u64 pgdval_t;
>> +
>>  typedef pteval_t pte_t;
>>  typedef pmdval_t pmd_t;
>> -typedef pmdval_t pgd_t[2];
>> +typedef pgdval_t pgd_t;
>>  typedef pteval_t pgprot_t;
>>  
>>  #define pte_val(x)      (x)
>>  #define pmd_val(x)      (x)
>> -#define pgd_val(x)	((x)[0])
>> +
>> +static inline pmdval_t pgd_val(pgd_t pgd)
>> +{
>> +	return (*(pmdval_t (*)[2])&pgd)[0];
> 
> Ugh. This isn't correct C code. It only works because the kernel passes
> -fno-strict-aliasing.

So it is correct C code? ;)

> I would recommend either forcing a struct here, or
> using a u64 with bitmasks/shifts.

I had that, but little vs. big endian was annoying.

-- 
Cheers,

David