[PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Kiryl Shutsemau (Meta)]

Zapping a range calls arch_check_zapped_pte() for every batch of PTEs,
which is every page for small folios, and arch_check_zapped_pmd() or
arch_check_zapped_pud() for every huge leaf. On x86 these are out-of-line
functions whose only content is a VM_WARN_ON_ONCE(), so without
CONFIG_DEBUG_VM they are empty, but the call remains. With the SRSO "Safe
RET" mitigation the return of every such call mispredicts.

Declare and define the checks only with CONFIG_DEBUG_VM. Otherwise the
empty inlines from <linux/pgtable.h> apply, as on every other
architecture. They cannot be inlines in <asm/pgtable.h>: VM_SHADOW_STACK
is defined in <linux/mm.h> after it includes <linux/pgtable.h>.

In a KVM guest on Zen 4 with Safe RET, zapping 4K pages takes one branch
miss per page fewer. MADV_DONTNEED of 1 GiB gets 6% faster on anonymous
memory and 8% on shmem, munmap() 5%. THP is unchanged.

Fixes: e5136e876581 ("mm: Warn on shadow stack memory in wrong vma")
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/x86/include/asm/pgtable.h | 2 ++
 arch/x86/mm/pgtable.c          | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 000fc6c1ca37..9b03a316fb75 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -1666,6 +1666,7 @@ static inline bool arch_has_pfn_modify_check(void)
 	return boot_cpu_has_bug(X86_BUG_L1TF);
 }
 
+#ifdef CONFIG_DEBUG_VM
 #define arch_check_zapped_pte arch_check_zapped_pte
 void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte);
 
@@ -1674,6 +1675,7 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd);
 
 #define arch_check_zapped_pud arch_check_zapped_pud
 void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud);
+#endif
 
 #ifdef CONFIG_XEN_PV
 #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 4a105f283cfb..3352bc6904f3 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -798,6 +798,7 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 	return pmd_clear_saveddirty(pmd);
 }
 
+#ifdef CONFIG_DEBUG_VM
 void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
 {
 	/*
@@ -823,3 +824,4 @@ void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud)
 	/* See note in arch_check_zapped_pte() */
 	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pud_shstk(pud));
 }
+#endif
-- 
2.54.0

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Jose A. Perez de Azpillaga]

On 11/2/26 3:39 PM, Kiryl Shutsemau (Meta) wrote:
> Zapping a range calls arch_check_zapped_pte() for every batch of PTEs,
> which is every page for small folios, and arch_check_zapped_pmd() or
> arch_check_zapped_pud() for every huge leaf. On x86 these are out-of-line
> functions whose only content is a VM_WARN_ON_ONCE(), so without
> CONFIG_DEBUG_VM they are empty, but the call remains. With the SRSO "Safe
> RET" mitigation the return of every such call mispredicts.
> 
> Declare and define the checks only with CONFIG_DEBUG_VM. Otherwise the
> empty inlines from <linux/pgtable.h> apply, as on every other
> architecture. They cannot be inlines in <asm/pgtable.h>: VM_SHADOW_STACK
> is defined in <linux/mm.h> after it includes <linux/pgtable.h>.
> 
> In a KVM guest on Zen 4 with Safe RET, zapping 4K pages takes one branch
> miss per page fewer. MADV_DONTNEED of 1 GiB gets 6% faster on anonymous
> memory and 8% on shmem, munmap() 5%. THP is unchanged.
> 
> Fixes: e5136e876581 ("mm: Warn on shadow stack memory in wrong vma")
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
LGTM:

Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>

-- 
cheers,
jose a. p-a

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Edgecombe, Rick P]

On Mon, 2026-11-02 at 14:39 +0000, Kiryl Shutsemau (Meta) wrote:
> Zapping a range calls arch_check_zapped_pte() for every batch of PTEs,
> which is every page for small folios, and arch_check_zapped_pmd() or
> arch_check_zapped_pud() for every huge leaf. On x86 these are out-of-line
> functions whose only content is a VM_WARN_ON_ONCE(), so without
> CONFIG_DEBUG_VM they are empty, but the call remains. With the SRSO "Safe
> RET" mitigation the return of every such call mispredicts.
> 
> Declare and define the checks only with CONFIG_DEBUG_VM. Otherwise the
> empty inlines from <linux/pgtable.h> apply, as on every other
> architecture. They cannot be inlines in <asm/pgtable.h>: VM_SHADOW_STACK
> is defined in <linux/mm.h> after it includes <linux/pgtable.h>.
> 
> In a KVM guest on Zen 4 with Safe RET, zapping 4K pages takes one branch
> miss per page fewer. MADV_DONTNEED of 1 GiB gets 6% faster on anonymous
> memory and 8% on shmem, munmap() 5%. THP is unchanged.

I guess this is the impact of this fix on Safe RET machines, but it's a little
ambiguous. With the assumption that a mitigation didn't result in the
performance improvement I can parse it.

Do you happen to know if this performance impact is relevant at all in the non
Safe RET case?

> 
> Fixes: e5136e876581 ("mm: Warn on shadow stack memory in wrong vma")
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  arch/x86/include/asm/pgtable.h | 2 ++
>  arch/x86/mm/pgtable.c          | 2 ++
>  2 files changed, 4 insertions(+)
> 
> diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
> index 000fc6c1ca37..9b03a316fb75 100644
> --- a/arch/x86/include/asm/pgtable.h
> +++ b/arch/x86/include/asm/pgtable.h
> @@ -1666,6 +1666,7 @@ static inline bool arch_has_pfn_modify_check(void)
>  	return boot_cpu_has_bug(X86_BUG_L1TF);
>  }
>  
> +#ifdef CONFIG_DEBUG_VM
>  #define arch_check_zapped_pte arch_check_zapped_pte
>  void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte);
>  
> @@ -1674,6 +1675,7 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma,
> pmd_t pmd);
>  
>  #define arch_check_zapped_pud arch_check_zapped_pud
>  void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud);
> +#endif
>  
>  #ifdef CONFIG_XEN_PV
>  #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young
> diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
> index 4a105f283cfb..3352bc6904f3 100644
> --- a/arch/x86/mm/pgtable.c
> +++ b/arch/x86/mm/pgtable.c
> @@ -798,6 +798,7 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
>  	return pmd_clear_saveddirty(pmd);
>  }
>  
> +#ifdef CONFIG_DEBUG_VM
>  void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
>  {
>  	/*
> @@ -823,3 +824,4 @@ void arch_check_zapped_pud(struct vm_area_struct *vma,
> pud_t pud)
>  	/* See note in arch_check_zapped_pte() */
>  	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
> pud_shstk(pud));

I guess all of these don't need to be the "VM_WARN_" varieties if they are
compiled out in the other case.

>  }
> +#endif

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Kiryl Shutsemau]

On Mon, Nov 02, 2026 at 04:53:17PM +0000, Edgecombe, Rick P wrote:
> Do you happen to know if this performance impact is relevant at all in the non
> Safe RET case?

Yes, both sides of the comparison are the same kernel with Safe RET on,
with and without the patch.

Without Safe RET (spec_rstack_overflow=off, same two kernels) the call
and return predict fine: branch misses are the same with and without the
patch, and the gain is under 2%, within noise.

> > diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
> > index 4a105f283cfb..3352bc6904f3 100644
> > --- a/arch/x86/mm/pgtable.c
> > +++ b/arch/x86/mm/pgtable.c
> > @@ -798,6 +798,7 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
> >  	return pmd_clear_saveddirty(pmd);
> >  }
> >  
> > +#ifdef CONFIG_DEBUG_VM
> >  void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
> >  {
> >  	/*
> > @@ -823,3 +824,4 @@ void arch_check_zapped_pud(struct vm_area_struct *vma,
> > pud_t pud)
> >  	/* See note in arch_check_zapped_pte() */
> >  	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
> > pud_shstk(pud));
> 
> I guess all of these don't need to be the "VM_WARN_" varieties if they are
> compiled out in the other case.

I can do either way. It is functionally equivalent.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Edgecombe, Rick P]

On Mon, 2026-11-02 at 17:32 +0000, Kiryl Shutsemau wrote:
> On Mon, Nov 02, 2026 at 04:53:17PM +0000, Edgecombe, Rick P wrote:
> > Do you happen to know if this performance impact is relevant at all in the
> > non Safe RET case?
> 
> Yes, both sides of the comparison are the same kernel with Safe RET on,
> with and without the patch.
> 
> Without Safe RET (spec_rstack_overflow=off, same two kernels) the call
> and return predict fine: branch misses are the same with and without the
> patch, and the gain is under 2%, within noise.

Hmm, I'd wonder if the fixes tag is appropriate then. Should it be the safe ret
commits?

> 
> > > diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
> > > index 4a105f283cfb..3352bc6904f3 100644
> > > --- a/arch/x86/mm/pgtable.c
> > > +++ b/arch/x86/mm/pgtable.c
> > > @@ -798,6 +798,7 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct
> > > *vma)
> > >  	return pmd_clear_saveddirty(pmd);
> > >  }
> > >  
> > > +#ifdef CONFIG_DEBUG_VM
> > >  void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
> > >  {
> > >  	/*
> > > @@ -823,3 +824,4 @@ void arch_check_zapped_pud(struct vm_area_struct *vma,
> > > pud_t pud)
> > >  	/* See note in arch_check_zapped_pte() */
> > >  	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
> > > pud_shstk(pud));
> > 
> > I guess all of these don't need to be the "VM_WARN_" varieties if they are
> > compiled out in the other case.
> 
> I can do either way. It is functionally equivalent.

Otherwise it's effectively a duplicate check.

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Dave Hansen]

On 11/2/26 06:39, Kiryl Shutsemau (Meta) wrote:
...
> +#ifdef CONFIG_DEBUG_VM
>  void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
>  {
>  	/*
> @@ -823,3 +824,4 @@ void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud)
>  	/* See note in arch_check_zapped_pte() */
>  	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pud_shstk(pud));
>  }
> +#endif

So all these #ifdefs are because VM_SHADOW_STACK, a *constant* is not
available? That's unfortunate.

I dislike doing this gunk with macros, but I'd take that over new
#ifdefs. Or, settle the "does the arch provide these debug functions?"
at Kconfig time with an ARCH_HAS_FOO option. That could even be driven
from the x86 side with:

	select ARCH_HAS_CHECK_ZAPPED_PTE if DEBUG_VM

Or, put them in a new header. They're only used in memory.c, so what's
keeping them in pgtable.h? Another header could reference
VM_SHADOW_STACK. No?

Or, heck, I'm not that attached to these checks. Let's just kill them.

But I really don't like the new #ifdefs. It would be about my fifth
favorite way to fix this.

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Kiryl Shutsemau]

On Mon, Nov 02, 2026 at 01:12:47PM -0800, Dave Hansen wrote:
> Or, heck, I'm not that attached to these checks. Let's just kill them.

Works for me. x86 is the only architecture with them, although arm64 and
riscv have shadow stacks too, and the x86 helpers already keep
Write=0,Dirty=1 out of other VMAs. Patch below, without the Fixes: tag.

Rick, any objection?

-- >8 --
From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
Subject: [PATCH] x86/mm: Remove the zapped-entry checks

Zapping a range calls arch_check_zapped_pte() for every batch of PTEs,
which is every page for small folios, and arch_check_zapped_pmd() or
arch_check_zapped_pud() for every huge leaf. Only x86 implements them, as
out-of-line functions that warn about a shadow stack entry (Write=0,
Dirty=1) in a VMA without VM_SHADOW_STACK.

The warning is a VM_WARN_ON_ONCE(), so without CONFIG_DEBUG_VM the
functions are empty, but the calls remain. With the SRSO "Safe RET"
mitigation the return of every such call mispredicts.

The x86 helpers already keep such entries out of other VMAs.
pte_wrprotect(), pte_mkdirty(), pte_modify() and their PMD versions go
through SavedDirty, and only pte_mkwrite() and pmd_mkwrite() on a
VM_SHADOW_STACK VMA create the combination.

Remove the checks, their callers and pud_shstk(), which has no other
user.

In a KVM guest on Zen 4 with Safe RET, zapping 4K pages takes one branch
miss per page fewer. MADV_DONTNEED of 1 GiB gets 6% faster on anonymous
memory and 8% on shmem, munmap() 5%. THP is unchanged. Without Safe RET
the difference is within noise.

Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/x86/include/asm/pgtable.h | 16 ----------------
 arch/x86/mm/pgtable.c          | 26 --------------------------
 include/linux/pgtable.h        | 20 --------------------
 mm/huge_memory.c               |  2 --
 mm/memory.c                    |  3 ---
 5 files changed, 67 deletions(-)

diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 000fc6c1ca37..613c22c7838a 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -199,13 +199,6 @@ static inline int pud_young(pud_t pud)
 	return pud_flags(pud) & _PAGE_ACCESSED;
 }
 
-static inline bool pud_shstk(pud_t pud)
-{
-	return cpu_feature_enabled(X86_FEATURE_SHSTK) &&
-	       (pud_flags(pud) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) ==
-	       (_PAGE_DIRTY | _PAGE_PSE);
-}
-
 static inline int pte_write(pte_t pte)
 {
 	/*
@@ -1666,15 +1659,6 @@ static inline bool arch_has_pfn_modify_check(void)
 	return boot_cpu_has_bug(X86_BUG_L1TF);
 }
 
-#define arch_check_zapped_pte arch_check_zapped_pte
-void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte);
-
-#define arch_check_zapped_pmd arch_check_zapped_pmd
-void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd);
-
-#define arch_check_zapped_pud arch_check_zapped_pud
-void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud);
-
 #ifdef CONFIG_XEN_PV
 #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young
 static inline bool arch_has_hw_nonleaf_pmd_young(void)
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 4a105f283cfb..58e5012b56b0 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -797,29 +797,3 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 
 	return pmd_clear_saveddirty(pmd);
 }
-
-void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
-{
-	/*
-	 * Hardware before shadow stack can (rarely) set Dirty=1
-	 * on a Write=0 PTE. So the below condition
-	 * only indicates a software bug when shadow stack is
-	 * supported by the HW. This checking is covered in
-	 * pte_shstk().
-	 */
-	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
-			pte_shstk(pte));
-}
-
-void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd)
-{
-	/* See note in arch_check_zapped_pte() */
-	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
-			pmd_shstk(pmd));
-}
-
-void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud)
-{
-	/* See note in arch_check_zapped_pte() */
-	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pud_shstk(pud));
-}
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 780fe849ff8b..fc32289af58a 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -621,26 +621,6 @@ static inline unsigned int exec_folio_order(void)
 }
 #endif
 
-#ifndef arch_check_zapped_pte
-static inline void arch_check_zapped_pte(struct vm_area_struct *vma,
-					 pte_t pte)
-{
-}
-#endif
-
-#ifndef arch_check_zapped_pmd
-static inline void arch_check_zapped_pmd(struct vm_area_struct *vma,
-					 pmd_t pmd)
-{
-}
-#endif
-
-#ifndef arch_check_zapped_pud
-static inline void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud)
-{
-}
-#endif
-
 #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 				       unsigned long address,
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd44d58e8684..6e49fa841911 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2558,7 +2558,6 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	 */
 	orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd,
 						tlb->fullmm);
-	arch_check_zapped_pmd(vma, orig_pmd);
 	tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
 
 	is_present = pmd_present(orig_pmd);
@@ -3045,7 +3044,6 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		return 0;
 
 	orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm);
-	arch_check_zapped_pud(vma, orig_pud);
 	tlb_remove_pud_tlb_entry(tlb, pud, addr);
 	if (!vma_is_mm_managed(vma)) {
 		spin_unlock(ptl);
diff --git a/mm/memory.c b/mm/memory.c
index 0fb676dd2bd6..25d630033642 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1755,8 +1755,6 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb,
 		clear_full_ptes(mm, addr, pte, nr, tlb->fullmm);
 		rss[MM_ANONPAGES] -= nr;
 	}
-	/* Checking a single PTE in a batch is sufficient. */
-	arch_check_zapped_pte(vma, ptent);
 	tlb_remove_tlb_entries(tlb, pte, nr, addr);
 	if (unlikely(userfaultfd_pte_wp(vma, ptent)))
 		*any_skipped = zap_install_uffd_wp_if_needed(vma, addr, pte,
@@ -1795,7 +1793,6 @@ static inline int zap_present_ptes(struct mmu_gather *tlb,
 	if (!page) {
 		/* We don't need up-to-date accessed/dirty bits. */
 		ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
-		arch_check_zapped_pte(vma, ptent);
 		tlb_remove_tlb_entry(tlb, pte, addr);
 		if (userfaultfd_pte_wp(vma, ptent))
 			*any_skipped = zap_install_uffd_wp_if_needed(vma, addr,
-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] x86/mm: Define the zapped-entry checks only with CONFIG_DEBUG_VM

[Edgecombe, Rick P]

On Tue, 2026-11-03 at 10:13 +0000, Kiryl Shutsemau wrote:
> On Mon, Nov 02, 2026 at 01:12:47PM -0800, Dave Hansen wrote:
> > Or, heck, I'm not that attached to these checks. Let's just kill them.
> 
> Works for me. x86 is the only architecture with them, although arm64 and
> riscv have shadow stacks too, and the x86 helpers already keep
> Write=0,Dirty=1 out of other VMAs. Patch below, without the Fixes: tag.
> 
> Rick, any objection?

Eh, I guess I'm lukewarm on it. These checks only catch the case at zap time, so
it's not a huge protection that is being pulled out. I thought they were useful.
At the same time, they have never triggered.