Linux Hardening
 help / color / mirror / Atom feed
* [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix
@ 2026-11-19 21:31 Usama Arif
  2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
  2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
  0 siblings, 2 replies; 17+ messages in thread
From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw)
  To: tglx, mingo, bp, dave.hansen, x86, kees
  Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo,
	justinstitt, linux-hardening, llvm, linux-kernel, riel,
	shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid,
	Usama Arif

With KSTACK_ERASE, every return to user space poisons the part of the
task stack used since the previous return, one 8-byte store at a time.
On Meta hosts running many small VMs, that loop costs about 1.5% of
non-idle host kernel cycles. Most of it is on the way out of futex,
epoll, procfs and socket syscalls that go 1-2 KiB deep.

Patch 2 makes x86-64 use REP STOSQ for spans above 512 bytes, through
the arch hook that s390 already uses. Shorter spans keep the loop, and
the range and value written do not change.

Patch 1 is an independent build fix I found while testing patch 2 with a
small KSTACK_ERASE_TRACK_MIN_SIZE. The startup code still uses the
pre-rename DISABLE_STACKLEAK_PLUGIN, so with Clang, AMD memory
encryption and a tracking minimum below about 90 bytes, vmlinux fails
to link. It can be applied on its own.

Zen 5c KVM guest, one vCPU, before -> after:

  stackleak_erase() per return to user       71 ns -> 46 ns
  futex wake/wait ping-pong                  5-7% faster
  Unix socket sendmsg()/recvmsg() ping-pong  5-8% faster
  open+read+close of /proc/self/stat         6-8% faster
  getppid() (spans below 512 bytes)          within 2%
 
Usama Arif (2):
  x86/boot/startup: Disable kernel stack depth tracking
  x86/kstack_erase: Use REP STOSQ for large stack fills

 arch/x86/boot/startup/Makefile   |  2 +-
 arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
 2 files changed, 26 insertions(+), 1 deletion(-)

-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking
  2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
@ 2026-11-19 21:31 ` Usama Arif
  2026-11-19 22:30   ` Nick Desaulniers
  2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
  1 sibling, 1 reply; 17+ messages in thread
From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw)
  To: tglx, mingo, bp, dave.hansen, x86, kees
  Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo,
	justinstitt, linux-hardening, llvm, linux-kernel, riel,
	shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid,
	Usama Arif

Commit 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE")
renamed DISABLE_STACKLEAK_PLUGIN to DISABLE_KSTACK_ERASE everywhere
except in arch/x86/boot/startup/Makefile. The old name now expands to
nothing, so the startup code is built with stack depth tracking enabled.

The startup code is linked with every symbol prefixed by __pi_, so that
it cannot call into the rest of the kernel by accident. It has no alias
for the tracking callback, and tracking stack depth there is of no use.

At the default KSTACK_ERASE_TRACK_MIN_SIZE of 100 no startup function is
large enough to be tracked, so this went unnoticed. With Clang,
CONFIG_AMD_MEM_ENCRYPT=y and a minimum below about 90 bytes,
pvalidate_4k_page(), snp_cpuid() and do_vc_no_ghcb() get a call to
__sanitizer_cov_stack_depth() and vmlinux fails to link:

  ld.lld: error: undefined symbol: __pi___sanitizer_cov_stack_depth

Use DISABLE_KSTACK_ERASE, which disables tracking with either compiler.

Fixes: 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/boot/startup/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile
index 5e499cfb29b5c0..0815105e7205c7 100644
--- a/arch/x86/boot/startup/Makefile
+++ b/arch/x86/boot/startup/Makefile
@@ -3,7 +3,7 @@
 KBUILD_AFLAGS		+= -D__DISABLE_EXPORTS
 KBUILD_CFLAGS		+= -D__DISABLE_EXPORTS -mcmodel=small -fPIC \
 			   -Os -DDISABLE_BRANCH_PROFILING \
-			   $(DISABLE_STACKLEAK_PLUGIN) \
+			   $(DISABLE_KSTACK_ERASE) \
 			   $(DISABLE_LATENT_ENTROPY_PLUGIN) \
 			   -fno-stack-protector -D__NO_FORTIFY \
 			   -fno-jump-tables \
-- 
2.53.1-Meta


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
  2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
@ 2026-11-19 21:31 ` Usama Arif
  2026-11-19 22:28   ` Nick Desaulniers
                     ` (2 more replies)
  1 sibling, 3 replies; 17+ messages in thread
From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw)
  To: tglx, mingo, bp, dave.hansen, x86, kees
  Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo,
	justinstitt, linux-hardening, llvm, linux-kernel, riel,
	shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid,
	Usama Arif

With KSTACK_ERASE, every return to user space poisons the part of the
task stack that was used since the previous return. The generic
__stackleak_poison() does that one word per loop iteration, so a syscall
that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
running many small VMs, stackleak_erase() accounts for about 1.5% of
non-idle host kernel cycles.

Override __stackleak_poison() on x86-64, as s390 already does, through
the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
specific stackleak poison function"). Fill spans larger than 512 bytes
with REP STOSQ and keep the word loop for shorter ones, where the fixed
startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
slower.

The fill is inline asm, so the noinstr function still makes no calls.
memset64() would be the same instruction, but it is only static inline,
and with KMSAN it is an out-of-line function.

In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c7c..049231d484ad0e 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
 			       current_stack_pointer) < THREAD_SIZE;
 }
 
+#ifdef CONFIG_X86_64
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+					       unsigned long erase_high,
+					       unsigned long poison)
+{
+	unsigned long count;
+
+	/* Short spans do not amortize the startup cost of REP STOSQ. */
+	if (erase_high <= erase_low + 512) {
+		while (erase_low < erase_high) {
+			*(unsigned long *)erase_low = poison;
+			erase_low += sizeof(unsigned long);
+		}
+		return;
+	}
+
+	count = (erase_high - erase_low) / sizeof(unsigned long);
+	asm volatile("rep stosq"
+		     : "+D" (erase_low), "+c" (count)
+		     : "a" (poison)
+		     : "memory");
+}
+#endif
+
 #ifdef CONFIG_PARAVIRT_XXL
 #include <asm/paravirt.h>
 #else
-- 
2.53.1-Meta


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
@ 2026-11-19 22:28   ` Nick Desaulniers
  2026-11-20 11:38     ` David Laight
  2026-11-20 10:45   ` Peter Zijlstra
  2026-11-20 18:55   ` David Laight
  2 siblings, 1 reply; 17+ messages in thread
From: Nick Desaulniers @ 2026-11-19 22:28 UTC (permalink / raw)
  To: Usama Arif
  Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb,
	hca, nathan, morbo, justinstitt, linux-hardening, llvm,
	linux-kernel, riel, shakeel.butt, kernel-team, rkeilty,
	atish.patra, dgreid

On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:
>
> With KSTACK_ERASE, every return to user space poisons the part of the
> task stack that was used since the previous return. The generic
> __stackleak_poison() does that one word per loop iteration, so a syscall
> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> running many small VMs, stackleak_erase() accounts for about 1.5% of
> non-idle host kernel cycles.
>
> Override __stackleak_poison() on x86-64, as s390 already does, through
> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> specific stackleak poison function"). Fill spans larger than 512 bytes
> with REP STOSQ and keep the word loop for shorter ones, where the fixed
> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> slower.
>
> The fill is inline asm, so the noinstr function still makes no calls.
> memset64() would be the same instruction, but it is only static inline,
> and with KMSAN it is an out-of-line function.
>
> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index 77488ff5eb8c7c..049231d484ad0e 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
>                                current_stack_pointer) < THREAD_SIZE;
>  }
>
> +#ifdef CONFIG_X86_64
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +                                              unsigned long erase_high,
> +                                              unsigned long poison)
> +{
> +       unsigned long count;
> +
> +       /* Short spans do not amortize the startup cost of REP STOSQ. */
> +       if (erase_high <= erase_low + 512) {
> +               while (erase_low < erase_high) {
> +                       *(unsigned long *)erase_low = poison;
> +                       erase_low += sizeof(unsigned long);
> +               }
> +               return;
> +       }
> +
> +       count = (erase_high - erase_low) / sizeof(unsigned long);
> +       asm volatile("rep stosq"
> +                    : "+D" (erase_low), "+c" (count)
> +                    : "a" (poison)
> +                    : "memory");

Surely this isn't the only place using `rep stosq` in the tree, right?
It's not, see memset64 in arch/x86/include/asm/string_64.h.

> +}
> +#endif
> +
>  #ifdef CONFIG_PARAVIRT_XXL
>  #include <asm/paravirt.h>
>  #else
> --
> 2.53.1-Meta
>


-- 
Thanks,
~Nick Desaulniers

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking
  2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
@ 2026-11-19 22:30   ` Nick Desaulniers
  0 siblings, 0 replies; 17+ messages in thread
From: Nick Desaulniers @ 2026-11-19 22:30 UTC (permalink / raw)
  To: Usama Arif, Kees Cook
  Cc: tglx, mingo, bp, dave.hansen, x86, hpa, gustavoars, ardb, hca,
	nathan, morbo, justinstitt, linux-hardening, llvm, linux-kernel,
	riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid

On Thu, Nov 19, 2026 at 1:31 PM Usama Arif <usama.arif@linux.dev> wrote:
>
> Commit 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE")
> renamed DISABLE_STACKLEAK_PLUGIN to DISABLE_KSTACK_ERASE everywhere
> except in arch/x86/boot/startup/Makefile. The old name now expands to
> nothing, so the startup code is built with stack depth tracking enabled.
>
> The startup code is linked with every symbol prefixed by __pi_, so that
> it cannot call into the rest of the kernel by accident. It has no alias
> for the tracking callback, and tracking stack depth there is of no use.
>
> At the default KSTACK_ERASE_TRACK_MIN_SIZE of 100 no startup function is
> large enough to be tracked, so this went unnoticed. With Clang,
> CONFIG_AMD_MEM_ENCRYPT=y and a minimum below about 90 bytes,
> pvalidate_4k_page(), snp_cpuid() and do_vc_no_ghcb() get a call to
> __sanitizer_cov_stack_depth() and vmlinux fails to link:
>
>   ld.lld: error: undefined symbol: __pi___sanitizer_cov_stack_depth
>
> Use DISABLE_KSTACK_ERASE, which disables tracking with either compiler.
>
> Fixes: 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE")

tsk tsk, Kees your `grep` might be borked!
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>

> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  arch/x86/boot/startup/Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile
> index 5e499cfb29b5c0..0815105e7205c7 100644
> --- a/arch/x86/boot/startup/Makefile
> +++ b/arch/x86/boot/startup/Makefile
> @@ -3,7 +3,7 @@
>  KBUILD_AFLAGS          += -D__DISABLE_EXPORTS
>  KBUILD_CFLAGS          += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \
>                            -Os -DDISABLE_BRANCH_PROFILING \
> -                          $(DISABLE_STACKLEAK_PLUGIN) \
> +                          $(DISABLE_KSTACK_ERASE) \
>                            $(DISABLE_LATENT_ENTROPY_PLUGIN) \
>                            -fno-stack-protector -D__NO_FORTIFY \
>                            -fno-jump-tables \
> --
> 2.53.1-Meta
>


-- 
Thanks,
~Nick Desaulniers

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
  2026-11-19 22:28   ` Nick Desaulniers
@ 2026-11-20 10:45   ` Peter Zijlstra
  2026-11-20 12:26     ` Peter Zijlstra
  2026-11-20 14:02     ` David Laight
  2026-11-20 18:55   ` David Laight
  2 siblings, 2 replies; 17+ messages in thread
From: Peter Zijlstra @ 2026-11-20 10:45 UTC (permalink / raw)
  To: Usama Arif
  Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb,
	hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening,
	llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty,
	atish.patra, dgreid

On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
> +#ifdef CONFIG_X86_64
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +					       unsigned long erase_high,
> +					       unsigned long poison)
> +{
> +	unsigned long count;
> +
> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> +	if (erase_high <= erase_low + 512) {
> +		while (erase_low < erase_high) {
> +			*(unsigned long *)erase_low = poison;
> +			erase_low += sizeof(unsigned long);
> +		}
> +		return;
> +	}
> +
> +	count = (erase_high - erase_low) / sizeof(unsigned long);
> +	asm volatile("rep stosq"
> +		     : "+D" (erase_low), "+c" (count)
> +		     : "a" (poison)
> +		     : "memory");
> +}
> +#endif

Gemini seems to suggest 16-32 words would be sufficient to amortize the
REP STOS setup. It was also adamant I try and unroll that earlier loop.

How about something like so, does that make sense? Also, I suppose we
should make memset64() __always_inline if we use it here. I'm not at all
sold on the unroll here, the asm is pretty horrid for what it does.

---
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -28,6 +28,7 @@ struct vm86;
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
 #include <asm/shstk.h>
+#include <asm/string.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -580,6 +581,33 @@ static __always_inline bool on_thread_st
 			       current_stack_pointer) < THREAD_SIZE;
 }
 
+#ifdef CONFIG_X86_64
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+					       unsigned long erase_high,
+					       unsigned long poison)
+{
+	unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
+
+	/* Amortize the startup cost of REP STOSQ. */
+	if (count < 32) {
+		unsigned long *ptr = (unsigned long *)erase_low;
+
+		while (count) {
+			switch ((count-1) % 4) {
+			case 3: ptr[--count] = poison; fallthrough;
+			case 2: ptr[--count] = poison; fallthrough;
+			case 1: ptr[--count] = poison; fallthrough;
+			case 0: ptr[--count] = poison; break;
+			}
+		}
+		return;
+	}
+
+	memset64((void *)erase_low, poison, count);
+}
+#endif
+
 #ifdef CONFIG_PARAVIRT_XXL
 #include <asm/paravirt.h>
 #else

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-19 22:28   ` Nick Desaulniers
@ 2026-11-20 11:38     ` David Laight
  2026-11-20 12:42       ` David Laight
  0 siblings, 1 reply; 17+ messages in thread
From: David Laight @ 2026-11-20 11:38 UTC (permalink / raw)
  To: Nick Desaulniers
  Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa,
	gustavoars, ardb, hca, nathan, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Thu, 19 Nov 2026 14:28:04 -0800
Nick Desaulniers <ndesaulniers@google.com> wrote:

> On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:
> >
> > With KSTACK_ERASE, every return to user space poisons the part of the
> > task stack that was used since the previous return. The generic
> > __stackleak_poison() does that one word per loop iteration, so a syscall
> > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> > running many small VMs, stackleak_erase() accounts for about 1.5% of
> > non-idle host kernel cycles.
> >
> > Override __stackleak_poison() on x86-64, as s390 already does, through
> > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> > specific stackleak poison function"). Fill spans larger than 512 bytes
> > with REP STOSQ and keep the word loop for shorter ones, where the fixed
> > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> > slower.
> >
> > The fill is inline asm, so the noinstr function still makes no calls.
> > memset64() would be the same instruction, but it is only static inline,
> > and with KMSAN it is an out-of-line function.
> >
> > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> >
> > Signed-off-by: Usama Arif <usama.arif@linux.dev>
> > ---
> >  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> >  1 file changed, 25 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> > index 77488ff5eb8c7c..049231d484ad0e 100644
> > --- a/arch/x86/include/asm/processor.h
> > +++ b/arch/x86/include/asm/processor.h
> > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> >                                current_stack_pointer) < THREAD_SIZE;
> >  }
> >
> > +#ifdef CONFIG_X86_64
> > +#define __stackleak_poison __stackleak_poison
> > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > +                                              unsigned long erase_high,
> > +                                              unsigned long poison)
> > +{
> > +       unsigned long count;
> > +
> > +       /* Short spans do not amortize the startup cost of REP STOSQ. */
> > +       if (erase_high <= erase_low + 512) {
> > +               while (erase_low < erase_high) {
> > +                       *(unsigned long *)erase_low = poison;
> > +                       erase_low += sizeof(unsigned long);
> > +               }
> > +               return;

gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
It does manage to unroll the loop once - which is good because the intel
cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
setup code is horrid.
But I also think some of the more recent cpu can execute more than
one write per clock.
If you unroll the loop 4 times it may run a lot faster.
Perhaps:
	long count = (erase_high - erase_low) / sizeof(unsigned long);
	if (count <= 64) {
		while ((count -= 4) >= 0) {
			*(unsigned long *)erase_low = poison;
			*(unsigned long *)(erase_low + 8) = poison;
			*(unsigned long *)(erase_low + 16) = poison;
			*(unsigned long *)(erase_low + 24) = poison;
			erase_low += 32;
		}
		if (count & 2) {
			*(unsigned long *)erase_low = poison;
			*(unsigned long *)(erase_low + 8) = poison;
			erase_low += 16;
		}
		if (count & 1)
			*(unsigned long *)erase_low = poison;
	} else {
		...
	}

Except that gcc makes a mess of it again.
I don't think the 'erase_low += 32' is going to execute at the
same time as the last (two probably) writes so costs a clock.
So it may need to be asm.

> > +       }
> > +
> > +       count = (erase_high - erase_low) / sizeof(unsigned long);
> > +       asm volatile("rep stosq"
> > +                    : "+D" (erase_low), "+c" (count)
> > +                    : "a" (poison)
> > +                    : "memory");  
> 
> Surely this isn't the only place using `rep stosq` in the tree, right?
> It's not, see memset64 in arch/x86/include/asm/string_64.h.

Which seems to have very few users.
memset_p() and memset_l() have almost no users.

David

> 
> > +}
> > +#endif
> > +
> >  #ifdef CONFIG_PARAVIRT_XXL
> >  #include <asm/paravirt.h>
> >  #else
> > --
> > 2.53.1-Meta
> >  
> 
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 10:45   ` Peter Zijlstra
@ 2026-11-20 12:26     ` Peter Zijlstra
  2026-11-20 18:43       ` Usama Arif
  2026-11-20 14:02     ` David Laight
  1 sibling, 1 reply; 17+ messages in thread
From: Peter Zijlstra @ 2026-11-20 12:26 UTC (permalink / raw)
  To: Usama Arif
  Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb,
	hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening,
	llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty,
	atish.patra, dgreid

On Fri, Nov 20, 2026 at 11:45:34AM +0100, Peter Zijlstra wrote:
> On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
> > +#ifdef CONFIG_X86_64
> > +#define __stackleak_poison __stackleak_poison
> > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > +					       unsigned long erase_high,
> > +					       unsigned long poison)
> > +{
> > +	unsigned long count;
> > +
> > +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> > +	if (erase_high <= erase_low + 512) {
> > +		while (erase_low < erase_high) {
> > +			*(unsigned long *)erase_low = poison;
> > +			erase_low += sizeof(unsigned long);
> > +		}
> > +		return;
> > +	}
> > +
> > +	count = (erase_high - erase_low) / sizeof(unsigned long);
> > +	asm volatile("rep stosq"
> > +		     : "+D" (erase_low), "+c" (count)
> > +		     : "a" (poison)
> > +		     : "memory");
> > +}
> > +#endif
> 
> Gemini seems to suggest 16-32 words would be sufficient to amortize the
> REP STOS setup. It was also adamant I try and unroll that earlier loop.
> 
> How about something like so, does that make sense? Also, I suppose we
> should make memset64() __always_inline if we use it here. I'm not at all
> sold on the unroll here, the asm is pretty horrid for what it does.

This one is much improved:

--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -28,6 +28,7 @@ struct vm86;
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
 #include <asm/shstk.h>
+#include <asm/string.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -580,6 +581,35 @@ static __always_inline bool on_thread_st
 			       current_stack_pointer) < THREAD_SIZE;
 }
 
+#ifdef CONFIG_X86_64
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+					       unsigned long erase_high,
+					       unsigned long poison)
+{
+	unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
+
+	/* Amortize the startup cost of REP STOSQ. */
+	if (count < 32) {
+		unsigned long *ptr = (unsigned long *)erase_low;
+
+		while (count) {
+			unsigned long mod = (count-1) % 4;
+			count -= mod+1;
+			switch (mod) {
+			case 3: ptr[count+3] = poison; fallthrough;
+			case 2: ptr[count+2] = poison; fallthrough;
+			case 1: ptr[count+1] = poison; fallthrough;
+			case 0: ptr[count+0] = poison; break;
+			}
+		}
+		return;
+	}
+
+	memset64((void *)erase_low, poison, count);
+}
+#endif
+
 #ifdef CONFIG_PARAVIRT_XXL
 #include <asm/paravirt.h>
 #else

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 11:38     ` David Laight
@ 2026-11-20 12:42       ` David Laight
  2026-11-20 18:46         ` Usama Arif
  0 siblings, 1 reply; 17+ messages in thread
From: David Laight @ 2026-11-20 12:42 UTC (permalink / raw)
  To: Nick Desaulniers
  Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa,
	gustavoars, ardb, hca, nathan, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Fri, 20 Nov 2026 11:38:25 +0000
David Laight <david.laight.linux@gmail.com> wrote:

> On Thu, 19 Nov 2026 14:28:04 -0800
> Nick Desaulniers <ndesaulniers@google.com> wrote:
> 
> > On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:  
> > >
> > > With KSTACK_ERASE, every return to user space poisons the part of the
> > > task stack that was used since the previous return. The generic
> > > __stackleak_poison() does that one word per loop iteration, so a syscall
> > > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> > > running many small VMs, stackleak_erase() accounts for about 1.5% of
> > > non-idle host kernel cycles.
> > >
> > > Override __stackleak_poison() on x86-64, as s390 already does, through
> > > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> > > specific stackleak poison function"). Fill spans larger than 512 bytes
> > > with REP STOSQ and keep the word loop for shorter ones, where the fixed
> > > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> > > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> > > slower.
> > >
> > > The fill is inline asm, so the noinstr function still makes no calls.
> > > memset64() would be the same instruction, but it is only static inline,
> > > and with KMSAN it is an out-of-line function.
> > >
> > > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> > > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> > > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> > > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> > >
> > > Signed-off-by: Usama Arif <usama.arif@linux.dev>
> > > ---
> > >  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> > >  1 file changed, 25 insertions(+)
> > >
> > > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> > > index 77488ff5eb8c7c..049231d484ad0e 100644
> > > --- a/arch/x86/include/asm/processor.h
> > > +++ b/arch/x86/include/asm/processor.h
> > > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> > >                                current_stack_pointer) < THREAD_SIZE;
> > >  }
> > >
> > > +#ifdef CONFIG_X86_64
> > > +#define __stackleak_poison __stackleak_poison
> > > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > > +                                              unsigned long erase_high,
> > > +                                              unsigned long poison)
> > > +{
> > > +       unsigned long count;
> > > +
> > > +       /* Short spans do not amortize the startup cost of REP STOSQ. */
> > > +       if (erase_high <= erase_low + 512) {
> > > +               while (erase_low < erase_high) {
> > > +                       *(unsigned long *)erase_low = poison;
> > > +                       erase_low += sizeof(unsigned long);
> > > +               }
> > > +               return;  
> 
> gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
> It does manage to unroll the loop once - which is good because the intel
> cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
> setup code is horrid.
> But I also think some of the more recent cpu can execute more than
> one write per clock.
> If you unroll the loop 4 times it may run a lot faster.
> Perhaps:
> 	long count = (erase_high - erase_low) / sizeof(unsigned long);
> 	if (count <= 64) {
> 		while ((count -= 4) >= 0) {
> 			*(unsigned long *)erase_low = poison;
> 			*(unsigned long *)(erase_low + 8) = poison;

The add (probably) needs to be in the middle...
 			erase_low += 32;
> 			*(unsigned long *)(erase_low - 32 + 16) = poison;
> 			*(unsigned long *)(erase_low - 32 + 24) = poison;
> 		}
> 		if (count & 2) {
> 			*(unsigned long *)erase_low = poison;
> 			*(unsigned long *)(erase_low + 8) = poison;
> 			erase_low += 16;
> 		}
> 		if (count & 1)
> 			*(unsigned long *)erase_low = poison;
> 	} else {
> 		...
> 	}
> 
> Except that gcc makes a mess of it again.
> I don't think the 'erase_low += 32' is going to execute at the
> same time as the last (two probably) writes so costs a clock.
> So it may need to be asm.

I threw in enough OPTIMIZER_HIDE_VAR() to stop the optimiser breaking
the code. https://godbolt.org/z/nx477T4M1
gcc then generates:
        movq    %rdx, (%rdi)
        movq    %rdx, 8(%rdi)
        addq    $32, %rdi
        movq    %rdx, -16(%rdi)
        movq    %rdx, -8(%rdi)
.L32:
        subq    $4, %rcx
        jns     .L22
for the code loop (clang is a bit worse - but should be ok).
I think that will be a 'two clock loop' on cpu that can do
two writes/clock; zen3 and ice lake).
For 'stack splatting' rounding up to a multiple of 32 bytes
will save all the 'remainder' code.

David

> 
> > > +       }
> > > +
> > > +       count = (erase_high - erase_low) / sizeof(unsigned long);
> > > +       asm volatile("rep stosq"
> > > +                    : "+D" (erase_low), "+c" (count)
> > > +                    : "a" (poison)
> > > +                    : "memory");    
> > 
> > Surely this isn't the only place using `rep stosq` in the tree, right?
> > It's not, see memset64 in arch/x86/include/asm/string_64.h.  
> 
> Which seems to have very few users.
> memset_p() and memset_l() have almost no users.
> 
> David
> 
> >   
> > > +}
> > > +#endif
> > > +
> > >  #ifdef CONFIG_PARAVIRT_XXL
> > >  #include <asm/paravirt.h>
> > >  #else
> > > --
> > > 2.53.1-Meta
> > >    
> > 
> >   
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 10:45   ` Peter Zijlstra
  2026-11-20 12:26     ` Peter Zijlstra
@ 2026-11-20 14:02     ` David Laight
  2026-11-20 15:35       ` Peter Zijlstra
  1 sibling, 1 reply; 17+ messages in thread
From: David Laight @ 2026-11-20 14:02 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa,
	gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Fri, 20 Nov 2026 11:45:34 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
> > +#ifdef CONFIG_X86_64
> > +#define __stackleak_poison __stackleak_poison
> > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > +					       unsigned long erase_high,
> > +					       unsigned long poison)
> > +{
> > +	unsigned long count;
> > +
> > +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> > +	if (erase_high <= erase_low + 512) {
> > +		while (erase_low < erase_high) {
> > +			*(unsigned long *)erase_low = poison;
> > +			erase_low += sizeof(unsigned long);
> > +		}
> > +		return;
> > +	}
> > +
> > +	count = (erase_high - erase_low) / sizeof(unsigned long);
> > +	asm volatile("rep stosq"
> > +		     : "+D" (erase_low), "+c" (count)
> > +		     : "a" (poison)
> > +		     : "memory");
> > +}
> > +#endif  
> 
> Gemini seems to suggest 16-32 words would be sufficient to amortize the
> REP STOS setup.

That really does sound like wishful thinking.
Especially if you go back a few generations on the cpu architecture.

> It was also adamant I try and unroll that earlier loop.
> 
> How about something like so, does that make sense? Also, I suppose we
> should make memset64() __always_inline if we use it here. I'm not at all
> sold on the unroll here, the asm is pretty horrid for what it does.
> 
> ---
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -28,6 +28,7 @@ struct vm86;
>  #include <asm/vmxfeatures.h>
>  #include <asm/vdso/processor.h>
>  #include <asm/shstk.h>
> +#include <asm/string.h>
>  
>  #include <linux/personality.h>
>  #include <linux/cache.h>
> @@ -580,6 +581,33 @@ static __always_inline bool on_thread_st
>  			       current_stack_pointer) < THREAD_SIZE;
>  }
>  
> +#ifdef CONFIG_X86_64
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +					       unsigned long erase_high,
> +					       unsigned long poison)
> +{
> +	unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
> +
> +	/* Amortize the startup cost of REP STOSQ. */
> +	if (count < 32) {
> +		unsigned long *ptr = (unsigned long *)erase_low;
> +
> +		while (count) {
> +			switch ((count-1) % 4) {
> +			case 3: ptr[--count] = poison; fallthrough;
> +			case 2: ptr[--count] = poison; fallthrough;
> +			case 1: ptr[--count] = poison; fallthrough;
> +			case 0: ptr[--count] = poison; break;
> +			}
> +		}

That looks horrid to me.
You want as few instructions (especially conditionals) inside the
loop as possible.
The Intel cpu will execute a two clock loop and the later ones will
do two writes per clock
(zen5 will execute a one clock loop and do two writes/clock).
So you want a loop with just 4 writes and the loop control code.
The 'pointer add' just needs to execute at the same time as some
of the stores - to avoid lengthening the register dependency chain.

For this code, can the length be a multiple of 32?
Also on some cpu the 'rep stos' might be twice as fast (per byte) if
the destination in 64 byte aligned - true for 'rep movsb'.

David


> +		return;
> +	}
> +
> +	memset64((void *)erase_low, poison, count);
> +}
> +#endif
> +
>  #ifdef CONFIG_PARAVIRT_XXL
>  #include <asm/paravirt.h>
>  #else
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 14:02     ` David Laight
@ 2026-11-20 15:35       ` Peter Zijlstra
  2026-11-20 16:18         ` David Laight
  0 siblings, 1 reply; 17+ messages in thread
From: Peter Zijlstra @ 2026-11-20 15:35 UTC (permalink / raw)
  To: David Laight
  Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa,
	gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Fri, Nov 20, 2026 at 02:02:00PM +0000, David Laight wrote:
> On Fri, 20 Nov 2026 11:45:34 +0100
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
> > > +#ifdef CONFIG_X86_64
> > > +#define __stackleak_poison __stackleak_poison
> > > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > > +					       unsigned long erase_high,
> > > +					       unsigned long poison)
> > > +{
> > > +	unsigned long count;
> > > +
> > > +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> > > +	if (erase_high <= erase_low + 512) {
> > > +		while (erase_low < erase_high) {
> > > +			*(unsigned long *)erase_low = poison;
> > > +			erase_low += sizeof(unsigned long);
> > > +		}
> > > +		return;
> > > +	}
> > > +
> > > +	count = (erase_high - erase_low) / sizeof(unsigned long);
> > > +	asm volatile("rep stosq"
> > > +		     : "+D" (erase_low), "+c" (count)
> > > +		     : "a" (poison)
> > > +		     : "memory");
> > > +}
> > > +#endif  
> > 
> > Gemini seems to suggest 16-32 words would be sufficient to amortize the
> > REP STOS setup.
> 
> That really does sound like wishful thinking.
> Especially if you go back a few generations on the cpu architecture.

How old? I mean, Interlagos old, sure, but nobody should care about
performance there I think.

AFAICT Nehalem and later should be good with the 16-32 words. As would
all the various Zen chips.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 15:35       ` Peter Zijlstra
@ 2026-11-20 16:18         ` David Laight
  0 siblings, 0 replies; 17+ messages in thread
From: David Laight @ 2026-11-20 16:18 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa,
	gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Fri, 20 Nov 2026 16:35:39 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> On Fri, Nov 20, 2026 at 02:02:00PM +0000, David Laight wrote:
> > On Fri, 20 Nov 2026 11:45:34 +0100
> > Peter Zijlstra <peterz@infradead.org> wrote:
> >   
> > > On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:  
> > > > +#ifdef CONFIG_X86_64
> > > > +#define __stackleak_poison __stackleak_poison
> > > > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > > > +					       unsigned long erase_high,
> > > > +					       unsigned long poison)
> > > > +{
> > > > +	unsigned long count;
> > > > +
> > > > +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> > > > +	if (erase_high <= erase_low + 512) {
> > > > +		while (erase_low < erase_high) {
> > > > +			*(unsigned long *)erase_low = poison;
> > > > +			erase_low += sizeof(unsigned long);
> > > > +		}
> > > > +		return;
> > > > +	}
> > > > +
> > > > +	count = (erase_high - erase_low) / sizeof(unsigned long);
> > > > +	asm volatile("rep stosq"
> > > > +		     : "+D" (erase_low), "+c" (count)
> > > > +		     : "a" (poison)
> > > > +		     : "memory");
> > > > +}
> > > > +#endif    
> > > 
> > > Gemini seems to suggest 16-32 words would be sufficient to amortize the
> > > REP STOS setup.  
> > 
> > That really does sound like wishful thinking.
> > Especially if you go back a few generations on the cpu architecture.  
> 
> How old? I mean, Interlagos old, sure, but nobody should care about
> performance there I think.
> 
> AFAICT Nehalem and later should be good with the 16-32 words. As would
> all the various Zen chips.

Have you got an accurate source for the setup costs for 'rep stos/movs'
for the various architectures?
I didn't find one last time I looked.
Last time I measured rep movs I was using mfence to synchronise with rdpmc
and that takes even longer - so you only get the per-byte costs.
(You can use data-dependencies which have minimal overhead unless you
go back to to some pre-zen amd cpu.)

You get 16 bytes/clock until Haswell which will do 32 if rdi is aligned.
Icelake might increase that to 64 bytes/clock.
Agner has zen2 onwards at 32 bytes/clock.

Agner also has zen3 and Icelake being able to do 2 writes per clock.
If that is achievable for the writes being done (the loop isn't a
problem) then it also takes a while for the 'rep stos' to catch up.
(Then you have the cost of the I-cache misses ...)

David


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 12:26     ` Peter Zijlstra
@ 2026-11-20 18:43       ` Usama Arif
  2026-11-20 22:22         ` David Laight
  0 siblings, 1 reply; 17+ messages in thread
From: Usama Arif @ 2026-11-20 18:43 UTC (permalink / raw)
  To: Peter Zijlstra, ndesaulniers, David Laight
  Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb,
	hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening,
	llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty,
	atish.patra, dgreid



On 20/11/2026 04:26, Peter Zijlstra wrote:
> On Fri, Nov 20, 2026 at 11:45:34AM +0100, Peter Zijlstra wrote:
>> On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
>>> +#ifdef CONFIG_X86_64
>>> +#define __stackleak_poison __stackleak_poison
>>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>>> +					       unsigned long erase_high,
>>> +					       unsigned long poison)
>>> +{
>>> +	unsigned long count;
>>> +
>>> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
>>> +	if (erase_high <= erase_low + 512) {
>>> +		while (erase_low < erase_high) {
>>> +			*(unsigned long *)erase_low = poison;
>>> +			erase_low += sizeof(unsigned long);
>>> +		}
>>> +		return;
>>> +	}
>>> +
>>> +	count = (erase_high - erase_low) / sizeof(unsigned long);
>>> +	asm volatile("rep stosq"
>>> +		     : "+D" (erase_low), "+c" (count)
>>> +		     : "a" (poison)
>>> +		     : "memory");
>>> +}
>>> +#endif
>>
>> Gemini seems to suggest 16-32 words would be sufficient to amortize the
>> REP STOS setup. It was also adamant I try and unroll that earlier loop.
>>
>> How about something like so, does that make sense? Also, I suppose we
>> should make memset64() __always_inline if we use it here. I'm not at all
>> sold on the unroll here, the asm is pretty horrid for what it does.
> 
> This one is much improved:
> 
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -28,6 +28,7 @@ struct vm86;
>  #include <asm/vmxfeatures.h>
>  #include <asm/vdso/processor.h>
>  #include <asm/shstk.h>
> +#include <asm/string.h>
>  
>  #include <linux/personality.h>
>  #include <linux/cache.h>
> @@ -580,6 +581,35 @@ static __always_inline bool on_thread_st
>  			       current_stack_pointer) < THREAD_SIZE;
>  }
>  
> +#ifdef CONFIG_X86_64
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +					       unsigned long erase_high,
> +					       unsigned long poison)
> +{
> +	unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
> +
> +	/* Amortize the startup cost of REP STOSQ. */
> +	if (count < 32) {
> +		unsigned long *ptr = (unsigned long *)erase_low;
> +
> +		while (count) {
> +			unsigned long mod = (count-1) % 4;
> +			count -= mod+1;
> +			switch (mod) {
> +			case 3: ptr[count+3] = poison; fallthrough;
> +			case 2: ptr[count+2] = poison; fallthrough;
> +			case 1: ptr[count+1] = poison; fallthrough;
> +			case 0: ptr[count+0] = poison; break;
> +			}
> +		}
> +		return;
> +	}
> +

Hello Nick, Peter and David,

Thanks for the reviews! I should have used memset64 from the get go :)

I have tested 2 possible revisions now as well as this one from Peter,
have added both of them at the end of the reply.

- v2.1 where you loop upto 512B and then memset
- v2.2: 4 stores per loop up to 512 B, then 64-byte aligned memset64()

I timed stackleak_erase() in a Zen 5c guest over a futex/epoll/procfs/unix
syscall:

  No patch                             68.3 ns per erase
  v1 (loop up to 512 B, then REP)           46.6 ns
  v2.1 (loop up to 512 B, memset64())       46.6 ns
  this version                              46.4 ns
  v2.2 (4 stores/iteration, aligned REP)    43.7 ns


The switch doesn't seem to help.

Either way I kept 512 bytes: REP STOSQ only wins from ~430 bytes on Zen
5c against the plain loop (~850 against the unrolled one), and from ~320
on Cooper Lake.


My personal preference is v2.1, as its much simpler code and the difference
in performance between v2.1 and v2.2 isn't actually that much. But if there is
a stronger preference for v2.2, I can send that as it is faster.


v2.1:

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c7c..3d205ae8566114 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -28,6 +28,7 @@ struct vm86;
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
 #include <asm/shstk.h>
+#include <asm/string.h>

 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -580,6 +581,26 @@ static __always_inline bool on_thread_stack(void)
                               current_stack_pointer) < THREAD_SIZE;
 }

+#if defined(CONFIG_X86_64) && !defined(CONFIG_KMSAN)
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+                                              unsigned long erase_high,
+                                              unsigned long poison)
+{
+       /* Short spans do not amortize the startup cost of REP STOSQ. */
+       if (erase_high <= erase_low + 512) {
+               while (erase_low < erase_high) {
+                       *(unsigned long *)erase_low = poison;
+                       erase_low += sizeof(unsigned long);
+               }
+               return;
+       }
+
+       memset64((u64 *)erase_low, poison,
+                (erase_high - erase_low) / sizeof(u64));
+}
+#endif
+
 #ifdef CONFIG_PARAVIRT_XXL
 #include <asm/paravirt.h>
 #else
diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
index 2dae70c2b01465..73cc91254b1381 100644
--- a/arch/x86/include/asm/string_64.h
+++ b/arch/x86/include/asm/string_64.h
@@ -54,7 +54,7 @@ static inline void *memset32(uint32_t *s, uint32_t v, size_t n)
 }

 #define __HAVE_ARCH_MEMSET64
-static inline void *memset64(uint64_t *s, uint64_t v, size_t n)
+static __always_inline void *memset64(uint64_t *s, uint64_t v, size_t n)
 {
        const auto s0 = s;
        asm volatile (





v2.2:

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c7c..5e76e1140908d7 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -28,7 +28,9 @@ struct vm86;
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
 #include <asm/shstk.h>
+#include <asm/string.h>

+#include <linux/align.h>
 #include <linux/personality.h>
 #include <linux/cache.h>
 #include <linux/threads.h>
@@ -580,6 +582,48 @@ static __always_inline bool on_thread_stack(void)
                               current_stack_pointer) < THREAD_SIZE;
 }

+#if defined(CONFIG_X86_64) && !defined(CONFIG_KMSAN)
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+                                              unsigned long erase_high,
+                                              unsigned long poison)
+{
+       unsigned long *p = (unsigned long *)erase_low;
+       unsigned long *end = (unsigned long *)erase_high;
+       unsigned long i;
+
+       /* Fewer than four words, or nothing to do. */
+       if (erase_high < erase_low + 4 * sizeof(*p)) {
+               while (p < end)
+                       *p++ = poison;
+               return;
+       }
+
+       /* Four stores per iteration; the last four may overlap. */
+       if (erase_high - erase_low <= 512) {
+               end -= 4;
+               while (p < end) {
+                       p[0] = poison;
+                       p[1] = poison;
+                       p[2] = poison;
+                       p[3] = poison;
+                       p += 4;
+               }
+               end[0] = poison;
+               end[1] = poison;
+               end[2] = poison;
+               end[3] = poison;
+               return;
+       }
+
+       /* REP STOSQ is faster from a 64-byte aligned destination. */
+       for (i = 0; i < 8; i++)
+               p[i] = poison;
+       p = (unsigned long *)ALIGN(erase_low + 1, 64);
+       memset64((u64 *)p, poison, end - p);
+}
+#endif
+
 #ifdef CONFIG_PARAVIRT_XXL
 #include <asm/paravirt.h>
 #else
diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
index 2dae70c2b01465..73cc91254b1381 100644
--- a/arch/x86/include/asm/string_64.h
+++ b/arch/x86/include/asm/string_64.h
@@ -54,7 +54,7 @@ static inline void *memset32(uint32_t *s, uint32_t v, size_t n)
 }

 #define __HAVE_ARCH_MEMSET64
-static inline void *memset64(uint64_t *s, uint64_t v, size_t n)
+static __always_inline void *memset64(uint64_t *s, uint64_t v, size_t n)
 {
        const auto s0 = s;
        asm volatile (

> +	memset64((void *)erase_low, poison, count);
> +}
> +#endif
> +
>  #ifdef CONFIG_PARAVIRT_XXL
>  #include <asm/paravirt.h>
>  #else


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 12:42       ` David Laight
@ 2026-11-20 18:46         ` Usama Arif
  2026-11-20 21:44           ` David Laight
  0 siblings, 1 reply; 17+ messages in thread
From: Usama Arif @ 2026-11-20 18:46 UTC (permalink / raw)
  To: David Laight, Nick Desaulniers
  Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb,
	hca, nathan, morbo, justinstitt, linux-hardening, llvm,
	linux-kernel, riel, shakeel.butt, kernel-team, rkeilty,
	atish.patra, dgreid



On 20/11/2026 04:42, David Laight wrote:
> On Fri, 20 Nov 2026 11:38:25 +0000
> David Laight <david.laight.linux@gmail.com> wrote:
> 
>> On Thu, 19 Nov 2026 14:28:04 -0800
>> Nick Desaulniers <ndesaulniers@google.com> wrote:
>>
>>> On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:  
>>>>
>>>> With KSTACK_ERASE, every return to user space poisons the part of the
>>>> task stack that was used since the previous return. The generic
>>>> __stackleak_poison() does that one word per loop iteration, so a syscall
>>>> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
>>>> running many small VMs, stackleak_erase() accounts for about 1.5% of
>>>> non-idle host kernel cycles.
>>>>
>>>> Override __stackleak_poison() on x86-64, as s390 already does, through
>>>> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
>>>> specific stackleak poison function"). Fill spans larger than 512 bytes
>>>> with REP STOSQ and keep the word loop for shorter ones, where the fixed
>>>> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
>>>> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
>>>> slower.
>>>>
>>>> The fill is inline asm, so the noinstr function still makes no calls.
>>>> memset64() would be the same instruction, but it is only static inline,
>>>> and with KMSAN it is an out-of-line function.
>>>>
>>>> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
>>>> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
>>>> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
>>>> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
>>>>
>>>> Signed-off-by: Usama Arif <usama.arif@linux.dev>
>>>> ---
>>>>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
>>>>  1 file changed, 25 insertions(+)
>>>>
>>>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
>>>> index 77488ff5eb8c7c..049231d484ad0e 100644
>>>> --- a/arch/x86/include/asm/processor.h
>>>> +++ b/arch/x86/include/asm/processor.h
>>>> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
>>>>                                current_stack_pointer) < THREAD_SIZE;
>>>>  }
>>>>
>>>> +#ifdef CONFIG_X86_64
>>>> +#define __stackleak_poison __stackleak_poison
>>>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>>>> +                                              unsigned long erase_high,
>>>> +                                              unsigned long poison)
>>>> +{
>>>> +       unsigned long count;
>>>> +
>>>> +       /* Short spans do not amortize the startup cost of REP STOSQ. */
>>>> +       if (erase_high <= erase_low + 512) {
>>>> +               while (erase_low < erase_high) {
>>>> +                       *(unsigned long *)erase_low = poison;
>>>> +                       erase_low += sizeof(unsigned long);
>>>> +               }
>>>> +               return;  
>>
>> gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
>> It does manage to unroll the loop once - which is good because the intel
>> cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
>> setup code is horrid.
>> But I also think some of the more recent cpu can execute more than
>> one write per clock.
>> If you unroll the loop 4 times it may run a lot faster.
>> Perhaps:
>> 	long count = (erase_high - erase_low) / sizeof(unsigned long);
>> 	if (count <= 64) {
>> 		while ((count -= 4) >= 0) {
>> 			*(unsigned long *)erase_low = poison;
>> 			*(unsigned long *)(erase_low + 8) = poison;
> 
> The add (probably) needs to be in the middle...
>  			erase_low += 32;
>> 			*(unsigned long *)(erase_low - 32 + 16) = poison;
>> 			*(unsigned long *)(erase_low - 32 + 24) = poison;
>> 		}
>> 		if (count & 2) {
>> 			*(unsigned long *)erase_low = poison;
>> 			*(unsigned long *)(erase_low + 8) = poison;
>> 			erase_low += 16;
>> 		}
>> 		if (count & 1)
>> 			*(unsigned long *)erase_low = poison;
>> 	} else {
>> 		...
>> 	}
>>
>> Except that gcc makes a mess of it again.
>> I don't think the 'erase_low += 32' is going to execute at the
>> same time as the last (two probably) writes so costs a clock.
>> So it may need to be asm.
> 
> I threw in enough OPTIMIZER_HIDE_VAR() to stop the optimiser breaking
> the code. https://godbolt.org/z/nx477T4M1
> gcc then generates:
>         movq    %rdx, (%rdi)
>         movq    %rdx, 8(%rdi)
>         addq    $32, %rdi
>         movq    %rdx, -16(%rdi)
>         movq    %rdx, -8(%rdi)
> .L32:
>         subq    $4, %rcx
>         jns     .L22
> for the code loop (clang is a bit worse - but should be ok).
> I think that will be a 'two clock loop' on cpu that can do
> two writes/clock; zen3 and ice lake).
> For 'stack splatting' rounding up to a multiple of 32 bytes
> will save all the 'remainder' code.


The range can't be rounded: erase_low is wherever the poison search
stopped, and erase_high is the stack pointer or the bottom of pt_regs
(168 bytes, so only 8-byte aligned). Rounding up writes live stack,
rounding down can write STACK_END_MAGIC when the stack was used to the
end.


> 
> David
> 
>>
>>>> +       }
>>>> +
>>>> +       count = (erase_high - erase_low) / sizeof(unsigned long);
>>>> +       asm volatile("rep stosq"
>>>> +                    : "+D" (erase_low), "+c" (count)
>>>> +                    : "a" (poison)
>>>> +                    : "memory");    
>>>
>>> Surely this isn't the only place using `rep stosq` in the tree, right?
>>> It's not, see memset64 in arch/x86/include/asm/string_64.h.  
>>
>> Which seems to have very few users.
>> memset_p() and memset_l() have almost no users.
>>
>> David
>>
>>>   
>>>> +}
>>>> +#endif
>>>> +
>>>>  #ifdef CONFIG_PARAVIRT_XXL
>>>>  #include <asm/paravirt.h>
>>>>  #else
>>>> --
>>>> 2.53.1-Meta
>>>>    
>>>
>>>   
>>
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
  2026-11-19 22:28   ` Nick Desaulniers
  2026-11-20 10:45   ` Peter Zijlstra
@ 2026-11-20 18:55   ` David Laight
  2 siblings, 0 replies; 17+ messages in thread
From: David Laight @ 2026-11-20 18:55 UTC (permalink / raw)
  To: Usama Arif
  Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb,
	hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening,
	llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty,
	atish.patra, dgreid

On Thu, 19 Nov 2026 13:31:08 -0800
Usama Arif <usama.arif@linux.dev> wrote:

> With KSTACK_ERASE, every return to user space poisons the part of the
> task stack that was used since the previous return. The generic
> __stackleak_poison() does that one word per loop iteration, so a syscall
> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> running many small VMs, stackleak_erase() accounts for about 1.5% of
> non-idle host kernel cycles.
> 
> Override __stackleak_poison() on x86-64, as s390 already does, through
> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> specific stackleak poison function"). Fill spans larger than 512 bytes
> with REP STOSQ and keep the word loop for shorter ones, where the fixed
> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> slower.

As a matter of interest (and separately from the exact code) how did you
measure the performance to get the 300-400 byte gain point?

If the count is variable - likely in real life, you'll get a branch
misprediction penalty (about 20 clocks on zen5) every time you change
between the two choices.
The copy loop will have another mispredict penalty for the loop itself.

So you are likely to get an overall improvement if you use 'rep stos'
for all lengths - even the ones for with it is slightly slower.
Unless, that is, you know at compile time that most of the length
will be short.

David

> 
> The fill is inline asm, so the noinstr function still makes no calls.
> memset64() would be the same instruction, but it is only static inline,
> and with KMSAN it is an out-of-line function.
> 
> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> 
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index 77488ff5eb8c7c..049231d484ad0e 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
>  			       current_stack_pointer) < THREAD_SIZE;
>  }
>  
> +#ifdef CONFIG_X86_64
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +					       unsigned long erase_high,
> +					       unsigned long poison)
> +{
> +	unsigned long count;
> +
> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> +	if (erase_high <= erase_low + 512) {
> +		while (erase_low < erase_high) {
> +			*(unsigned long *)erase_low = poison;
> +			erase_low += sizeof(unsigned long);
> +		}
> +		return;
> +	}
> +
> +	count = (erase_high - erase_low) / sizeof(unsigned long);
> +	asm volatile("rep stosq"
> +		     : "+D" (erase_low), "+c" (count)
> +		     : "a" (poison)
> +		     : "memory");
> +}
> +#endif
> +
>  #ifdef CONFIG_PARAVIRT_XXL
>  #include <asm/paravirt.h>
>  #else


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 18:46         ` Usama Arif
@ 2026-11-20 21:44           ` David Laight
  0 siblings, 0 replies; 17+ messages in thread
From: David Laight @ 2026-11-20 21:44 UTC (permalink / raw)
  To: Usama Arif
  Cc: Nick Desaulniers, tglx, mingo, bp, dave.hansen, x86, kees, hpa,
	gustavoars, ardb, hca, nathan, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Fri, 20 Nov 2026 18:46:01 +0000
Usama Arif <usama.arif@linux.dev> wrote:

> On 20/11/2026 04:42, David Laight wrote:
> > On Fri, 20 Nov 2026 11:38:25 +0000
> > David Laight <david.laight.linux@gmail.com> wrote:
> >   
> >> On Thu, 19 Nov 2026 14:28:04 -0800
> >> Nick Desaulniers <ndesaulniers@google.com> wrote:
> >>  
> >>> On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:    
> >>>>
> >>>> With KSTACK_ERASE, every return to user space poisons the part of the
> >>>> task stack that was used since the previous return. The generic
> >>>> __stackleak_poison() does that one word per loop iteration, so a syscall
> >>>> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> >>>> running many small VMs, stackleak_erase() accounts for about 1.5% of
> >>>> non-idle host kernel cycles.
> >>>>
> >>>> Override __stackleak_poison() on x86-64, as s390 already does, through
> >>>> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> >>>> specific stackleak poison function"). Fill spans larger than 512 bytes
> >>>> with REP STOSQ and keep the word loop for shorter ones, where the fixed
> >>>> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> >>>> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> >>>> slower.
> >>>>
> >>>> The fill is inline asm, so the noinstr function still makes no calls.
> >>>> memset64() would be the same instruction, but it is only static inline,
> >>>> and with KMSAN it is an out-of-line function.
> >>>>
> >>>> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> >>>> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> >>>> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> >>>> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> >>>>
> >>>> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> >>>> ---
> >>>>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> >>>>  1 file changed, 25 insertions(+)
> >>>>
> >>>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> >>>> index 77488ff5eb8c7c..049231d484ad0e 100644
> >>>> --- a/arch/x86/include/asm/processor.h
> >>>> +++ b/arch/x86/include/asm/processor.h
> >>>> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> >>>>                                current_stack_pointer) < THREAD_SIZE;
> >>>>  }
> >>>>
> >>>> +#ifdef CONFIG_X86_64
> >>>> +#define __stackleak_poison __stackleak_poison
> >>>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> >>>> +                                              unsigned long erase_high,
> >>>> +                                              unsigned long poison)
> >>>> +{
> >>>> +       unsigned long count;
> >>>> +
> >>>> +       /* Short spans do not amortize the startup cost of REP STOSQ. */
> >>>> +       if (erase_high <= erase_low + 512) {
> >>>> +               while (erase_low < erase_high) {
> >>>> +                       *(unsigned long *)erase_low = poison;
> >>>> +                       erase_low += sizeof(unsigned long);
> >>>> +               }
> >>>> +               return;    
> >>
> >> gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
> >> It does manage to unroll the loop once - which is good because the intel
> >> cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
> >> setup code is horrid.
> >> But I also think some of the more recent cpu can execute more than
> >> one write per clock.
> >> If you unroll the loop 4 times it may run a lot faster.
> >> Perhaps:
> >> 	long count = (erase_high - erase_low) / sizeof(unsigned long);
> >> 	if (count <= 64) {
> >> 		while ((count -= 4) >= 0) {
> >> 			*(unsigned long *)erase_low = poison;
> >> 			*(unsigned long *)(erase_low + 8) = poison;  
> > 
> > The add (probably) needs to be in the middle...
> >  			erase_low += 32;  
> >> 			*(unsigned long *)(erase_low - 32 + 16) = poison;
> >> 			*(unsigned long *)(erase_low - 32 + 24) = poison;
> >> 		}
> >> 		if (count & 2) {
> >> 			*(unsigned long *)erase_low = poison;
> >> 			*(unsigned long *)(erase_low + 8) = poison;
> >> 			erase_low += 16;
> >> 		}
> >> 		if (count & 1)
> >> 			*(unsigned long *)erase_low = poison;
> >> 	} else {
> >> 		...
> >> 	}
> >>
> >> Except that gcc makes a mess of it again.
> >> I don't think the 'erase_low += 32' is going to execute at the
> >> same time as the last (two probably) writes so costs a clock.
> >> So it may need to be asm.  
> > 
> > I threw in enough OPTIMIZER_HIDE_VAR() to stop the optimiser breaking
> > the code. https://godbolt.org/z/nx477T4M1
> > gcc then generates:
> >         movq    %rdx, (%rdi)
> >         movq    %rdx, 8(%rdi)
> >         addq    $32, %rdi
> >         movq    %rdx, -16(%rdi)
> >         movq    %rdx, -8(%rdi)
> > .L32:
> >         subq    $4, %rcx
> >         jns     .L22
> > for the code loop (clang is a bit worse - but should be ok).
> > I think that will be a 'two clock loop' on cpu that can do
> > two writes/clock; zen3 and ice lake).
> > For 'stack splatting' rounding up to a multiple of 32 bytes
> > will save all the 'remainder' code.  
> 
> 
> The range can't be rounded: erase_low is wherever the poison search
> stopped, and erase_high is the stack pointer or the bottom of pt_regs
> (168 bytes, so only 8-byte aligned). Rounding up writes live stack,
> rounding down can write STACK_END_MAGIC when the stack was used to the
> end.

If the stack down by STACK_END_MAGIC has been written you are almost in
'deep do-do land' so task_low_bound() could return an aligned address
and that would let you align down the low address.
(So leaving the very last few words alone.)
It also probably doesn't really matter about the top few words.
So the length could be rounded down.

Alternatively, do STACK_END_MAGIC and STACK_ERASE_POISON even need
to have different values?
(And aren't stacks now allocated with a guard page - so you get a
fault on overflow much earlier than any check for STACK_END_MAGIC.)

But does this code even work in a useful manner.
It seems to stop scanning after finding 128 bytes of KSTACK_ERASE_POISON.
Isn't that going to stop on a function with a large on-stack buffer
(especially since one is likely to be on the deepest part of the stack).
I'd have thought you'd jump down the stack in larger lumps than that
just looking for an unwritten word before even trying to decide if it
was the boundary.

David

> 
> 
> > 
> > David
> >   
> >>  
> >>>> +       }
> >>>> +
> >>>> +       count = (erase_high - erase_low) / sizeof(unsigned long);
> >>>> +       asm volatile("rep stosq"
> >>>> +                    : "+D" (erase_low), "+c" (count)
> >>>> +                    : "a" (poison)
> >>>> +                    : "memory");      
> >>>
> >>> Surely this isn't the only place using `rep stosq` in the tree, right?
> >>> It's not, see memset64 in arch/x86/include/asm/string_64.h.    
> >>
> >> Which seems to have very few users.
> >> memset_p() and memset_l() have almost no users.
> >>
> >> David
> >>  
> >>>     
> >>>> +}
> >>>> +#endif
> >>>> +
> >>>>  #ifdef CONFIG_PARAVIRT_XXL
> >>>>  #include <asm/paravirt.h>
> >>>>  #else
> >>>> --
> >>>> 2.53.1-Meta
> >>>>      
> >>>
> >>>     
> >>  
> >   
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
  2026-11-20 18:43       ` Usama Arif
@ 2026-11-20 22:22         ` David Laight
  0 siblings, 0 replies; 17+ messages in thread
From: David Laight @ 2026-11-20 22:22 UTC (permalink / raw)
  To: Usama Arif
  Cc: Peter Zijlstra, ndesaulniers, tglx, mingo, bp, dave.hansen, x86,
	kees, hpa, gustavoars, ardb, hca, nathan, morbo, justinstitt,
	linux-hardening, llvm, linux-kernel, riel, shakeel.butt,
	kernel-team, rkeilty, atish.patra, dgreid

On Fri, 20 Nov 2026 18:43:38 +0000
Usama Arif <usama.arif@linux.dev> wrote:

> On 20/11/2026 04:26, Peter Zijlstra wrote:
> > On Fri, Nov 20, 2026 at 11:45:34AM +0100, Peter Zijlstra wrote:  
> >> On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:  
> >>> +#ifdef CONFIG_X86_64
> >>> +#define __stackleak_poison __stackleak_poison
> >>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> >>> +					       unsigned long erase_high,
> >>> +					       unsigned long poison)
> >>> +{
> >>> +	unsigned long count;
> >>> +
> >>> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> >>> +	if (erase_high <= erase_low + 512) {
> >>> +		while (erase_low < erase_high) {
> >>> +			*(unsigned long *)erase_low = poison;
> >>> +			erase_low += sizeof(unsigned long);
> >>> +		}
> >>> +		return;
> >>> +	}
> >>> +
> >>> +	count = (erase_high - erase_low) / sizeof(unsigned long);
> >>> +	asm volatile("rep stosq"
> >>> +		     : "+D" (erase_low), "+c" (count)
> >>> +		     : "a" (poison)
> >>> +		     : "memory");
> >>> +}
> >>> +#endif  
> >>
> >> Gemini seems to suggest 16-32 words would be sufficient to amortize the
> >> REP STOS setup. It was also adamant I try and unroll that earlier loop.
> >>
> >> How about something like so, does that make sense? Also, I suppose we
> >> should make memset64() __always_inline if we use it here. I'm not at all
> >> sold on the unroll here, the asm is pretty horrid for what it does.  
> > 
> > This one is much improved:
> > 
> > --- a/arch/x86/include/asm/processor.h
> > +++ b/arch/x86/include/asm/processor.h
> > @@ -28,6 +28,7 @@ struct vm86;
> >  #include <asm/vmxfeatures.h>
> >  #include <asm/vdso/processor.h>
> >  #include <asm/shstk.h>
> > +#include <asm/string.h>
> >  
> >  #include <linux/personality.h>
> >  #include <linux/cache.h>
> > @@ -580,6 +581,35 @@ static __always_inline bool on_thread_st
> >  			       current_stack_pointer) < THREAD_SIZE;
> >  }
> >  
> > +#ifdef CONFIG_X86_64
> > +#define __stackleak_poison __stackleak_poison
> > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > +					       unsigned long erase_high,
> > +					       unsigned long poison)
> > +{
> > +	unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
> > +
> > +	/* Amortize the startup cost of REP STOSQ. */
> > +	if (count < 32) {
> > +		unsigned long *ptr = (unsigned long *)erase_low;
> > +
> > +		while (count) {
> > +			unsigned long mod = (count-1) % 4;
> > +			count -= mod+1;
> > +			switch (mod) {
> > +			case 3: ptr[count+3] = poison; fallthrough;
> > +			case 2: ptr[count+2] = poison; fallthrough;
> > +			case 1: ptr[count+1] = poison; fallthrough;
> > +			case 0: ptr[count+0] = poison; break;
> > +			}
> > +		}
> > +		return;
> > +	}
> > +  
> 
> Hello Nick, Peter and David,
> 
> Thanks for the reviews! I should have used memset64 from the get go :)
> 
> I have tested 2 possible revisions now as well as this one from Peter,
> have added both of them at the end of the reply.
> 
> - v2.1 where you loop upto 512B and then memset
> - v2.2: 4 stores per loop up to 512 B, then 64-byte aligned memset64()
> 
> I timed stackleak_erase() in a Zen 5c guest over a futex/epoll/procfs/unix
> syscall:
> 
>   No patch                             68.3 ns per erase
>   v1 (loop up to 512 B, then REP)           46.6 ns
>   v2.1 (loop up to 512 B, memset64())       46.6 ns
>   this version                              46.4 ns
>   v2.2 (4 stores/iteration, aligned REP)    43.7 ns

What are the lengths of those erases?
(and where does the 46ns come from??)
I'd definitely time 'just the code loop  - various loops' and
'just rep stos'.

> The switch doesn't seem to help.

To many conditions in the loop.
You could do it this way:
	switch (count & 3) {
	case 0: do { *ptr++ = poison;
	case 3:      *ptr++ = poison;
	case 2:      *ptr++ = poison;
	case 1:      *ptr++ = poison;
		} while ((count -= 4) > 0);
	}
(give or take some off-by-one here and there).
But I suspect that using Duff's device is frowned up.

> Either way I kept 512 bytes: REP STOSQ only wins from ~430 bytes on Zen
> 5c against the plain loop (~850 against the unrolled one),

For zen5 you should only need to unroll the loop once (two writes per iteration).
In my experiments it can execute a loop in one clock.
Unless the compiler unrolled it for you (and got it right!) the original
loop would be half the speed 

> and from ~320 on Cooper Lake.

I'd also try some older Intel cpu.
The setup costs are likely to be higher with FRMS and friends.

David

> 
> 
> My personal preference is v2.1, as its much simpler code and the difference
> in performance between v2.1 and v2.2 isn't actually that much. But if there is
> a stronger preference for v2.2, I can send that as it is faster.
> 
> 
> v2.1:
> 
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index 77488ff5eb8c7c..3d205ae8566114 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -28,6 +28,7 @@ struct vm86;
>  #include <asm/vmxfeatures.h>
>  #include <asm/vdso/processor.h>
>  #include <asm/shstk.h>
> +#include <asm/string.h>
> 
>  #include <linux/personality.h>
>  #include <linux/cache.h>
> @@ -580,6 +581,26 @@ static __always_inline bool on_thread_stack(void)
>                                current_stack_pointer) < THREAD_SIZE;
>  }
> 
> +#if defined(CONFIG_X86_64) && !defined(CONFIG_KMSAN)
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +                                              unsigned long erase_high,
> +                                              unsigned long poison)
> +{
> +       /* Short spans do not amortize the startup cost of REP STOSQ. */
> +       if (erase_high <= erase_low + 512) {
> +               while (erase_low < erase_high) {
> +                       *(unsigned long *)erase_low = poison;
> +                       erase_low += sizeof(unsigned long);
> +               }
> +               return;
> +       }
> +
> +       memset64((u64 *)erase_low, poison,
> +                (erase_high - erase_low) / sizeof(u64));
> +}
> +#endif
> +
>  #ifdef CONFIG_PARAVIRT_XXL
>  #include <asm/paravirt.h>
>  #else
> diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
> index 2dae70c2b01465..73cc91254b1381 100644
> --- a/arch/x86/include/asm/string_64.h
> +++ b/arch/x86/include/asm/string_64.h
> @@ -54,7 +54,7 @@ static inline void *memset32(uint32_t *s, uint32_t v, size_t n)
>  }
> 
>  #define __HAVE_ARCH_MEMSET64
> -static inline void *memset64(uint64_t *s, uint64_t v, size_t n)
> +static __always_inline void *memset64(uint64_t *s, uint64_t v, size_t n)
>  {
>         const auto s0 = s;
>         asm volatile (
> 
> 
> 
> 
> 
> v2.2:
> 
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index 77488ff5eb8c7c..5e76e1140908d7 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -28,7 +28,9 @@ struct vm86;
>  #include <asm/vmxfeatures.h>
>  #include <asm/vdso/processor.h>
>  #include <asm/shstk.h>
> +#include <asm/string.h>
> 
> +#include <linux/align.h>
>  #include <linux/personality.h>
>  #include <linux/cache.h>
>  #include <linux/threads.h>
> @@ -580,6 +582,48 @@ static __always_inline bool on_thread_stack(void)
>                                current_stack_pointer) < THREAD_SIZE;
>  }
> 
> +#if defined(CONFIG_X86_64) && !defined(CONFIG_KMSAN)
> +#define __stackleak_poison __stackleak_poison
> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> +                                              unsigned long erase_high,
> +                                              unsigned long poison)
> +{
> +       unsigned long *p = (unsigned long *)erase_low;
> +       unsigned long *end = (unsigned long *)erase_high;
> +       unsigned long i;
> +
> +       /* Fewer than four words, or nothing to do. */
> +       if (erase_high < erase_low + 4 * sizeof(*p)) {
> +               while (p < end)
> +                       *p++ = poison;
> +               return;
> +       }
> +
> +       /* Four stores per iteration; the last four may overlap. */
> +       if (erase_high - erase_low <= 512) {
> +               end -= 4;
> +               while (p < end) {
> +                       p[0] = poison;
> +                       p[1] = poison;
> +                       p[2] = poison;
> +                       p[3] = poison;
> +                       p += 4;
> +               }
> +               end[0] = poison;
> +               end[1] = poison;
> +               end[2] = poison;
> +               end[3] = poison;
> +               return;
> +       }
> +
> +       /* REP STOSQ is faster from a 64-byte aligned destination. */
> +       for (i = 0; i < 8; i++)
> +               p[i] = poison;
> +       p = (unsigned long *)ALIGN(erase_low + 1, 64);
> +       memset64((u64 *)p, poison, end - p);
> +}
> +#endif
> +
>  #ifdef CONFIG_PARAVIRT_XXL
>  #include <asm/paravirt.h>
>  #else
> diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
> index 2dae70c2b01465..73cc91254b1381 100644
> --- a/arch/x86/include/asm/string_64.h
> +++ b/arch/x86/include/asm/string_64.h
> @@ -54,7 +54,7 @@ static inline void *memset32(uint32_t *s, uint32_t v, size_t n)
>  }
> 
>  #define __HAVE_ARCH_MEMSET64
> -static inline void *memset64(uint64_t *s, uint64_t v, size_t n)
> +static __always_inline void *memset64(uint64_t *s, uint64_t v, size_t n)
>  {
>         const auto s0 = s;
>         asm volatile (
> 
> > +	memset64((void *)erase_low, poison, count);
> > +}
> > +#endif
> > +
> >  #ifdef CONFIG_PARAVIRT_XXL
> >  #include <asm/paravirt.h>
> >  #else  
> 
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2026-11-20 22:22 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30   ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28   ` Nick Desaulniers
2026-11-20 11:38     ` David Laight
2026-11-20 12:42       ` David Laight
2026-11-20 18:46         ` Usama Arif
2026-11-20 21:44           ` David Laight
2026-11-20 10:45   ` Peter Zijlstra
2026-11-20 12:26     ` Peter Zijlstra
2026-11-20 18:43       ` Usama Arif
2026-11-20 22:22         ` David Laight
2026-11-20 14:02     ` David Laight
2026-11-20 15:35       ` Peter Zijlstra
2026-11-20 16:18         ` David Laight
2026-11-20 18:55   ` David Laight

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox