* [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix @ 2026-11-19 21:31 Usama Arif 2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif 2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif 0 siblings, 2 replies; 12+ messages in thread From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw) To: tglx, mingo, bp, dave.hansen, x86, kees Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid, Usama Arif With KSTACK_ERASE, every return to user space poisons the part of the task stack used since the previous return, one 8-byte store at a time. On Meta hosts running many small VMs, that loop costs about 1.5% of non-idle host kernel cycles. Most of it is on the way out of futex, epoll, procfs and socket syscalls that go 1-2 KiB deep. Patch 2 makes x86-64 use REP STOSQ for spans above 512 bytes, through the arch hook that s390 already uses. Shorter spans keep the loop, and the range and value written do not change. Patch 1 is an independent build fix I found while testing patch 2 with a small KSTACK_ERASE_TRACK_MIN_SIZE. The startup code still uses the pre-rename DISABLE_STACKLEAK_PLUGIN, so with Clang, AMD memory encryption and a tracking minimum below about 90 bytes, vmlinux fails to link. It can be applied on its own. Zen 5c KVM guest, one vCPU, before -> after: stackleak_erase() per return to user 71 ns -> 46 ns futex wake/wait ping-pong 5-7% faster Unix socket sendmsg()/recvmsg() ping-pong 5-8% faster open+read+close of /proc/self/stat 6-8% faster getppid() (spans below 512 bytes) within 2% Usama Arif (2): x86/boot/startup: Disable kernel stack depth tracking x86/kstack_erase: Use REP STOSQ for large stack fills arch/x86/boot/startup/Makefile | 2 +- arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) -- 2.53.1-Meta ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking 2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif @ 2026-11-19 21:31 ` Usama Arif 2026-11-19 22:30 ` Nick Desaulniers 2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif 1 sibling, 1 reply; 12+ messages in thread From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw) To: tglx, mingo, bp, dave.hansen, x86, kees Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid, Usama Arif Commit 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE") renamed DISABLE_STACKLEAK_PLUGIN to DISABLE_KSTACK_ERASE everywhere except in arch/x86/boot/startup/Makefile. The old name now expands to nothing, so the startup code is built with stack depth tracking enabled. The startup code is linked with every symbol prefixed by __pi_, so that it cannot call into the rest of the kernel by accident. It has no alias for the tracking callback, and tracking stack depth there is of no use. At the default KSTACK_ERASE_TRACK_MIN_SIZE of 100 no startup function is large enough to be tracked, so this went unnoticed. With Clang, CONFIG_AMD_MEM_ENCRYPT=y and a minimum below about 90 bytes, pvalidate_4k_page(), snp_cpuid() and do_vc_no_ghcb() get a call to __sanitizer_cov_stack_depth() and vmlinux fails to link: ld.lld: error: undefined symbol: __pi___sanitizer_cov_stack_depth Use DISABLE_KSTACK_ERASE, which disables tracking with either compiler. Fixes: 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE") Signed-off-by: Usama Arif <usama.arif@linux.dev> --- arch/x86/boot/startup/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile index 5e499cfb29b5c0..0815105e7205c7 100644 --- a/arch/x86/boot/startup/Makefile +++ b/arch/x86/boot/startup/Makefile @@ -3,7 +3,7 @@ KBUILD_AFLAGS += -D__DISABLE_EXPORTS KBUILD_CFLAGS += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \ -Os -DDISABLE_BRANCH_PROFILING \ - $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_KSTACK_ERASE) \ $(DISABLE_LATENT_ENTROPY_PLUGIN) \ -fno-stack-protector -D__NO_FORTIFY \ -fno-jump-tables \ -- 2.53.1-Meta ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking 2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif @ 2026-11-19 22:30 ` Nick Desaulniers 0 siblings, 0 replies; 12+ messages in thread From: Nick Desaulniers @ 2026-11-19 22:30 UTC (permalink / raw) To: Usama Arif, Kees Cook Cc: tglx, mingo, bp, dave.hansen, x86, hpa, gustavoars, ardb, hca, nathan, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Thu, Nov 19, 2026 at 1:31 PM Usama Arif <usama.arif@linux.dev> wrote: > > Commit 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE") > renamed DISABLE_STACKLEAK_PLUGIN to DISABLE_KSTACK_ERASE everywhere > except in arch/x86/boot/startup/Makefile. The old name now expands to > nothing, so the startup code is built with stack depth tracking enabled. > > The startup code is linked with every symbol prefixed by __pi_, so that > it cannot call into the rest of the kernel by accident. It has no alias > for the tracking callback, and tracking stack depth there is of no use. > > At the default KSTACK_ERASE_TRACK_MIN_SIZE of 100 no startup function is > large enough to be tracked, so this went unnoticed. With Clang, > CONFIG_AMD_MEM_ENCRYPT=y and a minimum below about 90 bytes, > pvalidate_4k_page(), snp_cpuid() and do_vc_no_ghcb() get a call to > __sanitizer_cov_stack_depth() and vmlinux fails to link: > > ld.lld: error: undefined symbol: __pi___sanitizer_cov_stack_depth > > Use DISABLE_KSTACK_ERASE, which disables tracking with either compiler. > > Fixes: 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE") tsk tsk, Kees your `grep` might be borked! Reviewed-by: Nick Desaulniers <ndesaulniers@google.com> > Signed-off-by: Usama Arif <usama.arif@linux.dev> > --- > arch/x86/boot/startup/Makefile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile > index 5e499cfb29b5c0..0815105e7205c7 100644 > --- a/arch/x86/boot/startup/Makefile > +++ b/arch/x86/boot/startup/Makefile > @@ -3,7 +3,7 @@ > KBUILD_AFLAGS += -D__DISABLE_EXPORTS > KBUILD_CFLAGS += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \ > -Os -DDISABLE_BRANCH_PROFILING \ > - $(DISABLE_STACKLEAK_PLUGIN) \ > + $(DISABLE_KSTACK_ERASE) \ > $(DISABLE_LATENT_ENTROPY_PLUGIN) \ > -fno-stack-protector -D__NO_FORTIFY \ > -fno-jump-tables \ > -- > 2.53.1-Meta > -- Thanks, ~Nick Desaulniers ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif 2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif @ 2026-11-19 21:31 ` Usama Arif 2026-11-19 22:28 ` Nick Desaulniers 2026-11-20 10:45 ` Peter Zijlstra 1 sibling, 2 replies; 12+ messages in thread From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw) To: tglx, mingo, bp, dave.hansen, x86, kees Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid, Usama Arif With KSTACK_ERASE, every return to user space poisons the part of the task stack that was used since the previous return. The generic __stackleak_poison() does that one word per loop iteration, so a syscall that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts running many small VMs, stackleak_erase() accounts for about 1.5% of non-idle host kernel cycles. Override __stackleak_poison() on x86-64, as s390 already does, through the hook added by commit 491a78663e03 ("stackleak: allow to specify arch specific stackleak poison function"). Fill spans larger than 512 bytes with REP STOSQ and keep the word loop for shorter ones, where the fixed startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets slower. The fill is inline asm, so the noinstr function still makes no calls. memset64() would be the same instruction, but it is only static inline, and with KMSAN it is an out-of-line function. In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket sendmsg()/recvmsg() ping-pong round trips got 5-8% faster. Signed-off-by: Usama Arif <usama.arif@linux.dev> --- arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 77488ff5eb8c7c..049231d484ad0e 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void) current_stack_pointer) < THREAD_SIZE; } +#ifdef CONFIG_X86_64 +#define __stackleak_poison __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, + unsigned long erase_high, + unsigned long poison) +{ + unsigned long count; + + /* Short spans do not amortize the startup cost of REP STOSQ. */ + if (erase_high <= erase_low + 512) { + while (erase_low < erase_high) { + *(unsigned long *)erase_low = poison; + erase_low += sizeof(unsigned long); + } + return; + } + + count = (erase_high - erase_low) / sizeof(unsigned long); + asm volatile("rep stosq" + : "+D" (erase_low), "+c" (count) + : "a" (poison) + : "memory"); +} +#endif + #ifdef CONFIG_PARAVIRT_XXL #include <asm/paravirt.h> #else -- 2.53.1-Meta ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif @ 2026-11-19 22:28 ` Nick Desaulniers 2026-11-20 11:38 ` David Laight 2026-11-20 10:45 ` Peter Zijlstra 1 sibling, 1 reply; 12+ messages in thread From: Nick Desaulniers @ 2026-11-19 22:28 UTC (permalink / raw) To: Usama Arif Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote: > > With KSTACK_ERASE, every return to user space poisons the part of the > task stack that was used since the previous return. The generic > __stackleak_poison() does that one word per loop iteration, so a syscall > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts > running many small VMs, stackleak_erase() accounts for about 1.5% of > non-idle host kernel cycles. > > Override __stackleak_poison() on x86-64, as s390 already does, through > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch > specific stackleak poison function"). Fill spans larger than 512 bytes > with REP STOSQ and keep the word loop for shorter ones, where the fixed > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets > slower. > > The fill is inline asm, so the noinstr function still makes no calls. > memset64() would be the same instruction, but it is only static inline, > and with KMSAN it is an out-of-line function. > > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster. > > Signed-off-by: Usama Arif <usama.arif@linux.dev> > --- > arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h > index 77488ff5eb8c7c..049231d484ad0e 100644 > --- a/arch/x86/include/asm/processor.h > +++ b/arch/x86/include/asm/processor.h > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void) > current_stack_pointer) < THREAD_SIZE; > } > > +#ifdef CONFIG_X86_64 > +#define __stackleak_poison __stackleak_poison > +static __always_inline void __stackleak_poison(unsigned long erase_low, > + unsigned long erase_high, > + unsigned long poison) > +{ > + unsigned long count; > + > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > + if (erase_high <= erase_low + 512) { > + while (erase_low < erase_high) { > + *(unsigned long *)erase_low = poison; > + erase_low += sizeof(unsigned long); > + } > + return; > + } > + > + count = (erase_high - erase_low) / sizeof(unsigned long); > + asm volatile("rep stosq" > + : "+D" (erase_low), "+c" (count) > + : "a" (poison) > + : "memory"); Surely this isn't the only place using `rep stosq` in the tree, right? It's not, see memset64 in arch/x86/include/asm/string_64.h. > +} > +#endif > + > #ifdef CONFIG_PARAVIRT_XXL > #include <asm/paravirt.h> > #else > -- > 2.53.1-Meta > -- Thanks, ~Nick Desaulniers ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-19 22:28 ` Nick Desaulniers @ 2026-11-20 11:38 ` David Laight 2026-11-20 12:42 ` David Laight 0 siblings, 1 reply; 12+ messages in thread From: David Laight @ 2026-11-20 11:38 UTC (permalink / raw) To: Nick Desaulniers Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Thu, 19 Nov 2026 14:28:04 -0800 Nick Desaulniers <ndesaulniers@google.com> wrote: > On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote: > > > > With KSTACK_ERASE, every return to user space poisons the part of the > > task stack that was used since the previous return. The generic > > __stackleak_poison() does that one word per loop iteration, so a syscall > > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts > > running many small VMs, stackleak_erase() accounts for about 1.5% of > > non-idle host kernel cycles. > > > > Override __stackleak_poison() on x86-64, as s390 already does, through > > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch > > specific stackleak poison function"). Fill spans larger than 512 bytes > > with REP STOSQ and keep the word loop for shorter ones, where the fixed > > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at > > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets > > slower. > > > > The fill is inline asm, so the noinstr function still makes no calls. > > memset64() would be the same instruction, but it is only static inline, > > and with KMSAN it is an out-of-line function. > > > > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return > > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs > > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket > > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster. > > > > Signed-off-by: Usama Arif <usama.arif@linux.dev> > > --- > > arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++ > > 1 file changed, 25 insertions(+) > > > > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h > > index 77488ff5eb8c7c..049231d484ad0e 100644 > > --- a/arch/x86/include/asm/processor.h > > +++ b/arch/x86/include/asm/processor.h > > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void) > > current_stack_pointer) < THREAD_SIZE; > > } > > > > +#ifdef CONFIG_X86_64 > > +#define __stackleak_poison __stackleak_poison > > +static __always_inline void __stackleak_poison(unsigned long erase_low, > > + unsigned long erase_high, > > + unsigned long poison) > > +{ > > + unsigned long count; > > + > > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > > + if (erase_high <= erase_low + 512) { > > + while (erase_low < erase_high) { > > + *(unsigned long *)erase_low = poison; > > + erase_low += sizeof(unsigned long); > > + } > > + return; gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6 It does manage to unroll the loop once - which is good because the intel cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the setup code is horrid. But I also think some of the more recent cpu can execute more than one write per clock. If you unroll the loop 4 times it may run a lot faster. Perhaps: long count = (erase_high - erase_low) / sizeof(unsigned long); if (count <= 64) { while ((count -= 4) >= 0) { *(unsigned long *)erase_low = poison; *(unsigned long *)(erase_low + 8) = poison; *(unsigned long *)(erase_low + 16) = poison; *(unsigned long *)(erase_low + 24) = poison; erase_low += 32; } if (count & 2) { *(unsigned long *)erase_low = poison; *(unsigned long *)(erase_low + 8) = poison; erase_low += 16; } if (count & 1) *(unsigned long *)erase_low = poison; } else { ... } Except that gcc makes a mess of it again. I don't think the 'erase_low += 32' is going to execute at the same time as the last (two probably) writes so costs a clock. So it may need to be asm. > > + } > > + > > + count = (erase_high - erase_low) / sizeof(unsigned long); > > + asm volatile("rep stosq" > > + : "+D" (erase_low), "+c" (count) > > + : "a" (poison) > > + : "memory"); > > Surely this isn't the only place using `rep stosq` in the tree, right? > It's not, see memset64 in arch/x86/include/asm/string_64.h. Which seems to have very few users. memset_p() and memset_l() have almost no users. David > > > +} > > +#endif > > + > > #ifdef CONFIG_PARAVIRT_XXL > > #include <asm/paravirt.h> > > #else > > -- > > 2.53.1-Meta > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-20 11:38 ` David Laight @ 2026-11-20 12:42 ` David Laight 0 siblings, 0 replies; 12+ messages in thread From: David Laight @ 2026-11-20 12:42 UTC (permalink / raw) To: Nick Desaulniers Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Fri, 20 Nov 2026 11:38:25 +0000 David Laight <david.laight.linux@gmail.com> wrote: > On Thu, 19 Nov 2026 14:28:04 -0800 > Nick Desaulniers <ndesaulniers@google.com> wrote: > > > On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote: > > > > > > With KSTACK_ERASE, every return to user space poisons the part of the > > > task stack that was used since the previous return. The generic > > > __stackleak_poison() does that one word per loop iteration, so a syscall > > > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts > > > running many small VMs, stackleak_erase() accounts for about 1.5% of > > > non-idle host kernel cycles. > > > > > > Override __stackleak_poison() on x86-64, as s390 already does, through > > > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch > > > specific stackleak poison function"). Fill spans larger than 512 bytes > > > with REP STOSQ and keep the word loop for shorter ones, where the fixed > > > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at > > > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets > > > slower. > > > > > > The fill is inline asm, so the noinstr function still makes no calls. > > > memset64() would be the same instruction, but it is only static inline, > > > and with KMSAN it is an out-of-line function. > > > > > > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return > > > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs > > > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket > > > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster. > > > > > > Signed-off-by: Usama Arif <usama.arif@linux.dev> > > > --- > > > arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++ > > > 1 file changed, 25 insertions(+) > > > > > > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h > > > index 77488ff5eb8c7c..049231d484ad0e 100644 > > > --- a/arch/x86/include/asm/processor.h > > > +++ b/arch/x86/include/asm/processor.h > > > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void) > > > current_stack_pointer) < THREAD_SIZE; > > > } > > > > > > +#ifdef CONFIG_X86_64 > > > +#define __stackleak_poison __stackleak_poison > > > +static __always_inline void __stackleak_poison(unsigned long erase_low, > > > + unsigned long erase_high, > > > + unsigned long poison) > > > +{ > > > + unsigned long count; > > > + > > > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > > > + if (erase_high <= erase_low + 512) { > > > + while (erase_low < erase_high) { > > > + *(unsigned long *)erase_low = poison; > > > + erase_low += sizeof(unsigned long); > > > + } > > > + return; > > gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6 > It does manage to unroll the loop once - which is good because the intel > cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the > setup code is horrid. > But I also think some of the more recent cpu can execute more than > one write per clock. > If you unroll the loop 4 times it may run a lot faster. > Perhaps: > long count = (erase_high - erase_low) / sizeof(unsigned long); > if (count <= 64) { > while ((count -= 4) >= 0) { > *(unsigned long *)erase_low = poison; > *(unsigned long *)(erase_low + 8) = poison; The add (probably) needs to be in the middle... erase_low += 32; > *(unsigned long *)(erase_low - 32 + 16) = poison; > *(unsigned long *)(erase_low - 32 + 24) = poison; > } > if (count & 2) { > *(unsigned long *)erase_low = poison; > *(unsigned long *)(erase_low + 8) = poison; > erase_low += 16; > } > if (count & 1) > *(unsigned long *)erase_low = poison; > } else { > ... > } > > Except that gcc makes a mess of it again. > I don't think the 'erase_low += 32' is going to execute at the > same time as the last (two probably) writes so costs a clock. > So it may need to be asm. I threw in enough OPTIMIZER_HIDE_VAR() to stop the optimiser breaking the code. https://godbolt.org/z/nx477T4M1 gcc then generates: movq %rdx, (%rdi) movq %rdx, 8(%rdi) addq $32, %rdi movq %rdx, -16(%rdi) movq %rdx, -8(%rdi) .L32: subq $4, %rcx jns .L22 for the code loop (clang is a bit worse - but should be ok). I think that will be a 'two clock loop' on cpu that can do two writes/clock; zen3 and ice lake). For 'stack splatting' rounding up to a multiple of 32 bytes will save all the 'remainder' code. David > > > > + } > > > + > > > + count = (erase_high - erase_low) / sizeof(unsigned long); > > > + asm volatile("rep stosq" > > > + : "+D" (erase_low), "+c" (count) > > > + : "a" (poison) > > > + : "memory"); > > > > Surely this isn't the only place using `rep stosq` in the tree, right? > > It's not, see memset64 in arch/x86/include/asm/string_64.h. > > Which seems to have very few users. > memset_p() and memset_l() have almost no users. > > David > > > > > > +} > > > +#endif > > > + > > > #ifdef CONFIG_PARAVIRT_XXL > > > #include <asm/paravirt.h> > > > #else > > > -- > > > 2.53.1-Meta > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif 2026-11-19 22:28 ` Nick Desaulniers @ 2026-11-20 10:45 ` Peter Zijlstra 2026-11-20 12:26 ` Peter Zijlstra 2026-11-20 14:02 ` David Laight 1 sibling, 2 replies; 12+ messages in thread From: Peter Zijlstra @ 2026-11-20 10:45 UTC (permalink / raw) To: Usama Arif Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote: > +#ifdef CONFIG_X86_64 > +#define __stackleak_poison __stackleak_poison > +static __always_inline void __stackleak_poison(unsigned long erase_low, > + unsigned long erase_high, > + unsigned long poison) > +{ > + unsigned long count; > + > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > + if (erase_high <= erase_low + 512) { > + while (erase_low < erase_high) { > + *(unsigned long *)erase_low = poison; > + erase_low += sizeof(unsigned long); > + } > + return; > + } > + > + count = (erase_high - erase_low) / sizeof(unsigned long); > + asm volatile("rep stosq" > + : "+D" (erase_low), "+c" (count) > + : "a" (poison) > + : "memory"); > +} > +#endif Gemini seems to suggest 16-32 words would be sufficient to amortize the REP STOS setup. It was also adamant I try and unroll that earlier loop. How about something like so, does that make sense? Also, I suppose we should make memset64() __always_inline if we use it here. I'm not at all sold on the unroll here, the asm is pretty horrid for what it does. --- --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -28,6 +28,7 @@ struct vm86; #include <asm/vmxfeatures.h> #include <asm/vdso/processor.h> #include <asm/shstk.h> +#include <asm/string.h> #include <linux/personality.h> #include <linux/cache.h> @@ -580,6 +581,33 @@ static __always_inline bool on_thread_st current_stack_pointer) < THREAD_SIZE; } +#ifdef CONFIG_X86_64 +#define __stackleak_poison __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, + unsigned long erase_high, + unsigned long poison) +{ + unsigned long count = (erase_high - erase_low) / sizeof(unsigned long); + + /* Amortize the startup cost of REP STOSQ. */ + if (count < 32) { + unsigned long *ptr = (unsigned long *)erase_low; + + while (count) { + switch ((count-1) % 4) { + case 3: ptr[--count] = poison; fallthrough; + case 2: ptr[--count] = poison; fallthrough; + case 1: ptr[--count] = poison; fallthrough; + case 0: ptr[--count] = poison; break; + } + } + return; + } + + memset64((void *)erase_low, poison, count); +} +#endif + #ifdef CONFIG_PARAVIRT_XXL #include <asm/paravirt.h> #else ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-20 10:45 ` Peter Zijlstra @ 2026-11-20 12:26 ` Peter Zijlstra 2026-11-20 14:02 ` David Laight 1 sibling, 0 replies; 12+ messages in thread From: Peter Zijlstra @ 2026-11-20 12:26 UTC (permalink / raw) To: Usama Arif Cc: tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Fri, Nov 20, 2026 at 11:45:34AM +0100, Peter Zijlstra wrote: > On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote: > > +#ifdef CONFIG_X86_64 > > +#define __stackleak_poison __stackleak_poison > > +static __always_inline void __stackleak_poison(unsigned long erase_low, > > + unsigned long erase_high, > > + unsigned long poison) > > +{ > > + unsigned long count; > > + > > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > > + if (erase_high <= erase_low + 512) { > > + while (erase_low < erase_high) { > > + *(unsigned long *)erase_low = poison; > > + erase_low += sizeof(unsigned long); > > + } > > + return; > > + } > > + > > + count = (erase_high - erase_low) / sizeof(unsigned long); > > + asm volatile("rep stosq" > > + : "+D" (erase_low), "+c" (count) > > + : "a" (poison) > > + : "memory"); > > +} > > +#endif > > Gemini seems to suggest 16-32 words would be sufficient to amortize the > REP STOS setup. It was also adamant I try and unroll that earlier loop. > > How about something like so, does that make sense? Also, I suppose we > should make memset64() __always_inline if we use it here. I'm not at all > sold on the unroll here, the asm is pretty horrid for what it does. This one is much improved: --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -28,6 +28,7 @@ struct vm86; #include <asm/vmxfeatures.h> #include <asm/vdso/processor.h> #include <asm/shstk.h> +#include <asm/string.h> #include <linux/personality.h> #include <linux/cache.h> @@ -580,6 +581,35 @@ static __always_inline bool on_thread_st current_stack_pointer) < THREAD_SIZE; } +#ifdef CONFIG_X86_64 +#define __stackleak_poison __stackleak_poison +static __always_inline void __stackleak_poison(unsigned long erase_low, + unsigned long erase_high, + unsigned long poison) +{ + unsigned long count = (erase_high - erase_low) / sizeof(unsigned long); + + /* Amortize the startup cost of REP STOSQ. */ + if (count < 32) { + unsigned long *ptr = (unsigned long *)erase_low; + + while (count) { + unsigned long mod = (count-1) % 4; + count -= mod+1; + switch (mod) { + case 3: ptr[count+3] = poison; fallthrough; + case 2: ptr[count+2] = poison; fallthrough; + case 1: ptr[count+1] = poison; fallthrough; + case 0: ptr[count+0] = poison; break; + } + } + return; + } + + memset64((void *)erase_low, poison, count); +} +#endif + #ifdef CONFIG_PARAVIRT_XXL #include <asm/paravirt.h> #else ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-20 10:45 ` Peter Zijlstra 2026-11-20 12:26 ` Peter Zijlstra @ 2026-11-20 14:02 ` David Laight 2026-11-20 15:35 ` Peter Zijlstra 1 sibling, 1 reply; 12+ messages in thread From: David Laight @ 2026-11-20 14:02 UTC (permalink / raw) To: Peter Zijlstra Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Fri, 20 Nov 2026 11:45:34 +0100 Peter Zijlstra <peterz@infradead.org> wrote: > On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote: > > +#ifdef CONFIG_X86_64 > > +#define __stackleak_poison __stackleak_poison > > +static __always_inline void __stackleak_poison(unsigned long erase_low, > > + unsigned long erase_high, > > + unsigned long poison) > > +{ > > + unsigned long count; > > + > > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > > + if (erase_high <= erase_low + 512) { > > + while (erase_low < erase_high) { > > + *(unsigned long *)erase_low = poison; > > + erase_low += sizeof(unsigned long); > > + } > > + return; > > + } > > + > > + count = (erase_high - erase_low) / sizeof(unsigned long); > > + asm volatile("rep stosq" > > + : "+D" (erase_low), "+c" (count) > > + : "a" (poison) > > + : "memory"); > > +} > > +#endif > > Gemini seems to suggest 16-32 words would be sufficient to amortize the > REP STOS setup. That really does sound like wishful thinking. Especially if you go back a few generations on the cpu architecture. > It was also adamant I try and unroll that earlier loop. > > How about something like so, does that make sense? Also, I suppose we > should make memset64() __always_inline if we use it here. I'm not at all > sold on the unroll here, the asm is pretty horrid for what it does. > > --- > --- a/arch/x86/include/asm/processor.h > +++ b/arch/x86/include/asm/processor.h > @@ -28,6 +28,7 @@ struct vm86; > #include <asm/vmxfeatures.h> > #include <asm/vdso/processor.h> > #include <asm/shstk.h> > +#include <asm/string.h> > > #include <linux/personality.h> > #include <linux/cache.h> > @@ -580,6 +581,33 @@ static __always_inline bool on_thread_st > current_stack_pointer) < THREAD_SIZE; > } > > +#ifdef CONFIG_X86_64 > +#define __stackleak_poison __stackleak_poison > +static __always_inline void __stackleak_poison(unsigned long erase_low, > + unsigned long erase_high, > + unsigned long poison) > +{ > + unsigned long count = (erase_high - erase_low) / sizeof(unsigned long); > + > + /* Amortize the startup cost of REP STOSQ. */ > + if (count < 32) { > + unsigned long *ptr = (unsigned long *)erase_low; > + > + while (count) { > + switch ((count-1) % 4) { > + case 3: ptr[--count] = poison; fallthrough; > + case 2: ptr[--count] = poison; fallthrough; > + case 1: ptr[--count] = poison; fallthrough; > + case 0: ptr[--count] = poison; break; > + } > + } That looks horrid to me. You want as few instructions (especially conditionals) inside the loop as possible. The Intel cpu will execute a two clock loop and the later ones will do two writes per clock (zen5 will execute a one clock loop and do two writes/clock). So you want a loop with just 4 writes and the loop control code. The 'pointer add' just needs to execute at the same time as some of the stores - to avoid lengthening the register dependency chain. For this code, can the length be a multiple of 32? Also on some cpu the 'rep stos' might be twice as fast (per byte) if the destination in 64 byte aligned - true for 'rep movsb'. David > + return; > + } > + > + memset64((void *)erase_low, poison, count); > +} > +#endif > + > #ifdef CONFIG_PARAVIRT_XXL > #include <asm/paravirt.h> > #else > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-20 14:02 ` David Laight @ 2026-11-20 15:35 ` Peter Zijlstra 2026-11-20 16:18 ` David Laight 0 siblings, 1 reply; 12+ messages in thread From: Peter Zijlstra @ 2026-11-20 15:35 UTC (permalink / raw) To: David Laight Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Fri, Nov 20, 2026 at 02:02:00PM +0000, David Laight wrote: > On Fri, 20 Nov 2026 11:45:34 +0100 > Peter Zijlstra <peterz@infradead.org> wrote: > > > On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote: > > > +#ifdef CONFIG_X86_64 > > > +#define __stackleak_poison __stackleak_poison > > > +static __always_inline void __stackleak_poison(unsigned long erase_low, > > > + unsigned long erase_high, > > > + unsigned long poison) > > > +{ > > > + unsigned long count; > > > + > > > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > > > + if (erase_high <= erase_low + 512) { > > > + while (erase_low < erase_high) { > > > + *(unsigned long *)erase_low = poison; > > > + erase_low += sizeof(unsigned long); > > > + } > > > + return; > > > + } > > > + > > > + count = (erase_high - erase_low) / sizeof(unsigned long); > > > + asm volatile("rep stosq" > > > + : "+D" (erase_low), "+c" (count) > > > + : "a" (poison) > > > + : "memory"); > > > +} > > > +#endif > > > > Gemini seems to suggest 16-32 words would be sufficient to amortize the > > REP STOS setup. > > That really does sound like wishful thinking. > Especially if you go back a few generations on the cpu architecture. How old? I mean, Interlagos old, sure, but nobody should care about performance there I think. AFAICT Nehalem and later should be good with the 16-32 words. As would all the various Zen chips. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills 2026-11-20 15:35 ` Peter Zijlstra @ 2026-11-20 16:18 ` David Laight 0 siblings, 0 replies; 12+ messages in thread From: David Laight @ 2026-11-20 16:18 UTC (permalink / raw) To: Peter Zijlstra Cc: Usama Arif, tglx, mingo, bp, dave.hansen, x86, kees, hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo, justinstitt, linux-hardening, llvm, linux-kernel, riel, shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid On Fri, 20 Nov 2026 16:35:39 +0100 Peter Zijlstra <peterz@infradead.org> wrote: > On Fri, Nov 20, 2026 at 02:02:00PM +0000, David Laight wrote: > > On Fri, 20 Nov 2026 11:45:34 +0100 > > Peter Zijlstra <peterz@infradead.org> wrote: > > > > > On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote: > > > > +#ifdef CONFIG_X86_64 > > > > +#define __stackleak_poison __stackleak_poison > > > > +static __always_inline void __stackleak_poison(unsigned long erase_low, > > > > + unsigned long erase_high, > > > > + unsigned long poison) > > > > +{ > > > > + unsigned long count; > > > > + > > > > + /* Short spans do not amortize the startup cost of REP STOSQ. */ > > > > + if (erase_high <= erase_low + 512) { > > > > + while (erase_low < erase_high) { > > > > + *(unsigned long *)erase_low = poison; > > > > + erase_low += sizeof(unsigned long); > > > > + } > > > > + return; > > > > + } > > > > + > > > > + count = (erase_high - erase_low) / sizeof(unsigned long); > > > > + asm volatile("rep stosq" > > > > + : "+D" (erase_low), "+c" (count) > > > > + : "a" (poison) > > > > + : "memory"); > > > > +} > > > > +#endif > > > > > > Gemini seems to suggest 16-32 words would be sufficient to amortize the > > > REP STOS setup. > > > > That really does sound like wishful thinking. > > Especially if you go back a few generations on the cpu architecture. > > How old? I mean, Interlagos old, sure, but nobody should care about > performance there I think. > > AFAICT Nehalem and later should be good with the 16-32 words. As would > all the various Zen chips. Have you got an accurate source for the setup costs for 'rep stos/movs' for the various architectures? I didn't find one last time I looked. Last time I measured rep movs I was using mfence to synchronise with rdpmc and that takes even longer - so you only get the per-byte costs. (You can use data-dependencies which have minimal overhead unless you go back to to some pre-zen amd cpu.) You get 16 bytes/clock until Haswell which will do 32 if rdi is aligned. Icelake might increase that to 64 bytes/clock. Agner has zen2 onwards at 32 bytes/clock. Agner also has zen3 and Icelake being able to do 2 writes per clock. If that is achievable for the writes being done (the loop isn't a problem) then it also takes a while for the 'rep stos' to catch up. (Then you have the cost of the I-cache misses ...) David ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-11-20 16:18 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif 2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif 2026-11-19 22:30 ` Nick Desaulniers 2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif 2026-11-19 22:28 ` Nick Desaulniers 2026-11-20 11:38 ` David Laight 2026-11-20 12:42 ` David Laight 2026-11-20 10:45 ` Peter Zijlstra 2026-11-20 12:26 ` Peter Zijlstra 2026-11-20 14:02 ` David Laight 2026-11-20 15:35 ` Peter Zijlstra 2026-11-20 16:18 ` David Laight
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox