From: Usama Arif <usama.arif@linux.dev>
To: tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org
Cc: hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
hca@linux.ibm.com, nathan@kernel.org, ndesaulniers@google.com,
morbo@google.com, justinstitt@google.com,
linux-hardening@vger.kernel.org, llvm@lists.linux.dev,
linux-kernel@vger.kernel.org, riel@surriel.com,
shakeel.butt@linux.dev, kernel-team@meta.com, rkeilty@meta.com,
atish.patra@linux.dev, dgreid@meta.com,
Usama Arif <usama.arif@linux.dev>
Subject: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Thu, 19 Nov 2026 13:31:08 -0800 [thread overview]
Message-ID: <20261119213147.1094488-3-usama.arif@linux.dev> (raw)
In-Reply-To: <20261119213147.1094488-1-usama.arif@linux.dev>
With KSTACK_ERASE, every return to user space poisons the part of the
task stack that was used since the previous return. The generic
__stackleak_poison() does that one word per loop iteration, so a syscall
that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
running many small VMs, stackleak_erase() accounts for about 1.5% of
non-idle host kernel cycles.
Override __stackleak_poison() on x86-64, as s390 already does, through
the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
specific stackleak poison function"). Fill spans larger than 512 bytes
with REP STOSQ and keep the word loop for shorter ones, where the fixed
startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
slower.
The fill is inline asm, so the noinstr function still makes no calls.
memset64() would be the same instruction, but it is only static inline,
and with KMSAN it is an out-of-line function.
In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c7c..049231d484ad0e 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
current_stack_pointer) < THREAD_SIZE;
}
+#ifdef CONFIG_X86_64
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+ unsigned long erase_high,
+ unsigned long poison)
+{
+ unsigned long count;
+
+ /* Short spans do not amortize the startup cost of REP STOSQ. */
+ if (erase_high <= erase_low + 512) {
+ while (erase_low < erase_high) {
+ *(unsigned long *)erase_low = poison;
+ erase_low += sizeof(unsigned long);
+ }
+ return;
+ }
+
+ count = (erase_high - erase_low) / sizeof(unsigned long);
+ asm volatile("rep stosq"
+ : "+D" (erase_low), "+c" (count)
+ : "a" (poison)
+ : "memory");
+}
+#endif
+
#ifdef CONFIG_PARAVIRT_XXL
#include <asm/paravirt.h>
#else
--
2.53.1-Meta
prev parent reply other threads:[~2026-11-19 21:32 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 21:31 ` Usama Arif [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119213147.1094488-3-usama.arif@linux.dev \
--to=usama.arif@linux.dev \
--cc=ardb@kernel.org \
--cc=atish.patra@linux.dev \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=dgreid@meta.com \
--cc=gustavoars@kernel.org \
--cc=hca@linux.ibm.com \
--cc=hpa@zytor.com \
--cc=justinstitt@google.com \
--cc=kees@kernel.org \
--cc=kernel-team@meta.com \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=llvm@lists.linux.dev \
--cc=mingo@redhat.com \
--cc=morbo@google.com \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=riel@surriel.com \
--cc=rkeilty@meta.com \
--cc=shakeel.butt@linux.dev \
--cc=tglx@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox