From: David Laight <david.laight.linux@gmail.com>
To: Nick Desaulniers <ndesaulniers@google.com>
Cc: Usama Arif <usama.arif@linux.dev>,
tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org,
hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
hca@linux.ibm.com, nathan@kernel.org, morbo@google.com,
justinstitt@google.com, linux-hardening@vger.kernel.org,
llvm@lists.linux.dev, linux-kernel@vger.kernel.org,
riel@surriel.com, shakeel.butt@linux.dev, kernel-team@meta.com,
rkeilty@meta.com, atish.patra@linux.dev, dgreid@meta.com
Subject: Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Fri, 20 Nov 2026 11:38:25 +0000 [thread overview]
Message-ID: <20261120113825.234bee1c@pumpkin> (raw)
In-Reply-To: <CAKwvOdnzNU_xtLeb4vXSFe3DLN9QmdNSSReUmQahsLX+1pZxCw@mail.gmail.com>
On Thu, 19 Nov 2026 14:28:04 -0800
Nick Desaulniers <ndesaulniers@google.com> wrote:
> On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:
> >
> > With KSTACK_ERASE, every return to user space poisons the part of the
> > task stack that was used since the previous return. The generic
> > __stackleak_poison() does that one word per loop iteration, so a syscall
> > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> > running many small VMs, stackleak_erase() accounts for about 1.5% of
> > non-idle host kernel cycles.
> >
> > Override __stackleak_poison() on x86-64, as s390 already does, through
> > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> > specific stackleak poison function"). Fill spans larger than 512 bytes
> > with REP STOSQ and keep the word loop for shorter ones, where the fixed
> > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> > slower.
> >
> > The fill is inline asm, so the noinstr function still makes no calls.
> > memset64() would be the same instruction, but it is only static inline,
> > and with KMSAN it is an out-of-line function.
> >
> > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> >
> > Signed-off-by: Usama Arif <usama.arif@linux.dev>
> > ---
> > arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> > 1 file changed, 25 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> > index 77488ff5eb8c7c..049231d484ad0e 100644
> > --- a/arch/x86/include/asm/processor.h
> > +++ b/arch/x86/include/asm/processor.h
> > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> > current_stack_pointer) < THREAD_SIZE;
> > }
> >
> > +#ifdef CONFIG_X86_64
> > +#define __stackleak_poison __stackleak_poison
> > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > + unsigned long erase_high,
> > + unsigned long poison)
> > +{
> > + unsigned long count;
> > +
> > + /* Short spans do not amortize the startup cost of REP STOSQ. */
> > + if (erase_high <= erase_low + 512) {
> > + while (erase_low < erase_high) {
> > + *(unsigned long *)erase_low = poison;
> > + erase_low += sizeof(unsigned long);
> > + }
> > + return;
gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
It does manage to unroll the loop once - which is good because the intel
cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
setup code is horrid.
But I also think some of the more recent cpu can execute more than
one write per clock.
If you unroll the loop 4 times it may run a lot faster.
Perhaps:
long count = (erase_high - erase_low) / sizeof(unsigned long);
if (count <= 64) {
while ((count -= 4) >= 0) {
*(unsigned long *)erase_low = poison;
*(unsigned long *)(erase_low + 8) = poison;
*(unsigned long *)(erase_low + 16) = poison;
*(unsigned long *)(erase_low + 24) = poison;
erase_low += 32;
}
if (count & 2) {
*(unsigned long *)erase_low = poison;
*(unsigned long *)(erase_low + 8) = poison;
erase_low += 16;
}
if (count & 1)
*(unsigned long *)erase_low = poison;
} else {
...
}
Except that gcc makes a mess of it again.
I don't think the 'erase_low += 32' is going to execute at the
same time as the last (two probably) writes so costs a clock.
So it may need to be asm.
> > + }
> > +
> > + count = (erase_high - erase_low) / sizeof(unsigned long);
> > + asm volatile("rep stosq"
> > + : "+D" (erase_low), "+c" (count)
> > + : "a" (poison)
> > + : "memory");
>
> Surely this isn't the only place using `rep stosq` in the tree, right?
> It's not, see memset64 in arch/x86/include/asm/string_64.h.
Which seems to have very few users.
memset_p() and memset_l() have almost no users.
David
>
> > +}
> > +#endif
> > +
> > #ifdef CONFIG_PARAVIRT_XXL
> > #include <asm/paravirt.h>
> > #else
> > --
> > 2.53.1-Meta
> >
>
>
next prev parent reply other threads:[~2026-11-20 11:38 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30 ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28 ` Nick Desaulniers
2026-11-20 11:38 ` David Laight [this message]
2026-11-20 10:45 ` Peter Zijlstra
2026-11-20 12:26 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120113825.234bee1c@pumpkin \
--to=david.laight.linux@gmail.com \
--cc=ardb@kernel.org \
--cc=atish.patra@linux.dev \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=dgreid@meta.com \
--cc=gustavoars@kernel.org \
--cc=hca@linux.ibm.com \
--cc=hpa@zytor.com \
--cc=justinstitt@google.com \
--cc=kees@kernel.org \
--cc=kernel-team@meta.com \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=llvm@lists.linux.dev \
--cc=mingo@redhat.com \
--cc=morbo@google.com \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=riel@surriel.com \
--cc=rkeilty@meta.com \
--cc=shakeel.butt@linux.dev \
--cc=tglx@kernel.org \
--cc=usama.arif@linux.dev \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox