Building the Linux kernel with Clang and LLVM
 help / color / mirror / Atom feed
From: Usama Arif <usama.arif@linux.dev>
To: David Laight <david.laight.linux@gmail.com>
Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org,
	hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
	hca@linux.ibm.com, nathan@kernel.org, ndesaulniers@google.com,
	morbo@google.com, justinstitt@google.com,
	linux-hardening@vger.kernel.org, llvm@lists.linux.dev,
	linux-kernel@vger.kernel.org, riel@surriel.com,
	shakeel.butt@linux.dev, kernel-team@meta.com, rkeilty@meta.com,
	atish.patra@linux.dev, dgreid@meta.com
Subject: Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Sat, 21 Nov 2026 13:57:39 +0000	[thread overview]
Message-ID: <ee4847f9-53ba-4f0a-9297-7ca8b6408f7f@linux.dev> (raw)
In-Reply-To: <20261120185513.0445f0b1@pumpkin>



On 20/11/2026 10:55, David Laight wrote:
> On Thu, 19 Nov 2026 13:31:08 -0800
> Usama Arif <usama.arif@linux.dev> wrote:
> 
>> With KSTACK_ERASE, every return to user space poisons the part of the
>> task stack that was used since the previous return. The generic
>> __stackleak_poison() does that one word per loop iteration, so a syscall
>> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
>> running many small VMs, stackleak_erase() accounts for about 1.5% of
>> non-idle host kernel cycles.
>>
>> Override __stackleak_poison() on x86-64, as s390 already does, through
>> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
>> specific stackleak poison function"). Fill spans larger than 512 bytes
>> with REP STOSQ and keep the word loop for shorter ones, where the fixed
>> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
>> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
>> slower.
> 
> As a matter of interest (and separately from the exact code) how did you
> measure the performance to get the 300-400 byte gain point?
> 

With a userspace benchmark pinned to one CPU. Each sample is one fill of
a stack buffer straight after a getppid() syscall, with
lfence;rdtsc;lfence around the fill only, an empty call subtracted, and
the mean of 15 times taken. The size stayed fixed for a whole run,
so the branches were well predicted. The plain loop and REP STOSQ
cost the same at about 430 bytes on Zen 5c and 320 bytes on
Cooper Lake.

> If the count is variable - likely in real life, you'll get a branch
> misprediction penalty (about 20 clocks on zen5) every time you change
> between the two choices.
> The copy loop will have another mispredict penalty for the loop itself.
> 
> So you are likely to get an overall improvement if you use 'rep stos'
> for all lengths - even the ones for with it is slightly slower.
> Unless, that is, you know at compile time that most of the length
> will be short.


I timed every stackleak_erase() in a Zen 5c guest running a
futex, epoll, procfs and unix socket mix, with the variants switchable
at runtime and alternated over 15 rounds:

  Unpatched                     68.3 ns per erase
  REP STOSQ for all lengths     47.3 ns
  v1 / v2.1 (loop up to 512 B)  46.6 ns
  v2.2 (4 stores/iteration)     43.7 ns

The spans aren't random. Unix socket and /proc/self/stat exits were over
1 KiB every time, while getppid(), epoll_wait(), read() and write() exits
were under 128 bytes almost everytime.

My preference still is v2.1 because of the simplicity.

> 
> David
> 
>>
>> The fill is inline asm, so the noinstr function still makes no calls.
>> memset64() would be the same instruction, but it is only static inline,
>> and with KMSAN it is an out-of-line function.
>>
>> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
>> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
>> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
>> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
>>
>> Signed-off-by: Usama Arif <usama.arif@linux.dev>
>> ---
>>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
>>  1 file changed, 25 insertions(+)
>>
>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
>> index 77488ff5eb8c7c..049231d484ad0e 100644
>> --- a/arch/x86/include/asm/processor.h
>> +++ b/arch/x86/include/asm/processor.h
>> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
>>  			       current_stack_pointer) < THREAD_SIZE;
>>  }
>>  
>> +#ifdef CONFIG_X86_64
>> +#define __stackleak_poison __stackleak_poison
>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>> +					       unsigned long erase_high,
>> +					       unsigned long poison)
>> +{
>> +	unsigned long count;
>> +
>> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
>> +	if (erase_high <= erase_low + 512) {
>> +		while (erase_low < erase_high) {
>> +			*(unsigned long *)erase_low = poison;
>> +			erase_low += sizeof(unsigned long);
>> +		}
>> +		return;
>> +	}
>> +
>> +	count = (erase_high - erase_low) / sizeof(unsigned long);
>> +	asm volatile("rep stosq"
>> +		     : "+D" (erase_low), "+c" (count)
>> +		     : "a" (poison)
>> +		     : "memory");
>> +}
>> +#endif
>> +
>>  #ifdef CONFIG_PARAVIRT_XXL
>>  #include <asm/paravirt.h>
>>  #else
> 


      reply	other threads:[~2026-11-21 13:57 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30   ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28   ` Nick Desaulniers
2026-11-20 11:38     ` David Laight
2026-11-20 12:42       ` David Laight
2026-11-20 18:46         ` Usama Arif
2026-11-20 21:44           ` David Laight
2026-11-20 10:45   ` Peter Zijlstra
2026-11-20 12:26     ` Peter Zijlstra
2026-11-20 18:43       ` Usama Arif
2026-11-20 22:22         ` David Laight
2026-11-21 14:38           ` Usama Arif
2026-11-20 14:02     ` David Laight
2026-11-20 15:35       ` Peter Zijlstra
2026-11-20 16:18         ` David Laight
2026-11-20 18:55   ` David Laight
2026-11-21 13:57     ` Usama Arif [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ee4847f9-53ba-4f0a-9297-7ca8b6408f7f@linux.dev \
    --to=usama.arif@linux.dev \
    --cc=ardb@kernel.org \
    --cc=atish.patra@linux.dev \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=david.laight.linux@gmail.com \
    --cc=dgreid@meta.com \
    --cc=gustavoars@kernel.org \
    --cc=hca@linux.ibm.com \
    --cc=hpa@zytor.com \
    --cc=justinstitt@google.com \
    --cc=kees@kernel.org \
    --cc=kernel-team@meta.com \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=llvm@lists.linux.dev \
    --cc=mingo@redhat.com \
    --cc=morbo@google.com \
    --cc=nathan@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=riel@surriel.com \
    --cc=rkeilty@meta.com \
    --cc=shakeel.butt@linux.dev \
    --cc=tglx@kernel.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox