All of lore.kernel.org
 help / color / mirror / Atom feed
From: Usama Arif <usama.arif@linux.dev>
To: David Laight <david.laight.linux@gmail.com>
Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org,
	hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
	hca@linux.ibm.com, nathan@kernel.org, ndesaulniers@google.com,
	morbo@google.com, justinstitt@google.com,
	linux-hardening@vger.kernel.org, llvm@lists.linux.dev,
	linux-kernel@vger.kernel.org, riel@surriel.com,
	shakeel.butt@linux.dev, kernel-team@meta.com, rkeilty@meta.com,
	atish.patra@linux.dev, dgreid@meta.com
Subject: Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Sat, 21 Nov 2026 13:57:39 +0000	[thread overview]
Message-ID: <ee4847f9-53ba-4f0a-9297-7ca8b6408f7f@linux.dev> (raw)
In-Reply-To: <20261120185513.0445f0b1@pumpkin>



On 20/11/2026 10:55, David Laight wrote:
> On Thu, 19 Nov 2026 13:31:08 -0800
> Usama Arif <usama.arif@linux.dev> wrote:
> 
>> With KSTACK_ERASE, every return to user space poisons the part of the
>> task stack that was used since the previous return. The generic
>> __stackleak_poison() does that one word per loop iteration, so a syscall
>> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
>> running many small VMs, stackleak_erase() accounts for about 1.5% of
>> non-idle host kernel cycles.
>>
>> Override __stackleak_poison() on x86-64, as s390 already does, through
>> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
>> specific stackleak poison function"). Fill spans larger than 512 bytes
>> with REP STOSQ and keep the word loop for shorter ones, where the fixed
>> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
>> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
>> slower.
> 
> As a matter of interest (and separately from the exact code) how did you
> measure the performance to get the 300-400 byte gain point?
> 

With a userspace benchmark pinned to one CPU. Each sample is one fill of
a stack buffer straight after a getppid() syscall, with
lfence;rdtsc;lfence around the fill only, an empty call subtracted, and
the mean of 15 times taken. The size stayed fixed for a whole run,
so the branches were well predicted. The plain loop and REP STOSQ
cost the same at about 430 bytes on Zen 5c and 320 bytes on
Cooper Lake.

> If the count is variable - likely in real life, you'll get a branch
> misprediction penalty (about 20 clocks on zen5) every time you change
> between the two choices.
> The copy loop will have another mispredict penalty for the loop itself.
> 
> So you are likely to get an overall improvement if you use 'rep stos'
> for all lengths - even the ones for with it is slightly slower.
> Unless, that is, you know at compile time that most of the length
> will be short.


I timed every stackleak_erase() in a Zen 5c guest running a
futex, epoll, procfs and unix socket mix, with the variants switchable
at runtime and alternated over 15 rounds:

  Unpatched                     68.3 ns per erase
  REP STOSQ for all lengths     47.3 ns
  v1 / v2.1 (loop up to 512 B)  46.6 ns
  v2.2 (4 stores/iteration)     43.7 ns

The spans aren't random. Unix socket and /proc/self/stat exits were over
1 KiB every time, while getppid(), epoll_wait(), read() and write() exits
were under 128 bytes almost everytime.

My preference still is v2.1 because of the simplicity.

> 
> David
> 
>>
>> The fill is inline asm, so the noinstr function still makes no calls.
>> memset64() would be the same instruction, but it is only static inline,
>> and with KMSAN it is an out-of-line function.
>>
>> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
>> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
>> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
>> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
>>
>> Signed-off-by: Usama Arif <usama.arif@linux.dev>
>> ---
>>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
>>  1 file changed, 25 insertions(+)
>>
>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
>> index 77488ff5eb8c7c..049231d484ad0e 100644
>> --- a/arch/x86/include/asm/processor.h
>> +++ b/arch/x86/include/asm/processor.h
>> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
>>  			       current_stack_pointer) < THREAD_SIZE;
>>  }
>>  
>> +#ifdef CONFIG_X86_64
>> +#define __stackleak_poison __stackleak_poison
>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>> +					       unsigned long erase_high,
>> +					       unsigned long poison)
>> +{
>> +	unsigned long count;
>> +
>> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
>> +	if (erase_high <= erase_low + 512) {
>> +		while (erase_low < erase_high) {
>> +			*(unsigned long *)erase_low = poison;
>> +			erase_low += sizeof(unsigned long);
>> +		}
>> +		return;
>> +	}
>> +
>> +	count = (erase_high - erase_low) / sizeof(unsigned long);
>> +	asm volatile("rep stosq"
>> +		     : "+D" (erase_low), "+c" (count)
>> +		     : "a" (poison)
>> +		     : "memory");
>> +}
>> +#endif
>> +
>>  #ifdef CONFIG_PARAVIRT_XXL
>>  #include <asm/paravirt.h>
>>  #else
> 


      reply	other threads:[~2026-11-21 13:57 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30   ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28   ` Nick Desaulniers
2026-11-20 11:38     ` David Laight
2026-11-20 12:42       ` David Laight
2026-11-20 18:46         ` Usama Arif
2026-11-20 21:44           ` David Laight
2026-11-20 10:45   ` Peter Zijlstra
2026-11-20 12:26     ` Peter Zijlstra
2026-11-20 18:43       ` Usama Arif
2026-11-20 22:22         ` David Laight
2026-11-20 14:02     ` David Laight
2026-11-20 15:35       ` Peter Zijlstra
2026-11-20 16:18         ` David Laight
2026-11-20 18:55   ` David Laight
2026-11-21 13:57     ` Usama Arif [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ee4847f9-53ba-4f0a-9297-7ca8b6408f7f@linux.dev \
    --to=usama.arif@linux.dev \
    --cc=ardb@kernel.org \
    --cc=atish.patra@linux.dev \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=david.laight.linux@gmail.com \
    --cc=dgreid@meta.com \
    --cc=gustavoars@kernel.org \
    --cc=hca@linux.ibm.com \
    --cc=hpa@zytor.com \
    --cc=justinstitt@google.com \
    --cc=kees@kernel.org \
    --cc=kernel-team@meta.com \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=llvm@lists.linux.dev \
    --cc=mingo@redhat.com \
    --cc=morbo@google.com \
    --cc=nathan@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=riel@surriel.com \
    --cc=rkeilty@meta.com \
    --cc=shakeel.butt@linux.dev \
    --cc=tglx@kernel.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.