The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: David Laight <david.laight.linux@gmail.com>
To: Usama Arif <usama.arif@linux.dev>
Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org,
	hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
	hca@linux.ibm.com, nathan@kernel.org, ndesaulniers@google.com,
	morbo@google.com, justinstitt@google.com,
	linux-hardening@vger.kernel.org, llvm@lists.linux.dev,
	linux-kernel@vger.kernel.org, riel@surriel.com,
	shakeel.butt@linux.dev, kernel-team@meta.com, rkeilty@meta.com,
	atish.patra@linux.dev, dgreid@meta.com
Subject: Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Sat, 21 Nov 2026 18:22:08 +0000	[thread overview]
Message-ID: <20261121182208.19594445@pumpkin> (raw)
In-Reply-To: <ee4847f9-53ba-4f0a-9297-7ca8b6408f7f@linux.dev>

On Sat, 21 Nov 2026 13:57:39 +0000
Usama Arif <usama.arif@linux.dev> wrote:

> On 20/11/2026 10:55, David Laight wrote:
> > On Thu, 19 Nov 2026 13:31:08 -0800
> > Usama Arif <usama.arif@linux.dev> wrote:
> >   
> >> With KSTACK_ERASE, every return to user space poisons the part of the
> >> task stack that was used since the previous return. The generic
> >> __stackleak_poison() does that one word per loop iteration, so a syscall
> >> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> >> running many small VMs, stackleak_erase() accounts for about 1.5% of
> >> non-idle host kernel cycles.
> >>
> >> Override __stackleak_poison() on x86-64, as s390 already does, through
> >> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> >> specific stackleak poison function"). Fill spans larger than 512 bytes
> >> with REP STOSQ and keep the word loop for shorter ones, where the fixed
> >> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> >> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> >> slower.  
> > 
> > As a matter of interest (and separately from the exact code) how did you
> > measure the performance to get the 300-400 byte gain point?
> >   
> 
> With a userspace benchmark pinned to one CPU. Each sample is one fill of
> a stack buffer straight after a getppid() syscall, with
> lfence;rdtsc;lfence around the fill only, an empty call subtracted, and
> the mean of 15 times taken. The size stayed fixed for a whole run,
> so the branches were well predicted. The plain loop and REP STOSQ
> cost the same at about 430 bytes on Zen 5c and 320 bytes on
> Cooper Lake.

The lfence (or mfence - which is the one you really need) take long
enough that they don't give meaningful timings for anything short.
I think they manage to overlap with the setup times for 'rep xxx'.
You also get the problem that the tsc doesn't measure clocks and a lot
of benchmarks run long enough that the cpu speed changes.

I've updated my benchmark code and run some tests on zen5.
This uses the PERF_COUNT_CPU_CYCLES counter but directly executing rdpmc.
I then use data dependencies from the original count to the test code,
and from the test result to the second rdpmc (which nicely has a register
number) to get a time that includes the test function with little overhead.
The clock count for an empty test function is 25, 32 with a memory read.

Each test is run 10 times and the clock count for each shown separately.
The first column includes the cost of the I-cache loads and any branches
that get mispredicted because (in these cases) the length changes.
This is most obvious for the libc calls.

memset() to 4k aligned buffer
0: len   16 ok  445   32   32   32   32   32   32   32   32   32 no_memset
5: len   16 ok 10538  107   52   52   52   52   52   52   52   52 libc
6: len  256 ok  193  101   48   48   51   44   51   44   44   44 libc
7: len  512 ok  120   83   51   51   51   51   51   51   51   51 libc
8: len 1024 ok  111   56   56   55   56   55   56   55   56   56 libc
9: len 2048 ok  117   71   72   72   72   72   72   72   72   72 libc
10: len   16 ok   90   78   74   55   54   55   54   55   54   55 stosb
11: len  256 ok  112   78   78   78   78   78   78   78   78   78 stosb
12: len  512 ok   80   80   80   80   80   80   80   87   80   80 stosb
13: len 1024 ok   90   95   89   95   89   88   89   95   89   95 stosb
14: len 2048 ok  105  111  104  104  104  111  104  111  104  111 stosb
15: len   16 ok  104   88   84   58   58   58   58   58   58   58 stosq
16: len  256 ok  125   80   80   80   80   80   80   80   80   80 stosq
17: len  512 ok   82   83   83   90   82   82   83   83   82   82 stosq
18: len 1024 ok   90   91   91   98   91   98   91   98   91   91 stosq
19: len 2048 ok  106  113  106  113  106  113  106  106  106  113 stosq
20: len   16 ok  133   65   57   33   33   26   26   26   26   26 longs
21: len  256 ok  159  152   80   42   42   42   42   42   42   42 longs
22: len  512 ok  187  115   88   58   56   57   57   57   57   57 longs
23: len 1024 ok  146  211  167   89   88   88   88   88   88   90 longs
24: len 2048 ok  213  213  209  167  153  155  153  153  155  152 longs

memcpy() between 4k aligned buffers
0: len   16 ok  119   63   59   33   33   33   33   33   33   33 no_copy
5: len   16 ok 7333  103   51   51   51   51   51   51   51   51 libc
6: len  256 ok  447  105   51   51   51   51   51   51   51   51 libc
7: len  512 ok  405   56   56   56   56   56   56   56   56   56 libc
8: len 1024 ok  421   96   66   65   64   66   64   70   64   70 libc
9: len 2048 ok  406  139   81   83   81   85   81   83   83   81 libc
10: len   16 ok   91   81   76   56   56   56   56   56   56   56 movsb
11: len  256 ok  115   77   76   77   76   77   76   77   76   77 movsb
12: len  512 ok   80   81   80   81   80   81   81   81   80   81 movsb
13: len 1024 ok   90   89   90   89   90   89   90   89   90   89 movsb
14: len 2048 ok  112  106  105  106  105  106  105  106  105  106 movsb
15: len   16 ok  108   87   85   65   57   65   57   65   57   65 movsq
16: len  256 ok  119   78   78   78   79   78   79   78   79   78 movsq
17: len  512 ok   81   83   82   83   82   82   82   83   82   82 movsq
18: len 1024 ok   91   91   91   91   91   91   91   91   91   91 movsq
19: len 2048 ok  107  107  107  107  107  107  107  107  107  107 movsq
20: len   16 ok  110   62   56   32   32   32   32   32   32   32 longs
21: len  256 ok  203  128   58   57   57   57   57   47   47   47 longs
22: len  512 ok   97  152  190   95   65   63   65   63   65   63 longs
23: len 1024 ok  226  185  155  100   99   98  100   98   99   98 longs
24: len 2048 ok  201  262  260  179  171  166  170  169  171  166 longs

I added enough barrier() and OPTIMIZER_HIDE_VAR() to get the 'longs'
loop to be sane - it sets/copies two 64bit locations each iteration.
Both tests manage a loop iteration every clock (on zen5).

The cutoff for 'rep stos' is pretty much 1k.
For memcpy() it is a bit lower.

'rep movsq' is slightly slower than 'rep movsb' because of the 'len /= 8'.
But it is much slower for counts of 1 (included above) and 0.

I need to boot up one of my other systems to see how older Intel cpu fair.

David

> 
> > If the count is variable - likely in real life, you'll get a branch
> > misprediction penalty (about 20 clocks on zen5) every time you change
> > between the two choices.
> > The copy loop will have another mispredict penalty for the loop itself.
> > 
> > So you are likely to get an overall improvement if you use 'rep stos'
> > for all lengths - even the ones for with it is slightly slower.
> > Unless, that is, you know at compile time that most of the length
> > will be short.  
> 
> 
> I timed every stackleak_erase() in a Zen 5c guest running a
> futex, epoll, procfs and unix socket mix, with the variants switchable
> at runtime and alternated over 15 rounds:
> 
>   Unpatched                     68.3 ns per erase
>   REP STOSQ for all lengths     47.3 ns
>   v1 / v2.1 (loop up to 512 B)  46.6 ns
>   v2.2 (4 stores/iteration)     43.7 ns
> 
> The spans aren't random. Unix socket and /proc/self/stat exits were over
> 1 KiB every time, while getppid(), epoll_wait(), read() and write() exits
> were under 128 bytes almost everytime.
> 
> My preference still is v2.1 because of the simplicity.
> 
> > 
> > David
> >   
> >>
> >> The fill is inline asm, so the noinstr function still makes no calls.
> >> memset64() would be the same instruction, but it is only static inline,
> >> and with KMSAN it is an out-of-line function.
> >>
> >> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> >> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> >> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> >> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> >>
> >> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> >> ---
> >>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> >>  1 file changed, 25 insertions(+)
> >>
> >> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> >> index 77488ff5eb8c7c..049231d484ad0e 100644
> >> --- a/arch/x86/include/asm/processor.h
> >> +++ b/arch/x86/include/asm/processor.h
> >> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> >>  			       current_stack_pointer) < THREAD_SIZE;
> >>  }
> >>  
> >> +#ifdef CONFIG_X86_64
> >> +#define __stackleak_poison __stackleak_poison
> >> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> >> +					       unsigned long erase_high,
> >> +					       unsigned long poison)
> >> +{
> >> +	unsigned long count;
> >> +
> >> +	/* Short spans do not amortize the startup cost of REP STOSQ. */
> >> +	if (erase_high <= erase_low + 512) {
> >> +		while (erase_low < erase_high) {
> >> +			*(unsigned long *)erase_low = poison;
> >> +			erase_low += sizeof(unsigned long);
> >> +		}
> >> +		return;
> >> +	}
> >> +
> >> +	count = (erase_high - erase_low) / sizeof(unsigned long);
> >> +	asm volatile("rep stosq"
> >> +		     : "+D" (erase_low), "+c" (count)
> >> +		     : "a" (poison)
> >> +		     : "memory");
> >> +}
> >> +#endif
> >> +
> >>  #ifdef CONFIG_PARAVIRT_XXL
> >>  #include <asm/paravirt.h>
> >>  #else  
> >   
> 


      reply	other threads:[~2026-11-21 18:22 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30   ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28   ` Nick Desaulniers
2026-11-20 11:38     ` David Laight
2026-11-20 12:42       ` David Laight
2026-11-20 18:46         ` Usama Arif
2026-11-20 21:44           ` David Laight
2026-11-20 10:45   ` Peter Zijlstra
2026-11-20 12:26     ` Peter Zijlstra
2026-11-20 18:43       ` Usama Arif
2026-11-20 22:22         ` David Laight
2026-11-21 14:38           ` Usama Arif
2026-11-20 14:02     ` David Laight
2026-11-20 15:35       ` Peter Zijlstra
2026-11-20 16:18         ` David Laight
2026-11-20 18:55   ` David Laight
2026-11-21 13:57     ` Usama Arif
2026-11-21 18:22       ` David Laight [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261121182208.19594445@pumpkin \
    --to=david.laight.linux@gmail.com \
    --cc=ardb@kernel.org \
    --cc=atish.patra@linux.dev \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=dgreid@meta.com \
    --cc=gustavoars@kernel.org \
    --cc=hca@linux.ibm.com \
    --cc=hpa@zytor.com \
    --cc=justinstitt@google.com \
    --cc=kees@kernel.org \
    --cc=kernel-team@meta.com \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=llvm@lists.linux.dev \
    --cc=mingo@redhat.com \
    --cc=morbo@google.com \
    --cc=nathan@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=riel@surriel.com \
    --cc=rkeilty@meta.com \
    --cc=shakeel.butt@linux.dev \
    --cc=tglx@kernel.org \
    --cc=usama.arif@linux.dev \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox