[PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Martin Leitner-Ankerl]

x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from
<linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with
-mno-sse, and clang then drops __builtin_prefetch() without a warning,
while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8
prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc
16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it
needs ARCH_HAS_PREFETCH.

Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which
every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not
the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a
non-temporal prefetch, which would change what gcc builds emit today.

Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34
functions, e.g. __blk_mq_alloc_requests(), prepend_path(),
__alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2,
r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range()
adds 16 to hfi1's receive functions and 8 to scsi_debug, which a
defconfig does not build. Youling Tang proposed to define
ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].

KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang
22.1.8, 6 boots per kernel in turns, ns per call, median (range):

                    before                 after
  io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
  getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%

IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40
directories deep. Built with gcc, where both kernels have the same
prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP
and TCP over loopback, times() and clock_gettime() change no more than
that. Unfortunately the NIC receive paths need real hardware, so they
are not measured.

Fixes: ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
Link: https://lore.kernel.org/20240529032059.899347-2-youling.tang@linux.dev/ # [1]
Cc: Youling Tang <tangyouling@kylinos.cn>
Cc: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Martin Leitner-Ankerl <martin.ankerl@gmail.com>
---

Notes:
    Code generation, testing, compiling, analysis, measurements, etc. were
    all driven by an AI agent (Claude Code). I have reviewed the results and
    supervised the process.

    A few minutes ago this went to the x86 maintainers only, without the
    lists, by mistake. Sorry for the duplicate.

 arch/x86/include/asm/processor.h | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c..64748b8705e3 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -624,15 +624,25 @@ extern int			bootloader_version;
 extern char			ignore_fpu_irq;
 
 #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
+#define ARCH_HAS_PREFETCH
 #define ARCH_HAS_PREFETCHW
 
 #ifdef CONFIG_X86_32
 # define BASE_PREFETCH		""
-# define ARCH_HAS_PREFETCH
 #else
 # define BASE_PREFETCH		"prefetcht0 %1"
 #endif
 
+#ifdef CONFIG_X86_64
+/*
+ * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
+ * builds with -mno-sse, and clang then drops the builtin without a warning.
+ */
+static inline void prefetch(const void *x)
+{
+	asm("prefetcht0 %0" : : "m" (*(const char *)x));
+}
+#else
 /*
  * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
  *
@@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
 			  X86_FEATURE_XMM,
 			  "m" (*(const char *)x));
 }
+#endif /* CONFIG_X86_64 */
 
 /*
  * 3dnow prefetch to get an exclusive cache line.

base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc
-- 
2.56.0

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Nick Desaulniers]

On Mon, Nov 2, 2026 at 10:08 PM Martin Leitner-Ankerl
<martin.ankerl@gmail.com> wrote:
>
> x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from
> <linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with
> -mno-sse, and clang then drops __builtin_prefetch() without a warning,
> while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8
> prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc
> 16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it
> needs ARCH_HAS_PREFETCH.
>
> Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which
> every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not
> the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a
> non-temporal prefetch, which would change what gcc builds emit today.
>
> Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34
> functions, e.g. __blk_mq_alloc_requests(), prepend_path(),
> __alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2,
> r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range()
> adds 16 to hfi1's receive functions and 8 to scsi_debug, which a
> defconfig does not build. Youling Tang proposed to define
> ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].
>
> KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang
> 22.1.8, 6 boots per kernel in turns, ns per call, median (range):
>
>                     before                 after
>   io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
>   getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%
>
> IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40
> directories deep. Built with gcc, where both kernels have the same
> prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP
> and TCP over loopback, times() and clock_gettime() change no more than
> that. Unfortunately the NIC receive paths need real hardware, so they
> are not measured.
>
> Fixes: ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
> Link: https://lore.kernel.org/20240529032059.899347-2-youling.tang@linux.dev/ # [1]
> Cc: Youling Tang <tangyouling@kylinos.cn>
> Cc: Nikolay Borisov <nik.borisov@suse.com>
> Signed-off-by: Martin Leitner-Ankerl <martin.ankerl@gmail.com>
> ---
>
> Notes:
>     Code generation, testing, compiling, analysis, measurements, etc. were
>     all driven by an AI agent (Claude Code). I have reviewed the results and
>     supervised the process.
>
>     A few minutes ago this went to the x86 maintainers only, without the
>     lists, by mistake. Sorry for the duplicate.
>
>  arch/x86/include/asm/processor.h | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index 77488ff5eb8c..64748b8705e3 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -624,15 +624,25 @@ extern int                        bootloader_version;
>  extern char                    ignore_fpu_irq;
>
>  #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
> +#define ARCH_HAS_PREFETCH
>  #define ARCH_HAS_PREFETCHW
>
>  #ifdef CONFIG_X86_32
>  # define BASE_PREFETCH         ""
> -# define ARCH_HAS_PREFETCH
>  #else
>  # define BASE_PREFETCH         "prefetcht0 %1"
>  #endif
>
> +#ifdef CONFIG_X86_64
> +/*
> + * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
> + * builds with -mno-sse, and clang then drops the builtin without a warning.
> + */
> +static inline void prefetch(const void *x)
> +{
> +       asm("prefetcht0 %0" : : "m" (*(const char *)x));

Thanks for the patch!
Is there a reason to have the cast vs simply use "r" constraint and
deref in the inline asm?
asm("prefetcht0 (%0)" : : "r" (x));
https://godbolt.org/z/xezh15oMe

Either way:
Acked-by: Nick Desaulniers <ndesaulniers@google.com>

> +}
> +#else
>  /*
>   * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
>   *
> @@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
>                           X86_FEATURE_XMM,
>                           "m" (*(const char *)x));
>  }
> +#endif /* CONFIG_X86_64 */
>
>  /*
>   * 3dnow prefetch to get an exclusive cache line.
>
> base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc
> --
> 2.56.0
>


-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Martin Leitner-Ankerl]

> Thanks for the patch!
> Is there a reason to have the cast vs simply use "r" constraint and
> deref in the inline asm?
> asm("prefetcht0 (%0)" : : "r" (x));
> https://godbolt.org/z/xezh15oMe

I mostly did it to be consistent with the 32-bit prefetch() and prefetchw()
in the same file, but it also has a codegen benefit. With "m" the compiler
can put an offset or index in the prefetcht0 itself, while with "r" it needs
to compute the address into a register first:
https://godbolt.org/z/j311eYzsY

Thanks for the ack!
Martin

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Nick Desaulniers]

On Tue, Nov 3, 2026 at 10:49 PM Martin Leitner-Ankerl
<martin.ankerl@gmail.com> wrote:
>
> > Thanks for the patch!
> > Is there a reason to have the cast vs simply use "r" constraint and
> > deref in the inline asm?
> > asm("prefetcht0 (%0)" : : "r" (x));
> > https://godbolt.org/z/xezh15oMe
>
> I mostly did it to be consistent with the 32-bit prefetch() and prefetchw()
> in the same file, but it also has a codegen benefit. With "m" the compiler
> can put an offset or index in the prefetcht0 itself, while with "r" it needs
> to compute the address into a register first:
> https://godbolt.org/z/j311eYzsY

Interesting! I learn more about inline asm everyday...

>
> Thanks for the ack!
> Martin



-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Andi Kleen]

> +#ifdef CONFIG_X86_64
> +/*
> + * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
> + * builds with -mno-sse, and clang then drops the builtin without a warning.
> + */
> +static inline void prefetch(const void *x)
> +{
> +	asm("prefetcht0 %0" : : "m" (*(const char *)x));
> +}
> +#else

One thing you should check if the inline assembly pessimizes the code on
gcc. Normally builtins can be better optimized. Or keep using the builtin
on gcc to be safe.

-Andi

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Martin Leitner-Ankerl]

> One thing you should check if the inline assembly pessimizes the code on
> gcc. Normally builtins can be better optimized. Or keep using the builtin
> on gcc to be safe.

I built an x86-64 defconfig vmlinux with and without the patch with
gcc 16.2.1, and
compared them function by function. The .text size is exactly the same, both
have exactly 62 prefetcht0 in the binary, and instruction count differs by just
one: 7365619 without the patch, 7365620 with it

23 functions of 130789 actually differ: 21 only in the order of instructions,
where prefetcht0 comes mostly 1-4 instructions earlier, and tcp_ack() and
e1000_clean_rx_irq_ps() differ in register allocation, the latter has
an extra mov.

I timed system calls that reach 8 of the 23 functions in a KVM guest with one
CPU, 10 boots per kernel. The differences are very small and within
the noise, in
range from -0.7% to 0.0%. So the asm might be slightly faster, but
it's too close to
say for sure.

So I'd keep one prefetch() for both compilers. The other 15 functions
that differ
are mostly NIC receive paths, which would require real hardware to measure.

Martin