[PATCH][V2] fortify: add branch hints on unlikely fortify_panic paths

[PATCH][V2] fortify: add branch hints on unlikely fortify_panic paths

[Colin Ian King]

Analysis with gcov while running the stress-ng urandom stressor
shows that there are a couple of fortify panic paths that are highly
unlikely to be executed for well-behaving code. Adding appropriate
branch hints improves the stress-ng urandom stressor my a small
but statistically measureable amount. Ran 100 x 1 minute tests and
measured the stressor bogo-op rates on a Debian based Intel(R)
Core(TM) Ultra 9 285K with a 6.15 kernel with turbo disabled to
reduce jitter.

Results based on a Geometic Mean of 100 tests:

Without patch: 50512.95 bogo-ops/sec
With patch:    50819.58 bogo-ops/sec

%Std.Deviation of ~0.18%, so low jitter in results, improvement of ~0.6%

Branch hints can only be enabled if CONFIG_TRACE_BRANCH_PROFILING due
to a static variable being declared in __branch_check__ when using
trace branch profiling in the unlikely macro causing build issues.

Signed-off-by: Colin Ian King <colin.i.king@gmail.com>
---
V2: add fortify_unlikely macro wrapper for CONFIG_TRACE_BRANCH_PROFILING
    build config.
---
 include/linux/fortify-string.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h
index e4ce1cae03bf..c740114bcbf8 100644
--- a/include/linux/fortify-string.h
+++ b/include/linux/fortify-string.h
@@ -10,6 +10,12 @@
 #define __FORTIFY_INLINE extern __always_inline __gnu_inline __overloadable
 #define __RENAME(x) __asm__(#x)
 
+#if defined(CONFIG_TRACE_BRANCH_PROFILING)
+#define fortify_unlikely(x)		(x)
+#else
+#define fortify_unlikely(x)		unlikely(x)
+#endif
+
 #define FORTIFY_REASON_DIR(r)		FIELD_GET(BIT(0), r)
 #define FORTIFY_REASON_FUNC(r)		FIELD_GET(GENMASK(7, 1), r)
 #define FORTIFY_REASON(func, write)	(FIELD_PREP(BIT(0), write) | \
@@ -593,9 +599,9 @@ __FORTIFY_INLINE bool fortify_memcpy_chk(__kernel_size_t size,
 	 * (The SIZE_MAX test is to optimize away checks where the buffer
 	 * lengths are unknown.)
 	 */
-	if (p_size != SIZE_MAX && p_size < size)
+	if (fortify_unlikely(p_size != SIZE_MAX && p_size < size))
 		fortify_panic(func, FORTIFY_WRITE, p_size, size, true);
-	else if (q_size != SIZE_MAX && q_size < size)
+	else if (fortify_unlikely(q_size != SIZE_MAX && q_size < size))
 		fortify_panic(func, FORTIFY_READ, p_size, size, true);
 
 	/*
-- 
2.50.0

Re: [PATCH][V2] fortify: add branch hints on unlikely fortify_panic paths

[Alexander Lobakin]

From: Colin Ian King <colin.i.king@gmail.com>
Date: Mon,  7 Jul 2025 14:23:40 +0100

> Analysis with gcov while running the stress-ng urandom stressor
> shows that there are a couple of fortify panic paths that are highly
> unlikely to be executed for well-behaving code. Adding appropriate
> branch hints improves the stress-ng urandom stressor my a small
> but statistically measureable amount. Ran 100 x 1 minute tests and
> measured the stressor bogo-op rates on a Debian based Intel(R)
> Core(TM) Ultra 9 285K with a 6.15 kernel with turbo disabled to
> reduce jitter.
> 
> Results based on a Geometic Mean of 100 tests:
> 
> Without patch: 50512.95 bogo-ops/sec
> With patch:    50819.58 bogo-ops/sec
> 
> %Std.Deviation of ~0.18%, so low jitter in results, improvement of ~0.6%
> 
> Branch hints can only be enabled if CONFIG_TRACE_BRANCH_PROFILING due
> to a static variable being declared in __branch_check__ when using
> trace branch profiling in the unlikely macro causing build issues.
> 
> Signed-off-by: Colin Ian King <colin.i.king@gmail.com>
> ---
> V2: add fortify_unlikely macro wrapper for CONFIG_TRACE_BRANCH_PROFILING
>     build config.
> ---
>  include/linux/fortify-string.h | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h
> index e4ce1cae03bf..c740114bcbf8 100644
> --- a/include/linux/fortify-string.h
> +++ b/include/linux/fortify-string.h
> @@ -10,6 +10,12 @@
>  #define __FORTIFY_INLINE extern __always_inline __gnu_inline __overloadable
>  #define __RENAME(x) __asm__(#x)
>  
> +#if defined(CONFIG_TRACE_BRANCH_PROFILING)
> +#define fortify_unlikely(x)		(x)
> +#else
> +#define fortify_unlikely(x)		unlikely(x)
> +#endif
> +
>  #define FORTIFY_REASON_DIR(r)		FIELD_GET(BIT(0), r)
>  #define FORTIFY_REASON_FUNC(r)		FIELD_GET(GENMASK(7, 1), r)
>  #define FORTIFY_REASON(func, write)	(FIELD_PREP(BIT(0), write) | \
> @@ -593,9 +599,9 @@ __FORTIFY_INLINE bool fortify_memcpy_chk(__kernel_size_t size,

Why only memcpy()?

>  	 * (The SIZE_MAX test is to optimize away checks where the buffer
>  	 * lengths are unknown.)
>  	 */
> -	if (p_size != SIZE_MAX && p_size < size)
> +	if (fortify_unlikely(p_size != SIZE_MAX && p_size < size))
>  		fortify_panic(func, FORTIFY_WRITE, p_size, size, true);
> -	else if (q_size != SIZE_MAX && q_size < size)
> +	else if (fortify_unlikely(q_size != SIZE_MAX && q_size < size))
>  		fortify_panic(func, FORTIFY_READ, p_size, size, true);
>  
>  	/*

Thanks,
Olek