* [PATCH v8 1/7] typeinfo: Introduce KCFI typeinfo mangling API
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
2025-11-20 22:21 ` [PATCH v8 2/7] kcfi: Add core Kernel Control Flow Integrity infrastructure Kees Cook
` (5 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
To support the KCFI typeid and future type-based allocators, which need
to convert unique types into unique 32-bit values, add a mangling system
based on the Itanium C++ mangling ABI, adapted for C types. Introduce
__builtin_typeinfo_hash for the hash, and __builtin_typeinfo_name for
testing and debugging (to see the human-readable mangling form). Add
tests for typeinfo validation and error handling.
This ABI needs to match what is used by LLVM Rust (which matches the Clang
ABI) so that KCFI can work on mixed GCC with LLVM-Rust kernel builds.
Instead of inventing a new ABI, all use the existing Itanium C++ mangling
which matches KCFI's needs.
An important aspect of the C++ typeinfo behavior that is retained here
is that typedefs are treated as pass-through except when the underlying
type lacks a tag (i.e. anonymous struct, union, or enum). This provides a
distinction between those typedefs and typedefs used to provide _aliases_
(u8, uint16_t).
In the future, an additional "strict mode" builtin helper pair could
also be added to follow strict ISO C type equivalency instead of the
existing typeinfo used here, but that is out of scope for this patch.
gcc/ChangeLog:
* Makefile.in: Add kcfi-typeinfo.o.
* doc/extend.texi: Document typeinfo builtins.
* kcfi-typeinfo.h: New file, typeinfo mangling API.
* kcfi-typeinfo.cc: New file, implement typeinfo mangling.
gcc/c-family/ChangeLog:
* c-common.h (enum rid): Add typeinfo builtins.
* c-common.cc: Add typeinfo builtins.
gcc/c/ChangeLog:
* c-parser.cc (c_parser_get_builtin_type_arg): New function,
parse type.
(c_parser_postfix_expression): Add typeinfo builtins.
gcc/testsuite/ChangeLog:
* gcc.dg/builtin-typeinfo-errors.c: New test, validate bad
arguments are rejected.
* gcc.dg/builtin-typeinfo.c: New test, typeinfo mangling.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/doc/extend.texi | 95 ++++
.../gcc.dg/builtin-typeinfo-errors.c | 28 +
gcc/testsuite/gcc.dg/builtin-typeinfo.c | 350 ++++++++++++
gcc/Makefile.in | 1 +
gcc/c-family/c-common.h | 1 +
gcc/kcfi-typeinfo.h | 32 ++
gcc/c-family/c-common.cc | 2 +
gcc/c/c-parser.cc | 72 +++
gcc/kcfi-typeinfo.cc | 516 ++++++++++++++++++
9 files changed, 1097 insertions(+)
create mode 100644 gcc/testsuite/gcc.dg/builtin-typeinfo-errors.c
create mode 100644 gcc/testsuite/gcc.dg/builtin-typeinfo.c
create mode 100644 gcc/kcfi-typeinfo.h
create mode 100644 gcc/kcfi-typeinfo.cc
diff --git a/gcc/doc/extend.texi b/gcc/doc/extend.texi
index b9d1a6aac75a..c91b48bbdfe6 100644
--- a/gcc/doc/extend.texi
+++ b/gcc/doc/extend.texi
@@ -17925,6 +17925,101 @@ which will cause a @code{NULL} pointer to be used for the unsafe case.
@enddefbuiltin
+@defbuiltin{{unsigned int} __builtin_typeinfo_hash (@var{type})}
+
+The built-in function @code{__builtin_typeinfo_hash} returns a hash value
+for the given type @var{type} (which is a type, not an expression).
+The hash is computed using the FNV-1a algorithm on the type's mangled
+name representation, which follows a subset of the Itanium C++ ABI
+conventions adapted for C types. (See @code{__buitin_typeinfo_name}
+for the string representation.)
+
+This built-in is primarily intended for kernel control flow integrity (KCFI)
+implementations and other type-aware runtime systems that need to generate
+consistent type identifiers. The hash value is a 32-bit unsigned integer.
+
+Key characteristics of the hash:
+@itemize @bullet
+@item
+The hash is consistent for the same type across different translation units.
+@item
+Typedefs are recursively canonicalized down to integral type name or named
+struct, union, or enum tag name.
+@item
+Typedefs of anonymous structs, unions, and enums preserve the typedef name
+in the hash calculation (e.g., @code{typedef struct @{ int x; @} foo_t;}
+uses @code{foo_t} in the hash).
+@item
+Type qualifiers (@code{const}, @code{volatile}, @code{restrict}) affect
+the hash value.
+@item
+Function types include parameter types and variadic markers in the hash.
+@end itemize
+
+For example:
+@smallexample
+typedef struct @{ int x; @} mytype_t;
+unsigned int hash1 = __builtin_typeinfo_hash (mytype_t);
+unsigned int hash2 = __builtin_typeinfo_hash (struct @{ int x; @});
+/* hash1 != hash2 because the typedef name is preserved */
+
+void func(int x, char y);
+unsigned int hash3 = __builtin_typeinfo_hash (typeof (func));
+/* Returns hash for function type "void(int, char)" */
+@end smallexample
+
+@emph{Note:} This construct is only available for C@. For C++, see
+@code{std::type_info::hash_code}.
+
+@enddefbuiltin
+
+@defbuiltin{{const char *} __builtin_typeinfo_name (@var{type})}
+
+The built-in function @code{__builtin_typeinfo_name} returns a string
+containing the mangled name representation of the given type @var{type}
+(which is a type, not an expression). The string follows a subset of the
+Itanium C++ ABI mangling conventions adapted for C types. (See
+@code{__buitin_typeinfo_hash} for the unsigned 32-bit hash representation.)
+
+The returned string is a compile-time constant suitable for use in
+string comparisons, debugging output, or other type introspection needs.
+The string begins with @code{_ZTS} followed by the encoded type information.
+
+Mangling examples:
+@itemize @bullet
+@item
+@code{int} becomes @code{"_ZTSi"}
+@item
+@code{char *} becomes @code{"_ZTSPc"}
+@item
+@code{const int} becomes @code{"_ZTSKi"}
+@item
+@code{int[10]} becomes @code{"_ZTSA10_i"}
+@item
+@code{void (*)(int)} becomes @code{"_ZTSPFviE"}
+@item
+@code{struct foo} becomes @code{"_ZTS3foo"}
+@item
+@code{typedef struct @{ int x; @} bar_t;} becomes @code{"_ZTS5bar_t"}
+@end itemize
+
+The mangling preserves typedef names for anonymous compound types, which
+is particularly useful for distinguishing between different typedefs of
+structurally identical anonymous types:
+
+@smallexample
+typedef struct @{ int x; @} type_a;
+typedef struct @{ int x; @} type_b;
+const char *name_a = __builtin_typeinfo_name (type_a); /* "_ZTS6type_a" */
+const char *name_b = __builtin_typeinfo_name (type_b); /* "_ZTS6type_b" */
+/* name_a and name_b are different despite identical structure */
+@end smallexample
+
+@emph{Note:} This construct is only available for C@. For C++, see
+@code{std::type_info::name}.
+
+@enddefbuiltin
+
@defbuiltin{int __builtin_types_compatible_p (@var{type1}, @var{type2})}
You can use the built-in function @code{__builtin_types_compatible_p} to
diff --git a/gcc/testsuite/gcc.dg/builtin-typeinfo-errors.c b/gcc/testsuite/gcc.dg/builtin-typeinfo-errors.c
new file mode 100644
index 000000000000..71ad01337b4e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/builtin-typeinfo-errors.c
@@ -0,0 +1,28 @@
+/* Test error handling for __builtin_typeinfo_name and __builtin_typeinfo_hash. */
+/* { dg-do compile } */
+
+int main() {
+ /* Test missing arguments */
+ const char *result1 = __builtin_typeinfo_name(); /* { dg-error "expected specifier-qualifier-list before '\\)'" } */
+ /* { dg-error "expected type name in '__builtin_typeinfo_name'" "" { target *-*-* } .-1 } */
+ unsigned int result2 = __builtin_typeinfo_hash(); /* { dg-error "expected specifier-qualifier-list before '\\)'" } */
+ /* { dg-error "expected type name in '__builtin_typeinfo_hash'" "" { target *-*-* } .-1 } */
+
+ /* Test wrong argument types (expressions instead of type names) */
+ const char *result3 = __builtin_typeinfo_name(42); /* { dg-error "expected specifier-qualifier-list before numeric constant" } */
+ /* { dg-error "expected type name in '__builtin_typeinfo_name'" "" { target *-*-* } .-1 } */
+ unsigned int result4 = __builtin_typeinfo_hash(42); /* { dg-error "expected specifier-qualifier-list before numeric constant" } */
+ /* { dg-error "expected type name in '__builtin_typeinfo_hash'" "" { target *-*-* } .-1 } */
+
+ int x = 5;
+ const char *result5 = __builtin_typeinfo_name(x); /* { dg-error "expected specifier-qualifier-list before" } */
+ /* { dg-error "expected type name in '__builtin_typeinfo_name'" "" { target *-*-* } .-1 } */
+ unsigned int result6 = __builtin_typeinfo_hash(x); /* { dg-error "expected specifier-qualifier-list before" } */
+ /* { dg-error "expected type name in '__builtin_typeinfo_hash'" "" { target *-*-* } .-1 } */
+
+ /* Test too many arguments */
+ const char *result7 = __builtin_typeinfo_name(int, int); /* { dg-error "expected '\\)' before ','" } */
+ unsigned int result8 = __builtin_typeinfo_hash(int, int); /* { dg-error "expected '\\)' before ','" } */
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/builtin-typeinfo.c b/gcc/testsuite/gcc.dg/builtin-typeinfo.c
new file mode 100644
index 000000000000..307657310fec
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/builtin-typeinfo.c
@@ -0,0 +1,350 @@
+/* Test KCFI type mangling using __builtin_typeinfo_name. */
+/* { dg-do run } */
+/* { dg-options "-std=gnu99" } */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdarg.h>
+
+int pass, fail;
+
+#define TEST_STRING(expr, expected_string) \
+ do { \
+ const char *actual_string = __builtin_typeinfo_name(typeof(expr)); \
+ printf("Testing %s: ", #expr); \
+ if (strcmp(actual_string, expected_string) == 0) { \
+ printf("PASS (%s)\n", actual_string); \
+ pass ++; \
+ } else { \
+ printf("FAIL\n"); \
+ printf(" Expected: %s\n", expected_string); \
+ printf(" Actual: %s\n", actual_string); \
+ fail ++; \
+ } \
+ } while (0)
+
+int main(void)
+{
+ printf("Testing KCFI Typeinfo Mangling\n");
+ printf("======================================================\n");
+
+ /* Test basic types */
+ TEST_STRING(void, "v");
+ TEST_STRING(char, "c");
+ TEST_STRING(int, "i");
+ TEST_STRING(short, "s");
+ TEST_STRING(long, "l");
+ TEST_STRING(float, "f");
+ TEST_STRING(double, "d");
+
+ /* Test qualified types */
+ TEST_STRING(const int, "Ki");
+ TEST_STRING(volatile int, "Vi");
+
+ /* Test pointer types */
+ TEST_STRING(char*, "Pc");
+ TEST_STRING(int*, "Pi");
+ TEST_STRING(void*, "Pv");
+ TEST_STRING(const char*, "PKc");
+
+ /* Test array types */
+ TEST_STRING(int[10], "A10_i");
+ TEST_STRING(char[20], "A20_c");
+ TEST_STRING(short[], "A_s");
+
+ /* Test basic function types */
+ extern void func_void(void);
+ extern void func_char(char x);
+ extern void func_short(short x);
+ extern void func_int(int x);
+ extern void func_long(long x);
+ TEST_STRING(func_void, "FvvE");
+ TEST_STRING(func_char, "FvcE");
+ TEST_STRING(func_short, "FvsE");
+ TEST_STRING(func_int, "FviE");
+ TEST_STRING(func_long, "FvlE");
+
+ /* Test functions with unsigned types */
+ extern void func_unsigned_char(unsigned char x);
+ extern void func_unsigned_short(unsigned short x);
+ extern void func_unsigned_int(unsigned int x);
+ TEST_STRING(func_unsigned_char, "FvhE");
+ TEST_STRING(func_unsigned_short, "FvtE");
+ TEST_STRING(func_unsigned_int, "FvjE");
+
+ /* Test functions with signed types */
+ extern void func_signed_char(signed char x);
+ extern void func_signed_short(signed short x);
+ extern void func_signed_int(signed int x);
+ TEST_STRING(func_signed_char, "FvaE");
+ TEST_STRING(func_signed_short, "FvsE");
+ TEST_STRING(func_signed_int, "FviE");
+
+ /* Test functions with pointer types */
+ extern void func_void_ptr(void *x);
+ extern void func_char_ptr(char *x);
+ extern void func_short_ptr(short *x);
+ extern void func_int_ptr(int *x);
+ extern void func_int_array(int arr[]); /* Decays to "int *". */
+ extern void func_long_ptr(long *x);
+ TEST_STRING(func_void_ptr, "FvPvE");
+ TEST_STRING(func_char_ptr, "FvPcE");
+ TEST_STRING(func_short_ptr, "FvPsE");
+ TEST_STRING(func_int_ptr, "FvPiE");
+ TEST_STRING(func_int_array, "FvPiE");
+ TEST_STRING(func_long_ptr, "FvPlE");
+
+ /* Test functions with const qualifiers */
+ extern void func_const_void_ptr(const void *x);
+ extern void func_const_char_ptr(const char *x);
+ extern void func_const_short_ptr(const short *x);
+ extern void func_const_int_ptr(const int *x);
+ extern void func_const_long_ptr(const long *x);
+ TEST_STRING(func_const_void_ptr, "FvPKvE");
+ TEST_STRING(func_const_char_ptr, "FvPKcE");
+ TEST_STRING(func_const_short_ptr, "FvPKsE");
+ TEST_STRING(func_const_int_ptr, "FvPKiE");
+ TEST_STRING(func_const_long_ptr, "FvPKlE");
+
+ /* Test nested pointers */
+ extern void func_int_ptr_ptr(int **x);
+ extern void func_char_ptr_ptr(char **x);
+ TEST_STRING(func_int_ptr_ptr, "FvPPiE");
+ TEST_STRING(func_char_ptr_ptr, "FvPPcE");
+
+ /* Test multiple parameters */
+ extern void func_int_char(int x, char y);
+ extern void func_char_int(char x, int y);
+ extern void func_two_int(int x, int y);
+ TEST_STRING(func_int_char, "FvicE");
+ TEST_STRING(func_char_int, "FvciE");
+ TEST_STRING(func_two_int, "FviiE");
+
+ /* Test return types */
+ extern int func_return_int(void);
+ extern char func_return_char(void);
+ extern void* func_return_ptr(void);
+ TEST_STRING(func_return_int, "FivE");
+ TEST_STRING(func_return_char, "FcvE");
+ TEST_STRING(func_return_ptr, "FPvvE");
+
+ /* Test function pointer parameters */
+ extern void func_fptr_void(void (*fp)(void));
+ extern void func_fptr_int(void (*fp)(int));
+ extern void func_fptr_ret_int(int (*fp)(void));
+ TEST_STRING(func_fptr_void, "FvPFvvEE");
+ TEST_STRING(func_fptr_int, "FvPFviEE");
+ TEST_STRING(func_fptr_ret_int, "FvPFivEE");
+
+ /* Test variadic functions */
+ struct audit_context { int dummy; };
+ extern void func_variadic_simple(const char *fmt, ...);
+ extern void func_variadic_mixed(int x, const char *fmt, ...);
+ extern void func_variadic_multi(int x, char y, const char *fmt, ...);
+ extern void audit_log_pattern(struct audit_context *ctx, unsigned int gfp_mask,
+ int type, const char *fmt, ...);
+ TEST_STRING(func_variadic_simple, "FvPKczE");
+ TEST_STRING(func_variadic_mixed, "FviPKczE");
+ TEST_STRING(func_variadic_multi, "FvicPKczE");
+ TEST_STRING(audit_log_pattern, "FvP13audit_contextjiPKczE");
+
+ /* Test mixed const/non-const */
+ extern void func_const_mixed(int x, const char *fmt);
+ TEST_STRING(func_const_mixed, "FviPKcE");
+
+ /* Test named struct types */
+ struct test_struct_a { int x; };
+ struct test_struct_b { char y; };
+ struct test_struct_c { void *ptr; };
+ TEST_STRING(struct test_struct_a, "13test_struct_a");
+ extern void func_struct_a_ptr(struct test_struct_a *x);
+ extern void func_struct_b_ptr(struct test_struct_b *x);
+ extern void func_struct_c_ptr(struct test_struct_c *x);
+ TEST_STRING(func_struct_a_ptr, "FvP13test_struct_aE");
+ TEST_STRING(func_struct_b_ptr, "FvP13test_struct_bE");
+ TEST_STRING(func_struct_c_ptr, "FvP13test_struct_cE");
+
+ /* Test const named struct types */
+ extern void func_const_struct_a_ptr(const struct test_struct_a *x);
+ extern void func_const_struct_b_ptr(const struct test_struct_b *x);
+ extern void func_const_struct_c_ptr(const struct test_struct_c *x);
+ TEST_STRING(func_const_struct_a_ptr, "FvPK13test_struct_aE");
+ TEST_STRING(func_const_struct_b_ptr, "FvPK13test_struct_bE");
+ TEST_STRING(func_const_struct_c_ptr, "FvPK13test_struct_cE");
+
+ /* Test named union types */
+ union test_union_a { int x; float y; };
+ union test_union_b { char a; void *b; };
+ TEST_STRING(union test_union_a, "12test_union_a");
+ extern void func_union_a_ptr(union test_union_a *x);
+ extern void func_union_b_ptr(union test_union_b *x);
+ TEST_STRING(func_union_a_ptr, "FvP12test_union_aE");
+ TEST_STRING(func_union_b_ptr, "FvP12test_union_bE");
+
+ /* Test enum types: distinct from int */
+ enum test_enum_a { ENUM_A_VAL };
+ enum test_enum_b { ENUM_B_VAL };
+ TEST_STRING(enum test_enum_a, "11test_enum_a");
+ extern void func_enum_a_ptr(enum test_enum_a *x);
+ extern void func_enum_b_ptr(enum test_enum_b *x);
+ TEST_STRING(func_enum_a_ptr, "FvP11test_enum_aE");
+ TEST_STRING(func_enum_b_ptr, "FvP11test_enum_bE");
+
+ /* Test union member discrimination */
+ struct tasklet {
+ int state;
+ union {
+ void (*func)(unsigned long data);
+ void (*callback)(struct tasklet *t);
+ };
+ unsigned long data;
+ } tasklet_instance;
+ TEST_STRING(tasklet_instance, "7tasklet");
+ struct tasklet *p = &tasklet_instance;
+ extern void tasklet_callback_function(struct tasklet *t);
+ extern void tasklet_func_function(unsigned long data);
+ TEST_STRING(tasklet_func_function, "FvmE");
+ TEST_STRING(*p->func, "FvmE");
+ TEST_STRING(tasklet_callback_function, "FvP7taskletE");
+ TEST_STRING(*p->callback, "FvP7taskletE");
+
+ /* Test struct return pointers */
+ extern struct test_struct_a* func_ret_struct_a_ptr(void);
+ extern struct test_struct_b* func_ret_struct_b_ptr(void);
+ extern struct test_struct_c* func_ret_struct_c_ptr(void);
+ TEST_STRING(func_ret_struct_a_ptr, "FP13test_struct_avE");
+ TEST_STRING(func_ret_struct_b_ptr, "FP13test_struct_bvE");
+ TEST_STRING(func_ret_struct_c_ptr, "FP13test_struct_cvE");
+
+ /* Test struct by-value parameters */
+ extern void func_struct_a_val(struct test_struct_a x);
+ extern void func_struct_b_val(struct test_struct_b x);
+ extern void func_struct_c_val(struct test_struct_c x);
+ TEST_STRING(func_struct_a_val, "Fv13test_struct_aE");
+ TEST_STRING(func_struct_b_val, "Fv13test_struct_bE");
+ TEST_STRING(func_struct_c_val, "Fv13test_struct_cE");
+
+ /* Test struct return by-value */
+ extern struct test_struct_a func_ret_struct_a_val(void);
+ extern struct test_struct_b func_ret_struct_b_val(void);
+ extern struct test_struct_c func_ret_struct_c_val(void);
+ TEST_STRING(func_ret_struct_a_val, "F13test_struct_avE");
+ TEST_STRING(func_ret_struct_b_val, "F13test_struct_bvE");
+ TEST_STRING(func_ret_struct_c_val, "F13test_struct_cvE");
+
+ /* Test mixed struct parameters */
+ extern void func_struct_a_b(struct test_struct_a *a, struct test_struct_b *b);
+ extern void func_struct_b_a(struct test_struct_b *b, struct test_struct_a *a);
+ TEST_STRING(func_struct_a_b, "FvP13test_struct_aP13test_struct_bE");
+ TEST_STRING(func_struct_b_a, "FvP13test_struct_bP13test_struct_aE");
+
+ /* Test anonymous struct typedefs */
+ typedef struct { int x; } typedef_struct_x;
+ typedef struct { int y; } typedef_struct_y;
+ TEST_STRING(typedef_struct_x, "16typedef_struct_x");
+ extern void func_typedef_x_ptr(typedef_struct_x *x);
+ extern void func_typedef_y_ptr(typedef_struct_y *y);
+ TEST_STRING(func_typedef_x_ptr, "FvP16typedef_struct_xE");
+ TEST_STRING(func_typedef_y_ptr, "FvP16typedef_struct_yE");
+ extern void func_typedef_x(typedef_struct_x x);
+ TEST_STRING(func_typedef_x, "Fv16typedef_struct_xE");
+
+ /* Test anonymous union typedefs */
+ typedef union { int x; short a; } typedef_union_x;
+ typedef union { int y; short b; } typedef_union_y;
+ TEST_STRING(typedef_union_x, "15typedef_union_x");
+ extern void func_typedef_union_x_ptr(typedef_union_x *x);
+ extern void func_typedef_union_y_ptr(typedef_union_y *y);
+ TEST_STRING(func_typedef_union_x_ptr, "FvP15typedef_union_xE");
+ TEST_STRING(func_typedef_union_y_ptr, "FvP15typedef_union_yE");
+ extern void func_typedef_union_x(typedef_union_x x);
+ TEST_STRING(func_typedef_union_x, "Fv15typedef_union_xE");
+
+ /* Test anonymous enum typedefs */
+ typedef enum { STEP_1, STEP_2 } typedef_enum_x;
+ typedef enum { STEP_A, STEP_B } typedef_enum_y;
+ TEST_STRING(typedef_enum_x, "14typedef_enum_x");
+ extern void func_typedef_enum_x_ptr(typedef_enum_x *x);
+ extern void func_typedef_enum_y_ptr(typedef_enum_y *y);
+ TEST_STRING(func_typedef_enum_x_ptr, "FvP14typedef_enum_xE");
+ TEST_STRING(func_typedef_enum_y_ptr, "FvP14typedef_enum_yE");
+ extern void func_typedef_enum_x(typedef_enum_x x);
+ TEST_STRING(func_typedef_enum_x, "Fv14typedef_enum_xE");
+
+ /* Test basic typedef vs open-coded function types: should be the same. */
+ typedef void (*func_type_typedef)(int, char);
+ TEST_STRING(func_type_typedef, "PFvicE");
+ extern void func_with_typedef_param(func_type_typedef fp);
+ extern void func_with_opencoded_param(void (*fp)(int, char));
+ TEST_STRING(func_with_typedef_param, "FvPFvicEE");
+ TEST_STRING(func_with_opencoded_param, "FvPFvicEE");
+
+ /* Test return function pointer types */
+ typedef int (*ret_func_type_typedef)(void);
+ TEST_STRING(ret_func_type_typedef, "PFivE");
+ extern ret_func_type_typedef func_ret_typedef_param(void);
+ extern int (*func_ret_opencoded_param(void))(void);
+ TEST_STRING(func_ret_typedef_param, "FPFivEvE");
+ TEST_STRING(func_ret_opencoded_param, "FPFivEvE");
+
+ /* Test additional type combos */
+ extern void func_float(float x);
+ extern void func_double_ptr(double *x);
+ extern void func_float_ptr(float *x);
+ extern void func_void_ptr_ptr(void **x);
+ extern void func_ptr_val(int *x, int y);
+ extern void func_val_ptr(int x, int *y);
+ extern float func_return_float(void);
+ extern double func_return_double(void);
+ TEST_STRING(func_float, "FvfE");
+ TEST_STRING(func_double_ptr, "FvPdE");
+ TEST_STRING(func_float_ptr, "FvPfE");
+ TEST_STRING(func_void_ptr_ptr, "FvPPvE");
+ TEST_STRING(func_ptr_val, "FvPiiE");
+ TEST_STRING(func_val_ptr, "FviPiE");
+ TEST_STRING(func_return_float, "FfvE");
+ TEST_STRING(func_return_double, "FdvE");
+
+ /* Test VLA types: should be all the same. */
+ extern void func_vla_1d(int n, int arr[n]);
+ extern void func_vla_empty(int n, int arr[]);
+ extern void func_vla_ptr(int n, int *arr);
+ TEST_STRING(func_vla_1d, "FviPiE");
+ TEST_STRING(func_vla_empty, "FviPiE");
+ TEST_STRING(func_vla_ptr, "FviPiE");
+
+ /* Test 2D VLA with fixed dimension: should be all the same. */
+ extern void func_vla_2d_first(int n, int arr[n][10]);
+ extern void func_vla_2d_empty(int n, int arr[][10]);
+ extern void func_vla_2d_ptr(int n, int (*arr)[10]);
+ TEST_STRING(func_vla_2d_first, "FviPA10_iE");
+ TEST_STRING(func_vla_2d_empty, "FviPA10_iE");
+ TEST_STRING(func_vla_2d_ptr, "FviPA10_iE");
+
+ /* Test 2D VLA with both dimensions variable: should be all the same. */
+ extern void func_vla_2d_both(int rows, int cols, int arr[rows][cols]);
+ extern void func_vla_2d_second(int rows, int cols, int arr[][cols]);
+ extern void func_vla_2d_star(int rows, int cols, int arr[*][cols]);
+ TEST_STRING(func_vla_2d_both, "FviiPA_iE");
+ TEST_STRING(func_vla_2d_second, "FviiPA_iE");
+ TEST_STRING(func_vla_2d_star, "FviiPA_iE");
+
+ /* Test recursive typedef canonicalization */
+ struct recursive_struct_test { int field; };
+ typedef struct recursive_struct_test recursive_struct_typedef_1;
+ typedef recursive_struct_typedef_1 recursive_struct_typedef_2;
+ extern void func_recursive_struct_test(struct recursive_struct_test *x);
+ TEST_STRING(func_recursive_struct_test, "FvP21recursive_struct_testE");
+
+ /* Test anonymous struct, union, enum types */
+ struct { int a; short b; } anon_struct;
+ union { int x; float y; } anon_union;
+ enum { ANON_VAL1, ANON_VAL2 } anon_enum;
+ TEST_STRING(anon_struct, "3$_0"); // <length>$_<counter>
+ TEST_STRING(anon_union, "3$_1"); // <length>$_<counter>
+ TEST_STRING(anon_enum, "3$_2"); // <length>$_<counter>
+
+ printf("\n================================================================\n");
+ printf("Passed: %d Failed: %d (%d total tests)\n", pass, fail, pass + fail);
+ return fail;
+}
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 02cc38ce57e0..de41e77c5d00 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1600,6 +1600,7 @@ OBJS = \
ira-emit.o \
ira-lives.o \
jump.o \
+ kcfi-typeinfo.o \
langhooks.o \
late-combine.o \
lcm.o \
diff --git a/gcc/c-family/c-common.h b/gcc/c-family/c-common.h
index 8b7f4ae44717..8c3d3cef6128 100644
--- a/gcc/c-family/c-common.h
+++ b/gcc/c-family/c-common.h
@@ -112,6 +112,7 @@ enum rid
RID_BUILTIN_SHUFFLEVECTOR, RID_BUILTIN_CONVERTVECTOR, RID_BUILTIN_TGMATH,
RID_BUILTIN_HAS_ATTRIBUTE, RID_BUILTIN_ASSOC_BARRIER, RID_BUILTIN_STDC,
RID_BUILTIN_COUNTED_BY_REF,
+ RID_BUILTIN_TYPEINFO_NAME, RID_BUILTIN_TYPEINFO_HASH,
RID_DFLOAT32, RID_DFLOAT64, RID_DFLOAT128, RID_DFLOAT64X,
/* TS 18661-3 keywords, in the same sequence as the TI_* values. */
diff --git a/gcc/kcfi-typeinfo.h b/gcc/kcfi-typeinfo.h
new file mode 100644
index 000000000000..805f9ebaeca4
--- /dev/null
+++ b/gcc/kcfi-typeinfo.h
@@ -0,0 +1,32 @@
+/* KCFI-compatible type mangling, based on Itanium C++ ABI.
+ Copyright (C) 2025 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#ifndef GCC_KCFI_TYPEINFO_H
+#define GCC_KCFI_TYPEINFO_H
+
+#include "tree.h"
+#include <string>
+
+/* Get the typeinfo mangled name string for any C type. */
+extern std::string typeinfo_get_name (tree type);
+
+/* Get the typeinfo hash for any C type. */
+extern uint32_t typeinfo_get_hash (tree type);
+
+#endif /* GCC_KCFI_TYPEINFO_H */
diff --git a/gcc/c-family/c-common.cc b/gcc/c-family/c-common.cc
index 9b97a0fe6224..8cb1318525d5 100644
--- a/gcc/c-family/c-common.cc
+++ b/gcc/c-family/c-common.cc
@@ -462,6 +462,8 @@ const struct c_common_resword c_common_reswords[] =
{ "__builtin_stdc_trailing_zeros", RID_BUILTIN_STDC, D_CONLY },
{ "__builtin_tgmath", RID_BUILTIN_TGMATH, D_CONLY },
{ "__builtin_offsetof", RID_OFFSETOF, 0 },
+ { "__builtin_typeinfo_hash", RID_BUILTIN_TYPEINFO_HASH, D_CONLY },
+ { "__builtin_typeinfo_name", RID_BUILTIN_TYPEINFO_NAME, D_CONLY },
{ "__builtin_types_compatible_p", RID_TYPES_COMPATIBLE_P, D_CONLY },
{ "__builtin_c23_va_start", RID_C23_VA_START, D_C23 | D_CXX26 },
{ "__builtin_va_arg", RID_VA_ARG, 0 },
diff --git a/gcc/c/c-parser.cc b/gcc/c/c-parser.cc
index 360df52967b5..4adca5909c6c 100644
--- a/gcc/c/c-parser.cc
+++ b/gcc/c/c-parser.cc
@@ -77,6 +77,7 @@ along with GCC; see the file COPYING3. If not see
#include "asan.h"
#include "c-family/c-ubsan.h"
#include "gcc-urlifier.h"
+#include "kcfi-typeinfo.h"
\f
/* We need to walk over decls with incomplete struct/union/enum types
after parsing the whole translation unit.
@@ -11036,6 +11037,38 @@ c_parser_has_attribute_expression (c_parser *parser)
return result;
}
+/* Parse the single type name argument of a builtin that takes a type name.
+ Returns true on success and stores the parsed type in *OUT_TYPE.
+ If successful, *OUT_CLOSE_PAREN_LOC is written with the location of
+ the closing parenthesis. */
+
+static bool
+c_parser_get_builtin_type_arg (c_parser *parser, const char *bname,
+ tree *out_type, location_t *out_close_paren_loc)
+{
+ matching_parens parens;
+ if (!parens.require_open (parser))
+ return false;
+
+ struct c_type_name *type_name = c_parser_type_name (parser);
+ if (type_name == NULL)
+ {
+ error_at (c_parser_peek_token (parser)->location,
+ "expected type name in %qs", bname);
+ return false;
+ }
+
+ *out_close_paren_loc = c_parser_peek_token (parser)->location;
+ parens.skip_until_found_close (parser);
+
+ tree type = groktypename (type_name, NULL, NULL);
+ if (type == error_mark_node)
+ return false;
+
+ *out_type = type;
+ return true;
+}
+
/* Helper function to read arguments of builtins which are interfaces
for the middle-end nodes like COMPLEX_EXPR, VEC_PERM_EXPR and
others. The name of the builtin is passed using BNAME parameter.
@@ -12064,6 +12097,45 @@ c_parser_postfix_expression (c_parser *parser)
set_c_expr_source_range (&expr, loc, close_paren_loc);
}
break;
+ case RID_BUILTIN_TYPEINFO_NAME:
+ {
+ c_parser_consume_token (parser);
+ location_t close_paren_loc;
+ tree type;
+ if (!c_parser_get_builtin_type_arg (parser,
+ "__builtin_typeinfo_name",
+ &type, &close_paren_loc))
+ {
+ expr.set_error ();
+ break;
+ }
+
+ /* Call the typeinfo name function. */
+ std::string type_name = typeinfo_get_name (type);
+ expr.value = build_string_literal (type_name.length () + 1,
+ type_name.c_str ());
+ set_c_expr_source_range (&expr, loc, close_paren_loc);
+ }
+ break;
+ case RID_BUILTIN_TYPEINFO_HASH:
+ {
+ c_parser_consume_token (parser);
+ location_t close_paren_loc;
+ tree type;
+ if (!c_parser_get_builtin_type_arg (parser,
+ "__builtin_typeinfo_hash",
+ &type, &close_paren_loc))
+ {
+ expr.set_error ();
+ break;
+ }
+
+ /* Call the typeinfo hash function. */
+ uint32_t type_hash = typeinfo_get_hash (type);
+ expr.value = build_int_cst (unsigned_type_node, type_hash);
+ set_c_expr_source_range (&expr, loc, close_paren_loc);
+ }
+ break;
case RID_BUILTIN_TGMATH:
{
vec<c_expr_t, va_gc> *cexpr_list;
diff --git a/gcc/kcfi-typeinfo.cc b/gcc/kcfi-typeinfo.cc
new file mode 100644
index 000000000000..628a8538bf95
--- /dev/null
+++ b/gcc/kcfi-typeinfo.cc
@@ -0,0 +1,516 @@
+/* KCFI-compatible type mangling, based on Itanium C++ ABI.
+ Copyright (C) 2025 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+/* Produces typeinfo mangling similar to Itanium C++ Mangling ABI, but
+ limited to types exposed within GCC for C language handling. The
+ hashes are used by KCFI (and future type-aware allocator support).
+ The strings are used for testing and debugging. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "tree.h"
+#include "diagnostic-core.h"
+#include "stringpool.h"
+#include "stor-layout.h"
+#include "print-tree.h"
+#include "kcfi-typeinfo.h"
+
+/* Helper to update FNV-1a hash with a single character. HASH_STATE is
+ the hash accumulator to update. C is the character to hash. */
+
+static inline void
+fnv1a_hash_char (uint32_t *hash_state, unsigned char c)
+{
+ *hash_state ^= c;
+ *hash_state *= 16777619U; /* FNV-1a 32-bit prime. */
+}
+
+/* Helper to append character to optional string and update hash using
+ FNV-1a. C is the character to append. OUT_STR is the optional string
+ to append to (NULL if not needed). HASH_STATE is the optional hash
+ accumulator to update (NULL if not needed). */
+
+static void
+append_char (char c, std::string *out_str, uint32_t *hash_state)
+{
+ if (out_str)
+ *out_str += c;
+ if (!hash_state)
+ return;
+ fnv1a_hash_char (hash_state, (unsigned char) c);
+}
+
+/* Helper to append string to optional string and update hash using
+ FNV-1a. STR is the string to append. OUT_STR is the optional string
+ to append to (NULL if not needed). HASH_STATE is the optional hash
+ accumulator to update (NULL if not needed). */
+
+static void
+append_string (const char *str, std::string *out_str, uint32_t *hash_state)
+{
+ if (out_str)
+ *out_str += str;
+ if (!hash_state)
+ return;
+ for (const char *p = str; *p; p++)
+ fnv1a_hash_char (hash_state, (unsigned char) *p);
+}
+
+/* Forward declaration for recursive type mangling. */
+
+static void mangle_type (tree type, std::string *out_str, uint32_t *hash_state);
+
+/* Mangle a builtin type following Itanium C++ ABI for C types. TYPE is
+ the builtin type to mangle. OUT_STR is the optional string to append
+ mangling to (NULL if not needed). HASH_STATE is the optional hash
+ accumulator to update (NULL if not needed). */
+
+static void
+mangle_builtin_type (tree type, std::string *out_str, uint32_t *hash_state)
+{
+ gcc_assert (type != NULL_TREE);
+
+ switch (TREE_CODE (type))
+ {
+ case VOID_TYPE:
+ append_char ('v', out_str, hash_state);
+ return;
+
+ case BOOLEAN_TYPE:
+ append_char ('b', out_str, hash_state);
+ return;
+
+ case INTEGER_TYPE:
+ if (type == char_type_node)
+ append_char ('c', out_str, hash_state);
+ else if (type == signed_char_type_node)
+ append_char ('a', out_str, hash_state);
+ else if (type == unsigned_char_type_node)
+ append_char ('h', out_str, hash_state);
+ else if (type == short_integer_type_node)
+ append_char ('s', out_str, hash_state);
+ else if (type == short_unsigned_type_node)
+ append_char ('t', out_str, hash_state);
+ else if (type == integer_type_node)
+ append_char ('i', out_str, hash_state);
+ else if (type == unsigned_type_node)
+ append_char ('j', out_str, hash_state);
+ else if (type == long_integer_type_node)
+ append_char ('l', out_str, hash_state);
+ else if (type == long_unsigned_type_node)
+ append_char ('m', out_str, hash_state);
+ else if (type == long_long_integer_type_node)
+ append_char ('x', out_str, hash_state);
+ else if (type == long_long_unsigned_type_node)
+ append_char ('y', out_str, hash_state);
+ else
+ {
+ /* Fallback for other integer types - use precision-based
+ encoding. */
+ append_char ('i', out_str, hash_state);
+ append_string (std::to_string (TYPE_PRECISION (type)).c_str (),
+ out_str, hash_state);
+ }
+ return;
+
+ case REAL_TYPE:
+ if (type == float_type_node)
+ append_char ('f', out_str, hash_state);
+ else if (type == double_type_node)
+ append_char ('d', out_str, hash_state);
+ else if (type == long_double_type_node)
+ append_char ('e', out_str, hash_state);
+ else
+ {
+ /* Fallback for other real types. */
+ append_char ('f', out_str, hash_state);
+ append_string (std::to_string (TYPE_PRECISION (type)).c_str (),
+ out_str, hash_state);
+ }
+ return;
+
+ case VECTOR_TYPE:
+ {
+ /* Handle vector types:
+ Dv<num-elements>_<element-type-encoding>
+ Example: uint8x16_t -> Dv16_h (vector of 16 unsigned char) */
+ tree vector_size = TYPE_SIZE_UNIT (type);
+ tree element_type = TREE_TYPE (type);
+ tree element_size = TYPE_SIZE_UNIT (element_type);
+
+ if (vector_size && element_size
+ && TREE_CODE (vector_size) == INTEGER_CST
+ && TREE_CODE (element_size) == INTEGER_CST)
+ {
+ append_char ('D', out_str, hash_state);
+ append_char ('v', out_str, hash_state);
+
+ unsigned HOST_WIDE_INT vec_bytes = tree_to_uhwi (vector_size);
+ unsigned HOST_WIDE_INT elem_bytes = tree_to_uhwi (element_size);
+ unsigned HOST_WIDE_INT num_elements = vec_bytes / elem_bytes;
+
+ /* Append number of elements. */
+ append_string (std::to_string (num_elements).c_str (),
+ out_str, hash_state);
+ append_char ('_', out_str, hash_state);
+
+ /* Recursively mangle the element type. */
+ mangle_type (element_type, out_str, hash_state);
+ return;
+ }
+
+ /* Scalable vectors (like RISC-V RVV types): use type name if
+ available, otherwise use a generic scalable vector encoding. */
+ if (TYPE_NAME (type))
+ {
+ const char *name = NULL;
+ if (TREE_CODE (TYPE_NAME (type)) == TYPE_DECL)
+ {
+ tree decl_name = DECL_NAME (TYPE_NAME (type));
+ if (decl_name && TREE_CODE (decl_name) == IDENTIFIER_NODE)
+ name = IDENTIFIER_POINTER (decl_name);
+ }
+ else if (TREE_CODE (TYPE_NAME (type)) == IDENTIFIER_NODE)
+ {
+ name = IDENTIFIER_POINTER (TYPE_NAME (type));
+ }
+
+ if (name)
+ {
+ /* Use vendor-extended type name encoding: u<length><name>. */
+ append_char ('u', out_str, hash_state);
+ append_string (std::to_string (strlen (name)).c_str (),
+ out_str, hash_state);
+ append_string (name, out_str, hash_state);
+ return;
+ }
+ }
+
+ /* Fallback for unnamed scalable vectors: encode as Dv0_<element>.
+ Using 0 elements indicates scalable/unknown size. */
+ append_string ("Dv0_", out_str, hash_state);
+ mangle_type (element_type, out_str, hash_state);
+ return;
+ }
+
+ default:
+ break;
+ }
+
+ /* Unknown builtin type: this should never happen in a well-formed C. */
+ debug_tree (type);
+ internal_error ("mangle: Unknown builtin type - please report this as a bug");
+}
+
+/* Canonicalize typedef types to their underlying named struct/union types.
+ TYPE is the type to canonicalize. Returns the canonicalized type. */
+
+static tree
+canonicalize_typedef_type (tree type)
+{
+ /* Handle typedef types: canonicalize to named structs when possible. */
+ if (TYPE_NAME (type) && TREE_CODE (TYPE_NAME (type)) == TYPE_DECL)
+ {
+ tree type_decl = TYPE_NAME (type);
+
+ /* Check if this is a typedef (not the original struct declaration) */
+ if (DECL_ORIGINAL_TYPE (type_decl))
+ {
+ tree original_type = DECL_ORIGINAL_TYPE (type_decl);
+
+ /* Handle struct/union/enum types. */
+ if (TREE_CODE (original_type) == RECORD_TYPE
+ || TREE_CODE (original_type) == UNION_TYPE
+ || TREE_CODE (original_type) == ENUMERAL_TYPE)
+ {
+ /* Preserve typedef of anonymous struct/union/enum types. */
+ if (!TYPE_NAME (original_type))
+ return type;
+
+ /* Named compound type: canonicalize to it. */
+ return canonicalize_typedef_type (original_type);
+ }
+
+ /* For basic type typedefs (e.g., u8 -> unsigned char),
+ canonicalize to original type. */
+ if (TREE_CODE (original_type) == INTEGER_TYPE
+ || TREE_CODE (original_type) == REAL_TYPE
+ || TREE_CODE (original_type) == POINTER_TYPE
+ || TREE_CODE (original_type) == ARRAY_TYPE
+ || TREE_CODE (original_type) == FUNCTION_TYPE
+ || TREE_CODE (original_type) == METHOD_TYPE
+ || TREE_CODE (original_type) == BOOLEAN_TYPE
+ || TREE_CODE (original_type) == COMPLEX_TYPE
+ || TREE_CODE (original_type) == VECTOR_TYPE)
+ {
+ /* Recursively canonicalize in case the original type is
+ also a typedef. */
+ return canonicalize_typedef_type (original_type);
+ }
+ }
+ }
+
+ return type;
+}
+
+/* Recursively mangle a C type following Itanium C++ ABI. TYPE is the
+ type to mangle. OUT_STR is the optional string to append mangling to
+ (NULL if not needed). HASH_STATE is the optional hash accumulator to
+ update (NULL if not needed). */
+
+static void
+mangle_type (tree type, std::string *out_str, uint32_t *hash_state)
+{
+ gcc_assert (type != NULL_TREE);
+
+ /* Canonicalize typedef types to their underlying named struct types. */
+ type = canonicalize_typedef_type (type);
+
+ /* Save original qualified type for cases where we need typedef
+ information. */
+ tree qualified_type = type;
+
+ /* Centralized qualifier handling: emit qualifiers for this type,
+ then continue with unqualified version. */
+ if (TYPE_QUALS (type) != TYPE_UNQUALIFIED)
+ {
+ /* Emit qualifiers in Itanium ABI order: restrict, volatile, const. */
+ if (TYPE_QUALS (type) & TYPE_QUAL_RESTRICT)
+ append_char ('r', out_str, hash_state);
+ if (TYPE_QUALS (type) & TYPE_QUAL_VOLATILE)
+ append_char ('V', out_str, hash_state);
+ if (TYPE_QUALS (type) & TYPE_QUAL_CONST)
+ append_char ('K', out_str, hash_state);
+
+ /* Get unqualified version for further processing. */
+ type = TYPE_MAIN_VARIANT (type);
+ }
+
+ switch (TREE_CODE (type))
+ {
+ case POINTER_TYPE:
+ {
+ /* Pointer type: 'P' + pointed-to type. */
+ append_char ('P', out_str, hash_state);
+
+ /* Recursively mangle the pointed-to type. */
+ tree pointed_to_type = TREE_TYPE (type);
+ mangle_type (pointed_to_type, out_str, hash_state);
+ break;
+ }
+
+ case ARRAY_TYPE:
+ /* Array type: 'A' + size + '_' + element type (simplified). */
+ append_char ('A', out_str, hash_state);
+ if (TYPE_DOMAIN (type) && TYPE_MAX_VALUE (TYPE_DOMAIN (type)))
+ {
+ tree max_val = TYPE_MAX_VALUE (TYPE_DOMAIN (type));
+ /* Check if array size is compile-time constant to handle VLAs. */
+ if (TREE_CODE (max_val) == INTEGER_CST && tree_fits_shwi_p (max_val))
+ {
+ HOST_WIDE_INT size = tree_to_shwi (max_val) + 1;
+ append_string (std::to_string ((long) size).c_str (),
+ out_str, hash_state);
+ }
+ /* For VLAs or non-constant dimensions, emit empty size (A_). */
+ append_char ('_', out_str, hash_state);
+ }
+ else
+ {
+ /* No domain or no max value: emit A_. */
+ append_char ('_', out_str, hash_state);
+ }
+ mangle_type (TREE_TYPE (type), out_str, hash_state);
+ break;
+
+ case REFERENCE_TYPE:
+ /* Reference type: 'R' + referenced type.
+ Note: We must handle references to builtin types including compiler
+ builtins like __builtin_va_list used in functions like va_start. */
+ append_char ('R', out_str, hash_state);
+ mangle_type (TREE_TYPE (type), out_str, hash_state);
+ break;
+
+ case FUNCTION_TYPE:
+ {
+ /* Function type: 'F' + return type + parameter types + 'E' */
+ append_char ('F', out_str, hash_state);
+ mangle_type (TREE_TYPE (type), out_str, hash_state);
+
+ /* Add parameter types. */
+ tree param_types = TYPE_ARG_TYPES (type);
+
+ if (param_types == NULL_TREE)
+ {
+ /* func () - no parameter list (could be variadic). */
+ }
+ else
+ {
+ bool found_real_params = false;
+ for (tree param = param_types; param; param = TREE_CHAIN (param))
+ {
+ tree param_type = TREE_VALUE (param);
+ if (param_type == void_type_node)
+ {
+ /* Check if this is the first parameter (explicit void) or a
+ sentinel. */
+ if (!found_real_params)
+ {
+ /* func (void) - explicit empty parameter list.
+ Mangle void to distinguish from variadic func (). */
+ mangle_type (void_type_node, out_str, hash_state);
+ }
+ /* If we found real params before this void, it's a sentinel
+ so stop here. */
+ break;
+ }
+
+ found_real_params = true;
+
+ /* For value parameters, ignore const/volatile qualifiers as
+ they don't affect the calling convention. "const int" and
+ "int" are passed identically by value. */
+ tree canonical_param_type = param_type;
+
+ if (TREE_CODE (param_type) != POINTER_TYPE
+ && TREE_CODE (param_type) != REFERENCE_TYPE
+ && TREE_CODE (param_type) != ARRAY_TYPE)
+ {
+ /* For non-pointer/reference value parameters, strip
+ qualifiers by default. */
+ canonical_param_type = TYPE_MAIN_VARIANT (param_type);
+
+ /* Exception: preserve typedef information for anonymous
+ compound types. */
+ if (TYPE_NAME (param_type)
+ && TREE_CODE (TYPE_NAME (param_type)) == TYPE_DECL
+ && DECL_ORIGINAL_TYPE (TYPE_NAME (param_type)))
+ {
+ tree original_type
+ = DECL_ORIGINAL_TYPE (TYPE_NAME (param_type));
+ if ((TREE_CODE (original_type) == RECORD_TYPE
+ || TREE_CODE (original_type) == UNION_TYPE
+ || TREE_CODE (original_type) == ENUMERAL_TYPE)
+ && !TYPE_NAME (original_type))
+ {
+ /* Preserve typedef of an anonymous
+ struct/union/enum. */
+ canonical_param_type = param_type;
+ }
+ }
+ }
+
+ mangle_type (canonical_param_type, out_str, hash_state);
+ }
+ }
+
+ /* Check if this is a variadic function and add 'z' marker. */
+ if (stdarg_p (type))
+ {
+ append_char ('z', out_str, hash_state);
+ }
+
+ append_char ('E', out_str, hash_state);
+ break;
+ }
+
+ case RECORD_TYPE:
+ case UNION_TYPE:
+ case ENUMERAL_TYPE:
+ {
+ /* Struct/union/enum: use simplified representation for C types. */
+ const char *name = NULL;
+
+ /* For compound types, use the original qualified type to preserve
+ typedef info. */
+ if (TYPE_QUALS (qualified_type) != TYPE_UNQUALIFIED)
+ {
+ type = qualified_type;
+ }
+
+ if (TYPE_NAME (type))
+ {
+ if (TREE_CODE (TYPE_NAME (type)) == TYPE_DECL)
+ {
+ /* TYPE_DECL case: both named structs and typedef structs. */
+ tree decl_name = DECL_NAME (TYPE_NAME (type));
+ if (decl_name && TREE_CODE (decl_name) == IDENTIFIER_NODE)
+ {
+ name = IDENTIFIER_POINTER (decl_name);
+ }
+ }
+ else if (TREE_CODE (TYPE_NAME (type)) == IDENTIFIER_NODE)
+ {
+ /* Direct identifier case. */
+ name = IDENTIFIER_POINTER (TYPE_NAME (type));
+ }
+ }
+
+ if (name)
+ {
+ append_string (std::to_string (strlen (name)).c_str (),
+ out_str, hash_state);
+ append_string (name, out_str, hash_state);
+ break;
+ }
+
+ /* If no name found, use anonymous type format: <length>$_<counter>. */
+ static unsigned anon_counter = 0;
+ std::string anon_name = "$_" + std::to_string (anon_counter++);
+
+ append_string (std::to_string (anon_name.length ()).c_str (),
+ out_str, hash_state);
+ append_string (anon_name.c_str (), out_str, hash_state);
+ break;
+ }
+
+ default:
+ /* Handle builtin types. */
+ mangle_builtin_type (type, out_str, hash_state);
+ break;
+ }
+}
+
+/* Get the typeinfo mangled name string for any C type. TYPE is the type
+ to mangle. Returns the mangled type string following Itanium C++ ABI
+ conventions. */
+
+std::string
+typeinfo_get_name (tree type)
+{
+ gcc_assert (type != NULL_TREE);
+ std::string result = "";
+
+ mangle_type (type, &result, nullptr);
+ return result;
+}
+
+/* Get the typeinfo hash for any C type. TYPE is the type to hash.
+ Returns the FNV-1a hash of the mangled type string. */
+
+uint32_t
+typeinfo_get_hash (tree type)
+{
+ gcc_assert (type != NULL_TREE);
+ uint32_t hash_state = 2166136261U; /* FNV-1a 32-bit offset basis. */
+
+ mangle_type (type, nullptr, &hash_state);
+ return hash_state;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v8 2/7] kcfi: Add core Kernel Control Flow Integrity infrastructure
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
2025-11-20 22:21 ` [PATCH v8 1/7] typeinfo: Introduce KCFI typeinfo mangling API Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
2025-11-20 22:21 ` [PATCH v8 3/7] kcfi: Add regression test suite Kees Cook
` (4 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
Implements the Linux Kernel Control Flow Integrity ABI, which provides a
function prototype based forward edge control flow integrity protection
by instrumenting every indirect call to check for a hash value before
the target function address. If the hash at the call site and the hash
at the target do not match, execution will trap.
See the start of kcfi.cc for design details.
gcc/ChangeLog:
* kcfi.h: New file with KCFI public interface declarations.
* kcfi.cc: New file implementing Kernel Control Flow Integrity
infrastructure.
* Makefile.in (OBJS): Add kcfi.o.
* flag-types.h (enum sanitize_code): Add SANITIZE_KCFI.
* gimple.h (enum gf_mask): Add GF_CALL_INLINED_FROM_KCFI_NOSANTIZE.
(gimple_call_set_inlined_from_kcfi_nosantize): New function.
(gimple_call_inlined_from_kcfi_nosantize_p): New function.
* tree-pass.h Add kcfi passes.
* df-scan.cc (df_uses_record): Add KCFI case to handle KCFI RTL
patterns and process wrapped RTL.
* doc/extend.texi: Update nocf_check for kcfi.
* doc/invoke.texi (fsanitize=kcfi): Add documentation for KCFI
sanitizer option.
* doc/tm.texi.in: Add Kernel Control Flow Integrity section with
TARGET_KCFI_SUPPORTED, TARGET_KCFI_MASK_TYPE_ID,
TARGET_KCFI_EMIT_TYPE_ID hooks.
* doc/tm.texi: Regenerate.
* final.cc (call_from_call_insn): Add KCFI case to handle
KCFI-wrapped calls.
* opts.cc (sanitizer_opts): Add kcfi entry.
* passes.cc: Include kcfi.h.
* passes.def: Add KCFI IPA pass.
* rtl.def (KCFI): Add new RTL code for KCFI instrumentation.
* rtlanal.cc (reg_referenced_p): Add KCFI case.
* target.def: Add KCFI target hooks.
* toplev.cc (process_options): Add KCFI option processing.
* tree-inline.cc: Include kcfi.h and asan.h.
(copy_bb): Handle KCFI no_sanitize attribute propagation during
inlining.
* varasm.cc (assemble_start_function): Emit KCFI preambles.
(assemble_external_real): Emit KCFI typeid symbols.
(default_elf_asm_named_section): Handle .kcfi_traps using
SECTION_LINK_ORDER flag.
gcc/c-family/ChangeLog:
* c-attribs.cc: Include asan.h.
(handle_nocf_check_attribute): Enable nocf_check under kcfi.
(handle_patchable_function_entry_attribute): Add error for using
patchable_function_entry attribute with -fsanitize=kcfi.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/kcfi.h | 59 ++++
gcc/kcfi.cc | 696 ++++++++++++++++++++++++++++++++++++++
gcc/doc/extend.texi | 42 +++
gcc/doc/invoke.texi | 33 ++
gcc/doc/tm.texi | 32 ++
gcc/Makefile.in | 1 +
gcc/flag-types.h | 2 +
gcc/gimple.h | 22 ++
gcc/tree-pass.h | 1 +
gcc/c-family/c-attribs.cc | 17 +-
gcc/df-scan.cc | 7 +
gcc/doc/tm.texi.in | 12 +
gcc/final.cc | 3 +
gcc/opts.cc | 1 +
gcc/passes.cc | 1 +
gcc/passes.def | 1 +
gcc/rtl.def | 6 +
gcc/rtlanal.cc | 5 +
gcc/target.def | 39 +++
gcc/toplev.cc | 12 +
gcc/tree-inline.cc | 10 +
gcc/varasm.cc | 37 +-
22 files changed, 1028 insertions(+), 11 deletions(-)
create mode 100644 gcc/kcfi.h
create mode 100644 gcc/kcfi.cc
diff --git a/gcc/kcfi.h b/gcc/kcfi.h
new file mode 100644
index 000000000000..ea7e7881040f
--- /dev/null
+++ b/gcc/kcfi.h
@@ -0,0 +1,59 @@
+/* Kernel Control Flow Integrity (KCFI) support for GCC.
+ Copyright (C) 2025 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#ifndef GCC_KCFI_H
+#define GCC_KCFI_H
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "rtl.h"
+
+/* Common helper for RTL patterns to emit .kcfi_traps section entry.
+ Call after emitting trap label and instruction with the trap symbol
+ reference. */
+extern void kcfi_emit_traps_section (FILE *file, rtx trap_label_sym,
+ int labelno);
+
+/* Extract KCFI type ID from current GIMPLE statement. */
+extern rtx internal_kcfi_get_type_id_for_expanding_gimple_call (void);
+
+/* Convenience wrapper to check for SANITIZE_KCFI. */
+static inline rtx
+kcfi_get_type_id_for_expanding_gimple_call (void)
+{
+ if (!(flag_sanitize & SANITIZE_KCFI))
+ return NULL_RTX;
+ return internal_kcfi_get_type_id_for_expanding_gimple_call ();
+}
+
+/* Emit KCFI type ID symbol for external address-taken functions. */
+extern void kcfi_emit_typeid_symbol (FILE *asm_file, tree fndecl);
+
+/* Emit KCFI preamble for potential indirect call targets. */
+extern void kcfi_emit_preamble (FILE *asm_file, tree fndecl,
+ const char *actual_fname);
+
+/* Get next KCFI label number for trap/call/entry label numbering. */
+extern int kcfi_next_labelno (void);
+
+/* Get the KCFI typeid offset for calculating callsite typeid offset. */
+extern HOST_WIDE_INT kcfi_get_typeid_offset (void);
+
+#endif /* GCC_KCFI_H */
diff --git a/gcc/kcfi.cc b/gcc/kcfi.cc
new file mode 100644
index 000000000000..e61831eaa20d
--- /dev/null
+++ b/gcc/kcfi.cc
@@ -0,0 +1,696 @@
+/* Kernel Control Flow Integrity (KCFI) support for GCC.
+ Copyright (C) 2025 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+/* KCFI ABI Design:
+
+The Linux Kernel Control Flow Integrity ABI provides a function prototype
+based forward edge control flow integrity protection by instrumenting
+every indirect call to check for a hash value before the target function
+address. If the hash at the call site and the hash at the target do not
+match, execution will trap.
+
+The general CFI ideas are discussed here, but focuses more on a CFG
+analysis to construct valid call destinations, which tends to require LTO:
+https://users.soe.ucsc.edu/~abadi/Papers/cfi-tissec-revised.pdf
+
+Later refinement for using jump tables (constructed via CFG analysis
+during LTO) was proposed here:
+https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-tice.pdf
+
+Linux used the above implementation from 2018 to 2022:
+https://android-developers.googleblog.com/2018/10/control-flow-integrity-in-android-kernel.html
+but the corner cases for target addresses not being the actual functions
+(i.e. pointing into the jump table) was a continual source of problems,
+and generating the jump tables required full LTO, which had its own set
+of problems.
+
+Looking at function prototypes as the source of call validity was
+presented here, though still relied on LTO:
+https://www.blackhat.com/docs/asia-17/materials/asia-17-Moreira-Drop-The-Rop-Fine-Grained-Control-Flow-Integrity-For-The-Linux-Kernel-wp.pdf
+
+The KCFI approach built on the function-prototype idea, but avoided
+needing LTO, and could be further updated to deal with CPU errata
+(retpolines, etc):
+https://lpc.events/event/16/contributions/1315/
+
+KCFI has a number of specific constraints. Some are tied to the
+backend architecture, which are covered in arch-specific code.
+The constraints are:
+
+- The KCFI scheme generates a unique 32-bit hash ("typeid") for each
+ unique function prototype, allowing for indirect call sites to verify
+ that they are calling into a matching _type_ of function pointer.
+ This changes the semantics of some optimization logic because now
+ indirect calls to different types cannot be merged. For example:
+
+ if (p->func_type_1)
+ return p->func_type_1 ();
+ if (p->func_type_2)
+ return p->func_type_2 ();
+
+ In final asm, the optimizer may collapse the second indirect call
+ into a jump to the first indirect call once it has loaded the function
+ pointer. KCFI must block cross-type merging otherwise there will be a
+ single KCFI check happening for only 1 type but being used by 2 target
+ types. The distinguishing characteristic for call merging becomes the
+ type, not the address/register usage.
+
+- The check-call instruction sequence must be treated as a single unit: it
+ cannot be rearranged or split or optimized. The pattern is that
+ indirect calls, "call *%target", get converted into:
+
+ mov $target_expression, %target ; only present if the expression was
+ ; not already in %target register
+ load -$offset(%target), %tmp ; load typeid hash from target preamble
+ cmp $typeid, %tmp ; compare expected typeid with loaded
+ je .Lkcfi_call$N ; success: jump to the indirect call
+ .Lkcfi_trap$N: ; label of trap insn
+ trap ; trap on failure, but arranged so
+ ; "permissive mode" falls through
+ .Lkcfi_call$N: ; label of call insn
+ call *%target ; actual indirect call
+
+ This pattern of call immediately after trap provides for the
+ "permissive" checking mode automatically: the trap gets handled,
+ a warning emitted, and then execution continues after the trap to
+ the call.
+
+- KCFI check-call instrumentation must survive tail call optimization.
+ If an indirect call is turned into an indirect jump, KCFI checking
+ must still happen (but it will use a jmp rather than a call).
+
+- Functions that may be called indirectly have a preamble added,
+ __cfi_$original_func_name, which contains the $typeid value:
+
+ __cfi_target_func:
+ .word $typeid
+ target_func:
+ [regular function entry...]
+
+- The preamble needs to interact with patchable function entry so that
+ the typeid appears further away from the actual start of the function
+ (leaving the prefix NOPs of the patchable function entry unchanged).
+ This means only _globally defined_ patchable function entry is supported
+ with KCFI (indrect call sites must know in advance what the offset is,
+ which may not be possible with extern functions that use a function
+ attribute to change their patchable function entry characteristics).
+ For example, a "4,4" patchable function entry would end up like:
+
+ __cfi_target_func:
+ .data $typeid
+ nop nop nop nop
+ target_func:
+ [regular function entry...]
+
+ Architectures may need to add alignment nops prior to the typeid to keep
+ __cfi_target_func aligned for function call conventions.
+
+- An external function that is address-taken but does not have a definition has
+ a weak __kcfi_typeid_$func symbol added at the declaration site. This weak
+ symbol has the typeid value available so that the typeid can be referenced
+ from assembly linkages, etc, where the typeid values cannot be calculated
+ (i.e where C type information is missing):
+
+ .weak __kcfi_typeid_$func
+ .set __kcfi_typeid_$func, $typeid
+
+- On architectures that do not have a good way to encode additional
+ details in their trap insn (e.g. x86_64 and riscv64), the trap location
+ is identified as a KCFI trap via a relative address offset entry
+ emitted into the .kcfi_traps section for each indirect call site's
+ trap instruction. The previous check-call example's insn sequence would
+ then have section changes inserted between the trap and call:
+
+ ...
+ .Lkcfi_trap$N:
+ trap
+ .section .kcfi_traps,"ao",@progbits,.text
+ .Lkcfi_entry$N:
+ .long .Lkcfi_trap$N - .Lkcfi_entry$N
+ .text
+ .Lkcfi_call$N:
+ call *%target
+
+ It is up to such architectures to decode instructions prior to the
+ trap to locate the typeid that the callsite was expecting.
+
+ For architectures that can encode immediates in their trap function
+ (e.g. aarch64 and arm32), this isn't needed: they just use immediate
+ codes that indicate a KCFI trap.
+
+- The no_sanitize("kcfi") function attribute means that the marked
+ function must not produce KCFI checking for indirect calls, and this
+ attribute must survive inlining. This is used rarely by Linux, but
+ is required to make BPF JIT trampolines work on older Linux kernel
+ versions.
+
+- The "nocf_check" function attribute can be used to supress the
+ KCFI preamble for a function, making that function unavailable
+ for indirect calls.
+
+As a result of these constraints, there are some behavioral aspects
+that need to be preserved across the middle-end and back-end.
+
+For indirect call sites:
+
+- Make sure KCFI expansion is skipped for inline functions that
+ are marked with no_sanitize("kcfi") by marking these calls
+ during GIMPLE inlining with a new flag which is checked during
+ expansion.
+
+- All function types have their associated typeid attached as an
+ attribute during an IPA pass.
+
+- Keep typeid information available through to the RTL expansion
+ phase via a new KCFI insn RTL pattern that wraps the CALL
+ and the typeid as expressions. Since per-arch RTL call logic
+ can be very complex, we don't want to replace the "define_expand"
+ patterns, but rather hook the tail end of expansion. This is
+ where kcfi_get_type_id_for_expanding_gimple_call gets used, which
+ uses currently_expanding_gimple_stmt internally, as done in
+ other cases where GIMPLE needs to be examined during expansion.
+
+- Keep indirect calls from being merged (see earlier example)
+ naturally by having typeid be the second argument of the KCFI insn
+ RTL pattern, so jump2 pass's use of rtx_equal_p see differing
+ typeids in the RTL.
+
+- Update register liveness analysis to look inside the new KCFI
+ RTL to find the CALL RTL and examine the registers in use there
+ so the allocator can track register usage correctly.
+
+- KCFI insn emission interacts with patchable function entry to
+ load the typeid from the target preamble, offset by prefix NOPs.
+
+For indirect call targets:
+
+- kcfi_emit_preamble interacts with patchable function entry to add
+ any needed alignment padding prior to emitting the typeid.
+
+- assemble_external_real calls kcfi_emit_typeid_symbol to add the
+ __kcfi_typeid_$func symbols.
+
+*/
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "target.h"
+#include "function.h"
+#include "tree.h"
+#include "tree-pass.h"
+#include "dumpfile.h"
+#include "basic-block.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "cgraph.h"
+#include "kcfi.h"
+#include "stringpool.h"
+#include "attribs.h"
+#include "rtl.h"
+#include "cfg.h"
+#include "cfgrtl.h"
+#include "asan.h"
+#include "diagnostic-core.h"
+#include "memmodel.h"
+#include "print-tree.h"
+#include "emit-rtl.h"
+#include "output.h"
+#include "builtins.h"
+#include "varasm.h"
+#include "opts.h"
+#include "target.h"
+#include "flags.h"
+#include "kcfi-typeinfo.h"
+#include "insn-config.h"
+#include "recog.h"
+
+/* KCFI label counter, incremented by KCFI insn emission. */
+static int kcfi_labelno = 0;
+
+/* Get next KCFI label number. Returns the current KCFI label number
+ and increments the internal counter for the next call. The label is
+ used to provide a unique number for each indirect callsite of the
+ current module of the compilation. */
+
+int
+kcfi_next_labelno (void)
+{
+ return kcfi_labelno++;
+}
+
+/* Callsite typeid loading offset. */
+static HOST_WIDE_INT kcfi_typeid_offset = 0;
+
+/* Get the KCFI typeid offset. Returns the offset in bytes from the
+ function entry point where the KCFI type ID is stored. */
+
+HOST_WIDE_INT
+kcfi_get_typeid_offset (void)
+{
+ return kcfi_typeid_offset;
+}
+/* Count of needed __cfi_... preamble alignment padding NOPs. */
+static HOST_WIDE_INT kcfi_alignment_padding_nops = 0;
+/* NOP insn template. */
+static const char *kcfi_nop = NULL;
+
+/* Common helper for RTL patterns to emit .kcfi_traps section entry.
+ FILE is the output assembly file stream. TRAP_LABEL_SYM is the RTX
+ symbol reference for the trap instruction label. LABELNO is the KCFI
+ label number to use for the entry label. */
+
+void
+kcfi_emit_traps_section (FILE *file, rtx trap_label_sym, int labelno)
+{
+ /* Generate entry label name with custom prefix. */
+ char entry_name[32];
+ ASM_GENERATE_INTERNAL_LABEL (entry_name, "Lkcfi_entry", labelno);
+
+ /* Save current section to restore later. */
+ section *saved_section = in_section;
+
+ /* Use varasm infrastructure for section handling:
+ .section .kcfi_traps,"ao",@progbits,.text */
+ section *kcfi_traps_section = get_section (".kcfi_traps",
+ SECTION_LINK_ORDER, NULL);
+ switch_to_section (kcfi_traps_section);
+
+ /* Emit entry label for relative offset:
+ .Lkcfi_entry$N: */
+ ASM_OUTPUT_LABEL (file, entry_name);
+
+ /* Generate address difference using RTL infrastructure. */
+ rtx entry_label_sym = gen_rtx_SYMBOL_REF (Pmode, entry_name);
+ rtx addr_diff = gen_rtx_MINUS (Pmode, trap_label_sym, entry_label_sym);
+
+ /* Emit the address difference as a 4-byte value:
+ .long .Lkcfi_trap$N - .Lkcfi_entry$N */
+ assemble_integer (addr_diff, 4, BITS_PER_UNIT, 1);
+
+ /* Restore the previous section:
+ .text */
+ switch_to_section (saved_section);
+}
+
+/* Compute KCFI type ID for a function type. FNTYPE is the function
+ type tree node to compute the type ID for. Returns the 32-bit KCFI
+ type identifier hash. */
+
+static uint32_t
+compute_kcfi_type_id (tree fntype)
+{
+ gcc_assert (fntype);
+ gcc_assert (TREE_CODE (fntype) == FUNCTION_TYPE);
+
+ uint32_t type_id = typeinfo_get_hash (fntype);
+
+ /* Apply target-specific masking if supported. */
+ if (targetm.kcfi.mask_type_id)
+ type_id = targetm.kcfi.mask_type_id (type_id);
+
+ /* Output to dump file if enabled. */
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ {
+ std::string mangled_name = typeinfo_get_name (fntype);
+ fprintf (dump_file, "KCFI type ID: mangled='%s' typeid=0x%08x\n",
+ mangled_name.c_str (), type_id);
+ }
+
+ return type_id;
+}
+
+/* Function attribute to store KCFI type ID. */
+static tree kcfi_type_id_attr = NULL_TREE;
+
+/* Get KCFI type ID for a function type. Set it if missing. FN_TYPE
+ is the function type tree node. Returns the cached or newly computed
+ 32-bit KCFI type identifier, storing it as a type attribute. */
+
+static uint32_t
+kcfi_get_type_id (tree fn_type)
+{
+ uint32_t type_id;
+
+ /* Cache the attribute identifier for build_tree_list usage. */
+ if (!kcfi_type_id_attr)
+ kcfi_type_id_attr = get_identifier ("kcfi_type_id");
+
+ tree attr = lookup_attribute ("kcfi_type_id", TYPE_ATTRIBUTES (fn_type));
+ if (attr)
+ {
+ tree value = TREE_VALUE (attr);
+ gcc_assert (value && TREE_CODE (value) == INTEGER_CST);
+ type_id = (uint32_t) TREE_INT_CST_LOW (value);
+ }
+ else
+ {
+ type_id = compute_kcfi_type_id (fn_type);
+
+ tree type_id_tree = build_int_cst (unsigned_type_node, type_id);
+ tree attr = build_tree_list (kcfi_type_id_attr, type_id_tree);
+
+ TYPE_ATTRIBUTES (fn_type) = chainon (TYPE_ATTRIBUTES (fn_type), attr);
+ }
+
+ return type_id;
+}
+
+/* Prepare the global KCFI alignment NOPs calculation. Called once
+ during IPA pass to set global variables including kcfi_typeid_offset
+ and kcfi_alignment_padding_nops based on patchable function entry
+ settings and function alignment requirements. */
+
+static void
+kcfi_prepare_alignment_nops (void)
+{
+ /* Calculate where callsites load the 32-bit typeid from, based
+ on the target function address. Under default circumstances,
+ the typeid is located 4 bytes before the function entry, in
+ the __cfi_... preamble:
+
+ __cfi_func:
+ [typeid] // -4 from func
+ func:
+ [function body]
+
+ */
+ kcfi_typeid_offset = sizeof (uint32_t);
+
+ /* When Patchable Function Entry (PFE) is enabled, there may be NOP
+ instructions between the typeid and the function entry point.
+ Since KCFI callsites have no way to see PFE function attributes,
+ it can only base the calculations on the global
+ -fpatchable-function-entry=TOTAL[,PREFIX] flag. */
+ HOST_WIDE_INT prefix_nops = 0;
+ if (flag_patchable_function_entry)
+ {
+ HOST_WIDE_INT total_nops;
+ parse_and_check_patch_area (flag_patchable_function_entry, false,
+ &total_nops, &prefix_nops);
+ }
+
+ /* However, PFE is measured in NOP instruction counts, not bytes. So
+ we need to find out how many bytes they are, and we may need to
+ emit NOPs for alignment padding later. Prepare the NOP template. */
+ rtx_insn *nop_insn = make_insn_raw (gen_nop ());
+ int code_num = recog_memoized (nop_insn);
+ kcfi_nop = get_insn_template (code_num, nop_insn);
+ int nop_insn_bytes = get_attr_length (nop_insn);
+
+ /* Adjust the offset by how many NOP bytes may be between the typeid
+ and the function entry point.
+
+ __cfi_func:
+ [typeid] // -(4 + (prefix nop count * nop size)) from func
+ [prefix nops] // added when -fpatchable-function-entry is set
+ func:
+ [entry nops] // added when -fpatchable-function-entry is set
+ [function body]
+
+ At this point, kcfi_typeid_offset is ready and callsites can now
+ correctly find the typeid.
+
+ */
+ int prefix_nop_bytes = prefix_nops * nop_insn_bytes;
+ kcfi_typeid_offset += prefix_nop_bytes;
+
+ /* In the case where the KCFI preamble (and potentially the prefix NOPs)
+ are being used for alternative CFI implementations via live-patching,
+ the __cfi_... label itself needs to be usable as a callable function
+ target, so alignment NOPs may need to be added between the preamble
+ label and the typeid during KCFI preamble emission:
+
+ __cfi_func: // may need to be function entry aligned
+ [alignment padding nops] // may be needed when -falign-functions set
+ [typeid]
+ [prefix nops]
+ func:
+ [entry nops]
+ [function body]
+
+ But we only calculate alignment padding NOPs when -falign-functions
+ has been explicitly set.
+ */
+ if (align_functions.levels[0].log <= 0)
+ return;
+ int function_entry_alignment = align_functions.levels[0].get_value ();
+
+ /* Some architectures may be using an instruction for the typeid (though
+ this requires that the typeid is a trailing immediate value), but the
+ instruction will have a size greater than 4, which must be part of the
+ resulting alignment padding calculation. */
+ int typeid_insn_bytes = targetm.kcfi.emit_type_id
+ ? targetm.kcfi.emit_type_id (NULL, 0, NULL)
+ : sizeof (uint32_t);
+
+ /* Calculate needed architecture-specific alignment padding bytes. */
+ int needed_alignment_bytes = (function_entry_alignment
+ - ((prefix_nop_bytes + typeid_insn_bytes)
+ % function_entry_alignment))
+ % function_entry_alignment;
+
+ /* Calculate number of NOP instructions needed for alignment padding. */
+ if (needed_alignment_bytes % nop_insn_bytes != 0)
+ sorry ("KCFI function entry alignment padding bytes (%d) are not "
+ "a multiple of architecture NOP instruction size (%d)",
+ needed_alignment_bytes, nop_insn_bytes);
+ kcfi_alignment_padding_nops = needed_alignment_bytes / nop_insn_bytes;
+}
+
+/* Extract KCFI type ID from indirect call GIMPLE statement. Uses the
+ currently expanding GIMPLE statement to determine if KCFI instrumentation
+ is needed. Returns RTX constant with type ID, or NULL_RTX if no KCFI
+ instrumentation is required. */
+
+rtx
+internal_kcfi_get_type_id_for_expanding_gimple_call (void)
+{
+ gcc_assert (currently_expanding_gimple_stmt);
+ gcc_assert (is_gimple_call (currently_expanding_gimple_stmt));
+
+ /* Internally checks for no_sanitize("kcfi") with current_function_decl. */
+ if (!sanitize_flags_p (SANITIZE_KCFI))
+ return NULL_RTX;
+
+ gcall *call_stmt = as_a <gcall *> (currently_expanding_gimple_stmt);
+
+ /* Only indirect calls need KCFI instrumentation. */
+ if (gimple_call_fndecl (call_stmt))
+ return NULL_RTX;
+
+ /* Skip calls originating from inlined no_sanitize("kcfi") functions. */
+ if (gimple_call_inlined_from_kcfi_nosantize_p (call_stmt))
+ return NULL_RTX;
+
+ /* Get function type of call. */
+ tree fn_type = gimple_call_fntype (call_stmt);
+ gcc_assert (fn_type);
+
+ /* Return the type_id. */
+ return GEN_INT (kcfi_get_type_id (fn_type));
+}
+
+/* Emit KCFI type ID symbol for an address-taken external function.
+ ASM_FILE is the output assembly file stream. FNDECL is the external
+ function declaration tree node. Emits weak symbol definitions for
+ external functions that are address-taken. */
+
+void
+kcfi_emit_typeid_symbol (FILE *asm_file, tree fndecl)
+{
+ /* Only emit for external function declarations. */
+ if (TREE_CODE (fndecl) != FUNCTION_DECL || DECL_INITIAL (fndecl))
+ return;
+
+ /* Only emit for functions that are address-taken. */
+ struct cgraph_node *node = cgraph_node::get (fndecl);
+ if (!node || !node->address_taken)
+ return;
+
+ /* Get symbol name from RTL and strip encoding prefixes. */
+ rtx rtl = DECL_RTL (fndecl);
+ const char *name = XSTR (XEXP (rtl, 0), 0);
+ name = targetm.strip_name_encoding (name);
+
+ /* .weak __kcfi_typeid_{name} */
+ std::string symbol_name = std::string ("__kcfi_typeid_") + name;
+ ASM_WEAKEN_LABEL (asm_file, symbol_name.c_str ());
+
+ /* .set __kcfi_typeid_{name}, 0x{type_id} */
+ char val[16];
+ snprintf (val, sizeof (val), "0x%08x",
+ kcfi_get_type_id (TREE_TYPE (fndecl)));
+ ASM_OUTPUT_DEF (asm_file, symbol_name.c_str (), val);
+}
+
+/* Emit KCFI preamble before the function label. ASM_FILE is the output
+ assembly file stream. FNDECL is the function declaration tree node.
+ ACTUAL_FNAME is the actual function name to use, or NULL to use the
+ function's assembler name. Functions get preambles when -fsanitize=kcfi
+ is enabled, regardless of no_sanitize("kcfi") attribute. */
+
+void
+kcfi_emit_preamble (FILE *asm_file, tree fndecl, const char *actual_fname)
+{
+ /* Skip functions with nocf_check attribute. */
+ if (lookup_attribute ("nocf_check", TYPE_ATTRIBUTES (TREE_TYPE (fndecl))))
+ return;
+
+ struct cgraph_node *node = cgraph_node::get (fndecl);
+
+ /* Ignore cold partition functions: not reached via indirect call. */
+ if (node && node->split_part)
+ return;
+
+ /* Ignore cold partition sections: cold partitions are never indirect call
+ targets. Only skip preambles for cold partitions (has_bb_partition = true)
+ not for entire cold-attributed functions (has_bb_partition = false). */
+ if (in_cold_section_p && crtl && crtl->has_bb_partition)
+ return;
+
+ /* Check if function is truly address-taken using cgraph node analysis. */
+ bool addr_taken = (node && node->address_taken);
+
+ /* Only instrument functions that can be targets of indirect calls:
+ - Public functions (can be called externally)
+ - External declarations (from other modules)
+ - Functions with true address-taken status from cgraph analysis. */
+ if (!(TREE_PUBLIC (fndecl) || DECL_EXTERNAL (fndecl) || addr_taken))
+ return;
+
+ /* Use actual function name if provided, otherwise fall back to
+ DECL_ASSEMBLER_NAME. */
+ const char *fname = actual_fname
+ ? actual_fname
+ : IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (fndecl));
+
+ /* Create symbol name for reuse. */
+ std::string cfi_symbol_name = std::string ("__cfi_") + fname;
+
+ /* Emit __cfi_ symbol with proper visibility. */
+ if (TREE_PUBLIC (fndecl))
+ {
+ if (DECL_WEAK (fndecl))
+ ASM_WEAKEN_LABEL (asm_file, cfi_symbol_name.c_str ());
+ else
+ targetm.asm_out.globalize_label (asm_file, cfi_symbol_name.c_str ());
+ }
+
+ /* Emit .type directive. */
+ ASM_OUTPUT_TYPE_DIRECTIVE (asm_file, cfi_symbol_name.c_str (), "function");
+ ASM_OUTPUT_LABEL (asm_file, cfi_symbol_name.c_str ());
+
+ /* Emit any needed alignment padding NOPs using target's NOP template. */
+ for (int i = 0; i < kcfi_alignment_padding_nops; i++)
+ output_asm_insn (kcfi_nop, NULL);
+
+ /* Emit type ID bytes. */
+ uint32_t type_id = kcfi_get_type_id (TREE_TYPE (fndecl));
+ if (targetm.kcfi.emit_type_id)
+ targetm.kcfi.emit_type_id (asm_file, type_id, fndecl);
+ else
+ fprintf (asm_file, "\t.word\t0x%08x\n", type_id);
+
+ /* Mark end of __cfi_ symbol and emit size directive. */
+ std::string cfi_end_label = std::string (".Lcfi_func_end_") + fname;
+ ASM_OUTPUT_LABEL (asm_file, cfi_end_label.c_str ());
+
+ ASM_OUTPUT_MEASURED_SIZE (asm_file, cfi_symbol_name.c_str ());
+}
+
+namespace {
+
+/* IPA pass for KCFI type ID setting - runs once per compilation unit. */
+
+const pass_data pass_data_ipa_kcfi =
+{
+ SIMPLE_IPA_PASS, /* type */
+ "ipa_kcfi", /* name */
+ OPTGROUP_NONE, /* optinfo_flags */
+ TV_IPA_OPT, /* tv_id */
+ 0, /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0, /* todo_flags_finish */
+};
+
+/* Set KCFI type_ids for all usable function types in compilation unit.
+ Processes all functions in the current compilation unit and caches their
+ KCFI type identifiers. Returns 0 on completion. */
+
+static unsigned int
+ipa_kcfi_execute (void)
+{
+ struct cgraph_node *node;
+
+ /* Prepare global KCFI alignment NOPs calculation once for all functions. */
+ kcfi_prepare_alignment_nops ();
+
+ /* Process all functions - both local and external. */
+ FOR_EACH_FUNCTION (node)
+ {
+ tree fndecl = node->decl;
+
+ /* Skip all non-NORMAL builtins (MD, FRONTEND) entirely.
+ For NORMAL builtins, skip those that lack an implicit
+ implementation (closest way to distinguishing DEF_LIB_BUILTIN
+ from others). E.g. we need to have typeids for memset(). */
+ if (fndecl_built_in_p (fndecl))
+ {
+ if (DECL_BUILT_IN_CLASS (fndecl) != BUILT_IN_NORMAL)
+ continue;
+ if (!builtin_decl_implicit_p (DECL_FUNCTION_CODE (fndecl)))
+ continue;
+ }
+
+ /* Cache the type_id in the function type. */
+ kcfi_get_type_id (TREE_TYPE (fndecl));
+ }
+
+ return 0;
+}
+
+class pass_ipa_kcfi : public simple_ipa_opt_pass
+{
+public:
+ pass_ipa_kcfi (gcc::context *ctxt)
+ : simple_ipa_opt_pass (pass_data_ipa_kcfi, ctxt)
+ {}
+
+ bool gate (function *) final override
+ {
+ return sanitize_flags_p (SANITIZE_KCFI);
+ }
+
+ unsigned int execute (function *) final override
+ {
+ return ipa_kcfi_execute ();
+ }
+
+}; /* class pass_ipa_kcfi */
+
+} /* anon namespace */
+
+simple_ipa_opt_pass *
+make_pass_ipa_kcfi (gcc::context *ctxt)
+{
+ return new pass_ipa_kcfi (ctxt);
+}
diff --git a/gcc/doc/extend.texi b/gcc/doc/extend.texi
index c91b48bbdfe6..d7b3d9836df9 100644
--- a/gcc/doc/extend.texi
+++ b/gcc/doc/extend.texi
@@ -2789,6 +2789,44 @@ void __attribute__ ((no_sanitize ("alignment,object-size")))
g () @{ /* @r{Do something.} */; @}
@end smallexample
+When @code{no_sanitize ("kcfi")} is applied to a function, it disables
+the generation of Kernel Control Flow Integrity (KCFI) instrumentation
+for indirect function calls within that function. This means that
+indirect calls in the marked function will not be checked against the
+target function's type signature.
+
+However, the function itself will still receive a KCFI preamble (type
+identifier) when compiled with @option{-fsanitize=kcfi}, allowing it to
+be safely called indirectly from other functions that do perform KCFI
+checks. In other words, @code{no_sanitize ("kcfi")} affects outgoing
+calls from the function, not incoming calls to the function.
+
+@smallexample
+void __attribute__ ((no_sanitize ("kcfi")))
+trusted_function(void (*callback)(int))
+@{
+ /* This indirect call will NOT be instrumented with KCFI checks. */
+ callback (42);
+@}
+
+void regular_function(void (*callback)(int))
+@{
+ /* This indirect call WILL be instrumented with KCFI checks. */
+ callback (42);
+@}
+@end smallexample
+
+This attribute is primarily used in kernel code for special contexts such
+as BPF JIT trampolines or other low-level code where KCFI instrumentation
+might interfere with the intended operation. The attribute survives
+inlining to ensure that @code{no_sanitize("kcfi")} functions do not generate
+KCFI checks even when inlined into a function that otherwise performs KCFI
+checks.
+
+Note: To disable KCFI preamble generation for functions so that they may
+explicitly not be called indirectly, use the @code{nocf_check} function
+attribute instead.
+
@cindex @code{no_sanitize_address} function attribute
@item no_sanitize_address
@itemx no_address_safety_analysis
@@ -3161,6 +3199,10 @@ instrumentation on all functions that are part of the instrumentation
framework with the attribute @code{patchable_function_entry (0)}
to prevent recursion.
+This attribute cannot be used with @option{-fsanitize=kcfi} because KCFI
+callsites cannot know about function-specific patchable entry settings on
+a preamble in a different translation unit.
+
@cindex @code{pure} function attribute
@cindex functions that have no side effects
@item pure
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 60e740529b78..d11f8973ed1b 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -18661,6 +18661,39 @@ possible by specifying the command-line options
@option{--param hwasan-instrument-allocas=1} respectively. Using a random frame
tag is not implemented for kernel instrumentation.
+@opindex fsanitize=kcfi
+@item -fsanitize=kcfi
+Enable Kernel Control Flow Integrity (KCFI), a lightweight control
+flow integrity mechanism designed for operating system kernels.
+KCFI instruments indirect function calls to verify that the target
+function has the expected type signature at runtime. Each function
+receives a unique type identifier computed from a hash of its function
+prototype (including parameter types and return type). Before each
+indirect call, the implementation inserts a check to verify that the
+target function's type identifier matches the expected identifier
+for the call site, issuing a trap instruction if a mismatch is detected.
+This provides forward-edge control flow protection against attacks that
+attempt to redirect indirect calls to unintended targets.
+
+The implementation adds minimal runtime overhead and does not require
+runtime library support, making it suitable for kernel environments.
+The type identifier is placed before the function entry point,
+allowing runtime verification without additional metadata structures,
+and without changing the entry points of the target functions.
+
+KCFI is intended primarily for kernel code and may not be suitable
+for user-space applications that rely on techniques incompatible
+with strict type checking of indirect calls.
+
+Note that KCFI is incompatible with function-specific
+@code{patchable_function_entry} attributes because KCFI call sites
+cannot know about function-specific patchable entry settings in different
+translation units. Only the global @option{-fpatchable-function-entry}
+command-line option is supported with KCFI.
+
+Use @option{-fdump-ipa-kcfi-details} to examine the computed type identifier
+hashes and their corresponding mangled type strings during compilation.
+
@opindex fsanitize=pointer-compare
@item -fsanitize=pointer-compare
Instrument comparison operation (<, <=, >, >=) with pointer operands.
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index 25aad2eb74b6..e38dfba2d5ad 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -3175,6 +3175,7 @@ This describes the stack layout and calling conventions.
* Tail Calls::
* Shrink-wrapping separate components::
* Stack Smashing Protection::
+* Kernel Control Flow Integrity::
* Miscellaneous Register Hooks::
@end menu
@@ -5441,6 +5442,37 @@ should be allocated from heap memory and consumers should release them.
The result will be pruned to cases with PREFIX if not NULL.
@end deftypefn
+@node Kernel Control Flow Integrity
+@subsection Kernel Control Flow Integrity
+@cindex kernel control flow integrity
+@cindex KCFI
+
+@deftypefn {Target Hook} bool TARGET_KCFI_SUPPORTED (void)
+Return true if the target supports Kernel Control Flow Integrity (KCFI).
+This hook indicates whether the target has implemented the necessary RTL
+patterns and infrastructure to support KCFI instrumentation. The default
+implementation returns false.
+@end deftypefn
+
+@deftypefn {Target Hook} uint32_t TARGET_KCFI_MASK_TYPE_ID (uint32_t @var{type_id})
+Apply architecture-specific masking to KCFI type ID. This hook allows
+targets to apply bit masks or other transformations to the computed KCFI
+type identifier to match the target's specific requirements. The default
+implementation returns the type ID unchanged.
+@end deftypefn
+
+@deftypefn {Target Hook} int TARGET_KCFI_EMIT_TYPE_ID (FILE *@var{file}, uint32_t @var{type_id}, tree @var{fndecl})
+Emit architecture-specific type ID instruction for KCFI preambles
+and return the size of the instruction in bytes.
+@var{file} is the assembly output stream and @var{type_id} is the KCFI
+type identifier to emit. If @var{file} is NULL, skip emission and only
+return the size. If not overridden, the default fallback emits a
+@code{.word} directive with the type ID and returns 4 bytes. Targets can
+override this to emit different instruction sequences and return their
+corresponding sizes. @var{fndecl} is the function declaration, which
+targets can use to compute architecture-specific arity or other properties.
+@end deftypefn
+
@node Miscellaneous Register Hooks
@subsection Miscellaneous register hooks
@cindex miscellaneous register hooks
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index de41e77c5d00..1b5e63c29402 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1601,6 +1601,7 @@ OBJS = \
ira-lives.o \
jump.o \
kcfi-typeinfo.o \
+ kcfi.o \
langhooks.o \
late-combine.o \
lcm.o \
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index 44a90becb27a..5f9ea6a6d7b3 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -338,6 +338,8 @@ enum sanitize_code {
SANITIZE_KERNEL_HWADDRESS = 1UL << 30,
/* Shadow Call Stack. */
SANITIZE_SHADOW_CALL_STACK = 1UL << 31,
+ /* KCFI (Kernel Control Flow Integrity) */
+ SANITIZE_KCFI = 1ULL << 32,
SANITIZE_SHIFT = SANITIZE_SHIFT_BASE | SANITIZE_SHIFT_EXPONENT,
SANITIZE_UNDEFINED = SANITIZE_SHIFT | SANITIZE_DIVIDE | SANITIZE_UNREACHABLE
| SANITIZE_VLA | SANITIZE_NULL | SANITIZE_RETURN
diff --git a/gcc/gimple.h b/gcc/gimple.h
index b421850625e0..c554e8f64699 100644
--- a/gcc/gimple.h
+++ b/gcc/gimple.h
@@ -142,6 +142,7 @@ enum gf_mask {
GF_CALL_ALLOCA_FOR_VAR = 1 << 5,
GF_CALL_INTERNAL = 1 << 6,
GF_CALL_CTRL_ALTERING = 1 << 7,
+ GF_CALL_INLINED_FROM_KCFI_NOSANTIZE = 1 << 8,
GF_CALL_MUST_TAIL_CALL = 1 << 9,
GF_CALL_BY_DESCRIPTOR = 1 << 10,
GF_CALL_NOCF_CHECK = 1 << 11,
@@ -3487,6 +3488,27 @@ gimple_call_from_thunk_p (gcall *s)
return (s->subcode & GF_CALL_FROM_THUNK) != 0;
}
+/* If INLINED_FROM_KCFI_NOSANTIZE_P is true, mark GIMPLE_CALL S as being
+ inlined from a function with no_sanitize("kcfi"). */
+
+inline void
+gimple_call_set_inlined_from_kcfi_nosantize (gcall *s,
+ bool inlined_from_kcfi_nosantize_p)
+{
+ if (inlined_from_kcfi_nosantize_p)
+ s->subcode |= GF_CALL_INLINED_FROM_KCFI_NOSANTIZE;
+ else
+ s->subcode &= ~GF_CALL_INLINED_FROM_KCFI_NOSANTIZE;
+}
+
+/* Return true if GIMPLE_CALL S was inlined from a function with
+ no_sanitize("kcfi"). */
+
+inline bool
+gimple_call_inlined_from_kcfi_nosantize_p (const gcall *s)
+{
+ return (s->subcode & GF_CALL_INLINED_FROM_KCFI_NOSANTIZE) != 0;
+}
/* If FROM_NEW_OR_DELETE_P is true, mark GIMPLE_CALL S as being a call
to operator new or delete created from a new or delete expression. */
diff --git a/gcc/tree-pass.h b/gcc/tree-pass.h
index 410341d47119..60d20176dc76 100644
--- a/gcc/tree-pass.h
+++ b/gcc/tree-pass.h
@@ -543,6 +543,7 @@ extern ipa_opt_pass_d *make_pass_ipa_odr (gcc::context *ctxt);
extern ipa_opt_pass_d *make_pass_ipa_reference (gcc::context *ctxt);
extern ipa_opt_pass_d *make_pass_ipa_pure_const (gcc::context *ctxt);
extern simple_ipa_opt_pass *make_pass_ipa_pta (gcc::context *ctxt);
+extern simple_ipa_opt_pass *make_pass_ipa_kcfi (gcc::context *ctxt);
extern simple_ipa_opt_pass *make_pass_ipa_tm (gcc::context *ctxt);
extern simple_ipa_opt_pass *make_pass_target_clone (gcc::context *ctxt);
extern simple_ipa_opt_pass *make_pass_dispatcher_calls (gcc::context *ctxt);
diff --git a/gcc/c-family/c-attribs.cc b/gcc/c-family/c-attribs.cc
index 28a034f6c065..4daec35d8b73 100644
--- a/gcc/c-family/c-attribs.cc
+++ b/gcc/c-family/c-attribs.cc
@@ -48,6 +48,7 @@ along with GCC; see the file COPYING3. If not see
#include "gimplify.h"
#include "tree-pretty-print.h"
#include "gcc-rich-location.h"
+#include "asan.h"
#include "gcc-urlifier.h"
#include "attr-callback.h"
@@ -1767,8 +1768,11 @@ handle_nocf_check_attribute (tree *node, tree name,
warning (OPT_Wattributes, "%qE attribute ignored", name);
*no_add_attrs = true;
}
- else if (!(flag_cf_protection & CF_BRANCH))
+ else if (!(flag_cf_protection & CF_BRANCH)
+ && !(flag_sanitize & SANITIZE_KCFI))
{
+ /* Allow it with -fsanitize=kcfi, but leave this warning alone
+ to avoid confusion over this weird corner case. */
warning (OPT_Wattributes, "%qE attribute ignored. Use "
"%<-fcf-protection%> option to enable it",
name);
@@ -6651,6 +6655,17 @@ static tree
handle_patchable_function_entry_attribute (tree *, tree name, tree args,
int, bool *no_add_attrs)
{
+ /* Function-specific patchable_function_entry attribute is incompatible
+ with KCFI because KCFI callsites cannot know about function-specific
+ patchable entry settings on a preamble in a different translation
+ unit. */
+ if (sanitize_flags_p (SANITIZE_KCFI))
+ {
+ error ("%qE attribute cannot be used with %<-fsanitize=kcfi%>", name);
+ *no_add_attrs = true;
+ return NULL_TREE;
+ }
+
for (; args; args = TREE_CHAIN (args))
{
tree val = TREE_VALUE (args);
diff --git a/gcc/df-scan.cc b/gcc/df-scan.cc
index 1e4c6a2a4fb5..2be5e60786a3 100644
--- a/gcc/df-scan.cc
+++ b/gcc/df-scan.cc
@@ -2851,6 +2851,13 @@ df_uses_record (class df_collection_rec *collection_rec,
/* If we're clobbering a REG then we have a def so ignore. */
return;
+ case KCFI:
+ /* KCFI wraps other RTL - process the wrapped RTL. */
+ df_uses_record (collection_rec, &XEXP (x, 0), ref_type, bb, insn_info,
+ flags);
+ /* The type ID operand (XEXP (x, 1)) doesn't contain register uses. */
+ return;
+
case MEM:
df_uses_record (collection_rec,
&XEXP (x, 0), DF_REF_REG_MEM_LOAD,
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 24d47b19e5f1..dd1cbe01293d 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -2442,6 +2442,7 @@ This describes the stack layout and calling conventions.
* Tail Calls::
* Shrink-wrapping separate components::
* Stack Smashing Protection::
+* Kernel Control Flow Integrity::
* Miscellaneous Register Hooks::
@end menu
@@ -3816,6 +3817,17 @@ generic code.
@hook TARGET_GET_VALID_OPTION_VALUES
+@node Kernel Control Flow Integrity
+@subsection Kernel Control Flow Integrity
+@cindex kernel control flow integrity
+@cindex KCFI
+
+@hook TARGET_KCFI_SUPPORTED
+
+@hook TARGET_KCFI_MASK_TYPE_ID
+
+@hook TARGET_KCFI_EMIT_TYPE_ID
+
@node Miscellaneous Register Hooks
@subsection Miscellaneous register hooks
@cindex miscellaneous register hooks
diff --git a/gcc/final.cc b/gcc/final.cc
index afcb0bb9efbc..7f6aa9f9e480 100644
--- a/gcc/final.cc
+++ b/gcc/final.cc
@@ -2094,6 +2094,9 @@ call_from_call_insn (const rtx_call_insn *insn)
case SET:
x = XEXP (x, 1);
break;
+ case KCFI:
+ x = XEXP (x, 0);
+ break;
}
}
return x;
diff --git a/gcc/opts.cc b/gcc/opts.cc
index 38f96efe7669..f1b324c5a397 100644
--- a/gcc/opts.cc
+++ b/gcc/opts.cc
@@ -2182,6 +2182,7 @@ const struct sanitizer_opts_s sanitizer_opts[] =
SANITIZER_OPT (pointer-overflow, SANITIZE_POINTER_OVERFLOW, true, true),
SANITIZER_OPT (builtin, SANITIZE_BUILTIN, true, true),
SANITIZER_OPT (shadow-call-stack, SANITIZE_SHADOW_CALL_STACK, false, false),
+ SANITIZER_OPT (kcfi, SANITIZE_KCFI, false, true),
SANITIZER_OPT (all, ~sanitize_code_type (0), true, true),
#undef SANITIZER_OPT
{ NULL, sanitize_code_type (0), 0UL, false, false }
diff --git a/gcc/passes.cc b/gcc/passes.cc
index a33c8d924a52..4c6ceac740ff 100644
--- a/gcc/passes.cc
+++ b/gcc/passes.cc
@@ -63,6 +63,7 @@ along with GCC; see the file COPYING3. If not see
#include "diagnostic-core.h" /* for fnotice */
#include "stringpool.h"
#include "attribs.h"
+#include "kcfi.h"
/* Reserved TODOs */
#define TODO_verify_il (1u << 31)
diff --git a/gcc/passes.def b/gcc/passes.def
index fac04cd86c7d..3d17873d76da 100644
--- a/gcc/passes.def
+++ b/gcc/passes.def
@@ -52,6 +52,7 @@ along with GCC; see the file COPYING3. If not see
NEXT_PASS (pass_ipa_auto_profile_offline);
NEXT_PASS (pass_ipa_free_lang_data);
NEXT_PASS (pass_ipa_function_and_variable_visibility);
+ NEXT_PASS (pass_ipa_kcfi);
NEXT_PASS (pass_ipa_strub_mode);
NEXT_PASS (pass_build_ssa_passes);
PUSH_INSERT_PASSES_WITHIN (pass_build_ssa_passes)
diff --git a/gcc/rtl.def b/gcc/rtl.def
index 15ae7d10fcc1..af643d187b95 100644
--- a/gcc/rtl.def
+++ b/gcc/rtl.def
@@ -318,6 +318,12 @@ DEF_RTL_EXPR(CLOBBER, "clobber", "e", RTX_EXTRA)
DEF_RTL_EXPR(CALL, "call", "ee", RTX_EXTRA)
+/* KCFI wrapper for call expressions.
+ Operand 0 is the call expression.
+ Operand 1 is the KCFI type ID (const_int). */
+
+DEF_RTL_EXPR(KCFI, "kcfi", "ee", RTX_EXTRA)
+
/* Return from a subroutine. */
DEF_RTL_EXPR(RETURN, "return", "", RTX_EXTRA)
diff --git a/gcc/rtlanal.cc b/gcc/rtlanal.cc
index 63a1d08c46cf..5016fe93ccac 100644
--- a/gcc/rtlanal.cc
+++ b/gcc/rtlanal.cc
@@ -1177,6 +1177,11 @@ reg_referenced_p (const_rtx x, const_rtx body)
case IF_THEN_ELSE:
return reg_overlap_mentioned_p (x, body);
+ case KCFI:
+ /* For KCFI wrapper, check both the wrapped call and the type ID. */
+ return (reg_overlap_mentioned_p (x, XEXP (body, 0))
+ || reg_overlap_mentioned_p (x, XEXP (body, 1)));
+
case TRAP_IF:
return reg_overlap_mentioned_p (x, TRAP_CONDITION (body));
diff --git a/gcc/target.def b/gcc/target.def
index 61931278fd93..708489e99fce 100644
--- a/gcc/target.def
+++ b/gcc/target.def
@@ -7666,6 +7666,45 @@ DEFHOOKPOD
The default value is NULL.",
const char *, NULL)
+/* Kernel Control Flow Integrity (KCFI) hooks. */
+#undef HOOK_PREFIX
+#define HOOK_PREFIX "TARGET_KCFI_"
+HOOK_VECTOR (TARGET_KCFI, kcfi)
+
+DEFHOOK
+(supported,
+ "Return true if the target supports Kernel Control Flow Integrity (KCFI).\n\
+This hook indicates whether the target has implemented the necessary RTL\n\
+patterns and infrastructure to support KCFI instrumentation. The default\n\
+implementation returns false.",
+ bool, (void),
+ hook_bool_void_false)
+
+DEFHOOK
+(mask_type_id,
+ "Apply architecture-specific masking to KCFI type ID. This hook allows\n\
+targets to apply bit masks or other transformations to the computed KCFI\n\
+type identifier to match the target's specific requirements. The default\n\
+implementation returns the type ID unchanged.",
+ uint32_t, (uint32_t type_id),
+ NULL)
+
+DEFHOOK
+(emit_type_id,
+ "Emit architecture-specific type ID instruction for KCFI preambles\n\
+and return the size of the instruction in bytes.\n\
+@var{file} is the assembly output stream and @var{type_id} is the KCFI\n\
+type identifier to emit. If @var{file} is NULL, skip emission and only\n\
+return the size. If not overridden, the default fallback emits a\n\
+@code{.word} directive with the type ID and returns 4 bytes. Targets can\n\
+override this to emit different instruction sequences and return their\n\
+corresponding sizes. @var{fndecl} is the function declaration, which\n\
+targets can use to compute architecture-specific arity or other properties.",
+ int, (FILE *file, uint32_t type_id, tree fndecl),
+ NULL)
+
+HOOK_VECTOR_END (kcfi)
+
/* Close the 'struct gcc_target' definition. */
HOOK_VECTOR_END (C90_EMPTY_HACK)
diff --git a/gcc/toplev.cc b/gcc/toplev.cc
index bcf1b7604c83..65496818cf57 100644
--- a/gcc/toplev.cc
+++ b/gcc/toplev.cc
@@ -68,6 +68,7 @@ along with GCC; see the file COPYING3. If not see
#include "attribs.h"
#include "asan.h"
#include "tsan.h"
+#include "kcfi.h"
#include "plugin.h"
#include "context.h"
#include "pass_manager.h"
@@ -1741,6 +1742,17 @@ process_options ()
"requires %<-fno-exceptions%>");
}
+ if (flag_sanitize & SANITIZE_KCFI)
+ {
+ if (!targetm.kcfi.supported ())
+ sorry ("%<-fsanitize=kcfi%> not supported by this target");
+
+ /* Compiling with -flto results in frontend language of GNU GIMPLE.
+ Allow it since the original source was C. */
+ if (!lang_GNU_C () && !startswith (lang_hooks.name, "GNU GIMPLE"))
+ sorry ("%<-fsanitize=kcfi%> is only supported for C");
+ }
+
HOST_WIDE_INT patch_area_size, patch_area_start;
parse_and_check_patch_area (flag_patchable_function_entry, false,
&patch_area_size, &patch_area_start);
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 7fecf487af73..37eca44a5008 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -2110,6 +2110,16 @@ copy_bb (copy_body_data *id, basic_block bb,
/* Advance iterator now before stmt is moved to seq_gsi. */
gsi_next (&stmts_gsi);
+ /* If inlining from a function with no_sanitize("kcfi"), mark any
+ call statements in the inlined body with the flag so they skip
+ KCFI instrumentation. */
+ if (is_gimple_call (stmt)
+ && !sanitize_flags_p (SANITIZE_KCFI, id->src_fn))
+ {
+ gcall *call = as_a <gcall *> (stmt);
+ gimple_call_set_inlined_from_kcfi_nosantize (call, true);
+ }
+
if (gimple_nop_p (stmt))
continue;
diff --git a/gcc/varasm.cc b/gcc/varasm.cc
index 0d78f5b384fb..07f2127721b7 100644
--- a/gcc/varasm.cc
+++ b/gcc/varasm.cc
@@ -57,6 +57,7 @@ along with GCC; see the file COPYING3. If not see
#include "attribs.h"
#include "asan.h"
#include "rtl-iter.h"
+#include "kcfi.h"
#include "file-prefix-map.h" /* remap_debug_filename() */
#include "alloc-pool.h"
#include "toplev.h"
@@ -2199,6 +2200,10 @@ assemble_start_function (tree decl, const char *fnname)
unsigned short patch_area_size = crtl->patch_area_size;
unsigned short patch_area_entry = crtl->patch_area_entry;
+ /* Emit KCFI preamble before any patchable areas. */
+ if (flag_sanitize & SANITIZE_KCFI)
+ kcfi_emit_preamble (asm_out_file, decl, fnname);
+
/* Emit the patching area before the entry label, if any. */
if (patch_area_entry > 0)
targetm.asm_out.print_patchable_function_entry (asm_out_file,
@@ -2767,6 +2772,9 @@ assemble_external_real (tree decl)
/* Some systems do require some output. */
SYMBOL_REF_USED (XEXP (rtl, 0)) = 1;
ASM_OUTPUT_EXTERNAL (asm_out_file, decl, XSTR (XEXP (rtl, 0), 0));
+
+ if (flag_sanitize & SANITIZE_KCFI)
+ kcfi_emit_typeid_symbol (asm_out_file, decl);
}
}
#endif
@@ -7283,16 +7291,25 @@ default_elf_asm_named_section (const char *name, unsigned int flags,
fprintf (asm_out_file, ",%d", flags & SECTION_ENTSIZE);
if (flags & SECTION_LINK_ORDER)
{
- /* For now, only section "__patchable_function_entries"
- adopts flag SECTION_LINK_ORDER, internal label LPFE*
- was emitted in default_print_patchable_function_entry,
- just place it here for linked_to section. */
- gcc_assert (!strcmp (name, "__patchable_function_entries"));
- fprintf (asm_out_file, ",");
- char buf[256];
- ASM_GENERATE_INTERNAL_LABEL (buf, "LPFE",
- current_function_funcdef_no);
- assemble_name_raw (asm_out_file, buf);
+ if (!strcmp (name, "__patchable_function_entries"))
+ {
+ /* For patchable function entries, internal label LPFE*
+ was emitted in default_print_patchable_function_entry,
+ just place it here for linked_to section. */
+ fprintf (asm_out_file, ",");
+ char buf[256];
+ ASM_GENERATE_INTERNAL_LABEL (buf, "LPFE",
+ current_function_funcdef_no);
+ assemble_name_raw (asm_out_file, buf);
+ }
+ else if (!strcmp (name, ".kcfi_traps"))
+ {
+ /* KCFI traps section links to .text section. */
+ fprintf (asm_out_file, ",.text");
+ }
+ else
+ internal_error ("unexpected use of %<SECTION_LINK_ORDER%> by "
+ "section %qs", name);
}
if (HAVE_COMDAT_GROUP && (flags & SECTION_LINKONCE))
{
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v8 3/7] kcfi: Add regression test suite
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
2025-11-20 22:21 ` [PATCH v8 1/7] typeinfo: Introduce KCFI typeinfo mangling API Kees Cook
2025-11-20 22:21 ` [PATCH v8 2/7] kcfi: Add core Kernel Control Flow Integrity infrastructure Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
2025-11-20 22:21 ` [PATCH v8 4/7] x86: Add x86_64 Kernel Control Flow Integrity implementation Kees Cook
` (3 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
Add test suite for KCFI (Kernel Control Flow Integrity) ABI, covering
core functionality, optimization and code generation, addressing,
architecture-specific KCFI sequence emission, and integration with
patchable function entry.
The arch-specific patterns themselves are added with the subsequent
architecture patches.
Tests can be run via:
make check-c RUNTESTFLAGS='kcfi.exp'
gcc/testsuite/ChangeLog:
* lib/target-supports.exp: Add check_effective_target_kcfi.
* gcc.dg/kcfi/kcfi.exp: Add kcfi tests.
* gcc.dg/kcfi/kcfi-adjacency.c: New test.
* gcc.dg/kcfi/kcfi-basics.c: New test.
* gcc.dg/kcfi/kcfi-call-sharing.c: New test.
* gcc.dg/kcfi/kcfi-cold-partition.c: New test.
* gcc.dg/kcfi/kcfi-complex-addressing.c: New test.
* gcc.dg/kcfi/kcfi-ipa-robustness.c: New test.
* gcc.dg/kcfi/kcfi-move-preservation.c: New test.
* gcc.dg/kcfi/kcfi-no-sanitize-inline.c: New test.
* gcc.dg/kcfi/kcfi-no-sanitize.c: New test.
* gcc.dg/kcfi/kcfi-offset-validation.c: New test.
* gcc.dg/kcfi/kcfi-patchable-entry-only.c: New test.
* gcc.dg/kcfi/kcfi-patchable-incompatible.c: New test.
* gcc.dg/kcfi/kcfi-patchable-large.c: New test.
* gcc.dg/kcfi/kcfi-patchable-medium.c: New test.
* gcc.dg/kcfi/kcfi-patchable-prefix-only.c: New test.
* gcc.dg/kcfi/kcfi-runtime.c: New test.
* gcc.dg/kcfi/kcfi-tail-calls.c: New test.
* gcc.dg/kcfi/kcfi-trap-section.c: New test.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/testsuite/gcc.dg/kcfi/kcfi.exp | 51 ++++
gcc/testsuite/lib/target-supports.exp | 14 +
gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 49 ++++
gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 74 +++++
gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 61 ++++
.../gcc.dg/kcfi/kcfi-cold-partition.c | 126 ++++++++
.../gcc.dg/kcfi/kcfi-complex-addressing.c | 131 +++++++++
.../gcc.dg/kcfi/kcfi-complex-addressing.s | 0
.../gcc.dg/kcfi/kcfi-ipa-robustness.c | 54 ++++
.../gcc.dg/kcfi/kcfi-move-preservation.c | 41 +++
.../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 74 +++++
gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 31 ++
.../gcc.dg/kcfi/kcfi-offset-validation.c | 24 ++
.../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 14 +
.../gcc.dg/kcfi/kcfi-patchable-incompatible.c | 7 +
.../gcc.dg/kcfi/kcfi-patchable-large.c | 14 +
.../gcc.dg/kcfi/kcfi-patchable-medium.c | 14 +
.../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 14 +
gcc/testsuite/gcc.dg/kcfi/kcfi-runtime.c | 276 ++++++++++++++++++
gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 60 ++++
gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 17 ++
21 files changed, 1146 insertions(+)
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi.exp
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-cold-partition.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.s
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-ipa-robustness.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-incompatible.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-runtime.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi.exp b/gcc/testsuite/gcc.dg/kcfi/kcfi.exp
new file mode 100644
index 000000000000..131b24b8f140
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi.exp
@@ -0,0 +1,51 @@
+# Copyright (C) 2025 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING3. If not see
+# <http://www.gnu.org/licenses/>.
+
+# GCC testsuite for KCFI (Kernel Control Flow Integrity) tests.
+
+# Load support procs.
+load_lib gcc-dg.exp
+
+# Skip tests if KCFI is not supported on this target.
+if { ![check_effective_target_kcfi] } {
+ return
+}
+
+# Save DEFAULT_CFLAGS to restore later.
+global DEFAULT_CFLAGS
+if [info exists DEFAULT_CFLAGS] then {
+ set save_default_cflags $DEFAULT_CFLAGS
+}
+
+# Enable KCFI for all the kcfi/ tests.
+set DEFAULT_CFLAGS "-fsanitize=kcfi"
+
+# Initialize `dg'.
+dg-init
+
+# Main loop.
+dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/*.c]] \
+ "" $DEFAULT_CFLAGS
+
+# All done.
+dg-finish
+
+# Restore original DEFAULT_CFLAGS.
+if [info exists save_default_cflags] {
+ set DEFAULT_CFLAGS $save_default_cflags
+} else {
+ unset DEFAULT_CFLAGS
+}
diff --git a/gcc/testsuite/lib/target-supports.exp b/gcc/testsuite/lib/target-supports.exp
index b00e11e2ae52..0e56a104606e 100644
--- a/gcc/testsuite/lib/target-supports.exp
+++ b/gcc/testsuite/lib/target-supports.exp
@@ -14023,6 +14023,20 @@ proc check_effective_target_no_fsanitize_address {} {
return 0;
}
+# Return 1 if this target supports KCFI (Kernel Control Flow Integrity)
+
+proc check_effective_target_kcfi {} {
+ # We don't need a full execuable to test this flag
+ return [check_no_compiler_messages kcfi assembly {
+ void func(void) {}
+ int main(void) {
+ void (*ptr)(void) = func;
+ ptr();
+ return 0;
+ }
+ } "-fsanitize=kcfi"]
+}
+
# Return 1 if this target supports 'R' flag in .section directive, 0
# otherwise. Cache the result.
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
new file mode 100644
index 000000000000..7c1cff986c01
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
@@ -0,0 +1,49 @@
+/* Test KCFI check/transfer adjacency - regression test for instruction
+ insertion. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+/* This test ensures that KCFI security checks remain immediately adjacent
+ to their corresponding indirect calls/jumps, with no executable instructions
+ between the type ID check and the control flow transfer. */
+
+/* External function pointers to prevent optimization. */
+extern void (*complex_func_ptr)(int, int, int, int);
+extern int (*return_func_ptr)(int, int);
+
+/* Function with complex argument preparation that could tempt
+ the optimizer to insert instructions between KCFI check and call. */
+__attribute__((noinline)) void test_complex_args(int a, int b, int c, int d) {
+ /* Complex argument expressions that might cause instruction scheduling. */
+ complex_func_ptr(a * 2, b + c, d - a, (a << 1) | b);
+}
+
+/* Function with return value handling. */
+__attribute__((noinline)) int test_return_value(int x, int y) {
+ /* Return value handling that shouldn't interfere with adjacency. */
+ int result = return_func_ptr(x + 1, y * 2);
+ return result + 1;
+}
+
+/* Test struct field access that caused issues in try-catch.c. */
+struct call_info {
+ void (*handler)(void);
+ int status;
+ int data;
+};
+
+extern struct call_info *global_call_info;
+
+__attribute__((noinline)) void test_struct_field_call(void) {
+ /* This pattern caused adjacency issues before the fix. */
+ global_call_info->handler();
+}
+
+/* Test conditional indirect call. */
+__attribute__((noinline)) void test_conditional_call(int flag) {
+ if (flag) {
+ global_call_info->handler();
+ }
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
new file mode 100644
index 000000000000..ca833fed2971
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
@@ -0,0 +1,74 @@
+/* Test basic KCFI functionality - preamble generation. */
+/* { dg-do compile } */
+
+/* Extern function declarations - should NOT get KCFI preambles. */
+extern void external_func(void);
+extern int external_func_int(int x);
+
+void regular_function(int x) {
+ /* This should get KCFI preamble. */
+}
+
+void static_target_function(int x) {
+ /* Target function that can be called indirectly. */
+}
+
+__attribute__((nocf_check))
+void nocf_check_function(int x) {
+ /* This function has nocf_check attribute - should NOT get KCFI preamble. */
+}
+
+static void static_caller(void) {
+ /* Static function that makes an indirect call
+ Should NOT get KCFI preamble (not address-taken)
+ But must generate KCFI check for the indirect call. */
+ void (*local_ptr)(int) = static_target_function;
+ local_ptr(42); /* This should generate KCFI check. */
+}
+
+/* Make external_func address-taken. */
+void (*func_ptr)(int) = regular_function;
+void (*ext_ptr)(void) = external_func;
+void (__attribute__((nocf_check)) *nocf_ptr)(int) = nocf_check_function;
+
+int main() {
+ func_ptr(42);
+ ext_ptr(); /* Indirect call to external_func. */
+ external_func_int(10); /* Direct call to external_func_int. */
+ static_caller(); /* Direct call to static function. */
+ return 0;
+}
+
+/* Verify KCFI preamble exists for regular_function. */
+/* { dg-final { scan-assembler {__cfi_regular_function:} } } */
+
+/* Verify KCFI preamble symbol comes before main function symbol. */
+/* { dg-final { scan-assembler {__cfi_regular_function:.*regular_function:} } } */
+
+/* Target function should have preamble (address-taken). */
+/* { dg-final { scan-assembler {__cfi_static_target_function:} } } */
+
+/* Static caller should NOT have preamble (it's only called directly,
+ not address-taken). */
+/* { dg-final { scan-assembler-not {__cfi_static_caller:} } } */
+
+/* Function with nocf_check attribute should NOT have preamble. */
+/* { dg-final { scan-assembler-not {__cfi_nocf_check_function:} } } */
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
+
+/* Extern functions should NOT get KCFI preambles. */
+/* { dg-final { scan-assembler-not {__cfi_external_func:} } } */
+/* { dg-final { scan-assembler-not {__cfi_external_func_int:} } } */
+
+/* Local functions should NOT get __kcfi_typeid_ symbols. */
+/* Only external declarations that are address-taken should get __kcfi_typeid_ */
+/* { dg-final { scan-assembler-not {__kcfi_typeid_regular_function} } } */
+/* { dg-final { scan-assembler-not {__kcfi_typeid_main} } } */
+
+/* External address-taken functions should get __kcfi_typeid_ symbols. */
+/* { dg-final { scan-assembler {__kcfi_typeid_external_func} } } */
+
+/* External functions that are only called directly should NOT get
+ __kcfi_typeid_ symbols. */
+/* { dg-final { scan-assembler-not {__kcfi_typeid_external_func_int} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
new file mode 100644
index 000000000000..f72344f77124
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
@@ -0,0 +1,61 @@
+/* Test KCFI check sharing bug - optimizer incorrectly shares KCFI checks
+ between different function types. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+/* Reproduce the pattern from Linux kernel internal_create_group where:
+ - Two different function pointer types (is_visible vs is_bin_visible).
+ - Both get loaded into the same register (%rcx).
+ - Optimizer creates shared KCFI check with wrong type ID.
+ - This causes CFI failures in production kernel. */
+
+struct kobject { int dummy; };
+struct attribute { int dummy; };
+struct bin_attribute { int dummy; };
+
+struct attribute_group {
+ const char *name;
+ /* Type ID A. */
+ int (*is_visible)(struct kobject *, struct attribute *, int);
+ /* Type ID B. */
+ int (*is_bin_visible)(struct kobject *, const struct bin_attribute *, int);
+ /* Type ID B again. */
+ int (*is_bin_visible_again)(struct kobject *, const struct bin_attribute *, int);
+ struct attribute **attrs;
+ const struct bin_attribute **bin_attrs;
+};
+
+/* Function that mimics __first_visible from kernel - gets inlined into
+ caller. */
+static int __first_visible(const struct attribute_group *grp, struct kobject *kobj)
+{
+ /* Path 1: Call is_visible function pointer. */
+ if (grp->attrs && grp->attrs[0] && grp->is_visible)
+ return grp->is_visible(kobj, grp->attrs[0], 0);
+
+ /* Path 2: Call is_bin_visible function pointer. */
+ if (grp->bin_attrs && grp->bin_attrs[0] && grp->is_bin_visible)
+ return grp->is_bin_visible(kobj, grp->bin_attrs[0], 0);
+
+ /* Path 3: Call is_bin_visible_again function pointer. */
+ if (grp->bin_attrs && grp->bin_attrs[0] && grp->is_bin_visible_again)
+ return grp->is_bin_visible_again(kobj, grp->bin_attrs[0], 0);
+
+ return 0;
+}
+
+/* Main function that triggers the optimization bug. */
+int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *grp)
+{
+ /* This should inline __first_visible and create the problematic pattern where:
+ 1. Both function pointers get loaded into same register.
+ 2. Optimizer shares KCFI check between them.
+ 3. Uses wrong type ID for one of the calls. */
+ return __first_visible(grp, kobj);
+}
+
+/* Each indirect call should have its own KCFI check with correct type ID.
+
+ Should see:
+ 1. KCFI check for is_visible call with is_visible type ID A.
+ 2. KCFI check for is_bin_visible and is_bin_visible_again call with type ID B. */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-cold-partition.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-cold-partition.c
new file mode 100644
index 000000000000..87ffdba1f0ae
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-cold-partition.c
@@ -0,0 +1,126 @@
+/* Test KCFI cold function and cold partition behavior. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+/* { dg-additional-options "-freorder-blocks-and-partition" { target freorder } } */
+
+void regular_function(void) {
+ /* Regular function should get preamble. */
+}
+
+/* Cold-attributed function should STILL get preamble (it's a regular
+ function, just marked cold). */
+__attribute__((cold))
+void cold_attributed_function(void) {
+ /* This function has cold attribute but should still get KCFI preamble. */
+}
+
+/* Hot-attributed function should get preamble. */
+__attribute__((hot))
+void hot_attributed_function(void) {
+ /* This function is explicitly hot and should get KCFI preamble. */
+}
+
+/* Global to prevent optimization from eliminating cold paths. */
+extern void abort(void);
+
+/* Additional function to test that normal functions still get preambles. */
+__attribute__((noinline))
+int another_regular_function(int x) {
+ return x + 42;
+}
+
+/* Function designed to generate cold partitions under optimization. */
+__attribute__((noinline))
+void function_with_cold_partition(int condition) {
+ /* Hot path - very likely to execute. */
+ if (__builtin_expect(condition == 42, 1)) {
+ /* Simple hot path that optimizer will keep inline. */
+ return;
+ }
+
+ /* Cold paths that actually do something to prevent elimination. */
+ if (__builtin_expect(condition < 0, 0)) {
+ /* Error path 1 - call abort to prevent elimination. */
+ abort();
+ }
+
+ if (__builtin_expect(condition > 1000000, 0)) {
+ /* Error path 2 - call abort to prevent elimination. */
+ abort();
+ }
+
+ if (__builtin_expect(condition == 999999, 0)) {
+ /* Error path 3 - more substantial cold code. */
+ volatile int sum = 0;
+ for (volatile int i = 0; i < 100; i++) {
+ sum += i * condition;
+ }
+ if (sum > 0)
+ abort();
+ }
+
+ /* More cold paths - switch with many unlikely cases. */
+ switch (condition) {
+ case 1000001: case 1000002: case 1000003: case 1000004: case 1000005:
+ case 1000006: case 1000007: case 1000008: case 1000009: case 1000010:
+ /* Each case does some work before abort. */
+ volatile int work = condition * 2;
+ if (work > 0) abort();
+ break;
+ default:
+ if (condition != 42) {
+ /* Fallback cold path - substantial work. */
+ volatile int result = 0;
+ for (volatile int j = 0; j < condition % 50; j++) {
+ result += j;
+ }
+ if (result >= 0) abort();
+ }
+ }
+}
+
+/* Test function pointers to ensure address-taken detection works. */
+void test_function_pointers(void) {
+ void (*regular_ptr)(void) = regular_function;
+ void (*cold_ptr)(void) = cold_attributed_function;
+ void (*hot_ptr)(void) = hot_attributed_function;
+
+ regular_ptr();
+ cold_ptr();
+ hot_ptr();
+}
+
+int main() {
+ regular_function();
+ cold_attributed_function();
+ hot_attributed_function();
+ function_with_cold_partition(42); /* Normal case - stay in hot path. */
+ another_regular_function(5);
+ test_function_pointers();
+ return 0;
+}
+
+/* Regular function should have preamble. */
+/* { dg-final { scan-assembler "__cfi_regular_function:" } } */
+
+/* Cold-attributed function should STILL have preamble (it's a legitimate function) */
+/* { dg-final { scan-assembler "__cfi_cold_attributed_function:" } } */
+
+/* Hot-attributed function should have preamble. */
+/* { dg-final { scan-assembler "__cfi_hot_attributed_function:" } } */
+
+/* Function that generates cold partitions should have preamble for main entry. */
+/* { dg-final { scan-assembler "__cfi_function_with_cold_partition:" } } */
+
+/* Address-taken functions should have preambles. */
+/* { dg-final { scan-assembler "__cfi_test_function_pointers:" } } */
+
+/* The function should generate a .cold partition (only on targets that support freorder) */
+/* { dg-final { scan-assembler "function_with_cold_partition\\.cold:" { target freorder } } } */
+
+/* The .cold partition should NOT get a __cfi_ preamble since it's never
+ reached via indirect calls. */
+/* { dg-final { scan-assembler-not "__cfi_function_with_cold_partition\\.cold:" { target freorder } } } */
+
+/* Additional regular function should get preamble. */
+/* { dg-final { scan-assembler "__cfi_another_regular_function:" } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
new file mode 100644
index 000000000000..c48b8d7ad552
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
@@ -0,0 +1,131 @@
+/* Test KCFI with complex addressing modes (structure members, array
+ elements). This is a regression test for the change_address_1 RTL
+ error that occurred when target_addr was PLUS(reg, offset) instead
+ of a simple register. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+struct function_table {
+ int (*callback1)(int);
+ int (*callback2)(int, int);
+ void (*callback3)(void);
+ int (*callback4)(void *, void *, void *, void *, void *, void *);
+ int data;
+};
+
+static int handler1(int x) {
+ return x * 2;
+}
+
+static int handler2(int x, int y) {
+ return x + y;
+}
+
+static void handler3(void) {
+ /* Empty handler. */
+}
+
+/* Test indirect calls through structure members - this creates
+ PLUS(reg, offset) addressing. */
+__attribute__((noinline))
+int test_struct_members(struct function_table *table) {
+ int result = 0;
+ int loop;
+
+ /* These indirect calls will generate complex addressing modes:
+ * call *(%rdi) - callback1 at offset 0
+ * call *8(%rdi) - callback2 at offset 8
+ * call *16(%rdi) - callback3 at offset 16
+ * KCFI must handle PLUS(reg, struct_offset) + kcfi_offset. */
+
+ for (loop = 0; loop < 16; loop++) {
+ result += table->callback1(10);
+ result += table->callback2(5, 7);
+ table->callback3();
+ result += table->callback4(handler1, handler2, handler3, &result, test_struct_members, table);
+ }
+
+ return result;
+}
+
+/* Test indirect calls through array elements - another source of
+ complex addressing. */
+typedef int (*func_array_t)(int);
+
+int test_array_elements(func_array_t functions[], int index) {
+ /* This creates addressing like MEM[PLUS(PLUS(reg, index*8), 0)]
+ which should be simplified to MEM[PLUS(reg, index*8)]. */
+ return functions[index](42);
+}
+
+/* Test with global structure. */
+static struct function_table global_table = {
+ .callback1 = handler1,
+ .callback2 = handler2,
+ .callback3 = handler3,
+ .data = 100
+};
+
+int test_global_struct(void) {
+ /* Access through global structure - may generate different
+ addressing patterns. */
+ return global_table.callback1(20) + global_table.callback2(3, 4);
+}
+
+/* Test nested structure access. */
+struct nested_table {
+ struct function_table inner;
+ int extra_data;
+};
+
+int test_nested_struct(struct nested_table *nested) {
+ /* Even more complex addressing: nested structure member access. */
+ return nested->inner.callback1(15);
+}
+
+int test_many_args(void *one, void *two, void *three, void *four, void *five, void *six)
+{
+ return (unsigned long)one + (unsigned long)two + (unsigned long)three
+ + (unsigned long)four + (unsigned long)five + (unsigned long)six;
+}
+
+int target_func(int a, int b, int c, int d) { return a + b + c + d; }
+
+/* Function to force r3 spill/reload by using 4+ arguments in indirect call */
+__attribute__((noinline))
+int force_r3_spill(int (*fp)(int, int, int, int)) {
+ int (*ip_reg)(int, int, int, int) = fp;
+ volatile int val = 0;
+ /* This should force r3 as scratch since ip is the target */
+ return ip_reg(val, val, val, val) + 5;
+}
+
+int main() {
+ struct function_table local_table = {
+ .callback1 = handler1,
+ .callback2 = handler2,
+ .callback3 = handler3,
+ .callback4 = test_many_args,
+ .data = 50
+ };
+
+ func_array_t func_array[] = { handler1, handler1, handler1 };
+
+ int result = 0;
+ result += test_struct_members(&local_table);
+ result += test_array_elements(func_array, 1);
+ result += test_global_struct();
+
+ struct nested_table nested = { .inner = local_table, .extra_data = 200 };
+ result += test_nested_struct(&nested);
+
+ /* Force r3 spill/reload pattern in ARM32 KCFI */
+ volatile int (*four_arg_ptr)(int, int, int, int) = target_func;
+ result += force_r3_spill(four_arg_ptr);
+
+ result += local_table.callback4(handler1, handler2, handler3, &result, main, &local_table);
+
+ return result;
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.s b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.s
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-ipa-robustness.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-ipa-robustness.c
new file mode 100644
index 000000000000..a43bcd4f3e3f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-ipa-robustness.c
@@ -0,0 +1,54 @@
+/* Test KCFI IPA pass robustness with compiler-generated constructs. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+#include <stddef.h>
+
+/* Test various compiler-generated constructs that could confuse IPA pass. */
+
+/* static_assert - this was causing the original crash. */
+typedef struct {
+ int field1;
+ char field2;
+} test_struct_t;
+
+static_assert(offsetof(test_struct_t, field1) == 0, "layout check 1");
+static_assert(offsetof(test_struct_t, field2) == 4, "layout check 2");
+static_assert(sizeof(test_struct_t) >= 5, "size check");
+
+/* Regular functions that should get KCFI analysis. */
+void regular_function(void) {
+ /* Should get KCFI preamble. */
+}
+
+static void static_function(void) {
+ /* With -O2: correctly identified as not address-taken, no preamble. */
+}
+
+void address_taken_function(void) {
+ /* Should get KCFI preamble (address taken below) */
+}
+
+/* Function pointer to create address-taken scenario. */
+void (*func_ptr)(void) = address_taken_function;
+
+/* More static_asserts mixed with function definitions. */
+static_assert(sizeof(void*) >= 4, "pointer size check");
+
+int main(void) {
+ regular_function(); /* Direct call. */
+ static_function(); /* Direct call to static. */
+ func_ptr(); /* Indirect call. */
+
+ static_assert(sizeof(int) == 4, "int size check");
+
+ return 0;
+}
+
+/* Verify KCFI preambles are generated appropriately. */
+/* { dg-final { scan-assembler "__cfi_regular_function:" } } */
+/* { dg-final { scan-assembler "__cfi_address_taken_function:" } } */
+/* { dg-final { scan-assembler "__cfi_main:" } } */
+
+/* With -O2: static_function correctly identified as not address-taken. */
+/* { dg-final { scan-assembler-not "__cfi_static_function:" } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
new file mode 100644
index 000000000000..7d58fef3f920
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
@@ -0,0 +1,41 @@
+/* Test that KCFI preserves function pointer moves at -O2 optimization.
+ This test ensures that the combine pass doesn't incorrectly optimize away
+ the move instruction needed to transfer function pointers from argument
+ registers to the target registers used by KCFI patterns. */
+
+/* { dg-do compile } */
+/* { dg-additional-options "-O2 -std=gnu11" } */
+
+static int called_count = 0;
+
+/* Function taking one argument, returning void. */
+static __attribute__((noinline)) void increment_void(int *counter)
+{
+ (*counter)++;
+}
+
+/* Function taking one argument, returning int. */
+static __attribute__((noinline)) int increment_int(int *counter)
+{
+ (*counter)++;
+ return *counter;
+}
+
+/* Don't allow the compiler to inline the calls. */
+static __attribute__((noinline)) void indirect_call(void (*func)(int *))
+{
+ func(&called_count);
+}
+
+int main(void)
+{
+ /* This should work - matching prototype. */
+ indirect_call(increment_void);
+
+ /* This should trap - mismatched prototype. */
+ indirect_call((void *)increment_int);
+
+ return 0;
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
new file mode 100644
index 000000000000..4a90390d1934
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
@@ -0,0 +1,74 @@
+/* Test that no_sanitize("kcfi") attribute is preserved during inlining. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+extern void external_side_effect(int value);
+
+/* Regular function (should get KCFI checks) */
+__attribute__((noinline))
+void normal_function(void (*callback)(int))
+{
+ /* This indirect call must generate KCFI checks. */
+ callback(300);
+ external_side_effect(300);
+}
+
+/* Regular function marked with no_sanitize("kcfi") (positive control) */
+__attribute__((noinline, no_sanitize("kcfi")))
+void sensitive_non_inline_function(void (*callback)(int))
+{
+ /* This indirect call should NOT generate KCFI checks. */
+ callback(100);
+ external_side_effect(100);
+}
+
+/* Function marked with both no_sanitize("kcfi") and always_inline. */
+__attribute__((always_inline, no_sanitize("kcfi")))
+static inline void sensitive_inline_function(void (*callback)(int))
+{
+ /* This indirect call should NOT generate KCFI checks when inlined. */
+ callback(42);
+ external_side_effect(42);
+}
+
+/* Explicit wrapper for testing sensitive_inline_function behavior. */
+__attribute__((noinline))
+void wrap_sensitive_inline(void (*callback)(int))
+{
+ sensitive_inline_function(callback);
+}
+
+/* Function marked with only always_inline (should get KCFI checks) */
+__attribute__((always_inline))
+static inline void normal_inline_function(void (*callback)(int))
+{
+ /* This indirect call must generate KCFI checks when inlined. */
+ callback(200);
+ external_side_effect(200);
+}
+
+/* Explicit wrapper for testing normal_inline_function behavior. */
+__attribute__((noinline))
+void wrap_normal_inline(void (*callback)(int))
+{
+ normal_inline_function(callback);
+}
+
+void test_callback(int value)
+{
+ external_side_effect(value);
+}
+
+static void (*volatile function_pointer)(int) = test_callback;
+
+int main(void)
+{
+ void (*fn_ptr)(int) = function_pointer;
+
+ normal_function(fn_ptr);
+ wrap_normal_inline(fn_ptr);
+ sensitive_non_inline_function(fn_ptr);
+ wrap_sensitive_inline(fn_ptr);
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
new file mode 100644
index 000000000000..124d26488635
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
@@ -0,0 +1,31 @@
+/* Test KCFI with no_sanitize attribute. */
+/* { dg-do compile } */
+
+void target_function(void) {
+ /* This should get KCFI preamble. */
+}
+
+void caller_with_checks(void) {
+ /* This function should generate KCFI checks. */
+ void (*func_ptr)(void) = target_function;
+ func_ptr();
+}
+
+__attribute__((no_sanitize("kcfi")))
+void caller_no_checks(void) {
+ /* This function should NOT generate KCFI checks due to no_sanitize. */
+ void (*func_ptr)(void) = target_function;
+ func_ptr();
+}
+
+int main() {
+ caller_with_checks(); /* This should generate checks inside. */
+ caller_no_checks(); /* This should NOT generate checks inside. */
+ return 0;
+}
+
+/* All functions should get preambles regardless of no_sanitize. */
+/* { dg-final { scan-assembler "__cfi_target_function:" } } */
+/* { dg-final { scan-assembler "__cfi_caller_with_checks:" } } */
+/* { dg-final { scan-assembler "__cfi_caller_no_checks:" } } */
+/* { dg-final { scan-assembler "__cfi_main:" } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
new file mode 100644
index 000000000000..213a1a2892a5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
@@ -0,0 +1,24 @@
+/* Test KCFI call-site offset validation across architectures. */
+/* { dg-do compile } */
+
+void target_func_a(void) { }
+void target_func_b(int x) { }
+void target_func_c(int x, int y) { }
+
+int main() {
+ void (*ptr_a)(void) = target_func_a;
+ void (*ptr_b)(int) = target_func_b;
+ void (*ptr_c)(int, int) = target_func_c;
+
+ /* Multiple indirect calls. */
+ ptr_a();
+ ptr_b(1);
+ ptr_c(1, 2);
+
+ return 0;
+}
+
+/* Should have KCFI preambles for all functions. */
+/* { dg-final { scan-assembler "__cfi_target_func_a:" } } */
+/* { dg-final { scan-assembler "__cfi_target_func_b:" } } */
+/* { dg-final { scan-assembler "__cfi_target_func_c:" } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
new file mode 100644
index 000000000000..a6a2f4816fef
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
@@ -0,0 +1,14 @@
+/* Test KCFI with patchable function entries - entry NOPs only. */
+/* { dg-do compile } */
+/* { dg-additional-options "-fpatchable-function-entry=4,0" } */
+
+void test_function(void) {
+}
+
+int main() {
+ void (*func_ptr)(void) = test_function;
+ func_ptr();
+ return 0;
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-incompatible.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-incompatible.c
new file mode 100644
index 000000000000..c6cf9ab720a3
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-incompatible.c
@@ -0,0 +1,7 @@
+/* The patchable_function_entry attribute is incompatible with KCFI. */
+/* { dg-do compile } */
+
+__attribute__((patchable_function_entry(4, 2)))
+int test_function(void) { /* { dg-error "'patchable_function_entry' attribute cannot be used with '-fsanitize=kcfi'" } */
+ return 42;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
new file mode 100644
index 000000000000..8c4ec30cecc5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
@@ -0,0 +1,14 @@
+/* Test KCFI with large patchable function entries. */
+/* { dg-do compile } */
+/* { dg-additional-options "-fpatchable-function-entry=11,11" } */
+
+void test_function(void) {
+}
+
+int main() {
+ void (*func_ptr)(void) = test_function;
+ func_ptr();
+ return 0;
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
new file mode 100644
index 000000000000..78a834ef2a97
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
@@ -0,0 +1,14 @@
+/* Test KCFI with medium patchable function entries. */
+/* { dg-do compile } */
+/* { dg-additional-options "-fpatchable-function-entry=8,4" } */
+
+void test_function(void) {
+}
+
+int main() {
+ void (*func_ptr)(void) = test_function;
+ func_ptr();
+ return 0;
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
new file mode 100644
index 000000000000..1a4d8269ed56
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
@@ -0,0 +1,14 @@
+/* Test KCFI with patchable function entries - prefix NOPs only. */
+/* { dg-do compile } */
+/* { dg-additional-options "-fpatchable-function-entry=3,3" } */
+
+void test_function(void) {
+}
+
+int main() {
+ void (*func_ptr)(void) = test_function;
+ func_ptr();
+ return 0;
+}
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-runtime.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-runtime.c
new file mode 100644
index 000000000000..7593a421a4c1
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-runtime.c
@@ -0,0 +1,276 @@
+/* Test KCFI runtime behavior: working calls and type mismatch trapping. */
+/* { dg-do run { target native } } */
+/* { dg-additional-options "-O2" } */
+
+#include <stdio.h>
+#include <signal.h>
+#include <setjmp.h>
+#include <stdlib.h>
+#include <string.h>
+
+/* Test functions with different signatures */
+static int func_int_void(void)
+{
+ return 42;
+}
+
+__attribute__((nocf_check))
+static int func_int_void_nocf_check(void)
+{
+ return 42;
+}
+
+static int func_int_int(int x)
+{
+ return x * 4;
+}
+
+/* Complex functions with many arguments to create register pressure */
+static long complex_calc_8args(int a, long b, int c, long d,
+ int e, long f, int g, long h)
+{
+ /* Do actual work with all arguments to keep them live */
+ /* Each calculation uses specific operations to ensure values are preserved */
+ long result = (a * 2) + (b / 3) - (c * 4) + (d / 5) +
+ (e * 6) - (f / 7) + (g * 8) - (h / 9);
+
+ return result;
+}
+
+static int complex_intermediate_6args(int x1, int x2, int x3,
+ int x4, int x5, int x6)
+{
+ /* Keep variables live by using them in multiple calculations */
+ int temp1 = x1 + x2;
+ int temp2 = x3 * x4;
+ int temp3 = x5 - x6;
+
+ /* Call another function with many args through pointer */
+ typedef long (*calc_ptr)(int, long, int, long, int, long, int, long);
+ volatile calc_ptr ptr = complex_calc_8args;
+
+ /* Pass derived values to force register preservation */
+ long result = ptr(temp1, temp2, temp3, x1 * x2,
+ x3 + x4, x5 * x6, temp1 + temp2, temp3 - x1);
+
+ /* Use original args again to ensure they stayed live */
+ return (int)(result + x1 - x2 + x3 - x4 + x5 - x6);
+}
+
+static int complex_outer_4args(int a, int b, int c, int d)
+{
+ /* Create local variables that must be preserved */
+ int local1 = a * a;
+ int local2 = b * b;
+ int local3 = c * c;
+ int local4 = d * d;
+ volatile int force_spill1 = local1 + local2;
+ volatile int force_spill2 = local3 + local4;
+
+ /* Call through function pointer with many args */
+ typedef int (*inter_ptr)(int, int, int, int, int, int);
+ volatile inter_ptr ptr = complex_intermediate_6args;
+
+ /* Pass combinations that require preserving originals */
+ int result = ptr(local1, local2, local3, local4,
+ a + b, c + d);
+
+ /* Force use of spilled values */
+ result += force_spill1 + force_spill2;
+
+ /* Use original arguments again */
+ return result + a - b + c - d;
+}
+
+/* Entry point for complex call chain */
+static int complex_entry_point(void)
+{
+ /* Start with many live values */
+ int v1 = 10, v2 = 20, v3 = 30, v4 = 40;
+ int v5 = 50, v6 = 60, v7 = 70, v8 = 80;
+
+ /* Keep them live with volatile stores */
+ volatile int keep_live1 = v1 + v5;
+ volatile int keep_live2 = v2 + v6;
+ volatile int keep_live3 = v3 + v7;
+ volatile int keep_live4 = v4 + v8;
+
+ /* Call through function pointer */
+ typedef int (*outer_ptr)(int, int, int, int);
+ volatile outer_ptr ptr = complex_outer_4args;
+
+ /* Use derived values to maintain register pressure */
+ int result = ptr(v1 * v2, v3 * v4, v5 * v6, v7 * v8);
+
+ /* Force original values to stay live */
+ result += keep_live1 + keep_live2 + keep_live3 + keep_live4;
+ result += v1 + v2 + v3 + v4 + v5 + v6 + v7 + v8;
+
+ /* Return the calculated result */
+ return result;
+}
+
+/* Global state for signal handling */
+static volatile int trap_occurred = 0;
+static jmp_buf trap_env;
+
+/* Signal handler for KCFI traps */
+static void trap_handler(int sig)
+{
+ trap_occurred = 1;
+ longjmp(trap_env, 1);
+}
+
+/* Compatible indirect call should work - simple version */
+static int test_compatible_call(void)
+{
+ typedef int (*int_void_ptr)(void);
+ volatile int_void_ptr ptr = func_int_void;
+
+ fprintf(stderr, "Calling %s(0x%08x) through %s(0x%08x) ...\n",
+ __builtin_typeinfo_name(typeof(func_int_void)),
+ __builtin_typeinfo_hash(typeof(func_int_void)),
+ __builtin_typeinfo_name(typeof(*ptr)),
+ __builtin_typeinfo_hash(typeof(*ptr)));
+
+ trap_occurred = 0;
+ /* This should work - same signature */
+ int result = ptr();
+
+ return (trap_occurred == 0 && result == 42) ? 1 : 0;
+}
+
+/* Compatible indirect call with complex register pressure */
+static int test_compatible_call_complex(void)
+{
+ typedef int (*int_void_ptr)(void);
+ volatile int_void_ptr ptr = complex_entry_point;
+
+ fprintf(stderr, "Calling complex chain %s(0x%08x) through %s(0x%08x) ...\n",
+ __builtin_typeinfo_name(typeof(complex_entry_point)),
+ __builtin_typeinfo_hash(typeof(complex_entry_point)),
+ __builtin_typeinfo_name(typeof(*ptr)),
+ __builtin_typeinfo_hash(typeof(*ptr)));
+
+ trap_occurred = 0;
+
+ /* This should work - complex call chain with register pressure */
+ int result = ptr();
+
+ /* Verify the complete call chain computed correctly with all register
+ values preserved through the high-pressure call sequence */
+ if (result != 657383831) {
+ fprintf(stderr, "ERROR: Incorrect final result %d (expected 657383831)\n", result);
+ return 0;
+ }
+
+ /* Check that no trap occurred and result is correct */
+ return (trap_occurred == 0) ? 1 : 0;
+}
+
+/* Compatible indirect call to nocf_check should not work */
+static int test_nocf_check_trap(void)
+{
+ trap_occurred = 0;
+
+ if (setjmp(trap_env) == 0) {
+ typedef int (__attribute__((nocf_check)) *int_void_ptr_nocf)(void);
+ volatile int_void_ptr_nocf ptr = func_int_void_nocf_check;
+
+ fprintf(stderr, "Calling %s(0x%08x) through %s(0x%08x) ...\n",
+ __builtin_typeinfo_name(typeof(func_int_void_nocf_check)),
+ __builtin_typeinfo_hash(typeof(func_int_void_nocf_check)),
+ __builtin_typeinfo_name(typeof(*ptr)),
+ __builtin_typeinfo_hash(typeof(*ptr)));
+
+ int result = ptr();
+
+ fprintf(stderr, "Yikes! Survived nocf_check call\n");
+
+ /* If we get here, the trap didn't occur */
+ return 0;
+ } else {
+ /* We caught the trap - this is expected */
+ return trap_occurred;
+ }
+}
+
+/* Type mismatch should trap */
+static int test_type_mismatch_trap(void)
+{
+ trap_occurred = 0;
+
+ if (setjmp(trap_env) == 0) {
+ /* Cast func_int_void to incompatible void(*)(void) type */
+ typedef void (*void_void_ptr)(void);
+ volatile void_void_ptr ptr = (void_void_ptr)func_int_void;
+
+ fprintf(stderr, "Calling %s(0x%08x) through %s(0x%08x) ...\n",
+ __builtin_typeinfo_name(typeof(func_int_void)),
+ __builtin_typeinfo_hash(typeof(func_int_void)),
+ __builtin_typeinfo_name(typeof(*ptr)),
+ __builtin_typeinfo_hash(typeof(*ptr)));
+
+ /* This should trap because type IDs don't match:
+ - func_int_void has type ID for int(void)
+ - but we're calling through void(void) pointer type */
+ ptr();
+
+ fprintf(stderr, "Yikes! Survived mismatched call\n");
+
+ /* If we get here, the trap didn't occur */
+ return 0;
+ } else {
+ /* We caught the trap - this is expected */
+ return trap_occurred;
+ }
+}
+
+int main(void)
+{
+ struct sigaction sa = {
+ .sa_handler = trap_handler,
+ .sa_flags = SA_NODEFER,
+ };
+ int failed = 4;
+
+ /* Install trap handler. */
+ if (sigaction(SIGILL, &sa, NULL)) {
+ perror("sigaction");
+ return 1;
+ }
+
+ /* Simple compatible call should work */
+ if (test_compatible_call()) {
+ printf("OK: simple matched indirect call succeeded\n");
+ failed--;
+ } else {
+ printf("FAIL: simple matched call\n");
+ }
+
+ /* Complex compatible call chain should work */
+ if (test_compatible_call_complex()) {
+ printf("OK: complex matched indirect call chain succeeded\n");
+ failed--;
+ } else {
+ printf("FAIL: complex matched call chain\n");
+ }
+
+ /* Using nocf_check should trap */
+ if (test_nocf_check_trap()) {
+ printf("OK: indirect call to nocf_check correctly trapped\n");
+ failed--;
+ } else {
+ printf("FAIL: nocf_check trap\n");
+ }
+
+ /* Type mismatch should trap */
+ if (test_type_mismatch_trap()) {
+ printf("OK: mismatched indirect call correctly trapped\n");
+ failed--;
+ } else {
+ printf("FAIL: type mismatch trap\n");
+ }
+
+ return failed;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
new file mode 100644
index 000000000000..9ddf178aa2b1
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
@@ -0,0 +1,60 @@
+/* Test KCFI protection when indirect calls get converted to tail calls. */
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+typedef int (*func_ptr_t)(int);
+typedef void (*void_func_ptr_t)(void);
+
+struct function_table {
+ func_ptr_t process;
+ void_func_ptr_t cleanup;
+};
+
+/* Target functions. */
+int process_data(int x) { return x * 2; }
+void cleanup_data(void) {}
+
+/* Initialize function table. */
+volatile struct function_table vtable = {
+ .process = &process_data,
+ .cleanup = &cleanup_data
+};
+
+/* Indirect call through struct member that should become tail call. */
+int test_struct_indirect_call(int x) {
+ /* This is an indirect call that should be converted to tail call:
+ Without -fno-optimize-sibling-calls should become "jmp *vtable+0(%rip)"
+ With -fno-optimize-sibling-calls should become "call *vtable+0(%rip)" */
+ return vtable.process(x);
+}
+
+/* Indirect call through function pointer parameter. */
+int test_param_indirect_call(func_ptr_t handler, int x) {
+ /* This is an indirect call that should be converted to tail call:
+ Without -fno-optimize-sibling-calls should become "jmp *%rdi"
+ With -fno-optimize-sibling-calls should be "call *%rdi" */
+ return handler(x);
+}
+
+/* Void indirect call through struct member. */
+void test_void_indirect_call(void) {
+ /* This is an indirect call that should be converted to tail call:
+ * Without -fno-optimize-sibling-calls: should become "jmp *vtable+8(%rip)"
+ * With -fno-optimize-sibling-calls: should be "call *vtable+8(%rip)" */
+ vtable.cleanup();
+}
+
+/* Non-tail call for comparison (should always be call). */
+int test_non_tail_indirect_call(func_ptr_t handler, int x) {
+ /* This should never become a tail call - always "call *%rdi" */
+ int result = handler(x);
+ return result + 1; /* Prevents tail call optimization. */
+}
+
+/* Should have KCFI preambles for all functions. */
+/* { dg-final { scan-assembler-times "__cfi_process_data:" 1 } } */
+/* { dg-final { scan-assembler-times "__cfi_cleanup_data:" 1 } } */
+/* { dg-final { scan-assembler-times "__cfi_test_struct_indirect_call:" 1 } } */
+/* { dg-final { scan-assembler-times "__cfi_test_param_indirect_call:" 1 } } */
+/* { dg-final { scan-assembler-times "__cfi_test_void_indirect_call:" 1 } } */
+/* { dg-final { scan-assembler-times "__cfi_test_non_tail_indirect_call:" 1 } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
new file mode 100644
index 000000000000..6d34ad6e1a0c
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
@@ -0,0 +1,17 @@
+/* Test KCFI trap section generation. */
+/* { dg-do compile } */
+
+void target_function(void) {}
+
+int main() {
+ void (*func_ptr)(void) = target_function;
+
+ /* Multiple indirect calls to generate multiple trap entries. */
+ func_ptr();
+ func_ptr();
+
+ return 0;
+}
+
+/* Should have KCFI preamble. */
+/* { dg-final { scan-assembler "__cfi_target_function:" } } */
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v8 4/7] x86: Add x86_64 Kernel Control Flow Integrity implementation
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
` (2 preceding siblings ...)
2025-11-20 22:21 ` [PATCH v8 3/7] kcfi: Add regression test suite Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
2025-11-20 22:21 ` [PATCH v8 5/7] aarch64: Add AArch64 " Kees Cook
` (2 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
Implement x86_64-specific KCFI backend:
- Implies -mindirect-branch-register since KCFI needs call target in
a register for typeid hash loading.
- Function preamble generation with type IDs positioned at -(4+prefix_nops)
offset from function entry point.
- Function-aligned KCFI preambles using calculated alignment padding NOPs:
aligned(prefix_nops + 5, $func_align) to maintain ability to call the
__cfi_ preamble directly in the case of Linux's FineIBT alternative
CFI sequences (live patched into place).
- Type-id hash avoids generating ENDBR instruction in type IDs
(0xfa1e0ff3/0xfb1e0ff3 are incremented by 1 to prevent execution).
- On-demand scratch register allocation strategy (r11 as needed), with
the clobbers added when KCFI is used.
- Incompatible with -ffixed-r10 or -ffixed-r11.
- Uses the .kcfi_traps section for debugger/runtime metadata.
- Introduces -fsanitize-kcfi-arity to enable function arg count to be
represented in the kcfi hash preamble for FineIBT.
Assembly Code Pattern layout required by Linux kernel:
movl $inverse_type_id, %r10d ; Load expected type (0 - hash)
addl offset(%target), %r10d ; Add stored type ID from preamble
je .Lkcfi_call ; Branch if types match (sum == 0)
.Lkcfi_trap: ud2 ; Undefined instruction trap on mismatch
.Lkcfi_call: call/jmp *%target ; Execute validated indirect transfer
Build and run tested on x86_64 Linux kernel with various CPU errata
handling alternatives, with and without FineIBT patching.
gcc/ChangeLog:
config/i386/i386.h: KCFI enables TARGET_INDIRECT_BRANCH_REGISTER.
config/i386/i386-protos.h: Update ix86_output_call_insn.
config/i386/i386-expand.cc (ix86_expand_call): Expand indirect
calls into KCFI RTL.
config/i386/i386.cc (ix86_kcfi_mask_type_id): New function.
(ix86_output_call_insn): Emit KCFI assembly.
config/i386/i386.md: Add KCFI RTL substitution patterns.
doc/invoke.texi: Document x86 nuances.
gcc/testsuite/ChangeLog:
* gcc.dg/kcfi/kcfi-adjacency.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-basics.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-call-sharing.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-complex-addressing.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-move-preservation.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-no-sanitize.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-offset-validation.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-patchable-large.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-patchable-medium.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add x86 patterns.
* gcc.dg/kcfi/kcfi-tail-calls.c: Add x86 tail-call patterns.
* gcc.dg/kcfi/kcfi-trap-section.c: Add x86 trap patterns.
* gcc.dg/kcfi/kcfi-x86-arity.c: New test.
* gcc.dg/kcfi/kcfi-x86-fixed-r10.c: New test.
* gcc.dg/kcfi/kcfi-x86-fixed-r11.c: New test.
* gcc.dg/kcfi/kcfi-x86-retpoline-r11.c: New test.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/config/i386/i386-protos.h | 2 +-
gcc/config/i386/i386.h | 3 +-
gcc/config/i386/i386.md | 63 ++++--
gcc/config/i386/i386-expand.cc | 22 +-
gcc/config/i386/i386-options.cc | 13 ++
gcc/config/i386/i386.cc | 189 +++++++++++++++++-
gcc/doc/invoke.texi | 46 +++++
gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 17 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 15 ++
.../gcc.dg/kcfi/kcfi-complex-addressing.c | 18 ++
.../gcc.dg/kcfi/kcfi-move-preservation.c | 19 ++
.../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 11 +
gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 5 +
.../gcc.dg/kcfi/kcfi-offset-validation.c | 5 +
.../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 24 +++
.../gcc.dg/kcfi/kcfi-patchable-large.c | 13 ++
.../gcc.dg/kcfi/kcfi-patchable-medium.c | 20 ++
.../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 21 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 20 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 6 +
gcc/testsuite/gcc.dg/kcfi/kcfi-x86-arity.c | 93 +++++++++
.../gcc.dg/kcfi/kcfi-x86-fixed-r10.c | 17 ++
.../gcc.dg/kcfi/kcfi-x86-fixed-r11.c | 17 ++
.../gcc.dg/kcfi/kcfi-x86-retpoline-r11.c | 40 ++++
gcc/common.opt | 8 +
gcc/opts.cc | 1 +
27 files changed, 702 insertions(+), 27 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-x86-arity.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r10.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r11.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-x86-retpoline-r11.c
diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
index 5ff414a22a2a..e426b2f89a11 100644
--- a/gcc/config/i386/i386-protos.h
+++ b/gcc/config/i386/i386-protos.h
@@ -377,7 +377,7 @@ extern enum attr_cpu ix86_schedule;
#endif
extern bool ix86_nopic_noplt_attribute_p (rtx call_op);
-extern const char * ix86_output_call_insn (rtx_insn *insn, rtx call_op);
+extern const char * ix86_output_call_insn (rtx_insn *insn, rtx *operands);
extern const char * ix86_output_indirect_jmp (rtx call_op);
extern const char * ix86_output_function_return (bool long_p);
extern const char * ix86_output_indirect_function_return (rtx ret_op);
diff --git a/gcc/config/i386/i386.h b/gcc/config/i386/i386.h
index b93411796af3..49f8951d6d46 100644
--- a/gcc/config/i386/i386.h
+++ b/gcc/config/i386/i386.h
@@ -3058,7 +3058,8 @@ extern void debug_dispatch_window (int);
#define TARGET_INDIRECT_BRANCH_REGISTER \
(ix86_indirect_branch_register \
- || cfun->machine->indirect_branch_type != indirect_branch_keep)
+ || cfun->machine->indirect_branch_type != indirect_branch_keep \
+ || (flag_sanitize & SANITIZE_KCFI))
#define IX86_HLE_ACQUIRE (1 << 16)
#define IX86_HLE_RELEASE (1 << 17)
diff --git a/gcc/config/i386/i386.md b/gcc/config/i386/i386.md
index 6af7dcfcdd32..23c896fd7e7b 100644
--- a/gcc/config/i386/i386.md
+++ b/gcc/config/i386/i386.md
@@ -20431,11 +20431,19 @@
DONE;
})
-(define_insn "*call"
+(define_subst_attr "kcfi" "kcfi_subst" "" "_kcfi")
+
+(define_subst "kcfi_subst"
+ [(match_operand 0)]
+ ""
+ [(kcfi (match_dup 0)
+ (match_operand 2 "const_int_operand"))])
+
+(define_insn "*call<kcfi>"
[(call (mem:QI (match_operand:W 0 "call_insn_operand" "<c>BwBz"))
(match_operand 1))]
"!SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[0]);"
+ "* return ix86_output_call_insn (insn, operands);"
[(set_attr "type" "call")])
;; This covers both call and sibcall since only GOT slot is allowed.
@@ -20446,7 +20454,8 @@
"TARGET_X32 && !TARGET_INDIRECT_BRANCH_REGISTER"
{
rtx fnaddr = gen_const_mem (DImode, XEXP (operands[0], 0));
- return ix86_output_call_insn (insn, fnaddr);
+ rtx fn_operands[] = { fnaddr };
+ return ix86_output_call_insn (insn, fn_operands);
}
[(set_attr "type" "call")])
@@ -20464,15 +20473,17 @@
{
rtx fnaddr = gen_rtx_PLUS (SImode, operands[0], operands[1]);
fnaddr = gen_const_mem (SImode, fnaddr);
- return ix86_output_call_insn (insn, fnaddr);
+ rtx fn_operands[] = { fnaddr };
+ return ix86_output_call_insn (insn, fn_operands);
}
[(set_attr "type" "call")])
-(define_insn "*sibcall"
+;; KCFI sibling call
+(define_insn "*sibcall<kcfi>"
[(call (mem:QI (match_operand:W 0 "sibcall_insn_operand" "UBsBz"))
(match_operand 1))]
"SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[0]);"
+ "* return ix86_output_call_insn (insn, operands);"
[(set_attr "type" "call")])
(define_insn "*sibcall_memory"
@@ -20480,7 +20491,7 @@
(match_operand 1))
(unspec [(const_int 0)] UNSPEC_PEEPSIB)]
"!TARGET_X32 && !TARGET_INDIRECT_BRANCH_REGISTER"
- "* return ix86_output_call_insn (insn, operands[0]);"
+ "* return ix86_output_call_insn (insn, operands);"
[(set_attr "type" "call")])
(define_peephole2
@@ -20531,7 +20542,7 @@
(plus:SI (reg:SI SP_REG)
(match_operand:SI 2 "immediate_operand" "i")))]
"!TARGET_64BIT && !SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[0]);"
+ "* return ix86_output_call_insn (insn, operands);"
[(set_attr "type" "call")])
(define_insn "*sibcall_pop"
@@ -20541,7 +20552,7 @@
(plus:SI (reg:SI SP_REG)
(match_operand:SI 2 "immediate_operand" "i")))]
"!TARGET_64BIT && SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[0]);"
+ "* return ix86_output_call_insn (insn, operands);"
[(set_attr "type" "call")])
(define_insn "*sibcall_pop_memory"
@@ -20552,7 +20563,7 @@
(match_operand:SI 2 "immediate_operand" "i")))
(unspec [(const_int 0)] UNSPEC_PEEPSIB)]
"!TARGET_64BIT && !TARGET_INDIRECT_BRANCH_REGISTER"
- "* return ix86_output_call_insn (insn, operands[0]);"
+ "* return ix86_output_call_insn (insn, operands);"
[(set_attr "type" "call")])
(define_peephole2
@@ -20629,12 +20640,22 @@
DONE;
})
-(define_insn "*call_value"
+(define_subst_attr "kcfiv" "kcfiv_subst" "" "_kcfi")
+
+(define_subst "kcfiv_subst"
+ [(set (match_operand 0)
+ (match_operand 1))]
+ ""
+ [(set (match_dup 0)
+ (kcfi (match_dup 1)
+ (match_operand 3 "const_int_operand")))])
+
+(define_insn "*call_value<kcfiv>"
[(set (match_operand 0)
(call (mem:QI (match_operand:W 1 "call_insn_operand" "<c>BwBz"))
(match_operand 2)))]
"!SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[1]);"
+ "* return ix86_output_call_insn (insn, &operands[1]);"
[(set_attr "type" "callv")])
;; This covers both call and sibcall since only GOT slot is allowed.
@@ -20647,7 +20668,8 @@
"TARGET_X32 && !TARGET_INDIRECT_BRANCH_REGISTER"
{
rtx fnaddr = gen_const_mem (DImode, XEXP (operands[1], 0));
- return ix86_output_call_insn (insn, fnaddr);
+ rtx fn_operands[] = { fnaddr };
+ return ix86_output_call_insn (insn, fn_operands);
}
[(set_attr "type" "callv")])
@@ -20666,16 +20688,17 @@
{
rtx fnaddr = gen_rtx_PLUS (SImode, operands[1], operands[2]);
fnaddr = gen_const_mem (SImode, fnaddr);
- return ix86_output_call_insn (insn, fnaddr);
+ rtx fn_operands[] = { fnaddr };
+ return ix86_output_call_insn (insn, fn_operands);
}
[(set_attr "type" "callv")])
-(define_insn "*sibcall_value"
+(define_insn "*sibcall_value<kcfiv>"
[(set (match_operand 0)
(call (mem:QI (match_operand:W 1 "sibcall_insn_operand" "UBsBz"))
(match_operand 2)))]
"SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[1]);"
+ "* return ix86_output_call_insn (insn, &operands[1]);"
[(set_attr "type" "callv")])
(define_insn "*sibcall_value_memory"
@@ -20684,7 +20707,7 @@
(match_operand 2)))
(unspec [(const_int 0)] UNSPEC_PEEPSIB)]
"!TARGET_X32 && !TARGET_INDIRECT_BRANCH_REGISTER"
- "* return ix86_output_call_insn (insn, operands[1]);"
+ "* return ix86_output_call_insn (insn, &operands[1]);"
[(set_attr "type" "callv")])
(define_peephole2
@@ -20741,7 +20764,7 @@
(plus:SI (reg:SI SP_REG)
(match_operand:SI 3 "immediate_operand" "i")))]
"!TARGET_64BIT && !SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[1]);"
+ "* return ix86_output_call_insn (insn, &operands[1]);"
[(set_attr "type" "callv")])
(define_insn "*sibcall_value_pop"
@@ -20752,7 +20775,7 @@
(plus:SI (reg:SI SP_REG)
(match_operand:SI 3 "immediate_operand" "i")))]
"!TARGET_64BIT && SIBLING_CALL_P (insn)"
- "* return ix86_output_call_insn (insn, operands[1]);"
+ "* return ix86_output_call_insn (insn, &operands[1]);"
[(set_attr "type" "callv")])
(define_insn "*sibcall_value_pop_memory"
@@ -20764,7 +20787,7 @@
(match_operand:SI 3 "immediate_operand" "i")))
(unspec [(const_int 0)] UNSPEC_PEEPSIB)]
"!TARGET_64BIT && !TARGET_INDIRECT_BRANCH_REGISTER"
- "* return ix86_output_call_insn (insn, operands[1]);"
+ "* return ix86_output_call_insn (insn, &operands[1]);"
[(set_attr "type" "callv")])
(define_peephole2
diff --git a/gcc/config/i386/i386-expand.cc b/gcc/config/i386/i386-expand.cc
index c131f7c44c11..4f0557f40ac2 100644
--- a/gcc/config/i386/i386-expand.cc
+++ b/gcc/config/i386/i386-expand.cc
@@ -94,6 +94,7 @@ along with GCC; see the file COPYING3. If not see
#include "i386-builtins.h"
#include "i386-expand.h"
#include "asan.h"
+#include "kcfi.h"
/* Split one or more double-mode RTL references into pairs of half-mode
references. The RTL can be REG, offsettable MEM, integer constant, or
@@ -11071,8 +11072,9 @@ ix86_expand_call (rtx retval, rtx fnaddr, rtx callarg1,
unsigned int vec_len = 0;
tree fndecl;
bool call_no_callee_saved_registers = false;
+ bool is_direct_call = SYMBOL_REF_P (XEXP (fnaddr, 0));
- if (SYMBOL_REF_P (XEXP (fnaddr, 0)))
+ if (is_direct_call)
{
fndecl = SYMBOL_REF_DECL (XEXP (fnaddr, 0));
if (fndecl)
@@ -11109,7 +11111,7 @@ ix86_expand_call (rtx retval, rtx fnaddr, rtx callarg1,
if (TARGET_MACHO && !TARGET_64BIT)
{
#if TARGET_MACHO
- if (flag_pic && SYMBOL_REF_P (XEXP (fnaddr, 0)))
+ if (flag_pic && is_direct_call)
fnaddr = machopic_indirect_call_target (fnaddr);
#endif
}
@@ -11193,7 +11195,7 @@ ix86_expand_call (rtx retval, rtx fnaddr, rtx callarg1,
if (ix86_cmodel == CM_LARGE_PIC
&& !TARGET_PECOFF
&& MEM_P (fnaddr)
- && SYMBOL_REF_P (XEXP (fnaddr, 0))
+ && is_direct_call
&& !local_symbolic_operand (XEXP (fnaddr, 0), VOIDmode))
fnaddr = gen_rtx_MEM (QImode, construct_plt_address (XEXP (fnaddr, 0)));
/* Since x32 GOT slot is 64 bit with zero upper 32 bits, indirect
@@ -11225,6 +11227,20 @@ ix86_expand_call (rtx retval, rtx fnaddr, rtx callarg1,
call = gen_rtx_CALL (VOIDmode, fnaddr, callarg1);
+ /* Only indirect calls need KCFI instrumentation. */
+ rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
+ : kcfi_get_type_id_for_expanding_gimple_call ();
+ if (kcfi_type_rtx)
+ {
+ /* Wrap call with KCFI. */
+ call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
+
+ /* Add KCFI clobbers for the insn sequence. */
+ clobber_reg (&use, gen_rtx_REG (DImode, R10_REG));
+ clobber_reg (&use, gen_rtx_REG (DImode, R11_REG));
+ clobber_reg (&use, gen_rtx_REG (CCmode, FLAGS_REG));
+ }
+
if (retval)
call = gen_rtx_SET (retval, call);
vec[vec_len++] = call;
diff --git a/gcc/config/i386/i386-options.cc b/gcc/config/i386/i386-options.cc
index ba598a817f30..1a1a3c9b1e52 100644
--- a/gcc/config/i386/i386-options.cc
+++ b/gcc/config/i386/i386-options.cc
@@ -2186,6 +2186,19 @@ ix86_option_override_internal (bool main_args_p,
ix86_lam_type = lam_u57;
}
+ /* KCFI is only supported in 64-bit mode due to use of r10/r11 registers. */
+ if ((opts->x_flag_sanitize & SANITIZE_KCFI)
+ && (!TARGET_64BIT_P (opts->x_ix86_isa_flags)
+ || TARGET_X32_P (opts->x_ix86_isa_flags)))
+ sorry ("%<-fsanitize=kcfi%> is not supported for 32-bit x86 or x32 mode");
+
+ /* KCFI requires R10 and R11 registers for type checking. */
+ if ((opts->x_flag_sanitize & SANITIZE_KCFI)
+ && (fixed_regs[R10_REG] || fixed_regs[R11_REG]))
+ sorry ("%<-fsanitize=kcfi%> is not compatible with "
+ "%<-ffixed-r10%> or %<-ffixed-r11%> as KCFI requires these "
+ "registers for type checking");
+
/* For targets using ms ABI enable ms-extensions, if not
explicit turned off. For non-ms ABI we turn off this
option. */
diff --git a/gcc/config/i386/i386.cc b/gcc/config/i386/i386.cc
index 722c1713a6d3..20d63b2ee03c 100644
--- a/gcc/config/i386/i386.cc
+++ b/gcc/config/i386/i386.cc
@@ -98,6 +98,7 @@ along with GCC; see the file COPYING3. If not see
#include "i386-builtins.h"
#include "i386-expand.h"
#include "i386-features.h"
+#include "kcfi.h"
#include "function-abi.h"
#include "rtl-error.h"
#include "gimple-pretty-print.h"
@@ -323,6 +324,26 @@ unsigned int const svr4_debugger_register_map[FIRST_PSEUDO_REGISTER] =
93, 94, 95, 96, 97, 98, 99, 100
};
+/* KCFI arity (how many parameters a function has) is encoded in the
+ preamble's immediate register. The encoding is as follows:
+
+ Arity represented by
+ 0 by %eax (gcc regno = 0)
+ 1 by %ecx (gcc regno = 2)
+ 2 by %edx (gcc regno = 1)
+ 3 by %ebx (gcc regno = 3)
+ 4 by %esp (gcc regno = 7)
+ 5 by %ebp (gcc regno = 6)
+ 6 by %esi (gcc regno = 4)
+ 7 by %edi (gcc regno = 5)
+*/
+#define KCFI_ARITY_MAX_INDICATOR 7
+unsigned int const kcfi_arity_register_map[KCFI_ARITY_MAX_INDICATOR + 1] =
+{
+ /* eax to edi, as above. */
+ 0, 2, 1, 3, 7, 6, 4, 5
+};
+
/* Define parameter passing and return registers. */
static int const x86_64_int_parameter_registers[6] =
@@ -17525,11 +17546,89 @@ ix86_output_indirect_function_return (rtx ret_op)
return "";
}
-/* Output the assembly for a call instruction. */
+/* Output the assembly for a call instruction. INSN is the RTL instruction
+ being processed. OPERANDS is the array of RTL operands where operands[0]
+ is the call target, operands[1] is the call operand (unused on x86), and
+ operands[2] is the KCFI type ID constant (for KCFI-instrumented calls). */
const char *
-ix86_output_call_insn (rtx_insn *insn, rtx call_op)
+ix86_output_call_insn (rtx_insn *insn, rtx *operands)
{
+ rtx call_op = operands[0];
+ rtx pat = PATTERN (insn);
+
+ /* If KCFI, emit the check sequence first. */
+ if (GET_CODE (pat) == KCFI
+ || (GET_CODE (pat) == SET && GET_CODE (XEXP (pat, 1)) == KCFI))
+ {
+ /* Target is guaranteed to be in a register due to
+ TARGET_INDIRECT_BRANCH_REGISTER. */
+ gcc_assert (REG_P (call_op));
+
+ /* In thunk-extern mode, the register must be R11 for FineIBT
+ compatibility. */
+ if (cfun->machine->indirect_branch_type == indirect_branch_thunk_extern)
+ {
+ if (REGNO (call_op) != R11_REG)
+ {
+ /* Emit move from current target to R11. */
+ call_op = gen_rtx_REG (DImode, R11_REG);
+ rtx r11_operands[2] = { operands[0], call_op };
+ output_asm_insn ("movq\t%0, %1", r11_operands);
+ }
+ }
+
+ /* Get unique label number for this KCFI check. */
+ int labelno = kcfi_next_labelno ();
+
+ /* Generate custom label names. */
+ char trap_name[32];
+ char call_name[32];
+ ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
+ ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
+
+ /* Choose scratch register: r10 by default, r11 if r10 is the target. */
+ bool target_is_r10 = (REGNO (call_op) == R10_REG);
+ rtx scratch_reg = gen_rtx_REG (SImode, target_is_r10 ? R11_REG : R10_REG);
+
+ /* Get KCFI type ID from operand 2. */
+ uint32_t type_id = (uint32_t) INTVAL (operands[2]);
+
+ /* Convert to inverse for the check (0 - hash). */
+ uint32_t inverse_type_id = (uint32_t)(0 - type_id);
+
+ /* Calculate offset to typeid from target address. */
+ HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
+
+ /* Output complete KCFI check sequence. */
+ rtx inverse_type_id_rtx = gen_int_mode (inverse_type_id, SImode);
+ rtx mov_operands[2] = { inverse_type_id_rtx, scratch_reg };
+ output_asm_insn ("movl\t$%c0, %1", mov_operands);
+
+ /* Create memory operand for the addl instruction. */
+ rtx offset_rtx = gen_int_mode (offset, DImode);
+ rtx mem_op = gen_rtx_MEM (SImode,
+ gen_rtx_PLUS (DImode, call_op, offset_rtx));
+ rtx add_operands[2] = { mem_op, scratch_reg };
+ output_asm_insn ("addl\t%0, %1", add_operands);
+
+ /* Output conditional jump to call label. */
+ fputs ("\tje\t", asm_out_file);
+ assemble_name (asm_out_file, call_name);
+ fputc ('\n', asm_out_file);
+
+ /* Output trap label and instruction. */
+ ASM_OUTPUT_LABEL (asm_out_file, trap_name);
+ output_asm_insn ("ud2", operands);
+
+ /* Use common helper for trap section entry. */
+ rtx trap_label_sym = gen_rtx_SYMBOL_REF (Pmode, trap_name);
+ kcfi_emit_traps_section (asm_out_file, trap_label_sym, labelno);
+
+ /* Output pass/call label. */
+ ASM_OUTPUT_LABEL (asm_out_file, call_name);
+ }
+
bool direct_p = constant_call_address_operand (call_op, VOIDmode);
bool output_indirect_p
= (!TARGET_SEH
@@ -28735,6 +28834,92 @@ ix86_set_handled_components (sbitmap components)
}
}
+/* Apply x86-64 specific masking to KCFI type ID. TYPE_ID is the 32-bit
+ KCFI type identifier to potentially mask. Returns the type ID with
+ x86-64 specific adjustments to avoid embedding ENDBR instruction
+ sequences in the type identifier values. */
+
+static uint32_t
+ix86_kcfi_mask_type_id (uint32_t type_id)
+{
+ /* Avoid embedding ENDBR instructions in KCFI type IDs.
+ ENDBR64: 0xfa1e0ff3, ENDBR32: 0xfb1e0ff3
+ If the type ID matches either instruction encoding, increment by 1. */
+ if (type_id == 0xfa1e0ff3U || type_id == 0xfb1e0ff3U)
+ return type_id + 1;
+
+ return type_id;
+}
+
+/* Return x86-64 specific function arity (number of integer register
+ arguments) of the given FNDECL. */
+
+static uint8_t
+ix86_kcfi_compute_type_arity (tree fndecl)
+{
+ tree args;
+ uint8_t arity = 0;
+
+ /* Only count arity if requested. */
+ if (!flag_sanitize_kcfi_arity)
+ return 0;
+
+ /* If fndecl is NULL, we can't determine arity - return 0. */
+ if (!fndecl)
+ return 0;
+
+ /* Count the number of registers used, disregard SSE registers. */
+ for (args = DECL_ARGUMENTS (fndecl); args; args = TREE_CHAIN (args))
+ {
+ int int_nregs, sse_nregs;
+ bool args_on_stack = false;
+ machine_mode mode = TYPE_MODE (TREE_TYPE (args));
+
+ args_on_stack = examine_argument (mode, TREE_TYPE (args), 0, &int_nregs,
+ &sse_nregs);
+ /* If we place arguments on the stack, return highest arity indicator. */
+ if (args_on_stack)
+ return KCFI_ARITY_MAX_INDICATOR;
+ arity += int_nregs;
+ }
+
+ /* Return KCFI_ARITY_MAX_INDICATOR if we have counted more than
+ KCFI_ARITY_MAX_INDICATOR arguments. */
+ return arity > KCFI_ARITY_MAX_INDICATOR ? KCFI_ARITY_MAX_INDICATOR : arity;
+}
+
+/* Emit x86_64-specific type ID instruction and return instruction size.
+ FILE is the output assembly file stream, or NULL for size calculation only.
+ TYPE_ID is the 32-bit KCFI type identifier to emit. FNDECL is the function
+ declaration, used to compute arity if needed. Returns the number
+ of bytes the instruction occupies (5 bytes for x86_64 movl instruction). */
+
+static int
+ix86_kcfi_emit_type_id (FILE *file, uint32_t type_id, tree fndecl)
+{
+ /* Compute function arity. */
+ uint8_t arity = ix86_kcfi_compute_type_arity (fndecl);
+
+ /* Choose register for movl instruction. */
+ gcc_assert (arity <= KCFI_ARITY_MAX_INDICATOR);
+ uint32_t regno = kcfi_arity_register_map[arity];
+
+ if (file)
+ fprintf (file, "\tmovl\t$0x%08x, %%e%s\n", type_id, reg_names[regno]);
+
+ /* x86_64 uses 5-byte movl instruction for type ID. */
+ return 5;
+}
+
+#undef TARGET_KCFI_SUPPORTED
+#define TARGET_KCFI_SUPPORTED hook_bool_void_true
+
+#undef TARGET_KCFI_MASK_TYPE_ID
+#define TARGET_KCFI_MASK_TYPE_ID ix86_kcfi_mask_type_id
+
+#undef TARGET_KCFI_EMIT_TYPE_ID
+#define TARGET_KCFI_EMIT_TYPE_ID ix86_kcfi_emit_type_id
+
#undef TARGET_SHRINK_WRAP_GET_SEPARATE_COMPONENTS
#define TARGET_SHRINK_WRAP_GET_SEPARATE_COMPONENTS ix86_get_separate_components
#undef TARGET_SHRINK_WRAP_COMPONENTS_FOR_BB
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index d11f8973ed1b..de6aadfd5346 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -18681,6 +18681,52 @@ The type identifier is placed before the function entry point,
allowing runtime verification without additional metadata structures,
and without changing the entry points of the target functions.
+Platform-specific implementation details:
+
+On x86_64, KCFI type identifiers are emitted as a @code{movl $ID, %eax}
+instruction before the function entry. The implementation ensures that
+type IDs never collide with ENDBR instruction encodings. When used
+with @option{-fpatchable-function-entry}, the type identifier is
+placed before any patchable NOPs, with appropriate alignment to maintain
+the alignment specified by @code{-falign-functions}. KCFI automatically
+implies @option{-mindirect-branch-register}, forcing all indirect calls
+and jumps to use registers instead of memory operands. The runtime
+check loads the type ID from the target function into @code{%r10d} and
+uses an @code{addl} instruction to add the negative expected type ID,
+effectively zeroing the register if the types match. A conditional
+jump follows to either continue execution or trap on mismatch. The
+check sequence uses @code{%r10d} and @code{%r11d} as scratch registers.
+Trap locations are recorded in a special @code{.kcfi_traps} section
+that maps trap sites to their corresponding function entry points,
+enabling debuggers and crash handlers to identify KCFI violations.
+The exact instruction sequences for both the KCFI preamble and the
+check-call bundle are considered ABI, as the Linux kernel may
+optionally rewrite these areas at boot time to mitigate detected CPU
+errata.
+
+The @code{-fsanitize-kcfi-arity} option encodes the function arity
+(i.e., the number of arguments) into the @code{movl $ID, $REG} instruction,
+where the @code{$REG} encodes the function arity. This allows users, such
+as FineIBT to generate code in the kernel that is aware of how many register
+arguments a function takes. The encoding is as follows:
+
+@multitable @columnfractions 0.10 0.50 0.40
+@headitem Arity @tab Description @tab Register
+@item 0 @tab 0 parameters @tab @code{EAX}
+@item 1 @tab 1 parameter in RDI @tab @code{ECX}
+@item 2 @tab 2 parameters in RDI and RSI @tab @code{EDX}
+@item 3 @tab 3 parameters in RDI, RSI, and RDX @tab @code{EBX}
+@item 4 @tab 4 parameters in RDI, RSI, RDX, and RCX @tab @code{ESP}
+@item 5 @tab 5 parameters in RDI, RSI, RDX, RCX, and R8 @tab @code{EBP}
+@item 6 @tab 6 parameters in RDI, RSI, RDX, RCX, R8, and R9 @tab @code{ESI}
+@item 7 @tab At least one parameter may be passed on the stack @tab @code{EDI}
+@end multitable
+
+For example, if a function `foo` takes 3 register arguments, the KCFI
+header MOVri instruction would become something like this:
+
+@code{movl $199571451, %ebx # hash of foo's type = 0xBE537FB}
+
KCFI is intended primarily for kernel code and may not be suitable
for user-space applications that rely on techniques incompatible
with strict type checking of indirect calls.
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
index 7c1cff986c01..7c59921e630c 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
@@ -46,4 +46,21 @@ __attribute__((noinline)) void test_conditional_call(int flag) {
}
}
+/*
+** test_complex_args: { target x86_64-*-* }
+** ...
+** movl \$-?[0-9]+, %r10d
+** addl -4\((%r[a-z0-9]+)\), %r10d
+** je .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ud2
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .long .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** jmp \*\1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
index ca833fed2971..fe0a21d26df9 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
@@ -1,5 +1,6 @@
/* Test basic KCFI functionality - preamble generation. */
/* { dg-do compile } */
+/* { dg-additional-options "-falign-functions=16" { target x86_64-*-* } } */
/* Extern function declarations - should NOT get KCFI preambles. */
extern void external_func(void);
@@ -55,6 +56,26 @@ int main() {
/* Function with nocf_check attribute should NOT have preamble. */
/* { dg-final { scan-assembler-not {__cfi_nocf_check_function:} } } */
+/* x86_64: Verify type ID in preamble (after NOPs, before function label) */
+/* { dg-final { scan-assembler {__cfi_regular_function:\n\t+nop\n.*\n\t+movl\t+\$0x[0-9a-f]+, %eax} { target x86_64-*-* } } } */
+
+/*
+** static_caller: { target x86_64-*-* }
+** ...
+** movl \$-?[0-9]+, %r10d
+** addl -4\((%r[a-z0-9]+)\), %r10d
+** je .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ud2
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .long .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** call \*\1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
/* Extern functions should NOT get KCFI preambles. */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
index f72344f77124..05165f0e2851 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
@@ -59,3 +59,18 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
Should see:
1. KCFI check for is_visible call with is_visible type ID A.
2. KCFI check for is_bin_visible and is_bin_visible_again call with type ID B. */
+
+/* Verify we have TWO different KCFI check sequences. */
+/* Each check should have different type ID constants. */
+/* x86: { dg-final { scan-assembler-times {movl\s+\$-?[0-9]+,\s+%r10d} 2 { target i?86-*-* x86_64-*-* } } } */
+
+/* Verify the checks use DIFFERENT type IDs (not shared).
+ We should NOT see the same type ID used twice - that would indicate
+ unmerged sharing. */
+/* x86: { dg-final { scan-assembler-not {movl\s+\$(-?[0-9]+),\s+%r10d.*movl\s+\$\1,\s+%r10d} { target i?86-*-* x86_64-*-* } } } */
+
+/* Verify expected number of traps. */
+/* x86: { dg-final { scan-assembler-times {ud2} 2 { target i?86-*-* x86_64-*-* } } } */
+
+/* Verify 2 separate call sites. */
+/* x86: { dg-final { scan-assembler-times {jmp\s+\*%[a-z0-9]+} 2 { target i?86-*-* x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
index c48b8d7ad552..ed415033c5c9 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
@@ -128,4 +128,22 @@ int main() {
return result;
}
+/* Standard KCFI handling. */
+/*
+** test_struct_members: { target x86_64-*-* }
+** ...
+** movl \$-?[0-9]+, %r10d
+** addl -4\((%r[a-z0-9]+)\), %r10d
+** je .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ud2
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .long .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** call \*\1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
index 7d58fef3f920..5553ff47174b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
@@ -38,4 +38,23 @@ int main(void)
return 0;
}
+/*
+** indirect_call: { target x86_64-*-* }
+** ...
+** movq %rdi, (%rax)
+** movl \$called_count, %edi
+** movl \$-?[0-9]+, %r10d
+** addl -4\(\1\), %r10d
+** je .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ud2
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .long .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** jmp \*\1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
index 4a90390d1934..9ed7e21fe8eb 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
@@ -72,3 +72,14 @@ int main(void)
return 0;
}
+
+/* Verify correct number of KCFI checks: exactly 2 */
+/* { dg-final { scan-assembler-times {ud2} 2 { target x86_64-*-* } } } */
+
+/* Positive controls: these should have KCFI checks. */
+/* { dg-final { scan-assembler {normal_function:.*ud2.*\.size\s+normal_function} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {wrap_normal_inline:.*ud2.*\.size\s+wrap_normal_inline} { target x86_64-*-* } } } */
+
+/* Negative controls: these should NOT have KCFI checks. */
+/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ud2.*\.size\s+sensitive_non_inline_function} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*ud2.*\.size\s+wrap_sensitive_inline} { target x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
index 124d26488635..95a8e8419e00 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
@@ -29,3 +29,8 @@ int main() {
/* { dg-final { scan-assembler "__cfi_caller_with_checks:" } } */
/* { dg-final { scan-assembler "__cfi_caller_no_checks:" } } */
/* { dg-final { scan-assembler "__cfi_main:" } } */
+
+/* caller_with_checks() should generate KCFI check.
+ caller_no_checks() should NOT generate KCFI check (no_sanitize).
+ So a total of exactly 1 KCFI check in the entire program. */
+/* { dg-final { scan-assembler-times {addl\t-4\(%r[ad]x\), %r1[01]d} 1 { target x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
index 213a1a2892a5..97d964feebd3 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
@@ -1,5 +1,6 @@
/* Test KCFI call-site offset validation across architectures. */
/* { dg-do compile } */
+/* { dg-additional-options "-falign-functions=16" { target x86_64-*-* } } */
void target_func_a(void) { }
void target_func_b(int x) { }
@@ -22,3 +23,7 @@ int main() {
/* { dg-final { scan-assembler "__cfi_target_func_a:" } } */
/* { dg-final { scan-assembler "__cfi_target_func_b:" } } */
/* { dg-final { scan-assembler "__cfi_target_func_c:" } } */
+
+/* x86_64: All call sites should use -4 offset for KCFI type ID loads, even
+ with -falign-functions=16 (we're not using patchable entries here). */
+/* { dg-final { scan-assembler {movl\t\$-?[0-9]+, %r10d\n\taddl\t-4\(%r[a-z0-9]+\), %r10d} { target x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
index a6a2f4816fef..379356385a16 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
@@ -1,6 +1,7 @@
/* Test KCFI with patchable function entries - entry NOPs only. */
/* { dg-do compile } */
/* { dg-additional-options "-fpatchable-function-entry=4,0" } */
+/* { dg-additional-options "-falign-functions=16" { target x86_64-*-* } } */
void test_function(void) {
}
@@ -11,4 +12,27 @@ int main() {
return 0;
}
+/*
+** __cfi_test_function: { target x86_64-*-* }
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** movl \$0x[0-9a-f]+, %eax
+*/
+
+/*
+** main: { target x86_64-*-* }
+** ...
+** addl -4\(%r[a-z0-9]+\), %r10d
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
index 8c4ec30cecc5..06df3495bb23 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
@@ -1,6 +1,7 @@
/* Test KCFI with large patchable function entries. */
/* { dg-do compile } */
/* { dg-additional-options "-fpatchable-function-entry=11,11" } */
+/* { dg-additional-options "-falign-functions=16" { target x86_64-*-* } } */
void test_function(void) {
}
@@ -11,4 +12,16 @@ int main() {
return 0;
}
+/*
+** __cfi_test_function: { target x86_64-*-* }
+** movl \$0x[0-9a-f]+, %eax
+*/
+
+/*
+** main: { target x86_64-*-* }
+** ...
+** addl -15\(%r[a-z0-9]+\), %r10d
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
index 78a834ef2a97..ef87b135934b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
@@ -1,6 +1,7 @@
/* Test KCFI with medium patchable function entries. */
/* { dg-do compile } */
/* { dg-additional-options "-fpatchable-function-entry=8,4" } */
+/* { dg-additional-options "-falign-functions=16" { target x86_64-*-* } } */
void test_function(void) {
}
@@ -11,4 +12,23 @@ int main() {
return 0;
}
+/*
+** __cfi_test_function: { target x86_64-*-* }
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** movl \$0x[0-9a-f]+, %eax
+*/
+
+/*
+** main: { target x86_64-*-* }
+** ...
+** addl -8\(%r[a-z0-9]+\), %r10d
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
index 1a4d8269ed56..872814aa4171 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
@@ -1,6 +1,7 @@
/* Test KCFI with patchable function entries - prefix NOPs only. */
/* { dg-do compile } */
/* { dg-additional-options "-fpatchable-function-entry=3,3" } */
+/* { dg-additional-options "-falign-functions=16" { target x86_64-*-* } } */
void test_function(void) {
}
@@ -11,4 +12,24 @@ int main() {
return 0;
}
+/*
+** __cfi_test_function: { target x86_64-*-* }
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** nop
+** movl \$0x[0-9a-f]+, %eax
+*/
+
+/*
+** main: { target x86_64-*-* }
+** ...
+** addl -7\(%r[a-z0-9]+\), %r10d
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
index 9ddf178aa2b1..04a9eb1fd206 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
@@ -58,3 +58,23 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
/* { dg-final { scan-assembler-times "__cfi_test_param_indirect_call:" 1 } } */
/* { dg-final { scan-assembler-times "__cfi_test_void_indirect_call:" 1 } } */
/* { dg-final { scan-assembler-times "__cfi_test_non_tail_indirect_call:" 1 } } */
+
+/* Should have exactly 4 KCFI checks for indirect calls as
+ (load type ID + compare). */
+/* { dg-final { scan-assembler-times {movl\t\$-?[0-9]+, %r10d} 4 { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-times {addl\t-4\(%r[a-z0-9]+\), %r10d} 4 { target x86_64-*-* } } } */
+
+/* Should have exactly 4 trap sections and 4 trap instructions. */
+/* { dg-final { scan-assembler-times "\\.kcfi_traps" 4 { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-times "ud2" 4 { target x86_64-*-* } } } */
+
+/* Should NOT have unprotected direct jumps to vtable. */
+/* { dg-final { scan-assembler-not {jmp\t\*vtable\(%rip\)} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-not {jmp\t\*vtable\+8\(%rip\)} { target x86_64-*-* } } } */
+
+/* Should have exactly 3 protected tail calls (jmp through register after
+ KCFI check). */
+/* { dg-final { scan-assembler-times {jmp\t\*%[a-z0-9]+} 3 { target x86_64-*-* } } } */
+
+/* Should have exactly 1 regular call (non-tail call case). */
+/* { dg-final { scan-assembler-times {call\t\*%[a-z0-9]+} 1 { target x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
index 6d34ad6e1a0c..55c0829ccd7b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
@@ -15,3 +15,9 @@ int main() {
/* Should have KCFI preamble. */
/* { dg-final { scan-assembler "__cfi_target_function:" } } */
+
+/* Should have exactly 2 trap labels in code. */
+/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ud2} 2 { target x86_64-*-* } } } */
+
+/* x86_64 should exactly 2 .kcfi_traps sections. */
+/* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-arity.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-arity.c
new file mode 100644
index 000000000000..5c1d1ffcc619
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-arity.c
@@ -0,0 +1,93 @@
+/* Test KCFI -fsanitize-kcfi-arity - preamble generation */
+/* { dg-do compile { target x86_64-*-* } } */
+/* { dg-additional-options "-O2 -fsanitize-kcfi-arity" } */
+
+void regular_function(int x) {
+ /* This should get KCFI preamble */
+}
+
+/* These functions should get a KCFI arity corresponing to the number of
+ parameter.
+
+ On x86-64, the mov instruction in the preamble encodes the arity in the
+ used immediate register, as follows:
+
+ Arity Description Encoding in reg field
+ 0 0 parameters EAX
+ 1 1 parameter in RDI ECX
+ 2 2 parameters in RDI EDX
+ and RSI
+ 3 3 parameters in RDI, EBX
+ RSI, and RDX
+ 4 4 parameters in RDI, ESP
+ RSI, RDX, and RCX
+ 5 5 parameters in RDI, EBP
+ RSI, RDX, RCX, and R8
+ 6 6 parameters in RDI, ESI
+ RSI, RDX, RCX, R8, and R9
+ 7 At least one parameter EDI
+ may be passed on the stack
+*/
+void ind_fn_00(void) {}
+void ind_fn_01(int) {}
+void ind_fn_02(int, int) {}
+void ind_fn_03(int, int, int) {}
+void ind_fn_04(int, int, int, int) {}
+void ind_fn_05(int, int, int, int, int) {}
+void ind_fn_06(int, int, int, int, int, int) {}
+/* Arguments on stack arity from here on should be 7. */
+void ind_fn_07(int, int, int, int, int, int, int) {}
+void ind_fn_08(int, int, int, int, int, int, int, int) {}
+
+void ind_fn_float_01(int, float) {}
+void ind_fn_float_03(int, int, float, float, float) {}
+
+void (*func_ptr_00)(void) = ind_fn_00;
+void (*func_ptr_01)(int) = ind_fn_01;
+void (*func_ptr_02)(int, int) = ind_fn_02;
+void (*func_ptr_03)(int, int, int) = ind_fn_03;
+void (*func_ptr_04)(int, int, int, int) = ind_fn_04;
+void (*func_ptr_05)(int, int, int, int, int) = ind_fn_05;
+void (*func_ptr_06)(int, int, int, int, int, int) = ind_fn_06;
+void (*func_ptr_07)(int, int, int, int, int, int, int) = ind_fn_07;
+void (*func_ptr_08)(int, int, int, int, int, int, int, int) = ind_fn_08;
+
+
+void (*func_ptr_float_01)(int, float) = ind_fn_float_01;
+void (*func_ptr_float_03)(int, int, float, float, float) = ind_fn_float_03;
+
+int main() {
+ /* Function arity tests. */
+ func_ptr_00();
+ func_ptr_01(1);
+ func_ptr_02(1, 1);
+ func_ptr_03(1, 1, 1);
+ func_ptr_04(1, 1, 1, 1);
+ func_ptr_05(1, 1, 1, 1, 1);
+ func_ptr_06(1, 1, 1, 1, 1, 1);
+
+ /* Both of these put arguments on the stack so both get arity 7. */
+ func_ptr_07(1, 1, 1, 1, 1, 1, 1);
+ func_ptr_08(1, 1, 1, 1, 1, 1, 1, 1);
+
+ /* Float parameters should not be counted for arity. */
+ func_ptr_float_01(1, 1.0);
+ func_ptr_float_03(1, 1, 0.5, 0.5, 0.5);
+
+ return 0;
+}
+
+/* x86_64: Verify arity immediate register encoding in preamble. */
+/* { dg-final { scan-assembler {__cfi_ind_fn_00:.*\n\tmovl\t+\$0x[0-9a-f]+, %eax\n.*\nind_fn_00:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_01:.*\n\tmovl\t+\$0x[0-9a-f]+, %ecx\n.*\nind_fn_01:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_02:.*\n\tmovl\t+\$0x[0-9a-f]+, %edx\n.*\nind_fn_02:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_03:.*\n\tmovl\t+\$0x[0-9a-f]+, %ebx\n.*\nind_fn_03:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_04:.*\n\tmovl\t+\$0x[0-9a-f]+, %esp\n.*\nind_fn_04:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_05:.*\n\tmovl\t+\$0x[0-9a-f]+, %ebp\n.*\nind_fn_05:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_06:.*\n\tmovl\t+\$0x[0-9a-f]+, %esi\n.*\nind_fn_06:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_07:.*\n\tmovl\t+\$0x[0-9a-f]+, %edi\n.*\nind_fn_07:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_08:.*\n\tmovl\t+\$0x[0-9a-f]+, %edi\n.*\nind_fn_08:} { target x86_64-*-* } } } */
+
+/* x86_64: Verify arity is not affected by SSE registers. */
+/* { dg-final { scan-assembler {__cfi_ind_fn_float_01:.*\n\tmovl\t+\$0x[0-9a-f]+, %ecx\n.*\nind_fn_float_01:} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {__cfi_ind_fn_float_03:.*\n\tmovl\t+\$0x[0-9a-f]+, %edx\n.*\nind_fn_float_03:} { target x86_64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r10.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r10.c
new file mode 100644
index 000000000000..f509bc918a82
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r10.c
@@ -0,0 +1,17 @@
+/* Test that KCFI is incompatible with -ffixed-r10 on x86_64. */
+/* { dg-do compile { target x86_64-*-* } } */
+/* { dg-additional-options "-ffixed-r10" } */
+
+/* { dg-message "sorry, unimplemented: '-fsanitize=kcfi' is not compatible with '-ffixed-r10' or '-ffixed-r11' as KCFI requires these registers for type checking" "" { target *-*-* } 0 } */
+
+void test_function(void)
+{
+ /* Empty function. */
+}
+
+int main(void)
+{
+ void (*ptr)(void) = test_function;
+ ptr(); /* This would need KCFI instrumentation. */
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r11.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r11.c
new file mode 100644
index 000000000000..6b1ce8a1b6c5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-fixed-r11.c
@@ -0,0 +1,17 @@
+/* Test that KCFI is incompatible with -ffixed-r11 on x86_64. */
+/* { dg-do compile { target x86_64-*-* } } */
+/* { dg-additional-options "-ffixed-r11" } */
+
+/* { dg-message "sorry, unimplemented: '-fsanitize=kcfi' is not compatible with '-ffixed-r10' or '-ffixed-r11' as KCFI requires these registers for type checking" "" { target *-*-* } 0 } */
+
+void test_function(void)
+{
+ /* Empty function. */
+}
+
+int main(void)
+{
+ void (*ptr)(void) = test_function;
+ ptr(); /* This would need KCFI instrumentation. */
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-retpoline-r11.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-retpoline-r11.c
new file mode 100644
index 000000000000..056068b4f197
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-x86-retpoline-r11.c
@@ -0,0 +1,40 @@
+/* Test KCFI with retpoline thunk-extern flag forces r11 usage. */
+/* { dg-do compile { target x86_64-*-* } } */
+/* { dg-additional-options "-O2 -mindirect-branch=thunk-extern" } */
+
+extern int external_target(void);
+
+/* Test regular call (not tail call) */
+__attribute__((noinline))
+int call_test(int (*func_ptr)(void)) {
+ /* This indirect call should use r11 when both KCFI and
+ -mindirect-branch=thunk-extern are enabled. */
+ int result = func_ptr(); /* Function parameter prevents direct optimization. */
+ return result + 1; /* Prevent tail call optimization. */
+}
+
+/* Reference external_target to generate the required symbol. */
+int (*external_func_ptr)(void) = external_target;
+
+/* Test function for sibcalls (tail calls) */
+__attribute__((noinline))
+void sibcall_test(int (**func_ptr)(void)) {
+ /* This sibcall should use r11 when both KCFI and
+ -mindirect-branch=thunk-extern are enabled. */
+ (*func_ptr)(); /* Tail call - should be optimized to sibcall. */
+}
+
+/* Should have weak symbol for external function. */
+/* { dg-final { scan-assembler "__kcfi_typeid_external_target" } } */
+
+/* When both KCFI and -mindirect-branch=thunk-extern are enabled,
+ indirect calls should always use r11 register and convert to extern thunks. */
+/* { dg-final { scan-assembler-times {call\s+} 1 } } */
+/* { dg-final { scan-assembler-times {call\s+__x86_indirect_thunk_r11} 1 } } */
+
+/* Sibcalls should also use r11 register and convert to extern thunks. */
+/* { dg-final { scan-assembler-times {jmp\s+} 1 } } */
+/* { dg-final { scan-assembler-times {jmp\s+__x86_indirect_thunk_r11} 1 } } */
+
+/* Should have exactly 2 KCFI traps (one per function) */
+/* { dg-final { scan-assembler-times {ud2} 2 } } */
diff --git a/gcc/common.opt b/gcc/common.opt
index d44f713ae34f..4d41b1e225de 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1159,6 +1159,14 @@ Enum(sanitize_coverage) String(trace-pc) Value(SANITIZE_COV_TRACE_PC)
EnumValue
Enum(sanitize_coverage) String(trace-cmp) Value(SANITIZE_COV_TRACE_CMP)
+fsanitize-kcfi-arity
+Common Driver Var(flag_sanitize_kcfi_arity)
+For supported targets, this feature extends kCFI by telling the compiler
+to record information about each indirect-callable function’s
+arity (i.e., the number of arguments passed in registers) into the
+binary. Some kernel CFI techniques, such as FineIBT, may be able to use
+this information to provide enhanced security.
+
fasan-shadow-offset=
Common Joined RejectNegative Var(common_deferred_options) Defer
-fasan-shadow-offset=<number> Use custom shadow memory offset.
diff --git a/gcc/opts.cc b/gcc/opts.cc
index f1b324c5a397..086a1e5669c1 100644
--- a/gcc/opts.cc
+++ b/gcc/opts.cc
@@ -2806,6 +2806,7 @@ common_handle_option (struct gcc_options *opts,
SET_OPTION_IF_UNSET (opts, opts_set, param_asan_stack, 0);
SET_OPTION_IF_UNSET (opts, opts_set, param_asan_protect_allocas, 0);
SET_OPTION_IF_UNSET (opts, opts_set, param_asan_use_after_return, 0);
+ SET_OPTION_IF_UNSET (opts, opts_set, flag_sanitize_kcfi_arity, 0);
}
if (opts->x_flag_sanitize & SANITIZE_KERNEL_HWADDRESS)
{
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
` (3 preceding siblings ...)
2025-11-20 22:21 ` [PATCH v8 4/7] x86: Add x86_64 Kernel Control Flow Integrity implementation Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
2025-11-20 22:57 ` Andrew Pinski
2025-11-20 22:21 ` [PATCH v8 6/7] arm: Add ARM 32-bit " Kees Cook
2025-11-20 22:21 ` [PATCH v8 7/7] riscv: Add RISC-V " Kees Cook
6 siblings, 1 reply; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
Implement AArch64-specific KCFI backend.
- Trap debugging through ESR (Exception Syndrome Register) encoding
in BRK instruction immediate values.
- Scratch register allocation using w16/w17 (x16/x17) following
AArch64 procedure call standard for intra-procedure-call registers,
which already makes x16/x17 available through existing clobbers.
- Complementary with BTI (which uses a separate pass system to inject
landing instructions where needed).
- Does not interfere with SME, which uses attributes not function
prototypes for distinguishing functions.
Assembly Code Pattern for AArch64:
ldur w16, [target, #-4] ; Load actual type ID from preamble
mov w17, #type_id_low ; Load expected type (lower 16 bits)
movk w17, #type_id_high, lsl #16 ; Load upper 16 bits if needed
cmp w16, w17 ; Compare type IDs directly
b.eq .Lpass ; Branch if types match
.Ltrap: brk #esr_value ; Enhanced trap with register info
.Lpass: blr/br target ; Execute validated indirect transfer
ESR (Exception Syndrome Register) Integration:
- BRK instruction immediate encoding format:
0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31)
- TypeIndex indicates which W register contains expected type (W17 = 17)
- AddrIndex indicates which X register contains target address (0-30)
- Example: brk #33313 (0x8221) = expected type in W17, target address in X1
Build and run tested with Linux kernel ARCH=arm64.
gcc/ChangeLog:
config/aarch64/aarch64-protos.h: Declare aarch64_indirect_branch_asm,
and KCFI helpers.
config/aarch64/aarch64.cc (aarch64_expand_call): Wrap CALLs in
KCFI, with clobbers.
(aarch64_indirect_branch_asm): New function, extract common
logic for branch asm, like existing call asm helper.
(aarch64_output_kcfi_insn): Emit KCFI assembly.
config/aarch64/aarch64.md: Add KCFI RTL patterns and replace
open-coded branch emission with aarch64_indirect_branch_asm.
doc/invoke.texi: Document aarch64 nuances.
gcc/testsuite/ChangeLog:
* gcc.dg/kcfi/kcfi-adjacency.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-basics.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-call-sharing.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-complex-addressing.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-move-preservation.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-no-sanitize.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-offset-validation.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-patchable-large.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-patchable-medium.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-tail-calls.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-trap-section.c: Add aarch64 patterns.
* gcc.dg/kcfi/kcfi-trap-encoding.c: New test.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/config/aarch64/aarch64-protos.h | 5 +
gcc/config/aarch64/aarch64.md | 66 +++++++++--
gcc/config/aarch64/aarch64.cc | 105 ++++++++++++++++++
gcc/doc/invoke.texi | 14 +++
gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 15 +++
gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++++
gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
.../gcc.dg/kcfi/kcfi-complex-addressing.c | 16 +++
.../gcc.dg/kcfi/kcfi-move-preservation.c | 20 ++++
.../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
.../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
.../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 12 ++
.../gcc.dg/kcfi/kcfi-patchable-large.c | 12 ++
.../gcc.dg/kcfi/kcfi-patchable-medium.c | 12 ++
.../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 12 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 19 ++++
.../gcc.dg/kcfi/kcfi-trap-encoding.c | 41 +++++++
gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 4 +
19 files changed, 379 insertions(+), 8 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
index a9e407ba340e..c32d454fe277 100644
--- a/gcc/config/aarch64/aarch64-protos.h
+++ b/gcc/config/aarch64/aarch64-protos.h
@@ -1272,6 +1272,7 @@ tree aarch64_resolve_overloaded_builtin_general (location_t, tree, void *);
const char *aarch64_sls_barrier (int);
const char *aarch64_indirect_call_asm (rtx);
+const char *aarch64_indirect_branch_asm (rtx);
extern bool aarch64_harden_sls_retbr_p (void);
extern bool aarch64_harden_sls_blr_p (void);
@@ -1295,4 +1296,8 @@ extern unsigned aarch64_stack_alignment (const_tree exp, unsigned align);
extern rtx aarch64_gen_compare_zero_and_branch (rtx_code code, rtx x,
rtx_code_label *label);
+/* KCFI support. */
+extern void kcfi_emit_trap_with_section (FILE *file, rtx trap_label_rtx);
+extern const char *aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands);
+
#endif /* GCC_AARCH64_PROTOS_H */
diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
index 855df791bae7..84894816509c 100644
--- a/gcc/config/aarch64/aarch64.md
+++ b/gcc/config/aarch64/aarch64.md
@@ -1506,6 +1506,19 @@
}"
)
+;; KCFI indirect call
+(define_insn "*call_insn"
+ [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucr"))
+ (match_operand 1 "" ""))
+ (match_operand 3 "const_int_operand"))
+ (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI LR_REGNUM))]
+ "!SIBLING_CALL_P (insn)"
+{
+ return aarch64_output_kcfi_insn (insn, operands);
+}
+ [(set_attr "type" "call")])
+
(define_insn "*call_insn"
[(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand"))
(match_operand 1 "" ""))
@@ -1533,6 +1546,21 @@
}"
)
+;; KCFI call with return value
+(define_insn "*call_value_insn"
+ [(set (match_operand 0 "" "")
+ (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
+ "Ucr"))
+ (match_operand 2 "" ""))
+ (match_operand 4 "const_int_operand")))
+ (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI LR_REGNUM))]
+ "!SIBLING_CALL_P (insn)"
+{
+ return aarch64_output_kcfi_insn (insn, &operands[1]);
+}
+ [(set_attr "type" "call")])
+
(define_insn "*call_value_insn"
[(set (match_operand 0 "" "")
(call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"))
@@ -1573,6 +1601,19 @@
}
)
+;; KCFI sibling call
+(define_insn "*sibcall_insn"
+ [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs"))
+ (match_operand 1 ""))
+ (match_operand 3 "const_int_operand"))
+ (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
+ (return)]
+ "SIBLING_CALL_P (insn)"
+{
+ return aarch64_output_kcfi_insn (insn, operands);
+}
+ [(set_attr "type" "branch")])
+
(define_insn "*sibcall_insn"
[(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs, Usf"))
(match_operand 1 ""))
@@ -1581,16 +1622,28 @@
"SIBLING_CALL_P (insn)"
{
if (which_alternative == 0)
- {
- output_asm_insn ("br\\t%0", operands);
- return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
- }
+ return aarch64_indirect_branch_asm (operands[0]);
return "b\\t%c0";
}
[(set_attr "type" "branch, branch")
(set_attr "sls_length" "retbr,none")]
)
+;; KCFI sibling call with return value
+(define_insn "*sibcall_value_insn"
+ [(set (match_operand 0 "")
+ (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
+ "Ucs"))
+ (match_operand 2 ""))
+ (match_operand 4 "const_int_operand")))
+ (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
+ (return)]
+ "SIBLING_CALL_P (insn)"
+{
+ return aarch64_output_kcfi_insn (insn, &operands[1]);
+}
+ [(set_attr "type" "branch")])
+
(define_insn "*sibcall_value_insn"
[(set (match_operand 0 "")
(call (mem:DI
@@ -1601,10 +1654,7 @@
"SIBLING_CALL_P (insn)"
{
if (which_alternative == 0)
- {
- output_asm_insn ("br\\t%1", operands);
- return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
- }
+ return aarch64_indirect_branch_asm (operands[1]);
return "b\\t%c1";
}
[(set_attr "type" "branch, branch")
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index 6dfdaa4fb9b0..86e3af992d2a 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -83,6 +83,7 @@
#include "rtlanal.h"
#include "tree-dfa.h"
#include "asan.h"
+#include "kcfi.h"
#include "aarch64-elf-metadata.h"
#include "aarch64-feature-deps.h"
#include "config/arm/aarch-common.h"
@@ -11965,6 +11966,16 @@ aarch64_expand_call (rtx result, rtx mem, rtx cookie, bool sibcall)
call = gen_rtx_CALL (VOIDmode, mem, const0_rtx);
+ /* Only indirect calls need KCFI instrumentation. */
+ bool is_direct_call = SYMBOL_REF_P (XEXP (mem, 0));
+ rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
+ : kcfi_get_type_id_for_expanding_gimple_call ();
+ if (kcfi_type_rtx)
+ {
+ /* Wrap call in KCFI. */
+ call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
+ }
+
if (result != NULL_RTX)
call = gen_rtx_SET (result, call);
@@ -19225,6 +19236,9 @@ aarch64_override_options_internal (struct gcc_options *opts)
#endif
}
+ if ((flag_sanitize & SANITIZE_KCFI) && TARGET_ILP32)
+ sorry ("%<-fsanitize=kcfi%> is not supported for %<-mabi=ilp32%>");
+
aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
&& !(isa_flags & AARCH64_FL_SME))
@@ -30822,6 +30836,18 @@ aarch64_indirect_call_asm (rtx addr)
return "";
}
+/* Generate assembly for AArch64 indirect branch instruction. ADDR is the
+ target address register. Returns any additional barrier instructions
+ needed for SLS (Straight Line Speculation) mitigation. */
+
+const char *
+aarch64_indirect_branch_asm (rtx addr)
+{
+ gcc_assert (REG_P (addr));
+ output_asm_insn ("br\t%0", &addr);
+ return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
+}
+
/* Emit the assembly instruction to load the thread pointer into DEST.
Select between different tpidr_elN registers depending on -mtp= setting. */
@@ -33089,6 +33115,85 @@ aarch64_libgcc_floating_mode_supported_p
#undef TARGET_DOCUMENTATION_NAME
#define TARGET_DOCUMENTATION_NAME "AArch64"
+/* Output the assembly for a KCFI checked call instruction. INSN is the
+ RTL instruction being processed. OPERANDS is the array of RTL operands
+ where operands[0] is the call target register, operands[3] is the KCFI
+ type ID constant. Returns the appropriate call instruction string. */
+
+const char *
+aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands)
+{
+ /* Target register is operands[0]. */
+ rtx target_reg = operands[0];
+ gcc_assert (REG_P (target_reg));
+
+ /* Get KCFI type ID from operand[3]. */
+ uint32_t type_id = (uint32_t) INTVAL (operands[3]);
+
+ /* Calculate typeid offset from call target. */
+ HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
+
+ /* Get unique label number for this KCFI check. */
+ int labelno = kcfi_next_labelno ();
+
+ /* Generate custom label names. */
+ char trap_name[32];
+ char call_name[32];
+ ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
+ ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
+
+ rtx temp_operands[3];
+
+ /* Load actual type into w16 from memory at offset using ldur. */
+ temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
+ temp_operands[1] = target_reg;
+ temp_operands[2] = GEN_INT (offset);
+ output_asm_insn ("ldur\t%w0, [%1, #%2]", temp_operands);
+
+ /* Load expected type low 16 bits into w17. */
+ temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
+ temp_operands[1] = GEN_INT (type_id & 0xFFFF);
+ output_asm_insn ("mov\t%w0, #%1", temp_operands);
+
+ /* Load expected type high 16 bits into w17. */
+ temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
+ temp_operands[1] = GEN_INT ((type_id >> 16) & 0xFFFF);
+ output_asm_insn ("movk\t%w0, #%1, lsl #16", temp_operands);
+
+ /* Compare types. */
+ temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
+ temp_operands[1] = gen_rtx_REG (SImode, R17_REGNUM);
+ output_asm_insn ("cmp\t%w0, %w1", temp_operands);
+
+ /* Output conditional branch to call label. */
+ fputs ("\tb.eq\t", asm_out_file);
+ assemble_name (asm_out_file, call_name);
+ fputc ('\n', asm_out_file);
+
+ /* Output trap label and BRK instruction. */
+ ASM_OUTPUT_LABEL (asm_out_file, trap_name);
+
+ /* Calculate and emit BRK with ESR encoding. */
+ unsigned type_index = R17_REGNUM;
+ unsigned addr_index = REGNO (operands[0]) - R0_REGNUM;
+ unsigned esr_value = 0x8000 | ((type_index & 31) << 5) | (addr_index & 31);
+
+ temp_operands[0] = GEN_INT (esr_value);
+ output_asm_insn ("brk\t#%0", temp_operands);
+
+ /* Output call label. */
+ ASM_OUTPUT_LABEL (asm_out_file, call_name);
+
+ /* Return appropriate call instruction based on SIBLING_CALL_P. */
+ if (SIBLING_CALL_P (insn))
+ return aarch64_indirect_branch_asm (operands[0]);
+ else
+ return aarch64_indirect_call_asm (operands[0]);
+}
+
+#undef TARGET_KCFI_SUPPORTED
+#define TARGET_KCFI_SUPPORTED hook_bool_void_true
+
struct gcc_target targetm = TARGET_INITIALIZER;
#include "gt-aarch64.h"
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index de6aadfd5346..94bb4ffcce98 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -18727,6 +18727,20 @@ header MOVri instruction would become something like this:
@code{movl $199571451, %ebx # hash of foo's type = 0xBE537FB}
+On AArch64, KCFI type identifiers are emitted as a @code{.word ID}
+directive (a 32-bit constant) before the function entry. AArch64's
+natural 4-byte instruction alignment eliminates the need for additional
+alignment NOPs. When used with @option{-fpatchable-function-entry}, the
+type identifier is placed before any prefix NOPs. The runtime check
+uses @code{x16} and @code{x17} as scratch registers. Type mismatches
+trigger a @code{brk} instruction with an immediate value that encodes
+both the expected type register index and the target address register
+index in the format @code{0x8000 | (type_reg << 5) | addr_reg}. This
+encoding is captured in the ESR (Exception Syndrome Register) when the
+trap is taken, allowing the kernel to identify both the KCFI violation
+and the involved registers for detailed diagnostics (eliminating the need
+for a separate @code{.kcfi_traps} section as used on x86_64).
+
KCFI is intended primarily for kernel code and may not be suitable
for user-space applications that rely on techniques incompatible
with strict type checking of indirect calls.
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
index 7c59921e630c..f3d7d23e6af2 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
@@ -63,4 +63,19 @@ __attribute__((noinline)) void test_conditional_call(int flag) {
** ...
*/
+/*
+** test_complex_args: { target aarch64*-*-* }
+** ...
+** ldur w16, \[(x[0-9]+), #-4\]
+** mov w17, #[0-9]+
+** movk w17, #[0-9]+, lsl #16
+** cmp w16, w17
+** b.eq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** brk #[0-9]+
+** .Lkcfi_call\2:
+** br \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
index fe0a21d26df9..6eac946f7abf 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
@@ -59,6 +59,9 @@ int main() {
/* x86_64: Verify type ID in preamble (after NOPs, before function label) */
/* { dg-final { scan-assembler {__cfi_regular_function:\n\t+nop\n.*\n\t+movl\t+\$0x[0-9a-f]+, %eax} { target x86_64-*-* } } } */
+/* AArch64: Verify type ID word in preamble. */
+/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* } } } */
+
/*
** static_caller: { target x86_64-*-* }
** ...
@@ -76,6 +79,21 @@ int main() {
** ...
*/
+/*
+** static_caller: { target aarch64*-*-* }
+** ...
+** ldur w16, \[(x[0-9]+), #-4\]
+** mov w17, #[0-9]+
+** movk w17, #[0-9]+, lsl #16
+** cmp w16, w17
+** b.eq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** brk #[0-9]+
+** .Lkcfi_call\2:
+** blr \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
/* Extern functions should NOT get KCFI preambles. */
@@ -93,3 +111,6 @@ int main() {
/* External functions that are only called directly should NOT get
__kcfi_typeid_ symbols. */
/* { dg-final { scan-assembler-not {__kcfi_typeid_external_func_int} } } */
+
+/* AArch64 should NOT have trap section (use immediate instructions instead). */
+/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
index 05165f0e2851..6062d74ef62a 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
@@ -63,14 +63,18 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
/* Verify we have TWO different KCFI check sequences. */
/* Each check should have different type ID constants. */
/* x86: { dg-final { scan-assembler-times {movl\s+\$-?[0-9]+,\s+%r10d} 2 { target i?86-*-* x86_64-*-* } } } */
+/* AArch64: { dg-final { scan-assembler-times {mov\s+w17, #[0-9]+} 2 { target aarch64*-*-* } } } */
/* Verify the checks use DIFFERENT type IDs (not shared).
We should NOT see the same type ID used twice - that would indicate
unmerged sharing. */
/* x86: { dg-final { scan-assembler-not {movl\s+\$(-?[0-9]+),\s+%r10d.*movl\s+\$\1,\s+%r10d} { target i?86-*-* x86_64-*-* } } } */
+/* AArch64: { dg-final { scan-assembler-not {mov\s+w17, #([0-9]+).*mov\s+w17, #\1} { target aarch64*-*-* } } } */
/* Verify expected number of traps. */
/* x86: { dg-final { scan-assembler-times {ud2} 2 { target i?86-*-* x86_64-*-* } } } */
+/* AArch64: { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
/* Verify 2 separate call sites. */
/* x86: { dg-final { scan-assembler-times {jmp\s+\*%[a-z0-9]+} 2 { target i?86-*-* x86_64-*-* } } } */
+/* AArch64: { dg-final { scan-assembler-times {br\tx[0-9]+} 2 { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
index ed415033c5c9..3ffbd408a69e 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
@@ -146,4 +146,20 @@ int main() {
** ...
*/
+/* Standard KCFI handling. */
+/*
+** main: { target aarch64*-*-* }
+** ...
+** ldur w16, \[(x[0-9]+), #-4\]
+** mov w17, #[0-9]+
+** movk w17, #[0-9]+, lsl #16
+** cmp w16, w17
+** b.eq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** brk #[0-9]+
+** .Lkcfi_call\2:
+** blr \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
index 5553ff47174b..df39b7f0a8a3 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
@@ -57,4 +57,24 @@ int main(void)
** ...
*/
+/*
+** indirect_call: { target aarch64*-*-* }
+** ...
+** mov (x[0-9]+), x0
+** ...
+** ldur w16, \[\1, #-4\]
+** mov w17, #[0-9]+
+** movk w17, #[0-9]+, lsl #16
+** cmp w16, w17
+** b.eq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** brk #[0-9]+
+** .Lkcfi_call\2:
+** br \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
+
+/* AArch64 should NOT have trap section (use immediate instructions instead). */
+/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
index 9ed7e21fe8eb..cdeb202ffd12 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
@@ -75,11 +75,16 @@ int main(void)
/* Verify correct number of KCFI checks: exactly 2 */
/* { dg-final { scan-assembler-times {ud2} 2 { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
/* Positive controls: these should have KCFI checks. */
/* { dg-final { scan-assembler {normal_function:.*ud2.*\.size\s+normal_function} { target x86_64-*-* } } } */
/* { dg-final { scan-assembler {wrap_normal_inline:.*ud2.*\.size\s+wrap_normal_inline} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler {normal_function:.*brk\s+#[0-9]+.*\.size\s+normal_function} { target aarch64*-*-* } } } */
+/* { dg-final { scan-assembler {wrap_normal_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_normal_inline} { target aarch64*-*-* } } } */
/* Negative controls: these should NOT have KCFI checks. */
/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ud2.*\.size\s+sensitive_non_inline_function} { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*ud2.*\.size\s+wrap_sensitive_inline} { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*brk\s+#[0-9]+.*\.size\s+sensitive_non_inline_function} { target aarch64*-*-* } } } */
+/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_sensitive_inline} { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
index 95a8e8419e00..af6d86803576 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
@@ -34,3 +34,4 @@ int main() {
caller_no_checks() should NOT generate KCFI check (no_sanitize).
So a total of exactly 1 KCFI check in the entire program. */
/* { dg-final { scan-assembler-times {addl\t-4\(%r[ad]x\), %r1[01]d} 1 { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 1 { target aarch64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
index 97d964feebd3..0ced5c43ae92 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
@@ -27,3 +27,6 @@ int main() {
/* x86_64: All call sites should use -4 offset for KCFI type ID loads, even
with -falign-functions=16 (we're not using patchable entries here). */
/* { dg-final { scan-assembler {movl\t\$-?[0-9]+, %r10d\n\taddl\t-4\(%r[a-z0-9]+\), %r10d} { target x86_64-*-* } } } */
+
+/* AArch64: All call sites should use -4 offset. */
+/* { dg-final { scan-assembler {ldur\tw16, \[x[0-9]+, #-4\]} { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
index 379356385a16..7a251cbdee3b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
@@ -28,6 +28,11 @@ int main() {
** movl \$0x[0-9a-f]+, %eax
*/
+/*
+** __cfi_test_function: { target aarch64*-*-* }
+** .word 0x[0-9a-f]+
+*/
+
/*
** main: { target x86_64-*-* }
** ...
@@ -35,4 +40,11 @@ int main() {
** ...
*/
+/*
+** main: { target aarch64*-*-* }
+** ...
+** ldur w16, \[x[0-9]+, #-4\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
index 06df3495bb23..3ed5d16c8e91 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
@@ -17,6 +17,11 @@ int main() {
** movl \$0x[0-9a-f]+, %eax
*/
+/*
+** __cfi_test_function: { target aarch64*-*-* }
+** .word 0x[0-9a-f]+
+*/
+
/*
** main: { target x86_64-*-* }
** ...
@@ -24,4 +29,11 @@ int main() {
** ...
*/
+/*
+** main: { target aarch64*-*-* }
+** ...
+** ldur w16, \[x[0-9]+, #-48\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
index ef87b135934b..e354914209e9 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
@@ -24,6 +24,11 @@ int main() {
** movl \$0x[0-9a-f]+, %eax
*/
+/*
+** __cfi_test_function: { target aarch64*-*-* }
+** .word 0x[0-9a-f]+
+*/
+
/*
** main: { target x86_64-*-* }
** ...
@@ -31,4 +36,11 @@ int main() {
** ...
*/
+/*
+** main: { target aarch64*-*-* }
+** ...
+** ldur w16, \[x[0-9]+, #-20\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
index 872814aa4171..7a1dc4fa0e07 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
@@ -25,6 +25,11 @@ int main() {
** movl \$0x[0-9a-f]+, %eax
*/
+/*
+** __cfi_test_function: { target aarch64*-*-* }
+** .word 0x[0-9a-f]+
+*/
+
/*
** main: { target x86_64-*-* }
** ...
@@ -32,4 +37,11 @@ int main() {
** ...
*/
+/*
+** main: { target aarch64*-*-* }
+** ...
+** ldur w16, \[x[0-9]+, #-16\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
index 04a9eb1fd206..1a7cc4aa167f 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
@@ -78,3 +78,22 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
/* Should have exactly 1 regular call (non-tail call case). */
/* { dg-final { scan-assembler-times {call\t\*%[a-z0-9]+} 1 { target x86_64-*-* } } } */
+
+/* Should have exactly 4 KCFI checks for indirect calls (load type ID from
+ -4 offset + compare). */
+/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 4 { target aarch64-*-* } } } */
+/* { dg-final { scan-assembler-times {cmp\tw16, w17} 4 { target aarch64-*-* } } } */
+
+/* Should have exactly 4 trap instructions. */
+/* { dg-final { scan-assembler-times {brk\t#[0-9]+} 4 { target aarch64-*-* } } } */
+
+/* Should have exactly 3 protected tail calls (br through register after
+ KCFI check). */
+/* { dg-final { scan-assembler-times {br\tx[0-9]+} 3 { target aarch64-*-* } } } */
+
+/* Should have exactly 1 regular call (non-tail call case). */
+/* { dg-final { scan-assembler-times {blr\tx[0-9]+} 1 { target aarch64-*-* } } } */
+
+/* Type ID loading should use mov + movk pattern for 32-bit constants. */
+/* { dg-final { scan-assembler {mov\tw17, #[0-9]+} { target aarch64-*-* } } } */
+/* { dg-final { scan-assembler {movk\tw17, #[0-9]+, lsl #16} { target aarch64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
new file mode 100644
index 000000000000..0c257565c9e8
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
@@ -0,0 +1,41 @@
+/* Test AArch64 and ARM32 KCFI trap encoding in BRK/UDF instructions. */
+/* { dg-do compile { target aarch64*-*-* } } */
+
+void target_function(int x, char y) {
+}
+
+int main() {
+ void (*func_ptr)(int, char) = target_function;
+
+ /* This should generate trap with immediate encoding. */
+ func_ptr(42, 'a');
+
+ return 0;
+}
+
+/* Should have KCFI preamble. */
+/* { dg-final { scan-assembler "__cfi_target_function:" } } */
+
+/* AArch64 specific: Should have BRK instruction with proper ESR encoding
+ ESR format: 0x8000 | ((type_reg & 31) << 5) | (addr_reg & 31)
+
+ Test the ESR encoding by checking for the expected value.
+ Since we know this test uses x2, we expect ESR = 0x8000 | (17<<5) | 2 = 33314
+ */
+
+/*
+** main: { target aarch64*-*-* }
+** ...
+** ldur w16, \[x[0-9]+, #-4\]
+** mov w17, #[0-9]+
+** movk w17, #[0-9]+, lsl #16
+** cmp w16, w17
+** b\.eq .Lkcfi_call[0-9]+
+** .Lkcfi_trap[0-9]+:
+** brk #33314
+** .Lkcfi_call[0-9]+:
+** blr x2
+** ...
+*/
+
+/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
index 55c0829ccd7b..e92873e51321 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
@@ -18,6 +18,10 @@ int main() {
/* Should have exactly 2 trap labels in code. */
/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ud2} 2 { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*brk} 2 { target aarch64*-*-* } } } */
/* x86_64 should exactly 2 .kcfi_traps sections. */
/* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* } } } */
+
+/* AArch64 should NOT have .kcfi_traps section. */
+/* { dg-final { scan-assembler-not {\.section\t+\.kcfi_traps} { target aarch64*-*-* } } } */
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-20 22:21 ` [PATCH v8 5/7] aarch64: Add AArch64 " Kees Cook
@ 2025-11-20 22:57 ` Andrew Pinski
2025-11-20 23:10 ` Andrew Pinski
2025-11-21 0:30 ` Kees Cook
0 siblings, 2 replies; 14+ messages in thread
From: Andrew Pinski @ 2025-11-20 22:57 UTC (permalink / raw)
To: Kees Cook
Cc: Qing Zhao, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
On Thu, Nov 20, 2025 at 2:29 PM Kees Cook <kees@kernel.org> wrote:
>
> Implement AArch64-specific KCFI backend.
>
> - Trap debugging through ESR (Exception Syndrome Register) encoding
> in BRK instruction immediate values.
>
> - Scratch register allocation using w16/w17 (x16/x17) following
> AArch64 procedure call standard for intra-procedure-call registers,
> which already makes x16/x17 available through existing clobbers.
>
> - Complementary with BTI (which uses a separate pass system to inject
> landing instructions where needed).
>
> - Does not interfere with SME, which uses attributes not function
> prototypes for distinguishing functions.
>
> Assembly Code Pattern for AArch64:
> ldur w16, [target, #-4] ; Load actual type ID from preamble
> mov w17, #type_id_low ; Load expected type (lower 16 bits)
> movk w17, #type_id_high, lsl #16 ; Load upper 16 bits if needed
> cmp w16, w17 ; Compare type IDs directly
> b.eq .Lpass ; Branch if types match
> .Ltrap: brk #esr_value ; Enhanced trap with register info
> .Lpass: blr/br target ; Execute validated indirect transfer
>
> ESR (Exception Syndrome Register) Integration:
> - BRK instruction immediate encoding format:
> 0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31)
> - TypeIndex indicates which W register contains expected type (W17 = 17)
> - AddrIndex indicates which X register contains target address (0-30)
> - Example: brk #33313 (0x8221) = expected type in W17, target address in X1
>
> Build and run tested with Linux kernel ARCH=arm64.
>
> gcc/ChangeLog:
>
> config/aarch64/aarch64-protos.h: Declare aarch64_indirect_branch_asm,
> and KCFI helpers.
> config/aarch64/aarch64.cc (aarch64_expand_call): Wrap CALLs in
> KCFI, with clobbers.
> (aarch64_indirect_branch_asm): New function, extract common
> logic for branch asm, like existing call asm helper.
> (aarch64_output_kcfi_insn): Emit KCFI assembly.
> config/aarch64/aarch64.md: Add KCFI RTL patterns and replace
> open-coded branch emission with aarch64_indirect_branch_asm.
> doc/invoke.texi: Document aarch64 nuances.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.dg/kcfi/kcfi-adjacency.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-basics.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-call-sharing.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-complex-addressing.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-move-preservation.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-no-sanitize.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-offset-validation.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-patchable-large.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-patchable-medium.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-tail-calls.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-trap-section.c: Add aarch64 patterns.
> * gcc.dg/kcfi/kcfi-trap-encoding.c: New test.
>
> Signed-off-by: Kees Cook <kees@kernel.org>
> ---
> gcc/config/aarch64/aarch64-protos.h | 5 +
> gcc/config/aarch64/aarch64.md | 66 +++++++++--
> gcc/config/aarch64/aarch64.cc | 105 ++++++++++++++++++
> gcc/doc/invoke.texi | 14 +++
> gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 15 +++
> gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++++
> gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
> .../gcc.dg/kcfi/kcfi-complex-addressing.c | 16 +++
> .../gcc.dg/kcfi/kcfi-move-preservation.c | 20 ++++
> .../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
> gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
> .../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
> .../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 12 ++
> .../gcc.dg/kcfi/kcfi-patchable-large.c | 12 ++
> .../gcc.dg/kcfi/kcfi-patchable-medium.c | 12 ++
> .../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 12 ++
> gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 19 ++++
> .../gcc.dg/kcfi/kcfi-trap-encoding.c | 41 +++++++
> gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 4 +
> 19 files changed, 379 insertions(+), 8 deletions(-)
> create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
>
> diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
> index a9e407ba340e..c32d454fe277 100644
> --- a/gcc/config/aarch64/aarch64-protos.h
> +++ b/gcc/config/aarch64/aarch64-protos.h
> @@ -1272,6 +1272,7 @@ tree aarch64_resolve_overloaded_builtin_general (location_t, tree, void *);
>
> const char *aarch64_sls_barrier (int);
> const char *aarch64_indirect_call_asm (rtx);
> +const char *aarch64_indirect_branch_asm (rtx);
> extern bool aarch64_harden_sls_retbr_p (void);
> extern bool aarch64_harden_sls_blr_p (void);
>
> @@ -1295,4 +1296,8 @@ extern unsigned aarch64_stack_alignment (const_tree exp, unsigned align);
> extern rtx aarch64_gen_compare_zero_and_branch (rtx_code code, rtx x,
> rtx_code_label *label);
>
> +/* KCFI support. */
> +extern void kcfi_emit_trap_with_section (FILE *file, rtx trap_label_rtx);
> +extern const char *aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands);
> +
> #endif /* GCC_AARCH64_PROTOS_H */
> diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
> index 855df791bae7..84894816509c 100644
> --- a/gcc/config/aarch64/aarch64.md
> +++ b/gcc/config/aarch64/aarch64.md
> @@ -1506,6 +1506,19 @@
> }"
> )
>
> +;; KCFI indirect call
> +(define_insn "*call_insn"
> + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucr"))
> + (match_operand 1 "" ""))
> + (match_operand 3 "const_int_operand"))
> + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> + (clobber (reg:DI LR_REGNUM))]
> + "!SIBLING_CALL_P (insn)"
> +{
> + return aarch64_output_kcfi_insn (insn, operands);
> +}
> + [(set_attr "type" "call")])
> +
> (define_insn "*call_insn"
> [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand"))
> (match_operand 1 "" ""))
> @@ -1533,6 +1546,21 @@
> }"
> )
>
> +;; KCFI call with return value
> +(define_insn "*call_value_insn"
> + [(set (match_operand 0 "" "")
> + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> + "Ucr"))
> + (match_operand 2 "" ""))
> + (match_operand 4 "const_int_operand")))
> + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> + (clobber (reg:DI LR_REGNUM))]
> + "!SIBLING_CALL_P (insn)"
> +{
> + return aarch64_output_kcfi_insn (insn, &operands[1]);
> +}
> + [(set_attr "type" "call")])
> +
> (define_insn "*call_value_insn"
> [(set (match_operand 0 "" "")
> (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"))
> @@ -1573,6 +1601,19 @@
> }
> )
>
> +;; KCFI sibling call
> +(define_insn "*sibcall_insn"
> + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs"))
> + (match_operand 1 ""))
> + (match_operand 3 "const_int_operand"))
> + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> + (return)]
> + "SIBLING_CALL_P (insn)"
> +{
> + return aarch64_output_kcfi_insn (insn, operands);
> +}
> + [(set_attr "type" "branch")])
> +
> (define_insn "*sibcall_insn"
> [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs, Usf"))
> (match_operand 1 ""))
> @@ -1581,16 +1622,28 @@
> "SIBLING_CALL_P (insn)"
> {
> if (which_alternative == 0)
> - {
> - output_asm_insn ("br\\t%0", operands);
> - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> - }
> + return aarch64_indirect_branch_asm (operands[0]);
Can you extract aarch64_indirect_branch_asm into a different patch?
Since it is only slightly related to this.
The main reason is because applying that can happen before the rest of
the KFCI patches are approved. And I will handling applying that once
it has been extracted out.
> return "b\\t%c0";
> }
> [(set_attr "type" "branch, branch")
> (set_attr "sls_length" "retbr,none")]
> )
>
> +;; KCFI sibling call with return value
> +(define_insn "*sibcall_value_insn"
> + [(set (match_operand 0 "")
> + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> + "Ucs"))
> + (match_operand 2 ""))
> + (match_operand 4 "const_int_operand")))
> + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> + (return)]
> + "SIBLING_CALL_P (insn)"
> +{
> + return aarch64_output_kcfi_insn (insn, &operands[1]);
> +}
> + [(set_attr "type" "branch")])
> +
> (define_insn "*sibcall_value_insn"
> [(set (match_operand 0 "")
> (call (mem:DI
> @@ -1601,10 +1654,7 @@
> "SIBLING_CALL_P (insn)"
> {
> if (which_alternative == 0)
> - {
> - output_asm_insn ("br\\t%1", operands);
> - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> - }
> + return aarch64_indirect_branch_asm (operands[1]);
> return "b\\t%c1";
> }
> [(set_attr "type" "branch, branch")
> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> index 6dfdaa4fb9b0..86e3af992d2a 100644
> --- a/gcc/config/aarch64/aarch64.cc
> +++ b/gcc/config/aarch64/aarch64.cc
> @@ -83,6 +83,7 @@
> #include "rtlanal.h"
> #include "tree-dfa.h"
> #include "asan.h"
> +#include "kcfi.h"
> #include "aarch64-elf-metadata.h"
> #include "aarch64-feature-deps.h"
> #include "config/arm/aarch-common.h"
> @@ -11965,6 +11966,16 @@ aarch64_expand_call (rtx result, rtx mem, rtx cookie, bool sibcall)
>
> call = gen_rtx_CALL (VOIDmode, mem, const0_rtx);
>
> + /* Only indirect calls need KCFI instrumentation. */
> + bool is_direct_call = SYMBOL_REF_P (XEXP (mem, 0));
> + rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
> + : kcfi_get_type_id_for_expanding_gimple_call ();
> + if (kcfi_type_rtx)
> + {
> + /* Wrap call in KCFI. */
> + call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
> + }
> +
> if (result != NULL_RTX)
> call = gen_rtx_SET (result, call);
>
> @@ -19225,6 +19236,9 @@ aarch64_override_options_internal (struct gcc_options *opts)
> #endif
> }
>
> + if ((flag_sanitize & SANITIZE_KCFI) && TARGET_ILP32)
> + sorry ("%<-fsanitize=kcfi%> is not supported for %<-mabi=ilp32%>");
> +
> aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
> if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
> && !(isa_flags & AARCH64_FL_SME))
> @@ -30822,6 +30836,18 @@ aarch64_indirect_call_asm (rtx addr)
> return "";
> }
>
> +/* Generate assembly for AArch64 indirect branch instruction. ADDR is the
> + target address register. Returns any additional barrier instructions
> + needed for SLS (Straight Line Speculation) mitigation. */
> +
> +const char *
> +aarch64_indirect_branch_asm (rtx addr)
> +{
> + gcc_assert (REG_P (addr));
> + output_asm_insn ("br\t%0", &addr);
> + return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> +}
> +
> /* Emit the assembly instruction to load the thread pointer into DEST.
> Select between different tpidr_elN registers depending on -mtp= setting. */
>
> @@ -33089,6 +33115,85 @@ aarch64_libgcc_floating_mode_supported_p
> #undef TARGET_DOCUMENTATION_NAME
> #define TARGET_DOCUMENTATION_NAME "AArch64"
>
> +/* Output the assembly for a KCFI checked call instruction. INSN is the
> + RTL instruction being processed. OPERANDS is the array of RTL operands
> + where operands[0] is the call target register, operands[3] is the KCFI
> + type ID constant. Returns the appropriate call instruction string. */
> +
> +const char *
> +aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands)
> +{
> + /* Target register is operands[0]. */
> + rtx target_reg = operands[0];
> + gcc_assert (REG_P (target_reg));
> +
> + /* Get KCFI type ID from operand[3]. */
> + uint32_t type_id = (uint32_t) INTVAL (operands[3]);
> +
> + /* Calculate typeid offset from call target. */
> + HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
> +
> + /* Get unique label number for this KCFI check. */
> + int labelno = kcfi_next_labelno ();
> +
> + /* Generate custom label names. */
> + char trap_name[32];
> + char call_name[32];
> + ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
> + ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
> +
> + rtx temp_operands[3];
> +
> + /* Load actual type into w16 from memory at offset using ldur. */
> + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> + temp_operands[1] = target_reg;
> + temp_operands[2] = GEN_INT (offset);
> + output_asm_insn ("ldur\t%w0, [%1, #%2]", temp_operands);
Why not just:
temp_operands[0] = target_reg;
temp_operands[0] = GEN_INT (offset);
output_asm_insn ("ldur\tw16, [%0, #%1]", temp_operands);
> +
> + /* Load expected type low 16 bits into w17. */
> + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> + temp_operands[1] = GEN_INT (type_id & 0xFFFF);
> + output_asm_insn ("mov\t%w0, #%1", temp_operands);
Likewise.
Also why not use fprintf here?
Like:
fprintf(asm_out_file, "mov\t%w0, #%d\n", type_id & 0xFFFF);
> +
> + /* Load expected type high 16 bits into w17. */
> + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> + temp_operands[1] = GEN_INT ((type_id >> 16) & 0xFFFF);
> + output_asm_insn ("movk\t%w0, #%1, lsl #16", temp_operands);
Likewise on both comments/points.
At least reuse the gen_rtx_REG for 16/17.
> +
> + /* Compare types. */
> + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> + temp_operands[1] = gen_rtx_REG (SImode, R17_REGNUM);
> + output_asm_insn ("cmp\t%w0, %w1", temp_operands);
fprintf (asm_out_file, "cmp\tw16, w17\n");
> +
> + /* Output conditional branch to call label. */
> + fputs ("\tb.eq\t", asm_out_file);
> + assemble_name (asm_out_file, call_name);
> + fputc ('\n', asm_out_file);
> +
> + /* Output trap label and BRK instruction. */
> + ASM_OUTPUT_LABEL (asm_out_file, trap_name);
> +
> + /* Calculate and emit BRK with ESR encoding. */
> + unsigned type_index = R17_REGNUM;
> + unsigned addr_index = REGNO (operands[0]) - R0_REGNUM;
> + unsigned esr_value = 0x8000 | ((type_index & 31) << 5) | (addr_index & 31);
> +
> + temp_operands[0] = GEN_INT (esr_value);
> + output_asm_insn ("brk\t#%0", temp_operands);
> +
> + /* Output call label. */
> + ASM_OUTPUT_LABEL (asm_out_file, call_name);
> +
> + /* Return appropriate call instruction based on SIBLING_CALL_P. */
> + if (SIBLING_CALL_P (insn))
> + return aarch64_indirect_branch_asm (operands[0]);
> + else
> + return aarch64_indirect_call_asm (operands[0]);
> +}
> +
> +#undef TARGET_KCFI_SUPPORTED
> +#define TARGET_KCFI_SUPPORTED hook_bool_void_true
Why return true always for it being supported? I thought this is not
supported with ilp32, it is definitely not supported for mingw. Is it
supported with -elf targets rather than -linux-gnu targets?
Also I am still trying to figure out and understand the interaction
between x16 and x17 in some cases.
Because I thought indirect calls/jumps will be using x16/x17 for those
to support BTI.
Thanks,
Andrew
> +
> struct gcc_target targetm = TARGET_INITIALIZER;
>
> #include "gt-aarch64.h"
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index de6aadfd5346..94bb4ffcce98 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -18727,6 +18727,20 @@ header MOVri instruction would become something like this:
>
> @code{movl $199571451, %ebx # hash of foo's type = 0xBE537FB}
>
> +On AArch64, KCFI type identifiers are emitted as a @code{.word ID}
> +directive (a 32-bit constant) before the function entry. AArch64's
> +natural 4-byte instruction alignment eliminates the need for additional
> +alignment NOPs. When used with @option{-fpatchable-function-entry}, the
> +type identifier is placed before any prefix NOPs. The runtime check
> +uses @code{x16} and @code{x17} as scratch registers. Type mismatches
> +trigger a @code{brk} instruction with an immediate value that encodes
> +both the expected type register index and the target address register
> +index in the format @code{0x8000 | (type_reg << 5) | addr_reg}. This
> +encoding is captured in the ESR (Exception Syndrome Register) when the
> +trap is taken, allowing the kernel to identify both the KCFI violation
> +and the involved registers for detailed diagnostics (eliminating the need
> +for a separate @code{.kcfi_traps} section as used on x86_64).
> +
> KCFI is intended primarily for kernel code and may not be suitable
> for user-space applications that rely on techniques incompatible
> with strict type checking of indirect calls.
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
> index 7c59921e630c..f3d7d23e6af2 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
> @@ -63,4 +63,19 @@ __attribute__((noinline)) void test_conditional_call(int flag) {
> ** ...
> */
>
> +/*
> +** test_complex_args: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[(x[0-9]+), #-4\]
> +** mov w17, #[0-9]+
> +** movk w17, #[0-9]+, lsl #16
> +** cmp w16, w17
> +** b.eq .Lkcfi_call([0-9]+)
> +** .Lkcfi_trap[0-9]+:
> +** brk #[0-9]+
> +** .Lkcfi_call\2:
> +** br \1
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
> index fe0a21d26df9..6eac946f7abf 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
> @@ -59,6 +59,9 @@ int main() {
> /* x86_64: Verify type ID in preamble (after NOPs, before function label) */
> /* { dg-final { scan-assembler {__cfi_regular_function:\n\t+nop\n.*\n\t+movl\t+\$0x[0-9a-f]+, %eax} { target x86_64-*-* } } } */
>
> +/* AArch64: Verify type ID word in preamble. */
> +/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* } } } */
> +
> /*
> ** static_caller: { target x86_64-*-* }
> ** ...
> @@ -76,6 +79,21 @@ int main() {
> ** ...
> */
>
> +/*
> +** static_caller: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[(x[0-9]+), #-4\]
> +** mov w17, #[0-9]+
> +** movk w17, #[0-9]+, lsl #16
> +** cmp w16, w17
> +** b.eq .Lkcfi_call([0-9]+)
> +** .Lkcfi_trap[0-9]+:
> +** brk #[0-9]+
> +** .Lkcfi_call\2:
> +** blr \1
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
>
> /* Extern functions should NOT get KCFI preambles. */
> @@ -93,3 +111,6 @@ int main() {
> /* External functions that are only called directly should NOT get
> __kcfi_typeid_ symbols. */
> /* { dg-final { scan-assembler-not {__kcfi_typeid_external_func_int} } } */
> +
> +/* AArch64 should NOT have trap section (use immediate instructions instead). */
> +/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
> index 05165f0e2851..6062d74ef62a 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
> @@ -63,14 +63,18 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
> /* Verify we have TWO different KCFI check sequences. */
> /* Each check should have different type ID constants. */
> /* x86: { dg-final { scan-assembler-times {movl\s+\$-?[0-9]+,\s+%r10d} 2 { target i?86-*-* x86_64-*-* } } } */
> +/* AArch64: { dg-final { scan-assembler-times {mov\s+w17, #[0-9]+} 2 { target aarch64*-*-* } } } */
>
> /* Verify the checks use DIFFERENT type IDs (not shared).
> We should NOT see the same type ID used twice - that would indicate
> unmerged sharing. */
> /* x86: { dg-final { scan-assembler-not {movl\s+\$(-?[0-9]+),\s+%r10d.*movl\s+\$\1,\s+%r10d} { target i?86-*-* x86_64-*-* } } } */
> +/* AArch64: { dg-final { scan-assembler-not {mov\s+w17, #([0-9]+).*mov\s+w17, #\1} { target aarch64*-*-* } } } */
>
> /* Verify expected number of traps. */
> /* x86: { dg-final { scan-assembler-times {ud2} 2 { target i?86-*-* x86_64-*-* } } } */
> +/* AArch64: { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
>
> /* Verify 2 separate call sites. */
> /* x86: { dg-final { scan-assembler-times {jmp\s+\*%[a-z0-9]+} 2 { target i?86-*-* x86_64-*-* } } } */
> +/* AArch64: { dg-final { scan-assembler-times {br\tx[0-9]+} 2 { target aarch64*-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
> index ed415033c5c9..3ffbd408a69e 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
> @@ -146,4 +146,20 @@ int main() {
> ** ...
> */
>
> +/* Standard KCFI handling. */
> +/*
> +** main: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[(x[0-9]+), #-4\]
> +** mov w17, #[0-9]+
> +** movk w17, #[0-9]+, lsl #16
> +** cmp w16, w17
> +** b.eq .Lkcfi_call([0-9]+)
> +** .Lkcfi_trap[0-9]+:
> +** brk #[0-9]+
> +** .Lkcfi_call\2:
> +** blr \1
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
> index 5553ff47174b..df39b7f0a8a3 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
> @@ -57,4 +57,24 @@ int main(void)
> ** ...
> */
>
> +/*
> +** indirect_call: { target aarch64*-*-* }
> +** ...
> +** mov (x[0-9]+), x0
> +** ...
> +** ldur w16, \[\1, #-4\]
> +** mov w17, #[0-9]+
> +** movk w17, #[0-9]+, lsl #16
> +** cmp w16, w17
> +** b.eq .Lkcfi_call([0-9]+)
> +** .Lkcfi_trap[0-9]+:
> +** brk #[0-9]+
> +** .Lkcfi_call\2:
> +** br \1
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> +
> +/* AArch64 should NOT have trap section (use immediate instructions instead). */
> +/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
> index 9ed7e21fe8eb..cdeb202ffd12 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
> @@ -75,11 +75,16 @@ int main(void)
>
> /* Verify correct number of KCFI checks: exactly 2 */
> /* { dg-final { scan-assembler-times {ud2} 2 { target x86_64-*-* } } } */
> +/* { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
>
> /* Positive controls: these should have KCFI checks. */
> /* { dg-final { scan-assembler {normal_function:.*ud2.*\.size\s+normal_function} { target x86_64-*-* } } } */
> /* { dg-final { scan-assembler {wrap_normal_inline:.*ud2.*\.size\s+wrap_normal_inline} { target x86_64-*-* } } } */
> +/* { dg-final { scan-assembler {normal_function:.*brk\s+#[0-9]+.*\.size\s+normal_function} { target aarch64*-*-* } } } */
> +/* { dg-final { scan-assembler {wrap_normal_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_normal_inline} { target aarch64*-*-* } } } */
>
> /* Negative controls: these should NOT have KCFI checks. */
> /* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ud2.*\.size\s+sensitive_non_inline_function} { target x86_64-*-* } } } */
> /* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*ud2.*\.size\s+wrap_sensitive_inline} { target x86_64-*-* } } } */
> +/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*brk\s+#[0-9]+.*\.size\s+sensitive_non_inline_function} { target aarch64*-*-* } } } */
> +/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_sensitive_inline} { target aarch64*-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
> index 95a8e8419e00..af6d86803576 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
> @@ -34,3 +34,4 @@ int main() {
> caller_no_checks() should NOT generate KCFI check (no_sanitize).
> So a total of exactly 1 KCFI check in the entire program. */
> /* { dg-final { scan-assembler-times {addl\t-4\(%r[ad]x\), %r1[01]d} 1 { target x86_64-*-* } } } */
> +/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 1 { target aarch64-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
> index 97d964feebd3..0ced5c43ae92 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
> @@ -27,3 +27,6 @@ int main() {
> /* x86_64: All call sites should use -4 offset for KCFI type ID loads, even
> with -falign-functions=16 (we're not using patchable entries here). */
> /* { dg-final { scan-assembler {movl\t\$-?[0-9]+, %r10d\n\taddl\t-4\(%r[a-z0-9]+\), %r10d} { target x86_64-*-* } } } */
> +
> +/* AArch64: All call sites should use -4 offset. */
> +/* { dg-final { scan-assembler {ldur\tw16, \[x[0-9]+, #-4\]} { target aarch64*-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
> index 379356385a16..7a251cbdee3b 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
> @@ -28,6 +28,11 @@ int main() {
> ** movl \$0x[0-9a-f]+, %eax
> */
>
> +/*
> +** __cfi_test_function: { target aarch64*-*-* }
> +** .word 0x[0-9a-f]+
> +*/
> +
> /*
> ** main: { target x86_64-*-* }
> ** ...
> @@ -35,4 +40,11 @@ int main() {
> ** ...
> */
>
> +/*
> +** main: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[x[0-9]+, #-4\]
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
> index 06df3495bb23..3ed5d16c8e91 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
> @@ -17,6 +17,11 @@ int main() {
> ** movl \$0x[0-9a-f]+, %eax
> */
>
> +/*
> +** __cfi_test_function: { target aarch64*-*-* }
> +** .word 0x[0-9a-f]+
> +*/
> +
> /*
> ** main: { target x86_64-*-* }
> ** ...
> @@ -24,4 +29,11 @@ int main() {
> ** ...
> */
>
> +/*
> +** main: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[x[0-9]+, #-48\]
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
> index ef87b135934b..e354914209e9 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
> @@ -24,6 +24,11 @@ int main() {
> ** movl \$0x[0-9a-f]+, %eax
> */
>
> +/*
> +** __cfi_test_function: { target aarch64*-*-* }
> +** .word 0x[0-9a-f]+
> +*/
> +
> /*
> ** main: { target x86_64-*-* }
> ** ...
> @@ -31,4 +36,11 @@ int main() {
> ** ...
> */
>
> +/*
> +** main: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[x[0-9]+, #-20\]
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
> index 872814aa4171..7a1dc4fa0e07 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
> @@ -25,6 +25,11 @@ int main() {
> ** movl \$0x[0-9a-f]+, %eax
> */
>
> +/*
> +** __cfi_test_function: { target aarch64*-*-* }
> +** .word 0x[0-9a-f]+
> +*/
> +
> /*
> ** main: { target x86_64-*-* }
> ** ...
> @@ -32,4 +37,11 @@ int main() {
> ** ...
> */
>
> +/*
> +** main: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[x[0-9]+, #-16\]
> +** ...
> +*/
> +
> /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
> index 04a9eb1fd206..1a7cc4aa167f 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
> @@ -78,3 +78,22 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
>
> /* Should have exactly 1 regular call (non-tail call case). */
> /* { dg-final { scan-assembler-times {call\t\*%[a-z0-9]+} 1 { target x86_64-*-* } } } */
> +
> +/* Should have exactly 4 KCFI checks for indirect calls (load type ID from
> + -4 offset + compare). */
> +/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 4 { target aarch64-*-* } } } */
> +/* { dg-final { scan-assembler-times {cmp\tw16, w17} 4 { target aarch64-*-* } } } */
> +
> +/* Should have exactly 4 trap instructions. */
> +/* { dg-final { scan-assembler-times {brk\t#[0-9]+} 4 { target aarch64-*-* } } } */
> +
> +/* Should have exactly 3 protected tail calls (br through register after
> + KCFI check). */
> +/* { dg-final { scan-assembler-times {br\tx[0-9]+} 3 { target aarch64-*-* } } } */
> +
> +/* Should have exactly 1 regular call (non-tail call case). */
> +/* { dg-final { scan-assembler-times {blr\tx[0-9]+} 1 { target aarch64-*-* } } } */
> +
> +/* Type ID loading should use mov + movk pattern for 32-bit constants. */
> +/* { dg-final { scan-assembler {mov\tw17, #[0-9]+} { target aarch64-*-* } } } */
> +/* { dg-final { scan-assembler {movk\tw17, #[0-9]+, lsl #16} { target aarch64-*-* } } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> new file mode 100644
> index 000000000000..0c257565c9e8
> --- /dev/null
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> @@ -0,0 +1,41 @@
> +/* Test AArch64 and ARM32 KCFI trap encoding in BRK/UDF instructions. */
> +/* { dg-do compile { target aarch64*-*-* } } */
> +
> +void target_function(int x, char y) {
> +}
> +
> +int main() {
> + void (*func_ptr)(int, char) = target_function;
> +
> + /* This should generate trap with immediate encoding. */
> + func_ptr(42, 'a');
> +
> + return 0;
> +}
> +
> +/* Should have KCFI preamble. */
> +/* { dg-final { scan-assembler "__cfi_target_function:" } } */
> +
> +/* AArch64 specific: Should have BRK instruction with proper ESR encoding
> + ESR format: 0x8000 | ((type_reg & 31) << 5) | (addr_reg & 31)
> +
> + Test the ESR encoding by checking for the expected value.
> + Since we know this test uses x2, we expect ESR = 0x8000 | (17<<5) | 2 = 33314
> + */
> +
> +/*
> +** main: { target aarch64*-*-* }
> +** ...
> +** ldur w16, \[x[0-9]+, #-4\]
> +** mov w17, #[0-9]+
> +** movk w17, #[0-9]+, lsl #16
> +** cmp w16, w17
> +** b\.eq .Lkcfi_call[0-9]+
> +** .Lkcfi_trap[0-9]+:
> +** brk #33314
> +** .Lkcfi_call[0-9]+:
> +** blr x2
> +** ...
> +*/
> +
> +/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*} } } */
> diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
> index 55c0829ccd7b..e92873e51321 100644
> --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
> +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
> @@ -18,6 +18,10 @@ int main() {
>
> /* Should have exactly 2 trap labels in code. */
> /* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ud2} 2 { target x86_64-*-* } } } */
> +/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*brk} 2 { target aarch64*-*-* } } } */
>
> /* x86_64 should exactly 2 .kcfi_traps sections. */
> /* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* } } } */
> +
> +/* AArch64 should NOT have .kcfi_traps section. */
> +/* { dg-final { scan-assembler-not {\.section\t+\.kcfi_traps} { target aarch64*-*-* } } } */
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-20 22:57 ` Andrew Pinski
@ 2025-11-20 23:10 ` Andrew Pinski
2025-11-21 0:18 ` Kees Cook
2025-11-21 0:30 ` Kees Cook
1 sibling, 1 reply; 14+ messages in thread
From: Andrew Pinski @ 2025-11-20 23:10 UTC (permalink / raw)
To: Kees Cook
Cc: Qing Zhao, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
On Thu, Nov 20, 2025 at 2:57 PM Andrew Pinski
<andrew.pinski@oss.qualcomm.com> wrote:
>
> On Thu, Nov 20, 2025 at 2:29 PM Kees Cook <kees@kernel.org> wrote:
> >
> > Implement AArch64-specific KCFI backend.
> >
> > - Trap debugging through ESR (Exception Syndrome Register) encoding
> > in BRK instruction immediate values.
> >
> > - Scratch register allocation using w16/w17 (x16/x17) following
> > AArch64 procedure call standard for intra-procedure-call registers,
> > which already makes x16/x17 available through existing clobbers.
> >
> > - Complementary with BTI (which uses a separate pass system to inject
> > landing instructions where needed).
> >
> > - Does not interfere with SME, which uses attributes not function
> > prototypes for distinguishing functions.
> >
> > Assembly Code Pattern for AArch64:
> > ldur w16, [target, #-4] ; Load actual type ID from preamble
> > mov w17, #type_id_low ; Load expected type (lower 16 bits)
> > movk w17, #type_id_high, lsl #16 ; Load upper 16 bits if needed
> > cmp w16, w17 ; Compare type IDs directly
> > b.eq .Lpass ; Branch if types match
> > .Ltrap: brk #esr_value ; Enhanced trap with register info
> > .Lpass: blr/br target ; Execute validated indirect transfer
> >
> > ESR (Exception Syndrome Register) Integration:
> > - BRK instruction immediate encoding format:
> > 0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31)
> > - TypeIndex indicates which W register contains expected type (W17 = 17)
> > - AddrIndex indicates which X register contains target address (0-30)
> > - Example: brk #33313 (0x8221) = expected type in W17, target address in X1
> >
> > Build and run tested with Linux kernel ARCH=arm64.
> >
> > gcc/ChangeLog:
> >
> > config/aarch64/aarch64-protos.h: Declare aarch64_indirect_branch_asm,
> > and KCFI helpers.
> > config/aarch64/aarch64.cc (aarch64_expand_call): Wrap CALLs in
> > KCFI, with clobbers.
> > (aarch64_indirect_branch_asm): New function, extract common
> > logic for branch asm, like existing call asm helper.
> > (aarch64_output_kcfi_insn): Emit KCFI assembly.
> > config/aarch64/aarch64.md: Add KCFI RTL patterns and replace
> > open-coded branch emission with aarch64_indirect_branch_asm.
> > doc/invoke.texi: Document aarch64 nuances.
> >
> > gcc/testsuite/ChangeLog:
> >
> > * gcc.dg/kcfi/kcfi-adjacency.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-basics.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-call-sharing.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-complex-addressing.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-move-preservation.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-no-sanitize.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-offset-validation.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-large.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-medium.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-tail-calls.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-trap-section.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-trap-encoding.c: New test.
> >
> > Signed-off-by: Kees Cook <kees@kernel.org>
> > ---
> > gcc/config/aarch64/aarch64-protos.h | 5 +
> > gcc/config/aarch64/aarch64.md | 66 +++++++++--
> > gcc/config/aarch64/aarch64.cc | 105 ++++++++++++++++++
> > gcc/doc/invoke.texi | 14 +++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 15 +++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
> > .../gcc.dg/kcfi/kcfi-complex-addressing.c | 16 +++
> > .../gcc.dg/kcfi/kcfi-move-preservation.c | 20 ++++
> > .../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
> > gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
> > .../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
> > .../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 12 ++
> > .../gcc.dg/kcfi/kcfi-patchable-large.c | 12 ++
> > .../gcc.dg/kcfi/kcfi-patchable-medium.c | 12 ++
> > .../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 12 ++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 19 ++++
> > .../gcc.dg/kcfi/kcfi-trap-encoding.c | 41 +++++++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 4 +
> > 19 files changed, 379 insertions(+), 8 deletions(-)
> > create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> >
> > diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
> > index a9e407ba340e..c32d454fe277 100644
> > --- a/gcc/config/aarch64/aarch64-protos.h
> > +++ b/gcc/config/aarch64/aarch64-protos.h
> > @@ -1272,6 +1272,7 @@ tree aarch64_resolve_overloaded_builtin_general (location_t, tree, void *);
> >
> > const char *aarch64_sls_barrier (int);
> > const char *aarch64_indirect_call_asm (rtx);
> > +const char *aarch64_indirect_branch_asm (rtx);
> > extern bool aarch64_harden_sls_retbr_p (void);
> > extern bool aarch64_harden_sls_blr_p (void);
> >
> > @@ -1295,4 +1296,8 @@ extern unsigned aarch64_stack_alignment (const_tree exp, unsigned align);
> > extern rtx aarch64_gen_compare_zero_and_branch (rtx_code code, rtx x,
> > rtx_code_label *label);
> >
> > +/* KCFI support. */
> > +extern void kcfi_emit_trap_with_section (FILE *file, rtx trap_label_rtx);
> > +extern const char *aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands);
> > +
> > #endif /* GCC_AARCH64_PROTOS_H */
> > diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
> > index 855df791bae7..84894816509c 100644
> > --- a/gcc/config/aarch64/aarch64.md
> > +++ b/gcc/config/aarch64/aarch64.md
> > @@ -1506,6 +1506,19 @@
> > }"
> > )
> >
> > +;; KCFI indirect call
> > +(define_insn "*call_insn"
> > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucr"))
> > + (match_operand 1 "" ""))
> > + (match_operand 3 "const_int_operand"))
> > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (clobber (reg:DI LR_REGNUM))]
> > + "!SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, operands);
> > +}
> > + [(set_attr "type" "call")])
> > +
> > (define_insn "*call_insn"
> > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand"))
> > (match_operand 1 "" ""))
> > @@ -1533,6 +1546,21 @@
> > }"
> > )
> >
> > +;; KCFI call with return value
> > +(define_insn "*call_value_insn"
> > + [(set (match_operand 0 "" "")
> > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > + "Ucr"))
> > + (match_operand 2 "" ""))
> > + (match_operand 4 "const_int_operand")))
> > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (clobber (reg:DI LR_REGNUM))]
> > + "!SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > +}
> > + [(set_attr "type" "call")])
> > +
> > (define_insn "*call_value_insn"
> > [(set (match_operand 0 "" "")
> > (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"))
> > @@ -1573,6 +1601,19 @@
> > }
> > )
> >
> > +;; KCFI sibling call
> > +(define_insn "*sibcall_insn"
> > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs"))
> > + (match_operand 1 ""))
> > + (match_operand 3 "const_int_operand"))
> > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (return)]
> > + "SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, operands);
> > +}
> > + [(set_attr "type" "branch")])
> > +
> > (define_insn "*sibcall_insn"
> > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs, Usf"))
> > (match_operand 1 ""))
> > @@ -1581,16 +1622,28 @@
> > "SIBLING_CALL_P (insn)"
> > {
> > if (which_alternative == 0)
> > - {
> > - output_asm_insn ("br\\t%0", operands);
> > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > - }
> > + return aarch64_indirect_branch_asm (operands[0]);
>
> Can you extract aarch64_indirect_branch_asm into a different patch?
> Since it is only slightly related to this.
> The main reason is because applying that can happen before the rest of
> the KFCI patches are approved. And I will handling applying that once
> it has been extracted out.
>
> > return "b\\t%c0";
> > }
> > [(set_attr "type" "branch, branch")
> > (set_attr "sls_length" "retbr,none")]
> > )
> >
> > +;; KCFI sibling call with return value
> > +(define_insn "*sibcall_value_insn"
> > + [(set (match_operand 0 "")
> > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > + "Ucs"))
> > + (match_operand 2 ""))
> > + (match_operand 4 "const_int_operand")))
> > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (return)]
> > + "SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > +}
> > + [(set_attr "type" "branch")])
> > +
> > (define_insn "*sibcall_value_insn"
> > [(set (match_operand 0 "")
> > (call (mem:DI
> > @@ -1601,10 +1654,7 @@
> > "SIBLING_CALL_P (insn)"
> > {
> > if (which_alternative == 0)
> > - {
> > - output_asm_insn ("br\\t%1", operands);
> > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > - }
> > + return aarch64_indirect_branch_asm (operands[1]);
> > return "b\\t%c1";
> > }
> > [(set_attr "type" "branch, branch")
> > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > index 6dfdaa4fb9b0..86e3af992d2a 100644
> > --- a/gcc/config/aarch64/aarch64.cc
> > +++ b/gcc/config/aarch64/aarch64.cc
> > @@ -83,6 +83,7 @@
> > #include "rtlanal.h"
> > #include "tree-dfa.h"
> > #include "asan.h"
> > +#include "kcfi.h"
> > #include "aarch64-elf-metadata.h"
> > #include "aarch64-feature-deps.h"
> > #include "config/arm/aarch-common.h"
> > @@ -11965,6 +11966,16 @@ aarch64_expand_call (rtx result, rtx mem, rtx cookie, bool sibcall)
> >
> > call = gen_rtx_CALL (VOIDmode, mem, const0_rtx);
> >
> > + /* Only indirect calls need KCFI instrumentation. */
> > + bool is_direct_call = SYMBOL_REF_P (XEXP (mem, 0));
> > + rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
> > + : kcfi_get_type_id_for_expanding_gimple_call ();
> > + if (kcfi_type_rtx)
> > + {
> > + /* Wrap call in KCFI. */
> > + call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
> > + }
> > +
> > if (result != NULL_RTX)
> > call = gen_rtx_SET (result, call);
> >
> > @@ -19225,6 +19236,9 @@ aarch64_override_options_internal (struct gcc_options *opts)
> > #endif
> > }
> >
> > + if ((flag_sanitize & SANITIZE_KCFI) && TARGET_ILP32)
> > + sorry ("%<-fsanitize=kcfi%> is not supported for %<-mabi=ilp32%>");
> > +
> > aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
> > if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
> > && !(isa_flags & AARCH64_FL_SME))
> > @@ -30822,6 +30836,18 @@ aarch64_indirect_call_asm (rtx addr)
> > return "";
> > }
> >
> > +/* Generate assembly for AArch64 indirect branch instruction. ADDR is the
> > + target address register. Returns any additional barrier instructions
> > + needed for SLS (Straight Line Speculation) mitigation. */
> > +
> > +const char *
> > +aarch64_indirect_branch_asm (rtx addr)
> > +{
> > + gcc_assert (REG_P (addr));
> > + output_asm_insn ("br\t%0", &addr);
> > + return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > +}
> > +
> > /* Emit the assembly instruction to load the thread pointer into DEST.
> > Select between different tpidr_elN registers depending on -mtp= setting. */
> >
> > @@ -33089,6 +33115,85 @@ aarch64_libgcc_floating_mode_supported_p
> > #undef TARGET_DOCUMENTATION_NAME
> > #define TARGET_DOCUMENTATION_NAME "AArch64"
> >
> > +/* Output the assembly for a KCFI checked call instruction. INSN is the
> > + RTL instruction being processed. OPERANDS is the array of RTL operands
> > + where operands[0] is the call target register, operands[3] is the KCFI
> > + type ID constant. Returns the appropriate call instruction string. */
> > +
> > +const char *
> > +aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands)
> > +{
> > + /* Target register is operands[0]. */
> > + rtx target_reg = operands[0];
> > + gcc_assert (REG_P (target_reg));
> > +
> > + /* Get KCFI type ID from operand[3]. */
> > + uint32_t type_id = (uint32_t) INTVAL (operands[3]);
> > +
> > + /* Calculate typeid offset from call target. */
> > + HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
> > +
> > + /* Get unique label number for this KCFI check. */
> > + int labelno = kcfi_next_labelno ();
> > +
> > + /* Generate custom label names. */
> > + char trap_name[32];
> > + char call_name[32];
> > + ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
> > + ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
> > +
> > + rtx temp_operands[3];
> > +
> > + /* Load actual type into w16 from memory at offset using ldur. */
> > + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> > + temp_operands[1] = target_reg;
> > + temp_operands[2] = GEN_INT (offset);
> > + output_asm_insn ("ldur\t%w0, [%1, #%2]", temp_operands);
>
> Why not just:
> temp_operands[0] = target_reg;
> temp_operands[0] = GEN_INT (offset);
> output_asm_insn ("ldur\tw16, [%0, #%1]", temp_operands);
>
> > +
> > + /* Load expected type low 16 bits into w17. */
> > + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> > + temp_operands[1] = GEN_INT (type_id & 0xFFFF);
> > + output_asm_insn ("mov\t%w0, #%1", temp_operands);
> Likewise.
> Also why not use fprintf here?
> Like:
> fprintf(asm_out_file, "mov\t%w0, #%d\n", type_id & 0xFFFF);
> > +
> > + /* Load expected type high 16 bits into w17. */
> > + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> > + temp_operands[1] = GEN_INT ((type_id >> 16) & 0xFFFF);
> > + output_asm_insn ("movk\t%w0, #%1, lsl #16", temp_operands);
>
> Likewise on both comments/points.
> At least reuse the gen_rtx_REG for 16/17.
>
> > +
> > + /* Compare types. */
> > + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> > + temp_operands[1] = gen_rtx_REG (SImode, R17_REGNUM);
> > + output_asm_insn ("cmp\t%w0, %w1", temp_operands);
>
> fprintf (asm_out_file, "cmp\tw16, w17\n");
>
> > +
> > + /* Output conditional branch to call label. */
> > + fputs ("\tb.eq\t", asm_out_file);
> > + assemble_name (asm_out_file, call_name);
> > + fputc ('\n', asm_out_file);
> > +
> > + /* Output trap label and BRK instruction. */
> > + ASM_OUTPUT_LABEL (asm_out_file, trap_name);
> > +
> > + /* Calculate and emit BRK with ESR encoding. */
> > + unsigned type_index = R17_REGNUM;
> > + unsigned addr_index = REGNO (operands[0]) - R0_REGNUM;
> > + unsigned esr_value = 0x8000 | ((type_index & 31) << 5) | (addr_index & 31);
> > +
> > + temp_operands[0] = GEN_INT (esr_value);
> > + output_asm_insn ("brk\t#%0", temp_operands);
> > +
> > + /* Output call label. */
> > + ASM_OUTPUT_LABEL (asm_out_file, call_name);
> > +
> > + /* Return appropriate call instruction based on SIBLING_CALL_P. */
> > + if (SIBLING_CALL_P (insn))
> > + return aarch64_indirect_branch_asm (operands[0]);
> > + else
> > + return aarch64_indirect_call_asm (operands[0]);
> > +}
> > +
> > +#undef TARGET_KCFI_SUPPORTED
> > +#define TARGET_KCFI_SUPPORTED hook_bool_void_true
>
> Why return true always for it being supported? I thought this is not
> supported with ilp32, it is definitely not supported for mingw. Is it
> supported with -elf targets rather than -linux-gnu targets?
>
> Also I am still trying to figure out and understand the interaction
> between x16 and x17 in some cases.
> Because I thought indirect calls/jumps will be using x16/x17 for those
> to support BTI.
Oh yes:
(define_register_constraint "Ucr"
"aarch64_harden_sls_blr_p () ? STUB_REGS : GENERAL_REGS"
"@internal Registers to be used for an indirect call.
This is usually the general registers, but when we are hardening against
Straight Line Speculation we disallow x16, x17, and x30 so we can use
indirection stubs. These indirection stubs cannot use the above registers
since they will be reached by a BL that may have to go through a linker
veneer.")
But you don't change Ucr so in theory x16/x17 could be used for call_value_insn.
(I can't get that one using x16/x17 right now).
Oh and sibcall_insn uses Ucs which is defined as:
(define_register_constraint "Ucs" "TAILCALL_ADDR_REGS"
"@internal Registers suitable for an indirect tail call")
TAILCALL_ADDR_REGS is a register class which just contains x16/x17.
I don't see a testcase for indirect sibcall either.
```
typedef void (*fptr)(void);
void f(fptr a)
{
a();
}
```
Is a testcase for the indirect sibcall case.
Thanks,
Andrew Pinski
>
> Thanks,
> Andrew
>
> > +
> > struct gcc_target targetm = TARGET_INITIALIZER;
> >
> > #include "gt-aarch64.h"
> > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> > index de6aadfd5346..94bb4ffcce98 100644
> > --- a/gcc/doc/invoke.texi
> > +++ b/gcc/doc/invoke.texi
> > @@ -18727,6 +18727,20 @@ header MOVri instruction would become something like this:
> >
> > @code{movl $199571451, %ebx # hash of foo's type = 0xBE537FB}
> >
> > +On AArch64, KCFI type identifiers are emitted as a @code{.word ID}
> > +directive (a 32-bit constant) before the function entry. AArch64's
> > +natural 4-byte instruction alignment eliminates the need for additional
> > +alignment NOPs. When used with @option{-fpatchable-function-entry}, the
> > +type identifier is placed before any prefix NOPs. The runtime check
> > +uses @code{x16} and @code{x17} as scratch registers. Type mismatches
> > +trigger a @code{brk} instruction with an immediate value that encodes
> > +both the expected type register index and the target address register
> > +index in the format @code{0x8000 | (type_reg << 5) | addr_reg}. This
> > +encoding is captured in the ESR (Exception Syndrome Register) when the
> > +trap is taken, allowing the kernel to identify both the KCFI violation
> > +and the involved registers for detailed diagnostics (eliminating the need
> > +for a separate @code{.kcfi_traps} section as used on x86_64).
> > +
> > KCFI is intended primarily for kernel code and may not be suitable
> > for user-space applications that rely on techniques incompatible
> > with strict type checking of indirect calls.
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
> > index 7c59921e630c..f3d7d23e6af2 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
> > @@ -63,4 +63,19 @@ __attribute__((noinline)) void test_conditional_call(int flag) {
> > ** ...
> > */
> >
> > +/*
> > +** test_complex_args: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[(x[0-9]+), #-4\]
> > +** mov w17, #[0-9]+
> > +** movk w17, #[0-9]+, lsl #16
> > +** cmp w16, w17
> > +** b.eq .Lkcfi_call([0-9]+)
> > +** .Lkcfi_trap[0-9]+:
> > +** brk #[0-9]+
> > +** .Lkcfi_call\2:
> > +** br \1
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
> > index fe0a21d26df9..6eac946f7abf 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
> > @@ -59,6 +59,9 @@ int main() {
> > /* x86_64: Verify type ID in preamble (after NOPs, before function label) */
> > /* { dg-final { scan-assembler {__cfi_regular_function:\n\t+nop\n.*\n\t+movl\t+\$0x[0-9a-f]+, %eax} { target x86_64-*-* } } } */
> >
> > +/* AArch64: Verify type ID word in preamble. */
> > +/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* } } } */
> > +
> > /*
> > ** static_caller: { target x86_64-*-* }
> > ** ...
> > @@ -76,6 +79,21 @@ int main() {
> > ** ...
> > */
> >
> > +/*
> > +** static_caller: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[(x[0-9]+), #-4\]
> > +** mov w17, #[0-9]+
> > +** movk w17, #[0-9]+, lsl #16
> > +** cmp w16, w17
> > +** b.eq .Lkcfi_call([0-9]+)
> > +** .Lkcfi_trap[0-9]+:
> > +** brk #[0-9]+
> > +** .Lkcfi_call\2:
> > +** blr \1
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> >
> > /* Extern functions should NOT get KCFI preambles. */
> > @@ -93,3 +111,6 @@ int main() {
> > /* External functions that are only called directly should NOT get
> > __kcfi_typeid_ symbols. */
> > /* { dg-final { scan-assembler-not {__kcfi_typeid_external_func_int} } } */
> > +
> > +/* AArch64 should NOT have trap section (use immediate instructions instead). */
> > +/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
> > index 05165f0e2851..6062d74ef62a 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
> > @@ -63,14 +63,18 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
> > /* Verify we have TWO different KCFI check sequences. */
> > /* Each check should have different type ID constants. */
> > /* x86: { dg-final { scan-assembler-times {movl\s+\$-?[0-9]+,\s+%r10d} 2 { target i?86-*-* x86_64-*-* } } } */
> > +/* AArch64: { dg-final { scan-assembler-times {mov\s+w17, #[0-9]+} 2 { target aarch64*-*-* } } } */
> >
> > /* Verify the checks use DIFFERENT type IDs (not shared).
> > We should NOT see the same type ID used twice - that would indicate
> > unmerged sharing. */
> > /* x86: { dg-final { scan-assembler-not {movl\s+\$(-?[0-9]+),\s+%r10d.*movl\s+\$\1,\s+%r10d} { target i?86-*-* x86_64-*-* } } } */
> > +/* AArch64: { dg-final { scan-assembler-not {mov\s+w17, #([0-9]+).*mov\s+w17, #\1} { target aarch64*-*-* } } } */
> >
> > /* Verify expected number of traps. */
> > /* x86: { dg-final { scan-assembler-times {ud2} 2 { target i?86-*-* x86_64-*-* } } } */
> > +/* AArch64: { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
> >
> > /* Verify 2 separate call sites. */
> > /* x86: { dg-final { scan-assembler-times {jmp\s+\*%[a-z0-9]+} 2 { target i?86-*-* x86_64-*-* } } } */
> > +/* AArch64: { dg-final { scan-assembler-times {br\tx[0-9]+} 2 { target aarch64*-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
> > index ed415033c5c9..3ffbd408a69e 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
> > @@ -146,4 +146,20 @@ int main() {
> > ** ...
> > */
> >
> > +/* Standard KCFI handling. */
> > +/*
> > +** main: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[(x[0-9]+), #-4\]
> > +** mov w17, #[0-9]+
> > +** movk w17, #[0-9]+, lsl #16
> > +** cmp w16, w17
> > +** b.eq .Lkcfi_call([0-9]+)
> > +** .Lkcfi_trap[0-9]+:
> > +** brk #[0-9]+
> > +** .Lkcfi_call\2:
> > +** blr \1
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
> > index 5553ff47174b..df39b7f0a8a3 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
> > @@ -57,4 +57,24 @@ int main(void)
> > ** ...
> > */
> >
> > +/*
> > +** indirect_call: { target aarch64*-*-* }
> > +** ...
> > +** mov (x[0-9]+), x0
> > +** ...
> > +** ldur w16, \[\1, #-4\]
> > +** mov w17, #[0-9]+
> > +** movk w17, #[0-9]+, lsl #16
> > +** cmp w16, w17
> > +** b.eq .Lkcfi_call([0-9]+)
> > +** .Lkcfi_trap[0-9]+:
> > +** brk #[0-9]+
> > +** .Lkcfi_call\2:
> > +** br \1
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
> > +
> > +/* AArch64 should NOT have trap section (use immediate instructions instead). */
> > +/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
> > index 9ed7e21fe8eb..cdeb202ffd12 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
> > @@ -75,11 +75,16 @@ int main(void)
> >
> > /* Verify correct number of KCFI checks: exactly 2 */
> > /* { dg-final { scan-assembler-times {ud2} 2 { target x86_64-*-* } } } */
> > +/* { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
> >
> > /* Positive controls: these should have KCFI checks. */
> > /* { dg-final { scan-assembler {normal_function:.*ud2.*\.size\s+normal_function} { target x86_64-*-* } } } */
> > /* { dg-final { scan-assembler {wrap_normal_inline:.*ud2.*\.size\s+wrap_normal_inline} { target x86_64-*-* } } } */
> > +/* { dg-final { scan-assembler {normal_function:.*brk\s+#[0-9]+.*\.size\s+normal_function} { target aarch64*-*-* } } } */
> > +/* { dg-final { scan-assembler {wrap_normal_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_normal_inline} { target aarch64*-*-* } } } */
> >
> > /* Negative controls: these should NOT have KCFI checks. */
> > /* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ud2.*\.size\s+sensitive_non_inline_function} { target x86_64-*-* } } } */
> > /* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*ud2.*\.size\s+wrap_sensitive_inline} { target x86_64-*-* } } } */
> > +/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*brk\s+#[0-9]+.*\.size\s+sensitive_non_inline_function} { target aarch64*-*-* } } } */
> > +/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_sensitive_inline} { target aarch64*-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
> > index 95a8e8419e00..af6d86803576 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
> > @@ -34,3 +34,4 @@ int main() {
> > caller_no_checks() should NOT generate KCFI check (no_sanitize).
> > So a total of exactly 1 KCFI check in the entire program. */
> > /* { dg-final { scan-assembler-times {addl\t-4\(%r[ad]x\), %r1[01]d} 1 { target x86_64-*-* } } } */
> > +/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 1 { target aarch64-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
> > index 97d964feebd3..0ced5c43ae92 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
> > @@ -27,3 +27,6 @@ int main() {
> > /* x86_64: All call sites should use -4 offset for KCFI type ID loads, even
> > with -falign-functions=16 (we're not using patchable entries here). */
> > /* { dg-final { scan-assembler {movl\t\$-?[0-9]+, %r10d\n\taddl\t-4\(%r[a-z0-9]+\), %r10d} { target x86_64-*-* } } } */
> > +
> > +/* AArch64: All call sites should use -4 offset. */
> > +/* { dg-final { scan-assembler {ldur\tw16, \[x[0-9]+, #-4\]} { target aarch64*-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
> > index 379356385a16..7a251cbdee3b 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
> > @@ -28,6 +28,11 @@ int main() {
> > ** movl \$0x[0-9a-f]+, %eax
> > */
> >
> > +/*
> > +** __cfi_test_function: { target aarch64*-*-* }
> > +** .word 0x[0-9a-f]+
> > +*/
> > +
> > /*
> > ** main: { target x86_64-*-* }
> > ** ...
> > @@ -35,4 +40,11 @@ int main() {
> > ** ...
> > */
> >
> > +/*
> > +** main: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[x[0-9]+, #-4\]
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
> > index 06df3495bb23..3ed5d16c8e91 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
> > @@ -17,6 +17,11 @@ int main() {
> > ** movl \$0x[0-9a-f]+, %eax
> > */
> >
> > +/*
> > +** __cfi_test_function: { target aarch64*-*-* }
> > +** .word 0x[0-9a-f]+
> > +*/
> > +
> > /*
> > ** main: { target x86_64-*-* }
> > ** ...
> > @@ -24,4 +29,11 @@ int main() {
> > ** ...
> > */
> >
> > +/*
> > +** main: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[x[0-9]+, #-48\]
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
> > index ef87b135934b..e354914209e9 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
> > @@ -24,6 +24,11 @@ int main() {
> > ** movl \$0x[0-9a-f]+, %eax
> > */
> >
> > +/*
> > +** __cfi_test_function: { target aarch64*-*-* }
> > +** .word 0x[0-9a-f]+
> > +*/
> > +
> > /*
> > ** main: { target x86_64-*-* }
> > ** ...
> > @@ -31,4 +36,11 @@ int main() {
> > ** ...
> > */
> >
> > +/*
> > +** main: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[x[0-9]+, #-20\]
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
> > index 872814aa4171..7a1dc4fa0e07 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
> > @@ -25,6 +25,11 @@ int main() {
> > ** movl \$0x[0-9a-f]+, %eax
> > */
> >
> > +/*
> > +** __cfi_test_function: { target aarch64*-*-* }
> > +** .word 0x[0-9a-f]+
> > +*/
> > +
> > /*
> > ** main: { target x86_64-*-* }
> > ** ...
> > @@ -32,4 +37,11 @@ int main() {
> > ** ...
> > */
> >
> > +/*
> > +** main: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[x[0-9]+, #-16\]
> > +** ...
> > +*/
> > +
> > /* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
> > index 04a9eb1fd206..1a7cc4aa167f 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
> > @@ -78,3 +78,22 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
> >
> > /* Should have exactly 1 regular call (non-tail call case). */
> > /* { dg-final { scan-assembler-times {call\t\*%[a-z0-9]+} 1 { target x86_64-*-* } } } */
> > +
> > +/* Should have exactly 4 KCFI checks for indirect calls (load type ID from
> > + -4 offset + compare). */
> > +/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 4 { target aarch64-*-* } } } */
> > +/* { dg-final { scan-assembler-times {cmp\tw16, w17} 4 { target aarch64-*-* } } } */
> > +
> > +/* Should have exactly 4 trap instructions. */
> > +/* { dg-final { scan-assembler-times {brk\t#[0-9]+} 4 { target aarch64-*-* } } } */
> > +
> > +/* Should have exactly 3 protected tail calls (br through register after
> > + KCFI check). */
> > +/* { dg-final { scan-assembler-times {br\tx[0-9]+} 3 { target aarch64-*-* } } } */
> > +
> > +/* Should have exactly 1 regular call (non-tail call case). */
> > +/* { dg-final { scan-assembler-times {blr\tx[0-9]+} 1 { target aarch64-*-* } } } */
> > +
> > +/* Type ID loading should use mov + movk pattern for 32-bit constants. */
> > +/* { dg-final { scan-assembler {mov\tw17, #[0-9]+} { target aarch64-*-* } } } */
> > +/* { dg-final { scan-assembler {movk\tw17, #[0-9]+, lsl #16} { target aarch64-*-* } } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> > new file mode 100644
> > index 000000000000..0c257565c9e8
> > --- /dev/null
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> > @@ -0,0 +1,41 @@
> > +/* Test AArch64 and ARM32 KCFI trap encoding in BRK/UDF instructions. */
> > +/* { dg-do compile { target aarch64*-*-* } } */
> > +
> > +void target_function(int x, char y) {
> > +}
> > +
> > +int main() {
> > + void (*func_ptr)(int, char) = target_function;
> > +
> > + /* This should generate trap with immediate encoding. */
> > + func_ptr(42, 'a');
> > +
> > + return 0;
> > +}
> > +
> > +/* Should have KCFI preamble. */
> > +/* { dg-final { scan-assembler "__cfi_target_function:" } } */
> > +
> > +/* AArch64 specific: Should have BRK instruction with proper ESR encoding
> > + ESR format: 0x8000 | ((type_reg & 31) << 5) | (addr_reg & 31)
> > +
> > + Test the ESR encoding by checking for the expected value.
> > + Since we know this test uses x2, we expect ESR = 0x8000 | (17<<5) | 2 = 33314
> > + */
> > +
> > +/*
> > +** main: { target aarch64*-*-* }
> > +** ...
> > +** ldur w16, \[x[0-9]+, #-4\]
> > +** mov w17, #[0-9]+
> > +** movk w17, #[0-9]+, lsl #16
> > +** cmp w16, w17
> > +** b\.eq .Lkcfi_call[0-9]+
> > +** .Lkcfi_trap[0-9]+:
> > +** brk #33314
> > +** .Lkcfi_call[0-9]+:
> > +** blr x2
> > +** ...
> > +*/
> > +
> > +/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*} } } */
> > diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
> > index 55c0829ccd7b..e92873e51321 100644
> > --- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
> > +++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
> > @@ -18,6 +18,10 @@ int main() {
> >
> > /* Should have exactly 2 trap labels in code. */
> > /* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ud2} 2 { target x86_64-*-* } } } */
> > +/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*brk} 2 { target aarch64*-*-* } } } */
> >
> > /* x86_64 should exactly 2 .kcfi_traps sections. */
> > /* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* } } } */
> > +
> > +/* AArch64 should NOT have .kcfi_traps section. */
> > +/* { dg-final { scan-assembler-not {\.section\t+\.kcfi_traps} { target aarch64*-*-* } } } */
> > --
> > 2.34.1
> >
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-20 23:10 ` Andrew Pinski
@ 2025-11-21 0:18 ` Kees Cook
0 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-21 0:18 UTC (permalink / raw)
To: Andrew Pinski
Cc: Qing Zhao, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
On Thu, Nov 20, 2025 at 03:10:41PM -0800, Andrew Pinski wrote:
> On Thu, Nov 20, 2025 at 2:57 PM Andrew Pinski
> <andrew.pinski@oss.qualcomm.com> wrote:
> > Also I am still trying to figure out and understand the interaction
> > between x16 and x17 in some cases.
> > Because I thought indirect calls/jumps will be using x16/x17 for those
> > to support BTI.
>
> Oh yes:
> (define_register_constraint "Ucr"
> "aarch64_harden_sls_blr_p () ? STUB_REGS : GENERAL_REGS"
> "@internal Registers to be used for an indirect call.
> This is usually the general registers, but when we are hardening against
> Straight Line Speculation we disallow x16, x17, and x30 so we can use
> indirection stubs. These indirection stubs cannot use the above registers
> since they will be reached by a BL that may have to go through a linker
> veneer.")
>
> But you don't change Ucr so in theory x16/x17 could be used for call_value_insn.
> (I can't get that one using x16/x17 right now).
>
> Oh and sibcall_insn uses Ucs which is defined as:
> (define_register_constraint "Ucs" "TAILCALL_ADDR_REGS"
> "@internal Registers suitable for an indirect tail call")
> TAILCALL_ADDR_REGS is a register class which just contains x16/x17.
Hm, I will need to study this more closely. I wonder if both kcfi and sls
hardening end up being self-contained users of the scratch registers? I'll
double check that my kernel test builds have SLS hardening enabled. (And
I'll likely need to add some aarch64-specific sibcall tests with/without
SLS hardening to see the resulting asm.)
> I don't see a testcase for indirect sibcall either.
>
> ```
> typedef void (*fptr)(void);
> void f(fptr a)
> {
> a();
> }
> ```
> Is a testcase for the indirect sibcall case.
I did include basic tests for a variety of sibcalls in the patch that
added the general tests; see gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
(though I named it "tail calls"), and the fptr test includes an argument
(the comment is x86-specific, but it should be a valid test for all archs):
+/* Indirect call through function pointer parameter. */
+int test_param_indirect_call(func_ptr_t handler, int x) {
+ /* This is an indirect call that should be converted to tail call:
+ Without -fno-optimize-sibling-calls should become "jmp *%rdi"
+ With -fno-optimize-sibling-calls should be "call *%rdi" */
+ return handler(x);
+}
-Kees
--
Kees Cook
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-20 22:57 ` Andrew Pinski
2025-11-20 23:10 ` Andrew Pinski
@ 2025-11-21 0:30 ` Kees Cook
2025-11-21 3:28 ` Andrew Pinski
2025-11-21 3:43 ` Andrew Pinski
1 sibling, 2 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-21 0:30 UTC (permalink / raw)
To: Andrew Pinski
Cc: Qing Zhao, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
On Thu, Nov 20, 2025 at 02:57:49PM -0800, Andrew Pinski wrote:
> On Thu, Nov 20, 2025 at 2:29 PM Kees Cook <kees@kernel.org> wrote:
> >
> > Implement AArch64-specific KCFI backend.
> >
> > - Trap debugging through ESR (Exception Syndrome Register) encoding
> > in BRK instruction immediate values.
> >
> > - Scratch register allocation using w16/w17 (x16/x17) following
> > AArch64 procedure call standard for intra-procedure-call registers,
> > which already makes x16/x17 available through existing clobbers.
> >
> > - Complementary with BTI (which uses a separate pass system to inject
> > landing instructions where needed).
> >
> > - Does not interfere with SME, which uses attributes not function
> > prototypes for distinguishing functions.
> >
> > Assembly Code Pattern for AArch64:
> > ldur w16, [target, #-4] ; Load actual type ID from preamble
> > mov w17, #type_id_low ; Load expected type (lower 16 bits)
> > movk w17, #type_id_high, lsl #16 ; Load upper 16 bits if needed
> > cmp w16, w17 ; Compare type IDs directly
> > b.eq .Lpass ; Branch if types match
> > .Ltrap: brk #esr_value ; Enhanced trap with register info
> > .Lpass: blr/br target ; Execute validated indirect transfer
> >
> > ESR (Exception Syndrome Register) Integration:
> > - BRK instruction immediate encoding format:
> > 0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31)
> > - TypeIndex indicates which W register contains expected type (W17 = 17)
> > - AddrIndex indicates which X register contains target address (0-30)
> > - Example: brk #33313 (0x8221) = expected type in W17, target address in X1
> >
> > Build and run tested with Linux kernel ARCH=arm64.
> >
> > gcc/ChangeLog:
> >
> > config/aarch64/aarch64-protos.h: Declare aarch64_indirect_branch_asm,
> > and KCFI helpers.
> > config/aarch64/aarch64.cc (aarch64_expand_call): Wrap CALLs in
> > KCFI, with clobbers.
> > (aarch64_indirect_branch_asm): New function, extract common
> > logic for branch asm, like existing call asm helper.
> > (aarch64_output_kcfi_insn): Emit KCFI assembly.
> > config/aarch64/aarch64.md: Add KCFI RTL patterns and replace
> > open-coded branch emission with aarch64_indirect_branch_asm.
> > doc/invoke.texi: Document aarch64 nuances.
> >
> > gcc/testsuite/ChangeLog:
> >
> > * gcc.dg/kcfi/kcfi-adjacency.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-basics.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-call-sharing.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-complex-addressing.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-move-preservation.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-no-sanitize.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-offset-validation.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-large.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-medium.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-tail-calls.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-trap-section.c: Add aarch64 patterns.
> > * gcc.dg/kcfi/kcfi-trap-encoding.c: New test.
> >
> > Signed-off-by: Kees Cook <kees@kernel.org>
> > ---
> > gcc/config/aarch64/aarch64-protos.h | 5 +
> > gcc/config/aarch64/aarch64.md | 66 +++++++++--
> > gcc/config/aarch64/aarch64.cc | 105 ++++++++++++++++++
> > gcc/doc/invoke.texi | 14 +++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 15 +++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
> > .../gcc.dg/kcfi/kcfi-complex-addressing.c | 16 +++
> > .../gcc.dg/kcfi/kcfi-move-preservation.c | 20 ++++
> > .../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
> > gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
> > .../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
> > .../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 12 ++
> > .../gcc.dg/kcfi/kcfi-patchable-large.c | 12 ++
> > .../gcc.dg/kcfi/kcfi-patchable-medium.c | 12 ++
> > .../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 12 ++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 19 ++++
> > .../gcc.dg/kcfi/kcfi-trap-encoding.c | 41 +++++++
> > gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 4 +
> > 19 files changed, 379 insertions(+), 8 deletions(-)
> > create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> >
> > diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
> > index a9e407ba340e..c32d454fe277 100644
> > --- a/gcc/config/aarch64/aarch64-protos.h
> > +++ b/gcc/config/aarch64/aarch64-protos.h
> > @@ -1272,6 +1272,7 @@ tree aarch64_resolve_overloaded_builtin_general (location_t, tree, void *);
> >
> > const char *aarch64_sls_barrier (int);
> > const char *aarch64_indirect_call_asm (rtx);
> > +const char *aarch64_indirect_branch_asm (rtx);
> > extern bool aarch64_harden_sls_retbr_p (void);
> > extern bool aarch64_harden_sls_blr_p (void);
> >
> > @@ -1295,4 +1296,8 @@ extern unsigned aarch64_stack_alignment (const_tree exp, unsigned align);
> > extern rtx aarch64_gen_compare_zero_and_branch (rtx_code code, rtx x,
> > rtx_code_label *label);
> >
> > +/* KCFI support. */
> > +extern void kcfi_emit_trap_with_section (FILE *file, rtx trap_label_rtx);
> > +extern const char *aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands);
> > +
> > #endif /* GCC_AARCH64_PROTOS_H */
> > diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
> > index 855df791bae7..84894816509c 100644
> > --- a/gcc/config/aarch64/aarch64.md
> > +++ b/gcc/config/aarch64/aarch64.md
> > @@ -1506,6 +1506,19 @@
> > }"
> > )
> >
> > +;; KCFI indirect call
> > +(define_insn "*call_insn"
> > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucr"))
> > + (match_operand 1 "" ""))
> > + (match_operand 3 "const_int_operand"))
> > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (clobber (reg:DI LR_REGNUM))]
> > + "!SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, operands);
> > +}
> > + [(set_attr "type" "call")])
> > +
> > (define_insn "*call_insn"
> > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand"))
> > (match_operand 1 "" ""))
> > @@ -1533,6 +1546,21 @@
> > }"
> > )
> >
> > +;; KCFI call with return value
> > +(define_insn "*call_value_insn"
> > + [(set (match_operand 0 "" "")
> > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > + "Ucr"))
> > + (match_operand 2 "" ""))
> > + (match_operand 4 "const_int_operand")))
> > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (clobber (reg:DI LR_REGNUM))]
> > + "!SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > +}
> > + [(set_attr "type" "call")])
> > +
> > (define_insn "*call_value_insn"
> > [(set (match_operand 0 "" "")
> > (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"))
> > @@ -1573,6 +1601,19 @@
> > }
> > )
> >
> > +;; KCFI sibling call
> > +(define_insn "*sibcall_insn"
> > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs"))
> > + (match_operand 1 ""))
> > + (match_operand 3 "const_int_operand"))
> > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (return)]
> > + "SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, operands);
> > +}
> > + [(set_attr "type" "branch")])
> > +
> > (define_insn "*sibcall_insn"
> > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs, Usf"))
> > (match_operand 1 ""))
> > @@ -1581,16 +1622,28 @@
> > "SIBLING_CALL_P (insn)"
> > {
> > if (which_alternative == 0)
> > - {
> > - output_asm_insn ("br\\t%0", operands);
> > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > - }
> > + return aarch64_indirect_branch_asm (operands[0]);
>
> Can you extract aarch64_indirect_branch_asm into a different patch?
> Since it is only slightly related to this.
> The main reason is because applying that can happen before the rest of
> the KFCI patches are approved. And I will handling applying that once
> it has been extracted out.
Sure! I'll extract it.
>
> > return "b\\t%c0";
> > }
> > [(set_attr "type" "branch, branch")
> > (set_attr "sls_length" "retbr,none")]
> > )
> >
> > +;; KCFI sibling call with return value
> > +(define_insn "*sibcall_value_insn"
> > + [(set (match_operand 0 "")
> > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > + "Ucs"))
> > + (match_operand 2 ""))
> > + (match_operand 4 "const_int_operand")))
> > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > + (return)]
> > + "SIBLING_CALL_P (insn)"
> > +{
> > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > +}
> > + [(set_attr "type" "branch")])
> > +
> > (define_insn "*sibcall_value_insn"
> > [(set (match_operand 0 "")
> > (call (mem:DI
> > @@ -1601,10 +1654,7 @@
> > "SIBLING_CALL_P (insn)"
> > {
> > if (which_alternative == 0)
> > - {
> > - output_asm_insn ("br\\t%1", operands);
> > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > - }
> > + return aarch64_indirect_branch_asm (operands[1]);
> > return "b\\t%c1";
> > }
> > [(set_attr "type" "branch, branch")
> > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > index 6dfdaa4fb9b0..86e3af992d2a 100644
> > --- a/gcc/config/aarch64/aarch64.cc
> > +++ b/gcc/config/aarch64/aarch64.cc
> > @@ -83,6 +83,7 @@
> > #include "rtlanal.h"
> > #include "tree-dfa.h"
> > #include "asan.h"
> > +#include "kcfi.h"
> > #include "aarch64-elf-metadata.h"
> > #include "aarch64-feature-deps.h"
> > #include "config/arm/aarch-common.h"
> > @@ -11965,6 +11966,16 @@ aarch64_expand_call (rtx result, rtx mem, rtx cookie, bool sibcall)
> >
> > call = gen_rtx_CALL (VOIDmode, mem, const0_rtx);
> >
> > + /* Only indirect calls need KCFI instrumentation. */
> > + bool is_direct_call = SYMBOL_REF_P (XEXP (mem, 0));
> > + rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
> > + : kcfi_get_type_id_for_expanding_gimple_call ();
> > + if (kcfi_type_rtx)
> > + {
> > + /* Wrap call in KCFI. */
> > + call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
> > + }
> > +
> > if (result != NULL_RTX)
> > call = gen_rtx_SET (result, call);
> >
> > @@ -19225,6 +19236,9 @@ aarch64_override_options_internal (struct gcc_options *opts)
> > #endif
> > }
> >
> > + if ((flag_sanitize & SANITIZE_KCFI) && TARGET_ILP32)
> > + sorry ("%<-fsanitize=kcfi%> is not supported for %<-mabi=ilp32%>");
> > +
> > aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
> > if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
> > && !(isa_flags & AARCH64_FL_SME))
> > @@ -30822,6 +30836,18 @@ aarch64_indirect_call_asm (rtx addr)
> > return "";
> > }
> >
> > +/* Generate assembly for AArch64 indirect branch instruction. ADDR is the
> > + target address register. Returns any additional barrier instructions
> > + needed for SLS (Straight Line Speculation) mitigation. */
> > +
> > +const char *
> > +aarch64_indirect_branch_asm (rtx addr)
> > +{
> > + gcc_assert (REG_P (addr));
> > + output_asm_insn ("br\t%0", &addr);
> > + return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > +}
> > +
> > /* Emit the assembly instruction to load the thread pointer into DEST.
> > Select between different tpidr_elN registers depending on -mtp= setting. */
> >
> > @@ -33089,6 +33115,85 @@ aarch64_libgcc_floating_mode_supported_p
> > #undef TARGET_DOCUMENTATION_NAME
> > #define TARGET_DOCUMENTATION_NAME "AArch64"
> >
> > +/* Output the assembly for a KCFI checked call instruction. INSN is the
> > + RTL instruction being processed. OPERANDS is the array of RTL operands
> > + where operands[0] is the call target register, operands[3] is the KCFI
> > + type ID constant. Returns the appropriate call instruction string. */
> > +
> > +const char *
> > +aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands)
> > +{
> > + /* Target register is operands[0]. */
> > + rtx target_reg = operands[0];
> > + gcc_assert (REG_P (target_reg));
> > +
> > + /* Get KCFI type ID from operand[3]. */
> > + uint32_t type_id = (uint32_t) INTVAL (operands[3]);
> > +
> > + /* Calculate typeid offset from call target. */
> > + HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
> > +
> > + /* Get unique label number for this KCFI check. */
> > + int labelno = kcfi_next_labelno ();
> > +
> > + /* Generate custom label names. */
> > + char trap_name[32];
> > + char call_name[32];
> > + ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
> > + ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
> > +
> > + rtx temp_operands[3];
> > +
> > + /* Load actual type into w16 from memory at offset using ldur. */
> > + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> > + temp_operands[1] = target_reg;
> > + temp_operands[2] = GEN_INT (offset);
> > + output_asm_insn ("ldur\t%w0, [%1, #%2]", temp_operands);
>
> Why not just:
> temp_operands[0] = target_reg;
> temp_operands[0] = GEN_INT (offset);
> output_asm_insn ("ldur\tw16, [%0, #%1]", temp_operands);
Oops, yes. I have rewritten these handlers so many times I've lost my
mind. :)
>
> > +
> > + /* Load expected type low 16 bits into w17. */
> > + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> > + temp_operands[1] = GEN_INT (type_id & 0xFFFF);
> > + output_asm_insn ("mov\t%w0, #%1", temp_operands);
> Likewise.
> Also why not use fprintf here?
> Like:
> fprintf(asm_out_file, "mov\t%w0, #%d\n", type_id & 0xFFFF);
It seemed like directly using fprintf() was frowned on, and the
output_asm_insn with an operands array was the "correct" way to do all
of these? I'm happy to do it however! :)
> > +#undef TARGET_KCFI_SUPPORTED
> > +#define TARGET_KCFI_SUPPORTED hook_bool_void_true
>
> Why return true always for it being supported? I thought this is not
> supported with ilp32, it is definitely not supported for mingw. Is it
> supported with -elf targets rather than -linux-gnu targets?
Ah, it seemed like the only place I could do the checking for ilp32,
etc was in aarch64_override_options_internal. I'll see if I can figure
out how to move that into the TARGET hooks, but all the examples I could
find where in the override_options functions...
Thanks for the review! I'll do more fixing. :)
-Kees
--
Kees Cook
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-21 0:30 ` Kees Cook
@ 2025-11-21 3:28 ` Andrew Pinski
2025-11-21 3:43 ` Andrew Pinski
1 sibling, 0 replies; 14+ messages in thread
From: Andrew Pinski @ 2025-11-21 3:28 UTC (permalink / raw)
To: Kees Cook
Cc: Qing Zhao, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
On Thu, Nov 20, 2025 at 4:30 PM Kees Cook <kees@kernel.org> wrote:
>
> On Thu, Nov 20, 2025 at 02:57:49PM -0800, Andrew Pinski wrote:
> > On Thu, Nov 20, 2025 at 2:29 PM Kees Cook <kees@kernel.org> wrote:
> > >
> > > Implement AArch64-specific KCFI backend.
> > >
> > > - Trap debugging through ESR (Exception Syndrome Register) encoding
> > > in BRK instruction immediate values.
> > >
> > > - Scratch register allocation using w16/w17 (x16/x17) following
> > > AArch64 procedure call standard for intra-procedure-call registers,
> > > which already makes x16/x17 available through existing clobbers.
> > >
> > > - Complementary with BTI (which uses a separate pass system to inject
> > > landing instructions where needed).
> > >
> > > - Does not interfere with SME, which uses attributes not function
> > > prototypes for distinguishing functions.
> > >
> > > Assembly Code Pattern for AArch64:
> > > ldur w16, [target, #-4] ; Load actual type ID from preamble
> > > mov w17, #type_id_low ; Load expected type (lower 16 bits)
> > > movk w17, #type_id_high, lsl #16 ; Load upper 16 bits if needed
> > > cmp w16, w17 ; Compare type IDs directly
> > > b.eq .Lpass ; Branch if types match
> > > .Ltrap: brk #esr_value ; Enhanced trap with register info
> > > .Lpass: blr/br target ; Execute validated indirect transfer
> > >
> > > ESR (Exception Syndrome Register) Integration:
> > > - BRK instruction immediate encoding format:
> > > 0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31)
> > > - TypeIndex indicates which W register contains expected type (W17 = 17)
> > > - AddrIndex indicates which X register contains target address (0-30)
> > > - Example: brk #33313 (0x8221) = expected type in W17, target address in X1
> > >
> > > Build and run tested with Linux kernel ARCH=arm64.
> > >
> > > gcc/ChangeLog:
> > >
> > > config/aarch64/aarch64-protos.h: Declare aarch64_indirect_branch_asm,
> > > and KCFI helpers.
> > > config/aarch64/aarch64.cc (aarch64_expand_call): Wrap CALLs in
> > > KCFI, with clobbers.
> > > (aarch64_indirect_branch_asm): New function, extract common
> > > logic for branch asm, like existing call asm helper.
> > > (aarch64_output_kcfi_insn): Emit KCFI assembly.
> > > config/aarch64/aarch64.md: Add KCFI RTL patterns and replace
> > > open-coded branch emission with aarch64_indirect_branch_asm.
> > > doc/invoke.texi: Document aarch64 nuances.
> > >
> > > gcc/testsuite/ChangeLog:
> > >
> > > * gcc.dg/kcfi/kcfi-adjacency.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-basics.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-call-sharing.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-complex-addressing.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-move-preservation.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-no-sanitize.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-offset-validation.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-large.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-medium.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-tail-calls.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-trap-section.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-trap-encoding.c: New test.
> > >
> > > Signed-off-by: Kees Cook <kees@kernel.org>
> > > ---
> > > gcc/config/aarch64/aarch64-protos.h | 5 +
> > > gcc/config/aarch64/aarch64.md | 66 +++++++++--
> > > gcc/config/aarch64/aarch64.cc | 105 ++++++++++++++++++
> > > gcc/doc/invoke.texi | 14 +++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 15 +++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
> > > .../gcc.dg/kcfi/kcfi-complex-addressing.c | 16 +++
> > > .../gcc.dg/kcfi/kcfi-move-preservation.c | 20 ++++
> > > .../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
> > > .../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
> > > .../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 12 ++
> > > .../gcc.dg/kcfi/kcfi-patchable-large.c | 12 ++
> > > .../gcc.dg/kcfi/kcfi-patchable-medium.c | 12 ++
> > > .../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 12 ++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 19 ++++
> > > .../gcc.dg/kcfi/kcfi-trap-encoding.c | 41 +++++++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 4 +
> > > 19 files changed, 379 insertions(+), 8 deletions(-)
> > > create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> > >
> > > diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
> > > index a9e407ba340e..c32d454fe277 100644
> > > --- a/gcc/config/aarch64/aarch64-protos.h
> > > +++ b/gcc/config/aarch64/aarch64-protos.h
> > > @@ -1272,6 +1272,7 @@ tree aarch64_resolve_overloaded_builtin_general (location_t, tree, void *);
> > >
> > > const char *aarch64_sls_barrier (int);
> > > const char *aarch64_indirect_call_asm (rtx);
> > > +const char *aarch64_indirect_branch_asm (rtx);
> > > extern bool aarch64_harden_sls_retbr_p (void);
> > > extern bool aarch64_harden_sls_blr_p (void);
> > >
> > > @@ -1295,4 +1296,8 @@ extern unsigned aarch64_stack_alignment (const_tree exp, unsigned align);
> > > extern rtx aarch64_gen_compare_zero_and_branch (rtx_code code, rtx x,
> > > rtx_code_label *label);
> > >
> > > +/* KCFI support. */
> > > +extern void kcfi_emit_trap_with_section (FILE *file, rtx trap_label_rtx);
> > > +extern const char *aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands);
> > > +
> > > #endif /* GCC_AARCH64_PROTOS_H */
> > > diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
> > > index 855df791bae7..84894816509c 100644
> > > --- a/gcc/config/aarch64/aarch64.md
> > > +++ b/gcc/config/aarch64/aarch64.md
> > > @@ -1506,6 +1506,19 @@
> > > }"
> > > )
> > >
> > > +;; KCFI indirect call
> > > +(define_insn "*call_insn"
> > > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucr"))
> > > + (match_operand 1 "" ""))
> > > + (match_operand 3 "const_int_operand"))
> > > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (clobber (reg:DI LR_REGNUM))]
> > > + "!SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, operands);
> > > +}
> > > + [(set_attr "type" "call")])
> > > +
> > > (define_insn "*call_insn"
> > > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand"))
> > > (match_operand 1 "" ""))
> > > @@ -1533,6 +1546,21 @@
> > > }"
> > > )
> > >
> > > +;; KCFI call with return value
> > > +(define_insn "*call_value_insn"
> > > + [(set (match_operand 0 "" "")
> > > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > > + "Ucr"))
> > > + (match_operand 2 "" ""))
> > > + (match_operand 4 "const_int_operand")))
> > > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (clobber (reg:DI LR_REGNUM))]
> > > + "!SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > > +}
> > > + [(set_attr "type" "call")])
> > > +
> > > (define_insn "*call_value_insn"
> > > [(set (match_operand 0 "" "")
> > > (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"))
> > > @@ -1573,6 +1601,19 @@
> > > }
> > > )
> > >
> > > +;; KCFI sibling call
> > > +(define_insn "*sibcall_insn"
> > > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs"))
> > > + (match_operand 1 ""))
> > > + (match_operand 3 "const_int_operand"))
> > > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (return)]
> > > + "SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, operands);
> > > +}
> > > + [(set_attr "type" "branch")])
> > > +
> > > (define_insn "*sibcall_insn"
> > > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs, Usf"))
> > > (match_operand 1 ""))
> > > @@ -1581,16 +1622,28 @@
> > > "SIBLING_CALL_P (insn)"
> > > {
> > > if (which_alternative == 0)
> > > - {
> > > - output_asm_insn ("br\\t%0", operands);
> > > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > > - }
> > > + return aarch64_indirect_branch_asm (operands[0]);
> >
> > Can you extract aarch64_indirect_branch_asm into a different patch?
> > Since it is only slightly related to this.
> > The main reason is because applying that can happen before the rest of
> > the KFCI patches are approved. And I will handling applying that once
> > it has been extracted out.
>
> Sure! I'll extract it.
>
> >
> > > return "b\\t%c0";
> > > }
> > > [(set_attr "type" "branch, branch")
> > > (set_attr "sls_length" "retbr,none")]
> > > )
> > >
> > > +;; KCFI sibling call with return value
> > > +(define_insn "*sibcall_value_insn"
> > > + [(set (match_operand 0 "")
> > > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > > + "Ucs"))
> > > + (match_operand 2 ""))
> > > + (match_operand 4 "const_int_operand")))
> > > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (return)]
> > > + "SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > > +}
> > > + [(set_attr "type" "branch")])
> > > +
> > > (define_insn "*sibcall_value_insn"
> > > [(set (match_operand 0 "")
> > > (call (mem:DI
> > > @@ -1601,10 +1654,7 @@
> > > "SIBLING_CALL_P (insn)"
> > > {
> > > if (which_alternative == 0)
> > > - {
> > > - output_asm_insn ("br\\t%1", operands);
> > > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > > - }
> > > + return aarch64_indirect_branch_asm (operands[1]);
> > > return "b\\t%c1";
> > > }
> > > [(set_attr "type" "branch, branch")
> > > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > > index 6dfdaa4fb9b0..86e3af992d2a 100644
> > > --- a/gcc/config/aarch64/aarch64.cc
> > > +++ b/gcc/config/aarch64/aarch64.cc
> > > @@ -83,6 +83,7 @@
> > > #include "rtlanal.h"
> > > #include "tree-dfa.h"
> > > #include "asan.h"
> > > +#include "kcfi.h"
> > > #include "aarch64-elf-metadata.h"
> > > #include "aarch64-feature-deps.h"
> > > #include "config/arm/aarch-common.h"
> > > @@ -11965,6 +11966,16 @@ aarch64_expand_call (rtx result, rtx mem, rtx cookie, bool sibcall)
> > >
> > > call = gen_rtx_CALL (VOIDmode, mem, const0_rtx);
> > >
> > > + /* Only indirect calls need KCFI instrumentation. */
> > > + bool is_direct_call = SYMBOL_REF_P (XEXP (mem, 0));
> > > + rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
> > > + : kcfi_get_type_id_for_expanding_gimple_call ();
> > > + if (kcfi_type_rtx)
> > > + {
> > > + /* Wrap call in KCFI. */
> > > + call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
> > > + }
> > > +
> > > if (result != NULL_RTX)
> > > call = gen_rtx_SET (result, call);
> > >
> > > @@ -19225,6 +19236,9 @@ aarch64_override_options_internal (struct gcc_options *opts)
> > > #endif
> > > }
> > >
> > > + if ((flag_sanitize & SANITIZE_KCFI) && TARGET_ILP32)
> > > + sorry ("%<-fsanitize=kcfi%> is not supported for %<-mabi=ilp32%>");
> > > +
> > > aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
> > > if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
> > > && !(isa_flags & AARCH64_FL_SME))
> > > @@ -30822,6 +30836,18 @@ aarch64_indirect_call_asm (rtx addr)
> > > return "";
> > > }
> > >
> > > +/* Generate assembly for AArch64 indirect branch instruction. ADDR is the
> > > + target address register. Returns any additional barrier instructions
> > > + needed for SLS (Straight Line Speculation) mitigation. */
> > > +
> > > +const char *
> > > +aarch64_indirect_branch_asm (rtx addr)
> > > +{
> > > + gcc_assert (REG_P (addr));
> > > + output_asm_insn ("br\t%0", &addr);
> > > + return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > > +}
> > > +
> > > /* Emit the assembly instruction to load the thread pointer into DEST.
> > > Select between different tpidr_elN registers depending on -mtp= setting. */
> > >
> > > @@ -33089,6 +33115,85 @@ aarch64_libgcc_floating_mode_supported_p
> > > #undef TARGET_DOCUMENTATION_NAME
> > > #define TARGET_DOCUMENTATION_NAME "AArch64"
> > >
> > > +/* Output the assembly for a KCFI checked call instruction. INSN is the
> > > + RTL instruction being processed. OPERANDS is the array of RTL operands
> > > + where operands[0] is the call target register, operands[3] is the KCFI
> > > + type ID constant. Returns the appropriate call instruction string. */
> > > +
> > > +const char *
> > > +aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands)
> > > +{
> > > + /* Target register is operands[0]. */
> > > + rtx target_reg = operands[0];
> > > + gcc_assert (REG_P (target_reg));
> > > +
> > > + /* Get KCFI type ID from operand[3]. */
> > > + uint32_t type_id = (uint32_t) INTVAL (operands[3]);
> > > +
> > > + /* Calculate typeid offset from call target. */
> > > + HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
> > > +
> > > + /* Get unique label number for this KCFI check. */
> > > + int labelno = kcfi_next_labelno ();
> > > +
> > > + /* Generate custom label names. */
> > > + char trap_name[32];
> > > + char call_name[32];
> > > + ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
> > > + ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
> > > +
> > > + rtx temp_operands[3];
> > > +
> > > + /* Load actual type into w16 from memory at offset using ldur. */
> > > + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> > > + temp_operands[1] = target_reg;
> > > + temp_operands[2] = GEN_INT (offset);
> > > + output_asm_insn ("ldur\t%w0, [%1, #%2]", temp_operands);
> >
> > Why not just:
> > temp_operands[0] = target_reg;
> > temp_operands[0] = GEN_INT (offset);
> > output_asm_insn ("ldur\tw16, [%0, #%1]", temp_operands);
>
> Oops, yes. I have rewritten these handlers so many times I've lost my
> mind. :)
>
> >
> > > +
> > > + /* Load expected type low 16 bits into w17. */
> > > + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> > > + temp_operands[1] = GEN_INT (type_id & 0xFFFF);
> > > + output_asm_insn ("mov\t%w0, #%1", temp_operands);
> > Likewise.
> > Also why not use fprintf here?
> > Like:
> > fprintf(asm_out_file, "mov\t%w0, #%d\n", type_id & 0xFFFF);
>
> It seemed like directly using fprintf() was frowned on, and the
> output_asm_insn with an operands array was the "correct" way to do all
> of these? I'm happy to do it however! :)
>
> > > +#undef TARGET_KCFI_SUPPORTED
> > > +#define TARGET_KCFI_SUPPORTED hook_bool_void_true
> >
> > Why return true always for it being supported? I thought this is not
> > supported with ilp32, it is definitely not supported for mingw. Is it
> > supported with -elf targets rather than -linux-gnu targets?
>
> Ah, it seemed like the only place I could do the checking for ilp32,
> etc was in aarch64_override_options_internal. I'll see if I can figure
> out how to move that into the TARGET hooks, but all the examples I could
> find where in the override_options functions...
I have not checked which one is called first
aarch64_override_options_internal or the hook. It might be that both
are needed depending on the order.
That is `-mabi=ilp32 -fsanitize=kfci` vs `-fsanitize=kfci -mabi=ilp32
`. If it might be the case that you only need the hook. It is not
valid to change the ABI at function level.
What I Mean is you could define something like:
```
static bool
aarch64_kcfi_supported_p (void)
{
return !TARGET_ILP32;
}
```
And not need the error message in override_options.
Note, I think the error/sorry message is incorrect too:
sorry ("%<-fsanitize=kcfi%> not supported by this target");
More likely should be `error ("%<-fsanitize=kcfi%> not supported by
this configuration");`
And then unset SANITIZE_KCFI from flag_sanitize.
`Sorry` is incorrect here I think. `error` is more friendly.
Thanks,
Andrew
>
> Thanks for the review! I'll do more fixing. :)
>
> -Kees
>
> --
> Kees Cook
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v8 5/7] aarch64: Add AArch64 Kernel Control Flow Integrity implementation
2025-11-21 0:30 ` Kees Cook
2025-11-21 3:28 ` Andrew Pinski
@ 2025-11-21 3:43 ` Andrew Pinski
1 sibling, 0 replies; 14+ messages in thread
From: Andrew Pinski @ 2025-11-21 3:43 UTC (permalink / raw)
To: Kees Cook
Cc: Qing Zhao, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
On Thu, Nov 20, 2025 at 4:30 PM Kees Cook <kees@kernel.org> wrote:
>
> On Thu, Nov 20, 2025 at 02:57:49PM -0800, Andrew Pinski wrote:
> > On Thu, Nov 20, 2025 at 2:29 PM Kees Cook <kees@kernel.org> wrote:
> > >
> > > Implement AArch64-specific KCFI backend.
> > >
> > > - Trap debugging through ESR (Exception Syndrome Register) encoding
> > > in BRK instruction immediate values.
> > >
> > > - Scratch register allocation using w16/w17 (x16/x17) following
> > > AArch64 procedure call standard for intra-procedure-call registers,
> > > which already makes x16/x17 available through existing clobbers.
> > >
> > > - Complementary with BTI (which uses a separate pass system to inject
> > > landing instructions where needed).
> > >
> > > - Does not interfere with SME, which uses attributes not function
> > > prototypes for distinguishing functions.
> > >
> > > Assembly Code Pattern for AArch64:
> > > ldur w16, [target, #-4] ; Load actual type ID from preamble
> > > mov w17, #type_id_low ; Load expected type (lower 16 bits)
> > > movk w17, #type_id_high, lsl #16 ; Load upper 16 bits if needed
> > > cmp w16, w17 ; Compare type IDs directly
> > > b.eq .Lpass ; Branch if types match
> > > .Ltrap: brk #esr_value ; Enhanced trap with register info
> > > .Lpass: blr/br target ; Execute validated indirect transfer
> > >
> > > ESR (Exception Syndrome Register) Integration:
> > > - BRK instruction immediate encoding format:
> > > 0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31)
> > > - TypeIndex indicates which W register contains expected type (W17 = 17)
> > > - AddrIndex indicates which X register contains target address (0-30)
> > > - Example: brk #33313 (0x8221) = expected type in W17, target address in X1
> > >
> > > Build and run tested with Linux kernel ARCH=arm64.
> > >
> > > gcc/ChangeLog:
> > >
> > > config/aarch64/aarch64-protos.h: Declare aarch64_indirect_branch_asm,
> > > and KCFI helpers.
> > > config/aarch64/aarch64.cc (aarch64_expand_call): Wrap CALLs in
> > > KCFI, with clobbers.
> > > (aarch64_indirect_branch_asm): New function, extract common
> > > logic for branch asm, like existing call asm helper.
> > > (aarch64_output_kcfi_insn): Emit KCFI assembly.
> > > config/aarch64/aarch64.md: Add KCFI RTL patterns and replace
> > > open-coded branch emission with aarch64_indirect_branch_asm.
> > > doc/invoke.texi: Document aarch64 nuances.
> > >
> > > gcc/testsuite/ChangeLog:
> > >
> > > * gcc.dg/kcfi/kcfi-adjacency.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-basics.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-call-sharing.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-complex-addressing.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-move-preservation.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-no-sanitize.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-offset-validation.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-large.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-medium.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-tail-calls.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-trap-section.c: Add aarch64 patterns.
> > > * gcc.dg/kcfi/kcfi-trap-encoding.c: New test.
> > >
> > > Signed-off-by: Kees Cook <kees@kernel.org>
> > > ---
> > > gcc/config/aarch64/aarch64-protos.h | 5 +
> > > gcc/config/aarch64/aarch64.md | 66 +++++++++--
> > > gcc/config/aarch64/aarch64.cc | 105 ++++++++++++++++++
> > > gcc/doc/invoke.texi | 14 +++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 15 +++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
> > > .../gcc.dg/kcfi/kcfi-complex-addressing.c | 16 +++
> > > .../gcc.dg/kcfi/kcfi-move-preservation.c | 20 ++++
> > > .../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
> > > .../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
> > > .../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 12 ++
> > > .../gcc.dg/kcfi/kcfi-patchable-large.c | 12 ++
> > > .../gcc.dg/kcfi/kcfi-patchable-medium.c | 12 ++
> > > .../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 12 ++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 19 ++++
> > > .../gcc.dg/kcfi/kcfi-trap-encoding.c | 41 +++++++
> > > gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 4 +
> > > 19 files changed, 379 insertions(+), 8 deletions(-)
> > > create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
> > >
> > > diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
> > > index a9e407ba340e..c32d454fe277 100644
> > > --- a/gcc/config/aarch64/aarch64-protos.h
> > > +++ b/gcc/config/aarch64/aarch64-protos.h
> > > @@ -1272,6 +1272,7 @@ tree aarch64_resolve_overloaded_builtin_general (location_t, tree, void *);
> > >
> > > const char *aarch64_sls_barrier (int);
> > > const char *aarch64_indirect_call_asm (rtx);
> > > +const char *aarch64_indirect_branch_asm (rtx);
> > > extern bool aarch64_harden_sls_retbr_p (void);
> > > extern bool aarch64_harden_sls_blr_p (void);
> > >
> > > @@ -1295,4 +1296,8 @@ extern unsigned aarch64_stack_alignment (const_tree exp, unsigned align);
> > > extern rtx aarch64_gen_compare_zero_and_branch (rtx_code code, rtx x,
> > > rtx_code_label *label);
> > >
> > > +/* KCFI support. */
> > > +extern void kcfi_emit_trap_with_section (FILE *file, rtx trap_label_rtx);
> > > +extern const char *aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands);
> > > +
> > > #endif /* GCC_AARCH64_PROTOS_H */
> > > diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
> > > index 855df791bae7..84894816509c 100644
> > > --- a/gcc/config/aarch64/aarch64.md
> > > +++ b/gcc/config/aarch64/aarch64.md
> > > @@ -1506,6 +1506,19 @@
> > > }"
> > > )
> > >
> > > +;; KCFI indirect call
> > > +(define_insn "*call_insn"
> > > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucr"))
> > > + (match_operand 1 "" ""))
> > > + (match_operand 3 "const_int_operand"))
> > > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (clobber (reg:DI LR_REGNUM))]
> > > + "!SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, operands);
> > > +}
> > > + [(set_attr "type" "call")])
> > > +
> > > (define_insn "*call_insn"
> > > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand"))
> > > (match_operand 1 "" ""))
> > > @@ -1533,6 +1546,21 @@
> > > }"
> > > )
> > >
> > > +;; KCFI call with return value
> > > +(define_insn "*call_value_insn"
> > > + [(set (match_operand 0 "" "")
> > > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > > + "Ucr"))
> > > + (match_operand 2 "" ""))
> > > + (match_operand 4 "const_int_operand")))
> > > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (clobber (reg:DI LR_REGNUM))]
> > > + "!SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > > +}
> > > + [(set_attr "type" "call")])
> > > +
> > > (define_insn "*call_value_insn"
> > > [(set (match_operand 0 "" "")
> > > (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"))
> > > @@ -1573,6 +1601,19 @@
> > > }
> > > )
> > >
> > > +;; KCFI sibling call
> > > +(define_insn "*sibcall_insn"
> > > + [(kcfi (call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs"))
> > > + (match_operand 1 ""))
> > > + (match_operand 3 "const_int_operand"))
> > > + (unspec:DI [(match_operand:DI 2 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (return)]
> > > + "SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, operands);
> > > +}
> > > + [(set_attr "type" "branch")])
> > > +
> > > (define_insn "*sibcall_insn"
> > > [(call (mem:DI (match_operand:DI 0 "aarch64_call_insn_operand" "Ucs, Usf"))
> > > (match_operand 1 ""))
> > > @@ -1581,16 +1622,28 @@
> > > "SIBLING_CALL_P (insn)"
> > > {
> > > if (which_alternative == 0)
> > > - {
> > > - output_asm_insn ("br\\t%0", operands);
> > > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > > - }
> > > + return aarch64_indirect_branch_asm (operands[0]);
> >
> > Can you extract aarch64_indirect_branch_asm into a different patch?
> > Since it is only slightly related to this.
> > The main reason is because applying that can happen before the rest of
> > the KFCI patches are approved. And I will handling applying that once
> > it has been extracted out.
>
> Sure! I'll extract it.
>
> >
> > > return "b\\t%c0";
> > > }
> > > [(set_attr "type" "branch, branch")
> > > (set_attr "sls_length" "retbr,none")]
> > > )
> > >
> > > +;; KCFI sibling call with return value
> > > +(define_insn "*sibcall_value_insn"
> > > + [(set (match_operand 0 "")
> > > + (kcfi (call (mem:DI (match_operand:DI 1 "aarch64_call_insn_operand"
> > > + "Ucs"))
> > > + (match_operand 2 ""))
> > > + (match_operand 4 "const_int_operand")))
> > > + (unspec:DI [(match_operand:DI 3 "const_int_operand")] UNSPEC_CALLEE_ABI)
> > > + (return)]
> > > + "SIBLING_CALL_P (insn)"
> > > +{
> > > + return aarch64_output_kcfi_insn (insn, &operands[1]);
> > > +}
> > > + [(set_attr "type" "branch")])
> > > +
> > > (define_insn "*sibcall_value_insn"
> > > [(set (match_operand 0 "")
> > > (call (mem:DI
> > > @@ -1601,10 +1654,7 @@
> > > "SIBLING_CALL_P (insn)"
> > > {
> > > if (which_alternative == 0)
> > > - {
> > > - output_asm_insn ("br\\t%1", operands);
> > > - return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > > - }
> > > + return aarch64_indirect_branch_asm (operands[1]);
> > > return "b\\t%c1";
> > > }
> > > [(set_attr "type" "branch, branch")
> > > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> > > index 6dfdaa4fb9b0..86e3af992d2a 100644
> > > --- a/gcc/config/aarch64/aarch64.cc
> > > +++ b/gcc/config/aarch64/aarch64.cc
> > > @@ -83,6 +83,7 @@
> > > #include "rtlanal.h"
> > > #include "tree-dfa.h"
> > > #include "asan.h"
> > > +#include "kcfi.h"
> > > #include "aarch64-elf-metadata.h"
> > > #include "aarch64-feature-deps.h"
> > > #include "config/arm/aarch-common.h"
> > > @@ -11965,6 +11966,16 @@ aarch64_expand_call (rtx result, rtx mem, rtx cookie, bool sibcall)
> > >
> > > call = gen_rtx_CALL (VOIDmode, mem, const0_rtx);
> > >
> > > + /* Only indirect calls need KCFI instrumentation. */
> > > + bool is_direct_call = SYMBOL_REF_P (XEXP (mem, 0));
> > > + rtx kcfi_type_rtx = is_direct_call ? NULL_RTX
> > > + : kcfi_get_type_id_for_expanding_gimple_call ();
> > > + if (kcfi_type_rtx)
> > > + {
> > > + /* Wrap call in KCFI. */
> > > + call = gen_rtx_KCFI (VOIDmode, call, kcfi_type_rtx);
> > > + }
> > > +
> > > if (result != NULL_RTX)
> > > call = gen_rtx_SET (result, call);
> > >
> > > @@ -19225,6 +19236,9 @@ aarch64_override_options_internal (struct gcc_options *opts)
> > > #endif
> > > }
> > >
> > > + if ((flag_sanitize & SANITIZE_KCFI) && TARGET_ILP32)
> > > + sorry ("%<-fsanitize=kcfi%> is not supported for %<-mabi=ilp32%>");
> > > +
> > > aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
> > > if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
> > > && !(isa_flags & AARCH64_FL_SME))
> > > @@ -30822,6 +30836,18 @@ aarch64_indirect_call_asm (rtx addr)
> > > return "";
> > > }
> > >
> > > +/* Generate assembly for AArch64 indirect branch instruction. ADDR is the
> > > + target address register. Returns any additional barrier instructions
> > > + needed for SLS (Straight Line Speculation) mitigation. */
> > > +
> > > +const char *
> > > +aarch64_indirect_branch_asm (rtx addr)
> > > +{
> > > + gcc_assert (REG_P (addr));
> > > + output_asm_insn ("br\t%0", &addr);
> > > + return aarch64_sls_barrier (aarch64_harden_sls_retbr_p ());
> > > +}
> > > +
> > > /* Emit the assembly instruction to load the thread pointer into DEST.
> > > Select between different tpidr_elN registers depending on -mtp= setting. */
> > >
> > > @@ -33089,6 +33115,85 @@ aarch64_libgcc_floating_mode_supported_p
> > > #undef TARGET_DOCUMENTATION_NAME
> > > #define TARGET_DOCUMENTATION_NAME "AArch64"
> > >
> > > +/* Output the assembly for a KCFI checked call instruction. INSN is the
> > > + RTL instruction being processed. OPERANDS is the array of RTL operands
> > > + where operands[0] is the call target register, operands[3] is the KCFI
> > > + type ID constant. Returns the appropriate call instruction string. */
> > > +
> > > +const char *
> > > +aarch64_output_kcfi_insn (rtx_insn *insn, rtx *operands)
> > > +{
> > > + /* Target register is operands[0]. */
> > > + rtx target_reg = operands[0];
> > > + gcc_assert (REG_P (target_reg));
> > > +
> > > + /* Get KCFI type ID from operand[3]. */
> > > + uint32_t type_id = (uint32_t) INTVAL (operands[3]);
> > > +
> > > + /* Calculate typeid offset from call target. */
> > > + HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
> > > +
> > > + /* Get unique label number for this KCFI check. */
> > > + int labelno = kcfi_next_labelno ();
> > > +
> > > + /* Generate custom label names. */
> > > + char trap_name[32];
> > > + char call_name[32];
> > > + ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
> > > + ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
> > > +
> > > + rtx temp_operands[3];
> > > +
> > > + /* Load actual type into w16 from memory at offset using ldur. */
> > > + temp_operands[0] = gen_rtx_REG (SImode, R16_REGNUM);
> > > + temp_operands[1] = target_reg;
> > > + temp_operands[2] = GEN_INT (offset);
> > > + output_asm_insn ("ldur\t%w0, [%1, #%2]", temp_operands);
> >
> > Why not just:
> > temp_operands[0] = target_reg;
> > temp_operands[0] = GEN_INT (offset);
> > output_asm_insn ("ldur\tw16, [%0, #%1]", temp_operands);
>
> Oops, yes. I have rewritten these handlers so many times I've lost my
> mind. :)
>
> >
> > > +
> > > + /* Load expected type low 16 bits into w17. */
> > > + temp_operands[0] = gen_rtx_REG (SImode, R17_REGNUM);
> > > + temp_operands[1] = GEN_INT (type_id & 0xFFFF);
> > > + output_asm_insn ("mov\t%w0, #%1", temp_operands);
> > Likewise.
> > Also why not use fprintf here?
> > Like:
> > fprintf(asm_out_file, "mov\t%w0, #%d\n", type_id & 0xFFFF);
>
> It seemed like directly using fprintf() was frowned on, and the
> output_asm_insn with an operands array was the "correct" way to do all
> of these? I'm happy to do it however! :)
the other option is doing:
```
fputs("\tmov\t%w0, #", asm_out_file);
fprint_whex (type_id & 0xFFFF);
fputc ('\n', asm_out_file);
```
Which might be faster than both fprintf and output_asm_insn here and
stays away from fprintf.
You could also use asm_fprintf which calls fprintf in some cases :).
Thanks,
Andrew
>
> > > +#undef TARGET_KCFI_SUPPORTED
> > > +#define TARGET_KCFI_SUPPORTED hook_bool_void_true
> >
> > Why return true always for it being supported? I thought this is not
> > supported with ilp32, it is definitely not supported for mingw. Is it
> > supported with -elf targets rather than -linux-gnu targets?
>
> Ah, it seemed like the only place I could do the checking for ilp32,
> etc was in aarch64_override_options_internal. I'll see if I can figure
> out how to move that into the TARGET hooks, but all the examples I could
> find where in the override_options functions...
>
> Thanks for the review! I'll do more fixing. :)
>
> -Kees
>
> --
> Kees Cook
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v8 6/7] arm: Add ARM 32-bit Kernel Control Flow Integrity implementation
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
` (4 preceding siblings ...)
2025-11-20 22:21 ` [PATCH v8 5/7] aarch64: Add AArch64 " Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
2025-11-20 22:21 ` [PATCH v8 7/7] riscv: Add RISC-V " Kees Cook
6 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
Implement ARM 32-bit KCFI backend:
- Use eor instructions for 32-bit immediate loading.
- Trap debugging through UDF instruction immediate encoding following
AArch64 BRK pattern for encoding registers with useful contents.
- Scratch register allocation uses ip by default since it is most
commonly available as a caller-saved register. When, due to register
pressure, ip is the call target register, use r3. Since r3 is already
caller-saved, the allocator will have already arranged to reload r3
after the call if it is needed again. However, if r3 is being used
as the 4th argument to the call, we must internally spill/reload
it. Also uses r3 with -ffixed-ip or -ffixed-r12.
Assembly Code Pattern for ARM 32-bit:
ldr ip, [target, #-4] ; Load actual type ID from preamble
eor ip, ip, #byte1 ; OR type id byte 1
eor ip, ip, #byte2 << 8 ; OR type id byte 2
eor ip, ip, #byte3 << 16 ; OR type id byte 3
eors ip, ip, #byte4 << 24 ; OR type id byte 4 and set if reg is 0
beq .Lkcfi_call ; Branch if typeids matched
.Lkcfi_trap: udf #udf_value ; Undefined instruction trap with encoding
.Lkcfi_call: blx/bx target ; Execute validated indirect transfer
UDF Immediate Encoding (following AArch64 ESR pattern):
- UDF instruction immediate encoding format:
0x8000 | ((ExpectedTypeReg & 31) << 5) | (TargetAddrReg & 31)
- ExpectedTypeReg indicates which register contains expected type (R12 = 12)
- TargetAddrReg indicates which register contains target address (0-15)
- Example: udf #33154 (0x817A) = expected type in R12, target address in R2
Build and run tested with Linux kernel ARCH=arm.
gcc/ChangeLog:
config/arm/arm-protos.h: Declare KCFI helpers.
config/arm/arm.cc (arm_maybe_wrap_call_with_kcfi): New function.
(arm_maybe_wrap_call_value_with_kcfi): New function.
(arm_output_kcfi_insn): Emit KCFI assembly.
config/arm/arm.md: Add KCFI RTL patterns and hook expansion.
doc/invoke.texi: Document arm32 nuances.
gcc/testsuite/ChangeLog:
* gcc.dg/kcfi/kcfi-adjacency.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-basics.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-call-sharing.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-complex-addressing.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-move-preservation.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-no-sanitize.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-offset-validation.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-patchable-large.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-patchable-medium.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-tail-calls.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-trap-encoding.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-trap-section.c: Add arm patterns.
* gcc.dg/kcfi/kcfi-arm-fixed-ip.c: New test.
* gcc.dg/kcfi/kcfi-arm-fixed-r12.c: New test.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/config/arm/arm-protos.h | 4 +
gcc/config/arm/arm.md | 62 +++++++
gcc/config/arm/arm.cc | 170 ++++++++++++++++++
gcc/doc/invoke.texi | 17 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 16 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-ip.c | 15 ++
.../gcc.dg/kcfi/kcfi-arm-fixed-r12.c | 15 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 24 ++-
gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 12 +-
.../gcc.dg/kcfi/kcfi-complex-addressing.c | 19 ++
.../gcc.dg/kcfi/kcfi-move-preservation.c | 22 ++-
.../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
.../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
.../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 9 +-
.../gcc.dg/kcfi/kcfi-patchable-large.c | 9 +-
.../gcc.dg/kcfi/kcfi-patchable-medium.c | 9 +-
.../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 9 +-
gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 18 ++
.../gcc.dg/kcfi/kcfi-trap-encoding.c | 30 +++-
gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 5 +-
21 files changed, 458 insertions(+), 16 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-ip.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-r12.c
diff --git a/gcc/config/arm/arm-protos.h b/gcc/config/arm/arm-protos.h
index ff7e7658f912..ad3dc522e2b9 100644
--- a/gcc/config/arm/arm-protos.h
+++ b/gcc/config/arm/arm-protos.h
@@ -607,6 +607,10 @@ void arm_initialize_isa (sbitmap, const enum isa_feature *);
const char * arm_gen_far_branch (rtx *, int, const char * , const char *);
+rtx arm_maybe_wrap_call_with_kcfi (rtx, rtx);
+rtx arm_maybe_wrap_call_value_with_kcfi (rtx, rtx);
+const char *arm_output_kcfi_insn (rtx_insn *, rtx *);
+
bool arm_mve_immediate_check(rtx, machine_mode, bool);
opt_machine_mode arm_mve_data_mode (scalar_mode, poly_uint64);
diff --git a/gcc/config/arm/arm.md b/gcc/config/arm/arm.md
index b5d92aebf4d8..f1149b3e291d 100644
--- a/gcc/config/arm/arm.md
+++ b/gcc/config/arm/arm.md
@@ -8622,6 +8622,7 @@
else
{
pat = gen_call_internal (operands[0], operands[1], operands[2]);
+ pat = arm_maybe_wrap_call_with_kcfi (pat, XEXP (operands[0], 0));
arm_emit_call_insn (pat, XEXP (operands[0], 0), false);
}
@@ -8680,6 +8681,20 @@
}
)
+;; KCFI indirect call - KCFI wraps just the call pattern
+(define_insn "*kcfi_call_reg"
+ [(kcfi (call (mem:SI (match_operand:SI 0 "s_register_operand" "r"))
+ (match_operand 1 "" ""))
+ (match_operand 2 "const_int_operand"))
+ (use (match_operand 3 "" ""))
+ (clobber (reg:SI LR_REGNUM))]
+ "TARGET_32BIT && !SIBLING_CALL_P (insn) && arm_ccfsm_state == 0"
+{
+ return arm_output_kcfi_insn (insn, operands);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "40")])
+
(define_insn "*call_reg_armv5"
[(call (mem:SI (match_operand:SI 0 "s_register_operand" "r"))
(match_operand 1 "" ""))
@@ -8746,6 +8761,7 @@
{
pat = gen_call_value_internal (operands[0], operands[1],
operands[2], operands[3]);
+ pat = arm_maybe_wrap_call_value_with_kcfi (pat, XEXP (operands[1], 0));
arm_emit_call_insn (pat, XEXP (operands[1], 0), false);
}
@@ -8792,6 +8808,21 @@
}
}")
+;; KCFI indirect call_value - KCFI wraps just the call pattern
+(define_insn "*kcfi_call_value_reg"
+ [(set (match_operand 0 "" "")
+ (kcfi (call (mem:SI (match_operand:SI 1 "s_register_operand" "r"))
+ (match_operand 2 "" ""))
+ (match_operand 3 "const_int_operand")))
+ (use (match_operand 4 "" ""))
+ (clobber (reg:SI LR_REGNUM))]
+ "TARGET_32BIT && !SIBLING_CALL_P (insn) && arm_ccfsm_state == 0"
+{
+ return arm_output_kcfi_insn (insn, &operands[1]);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "40")])
+
(define_insn "*call_value_reg_armv5"
[(set (match_operand 0 "" "")
(call (mem:SI (match_operand:SI 1 "s_register_operand" "r"))
@@ -8894,6 +8925,7 @@
operands[2] = const0_rtx;
pat = gen_sibcall_internal (operands[0], operands[1], operands[2]);
+ pat = arm_maybe_wrap_call_with_kcfi (pat, XEXP (operands[0], 0));
arm_emit_call_insn (pat, operands[0], true);
DONE;
}"
@@ -8928,11 +8960,26 @@
pat = gen_sibcall_value_internal (operands[0], operands[1],
operands[2], operands[3]);
+ pat = arm_maybe_wrap_call_value_with_kcfi (pat, XEXP (operands[1], 0));
arm_emit_call_insn (pat, operands[1], true);
DONE;
}"
)
+;; KCFI sibling call - KCFI wraps just the call pattern
+(define_insn "*kcfi_sibcall_insn"
+ [(kcfi (call (mem:SI (match_operand:SI 0 "s_register_operand" "r"))
+ (match_operand 1 "" ""))
+ (match_operand 2 "const_int_operand"))
+ (return)
+ (use (match_operand 3 "" ""))]
+ "TARGET_32BIT && SIBLING_CALL_P (insn) && arm_ccfsm_state == 0"
+{
+ return arm_output_kcfi_insn (insn, operands);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "40")])
+
(define_insn "*sibcall_insn"
[(call (mem:SI (match_operand:SI 0 "call_insn_operand" "Cs, US"))
(match_operand 1 "" ""))
@@ -8953,6 +9000,21 @@
[(set_attr "type" "call")]
)
+;; KCFI sibling call with return value - KCFI wraps just the call pattern
+(define_insn "*kcfi_sibcall_value_insn"
+ [(set (match_operand 0 "" "")
+ (kcfi (call (mem:SI (match_operand:SI 1 "s_register_operand" "r"))
+ (match_operand 2 "" ""))
+ (match_operand 3 "const_int_operand")))
+ (return)
+ (use (match_operand 4 "" ""))]
+ "TARGET_32BIT && SIBLING_CALL_P (insn) && arm_ccfsm_state == 0"
+{
+ return arm_output_kcfi_insn (insn, &operands[1]);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "40")])
+
(define_insn "*sibcall_value_insn"
[(set (match_operand 0 "" "")
(call (mem:SI (match_operand:SI 1 "call_insn_operand" "Cs,US"))
diff --git a/gcc/config/arm/arm.cc b/gcc/config/arm/arm.cc
index 07d24d1f67ed..a65de835b580 100644
--- a/gcc/config/arm/arm.cc
+++ b/gcc/config/arm/arm.cc
@@ -77,6 +77,8 @@
#include "aarch-common-protos.h"
#include "machmode.h"
#include "arm-builtins.h"
+#include "kcfi.h"
+#include "flags.h"
/* This file should be included last. */
#include "target-def.h"
@@ -35842,6 +35844,174 @@ arm_mode_base_reg_class (machine_mode mode)
return MODE_BASE_REG_REG_CLASS (mode);
}
+/* Apply KCFI wrapping to call pattern if needed. PAT is the RTL call
+ pattern to potentially wrap with KCFI instrumentation. ADDR is the
+ call target address RTL expression. Returns the possibly modified
+ call pattern with KCFI wrapper applied for indirect calls. */
+
+rtx
+arm_maybe_wrap_call_with_kcfi (rtx pat, rtx addr)
+{
+ /* Only indirect calls need KCFI instrumentation. */
+ bool is_direct_call = SYMBOL_REF_P (addr);
+ if (!is_direct_call)
+ {
+ rtx kcfi_type_rtx = kcfi_get_type_id_for_expanding_gimple_call ();
+ if (kcfi_type_rtx)
+ {
+ /* Extract the CALL from the PARALLEL and wrap it with KCFI. */
+ rtx call_rtx = XVECEXP (pat, 0, 0);
+ rtx kcfi_call = gen_rtx_KCFI (VOIDmode, call_rtx, kcfi_type_rtx);
+
+ /* Replace the CALL in the PARALLEL with the KCFI-wrapped call. */
+ XVECEXP (pat, 0, 0) = kcfi_call;
+ }
+ }
+ return pat;
+}
+
+/* Apply KCFI wrapping to call_value pattern if needed. PAT is the RTL
+ call_value pattern to potentially wrap with KCFI instrumentation. ADDR
+ is the call target address RTL expression. Returns the possibly modified
+ call pattern with KCFI wrapper applied for indirect calls. */
+
+rtx
+arm_maybe_wrap_call_value_with_kcfi (rtx pat, rtx addr)
+{
+ /* Only indirect calls need KCFI instrumentation. */
+ bool is_direct_call = SYMBOL_REF_P (addr);
+ if (!is_direct_call)
+ {
+ rtx kcfi_type_rtx = kcfi_get_type_id_for_expanding_gimple_call ();
+ if (kcfi_type_rtx)
+ {
+ /* Extract the SET from the PARALLEL and wrap its CALL with KCFI. */
+ rtx set_rtx = XVECEXP (pat, 0, 0);
+ rtx call_rtx = SET_SRC (set_rtx);
+ rtx kcfi_call = gen_rtx_KCFI (VOIDmode, call_rtx, kcfi_type_rtx);
+
+ /* Replace the CALL in the SET with the KCFI-wrapped call. */
+ SET_SRC (set_rtx) = kcfi_call;
+ }
+ }
+ return pat;
+}
+
+/* Output the assembly for a KCFI checked call instruction. INSN is the
+ RTL instruction being processed. OPERANDS is the array of RTL operands
+ where operands[0] is the call target register, operands[2] is the KCFI
+ type ID constant. Returns an empty string as all output is handled by
+ direct assembly generation. */
+
+const char *
+arm_output_kcfi_insn (rtx_insn *insn, rtx *operands)
+{
+ /* KCFI type id. */
+ uint32_t type_id = INTVAL (operands[2]);
+
+ /* Calculate typeid offset from call target. */
+ HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
+
+ /* Get unique label number for this KCFI check. */
+ int labelno = kcfi_next_labelno ();
+
+ /* Generate custom label names. */
+ char trap_name[32];
+ char call_name[32];
+ ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
+ ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
+
+ /* Create memory operand for the type load. */
+ rtx mem_op = gen_rtx_MEM (SImode,
+ gen_rtx_PLUS (SImode, operands[0],
+ GEN_INT (offset)));
+ rtx temp_operands[6];
+
+ /* Normally we can use r12 as our scratch register. */
+ unsigned scratch_reg_num = IP_REGNUM;
+ /* If register pressure has made r12 our target register, we need to pick
+ a different register. We don't want to spill our target register
+ because on reload at the end of the KCFI check, we'd be producing
+ the very kind of call gadget we were trying to protect against:
+ "pop %target; call %target". In this case, use r3 as our scratch
+ register. But since r3 may be used for function arguments, we need
+ to check if it is being used for that and only spill/reload if that
+ happens. Any spill/reload of r3 due to making a call will already
+ have been managed by the register allocator, so we only have to care
+ about not clobbering the argument value it may be carrying into the
+ call here. Also use r3 when r12 is a fixed register. */
+ if (REGNO (operands[0]) == scratch_reg_num
+ || fixed_regs[scratch_reg_num])
+ scratch_reg_num = LAST_ARG_REGNUM;
+ rtx scratch_reg = gen_rtx_REG (SImode, scratch_reg_num);
+
+ /* We only need to spill r3 if it's actually used by the call. */
+ bool need_spill = (scratch_reg_num == LAST_ARG_REGNUM)
+ && reg_overlap_mentioned_p (scratch_reg, insn);
+
+ /* Calculate trap immediate. */
+ unsigned addr_reg_num = REGNO (operands[0]);
+ /* The scratch register is always clobbered by eor seq: use 0x1F. */
+ unsigned udf_immediate = 0x8000 | (0x1F << 5) | (addr_reg_num & 31);
+
+ /* Spill if needed. */
+ if (need_spill)
+ output_asm_insn ("push\t{%0}", &scratch_reg);
+
+ /* Load actual type from memory into scratch register. */
+ temp_operands[0] = scratch_reg;
+ temp_operands[1] = mem_op;
+ output_asm_insn ("ldr\t%0, %1", temp_operands);
+
+ /* Set up operands for EOR instructions - source and dest are the same. */
+ temp_operands[0] = scratch_reg;
+ temp_operands[1] = scratch_reg;
+
+ /* XOR with type_id byte 0. */
+ temp_operands[2] = GEN_INT (type_id & 0xFF);
+ output_asm_insn ("eor\t%0, %1, %2", temp_operands);
+
+ /* XOR with type_id byte 1 << 8. */
+ temp_operands[2] = GEN_INT (((type_id >> 8) & 0xFF) << 8);
+ output_asm_insn ("eor\t%0, %1, %2", temp_operands);
+
+ /* XOR with type_id byte 2 << 16. */
+ temp_operands[2] = GEN_INT (((type_id >> 16) & 0xFF) << 16);
+ output_asm_insn ("eor\t%0, %1, %2", temp_operands);
+
+ /* EORS with type_id byte 3 << 24 (sets flags). */
+ temp_operands[2] = GEN_INT (((type_id >> 24) & 0xFF) << 24);
+ output_asm_insn ("eors\t%0, %1, %2", temp_operands);
+
+ /* Reload if needed. */
+ if (need_spill)
+ output_asm_insn ("pop\t{%0}", &scratch_reg);
+
+ /* Output conditional branch to call label. */
+ fputs ("\tbeq\t", asm_out_file);
+ assemble_name (asm_out_file, call_name);
+ fputc ('\n', asm_out_file);
+
+ /* Output trap label and UDF instruction. */
+ ASM_OUTPUT_LABEL (asm_out_file, trap_name);
+ temp_operands[0] = GEN_INT (udf_immediate);
+ output_asm_insn ("udf\t%0", temp_operands);
+
+ /* Output pass/call label. */
+ ASM_OUTPUT_LABEL (asm_out_file, call_name);
+
+ /* Call or tail call instruction. */
+ if (SIBLING_CALL_P (insn))
+ output_asm_insn ("bx\t%0", operands);
+ else
+ output_asm_insn ("blx\t%0", operands);
+
+ return "";
+}
+
+#undef TARGET_KCFI_SUPPORTED
+#define TARGET_KCFI_SUPPORTED hook_bool_void_true
+
#undef TARGET_DOCUMENTATION_NAME
#define TARGET_DOCUMENTATION_NAME "ARM"
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 94bb4ffcce98..2f8eef03f953 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -18741,6 +18741,23 @@ trap is taken, allowing the kernel to identify both the KCFI violation
and the involved registers for detailed diagnostics (eliminating the need
for a separate @code{.kcfi_traps} section as used on x86_64).
+On ARM 32-bit, KCFI type identifiers are emitted as a @code{.word ID}
+directive (a 32-bit constant) before the function entry. ARM's
+natural 4-byte instruction alignment eliminates the need for additional
+alignment NOPs. When used with @option{-fpatchable-function-entry}, the
+type identifier is placed before any prefix NOPs. The runtime check
+preserves argument registers @code{r0} and @code{r1} using @code{push}
+and @code{pop} instructions, then uses them as scratch registers for
+the type comparison. The expected type is loaded using @code{movw} and
+@code{movt} instruction pairs for 32-bit immediate values. Type mismatches
+trigger a @code{udf} instruction with an immediate value that encodes
+both the expected type register index and the target address register
+index in the format @code{0x8000 | (type_reg << 5) | addr_reg}. This
+encoding is captured in the UDF immediate field when the trap is taken,
+allowing the kernel to identify both the KCFI violation and the involved
+registers for detailed diagnostics (eliminating the need for a separate
+@code{.kcfi_traps} section as used on x86_64).
+
KCFI is intended primarily for kernel code and may not be suitable
for user-space applications that rely on techniques incompatible
with strict type checking of indirect calls.
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
index f3d7d23e6af2..00c14c0375cd 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
@@ -78,4 +78,20 @@ __attribute__((noinline)) void test_conditional_call(int flag) {
** ...
*/
+/*
+** test_complex_args: { target arm*-*-* }
+** ...
+** ldr ip, \[(r[0-9]+|lr), #-4\]
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eors ip, ip, #[0-9]+
+** beq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** udf #[0-9]+
+** .Lkcfi_call\2:
+** bx \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-ip.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-ip.c
new file mode 100644
index 000000000000..bb76078aa800
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-ip.c
@@ -0,0 +1,15 @@
+/* Test that KCFI works with -ffixed-ip on ARM by using r3 fallback. */
+/* { dg-do compile { target arm*-*-* } } */
+/* { dg-additional-options "-ffixed-ip" } */
+
+void target_function(void) {}
+
+int main() {
+ void (*func_ptr)(void) = target_function;
+ func_ptr();
+ return 0;
+}
+
+/* Should use r3 instead of ip for scratch register when ip is fixed. */
+/* { dg-final { scan-assembler "ldr\tr3, \\\[r\[0-9\]+, #-4\\\]" } } */
+/* { dg-final { scan-assembler "eor\tr3, r3, #\[0-9\]+" } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-r12.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-r12.c
new file mode 100644
index 000000000000..dd3f1f001f5b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-arm-fixed-r12.c
@@ -0,0 +1,15 @@
+/* Test that KCFI works with -ffixed-r12 on ARM by using r3 fallback. */
+/* { dg-do compile { target arm*-*-* } } */
+/* { dg-additional-options "-ffixed-r12" } */
+
+void target_function(void) {}
+
+int main() {
+ void (*func_ptr)(void) = target_function;
+ func_ptr();
+ return 0;
+}
+
+/* Should use r3 instead of ip for scratch register when ip is fixed. */
+/* { dg-final { scan-assembler "ldr\tr3, \\\[r\[0-9\]+, #-4\\\]" } } */
+/* { dg-final { scan-assembler "eor\tr3, r3, #\[0-9\]+" } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
index 6eac946f7abf..4c9a1e7aa552 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
@@ -59,8 +59,8 @@ int main() {
/* x86_64: Verify type ID in preamble (after NOPs, before function label) */
/* { dg-final { scan-assembler {__cfi_regular_function:\n\t+nop\n.*\n\t+movl\t+\$0x[0-9a-f]+, %eax} { target x86_64-*-* } } } */
-/* AArch64: Verify type ID word in preamble. */
-/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* } } } */
+/* AArch64, ARM32: Verify type ID word in preamble. */
+/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* arm*-*-* } } } */
/*
** static_caller: { target x86_64-*-* }
@@ -94,6 +94,22 @@ int main() {
** ...
*/
+/*
+** static_caller: { target arm*-*-* }
+** ...
+** ldr ip, \[(r[0-9]+), #-4\]
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eors ip, ip, #[0-9]+
+** beq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** udf #[0-9]+
+** .Lkcfi_call\2:
+** blx \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
/* Extern functions should NOT get KCFI preambles. */
@@ -112,5 +128,5 @@ int main() {
__kcfi_typeid_ symbols. */
/* { dg-final { scan-assembler-not {__kcfi_typeid_external_func_int} } } */
-/* AArch64 should NOT have trap section (use immediate instructions instead). */
-/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
+/* AArch64, ARM32 should NOT have trap section (use immediate instructions instead). */
+/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
index 6062d74ef62a..625cfa350a53 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
@@ -60,21 +60,27 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
1. KCFI check for is_visible call with is_visible type ID A.
2. KCFI check for is_bin_visible and is_bin_visible_again call with type ID B. */
-/* Verify we have TWO different KCFI check sequences. */
+/* Verify we have TWO different KCFI check sequences (except on arm, which
+ due to heavy register pressure ends up generating an unmergeable series
+ of instructions: the call target registers differ). */
/* Each check should have different type ID constants. */
/* x86: { dg-final { scan-assembler-times {movl\s+\$-?[0-9]+,\s+%r10d} 2 { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-times {mov\s+w17, #[0-9]+} 2 { target aarch64*-*-* } } } */
+/* ARM 32-bit: { dg-final { scan-assembler-times {ldr\s+ip, \[(?:r[0-9]+|lr), #-4\]} 3 { target arm*-*-* } } } */
-/* Verify the checks use DIFFERENT type IDs (not shared).
+/* Verify the checks use DIFFERENT type IDs (not shared, except arm: see above).
We should NOT see the same type ID used twice - that would indicate
unmerged sharing. */
/* x86: { dg-final { scan-assembler-not {movl\s+\$(-?[0-9]+),\s+%r10d.*movl\s+\$\1,\s+%r10d} { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-not {mov\s+w17, #([0-9]+).*mov\s+w17, #\1} { target aarch64*-*-* } } } */
+/* ARM 32-bit: { dg-final { scan-assembler-not {eor\s+ip, ip, #([0-9]+)\n\teor\s+r3, r3, #([0-9]+)\n\teor\s+r3, r3, #([0-9]+)\n\teors\s+r3, r3, #([0-9]+).*eor\s+r3, r3, #\1\n\teor\s+r3, r3, #[0-9]+\n\teor\s+r3, r3, #[0-9]+\n\teors\s+r3, r3, #[0-9]+.*eor\s+r3, r3, #\1\n\teor\s+r3, r3, #[0-9]+\n\teor\s+r3, r3, #[0-9]+\n\teors\s+r3, r3, #[0-9]+} { target arm*-*-* } } } */
/* Verify expected number of traps. */
/* x86: { dg-final { scan-assembler-times {ud2} 2 { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
+/* ARM 32-bit: { dg-final { scan-assembler-times {udf\s+#[0-9]+} 3 { target arm*-*-* } } } */
-/* Verify 2 separate call sites. */
+/* Verify 2 separate call sites (except arm). */
/* x86: { dg-final { scan-assembler-times {jmp\s+\*%[a-z0-9]+} 2 { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-times {br\tx[0-9]+} 2 { target aarch64*-*-* } } } */
+/* ARM 32-bit: { dg-final { scan-assembler-times {bx\s+(?:r[0-9]+|lr)} 3 { target arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
index 3ffbd408a69e..92d9dbb7906b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
@@ -162,4 +162,23 @@ int main() {
** ...
*/
+/* Looking for r3 fall-back due to register pressure. */
+/*
+** force_r3_spill: { target arm*-*-* }
+** ...
+** push \{r3\}
+** ldr r3, \[(ip), #-4\]
+** eor r3, r3, #[0-9]+
+** eor r3, r3, #[0-9]+
+** eor r3, r3, #[0-9]+
+** eors r3, r3, #[0-9]+
+** pop \{r3\}
+** beq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** udf #[0-9]+
+** .Lkcfi_call\2:
+** blx \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
index df39b7f0a8a3..c259620a3ed8 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
@@ -74,7 +74,25 @@ int main(void)
** ...
*/
+/*
+** indirect_call: { target arm*-*-* }
+** ...
+** mov (r[0-9]+), r0
+** ...
+** ldr ip, \[\1, #-4\]
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eors ip, ip, #[0-9]+
+** beq .Lkcfi_call([0-9]+)
+** .Lkcfi_trap[0-9]+:
+** udf #[0-9]+
+** .Lkcfi_call\2:
+** bx \1
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
-/* AArch64 should NOT have trap section (use immediate instructions instead). */
-/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* } } } */
+/* AArch64, ARM32 should NOT have trap section (use immediate instructions instead). */
+/* { dg-final { scan-assembler-not {\.kcfi_traps} { target aarch64*-*-* arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
index cdeb202ffd12..f8103466816a 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
@@ -76,15 +76,20 @@ int main(void)
/* Verify correct number of KCFI checks: exactly 2 */
/* { dg-final { scan-assembler-times {ud2} 2 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
+/* { dg-final { scan-assembler-times {udf\s+#[0-9]+} 2 { target arm*-*-* } } } */
/* Positive controls: these should have KCFI checks. */
/* { dg-final { scan-assembler {normal_function:.*ud2.*\.size\s+normal_function} { target x86_64-*-* } } } */
/* { dg-final { scan-assembler {wrap_normal_inline:.*ud2.*\.size\s+wrap_normal_inline} { target x86_64-*-* } } } */
/* { dg-final { scan-assembler {normal_function:.*brk\s+#[0-9]+.*\.size\s+normal_function} { target aarch64*-*-* } } } */
/* { dg-final { scan-assembler {wrap_normal_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_normal_inline} { target aarch64*-*-* } } } */
+/* { dg-final { scan-assembler {normal_function:.*udf\t#[0-9]+.*\.size\s+normal_function} { target arm*-*-* } } } */
+/* { dg-final { scan-assembler {wrap_normal_inline:.*udf\t#[0-9]+.*\.size\s+wrap_normal_inline} { target arm*-*-* } } } */
/* Negative controls: these should NOT have KCFI checks. */
/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ud2.*\.size\s+sensitive_non_inline_function} { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*ud2.*\.size\s+wrap_sensitive_inline} { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*brk\s+#[0-9]+.*\.size\s+sensitive_non_inline_function} { target aarch64*-*-* } } } */
/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_sensitive_inline} { target aarch64*-*-* } } } */
+/* { dg-final { scan-assembler-not {sensitive_non_inline_function:[^\n]*udf\t#[0-9]+[^\n]*\.size\tsensitive_non_inline_function} { target arm*-*-* } } } */
+/* { dg-final { scan-assembler-not {wrap_sensitive_inline:[^\n]*udf\t#[0-9]+[^\n]*\.size\twrap_sensitive_inline} { target arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
index af6d86803576..95bb46304493 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
@@ -35,3 +35,4 @@ int main() {
So a total of exactly 1 KCFI check in the entire program. */
/* { dg-final { scan-assembler-times {addl\t-4\(%r[ad]x\), %r1[01]d} 1 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 1 { target aarch64-*-* } } } */
+/* { dg-final { scan-assembler-times {ldr\tip, \[r[0-9]+, #-4\]} 1 { target arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
index 0ced5c43ae92..88f0ae64091b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
@@ -30,3 +30,6 @@ int main() {
/* AArch64: All call sites should use -4 offset. */
/* { dg-final { scan-assembler {ldur\tw16, \[x[0-9]+, #-4\]} { target aarch64*-*-* } } } */
+
+/* ARM 32-bit: All call sites should use -4 offset with EOR sequence. */
+/* { dg-final { scan-assembler {ldr\tip, \[r[0-9]+, #-4\]} { target arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
index 7a251cbdee3b..612140a0f509 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
@@ -29,7 +29,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -47,4 +47,11 @@ int main() {
** ...
*/
+/*
+** main: { target arm*-*-* }
+** ...
+** ldr (r[0-9]+|ip), \[(r[0-9]+|ip), #-4\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
index 3ed5d16c8e91..da3d8dc41f60 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
@@ -18,7 +18,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -36,4 +36,11 @@ int main() {
** ...
*/
+/*
+** main: { target arm*-*-* }
+** ...
+** ldr (r[0-9]+|ip), \[(r[0-9]+|ip), #-48\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
index e354914209e9..d61274e70157 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
@@ -25,7 +25,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -43,4 +43,11 @@ int main() {
** ...
*/
+/*
+** main: { target arm*-*-* }
+** ...
+** ldr ip, \[r[0-9]+, #-20\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
index 7a1dc4fa0e07..93df4d7ea5b6 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
@@ -26,7 +26,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -44,4 +44,11 @@ int main() {
** ...
*/
+/*
+** main: { target arm*-*-* }
+** ...
+** ldr (r[0-9]+|ip), \[(r[0-9]+|ip), #-16\]
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
index 1a7cc4aa167f..ee9d2c6dc741 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
@@ -97,3 +97,21 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
/* Type ID loading should use mov + movk pattern for 32-bit constants. */
/* { dg-final { scan-assembler {mov\tw17, #[0-9]+} { target aarch64-*-* } } } */
/* { dg-final { scan-assembler {movk\tw17, #[0-9]+, lsl #16} { target aarch64-*-* } } } */
+
+/* Should have exactly 4 KCFI checks for indirect calls (load type ID from
+ -4 offset + EOR sequence). */
+/* { dg-final { scan-assembler-times {ldr\tip, \[r[0-9]+, #-4\]} 4 { target arm*-*-* } } } */
+/* { dg-final { scan-assembler-times {eors\tip, ip, #[0-9]+} 4 { target arm*-*-* } } } */
+
+/* Should have exactly 4 trap instructions. */
+/* { dg-final { scan-assembler-times {udf\t#[0-9]+} 4 { target arm*-*-* } } } */
+
+/* Should have exactly 3 protected tail calls (bx through register after
+ KCFI check). */
+/* { dg-final { scan-assembler-times {bx\tr[0-9]+} 3 { target arm*-*-* } } } */
+
+/* Should have exactly 1 regular call (non-tail call case). */
+/* { dg-final { scan-assembler-times {blx\tr[0-9]+} 1 { target arm*-*-* } } } */
+
+/* Type ID loading should use 4 EOR instructions for 32-bit constants. */
+/* { dg-final { scan-assembler-times {eors?\tip, ip, #[0-9]+} 16 { target arm*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
index 0c257565c9e8..f302283db943 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-encoding.c
@@ -1,5 +1,5 @@
/* Test AArch64 and ARM32 KCFI trap encoding in BRK/UDF instructions. */
-/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-do compile { target aarch64*-*-* arm*-*-* } } */
void target_function(int x, char y) {
}
@@ -38,4 +38,32 @@ int main() {
** ...
*/
+/* ARM32 specific: Should have UDF instruction with proper encoding
+ UDF format: 0x8000 | ((type_reg & 31) << 5) | (addr_reg & 31)
+
+ Since ARM32 spills and restores r3 before the trap, the type_reg
+ field uses 0x1F (31) to indicate "register was spilled" rather than
+ pointing to a live register. The addr_reg field contains the actual
+ target register number.
+
+ For this test case using r3, we expect:
+ UDF = 0x8000 | (31 << 5) | 3 = 33763
+ */
+
+/*
+** main: { target arm*-*-* }
+** ...
+** ldr ip, \[r[0-9]+, #-4\]
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eor ip, ip, #[0-9]+
+** eors ip, ip, #[0-9]+
+** beq .Lkcfi_call[0-9]+
+** .Lkcfi_trap[0-9]+:
+** udf #33763
+** .Lkcfi_call[0-9]+:
+** blx r3
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
index e92873e51321..e02a320f2f92 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
@@ -19,9 +19,10 @@ int main() {
/* Should have exactly 2 trap labels in code. */
/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ud2} 2 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*brk} 2 { target aarch64*-*-* } } } */
+/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*udf} 2 { target arm*-*-* } } } */
/* x86_64 should exactly 2 .kcfi_traps sections. */
/* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* } } } */
-/* AArch64 should NOT have .kcfi_traps section. */
-/* { dg-final { scan-assembler-not {\.section\t+\.kcfi_traps} { target aarch64*-*-* } } } */
+/* AArch64 and ARM 32-bit should NOT have .kcfi_traps section. */
+/* { dg-final { scan-assembler-not {\.section\t+\.kcfi_traps} { target aarch64*-*-* arm*-*-* } } } */
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v8 7/7] riscv: Add RISC-V Kernel Control Flow Integrity implementation
2025-11-20 22:21 [PATCH v8 0/7] Introduce Kernel Control Flow Integrity ABI [PR107048] Kees Cook
` (5 preceding siblings ...)
2025-11-20 22:21 ` [PATCH v8 6/7] arm: Add ARM 32-bit " Kees Cook
@ 2025-11-20 22:21 ` Kees Cook
6 siblings, 0 replies; 14+ messages in thread
From: Kees Cook @ 2025-11-20 22:21 UTC (permalink / raw)
To: Qing Zhao
Cc: Kees Cook, Uros Bizjak, Joseph Myers, Richard Biener, Jeff Law,
Andrew Pinski, Jakub Jelinek, Martin Uecker, Peter Zijlstra,
Ard Biesheuvel, Jan Hubicka, Richard Earnshaw, Richard Sandiford,
Marcus Shawcroft, Kyrylo Tkachov, Kito Cheng, Palmer Dabbelt,
Andrew Waterman, Jim Wilson, Dan Li, Sami Tolvanen,
Ramon de C Valle, Joao Moreira, Nathan Chancellor, Bill Wendling,
Osterlund, Sebastian, Constable, Scott D, gcc-patches,
linux-hardening
Implement RISC-V-specific KCFI backend. Nothing is conceptually rv64
specific, but using an alternative set of instructions for rv32 would be
needed, and at present the only user of KCFI on riscv is the rv64 build
of the Linux kernel.
- Scratch register allocation using t1/t2 (x6/x7) following RISC-V
procedure call standard for temporary registers (already
caller-saved), and t3 (x8) when either t1 or t2 is already the call
target register.
- Incompatible with -ffixed-t1, -ffixed-t2, or -ffixed-t3.
- Integration with .kcfi_traps section for debugger/runtime metadata
(like x86_64).
Assembly Code Pattern for RISC-V:
lw t1, -4(target_reg) ; Load actual type ID from preamble
lui t2, %hi(expected_type) ; Load expected type (upper 20 bits)
addiw t2, t2, %lo(expected_type) ; Add lower 12 bits (sign-extended)
beq t1, t2, .Lkcfi_call ; Branch if types match
.Lkcfi_trap: ebreak ; Environment break trap on mismatch
.Lkcfi_call: jalr/jr target_reg ; Execute validated indirect transfer
Build and run tested with Linux kernel ARCH=riscv.
gcc/ChangeLog:
config/riscv/riscv-protos.h: Declare KCFI helpers.
config/riscv/riscv.cc (riscv_maybe_wrap_call_with_kcfi): New
function, to wrap calls.
(riscv_maybe_wrap_call_value_with_kcfi): New function, to
wrap calls with return values.
(riscv_output_kcfi_insn): New function to emit KCFI assembly.
config/riscv/riscv.md: Add KCFI RTL patterns and hook expansion.
doc/invoke.texi: Document riscv nuances.
gcc/testsuite/ChangeLog:
* gcc.dg/kcfi/kcfi-adjacency.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-basics.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-call-sharing.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-complex-addressing.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-move-preservation.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-no-sanitize-inline.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-no-sanitize.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-offset-validation.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-patchable-entry-only.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-patchable-large.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-patchable-medium.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-patchable-prefix-only.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-tail-calls.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-trap-section.c: Add riscv patterns.
* gcc.dg/kcfi/kcfi-riscv-fixed-t1.c: New test.
* gcc.dg/kcfi/kcfi-riscv-fixed-t2.c: New test.
* gcc.dg/kcfi/kcfi-riscv-fixed-t3.c: New test.
Signed-off-by: Kees Cook <kees@kernel.org>
---
gcc/config/riscv/riscv-protos.h | 3 +
gcc/config/riscv/riscv.md | 76 +++++++-
gcc/config/riscv/riscv.cc | 170 ++++++++++++++++++
gcc/doc/invoke.texi | 17 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c | 17 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c | 21 ++-
gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c | 4 +
.../gcc.dg/kcfi/kcfi-complex-addressing.c | 19 ++
.../gcc.dg/kcfi/kcfi-move-preservation.c | 20 +++
.../gcc.dg/kcfi/kcfi-no-sanitize-inline.c | 5 +
gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c | 1 +
.../gcc.dg/kcfi/kcfi-offset-validation.c | 3 +
.../gcc.dg/kcfi/kcfi-patchable-entry-only.c | 9 +-
.../gcc.dg/kcfi/kcfi-patchable-large.c | 10 +-
.../gcc.dg/kcfi/kcfi-patchable-medium.c | 9 +-
.../gcc.dg/kcfi/kcfi-patchable-prefix-only.c | 9 +-
.../gcc.dg/kcfi/kcfi-riscv-fixed-t1.c | 17 ++
.../gcc.dg/kcfi/kcfi-riscv-fixed-t2.c | 17 ++
.../gcc.dg/kcfi/kcfi-riscv-fixed-t3.c | 17 ++
gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c | 23 +++
gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c | 5 +-
21 files changed, 458 insertions(+), 14 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t1.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t2.c
create mode 100644 gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t3.c
diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h
index a372779cf9f5..c62801115c6e 100644
--- a/gcc/config/riscv/riscv-protos.h
+++ b/gcc/config/riscv/riscv-protos.h
@@ -126,6 +126,9 @@ extern bool riscv_split_64bit_move_p (rtx, rtx);
extern void riscv_split_doubleword_move (rtx, rtx);
extern const char *riscv_output_move (rtx, rtx);
extern const char *riscv_output_return ();
+extern rtx riscv_maybe_wrap_call_with_kcfi (rtx, rtx);
+extern rtx riscv_maybe_wrap_call_value_with_kcfi (rtx, rtx);
+extern const char *riscv_output_kcfi_insn (rtx_insn *, rtx *);
extern void riscv_declare_function_name (FILE *, const char *, tree);
extern void riscv_declare_function_size (FILE *, const char *, tree);
extern void riscv_asm_output_alias (FILE *, const tree, const tree);
diff --git a/gcc/config/riscv/riscv.md b/gcc/config/riscv/riscv.md
index 59233babfc32..b510ebd4bb3e 100644
--- a/gcc/config/riscv/riscv.md
+++ b/gcc/config/riscv/riscv.md
@@ -4008,10 +4008,25 @@
""
{
rtx target = riscv_legitimize_call_address (XEXP (operands[0], 0));
- emit_call_insn (gen_sibcall_internal (target, operands[1], operands[2]));
+ rtx pat = gen_sibcall_internal (target, operands[1], operands[2]);
+ pat = riscv_maybe_wrap_call_with_kcfi (pat, target);
+ emit_call_insn (pat);
DONE;
})
+;; KCFI sibling call
+(define_insn "*kcfi_sibcall_insn"
+ [(kcfi (call (mem:SI (match_operand:DI 0 "call_insn_operand" "l"))
+ (match_operand 1 ""))
+ (match_operand 3 "const_int_operand"))
+ (use (unspec:SI [(match_operand 2 "const_int_operand")] UNSPEC_CALLEE_CC))]
+ "SIBLING_CALL_P (insn)"
+{
+ return riscv_output_kcfi_insn (insn, operands);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "24")])
+
(define_insn "sibcall_internal"
[(call (mem:SI (match_operand 0 "call_insn_operand" "j,S,U"))
(match_operand 1 "" ""))
@@ -4035,11 +4050,27 @@
""
{
rtx target = riscv_legitimize_call_address (XEXP (operands[1], 0));
- emit_call_insn (gen_sibcall_value_internal (operands[0], target, operands[2],
- operands[3]));
+ rtx pat = gen_sibcall_value_internal (operands[0], target, operands[2],
+ operands[3]);
+ pat = riscv_maybe_wrap_call_value_with_kcfi (pat, target);
+ emit_call_insn (pat);
DONE;
})
+;; KCFI sibling call with return value
+(define_insn "*kcfi_sibcall_value_insn"
+ [(set (match_operand 0 "")
+ (kcfi (call (mem:SI (match_operand:DI 1 "call_insn_operand" "l"))
+ (match_operand 2 ""))
+ (match_operand 4 "const_int_operand")))
+ (use (unspec:SI [(match_operand 3 "const_int_operand")] UNSPEC_CALLEE_CC))]
+ "SIBLING_CALL_P (insn)"
+{
+ return riscv_output_kcfi_insn (insn, &operands[1]);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "24")])
+
(define_insn "sibcall_value_internal"
[(set (match_operand 0 "" "")
(call (mem:SI (match_operand 1 "call_insn_operand" "j,S,U"))
@@ -4063,10 +4094,26 @@
""
{
rtx target = riscv_legitimize_call_address (XEXP (operands[0], 0));
- emit_call_insn (gen_call_internal (target, operands[1], operands[2]));
+ rtx pat = gen_call_internal (target, operands[1], operands[2]);
+ pat = riscv_maybe_wrap_call_with_kcfi (pat, target);
+ emit_call_insn (pat);
DONE;
})
+;; KCFI indirect call
+(define_insn "*kcfi_call_internal"
+ [(kcfi (call (mem:SI (match_operand:DI 0 "call_insn_operand" "l"))
+ (match_operand 1 "" ""))
+ (match_operand 3 "const_int_operand"))
+ (use (unspec:SI [(match_operand 2 "const_int_operand")] UNSPEC_CALLEE_CC))
+ (clobber (reg:SI RETURN_ADDR_REGNUM))]
+ "!SIBLING_CALL_P (insn)"
+{
+ return riscv_output_kcfi_insn (insn, operands);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "24")])
+
(define_insn "call_internal"
[(call (mem:SI (match_operand 0 "call_insn_operand" "l,S,U"))
(match_operand 1 "" ""))
@@ -4091,11 +4138,28 @@
""
{
rtx target = riscv_legitimize_call_address (XEXP (operands[1], 0));
- emit_call_insn (gen_call_value_internal (operands[0], target, operands[2],
- operands[3]));
+ rtx pat = gen_call_value_internal (operands[0], target, operands[2],
+ operands[3]);
+ pat = riscv_maybe_wrap_call_value_with_kcfi (pat, target);
+ emit_call_insn (pat);
DONE;
})
+;; KCFI call with return value
+(define_insn "*kcfi_call_value_insn"
+ [(set (match_operand 0 "" "")
+ (kcfi (call (mem:SI (match_operand:DI 1 "call_insn_operand" "l"))
+ (match_operand 2 "" ""))
+ (match_operand 4 "const_int_operand")))
+ (use (unspec:SI [(match_operand 3 "const_int_operand")] UNSPEC_CALLEE_CC))
+ (clobber (reg:SI RETURN_ADDR_REGNUM))]
+ "!SIBLING_CALL_P (insn)"
+{
+ return riscv_output_kcfi_insn (insn, &operands[1]);
+}
+ [(set_attr "type" "call")
+ (set_attr "length" "24")])
+
(define_insn "call_value_internal"
[(set (match_operand 0 "" "")
(call (mem:SI (match_operand 1 "call_insn_operand" "l,S,U"))
diff --git a/gcc/config/riscv/riscv.cc b/gcc/config/riscv/riscv.cc
index 2d14b3c92f57..89d6d0777f12 100644
--- a/gcc/config/riscv/riscv.cc
+++ b/gcc/config/riscv/riscv.cc
@@ -81,6 +81,7 @@ along with GCC; see the file COPYING3. If not see
#include "cgraph.h"
#include "langhooks.h"
#include "gimplify.h"
+#include "kcfi.h"
/* This file should be included last. */
#include "target-def.h"
@@ -12070,6 +12071,160 @@ riscv_convert_vector_chunks (struct gcc_options *opts)
return 1;
}
+/* Apply KCFI wrapping to call pattern if needed. PAT is the RTL call
+ pattern to potentially wrap with KCFI instrumentation. ADDR is the
+ call target address RTL expression. Returns the possibly modified
+ call pattern with KCFI wrapper applied for indirect calls. */
+
+rtx
+riscv_maybe_wrap_call_with_kcfi (rtx pat, rtx addr)
+{
+ /* Only indirect calls need KCFI instrumentation. */
+ bool is_direct_call = SYMBOL_REF_P (addr);
+ if (!is_direct_call)
+ {
+ rtx kcfi_type_rtx = kcfi_get_type_id_for_expanding_gimple_call ();
+ if (kcfi_type_rtx)
+ {
+ /* Extract the CALL from the PARALLEL and wrap it with KCFI. */
+ rtx call_rtx = XVECEXP (pat, 0, 0);
+ rtx kcfi_call = gen_rtx_KCFI (VOIDmode, call_rtx, kcfi_type_rtx);
+
+ /* Replace the CALL in the PARALLEL with the KCFI-wrapped call. */
+ XVECEXP (pat, 0, 0) = kcfi_call;
+ }
+ }
+ return pat;
+}
+
+/* Apply KCFI wrapping to call_value pattern if needed. PAT is the RTL
+ call_value pattern to potentially wrap with KCFI instrumentation. ADDR
+ is the call target address RTL expression. Returns the possibly modified
+ call pattern with KCFI wrapper applied for indirect calls. */
+
+rtx
+riscv_maybe_wrap_call_value_with_kcfi (rtx pat, rtx addr)
+{
+ /* Only indirect calls need KCFI instrumentation. */
+ bool is_direct_call = SYMBOL_REF_P (addr);
+ if (!is_direct_call)
+ {
+ rtx kcfi_type_rtx = kcfi_get_type_id_for_expanding_gimple_call ();
+ if (kcfi_type_rtx)
+ {
+ /* Extract the SET from the PARALLEL and wrap its CALL with KCFI. */
+ rtx set_rtx = XVECEXP (pat, 0, 0);
+ rtx call_rtx = SET_SRC (set_rtx);
+ rtx kcfi_call = gen_rtx_KCFI (VOIDmode, call_rtx, kcfi_type_rtx);
+
+ /* Replace the CALL in the SET with the KCFI-wrapped call. */
+ SET_SRC (set_rtx) = kcfi_call;
+ }
+ }
+ return pat;
+}
+
+/* Output the assembly for a KCFI checked call instruction. INSN is the
+ RTL instruction being processed. OPERANDS is the array of RTL operands
+ where operands[0] is the call target register, operands[3] is the KCFI
+ type ID constant. Returns an empty string as all output is handled by
+ direct assembly generation. */
+
+const char *
+riscv_output_kcfi_insn (rtx_insn *insn, rtx *operands)
+{
+ /* Target register. */
+ rtx target_reg = operands[0];
+ gcc_assert (REG_P (target_reg));
+
+ /* Get KCFI type ID. */
+ uint32_t expected_type = (uint32_t) INTVAL (operands[3]);
+
+ /* Calculate typeid offset from call target. */
+ HOST_WIDE_INT offset = -kcfi_get_typeid_offset ();
+
+ /* Choose scratch registers that don't conflict with target. */
+ unsigned temp1_regnum = T1_REGNUM;
+ unsigned temp2_regnum = T2_REGNUM;
+
+ if (REGNO (target_reg) == T1_REGNUM)
+ temp1_regnum = T3_REGNUM;
+ else if (REGNO (target_reg) == T2_REGNUM)
+ temp2_regnum = T3_REGNUM;
+
+ /* Get unique label number for this KCFI check. */
+ int labelno = kcfi_next_labelno ();
+
+ /* Generate custom label names. */
+ char trap_name[32];
+ char call_name[32];
+ ASM_GENERATE_INTERNAL_LABEL (trap_name, "Lkcfi_trap", labelno);
+ ASM_GENERATE_INTERNAL_LABEL (call_name, "Lkcfi_call", labelno);
+
+ /* Split expected_type for RISC-V immediate encoding.
+ If bit 11 is set, increment upper 20 bits to compensate for sign
+ extension. */
+ int32_t lo12 = ((int32_t)(expected_type << 20)) >> 20;
+ uint32_t hi20 = ((expected_type >> 12)
+ + ((expected_type & 0x800) ? 1 : 0)) & 0xFFFFF;
+
+ rtx temp_operands[3];
+
+ /* Load actual type from memory at offset. */
+ temp_operands[0] = gen_rtx_REG (SImode, temp1_regnum);
+ temp_operands[1] = gen_rtx_MEM (SImode,
+ gen_rtx_PLUS (DImode, target_reg,
+ GEN_INT (offset)));
+ output_asm_insn ("lw\t%0, %1", temp_operands);
+
+ /* Load expected type using lui + addiw for proper sign extension. */
+ temp_operands[0] = gen_rtx_REG (SImode, temp2_regnum);
+ temp_operands[1] = GEN_INT (hi20);
+ output_asm_insn ("lui\t%0, %1", temp_operands);
+
+ temp_operands[0] = gen_rtx_REG (SImode, temp2_regnum);
+ temp_operands[1] = gen_rtx_REG (SImode, temp2_regnum);
+ temp_operands[2] = GEN_INT (lo12);
+ output_asm_insn ("addiw\t%0, %1, %2", temp_operands);
+
+ /* Output conditional branch to call label. */
+ fprintf (asm_out_file, "\tbeq\t%s, %s, ",
+ reg_names[temp1_regnum], reg_names[temp2_regnum]);
+ assemble_name (asm_out_file, call_name);
+ fputc ('\n', asm_out_file);
+
+ /* Output trap label and ebreak instruction. */
+ ASM_OUTPUT_LABEL (asm_out_file, trap_name);
+ output_asm_insn ("ebreak", operands);
+
+ /* Use common helper for trap section entry. */
+ rtx trap_label_sym = gen_rtx_SYMBOL_REF (Pmode, trap_name);
+ kcfi_emit_traps_section (asm_out_file, trap_label_sym, labelno);
+
+ /* Output pass/call label. */
+ ASM_OUTPUT_LABEL (asm_out_file, call_name);
+
+ /* Execute the indirect call. */
+ if (SIBLING_CALL_P (insn))
+ {
+ /* Tail call uses x0 (zero register) to avoid saving return address. */
+ temp_operands[0] = gen_rtx_REG (DImode, 0);
+ temp_operands[1] = target_reg;
+ temp_operands[2] = const0_rtx;
+ output_asm_insn ("jalr\t%0, %1, %2", temp_operands);
+ }
+ else
+ {
+ /* Regular call uses x1 (return address register). */
+ temp_operands[0] = gen_rtx_REG (DImode, RETURN_ADDR_REGNUM);
+ temp_operands[1] = target_reg;
+ temp_operands[2] = const0_rtx;
+ output_asm_insn ("jalr\t%0, %1, %2", temp_operands);
+ }
+
+ return "";
+}
+
/* 'Unpack' up the internal tuning structs and update the options
in OPTS. The caller must have set up selected_tune and selected_arch
as all the other target-specific codegen decisions are
@@ -12177,6 +12332,18 @@ riscv_override_options_internal (struct gcc_options *opts)
opts->x_flag_cf_protection
= (cf_protection_level) (opts->x_flag_cf_protection | CF_SET);
}
+
+ /* KCFI is only supported in 64-bit mode. */
+ if ((opts->x_flag_sanitize & SANITIZE_KCFI) && !TARGET_64BIT)
+ sorry ("%<-fsanitize=kcfi%> is not supported for 32-bit RISC-V");
+
+ /* KCFI requires T1, T2, and T3 registers for type checking. */
+ if ((opts->x_flag_sanitize & SANITIZE_KCFI)
+ && (fixed_regs[T1_REGNUM] || fixed_regs[T2_REGNUM]
+ || fixed_regs[T3_REGNUM]))
+ sorry ("%<-fsanitize=kcfi%> is not compatible with %<-ffixed-t1%>, "
+ "%<-ffixed-t2%>, or %<-ffixed-t3%> as KCFI requires these registers "
+ "for type checking");
}
/* Implement TARGET_OPTION_OVERRIDE. */
@@ -16647,6 +16814,9 @@ riscv_prefetch_offset_address_p (rtx x, machine_mode mode)
#define TARGET_GET_FUNCTION_VERSIONS_DISPATCHER \
riscv_get_function_versions_dispatcher
+#undef TARGET_KCFI_SUPPORTED
+#define TARGET_KCFI_SUPPORTED hook_bool_void_true
+
#undef TARGET_DOCUMENTATION_NAME
#define TARGET_DOCUMENTATION_NAME "RISC-V"
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 2f8eef03f953..9a4e7257d1eb 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -18758,6 +18758,23 @@ allowing the kernel to identify both the KCFI violation and the involved
registers for detailed diagnostics (eliminating the need for a separate
@code{.kcfi_traps} section as used on x86_64).
+On 64-bit RISC-V, KCFI type identifiers are emitted as a @code{.word ID}
+directive (a 32-bit constant) before the function entry, similar to AArch64.
+RISC-V's natural instruction alignment eliminates the need for
+additional alignment NOPs. When used with @option{-fpatchable-function-entry},
+the type identifier is placed before any prefix NOPs. The runtime check
+loads the actual type using @code{lw t1, OFFSET(target_reg)}, where the
+offset accounts for any prefix NOPs, constructs the expected type using
+@code{lui} and @code{addiw} instructions into @code{t2}, and compares them
+with @code{beq}. @code{t3} is used as an alternative when @code{t1} or
+@code{t2} is the target call register. Because of the use of these
+register, they cannot be fixed registers, so KCFI cannot be used with any
+of @code{-ffixed-t1}, @code{-ffixed-t2}, nor @code{-ffixed-t3}. Type
+mismatches trigger an @code{ebreak} instruction. Like x86_64, RISC-V
+uses a @code{.kcfi_traps} section to map trap locations to their
+corresponding function entry points for debugging (RISC-V lacks
+ESR-style trap encoding like used on AArch64).
+
KCFI is intended primarily for kernel code and may not be suitable
for user-space applications that rely on techniques incompatible
with strict type checking of indirect calls.
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
index 00c14c0375cd..899892b3142d 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-adjacency.c
@@ -94,4 +94,21 @@ __attribute__((noinline)) void test_conditional_call(int flag) {
** ...
*/
+/*
+** test_complex_args: { target riscv*-*-* }
+** ...
+** lw t1, -4\((a[0-9]+)\)
+** lui t2, [0-9]+
+** addiw t2, t2, -?[0-9]+
+** beq t1, t2, .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ebreak
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .4byte .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** jalr zero, \1, 0
+** ...
+*/
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
index 4c9a1e7aa552..6595c940c2d9 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-basics.c
@@ -59,8 +59,8 @@ int main() {
/* x86_64: Verify type ID in preamble (after NOPs, before function label) */
/* { dg-final { scan-assembler {__cfi_regular_function:\n\t+nop\n.*\n\t+movl\t+\$0x[0-9a-f]+, %eax} { target x86_64-*-* } } } */
-/* AArch64, ARM32: Verify type ID word in preamble. */
-/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* arm*-*-* } } } */
+/* AArch64, ARM32, RISC-V: Verify type ID word in preamble. */
+/* { dg-final { scan-assembler {__cfi_regular_function:\n\t\.word\t0x[0-9a-f]+} { target aarch64*-*-* arm*-*-* riscv*-*-* } } } */
/*
** static_caller: { target x86_64-*-* }
@@ -110,6 +110,23 @@ int main() {
** ...
*/
+/*
+** static_caller: { target riscv*-*-* }
+** ...
+** lw t1, -4\((a[0-9]+)\)
+** lui t2, [0-9]+
+** addiw t2, t2, -?[0-9]+
+** beq t1, t2, .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ebreak
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .4byte .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** jalr ra, \1, 0
+** ...
+*/
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
/* Extern functions should NOT get KCFI preambles. */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
index 625cfa350a53..92af591f2714 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-call-sharing.c
@@ -67,6 +67,7 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
/* x86: { dg-final { scan-assembler-times {movl\s+\$-?[0-9]+,\s+%r10d} 2 { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-times {mov\s+w17, #[0-9]+} 2 { target aarch64*-*-* } } } */
/* ARM 32-bit: { dg-final { scan-assembler-times {ldr\s+ip, \[(?:r[0-9]+|lr), #-4\]} 3 { target arm*-*-* } } } */
+/* RISC-V: { dg-final { scan-assembler-times {lui\tt2, [0-9]+} 2 { target riscv*-*-* } } } */
/* Verify the checks use DIFFERENT type IDs (not shared, except arm: see above).
We should NOT see the same type ID used twice - that would indicate
@@ -74,13 +75,16 @@ int test_kcfi_check_sharing(struct kobject *kobj, const struct attribute_group *
/* x86: { dg-final { scan-assembler-not {movl\s+\$(-?[0-9]+),\s+%r10d.*movl\s+\$\1,\s+%r10d} { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-not {mov\s+w17, #([0-9]+).*mov\s+w17, #\1} { target aarch64*-*-* } } } */
/* ARM 32-bit: { dg-final { scan-assembler-not {eor\s+ip, ip, #([0-9]+)\n\teor\s+r3, r3, #([0-9]+)\n\teor\s+r3, r3, #([0-9]+)\n\teors\s+r3, r3, #([0-9]+).*eor\s+r3, r3, #\1\n\teor\s+r3, r3, #[0-9]+\n\teor\s+r3, r3, #[0-9]+\n\teors\s+r3, r3, #[0-9]+.*eor\s+r3, r3, #\1\n\teor\s+r3, r3, #[0-9]+\n\teor\s+r3, r3, #[0-9]+\n\teors\s+r3, r3, #[0-9]+} { target arm*-*-* } } } */
+/* RISC-V: { dg-final { scan-assembler-not {lui\s+t2, ([0-9]+)\s.*lui\s+t2, \1\s} { target riscv*-*-* } } } */
/* Verify expected number of traps. */
/* x86: { dg-final { scan-assembler-times {ud2} 2 { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
/* ARM 32-bit: { dg-final { scan-assembler-times {udf\s+#[0-9]+} 3 { target arm*-*-* } } } */
+/* RISC-V: { dg-final { scan-assembler-times {ebreak} 2 { target riscv*-*-* } } } */
/* Verify 2 separate call sites (except arm). */
/* x86: { dg-final { scan-assembler-times {jmp\s+\*%[a-z0-9]+} 2 { target i?86-*-* x86_64-*-* } } } */
/* AArch64: { dg-final { scan-assembler-times {br\tx[0-9]+} 2 { target aarch64*-*-* } } } */
/* ARM 32-bit: { dg-final { scan-assembler-times {bx\s+(?:r[0-9]+|lr)} 3 { target arm*-*-* } } } */
+/* RISC-V: { dg-final { scan-assembler-times {jalr\t[a-z0-9]+} 2 { target riscv*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
index 92d9dbb7906b..b55e9dccd50a 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-complex-addressing.c
@@ -181,4 +181,23 @@ int main() {
** ...
*/
+/* Standard KCFI handling. */
+/*
+** test_struct_members: { target riscv*-*-* }
+** ...
+** lw t1, -4\((a[0-9]+)\)
+** lui t2, [0-9]+
+** addiw t2, t2, -?[0-9]+
+** beq t1, t2, .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ebreak
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .4byte .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** jalr ra, \1, 0
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
index c259620a3ed8..600c04d5eba3 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-move-preservation.c
@@ -92,6 +92,26 @@ int main(void)
** ...
*/
+/*
+** indirect_call: { target riscv*-*-* }
+** ...
+** mv (a[0-9]+),a0
+** addi a0,a4,%lo\(called_count\)
+** lw t1, -4\(\1\)
+** lui t2, [0-9]+
+** addiw t2, t2, -?[0-9]+
+** beq t1, t2, .Lkcfi_call([0-9]+)
+** .Lkcfi_trap([0-9]+):
+** ebreak
+** .section .kcfi_traps,"ao",@progbits,.text
+** .Lkcfi_entry([0-9]+):
+** .4byte .Lkcfi_trap\3-.Lkcfi_entry\4
+** .text
+** .Lkcfi_call\2:
+** jalr zero, \1, 0
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.L.*|\.section|\.text} } } */
/* AArch64, ARM32 should NOT have trap section (use immediate instructions instead). */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
index f8103466816a..36ccf24a41cc 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize-inline.c
@@ -77,6 +77,7 @@ int main(void)
/* { dg-final { scan-assembler-times {ud2} 2 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times {brk\s+#[0-9]+} 2 { target aarch64*-*-* } } } */
/* { dg-final { scan-assembler-times {udf\s+#[0-9]+} 2 { target arm*-*-* } } } */
+/* { dg-final { scan-assembler-times {ebreak} 2 { target riscv*-*-* } } } */
/* Positive controls: these should have KCFI checks. */
/* { dg-final { scan-assembler {normal_function:.*ud2.*\.size\s+normal_function} { target x86_64-*-* } } } */
@@ -85,6 +86,8 @@ int main(void)
/* { dg-final { scan-assembler {wrap_normal_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_normal_inline} { target aarch64*-*-* } } } */
/* { dg-final { scan-assembler {normal_function:.*udf\t#[0-9]+.*\.size\s+normal_function} { target arm*-*-* } } } */
/* { dg-final { scan-assembler {wrap_normal_inline:.*udf\t#[0-9]+.*\.size\s+wrap_normal_inline} { target arm*-*-* } } } */
+/* { dg-final { scan-assembler {normal_function:.*ebreak.*\.size\s+normal_function} { target riscv*-*-* } } } */
+/* { dg-final { scan-assembler {wrap_normal_inline:.*ebreak.*\.size\s+wrap_normal_inline} { target riscv*-*-* } } } */
/* Negative controls: these should NOT have KCFI checks. */
/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ud2.*\.size\s+sensitive_non_inline_function} { target x86_64-*-* } } } */
@@ -93,3 +96,5 @@ int main(void)
/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*brk\s+#[0-9]+.*\.size\s+wrap_sensitive_inline} { target aarch64*-*-* } } } */
/* { dg-final { scan-assembler-not {sensitive_non_inline_function:[^\n]*udf\t#[0-9]+[^\n]*\.size\tsensitive_non_inline_function} { target arm*-*-* } } } */
/* { dg-final { scan-assembler-not {wrap_sensitive_inline:[^\n]*udf\t#[0-9]+[^\n]*\.size\twrap_sensitive_inline} { target arm*-*-* } } } */
+/* { dg-final { scan-assembler-not {sensitive_non_inline_function:.*ebreak.*\.size\s+sensitive_non_inline_function} { target riscv*-*-* } } } */
+/* { dg-final { scan-assembler-not {wrap_sensitive_inline:.*ebreak.*\.size\s+wrap_sensitive_inline} { target riscv*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
index 95bb46304493..b315e6f65e00 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-no-sanitize.c
@@ -36,3 +36,4 @@ int main() {
/* { dg-final { scan-assembler-times {addl\t-4\(%r[ad]x\), %r1[01]d} 1 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 1 { target aarch64-*-* } } } */
/* { dg-final { scan-assembler-times {ldr\tip, \[r[0-9]+, #-4\]} 1 { target arm*-*-* } } } */
+/* { dg-final { scan-assembler-times {lw\tt1, -[0-9]+\(} 1 { target riscv*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
index 88f0ae64091b..b0dd7b6fc45d 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-offset-validation.c
@@ -33,3 +33,6 @@ int main() {
/* ARM 32-bit: All call sites should use -4 offset with EOR sequence. */
/* { dg-final { scan-assembler {ldr\tip, \[r[0-9]+, #-4\]} { target arm*-*-* } } } */
+
+/* RISC-V: All call sites should use -4 offset. */
+/* { dg-final { scan-assembler {lw\tt1, -4\(} { target riscv*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
index 612140a0f509..db2ec8a5b64f 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-entry-only.c
@@ -29,7 +29,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* riscv*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -54,4 +54,11 @@ int main() {
** ...
*/
+/*
+** main: { target riscv*-*-* }
+** ...
+** lw t1, -4\(a[0-9]+\)
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
index da3d8dc41f60..0c026e6d8615 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-large.c
@@ -18,7 +18,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* riscv*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -43,4 +43,12 @@ int main() {
** ...
*/
+
+/*
+** main: { target riscv*-*-* }
+** ...
+** lw t1, -48\(a[0-9]+\)
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
index d61274e70157..e60c8546ef5c 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-medium.c
@@ -25,7 +25,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* riscv*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -50,4 +50,11 @@ int main() {
** ...
*/
+/*
+** main: { target riscv*-*-* }
+** ...
+** lw t1, -20\(a[0-9]+\)
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
index 93df4d7ea5b6..e90979a9b433 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-patchable-prefix-only.c
@@ -26,7 +26,7 @@ int main() {
*/
/*
-** __cfi_test_function: { target aarch64*-*-* arm*-*-* }
+** __cfi_test_function: { target aarch64*-*-* arm*-*-* riscv*-*-* }
** .word 0x[0-9a-f]+
*/
@@ -51,4 +51,11 @@ int main() {
** ...
*/
+/*
+** main: { target riscv*-*-* }
+** ...
+** lw t1, -16\(a[0-9]+\)
+** ...
+*/
+
/* { dg-final { check-function-bodies "**" "" "" { target *-*-* } {\.word} } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t1.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t1.c
new file mode 100644
index 000000000000..74fe44420688
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t1.c
@@ -0,0 +1,17 @@
+/* Test that KCFI is incompatible with -ffixed-t1 on RISC-V. */
+/* { dg-do compile { target riscv*-*-* } } */
+/* { dg-additional-options "-ffixed-t1" } */
+
+/* { dg-message "sorry, unimplemented: '-fsanitize=kcfi' is not compatible with '-ffixed-t1', '-ffixed-t2', or '-ffixed-t3' as KCFI requires these registers for type checking" "" { target *-*-* } 0 } */
+
+void test_function(void)
+{
+ /* Empty function. */
+}
+
+int main(void)
+{
+ void (*ptr)(void) = test_function;
+ ptr(); /* This would need KCFI instrumentation. */
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t2.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t2.c
new file mode 100644
index 000000000000..25736adf7ba7
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t2.c
@@ -0,0 +1,17 @@
+/* Test that KCFI is incompatible with -ffixed-t2 on RISC-V. */
+/* { dg-do compile { target riscv*-*-* } } */
+/* { dg-additional-options "-ffixed-t2" } */
+
+/* { dg-message "sorry, unimplemented: '-fsanitize=kcfi' is not compatible with '-ffixed-t1', '-ffixed-t2', or '-ffixed-t3' as KCFI requires these registers for type checking" "" { target *-*-* } 0 } */
+
+void test_function(void)
+{
+ /* Empty function. */
+}
+
+int main(void)
+{
+ void (*ptr)(void) = test_function;
+ ptr(); /* This would need KCFI instrumentation. */
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t3.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t3.c
new file mode 100644
index 000000000000..0f557774fe7f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-riscv-fixed-t3.c
@@ -0,0 +1,17 @@
+/* Test that KCFI is incompatible with -ffixed-t3 on RISC-V. */
+/* { dg-do compile { target riscv*-*-* } } */
+/* { dg-additional-options "-ffixed-t3" } */
+
+/* { dg-message "sorry, unimplemented: '-fsanitize=kcfi' is not compatible with '-ffixed-t1', '-ffixed-t2', or '-ffixed-t3' as KCFI requires these registers for type checking" "" { target *-*-* } 0 } */
+
+void test_function(void)
+{
+ /* Empty function. */
+}
+
+int main(void)
+{
+ void (*ptr)(void) = test_function;
+ ptr(); /* This would need KCFI instrumentation. */
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
index ee9d2c6dc741..34d18e6e0a03 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-tail-calls.c
@@ -67,6 +67,8 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
/* Should have exactly 4 trap sections and 4 trap instructions. */
/* { dg-final { scan-assembler-times "\\.kcfi_traps" 4 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times "ud2" 4 { target x86_64-*-* } } } */
+/* { dg-final { scan-assembler-times "\\.kcfi_traps" 4 { target riscv*-*-* } } } */
+/* { dg-final { scan-assembler-times "ebreak" 4 { target riscv*-*-* } } } */
/* Should NOT have unprotected direct jumps to vtable. */
/* { dg-final { scan-assembler-not {jmp\t\*vtable\(%rip\)} { target x86_64-*-* } } } */
@@ -79,6 +81,27 @@ int test_non_tail_indirect_call(func_ptr_t handler, int x) {
/* Should have exactly 1 regular call (non-tail call case). */
/* { dg-final { scan-assembler-times {call\t\*%[a-z0-9]+} 1 { target x86_64-*-* } } } */
+/* RISC-V: Should have exactly 4 KCFI checks for indirect calls
+ (comparison instruction). */
+/* { dg-final { scan-assembler-times {beq\tt1, t2, \.Lkcfi_call[0-9]+} 4 { target riscv*-*-* } } } */
+
+/* RISC-V: Should have exactly 4 KCFI checks for indirect calls as
+ (load type ID + compare). */
+/* { dg-final { scan-assembler-times {lw\tt1, -4\([a-z0-9]+\)} 4 { target riscv*-*-* } } } */
+/* { dg-final { scan-assembler-times {lui\tt2, [0-9]+} 4 { target riscv*-*-* } } } */
+
+/* RISC-V: Should have exactly 3 protected tail calls (jr after
+ KCFI check - no return address save). */
+/* { dg-final { scan-assembler-times {jalr\t(x0|zero), [a-z0-9]+, 0} 3 { target riscv*-*-* } } } */
+
+/* RISC-V: Should have exactly 1 regular call (non-tail call case - saves
+ return address). */
+/* { dg-final { scan-assembler-times {jalr\t(x1|ra), [a-z0-9]+, 0} 1 { target riscv*-*-* } } } */
+
+/* Type ID loading should use lui + addiw pattern for 32-bit constants. */
+/* { dg-final { scan-assembler {lui\tt2, [0-9]+} { target riscv*-*-* } } } */
+/* { dg-final { scan-assembler {addiw\tt2, t2, -?[0-9]+} { target riscv*-*-* } } } */
+
/* Should have exactly 4 KCFI checks for indirect calls (load type ID from
-4 offset + compare). */
/* { dg-final { scan-assembler-times {ldur\tw16, \[x[0-9]+, #-4\]} 4 { target aarch64-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
index e02a320f2f92..5d1159cfb39b 100644
--- a/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
+++ b/gcc/testsuite/gcc.dg/kcfi/kcfi-trap-section.c
@@ -20,9 +20,10 @@ int main() {
/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ud2} 2 { target x86_64-*-* } } } */
/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*brk} 2 { target aarch64*-*-* } } } */
/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*udf} 2 { target arm*-*-* } } } */
+/* { dg-final { scan-assembler-times {\.L[^:]+:\n\s*ebreak} 2 { target riscv*-*-* } } } */
-/* x86_64 should exactly 2 .kcfi_traps sections. */
-/* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* } } } */
+/* x86_64 and RISC-V should exactly 2 .kcfi_traps sections. */
+/* { dg-final { scan-assembler-times {\.section\t\.kcfi_traps,"ao",@progbits,\.text} 2 { target x86_64-*-* riscv*-*-* } } } */
/* AArch64 and ARM 32-bit should NOT have .kcfi_traps section. */
/* { dg-final { scan-assembler-not {\.section\t+\.kcfi_traps} { target aarch64*-*-* arm*-*-* } } } */
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread