* Re: [PATCH v2] kselftest/arm64: Fix build failure with GCC-15
From: Mark Brown @ 2026-04-22 17:50 UTC (permalink / raw)
To: Leo Yan
Cc: Catalin Marinas, Will Deacon, Shuah Khan, Thiago Jung Bauermann,
linux-arm-kernel, linux-kselftest, linux-kernel
In-Reply-To: <20260422-selftests_arm64_gcc15-v2-1-c0134de8838a@arm.com>
[-- Attachment #1: Type: text/plain, Size: 585 bytes --]
On Wed, Apr 22, 2026 at 06:42:54PM +0100, Leo Yan wrote:
> For GCC-15, NT_ARM_GCS is defined in the libc header, causing gcs-util.h
> to skip its fallback definition of struct user_gcs. This leads to the
> compiler error.
> Fix this by including <asm/ptrace.h> to provide the proper definition.
> +++ b/tools/testing/selftests/arm64/gcs/gcs-util.h
> @@ -7,6 +7,7 @@
> #define GCS_UTIL_H
>
> #include <stdbool.h>
> +#include <asm/ptrace.h>
>
> #ifndef __NR_map_shadow_stack
> #define __NR_map_shadow_stack 453
Reviewed-by: Mark Brown <broonie@kernel.org>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [REGRESSION] rseq: refactoring in v6.19 broke everyone on arm64 and tcmalloc everywhere
From: Thomas Gleixner @ 2026-04-22 17:49 UTC (permalink / raw)
To: Mark Rutland, Mathias Stearn
Cc: Mathieu Desnoyers, Catalin Marinas, Will Deacon, Boqun Feng,
Paul E. McKenney, Chris Kennelly, Dmitry Vyukov, regressions,
linux-kernel, linux-arm-kernel, Peter Zijlstra, Ingo Molnar,
Jinjie Ruan, Blake Oler
In-Reply-To: <aejCaG6n9s7ak5TO@J2N7QTR9R3.cambridge.arm.com>
On Wed, Apr 22 2026 at 14:09, Mark Rutland wrote:
> On Wed, Apr 22, 2026 at 11:50:26AM +0200, Mathias Stearn wrote:
>> TL;DR: As of 6.19, rseq no longer provides the documented atomicity
>> guarantees on arm64 by failing to abort the critical section on same-core
>> preemption/resumption. Additionally, it breaks tcmalloc specifically by
>> failing to overwrite the cpu_id_start field at points where it was relied
>> on for correctness.
>
> Thanks for the report, and the test case.
>
> As a holding reply, I'm looking into this now from the arm64 side.
I assume it's the partial conversion to the generic entry code which
screws that up. The problem reproduces with rseq selftests nicely.
The patch below fixes it as it puts ARM64 back to the non-optimized code
for now. Once ARM64 is fully converted it gets all the nice improvements.
Thanks,
tglx
---
diff --git a/include/linux/rseq.h b/include/linux/rseq.h
index 2266f4dc77b6..d55476e2a336 100644
--- a/include/linux/rseq.h
+++ b/include/linux/rseq.h
@@ -30,7 +30,7 @@ void __rseq_signal_deliver(int sig, struct pt_regs *regs);
*/
static inline void rseq_signal_deliver(struct ksignal *ksig, struct pt_regs *regs)
{
- if (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY)) {
+ if (IS_ENABLED(CONFIG_GENERIC_ENTRY)) {
/* '&' is intentional to spare one conditional branch */
if (current->rseq.event.has_rseq & current->rseq.event.user_irq)
__rseq_signal_deliver(ksig->sig, regs);
@@ -50,7 +50,7 @@ static __always_inline void rseq_sched_switch_event(struct task_struct *t)
{
struct rseq_event *ev = &t->rseq.event;
- if (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY)) {
+ if (IS_ENABLED(CONFIG_GENERIC_ENTRY)) {
/*
* Avoid a boat load of conditionals by using simple logic
* to determine whether NOTIFY_RESUME needs to be raised.
diff --git a/include/linux/rseq_entry.h b/include/linux/rseq_entry.h
index a36b472627de..8ccd464a108d 100644
--- a/include/linux/rseq_entry.h
+++ b/include/linux/rseq_entry.h
@@ -80,7 +80,7 @@ bool rseq_debug_validate_ids(struct task_struct *t);
static __always_inline void rseq_note_user_irq_entry(void)
{
- if (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY))
+ if (IS_ENABLED(CONFIG_GENERIC_ENTRY))
current->rseq.event.user_irq = true;
}
@@ -171,8 +171,8 @@ bool rseq_debug_update_user_cs(struct task_struct *t, struct pt_regs *regs,
if (unlikely(usig != t->rseq.sig))
goto die;
- /* rseq_event.user_irq is only valid if CONFIG_GENERIC_IRQ_ENTRY=y */
- if (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY)) {
+ /* rseq_event.user_irq is only valid if CONFIG_GENERIC_ENTRY=y */
+ if (IS_ENABLED(CONFIG_GENERIC_ENTRY)) {
/* If not in interrupt from user context, let it die */
if (unlikely(!t->rseq.event.user_irq))
goto die;
@@ -387,7 +387,7 @@ static rseq_inline bool rseq_update_usr(struct task_struct *t, struct pt_regs *r
* allows to skip the critical section when the entry was not from
* a user space interrupt, unless debug mode is enabled.
*/
- if (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY)) {
+ if (IS_ENABLED(CONFIG_GENERIC_ENTRY)) {
if (!static_branch_unlikely(&rseq_debug_enabled)) {
if (likely(!t->rseq.event.user_irq))
return true;
^ permalink raw reply related
* [PATCH v2] kselftest/arm64: Fix build failure with GCC-15
From: Leo Yan @ 2026-04-22 17:42 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Shuah Khan, Mark Brown,
Thiago Jung Bauermann
Cc: linux-arm-kernel, linux-kselftest, linux-kernel, Leo Yan
Building on Debian sid with GCC 15 fails:
CC libc-gcs
libc-gcs.c: In function 'ptrace_read_write':
libc-gcs.c:142:25: error: storage size of 'child_gcs' isn't known
142 | struct user_gcs child_gcs;
| ^~~~~~~~~
libc-gcs.c:142:25: warning: unused variable 'child_gcs' [-Wunused-variable]
For GCC-15, NT_ARM_GCS is defined in the libc header, causing gcs-util.h
to skip its fallback definition of struct user_gcs. This leads to the
compiler error.
Fix this by including <asm/ptrace.h> to provide the proper definition.
Fixes: a505a52b4e29 ("kselftest/arm64: Add a GCS test program built with the system libc")
Signed-off-by: Leo Yan <leo.yan@arm.com>
---
Changes in v2:
- Refined commit log (Mark Brown).
- Moving header to gcs-util.h (Mark Brown).
- Link to v1: https://lore.kernel.org/r/20260422-selftests_arm64_gcc15-v1-1-0d919ea5ac5f@arm.com
---
tools/testing/selftests/arm64/gcs/gcs-util.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/testing/selftests/arm64/gcs/gcs-util.h b/tools/testing/selftests/arm64/gcs/gcs-util.h
index c99a6b39ac147b4efbc9b5fbadb43daf4da2c85e..f583d4dbd0bfd964e08cf40b3951664c70bb031a 100644
--- a/tools/testing/selftests/arm64/gcs/gcs-util.h
+++ b/tools/testing/selftests/arm64/gcs/gcs-util.h
@@ -7,6 +7,7 @@
#define GCS_UTIL_H
#include <stdbool.h>
+#include <asm/ptrace.h>
#ifndef __NR_map_shadow_stack
#define __NR_map_shadow_stack 453
---
base-commit: 4ee64205ffaa587e8114d84a67ac721399ccb369
change-id: 20260421-selftests_arm64_gcc15-15f6f6d07dd8
Best regards,
--
Leo Yan <leo.yan@arm.com>
^ permalink raw reply related
* Re: [PATCH] kselftest/arm64: Fix build failure with GCC-15
From: Leo Yan @ 2026-04-22 17:41 UTC (permalink / raw)
To: Mark Brown
Cc: Catalin Marinas, Will Deacon, Shuah Khan, Thiago Jung Bauermann,
linux-arm-kernel, linux-kselftest, linux-kernel
In-Reply-To: <8690e67b-3637-45de-8956-e27486470866@sirena.org.uk>
On Wed, Apr 22, 2026 at 06:20:16PM +0100, Mark Brown wrote:
[...]
> > +++ b/tools/testing/selftests/arm64/gcs/libc-gcs.c
> > @@ -16,6 +16,7 @@
> >
> > #include <asm/hwcap.h>
> > #include <asm/mman.h>
> > +#include <asm/ptrace.h>
>
> Shouldn't this be in gcs-util.h where the local definition is?
Will update commit log and this one. Thanks for review!
Leo
^ permalink raw reply
* Re: [RFC PATCH v2 1/4] security: ima: call ima_init() again at late_initcall_sync for defered TPM
From: Mimi Zohar @ 2026-04-22 17:20 UTC (permalink / raw)
To: Yeoreum Yun, linux-security-module, linux-kernel, linux-integrity,
linux-arm-kernel, kvmarm
Cc: paul, jmorris, serge, roberto.sassu, dmitry.kasatkin,
eric.snowberg, jarkko, jgg, sudeep.holla, maz, oupton, joey.gouly,
suzuki.poulose, yuzenghui, catalin.marinas, will, noodles,
sebastianene
In-Reply-To: <20260422162449.1814615-2-yeoreum.yun@arm.com>
On Wed, 2026-04-22 at 17:24 +0100, Yeoreum Yun wrote:
> To generate the boot_aggregate log in the IMA subsystem with TPM PCR values,
> the TPM driver must be built as built-in and
> must be probed before the IMA subsystem is initialized.
>
> However, when the TPM device operates over the FF-A protocol using
> the CRB interface, probing fails and returns -EPROBE_DEFER if
> the tpm_crb_ffa device — an FF-A device that provides the communication
> interface to the tpm_crb driver — has not yet been probed.
>
> To ensure the TPM device operating over the FF-A protocol with
> the CRB interface is probed before IMA initialization,
> the following conditions must be met:
>
> 1. The corresponding ffa_device must be registered,
> which is done via ffa_init().
>
> 2. The tpm_crb_driver must successfully probe this device via
> tpm_crb_ffa_init().
>
> 3. The tpm_crb driver using CRB over FF-A can then
> be probed successfully. (See crb_acpi_add() and
> tpm_crb_ffa_init() for reference.)
>
> Unfortunately, ffa_init(), tpm_crb_ffa_init(), and crb_acpi_driver_init() are
> all registered with device_initcall, which means crb_acpi_driver_init() may
> be invoked before ffa_init() and tpm_crb_ffa_init() are completed.
>
> When this occurs, probing the TPM device is deferred.
> However, the deferred probe can happen after the IMA subsystem
> has already been initialized, since IMA initialization is performed
> during late_initcall, and deferred_probe_initcall() is performed
> at the same level.
>
> To resolve this, call ima_init() again at late_inicall_sync level
> so that let IMA not miss TPM PCR value when generating boot_aggregate
> log though TPM device presents in the system.
>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
A lot of change for just detecting whether ima_init() is being called on
late_initcall or late_initcall_sync(), without any explanation for all the other
changes (e.g. ima_init_core).
Please just limit the change to just calling ima_init() twice.
Mimi
> ---
> include/linux/lsm_hooks.h | 2 +
> security/integrity/ima/ima.h | 4 +-
> security/integrity/ima/ima_init.c | 10 +++-
> security/integrity/ima/ima_main.c | 76 +++++++++++++++++++++++------
> security/integrity/ima/ima_policy.c | 3 ++
> security/lsm_init.c | 13 ++++-
> 6 files changed, 87 insertions(+), 21 deletions(-)
>
> diff --git a/include/linux/lsm_hooks.h b/include/linux/lsm_hooks.h
> index d48bf0ad26f4..88fe105b7f00 100644
> --- a/include/linux/lsm_hooks.h
> +++ b/include/linux/lsm_hooks.h
> @@ -166,6 +166,7 @@ enum lsm_order {
> * @initcall_fs: LSM callback for fs_initcall setup, optional
> * @initcall_device: LSM callback for device_initcall() setup, optional
> * @initcall_late: LSM callback for late_initcall() setup, optional
> + * @initcall_late_sync: LSM callback for late_initcall_sync() setup, optional
> */
> struct lsm_info {
> const struct lsm_id *id;
> @@ -181,6 +182,7 @@ struct lsm_info {
> int (*initcall_fs)(void);
> int (*initcall_device)(void);
> int (*initcall_late)(void);
> + int (*initcall_late_sync)(void);
> };
>
> #define DEFINE_LSM(lsm) \
> diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h
> index 89ebe98ffc5e..75ee7ad184d0 100644
> --- a/security/integrity/ima/ima.h
> +++ b/security/integrity/ima/ima.h
> @@ -62,6 +62,8 @@ extern int ima_hash_algo_idx __ro_after_init;
> extern int ima_extra_slots __ro_after_init;
> extern struct ima_algo_desc *ima_algo_array __ro_after_init;
>
> +extern bool ima_initialised __ro_after_init;
> +
> extern int ima_appraise;
> extern struct tpm_chip *ima_tpm_chip;
> extern const char boot_aggregate_name[];
> @@ -257,7 +259,7 @@ static inline void ima_measure_kexec_event(const char *event_name) {}
> extern bool ima_canonical_fmt;
>
> /* Internal IMA function definitions */
> -int ima_init(void);
> +int ima_init(bool late);
> int ima_fs_init(void);
> int ima_add_template_entry(struct ima_template_entry *entry, int violation,
> const char *op, struct inode *inode,
> diff --git a/security/integrity/ima/ima_init.c b/security/integrity/ima/ima_init.c
> index a2f34f2d8ad7..c28c71090ad2 100644
> --- a/security/integrity/ima/ima_init.c
> +++ b/security/integrity/ima/ima_init.c
> @@ -115,13 +115,19 @@ void __init ima_load_x509(void)
> }
> #endif
>
> -int __init ima_init(void)
> +int __init ima_init(bool late)
> {
> int rc;
>
> ima_tpm_chip = tpm_default_chip();
> - if (!ima_tpm_chip)
> + if (!ima_tpm_chip) {
> + if (!late) {
> + pr_info("Defer initialisation to the late_initcall_sync stage.\n");
> + return -EPROBE_DEFER;
> + }
> +
> pr_info("No TPM chip found, activating TPM-bypass!\n");
> + }
>
> rc = integrity_init_keyring(INTEGRITY_KEYRING_IMA);
> if (rc)
> diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
> index 1d6229b156fb..ac444ee600e2 100644
> --- a/security/integrity/ima/ima_main.c
> +++ b/security/integrity/ima/ima_main.c
> @@ -38,6 +38,7 @@ int ima_appraise;
> #endif
>
> int __ro_after_init ima_hash_algo = HASH_ALGO_SHA1;
> +bool ima_initialised __ro_after_init = false;
> static int hash_setup_done;
> static int ima_disabled __ro_after_init;
>
> @@ -1237,6 +1238,35 @@ static int ima_kernel_module_request(char *kmod_name)
>
> #endif /* CONFIG_INTEGRITY_ASYMMETRIC_KEYS */
>
> +static int __init init_ima_core(bool late)
> +{
> + int err;
> +
> + if (ima_initialised)
> + return 0;
> +
> + err = ima_init(late);
> + if (err == -EPROBE_DEFER)
> + return 0;
> +
> + if (err && strcmp(hash_algo_name[ima_hash_algo],
> + CONFIG_IMA_DEFAULT_HASH) != 0) {
> + pr_info("Allocating %s failed, going to use default hash algorithm %s\n",
> + hash_algo_name[ima_hash_algo], CONFIG_IMA_DEFAULT_HASH);
> + hash_setup_done = 0;
> + hash_setup(CONFIG_IMA_DEFAULT_HASH);
> + err = ima_init(late);
> + }
> +
> + if (!err) {
> + ima_update_policy_flags();
> + ima_initialised = true;
> + } else
> + ima_disabled = 1;
> +
> + return err;
> +}
> +
> static int __init init_ima(void)
> {
> int error;
> @@ -1250,30 +1280,42 @@ static int __init init_ima(void)
> ima_appraise_parse_cmdline();
> ima_init_template_list();
> hash_setup(CONFIG_IMA_DEFAULT_HASH);
> - error = ima_init();
> -
> - if (error && strcmp(hash_algo_name[ima_hash_algo],
> - CONFIG_IMA_DEFAULT_HASH) != 0) {
> - pr_info("Allocating %s failed, going to use default hash algorithm %s\n",
> - hash_algo_name[ima_hash_algo], CONFIG_IMA_DEFAULT_HASH);
> - hash_setup_done = 0;
> - hash_setup(CONFIG_IMA_DEFAULT_HASH);
> - error = ima_init();
> - }
> -
> - if (error)
> - return error;
>
> error = register_blocking_lsm_notifier(&ima_lsm_policy_notifier);
> - if (error)
> + if (error) {
> pr_warn("Couldn't register LSM notifier, error %d\n", error);
> + goto disable_ima;
> + }
>
> - if (!error)
> - ima_update_policy_flags();
> + error = init_ima_core(false);
> + if (error) {
> + unregister_blocking_lsm_notifier(&ima_lsm_policy_notifier);
> + goto disable_ima;
> + }
> +
> + return 0;
>
> +disable_ima:
> + ima_disabled = 1;
> return error;
> }
>
> +static int __init late_init_ima(void)
> +{
> + int err;
> +
> + if (ima_disabled)
> + return 0;
> +
> + err = init_ima_core(true);
> + if (err) {
> + unregister_blocking_lsm_notifier(&ima_lsm_policy_notifier);
> + ima_disabled = 1;
> + }
> +
> + return err;
> +}
> +
> static struct security_hook_list ima_hooks[] __ro_after_init = {
> LSM_HOOK_INIT(bprm_check_security, ima_bprm_check),
> LSM_HOOK_INIT(bprm_creds_for_exec, ima_bprm_creds_for_exec),
> @@ -1321,4 +1363,6 @@ DEFINE_LSM(ima) = {
> .blobs = &ima_blob_sizes,
> /* Start IMA after the TPM is available */
> .initcall_late = init_ima,
> + /* Start IMA late in case of probing TPM is deferred. */
> + .initcall_late_sync = late_init_ima,
> };
> diff --git a/security/integrity/ima/ima_policy.c b/security/integrity/ima/ima_policy.c
> index bf2d7ba4c14a..c3bcc3521c81 100644
> --- a/security/integrity/ima/ima_policy.c
> +++ b/security/integrity/ima/ima_policy.c
> @@ -501,6 +501,9 @@ static void ima_lsm_update_rules(void)
> int ima_lsm_policy_change(struct notifier_block *nb, unsigned long event,
> void *lsm_data)
> {
> + if (!ima_initialised)
> + return NOTIFY_DONE;
> +
> if (event != LSM_POLICY_CHANGE)
> return NOTIFY_DONE;
>
> diff --git a/security/lsm_init.c b/security/lsm_init.c
> index 573e2a7250c4..4e5c59beb82a 100644
> --- a/security/lsm_init.c
> +++ b/security/lsm_init.c
> @@ -547,13 +547,22 @@ device_initcall(security_initcall_device);
> * security_initcall_late - Run the LSM late initcalls
> */
> static int __init security_initcall_late(void)
> +{
> + return lsm_initcall(late);
> +}
> +late_initcall(security_initcall_late);
> +
> +/**
> + * security_initcall_late_sync - Run the LSM late initcalls sync
> + */
> +static int __init security_initcall_late_sync(void)
> {
> int rc;
>
> - rc = lsm_initcall(late);
> + rc = lsm_initcall(late_sync);
> lsm_pr_dbg("all enabled LSMs fully activated\n");
> call_blocking_lsm_notifier(LSM_STARTED_ALL, NULL);
>
> return rc;
> }
> -late_initcall(security_initcall_late);
> +late_initcall_sync(security_initcall_late_sync);
> --
> LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply
* Re: [PATCH] kselftest/arm64: Fix build failure with GCC-15
From: Mark Brown @ 2026-04-22 17:20 UTC (permalink / raw)
To: Leo Yan
Cc: Catalin Marinas, Will Deacon, Shuah Khan, Thiago Jung Bauermann,
linux-arm-kernel, linux-kselftest, linux-kernel
In-Reply-To: <20260422-selftests_arm64_gcc15-v1-1-0d919ea5ac5f@arm.com>
[-- Attachment #1: Type: text/plain, Size: 905 bytes --]
On Wed, Apr 22, 2026 at 06:01:39PM +0100, Leo Yan wrote:
> Building on Debian sid with GCC 15 fails:
> GCC-15 defines NT_ARM_GCS in the toolchain headers, causing gcs-util.h
> to skip its fallback definition of struct user_gcs. This leads to the
> compiler error.
Are you sure it's GCC and not libc? Anyway...
> tools/testing/selftests/arm64/gcs/libc-gcs.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/tools/testing/selftests/arm64/gcs/libc-gcs.c b/tools/testing/selftests/arm64/gcs/libc-gcs.c
> index 17b2fabfec386e523e788d8bcfcafa553c93bd9b..72e82bfbecc99e34da71d56cf5401c137d6e9a48 100644
> --- a/tools/testing/selftests/arm64/gcs/libc-gcs.c
> +++ b/tools/testing/selftests/arm64/gcs/libc-gcs.c
> @@ -16,6 +16,7 @@
>
> #include <asm/hwcap.h>
> #include <asm/mman.h>
> +#include <asm/ptrace.h>
Shouldn't this be in gcs-util.h where the local definition is?
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* [PATCH 8/8] ARM: Remove hacked-up asm/types.h header
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
ARM has a special version of asm/types.h which contains overrides for
certain #define's related to the C types used to back C99 types such as
uint32_t and uintptr_t.
This is only needed when pulling in system headers such as stdint.h
during the build, and this only happens when using NEON intrinsics,
for which there is now a dedicated header file.
So drop this header entirely, and revert to the asm-generic one.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/uapi/asm/types.h | 41 --------------------
1 file changed, 41 deletions(-)
diff --git a/arch/arm/include/uapi/asm/types.h b/arch/arm/include/uapi/asm/types.h
deleted file mode 100644
index 1a667bc26510..000000000000
--- a/arch/arm/include/uapi/asm/types.h
+++ /dev/null
@@ -1,41 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _UAPI_ASM_TYPES_H
-#define _UAPI_ASM_TYPES_H
-
-#include <asm-generic/int-ll64.h>
-
-/*
- * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
- * unambiguous on ARM as you would expect. For the types below, there is a
- * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
- * and the kernel itself, which results in build errors if you try to build with
- * -ffreestanding and include 'stdint.h' (such as when you include 'arm_neon.h'
- * in order to use NEON intrinsics)
- *
- * As the typedefs for these types in 'stdint.h' are based on builtin defines
- * supplied by GCC, we can tweak these to align with the kernel's idea of those
- * types, so 'linux/types.h' and 'stdint.h' can be safely included from the same
- * source file (provided that -ffreestanding is used).
- *
- * int32_t uint32_t uintptr_t
- * bare metal GCC long unsigned long unsigned int
- * glibc GCC int unsigned int unsigned int
- * kernel int unsigned int unsigned long
- */
-
-#ifdef __INT32_TYPE__
-#undef __INT32_TYPE__
-#define __INT32_TYPE__ int
-#endif
-
-#ifdef __UINT32_TYPE__
-#undef __UINT32_TYPE__
-#define __UINT32_TYPE__ unsigned int
-#endif
-
-#ifdef __UINTPTR_TYPE__
-#undef __UINTPTR_TYPE__
-#define __UINTPTR_TYPE__ unsigned long
-#endif
-
-#endif /* _UAPI_ASM_TYPES_H */
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
arm_neon.h is a compiler header which needs some scaffolding to work
correctly in the linux context, and so it is better not to include it
directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which
takes care of this.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid6/neon.uc | 2 +-
lib/raid6/recov_neon_inner.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/raid6/neon.uc b/lib/raid6/neon.uc
index 355270af0cd6..3dc20511103a 100644
--- a/lib/raid6/neon.uc
+++ b/lib/raid6/neon.uc
@@ -24,7 +24,7 @@
* This file is postprocessed using unroll.awk
*/
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
#include "neon.h"
typedef uint8x16_t unative_t;
diff --git a/lib/raid6/recov_neon_inner.c b/lib/raid6/recov_neon_inner.c
index f9e7e8f5a151..06b2967fb8b6 100644
--- a/lib/raid6/recov_neon_inner.c
+++ b/lib/raid6/recov_neon_inner.c
@@ -4,7 +4,7 @@
* Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
#include "neon.h"
#ifdef CONFIG_ARM
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 5/8] lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Tweak the NEON intrinsics crc64 code written for arm64 so it can be
built for 32-bit ARM as well. The only workaround needed is to provide
alternatives for vmull_p64() and vmull_high_p64() on Clang, which only
defines those when building for the AArch64 or arm64ec ISA. Use the same
helpers for GCC too, to avoid doubling the size of the test/validation
matrix.
KUnit benchmark results (Cortex-A53 @ 1 Ghz)
Before:
# crc64_nvme_benchmark: len=1: 35 MB/s
# crc64_nvme_benchmark: len=16: 78 MB/s
# crc64_nvme_benchmark: len=64: 87 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 88 MB/s
# crc64_nvme_benchmark: len=200: 89 MB/s
# crc64_nvme_benchmark: len=256: 89 MB/s
# crc64_nvme_benchmark: len=511: 89 MB/s
# crc64_nvme_benchmark: len=512: 89 MB/s
# crc64_nvme_benchmark: len=1024: 90 MB/s
# crc64_nvme_benchmark: len=3173: 90 MB/s
# crc64_nvme_benchmark: len=4096: 90 MB/s
# crc64_nvme_benchmark: len=16384: 90 MB/s
After:
# crc64_nvme_benchmark: len=1: 32 MB/s
# crc64_nvme_benchmark: len=16: 76 MB/s
# crc64_nvme_benchmark: len=64: 71 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 618 MB/s
# crc64_nvme_benchmark: len=200: 542 MB/s
# crc64_nvme_benchmark: len=256: 920 MB/s
# crc64_nvme_benchmark: len=511: 836 MB/s
# crc64_nvme_benchmark: len=512: 1261 MB/s
# crc64_nvme_benchmark: len=1024: 1531 MB/s
# crc64_nvme_benchmark: len=3173: 1731 MB/s
# crc64_nvme_benchmark: len=4096: 1851 MB/s
# crc64_nvme_benchmark: len=16384: 1858 MB/s
Don't bother with big-endian, as it doesn't work correctly on Clang, and
is barely used these days.
Note that ARM disables preemption and softirq processing when using
kernel mode SIMD, so take care not to hog the CPU for too long.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 5 ++-
lib/crc/arm/crc64-neon.h | 34 ++++++++++++++++++
lib/crc/arm/crc64.h | 36 ++++++++++++++++++++
4 files changed, 75 insertions(+), 1 deletion(-)
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 31038c8d111a..86a0e4bfec77 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -82,6 +82,7 @@ config CRC64
config CRC64_ARCH
bool
depends on CRC64 && CRC_OPTIMIZATIONS
+ default y if ARM && KERNEL_MODE_NEON && !CPU_BIG_ENDIAN
default y if ARM64
default y if RISCV && RISCV_ISA_ZBC && 64BIT
default y if X86_64
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 193257ae466f..386e9c175263 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -39,8 +39,11 @@ crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH)
+crc64-cflags-$(CONFIG_ARM) += -march=armv8-a -mfpu=crypto-neon-fp-armv8
+crc64-cflags-$(CONFIG_ARM64) += -march=armv8-a+crypto
CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto
+CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) $(crc64-cflags-y)
+crc64-$(CONFIG_ARM) += crc64-neon.o
crc64-$(CONFIG_ARM64) += crc64-neon.o
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
diff --git a/lib/crc/arm/crc64-neon.h b/lib/crc/arm/crc64-neon.h
new file mode 100644
index 000000000000..645f553220ff
--- /dev/null
+++ b/lib/crc/arm/crc64-neon.h
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{
+ uint64_t l = vgetq_lane_u64(a, 0);
+ uint64_t m = vgetq_lane_u64(b, 0);
+ uint64x2_t result;
+
+ asm("vmull.p64 %q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+ return result;
+}
+
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
+{
+ uint64_t l = vgetq_lane_u64(a, 1);
+ uint64_t m = vgetq_lane_u64(b, 1);
+ uint64x2_t result;
+
+ asm("vmull.p64 %q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+ return result;
+}
+
+static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
+{
+ uint64_t l = vgetq_lane_u64(a, 1);
+ uint64_t m = vgetq_lane_u64(b, 0);
+ uint64x2_t result;
+
+ asm("vmull.p64 %q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+ return result;
+}
diff --git a/lib/crc/arm/crc64.h b/lib/crc/arm/crc64.h
new file mode 100644
index 000000000000..de274288af61
--- /dev/null
+++ b/lib/crc/arm/crc64.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CRC64 using ARM PMULL instructions
+ */
+
+#include <asm/simd.h>
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
+
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
+
+#define crc64_be_arch crc64_be_generic
+
+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
+{
+ if (len >= 128 && static_branch_likely(&have_pmull) &&
+ likely(may_use_simd())) {
+ do {
+ size_t chunk = min_t(size_t, len & ~15, SZ_4K);
+
+ scoped_ksimd()
+ crc = crc64_nvme_neon(crc, p, chunk);
+
+ p += chunk;
+ len -= chunk;
+ } while (len >= 128);
+ }
+ return crc64_nvme_generic(crc, p, len);
+}
+
+#define crc64_mod_init_arch crc64_mod_init_arch
+static void crc64_mod_init_arch(void)
+{
+ if (elf_hwcap2 & HWCAP2_PMULL)
+ static_branch_enable(&have_pmull);
+}
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 6/8] crypto: aegis128 - Use neon-intrinsics.h on ARM too
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Use the asm/neon-intrinsics.h header on ARM as well as arm64, so that
the calling code does not have to know the difference.
Clean up the Makefile a bit while at it.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
crypto/Makefile | 10 ++++------
crypto/aegis128-neon-inner.c | 4 +---
2 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/crypto/Makefile b/crypto/Makefile
index 162242593c7c..69d1a18e8519 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -103,13 +103,14 @@ obj-$(CONFIG_CRYPTO_CHACHA20POLY1305) += chacha20poly1305.o
obj-$(CONFIG_CRYPTO_AEGIS128) += aegis128.o
aegis128-y := aegis128-core.o
+CFLAGS_aegis128-neon-inner.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_aegis128-neon-inner.o += $(CC_FLAGS_NO_FPU)
ifeq ($(ARCH),arm)
-CFLAGS_aegis128-neon-inner.o += -ffreestanding -march=armv8-a -mfloat-abi=softfp
-CFLAGS_aegis128-neon-inner.o += -mfpu=crypto-neon-fp-armv8
+CFLAGS_aegis128-neon-inner.o += -march=armv8-a -mfpu=crypto-neon-fp-armv8
aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
endif
ifeq ($(ARCH),arm64)
-aegis128-cflags-y := -ffreestanding -mcpu=generic+crypto
+aegis128-cflags-y := -mcpu=generic+crypto
aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \
-ffixed-q19 -ffixed-q20 -ffixed-q21 \
-ffixed-q22 -ffixed-q23 -ffixed-q24 \
@@ -117,11 +118,8 @@ aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \
-ffixed-q28 -ffixed-q29 -ffixed-q30 \
-ffixed-q31
CFLAGS_aegis128-neon-inner.o += $(aegis128-cflags-y)
-CFLAGS_REMOVE_aegis128-neon-inner.o += -mgeneral-regs-only
aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
endif
-# Enable <arm_neon.h>
-CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include)
obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o
obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o
diff --git a/crypto/aegis128-neon-inner.c b/crypto/aegis128-neon-inner.c
index b6a52a386b22..56b534eeb680 100644
--- a/crypto/aegis128-neon-inner.c
+++ b/crypto/aegis128-neon-inner.c
@@ -3,13 +3,11 @@
* Copyright (C) 2019 Linaro, Ltd. <ard.biesheuvel@linaro.org>
*/
-#ifdef CONFIG_ARM64
#include <asm/neon-intrinsics.h>
+#ifdef CONFIG_ARM64
#define AES_ROUND "aese %0.16b, %1.16b \n\t aesmc %0.16b, %0.16b"
#else
-#include <arm_neon.h>
-
#define AES_ROUND "aese.8 %q0, %q1 \n\t aesmc.8 %q0, %q0"
#endif
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Tweak the arm64 code so that the pure NEON intrinsics implementation of
XOR is shared between arm64 and ARM. While at it, rename the arm64
specific piece xor-eor3.c to reflect that only the version based on the
EOR3 instruction is kept there.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid/xor/Makefile | 7 +-
lib/raid/xor/arm64/xor-eor3.c | 146 +++++++++
lib/raid/xor/arm64/xor-neon.c | 312 --------------------
lib/raid/xor/xor-neon.c | 4 +
4 files changed, 154 insertions(+), 315 deletions(-)
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index d78400f2427a..e8ecec3c09f9 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -19,7 +19,8 @@ xor-$(CONFIG_ARM) += arm/xor.o
ifeq ($(CONFIG_ARM),y)
xor-$(CONFIG_KERNEL_MODE_NEON) += xor-neon.o arm/xor-neon-glue.o
endif
-xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
+xor-$(CONFIG_ARM64) += xor-neon.o arm64/xor-eor3.o \
+ arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
@@ -34,8 +35,8 @@ obj-y += tests/
CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_arm64/xor-eor3.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_arm64/xor-eor3.o += $(CC_FLAGS_NO_FPU)
CFLAGS_powerpc/xor_vmx.o += -mhard-float -maltivec \
$(call cc-option,-mabi=altivec) \
diff --git a/lib/raid/xor/arm64/xor-eor3.c b/lib/raid/xor/arm64/xor-eor3.c
new file mode 100644
index 000000000000..e44016c363f1
--- /dev/null
+++ b/lib/raid/xor/arm64/xor-eor3.c
@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/cache.h>
+#include <asm/neon-intrinsics.h>
+#include "xor_impl.h"
+#include "xor_arch.h"
+#include "xor-neon.h"
+
+extern void __xor_eor3_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2);
+
+static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
+{
+ uint64x2_t res;
+
+ asm(ARM64_ASM_PREAMBLE ".arch_extension sha3\n"
+ "eor3 %0.16b, %1.16b, %2.16b, %3.16b"
+ : "=w"(res) : "w"(p), "w"(q), "w"(r));
+ return res;
+}
+
+static void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 ^ p3 */
+ v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
+ vld1q_u64(dp3 + 0));
+ v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
+ vld1q_u64(dp3 + 2));
+ v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
+ vld1q_u64(dp3 + 4));
+ v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
+ vld1q_u64(dp3 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 ^ p3 */
+ v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
+ vld1q_u64(dp3 + 0));
+ v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
+ vld1q_u64(dp3 + 2));
+ v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
+ vld1q_u64(dp3 + 4));
+ v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
+ vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+ uint64_t *dp5 = (uint64_t *)p5;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 ^ p3 */
+ v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
+ vld1q_u64(dp3 + 0));
+ v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
+ vld1q_u64(dp3 + 2));
+ v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
+ vld1q_u64(dp3 + 4));
+ v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
+ vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 ^ p5 */
+ v0 = eor3(v0, vld1q_u64(dp4 + 0), vld1q_u64(dp5 + 0));
+ v1 = eor3(v1, vld1q_u64(dp4 + 2), vld1q_u64(dp5 + 2));
+ v2 = eor3(v2, vld1q_u64(dp4 + 4), vld1q_u64(dp5 + 4));
+ v3 = eor3(v3, vld1q_u64(dp4 + 6), vld1q_u64(dp5 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ dp5 += 8;
+ } while (--lines > 0);
+}
+
+__DO_XOR_BLOCKS(eor3_inner, __xor_eor3_2, __xor_eor3_3, __xor_eor3_4,
+ __xor_eor3_5);
diff --git a/lib/raid/xor/arm64/xor-neon.c b/lib/raid/xor/arm64/xor-neon.c
deleted file mode 100644
index 97ef3cb92496..000000000000
--- a/lib/raid/xor/arm64/xor-neon.c
+++ /dev/null
@@ -1,312 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Authors: Jackie Liu <liuyun01@kylinos.cn>
- * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
- */
-
-#include <linux/cache.h>
-#include <asm/neon-intrinsics.h>
-#include "xor_impl.h"
-#include "xor_arch.h"
-#include "xor-neon.h"
-
-static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
- uint64_t *dp5 = (uint64_t *)p5;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* p1 ^= p5 */
- v0 = veorq_u64(v0, vld1q_u64(dp5 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp5 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp5 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp5 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- dp5 += 8;
- } while (--lines > 0);
-}
-
-__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
- __xor_neon_5);
-
-static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
-{
- uint64x2_t res;
-
- asm(ARM64_ASM_PREAMBLE ".arch_extension sha3\n"
- "eor3 %0.16b, %1.16b, %2.16b, %3.16b"
- : "=w"(res) : "w"(p), "w"(q), "w"(r));
- return res;
-}
-
-static void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 ^ p3 */
- v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
- vld1q_u64(dp3 + 0));
- v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
- vld1q_u64(dp3 + 2));
- v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
- vld1q_u64(dp3 + 4));
- v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
- vld1q_u64(dp3 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 ^ p3 */
- v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
- vld1q_u64(dp3 + 0));
- v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
- vld1q_u64(dp3 + 2));
- v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
- vld1q_u64(dp3 + 4));
- v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
- vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
- uint64_t *dp5 = (uint64_t *)p5;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 ^ p3 */
- v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
- vld1q_u64(dp3 + 0));
- v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
- vld1q_u64(dp3 + 2));
- v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
- vld1q_u64(dp3 + 4));
- v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
- vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 ^ p5 */
- v0 = eor3(v0, vld1q_u64(dp4 + 0), vld1q_u64(dp5 + 0));
- v1 = eor3(v1, vld1q_u64(dp4 + 2), vld1q_u64(dp5 + 2));
- v2 = eor3(v2, vld1q_u64(dp4 + 4), vld1q_u64(dp5 + 4));
- v3 = eor3(v3, vld1q_u64(dp4 + 6), vld1q_u64(dp5 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- dp5 += 8;
- } while (--lines > 0);
-}
-
-__DO_XOR_BLOCKS(eor3_inner, __xor_neon_2, __xor_eor3_3, __xor_eor3_4,
- __xor_eor3_5);
diff --git a/lib/raid/xor/xor-neon.c b/lib/raid/xor/xor-neon.c
index a3e2b4af8d36..c7c3cf634e23 100644
--- a/lib/raid/xor/xor-neon.c
+++ b/lib/raid/xor/xor-neon.c
@@ -173,3 +173,7 @@ static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
__xor_neon_5);
+
+#ifdef CONFIG_ARM64
+extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2);
+#endif
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 4/8] lib/crc: Turn NEON intrinsics crc64 implementation into common code
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Move and rename the CRC64 NEON intrinsics implementation source file and
rename the function name to reflect that it is NEON code that can be
shared. This will be wired up for 32-bit ARM in a subsequent patch.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/crc/Makefile | 6 ++---
lib/crc/arm64/crc64-neon.h | 21 ++++++++++++++++
lib/crc/arm64/crc64.h | 4 +--
lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +++-----------------
4 files changed, 30 insertions(+), 27 deletions(-)
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index ff213590e4e3..193257ae466f 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -39,9 +39,9 @@ crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH)
-CFLAGS_REMOVE_arm64/crc64-neon-inner.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_arm64/crc64-neon-inner.o += $(CC_FLAGS_FPU) -march=armv8-a+crypto
-crc64-$(CONFIG_ARM64) += arm64/crc64-neon-inner.o
+CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto
+crc64-$(CONFIG_ARM64) += crc64-neon.o
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
crc64-$(CONFIG_X86) += x86/crc64-pclmul.o
diff --git a/lib/crc/arm64/crc64-neon.h b/lib/crc/arm64/crc64-neon.h
new file mode 100644
index 000000000000..fcd5b1e6f812
--- /dev/null
+++ b/lib/crc/arm64/crc64-neon.h
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{
+ return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0),
+ vgetq_lane_u64(b, 0)));
+}
+
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
+{
+ poly64x2_t l = vreinterpretq_p64_u64(a);
+ poly64x2_t m = vreinterpretq_p64_u64(b);
+
+ return vreinterpretq_u64_p128(vmull_high_p64(l, m));
+}
+
+static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
+{
+ return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 1),
+ vgetq_lane_u64(b, 0)));
+}
diff --git a/lib/crc/arm64/crc64.h b/lib/crc/arm64/crc64.h
index 60151ec3035a..c7a69e1f3d8f 100644
--- a/lib/crc/arm64/crc64.h
+++ b/lib/crc/arm64/crc64.h
@@ -8,7 +8,7 @@
#include <linux/minmax.h>
#include <linux/sizes.h>
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len);
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
#define crc64_be_arch crc64_be_generic
@@ -19,7 +19,7 @@ static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
size_t chunk = len & ~15;
scoped_ksimd()
- crc = crc64_nvme_arm64_c(crc, p, chunk);
+ crc = crc64_nvme_neon(crc, p, chunk);
p += chunk;
len &= 15;
diff --git a/lib/crc/arm64/crc64-neon-inner.c b/lib/crc/crc64-neon.c
similarity index 62%
rename from lib/crc/arm64/crc64-neon-inner.c
rename to lib/crc/crc64-neon.c
index 28527e544ff6..4753fb94a4be 100644
--- a/lib/crc/arm64/crc64-neon-inner.c
+++ b/lib/crc/crc64-neon.c
@@ -6,7 +6,9 @@
#include <linux/types.h>
#include <asm/neon-intrinsics.h>
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len);
+#include "crc64-neon.h"
+
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
/* x^191 mod G, x^127 mod G */
static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,
@@ -15,27 +17,7 @@ static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,
static const u64 bconsts_val[2] = { 0x27ecfa329aef9f77ULL,
0x34d926535897936aULL };
-static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
-{
- return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0),
- vgetq_lane_u64(b, 0)));
-}
-
-static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
-{
- poly64x2_t l = vreinterpretq_p64_u64(a);
- poly64x2_t m = vreinterpretq_p64_u64(b);
-
- return vreinterpretq_u64_p128(vmull_high_p64(l, m));
-}
-
-static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
-{
- return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 1),
- vgetq_lane_u64(b, 0)));
-}
-
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len)
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len)
{
uint64x2_t fold_consts = vld1q_u64(fold_consts_val);
uint64x2_t v0 = { crc, 0 };
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 2/8] xor/arm: Replace vectorized implementation with arm64's intrinsics
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Drop the XOR implementation generated by the vectorizer: this has always
been a bit of a hack, and now that arm64 has an intrinsics version that
works on ARM too, let's use that instead.
So copy the part of the arm64 code that can be shared (so not the EOR3
version). The arm64 code will be updated in a subsequent patch to share
this implementation.
Performance (QEMU mach-virt VM running on Synquacer [Cortex-A53 @ 1 GHz]
Before:
[ 3.519687] xor: measuring software checksum speed
[ 3.521725] neon : 1660 MB/sec
[ 3.524733] 32regs : 1105 MB/sec
[ 3.527751] 8regs : 1098 MB/sec
[ 3.529911] arm4regs : 1540 MB/sec
After:
[ 3.517654] xor: measuring software checksum speed
[ 3.519454] neon : 1896 MB/sec
[ 3.522499] 32regs : 1090 MB/sec
[ 3.525560] 8regs : 1083 MB/sec
[ 3.527700] arm4regs : 1556 MB/sec
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid/xor/Makefile | 6 +-
lib/raid/xor/arm/xor-neon.c | 26 ---
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/xor-8regs.c | 2 -
lib/raid/xor/xor-neon.c | 175 ++++++++++++++++++++
6 files changed, 186 insertions(+), 37 deletions(-)
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 4d633dfd5b90..d78400f2427a 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -17,7 +17,7 @@ endif
xor-$(CONFIG_ALPHA) += alpha/xor.o
xor-$(CONFIG_ARM) += arm/xor.o
ifeq ($(CONFIG_ARM),y)
-xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o
+xor-$(CONFIG_KERNEL_MODE_NEON) += xor-neon.o arm/xor-neon-glue.o
endif
xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
@@ -31,8 +31,8 @@ xor-$(CONFIG_X86_32) += x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o
xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o
obj-y += tests/
-CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
+CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU)
CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU)
CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
diff --git a/lib/raid/xor/arm/xor-neon.c b/lib/raid/xor/arm/xor-neon.c
deleted file mode 100644
index 23147e3a7904..000000000000
--- a/lib/raid/xor/arm/xor-neon.c
+++ /dev/null
@@ -1,26 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
- */
-
-#include "xor_impl.h"
-#include "xor_arch.h"
-
-#ifndef __ARM_NEON__
-#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
-#endif
-
-/*
- * Pull in the reference implementations while instructing GCC (through
- * -ftree-vectorize) to attempt to exploit implicit parallelism and emit
- * NEON instructions. Clang does this by default at O2 so no pragma is
- * needed.
- */
-#ifdef CONFIG_CC_IS_GCC
-#pragma GCC optimize "tree-vectorize"
-#endif
-
-#define NO_TEMPLATE
-#include "../xor-8regs.c"
-
-__DO_XOR_BLOCKS(neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
diff --git a/lib/raid/xor/arm/xor-neon.h b/lib/raid/xor/arm/xor-neon.h
new file mode 100644
index 000000000000..406e0356f05b
--- /dev/null
+++ b/lib/raid/xor/arm/xor-neon.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+extern struct xor_block_template xor_block_arm4regs;
+extern struct xor_block_template xor_block_neon;
+
+void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
diff --git a/lib/raid/xor/arm/xor_arch.h b/lib/raid/xor/arm/xor_arch.h
index 775ff835df65..f1ddb64fe62a 100644
--- a/lib/raid/xor/arm/xor_arch.h
+++ b/lib/raid/xor/arm/xor_arch.h
@@ -3,12 +3,7 @@
* Copyright (C) 2001 Russell King
*/
#include <asm/neon.h>
-
-extern struct xor_block_template xor_block_arm4regs;
-extern struct xor_block_template xor_block_neon;
-
-void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
- unsigned int bytes);
+#include "xor-neon.h"
static __always_inline void __init arch_xor_init(void)
{
diff --git a/lib/raid/xor/xor-8regs.c b/lib/raid/xor/xor-8regs.c
index 1edaed8acffe..46b3c8bdc27f 100644
--- a/lib/raid/xor/xor-8regs.c
+++ b/lib/raid/xor/xor-8regs.c
@@ -93,11 +93,9 @@ xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-#ifndef NO_TEMPLATE
DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
struct xor_block_template xor_block_8regs = {
.name = "8regs",
.xor_gen = xor_gen_8regs,
};
-#endif /* NO_TEMPLATE */
diff --git a/lib/raid/xor/xor-neon.c b/lib/raid/xor/xor-neon.c
new file mode 100644
index 000000000000..a3e2b4af8d36
--- /dev/null
+++ b/lib/raid/xor/xor-neon.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Authors: Jackie Liu <liuyun01@kylinos.cn>
+ * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
+ */
+
+#include "xor_impl.h"
+#include "xor-neon.h"
+
+#include <asm/neon-intrinsics.h>
+
+static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+ uint64_t *dp5 = (uint64_t *)p5;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* p1 ^= p5 */
+ v0 = veorq_u64(v0, vld1q_u64(dp5 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp5 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp5 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp5 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ dp5 += 8;
+ } while (--lines > 0);
+}
+
+__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
+ __xor_neon_5);
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 1/8] ARM: Add a neon-intrinsics.h header like on arm64
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Add a header asm/neon-intrinsics.h similar to the one that arm64 has.
This makes it possible for NEON intrinsics code to be shared seamlessly
between ARM and arm64.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 60 ++++++++++++++++++++
2 files changed, 63 insertions(+), 1 deletion(-)
diff --git a/Documentation/arch/arm/kernel_mode_neon.rst b/Documentation/arch/arm/kernel_mode_neon.rst
index 9bfb71a2a9b9..1efb6d35b7bd 100644
--- a/Documentation/arch/arm/kernel_mode_neon.rst
+++ b/Documentation/arch/arm/kernel_mode_neon.rst
@@ -121,4 +121,6 @@ observe the following in addition to the rules above:
* Compile the unit containing the NEON intrinsics with '-ffreestanding' so GCC
uses its builtin version of <stdint.h> (this is a C99 header which the kernel
does not supply);
-* Include <arm_neon.h> last, or at least after <linux/types.h>
+* Do not include <arm_neon.h> directly: instead, include <asm/neon-intrinsics.h>,
+ which tweaks some macro definitions so that system headers can be included
+ safely.
diff --git a/arch/arm/include/asm/neon-intrinsics.h b/arch/arm/include/asm/neon-intrinsics.h
new file mode 100644
index 000000000000..8b80c05ce1d7
--- /dev/null
+++ b/arch/arm/include/asm/neon-intrinsics.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __ASM_NEON_INTRINSICS_H
+#define __ASM_NEON_INTRINSICS_H
+
+#ifndef __ARM_NEON__
+#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
+#endif
+
+#include <asm-generic/int-ll64.h>
+
+/*
+ * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
+ * unambiguous on ARM as you would expect. For the types below, there is a
+ * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
+ * and the kernel itself, which results in build errors if you try to build
+ * with -ffreestanding and include 'stdint.h' (such as when you include
+ * 'arm_neon.h' in order to use NEON intrinsics)
+ *
+ * As the typedefs for these types in 'stdint.h' are based on builtin defines
+ * supplied by GCC, we can tweak these to align with the kernel's idea of those
+ * types, so 'linux/types.h' and 'stdint.h' can be safely included from the
+ * same source file (provided that -ffreestanding is used).
+ *
+ * int32_t uint32_t intptr_t uintptr_t
+ * bare metal GCC long unsigned long int unsigned int
+ * glibc GCC int unsigned int int unsigned int
+ * kernel int unsigned int long unsigned long
+ */
+
+#ifdef __INT32_TYPE__
+#undef __INT32_TYPE__
+#define __INT32_TYPE__ int
+#endif
+
+#ifdef __UINT32_TYPE__
+#undef __UINT32_TYPE__
+#define __UINT32_TYPE__ unsigned int
+#endif
+
+#ifdef __INTPTR_TYPE__
+#undef __INTPTR_TYPE__
+#define __INTPTR_TYPE__ long
+#endif
+
+#ifdef __UINTPTR_TYPE__
+#undef __UINTPTR_TYPE__
+#define __UINTPTR_TYPE__ unsigned long
+#endif
+
+/*
+ * genksyms chokes on the ARM NEON instrinsics system header, but we
+ * don't export anything it defines anyway, so just disregard when
+ * genksyms execute.
+ */
+#ifndef __GENKSYMS__
+#include <arm_neon.h>
+#endif
+
+#endif /* __ASM_NEON_INTRINSICS_H */
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 0/8] ARM crc64 and XOR using NEON intrinsics
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
From: Ard Biesheuvel <ardb@kernel.org>
This is a follow-up to both [0] and [1], both of which included patch #1
of this series, which introduces the asm/neon-intrinsics.h header on
32-bit ARM. The remaining changes rely on this.
The purpose of this series is to streamline / clean up the use of NEON
intrinsics on 32-bit ARM, by sharing more code, clean up Make rules and
finally, getting rid of the hacked up types.h header, which does some
nasty things that are only needed when building NEON intrinsics code.
Patches #2 and #3 replace the ARM autovectorized XOR implementation with
the NEON intrinsics version used by arm64.
Patches #4 and #5 enable the arm64 NEON intrinsics implementation of
crc64 on 32-bit ARM.
Patches #6 and #7 drop the direct includes of <arm_neon.h> and perform
some additional cleanup to reduce the delta between ARM and arm64 code
and Make rules.
It would probably be easiest to take all these changes through a single
tree, and the CRC tree seems like a suitable candidate, if Eric agrees.
Cc: Christoph Hellwig <hch@lst.de>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Eric Biggers <ebiggers@kernel.org>
[0] https://lore.kernel.org/all/20260331074940.55502-7-ardb+git@google.com/
[1] https://lore.kernel.org/all/20260330144630.33026-7-ardb@kernel.org/
Ard Biesheuvel (8):
ARM: Add a neon-intrinsics.h header like on arm64
xor/arm: Replace vectorized implementation with arm64's intrinsics
xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
lib/crc: Turn NEON intrinsics crc64 implementation into common code
lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
crypto: aegis128 - Use neon-intrinsics.h on ARM too
lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
ARM: Remove hacked-up asm/types.h header
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 60 ++++++++
arch/arm/include/uapi/asm/types.h | 41 ------
crypto/Makefile | 10 +-
crypto/aegis128-neon-inner.c | 4 +-
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 9 +-
lib/crc/arm/crc64-neon.h | 34 +++++
lib/crc/arm/crc64.h | 36 +++++
lib/crc/arm64/crc64-neon.h | 21 +++
lib/crc/arm64/crc64.h | 4 +-
lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +---
lib/raid/xor/Makefile | 13 +-
lib/raid/xor/arm/xor-neon.c | 26 ----
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/arm64/xor-eor3.c | 146 ++++++++++++++++++++
lib/raid/xor/xor-8regs.c | 2 -
lib/raid/xor/{arm64 => }/xor-neon.c | 143 +------------------
lib/raid6/neon.uc | 2 +-
lib/raid6/recov_neon_inner.c | 2 +-
21 files changed, 340 insertions(+), 258 deletions(-)
create mode 100644 arch/arm/include/asm/neon-intrinsics.h
delete mode 100644 arch/arm/include/uapi/asm/types.h
create mode 100644 lib/crc/arm/crc64-neon.h
create mode 100644 lib/crc/arm/crc64.h
create mode 100644 lib/crc/arm64/crc64-neon.h
rename lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} (62%)
delete mode 100644 lib/raid/xor/arm/xor-neon.c
create mode 100644 lib/raid/xor/arm/xor-neon.h
create mode 100644 lib/raid/xor/arm64/xor-eor3.c
rename lib/raid/xor/{arm64 => }/xor-neon.c (56%)
base-commit: 6596a02b207886e9e00bb0161c7fd59fea53c081
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply
* Re: [PATCH] arm64: smp: Limit nr_cpu_ids under nosmp
From: Catalin Marinas @ 2026-04-22 17:16 UTC (permalink / raw)
To: Pengjie Zhang
Cc: will, maz, timothy.hayes, lpieralisi, mrigendra.chaubey, arnd,
linux-arm-kernel, linux-kernel, zhanjie9, zhenglifeng1, lihuisong,
yubowen8, linhongye, linuxarm, wangzhi12
In-Reply-To: <20260422095831.2926775-1-zhangpengjie2@huawei.com>
On Wed, Apr 22, 2026 at 05:58:31PM +0800, Pengjie Zhang wrote:
> Under nosmp (maxcpus=0), arm64 never brings up secondary CPUs.
>
> However, arm64 still enumerates firmware-described CPUs during SMP
> initialization, so secondary CPUs can remain visible to
> for_each_possible_cpu() users even though they never reach the
> bringup path in this configuration.
>
> This is not just a cosmetic mask mismatch: code iterating over
> possible CPUs may observe secondary CPU per-CPU state that is never
> fully initialized under nosmp.
>
> Limit nr_cpu_ids to 1 in arch_disable_smp_support() so that
> secondary CPUs are not set up on arm64 when nosmp/maxcpus=0 is in
> effect.
>
> Signed-off-by: Pengjie Zhang <zhangpengjie2@huawei.com>
> ---
> arch/arm64/kernel/smp.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index 1aa324104afb..cc34c68871e9 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -435,6 +435,15 @@ static void __init hyp_mode_check(void)
> }
> }
>
> +void __init arch_disable_smp_support(void)
> +{
> + /*
> + * Under nosmp/maxcpus=0, only the boot CPU can ever be brought up.
> + * Limit nr_cpu_ids so that secondary CPUs are never set up.
> + */
> + set_nr_cpu_ids(1);
> +}
I don't think that's the right fix. We don't have anything like the x86
ioapic to disable in this function, so no need to implement it. If
nr_cpu_ids must be 1 with nosmp/maxcpus=0, I'd rather do this in the
generic code. It need some alignment with other architectures if we are
to do this early. IOW, is nosmp equivalent to nr_cpus=1?
In the meantime, for arm64, we can do something like below and let the
generic code set nr_cpu_ids() via start_kernel() -> setup_nr_cpu_ids().
-------------8<-------------------
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 1aa324104afb..7364481cc03a 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -754,6 +754,13 @@ void __init smp_init_cpus(void)
return;
}
+ /*
+ * For the nosmp/maxcpus=0 case, do not mark the secondary CPUs
+ * possible.
+ */
+ if (!setup_max_cpus)
+ return;
+
/*
* We need to set the cpu_logical_map entries before enabling
* the cpus so that cpu processor description entries (DT cpu nodes
^ permalink raw reply related
* [PATCH] kselftest/arm64: Fix build failure with GCC-15
From: Leo Yan @ 2026-04-22 17:01 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Shuah Khan, Mark Brown,
Thiago Jung Bauermann
Cc: linux-arm-kernel, linux-kselftest, linux-kernel, Leo Yan
Building on Debian sid with GCC 15 fails:
CC libc-gcs
libc-gcs.c: In function 'ptrace_read_write':
libc-gcs.c:142:25: error: storage size of 'child_gcs' isn't known
142 | struct user_gcs child_gcs;
| ^~~~~~~~~
libc-gcs.c:142:25: warning: unused variable 'child_gcs' [-Wunused-variable]
GCC-15 defines NT_ARM_GCS in the toolchain headers, causing gcs-util.h
to skip its fallback definition of struct user_gcs. This leads to the
compiler error.
Fix this by including <asm/ptrace.h> to provide the proper definition.
Fixes: a505a52b4e29 ("kselftest/arm64: Add a GCS test program built with the system libc")
Signed-off-by: Leo Yan <leo.yan@arm.com>
---
tools/testing/selftests/arm64/gcs/libc-gcs.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/testing/selftests/arm64/gcs/libc-gcs.c b/tools/testing/selftests/arm64/gcs/libc-gcs.c
index 17b2fabfec386e523e788d8bcfcafa553c93bd9b..72e82bfbecc99e34da71d56cf5401c137d6e9a48 100644
--- a/tools/testing/selftests/arm64/gcs/libc-gcs.c
+++ b/tools/testing/selftests/arm64/gcs/libc-gcs.c
@@ -16,6 +16,7 @@
#include <asm/hwcap.h>
#include <asm/mman.h>
+#include <asm/ptrace.h>
#include <linux/compiler.h>
---
base-commit: 4ee64205ffaa587e8114d84a67ac721399ccb369
change-id: 20260421-selftests_arm64_gcc15-15f6f6d07dd8
Best regards,
--
Leo Yan <leo.yan@arm.com>
^ permalink raw reply related
* Re: [PATCH] iommu/arm-smmu-v3: Allow disabling Stage 1 translation
From: Robin Murphy @ 2026-04-22 16:36 UTC (permalink / raw)
To: Jason Gunthorpe, Evangelos Petrongonas
Cc: Will Deacon, Joerg Roedel, Nicolin Chen, Pranjal Shrivastava,
Lu Baolu, linux-arm-kernel, iommu, linux-kernel, nh-open-source,
Zeev Zilberman
In-Reply-To: <20260422162351.GK3611611@ziepe.ca>
On 2026-04-22 5:23 pm, Jason Gunthorpe wrote:
> On Wed, Apr 22, 2026 at 06:44:31AM +0000, Evangelos Petrongonas wrote:
>> The motivation is live update of the hypervisor: we want to kexec into a
>> new kernel while keeping DMA from passthrough devices flowing, which
>> means the SMMU's translation state has to survive the handover. The Live
>> Update Orchestrator work [1] and the in-progress "iommu: Add live
>> update state preservation" series [2] are building exactly this plumbing
>> on top of KHO; [2]'s cover letter calls out Arm SMMUv3 support as future
>> work, and an earlier RFC from Amazon [3] sketched the same idea for
>> iommufd.
>
> It would be appropriate to keep this patch with the rest of that out
> of tree pile, for example in the series that enables s2 only support
> in smmuv3.
Or even better, just make sure that whatever hypervisor supports this
half-finished WIP mechanism also uses IOMMU_HWPT_ALLOC_NEST_PARENT to
explicitly get stage 2 domains for VM-assigned devices in the first
place, rather than swing a big hammer at the kernel (that takes out
SVA/PASID support as collateral damage...)
Thanks,
Robin.
>> For this use case, Stage 2 is materially easier to persist than Stage 1,
>> for structural rather than performance reasons:
>
> I don't think so. The driver needs to know each and every STE that
> will survive KHO. The ones that don't survive need to be reset to
> abort STEs. From that point it is trivial enough to include the CD
> memory in the preservation.
>
> It would help to send a preparation series to switch the ARM STE and
> CD logic away from dma_alloc_coherent and use iommu-pages instead,
> since we only expect iommu-pages to support preservation..
>
> I could maybe see only supporting non-PASID as a first-series, but a
> CD table with SSID 0 only populated is still pretty trivial.
>
> Jason
^ permalink raw reply
* Re: [PATCH v6 3/3] dts: s32g: Add GPR syscon region
From: Jared Kangas @ 2026-04-22 16:25 UTC (permalink / raw)
To: Dan Carpenter
Cc: Chester Lin, Matthias Brugger, Ghennadi Procopciuc,
NXP S32 Linux Team, Frank Li, Sascha Hauer,
Pengutronix Kernel Team, Fabio Estevam, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, linux-arm-kernel, imx,
devicetree, linux-kernel, linaro-s32, netdev
In-Reply-To: <aeZYQp9b5aoV7Ihv@stanley.mountain>
On Mon, Apr 20, 2026 at 07:45:54PM +0300, Dan Carpenter wrote:
> On Mon, Apr 20, 2026 at 09:04:00AM -0700, Jared Kangas wrote:
> > Fixing Dan's address based on mailmap update, sorry for the noise.
> >
> > On Fri, Apr 17, 2026 at 02:36:25PM -0700, Jared Kangas wrote:
> > > Hi Dan,
> > >
> > > [snip]
> > >
> > > I gave this a test on an S32G-VNP-RDB3 and didn't see any issues on the
> > > dwmac-s32 side, but this appears to trigger a panic when reading the new
> > > debugfs regmap/*/registers file for the syscon node:
> > >
> > > [snip]
>
> Oh, ugh... I didn't realize that this wasn't merged. I don't have a
> way to test this any more. The simplest fix would be to do change the
> 0x3000 to 0x100. The GPR63 register is at 0xFC.
>
> reg = <0x4007c000 0x100>;
>
> That's probably the best fix as well. The later register areas would
> be their own syscons.
Tried that out and it looks good to me. With the write routed through
syscon:
# xxd -g4 /proc/device-tree/soc@0/syscon@4007c000/reg
00000000: 4007c000 00000100 @.......
# cat /sys/kernel/debug/regmap/dummy-syscon@0x000000004007c000/registers
00: 00000000
04: 00000002
08: 000000e7
0c: 00000001
10: ffffffff
14: 1fffffff
18: 00007fff
1c: 00000000
20: 00000000
...
f4: 00000000
f8: 00000000
fc: 00000000
No more crashes and 04's value lines up with the S32_PHY_INTF_SEL_RGMII
(0x2) write, so if you're able to post a revision, feel free to add my
T-b:
Tested-by: Jared Kangas <jkangas@redhat.com>
^ permalink raw reply
* [RFC PATCH v2 4/4] firmware: arm_ffa: check pkvm initailised when initailise ffa driver
From: Yeoreum Yun @ 2026-04-22 16:24 UTC (permalink / raw)
To: linux-security-module, linux-kernel, linux-integrity,
linux-arm-kernel, kvmarm
Cc: paul, jmorris, serge, zohar, roberto.sassu, dmitry.kasatkin,
eric.snowberg, jarkko, jgg, sudeep.holla, maz, oupton, joey.gouly,
suzuki.poulose, yuzenghui, catalin.marinas, will, noodles,
sebastianene, Yeoreum Yun
In-Reply-To: <20260422162449.1814615-1-yeoreum.yun@arm.com>
When pKVM is enabled, the FF-A driver must be initialized after pKVM.
Otherwise, pKVM cannot negotiate the FF-A version or
obtain RX/TX buffer information, leading to failures in FF-A calls.
During FF-A driver initialization, check whether pKVM has been initialized.
If pKVM isn't initailised, register notifier and do initialisation
of FF-A driver when pKVM is initialized.
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
arch/arm64/include/asm/virt.h | 11 ++++++++++
arch/arm64/kvm/arm.c | 21 ++++++++++++++++++
arch/arm64/kvm/pkvm.c | 2 ++
drivers/firmware/arm_ffa/common.h | 4 ++--
drivers/firmware/arm_ffa/driver.c | 36 ++++++++++++++++++++++++++++++-
drivers/firmware/arm_ffa/smccc.c | 2 +-
6 files changed, 72 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h
index b51ab6840f9c..ad038a3b8727 100644
--- a/arch/arm64/include/asm/virt.h
+++ b/arch/arm64/include/asm/virt.h
@@ -68,6 +68,8 @@
#include <asm/sysreg.h>
#include <asm/cpufeature.h>
+struct notifier_block;
+
/*
* __boot_cpu_mode records what mode CPUs were booted in.
* A correctly-implemented bootloader must start all CPUs in the same mode:
@@ -166,6 +168,15 @@ static inline bool is_hyp_nvhe(void)
return is_hyp_mode_available() && !is_kernel_in_hyp_mode();
}
+enum kvm_arm_event {
+ PKVM_INITIALISED,
+ KVM_ARM_EVENT_MAX,
+};
+
+extern int kvm_arm_event_notifier_call_chain(enum kvm_arm_event event, void *data);
+extern int kvm_arm_event_notifier_register(struct notifier_block *nb);
+extern int kvm_arm_event_notifier_unregister(struct notifier_block *nb);
+
#endif /* __ASSEMBLER__ */
#endif /* ! __ASM__VIRT_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 410ffd41fd73..8da10049ab65 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -14,6 +14,7 @@
#include <linux/vmalloc.h>
#include <linux/fs.h>
#include <linux/mman.h>
+#include <linux/notifier.h>
#include <linux/sched.h>
#include <linux/kvm.h>
#include <linux/kvm_irqfd.h>
@@ -111,6 +112,8 @@ DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
DECLARE_KVM_NVHE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt);
+BLOCKING_NOTIFIER_HEAD(kvm_arm_event_notifier_head);
+
static bool vgic_present, kvm_arm_initialised;
static DEFINE_PER_CPU(unsigned char, kvm_hyp_initialized);
@@ -3064,4 +3067,22 @@ enum kvm_mode kvm_get_mode(void)
return kvm_mode;
}
+int kvm_arm_event_notifier_call_chain(enum kvm_arm_event event, void *data)
+{
+ return blocking_notifier_call_chain(&kvm_arm_event_notifier_head,
+ event, data);
+}
+
+int kvm_arm_event_notifier_register(struct notifier_block *nb)
+{
+ return blocking_notifier_chain_register(&kvm_arm_event_notifier_head, nb);
+}
+EXPORT_SYMBOL_GPL(kvm_arm_event_notifier_register);
+
+int kvm_arm_event_notifier_unregister(struct notifier_block *nb)
+{
+ return blocking_notifier_chain_unregister(&kvm_arm_event_notifier_head, nb);
+}
+EXPORT_SYMBOL_GPL(kvm_arm_event_notifier_unregister);
+
module_init(kvm_arm_init);
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index d7a0f69a9982..e76562b0a45a 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -280,6 +280,8 @@ static int __init finalize_pkvm(void)
ret = pkvm_drop_host_privileges();
if (ret)
pr_err("Failed to finalize Hyp protection: %d\n", ret);
+ else
+ kvm_arm_event_notifier_call_chain(PKVM_INITIALISED, NULL);
return ret;
}
diff --git a/drivers/firmware/arm_ffa/common.h b/drivers/firmware/arm_ffa/common.h
index 9c6425a81d0d..5cdf4bd222c6 100644
--- a/drivers/firmware/arm_ffa/common.h
+++ b/drivers/firmware/arm_ffa/common.h
@@ -18,9 +18,9 @@ bool ffa_device_is_valid(struct ffa_device *ffa_dev);
void ffa_device_match_uuid(struct ffa_device *ffa_dev, const uuid_t *uuid);
#ifdef CONFIG_ARM_FFA_SMCCC
-int __init ffa_transport_init(ffa_fn **invoke_ffa_fn);
+int ffa_transport_init(ffa_fn **invoke_ffa_fn);
#else
-static inline int __init ffa_transport_init(ffa_fn **invoke_ffa_fn)
+static inline int ffa_transport_init(ffa_fn **invoke_ffa_fn)
{
return -EOPNOTSUPP;
}
diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index 02c76ac1570b..67df053e65b8 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -35,6 +35,7 @@
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/mutex.h>
+#include <linux/notifier.h>
#include <linux/of_irq.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
@@ -42,6 +43,8 @@
#include <linux/uuid.h>
#include <linux/xarray.h>
+#include <asm/virt.h>
+
#include "common.h"
#define FFA_DRIVER_VERSION FFA_VERSION_1_2
@@ -2029,7 +2032,7 @@ static void ffa_notifications_setup(void)
ffa_notifications_cleanup();
}
-static int __init ffa_init(void)
+static int __ffa_init(void)
{
int ret;
u32 buf_sz;
@@ -2105,11 +2108,42 @@ static int __init ffa_init(void)
free_drv_info:
kfree(drv_info);
return ret;
+
+}
+
+static int ffa_kvm_arm_event_handler(struct notifier_block *nb,
+ unsigned long event, void *unused)
+{
+ if (event == PKVM_INITIALISED)
+ __ffa_init();
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block ffa_kvm_arm_event_notifier = {
+ .notifier_call = ffa_kvm_arm_event_handler,
+};
+
+static int __init ffa_init(void)
+{
+ /*
+ * When pKVM is enabled, the FF-A driver must be initialized
+ * after pKVM initialization. Otherwise, pKVM cannot negotiate
+ * the FF-A version or obtain RX/TX buffer information,
+ * which leads to failures in FF-A calls.
+ */
+ if (IS_ENABLED(CONFIG_KVM) && is_protected_kvm_enabled() &&
+ !is_pkvm_initialized())
+ return kvm_arm_event_notifier_register(&ffa_kvm_arm_event_notifier);
+
+ return __ffa_init();
}
device_initcall(ffa_init);
static void __exit ffa_exit(void)
{
+ if (IS_ENABLED(CONFIG_KVM))
+ kvm_arm_event_notifier_unregister(&ffa_kvm_arm_event_notifier);
ffa_notifications_cleanup();
ffa_partitions_cleanup();
ffa_rxtx_unmap();
diff --git a/drivers/firmware/arm_ffa/smccc.c b/drivers/firmware/arm_ffa/smccc.c
index 4d85bfff0a4e..e6125dd9f58f 100644
--- a/drivers/firmware/arm_ffa/smccc.c
+++ b/drivers/firmware/arm_ffa/smccc.c
@@ -17,7 +17,7 @@ static void __arm_ffa_fn_hvc(ffa_value_t args, ffa_value_t *res)
arm_smccc_1_2_hvc(&args, res);
}
-int __init ffa_transport_init(ffa_fn **invoke_ffa_fn)
+int ffa_transport_init(ffa_fn **invoke_ffa_fn)
{
enum arm_smccc_conduit conduit;
--
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply related
* [RFC PATCH v2 3/4] firmware: arm_ffa: revert ffa_init() initcall level to device_initcall
From: Yeoreum Yun @ 2026-04-22 16:24 UTC (permalink / raw)
To: linux-security-module, linux-kernel, linux-integrity,
linux-arm-kernel, kvmarm
Cc: paul, jmorris, serge, zohar, roberto.sassu, dmitry.kasatkin,
eric.snowberg, jarkko, jgg, sudeep.holla, maz, oupton, joey.gouly,
suzuki.poulose, yuzenghui, catalin.marinas, will, noodles,
sebastianene, Yeoreum Yun
In-Reply-To: <20260422162449.1814615-1-yeoreum.yun@arm.com>
commit 0e0546eabcd6 ("firmware: arm_ffa: Change initcall level of ffa_init() to rootfs_initcall")
changed the initcall level of ffa_init() to rootfs_initcall to address
an issue where IMA could not properly recognize the TPM device.
However, this introduces a problem: pKVM fails to handle any FF-A calls
because it cannot trap the FFA_VERSION call invoked by ffa_init().
Since the IMA init function level has been changed to late_initcall_sync,
there is no longer a need to keep ffa_init() at rootfs_initcall.
Revert it back to device_initcall.
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
drivers/firmware/arm_ffa/driver.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index f2f94d4d533e..02c76ac1570b 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -2106,7 +2106,7 @@ static int __init ffa_init(void)
kfree(drv_info);
return ret;
}
-rootfs_initcall(ffa_init);
+device_initcall(ffa_init);
static void __exit ffa_exit(void)
{
--
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply related
* [RFC PATCH v2 2/4] tpm: tpm_crb_ffa: revert defered_probed when tpm_crb_ffa is built-in
From: Yeoreum Yun @ 2026-04-22 16:24 UTC (permalink / raw)
To: linux-security-module, linux-kernel, linux-integrity,
linux-arm-kernel, kvmarm
Cc: paul, jmorris, serge, zohar, roberto.sassu, dmitry.kasatkin,
eric.snowberg, jarkko, jgg, sudeep.holla, maz, oupton, joey.gouly,
suzuki.poulose, yuzenghui, catalin.marinas, will, noodles,
sebastianene, Yeoreum Yun
In-Reply-To: <20260422162449.1814615-1-yeoreum.yun@arm.com>
commit 746d9e9f62a6 ("tpm: tpm_crb_ffa: try to probe tpm_crb_ffa when it's build_in")
probe tpm_crb_ffa forcefully when it's built-in to integrate with IMA.
However, as IMA init function is changed to late_initcall_sync level.
So, this change isn't required anymore.
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
drivers/char/tpm/tpm_crb_ffa.c | 18 +++---------------
1 file changed, 3 insertions(+), 15 deletions(-)
diff --git a/drivers/char/tpm/tpm_crb_ffa.c b/drivers/char/tpm/tpm_crb_ffa.c
index 99f1c1e5644b..025c4d4b17ca 100644
--- a/drivers/char/tpm/tpm_crb_ffa.c
+++ b/drivers/char/tpm/tpm_crb_ffa.c
@@ -177,23 +177,13 @@ static int tpm_crb_ffa_to_linux_errno(int errno)
*/
int tpm_crb_ffa_init(void)
{
- int ret = 0;
-
- if (!IS_MODULE(CONFIG_TCG_ARM_CRB_FFA)) {
- ret = ffa_register(&tpm_crb_ffa_driver);
- if (ret) {
- tpm_crb_ffa = ERR_PTR(-ENODEV);
- return ret;
- }
- }
-
if (!tpm_crb_ffa)
- ret = -ENOENT;
+ return -ENOENT;
if (IS_ERR_VALUE(tpm_crb_ffa))
- ret = -ENODEV;
+ return -ENODEV;
- return ret;
+ return 0;
}
EXPORT_SYMBOL_GPL(tpm_crb_ffa_init);
@@ -405,9 +395,7 @@ static struct ffa_driver tpm_crb_ffa_driver = {
.id_table = tpm_crb_ffa_device_id,
};
-#ifdef MODULE
module_ffa_driver(tpm_crb_ffa_driver);
-#endif
MODULE_AUTHOR("Arm");
MODULE_DESCRIPTION("TPM CRB FFA driver");
--
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply related
* [RFC PATCH v2 1/4] security: ima: call ima_init() again at late_initcall_sync for defered TPM
From: Yeoreum Yun @ 2026-04-22 16:24 UTC (permalink / raw)
To: linux-security-module, linux-kernel, linux-integrity,
linux-arm-kernel, kvmarm
Cc: paul, jmorris, serge, zohar, roberto.sassu, dmitry.kasatkin,
eric.snowberg, jarkko, jgg, sudeep.holla, maz, oupton, joey.gouly,
suzuki.poulose, yuzenghui, catalin.marinas, will, noodles,
sebastianene, Yeoreum Yun
In-Reply-To: <20260422162449.1814615-1-yeoreum.yun@arm.com>
To generate the boot_aggregate log in the IMA subsystem with TPM PCR values,
the TPM driver must be built as built-in and
must be probed before the IMA subsystem is initialized.
However, when the TPM device operates over the FF-A protocol using
the CRB interface, probing fails and returns -EPROBE_DEFER if
the tpm_crb_ffa device — an FF-A device that provides the communication
interface to the tpm_crb driver — has not yet been probed.
To ensure the TPM device operating over the FF-A protocol with
the CRB interface is probed before IMA initialization,
the following conditions must be met:
1. The corresponding ffa_device must be registered,
which is done via ffa_init().
2. The tpm_crb_driver must successfully probe this device via
tpm_crb_ffa_init().
3. The tpm_crb driver using CRB over FF-A can then
be probed successfully. (See crb_acpi_add() and
tpm_crb_ffa_init() for reference.)
Unfortunately, ffa_init(), tpm_crb_ffa_init(), and crb_acpi_driver_init() are
all registered with device_initcall, which means crb_acpi_driver_init() may
be invoked before ffa_init() and tpm_crb_ffa_init() are completed.
When this occurs, probing the TPM device is deferred.
However, the deferred probe can happen after the IMA subsystem
has already been initialized, since IMA initialization is performed
during late_initcall, and deferred_probe_initcall() is performed
at the same level.
To resolve this, call ima_init() again at late_inicall_sync level
so that let IMA not miss TPM PCR value when generating boot_aggregate
log though TPM device presents in the system.
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
include/linux/lsm_hooks.h | 2 +
security/integrity/ima/ima.h | 4 +-
security/integrity/ima/ima_init.c | 10 +++-
security/integrity/ima/ima_main.c | 76 +++++++++++++++++++++++------
security/integrity/ima/ima_policy.c | 3 ++
security/lsm_init.c | 13 ++++-
6 files changed, 87 insertions(+), 21 deletions(-)
diff --git a/include/linux/lsm_hooks.h b/include/linux/lsm_hooks.h
index d48bf0ad26f4..88fe105b7f00 100644
--- a/include/linux/lsm_hooks.h
+++ b/include/linux/lsm_hooks.h
@@ -166,6 +166,7 @@ enum lsm_order {
* @initcall_fs: LSM callback for fs_initcall setup, optional
* @initcall_device: LSM callback for device_initcall() setup, optional
* @initcall_late: LSM callback for late_initcall() setup, optional
+ * @initcall_late_sync: LSM callback for late_initcall_sync() setup, optional
*/
struct lsm_info {
const struct lsm_id *id;
@@ -181,6 +182,7 @@ struct lsm_info {
int (*initcall_fs)(void);
int (*initcall_device)(void);
int (*initcall_late)(void);
+ int (*initcall_late_sync)(void);
};
#define DEFINE_LSM(lsm) \
diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h
index 89ebe98ffc5e..75ee7ad184d0 100644
--- a/security/integrity/ima/ima.h
+++ b/security/integrity/ima/ima.h
@@ -62,6 +62,8 @@ extern int ima_hash_algo_idx __ro_after_init;
extern int ima_extra_slots __ro_after_init;
extern struct ima_algo_desc *ima_algo_array __ro_after_init;
+extern bool ima_initialised __ro_after_init;
+
extern int ima_appraise;
extern struct tpm_chip *ima_tpm_chip;
extern const char boot_aggregate_name[];
@@ -257,7 +259,7 @@ static inline void ima_measure_kexec_event(const char *event_name) {}
extern bool ima_canonical_fmt;
/* Internal IMA function definitions */
-int ima_init(void);
+int ima_init(bool late);
int ima_fs_init(void);
int ima_add_template_entry(struct ima_template_entry *entry, int violation,
const char *op, struct inode *inode,
diff --git a/security/integrity/ima/ima_init.c b/security/integrity/ima/ima_init.c
index a2f34f2d8ad7..c28c71090ad2 100644
--- a/security/integrity/ima/ima_init.c
+++ b/security/integrity/ima/ima_init.c
@@ -115,13 +115,19 @@ void __init ima_load_x509(void)
}
#endif
-int __init ima_init(void)
+int __init ima_init(bool late)
{
int rc;
ima_tpm_chip = tpm_default_chip();
- if (!ima_tpm_chip)
+ if (!ima_tpm_chip) {
+ if (!late) {
+ pr_info("Defer initialisation to the late_initcall_sync stage.\n");
+ return -EPROBE_DEFER;
+ }
+
pr_info("No TPM chip found, activating TPM-bypass!\n");
+ }
rc = integrity_init_keyring(INTEGRITY_KEYRING_IMA);
if (rc)
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index 1d6229b156fb..ac444ee600e2 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -38,6 +38,7 @@ int ima_appraise;
#endif
int __ro_after_init ima_hash_algo = HASH_ALGO_SHA1;
+bool ima_initialised __ro_after_init = false;
static int hash_setup_done;
static int ima_disabled __ro_after_init;
@@ -1237,6 +1238,35 @@ static int ima_kernel_module_request(char *kmod_name)
#endif /* CONFIG_INTEGRITY_ASYMMETRIC_KEYS */
+static int __init init_ima_core(bool late)
+{
+ int err;
+
+ if (ima_initialised)
+ return 0;
+
+ err = ima_init(late);
+ if (err == -EPROBE_DEFER)
+ return 0;
+
+ if (err && strcmp(hash_algo_name[ima_hash_algo],
+ CONFIG_IMA_DEFAULT_HASH) != 0) {
+ pr_info("Allocating %s failed, going to use default hash algorithm %s\n",
+ hash_algo_name[ima_hash_algo], CONFIG_IMA_DEFAULT_HASH);
+ hash_setup_done = 0;
+ hash_setup(CONFIG_IMA_DEFAULT_HASH);
+ err = ima_init(late);
+ }
+
+ if (!err) {
+ ima_update_policy_flags();
+ ima_initialised = true;
+ } else
+ ima_disabled = 1;
+
+ return err;
+}
+
static int __init init_ima(void)
{
int error;
@@ -1250,30 +1280,42 @@ static int __init init_ima(void)
ima_appraise_parse_cmdline();
ima_init_template_list();
hash_setup(CONFIG_IMA_DEFAULT_HASH);
- error = ima_init();
-
- if (error && strcmp(hash_algo_name[ima_hash_algo],
- CONFIG_IMA_DEFAULT_HASH) != 0) {
- pr_info("Allocating %s failed, going to use default hash algorithm %s\n",
- hash_algo_name[ima_hash_algo], CONFIG_IMA_DEFAULT_HASH);
- hash_setup_done = 0;
- hash_setup(CONFIG_IMA_DEFAULT_HASH);
- error = ima_init();
- }
-
- if (error)
- return error;
error = register_blocking_lsm_notifier(&ima_lsm_policy_notifier);
- if (error)
+ if (error) {
pr_warn("Couldn't register LSM notifier, error %d\n", error);
+ goto disable_ima;
+ }
- if (!error)
- ima_update_policy_flags();
+ error = init_ima_core(false);
+ if (error) {
+ unregister_blocking_lsm_notifier(&ima_lsm_policy_notifier);
+ goto disable_ima;
+ }
+
+ return 0;
+disable_ima:
+ ima_disabled = 1;
return error;
}
+static int __init late_init_ima(void)
+{
+ int err;
+
+ if (ima_disabled)
+ return 0;
+
+ err = init_ima_core(true);
+ if (err) {
+ unregister_blocking_lsm_notifier(&ima_lsm_policy_notifier);
+ ima_disabled = 1;
+ }
+
+ return err;
+}
+
static struct security_hook_list ima_hooks[] __ro_after_init = {
LSM_HOOK_INIT(bprm_check_security, ima_bprm_check),
LSM_HOOK_INIT(bprm_creds_for_exec, ima_bprm_creds_for_exec),
@@ -1321,4 +1363,6 @@ DEFINE_LSM(ima) = {
.blobs = &ima_blob_sizes,
/* Start IMA after the TPM is available */
.initcall_late = init_ima,
+ /* Start IMA late in case of probing TPM is deferred. */
+ .initcall_late_sync = late_init_ima,
};
diff --git a/security/integrity/ima/ima_policy.c b/security/integrity/ima/ima_policy.c
index bf2d7ba4c14a..c3bcc3521c81 100644
--- a/security/integrity/ima/ima_policy.c
+++ b/security/integrity/ima/ima_policy.c
@@ -501,6 +501,9 @@ static void ima_lsm_update_rules(void)
int ima_lsm_policy_change(struct notifier_block *nb, unsigned long event,
void *lsm_data)
{
+ if (!ima_initialised)
+ return NOTIFY_DONE;
+
if (event != LSM_POLICY_CHANGE)
return NOTIFY_DONE;
diff --git a/security/lsm_init.c b/security/lsm_init.c
index 573e2a7250c4..4e5c59beb82a 100644
--- a/security/lsm_init.c
+++ b/security/lsm_init.c
@@ -547,13 +547,22 @@ device_initcall(security_initcall_device);
* security_initcall_late - Run the LSM late initcalls
*/
static int __init security_initcall_late(void)
+{
+ return lsm_initcall(late);
+}
+late_initcall(security_initcall_late);
+
+/**
+ * security_initcall_late_sync - Run the LSM late initcalls sync
+ */
+static int __init security_initcall_late_sync(void)
{
int rc;
- rc = lsm_initcall(late);
+ rc = lsm_initcall(late_sync);
lsm_pr_dbg("all enabled LSMs fully activated\n");
call_blocking_lsm_notifier(LSM_STARTED_ALL, NULL);
return rc;
}
-late_initcall(security_initcall_late);
+late_initcall_sync(security_initcall_late_sync);
--
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply related
* [RFC PATCH v2 0/4] fix FF-A call failed with pKVM when ff-a driver is built-in
From: Yeoreum Yun @ 2026-04-22 16:24 UTC (permalink / raw)
To: linux-security-module, linux-kernel, linux-integrity,
linux-arm-kernel, kvmarm
Cc: paul, jmorris, serge, zohar, roberto.sassu, dmitry.kasatkin,
eric.snowberg, jarkko, jgg, sudeep.holla, maz, oupton, joey.gouly,
suzuki.poulose, yuzenghui, catalin.marinas, will, noodles,
sebastianene, Yeoreum Yun
commit 0e0546eabcd6 ("firmware: arm_ffa: Change initcall level of ffa_init() to rootfs_initcall")
changed the initcall level of ffa_init() to rootfs_initcall to address
an issue where IMA could not properly recognize the TPM device
when FF-A driver is built as built-in.
However, this introduces another problem: pKVM fails to handle FF-A calls
because it cannot trap the FFA_VERSION call invoked by ffa_init().
To ensure the TPM device is recognized when present in the system,
it is preferable to invoke again ima_init() at a later stage.
Deferred probing is resolved by deferred_probe_initcall(),
which runs at the late_initcall level.
Therefore, introduce an LSM initcall at late_initcall_sync and
invode ima_init() again to this level in case of TPM is probed yet
at the late_initcall stage.
With this change, revert the initcall level of ffa_init() back to
device_initcall. Additionally, to handle the case where ffa_init() runs
before kvm_init(), check whether pKVM has been initialized during ffa_init().
If not, defer initialization to prevent failures of FF-A calls
due to the inability to trap FFA_VERSION and FFA_RXTX_MAP in pKVM.
This patch is based on v7.0
Patch History
=============
from v1 to v2:
- add notifier to make ffa-driver pkvm initialised.
- modify to try initailisation again when IMA coudln't find proper TPM device.
- https://lore.kernel.org/all/20260417175759.3191279-1-yeoreum.yun@arm.com/#t
Yeoreum Yun (4):
security: ima: call ima_init() again at late_initcall_sync for defered
TPM
tpm: tpm_crb_ffa: revert defered_probed when tpm_crb_ffa is built-in
firmware: arm_ffa: revert ffa_init() initcall level to device_initcall
firmware: arm_ffa: check pkvm initailised when initailise ffa driver
arch/arm64/include/asm/virt.h | 11 +++++
arch/arm64/kvm/arm.c | 21 ++++++++
arch/arm64/kvm/pkvm.c | 2 +
drivers/char/tpm/tpm_crb_ffa.c | 18 ++-----
drivers/firmware/arm_ffa/common.h | 4 +-
drivers/firmware/arm_ffa/driver.c | 38 ++++++++++++++-
drivers/firmware/arm_ffa/smccc.c | 2 +-
include/linux/lsm_hooks.h | 2 +
security/integrity/ima/ima.h | 4 +-
security/integrity/ima/ima_init.c | 10 +++-
security/integrity/ima/ima_main.c | 76 +++++++++++++++++++++++------
security/integrity/ima/ima_policy.c | 3 ++
security/lsm_init.c | 13 ++++-
13 files changed, 163 insertions(+), 41 deletions(-)
base-commit: 028ef9c96e96197026887c0f092424679298aae8
--
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply
* Re: [PATCH] iommu/arm-smmu-v3: Allow disabling Stage 1 translation
From: Jason Gunthorpe @ 2026-04-22 16:23 UTC (permalink / raw)
To: Evangelos Petrongonas
Cc: Will Deacon, Robin Murphy, Joerg Roedel, Nicolin Chen,
Pranjal Shrivastava, Lu Baolu, linux-arm-kernel, iommu,
linux-kernel, nh-open-source, Zeev Zilberman
In-Reply-To: <20260422064431.GA49867@dev-dsk-epetron-1c-1d4d9719.eu-west-1.amazon.com>
On Wed, Apr 22, 2026 at 06:44:31AM +0000, Evangelos Petrongonas wrote:
> The motivation is live update of the hypervisor: we want to kexec into a
> new kernel while keeping DMA from passthrough devices flowing, which
> means the SMMU's translation state has to survive the handover. The Live
> Update Orchestrator work [1] and the in-progress "iommu: Add live
> update state preservation" series [2] are building exactly this plumbing
> on top of KHO; [2]'s cover letter calls out Arm SMMUv3 support as future
> work, and an earlier RFC from Amazon [3] sketched the same idea for
> iommufd.
It would be appropriate to keep this patch with the rest of that out
of tree pile, for example in the series that enables s2 only support
in smmuv3.
> For this use case, Stage 2 is materially easier to persist than Stage 1,
> for structural rather than performance reasons:
I don't think so. The driver needs to know each and every STE that
will survive KHO. The ones that don't survive need to be reset to
abort STEs. From that point it is trivial enough to include the CD
memory in the preservation.
It would help to send a preparation series to switch the ARM STE and
CD logic away from dma_alloc_coherent and use iommu-pages instead,
since we only expect iommu-pages to support preservation..
I could maybe see only supporting non-PASID as a first-series, but a
CD table with SSID 0 only populated is still pretty trivial.
Jason
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox