[PATCH v1 00/17] crypto: Standalone crypto module (Series 1/4): Core implementation

[PATCH v1 00/17] crypto: Standalone crypto module (Series 1/4): Core implementation

[Jay Wang]

This feature is organized into four patch series for submission to the mainline (up to the "Merge tag 'landlock-7.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/mic/linux"):
- Patch Series 1/4: "[PATCH v1 0...17] crypto: Standalone crypto module (Series 1/4): Core implementation"
- Patch Series 2/4: "[PATCH v1 0...106] crypto: Standalone crypto module (Series 2/4): Arch-independent crypto"
- Patch Series 3/4: "[PATCH v1 0...21] crypto: Standalone crypto module (Series 3/4): x86_64 crypto"
- Patch Series 4/4: "[PATCH v1 0...12] crypto: Standalone crypto module (Series 4/4): arm64 crypto"

The full source code is available at https://github.com/amazonlinux/linux/tree/fips-kernel-module.
Details on this feature and what each patch series covers can be found below.

## 1. Introduction

Amazon Linux is releasing a new kernel feature that converts the previously built-in kernel crypto subsystem into a standalone kernel module. This module becomes the carrier of the kernel crypto subsystem and can be loaded at early boot to provide the same functionality as the original built-in crypto. The primary motivation for this modularization is to streamline Federal Information Processing Standards (FIPS) validation, a critical cryptographic certification for cloud computing users doing business with the U.S. government.
 
In a bit more detail, previously, FIPS certification was tied to the entire kernel image, meaning non-crypto updates could potentially invalidate certification. With this feature, FIPS certification is tied only to the crypto module. Therefore, once the module is certified, loading this certified module on newer kernels automatically makes those kernels FIPS-certified. As a result, this approach can save re-certification costs and 12-18 months of waiting time by reducing the need for repeated FIPS re-certification cycles.

This feature is organized into four patch series:
- Patch Series 1 - Core feature implementation
- Patch Series 2 - Architecture-independent crypto: Modularize built-in crypto under `crypto/`
- Patch Series 3 - x86_64 crypto: Modularize built-in crypto under `arch/x86/crypto/`
- Patch Series 4 - arm64 crypto: Modularize built-in crypto under `arch/arm64/crypto/`

This document provides technical details on how this feature is designed and implemented for users or developers who are interested in developing upon it, and is organized as follows:
- Section 2 - Getting Started: Quick start on how to enable the feature
- Section 3 - Workflow Overview: Changes this feature brings to build and runtime
- Section 4 - Design Implementation Details: Technical deep-dive into each component
- Section 5 - Customizing and Extending Crypto Module: How to select crypto to be included and extend to new crypto/architectures
- Section 6 - Related Work and Comparison
- Section 7 - Summary

## 2. Getting Started

This section provides a quick start guide for developers on how to enable, compile and use the standalone cryptography module feature.

### 2.1 Basic Configuration

The feature is controlled by a single configuration option:
```
CONFIG_CRYPTO_FIPS140_EXTMOD=y
```
What it does: When enabled, automatically redirects a set of cryptographic algorithms from the main kernel into a standalone module `crypto/fips140/fips140.ko`. The cryptographic algorithms that are redirected need to satisfy all the following conditions, otherwise the cryptography will remain in its original form:
1. Must be configured as built-in (i.e., `CONFIG_CRYPTO_*=y`). This means cryptography already configured as modular (i.e., `CONFIG_CRYPTO_*=m`) are not redirected as they are already modularized.
2. Must be among a list, which can be customized by developers as described in Section 5.

When disabled, the kernel behaves as before.

### 2.2 Build Process

Once `CONFIG_CRYPTO_FIPS140_EXTMOD=y` is set, no additional steps are required. The standalone module will be built automatically as part of the standard kernel build process:
```
make -j$(nproc)
# or
make vmlinux
```
**What happens automatically (No user action required):**
1. Build the module as `crypto/fips140/fips140.ko`
2. The cryptography module will be loaded at boot time
3. All kernel cryptographic services will provide the same functionality as before (i.e., prior to introducing this new feature) once boot completes.

### 2.3 Advanced Configuration Options

**Using External Cryptography Module:**
```
CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE=y
```
By default, `CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE` is not set, meaning the freshly built cryptography module is used. Otherwise, the pre-built standalone cryptography module from `fips140_build/crypto/fips140/fips140.ko` and modular cryptography such as `fips140_build/crypto/aes.ko` (need to manually place pre-built modules in these locations before the build) are included in kernel packaging (e.g., during `make modules_install`) and are used at later boot time.

**Dual Version Support:**
```
CONFIG_CRYPTO_FIPS140_DUAL_VERSION=y
```
Encapsulate two versions of `fips140.ko` into kernel: one is freshly built for non-FIPS mode usage, another is pre-built specified by `fips140_build/crypto/fips140/fips140.ko` for FIPS mode usage. The appropriate version is selected and loaded at boot time based on boot time FIPS mode status.

### 2.4 Verification

To verify the feature is working, after install and boot with the new kernel:
```
# Check if fips140.ko module is loaded
lsmod | grep fips140
```

## 3. Workflow Overview

This section provides an overview without delving into deep technical details of the changes the standalone cryptography module feature introduces. When this feature is enabled, it introduces changes to both the kernel build and booting process. 

3.1 Build-Time Changes

Kernel cryptography subsystem consists of both cryptography management infrastructure (e.g., `crypto/api.c`, `crypto/algapi.c`, etc), along with hundreds of different cryptography algorithms (e.g., `crypto/arc4.c`).

**Traditional Build Process:**
Traditionally, cryptography management infrastructure are always built-in to the kernel, while cryptographic algorithms can be configured to be built either as built-in (`CONFIG_CRYPTO_*=y`) or as separate modular (`CONFIG_CRYPTO_*=m`) `.ko` file depending on kernel configuration:
As a result, the builtin cryptography management infrastructure and cryptographic algorithms are statically linked into the kernel binary:
```
cryptographic algorithms source files → compiled as .o objfiles →  linked into vmlinux → single kernel binary
```
**With Standalone Cryptography Module:**
This feature automatically transforms the builtin cryptographic components into a standalone cryptography module, `fips140.ko`. To do so, it develops a new kernel build rule `crypto-objs-$(CONFIG_CRYPTO_*)` such that, once this build rule is applied to a cryptographic algorithm, such cryptographic algorithm will be automatically collected into the cryptography module if it is configured as built-in (i.e, `CONFIG_CRYPTO_*=y`), for example:
```
// in crypto/asymmetric_keys/Makefile
- obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
+ crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
```
Such build change allows the modularization transformation to only affect selected cryptographic algorithms (i.e, where the `crypto-objs-$(CONFIG_CRYPTO_*`) is applied).

Then, after the `fips140.ko` is generated, it will be embedded back into main kernel vmlinux as a replacement part. The purpose of this embedding, instead of traditionally putting the `fips140.ko` into filesystem, is a preparation to allow the module to be loaded early enough even before the filesystem is ready.

The new build process is illustrated below.
```
cryptographic algorithms source files → compiled as .o objfiles → automatically collected and linked into fips140.ko → embedded fips140.ko into vmlinux as a replaceable binary
```

### 3.2 Runtime Changes

**Traditional Boot Process:**
The kernel initializes the cryptographic subsystem early during boot, executing each cryptographic initialization routine accordingly. These initialization routines may depend on other cryptographic components or other kernel subsystems, so their invocation follows a well-defined execution order to ensure they are initialized before their first use.
```
kernel starts → cryptography subsystem initialization → cryptography subsystem available → other components use cryptography
```
**With Standalone Cryptography Module:**
At the start of kernel boot, compared to a regular kernel, the first major change introduced by this feature is that no cryptography services are initially available — since the entire cryptography subsystem has been decoupled from the main kernel.
To ensure that the cryptography subsystem becomes available early enough (before the first kernel component that requires cryptography services), the standalone cryptography kernel module must be loaded at a very early stage, even before the filesystem becomes available.

However, the regular module loading mechanism relies on placing kernel modules in the filesystem and loading them from there, which creates a chicken-and-egg problem — the cryptography module cannot be loaded until the filesystem is ready, yet some kernel components may require cryptography services even before that point.

To address this, the second change introduced by this feature is that the cryptography kernel module is loaded directly from memory, leveraging the earlier compilation changes that embed the module binary into the main kernel image. Afterward, the feature includes a “plug-in” mechanism that connects the decoupled cryptography subsystem back to the main kernel, ensuring that kernel cryptography users can correctly locate and invoke the cryptography routine entry points.

Finally, to ensure proper initialization, the feature guarantees that all cryptography algorithms and the cryptography management infra execute their initialization routines in the exact same order as they would if they were built-in.

The process described above is illustrated below.
```
kernel starts → no cryptography available → load fips140.ko from memory → plug cryptography back to kernel → module initialization → cryptographic services available → other components use cryptography
```

## 4. Design Implementation Details

While the earlier sections provide a holistic view of how this feature shapes the kernel, this section provides deeper design details on how these functionalities are realized. There are three key design components:
1. A specialized compile rule that automatically compiles and collects all built-in cryptographic algorithm object files to generate the final module binary under arbitrary kernel configurations, and then embeds the generated binary into the main kernel image for early loading.
2. A mechanism to convert interactions between the cryptography subsystem and the main kernel into a pluggable interface.
3. A module loading and initialization process that ensures the cryptography subsystem is properly initialized as if it were built-in.

### 4.1. Specialized Compilation System

**Automatic Collection and Linking of Built-in Cryptographic Algorithm Objects:**
The first step in generating the `fips140.ko` module is to compile and collect built-in cryptographic components (i.e., those specified by `CONFIG_CRYPTO_*=y`).
Traditionally, the existing module build process requires all module components (e.g., source files) to reside in a single directory. However, this approach is not suitable for our case, where hundreds of cryptographic algorithm source files are scattered across multiple directories.

A naïve approach would be to create a separate Makefile that duplicates the original build rules with adjusted paths.
However, this method is not scalable due to the large number of cryptographic build rules, many of which are highly customized and can vary under different Kconfig settings, making such a separate Makefile even more complex.
Moreover, this approach cannot ensure that built-in cryptographic algorithms are completely removed from the main kernel, which would result in redundant cryptographic code being included in both the kernel and the module.

To tackle this challenge, we automated the object collection and linking process by introducing special build logic for the kernel cryptography subsystem.
Specifically, to automatically collect cryptography object files while preserving their original compilation settings (such as flags, headers, and paths), we introduced a new compilation rule:
```
crypto-objs-y += *.o
```
This replaces the original `obj-y += *.o` rule in cryptography Makefiles later, for example:
```
// in crypto/asymmetric_keys/Makefile
- obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
+ crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
asymmetric_keys-y := \    asymmetric_type.o \    restrict.o \    signature.o
```
in the cryptography subsystem Makefiles, allowing most of the existing Makefile logic to be reused.
As a result, when the standalone cryptography module feature is enabled, any cryptographic algorithm configured as built-in (for example, `crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o` where `CONFIG_ASYMMETRIC_KEY_TYPE=y`) will be automatically collected and linked into a single final object binary, `fips140.o`.
During this process, a special compilation flag (`-DFIPS_MODULE=1`) is applied to instruct each object file to be compiled in a module-specific manner. This flag will later be used to generate the pluggable interface on both the main kernel side and the module side from the same source code.

The implementation details are as follows: it follows a similar methodology used by the `obj-y` collection process for building `vmlinux.o`. The `crypto-objs-y` rule is placed in `scripts/Makefile.build`, which is executed by each directory Makefile to collect the corresponding crypto object files. Each directory then creates a `crypto-module.a` archive that contains all `crypto-objs-y += <object>.o` files under that directory. In the parent directories, these `crypto-module.a` archives are recursively included into the parent’s own `crypto-module.a`, and this process continues upward until the final `fips140.o` is generated.

**A Separate Module Generation Pipeline for Building the Final Kernel Module from Linked Cryptographic Algorithm Object:**
With the linked cryptographic algorithm object (i.e., `fips140.o`), the next step is to generate the final kernel module, `fips140.ko`.

A direct approach would be to inject the `fips140.ko` module build into the existing modules generation pipeline (i.e., `make modules`) by providing our pre-generated `fips140.o`. However, we choose not to do this because it would create a circular make rule dependency (which is invalid in Makefiles and causes build failures), resulting in mutual dependencies between the modules and vmlinux targets (i.e., `modules:vmlinux` and `vmlinux:modules` at the same time).
This happens for the following reasons:
1. Since we will later embed `fips140.ko` into the final kernel image (as described in the next section), we must make vmlinux depend on `fips140.ko`. In other words: `vmlinux: fips140.ko`.
2. When the kernel is built with `CONFIG_DEBUG_INFO_BTF_MODULES=y`, it requires: modules: vmlinux. This is because `CONFIG_DEBUG_INFO_BTF_MODULES=y` takes vmlinux as input to generate BTF info for the module, and inserts such info into the `.ko` module by default.
3. If we choose to inject `fips140.ko` into make modules, this would create a make rule dependency: `fips140.ko: modules`. Combined with items 1 and 2, this eventually creates an invalid circular dependency between vmlinux and modules.

Due to these reasons, the design choice is to use a separate make pipeline (defined as `fips140-ready` in the Makefile). This new pipeline reuses the same module generation scripts used by make modules but adds additional logic in `scripts/Makefile.{modfinal|modinst|modpost}` and `scripts/mod/modpost.c` to handle module symbol generation and verification correctly. 

**A Seamless Process That Embeds the Generated Binary Into the Main Kernel Image for Early Loading:**
As mentioned earlier, in order to load the standalone cryptography module early in the boot process—before the filesystem is ready—the module binary must be embedded into the final kernel image (i.e., vmlinux) so that it can be loaded directly from memory.
We intend for this embedding process to be completely seamless and automatically triggered whenever vmlinux is built (i.e., during `make vmlinux`).

To achieve this, the feature adds a Make dependency rule so that vmlinux depends on `fips140.ko`.
It also modifies the vmlinux link rules (i.e., `arch/<arch>/kernel/vmlinux.lds.S`, `scripts/Makefile.vmlinux`, and `scripts/link-vmlinux.sh`) so that the generated module binary is finally combined with `vmlinux.o`.

In addition, we allow multiple cryptography module binary versions (for example, a certified cryptography binary and a latest, up-to-date but uncertified one) to be embedded into the main kernel image to serve different user needs. This design allows regular (non-FIPS) users to benefit from the latest cryptographic updates, while FIPS-mode users continue to use the certified cryptography module.

To support this, we introduce an optional configuration, `CONFIG_CRYPTO_FIPS140_DUAL_VERSION`. When enabled, this option allows two cryptography module versions to be embedded within a single kernel build and ensures that the appropriate module is selected and loaded at boot time based on the system’s FIPS mode status.

### 4.2. Pluggable Interface Between the Built-in Cryptography Subsystem and the Main Kernel

Although the module binary (`fips140.ko`) has been embedded into the final kernel image (`vmlinux`) as described in the previous section, it is not linked to the kernel in any way. This is because `fips140.ko` is embedded in a data-only manner, so the main kernel cannot directly call any functions or access any data defined in the module binary.

Even worse, simply removing the built-in cryptographic algorithms without additional handling would cause the kernel to fail to compile, because traditionally, built-in cryptographic algorithms and the main kernel can interact only through functions and variables whose addresses they assume to know. As a result, even if they have been removed, kernel cryptography users still expect the symbol addresses of cryptographic routines and data to be available at compile time.

To address this, we introduce a pluggable interface between built-in cryptographic functions and variables by placing **address placeholders**. During runtime, once the cryptography kernel module is loaded, these placeholders are updated to the correct addresses. In the rest of this section, we first introduce this pluggable interface mechanism, and then explain how to apply it to the built-in cryptographic algorithms.

**The Pluggable Interface Mechanism:**
There are two types of address holders used to achieve this pluggable interface:
- Function addresses:  We use the “static call” mechanism. Static calls are a Linux mechanism that converts an “indirect call” into something with performance equivalent to a “direct call,” while avoiding the introduction of additional security concerns, such as control-flow–hijacking attack gadgets. We implement this function-address placeholder as the `DECLARE_STATIC_CALL()` and `DEFINE_CRYPTO_API_STUB()` wrappers.
- Variable addresses (the remaining smaller portion): For these, we use a pointer of the corresponding data type. We implement this address placeholder as the `DECLARE_CRYPTO_VAR()` and `DEFINE_CRYPTO_API_STUB()` wrappers:

These wrappers are applied to each symbol-exported (i.e., `EXPORT_SYMBOL()`) cryptographic function and variable (details on how to apply them are described later). Once applied, the wrappers are compiled differently for the main kernel and for the built-in cryptographic algorithm source code—acting as the “outlet” and the “plug,” respectively—using different compilation flags (`-DFIPS_MODULE`) introduced by our customized build rules described earlier.

As a result, the kernel can successfully compile even when the built-in cryptographic algorithms are removed, thanks to these address placeholders. At boot time, the placeholders initially hold NULL, but since no cryptography users exist at that stage, the kernel can still start booting correctly. After the cryptography module is loaded, the placeholders are dynamically updated to the correct addresses later (by `do_crypto_api()` and `do_crypto_var()`, described in later section).

In addition to these address placeholders, there is another important interaction point between the cryptography subsystem and the main kernel—the cryptographic initialization routines. Therefore, we also add a mechanism to collect all cryptographic initialization functions (e.g., those defined using `module_init()`) into a dedicated ELF section. This serves as preparation for the later module and cryptography-subsystem initialization steps described in subsequent sections.

**Applying the Pluggable Interface Mechanism to Cryptographic Algorithms:**

To apply these pluggable interface wrappers to a cryptographic algorithm and make them take effect, we follow the steps below (using `crypto/asymmetric_keys/asymmetric_type.c` as an example):
1. **Apply `crypto-objs-y` compile rule to the cryptographic algorithm:**
```
// in crypto/asymmetric_keys/Makefile
- obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
+ crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
asymmetric_keys-y := \    asymmetric_type.o \    restrict.o \    signature.o
```
2. **Locate the communication point between the cryptographic algorithm and the main kernel:**

The cryptography subsystem is designed such that most interactions between the main kernel and cryptographic algorithms occur through exported symbols using `EXPORT_SYMBOL()` wrappers.
This kernel design exists because most cryptographic algorithm implementations must support both built-in and modular modes. 

Consequently, the cryptographic functions and variables exported by `EXPORT_SYMBOL()` are a well-defined and identifiable interface between the cryptography subsystem and the main kernel: 
```
// in crypto/asymmetric_keys/asymmetric_type.c 
//Exported cryptographic function:
bool asymmetric_key_id_same(const struct asymmetric_key_id *kid1,
                const struct asymmetric_key_id *kid2) {...}
EXPORT_SYMBOL_GPL(asymmetric_key_id_same); 
//Exported cryptographic variable:
struct key_type key_type_asymmetric = {...};
EXPORT_SYMBOL_GPL(key_type_asymmetric); 
```
3. **Replace their declarations in the header file with the address-placeholder declaration wrappers:**
```
// in include/keys/asymmetric-type.h
// for exported cryptographic function:
- bool asymmetric_key_id_same const struct asymmetric_key_id *kid1, const struct asymmetric_key_id *kid2);
+ DECLARE_CRYPTO_API(CONFIG_ASYMMETRIC_KEY_TYPE, asymmetric_key_id_same, bool,
    (const struct asymmetric_key_id *kid1, const struct asymmetric_key_id *kid2),
    (kid1, kid2)); 
// for exported cryptographic variables:
- struct key_type key_type_asymmetric;
+ DECLARE_CRYPTO_VAR(CONFIG_ASYMMETRIC_KEY_TYPE, key_type_asymmetric, struct key_type, );
+ #if defined(CONFIG_CRYPTO_FIPS140_EXTMOD) && !defined(FIPS_MODULE) && IS_BUILTIN(CONFIG_ASYMMETRIC_KEY_TYPE)
+ #define key_type_asymmetric (*((struct key_type*)CRYPTO_VAR_NAME(key_type_asymmetric)))
+ #endif 
```
By replacing the original declarations with the address-placeholder declaration wrappers, we can automatically force all cryptography users to go through the placeholders, because those users already include the same header file.
The wrapper also takes the cryptographic algorithm Kconfig symbol as a parameter, so that when a cryptographic algorithm is built as a module (for example, `CONFIG_ASYMMETRIC_KEY_TYPE=m`), the original function declarations remain unchanged and are not affected.

4. **Add the address-placeholder definition wrappers into a dedicated file `fips140-api.c`:**

This file will be compiled separately and acts as both the “outlet” and the “plug” for the main kernel and the cryptography module, respectively:
```
// in crypto/fips140/fips140-api.c
+ #if IS_BUILTIN(CONFIG_ASYMMETRIC_KEY_TYPE)
+ #include <keys/asymmetric-type.h>
// for exported cryptographic function:
+ DEFINE_CRYPTO_API_STUB(asymmetric_key_id_same);
// for exported cryptographic variables:
+ #undef key_type_asymmetric
+ DEFINE_CRYPTO_VAR_STUB(key_type_asymmetric);
+ #endif
```

5. **Lastly, collect the cryptographic initialization routines for later module and cryptography-subsystem initialization by wrapping the original cryptographic initialization functions:**
```
// in crypto/asymmetric_keys/asymmetric_type.c 
- module_init(asymmetric_key_init);
- module_exit(asymmetric_key_cleanup); 
+ crypto_module_init(asymmetric_key_init);
+ crypto_module_exit(asymmetric_key_cleanup); 
```
We apply the above steps to both architecture-independent and architecture-specific cryptographic algorithms.

### 4.3. Initialization Synchronization

To ensure the embedded `fips140.ko` module binary provides the same cryptography functionality as the regular kernel, the kernel needs:
1. A module loader to load the module binary directly from memory,
2. A mechanism to plug the module back into the kernel by updating the address placeholders, and
3. Correct cryptography subsystem initialization, as if the cryptographic algorithms were still built-in.

**Directly Load Module Binary from Memory:**
Regular modules are loaded from the filesystem and undergo signature verification on the module binary, which relies on cryptographic operations. However, since we have already fully decoupled the cryptography subsystem, we must skip this step for this `fips140.ko` module.
To achieve this, we add a new loader function `load_crypto_module_mem()` that can load the module binary directly from memory at the designed address without checking the signature. Since the module binary is embedded into main kernel in an ELF section, as specified in the linker script:
```
// in arch/<arch>/kernel/vmlinux.lds.S
    .fips140_embedded : AT(ADDR(.fips140_embedded) - LOAD_OFFSET) {
        . = ALIGN(8);
        _binary_fips140_ko_start = .;
        KEEP(*(.fips140_module_data))
        _binary_fips140_ko_end = .;
    }
```
Therefore, the runtime memory address of the module can be accessed directly by the module loader to invoke the new loader function `load_crypto_module_mem()`.

**Plug Back the Module by Updating Address Placeholder Values:**
To update the address placeholders in the main kernel to the correct addresses matching the loaded module, after compilation the placeholders are placed into dedicated ELF sections `_crypto_api_keys` and `_crypto_var_keys`.
This can be seen from the definition of the placeholder-declaration wrappers:
```
#define DEFINE_CRYPTO_API_STUB(name) \    static struct crypto_api_key __##name##_key \        __used \        __section("__crypto_api_keys")  // Place in a dedicated ELF Section
        __aligned(__alignof__(struct crypto_api_key)) = \    { \        .key = &STATIC_CALL_KEY(crypto_##name##_key), \        .tramp = STATIC_CALL_TRAMP_ADDR(crypto_##name##_key), \        .func = &name, \    };

#define DEFINE_CRYPTO_VAR_STUB(name) \    static struct crypto_var_key __crypto_##name##_var_key \        __used \        __section("__crypto_var_keys")  // Place in a dedicated ELF Section
        __aligned(__alignof__(struct crypto_var_key)) = \    { \        .ptr = &CRYPTO_VAR_NAME(name), \        .var = (void*)&name, \    };
```
The purpose of doing this is to allow the main kernel to quickly locate the placeholders and update them to the correct addresses. The update functions are defined as `do_crypto_var()` and `do_crypto_api()`, which are executed at module load.

As a result, all cryptography users in the main kernel can now call the cryptographic functions as if they were built-in.

**Initialize Cryptography Subsystem as if it Were Built-in:**
Cryptographic components must be properly initialized before use, and this initialization is typically achieved through dedicated initialization functions (e.g., `module_init(crypto_init_func)` or `late_initcall(crypto_init_func)`). These functions often have strict execution order requirements and must run during the appropriate boot phase.

Therefore, for our standalone cryptography module feature, we must ensure that these decoupled “built-in” cryptographic algorithms are properly initialized and that their initialization order is preserved as before because failure to follow the correct order can result in kernel panic.

To address this, we introduce a synchronization mechanism between the main kernel and the module to ensure all cryptographic algorithms are executed in the correct kernel boot phase. In more details, we spawn the module initialization process `fips_loader_init()` as an async thread `fips140_sync_thread()`, in which we call `run_initcalls()` to execute the initialization calls of each cryptographic algorithm.
Then, we introduce synchronization helpers such as `wait_until_fips140_level_sync(int level)` to ensure the initialization order of all cryptographic algorithms is synchronized with the main kernel.

## 5. Customization and Extension of Cryptography Module

This section describes how developers can customize which cryptographic algorithms are included in the standalone cryptography module, as well as extend this feature to other cryptographic algorithms or hardware architectures.

### 5.1. Cryptography Selection Mechanism

The feature automatically includes cryptographic algorithms that meet specific criteria:
1. **Built-in Configuration**: Only cryptographic algorithms configured as `CONFIG_CRYPTO_*=y` are candidates for inclusion
2. **Explicit Inclusion**: Cryptographic algorithms must be explicitly converted using the `crypto-objs-$(CONFIG__CRYPTO_*`) build rule

### 5.2. Extend Support to New Cryptographic Algorithms

To extend support to a new cryptographic algorithm in the standalone module, follow these steps:

**Step 1: Update the Makefile**
```
# in crypto/[algorithm]/Makefile
- obj-$(CONFIG_CRYPTO_ALGORITHM) += algorithm.o
+ crypto-objs-$(CONFIG_CRYPTO_ALGORITHM) += algorithm.o
```
For Architecture-Specific Cryptographic Algorithms:
- Apply the `crypto-objs-` rule in the appropriate `arch/*/crypto/Makefile`

**Step 2: Add Pluggable Interface Support**
If the cryptographic algorithm exports symbols via `EXPORT_SYMBOL()`, add the pluggable interface wrappers:
```
# Example: in include/crypto/algorithm.h
- extern int crypto_algorithm_transform(struct crypto_tfm *tfm, const u8 *src, 
                                       u8 *dst, unsigned int len, u32 flags);
+ DECLARE_CRYPTO_API(CONFIG_CRYPTO_ALGORITHM, crypto_algorithm_transform, int,
    (struct crypto_tfm *tfm, const u8 *src, u8 *dst, unsigned int len, u32 flags),
    (tfm, src, dst, len, flags));
```
Then, add the corresponding stubs in `crypto/fips140/fips140-api.c`:
```
#if IS_BUILTIN(CONFIG_CRYPTO_ALGORITHM)
#include <crypto/algorithm.h>
DEFINE_CRYPTO_API_STUB(crypto_algorithm_transform);
#endif
```
For Architecture-Specific Cryptographic Algorithms:
- Include architecture-specific stubs in `arch/*/crypto/fips140/fips140-api.c`:
```
# Example: in arch/arm64/crypto/fips140/fips140-api.c
#if IS_BUILTIN(CONFIG_CRYPTO_AES_ARM64_CE)
#include <arch/arm64/crypto/aes-ce-setkey.h>
DEFINE_CRYPTO_API_STUB(ce_aes_setkey);
DEFINE_CRYPTO_API_STUB(ce_aes_expandkey);
#endif
```
**Step 3: Update Initialization**
Replace module initialization calls:
```
# in crypto/algorithm/algorithm.c
- module_init(algorithm_init);
- module_exit(algorithm_exit);
+ crypto_module_init(algorithm_init);
+ crypto_module_exit(algorithm_exit);
```

### 5.3. Architecture-Specific Extensions

**Extending to New Architectures:**
Currently supported architectures are x86_64 and ARM64. To extend this feature to additional architectures:
1. **Update Linker Scripts**: Add ELF sections in `arch/[new-arch]/kernel/vmlinux.lds.S`:
```
.fips140_embedded : AT(ADDR(.fips140_embedded) - LOAD_OFFSET) {
    . = ALIGN(8);
    _binary_fips140_ko_start = .;
    KEEP(*(.fips140_module_data))
    _binary_fips140_ko_end = .;
}
```
2. **Create Architecture-Specific Files**: Set up `arch/[new-arch]/crypto/fips140/` directory with Makefile and `fips140-api.c` following the pattern used in x86_64 and ARM64.

## 6. Related Work and Comparison

The idea of modularizing kernel cryptographic functionality has also attracted attention from other Linux distributions as well as Linux-kernel-based platforms that are not traditional distributions. Specifically, there are two related efforts: one from [Android's GKI kernel](https://source.android.com/docs/core/architecture/kernel/gki-fips140-module) and another from [Oracle Linux](https://git.kernel.org/pub/scm/linux/kernel/git/vegard/linux-fips140.git/log/?h=fips140). While Amazon Linux incorporated several valuable ideas from these efforts (and have acknowledged them in the patch commits—thank you again!), this section highlights the key differences between those approaches and this approach. The goal is to describe the trade-offs and design choices objectively, rather than to criticize other implementations.

### 6.1. Comparison with Android's GKI

Android's work is the earliest one on modularizing kernel cryptographic code, and it targets a non-intrusive approach to the core GKI kernel, with the goal of minimizing modifications to the kernel source. To achieve this, the crypto module relies on several interception or "hijacking" techniques that intervene in the core kernel execution path.

While this approach minimizes kernel code changes, we don't adopt such an approach for several reasons. First, these interception mechanisms tightly depend on internal kernel crypto subsystem behavior, making them fragile across major kernel updates thus less suitable to reuse the same module on newer major kernel versions. Second, this design requires substantial additional cryptographic code duplication, which impacts maintainability. Finally, the solution only supports a fixed set of cryptographic algorithms, making it non-general and difficult to extend.

In contrast, our design integrates directly into the Linux kernel source tree, avoids duplicated cryptographic implementations, supports arbitrary kernel configuration settings, and works with any chosen set of cryptographic algorithms.

### 6.2. Comparison with Oracle Linux

Oracle’s work was developed concurrently with this approach. The primary differences between Oracle’s approach and Amazon's lie in build integration, pluggable interface design, and module initialization.

**Build Integration:**
Oracle's module is implemented as an out-of-tree module with a separate Makefile. This introduces three major reasons we don't adopt such an approach: 

*First*, the separate Makefile duplicates many kernel build rules, which increases maintenance cost, as upstream kernel build changes must be tracked and replicated. One concrete example can be seen below:

in Oracle's module makefile
```
fips140-y += crypto/skcipher.o
fips140-y += crypto/lskcipher.o
```
However, in upstream, the corresponding build logic is more complex and configuration-dependent:
```
crypto_skcipher-y += lskcipher.o
crypto_skcipher-y += skcipher.o
obj-$(CONFIG_CRYPTO_SKCIPHER2) += crypto_skcipher.o
ifeq ($(CONFIG_BPF_SYSCALL),y)
obj-$(CONFIG_CRYPTO_SKCIPHER2) += bpf_crypto_skcipher.o
endif
```
As shown above, when `CONFIG_BPF_SYSCALL` is enabled, `bpf_crypto_skcipher.o` must also be included. Tracking such dependencies is hard in the duplicated Makefile approach. In contrast, our approach integrates seamlessly into the kernel build system by introducing a customized build rule (`crypto-objs-*`) rather than relying on a duplicated Makefile, such that this is handled correctly by reusing the existing kernel build logic:
```
crypto_skcipher-y += lskcipher.o
crypto_skcipher-y += skcipher.o
- obj-$(CONFIG_CRYPTO_SKCIPHER2) += crypto_skcipher.o
+ crypto-objs-$(CONFIG_CRYPTO_SKCIPHER2) += crypto_skcipher.o
ifeq ($(CONFIG_BPF_SYSCALL),y)
- obj-$(CONFIG_CRYPTO_SKCIPHER2) += bpf_crypto_skcipher.o
+ crypto-objs-$(CONFIG_CRYPTO_SKCIPHER2) += bpf_crypto_skcipher.o
endif
```
As a result, such a Makefile-duplication approach does not scale well across all kernel configurations and does not easily support arbitrary sets of cryptographic algorithms.

*Second*, since the module is to be embedded as part of the kernel image (i.e., `vmlinux`) as described earlier, the module build must be triggered automatically as part of the `vmlinux` build process to achieve a seamless build workflow. However, Oracle's module build is not tightly integrated into the kernel build framework and requires special build commands (e.g., first do `make M=fips140/` specifically, then do some shell command and finally `make`). 

In contrast, our approach improves this aspect by integrating the module build tightly into the regular kernel build, so the build process is seamless and automatic with regular build and packaging processes such as `make` or `make vmlinux` or `make install`.

**Pluggable Interface:**
There are several differences in the pluggable interface design.

*First*, we avoid duplicate crypto code so only keep one crypto code in kernel memory, while existing work keeps two crypto code even if these crypto code are from the same source code. This is due to the way Oracle defines pluggable interface macros in `crypto/api`, where its design requires some cryptographic code to remain compiled into the main kernel image in addition to the code inside the standalone cryptography module. Keeping two crypto code is ok if these code are different and designed to be used for different runtime modes (i.e., FIPS/non-FIPS mode), but will be unnecessary if both crypto code are the same.

In contrast, the approach we use can flexibly support both choices: keep one cryptography subsystem, or two different crypto subsystems. To do so, we introduce an option `CRYPTO_FIPS140_DUAL_VERSION` such that when it is disabled, we only keep one cryptographic subsystem in the cryptography module while completely removing it from the main kernel; and when it is enabled, we allow having two different modules carrying different cryptography for different kernel runtime modes (i.e., FIPS and non-FIPS mode).

*Second*, existing approach requires modifications to both the cryptographic implementation (.c) files and the declaration (.h) header files while our approach only requires modifying the header file, making the change less intrusive to the kernel codebase.

*Third*, prior approaches mainly support making cryptographic function calls pluggable, while our approach extends pluggability to cryptographic variables as well.

*Fourth*, prior approach requires all cryptographics that we care (for any purpose such as those within FIPS boundary) to be included within a single kernel module `fips140.ko` (e.g., when `CONFIG_CRYPTO_AES=m`, it cannot be `aes.ko` but must be within fips140.ko). However, this requirement limits the inherent benefit of a kernel module (i.e., on-demand loading for memory efficiency). In contrast, our approach allows the cryptographic we care remain its original modular if it is configured as being so (i.e., if `CONFIG_CRYPTO_AES=m`, the aes will still be as `aes.ko` but not forced to `fips140.ko`) up to the `.config` setting. One benefit of this design is that it does not impose strict requirements on `.config` setting (i.e., a cryptography `.config` can be set to both `=y|m` while existing work must be set as `=y`), preserving configuration flexibility.

To support so, for any cryptography within the interest (i.e., whose makerule has been replaced with `crypto-objs-*`) but configured as build-as-module (i.e., `CONFIG_CRYPTO_*=m`), its compiled `.ko` binary will be marked automatically, such that the loader will have a way to recognize to perform some interest-specific processing (e.g., registered as FIPS-required flag) if needed. And the pluggable interface can also adjust its coverage automatically based on different `CONFIG_CRYPTO_*=y|=m` settings. This is achieved by letting the pluggable interface macro to take `CONFIG_CRYPTO_*` option as a parameter to recognize the `.config` setting.

**Module Initialization:**
Oracle's initialization routine does not guarantee preservation of the original crypto initialization order (i.e., the order they should follow if they were originally built-in in the main kernel), which limits its ability to support arbitrary combinations of cryptographic algorithms. This is because the crypto initialization routine in the module is executed too early, such that all module crypto is initialized before the cryptography init in the main kernel. So if there is a crypto in the module (e.g., a crypto init defined as `late_init()`) that depends on a cryptography (whose init is defined as `module_init()`) in the main kernel, since the one in the main kernel should be executed earlier (but because the module init is too early, it makes the crypto in the main kernel executed too late), such a case will break the kernel boot process.

Our design, on the other hand, introduces explicit initialization synchronization mechanisms between cryptography's init routine in the module and in the main kernel that can preserve the original built-in initialization order. As a result, our module supports any chosen crypto set to be included in the module.

### 6.3. Comparison Summary

Overall, combined with differences in coding style and integration strategy, the proposed approach is more seamlessly integrated with the upstream Linux kernel, making it more generalizable across different kernel configuration settings, and the changed behavior more invisible to kernel users.

## 7. Summary
In this patch series, Amazon Linux proposes a new kernel feature that decouples the built-in crypto subsystem into a dedicated kernel module. To achieve this, several key mechanisms are designed, including specialized compile rules, a novel pluggable interface mechanism, and a module-loading initialization process. This feature is designed in an upstream-friendly manner so that it can support arbitrary kernel configuration settings and arbitrary chosen sets of cryptographic algorithms. It is planned to be officially launched with the Amazon Linux Kernel 6.18 and future kernels.

---
Written by Jay Wang <wanjay@amazon.com>, Amazon Linux


Jay Wang (15):
  crypto: add Kconfig options for standalone crypto module
  crypto: add module entry for standalone crypto kernel module
  build: special compilation rule for building the standalone crypto
    module
  build: Add ELF marker for crypto-objs-m modules
  crypto: add pluggable interface for builtin crypto modules
  crypto: dedicated ELF sections for collected crypto initcalls
  crypto: fips140: add crypto module loader
  build: embed the standalone crypto module into vmlinux
  build: add CONFIG_DEBUG_INFO_BTF_MODULES support for the standalone
    crypto kernel module
  Allow selective crypto module loading at boot based on FIPS mode
  Execute crypto initcalls during module initialization
  crypto: fips140: add module integrity self-check
  x86: crypto: to convert exported crypto symbols into pluggable
    interface for x86 cryptos
  arm64: crypto: to convert exported crypto symbols into pluggable
    interface for arm64 cryptos
  Add standalone crypto kernel module technical documentation

Vegard Nossum (2):
  module: allow kernel module loading directly from memory
  crypto/algapi.c: skip crypto_check_module_sig() for the standalone
    crypto module

 Makefile                                      |  68 ++-
 arch/arm64/crypto/Makefile                    |   3 +
 arch/arm64/crypto/fips140/Makefile            |  14 +
 arch/arm64/crypto/fips140/fips140-api.c       |   0
 arch/arm64/kernel/vmlinux.lds.S               |  40 ++
 arch/x86/crypto/Makefile                      |   3 +
 arch/x86/crypto/fips140/Makefile              |  14 +
 arch/x86/crypto/fips140/fips140-api.c         |   0
 arch/x86/kernel/vmlinux.lds.S                 |  40 ++
 crypto/Kconfig                                |   1 +
 crypto/Makefile                               |   5 +
 crypto/algapi.c                               |  14 +-
 crypto/fips140/Kconfig                        |  68 +++
 crypto/fips140/Makefile                       |  21 +
 crypto/fips140/README                         | 401 ++++++++++++++++++
 crypto/fips140/fips140-api.c                  |   7 +
 crypto/fips140/fips140-crypto-module-marker.h |   8 +
 crypto/fips140/fips140-loader.c               | 201 +++++++++
 crypto/fips140/fips140-module.c               | 138 ++++++
 crypto/fips140/fips140-module.h               |  30 ++
 crypto/fips140/fips140.lds                    |  17 +
 include/asm-generic/vmlinux.lds.h             |   1 +
 include/crypto/api.h                          | 197 +++++++++
 include/linux/init.h                          |  10 +
 include/linux/module.h                        |   2 +
 include/uapi/linux/module.h                   |   5 +
 init/main.c                                   |   4 +
 kernel/bpf/btf.c                              |  20 +
 kernel/module/main.c                          | 169 ++++++--
 kernel/params.c                               |   3 +-
 scripts/Makefile.build                        | 123 +++++-
 scripts/Makefile.modfinal                     |  37 ++
 scripts/Makefile.modinst                      |  13 +-
 scripts/Makefile.modpost                      |  25 ++
 scripts/Makefile.vmlinux                      |  59 ++-
 scripts/link-vmlinux.sh                       |  14 +
 scripts/mod/modpost.c                         |  24 +-
 37 files changed, 1759 insertions(+), 40 deletions(-)
 create mode 100644 arch/arm64/crypto/fips140/Makefile
 create mode 100644 arch/arm64/crypto/fips140/fips140-api.c
 create mode 100644 arch/x86/crypto/fips140/Makefile
 create mode 100644 arch/x86/crypto/fips140/fips140-api.c
 create mode 100644 crypto/fips140/Kconfig
 create mode 100644 crypto/fips140/Makefile
 create mode 100644 crypto/fips140/README
 create mode 100644 crypto/fips140/fips140-api.c
 create mode 100644 crypto/fips140/fips140-crypto-module-marker.h
 create mode 100644 crypto/fips140/fips140-loader.c
 create mode 100644 crypto/fips140/fips140-module.c
 create mode 100644 crypto/fips140/fips140-module.h
 create mode 100644 crypto/fips140/fips140.lds
 create mode 100644 include/crypto/api.h

-- 
2.47.3

[PATCH 01/17] crypto: add Kconfig options for standalone crypto module

[Jay Wang]

Add Kconfig option `CRYPTO_FIPS140_EXTMOD` to enable standalone crypto
module support that can override built-in cryptographic implementations.

Currently supports X86_64 and ARM64 architectures and requires CRYPTO
and MODULES to be enabled.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/Kconfig         |  1 +
 crypto/fips140/Kconfig | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 crypto/fips140/Kconfig

diff --git a/crypto/Kconfig b/crypto/Kconfig
index e2b4106ac961..b4ce3c1cfa1b 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1415,6 +1415,7 @@ endif
 endif
 
 source "drivers/crypto/Kconfig"
+source "crypto/fips140/Kconfig"
 source "crypto/asymmetric_keys/Kconfig"
 source "certs/Kconfig"
 source "crypto/krb5/Kconfig"
diff --git a/crypto/fips140/Kconfig b/crypto/fips140/Kconfig
new file mode 100644
index 000000000000..0665e94b9fe0
--- /dev/null
+++ b/crypto/fips140/Kconfig
@@ -0,0 +1,15 @@
+config CRYPTO_FIPS140_EXTMOD
+	bool "FIPS 140 compliant algorithms as a kernel module"
+	depends on CRYPTO && (X86_64 || ARM64) && MODULES
+	select CRYPTO_FIPS
+	help
+	  This option enables building a kernel module that contains
+	  copies of crypto algorithms that are built in a way that
+	  complies with the FIPS 140 standard.
+
+	  The module registers the algorithms it contains with the
+	  kernel crypto API, and the kernel crypto API's FIPS 140 mode
+	  can be enabled to restrict crypto algorithm usage to only
+	  those provided by this module.
+
+	  If unsure, say N.
-- 
2.47.3

[PATCH 02/17] crypto: add module entry for standalone crypto kernel module

[Jay Wang]

Add the module entry for the standalone FIPS kernel crypto module.
This creates the basic structure for fips140.ko that will be linked
with built-in crypto implementations in later patches.

The implementation includes module initialization and exit functions
and add into build system.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/Makefile                 |  5 +++++
 crypto/fips140/Makefile         |  3 +++
 crypto/fips140/fips140-module.c | 36 +++++++++++++++++++++++++++++++++
 crypto/fips140/fips140-module.h | 14 +++++++++++++
 4 files changed, 58 insertions(+)
 create mode 100644 crypto/fips140/Makefile
 create mode 100644 crypto/fips140/fips140-module.c
 create mode 100644 crypto/fips140/fips140-module.h

diff --git a/crypto/Makefile b/crypto/Makefile
index 04e269117589..5129be5e7208 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -210,3 +210,8 @@ obj-$(CONFIG_CRYPTO_KDF800108_CTR) += kdf_sp800108.o
 obj-$(CONFIG_CRYPTO_DF80090A) += df_sp80090a.o
 
 obj-$(CONFIG_CRYPTO_KRB5) += krb5/
+
+# FIPS 140 kernel module
+obj-$(CONFIG_CRYPTO_FIPS140_EXTMOD) += fips140/
+
+
diff --git a/crypto/fips140/Makefile b/crypto/fips140/Makefile
new file mode 100644
index 000000000000..364ef52c190f
--- /dev/null
+++ b/crypto/fips140/Makefile
@@ -0,0 +1,3 @@
+
+
+	
\ No newline at end of file
diff --git a/crypto/fips140/fips140-module.c b/crypto/fips140/fips140-module.c
new file mode 100644
index 000000000000..a942de8780ef
--- /dev/null
+++ b/crypto/fips140/fips140-module.c
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * FIPS 140 Kernel Cryptographic Module
+ *
+ * This file is the module entry point for fips140.ko, which is linked with previously built-in cryptos
+ * to generate the fips140.ko module.
+ * At load time, this module plugs the previously built-in implementations contained within itself back to the kernel.
+ * It also runs self-tests on these algorithms and verifies the integrity of its code and data.
+ * If either of these steps fails, the kernel will panic.
+ */
+
+#include "fips140-module.h"
+
+#define FIPS140_MODULE_NAME "FIPS 140 Kernel Cryptographic Module"
+#define FIPS140_MODULE_VERSION "1.0.0"
+
+#define CRYPTO_INTERNAL "CRYPTO_INTERNAL"
+
+/* Initialize the FIPS 140 module */
+static int __init fips140_init(void)
+{
+    return 0;
+}
+
+static void __exit fips140_exit(void)
+{
+    pr_info("Unloading " FIPS140_MODULE_NAME "\n");
+}
+
+module_init(fips140_init);
+module_exit(fips140_exit);
+
+MODULE_IMPORT_NS(CRYPTO_INTERNAL);
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION(FIPS140_MODULE_NAME);
+MODULE_VERSION(FIPS140_MODULE_VERSION);
diff --git a/crypto/fips140/fips140-module.h b/crypto/fips140/fips140-module.h
new file mode 100644
index 000000000000..ed2b6e17969f
--- /dev/null
+++ b/crypto/fips140/fips140-module.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * FIPS 140 Kernel Cryptographic Module - Header File
+ */
+
+#ifndef _CRYPTO_FIPS140_MODULE_H
+#define _CRYPTO_FIPS140_MODULE_H
+
+#include <linux/module.h>
+#include <linux/crypto.h>
+#include <crypto/algapi.h>
+#include <linux/init.h>
+
+#endif /* _CRYPTO_FIPS140_MODULE_H */
-- 
2.47.3

[PATCH 03/17] build: special compilation rule for building the standalone crypto module

[Jay Wang]

Add special build rules to redirect builtin crypto algorithms into a
standalone module `fips140.ko` instead of the main kernel, by introducing
a new crypto-objs-y compilation rule that collects crypto objects to
fips140.ko when CONFIG_CRYPTO_FIPS140_EXTMOD is enabled. Details as below:

Originally, builtin crypto (i.e., which specified by
CONFIG_CRYPTO_{NAME}=y) are compiled into main kernel by default. On the
contrary, this standalone crypto module feature requires compiling builtin
crypto into modules while ensuring such cryptos are not included into main
kernel anymore.

A naive way is to have a separate makefile containing all compilation
rules for all cryptos for such standalone module, but such makefile could
be too large to be scalable to arbitrary CONFIG_CRYPTO_{NAME}
configuration, plus, this method cannot ensure the builtin crypto is
completely removed from main kernel either.

To tackle this challenge, this patch automates object linking redirection by
introducing special build logic for kernel crypto subsystem. Specifically:

First, to automatically collect crypto objects file while preserving the
original compilation setting (such as flags, headers, path etc), it
introduces a specific compilation rule `crypto-objs-y += *.o` that replaces
the original rule `obj-y += *.o` in the crypto subsystem Makefile. As a
result, when the standalone crypto module feature is turned on, for any
crypto chosen to be builtin (e.g., crypto-objs-$(CONFIG_CRYPTO_SKCIPHER2)
+= *.o where CONFIG_CRYPTO_SKCIPHER2=y), it will be automatically collected
and linked into a final object binary: `fips140.o`, with special
compilation flag (-DFIPS_MODULE=1) to tell each individual obj files
compiled in a specific way (will be used in later patches for generating
pluggable interface on main kernel side and module side respectively from
same source code). The implementation details are: it refers to the
methodology of how obj-y collecting method of vmlinux.o, it places the
`crypto-objs-y` rule in scripts/Makefile.build that will be run in each
directory Makefile, and create crypto-module.a that comprises all captured
`crypto-objs-y += *.o` under same folders. In its parent folders, such
crypto-module.a will be included into parents' crypto-module.a recursively
and eventually used to generate final fips140.o.

Second, to generate the final kernel module fips140.ko with the above
fips140.o, A naïve approach would be to directly inject the fips140.ko
module build into the existing modules generation pipeline (i.e., `make
modules`) by providing our pre-generated fips140.o. However, we choose
not to do this because it would create a circular make rule dependency
(which is invalid in Makefiles and causes build failures), resulting in
mutual dependencies between the modules and vmlinux targets (i.e.,
`modules:vmlinux` and `vmlinux:modules` at the same time).
This happens for the following reasons:

1. Since we will later embed fips140.ko into the final kernel image (as
   described in the later patch), we must make vmlinux depend on
   fips140.ko. In other words: vmlinux: fips140.ko.
2. When the kernel is built with CONFIG_DEBUG_INFO_BTF_MODULES=y, it
   requires: `modules: vmlinux`. This is because
   `CONFIG_DEBUG_INFO_BTF_MODULES=y` will take vmlinux as input to
   generate a `BTF` info for the module, and insert such info into the
   `.ko` module by default.
3. If we choose to inject fips140.ko into make modules, this would
   create a make rule dependency: `fips140.ko: modules`. Combined with
   items 1 and 2, this eventually creates an invalid circular dependency
   between vmlinux and modules.

Due to these reasons, the design choice is to use a separate make
pipeline (defined as `fips140-ready` in the Makefile). This new
pipeline reuses the same module generation scripts used by make
modules but adds additional logic in
scripts/Makefile.{modfinal|modinst|modpost} and scripts/mod/modpost.c
to handle module symbol generation and verification correctly. In such
pipeline, to eliminate the make rule dependency of
`fips140.ko:vmlinux` which imposed by `CONFIG_DEBUG_INFO_BTF_MODULES=y`,
we temporarily not generating BTF info for fips140.ko in this patch,
where the BTF info for fips140.ko will be generated in later patch in
a special manner.

Finally, with these special build rules, crypto/fips140/fips140.ko is generated.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 Makefile                  |  34 +++++++++++-
 crypto/fips140/Makefile   |   4 +-
 scripts/Makefile.build    | 108 +++++++++++++++++++++++++++++++++++++-
 scripts/Makefile.modfinal |  21 ++++++++
 scripts/Makefile.modinst  |  13 ++++-
 scripts/Makefile.modpost  |  25 +++++++++
 scripts/mod/modpost.c     |  24 +++++++--
 7 files changed, 221 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index 832992a7b4fc..b5ae385ed5f3 100644
--- a/Makefile
+++ b/Makefile
@@ -1291,6 +1291,38 @@ PHONY += vmlinux
 #  vmlinux: private export LDFLAGS_vmlinux := $(LDFLAGS_vmlinux)
 vmlinux: private _LDFLAGS_vmlinux := $(LDFLAGS_vmlinux)
 vmlinux: export LDFLAGS_vmlinux = $(_LDFLAGS_vmlinux)
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+vmlinux: fips140-ready
+# Ensure fips140.ko is built before embedding
+fips140-ready: crypto/fips140/fips140.o crypto/fips140/.fips140.order crypto/fips140/fips140.mod vmlinux.o | modules_prepare
+	$(Q)$(MAKE) KBUILD_MODULES= -f $(srctree)/scripts/Makefile.modpost
+	$(Q)$(MAKE) KBUILD_MODULES=y crypto-module-gen=1 -f $(srctree)/scripts/Makefile.modpost
+ifneq ($(KBUILD_MODPOST_NOFINAL),1)
+	$(Q)$(MAKE) KBUILD_MODULES=y crypto-module-gen=1 -f $(srctree)/scripts/Makefile.modfinal
+endif
+	@:
+
+# Generate fips140.o from crypto-module.a files
+crypto/fips140/fips140.o: crypto-module.a FORCE
+	$(call if_changed,ld_fips140)
+crypto/fips140/.fips140.order: crypto/fips140/fips140.o
+	echo "crypto/fips140/fips140.o" > $@
+crypto/fips140/fips140.mod: crypto-module.a FORCE
+	$(call if_changed,fips140_mod)
+crypto/fips140/.fips140.symvers: fips140-ready
+	@:
+modpost: crypto/fips140/.fips140.symvers
+quiet_cmd_ld_fips140 = LD [M]  $@
+      cmd_ld_fips140 = $(LD) -r $(KBUILD_LDFLAGS) $(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) --build-id=none --whole-archive $< --no-whole-archive -o $@
+
+cmd_fips140_mod = ar -t $< > $@
+
+# Add to targets so .cmd file is included
+targets += crypto/fips140/fips140.o crypto/fips140/fips140.mod
+
+# Bridge rule: crypto-module.a depends on directory build (like built-in.a)
+crypto-module.a: . ;
+endif
 vmlinux: vmlinux.o $(KBUILD_LDS) modpost
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux
 
@@ -2083,7 +2115,7 @@ prepare: outputmakefile
 # Error messages still appears in the original language
 PHONY += $(build-dir)
 $(build-dir): prepare
-	$(Q)$(MAKE) $(build)=$@ need-builtin=1 need-modorder=1 $(single-goals)
+	$(Q)$(MAKE) $(build)=$@ need-builtin=1 need-modorder=1 need-crypto=1 $(single-goals)
 
 clean-dirs := $(addprefix _clean_, $(clean-dirs))
 PHONY += $(clean-dirs) clean
diff --git a/crypto/fips140/Makefile b/crypto/fips140/Makefile
index 364ef52c190f..3b4a74ccf41e 100644
--- a/crypto/fips140/Makefile
+++ b/crypto/fips140/Makefile
@@ -1,3 +1,5 @@
 
+crypto-objs-y += \
+	fips140-module.o 
 
-	
\ No newline at end of file
+clean-files:= .fips140.order .fips140.symvers
\ No newline at end of file
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 32e209bc7985..018289da4ccd 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -29,6 +29,21 @@ ldflags-y  :=
 subdir-asflags-y :=
 subdir-ccflags-y :=
 
+crypto-objs-flags-y := -DFIPS_MODULE=1
+crypto-objs-y :=
+crypto-module-folders := $(srcroot)/crypto $(srcroot)/arch/$(ARCH)/crypto $(srcroot)/lib/crypto
+# Global crypto directory checking logic
+define is-crypto-related-dir
+$(eval full-path := $(abspath $(obj)/$(1)))
+$(eval norm-folders := $(foreach dir,$(crypto-module-folders),$(abspath $(dir))))
+$(eval check-exact := $(filter $(norm-folders),$(full-path)))
+$(eval check-subdir := $(foreach dir,$(norm-folders),$(filter $(dir)/%,$(full-path))))
+$(eval check-ancestor := $(foreach dir,$(norm-folders),$(if $(filter $(full-path)/%,$(dir)),$(full-path))))
+$(eval result := $(strip $(check-exact) $(check-subdir) $(check-ancestor)))
+$(result)
+endef
+is-crypto-related := $(call is-crypto-related-dir, ./)
+
 # Read auto.conf if it exists, otherwise ignore
 -include $(objtree)/include/config/auto.conf
 
@@ -45,6 +60,16 @@ KBUILD_RUSTFLAGS += $(subdir-rustflags-y)
 # Figure out what we need to build from the various variables
 # ===========================================================================
 
+
+# Add crypto-objs-m to obj-m unconditionally
+obj-m += $(crypto-objs-m)
+
+# When CRYPTO_FIPS140_EXTMOD is not defined, add crypto-objs-y to obj-y and clear crypto-objs-y
+ifndef CONFIG_CRYPTO_FIPS140_EXTMOD
+obj-y += $(crypto-objs-y)
+crypto-objs-y :=
+endif
+
 # When an object is listed to be built compiled-in and modular,
 # only build the compiled-in version
 obj-m := $(filter-out $(obj-y),$(obj-m))
@@ -71,12 +96,27 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+# Capture obj-y directories before conversion
+obj-y-dirs := $(filter %/, $(obj-y))
+
 ifdef need-builtin
 obj-y		:= $(patsubst %/, %/built-in.a, $(obj-y))
 else
 obj-y		:= $(filter-out %/, $(obj-y))
 endif
 
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+# Add obj-y directories to crypto-objs-y only if they are crypto-related
+crypto-objs-y	+= $(strip $(foreach dir,$(obj-y-dirs),$(if $(strip $(call is-crypto-related-dir,$(patsubst %/,%,$(dir)))),$(dir))))
+
+ifdef need-crypto
+crypto-objs-y	:= $(patsubst %/, %/crypto-module.a, $(crypto-objs-y))
+else
+crypto-objs-y	:= $(filter-out %/, $(crypto-objs-y))
+endif
+endif
+
+
 # Expand $(foo-objs) $(foo-y) etc. by replacing their individuals
 suffix-search = $(strip $(foreach s, $3, $($(1:%$(strip $2)=%$s))))
 # List composite targets that are constructed by combining other targets
@@ -88,11 +128,17 @@ real-search = $(foreach m, $1, $(if $(call suffix-search, $m, $2, $3 -), $(call
 multi-obj-y := $(call multi-search, $(obj-y), .o, -objs -y)
 multi-obj-m := $(call multi-search, $(obj-m), .o, -objs -y -m)
 multi-obj-ym := $(multi-obj-y) $(multi-obj-m)
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+multi-crypto-objs-y := $(call multi-search, $(crypto-objs-y), .o, -objs -y)
+endif
 
 # Replace multi-part objects by their individual parts,
 # including built-in.a from subdirectories
 real-obj-y := $(call real-search, $(obj-y), .o, -objs -y)
 real-obj-m := $(call real-search, $(obj-m), .o, -objs -y -m)
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+real-crypto-objs-y := $(strip $(call real-search, $(crypto-objs-y), .o, -objs -y))
+endif
 
 always-y += $(always-m)
 
@@ -117,8 +163,14 @@ obj-m		:= $(addprefix $(obj)/, $(obj-m))
 lib-y		:= $(addprefix $(obj)/, $(lib-y))
 real-obj-y	:= $(addprefix $(obj)/, $(real-obj-y))
 real-obj-m	:= $(addprefix $(obj)/, $(real-obj-m))
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+real-crypto-objs-y := $(addprefix $(obj)/, $(real-crypto-objs-y))
+endif
 multi-obj-m	:= $(addprefix $(obj)/, $(multi-obj-m))
 subdir-ym	:= $(addprefix $(obj)/, $(subdir-ym))
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+multi-crypto-objs-y := $(addprefix $(obj)/, $(multi-crypto-objs-y))
+endif
 endif
 
 ifndef obj
@@ -137,7 +189,9 @@ endif
 # subdir-builtin and subdir-modorder may contain duplications. Use $(sort ...)
 subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y)))
 subdir-modorder := $(sort $(filter %/modules.order, $(obj-m)))
-
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+subdir-crypto := $(sort $(filter %/crypto-module.a, $(real-crypto-objs-y)))
+endif
 targets-for-builtin := $(extra-y)
 
 ifneq ($(strip $(lib-y) $(lib-m) $(lib-)),)
@@ -148,6 +202,16 @@ ifdef need-builtin
 targets-for-builtin += $(obj)/built-in.a
 endif
 
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+targets-for-crypto :=
+ifneq ($(is-crypto-related),)
+ifdef need-crypto
+targets-for-crypto += $(obj)/crypto-module.a
+endif
+targets += $(obj)/crypto-module.a
+endif
+endif
+
 targets-for-modules := $(foreach x, o mod, \
 				$(patsubst %.o, %.$x, $(filter %.o, $(obj-m))))
 
@@ -222,6 +286,12 @@ $(obj)/%.ll: $(obj)/%.c FORCE
 
 is-single-obj-m = $(and $(part-of-module),$(filter $@, $(obj-m)),y)
 
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+is-single-crypto-obj-y = $(and $(part-of-module),$(filter $@, $(real-crypto-objs-y)),y)
+else
+is-single-crypto-obj-y =
+endif
+
 ifdef CONFIG_MODVERSIONS
 # When module versioning is enabled the following steps are executed:
 # o compile a <file>.o from <file>.c
@@ -277,7 +347,7 @@ endif # CONFIG_FTRACE_MCOUNT_USE_RECORDMCOUNT
 is-standard-object = $(if $(filter-out y%, $(OBJECT_FILES_NON_STANDARD_$(target-stem).o)$(OBJECT_FILES_NON_STANDARD)n),$(is-kernel-object))
 
 ifdef CONFIG_OBJTOOL
-$(obj)/%.o: private objtool-enabled = $(if $(is-standard-object),$(if $(delay-objtool),$(is-single-obj-m),y))
+$(obj)/%.o: private objtool-enabled = $(if $(is-standard-object),$(if $(delay-objtool),$(or $(is-single-obj-m),$(is-single-crypto-obj-y)),y))
 endif
 
 ifneq ($(findstring 1, $(KBUILD_EXTRA_WARN)),)
@@ -433,6 +503,9 @@ $(obj)/%.o: $(obj)/%.S FORCE
 
 targets += $(filter-out $(subdir-builtin), $(real-obj-y))
 targets += $(filter-out $(subdir-modorder), $(real-obj-m))
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+targets += $(filter-out $(subdir-crypto), $(real-crypto-objs-y))
+endif
 targets += $(lib-y) $(always-y)
 
 # Linker scripts preprocessor (.lds.S -> .lds)
@@ -459,6 +532,9 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler
 # To build objects in subdirs, we need to descend into the directories
 $(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ;
 $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ;
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+$(subdir-crypto): $(obj)/%/crypto-module.a: $(obj)/% ;
+endif
 
 #
 # Rule to compile a set of .o files into one .a file (without symbol table)
@@ -474,6 +550,32 @@ quiet_cmd_ar_builtin = AR      $@
 $(obj)/built-in.a: $(real-obj-y) FORCE
 	$(call if_changed,ar_builtin)
 
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifneq ($(is-crypto-related),)
+$(obj)/crypto-module.a: $(real-crypto-objs-y) FORCE
+	$(call if_changed,ar_builtin)
+
+# Set module flags for crypto objects when building crypto-module.a
+ifdef need-crypto
+$(real-crypto-objs-y): private part-of-module := y
+$(real-crypto-objs-y): private modname := fips140
+$(real-crypto-objs-y): private KBUILD_CFLAGS += $(crypto-objs-flags-y)
+
+# Also set flags for individual objects that make up composite crypto objects
+$(foreach obj,$(multi-crypto-objs-y),$($(obj:.o=-y))): private part-of-module := y
+$(foreach obj,$(multi-crypto-objs-y),$($(obj:.o=-y))): private modname := fips140
+$(foreach obj,$(multi-crypto-objs-y),$($(obj:.o=-y))): private KBUILD_CFLAGS += $(crypto-objs-flags-y)
+
+# Multi-part crypto objects
+$(multi-crypto-objs-y): private part-of-module := y
+$(multi-crypto-objs-y): private modname := fips140
+$(multi-crypto-objs-y): private KBUILD_CFLAGS += $(crypto-objs-flags-y)
+$(multi-crypto-objs-y): %.o: %.mod FORCE
+	$(call if_changed_rule,ld_multi_m)
+$(call multi_depend, $(multi-crypto-objs-y), .o, -objs -y -m)
+endif
+endif
+endif
 # This is a list of build artifacts from the current Makefile and its
 # sub-directories. The timestamp should be updated when any of the member files.
 
@@ -546,6 +648,7 @@ $(subdir-ym):
 	$(Q)$(MAKE) $(build)=$@ \
 	need-builtin=$(if $(filter $@/built-in.a, $(subdir-builtin)),1) \
 	need-modorder=$(if $(filter $@/modules.order, $(subdir-modorder)),1) \
+	need-crypto=$(if $(and $(CONFIG_CRYPTO_FIPS140_EXTMOD), $(filter $@/crypto-module.a, $(subdir-crypto))),1) \
 	$(filter $@/%, $(single-subdir-goals))
 
 # Add FORCE to the prerequisites of a target to force it to be always rebuilt.
@@ -569,6 +672,7 @@ endif
 
 $(obj)/: $(if $(KBUILD_BUILTIN), $(targets-for-builtin)) \
 	 $(if $(KBUILD_MODULES), $(targets-for-modules)) \
+	 $(targets-for-crypto) \
 	 $(subdir-ym) $(always-y)
 	@:
 
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index adcbcde16a07..c68dab4d6584 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -11,7 +11,15 @@ include $(srctree)/scripts/Kbuild.include
 include $(srctree)/scripts/Makefile.lib
 
 # find all modules listed in modules.order
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifeq ($(crypto-module-gen),1)
+modules := $(call read-file, crypto/fips140/.fips140.order)
+else
 modules := $(call read-file, modules.order)
+endif
+else
+modules := $(call read-file, modules.order)
+endif
 
 __modfinal: $(modules:%.o=%.ko)
 	@:
@@ -55,12 +63,25 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
 	printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
 
 # Re-generate module BTFs if either module's .ko or vmlinux changed
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifeq ($(crypto-module-gen),1)
+%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds FORCE
+	+$(call if_changed,ld_ko_o)
+else
+%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
+	+$(call if_changed_except,ld_ko_o,$(objtree)/vmlinux)
+ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	+$(if $(newer-prereqs),$(call cmd,btf_ko))
+endif
+endif
+else
 %.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,$(objtree)/vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
 endif
 	+$(call cmd,check_tracepoint)
+endif
 
 targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o) .module-common.o
 
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 9ba45e5b32b1..32b6d0986922 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -28,7 +28,12 @@ $(MODLIB)/modules.order: modules.order FORCE
 	$(call cmd,install_modorder)
 
 quiet_cmd_install_modorder = INSTALL $@
-      cmd_install_modorder = sed 's:^\(.*\)\.o$$:kernel/\1.ko:' $< > $@
+      cmd_install_modorder = sed 's:^\(.*\)\.o$$:kernel/\1.ko:' $< > $@; \
+	$(if $(CONFIG_CRYPTO_FIPS140_EXTMOD), \
+		if [ -f crypto/fips140/.fips140.order ]; then \
+			sed 's:^\(.*\)\.o$$:kernel/\1.ko:' crypto/fips140/.fips140.order >> $@; \
+		fi \
+	)
 
 # Install modules.builtin(.modinfo,.ranges) even when CONFIG_MODULES is disabled.
 install-y += $(addprefix $(MODLIB)/, modules.builtin modules.builtin.modinfo)
@@ -42,6 +47,12 @@ endif
 
 modules := $(call read-file, modules.order)
 
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifneq ($(wildcard crypto/fips140/.fips140.order),)
+modules += $(call read-file, crypto/fips140/.fips140.order)
+endif
+endif
+
 ifeq ($(KBUILD_EXTMOD),)
 dst := $(MODLIB)/kernel
 else
diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index d7d45067d08b..18b5a5de74d9 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -51,6 +51,7 @@ modpost-args =										\
 	$(if $(KBUILD_NSDEPS),-d modules.nsdeps)					\
 	$(if $(CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS)$(KBUILD_NSDEPS),-N)	\
 	$(if $(findstring 1, $(KBUILD_EXTRA_WARN)),-W)					\
+	$(if $(crypto-module-gen),-c)							\
 	-o $@
 
 modpost-deps := $(MODPOST)
@@ -63,10 +64,22 @@ endif
 # Read out modules.order to pass in modpost.
 # Otherwise, allmodconfig would fail with "Argument list too long".
 ifdef KBUILD_MODULES
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifeq ($(crypto-module-gen),1)
+# For crypto-module-gen, use .fips140.order instead of modules.order
+modpost-args += -T crypto/fips140/.fips140.order
+modpost-deps += crypto/fips140/.fips140.order
+else
+modpost-args += -T modules.order
+modpost-deps += modules.order
+endif
+else
 modpost-args += -T modules.order
 modpost-deps += modules.order
 endif
 
+endif
+
 ifeq ($(KBUILD_EXTMOD),)
 
 # Generate the list of in-tree objects in vmlinux
@@ -110,6 +123,18 @@ modpost-deps += vmlinux.o
 output-symdump := $(if $(KBUILD_MODULES), Module.symvers, vmlinux.symvers)
 endif
 
+# Include .fips140.symvers for regular module builds if it exists
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifeq ($(KBUILD_MODULES),y)
+ifneq ($(crypto-module-gen),1)
+ifneq ($(wildcard crypto/fips140/.fips140.symvers),)
+modpost-args += -i crypto/fips140/.fips140.symvers
+modpost-deps += crypto/fips140/.fips140.symvers
+endif
+endif
+endif
+endif
+
 else
 
 # set src + obj - they may be used in the modules's Makefile
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 0c25b5ad497b..63f123f077e8 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -43,6 +43,8 @@ static bool extended_modversions;
 static bool external_module;
 /* Only warn about unresolved symbols */
 static bool warn_unresolved;
+/* Crypto module generation mode */
+static bool crypto_module_gen;
 
 static int sec_mismatch_count;
 static bool sec_mismatch_warn_only = true;
@@ -1746,6 +1748,9 @@ static void check_exports(struct module *mod)
 		const char *basename;
 		exp = find_symbol(s->name);
 		if (!exp) {
+			/* Skip undefined symbol errors for crypto module generation */
+			if (crypto_module_gen)
+				continue;
 			if (!s->weak && nr_unresolved++ < MAX_UNRESOLVED_REPORTS)
 				modpost_log(!warn_unresolved,
 					    "\"%s\" [%s.ko] undefined!\n",
@@ -2202,10 +2207,14 @@ static void write_dump(const char *fname)
 	struct buffer buf = { };
 	struct module *mod;
 	struct symbol *sym;
+	bool is_fips_symvers = crypto_module_gen && (strcmp(fname, "crypto/fips140/.fips140.symvers") == 0);
 
 	list_for_each_entry(mod, &modules, list) {
 		if (mod->dump_file)
 			continue;
+		/* Skip vmlinux symbols when writing .fips140.symvers */
+		if (is_fips_symvers && mod->is_vmlinux)
+			continue;
 		list_for_each_entry(sym, &mod->exported_symbols, list) {
 			if (trim_unused_exports && !sym->used)
 				continue;
@@ -2277,7 +2286,7 @@ int main(int argc, char **argv)
 	LIST_HEAD(dump_lists);
 	struct dump_list *dl, *dl2;
 
-	while ((opt = getopt(argc, argv, "ei:MmnT:to:au:WwENd:xb")) != -1) {
+	while ((opt = getopt(argc, argv, "ei:MmnT:to:au:WwENd:xbc")) != -1) {
 		switch (opt) {
 		case 'e':
 			external_module = true;
@@ -2332,6 +2341,9 @@ int main(int argc, char **argv)
 		case 'x':
 			extended_modversions = true;
 			break;
+		case 'c':
+			crypto_module_gen = true;
+			break;
 		default:
 			exit(1);
 		}
@@ -2377,8 +2389,14 @@ int main(int argc, char **argv)
 	if (missing_namespace_deps)
 		write_namespace_deps_files(missing_namespace_deps);
 
-	if (dump_write)
-		write_dump(dump_write);
+	if (dump_write) {
+		if (crypto_module_gen) {
+		 	/* generate .fips140.symvers for FIPS crypto modules */
+			write_dump("crypto/fips140/.fips140.symvers");
+		}else {
+			write_dump(dump_write);
+		}
+	}
 	if (sec_mismatch_count && !sec_mismatch_warn_only)
 		error("Section mismatches detected.\n"
 		      "Set CONFIG_SECTION_MISMATCH_WARN_ONLY=y to allow them.\n");
-- 
2.47.3

[PATCH 04/17] build: Add ELF marker for crypto-objs-m modules

[Jay Wang]

Previously, crypto-objs-$(CONFIG_*) behavior depends on the config value.
When CONFIG_*=y, crypto is built into fips140.ko. When CONFIG_*=m, crypto
is already built as a separate module (e.g., aes.ko), so previous patches
do not affect such modules.

This patch adds an ELF marker to identify modules built with CONFIG_*=m
so they can be distinguished as part of the CONFIG_CRYPTO_FIPS140_EXTMOD
framework. This gives module loaders a way to tell the module is included
in crypto-objs-m.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/fips140/fips140-crypto-module-marker.h |  8 ++++++++
 scripts/Makefile.build                        | 15 +++++++++++++++
 2 files changed, 23 insertions(+)
 create mode 100644 crypto/fips140/fips140-crypto-module-marker.h

diff --git a/crypto/fips140/fips140-crypto-module-marker.h b/crypto/fips140/fips140-crypto-module-marker.h
new file mode 100644
index 000000000000..eadca087cee2
--- /dev/null
+++ b/crypto/fips140/fips140-crypto-module-marker.h
@@ -0,0 +1,8 @@
+#ifndef _FIPS140_CRYPTO_MODULE_MARKER_H
+#define _FIPS140_CRYPTO_MODULE_MARKER_H
+
+/* Crypto module marker - automatically included for crypto-objs-m modules */
+static const char __fips140_crypto_marker[] 
+    __attribute__((section(".fips140_crypto_marker"), used)) = "FIPS140_CRYPTO_OBJS_M";
+
+#endif /* _FIPS140_CRYPTO_MODULE_MARKER_H */
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 018289da4ccd..cb21112472d4 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -68,6 +68,7 @@ obj-m += $(crypto-objs-m)
 ifndef CONFIG_CRYPTO_FIPS140_EXTMOD
 obj-y += $(crypto-objs-y)
 crypto-objs-y :=
+crypto-objs-m := $(filter-out $(crypto-objs-y),$(crypto-objs-m))
 endif
 
 # When an object is listed to be built compiled-in and modular,
@@ -130,6 +131,7 @@ multi-obj-m := $(call multi-search, $(obj-m), .o, -objs -y -m)
 multi-obj-ym := $(multi-obj-y) $(multi-obj-m)
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 multi-crypto-objs-y := $(call multi-search, $(crypto-objs-y), .o, -objs -y)
+multi-crypto-objs-m := $(call multi-search, $(crypto-objs-m), .o, -objs -y -m)
 endif
 
 # Replace multi-part objects by their individual parts,
@@ -138,6 +140,7 @@ real-obj-y := $(call real-search, $(obj-y), .o, -objs -y)
 real-obj-m := $(call real-search, $(obj-m), .o, -objs -y -m)
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 real-crypto-objs-y := $(strip $(call real-search, $(crypto-objs-y), .o, -objs -y))
+real-crypto-objs-m := $(strip $(call real-search, $(crypto-objs-m), .o, -objs -y -m))
 endif
 
 always-y += $(always-m)
@@ -165,11 +168,13 @@ real-obj-y	:= $(addprefix $(obj)/, $(real-obj-y))
 real-obj-m	:= $(addprefix $(obj)/, $(real-obj-m))
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 real-crypto-objs-y := $(addprefix $(obj)/, $(real-crypto-objs-y))
+real-crypto-objs-m := $(addprefix $(obj)/, $(real-crypto-objs-m))
 endif
 multi-obj-m	:= $(addprefix $(obj)/, $(multi-obj-m))
 subdir-ym	:= $(addprefix $(obj)/, $(subdir-ym))
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 multi-crypto-objs-y := $(addprefix $(obj)/, $(multi-crypto-objs-y))
+multi-crypto-objs-m := $(addprefix $(obj)/, $(multi-crypto-objs-m))
 endif
 endif
 
@@ -575,6 +580,16 @@ $(multi-crypto-objs-y): %.o: %.mod FORCE
 $(call multi_depend, $(multi-crypto-objs-y), .o, -objs -y -m)
 endif
 endif
+
+# Individual object compilation with version-specific flags
+$(real-crypto-objs-m): private KBUILD_CFLAGS += -DFIPS140_CRYPTO_OBJS_M=1 -include $(srctree)/crypto/fips140/fips140-crypto-module-marker.h
+
+# Also set flags for individual objects that make up composite crypto objects
+$(foreach obj,$(multi-crypto-objs-m),$($(obj:.o=-y))): private KBUILD_CFLAGS += -DFIPS140_CRYPTO_OBJS_M=1
+$(foreach obj,$(multi-crypto-objs-m),$($(obj:.o=-objs))): private KBUILD_CFLAGS += -DFIPS140_CRYPTO_OBJS_M=1
+
+# Multi-part crypto objects
+$(multi-crypto-objs-m): private KBUILD_CFLAGS += -DFIPS140_CRYPTO_OBJS_M=1 -include $(srctree)/crypto/fips140/fips140-crypto-module-marker.h
 endif
 # This is a list of build artifacts from the current Makefile and its
 # sub-directories. The timestamp should be updated when any of the member files.
-- 
2.47.3

[PATCH 05/17] module: allow kernel module loading directly from memory

[Jay Wang]

From: Vegard Nossum <vegard.nossum@oracle.com>

To enable loading the crypto module earlier before file system is ready,
add a new helper function, load_crypto_module_mem(), which can load a kernel
module from a byte array in memory. When loading in this way, we don't
do signature verification as crypto is not ready yet before loaded.
To tell that a module is loaded in this way, a new module loader flag,
MODULE_INIT_CRYPTO_FROM_MEM, is added.

Co-developed-by: Saeed Mirzamohammadi <saeed.mirzamohammadi@oracle.com>
Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
[With code change and revise commit message]
Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 include/linux/module.h      |   2 +
 include/uapi/linux/module.h |   5 ++
 kernel/module/main.c        | 100 +++++++++++++++++++++++++-----------
 kernel/params.c             |   3 +-
 4 files changed, 79 insertions(+), 31 deletions(-)

diff --git a/include/linux/module.h b/include/linux/module.h
index 20ddfd97630d..22a1d8459ce4 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -589,6 +589,8 @@ struct module {
 
 #ifdef CONFIG_MODULES
 
+extern int load_crypto_module_mem(const char *mem, size_t size);
+
 /* Get/put a kernel symbol (calls must be symmetric) */
 void *__symbol_get(const char *symbol);
 void *__symbol_get_gpl(const char *symbol);
diff --git a/include/uapi/linux/module.h b/include/uapi/linux/module.h
index 03a33ffffcba..30e9a7813eac 100644
--- a/include/uapi/linux/module.h
+++ b/include/uapi/linux/module.h
@@ -7,4 +7,9 @@
 #define MODULE_INIT_IGNORE_VERMAGIC	2
 #define MODULE_INIT_COMPRESSED_FILE	4
 
+#ifdef __KERNEL__
+/* Internal flags */
+#define MODULE_INIT_CRYPTO_FROM_MEM			30
+#endif
+
 #endif /* _UAPI_LINUX_MODULE_H */
diff --git a/kernel/module/main.c b/kernel/module/main.c
index 710ee30b3bea..2914e7619766 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -2572,11 +2572,14 @@ static void module_augment_kernel_taints(struct module *mod, struct load_info *i
 
 static int check_modinfo(struct module *mod, struct load_info *info, int flags)
 {
-	const char *modmagic = get_modinfo(info, "vermagic");
+	const char *modmagic = NULL;
 	int err;
 
-	if (flags & MODULE_INIT_IGNORE_VERMAGIC)
-		modmagic = NULL;
+	if (flags & MODULE_INIT_CRYPTO_FROM_MEM)
+		return 0;
+
+	if (!(flags & MODULE_INIT_IGNORE_VERMAGIC))
+		modmagic = get_modinfo(info, "vermagic");
 
 	/* This is allowed: modprobe --force will invalidate it. */
 	if (!modmagic) {
@@ -3007,7 +3010,7 @@ module_param(async_probe, bool, 0644);
  * Keep it uninlined to provide a reliable breakpoint target, e.g. for the gdb
  * helper command 'lx-symbols'.
  */
-static noinline int do_init_module(struct module *mod)
+static noinline int do_init_module(struct module *mod, int flags)
 {
 	int ret = 0;
 	struct mod_initfree *freeinit;
@@ -3070,8 +3073,10 @@ static noinline int do_init_module(struct module *mod)
 	ftrace_free_mem(mod, mod->mem[MOD_INIT_TEXT].base,
 			mod->mem[MOD_INIT_TEXT].base + mod->mem[MOD_INIT_TEXT].size);
 	mutex_lock(&module_mutex);
-	/* Drop initial reference. */
-	module_put(mod);
+	/* Drop initial reference for normal modules to allow unloading.
+	 * Keep reference for MODULE_INIT_CRYPTO_FROM_MEM modules to prevent unloading. */
+	if (!(flags & MODULE_INIT_CRYPTO_FROM_MEM))
+		module_put(mod);  
 	trim_init_extable(mod);
 #ifdef CONFIG_KALLSYMS
 	/* Switch to core kallsyms now init is done: kallsyms may be walking! */
@@ -3347,31 +3352,17 @@ static int early_mod_check(struct load_info *info, int flags)
 /*
  * Allocate and load the module: note that size of section 0 is always
  * zero, and we rely on this for optional sections.
+ *
+ * NOTE: module signature verification must have been done already.
  */
-static int load_module(struct load_info *info, const char __user *uargs,
-		       int flags)
+static int _load_module(struct load_info *info, const char __user *uargs,
+			int flags)
 {
 	struct module *mod;
 	bool module_allocated = false;
 	long err = 0;
 	char *after_dashes;
 
-	/*
-	 * Do the signature check (if any) first. All that
-	 * the signature check needs is info->len, it does
-	 * not need any of the section info. That can be
-	 * set up later. This will minimize the chances
-	 * of a corrupt module causing problems before
-	 * we even get to the signature check.
-	 *
-	 * The check will also adjust info->len by stripping
-	 * off the sig length at the end of the module, making
-	 * checks against info->len more correct.
-	 */
-	err = module_sig_check(info, flags);
-	if (err)
-		goto free_copy;
-
 	/*
 	 * Do basic sanity checks against the ELF header and
 	 * sections. Cache useful sections and set the
@@ -3405,7 +3396,8 @@ static int load_module(struct load_info *info, const char __user *uargs,
 	 * We are tainting your kernel if your module gets into
 	 * the modules linked list somehow.
 	 */
-	module_augment_kernel_taints(mod, info);
+	if (!(flags & MODULE_INIT_CRYPTO_FROM_MEM))
+		module_augment_kernel_taints(mod, info);
 
 	/* To avoid stressing percpu allocator, do this once we're unique. */
 	err = percpu_modalloc(mod, info);
@@ -3452,7 +3444,11 @@ static int load_module(struct load_info *info, const char __user *uargs,
 	flush_module_icache(mod);
 
 	/* Now copy in args */
-	mod->args = strndup_user(uargs, ~0UL >> 1);
+	if ((flags & MODULE_INIT_CRYPTO_FROM_MEM))
+		mod->args = kstrdup("", GFP_KERNEL);
+	else
+		mod->args = strndup_user(uargs, ~0UL >> 1);
+
 	if (IS_ERR(mod->args)) {
 		err = PTR_ERR(mod->args);
 		goto free_arch_cleanup;
@@ -3500,13 +3496,10 @@ static int load_module(struct load_info *info, const char __user *uargs,
 	if (codetag_load_module(mod))
 		goto sysfs_cleanup;
 
-	/* Get rid of temporary copy. */
-	free_copy(info, flags);
-
 	/* Done! */
 	trace_module_load(mod);
 
-	return do_init_module(mod);
+	return do_init_module(mod, flags);
 
  sysfs_cleanup:
 	mod_sysfs_teardown(mod);
@@ -3562,7 +3555,54 @@ static int load_module(struct load_info *info, const char __user *uargs,
 		audit_log_kern_module(info->name ? info->name : "?");
 		mod_stat_bump_becoming(info, flags);
 	}
+	return err;
+}
+
+/*
+ * Load crypto module from kernel memory without signature check.
+ */
+int __init load_crypto_module_mem(const char *mem, size_t size)
+{
+	int err;
+	struct load_info info = { };
+
+	if (!mem) {
+		pr_err("load_crypto_module_mem: mem parameter is NULL\n");
+		return -EINVAL;
+	}
+
+	info.sig_ok = true;
+	info.hdr = (Elf_Ehdr *) mem;
+	info.len = size;
+
+	err = _load_module(&info, NULL, MODULE_INIT_CRYPTO_FROM_MEM);
+	return err;
+}
+
+static int load_module(struct load_info *info, const char __user *uargs,
+		       int flags)
+{
+	int err;
+
+	/*
+	 * Do the signature check (if any) first. All that
+	 * the signature check needs is info->len, it does
+	 * not need any of the section info. That can be
+	 * set up later. This will minimize the chances
+	 * of a corrupt module causing problems before
+	 * we even get to the signature check.
+	 *
+	 * The check will also adjust info->len by stripping
+	 * off the sig length at the end of the module, making
+	 * checks against info->len more correct.
+	 */
+	err = module_sig_check(info, flags);
+	if (!err)
+		err = _load_module(info, uargs, flags);
+
+	/* Get rid of temporary copy. */
 	free_copy(info, flags);
+
 	return err;
 }
 
diff --git a/kernel/params.c b/kernel/params.c
index 7c2242f64bf0..b0671d752ff1 100644
--- a/kernel/params.c
+++ b/kernel/params.c
@@ -967,7 +967,8 @@ static int __init param_sysfs_init(void)
 
 	return 0;
 }
-subsys_initcall(param_sysfs_init);
+/* Use arch_initcall instead of subsys_initcall for early module loading */
+arch_initcall(param_sysfs_init);
 
 /*
  * param_sysfs_builtin_init - add sysfs version and parameter
-- 
2.47.3

[PATCH 06/17] crypto: add pluggable interface for builtin crypto modules

[Jay Wang]

Traditionally, builtin cryptos interact with the main kernel through
exported functions and variables via EXPORT_SYMBOL. The exported
symbols are statically tied with the main kernel, so not suitable
after those builtin cryptos moved to a standalone kernel module
(e.g., main kernel will not even be compilable since users need the
exact function/variable address).

To address this, introduce a pluggable interface between builtin
crypto functions and variables by placing address placeholders.
During runtime once the crypto kernel module is loaded, the address
placeholder gets updated (by `do_crypto_api` and `do_crypto_var`) to
correct address.

In more details, there are two types of address holders: for function
addresses, "static call" mechanism is used; for variable addresses, a
var_type* pointer is used. To apply this pluggable interface, just
wrap the function/variable header declaration with the placeholder
declaration wrappers (implemented as `DECLARE_STATIC_CALL()`/
`DECLARE_CRYPTO_VAR()`), and place the placeholder definition wrappers
for each crypto function/variable in fips140-api.c (implemented as
`DEFINE_CRYPTO_API_STUB()`/`DEFINE_CRYPTO_VAR_STUB()`). Those wrappers
will be compiled differently for main kernel and the crypto kernel
module using different compilation flags. To make such pluggable
interface only affect cryptos chosen as builtin but not crypto chosen
built as modules already, associate wrappers with CONFIG_CRYPTO_{NAME}
parameter.

In addition to the pluggable interface, this patch also adds a way to
collect crypto init functions, such as `module_init()` into a
dedicated ELF section for later module entry to initialize each
crypto.

The idea of using "static call" as pluggable interface for function
address stems from Vegard Nossum <vegard.nossum@oracle.com>, while
we additionally make the following core differences: avoid duplicate
crypto code in main kernel, allow variable pluggable, make this
feature not affect cryptos already chosen to be module-built, and
make this adapt to any .config choices.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/fips140/Makefile           |   5 +-
 crypto/fips140/fips140-api.c      |   7 ++
 include/asm-generic/vmlinux.lds.h |   1 +
 include/crypto/api.h              | 197 ++++++++++++++++++++++++++++++
 kernel/module/main.c              |  46 ++++++-
 5 files changed, 253 insertions(+), 3 deletions(-)
 create mode 100644 crypto/fips140/fips140-api.c
 create mode 100644 include/crypto/api.h

diff --git a/crypto/fips140/Makefile b/crypto/fips140/Makefile
index 3b4a74ccf41e..fb083022efbb 100644
--- a/crypto/fips140/Makefile
+++ b/crypto/fips140/Makefile
@@ -1,5 +1,8 @@
 
 crypto-objs-y += \
-	fips140-module.o 
+	fips140-module.o \
+	fips140-api.o
+
+obj-y += fips140-api.o
 
 clean-files:= .fips140.order .fips140.symvers
\ No newline at end of file
diff --git a/crypto/fips140/fips140-api.c b/crypto/fips140/fips140-api.c
new file mode 100644
index 000000000000..a11e898ff4bc
--- /dev/null
+++ b/crypto/fips140/fips140-api.c
@@ -0,0 +1,7 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+/*
+ * Define static call keys for any functions which are part of the crypto
+ * API and used by the standalone FIPS module but which are not built into
+ * vmlinux.
+ */
\ No newline at end of file
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index eeb070f330bd..e25b44d29362 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -724,6 +724,7 @@
 	KERNEL_CTORS()							\
 	MCOUNT_REC()							\
 	*(.init.rodata .init.rodata.*)					\
+	BOUNDED_SECTION(__crypto_api_keys)				\
 	FTRACE_EVENTS()							\
 	TRACE_SYSCALLS()						\
 	KPROBE_BLACKLIST()						\
diff --git a/include/crypto/api.h b/include/crypto/api.h
new file mode 100644
index 000000000000..b58240ffd173
--- /dev/null
+++ b/include/crypto/api.h
@@ -0,0 +1,197 @@
+#ifndef _CRYPTO_API_H
+#define _CRYPTO_API_H
+
+#include <linux/static_call.h>
+
+#define CRYPTO_VAR_NAME(name) __crypto_##name##_ptr
+
+#define __CAT(a,b) a##b
+#define _CAT(a,b)  __CAT(a,b)
+
+#define __IF_1(...) __VA_ARGS__
+#define __IF_0(...)
+#define __IFNOT_1(...)
+#define __IFNOT_0(...) __VA_ARGS__
+
+/* Emit __VA_ARGS__ only if cfg is built into vmlinux (=y) */
+#define IF_BUILTIN(cfg, ...)     _CAT(__IF_,    IS_BUILTIN(cfg))(__VA_ARGS__)
+/* Emit __VA_ARGS__ only if cfg is NOT built in (i.e., =m or unset) */
+#define IF_NOT_BUILTIN(cfg, ...) _CAT(__IFNOT_, IS_BUILTIN(cfg))(__VA_ARGS__)
+
+#if !defined(CONFIG_CRYPTO_FIPS140_EXTMOD)
+
+/*
+ * These are the definitions that get used when no standalone FIPS module
+ * is used: we simply forward everything to normal functions and function
+ * calls.
+ */
+
+#define DECLARE_CRYPTO_API(cfg, name, ret_type, args_decl, args_call)       \
+	ret_type name args_decl;
+
+#define DECLARE_CRYPTO_VAR(cfg, name, var_type, ...) \
+	extern var_type name __VA_ARGS__;
+
+#define crypto_module_init(fn) module_init(fn)
+#define crypto_module_exit(fn) module_exit(fn)
+
+#define crypto_subsys_initcall(fn)	subsys_initcall(fn)
+#define crypto_late_initcall(fn)	late_initcall(fn)
+
+#define crypto_module_cpu_feature_match(x, __initfunc) \
+	module_cpu_feature_match(x, __initfunc)
+
+#else
+
+struct crypto_api_key {
+	struct static_call_key *key;
+	void *tramp;
+	void *func;
+};
+
+struct crypto_var_key {
+	void **ptr;
+	void *var;
+};
+
+#ifndef FIPS_MODULE
+
+/*
+ * These are the definitions that get used for vmlinux and in-tree
+ * kernel modules.
+ *
+ * In this case, all references to the kernel crypto API functions will
+ * be replaced by wrappers that perform a call using the kernel's static_call
+ * functionality.
+ */
+
+/* Consolidated version of different DECLARE_CRYPTO_API versions */
+
+/*
+ *  - If cfg is built-in (=y): declare nonfips_<name>, a static_call key,
+ *    and an inline wrapper <name>() that dispatches via static_call.
+ *  - Else (cfg =m or unset): only declare <name>() prototype.
+ */
+#define DECLARE_CRYPTO_API(cfg, name, ret_type, args_decl, args_call)       \
+	IF_BUILTIN(cfg,                                                      \
+		ret_type nonfips_##name args_decl;                           \
+		DECLARE_STATIC_CALL(crypto_##name##_key, nonfips_##name);    \
+		static inline ret_type name args_decl                        \
+		{                                                            \
+			return static_call(crypto_##name##_key) args_call;   \
+		}                                                            \
+	)                                                                   \
+	IF_NOT_BUILTIN(cfg,                                                 \
+		ret_type name args_decl;                                    \
+	)
+
+#define DECLARE_CRYPTO_VAR(cfg, name, var_type, ...) \
+	IF_BUILTIN(cfg, \
+		extern void *CRYPTO_VAR_NAME(name); \
+	) \
+	IF_NOT_BUILTIN(cfg, \
+		extern var_type name __VA_ARGS__; \
+	)
+
+#define DEFINE_CRYPTO_API_STUB(name) \
+	DEFINE_STATIC_CALL_NULL(crypto_##name##_key, name); \
+	EXPORT_STATIC_CALL(crypto_##name##_key)
+
+#define DEFINE_CRYPTO_VAR_STUB(name) \
+	void* CRYPTO_VAR_NAME(name) = NULL;\
+	EXPORT_SYMBOL(CRYPTO_VAR_NAME(name));
+
+#define crypto_module_init(fn) module_init(fn)
+#define crypto_module_exit(fn) module_exit(fn)
+
+#define crypto_subsys_initcall(fn)	subsys_initcall(fn)
+#define crypto_late_initcall(fn)	late_initcall(fn)
+
+#define crypto_module_cpu_feature_match(x, __initfunc) \
+	module_cpu_feature_match(x, __initfunc)
+	
+#else /* defined(FIPS_MODULE) */
+
+/* Consolidated version of different DECLARE_CRYPTO_API versions,
+   within FIPS module, API remains the same, only declare static 
+   call key */
+#define DECLARE_CRYPTO_API(cfg, name, ret_type, args_decl, args_call) \
+	IF_BUILTIN(cfg,                                                   \
+		ret_type name args_decl;                                      \
+		DECLARE_STATIC_CALL(crypto_##name##_key, name)                \
+	)                                                                 \
+	IF_NOT_BUILTIN(cfg,                                               \
+		ret_type name args_decl;                                      \
+	)
+
+#define DECLARE_CRYPTO_VAR(cfg, name, var_type, ...)               \
+	IF_BUILTIN(cfg,                                             \
+		extern var_type name __VA_ARGS__;                   \
+		extern void *CRYPTO_VAR_NAME(name);                  \
+	)                                                            \
+	IF_NOT_BUILTIN(cfg,                                          \
+		extern var_type name __VA_ARGS__;                   \
+	)
+
+#define DEFINE_CRYPTO_VAR_STUB(name) \
+	static struct crypto_var_key __crypto_##name##_var_key \
+		__used \
+		__section("__crypto_var_keys") \
+		__aligned(__alignof__(struct crypto_var_key)) = \
+	{ \
+		.ptr = &CRYPTO_VAR_NAME(name), \
+		.var = (void*)&name, \
+	};
+
+/*
+ * These are the definitions that get used for the main kernel.
+ *
+ * In this case, initialize crypto static call key with original name
+ */
+
+#define DEFINE_CRYPTO_API_STUB(name) \
+	static struct crypto_api_key __##name##_key \
+		__used \
+		__section("__crypto_api_keys") \
+		__aligned(__alignof__(struct crypto_api_key)) = \
+	{ \
+		.key = &STATIC_CALL_KEY(crypto_##name##_key), \
+		.tramp = STATIC_CALL_TRAMP_ADDR(crypto_##name##_key), \
+		.func = &name, \
+	};
+
+#define crypto_module_init(fn) \
+	static initcall_t __used __section(".fips_initcall1") \
+		__fips_##fn = fn;
+#define crypto_module_exit(fn) \
+		static unsigned long __used __section(".fips_exitcall") \
+		__fips_##fn = (unsigned long) &fn;
+#define crypto_subsys_initcall(fn) \
+	static initcall_t __used __section(".fips_initcall0") \
+		__fips_##fn = fn;
+#define crypto_subsys_exitcall(fn) \
+		static unsigned long __used __section(".fips_exitcall") \
+		__fips_##fn = (unsigned long) &fn;
+#define crypto_late_initcall(fn) \
+	static initcall_t __used __section(".fips_initcall2") \
+		__fips_##fn = fn;
+#define crypto_late_exitcall(fn) \
+		static unsigned long __used __section(".fips_exitcall") \
+		__fips_##fn = (unsigned long) &fn;
+
+#define crypto_module_cpu_feature_match(x, __initfunc) \
+static struct cpu_feature const __maybe_unused cpu_feature_match_ ## x[] = \
+	{ { .feature = cpu_feature(x) }, { } }; \
+MODULE_DEVICE_TABLE(cpu, cpu_feature_match_ ## x); \
+static int __init cpu_feature_match_ ## x ## _init(void) \
+{ \
+	if (!cpu_have_feature(cpu_feature(x))) \
+		return -ENODEV; \
+	return __initfunc(); \
+} \
+crypto_module_init(cpu_feature_match_ ## x ## _init)
+
+#endif /* defined(FIPS_MODULE) */
+#endif /* defined(CONFIG_CRYPTO_FIPS140_EXTMOD) */
+
+#endif /* !_CRYPTO_API_H */
diff --git a/kernel/module/main.c b/kernel/module/main.c
index 2914e7619766..dad84f0548ac 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -7,6 +7,7 @@
 
 #define INCLUDE_VERMAGIC
 
+#include <crypto/api.h>
 #include <linux/export.h>
 #include <linux/extable.h>
 #include <linux/moduleloader.h>
@@ -2956,6 +2957,39 @@ static int post_relocation(struct module *mod, const struct load_info *info)
 	return module_finalize(info->hdr, info->sechdrs, mod);
 }
 
+#ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+static void do_crypto_api(struct load_info *info)
+{
+	struct crypto_api_key *crypto_api_keys;
+	unsigned int num_crypto_api_keys;
+
+	unsigned int i;
+
+	crypto_api_keys = section_objs(info, "__crypto_api_keys",
+		sizeof(*crypto_api_keys), &num_crypto_api_keys);
+
+	for (i = 0; i < num_crypto_api_keys; ++i) {
+		struct crypto_api_key *key = &crypto_api_keys[i];
+		__static_call_update(key->key, key->tramp, key->func);
+	}
+}
+
+static void do_crypto_var(struct load_info *info)
+{
+	struct crypto_var_key *crypto_var_keys;
+	unsigned int num_crypto_var_keys;
+	unsigned int i;
+
+	crypto_var_keys = section_objs(info, "__crypto_var_keys",
+		sizeof(*crypto_var_keys), &num_crypto_var_keys);
+
+	for (i = 0; i < num_crypto_var_keys; ++i) {
+		struct crypto_var_key *var_key = &crypto_var_keys[i];
+		*(var_key->ptr) = var_key->var;
+	}
+}
+#endif
+
 /* Call module constructors. */
 static void do_mod_ctors(struct module *mod)
 {
@@ -3010,7 +3044,7 @@ module_param(async_probe, bool, 0644);
  * Keep it uninlined to provide a reliable breakpoint target, e.g. for the gdb
  * helper command 'lx-symbols'.
  */
-static noinline int do_init_module(struct module *mod, int flags)
+static noinline int do_init_module(struct load_info *info, struct module *mod, int flags)
 {
 	int ret = 0;
 	struct mod_initfree *freeinit;
@@ -3036,6 +3070,14 @@ static noinline int do_init_module(struct module *mod, int flags)
 	freeinit->init_data = mod->mem[MOD_INIT_DATA].base;
 	freeinit->init_rodata = mod->mem[MOD_INIT_RODATA].base;
 
+#ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+	if (flags & MODULE_INIT_CRYPTO_FROM_MEM)
+		{
+			do_crypto_api(info);
+			do_crypto_var(info);
+		}
+#endif
+
 	do_mod_ctors(mod);
 	/* Start the module */
 	if (mod->init != NULL)
@@ -3499,7 +3541,7 @@ static int _load_module(struct load_info *info, const char __user *uargs,
 	/* Done! */
 	trace_module_load(mod);
 
-	return do_init_module(mod, flags);
+	return do_init_module(info, mod, flags);
 
  sysfs_cleanup:
 	mod_sysfs_teardown(mod);
-- 
2.47.3

[PATCH 07/17] crypto: dedicated ELF sections for collected crypto initcalls

[Jay Wang]

The addresses of init functions for crypto collected into the standalone
modules are placed into a dedicated ELF section. This is achieved by
introducing a custom linker script crypto/fips140/fips140.lds. Since in
later patch the init crypto will be called in the proper ordering (e.g.,
device_initcall should be earlier than late_initcall), the linker script
allocates different levels of ELF section (e.g.,
__fips140_initcall0_start and __fips140_initcall1_start) to preserve the
proper ordering.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 Makefile                   |  2 +-
 crypto/fips140/fips140.lds | 17 +++++++++++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
 create mode 100644 crypto/fips140/fips140.lds

diff --git a/Makefile b/Makefile
index b5ae385ed5f3..8413cc02665c 100644
--- a/Makefile
+++ b/Makefile
@@ -1313,7 +1313,7 @@ crypto/fips140/.fips140.symvers: fips140-ready
 	@:
 modpost: crypto/fips140/.fips140.symvers
 quiet_cmd_ld_fips140 = LD [M]  $@
-      cmd_ld_fips140 = $(LD) -r $(KBUILD_LDFLAGS) $(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) --build-id=none --whole-archive $< --no-whole-archive -o $@
+      cmd_ld_fips140 = $(LD) -r $(KBUILD_LDFLAGS) $(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) --build-id=none -T $(srctree)/crypto/fips140/fips140.lds --whole-archive $< --no-whole-archive -o $@
 
 cmd_fips140_mod = ar -t $< > $@
 
diff --git a/crypto/fips140/fips140.lds b/crypto/fips140/fips140.lds
new file mode 100644
index 000000000000..1adb3ccb5b9d
--- /dev/null
+++ b/crypto/fips140/fips140.lds
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+SECTIONS {
+	.init.data : {
+		__fips140_initcalls_start = .;
+		__fips140_initcall0_start = .;
+		*(.fips_initcall0)
+		__fips140_initcall0_end = .;
+		__fips140_initcall1_start = .;
+		*(.fips_initcall1)
+		__fips140_initcall1_end = .;
+		__fips140_initcall2_start = .;
+		*(.fips_initcall2)
+		__fips140_initcall2_end = .;
+		__fips140_initcalls_end = .;
+	}
+}
\ No newline at end of file
-- 
2.47.3

[PATCH 08/17] crypto: fips140: add crypto module loader

[Jay Wang]

Add a crypto module loader mechanism that loads a precompiled crypto
kernel module that is embedded in vmlinux image directly from memory
(whose address stored in _binary_fips140_ko_start/end) during early boot.

This is built based on Vegard Nossum <vegard.nossum@oracle.com> and
Saeed Mirzamohammadi <saeed.mirzamohammadi@oracle.com>, the
fips_loader_init is picked up. But different from them, such loader is
not executed as arch_initcall_sync(), but rather as a thread along main
kernel init to ensure proper initialization sequencing (Details are in
later patch).

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/fips140/fips140-loader.c | 55 +++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
 create mode 100644 crypto/fips140/fips140-loader.c

diff --git a/crypto/fips140/fips140-loader.c b/crypto/fips140/fips140-loader.c
new file mode 100644
index 000000000000..cbf5ff59eb2e
--- /dev/null
+++ b/crypto/fips140/fips140-loader.c
@@ -0,0 +1,55 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * FIPS 140 Early Loader
+ */
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/printk.h>
+#include <linux/vmalloc.h>
+#include <linux/string.h>
+#include <linux/elf.h>
+#include <linux/kthread.h>
+#include <linux/wait.h>
+
+extern const u8 _binary_fips140_ko_start[];
+extern const u8 _binary_fips140_ko_end[];
+const u8 *_binary_crypto_ko_start;
+EXPORT_SYMBOL_GPL(_binary_crypto_ko_start);
+const u8 *_binary_crypto_ko_end;
+EXPORT_SYMBOL_GPL(_binary_crypto_ko_end);
+
+/* Function to load crypto module from memory */
+extern int load_crypto_module_mem(const char *mem, size_t size);
+
+static void load_prepare(void)
+{
+	_binary_crypto_ko_start = _binary_fips140_ko_start;
+	_binary_crypto_ko_end = _binary_fips140_ko_end;
+}
+
+static int __init fips_loader_init(void)
+{
+	load_prepare();
+	
+	const void *ko_mem = _binary_crypto_ko_start;
+	size_t ko_size = _binary_crypto_ko_end - _binary_crypto_ko_start;
+	void *vmalloc_mem;
+	int ret;
+	
+	// Copy to vmalloc'd memory since load_module expects to free it
+	vmalloc_mem = vmalloc(ko_size);
+	if (!vmalloc_mem) {
+		pr_err("FIPS140 loader: failed to allocate memory\n");
+		return -ENOMEM;
+	}
+	
+	memcpy(vmalloc_mem, ko_mem, ko_size);
+	
+	ret = load_crypto_module_mem(vmalloc_mem, ko_size); // Skip signature check
+	if (ret)
+		panic("FIPS140 loader: module loading error\n");
+
+	vfree(vmalloc_mem); // Free after successful module loading
+	return ret;
+}
-- 
2.47.3

[PATCH 09/17] build: embed the standalone crypto module into vmlinux

[Jay Wang]

As mentioned in earlier patch, in order to load standalone crypto module
in early boot before filesystem is ready, the module needs to be embedded
into vmlinux image. This patch intends to make such embedded process a
seamless process that will automatically trigger as building vmlinux (i.e.,
during `make vmlinux`). So it adds make dependency rule such that vmlinux
will depend on the `fips140.ko` and its signature `.fips140.hmac`
generation rule. It also modifies vmlinux link rule to finally link them
with vmlinux.o.

The high level idea of embedding fips140.ko into vmlinux stems from
Vegard Nossum <vegard.nossum@oracle.com>.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 Makefile                        | 25 +++++++++++++++++++++++--
 arch/arm64/kernel/vmlinux.lds.S | 16 ++++++++++++++++
 arch/x86/kernel/vmlinux.lds.S   | 16 ++++++++++++++++
 crypto/fips140/Kconfig          | 29 +++++++++++++++++++++++++++++
 crypto/fips140/Makefile         | 16 +++++++++++++---
 crypto/fips140/fips140-loader.c |  9 +++++++++
 scripts/Makefile.modfinal       | 18 +++++++++++++++++-
 scripts/Makefile.vmlinux        |  6 +++++-
 scripts/link-vmlinux.sh         |  5 +++++
 9 files changed, 133 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index 8413cc02665c..7530009d8081 100644
--- a/Makefile
+++ b/Makefile
@@ -1292,7 +1292,26 @@ PHONY += vmlinux
 vmlinux: private _LDFLAGS_vmlinux := $(LDFLAGS_vmlinux)
 vmlinux: export LDFLAGS_vmlinux = $(_LDFLAGS_vmlinux)
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
-vmlinux: fips140-ready
+vmlinux: crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o
+fips140_build = .
+ifeq ($(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE),y)
+fips140_build = fips140_build
+endif
+crypto/fips140/fips140-embedded.o: fips140-ready
+	@echo "  LD      $@"
+	@$(LD) -r -b binary -o $@ $(fips140_build)/crypto/fips140/fips140.ko
+	@$(OBJCOPY) --rename-section .data=.fips140_module_data $@
+
+crypto/fips140/.fips140.hmac: crypto/fips140/fips140-embedded.o
+	@echo "  HMAC    $@"
+	@hmac_key=$$(awk -F'"' '/^CONFIG_CRYPTO_FIPS140_HMAC_KEY=/{print $$2}' .config); \
+	openssl dgst -sha256 -hmac "$$hmac_key" -binary -out $@ $(fips140_build)/crypto/fips140/fips140.ko
+
+crypto/fips140/fips140-digest.o: crypto/fips140/.fips140.hmac
+	@echo "  LD      $@"
+	@$(LD) -r -b binary -o $@ crypto/fips140/.fips140.hmac
+	@$(OBJCOPY) --rename-section .data=.fips140_digest $@
+
 # Ensure fips140.ko is built before embedding
 fips140-ready: crypto/fips140/fips140.o crypto/fips140/.fips140.order crypto/fips140/fips140.mod vmlinux.o | modules_prepare
 	$(Q)$(MAKE) KBUILD_MODULES= -f $(srctree)/scripts/Makefile.modpost
@@ -1300,7 +1319,9 @@ fips140-ready: crypto/fips140/fips140.o crypto/fips140/.fips140.order crypto/fip
 ifneq ($(KBUILD_MODPOST_NOFINAL),1)
 	$(Q)$(MAKE) KBUILD_MODULES=y crypto-module-gen=1 -f $(srctree)/scripts/Makefile.modfinal
 endif
-	@:
+ifeq ($(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE),y)
+	cp "$(fips140_build)/crypto/fips140/fips140.ko" crypto/fips140/fips140.ko;
+endif
 
 # Generate fips140.o from crypto-module.a files
 crypto/fips140/fips140.o: crypto-module.a FORCE
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index ad6133b89e7a..8de7dba3ac12 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -200,6 +200,22 @@ SECTIONS
 	/* everything from this point to __init_begin will be marked RO NX */
 	RO_DATA(PAGE_SIZE)
 
+#ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+	/* FIPS 140 embedded module data */
+	.fips140_embedded : {
+		. = ALIGN(8);
+		_binary_fips140_ko_start = .;
+		KEEP(*(.fips140_module_data))
+		_binary_fips140_ko_end = .;
+	}
+	.fips140_digest : {
+		. = ALIGN(8);
+		_binary_fips140_hmac_start = .;
+		KEEP(*(.fips140_digest))
+		_binary_fips140_hmac_end = .;
+	}
+#endif
+
 	HYPERVISOR_RODATA_SECTIONS
 
 	.got : { *(.got) }
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 3a24a3fc55f5..25fdea6b9f27 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -171,6 +171,22 @@ SECTIONS
 	RO_DATA(PAGE_SIZE)
 	X86_ALIGN_RODATA_END
 
+#ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+	/* FIPS 140 embedded module data */
+	.fips140_embedded : AT(ADDR(.fips140_embedded) - LOAD_OFFSET) {
+		. = ALIGN(8);
+		_binary_fips140_ko_start = .;
+		KEEP(*(.fips140_module_data))
+		_binary_fips140_ko_end = .;
+	}
+	.fips140_digest : AT(ADDR(.fips140_digest) - LOAD_OFFSET) {
+		. = ALIGN(8);
+		_binary_fips140_hmac_start = .;
+		KEEP(*(.fips140_digest))
+		_binary_fips140_hmac_end = .;
+	}
+#endif
+
 	/* Data */
 	.data : AT(ADDR(.data) - LOAD_OFFSET) {
 		/* Start of data section */
diff --git a/crypto/fips140/Kconfig b/crypto/fips140/Kconfig
index 0665e94b9fe0..68b877f0dbab 100644
--- a/crypto/fips140/Kconfig
+++ b/crypto/fips140/Kconfig
@@ -12,4 +12,33 @@ config CRYPTO_FIPS140_EXTMOD
 	  can be enabled to restrict crypto algorithm usage to only
 	  those provided by this module.
 
+	  If unsure, say N.
+config CRYPTO_FIPS140_HMAC_KEY
+	string "FIPS 140-3 external module HMAC key"
+	depends on CRYPTO_FIPS140_EXTMOD
+	default "The quick brown fox jumps over the lazy dog while the sphinx of black quartz judges my vow"
+	help
+	  This is the HMAC key used to build and verify the integrity of
+	  the FIPS module.
+
+	  Must be at least 80 characters.
+config CRYPTO_FIPS140_EXTMOD_SOURCE
+	bool "Use external FIPS module source"
+	depends on CRYPTO_FIPS140_EXTMOD
+	default n
+	help
+	  Use pre-built FIPS modules from an external build directory instead
+	  of freshly built modules from the current kernel build.
+	  
+	  If N, the kernel uses freshly generated crypto modules from the
+	  current build directory:
+	    - crypto/fips140/fips140.ko
+	    - crypto/aes.ko
+	    - crypto/sha256.ko
+	  
+	  If Y, pre-built modules from fips140_build/ are used:
+	    - fips140_build/crypto/fips140/fips140.ko
+	    - fips140_build/crypto/aes.ko
+	    - fips140_build/crypto/sha256.ko
+	  
 	  If unsure, say N.
diff --git a/crypto/fips140/Makefile b/crypto/fips140/Makefile
index fb083022efbb..8f9d3c06ac98 100644
--- a/crypto/fips140/Makefile
+++ b/crypto/fips140/Makefile
@@ -1,8 +1,18 @@
 
 crypto-objs-y += \
 	fips140-module.o \
-	fips140-api.o
+	fips140-api-fips.o
 
-obj-y += fips140-api.o
+obj-y += fips140-api-main.o fips140-loader.o
 
-clean-files:= .fips140.order .fips140.symvers
\ No newline at end of file
+# Explicit rules to compile same source to different objects
+$(obj)/fips140-api-main.o: $(src)/fips140-api.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+$(obj)/fips140-api-fips.o: $(src)/fips140-api.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+CFLAGS_fips140-api-main.o += -I$(srctree)
+CFLAGS_fips140-api-fips.o += -I$(srctree)
+
+clean-files:= .fips140.order .fips140.symvers .fips140.hmac
\ No newline at end of file
diff --git a/crypto/fips140/fips140-loader.c b/crypto/fips140/fips140-loader.c
index cbf5ff59eb2e..7fcb420997cb 100644
--- a/crypto/fips140/fips140-loader.c
+++ b/crypto/fips140/fips140-loader.c
@@ -14,10 +14,17 @@
 
 extern const u8 _binary_fips140_ko_start[];
 extern const u8 _binary_fips140_ko_end[];
+extern const u8 _binary_fips140_hmac_start[];
+extern const u8 _binary_fips140_hmac_end[];
+
 const u8 *_binary_crypto_ko_start;
 EXPORT_SYMBOL_GPL(_binary_crypto_ko_start);
 const u8 *_binary_crypto_ko_end;
 EXPORT_SYMBOL_GPL(_binary_crypto_ko_end);
+const u8 *_binary_crypto_hmac_start;
+EXPORT_SYMBOL_GPL(_binary_crypto_hmac_start);
+const u8 *_binary_crypto_hmac_end;
+EXPORT_SYMBOL_GPL(_binary_crypto_hmac_end);
 
 /* Function to load crypto module from memory */
 extern int load_crypto_module_mem(const char *mem, size_t size);
@@ -26,6 +33,8 @@ static void load_prepare(void)
 {
 	_binary_crypto_ko_start = _binary_fips140_ko_start;
 	_binary_crypto_ko_end = _binary_fips140_ko_end;
+	_binary_crypto_hmac_start = _binary_fips140_hmac_start;
+	_binary_crypto_hmac_end = _binary_fips140_hmac_end;
 }
 
 static int __init fips_loader_init(void)
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index c68dab4d6584..6a2219582258 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -69,11 +69,27 @@ ifeq ($(crypto-module-gen),1)
 	+$(call if_changed,ld_ko_o)
 else
 %.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
-	+$(call if_changed_except,ld_ko_o,$(objtree)/vmlinux)
+	+$(call if_changed_except,ld_ko_o_and_cp_extmod,$(objtree)/vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
 endif
 endif
+
+fips140_build = .
+ifeq ($(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE),y)
+fips140_build = fips140_build
+endif
+
+quiet_cmd_ld_ko_o_and_cp_extmod = LD [M]  $@
+      cmd_ld_ko_o_and_cp_extmod = \
+	$(LD) -r $(KBUILD_LDFLAGS) \
+		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) \
+		-T $(objtree)/scripts/module.lds -o $@ $(filter %.o, $^); \
+	if [ "$(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE)" = "y" ] && \
+	   [ -f "$(fips140_build)/$@" ]; then \
+		echo "  CP [M]  $@"; \
+		cp "$(fips140_build)/$@" "$@"; \
+	fi
 else
 %.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,$(objtree)/vmlinux)
diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
index fcae1e432d9a..93b382e08892 100644
--- a/scripts/Makefile.vmlinux
+++ b/scripts/Makefile.vmlinux
@@ -67,8 +67,12 @@ cmd_link_vmlinux =							\
 	$< "$(LD)" "$(KBUILD_LDFLAGS)" "$(LDFLAGS_vmlinux)" "$@";	\
 	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
 
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+fips140-deps := crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o
+endif
+
 targets += vmlinux.unstripped .vmlinux.export.o
-vmlinux.unstripped: scripts/link-vmlinux.sh vmlinux.o .vmlinux.export.o $(KBUILD_LDS) FORCE
+vmlinux.unstripped: scripts/link-vmlinux.sh vmlinux.o .vmlinux.export.o $(KBUILD_LDS) $(fips140-deps) FORCE
 	+$(call if_changed_dep,link_vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF
 vmlinux.unstripped: $(RESOLVE_BTFIDS) $(srctree)/scripts/gen-btf.sh
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index f99e196abeea..34c0a162da84 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -75,6 +75,11 @@ vmlinux_link()
 	fi
 
 	objs="${objs} .vmlinux.export.o"
+
+	if is_enabled CONFIG_CRYPTO_FIPS140_EXTMOD; then
+		objs="${objs} crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o"
+	fi
+
 	objs="${objs} init/version-timestamp.o"
 
 	if [ "${SRCARCH}" = "um" ]; then
-- 
2.47.3

[PATCH 10/17] build: add CONFIG_DEBUG_INFO_BTF_MODULES support for the standalone crypto kernel module

[Jay Wang]

When CONFIG_DEBUG_INFO_BTF_MODULES=y, there are two problems for the
standalone crypto kernel module.

First, it requires a make dependency `.ko: vmlinux` because it takes
vmlinux as input to generate BTF info for the module, and inserts such
info into the `.ko` module binary as a dedicated ELF section. This can
cause an unwanted circular make rule dependency `fips140.ko:vmlinux`
because fips140.ko is already embedded into vmlinux.

To address this issue, we reuse the same script to explicitly generate
fips140.ko's BTF info in the vmlinux generation process to avoid the
circular make dependency. We link vmlinux first, then use it to generate
fips140.ko's BTF info, and then embed the fips140.ko and its BTF info
back with vmlinux by re-linking. Since the fips140.ko's BTF info is
embedded as data only into vmlinux, the BTF info generated using the
first linked vmlinux will be the same as if using the latest vmlinux.

Second, CONFIG_DEBUG_INFO_BTF_MODULES=y will insert BTF info into
fips140.ko binary, which means the previously generated module signature
on "fips140.ko" binary becomes invalid, thus needing regeneration.

To avoid this issue, we don't re-insert module's BTF info into
fips140.ko binary (as normally done), but keep such info as a separate
file, and embed into vmlinux as separate ELF section. By doing this,
the fips140.ko binary remains unchanged while its latest up-to-date BTF
info is available to kernel.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 arch/arm64/kernel/vmlinux.lds.S |  8 ++++++++
 arch/x86/kernel/vmlinux.lds.S   |  8 ++++++++
 crypto/fips140/Makefile         |  2 +-
 crypto/fips140/fips140-loader.c | 19 +++++++++++++++++++
 kernel/bpf/btf.c                | 20 ++++++++++++++++++++
 kernel/module/main.c            | 27 +++++++++++++++++++++++++++
 scripts/Makefile.vmlinux        | 28 ++++++++++++++++++++++++++++
 scripts/link-vmlinux.sh         |  3 +++
 8 files changed, 114 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 8de7dba3ac12..41223fa3f14e 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -214,6 +214,14 @@ SECTIONS
 		KEEP(*(.fips140_digest))
 		_binary_fips140_hmac_end = .;
 	}
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	.fips140_btf : {
+		. = ALIGN(8);
+		__start_fips140_btf = .;
+		KEEP(*(.fips140_btf))
+		__stop_fips140_btf = .;
+	}
+#endif
 #endif
 
 	HYPERVISOR_RODATA_SECTIONS
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 25fdea6b9f27..e07c1b5c52cf 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -185,6 +185,14 @@ SECTIONS
 		KEEP(*(.fips140_digest))
 		_binary_fips140_hmac_end = .;
 	}
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	.fips140_btf : AT(ADDR(.fips140_btf) - LOAD_OFFSET) {
+		. = ALIGN(8);
+		__start_fips140_btf = .;
+		KEEP(*(.fips140_btf))
+		__stop_fips140_btf = .;
+	}
+#endif
 #endif
 
 	/* Data */
diff --git a/crypto/fips140/Makefile b/crypto/fips140/Makefile
index 8f9d3c06ac98..ac8ae42eb0fa 100644
--- a/crypto/fips140/Makefile
+++ b/crypto/fips140/Makefile
@@ -15,4 +15,4 @@ $(obj)/fips140-api-fips.o: $(src)/fips140-api.c FORCE
 CFLAGS_fips140-api-main.o += -I$(srctree)
 CFLAGS_fips140-api-fips.o += -I$(srctree)
 
-clean-files:= .fips140.order .fips140.symvers .fips140.hmac
\ No newline at end of file
+clean-files:= .fips140.order .fips140.symvers .fips140.hmac .fips140.ko.btf
\ No newline at end of file
diff --git a/crypto/fips140/fips140-loader.c b/crypto/fips140/fips140-loader.c
index 7fcb420997cb..13c82ffdc65b 100644
--- a/crypto/fips140/fips140-loader.c
+++ b/crypto/fips140/fips140-loader.c
@@ -26,6 +26,20 @@ EXPORT_SYMBOL_GPL(_binary_crypto_hmac_start);
 const u8 *_binary_crypto_hmac_end;
 EXPORT_SYMBOL_GPL(_binary_crypto_hmac_end);
 
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+extern const u8 __start_fips140_btf[];
+extern const u8 __stop_fips140_btf[];
+const u8 *__start_crypto_btf;
+const u8 *__stop_crypto_btf;
+#endif
+
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+extern const u8 __start_fips140_btf[];
+extern const u8 __stop_fips140_btf[];
+const u8 *__start_crypto_btf;
+const u8 *__stop_crypto_btf;
+#endif
+
 /* Function to load crypto module from memory */
 extern int load_crypto_module_mem(const char *mem, size_t size);
 
@@ -35,6 +49,11 @@ static void load_prepare(void)
 	_binary_crypto_ko_end = _binary_fips140_ko_end;
 	_binary_crypto_hmac_start = _binary_fips140_hmac_start;
 	_binary_crypto_hmac_end = _binary_fips140_hmac_end;
+	
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	__start_crypto_btf = __start_fips140_btf;
+	__stop_crypto_btf = __stop_fips140_btf;
+#endif
 }
 
 static int __init fips_loader_init(void)
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 7708958e3fb8..7e3b0d0acb9c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -8408,6 +8408,26 @@ static int __init btf_module_init(void)
 	return 0;
 }
 
+#if defined(CONFIG_CRYPTO_FIPS140_EXTMOD) && defined(CONFIG_DEBUG_INFO_BTF_MODULES)
+/* Handle deferred BTF registration for FIPS140 loaded before btf_kobj exists */
+struct module *fips140_deferred_mod = NULL;
+
+static int __init register_deferred_fips140_btf(void)
+{	
+	if (fips140_deferred_mod && btf_kobj) {
+		/* Manually trigger BTF registration for FIPS140 */
+		btf_module_notify(NULL, MODULE_STATE_COMING, fips140_deferred_mod);
+		fips140_deferred_mod = NULL;
+		pr_info("FIPS140 BTF registration completed\n");
+	} else {
+		pr_info("FIPS140 BTF registration skipped: deferred_mod=%p, btf_kobj=%p\n",
+			fips140_deferred_mod, btf_kobj);
+	}
+	return 0;
+}
+late_initcall(register_deferred_fips140_btf);
+#endif
+
 fs_initcall(btf_module_init);
 #endif /* CONFIG_DEBUG_INFO_BTF_MODULES */
 
diff --git a/kernel/module/main.c b/kernel/module/main.c
index dad84f0548ac..7e34c8408705 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -2651,6 +2651,23 @@ static int find_module_sections(struct module *mod, struct load_info *info)
 	mod->btf_data = any_section_objs(info, ".BTF", 1, &mod->btf_data_size);
 	mod->btf_base_data = any_section_objs(info, ".BTF.base", 1,
 					      &mod->btf_base_data_size);
+	
+#ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+	/* Inject embedded BTF for FIPS140 module */
+	if (!mod->btf_data && !strcmp(mod->name, "fips140")) {
+		extern u8 *__start_crypto_btf;
+		extern u8 *__stop_crypto_btf;
+		size_t btf_size = __stop_crypto_btf - __start_crypto_btf;
+		
+		pr_info("FIPS140: Attempting BTF injection, btf_size=%zu\n", btf_size);
+		
+		if (btf_size > 0) {
+			mod->btf_data = __start_crypto_btf;
+			mod->btf_data_size = btf_size;
+			pr_info("FIPS140: Injected embedded BTF data, size %zu\n", btf_size);
+		}
+	}
+#endif
 #endif
 #ifdef CONFIG_JUMP_LABEL
 	mod->jump_entries = section_objs(info, "__jump_table",
@@ -3333,6 +3350,16 @@ static int prepare_coming_module(struct module *mod)
 	err = blocking_notifier_call_chain_robust(&module_notify_list,
 			MODULE_STATE_COMING, MODULE_STATE_GOING, mod);
 	err = notifier_to_errno(err);
+#if defined(CONFIG_CRYPTO_FIPS140_EXTMOD) && defined(CONFIG_DEBUG_INFO_BTF_MODULES)
+	/* Since fips140 module is loaded too early when BTF subsystem is not ready,
+	 * record this module for later BTF registration processing */
+	if (!strcmp(mod->name, "fips140")) {
+		pr_info("FIPS140 BTF MODULE_STATE_COMING: processing BTF registration\n");
+		extern struct module *fips140_deferred_mod;
+		fips140_deferred_mod = mod;  /* Store for later reference */
+	}
+#endif
+
 	if (err)
 		klp_module_going(mod);
 
diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
index 93b382e08892..b30d65f8b6b3 100644
--- a/scripts/Makefile.vmlinux
+++ b/scripts/Makefile.vmlinux
@@ -62,10 +62,38 @@ endif
 
 ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
 
+fips140_build = .
+ifeq ($(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE),y)
+fips140_build = fips140_build
+endif
+
 # Final link of vmlinux with optional arch pass after final link
 cmd_link_vmlinux =							\
 	$< "$(LD)" "$(KBUILD_LDFLAGS)" "$(LDFLAGS_vmlinux)" "$@";	\
 	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
+ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+      cmd_link_vmlinux += ; \
+	cp $(fips140_build)/crypto/fips140/fips140.ko crypto/fips140/fips140.ko.tmp; \
+	LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base $@ crypto/fips140/fips140.ko.tmp; \
+	$(RESOLVE_BTFIDS) -b $@ crypto/fips140/fips140.ko.tmp; \
+	$(OBJCOPY) --dump-section=.BTF=crypto/fips140/.fips140.ko.btf crypto/fips140/fips140.ko.tmp; \
+	cp crypto/fips140/.fips140.ko.btf crypto/fips140/.fips140.ko.btf.first; \
+	rm -f crypto/fips140/fips140.ko.tmp; \
+	$(LD) -r -b binary -o crypto/fips140/fips140_btf.o crypto/fips140/.fips140.ko.btf; \
+	$(OBJCOPY) --rename-section .data=.fips140_btf crypto/fips140/fips140_btf.o; \
+	rm -f $@; \
+	FIPS140_BTF_RELINK=1 $< "$(LD)" "$(KBUILD_LDFLAGS)" "$(LDFLAGS_vmlinux)" "$@"; \
+	cp $(fips140_build)/crypto/fips140/fips140.ko crypto/fips140/fips140.ko.tmp2; \
+	LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base $@ crypto/fips140/fips140.ko.tmp2; \
+	$(RESOLVE_BTFIDS) -b $@ crypto/fips140/fips140.ko.tmp2; \
+	$(OBJCOPY) --dump-section=.BTF=crypto/fips140/.fips140.ko.btf.second crypto/fips140/fips140.ko.tmp2; \
+	rm -f crypto/fips140/fips140.ko.tmp2; \
+	diff crypto/fips140/.fips140.ko.btf.first crypto/fips140/.fips140.ko.btf.second >/dev/null || echo "Module BTF differs"; \
+	rm -f crypto/fips140/.fips140.ko.btf.first crypto/fips140/.fips140.ko.btf.second; \
+	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
+endif
+endif
 
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 fips140-deps := crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 34c0a162da84..37c9b8576ec7 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -78,6 +78,9 @@ vmlinux_link()
 
 	if is_enabled CONFIG_CRYPTO_FIPS140_EXTMOD; then
 		objs="${objs} crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o"
+		if is_enabled CONFIG_DEBUG_INFO_BTF_MODULES && [ -n "${FIPS140_BTF_RELINK}" ] && [ -f crypto/fips140/fips140_btf.o ]; then
+			objs="${objs} crypto/fips140/fips140_btf.o"
+		fi
 	fi
 
 	objs="${objs} init/version-timestamp.o"
-- 
2.47.3

[PATCH 11/17] Allow selective crypto module loading at boot based on FIPS mode

[Jay Wang]

Introduce CONFIG_CRYPTO_FIPS140_DUAL_VERSION to enable dual crypto module
versions within a single kernel build, allowing boot-time selection based on
FIPS mode status.

This configuration allows FIPS mode to use pre-compiled certified crypto
modules from external source, while regular mode uses freshly built kernel
crypto implementation for optimal performance and latest security features.

The implementation embeds both certified and non-certified fips140.ko
modules in vmlinux and adds new linker sections (.nonfips140_embedded,
.nonfips140_btf) for non-FIPS crypto module storage. It modifies
fips140-loader.c to select appropriate module at boot time based on
fips_enabled flag, updates build system to generate and embed both module
versions, and includes BTF support for both module variants when
CONFIG_DEBUG_INFO_BTF_MODULES is enabled.

For modular crypto algorithms (e.g., aes.ko), they are not automatically
duplicated. They should either be built-in to fips140.ko for automatic
duplication, or require userspace utilities like modprobe to handle
proper isolation between FIPS and non-FIPS modular crypto implementations.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 Makefile                        | 13 +++++++++++++
 arch/arm64/kernel/vmlinux.lds.S | 16 ++++++++++++++++
 arch/x86/kernel/vmlinux.lds.S   | 16 ++++++++++++++++
 crypto/fips140/Kconfig          | 24 ++++++++++++++++++++++++
 crypto/fips140/Makefile         |  5 ++++-
 crypto/fips140/fips140-loader.c | 28 +++++++++++++++++++++++++++-
 scripts/Makefile.vmlinux        | 29 +++++++++++++++++++++++++++--
 scripts/link-vmlinux.sh         |  6 ++++++
 8 files changed, 133 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 7530009d8081..b3a9f7a17ddf 100644
--- a/Makefile
+++ b/Makefile
@@ -1293,6 +1293,9 @@ vmlinux: private _LDFLAGS_vmlinux := $(LDFLAGS_vmlinux)
 vmlinux: export LDFLAGS_vmlinux = $(_LDFLAGS_vmlinux)
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 vmlinux: crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o
+ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+vmlinux: crypto/fips140/nonfips140-embedded.o
+endif
 fips140_build = .
 ifeq ($(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE),y)
 fips140_build = fips140_build
@@ -1302,6 +1305,14 @@ crypto/fips140/fips140-embedded.o: fips140-ready
 	@$(LD) -r -b binary -o $@ $(fips140_build)/crypto/fips140/fips140.ko
 	@$(OBJCOPY) --rename-section .data=.fips140_module_data $@
 
+ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+crypto/fips140/nonfips140-embedded.o: fips140-ready
+	@echo "  LD      $@"
+	@$(LD) -r -b binary -o $@ crypto/fips140/fips140.ko
+	@$(OBJCOPY) --rename-section .data=.nonfips140_module_data \
+		--prefix-symbols nonfips140_ $@
+endif
+
 crypto/fips140/.fips140.hmac: crypto/fips140/fips140-embedded.o
 	@echo "  HMAC    $@"
 	@hmac_key=$$(awk -F'"' '/^CONFIG_CRYPTO_FIPS140_HMAC_KEY=/{print $$2}' .config); \
@@ -1319,9 +1330,11 @@ fips140-ready: crypto/fips140/fips140.o crypto/fips140/.fips140.order crypto/fip
 ifneq ($(KBUILD_MODPOST_NOFINAL),1)
 	$(Q)$(MAKE) KBUILD_MODULES=y crypto-module-gen=1 -f $(srctree)/scripts/Makefile.modfinal
 endif
+ifndef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
 ifeq ($(CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE),y)
 	cp "$(fips140_build)/crypto/fips140/fips140.ko" crypto/fips140/fips140.ko;
 endif
+endif
 
 # Generate fips140.o from crypto-module.a files
 crypto/fips140/fips140.o: crypto-module.a FORCE
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 41223fa3f14e..0722e07c5551 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -222,6 +222,22 @@ SECTIONS
 		__stop_fips140_btf = .;
 	}
 #endif
+#ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+	.nonfips140_embedded : {
+		. = ALIGN(8);
+		_binary_nonfips140_ko_start = .;
+		KEEP(*(.nonfips140_module_data))
+		_binary_nonfips140_ko_end = .;
+	}
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	.nonfips140_btf : {
+		. = ALIGN(8);
+		__start_nonfips140_btf = .;
+		KEEP(*(.nonfips140_btf))
+		__stop_nonfips140_btf = .;
+	}
+#endif
+#endif
 #endif
 
 	HYPERVISOR_RODATA_SECTIONS
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index e07c1b5c52cf..aa1b97d7aabd 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -193,6 +193,22 @@ SECTIONS
 		__stop_fips140_btf = .;
 	}
 #endif
+#ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+	.nonfips140_embedded : AT(ADDR(.nonfips140_embedded) - LOAD_OFFSET) {
+		. = ALIGN(8);
+		_binary_nonfips140_ko_start = .;
+		KEEP(*(.nonfips140_module_data))
+		_binary_nonfips140_ko_end = .;
+	}
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+	.nonfips140_btf : AT(ADDR(.nonfips140_btf) - LOAD_OFFSET) {
+		. = ALIGN(8);
+		__start_nonfips140_btf = .;
+		KEEP(*(.nonfips140_btf))
+		__stop_nonfips140_btf = .;
+	}
+#endif
+#endif
 #endif
 
 	/* Data */
diff --git a/crypto/fips140/Kconfig b/crypto/fips140/Kconfig
index 68b877f0dbab..7d8997aa1094 100644
--- a/crypto/fips140/Kconfig
+++ b/crypto/fips140/Kconfig
@@ -42,3 +42,27 @@ config CRYPTO_FIPS140_EXTMOD_SOURCE
 	    - fips140_build/crypto/sha256.ko
 	  
 	  If unsure, say N.
+config CRYPTO_FIPS140_DUAL_VERSION
+	bool "Enable dual crypto versions for FIPS and regular modes"
+	depends on CRYPTO_FIPS140_EXTMOD && CRYPTO_FIPS140_EXTMOD_SOURCE
+	default n
+	help
+	  Enable keeping two crypto module versions in the same kernel build
+	  for boot-time switching based on FIPS mode status. This allows:
+	  - Non-FIPS users: Get latest crypto algorithms built from current
+	    kernel sources for optimal performance and security features
+	  - FIPS users: Get pre-compiled certified crypto modules that have
+	    undergone formal validation and certification processes
+
+	  When enabled:
+
+	  For core fips140.ko:
+	  - FIPS mode: Uses certified module from CRYPTO_FIPS140_EXTMOD_SOURCE
+	  - Regular mode: Uses freshly built kernel crypto implementation
+
+	  For modular algorithms (e.g., aes.ko), they are not duplicated
+	  automatically. Either make them built-in to be included into
+	  fips140.ko for automatic duplication, or require OS utilities such
+	  as `modprobe` to correctly isolate modular cryptos in filesystems.
+
+	  If unsure, say N.
\ No newline at end of file
diff --git a/crypto/fips140/Makefile b/crypto/fips140/Makefile
index ac8ae42eb0fa..c99bf2948432 100644
--- a/crypto/fips140/Makefile
+++ b/crypto/fips140/Makefile
@@ -15,4 +15,7 @@ $(obj)/fips140-api-fips.o: $(src)/fips140-api.c FORCE
 CFLAGS_fips140-api-main.o += -I$(srctree)
 CFLAGS_fips140-api-fips.o += -I$(srctree)
 
-clean-files:= .fips140.order .fips140.symvers .fips140.hmac .fips140.ko.btf
\ No newline at end of file
+clean-files:= .fips140.order .fips140.symvers .fips140.hmac .fips140.ko.btf
+ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+clean-files += .nonfips140.ko.btf
+endif
\ No newline at end of file
diff --git a/crypto/fips140/fips140-loader.c b/crypto/fips140/fips140-loader.c
index 13c82ffdc65b..826075928723 100644
--- a/crypto/fips140/fips140-loader.c
+++ b/crypto/fips140/fips140-loader.c
@@ -11,12 +11,20 @@
 #include <linux/elf.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>
+#include <linux/fips.h>
 
 extern const u8 _binary_fips140_ko_start[];
 extern const u8 _binary_fips140_ko_end[];
 extern const u8 _binary_fips140_hmac_start[];
 extern const u8 _binary_fips140_hmac_end[];
 
+#ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+/* For non-FIPS mode: no module signature/HMAC is required,
+ * so only include binary start/end address without module sig address */
+extern const u8 _binary_nonfips140_ko_start[];
+extern const u8 _binary_nonfips140_ko_end[];
+#endif
+
 const u8 *_binary_crypto_ko_start;
 EXPORT_SYMBOL_GPL(_binary_crypto_ko_start);
 const u8 *_binary_crypto_ko_end;
@@ -29,6 +37,10 @@ EXPORT_SYMBOL_GPL(_binary_crypto_hmac_end);
 #ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 extern const u8 __start_fips140_btf[];
 extern const u8 __stop_fips140_btf[];
+#ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+extern const u8 __start_nonfips140_btf[];
+extern const u8 __stop_nonfips140_btf[];
+#endif
 const u8 *__start_crypto_btf;
 const u8 *__stop_crypto_btf;
 #endif
@@ -49,11 +61,25 @@ static void load_prepare(void)
 	_binary_crypto_ko_end = _binary_fips140_ko_end;
 	_binary_crypto_hmac_start = _binary_fips140_hmac_start;
 	_binary_crypto_hmac_end = _binary_fips140_hmac_end;
-	
+
 #ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	__start_crypto_btf = __start_fips140_btf;
 	__stop_crypto_btf = __stop_fips140_btf;
 #endif
+
+#ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+	if (!fips_enabled) {
+		_binary_crypto_ko_start = _binary_nonfips140_ko_start;
+		_binary_crypto_ko_end = _binary_nonfips140_ko_end;
+		_binary_crypto_hmac_start = NULL;
+		_binary_crypto_hmac_end = NULL;
+
+#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+		__start_crypto_btf = __start_nonfips140_btf;
+		__stop_crypto_btf = __stop_nonfips140_btf;
+#endif
+		}
+#endif
 }
 
 static int __init fips_loader_init(void)
diff --git a/scripts/Makefile.vmlinux b/scripts/Makefile.vmlinux
index b30d65f8b6b3..996d016e518c 100644
--- a/scripts/Makefile.vmlinux
+++ b/scripts/Makefile.vmlinux
@@ -81,7 +81,18 @@ ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	cp crypto/fips140/.fips140.ko.btf crypto/fips140/.fips140.ko.btf.first; \
 	rm -f crypto/fips140/fips140.ko.tmp; \
 	$(LD) -r -b binary -o crypto/fips140/fips140_btf.o crypto/fips140/.fips140.ko.btf; \
-	$(OBJCOPY) --rename-section .data=.fips140_btf crypto/fips140/fips140_btf.o; \
+	$(OBJCOPY) --rename-section .data=.fips140_btf crypto/fips140/fips140_btf.o
+ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+      cmd_link_vmlinux += ; \
+	cp crypto/fips140/fips140.ko crypto/fips140/nonfips140.ko.tmp; \
+	LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base $@ crypto/fips140/nonfips140.ko.tmp; \
+	$(RESOLVE_BTFIDS) -b $@ crypto/fips140/nonfips140.ko.tmp; \
+	$(OBJCOPY) --dump-section=.BTF=crypto/fips140/.nonfips140.ko.btf crypto/fips140/nonfips140.ko.tmp; \
+	rm -f crypto/fips140/nonfips140.ko.tmp; \
+	$(LD) -r -b binary -o crypto/fips140/nonfips140_btf.o crypto/fips140/.nonfips140.ko.btf; \
+	$(OBJCOPY) --rename-section .data=.nonfips140_btf --prefix-symbols nonfips140_ crypto/fips140/nonfips140_btf.o
+endif
+	  cmd_link_vmlinux += ; \
 	rm -f $@; \
 	FIPS140_BTF_RELINK=1 $< "$(LD)" "$(KBUILD_LDFLAGS)" "$(LDFLAGS_vmlinux)" "$@"; \
 	cp $(fips140_build)/crypto/fips140/fips140.ko crypto/fips140/fips140.ko.tmp2; \
@@ -90,13 +101,27 @@ ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	$(OBJCOPY) --dump-section=.BTF=crypto/fips140/.fips140.ko.btf.second crypto/fips140/fips140.ko.tmp2; \
 	rm -f crypto/fips140/fips140.ko.tmp2; \
 	diff crypto/fips140/.fips140.ko.btf.first crypto/fips140/.fips140.ko.btf.second >/dev/null || echo "Module BTF differs"; \
-	rm -f crypto/fips140/.fips140.ko.btf.first crypto/fips140/.fips140.ko.btf.second; \
+	rm -f crypto/fips140/.fips140.ko.btf.first crypto/fips140/.fips140.ko.btf.second
+ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+      cmd_link_vmlinux += ; \
+	cp crypto/fips140/fips140.ko crypto/fips140/nonfips140.ko.tmp2; \
+	LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base $@ crypto/fips140/nonfips140.ko.tmp2; \
+	$(RESOLVE_BTFIDS) -b $@ crypto/fips140/nonfips140.ko.tmp2; \
+	$(OBJCOPY) --dump-section=.BTF=crypto/fips140/.nonfips140.ko.btf.second crypto/fips140/nonfips140.ko.tmp2; \
+	rm -f crypto/fips140/nonfips140.ko.tmp2; \
+	diff crypto/fips140/.nonfips140.ko.btf crypto/fips140/.nonfips140.ko.btf.second >/dev/null || echo "Nonfips140 Module BTF differs"; \
+	rm -f crypto/fips140/.nonfips140.ko.btf.second
+endif
+	  cmd_link_vmlinux += ; \
 	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
 endif
 endif
 
 ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
 fips140-deps := crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o
+ifdef CONFIG_CRYPTO_FIPS140_DUAL_VERSION
+fips140-deps += crypto/fips140/nonfips140-embedded.o
+endif
 endif
 
 targets += vmlinux.unstripped .vmlinux.export.o
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 37c9b8576ec7..43a272e8d3a4 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -78,8 +78,14 @@ vmlinux_link()
 
 	if is_enabled CONFIG_CRYPTO_FIPS140_EXTMOD; then
 		objs="${objs} crypto/fips140/fips140-embedded.o crypto/fips140/fips140-digest.o"
+		if is_enabled CONFIG_CRYPTO_FIPS140_DUAL_VERSION; then
+			objs="${objs} crypto/fips140/nonfips140-embedded.o"
+		fi
 		if is_enabled CONFIG_DEBUG_INFO_BTF_MODULES && [ -n "${FIPS140_BTF_RELINK}" ] && [ -f crypto/fips140/fips140_btf.o ]; then
 			objs="${objs} crypto/fips140/fips140_btf.o"
+			if is_enabled CONFIG_CRYPTO_FIPS140_DUAL_VERSION && [ -f crypto/fips140/nonfips140_btf.o ]; then
+				objs="${objs} crypto/fips140/nonfips140_btf.o"
+			fi
 		fi
 	fi
 
-- 
2.47.3

[PATCH 12/17] Execute crypto initcalls during module initialization

[Jay Wang]

After moving builtin cryptos into the standalone crypto module, to ensure
all such cryptos are properly initialized as they were builtin, the
initcalls of the cryptos should be executed at proper booting time.

To do so, this patch adds run_initcalls() function to execute the
initialization calls of the collected cryptos right after the module is
loaded. The function iterates through initcalls in the __fips_initcalls
section (which stores all function addresses of the collected crypto
initcalls) and executes them.

A key consideration is to ensure the crypto initcalls are executed in
proper order, for instance, some crypto initcalls are ought to execute
at `late_initcall` boot time level while some should be executed at
`module_init` boot time level. To do so, this patch enables coordination
between kernel and the crypto module initialization to preserve proper
execution order by hijacking the kernel initialization function
`do_initcall_level` with added synchronization helpers.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/fips140/fips140-loader.c | 92 +++++++++++++++++++++++++++++++++
 crypto/fips140/fips140-module.c | 50 ++++++++++++++++++
 crypto/fips140/fips140-module.h | 12 +++++
 include/linux/init.h            | 10 ++++
 init/main.c                     |  4 ++
 5 files changed, 168 insertions(+)

diff --git a/crypto/fips140/fips140-loader.c b/crypto/fips140/fips140-loader.c
index 826075928723..d9a477421063 100644
--- a/crypto/fips140/fips140-loader.c
+++ b/crypto/fips140/fips140-loader.c
@@ -107,3 +107,95 @@ static int __init fips_loader_init(void)
 	vfree(vmalloc_mem); // Free after successful module loading
 	return ret;
 }
+
+/* FIPS140 synchronization between kernel and module
+ *
+ * Synchronization levels map kernel initcall levels to FIPS module levels:
+ * - Level 0: subsys_initcall (kernel init level 4) - Basic subsystem initialization
+ * - Level 1: device_initcall (kernel init level 6) - Device driver initialization  
+ * - Level 2: late_initcall (kernel init level 7) - Late system initialization
+ *
+ * The kernel marks each level complete and waits for the FIPS module to
+ * complete the corresponding level before proceeding to ensure proper
+ * initialization ordering between kernel crypto and FIPS module.
+ */
+atomic_t fips140_kernel_level_complete = ATOMIC_INIT(0);
+atomic_t fips140_module_level_complete = ATOMIC_INIT(0);
+
+/* Wait queues for efficient synchronization */
+DECLARE_WAIT_QUEUE_HEAD(fips140_kernel_wq);
+DECLARE_WAIT_QUEUE_HEAD(fips140_module_wq);
+
+void fips140_mark_kernel_level_complete(int level)
+{
+	atomic_or(1 << level, &fips140_kernel_level_complete);
+	wake_up(&fips140_kernel_wq);
+}
+
+bool fips140_is_kernel_level_complete(int level)
+{
+	return atomic_read(&fips140_kernel_level_complete) & (1 << level);
+}
+
+bool fips140_is_module_level_complete(int level)
+{
+	return atomic_read(&fips140_module_level_complete) & (1 << level);
+}
+
+void fips140_mark_module_level_complete(int level)
+{
+	atomic_or(1 << level, &fips140_module_level_complete);
+	wake_up(&fips140_module_wq);
+}
+
+static int __init fips140_sync_thread(void *data)
+{
+	pr_info("FIPS 140: starting sync thread\n");
+	
+	/* Call FIPS loader explicitly */
+	int ret = fips_loader_init();
+	if (ret)
+		panic("FIPS 140: loader initialization failed: %d\n", ret);
+	
+	pr_info("FIPS 140: sync thread finished\n");
+	return 0;
+}
+
+void __init start_fips140_loader(void)
+{
+	struct task_struct *task;
+	
+	task = kthread_run(fips140_sync_thread, NULL, "fips140_sync");
+	if (IS_ERR(task)) {
+		panic("FIPS 140: failed to create sync thread\n");
+	}
+}
+
+void __init wait_until_fips140_level_sync(int level)
+{
+	/* Map kernel initcall levels to FIPS module levels */
+	int fips_level = -1;
+	if (level == 3) /* Start FIPS loader thread at arch_initcall_sync level */
+		start_fips140_loader(); 
+	if (level == 4) /* subsys_initcall */
+		fips_level = 0;
+	else if (level == 6) /* device_initcall */
+		fips_level = 1;
+	else if (level == 7) /* late_initcall */
+		fips_level = 2;
+
+	if (fips_level >= 0) {
+		/* Mark kernel level complete and wait for module level completion */
+		fips140_mark_kernel_level_complete(fips_level);
+		wait_event(fips140_module_wq, fips140_is_module_level_complete(fips_level));
+	}
+}
+
+EXPORT_SYMBOL(fips140_kernel_level_complete);
+EXPORT_SYMBOL(fips140_module_level_complete);
+EXPORT_SYMBOL(fips140_kernel_wq);
+EXPORT_SYMBOL(fips140_module_wq);
+EXPORT_SYMBOL(fips140_mark_kernel_level_complete);
+EXPORT_SYMBOL(fips140_is_kernel_level_complete);
+EXPORT_SYMBOL(fips140_is_module_level_complete);
+EXPORT_SYMBOL(fips140_mark_module_level_complete);
\ No newline at end of file
diff --git a/crypto/fips140/fips140-module.c b/crypto/fips140/fips140-module.c
index a942de8780ef..e0e669ba1b5e 100644
--- a/crypto/fips140/fips140-module.c
+++ b/crypto/fips140/fips140-module.c
@@ -16,9 +16,59 @@
 
 #define CRYPTO_INTERNAL "CRYPTO_INTERNAL"
 
+static int __init run_initcalls(void)
+{
+	typedef int (*initcall_t)(void);
+	
+	extern initcall_t __fips140_initcall0_start[], __fips140_initcall0_end[];
+	extern initcall_t __fips140_initcall1_start[], __fips140_initcall1_end[];
+	extern initcall_t __fips140_initcall2_start[], __fips140_initcall2_end[];
+
+	initcall_t *starts[] = {
+		__fips140_initcall0_start,
+		__fips140_initcall1_start,
+		__fips140_initcall2_start,
+	};
+	
+	initcall_t *ends[] = {
+		__fips140_initcall0_end,
+		__fips140_initcall1_end,
+		__fips140_initcall2_end,
+	};
+
+	pr_info("FIPS 140: run_initcalls starting\n");
+
+	for (int level = 0; level < ARRAY_SIZE(starts); level++) {
+		
+		/* Run FIPS initcalls for this level */
+		for (initcall_t *initcall = starts[level]; initcall < ends[level]; ++initcall) {
+			int ret;
+			initcall_t fn = *initcall;
+			
+			ret = fn();
+			if (!ret || ret == -ENODEV)
+				continue;
+
+			pr_err("FIPS 140: initcall %pS failed: %d\n", fn, ret);
+		}
+	
+		if (level < 2)
+			fips140_mark_module_level_complete(level);
+		/* Wait for kernel to complete this level */
+		wait_event(fips140_kernel_wq, fips140_is_kernel_level_complete(level));
+	}
+
+	pr_info("FIPS 140: run_initcalls finished\n");
+	return 0;
+}
+
 /* Initialize the FIPS 140 module */
 static int __init fips140_init(void)
 {
+    pr_info("loading " FIPS140_MODULE_NAME "\n");
+
+	run_initcalls();
+	fips140_mark_module_level_complete(2);
     return 0;
 }
 
diff --git a/crypto/fips140/fips140-module.h b/crypto/fips140/fips140-module.h
index ed2b6e17969f..e95dac8eeda9 100644
--- a/crypto/fips140/fips140-module.h
+++ b/crypto/fips140/fips140-module.h
@@ -10,5 +10,17 @@
 #include <linux/crypto.h>
 #include <crypto/algapi.h>
 #include <linux/init.h>
+#include <linux/atomic.h>
+#include <linux/wait.h>
+
+/* FIPS140 synchronization between kernel and module */
+extern atomic_t fips140_kernel_level_complete;
+extern atomic_t fips140_module_level_complete;
+extern wait_queue_head_t fips140_kernel_wq;
+
+void fips140_mark_kernel_level_complete(int level);
+bool fips140_is_kernel_level_complete(int level);
+bool fips140_is_module_level_complete(int level);
+void fips140_mark_module_level_complete(int level);
 
 #endif /* _CRYPTO_FIPS140_MODULE_H */
diff --git a/include/linux/init.h b/include/linux/init.h
index 40331923b9f4..eefbdfac1d41 100644
--- a/include/linux/init.h
+++ b/include/linux/init.h
@@ -392,4 +392,14 @@ void __init parse_early_options(char *cmdline);
 #define __exit_p(x) NULL
 #endif
 
+#if defined(CONFIG_CRYPTO_FIPS140_EXTMOD) && !defined(__ASSEMBLY__)
+/* FIPS140 synchronization between kernel and module */
+void fips140_mark_kernel_level_complete(int level);
+bool fips140_is_kernel_level_complete(int level);
+bool fips140_is_module_level_complete(int level);
+void fips140_mark_module_level_complete(int level);
+void start_fips140_loader(void);
+void wait_until_fips140_level_sync(int level);
+#endif /* CONFIG_CRYPTO_FIPS140_EXTMOD && !__ASSEMBLY__ */
+
 #endif /* _LINUX_INIT_H */
diff --git a/init/main.c b/init/main.c
index b84818ad9685..0324226fe7b9 100644
--- a/init/main.c
+++ b/init/main.c
@@ -1438,6 +1438,10 @@ static void __init do_initcall_level(int level, char *command_line)
 	do_trace_initcall_level(initcall_level_names[level]);
 	for (fn = initcall_levels[level]; fn < initcall_levels[level+1]; fn++)
 		do_one_initcall(initcall_from_entry(fn));
+
+#ifdef CONFIG_CRYPTO_FIPS140_EXTMOD
+	wait_until_fips140_level_sync(level);
+#endif
 }
 
 static void __init do_initcalls(void)
-- 
2.47.3

[PATCH 13/17] crypto/algapi.c: skip crypto_check_module_sig() for the standalone crypto module

[Jay Wang]

From: Vegard Nossum <vegard.nossum@oracle.com>

The signature check in regular module verification process
`crypto_check_module_sig()` is skipped for this standalone crypto module
because its signature has already been checked during load (as described
in later patch).

Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
[Revise commit message]
Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/algapi.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/crypto/algapi.c b/crypto/algapi.c
index 37de377719ae..663698e0cd65 100644
--- a/crypto/algapi.c
+++ b/crypto/algapi.c
@@ -24,7 +24,19 @@ static LIST_HEAD(crypto_template_list);
 
 static inline void crypto_check_module_sig(struct module *mod)
 {
-	if (fips_enabled && mod && !module_sig_ok(mod))
+#ifdef FIPS_MODULE
+	/*
+	 * The FIPS module should ignore its own signature check, as it was
+	 * already been verified elsewhere during loading.
+	 */
+	if (mod == THIS_MODULE)
+		return;
+#else
+	if (!fips_enabled)
+		return;
+#endif
+
+	if (mod && !module_sig_ok(mod))
 		panic("Module %s signature verification failed in FIPS mode\n",
 		      module_name(mod));
 }
-- 
2.47.3

[PATCH 14/17] crypto: fips140: add module integrity self-check

[Jay Wang]

Add verify_integrity() function to perform HMAC-SHA256 self verification
of this standalone crypto module against pre-computed hash. This integrity
self-check is required by FIPS 140-3.

This patch is picked from Vegard Nossum <vegard.nossum@oracle.com> with
minor modifications.

Co-developed-by: Vegard Nossum <vegard.nossum@oracle.com>
Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/fips140/fips140-module.c | 52 +++++++++++++++++++++++++++++++++
 crypto/fips140/fips140-module.h |  4 +++
 2 files changed, 56 insertions(+)

diff --git a/crypto/fips140/fips140-module.c b/crypto/fips140/fips140-module.c
index e0e669ba1b5e..3bc8865567cb 100644
--- a/crypto/fips140/fips140-module.c
+++ b/crypto/fips140/fips140-module.c
@@ -16,6 +16,54 @@
 
 #define CRYPTO_INTERNAL "CRYPTO_INTERNAL"
 
+static const u8 fips140_integ_hmac_key[] = CONFIG_CRYPTO_FIPS140_HMAC_KEY;
+
+static int verify_integrity(void)
+{
+	extern const u8 *_binary_crypto_ko_start;
+	extern const u8 *_binary_crypto_ko_end;
+	extern const u8 *_binary_crypto_hmac_start;
+	
+	struct crypto_shash *tfm;
+	SHASH_DESC_ON_STACK(desc, tfm);
+	u8 digest[SHA256_DIGEST_SIZE];
+	int err;
+
+	tfm = crypto_alloc_shash("hmac(sha256)", 0, 0);
+	if (IS_ERR(tfm))
+		panic("FIPS 140: failed to allocate hmac tfm (%ld)\n", PTR_ERR(tfm));
+
+	desc->tfm = tfm;
+
+	err = crypto_shash_setkey(tfm, fips140_integ_hmac_key, sizeof(fips140_integ_hmac_key) - 1);
+	if (err)
+		panic("FIPS 140: crypto_shash_setkey() failed: %d\n", err);
+
+	err = crypto_shash_init(desc);
+	if (err)
+		panic("FIPS 140: crypto_shash_init() failed: %d\n", err);
+
+	err = crypto_shash_update(desc, _binary_crypto_ko_start, _binary_crypto_ko_end - _binary_crypto_ko_start);
+	if (err)
+		panic("FIPS 140: crypto_shash_update() failed: %d\n", err);
+
+	err = crypto_shash_final(desc, digest);
+	if (err)
+		panic("FIPS 140: crypto_shash_final() failed: %d\n", err);
+
+	shash_desc_zero(desc);
+
+	if (memcmp(digest, _binary_crypto_hmac_start, sizeof(digest)))
+		panic("FIPS 140: failed integrity check\n");
+
+	pr_info("FIPS 140: integrity verification passed\n");
+
+	crypto_free_shash(tfm);
+	memzero_explicit(digest, sizeof(digest));
+
+	return 0;
+}
+
 static int __init run_initcalls(void)
 {
 	typedef int (*initcall_t)(void);
@@ -68,6 +116,10 @@ static int __init fips140_init(void)
     pr_info("loading " FIPS140_MODULE_NAME "\n");
 
 	run_initcalls();
+
+	if (fips_enabled){
+		verify_integrity(); /* Panics if integrity check fails */
+	}
 	fips140_mark_module_level_complete(2);
     return 0;
 }
diff --git a/crypto/fips140/fips140-module.h b/crypto/fips140/fips140-module.h
index e95dac8eeda9..b8968d54800e 100644
--- a/crypto/fips140/fips140-module.h
+++ b/crypto/fips140/fips140-module.h
@@ -9,9 +9,13 @@
 #include <linux/module.h>
 #include <linux/crypto.h>
 #include <crypto/algapi.h>
+#include <crypto/hash.h>
+#include <crypto/sha2.h>
 #include <linux/init.h>
+#include <linux/string.h>
 #include <linux/atomic.h>
 #include <linux/wait.h>
+#include <linux/fips.h>
 
 /* FIPS140 synchronization between kernel and module */
 extern atomic_t fips140_kernel_level_complete;
-- 
2.47.3

[PATCH 15/17] x86: crypto: to convert exported crypto symbols into pluggable interface for x86 cryptos

[Jay Wang]

Prepare to convert exported crypto symbols in x86 cryptos
into pluggable interface. This patch introduces architecture specific
crypto API definition file `fips140-api.c`.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 arch/x86/crypto/Makefile              |  3 +++
 arch/x86/crypto/fips140/Makefile      | 14 ++++++++++++++
 arch/x86/crypto/fips140/fips140-api.c |  0
 3 files changed, 17 insertions(+)
 create mode 100644 arch/x86/crypto/fips140/Makefile
 create mode 100644 arch/x86/crypto/fips140/fips140-api.c

diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index b21ad0978c52..ab248a1beb00 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -70,3 +70,6 @@ aria-aesni-avx2-x86_64-y := aria-aesni-avx2-asm_64.o aria_aesni_avx2_glue.o
 
 obj-$(CONFIG_CRYPTO_ARIA_GFNI_AVX512_X86_64) += aria-gfni-avx512-x86_64.o
 aria-gfni-avx512-x86_64-y := aria-gfni-avx512-asm_64.o aria_gfni_avx512_glue.o
+
+# FIPS 140 kernel module
+obj-$(CONFIG_CRYPTO_FIPS140_EXTMOD) += fips140/
diff --git a/arch/x86/crypto/fips140/Makefile b/arch/x86/crypto/fips140/Makefile
new file mode 100644
index 000000000000..0a150ab16027
--- /dev/null
+++ b/arch/x86/crypto/fips140/Makefile
@@ -0,0 +1,14 @@
+
+crypto-objs-y += fips140-api-fips.o
+
+obj-y += fips140-api-main.o
+
+# Explicit rules to compile same source to different objects
+$(obj)/fips140-api-main.o: $(src)/fips140-api.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+$(obj)/fips140-api-fips.o: $(src)/fips140-api.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+CFLAGS_fips140-api-main.o += -I$(srctree)
+CFLAGS_fips140-api-fips.o += -I$(srctree)
\ No newline at end of file
diff --git a/arch/x86/crypto/fips140/fips140-api.c b/arch/x86/crypto/fips140/fips140-api.c
new file mode 100644
index 000000000000..e69de29bb2d1
-- 
2.47.3

[PATCH 16/17] arm64: crypto: to convert exported crypto symbols into pluggable interface for arm64 cryptos

[Jay Wang]

Prepare to convert exported crypto symbols in arm64 cryptos
into pluggable interface. This patch introduces architecture specific
crypto API definition file `fips140-api.c`.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 arch/arm64/crypto/Makefile              |  3 +++
 arch/arm64/crypto/fips140/Makefile      | 14 ++++++++++++++
 arch/arm64/crypto/fips140/fips140-api.c |  0
 3 files changed, 17 insertions(+)
 create mode 100644 arch/arm64/crypto/fips140/Makefile
 create mode 100644 arch/arm64/crypto/fips140/fips140-api.c

diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile
index 3574e917bc37..9f7106424d66 100644
--- a/arch/arm64/crypto/Makefile
+++ b/arch/arm64/crypto/Makefile
@@ -40,3 +40,6 @@ aes-neon-blk-y := aes-glue-neon.o aes-neon.o
 
 obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o
 aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o
+
+# FIPS 140 kernel module
+obj-$(CONFIG_CRYPTO_FIPS140_EXTMOD) += fips140/
\ No newline at end of file
diff --git a/arch/arm64/crypto/fips140/Makefile b/arch/arm64/crypto/fips140/Makefile
new file mode 100644
index 000000000000..0a150ab16027
--- /dev/null
+++ b/arch/arm64/crypto/fips140/Makefile
@@ -0,0 +1,14 @@
+
+crypto-objs-y += fips140-api-fips.o
+
+obj-y += fips140-api-main.o
+
+# Explicit rules to compile same source to different objects
+$(obj)/fips140-api-main.o: $(src)/fips140-api.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+$(obj)/fips140-api-fips.o: $(src)/fips140-api.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+CFLAGS_fips140-api-main.o += -I$(srctree)
+CFLAGS_fips140-api-fips.o += -I$(srctree)
\ No newline at end of file
diff --git a/arch/arm64/crypto/fips140/fips140-api.c b/arch/arm64/crypto/fips140/fips140-api.c
new file mode 100644
index 000000000000..e69de29bb2d1
-- 
2.47.3

[PATCH 17/17] Add standalone crypto kernel module technical documentation

[Jay Wang]

Technical guide covering implementation details and usage of the
standalone crypto kernel module feature.

Signed-off-by: Jay Wang <wanjay@amazon.com>
---
 crypto/fips140/README | 401 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 401 insertions(+)
 create mode 100644 crypto/fips140/README

diff --git a/crypto/fips140/README b/crypto/fips140/README
new file mode 100644
index 000000000000..45001037e855
--- /dev/null
+++ b/crypto/fips140/README
@@ -0,0 +1,401 @@
+## 1. Introduction
+
+Amazon Linux is releasing a new kernel feature that converts the previously built-in kernel crypto subsystem into a standalone kernel module. This module becomes the carrier of the kernel crypto subsystem and can be loaded at early boot to provide the same functionality as the original built-in crypto. The primary motivation for this modularization is to streamline Federal Information Processing Standards (FIPS) validation, a critical cryptographic certification for cloud computing users doing business with the U.S. government.
+ 
+In a bit more detail, previously, FIPS certification was tied to the entire kernel image, meaning non-crypto updates could potentially invalidate certification. With this feature, FIPS certification is tied only to the crypto module. Therefore, once the module is certified, loading this certified module on newer kernels automatically makes those kernels FIPS-certified. As a result, this approach can save re-certification costs and 12-18 months of waiting time by reducing the need for repeated FIPS re-certification cycles.
+
+This document provides technical details on how this feature is designed and implemented for users or developers who are interested in developing upon it, and is organized as follows:
+- Section 2 - Getting Started: Quick start on how to enable the feature
+- Section 3 - Workflow Overview: Changes this feature brings to build and runtime
+- Section 4 - Design Implementation Details: Technical deep-dive into each component
+- Section 5 - Customizing and Extending Crypto Module: How to select crypto to be included and extend to new crypto/architectures
+
+## 2. Getting Started
+
+This section provides a quick start guide for developers on how to enable, compile and use the standalone cryptography module feature.
+
+### 2.1 Basic Configuration
+
+The feature is controlled by a single configuration option:
+```
+CONFIG_CRYPTO_FIPS140_EXTMOD=y
+```
+What it does: When enabled, automatically redirects a set of cryptographic algorithms from the main kernel into a standalone module `crypto/fips140/fips140.ko`. The cryptographic algorithms that are redirected need to satisfy all the following conditions, otherwise the cryptography will remain in its original form:
+1. Must be configured as built-in (i.e., `CONFIG_CRYPTO_*=y`). This means cryptography already configured as modular (i.e., `CONFIG_CRYPTO_*=m`) are not redirected as they are already modularized.
+2. Must be among a list, which can be customized by developers as described in Section 5.
+
+When disabled, the kernel behaves as before.
+
+### 2.2 Build Process
+
+Once `CONFIG_CRYPTO_FIPS140_EXTMOD=y` is set, no additional steps are required. The standalone module will be built automatically as part of the standard kernel build process:
+```
+make -j$(nproc)
+# or
+make vmlinux
+```
+**What happens automatically (No user action required):**
+1. Build the module as `crypto/fips140/fips140.ko`
+2. The cryptography module will be loaded at boot time
+3. All kernel cryptographic services will provide the same functionality as before (i.e., prior to introducing this new feature) once boot completes.
+
+### 2.3 Advanced Configuration Options
+
+**Using External Cryptography Module:**
+```
+CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE=y
+```
+By default, `CONFIG_CRYPTO_FIPS140_EXTMOD_SOURCE` is not set, meaning the freshly built cryptography module is used. Otherwise, the pre-built standalone cryptography module from `fips140_build/crypto/fips140/fips140.ko` and modular cryptography such as `fips140_build/crypto/aes.ko` (need to manually place pre-built modules in these locations before the build) are included in kernel packaging (e.g., during `make modules_install`) and are used at later boot time.
+
+**Dual Version Support:**
+```
+CONFIG_CRYPTO_FIPS140_DUAL_VERSION=y
+```
+Encapsulate two versions of `fips140.ko` into kernel: one is freshly built for non-FIPS mode usage, another is pre-built specified by `fips140_build/crypto/fips140/fips140.ko` for FIPS mode usage. The appropriate version is selected and loaded at boot time based on boot time FIPS mode status.
+
+### 2.4 Verification
+
+To verify the feature is working, after install and boot with the new kernel:
+```
+# Check if fips140.ko module is loaded
+lsmod | grep fips140
+```
+
+## 3. Workflow Overview
+
+This section provides an overview without delving into deep technical details of the changes the standalone cryptography module feature introduces. When this feature is enabled, it introduces changes to both the kernel build and booting process. 
+
+3.1 Build-Time Changes
+
+Kernel cryptography subsystem consists of both cryptography management infrastructure (e.g., `crypto/api.c`, `crypto/algapi.c`, etc), along with hundreds of different cryptography algorithms (e.g., `crypto/arc4.c`).
+
+**Traditional Build Process:**
+Traditionally, cryptography management infrastructure are always built-in to the kernel, while cryptographic algorithms can be configured to be built either as built-in (`CONFIG_CRYPTO_*=y`) or as separate modular (`CONFIG_CRYPTO_*=m`) `.ko` file depending on kernel configuration:
+As a result, the builtin cryptography management infrastructure and cryptographic algorithms are statically linked into the kernel binary:
+```
+cryptographic algorithms source files → compiled as .o objfiles →  linked into vmlinux → single kernel binary
+```
+**With Standalone Cryptography Module:**
+This feature automatically transforms the builtin cryptographic components into a standalone cryptography module, `fips140.ko`. To do so, it develops a new kernel build rule `crypto-objs-$(CONFIG_CRYPTO_*)` such that, once this build rule is applied to a cryptographic algorithm, such cryptographic algorithm will be automatically collected into the cryptography module if it is configured as built-in (i.e, `CONFIG_CRYPTO_*=y`), for example:
+```
+// in crypto/asymmetric_keys/Makefile
+- obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
++ crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
+```
+Such build change allows the modularization transformation to only affect selected cryptographic algorithms (i.e, where the `crypto-objs-$(CONFIG_CRYPTO_*`) is applied).
+
+Then, after the `fips140.ko` is generated, it will be embedded back into main kernel vmlinux as a replacement part. The purpose of this embedding, instead of traditionally putting the `fips140.ko` into filesystem, is a preparation to allow the module to be loaded early enough even before the filesystem is ready.
+
+The new build process is illustrated below.
+```
+cryptographic algorithms source files → compiled as .o objfiles → automatically collected and linked into fips140.ko → embedded fips140.ko into vmlinux as a replaceable binary
+```
+
+### 3.2 Runtime Changes
+
+**Traditional Boot Process:**
+The kernel initializes the cryptographic subsystem early during boot, executing each cryptographic initialization routine accordingly. These initialization routines may depend on other cryptographic components or other kernel subsystems, so their invocation follows a well-defined execution order to ensure they are initialized before their first use.
+```
+kernel starts → cryptography subsystem initialization → cryptography subsystem available → other components use cryptography
+```
+**With Standalone Cryptography Module:**
+At the start of kernel boot, compared to a regular kernel, the first major change introduced by this feature is that no cryptography services are initially available — since the entire cryptography subsystem has been decoupled from the main kernel.
+To ensure that the cryptography subsystem becomes available early enough (before the first kernel component that requires cryptography services), the standalone cryptography kernel module must be loaded at a very early stage, even before the filesystem becomes available.
+
+However, the regular module loading mechanism relies on placing kernel modules in the filesystem and loading them from there, which creates a chicken-and-egg problem — the cryptography module cannot be loaded until the filesystem is ready, yet some kernel components may require cryptography services even before that point.
+
+To address this, the second change introduced by this feature is that the cryptography kernel module is loaded directly from memory, leveraging the earlier compilation changes that embed the module binary into the main kernel image. Afterward, the feature includes a “plug-in” mechanism that connects the decoupled cryptography subsystem back to the main kernel, ensuring that kernel cryptography users can correctly locate and invoke the cryptography routine entry points.
+
+Finally, to ensure proper initialization, the feature guarantees that all cryptography algorithms and the cryptography management infra execute their initialization routines in the exact same order as they would if they were built-in.
+
+The process described above is illustrated below.
+```
+kernel starts → no cryptography available → load fips140.ko from memory → plug cryptography back to kernel → module initialization → cryptographic services available → other components use cryptography
+```
+
+## 4. Design Implementation Details
+
+While the earlier sections provide a holistic view of how this feature shapes the kernel, this section provides deeper design details on how these functionalities are realized. There are three key design components:
+1. A specialized compile rule that automatically compiles and collects all built-in cryptographic algorithm object files to generate the final module binary under arbitrary kernel configurations, and then embeds the generated binary into the main kernel image for early loading.
+2. A mechanism to convert interactions between the cryptography subsystem and the main kernel into a pluggable interface.
+3. A module loading and initialization process that ensures the cryptography subsystem is properly initialized as if it were built-in.
+
+### 4.1. Specialized Compilation System
+
+**Automatic Collection and Linking of Built-in Cryptographic Algorithm Objects:**
+The first step in generating the `fips140.ko` module is to compile and collect built-in cryptographic components (i.e., those specified by `CONFIG_CRYPTO_*=y`).
+Traditionally, the existing module build process requires all module components (e.g., source files) to reside in a single directory. However, this approach is not suitable for our case, where hundreds of cryptographic algorithm source files are scattered across multiple directories.
+
+A naïve approach would be to create a separate Makefile that duplicates the original build rules with adjusted paths.
+However, this method is not scalable due to the large number of cryptographic build rules, many of which are highly customized and can vary under different Kconfig settings, making such a separate Makefile even more complex.
+Moreover, this approach cannot ensure that built-in cryptographic algorithms are completely removed from the main kernel, which would result in redundant cryptographic code being included in both the kernel and the module.
+
+To tackle this challenge, we automated the object collection and linking process by introducing special build logic for the kernel cryptography subsystem.
+Specifically, to automatically collect cryptography object files while preserving their original compilation settings (such as flags, headers, and paths), we introduced a new compilation rule:
+```
+crypto-objs-y += *.o
+```
+This replaces the original `obj-y += *.o` rule in cryptography Makefiles later, for example:
+```
+// in crypto/asymmetric_keys/Makefile
+- obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
++ crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
+asymmetric_keys-y := \
+    asymmetric_type.o \
+    restrict.o \
+    signature.o
+```
+in the cryptography subsystem Makefiles, allowing most of the existing Makefile logic to be reused.
+As a result, when the standalone cryptography module feature is enabled, any cryptographic algorithm configured as built-in (for example, `crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o` where `CONFIG_ASYMMETRIC_KEY_TYPE=y`) will be automatically collected and linked into a single final object binary, `fips140.o`.
+During this process, a special compilation flag (`-DFIPS_MODULE=1`) is applied to instruct each object file to be compiled in a module-specific manner. This flag will later be used to generate the pluggable interface on both the main kernel side and the module side from the same source code.
+
+The implementation details are as follows: it follows a similar methodology used by the `obj-y` collection process for building `vmlinux.o`. The `crypto-objs-y` rule is placed in `scripts/Makefile.build`, which is executed by each directory Makefile to collect the corresponding crypto object files. Each directory then creates a `crypto-module.a` archive that contains all `crypto-objs-y += <object>.o` files under that directory. In the parent directories, these `crypto-module.a` archives are recursively included into the parent’s own `crypto-module.a`, and this process continues upward until the final `fips140.o` is generated.
+
+**A Separate Module Generation Pipeline for Building the Final Kernel Module from Linked Cryptographic Algorithm Object:**
+With the linked cryptographic algorithm object (i.e., `fips140.o`), the next step is to generate the final kernel module, `fips140.ko`.
+
+A direct approach would be to inject the `fips140.ko` module build into the existing modules generation pipeline (i.e., `make modules`) by providing our pre-generated `fips140.o`. However, we choose not to do this because it would create a circular make rule dependency (which is invalid in Makefiles and causes build failures), resulting in mutual dependencies between the modules and vmlinux targets (i.e., `modules:vmlinux` and `vmlinux:modules` at the same time).
+This happens for the following reasons:
+1. Since we will later embed `fips140.ko` into the final kernel image (as described in the next section), we must make vmlinux depend on `fips140.ko`. In other words: `vmlinux: fips140.ko`.
+2. When the kernel is built with `CONFIG_DEBUG_INFO_BTF_MODULES=y`, it requires: modules: vmlinux. This is because `CONFIG_DEBUG_INFO_BTF_MODULES=y` takes vmlinux as input to generate BTF info for the module, and inserts such info into the `.ko` module by default.
+3. If we choose to inject `fips140.ko` into make modules, this would create a make rule dependency: `fips140.ko: modules`. Combined with items 1 and 2, this eventually creates an invalid circular dependency between vmlinux and modules.
+
+Due to these reasons, the design choice is to use a separate make pipeline (defined as `fips140-ready` in the Makefile). This new pipeline reuses the same module generation scripts used by make modules but adds additional logic in `scripts/Makefile.{modfinal|modinst|modpost}` and `scripts/mod/modpost.c` to handle module symbol generation and verification correctly. 
+
+**A Seamless Process That Embeds the Generated Binary Into the Main Kernel Image for Early Loading:**
+As mentioned earlier, in order to load the standalone cryptography module early in the boot process—before the filesystem is ready—the module binary must be embedded into the final kernel image (i.e., vmlinux) so that it can be loaded directly from memory.
+We intend for this embedding process to be completely seamless and automatically triggered whenever vmlinux is built (i.e., during `make vmlinux`).
+
+To achieve this, the feature adds a Make dependency rule so that vmlinux depends on `fips140.ko`.
+It also modifies the vmlinux link rules (i.e., `arch/<arch>/kernel/vmlinux.lds.S`, `scripts/Makefile.vmlinux`, and `scripts/link-vmlinux.sh`) so that the generated module binary is finally combined with `vmlinux.o`.
+
+In addition, we allow multiple cryptography module binary versions (for example, a certified cryptography binary and a latest, up-to-date but uncertified one) to be embedded into the main kernel image to serve different user needs. This design allows regular (non-FIPS) users to benefit from the latest cryptographic updates, while FIPS-mode users continue to use the certified cryptography module.
+
+To support this, we introduce an optional configuration, `CONFIG_CRYPTO_FIPS140_DUAL_VERSION`. When enabled, this option allows two cryptography module versions to be embedded within a single kernel build and ensures that the appropriate module is selected and loaded at boot time based on the system’s FIPS mode status.
+
+### 4.2. Pluggable Interface Between the Built-in Cryptography Subsystem and the Main Kernel
+
+Although the module binary (`fips140.ko`) has been embedded into the final kernel image (`vmlinux`) as described in the previous section, it is not linked to the kernel in any way. This is because `fips140.ko` is embedded in a data-only manner, so the main kernel cannot directly call any functions or access any data defined in the module binary.
+
+Even worse, simply removing the built-in cryptographic algorithms without additional handling would cause the kernel to fail to compile, because traditionally, built-in cryptographic algorithms and the main kernel can interact only through functions and variables whose addresses they assume to know. As a result, even if they have been removed, kernel cryptography users still expect the symbol addresses of cryptographic routines and data to be available at compile time.
+
+To address this, we introduce a pluggable interface between built-in cryptographic functions and variables by placing **address placeholders**. During runtime, once the cryptography kernel module is loaded, these placeholders are updated to the correct addresses. In the rest of this section, we first introduce this pluggable interface mechanism, and then explain how to apply it to the built-in cryptographic algorithms.
+
+**The Pluggable Interface Mechanism:**
+There are two types of address holders used to achieve this pluggable interface:
+- Function addresses:  We use the “static call” mechanism. Static calls are a Linux mechanism that converts an “indirect call” into something with performance equivalent to a “direct call,” while avoiding the introduction of additional security concerns, such as control-flow–hijacking attack gadgets. We implement this function-address placeholder as the `DECLARE_STATIC_CALL()` and `DEFINE_CRYPTO_API_STUB()` wrappers.
+- Variable addresses (the remaining smaller portion): For these, we use a pointer of the corresponding data type. We implement this address placeholder as the `DECLARE_CRYPTO_VAR()` and `DEFINE_CRYPTO_API_STUB()` wrappers:
+
+These wrappers are applied to each symbol-exported (i.e., `EXPORT_SYMBOL()`) cryptographic function and variable (details on how to apply them are described later). Once applied, the wrappers are compiled differently for the main kernel and for the built-in cryptographic algorithm source code—acting as the “outlet” and the “plug,” respectively—using different compilation flags (`-DFIPS_MODULE`) introduced by our customized build rules described earlier.
+
+As a result, the kernel can successfully compile even when the built-in cryptographic algorithms are removed, thanks to these address placeholders. At boot time, the placeholders initially hold NULL, but since no cryptography users exist at that stage, the kernel can still start booting correctly. After the cryptography module is loaded, the placeholders are dynamically updated to the correct addresses later (by `do_crypto_api()` and `do_crypto_var()`, described in later section).
+
+In addition to these address placeholders, there is another important interaction point between the cryptography subsystem and the main kernel—the cryptographic initialization routines. Therefore, we also add a mechanism to collect all cryptographic initialization functions (e.g., those defined using `module_init()`) into a dedicated ELF section. This serves as preparation for the later module and cryptography-subsystem initialization steps described in subsequent sections.
+
+**Applying the Pluggable Interface Mechanism to Cryptographic Algorithms:**
+
+To apply these pluggable interface wrappers to a cryptographic algorithm and make them take effect, we follow the steps below (using `crypto/asymmetric_keys/asymmetric_type.c` as an example):
+1. **Apply `crypto-objs-y` compile rule to the cryptographic algorithm:**
+```
+// in crypto/asymmetric_keys/Makefile
+- obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
++ crypto-objs-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys.o
+asymmetric_keys-y := \
+    asymmetric_type.o \
+    restrict.o \
+    signature.o
+```
+2. **Locate the communication point between the cryptographic algorithm and the main kernel:**
+
+The cryptography subsystem is designed such that most interactions between the main kernel and cryptographic algorithms occur through exported symbols using `EXPORT_SYMBOL()` wrappers.
+This kernel design exists because most cryptographic algorithm implementations must support both built-in and modular modes. 
+
+Consequently, the cryptographic functions and variables exported by `EXPORT_SYMBOL()` are a well-defined and identifiable interface between the cryptography subsystem and the main kernel: 
+```
+// in crypto/asymmetric_keys/asymmetric_type.c 
+//Exported cryptographic function:
+bool asymmetric_key_id_same(const struct asymmetric_key_id *kid1,
+                const struct asymmetric_key_id *kid2) {...}
+EXPORT_SYMBOL_GPL(asymmetric_key_id_same); 
+//Exported cryptographic variable:
+struct key_type key_type_asymmetric = {...};
+EXPORT_SYMBOL_GPL(key_type_asymmetric); 
+```
+3. **Replace their declarations in the header file with the address-placeholder declaration wrappers:**
+```
+// in include/keys/asymmetric-type.h
+// for exported cryptographic function:
+- bool asymmetric_key_id_same const struct asymmetric_key_id *kid1, const struct asymmetric_key_id *kid2);
++ DECLARE_CRYPTO_API(CONFIG_ASYMMETRIC_KEY_TYPE, asymmetric_key_id_same, bool,
+    (const struct asymmetric_key_id *kid1, const struct asymmetric_key_id *kid2),
+    (kid1, kid2)); 
+// for exported cryptographic variables:
+- struct key_type key_type_asymmetric;
++ DECLARE_CRYPTO_VAR(CONFIG_ASYMMETRIC_KEY_TYPE, key_type_asymmetric, struct key_type, );
++ #if defined(CONFIG_CRYPTO_FIPS140_EXTMOD) && !defined(FIPS_MODULE) && IS_BUILTIN(CONFIG_ASYMMETRIC_KEY_TYPE)
++ #define key_type_asymmetric (*((struct key_type*)CRYPTO_VAR_NAME(key_type_asymmetric)))
++ #endif 
+```
+By replacing the original declarations with the address-placeholder declaration wrappers, we can automatically force all cryptography users to go through the placeholders, because those users already include the same header file.
+The wrapper also takes the cryptographic algorithm Kconfig symbol as a parameter, so that when a cryptographic algorithm is built as a module (for example, `CONFIG_ASYMMETRIC_KEY_TYPE=m`), the original function declarations remain unchanged and are not affected.
+
+4. **Add the address-placeholder definition wrappers into a dedicated file `fips140-api.c`:**
+
+This file will be compiled separately and acts as both the “outlet” and the “plug” for the main kernel and the cryptography module, respectively:
+```
+// in crypto/fips140/fips140-api.c
++ #if IS_BUILTIN(CONFIG_ASYMMETRIC_KEY_TYPE)
++ #include <keys/asymmetric-type.h>
+// for exported cryptographic function:
++ DEFINE_CRYPTO_API_STUB(asymmetric_key_id_same);
+// for exported cryptographic variables:
++ #undef key_type_asymmetric
++ DEFINE_CRYPTO_VAR_STUB(key_type_asymmetric);
++ #endif
+```
+
+5. **Lastly, collect the cryptographic initialization routines for later module and cryptography-subsystem initialization by wrapping the original cryptographic initialization functions:**
+```
+// in crypto/asymmetric_keys/asymmetric_type.c 
+- module_init(asymmetric_key_init);
+- module_exit(asymmetric_key_cleanup); 
++ crypto_module_init(asymmetric_key_init);
++ crypto_module_exit(asymmetric_key_cleanup); 
+```
+We apply the above steps to both architecture-independent and architecture-specific cryptographic algorithms.
+
+### 4.3. Initialization Synchronization
+
+To ensure the embedded `fips140.ko` module binary provides the same cryptography functionality as the regular kernel, the kernel needs:
+1. A module loader to load the module binary directly from memory,
+2. A mechanism to plug the module back into the kernel by updating the address placeholders, and
+3. Correct cryptography subsystem initialization, as if the cryptographic algorithms were still built-in.
+
+**Directly Load Module Binary from Memory:**
+Regular modules are loaded from the filesystem and undergo signature verification on the module binary, which relies on cryptographic operations. However, since we have already fully decoupled the cryptography subsystem, we must skip this step for this `fips140.ko` module.
+To achieve this, we add a new loader function `load_crypto_module_mem()` that can load the module binary directly from memory at the designed address without checking the signature. Since the module binary is embedded into main kernel in an ELF section, as specified in the linker script:
+```
+// in arch/<arch>/kernel/vmlinux.lds.S
+    .fips140_embedded : AT(ADDR(.fips140_embedded) - LOAD_OFFSET) {
+        . = ALIGN(8);
+        _binary_fips140_ko_start = .;
+        KEEP(*(.fips140_module_data))
+        _binary_fips140_ko_end = .;
+    }
+```
+Therefore, the runtime memory address of the module can be accessed directly by the module loader to invoke the new loader function `load_crypto_module_mem()`.
+
+**Plug Back the Module by Updating Address Placeholder Values:**
+To update the address placeholders in the main kernel to the correct addresses matching the loaded module, after compilation the placeholders are placed into dedicated ELF sections `_crypto_api_keys` and `_crypto_var_keys`.
+This can be seen from the definition of the placeholder-declaration wrappers:
+```
+#define DEFINE_CRYPTO_API_STUB(name) \
+    static struct crypto_api_key __##name##_key \
+        __used \
+        __section("__crypto_api_keys") \ // Place in a dedicated ELF Section
+        __aligned(__alignof__(struct crypto_api_key)) = \
+    { \
+        .key = &STATIC_CALL_KEY(crypto_##name##_key), \
+        .tramp = STATIC_CALL_TRAMP_ADDR(crypto_##name##_key), \
+        .func = &name, \
+    };
+
+#define DEFINE_CRYPTO_VAR_STUB(name) \
+    static struct crypto_var_key __crypto_##name##_var_key \
+        __used \
+        __section("__crypto_var_keys") \ // Place in a dedicated ELF Section
+        __aligned(__alignof__(struct crypto_var_key)) = \
+    { \
+        .ptr = &CRYPTO_VAR_NAME(name), \
+        .var = (void*)&name, \
+    };
+```
+The purpose of doing this is to allow the main kernel to quickly locate the placeholders and update them to the correct addresses. The update functions are defined as `do_crypto_var()` and `do_crypto_api()`, which are executed at module load.
+
+As a result, all cryptography users in the main kernel can now call the cryptographic functions as if they were built-in.
+
+**Initialize Cryptography Subsystem as if it Were Built-in:**
+Cryptographic components must be properly initialized before use, and this initialization is typically achieved through dedicated initialization functions (e.g., `module_init(crypto_init_func)` or `late_initcall(crypto_init_func)`). These functions often have strict execution order requirements and must run during the appropriate boot phase.
+
+Therefore, for our standalone cryptography module feature, we must ensure that these decoupled “built-in” cryptographic algorithms are properly initialized and that their initialization order is preserved as before because failure to follow the correct order can result in kernel panic.
+
+To address this, we introduce a synchronization mechanism between the main kernel and the module to ensure all cryptographic algorithms are executed in the correct kernel boot phase. In more details, we spawn the module initialization process `fips_loader_init()` as an async thread `fips140_sync_thread()`, in which we call `run_initcalls()` to execute the initialization calls of each cryptographic algorithm.
+Then, we introduce synchronization helpers such as `wait_until_fips140_level_sync(int level)` to ensure the initialization order of all cryptographic algorithms is synchronized with the main kernel.
+
+## 5. Customization and Extension of Cryptography Module
+
+This section describes how developers can customize which cryptographic algorithms are included in the standalone cryptography module, as well as extend this feature to other cryptographic algorithms or hardware architectures.
+
+### 5.1. Cryptography Selection Mechanism
+
+The feature automatically includes cryptographic algorithms that meet specific criteria:
+1. **Built-in Configuration**: Only cryptographic algorithms configured as `CONFIG_CRYPTO_*=y` are candidates for inclusion
+2. **Explicit Inclusion**: Cryptographic algorithms must be explicitly converted using the `crypto-objs-$(CONFIG__CRYPTO_*`) build rule
+
+### 5.2. Extend Support to New Cryptographic Algorithms
+
+To extend support to a new cryptographic algorithm in the standalone module, follow these steps:
+
+**Step 1: Update the Makefile**
+```
+# in crypto/[algorithm]/Makefile
+- obj-$(CONFIG_CRYPTO_ALGORITHM) += algorithm.o
++ crypto-objs-$(CONFIG_CRYPTO_ALGORITHM) += algorithm.o
+```
+For Architecture-Specific Cryptographic Algorithms:
+- Apply the `crypto-objs-` rule in the appropriate `arch/*/crypto/Makefile`
+
+**Step 2: Add Pluggable Interface Support**
+If the cryptographic algorithm exports symbols via `EXPORT_SYMBOL()`, add the pluggable interface wrappers:
+```
+# Example: in include/crypto/algorithm.h
+- extern int crypto_algorithm_transform(struct crypto_tfm *tfm, const u8 *src, 
+                                       u8 *dst, unsigned int len, u32 flags);
++ DECLARE_CRYPTO_API(CONFIG_CRYPTO_ALGORITHM, crypto_algorithm_transform, int,
+    (struct crypto_tfm *tfm, const u8 *src, u8 *dst, unsigned int len, u32 flags),
+    (tfm, src, dst, len, flags));
+```
+Then, add the corresponding stubs in `crypto/fips140/fips140-api.c`:
+```
+#if IS_BUILTIN(CONFIG_CRYPTO_ALGORITHM)
+#include <crypto/algorithm.h>
+DEFINE_CRYPTO_API_STUB(crypto_algorithm_transform);
+#endif
+```
+For Architecture-Specific Cryptographic Algorithms:
+- Include architecture-specific stubs in `arch/*/crypto/fips140/fips140-api.c`:
+```
+# Example: in arch/arm64/crypto/fips140/fips140-api.c
+#if IS_BUILTIN(CONFIG_CRYPTO_AES_ARM64_CE)
+#include <arch/arm64/crypto/aes-ce-setkey.h>
+DEFINE_CRYPTO_API_STUB(ce_aes_setkey);
+DEFINE_CRYPTO_API_STUB(ce_aes_expandkey);
+#endif
+```
+**Step 3: Update Initialization**
+Replace module initialization calls:
+```
+# in crypto/algorithm/algorithm.c
+- module_init(algorithm_init);
+- module_exit(algorithm_exit);
++ crypto_module_init(algorithm_init);
++ crypto_module_exit(algorithm_exit);
+```
+
+### 5.3. Architecture-Specific Extensions
+
+**Extending to New Architectures:**
+Currently supported architectures are x86_64 and ARM64. To extend this feature to additional architectures:
+1. **Update Linker Scripts**: Add ELF sections in `arch/[new-arch]/kernel/vmlinux.lds.S`:
+```
+.fips140_embedded : AT(ADDR(.fips140_embedded) - LOAD_OFFSET) {
+    . = ALIGN(8);
+    _binary_fips140_ko_start = .;
+    KEEP(*(.fips140_module_data))
+    _binary_fips140_ko_end = .;
+}
+```
+2. **Create Architecture-Specific Files**: Set up `arch/[new-arch]/crypto/fips140/` directory with Makefile and `fips140-api.c` following the pattern used in x86_64 and ARM64.
+
+---
+Written by Jay Wang <wanjay@amazon.com>, Amazon Linux
\ No newline at end of file
-- 
2.47.3

Re: [PATCH 17/17] Add standalone crypto kernel module technical documentation

[Eric Biggers]

On Thu, Feb 12, 2026 at 02:42:21AM +0000, Jay Wang wrote:
> With this feature, FIPS certification is tied only to the crypto
> module. Therefore, once the module is certified, loading this
> certified module on newer kernels automatically makes those kernels
> FIPS-certified. As a result, this approach can save re-certification
> costs and 12-18 months of waiting time by reducing the need for
> repeated FIPS re-certification cycles.

Let's be clear: this is possible only when the kernel has a stable ABI
to the crypto module, which realistically isn't something that is going
to be supported upstream.  The Linux kernel is well-known for not
maintaining a stable in-kernel ABI, for good reasons.

So, the only case where this feature would have a benefit over the
kernel's existing approach to FIPS 140 is in downstream kernels that
maintain a stable in-kernel ABI.  There would be no benefit to direct
users of the mainline kernel or even the stable release series.

For this to be considered for upstream there would need to be some level
of consensus in the community to support this feature despite this.

- Eric

Re: [PATCH 17/17] Add standalone crypto kernel module technical documentation

[Christoph Hellwig]

On Tue, Feb 24, 2026 at 05:55:17PM -0800, Eric Biggers wrote:
> Let's be clear: this is possible only when the kernel has a stable ABI
> to the crypto module, which realistically isn't something that is going
> to be supported upstream.  The Linux kernel is well-known for not
> maintaining a stable in-kernel ABI, for good reasons.
> 
> So, the only case where this feature would have a benefit over the
> kernel's existing approach to FIPS 140 is in downstream kernels that
> maintain a stable in-kernel ABI.  There would be no benefit to direct
> users of the mainline kernel or even the stable release series.
> 
> For this to be considered for upstream there would need to be some level
> of consensus in the community to support this feature despite this.

That's a very nice way to say this goes against all the established
principles for kernel development.

RE: [PATCH 17/17] Add standalone crypto kernel module technical documentation

[Jay Wang]

Thanks to everyone who has taken the time to review and comment.

Based on the discussion so far, it seems the core question is whether this feature is appropriate for upstream at all, or whether it should remain entirely downstream.

We’ve discussed this with a few folks, and to help clarify the discussion, let me outline what is actually required to use this approach and what the community would gain from upstreaming it:

1. Maintaining a crypto snapshot (for example, a source code snapshot used for streamlined FIPS re-validation processes such as NSRL).
2. Maintaining the loadable crypto build infrastructure itself (i.e., this patch set).

For (1), since this requires maintaining a specific snapshot, we acknowledge that this is inherently a downstream responsibility. We are not expecting upstream to provide or guarantee a stable in-kernel API (to clarify, a stable ABI is not strictly required here, since the crypto module can be rebuilt against newer kernels and still benefit from FIPS through the shorter NSRL re-validation process of roughly 3 months, compared to the full 12–18 month certification cycle). The upstream crypto subsystem should continue evolving independently.

For (2), since this feature mainly serves as infrastructure and is of interest to multiple distributions, upstreaming it could help reduce the effort each distribution would otherwise spend maintaining similar infrastructure patches independently.

We’d love to hear more thoughts on this. If the general consensus is that the downsides outweigh the benefits of merging this into mainline, we are happy to maintain it in a separate repository tracking the latest mainline and stable releases in order to keep the work publicly available.

Thanks again for taking the time to read and discuss this.

Jay