* [PATCH 0/4] crypto: introduce generic dynamic software fallback and EIP93 support
@ 2026-07-28 8:43 Jihong Min
2026-07-28 8:43 ` [PATCH 1/4] crypto: move cycle benchmark helpers out of tcrypt Jihong Min
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Jihong Min @ 2026-07-28 8:43 UTC (permalink / raw)
To: Herbert Xu, Christian Marangi, linux-crypto
Cc: Jihong Min, David S. Miller, Antoine Tenart, linux-kernel
This series introduces a generic in-kernel Crypto API proxy for dynamic
CPU software fallback, inspired by tcrypt's in-kernel benchmarking and
RAID6's algorithm selection at module load time. It also adds support
for using the proxy with EIP93.
The recent QCE discussion [1][2][3] makes it clear that a crypto device
cannot be evaluated solely by whether there is any performance
improvement over the CPU software path. But at the same time,
performance remains one of the main factors.
Crypto accelerators generally incur submission, DMA, interrupt, and
completion costs. Legacy devices pair those costs with lower hardware
throughput, so small requests can be slower than CPU software. This
series lets drivers describe algorithm groups that share similar scaling
behavior, then opt in at runtime to dispatch requests only to the
implementation that provides a measured performance benefit. This is
possible because Crypto API requests of different sizes have no
dependency on one another, allowing the proxy to select a provider
independently for each request while keeping each stateful operation on
one provider.
EIP93 was chosen as the first target not only because I own and use it,
but also because it illustrates this problem. It still improves
performance for large requests, yet it is much slower for small requests
even when compared with a relatively weak CPU by modern standards, such
as the one in the AN7581. Community patches such as [4] have previously
addressed this with a static size override for AES only. They provide a
useful workaround, but neither determine the crossover point
systematically nor offer a generic framework that is easy to extend to
other devices or algorithms.
The control is exposed through a per-crypto-device sysfs "enabled"
attribute and is disabled by default. While disabled, the proxy
providers are unregistered, so newly allocated transforms bind directly
to the hardware providers and the existing hardware hot path is
unaffected. Proxy generations that still have live transforms are
retained until those transforms are released.
Adding dynamic fallback to a driver
===================================
Select CRYPTO_DYNAMIC_FALLBACK and the software algorithms used for
benchmarking in the driver's Kconfig. Define a crypto_fallback_group
table that groups hardware driver names with similar request-size
scaling. For each group, choose one representative hardware/software
pair and provide the benchmark key and parameters. The
CRYPTO_FALLBACK_GROUP() and CRYPTO_FALLBACK_GROUP_AUTHENC() macros can
be used to initialize these entries.
After registering the hardware algorithms, call
crypto_fallback_register(THIS_MODULE, dev, groups, ARRAY_SIZE(groups))
and keep the returned handle. Call crypto_fallback_unregister() with
that handle before unregistering the hardware algorithms. The framework
creates the per-device enable and group threshold sysfs attributes and
manages the proxy providers, so the driver's request path needs no
changes.
Benchmark method
================
Whenever fallback is enabled or re-enabled, one representative hardware
implementation from each group and its CPU software equivalent are
tested at these request sizes:
16, 64, 128, 256, 512, 1024, 1420, 2048, 4096, 8192, 16384 bytes
Each implementation gets four warm-up operations followed by eight
measured operations. If their cycle totals differ by at most 10% of the
lower total, both get 16 additional measurements. Equal totals count as
a software win.
The threshold is the tested size before hardware first becomes strictly
faster. It is 0 if hardware wins at 16 bytes and -1 if hardware never
wins. At request time:
use_software = T < 0 || (T > 0 && request_size <= T)
A failed benchmark uses T = 0, selecting hardware only. Each algorithm
group's threshold can also be overridden by writing to its sysfs
attribute.
Testing
=======
This series was integrated into an OpenWrt community image based on
Linux 6.18 and tested on a W1700K2 HW2.1 equipped with an Airoha AN7581
E2 SoC. Results from tcrypt and OpenSSL:
Enabling fallback took 0.82 seconds and selected these thresholds. A
value of -1 means that software was faster at every tested size.
Group Threshold (bytes)
raw_aes 512
raw_des 256
raw_3des 64
hash_md5 -1
hash_sha1 -1
hash_sha2 -1
authenc_aes 512
authenc_des 128
authenc_3des 64
The tcrypt AES-128-CBC results are median cycles from five runs. Lower
is better. Disabled uses cbc(aes-eip93) directly, while enabled uses
fallback(cbc(aes-eip93)).
Request Disabled Enabled Cycle reduction
(bytes) (cycles) (cycles)
16 322 18 94.4%
64 274 43 84.3%
128 292 74 74.7%
256 312 138 55.8%
1024 434 439 -1.2%
1424 492 488 0.8%
4096 980 1023 -4.4%
OpenSSL 3.5.7 AF_ALG AES-128-CBC throughput is the mean of two
two-second runs at each request size.
Request Disabled Enabled Throughput gain
(bytes) (MB/s) (MB/s)
16 0.560 1.257 124.5%
64 2.430 4.639 90.9%
256 9.106 14.168 55.6%
1024 30.279 30.332 0.2%
8192 88.807 89.383 0.6%
16384 101.220 102.068 0.8%
All tcrypt KATs and OpenSSL AF_ALG round-trip tests passed with fallback
both disabled and enabled.
LLM disclosure
==============
Development of patch 2/4 was aided by an LLM, while patch 4/4 was
developed more extensively with LLM assistance. All testing was
performed by me.
[1] https://lore.kernel.org/linux-crypto/20260712-qce-broken-v1-1-85e2bff17871@gmail.com/
[2] https://lore.kernel.org/linux-crypto/20260712-qce-broken-v2-1-b2dfff47f7f5@gmail.com/
[3] https://lore.kernel.org/linux-crypto/20260724050645.223799-1-ebiggers@kernel.org/
[4] https://github.com/stevenj/ph-openwrt-mtk-eip93
Sincerely,
Jihong Min
Jihong Min (4):
crypto: move cycle benchmark helpers out of tcrypt
crypto: introduce dynamic software fallback
crypto: eip93 - add dynamic software fallback support
crypto: eliminate fallback proxy overhead while disabled
crypto/Kconfig | 24 +-
crypto/Makefile | 3 +
crypto/benchmark.c | 180 ++
crypto/fallback.c | 1479 +++++++++++++++++
crypto/tcrypt.c | 213 +--
drivers/crypto/inside-secure/eip93/Kconfig | 7 +
drivers/crypto/inside-secure/eip93/Makefile | 1 +
.../inside-secure/eip93/eip93-fallback.c | 164 ++
.../inside-secure/eip93/eip93-fallback.h | 10 +
.../crypto/inside-secure/eip93/eip93-main.c | 19 +-
include/crypto/benchmark.h | 24 +
include/crypto/fallback.h | 96 ++
12 files changed, 2024 insertions(+), 196 deletions(-)
create mode 100644 crypto/benchmark.c
create mode 100644 crypto/fallback.c
create mode 100644 drivers/crypto/inside-secure/eip93/eip93-fallback.c
create mode 100644 drivers/crypto/inside-secure/eip93/eip93-fallback.h
create mode 100644 include/crypto/benchmark.h
create mode 100644 include/crypto/fallback.h
base-commit: 1163a476a568f6c0f852d469c8e4c5a5f805adac
--
2.53.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/4] crypto: move cycle benchmark helpers out of tcrypt
2026-07-28 8:43 [PATCH 0/4] crypto: introduce generic dynamic software fallback and EIP93 support Jihong Min
@ 2026-07-28 8:43 ` Jihong Min
2026-07-28 8:43 ` [PATCH 2/4] crypto: introduce dynamic software fallback Jihong Min
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Jihong Min @ 2026-07-28 8:43 UTC (permalink / raw)
To: Herbert Xu, Christian Marangi, linux-crypto
Cc: Jihong Min, David S. Miller, Antoine Tenart, linux-kernel
Move the reusable AEAD, ahash, and skcipher cycle measurement loops
into a dedicated benchmark module. Preserve tcrypt's existing output
and benchmark behavior while allowing crypto drivers to use the same
helpers for runtime comparisons.
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
crypto/Kconfig | 15 ++-
crypto/Makefile | 2 +
crypto/benchmark.c | 180 +++++++++++++++++++++++++++++++
crypto/tcrypt.c | 213 +++++--------------------------------
include/crypto/benchmark.h | 24 +++++
5 files changed, 242 insertions(+), 192 deletions(-)
create mode 100644 crypto/benchmark.c
create mode 100644 include/crypto/benchmark.h
diff --git a/crypto/Kconfig b/crypto/Kconfig
index f1e372195273..0cba261e3b88 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -244,13 +244,22 @@ config CRYPTO_KRB5ENC
profile. This is required for Kerberos 5-style encryption, used by
sunrpc/NFS and rxrpc/AFS.
-config CRYPTO_BENCHMARK
- tristate "Crypto benchmarking module"
- depends on m || EXPERT
+config CRYPTO_BENCHMARK_LIB
+ tristate
select CRYPTO_AEAD
select CRYPTO_HASH
select CRYPTO_MANAGER
select CRYPTO_SKCIPHER
+ help
+ Build reusable benchmark helpers for Crypto API algorithms. This
+ internal library is shared by tcrypt and drivers that select
+ implementations from measurements made while their devices are
+ initialized.
+
+config CRYPTO_BENCHMARK
+ tristate "Crypto benchmarking module"
+ depends on m || EXPERT
+ select CRYPTO_BENCHMARK_LIB
help
Quick & dirty crypto benchmarking module.
diff --git a/crypto/Makefile b/crypto/Makefile
index 8386d55a9755..77720b746503 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -165,6 +165,8 @@ UBSAN_SANITIZE_jitterentropy.o = n
jitterentropy_rng-y := jitterentropy.o jitterentropy-kcapi.o
obj-$(CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE) += jitterentropy-testing.o
obj-$(CONFIG_CRYPTO_BENCHMARK) += tcrypt.o
+obj-$(CONFIG_CRYPTO_BENCHMARK_LIB) += crypt-benchmark.o
+crypt-benchmark-y := benchmark.o
obj-$(CONFIG_CRYPTO_USER_API) += af_alg.o
obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o
obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o
diff --git a/crypto/benchmark.c b/crypto/benchmark.c
new file mode 100644
index 000000000000..ad27cdaa7491
--- /dev/null
+++ b/crypto/benchmark.c
@@ -0,0 +1,180 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <crypto/aead.h>
+#include <crypto/benchmark.h>
+#include <crypto/hash.h>
+#include <crypto/skcipher.h>
+#include <linux/module.h>
+#include <linux/timex.h>
+
+int crypto_benchmark_aead_cycles(struct aead_request *req, bool encrypt,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles)
+{
+ struct crypto_wait *wait;
+ unsigned int i;
+ u64 cycles = 0;
+ int ret;
+
+ if (!req || !req->base.data || !runs || !total_cycles)
+ return -EINVAL;
+
+ wait = req->base.data;
+ *total_cycles = 0;
+
+ for (i = 0; i < warmup_runs; i++) {
+ if (encrypt)
+ ret = crypto_wait_req(crypto_aead_encrypt(req), wait);
+ else
+ ret = crypto_wait_req(crypto_aead_decrypt(req), wait);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < runs; i++) {
+ cycles_t start, end;
+
+ start = get_cycles();
+ if (encrypt)
+ ret = crypto_wait_req(crypto_aead_encrypt(req), wait);
+ else
+ ret = crypto_wait_req(crypto_aead_decrypt(req), wait);
+ end = get_cycles();
+ if (ret)
+ return ret;
+
+ cycles += end - start;
+ }
+
+ *total_cycles = cycles;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(crypto_benchmark_aead_cycles);
+
+int crypto_benchmark_ahash_cycles(struct ahash_request *req,
+ unsigned int block_size,
+ unsigned int update_size,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles)
+{
+ struct crypto_wait *wait;
+ unsigned int processed;
+ unsigned int i;
+ u64 cycles = 0;
+ int ret;
+
+ if (!req || !req->base.data || !block_size || !update_size ||
+ block_size % update_size || !runs || !total_cycles)
+ return -EINVAL;
+
+ wait = req->base.data;
+ *total_cycles = 0;
+
+ for (i = 0; i < warmup_runs; i++) {
+ if (update_size == block_size) {
+ ret = crypto_wait_req(crypto_ahash_digest(req), wait);
+ if (ret)
+ return ret;
+ continue;
+ }
+
+ ret = crypto_wait_req(crypto_ahash_init(req), wait);
+ if (ret)
+ return ret;
+ for (processed = 0; processed < block_size;
+ processed += update_size) {
+ ret = crypto_wait_req(crypto_ahash_update(req), wait);
+ if (ret)
+ return ret;
+ }
+ ret = crypto_wait_req(crypto_ahash_final(req), wait);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < runs; i++) {
+ cycles_t start, end;
+
+ start = get_cycles();
+ if (update_size == block_size) {
+ ret = crypto_wait_req(crypto_ahash_digest(req), wait);
+ } else {
+ ret = crypto_wait_req(crypto_ahash_init(req), wait);
+ if (ret)
+ goto measure_end;
+ for (processed = 0; processed < block_size;
+ processed += update_size) {
+ ret = crypto_wait_req(crypto_ahash_update(req),
+ wait);
+ if (ret)
+ goto measure_end;
+ }
+ ret = crypto_wait_req(crypto_ahash_final(req), wait);
+ }
+measure_end:
+ end = get_cycles();
+ if (ret)
+ return ret;
+
+ cycles += end - start;
+ }
+
+ *total_cycles = cycles;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(crypto_benchmark_ahash_cycles);
+
+int crypto_benchmark_skcipher_cycles(struct skcipher_request *req, bool encrypt,
+ unsigned int warmup_runs,
+ unsigned int runs, u64 *total_cycles)
+{
+ struct crypto_wait *wait;
+ unsigned int i;
+ u64 cycles = 0;
+ int ret;
+
+ if (!req || !req->base.data || !runs || !total_cycles)
+ return -EINVAL;
+
+ wait = req->base.data;
+ *total_cycles = 0;
+
+ for (i = 0; i < warmup_runs; i++) {
+ if (encrypt)
+ ret = crypto_wait_req(crypto_skcipher_encrypt(req),
+ wait);
+ else
+ ret = crypto_wait_req(crypto_skcipher_decrypt(req),
+ wait);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < runs; i++) {
+ cycles_t start, end;
+
+ start = get_cycles();
+ if (encrypt)
+ ret = crypto_wait_req(crypto_skcipher_encrypt(req),
+ wait);
+ else
+ ret = crypto_wait_req(crypto_skcipher_decrypt(req),
+ wait);
+ end = get_cycles();
+ if (ret)
+ return ret;
+
+ cycles += end - start;
+ }
+
+ *total_cycles = cycles;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(crypto_benchmark_skcipher_cycles);
+
+MODULE_AUTHOR("Jihong Min <hurryman2212@gmail.com>");
+MODULE_DESCRIPTION("Crypto API benchmark helpers");
+MODULE_LICENSE("GPL");
diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c
index 61a2501bfe9b..81c77f3d7a17 100644
--- a/crypto/tcrypt.c
+++ b/crypto/tcrypt.c
@@ -20,6 +20,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <crypto/aead.h>
+#include <crypto/benchmark.h>
#include <crypto/hash.h>
#include <crypto/skcipher.h>
#include <linux/err.h>
@@ -473,48 +474,6 @@ static int test_aead_jiffies(struct aead_request *req, int enc,
return 0;
}
-static int test_aead_cycles(struct aead_request *req, int enc, int blen)
-{
- unsigned long cycles = 0;
- int ret = 0;
- int i;
-
- /* Warm-up run. */
- for (i = 0; i < 4; i++) {
- if (enc)
- ret = do_one_aead_op(req, crypto_aead_encrypt(req));
- else
- ret = do_one_aead_op(req, crypto_aead_decrypt(req));
-
- if (ret)
- goto out;
- }
-
- /* The real thing. */
- for (i = 0; i < 8; i++) {
- cycles_t start, end;
-
- start = get_cycles();
- if (enc)
- ret = do_one_aead_op(req, crypto_aead_encrypt(req));
- else
- ret = do_one_aead_op(req, crypto_aead_decrypt(req));
- end = get_cycles();
-
- if (ret)
- goto out;
-
- cycles += end - start;
- }
-
-out:
- if (ret == 0)
- pr_cont("1 operation in %lu cycles (%d bytes)\n",
- (cycles + 4) / 8, blen);
-
- return ret;
-}
-
static void test_aead_speed(const char *algo, int enc, unsigned int secs,
struct aead_speed_template *template,
unsigned int tcount, u8 authsize,
@@ -669,7 +628,13 @@ static void test_aead_speed(const char *algo, int enc, unsigned int secs,
secs);
cond_resched();
} else {
- ret = test_aead_cycles(req, enc, bs);
+ u64 cycles;
+
+ ret = crypto_benchmark_aead_cycles(req, enc, 4,
+ 8, &cycles);
+ if (!ret)
+ pr_cont("1 operation in %llu cycles (%d bytes)\n",
+ (cycles + 4) / 8, bs);
}
if (ret) {
@@ -768,101 +733,6 @@ static int test_ahash_jiffies(struct ahash_request *req, int blen,
return 0;
}
-static int test_ahash_cycles_digest(struct ahash_request *req, int blen,
- char *out)
-{
- unsigned long cycles = 0;
- int ret, i;
-
- /* Warm-up run. */
- for (i = 0; i < 4; i++) {
- ret = do_one_ahash_op(req, crypto_ahash_digest(req));
- if (ret)
- goto out;
- }
-
- /* The real thing. */
- for (i = 0; i < 8; i++) {
- cycles_t start, end;
-
- start = get_cycles();
-
- ret = do_one_ahash_op(req, crypto_ahash_digest(req));
- if (ret)
- goto out;
-
- end = get_cycles();
-
- cycles += end - start;
- }
-
-out:
- if (ret)
- return ret;
-
- pr_cont("%6lu cycles/operation, %4lu cycles/byte\n",
- cycles / 8, cycles / (8 * blen));
-
- return 0;
-}
-
-static int test_ahash_cycles(struct ahash_request *req, int blen,
- int plen, char *out)
-{
- unsigned long cycles = 0;
- int i, pcount, ret;
-
- if (plen == blen)
- return test_ahash_cycles_digest(req, blen, out);
-
- /* Warm-up run. */
- for (i = 0; i < 4; i++) {
- ret = do_one_ahash_op(req, crypto_ahash_init(req));
- if (ret)
- goto out;
- for (pcount = 0; pcount < blen; pcount += plen) {
- ret = do_one_ahash_op(req, crypto_ahash_update(req));
- if (ret)
- goto out;
- }
- ret = do_one_ahash_op(req, crypto_ahash_final(req));
- if (ret)
- goto out;
- }
-
- /* The real thing. */
- for (i = 0; i < 8; i++) {
- cycles_t start, end;
-
- start = get_cycles();
-
- ret = do_one_ahash_op(req, crypto_ahash_init(req));
- if (ret)
- goto out;
- for (pcount = 0; pcount < blen; pcount += plen) {
- ret = do_one_ahash_op(req, crypto_ahash_update(req));
- if (ret)
- goto out;
- }
- ret = do_one_ahash_op(req, crypto_ahash_final(req));
- if (ret)
- goto out;
-
- end = get_cycles();
-
- cycles += end - start;
- }
-
-out:
- if (ret)
- return ret;
-
- pr_cont("%6lu cycles/operation, %4lu cycles/byte\n",
- cycles / 8, cycles / (8 * blen));
-
- return 0;
-}
-
static void test_ahash_speed_common(const char *algo, unsigned int secs,
struct hash_speed *speed, unsigned mask)
{
@@ -931,8 +801,15 @@ static void test_ahash_speed_common(const char *algo, unsigned int secs,
speed[i].plen, output, secs);
cond_resched();
} else {
- ret = test_ahash_cycles(req, speed[i].blen,
- speed[i].plen, output);
+ u64 cycles;
+
+ ret = crypto_benchmark_ahash_cycles(req, speed[i].blen,
+ speed[i].plen, 4, 8,
+ &cycles);
+ if (!ret)
+ pr_cont("%6llu cycles/operation, %4llu cycles/byte\n",
+ cycles / 8,
+ cycles / (8 * speed[i].blen));
}
if (ret) {
@@ -1249,53 +1126,6 @@ static int test_acipher_jiffies(struct skcipher_request *req, int enc,
return 0;
}
-static int test_acipher_cycles(struct skcipher_request *req, int enc,
- int blen)
-{
- unsigned long cycles = 0;
- int ret = 0;
- int i;
-
- /* Warm-up run. */
- for (i = 0; i < 4; i++) {
- if (enc)
- ret = do_one_acipher_op(req,
- crypto_skcipher_encrypt(req));
- else
- ret = do_one_acipher_op(req,
- crypto_skcipher_decrypt(req));
-
- if (ret)
- goto out;
- }
-
- /* The real thing. */
- for (i = 0; i < 8; i++) {
- cycles_t start, end;
-
- start = get_cycles();
- if (enc)
- ret = do_one_acipher_op(req,
- crypto_skcipher_encrypt(req));
- else
- ret = do_one_acipher_op(req,
- crypto_skcipher_decrypt(req));
- end = get_cycles();
-
- if (ret)
- goto out;
-
- cycles += end - start;
- }
-
-out:
- if (ret == 0)
- pr_cont("1 operation in %lu cycles (%d bytes)\n",
- (cycles + 4) / 8, blen);
-
- return ret;
-}
-
static void test_skcipher_speed(const char *algo, int enc, unsigned int secs,
struct cipher_speed_template *template,
unsigned int tcount, u8 *keysize, bool async)
@@ -1405,8 +1235,13 @@ static void test_skcipher_speed(const char *algo, int enc, unsigned int secs,
bs, secs);
cond_resched();
} else {
- ret = test_acipher_cycles(req, enc,
- bs);
+ u64 cycles;
+
+ ret = crypto_benchmark_skcipher_cycles(req, enc, 4,
+ 8, &cycles);
+ if (!ret)
+ pr_cont("1 operation in %llu cycles (%d bytes)\n",
+ (cycles + 4) / 8, bs);
}
if (ret) {
diff --git a/include/crypto/benchmark.h b/include/crypto/benchmark.h
new file mode 100644
index 000000000000..09dc0fb38483
--- /dev/null
+++ b/include/crypto/benchmark.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _CRYPTO_BENCHMARK_H
+#define _CRYPTO_BENCHMARK_H
+
+#include <linux/types.h>
+
+struct aead_request;
+struct ahash_request;
+struct skcipher_request;
+
+/* Requests must use crypto_req_done() with struct crypto_wait callback data. */
+int crypto_benchmark_aead_cycles(struct aead_request *req, bool encrypt,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles);
+int crypto_benchmark_ahash_cycles(struct ahash_request *req,
+ unsigned int block_size,
+ unsigned int update_size,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles);
+int crypto_benchmark_skcipher_cycles(struct skcipher_request *req, bool encrypt,
+ unsigned int warmup_runs,
+ unsigned int runs, u64 *total_cycles);
+
+#endif /* _CRYPTO_BENCHMARK_H */
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/4] crypto: introduce dynamic software fallback
2026-07-28 8:43 [PATCH 0/4] crypto: introduce generic dynamic software fallback and EIP93 support Jihong Min
2026-07-28 8:43 ` [PATCH 1/4] crypto: move cycle benchmark helpers out of tcrypt Jihong Min
@ 2026-07-28 8:43 ` Jihong Min
2026-07-28 8:43 ` [PATCH 3/4] crypto: eip93 - add dynamic software fallback support Jihong Min
2026-07-28 8:43 ` [PATCH 4/4] crypto: eliminate fallback proxy overhead while disabled Jihong Min
3 siblings, 0 replies; 5+ messages in thread
From: Jihong Min @ 2026-07-28 8:43 UTC (permalink / raw)
To: Herbert Xu, Christian Marangi, linux-crypto
Cc: Jihong Min, David S. Miller, Antoine Tenart, linux-kernel
Add an opt-in framework that lets crypto drivers compare hardware and
software implementations for algorithm groups whose members share
similar hardware scaling characteristics. Benchmark each group at fixed
block sizes from 16 to 16384 bytes, then expose the resulting per-device
CPU fallback thresholds through sysfs.
Use four warm-up runs and eight measured runs for each implementation.
Extend results within ten percent with sixteen additional runs. Treat
equal totals as a software win. After any software win, continue with
the next block size if one remains; otherwise select software-only
fallback. Select software or hardware per request using the threshold
computed at sysfs (re-)enablement for each crypto device.
Assisted-by: Codex:gpt-5.6
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
crypto/Kconfig | 9 +
crypto/Makefile | 1 +
crypto/fallback.c | 1335 +++++++++++++++++++++++++++++++++++++
include/crypto/fallback.h | 96 +++
4 files changed, 1441 insertions(+)
create mode 100644 crypto/fallback.c
create mode 100644 include/crypto/fallback.h
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 0cba261e3b88..c14b00768788 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -267,6 +267,15 @@ config CRYPTO_BENCHMARK
algorithms in the kernel. It should not be enabled in production
kernels.
+config CRYPTO_DYNAMIC_FALLBACK
+ tristate
+ select CRYPTO_BENCHMARK_LIB
+ help
+ Build generic support for selecting software fallback thresholds
+ from Crypto API benchmark results. Crypto drivers select this
+ internal library when they support runtime fallback tuning for
+ their hardware implementations.
+
config CRYPTO_SIMD
tristate
select CRYPTO_AEAD
diff --git a/crypto/Makefile b/crypto/Makefile
index 77720b746503..babbd7c6e984 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -167,6 +167,7 @@ obj-$(CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE) += jitterentropy-testing.o
obj-$(CONFIG_CRYPTO_BENCHMARK) += tcrypt.o
obj-$(CONFIG_CRYPTO_BENCHMARK_LIB) += crypt-benchmark.o
crypt-benchmark-y := benchmark.o
+obj-$(CONFIG_CRYPTO_DYNAMIC_FALLBACK) += fallback.o
obj-$(CONFIG_CRYPTO_USER_API) += af_alg.o
obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o
obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o
diff --git a/crypto/fallback.c b/crypto/fallback.c
new file mode 100644
index 000000000000..af8830f1077c
--- /dev/null
+++ b/crypto/fallback.c
@@ -0,0 +1,1335 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <crypto/authenc.h>
+#include <crypto/benchmark.h>
+#include <crypto/fallback.h>
+#include <crypto/internal/aead.h>
+#include <crypto/internal/hash.h>
+#include <crypto/internal/skcipher.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/kernel.h>
+#include <linux/limits.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/random.h>
+#include <linux/rtnetlink.h>
+#include <linux/scatterlist.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/sysfs.h>
+
+static const unsigned int crypto_fallback_block_sizes[] = {
+ 16, 64, 128, 256, 512, 1024, 1420, 2048, 4096, 8192, 16384,
+};
+
+enum crypto_fallback_implementation {
+ CRYPTO_FALLBACK_HARDWARE,
+ CRYPTO_FALLBACK_SOFTWARE,
+};
+
+struct fallback_threshold {
+ int value;
+};
+
+struct fallback_alg;
+
+struct crypto_fallback_group_state {
+ struct fallback_threshold threshold;
+ int value;
+ struct crypto_fallback_group group;
+ struct crypto_fallback *fallback;
+ struct device_attribute dev_attr;
+};
+
+struct crypto_fallback {
+ struct device *dev;
+ struct crypto_fallback_group_state *groups;
+ struct attribute_group sysfs_group;
+ struct attribute **sysfs_attrs;
+ struct device_attribute enabled_attr;
+ struct mutex lock; /* Serializes tuning and sysfs writes. */
+ struct fallback_alg *algs;
+ unsigned int num_groups;
+ unsigned int num_algs;
+ bool enabled;
+};
+
+static const char *
+crypto_fallback_alg_name(const struct crypto_fallback_benchmark *alg,
+ enum crypto_fallback_implementation implementation)
+{
+ if (implementation == CRYPTO_FALLBACK_HARDWARE)
+ return alg->driver_name;
+
+ return alg->name;
+}
+
+static u32
+crypto_fallback_alg_mask(enum crypto_fallback_implementation implementation)
+{
+ if (implementation == CRYPTO_FALLBACK_HARDWARE)
+ return 0;
+
+ return CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK;
+}
+
+static int fallback_bench_skcipher(const struct crypto_fallback_benchmark *alg,
+ enum crypto_fallback_implementation impl,
+ unsigned int block_size,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles)
+{
+ struct skcipher_request *req = NULL;
+ struct crypto_skcipher *tfm;
+ struct crypto_wait wait;
+ struct scatterlist sg;
+ unsigned int ivsize;
+ unsigned int len;
+ u8 *data = NULL;
+ u8 *iv = NULL;
+ int err;
+
+ tfm = crypto_alloc_skcipher(crypto_fallback_alg_name(alg, impl), 0,
+ crypto_fallback_alg_mask(impl));
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ err = crypto_skcipher_setkey(tfm, alg->key, alg->keylen);
+ if (err)
+ goto out_free_tfm;
+
+ len = round_up(block_size, crypto_skcipher_blocksize(tfm));
+ ivsize = crypto_skcipher_ivsize(tfm);
+ data = kmalloc(len, GFP_KERNEL);
+ iv = kzalloc(ivsize, GFP_KERNEL);
+ if (!data || (ivsize && !iv)) {
+ err = -ENOMEM;
+ goto out_free_buffers;
+ }
+
+ get_random_bytes(data, len);
+ sg_init_one(&sg, data, len);
+
+ req = skcipher_request_alloc(tfm, GFP_KERNEL);
+ if (!req) {
+ err = -ENOMEM;
+ goto out_free_buffers;
+ }
+
+ crypto_init_wait(&wait);
+ skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ crypto_req_done, &wait);
+ skcipher_request_set_crypt(req, &sg, &sg, len, iv);
+ err = crypto_benchmark_skcipher_cycles(req, true, warmup_runs, runs,
+ total_cycles);
+
+out_free_buffers:
+ skcipher_request_free(req);
+ kfree(iv);
+ kfree(data);
+out_free_tfm:
+ crypto_free_skcipher(tfm);
+
+ return err;
+}
+
+static int fallback_bench_ahash(const struct crypto_fallback_benchmark *alg,
+ enum crypto_fallback_implementation impl,
+ unsigned int block_size,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles)
+{
+ struct ahash_request *req = NULL;
+ struct crypto_ahash *tfm;
+ struct crypto_wait wait;
+ struct scatterlist src;
+ u8 *input = NULL;
+ u8 *output = NULL;
+ int err;
+
+ tfm = crypto_alloc_ahash(crypto_fallback_alg_name(alg, impl), 0,
+ crypto_fallback_alg_mask(impl));
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ if (alg->keylen) {
+ err = crypto_ahash_setkey(tfm, alg->key, alg->keylen);
+ if (err)
+ goto out_free_tfm;
+ }
+
+ input = kmalloc(block_size, GFP_KERNEL);
+ output = kmalloc(crypto_ahash_digestsize(tfm), GFP_KERNEL);
+ if (!input || !output) {
+ err = -ENOMEM;
+ goto out_free_buffers;
+ }
+
+ get_random_bytes(input, block_size);
+ sg_init_one(&src, input, block_size);
+
+ req = ahash_request_alloc(tfm, GFP_KERNEL);
+ if (!req) {
+ err = -ENOMEM;
+ goto out_free_buffers;
+ }
+
+ crypto_init_wait(&wait);
+ ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ crypto_req_done, &wait);
+ ahash_request_set_crypt(req, &src, output, block_size);
+ err = crypto_benchmark_ahash_cycles(req, block_size, block_size,
+ warmup_runs, runs, total_cycles);
+
+out_free_buffers:
+ ahash_request_free(req);
+ kfree(output);
+ kfree(input);
+out_free_tfm:
+ crypto_free_ahash(tfm);
+
+ return err;
+}
+
+static int
+crypto_fallback_aead_setkey(struct crypto_aead *tfm,
+ const struct crypto_fallback_benchmark *alg)
+{
+ struct crypto_authenc_key_param *param;
+ struct rtattr *rta;
+ unsigned int len;
+ u8 *key;
+ int err;
+
+ if (!alg->authkeylen)
+ return crypto_aead_setkey(tfm, alg->key, alg->keylen);
+
+ len = RTA_LENGTH(sizeof(*param)) + alg->authkeylen + alg->keylen;
+ key = kzalloc(len, GFP_KERNEL);
+ if (!key)
+ return -ENOMEM;
+
+ rta = (struct rtattr *)key;
+ rta->rta_type = CRYPTO_AUTHENC_KEYA_PARAM;
+ rta->rta_len = RTA_LENGTH(sizeof(*param));
+ param = RTA_DATA(rta);
+ param->enckeylen = cpu_to_be32(alg->keylen);
+ memcpy(key + rta->rta_len, alg->authkey, alg->authkeylen);
+ memcpy(key + rta->rta_len + alg->authkeylen, alg->key, alg->keylen);
+
+ err = crypto_aead_setkey(tfm, key, len);
+ kfree_sensitive(key);
+
+ return err;
+}
+
+static int fallback_bench_aead(const struct crypto_fallback_benchmark *alg,
+ enum crypto_fallback_implementation impl,
+ unsigned int block_size,
+ unsigned int warmup_runs, unsigned int runs,
+ u64 *total_cycles)
+{
+ struct aead_request *req = NULL;
+ struct crypto_aead *tfm;
+ struct crypto_wait wait;
+ struct scatterlist src;
+ struct scatterlist dst;
+ unsigned int ivsize;
+ unsigned int len;
+ u8 *input = NULL;
+ u8 *output = NULL;
+ u8 *iv = NULL;
+ int err;
+
+ tfm = crypto_alloc_aead(crypto_fallback_alg_name(alg, impl), 0,
+ crypto_fallback_alg_mask(impl));
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ err = crypto_aead_setauthsize(tfm, alg->authsize);
+ if (err)
+ goto out_free_tfm;
+ err = crypto_fallback_aead_setkey(tfm, alg);
+ if (err)
+ goto out_free_tfm;
+
+ len = round_up(block_size, crypto_aead_blocksize(tfm));
+ ivsize = crypto_aead_ivsize(tfm);
+ input = kmalloc(len, GFP_KERNEL);
+ output = kmalloc(len + alg->authsize, GFP_KERNEL);
+ iv = kzalloc(ivsize, GFP_KERNEL);
+ if (!input || !output || (ivsize && !iv)) {
+ err = -ENOMEM;
+ goto out_free_buffers;
+ }
+
+ get_random_bytes(input, len);
+ sg_init_one(&src, input, len);
+ sg_init_one(&dst, output, len + alg->authsize);
+
+ req = aead_request_alloc(tfm, GFP_KERNEL);
+ if (!req) {
+ err = -ENOMEM;
+ goto out_free_buffers;
+ }
+
+ crypto_init_wait(&wait);
+ aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ crypto_req_done, &wait);
+ aead_request_set_crypt(req, &src, &dst, len, iv);
+ aead_request_set_ad(req, 0);
+ err = crypto_benchmark_aead_cycles(req, true, warmup_runs, runs,
+ total_cycles);
+
+out_free_buffers:
+ aead_request_free(req);
+ kfree(iv);
+ kfree(output);
+ kfree(input);
+out_free_tfm:
+ crypto_free_aead(tfm);
+
+ return err;
+}
+
+static int fallback_bench(const struct crypto_fallback_benchmark *alg,
+ enum crypto_fallback_implementation impl,
+ unsigned int block_size, unsigned int warmup_runs,
+ unsigned int runs, u64 *total_cycles)
+{
+ switch (alg->type) {
+ case CRYPTO_FALLBACK_SKCIPHER:
+ return fallback_bench_skcipher(alg, impl, block_size,
+ warmup_runs, runs, total_cycles);
+ case CRYPTO_FALLBACK_AHASH:
+ return fallback_bench_ahash(alg, impl, block_size, warmup_runs,
+ runs, total_cycles);
+ case CRYPTO_FALLBACK_AEAD:
+ return fallback_bench_aead(alg, impl, block_size, warmup_runs,
+ runs, total_cycles);
+ }
+
+ return -EINVAL;
+}
+
+static ssize_t fallback_threshold_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct crypto_fallback_group_state *group;
+
+ group = container_of(attr, struct crypto_fallback_group_state,
+ dev_attr);
+
+ return sysfs_emit(buf, "%d\n", READ_ONCE(group->value));
+}
+
+static ssize_t fallback_threshold_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct crypto_fallback_group_state *group;
+ int threshold;
+ int err;
+
+ err = kstrtoint(buf, 0, &threshold);
+ if (err)
+ return err;
+ if (threshold < -1)
+ return -ERANGE;
+
+ group = container_of(attr, struct crypto_fallback_group_state,
+ dev_attr);
+ mutex_lock(&group->fallback->lock);
+ WRITE_ONCE(group->value, threshold);
+ if (group->fallback->enabled)
+ WRITE_ONCE(group->threshold.value, threshold);
+ mutex_unlock(&group->fallback->lock);
+
+ return count;
+}
+
+static int crypto_fallback_tune_group(const struct crypto_fallback_group *group,
+ int *threshold)
+{
+ u64 hardware_cycles;
+ u64 software_cycles;
+ unsigned int i;
+ int err;
+
+ for (i = 0; i < ARRAY_SIZE(crypto_fallback_block_sizes); i++) {
+ u64 difference;
+ u64 lower;
+
+ err = fallback_bench(&group->benchmark,
+ CRYPTO_FALLBACK_HARDWARE,
+ crypto_fallback_block_sizes[i], 4, 8,
+ &hardware_cycles);
+ if (err)
+ return err;
+
+ err = fallback_bench(&group->benchmark,
+ CRYPTO_FALLBACK_SOFTWARE,
+ crypto_fallback_block_sizes[i], 4, 8,
+ &software_cycles);
+ if (err)
+ return err;
+ if (!hardware_cycles || !software_cycles)
+ return -EOPNOTSUPP;
+
+ difference = hardware_cycles > software_cycles ?
+ hardware_cycles - software_cycles :
+ software_cycles - hardware_cycles;
+ lower = min(hardware_cycles, software_cycles);
+
+ /* Extend results whose initial averages differ by at most 10%. */
+ if (difference <= lower / 10) {
+ u64 extra_cycles;
+
+ err = fallback_bench(&group->benchmark,
+ CRYPTO_FALLBACK_HARDWARE,
+ crypto_fallback_block_sizes[i], 0,
+ 16, &extra_cycles);
+ if (err)
+ return err;
+ if (!extra_cycles)
+ return -EOPNOTSUPP;
+ hardware_cycles += extra_cycles;
+
+ err = fallback_bench(&group->benchmark,
+ CRYPTO_FALLBACK_SOFTWARE,
+ crypto_fallback_block_sizes[i], 0,
+ 16, &extra_cycles);
+ if (err)
+ return err;
+ if (!extra_cycles)
+ return -EOPNOTSUPP;
+ software_cycles += extra_cycles;
+ }
+
+ /*
+ * Both totals contain the same number of samples, so comparing
+ * them is equivalent to comparing their arithmetic averages.
+ */
+ if (hardware_cycles < software_cycles) {
+ *threshold = i ? crypto_fallback_block_sizes[i - 1] : 0;
+ return 0;
+ }
+ }
+
+ *threshold = -1;
+
+ return 0;
+}
+
+static ssize_t fallback_enabled_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct crypto_fallback *fallback;
+
+ fallback = container_of(attr, struct crypto_fallback, enabled_attr);
+
+ return sysfs_emit(buf, "%u\n", READ_ONCE(fallback->enabled));
+}
+
+static ssize_t fallback_enabled_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct crypto_fallback *fallback;
+ bool enabled;
+ unsigned int i;
+ int err;
+
+ err = kstrtobool(buf, &enabled);
+ if (err)
+ return err;
+
+ fallback = container_of(attr, struct crypto_fallback, enabled_attr);
+ mutex_lock(&fallback->lock);
+ if (!enabled && !fallback->enabled)
+ goto out_unlock;
+
+ if (enabled) {
+ for (i = 0; i < fallback->num_groups; i++)
+ WRITE_ONCE(fallback->groups[i].threshold.value, 0);
+
+ for (i = 0; i < fallback->num_groups; i++) {
+ struct crypto_fallback_group_state *state;
+ int threshold;
+
+ state = &fallback->groups[i];
+ err = crypto_fallback_tune_group(&state->group,
+ &threshold);
+ if (err) {
+ dev_warn(fallback->dev,
+ "%s benchmark failed: %d; using hardware\n",
+ state->group.name, err);
+ threshold = 0;
+ }
+ WRITE_ONCE(state->value, threshold);
+ dev_info(fallback->dev,
+ "%s CPU fallback threshold: %d\n",
+ state->group.name, threshold);
+ }
+
+ for (i = 0; i < fallback->num_groups; i++)
+ WRITE_ONCE(fallback->groups[i].threshold.value,
+ fallback->groups[i].value);
+ } else {
+ for (i = 0; i < fallback->num_groups; i++)
+ WRITE_ONCE(fallback->groups[i].threshold.value, 0);
+ }
+ WRITE_ONCE(fallback->enabled, enabled);
+
+out_unlock:
+ mutex_unlock(&fallback->lock);
+
+ return count;
+}
+
+static void crypto_fallback_free(struct crypto_fallback *fallback)
+{
+ unsigned int i;
+
+ if (!fallback)
+ return;
+
+ for (i = 0; i < fallback->num_groups; i++)
+ kfree_const(fallback->groups[i].group.name);
+ kfree(fallback->algs);
+ kfree(fallback->sysfs_attrs);
+ kfree(fallback->groups);
+ mutex_destroy(&fallback->lock);
+ put_device(fallback->dev);
+ kfree(fallback);
+}
+
+static struct crypto_fallback *
+fallback_alloc(struct device *dev, const struct crypto_fallback_group *groups,
+ unsigned int num_groups)
+{
+ struct crypto_fallback *fallback;
+ unsigned int i;
+ int err;
+
+ if (!dev || !groups || !num_groups)
+ return ERR_PTR(-EINVAL);
+
+ fallback = kzalloc_obj(*fallback, GFP_KERNEL);
+ if (!fallback)
+ return ERR_PTR(-ENOMEM);
+
+ mutex_init(&fallback->lock);
+ fallback->dev = get_device(dev);
+ fallback->num_groups = num_groups;
+ fallback->groups =
+ kcalloc(num_groups, sizeof(*fallback->groups), GFP_KERNEL);
+ fallback->sysfs_attrs =
+ kcalloc(num_groups + 2, sizeof(*fallback->sysfs_attrs),
+ GFP_KERNEL);
+ if (!fallback->groups || !fallback->sysfs_attrs) {
+ err = -ENOMEM;
+ goto err_free;
+ }
+
+ for (i = 0; i < num_groups; i++) {
+ struct crypto_fallback_group_state *state;
+
+ if (!groups[i].name || !groups[i].name[0] ||
+ !strcmp(groups[i].name, "enabled") ||
+ !groups[i].benchmark.name ||
+ !groups[i].benchmark.driver_name ||
+ groups[i].benchmark.type > CRYPTO_FALLBACK_AEAD ||
+ (groups[i].benchmark.keylen && !groups[i].benchmark.key) ||
+ (groups[i].benchmark.authkeylen &&
+ !groups[i].benchmark.authkey) ||
+ (groups[i].benchmark.type == CRYPTO_FALLBACK_AEAD &&
+ !groups[i].benchmark.authsize)) {
+ err = -EINVAL;
+ goto err_free;
+ }
+
+ state = &fallback->groups[i];
+ state->fallback = fallback;
+ state->group = groups[i];
+ state->group.name = kstrdup_const(groups[i].name, GFP_KERNEL);
+ if (!state->group.name) {
+ err = -ENOMEM;
+ goto err_free;
+ }
+
+ sysfs_attr_init(&state->dev_attr.attr);
+ state->dev_attr.attr.name = state->group.name;
+ state->dev_attr.attr.mode = 0644;
+ state->dev_attr.show = fallback_threshold_show;
+ state->dev_attr.store = fallback_threshold_store;
+ fallback->sysfs_attrs[i + 1] = &state->dev_attr.attr;
+ }
+
+ sysfs_attr_init(&fallback->enabled_attr.attr);
+ fallback->enabled_attr.attr.name = "enabled";
+ fallback->enabled_attr.attr.mode = 0644;
+ fallback->enabled_attr.show = fallback_enabled_show;
+ fallback->enabled_attr.store = fallback_enabled_store;
+ fallback->sysfs_attrs[0] = &fallback->enabled_attr.attr;
+
+ fallback->sysfs_group.name = "cpu_fallback_thresholds";
+ fallback->sysfs_group.attrs = fallback->sysfs_attrs;
+ err = sysfs_create_group(&dev->kobj, &fallback->sysfs_group);
+ if (err)
+ goto err_free;
+
+ return fallback;
+
+err_free:
+ crypto_fallback_free(fallback);
+
+ return ERR_PTR(err);
+}
+
+static void fallback_release(struct crypto_fallback *fallback)
+{
+ if (!fallback || IS_ERR(fallback))
+ return;
+
+ sysfs_remove_group(&fallback->dev->kobj, &fallback->sysfs_group);
+ crypto_fallback_free(fallback);
+}
+
+static bool fallback_use_software(const struct fallback_threshold *threshold,
+ unsigned int input_size)
+{
+ int value = READ_ONCE(threshold->value);
+
+ return value < 0 || (value > 0 && input_size <= value);
+}
+
+struct fallback_skcipher_ctx {
+ struct crypto_skcipher *hardware;
+ struct crypto_skcipher *software;
+};
+
+struct fallback_aead_ctx {
+ struct crypto_aead *hardware;
+ struct crypto_aead *software;
+};
+
+struct fallback_ahash_ctx {
+ struct crypto_ahash *hardware;
+ struct crypto_ahash *software;
+};
+
+struct fallback_ahash_reqctx {
+ bool use_software;
+ struct ahash_request subreq;
+};
+
+struct fallback_ahash_state {
+ bool use_software;
+ u8 state[] CRYPTO_MINALIGN_ATTR;
+};
+
+struct fallback_alg {
+ const struct fallback_threshold *threshold;
+ char driver_name[CRYPTO_MAX_ALG_NAME];
+ enum crypto_fallback_alg_type type;
+ union {
+ struct skcipher_alg skcipher;
+ struct aead_alg aead;
+ struct ahash_alg ahash;
+ } alg;
+};
+
+static struct fallback_alg *fallback_skcipher_alg(struct crypto_skcipher *tfm)
+{
+ return container_of(crypto_skcipher_alg(tfm), struct fallback_alg,
+ alg.skcipher);
+}
+
+static struct fallback_alg *fallback_aead_alg(struct crypto_aead *tfm)
+{
+ return container_of(crypto_aead_alg(tfm), struct fallback_alg,
+ alg.aead);
+}
+
+static struct fallback_alg *fallback_ahash_alg(struct crypto_ahash *tfm)
+{
+ return container_of(crypto_ahash_alg(tfm), struct fallback_alg,
+ alg.ahash);
+}
+
+static int fallback_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct fallback_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+ u32 flags = crypto_skcipher_get_flags(tfm) & CRYPTO_TFM_REQ_MASK;
+ int err;
+
+ crypto_skcipher_clear_flags(ctx->software, CRYPTO_TFM_REQ_MASK);
+ crypto_skcipher_set_flags(ctx->software, flags);
+ err = crypto_skcipher_setkey(ctx->software, key, keylen);
+ if (err)
+ return err;
+
+ crypto_skcipher_clear_flags(ctx->hardware, CRYPTO_TFM_REQ_MASK);
+ crypto_skcipher_set_flags(ctx->hardware, flags);
+
+ return crypto_skcipher_setkey(ctx->hardware, key, keylen);
+}
+
+static int fallback_skcipher_crypt(struct skcipher_request *req, bool encrypt)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct fallback_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+ struct fallback_alg *alg = fallback_skcipher_alg(tfm);
+ struct crypto_skcipher *child = ctx->hardware;
+ struct skcipher_request *subreq = skcipher_request_ctx(req);
+
+ if (fallback_use_software(alg->threshold, req->cryptlen))
+ child = ctx->software;
+
+ skcipher_request_set_tfm(subreq, child);
+ skcipher_request_set_callback(subreq, req->base.flags,
+ req->base.complete, req->base.data);
+ skcipher_request_set_crypt(subreq, req->src, req->dst, req->cryptlen,
+ req->iv);
+
+ if (encrypt)
+ return crypto_skcipher_encrypt(subreq);
+
+ return crypto_skcipher_decrypt(subreq);
+}
+
+static int fallback_skcipher_encrypt(struct skcipher_request *req)
+{
+ return fallback_skcipher_crypt(req, true);
+}
+
+static int fallback_skcipher_decrypt(struct skcipher_request *req)
+{
+ return fallback_skcipher_crypt(req, false);
+}
+
+static int fallback_skcipher_init(struct crypto_skcipher *tfm)
+{
+ struct fallback_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+ struct fallback_alg *alg = fallback_skcipher_alg(tfm);
+ struct crypto_skcipher *hardware;
+ struct crypto_skcipher *software;
+ const char *name;
+ u32 mask = CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK;
+
+ hardware = crypto_alloc_skcipher(alg->driver_name, 0, 0);
+ if (IS_ERR(hardware))
+ return PTR_ERR(hardware);
+
+ name = crypto_tfm_alg_name(crypto_skcipher_tfm(tfm));
+ software = crypto_alloc_skcipher(name, 0, mask);
+ if (IS_ERR(software)) {
+ crypto_free_skcipher(hardware);
+ return PTR_ERR(software);
+ }
+
+ ctx->hardware = hardware;
+ ctx->software = software;
+ crypto_skcipher_set_reqsize(tfm,
+ sizeof(struct skcipher_request) +
+ max(crypto_skcipher_reqsize(hardware),
+ crypto_skcipher_reqsize(software)));
+
+ return 0;
+}
+
+static void fallback_skcipher_exit(struct crypto_skcipher *tfm)
+{
+ struct fallback_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+
+ crypto_free_skcipher(ctx->software);
+ crypto_free_skcipher(ctx->hardware);
+}
+
+static int fallback_aead_setkey(struct crypto_aead *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct fallback_aead_ctx *ctx = crypto_aead_ctx(tfm);
+ u32 flags = crypto_aead_get_flags(tfm) & CRYPTO_TFM_REQ_MASK;
+ int err;
+
+ crypto_aead_clear_flags(ctx->software, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(ctx->software, flags);
+ err = crypto_aead_setkey(ctx->software, key, keylen);
+ if (err)
+ return err;
+
+ crypto_aead_clear_flags(ctx->hardware, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(ctx->hardware, flags);
+
+ return crypto_aead_setkey(ctx->hardware, key, keylen);
+}
+
+static int fallback_aead_setauthsize(struct crypto_aead *tfm,
+ unsigned int authsize)
+{
+ struct fallback_aead_ctx *ctx = crypto_aead_ctx(tfm);
+ int err;
+
+ err = crypto_aead_setauthsize(ctx->software, authsize);
+ if (err)
+ return err;
+
+ return crypto_aead_setauthsize(ctx->hardware, authsize);
+}
+
+static int fallback_aead_crypt(struct aead_request *req, bool encrypt)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ struct fallback_aead_ctx *ctx = crypto_aead_ctx(tfm);
+ struct fallback_alg *alg = fallback_aead_alg(tfm);
+ struct crypto_aead *child = ctx->hardware;
+ struct aead_request *subreq = aead_request_ctx(req);
+ unsigned int input_size = req->cryptlen;
+
+ if (!encrypt) {
+ unsigned int authsize = crypto_aead_authsize(tfm);
+
+ if (input_size < authsize)
+ return -EINVAL;
+ input_size -= authsize;
+ }
+
+ if (fallback_use_software(alg->threshold, input_size))
+ child = ctx->software;
+
+ aead_request_set_tfm(subreq, child);
+ aead_request_set_callback(subreq, req->base.flags, req->base.complete,
+ req->base.data);
+ aead_request_set_crypt(subreq, req->src, req->dst, req->cryptlen,
+ req->iv);
+ aead_request_set_ad(subreq, req->assoclen);
+
+ if (encrypt)
+ return crypto_aead_encrypt(subreq);
+
+ return crypto_aead_decrypt(subreq);
+}
+
+static int fallback_aead_encrypt(struct aead_request *req)
+{
+ return fallback_aead_crypt(req, true);
+}
+
+static int fallback_aead_decrypt(struct aead_request *req)
+{
+ return fallback_aead_crypt(req, false);
+}
+
+static int fallback_aead_init(struct crypto_aead *tfm)
+{
+ struct fallback_aead_ctx *ctx = crypto_aead_ctx(tfm);
+ struct fallback_alg *alg = fallback_aead_alg(tfm);
+ struct crypto_aead *hardware;
+ struct crypto_aead *software;
+
+ hardware = crypto_alloc_aead(alg->driver_name, 0, 0);
+ if (IS_ERR(hardware))
+ return PTR_ERR(hardware);
+
+ software =
+ crypto_alloc_aead(crypto_tfm_alg_name(crypto_aead_tfm(tfm)), 0,
+ CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(software)) {
+ crypto_free_aead(hardware);
+ return PTR_ERR(software);
+ }
+
+ ctx->hardware = hardware;
+ ctx->software = software;
+ crypto_aead_set_reqsize(tfm,
+ sizeof(struct aead_request) +
+ max(crypto_aead_reqsize(hardware),
+ crypto_aead_reqsize(software)));
+
+ return 0;
+}
+
+static void fallback_aead_exit(struct crypto_aead *tfm)
+{
+ struct fallback_aead_ctx *ctx = crypto_aead_ctx(tfm);
+
+ crypto_free_aead(ctx->software);
+ crypto_free_aead(ctx->hardware);
+}
+
+static struct ahash_request *fallback_ahash_subreq(struct ahash_request *req,
+ bool reset)
+{
+ struct fallback_ahash_reqctx *rctx = ahash_request_ctx(req);
+ struct fallback_ahash_ctx *ctx =
+ crypto_ahash_ctx(crypto_ahash_reqtfm(req));
+ struct crypto_ahash *child = rctx->use_software ? ctx->software :
+ ctx->hardware;
+ struct ahash_request *subreq = &rctx->subreq;
+
+ if (reset)
+ memset(subreq, 0, sizeof(*subreq));
+ ahash_request_set_tfm(subreq, child);
+ ahash_request_set_callback(subreq, req->base.flags, req->base.complete,
+ req->base.data);
+ if (ahash_request_isvirt(req))
+ ahash_request_set_virt(subreq, req->svirt, req->result,
+ req->nbytes);
+ else
+ ahash_request_set_crypt(subreq, req->src, req->result,
+ req->nbytes);
+
+ return subreq;
+}
+
+static int fallback_ahash_setkey(struct crypto_ahash *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct fallback_ahash_ctx *ctx = crypto_ahash_ctx(tfm);
+ u32 flags = crypto_ahash_get_flags(tfm) & CRYPTO_TFM_REQ_MASK;
+ int err;
+
+ crypto_ahash_clear_flags(ctx->software, CRYPTO_TFM_REQ_MASK);
+ crypto_ahash_set_flags(ctx->software, flags);
+ err = crypto_ahash_setkey(ctx->software, key, keylen);
+ if (err)
+ return err;
+
+ crypto_ahash_clear_flags(ctx->hardware, CRYPTO_TFM_REQ_MASK);
+ crypto_ahash_set_flags(ctx->hardware, flags);
+
+ return crypto_ahash_setkey(ctx->hardware, key, keylen);
+}
+
+static int fallback_ahash_init_req(struct ahash_request *req)
+{
+ struct fallback_ahash_reqctx *rctx = ahash_request_ctx(req);
+ struct fallback_alg *alg = fallback_ahash_alg(crypto_ahash_reqtfm(req));
+
+ rctx->use_software = fallback_use_software(alg->threshold, UINT_MAX);
+
+ return crypto_ahash_init(fallback_ahash_subreq(req, true));
+}
+
+static int fallback_ahash_update(struct ahash_request *req)
+{
+ return crypto_ahash_update(fallback_ahash_subreq(req, false));
+}
+
+static int fallback_ahash_final(struct ahash_request *req)
+{
+ return crypto_ahash_final(fallback_ahash_subreq(req, false));
+}
+
+static int fallback_ahash_finup(struct ahash_request *req)
+{
+ return crypto_ahash_finup(fallback_ahash_subreq(req, false));
+}
+
+static int fallback_ahash_digest(struct ahash_request *req)
+{
+ struct fallback_ahash_reqctx *rctx = ahash_request_ctx(req);
+ struct fallback_alg *alg = fallback_ahash_alg(crypto_ahash_reqtfm(req));
+
+ rctx->use_software = fallback_use_software(alg->threshold, req->nbytes);
+
+ return crypto_ahash_digest(fallback_ahash_subreq(req, true));
+}
+
+static int fallback_ahash_export(struct ahash_request *req, void *out)
+{
+ struct fallback_ahash_reqctx *rctx = ahash_request_ctx(req);
+ struct fallback_ahash_state *state = out;
+
+ state->use_software = rctx->use_software;
+
+ return crypto_ahash_export(fallback_ahash_subreq(req, false),
+ state->state);
+}
+
+static int fallback_ahash_import(struct ahash_request *req, const void *in)
+{
+ const struct fallback_ahash_state *state = in;
+ struct fallback_ahash_reqctx *rctx = ahash_request_ctx(req);
+
+ rctx->use_software = state->use_software;
+
+ return crypto_ahash_import(fallback_ahash_subreq(req, true),
+ state->state);
+}
+
+static int fallback_ahash_init(struct crypto_ahash *tfm)
+{
+ struct fallback_ahash_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct fallback_alg *alg = fallback_ahash_alg(tfm);
+ struct crypto_ahash *hardware;
+ struct crypto_ahash *software;
+
+ hardware = crypto_alloc_ahash(alg->driver_name, 0, 0);
+ if (IS_ERR(hardware))
+ return PTR_ERR(hardware);
+
+ software = crypto_alloc_ahash(crypto_ahash_alg_name(tfm),
+ CRYPTO_ALG_REQ_VIRT,
+ CRYPTO_ALG_ASYNC | CRYPTO_ALG_REQ_VIRT |
+ CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(software)) {
+ crypto_free_ahash(hardware);
+ return PTR_ERR(software);
+ }
+
+ ctx->hardware = hardware;
+ ctx->software = software;
+ crypto_ahash_set_reqsize(tfm,
+ sizeof(struct fallback_ahash_reqctx) +
+ max(crypto_ahash_reqsize(hardware),
+ crypto_ahash_reqsize(software)));
+ crypto_ahash_set_statesize(tfm,
+ offsetof(struct fallback_ahash_state, state) +
+ max(crypto_ahash_statesize(hardware),
+ crypto_ahash_statesize(software)));
+
+ return 0;
+}
+
+static void fallback_ahash_exit(struct crypto_ahash *tfm)
+{
+ struct fallback_ahash_ctx *ctx = crypto_ahash_ctx(tfm);
+
+ crypto_free_ahash(ctx->software);
+ crypto_free_ahash(ctx->hardware);
+}
+
+static int fallback_alg_name(struct crypto_alg *alg, const char *name,
+ const char *driver_name)
+{
+ if (strscpy(alg->cra_name, name, CRYPTO_MAX_ALG_NAME) < 0)
+ return -ENAMETOOLONG;
+
+ if (snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "fallback(%s)",
+ driver_name) >= CRYPTO_MAX_ALG_NAME)
+ return -ENAMETOOLONG;
+
+ return 0;
+}
+
+static void fallback_init_base(struct crypto_alg *alg,
+ const struct crypto_alg *hardware,
+ const struct crypto_alg *software,
+ struct module *owner)
+{
+ alg->cra_flags =
+ CRYPTO_ALG_ASYNC | CRYPTO_ALG_NO_FALLBACK |
+ ((hardware->cra_flags | software->cra_flags) &
+ (CRYPTO_ALG_ALLOCATES_MEMORY | CRYPTO_ALG_KERN_DRIVER_ONLY));
+ alg->cra_priority = hardware->cra_priority + 1;
+ alg->cra_blocksize = hardware->cra_blocksize;
+ alg->cra_alignmask =
+ max(hardware->cra_alignmask, software->cra_alignmask);
+ alg->cra_module = owner;
+}
+
+static int fallback_register_skcipher(struct fallback_alg *entry,
+ struct module *owner)
+{
+ struct crypto_skcipher *hardware;
+ struct crypto_skcipher *software;
+ struct skcipher_alg *alg = &entry->alg.skcipher;
+ struct skcipher_alg_common *hcommon;
+ struct skcipher_alg_common *scommon;
+ int err;
+
+ hardware = crypto_alloc_skcipher(entry->driver_name, 0, 0);
+ if (IS_ERR(hardware))
+ return PTR_ERR(hardware);
+
+ hcommon = crypto_skcipher_alg_common(hardware);
+ software = crypto_alloc_skcipher(hcommon->base.cra_name, 0,
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(software)) {
+ err = PTR_ERR(software);
+ goto out_free_hardware;
+ }
+ scommon = crypto_skcipher_alg_common(software);
+
+ if (hcommon->base.cra_blocksize != scommon->base.cra_blocksize ||
+ hcommon->ivsize != scommon->ivsize) {
+ err = -EINVAL;
+ goto out_free_software;
+ }
+
+ err = fallback_alg_name(&alg->base, hcommon->base.cra_name,
+ entry->driver_name);
+ if (err)
+ goto out_free_software;
+ fallback_init_base(&alg->base, &hcommon->base, &scommon->base, owner);
+ alg->min_keysize = max(hcommon->min_keysize, scommon->min_keysize);
+ alg->max_keysize = min(hcommon->max_keysize, scommon->max_keysize);
+ if (alg->min_keysize > alg->max_keysize) {
+ err = -EINVAL;
+ goto out_free_software;
+ }
+ alg->ivsize = hcommon->ivsize;
+ alg->chunksize = max(hcommon->chunksize, scommon->chunksize);
+ alg->walksize = alg->chunksize;
+ alg->base.cra_ctxsize = sizeof(struct fallback_skcipher_ctx);
+ alg->init = fallback_skcipher_init;
+ alg->exit = fallback_skcipher_exit;
+ alg->setkey = fallback_skcipher_setkey;
+ alg->encrypt = fallback_skcipher_encrypt;
+ alg->decrypt = fallback_skcipher_decrypt;
+
+ err = crypto_register_skcipher(alg);
+
+out_free_software:
+ crypto_free_skcipher(software);
+out_free_hardware:
+ crypto_free_skcipher(hardware);
+
+ return err;
+}
+
+static int fallback_register_aead(struct fallback_alg *entry,
+ struct module *owner)
+{
+ struct crypto_aead *hardware;
+ struct crypto_aead *software;
+ struct aead_alg *alg = &entry->alg.aead;
+ struct aead_alg *halg;
+ struct aead_alg *salg;
+ int err;
+
+ hardware = crypto_alloc_aead(entry->driver_name, 0, 0);
+ if (IS_ERR(hardware))
+ return PTR_ERR(hardware);
+ halg = crypto_aead_alg(hardware);
+
+ software =
+ crypto_alloc_aead(halg->base.cra_name, 0,
+ CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(software)) {
+ err = PTR_ERR(software);
+ goto out_free_hardware;
+ }
+ salg = crypto_aead_alg(software);
+
+ if (halg->base.cra_blocksize != salg->base.cra_blocksize ||
+ halg->ivsize != salg->ivsize) {
+ err = -EINVAL;
+ goto out_free_software;
+ }
+
+ err = fallback_alg_name(&alg->base, halg->base.cra_name,
+ entry->driver_name);
+ if (err)
+ goto out_free_software;
+ fallback_init_base(&alg->base, &halg->base, &salg->base, owner);
+ alg->ivsize = halg->ivsize;
+ alg->maxauthsize = min(halg->maxauthsize, salg->maxauthsize);
+ alg->chunksize = max(halg->chunksize, salg->chunksize);
+ alg->base.cra_ctxsize = sizeof(struct fallback_aead_ctx);
+ alg->init = fallback_aead_init;
+ alg->exit = fallback_aead_exit;
+ alg->setkey = fallback_aead_setkey;
+ alg->setauthsize = fallback_aead_setauthsize;
+ alg->encrypt = fallback_aead_encrypt;
+ alg->decrypt = fallback_aead_decrypt;
+
+ err = crypto_register_aead(alg);
+
+out_free_software:
+ crypto_free_aead(software);
+out_free_hardware:
+ crypto_free_aead(hardware);
+
+ return err;
+}
+
+static int fallback_register_ahash(struct fallback_alg *entry,
+ struct module *owner)
+{
+ struct crypto_ahash *hardware;
+ struct crypto_ahash *software;
+ struct ahash_alg *alg = &entry->alg.ahash;
+ struct hash_alg_common *hcommon;
+ struct hash_alg_common *scommon;
+ int err;
+
+ hardware = crypto_alloc_ahash(entry->driver_name, 0, 0);
+ if (IS_ERR(hardware))
+ return PTR_ERR(hardware);
+ hcommon = crypto_hash_alg_common(hardware);
+
+ software = crypto_alloc_ahash(hcommon->base.cra_name,
+ CRYPTO_ALG_REQ_VIRT,
+ CRYPTO_ALG_ASYNC | CRYPTO_ALG_REQ_VIRT |
+ CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(software)) {
+ err = PTR_ERR(software);
+ goto out_free_hardware;
+ }
+ scommon = crypto_hash_alg_common(software);
+
+ if (hcommon->base.cra_blocksize != scommon->base.cra_blocksize ||
+ hcommon->digestsize != scommon->digestsize) {
+ err = -EINVAL;
+ goto out_free_software;
+ }
+
+ err = fallback_alg_name(&alg->halg.base, hcommon->base.cra_name,
+ entry->driver_name);
+ if (err)
+ goto out_free_software;
+ fallback_init_base(&alg->halg.base, &hcommon->base, &scommon->base,
+ owner);
+ alg->halg.base.cra_flags |= CRYPTO_ALG_REQ_VIRT;
+ alg->halg.base.cra_flags |= hcommon->base.cra_flags &
+ CRYPTO_ALG_OPTIONAL_KEY;
+ alg->halg.digestsize = hcommon->digestsize;
+ alg->halg.statesize = offsetof(struct fallback_ahash_state, state) +
+ max(crypto_ahash_statesize(hardware),
+ crypto_ahash_statesize(software));
+ alg->halg.base.cra_ctxsize = sizeof(struct fallback_ahash_ctx);
+ alg->init_tfm = fallback_ahash_init;
+ alg->exit_tfm = fallback_ahash_exit;
+ alg->init = fallback_ahash_init_req;
+ alg->update = fallback_ahash_update;
+ alg->final = fallback_ahash_final;
+ alg->finup = fallback_ahash_finup;
+ alg->digest = fallback_ahash_digest;
+ alg->export = fallback_ahash_export;
+ alg->import = fallback_ahash_import;
+ if (crypto_hash_alg_has_setkey(hcommon))
+ alg->setkey = fallback_ahash_setkey;
+
+ err = crypto_register_ahash(alg);
+
+out_free_software:
+ crypto_free_ahash(software);
+out_free_hardware:
+ crypto_free_ahash(hardware);
+
+ return err;
+}
+
+static int fallback_register_alg(struct fallback_alg *alg, struct module *owner)
+{
+ switch (alg->type) {
+ case CRYPTO_FALLBACK_SKCIPHER:
+ return fallback_register_skcipher(alg, owner);
+ case CRYPTO_FALLBACK_AHASH:
+ return fallback_register_ahash(alg, owner);
+ case CRYPTO_FALLBACK_AEAD:
+ return fallback_register_aead(alg, owner);
+ }
+
+ return -EINVAL;
+}
+
+static void fallback_unregister_alg(struct fallback_alg *alg)
+{
+ switch (alg->type) {
+ case CRYPTO_FALLBACK_SKCIPHER:
+ crypto_unregister_skcipher(&alg->alg.skcipher);
+ break;
+ case CRYPTO_FALLBACK_AHASH:
+ crypto_unregister_ahash(&alg->alg.ahash);
+ break;
+ case CRYPTO_FALLBACK_AEAD:
+ crypto_unregister_aead(&alg->alg.aead);
+ break;
+ }
+}
+
+struct crypto_fallback *
+crypto_fallback_register(struct module *owner, struct device *dev,
+ const struct crypto_fallback_group *groups,
+ unsigned int num_groups)
+{
+ struct crypto_fallback *fallback;
+ unsigned int num_algs = 0;
+ unsigned int i, j;
+ int err;
+
+ if (!dev || !groups || !num_groups)
+ return ERR_PTR(-EINVAL);
+
+ for (i = 0; i < num_groups; i++) {
+ if (!groups[i].algs || !groups[i].num_algs)
+ return ERR_PTR(-EINVAL);
+ if (groups[i].num_algs > UINT_MAX - num_algs)
+ return ERR_PTR(-EOVERFLOW);
+ num_algs += groups[i].num_algs;
+ }
+
+ fallback = fallback_alloc(dev, groups, num_groups);
+ if (IS_ERR(fallback))
+ return fallback;
+
+ fallback->algs = kcalloc(num_algs, sizeof(*fallback->algs), GFP_KERNEL);
+ if (!fallback->algs) {
+ err = -ENOMEM;
+ goto err_release;
+ }
+ for (i = 0; i < num_groups; i++) {
+ const struct fallback_threshold *threshold;
+
+ threshold = &fallback->groups[i].threshold;
+ for (j = 0; j < groups[i].num_algs; j++) {
+ const char *driver_name = groups[i].algs[j];
+ struct fallback_alg *alg =
+ &fallback->algs[fallback->num_algs];
+
+ if (!driver_name) {
+ err = -EINVAL;
+ goto err_unregister_algs;
+ }
+
+ alg->type = groups[i].benchmark.type;
+ alg->threshold = threshold;
+ if (strscpy(alg->driver_name, driver_name,
+ sizeof(alg->driver_name)) < 0) {
+ err = -ENAMETOOLONG;
+ goto err_unregister_algs;
+ }
+
+ err = fallback_register_alg(alg, owner);
+ if (err == -ENOENT)
+ continue;
+ if (err)
+ goto err_unregister_algs;
+ fallback->num_algs++;
+ }
+ }
+
+ return fallback;
+
+err_unregister_algs:
+ while (fallback->num_algs)
+ fallback_unregister_alg(&fallback->algs[--fallback->num_algs]);
+err_release:
+ fallback_release(fallback);
+
+ return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(crypto_fallback_register);
+
+void crypto_fallback_unregister(struct crypto_fallback *fallback)
+{
+ if (!fallback || IS_ERR(fallback))
+ return;
+
+ while (fallback->num_algs)
+ fallback_unregister_alg(&fallback->algs[--fallback->num_algs]);
+ fallback_release(fallback);
+}
+EXPORT_SYMBOL_GPL(crypto_fallback_unregister);
+
+MODULE_AUTHOR("Jihong Min <hurryman2212@gmail.com>");
+MODULE_DESCRIPTION("Crypto API dynamic fallback support");
+MODULE_LICENSE("GPL");
diff --git a/include/crypto/fallback.h b/include/crypto/fallback.h
new file mode 100644
index 000000000000..6b6f35bb20e4
--- /dev/null
+++ b/include/crypto/fallback.h
@@ -0,0 +1,96 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _CRYPTO_FALLBACK_H
+#define _CRYPTO_FALLBACK_H
+
+#include <linux/stddef.h>
+#include <linux/types.h>
+
+struct crypto_fallback;
+struct device;
+struct module;
+
+/**
+ * enum crypto_fallback_alg_type - algorithm API used for benchmarking
+ * @CRYPTO_FALLBACK_SKCIPHER: symmetric key cipher
+ * @CRYPTO_FALLBACK_AHASH: asynchronous hash
+ * @CRYPTO_FALLBACK_AEAD: authenticated encryption
+ */
+enum crypto_fallback_alg_type {
+ CRYPTO_FALLBACK_SKCIPHER,
+ CRYPTO_FALLBACK_AHASH,
+ CRYPTO_FALLBACK_AEAD,
+};
+
+/**
+ * struct crypto_fallback_benchmark - benchmark algorithm description
+ * @type: algorithm API
+ * @name: generic algorithm name used to allocate software
+ * @driver_name: hardware implementation driver name
+ * @key: cipher or hash key
+ * @keylen: length of @key
+ * @authkey: authentication key for an authenc AEAD, or NULL
+ * @authkeylen: length of @authkey
+ * @authsize: AEAD authentication tag size
+ */
+struct crypto_fallback_benchmark {
+ enum crypto_fallback_alg_type type;
+ const char *name;
+ const char *driver_name;
+ const u8 *key;
+ unsigned int keylen;
+ const u8 *authkey;
+ unsigned int authkeylen;
+ unsigned int authsize;
+};
+
+/**
+ * struct crypto_fallback_group - algorithms sharing one fallback threshold
+ * @name: unique sysfs name for the group
+ * @benchmark: representative algorithm used to find the threshold
+ * @algs: hardware algorithms using the threshold
+ * @num_algs: number of algorithms in @algs
+ */
+struct crypto_fallback_group {
+ const char *name;
+ struct crypto_fallback_benchmark benchmark;
+ const char *const *algs;
+ unsigned int num_algs;
+};
+
+#define __CRYPTO_FALLBACK_GROUP(_name, _type, _alg_name, _driver, _key, \
+ _authkey, _authkeylen, _authsize, _algs, \
+ _num_algs) \
+ { \
+ .name = (_name), \
+ .benchmark = { \
+ .type = (_type), \
+ .name = (_alg_name), \
+ .driver_name = (_driver), \
+ .key = (_key), \
+ .keylen = sizeof(_key), \
+ .authkey = (_authkey), \
+ .authkeylen = (_authkeylen), \
+ .authsize = (_authsize), \
+ }, \
+ .algs = (_algs), \
+ .num_algs = (_num_algs), \
+ }
+
+#define CRYPTO_FALLBACK_GROUP(_name, _type, _alg_name, _driver, _key, _algs, \
+ _num_algs) \
+ __CRYPTO_FALLBACK_GROUP(_name, _type, _alg_name, _driver, _key, NULL, \
+ 0, 0, _algs, _num_algs)
+
+#define CRYPTO_FALLBACK_GROUP_AUTHENC(_name, _alg_name, _driver, _key, \
+ _authkey, _authsize, _algs, _num_algs) \
+ __CRYPTO_FALLBACK_GROUP(_name, CRYPTO_FALLBACK_AEAD, _alg_name, \
+ _driver, _key, _authkey, sizeof(_authkey), \
+ _authsize, _algs, _num_algs)
+
+struct crypto_fallback *
+crypto_fallback_register(struct module *owner, struct device *dev,
+ const struct crypto_fallback_group *groups,
+ unsigned int num_groups);
+void crypto_fallback_unregister(struct crypto_fallback *fallback);
+
+#endif /* _CRYPTO_FALLBACK_H */
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/4] crypto: eip93 - add dynamic software fallback support
2026-07-28 8:43 [PATCH 0/4] crypto: introduce generic dynamic software fallback and EIP93 support Jihong Min
2026-07-28 8:43 ` [PATCH 1/4] crypto: move cycle benchmark helpers out of tcrypt Jihong Min
2026-07-28 8:43 ` [PATCH 2/4] crypto: introduce dynamic software fallback Jihong Min
@ 2026-07-28 8:43 ` Jihong Min
2026-07-28 8:43 ` [PATCH 4/4] crypto: eliminate fallback proxy overhead while disabled Jihong Min
3 siblings, 0 replies; 5+ messages in thread
From: Jihong Min @ 2026-07-28 8:43 UTC (permalink / raw)
To: Herbert Xu, Christian Marangi, linux-crypto
Cc: Jihong Min, David S. Miller, Antoine Tenart, linux-kernel
Group EIP93 skcipher, ahash, and authenc implementations by their
request-size performance trends. Register generic fallback proxies that
use one representative benchmark per group and dispatch each request to
EIP93 or software according to the per-device threshold.
Expose the fallback controls through the EIP93 device sysfs hierarchy
and add the software algorithm dependencies required for benchmarking.
Unregister the fallback proxies before removing the hardware providers.
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
drivers/crypto/inside-secure/eip93/Kconfig | 7 +
drivers/crypto/inside-secure/eip93/Makefile | 1 +
.../inside-secure/eip93/eip93-fallback.c | 164 ++++++++++++++++++
.../inside-secure/eip93/eip93-fallback.h | 10 ++
.../crypto/inside-secure/eip93/eip93-main.c | 19 +-
5 files changed, 197 insertions(+), 4 deletions(-)
create mode 100644 drivers/crypto/inside-secure/eip93/eip93-fallback.c
create mode 100644 drivers/crypto/inside-secure/eip93/eip93-fallback.h
diff --git a/drivers/crypto/inside-secure/eip93/Kconfig b/drivers/crypto/inside-secure/eip93/Kconfig
index 29523f6927dd..ea8c417653d4 100644
--- a/drivers/crypto/inside-secure/eip93/Kconfig
+++ b/drivers/crypto/inside-secure/eip93/Kconfig
@@ -10,6 +10,13 @@ config CRYPTO_DEV_EIP93
select CRYPTO_MD5
select CRYPTO_SHA1
select CRYPTO_SHA256
+ select CRYPTO_DYNAMIC_FALLBACK
+ select CRYPTO_AES
+ select CRYPTO_CBC
+ select CRYPTO_CTR
+ select CRYPTO_DES
+ select CRYPTO_ECB
+ select CRYPTO_HMAC
help
EIP93 have various crypto HW accelerators. Select this if
you want to use the EIP93 modules for any of the crypto algorithms.
diff --git a/drivers/crypto/inside-secure/eip93/Makefile b/drivers/crypto/inside-secure/eip93/Makefile
index a3d3d3677cdc..b408d509bdd7 100644
--- a/drivers/crypto/inside-secure/eip93/Makefile
+++ b/drivers/crypto/inside-secure/eip93/Makefile
@@ -3,3 +3,4 @@ obj-$(CONFIG_CRYPTO_DEV_EIP93) += crypto-hw-eip93.o
crypto-hw-eip93-y += eip93-main.o eip93-common.o
crypto-hw-eip93-y += eip93-cipher.o eip93-aead.o
crypto-hw-eip93-y += eip93-hash.o
+crypto-hw-eip93-y += eip93-fallback.o
diff --git a/drivers/crypto/inside-secure/eip93/eip93-fallback.c b/drivers/crypto/inside-secure/eip93/eip93-fallback.c
new file mode 100644
index 000000000000..78e4394b7e9e
--- /dev/null
+++ b/drivers/crypto/inside-secure/eip93/eip93-fallback.c
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026 Jihong Min <hurryman2212@gmail.com>
+ */
+
+#include <crypto/fallback.h>
+#include <crypto/md5.h>
+#include <linux/err.h>
+#include <linux/module.h>
+
+#include "eip93-fallback.h"
+#include "eip93-main.h"
+
+/*
+ * Fixed, non-secret keys used only for benchmark setup. The RFC 3686 key
+ * contains a 16-byte AES-128 key followed by its required 4-byte nonce.
+ */
+
+static const u8 eip93_fallback_aes_key[] = {
+ 0x2b, 0x7e, 0x15, 0x16, 0x28, 0xae, 0xd2, 0xa6, 0xab, 0xf7,
+ 0x15, 0x88, 0x09, 0xcf, 0x4f, 0x3c, 0x00, 0x00, 0x00, 0x01,
+};
+
+static const u8 eip93_fallback_des_key[] = {
+ 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
+};
+
+static const u8 eip93_fallback_3des_key[] = {
+ 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x23, 0x45, 0x67, 0x89,
+ 0xab, 0xcd, 0xef, 0x01, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23,
+};
+
+static const u8 eip93_fallback_auth_key[] = {
+ 0x0b, 0x0b, 0x0b, 0x0b, 0x0b, 0x0b, 0x0b, 0x0b,
+ 0x0b, 0x0b, 0x0b, 0x0b, 0x0b, 0x0b, 0x0b, 0x0b,
+};
+
+/* Group EIP93 algorithms by their observed request-size performance trends. */
+
+static const char *const eip93_fallback_raw_aes[] = {
+ "ecb(aes-eip93)",
+ "cbc(aes-eip93)",
+ "ctr(aes-eip93)",
+ "rfc3686(ctr(aes-eip93))",
+};
+
+static const char *const eip93_fallback_raw_des[] = {
+ "ecb(des-eip93)",
+ "cbc(des-eip93)",
+};
+
+static const char *const eip93_fallback_raw_3des[] = {
+ "ecb(des3_ede-eip93)",
+ "cbc(des3_ede-eip93)",
+};
+
+static const char *const eip93_fallback_hash_md5[] = {
+ "md5-eip93",
+ "hmac(md5-eip93)",
+};
+
+static const char *const eip93_fallback_hash_sha1[] = {
+ "sha1-eip93",
+ "hmac(sha1-eip93)",
+};
+
+static const char *const eip93_fallback_hash_sha2[] = {
+ "sha224-eip93",
+ "sha256-eip93",
+ "hmac(sha224-eip93)",
+ "hmac(sha256-eip93)",
+};
+
+static const char *const eip93_fallback_authenc_aes[] = {
+ "authenc(hmac(md5-eip93), cbc(aes-eip93))",
+ "authenc(hmac(sha1-eip93),cbc(aes-eip93))",
+ "authenc(hmac(sha224-eip93),cbc(aes-eip93))",
+ "authenc(hmac(sha256-eip93),cbc(aes-eip93))",
+ "authenc(hmac(md5-eip93),rfc3686(ctr(aes-eip93)))",
+ "authenc(hmac(sha1-eip93),rfc3686(ctr(aes-eip93)))",
+ "authenc(hmac(sha224-eip93),rfc3686(ctr(aes-eip93)))",
+ "authenc(hmac(sha256-eip93),rfc3686(ctr(aes-eip93)))",
+};
+
+static const char *const eip93_fallback_authenc_des[] = {
+ "authenc(hmac(md5-eip93),cbc(des-eip93))",
+ "authenc(hmac(sha1-eip93),cbc(des-eip93))",
+ "authenc(hmac(sha224-eip93),cbc(des-eip93))",
+ "authenc(hmac(sha256-eip93),cbc(des-eip93))",
+};
+
+static const char *const eip93_fallback_authenc_3des[] = {
+ "authenc(hmac(md5-eip93),cbc(des3_ede-eip93))",
+ "authenc(hmac(sha1-eip93),cbc(des3_ede-eip93))",
+ "authenc(hmac(sha224-eip93),cbc(des3_ede-eip93))",
+ "authenc(hmac(sha256-eip93),cbc(des3_ede-eip93))",
+};
+
+static const struct crypto_fallback_group eip93_fallback_groups[] = {
+ CRYPTO_FALLBACK_GROUP("raw_aes", CRYPTO_FALLBACK_SKCIPHER,
+ "rfc3686(ctr(aes))", "rfc3686(ctr(aes-eip93))",
+ eip93_fallback_aes_key, eip93_fallback_raw_aes,
+ ARRAY_SIZE(eip93_fallback_raw_aes)),
+ CRYPTO_FALLBACK_GROUP("raw_des", CRYPTO_FALLBACK_SKCIPHER, "ecb(des)",
+ "ecb(des-eip93)", eip93_fallback_des_key,
+ eip93_fallback_raw_des,
+ ARRAY_SIZE(eip93_fallback_raw_des)),
+ CRYPTO_FALLBACK_GROUP("raw_3des", CRYPTO_FALLBACK_SKCIPHER,
+ "cbc(des3_ede)", "cbc(des3_ede-eip93)",
+ eip93_fallback_3des_key, eip93_fallback_raw_3des,
+ ARRAY_SIZE(eip93_fallback_raw_3des)),
+ CRYPTO_FALLBACK_GROUP("hash_md5", CRYPTO_FALLBACK_AHASH, "hmac(md5)",
+ "hmac(md5-eip93)", eip93_fallback_auth_key,
+ eip93_fallback_hash_md5,
+ ARRAY_SIZE(eip93_fallback_hash_md5)),
+ CRYPTO_FALLBACK_GROUP("hash_sha1", CRYPTO_FALLBACK_AHASH, "hmac(sha1)",
+ "hmac(sha1-eip93)", eip93_fallback_auth_key,
+ eip93_fallback_hash_sha1,
+ ARRAY_SIZE(eip93_fallback_hash_sha1)),
+ CRYPTO_FALLBACK_GROUP("hash_sha2", CRYPTO_FALLBACK_AHASH,
+ "hmac(sha256)", "hmac(sha256-eip93)",
+ eip93_fallback_auth_key, eip93_fallback_hash_sha2,
+ ARRAY_SIZE(eip93_fallback_hash_sha2)),
+ CRYPTO_FALLBACK_GROUP_AUTHENC("authenc_aes",
+ "authenc(hmac(md5),rfc3686(ctr(aes)))",
+ "authenc(hmac(md5-eip93),rfc3686(ctr(aes-eip93)))",
+ eip93_fallback_aes_key,
+ eip93_fallback_auth_key, MD5_DIGEST_SIZE,
+ eip93_fallback_authenc_aes,
+ ARRAY_SIZE(eip93_fallback_authenc_aes)),
+ CRYPTO_FALLBACK_GROUP_AUTHENC("authenc_des",
+ "authenc(hmac(md5),cbc(des))",
+ "authenc(hmac(md5-eip93),cbc(des-eip93))",
+ eip93_fallback_des_key,
+ eip93_fallback_auth_key, MD5_DIGEST_SIZE,
+ eip93_fallback_authenc_des,
+ ARRAY_SIZE(eip93_fallback_authenc_des)),
+ CRYPTO_FALLBACK_GROUP_AUTHENC("authenc_3des",
+ "authenc(hmac(md5),cbc(des3_ede))",
+ "authenc(hmac(md5-eip93),cbc(des3_ede-eip93))",
+ eip93_fallback_3des_key,
+ eip93_fallback_auth_key, MD5_DIGEST_SIZE,
+ eip93_fallback_authenc_3des,
+ ARRAY_SIZE(eip93_fallback_authenc_3des)),
+};
+
+static struct crypto_fallback *eip93_fallback;
+
+int eip93_fallback_register(struct eip93_device *eip93)
+{
+ eip93_fallback =
+ crypto_fallback_register(THIS_MODULE, eip93->dev,
+ eip93_fallback_groups,
+ ARRAY_SIZE(eip93_fallback_groups));
+
+ return PTR_ERR_OR_ZERO(eip93_fallback);
+}
+
+void eip93_fallback_unregister(void)
+{
+ crypto_fallback_unregister(eip93_fallback);
+
+ eip93_fallback = NULL;
+}
diff --git a/drivers/crypto/inside-secure/eip93/eip93-fallback.h b/drivers/crypto/inside-secure/eip93/eip93-fallback.h
new file mode 100644
index 000000000000..e3ba9a66abc7
--- /dev/null
+++ b/drivers/crypto/inside-secure/eip93/eip93-fallback.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _EIP93_FALLBACK_H_
+#define _EIP93_FALLBACK_H_
+
+struct eip93_device;
+
+int eip93_fallback_register(struct eip93_device *eip93);
+void eip93_fallback_unregister(void);
+
+#endif /* _EIP93_FALLBACK_H_ */
diff --git a/drivers/crypto/inside-secure/eip93/eip93-main.c b/drivers/crypto/inside-secure/eip93/eip93-main.c
index 1a8dabc4ada4..96ecffeb75df 100644
--- a/drivers/crypto/inside-secure/eip93/eip93-main.c
+++ b/drivers/crypto/inside-secure/eip93/eip93-main.c
@@ -25,6 +25,7 @@
#include "eip93-aes.h"
#include "eip93-des.h"
#include "eip93-aead.h"
+#include "eip93-fallback.h"
#include "eip93-hash.h"
static struct eip93_alg_template *eip93_algs[] = {
@@ -460,10 +461,12 @@ static int eip93_crypto_probe(struct platform_device *pdev)
eip93_irq_enable(eip93, EIP93_INT_RDR_THRESH);
ret = eip93_register_algs(eip93, algo_flags);
- if (ret) {
- eip93_cleanup(eip93);
- return ret;
- }
+ if (ret)
+ goto err_cleanup;
+
+ ret = eip93_fallback_register(eip93);
+ if (ret)
+ goto err_unregister_algs;
ver = readl(eip93->base + EIP93_REG_PE_REVISION);
/* EIP_EIP_NO:MAJOR_HW_REV:MINOR_HW_REV:HW_PATCH,PE(ALGO_FLAGS) */
@@ -476,6 +479,13 @@ static int eip93_crypto_probe(struct platform_device *pdev)
readl(eip93->base + EIP93_REG_PE_OPTION_0));
return 0;
+
+err_unregister_algs:
+ eip93_unregister_algs(algo_flags, ARRAY_SIZE(eip93_algs));
+err_cleanup:
+ eip93_cleanup(eip93);
+
+ return ret;
}
static void eip93_crypto_remove(struct platform_device *pdev)
@@ -485,6 +495,7 @@ static void eip93_crypto_remove(struct platform_device *pdev)
algo_flags = readl(eip93->base + EIP93_REG_PE_OPTION_1);
+ eip93_fallback_unregister();
eip93_unregister_algs(algo_flags, ARRAY_SIZE(eip93_algs));
eip93_cleanup(eip93);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 4/4] crypto: eliminate fallback proxy overhead while disabled
2026-07-28 8:43 [PATCH 0/4] crypto: introduce generic dynamic software fallback and EIP93 support Jihong Min
` (2 preceding siblings ...)
2026-07-28 8:43 ` [PATCH 3/4] crypto: eip93 - add dynamic software fallback support Jihong Min
@ 2026-07-28 8:43 ` Jihong Min
3 siblings, 0 replies; 5+ messages in thread
From: Jihong Min @ 2026-07-28 8:43 UTC (permalink / raw)
To: Herbert Xu, Christian Marangi, linux-crypto
Cc: Jihong Min, David S. Miller, Antoine Tenart, linux-kernel
Register fallback proxy providers only when the per-device control is
enabled. Unregister them again on disable so newly allocated transforms
select the hardware provider directly.
Allocate proxy algorithms per enable cycle and retain retired
generations until existing transforms release them. This keeps repeated
runtime enable and disable transitions safe while forcing retired
proxies to use hardware after disable.
The Crypto API cannot migrate an already allocated transform between
providers. A transform created while disabled remains bound to its
native hardware provider after enable. A transform created while
enabled keeps its proxy after disable until it is released, with its
threshold forced to hardware. Newly allocated transforms follow the
current enabled state.
Assisted-by: Codex:gpt-5.6
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
crypto/fallback.c | 284 ++++++++++++++++++++++++++++++++++------------
1 file changed, 214 insertions(+), 70 deletions(-)
diff --git a/crypto/fallback.c b/crypto/fallback.c
index af8830f1077c..40755d409e7a 100644
--- a/crypto/fallback.c
+++ b/crypto/fallback.c
@@ -9,13 +9,16 @@
#include <linux/device.h>
#include <linux/err.h>
#include <linux/kernel.h>
+#include <linux/kref.h>
#include <linux/limits.h>
+#include <linux/list.h>
#include <linux/module.h>
#include <linux/mutex.h>
#include <linux/random.h>
#include <linux/rtnetlink.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
+#include <linux/spinlock.h>
#include <linux/string.h>
#include <linux/sysfs.h>
@@ -48,10 +51,12 @@ struct crypto_fallback {
struct attribute_group sysfs_group;
struct attribute **sysfs_attrs;
struct device_attribute enabled_attr;
+ struct kref refcount;
struct mutex lock; /* Serializes tuning and sysfs writes. */
- struct fallback_alg *algs;
+ spinlock_t alg_lock; /* Protects algs and fallback_alg state. */
+ struct list_head algs;
+ struct module *owner;
unsigned int num_groups;
- unsigned int num_algs;
bool enabled;
};
@@ -422,6 +427,9 @@ static int crypto_fallback_tune_group(const struct crypto_fallback_group *group,
return 0;
}
+static int fallback_register_algs(struct crypto_fallback *fallback);
+static void fallback_unregister_algs(struct crypto_fallback *fallback);
+
static ssize_t fallback_enabled_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -451,9 +459,6 @@ static ssize_t fallback_enabled_store(struct device *dev,
goto out_unlock;
if (enabled) {
- for (i = 0; i < fallback->num_groups; i++)
- WRITE_ONCE(fallback->groups[i].threshold.value, 0);
-
for (i = 0; i < fallback->num_groups; i++) {
struct crypto_fallback_group_state *state;
int threshold;
@@ -463,9 +468,10 @@ static ssize_t fallback_enabled_store(struct device *dev,
&threshold);
if (err) {
dev_warn(fallback->dev,
- "%s benchmark failed: %d; using hardware\n",
- state->group.name, err);
+ "%s benchmark failed: %d; using hardware\n",
+ state->group.name, err);
threshold = 0;
+ err = 0;
}
WRITE_ONCE(state->value, threshold);
dev_info(fallback->dev,
@@ -476,28 +482,39 @@ static ssize_t fallback_enabled_store(struct device *dev,
for (i = 0; i < fallback->num_groups; i++)
WRITE_ONCE(fallback->groups[i].threshold.value,
fallback->groups[i].value);
+
+ if (!fallback->enabled) {
+ err = fallback_register_algs(fallback);
+ if (err) {
+ for (i = 0; i < fallback->num_groups; i++)
+ WRITE_ONCE(fallback->groups[i].threshold.value,
+ 0);
+ goto out_unlock;
+ }
+ }
} else {
+ WRITE_ONCE(fallback->enabled, false);
for (i = 0; i < fallback->num_groups; i++)
WRITE_ONCE(fallback->groups[i].threshold.value, 0);
+ fallback_unregister_algs(fallback);
}
WRITE_ONCE(fallback->enabled, enabled);
out_unlock:
mutex_unlock(&fallback->lock);
- return count;
+ return err ?: count;
}
-static void crypto_fallback_free(struct crypto_fallback *fallback)
+static void crypto_fallback_free(struct kref *ref)
{
+ struct crypto_fallback *fallback;
unsigned int i;
- if (!fallback)
- return;
+ fallback = container_of(ref, struct crypto_fallback, refcount);
for (i = 0; i < fallback->num_groups; i++)
kfree_const(fallback->groups[i].group.name);
- kfree(fallback->algs);
kfree(fallback->sysfs_attrs);
kfree(fallback->groups);
mutex_destroy(&fallback->lock);
@@ -506,7 +523,8 @@ static void crypto_fallback_free(struct crypto_fallback *fallback)
}
static struct crypto_fallback *
-fallback_alloc(struct device *dev, const struct crypto_fallback_group *groups,
+fallback_alloc(struct module *owner, struct device *dev,
+ const struct crypto_fallback_group *groups,
unsigned int num_groups)
{
struct crypto_fallback *fallback;
@@ -520,8 +538,12 @@ fallback_alloc(struct device *dev, const struct crypto_fallback_group *groups,
if (!fallback)
return ERR_PTR(-ENOMEM);
+ kref_init(&fallback->refcount);
mutex_init(&fallback->lock);
+ spin_lock_init(&fallback->alg_lock);
+ INIT_LIST_HEAD(&fallback->algs);
fallback->dev = get_device(dev);
+ fallback->owner = owner;
fallback->num_groups = num_groups;
fallback->groups =
kcalloc(num_groups, sizeof(*fallback->groups), GFP_KERNEL);
@@ -583,20 +605,11 @@ fallback_alloc(struct device *dev, const struct crypto_fallback_group *groups,
return fallback;
err_free:
- crypto_fallback_free(fallback);
+ kref_put(&fallback->refcount, crypto_fallback_free);
return ERR_PTR(err);
}
-static void fallback_release(struct crypto_fallback *fallback)
-{
- if (!fallback || IS_ERR(fallback))
- return;
-
- sysfs_remove_group(&fallback->dev->kobj, &fallback->sysfs_group);
- crypto_fallback_free(fallback);
-}
-
static bool fallback_use_software(const struct fallback_threshold *threshold,
unsigned int input_size)
{
@@ -630,10 +643,20 @@ struct fallback_ahash_state {
u8 state[] CRYPTO_MINALIGN_ATTR;
};
+/*
+ * Allocate proxy algorithms per enable cycle because their destruction can be
+ * deferred until transforms allocated before disable release them.
+ */
struct fallback_alg {
+ struct list_head list;
+ struct crypto_fallback *fallback;
+ struct kref refcount;
const struct fallback_threshold *threshold;
char driver_name[CRYPTO_MAX_ALG_NAME];
enum crypto_fallback_alg_type type;
+ bool registration_complete;
+ bool registered;
+ bool dead;
union {
struct skcipher_alg skcipher;
struct aead_alg aead;
@@ -641,6 +664,87 @@ struct fallback_alg {
} alg;
};
+static void fallback_alg_release(struct kref *ref)
+{
+ struct fallback_alg *alg;
+ struct crypto_fallback *fallback;
+
+ alg = container_of(ref, struct fallback_alg, refcount);
+ fallback = alg->fallback;
+ kfree(alg);
+ kref_put(&fallback->refcount, crypto_fallback_free);
+}
+
+static void fallback_alg_destroy(struct fallback_alg *alg)
+{
+ struct crypto_fallback *fallback = alg->fallback;
+ unsigned long flags;
+ bool release = false;
+
+ spin_lock_irqsave(&fallback->alg_lock, flags);
+ alg->dead = true;
+ alg->registered = false;
+ if (alg->registration_complete) {
+ list_del_init(&alg->list);
+ release = true;
+ }
+ spin_unlock_irqrestore(&fallback->alg_lock, flags);
+
+ if (release)
+ kref_put(&alg->refcount, fallback_alg_release);
+}
+
+static void fallback_skcipher_destroy(struct crypto_alg *base)
+{
+ struct skcipher_alg *alg;
+
+ alg = container_of(base, struct skcipher_alg, base);
+ fallback_alg_destroy(container_of(alg, struct fallback_alg,
+ alg.skcipher));
+}
+
+static void fallback_aead_destroy(struct crypto_alg *base)
+{
+ struct aead_alg *alg;
+
+ alg = container_of(base, struct aead_alg, base);
+ fallback_alg_destroy(container_of(alg, struct fallback_alg, alg.aead));
+}
+
+static void fallback_ahash_destroy(struct crypto_alg *base)
+{
+ struct hash_alg_common *halg;
+ struct ahash_alg *alg;
+
+ halg = container_of(base, struct hash_alg_common, base);
+ alg = container_of(halg, struct ahash_alg, halg);
+ fallback_alg_destroy(container_of(alg, struct fallback_alg, alg.ahash));
+}
+
+static int fallback_alg_complete_registration(struct fallback_alg *alg, int err)
+{
+ struct crypto_fallback *fallback = alg->fallback;
+ unsigned long flags;
+ bool release = false;
+
+ spin_lock_irqsave(&fallback->alg_lock, flags);
+ alg->registration_complete = true;
+ if (err || alg->dead) {
+ list_del_init(&alg->list);
+ release = true;
+ if (!err)
+ err = -ENODEV;
+ } else {
+ alg->registered = true;
+ }
+ spin_unlock_irqrestore(&fallback->alg_lock, flags);
+
+ if (release)
+ kref_put(&alg->refcount, fallback_alg_release);
+
+ return err;
+}
+
static struct fallback_alg *fallback_skcipher_alg(struct crypto_skcipher *tfm)
{
return container_of(crypto_skcipher_alg(tfm), struct fallback_alg,
@@ -1068,6 +1172,7 @@ static int fallback_register_skcipher(struct fallback_alg *entry,
if (err)
goto out_free_software;
fallback_init_base(&alg->base, &hcommon->base, &scommon->base, owner);
+ alg->base.cra_destroy = fallback_skcipher_destroy;
alg->min_keysize = max(hcommon->min_keysize, scommon->min_keysize);
alg->max_keysize = min(hcommon->max_keysize, scommon->max_keysize);
if (alg->min_keysize > alg->max_keysize) {
@@ -1129,6 +1234,7 @@ static int fallback_register_aead(struct fallback_alg *entry,
if (err)
goto out_free_software;
fallback_init_base(&alg->base, &halg->base, &salg->base, owner);
+ alg->base.cra_destroy = fallback_aead_destroy;
alg->ivsize = halg->ivsize;
alg->maxauthsize = min(halg->maxauthsize, salg->maxauthsize);
alg->chunksize = max(halg->chunksize, salg->chunksize);
@@ -1187,6 +1293,7 @@ static int fallback_register_ahash(struct fallback_alg *entry,
goto out_free_software;
fallback_init_base(&alg->halg.base, &hcommon->base, &scommon->base,
owner);
+ alg->halg.base.cra_destroy = fallback_ahash_destroy;
alg->halg.base.cra_flags |= CRYPTO_ALG_REQ_VIRT;
alg->halg.base.cra_flags |= hcommon->base.cra_flags &
CRYPTO_ALG_OPTIONAL_KEY;
@@ -1246,87 +1353,124 @@ static void fallback_unregister_alg(struct fallback_alg *alg)
}
}
-struct crypto_fallback *
-crypto_fallback_register(struct module *owner, struct device *dev,
- const struct crypto_fallback_group *groups,
- unsigned int num_groups)
+static void fallback_unregister_algs(struct crypto_fallback *fallback)
{
- struct crypto_fallback *fallback;
- unsigned int num_algs = 0;
- unsigned int i, j;
- int err;
+ struct fallback_alg *alg;
+ struct fallback_alg *candidate;
+ unsigned long flags;
+
+ for (;;) {
+ alg = NULL;
+ spin_lock_irqsave(&fallback->alg_lock, flags);
+ list_for_each_entry(candidate, &fallback->algs, list) {
+ if (!candidate->registered)
+ continue;
- if (!dev || !groups || !num_groups)
- return ERR_PTR(-EINVAL);
+ alg = candidate;
+ alg->registered = false;
+ kref_get(&alg->refcount);
+ break;
+ }
+ spin_unlock_irqrestore(&fallback->alg_lock, flags);
- for (i = 0; i < num_groups; i++) {
- if (!groups[i].algs || !groups[i].num_algs)
- return ERR_PTR(-EINVAL);
- if (groups[i].num_algs > UINT_MAX - num_algs)
- return ERR_PTR(-EOVERFLOW);
- num_algs += groups[i].num_algs;
+ if (!alg)
+ break;
+
+ fallback_unregister_alg(alg);
+ kref_put(&alg->refcount, fallback_alg_release);
}
+}
- fallback = fallback_alloc(dev, groups, num_groups);
- if (IS_ERR(fallback))
- return fallback;
+static int fallback_register_algs(struct crypto_fallback *fallback)
+{
+ unsigned int i, j;
+ int err;
- fallback->algs = kcalloc(num_algs, sizeof(*fallback->algs), GFP_KERNEL);
- if (!fallback->algs) {
- err = -ENOMEM;
- goto err_release;
- }
- for (i = 0; i < num_groups; i++) {
- const struct fallback_threshold *threshold;
+ for (i = 0; i < fallback->num_groups; i++) {
+ const struct crypto_fallback_group *group;
- threshold = &fallback->groups[i].threshold;
- for (j = 0; j < groups[i].num_algs; j++) {
- const char *driver_name = groups[i].algs[j];
- struct fallback_alg *alg =
- &fallback->algs[fallback->num_algs];
+ group = &fallback->groups[i].group;
+ for (j = 0; j < group->num_algs; j++) {
+ struct fallback_alg *alg;
+ unsigned long flags;
- if (!driver_name) {
- err = -EINVAL;
+ alg = kzalloc_obj(*alg, GFP_KERNEL);
+ if (!alg) {
+ err = -ENOMEM;
goto err_unregister_algs;
}
- alg->type = groups[i].benchmark.type;
- alg->threshold = threshold;
- if (strscpy(alg->driver_name, driver_name,
+ INIT_LIST_HEAD(&alg->list);
+ kref_init(&alg->refcount);
+ alg->fallback = fallback;
+ alg->threshold = &fallback->groups[i].threshold;
+ alg->type = group->benchmark.type;
+ if (strscpy(alg->driver_name, group->algs[j],
sizeof(alg->driver_name)) < 0) {
+ kfree(alg);
err = -ENAMETOOLONG;
goto err_unregister_algs;
}
- err = fallback_register_alg(alg, owner);
+ kref_get(&fallback->refcount);
+ spin_lock_irqsave(&fallback->alg_lock, flags);
+ list_add_tail(&alg->list, &fallback->algs);
+ spin_unlock_irqrestore(&fallback->alg_lock, flags);
+
+ err = fallback_register_alg(alg, fallback->owner);
+ err = fallback_alg_complete_registration(alg, err);
if (err == -ENOENT)
continue;
if (err)
goto err_unregister_algs;
- fallback->num_algs++;
}
}
- return fallback;
+ return 0;
err_unregister_algs:
- while (fallback->num_algs)
- fallback_unregister_alg(&fallback->algs[--fallback->num_algs]);
-err_release:
- fallback_release(fallback);
+ fallback_unregister_algs(fallback);
- return ERR_PTR(err);
+ return err;
+}
+
+struct crypto_fallback *
+crypto_fallback_register(struct module *owner, struct device *dev,
+ const struct crypto_fallback_group *groups,
+ unsigned int num_groups)
+{
+ unsigned int i, j;
+
+ if (!owner || !dev || !groups || !num_groups)
+ return ERR_PTR(-EINVAL);
+
+ for (i = 0; i < num_groups; i++) {
+ if (!groups[i].algs || !groups[i].num_algs)
+ return ERR_PTR(-EINVAL);
+ for (j = 0; j < groups[i].num_algs; j++)
+ if (!groups[i].algs[j])
+ return ERR_PTR(-EINVAL);
+ }
+
+ return fallback_alloc(owner, dev, groups, num_groups);
}
EXPORT_SYMBOL_GPL(crypto_fallback_register);
void crypto_fallback_unregister(struct crypto_fallback *fallback)
{
+ unsigned int i;
+
if (!fallback || IS_ERR(fallback))
return;
- while (fallback->num_algs)
- fallback_unregister_alg(&fallback->algs[--fallback->num_algs]);
- fallback_release(fallback);
+ sysfs_remove_group(&fallback->dev->kobj, &fallback->sysfs_group);
+ mutex_lock(&fallback->lock);
+ WRITE_ONCE(fallback->enabled, false);
+ for (i = 0; i < fallback->num_groups; i++)
+ WRITE_ONCE(fallback->groups[i].threshold.value, 0);
+ fallback_unregister_algs(fallback);
+ mutex_unlock(&fallback->lock);
+ kref_put(&fallback->refcount, crypto_fallback_free);
}
EXPORT_SYMBOL_GPL(crypto_fallback_unregister);
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-07-28 8:44 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-28 8:43 [PATCH 0/4] crypto: introduce generic dynamic software fallback and EIP93 support Jihong Min
2026-07-28 8:43 ` [PATCH 1/4] crypto: move cycle benchmark helpers out of tcrypt Jihong Min
2026-07-28 8:43 ` [PATCH 2/4] crypto: introduce dynamic software fallback Jihong Min
2026-07-28 8:43 ` [PATCH 3/4] crypto: eip93 - add dynamic software fallback support Jihong Min
2026-07-28 8:43 ` [PATCH 4/4] crypto: eliminate fallback proxy overhead while disabled Jihong Min
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox