From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C4DECA5FA3 for ; Sun, 27 Sep 2026 22:45:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MNqzsYQBrsKFOIrvMvHOE8qz4hZBv6/HSc2rwJfO1KY=; b=CGuKnoBTHo5hDx KaR4EwhQ+SVqfuw2h2TVbFNiSDfNiy+P6eYwaoF3ViT68KiNmy7RIT9EkwT1eTYAoB5W/W5OUP2/L Rchx40Sk0I/EwVII55EB68FHagPpqoDxy3+uob5ax28/nLM2I8+8fgyx/Z/y+/PflOBhAUo/oz1WD D5Cq6f3u7d99Og7Fsh+f3eu1Vrpi5kQFN8HJAsPjyg3+4RvYza6kSG+7iIRKVGz5hWudTYut6yDpO JRITFgWnXWUD1RvlEwzk7VijDnttHojDL0fC1jyiN4UT7Hw//G2PyLevrZvjQk8n5AjfH3cNJkHim OG/ThLQ4/OQOPj2G1PDg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1xAxcI-0000000Gwjl-3FFB; Sun, 27 Sep 2026 22:45:06 +0000 Received: from tor.source.kernel.org ([2600:3c04:e001:324:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1xAxbx-0000000GwSh-3B3w for linux-riscv@lists.infradead.org; Sun, 27 Sep 2026 22:44:45 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by tor.source.kernel.org (Postfix) with ESMTP id 9532860D79; Sun, 27 Sep 2026 22:44:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 146251F0089F; Sun, 27 Sep 2026 22:44:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1790549085; bh=lphM/Brm+7rxF8TBpgy6Mr3aozf2S07ib6PKnHYEgDA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=AwLMsfkHBrX+u6qMIMwq1y2pA8VHzNx7/jsfcXLCnQWUo2DcalGu+L8kciTBlOk76 VbQvFY1c9C7k0CJZd4g23XcvLpyd22dPDVCC6tNjviqT5cTXA1qS7uUYPlhIcxk4EG 1nzMCh7PS9/GOZ0Acg+HF4Ah2bFF54Jh8/7tQo3YxMfMCPAXue92F9FOiKLqAVD45N 11neXV/FHpb2J6NlSMgb3A737LR5CHZ0OZnjngMHvH7hCRYcqV1p1SnPifONhZHB8g X4s+XQh+/uLYLBs1MylACwtqscXNGaMamOiJ0E08q3Gui+27EcarBiABWPJxtJ9SbH xszVb+6OBnDew== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH v2 17/20] lib/crypto: riscv/aes-ecb: Migrate optimized code into library Date: Sun, 27 Sep 2026 15:43:08 -0700 Message-ID: <20260927224418.109759-18-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260927224418.109759-1-ebiggers@kernel.org> References: <20260927224418.109759-1-ebiggers@kernel.org> MIME-Version: 1.0 X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Instead of exposing the riscv-optimized AES-ECB code via a riscv-specific crypto_skcipher algorithm, just implement the AES-ECB library functions. This is simpler, it makes the AES-ECB library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-ECB code was disabled by default. AES-ECB support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it. To match what the library expects, update the assembly functions to operate on struct aes_enckey or struct aes_key rather than struct crypto_aes_ctx, and adjust the argument order. Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/. Signed-off-by: Eric Biggers --- arch/riscv/crypto/Kconfig | 4 +- arch/riscv/crypto/aes-riscv64-glue.c | 61 +------------------------- arch/riscv/crypto/aes-riscv64-zvkned.S | 39 ---------------- crypto/aes.c | 3 +- lib/crypto/riscv/aes-riscv64-zvkned.S | 51 +++++++++++++++++++++ lib/crypto/riscv/aes.h | 33 ++++++++++++++ 6 files changed, 89 insertions(+), 102 deletions(-) diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig index 6905232ddb03..84c41824b433 100644 --- a/arch/riscv/crypto/Kconfig +++ b/arch/riscv/crypto/Kconfig @@ -3,13 +3,13 @@ menu "Accelerated Cryptographic Algorithms for CPU (riscv)" config CRYPTO_AES_RISCV64 - tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XTS" + tristate "Ciphers: AES, modes: CBC, CTS, CTR, XTS" depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \ RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS select CRYPTO_LIB_AES select CRYPTO_SKCIPHER help - Length-preserving ciphers: AES with ECB, CBC, CTS, CTR, XTS + Length-preserving ciphers: AES with CBC, CTS, CTR, XTS Architecture: riscv64 using: - Zvkned vector crypto extension diff --git a/arch/riscv/crypto/aes-riscv64-glue.c b/arch/riscv/crypto/aes-riscv64-glue.c index bbd920c9e29d..f7c492dcfd57 100644 --- a/arch/riscv/crypto/aes-riscv64-glue.c +++ b/arch/riscv/crypto/aes-riscv64-glue.c @@ -22,11 +22,6 @@ #include #include -asmlinkage void aes_ecb_encrypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len); -asmlinkage void aes_ecb_decrypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len); - asmlinkage void aes_cbc_encrypt_zvkned(const struct crypto_aes_ctx *key, const u8 *in, u8 *out, size_t len, u8 iv[AES_BLOCK_SIZE]); @@ -86,44 +81,6 @@ static int riscv64_aes_setkey_skcipher(struct crypto_skcipher *tfm, return riscv64_aes_setkey(ctx, key, keylen); } -/* AES-ECB */ - -static inline int riscv64_aes_ecb_crypt(struct skcipher_request *req, bool enc) -{ - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - while ((nbytes = walk.nbytes) != 0) { - kernel_vector_begin(); - if (enc) - aes_ecb_encrypt_zvkned(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - nbytes & ~(AES_BLOCK_SIZE - 1)); - else - aes_ecb_decrypt_zvkned(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - nbytes & ~(AES_BLOCK_SIZE - 1)); - kernel_vector_end(); - err = skcipher_walk_done(&walk, nbytes & (AES_BLOCK_SIZE - 1)); - } - - return err; -} - -static int riscv64_aes_ecb_encrypt(struct skcipher_request *req) -{ - return riscv64_aes_ecb_crypt(req, true); -} - -static int riscv64_aes_ecb_decrypt(struct skcipher_request *req) -{ - return riscv64_aes_ecb_crypt(req, false); -} - /* AES-CBC */ static int riscv64_aes_cbc_crypt(struct skcipher_request *req, bool enc) @@ -411,21 +368,6 @@ static int riscv64_aes_xts_decrypt(struct skcipher_request *req) static struct skcipher_alg riscv64_zvkned_aes_skcipher_algs[] = { { - .setkey = riscv64_aes_setkey_skcipher, - .encrypt = riscv64_aes_ecb_encrypt, - .decrypt = riscv64_aes_ecb_decrypt, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .walksize = 8 * AES_BLOCK_SIZE, /* matches LMUL=8 */ - .base = { - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_priority = 300, - .cra_name = "ecb(aes)", - .cra_driver_name = "ecb-aes-riscv64-zvkned", - .cra_module = THIS_MODULE, - }, - }, { .setkey = riscv64_aes_setkey_skcipher, .encrypt = riscv64_aes_cbc_encrypt, .decrypt = riscv64_aes_cbc_decrypt, @@ -555,11 +497,10 @@ static void __exit riscv64_aes_mod_exit(void) module_init(riscv64_aes_mod_init); module_exit(riscv64_aes_mod_exit); -MODULE_DESCRIPTION("AES-ECB/CBC/CTS/CTR/XTS (RISC-V accelerated)"); +MODULE_DESCRIPTION("AES-CBC/CTS/CTR/XTS (RISC-V accelerated)"); MODULE_AUTHOR("Jerry Shih "); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("aes"); -MODULE_ALIAS_CRYPTO("ecb(aes)"); MODULE_ALIAS_CRYPTO("cbc(aes)"); MODULE_ALIAS_CRYPTO("cts(cbc(aes))"); MODULE_ALIAS_CRYPTO("ctr(aes)"); diff --git a/arch/riscv/crypto/aes-riscv64-zvkned.S b/arch/riscv/crypto/aes-riscv64-zvkned.S index d0fc4581a380..00f8a06596d3 100644 --- a/arch/riscv/crypto/aes-riscv64-zvkned.S +++ b/arch/riscv/crypto/aes-riscv64-zvkned.S @@ -56,45 +56,6 @@ #define LEN a3 #define IVP a4 -.macro __aes_ecb_crypt enc, keylen - srli t0, LEN, 2 - // t0 is the remaining length in 32-bit words. It's a multiple of 4. -1: - vsetvli t1, t0, e32, m8, ta, ma - sub t0, t0, t1 // Subtract number of words processed - slli t1, t1, 2 // Words to bytes - vle32.v v16, (INP) - aes_crypt v16, \enc, \keylen - vse32.v v16, (OUTP) - add INP, INP, t1 - add OUTP, OUTP, t1 - bnez t0, 1b - - ret -.endm - -.macro aes_ecb_crypt enc - aes_begin KEYP, 128f, 192f - __aes_ecb_crypt \enc, 256 -128: - __aes_ecb_crypt \enc, 128 -192: - __aes_ecb_crypt \enc, 192 -.endm - -// void aes_ecb_encrypt_zvkned(const struct crypto_aes_ctx *key, -// const u8 *in, u8 *out, size_t len); -// -// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). -SYM_FUNC_START(aes_ecb_encrypt_zvkned) - aes_ecb_crypt 1 -SYM_FUNC_END(aes_ecb_encrypt_zvkned) - -// Same prototype and calling convention as the encryption function -SYM_FUNC_START(aes_ecb_decrypt_zvkned) - aes_ecb_crypt 0 -SYM_FUNC_END(aes_ecb_decrypt_zvkned) - .macro aes_cbc_encrypt keylen vle32.v v16, (IVP) // Load IV 1: diff --git a/crypto/aes.c b/crypto/aes.c index 756fce2157be..9b9e5b413fc4 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -610,7 +610,8 @@ static struct skcipher_alg skcipher_algs[] = { { .base.cra_name = "ecb(aes)", .base.cra_driver_name = "ecb-aes-lib", - .base.cra_priority = IS_ENABLED(CONFIG_X86) ? 300 : 110, + .base.cra_priority = (IS_ENABLED(CONFIG_RISCV) || + IS_ENABLED(CONFIG_X86)) ? 300 : 110, .base.cra_blocksize = AES_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct aes_key), .base.cra_module = THIS_MODULE, diff --git a/lib/crypto/riscv/aes-riscv64-zvkned.S b/lib/crypto/riscv/aes-riscv64-zvkned.S index fb35f694b5ac..3700028e62ea 100644 --- a/lib/crypto/riscv/aes-riscv64-zvkned.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned.S @@ -81,3 +81,54 @@ SYM_FUNC_END(aes_encrypt_zvkned) SYM_FUNC_START(aes_decrypt_zvkned) aes_crypt_zvkned 0 SYM_FUNC_END(aes_decrypt_zvkned) + +#undef KEYP +#undef OUTP +#undef INP + +#define DST a0 +#define SRC a1 +#define LEN a2 +#define KEYP a3 + +.macro __aes_ecb_crypt enc, keylen + srli t0, LEN, 2 + // t0 is the remaining length in 32-bit words. It's a multiple of 4. +1: + vsetvli t1, t0, e32, m8, ta, ma + sub t0, t0, t1 // Subtract number of words processed + slli t1, t1, 2 // Words to bytes + vle32.v v16, (SRC) + aes_crypt v16, \enc, \keylen + vse32.v v16, (DST) + add SRC, SRC, t1 + add DST, DST, t1 + bnez t0, 1b + + ret +.endm + +.macro aes_ecb_crypt enc + aes_begin KEYP, 128f, 192f + __aes_ecb_crypt \enc, 256 +128: + __aes_ecb_crypt \enc, 128 +192: + __aes_ecb_crypt \enc, 192 +.endm + +// void aes_ecb_encrypt_zvkned(u8 *dst, const u8 *src, size_t len, +// const struct aes_enckey *key); +// +// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). +SYM_FUNC_START(aes_ecb_encrypt_zvkned) + aes_ecb_crypt 1 +SYM_FUNC_END(aes_ecb_encrypt_zvkned) + +// void aes_ecb_decrypt_zvkned(u8 *dst, const u8 *src, size_t len, +// const struct aes_key *key); +// +// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). +SYM_FUNC_START(aes_ecb_decrypt_zvkned) + aes_ecb_crypt 0 +SYM_FUNC_END(aes_ecb_decrypt_zvkned) diff --git a/lib/crypto/riscv/aes.h b/lib/crypto/riscv/aes.h index 9de9dbd1e887..5b36d0147d93 100644 --- a/lib/crypto/riscv/aes.h +++ b/lib/crypto/riscv/aes.h @@ -60,6 +60,39 @@ static void aes_decrypt_arch(const struct aes_key *key, } } +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_ECB) +void aes_ecb_encrypt_zvkned(u8 *dst, const u8 *src, size_t len, + const struct aes_enckey *key); +void aes_ecb_decrypt_zvkned(u8 *dst, const u8 *src, size_t len, + const struct aes_key *key); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_ecb_encrypt_arch aes_ecb_encrypt_arch +static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len, + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + aes_ecb_encrypt_zvkned(dst, src, len, key); + kernel_vector_end(); + return true; +} + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_ecb_decrypt_arch aes_ecb_decrypt_arch +static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len, + const struct aes_key *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + aes_ecb_decrypt_zvkned(dst, src, len, key); + kernel_vector_end(); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_ECB */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { -- 2.55.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv