* [PATCH 1/2] crypto: Delete Qualcomm crypto engine driver
From: Demi Marie Obenour via B4 Relay @ 2026-05-23 19:03 UTC (permalink / raw)
To: Herbert Xu, David S. Miller, Thara Gopinath, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
Russell King
Cc: linux-kernel, linux-crypto, linux-arm-msm, Eric Biggers,
Ard Biesheuvel, devicetree, linux-arm-kernel, Demi Marie Obenour
In-Reply-To: <20260523-delete-qce-v1-0-86105cd7f406@gmail.com>
From: Demi Marie Obenour <demiobenour@gmail.com>
It's slower than the generic C code and causes problems.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
MAINTAINERS | 8 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm64/configs/defconfig | 1 -
drivers/crypto/Kconfig | 111 -----
drivers/crypto/Makefile | 1 -
drivers/crypto/qce/Makefile | 9 -
drivers/crypto/qce/aead.c | 841 ------------------------------------
drivers/crypto/qce/aead.h | 56 ---
drivers/crypto/qce/cipher.h | 56 ---
drivers/crypto/qce/common.c | 595 -------------------------
drivers/crypto/qce/common.h | 104 -----
drivers/crypto/qce/core.c | 271 ------------
drivers/crypto/qce/core.h | 64 ---
drivers/crypto/qce/dma.c | 135 ------
drivers/crypto/qce/dma.h | 47 --
drivers/crypto/qce/regs-v5.h | 326 --------------
drivers/crypto/qce/sha.c | 545 -----------------------
drivers/crypto/qce/sha.h | 72 ---
drivers/crypto/qce/skcipher.c | 529 -----------------------
19 files changed, 3772 deletions(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 882214b0e7db53bb8cc8e75b5d2269ee0591ea20..ff631ec4b025ed256d7ef74c313a88755c205797 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -21898,14 +21898,6 @@ F: Documentation/devicetree/bindings/cpufreq/qcom-cpufreq-nvmem.yaml
F: Documentation/devicetree/bindings/opp/opp-v2-kryo-cpu.yaml
F: drivers/cpufreq/qcom-cpufreq-nvmem.c
-QUALCOMM CRYPTO DRIVERS
-M: Thara Gopinath <thara.gopinath@gmail.com>
-L: linux-crypto@vger.kernel.org
-L: linux-arm-msm@vger.kernel.org
-S: Maintained
-F: Documentation/devicetree/bindings/crypto/qcom-qce.yaml
-F: drivers/crypto/qce/
-
QUALCOMM EMAC GIGABIT ETHERNET DRIVER
M: Timur Tabi <timur@kernel.org>
L: netdev@vger.kernel.org
diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig
index bcc9aabc120283e8b98584964e0db3e24679724f..54960c8dc0989226ffd16c69cd26b69c33d22b79 100644
--- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -1335,7 +1335,6 @@ CONFIG_CRYPTO_DEV_ATMEL_AES=m
CONFIG_CRYPTO_DEV_ATMEL_TDES=m
CONFIG_CRYPTO_DEV_ATMEL_SHA=m
CONFIG_CRYPTO_DEV_MARVELL_CESA=m
-CONFIG_CRYPTO_DEV_QCE=m
CONFIG_CRYPTO_DEV_QCOM_RNG=m
CONFIG_CRYPTO_DEV_ROCKCHIP=m
CONFIG_CRYPTO_DEV_STM32_HASH=m
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index d905a0777f939c51cc39df6230591a31058b765f..6ba10b76026867dd4f4d3dead6d59dadf0e7d9aa 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -1952,7 +1952,6 @@ CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
CONFIG_CRYPTO_DEV_SUN8I_CE=m
CONFIG_CRYPTO_DEV_FSL_CAAM=m
CONFIG_CRYPTO_DEV_FSL_DPAA2_CAAM=m
-CONFIG_CRYPTO_DEV_QCE=m
CONFIG_CRYPTO_DEV_QCOM_RNG=m
CONFIG_CRYPTO_DEV_TEGRA=m
CONFIG_CRYPTO_DEV_XILINX_TRNG=m
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 3449b3c9c6adfdaf1ab5740d6b3542c39c7d5745..6da104e8c3c270968f4d7f0bdd2a03c90e2621a1 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -545,117 +545,6 @@ source "drivers/crypto/cavium/nitrox/Kconfig"
source "drivers/crypto/marvell/Kconfig"
source "drivers/crypto/intel/Kconfig"
-config CRYPTO_DEV_QCE
- tristate "Qualcomm crypto engine accelerator"
- depends on ARCH_QCOM || COMPILE_TEST
- depends on HAS_IOMEM
- help
- This driver supports Qualcomm crypto engine accelerator
- hardware. To compile this driver as a module, choose M here. The
- module will be called qcrypto.
-
-config CRYPTO_DEV_QCE_SKCIPHER
- bool
- depends on CRYPTO_DEV_QCE
- select CRYPTO_AES
- select CRYPTO_LIB_DES
- select CRYPTO_ECB
- select CRYPTO_CBC
- select CRYPTO_XTS
- select CRYPTO_CTR
- select CRYPTO_SKCIPHER
-
-config CRYPTO_DEV_QCE_SHA
- bool
- depends on CRYPTO_DEV_QCE
- select CRYPTO_SHA1
- select CRYPTO_SHA256
-
-config CRYPTO_DEV_QCE_AEAD
- bool
- depends on CRYPTO_DEV_QCE
- select CRYPTO_AUTHENC
- select CRYPTO_LIB_DES
-
-choice
- prompt "Algorithms enabled for QCE acceleration"
- default CRYPTO_DEV_QCE_ENABLE_ALL
- depends on CRYPTO_DEV_QCE
- help
- This option allows to choose whether to build support for all algorithms
- (default), hashes-only, or skciphers-only.
-
- The QCE engine does not appear to scale as well as the CPU to handle
- multiple crypto requests. While the ipq40xx chips have 4-core CPUs, the
- QCE handles only 2 requests in parallel.
-
- Ipsec throughput seems to improve when disabling either family of
- algorithms, sharing the load with the CPU. Enabling skciphers-only
- appears to work best.
-
- config CRYPTO_DEV_QCE_ENABLE_ALL
- bool "All supported algorithms"
- select CRYPTO_DEV_QCE_SKCIPHER
- select CRYPTO_DEV_QCE_SHA
- select CRYPTO_DEV_QCE_AEAD
- help
- Enable all supported algorithms:
- - AES (CBC, CTR, ECB, XTS)
- - 3DES (CBC, ECB)
- - DES (CBC, ECB)
- - SHA1, HMAC-SHA1
- - SHA256, HMAC-SHA256
-
- config CRYPTO_DEV_QCE_ENABLE_SKCIPHER
- bool "Symmetric-key ciphers only"
- select CRYPTO_DEV_QCE_SKCIPHER
- help
- Enable symmetric-key ciphers only:
- - AES (CBC, CTR, ECB, XTS)
- - 3DES (ECB, CBC)
- - DES (ECB, CBC)
-
- config CRYPTO_DEV_QCE_ENABLE_SHA
- bool "Hash/HMAC only"
- select CRYPTO_DEV_QCE_SHA
- help
- Enable hashes/HMAC algorithms only:
- - SHA1, HMAC-SHA1
- - SHA256, HMAC-SHA256
-
- config CRYPTO_DEV_QCE_ENABLE_AEAD
- bool "AEAD algorithms only"
- select CRYPTO_DEV_QCE_AEAD
- help
- Enable AEAD algorithms only:
- - authenc()
- - ccm(aes)
- - rfc4309(ccm(aes))
-endchoice
-
-config CRYPTO_DEV_QCE_SW_MAX_LEN
- int "Default maximum request size to use software for AES"
- depends on CRYPTO_DEV_QCE && CRYPTO_DEV_QCE_SKCIPHER
- default 512
- help
- This sets the default maximum request size to perform AES requests
- using software instead of the crypto engine. It can be changed by
- setting the aes_sw_max_len parameter.
-
- Small blocks are processed faster in software than hardware.
- Considering the 256-bit ciphers, software is 2-3 times faster than
- qce at 256-bytes, 30% faster at 512, and about even at 768-bytes.
- With 128-bit keys, the break-even point would be around 1024-bytes.
-
- The default is set a little lower, to 512 bytes, to balance the
- cost in CPU usage. The minimum recommended setting is 16-bytes
- (1 AES block), since AES-GCM will fail if you set it lower.
- Setting this to zero will send all requests to the hardware.
-
- Note that 192-bit keys are not supported by the hardware and are
- always processed by the software fallback, and all DES requests
- are done by the hardware.
-
config CRYPTO_DEV_QCOM_RNG
tristate "Qualcomm Random Number Generator Driver"
depends on ARCH_QCOM || COMPILE_TEST
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index 283bbc650b5b22a3f2a5a1ec81ca42ae3d37a80f..9fcd55a8474e5a95c0c189d9ae5d890abcd5dbdc 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -27,7 +27,6 @@ obj-$(CONFIG_CRYPTO_DEV_OMAP_SHAM) += omap-sham.o
obj-$(CONFIG_CRYPTO_DEV_PADLOCK_AES) += padlock-aes.o
obj-$(CONFIG_CRYPTO_DEV_PADLOCK_SHA) += padlock-sha.o
obj-$(CONFIG_CRYPTO_DEV_PPC4XX) += amcc/
-obj-$(CONFIG_CRYPTO_DEV_QCE) += qce/
obj-$(CONFIG_CRYPTO_DEV_QCOM_RNG) += qcom-rng.o
obj-$(CONFIG_CRYPTO_DEV_ROCKCHIP) += rockchip/
obj-$(CONFIG_CRYPTO_DEV_S5P) += s5p-sss.o
diff --git a/drivers/crypto/qce/Makefile b/drivers/crypto/qce/Makefile
deleted file mode 100644
index 2cf8984e1b85161ab468bdec4d50950f40d070d0..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/Makefile
+++ /dev/null
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_CRYPTO_DEV_QCE) += qcrypto.o
-qcrypto-objs := core.o \
- common.o \
- dma.o
-
-qcrypto-$(CONFIG_CRYPTO_DEV_QCE_SHA) += sha.o
-qcrypto-$(CONFIG_CRYPTO_DEV_QCE_SKCIPHER) += skcipher.o
-qcrypto-$(CONFIG_CRYPTO_DEV_QCE_AEAD) += aead.o
diff --git a/drivers/crypto/qce/aead.c b/drivers/crypto/qce/aead.c
deleted file mode 100644
index 9cb11fada2c4ddecebcb5c48416245f402389bb1..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/aead.c
+++ /dev/null
@@ -1,841 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-
-/*
- * Copyright (C) 2021, Linaro Limited. All rights reserved.
- */
-#include <linux/dma-mapping.h>
-#include <linux/interrupt.h>
-#include <linux/string.h>
-#include <crypto/gcm.h>
-#include <crypto/authenc.h>
-#include <crypto/internal/aead.h>
-#include <crypto/internal/des.h>
-#include <crypto/sha1.h>
-#include <crypto/sha2.h>
-#include <crypto/scatterwalk.h>
-#include "aead.h"
-
-#define CCM_NONCE_ADATA_SHIFT 6
-#define CCM_NONCE_AUTHSIZE_SHIFT 3
-#define MAX_CCM_ADATA_HEADER_LEN 6
-
-static LIST_HEAD(aead_algs);
-
-static void qce_aead_done(void *data)
-{
- struct crypto_async_request *async_req = data;
- struct aead_request *req = aead_request_cast(async_req);
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_aead_ctx *ctx = crypto_tfm_ctx(async_req->tfm);
- struct qce_alg_template *tmpl = to_aead_tmpl(crypto_aead_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- struct qce_result_dump *result_buf = qce->dma.result_buf;
- enum dma_data_direction dir_src, dir_dst;
- bool diff_dst;
- int error;
- u32 status;
- unsigned int totallen;
- unsigned char tag[SHA256_DIGEST_SIZE] = {0};
-
- diff_dst = (req->src != req->dst) ? true : false;
- dir_src = diff_dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL;
- dir_dst = diff_dst ? DMA_FROM_DEVICE : DMA_BIDIRECTIONAL;
-
- error = qce_dma_terminate_all(&qce->dma);
- if (error)
- dev_dbg(qce->dev, "aead dma termination error (%d)\n",
- error);
- if (diff_dst)
- dma_unmap_sg(qce->dev, rctx->src_sg, rctx->src_nents, dir_src);
-
- dma_unmap_sg(qce->dev, rctx->dst_sg, rctx->dst_nents, dir_dst);
-
- if (IS_CCM(rctx->flags)) {
- if (req->assoclen) {
- sg_free_table(&rctx->src_tbl);
- if (diff_dst)
- sg_free_table(&rctx->dst_tbl);
- } else {
- if (!(IS_DECRYPT(rctx->flags) && !diff_dst))
- sg_free_table(&rctx->dst_tbl);
- }
- } else {
- sg_free_table(&rctx->dst_tbl);
- }
-
- error = qce_check_status(qce, &status);
- if (error < 0 && (error != -EBADMSG))
- dev_err(qce->dev, "aead operation error (%x)\n", status);
-
- if (IS_ENCRYPT(rctx->flags)) {
- totallen = req->cryptlen + req->assoclen;
- if (IS_CCM(rctx->flags))
- scatterwalk_map_and_copy(rctx->ccmresult_buf, req->dst,
- totallen, ctx->authsize, 1);
- else
- scatterwalk_map_and_copy(result_buf->auth_iv, req->dst,
- totallen, ctx->authsize, 1);
-
- } else if (!IS_CCM(rctx->flags)) {
- totallen = req->cryptlen + req->assoclen - ctx->authsize;
- scatterwalk_map_and_copy(tag, req->src, totallen, ctx->authsize, 0);
- if (memcmp(result_buf->auth_iv, tag, ctx->authsize)) {
- pr_err("Bad message error\n");
- error = -EBADMSG;
- }
- }
-
- qce->async_req_done(qce, error);
-}
-
-static struct scatterlist *
-qce_aead_prepare_result_buf(struct sg_table *tbl, struct aead_request *req)
-{
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_aead_tmpl(crypto_aead_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
-
- sg_init_one(&rctx->result_sg, qce->dma.result_buf, QCE_RESULT_BUF_SZ);
- return qce_sgtable_add(tbl, &rctx->result_sg, QCE_RESULT_BUF_SZ);
-}
-
-static struct scatterlist *
-qce_aead_prepare_ccm_result_buf(struct sg_table *tbl, struct aead_request *req)
-{
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
-
- sg_init_one(&rctx->result_sg, rctx->ccmresult_buf, QCE_BAM_BURST_SIZE);
- return qce_sgtable_add(tbl, &rctx->result_sg, QCE_BAM_BURST_SIZE);
-}
-
-static struct scatterlist *
-qce_aead_prepare_dst_buf(struct aead_request *req)
-{
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_aead_tmpl(crypto_aead_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- struct scatterlist *sg, *msg_sg, __sg[2];
- gfp_t gfp;
- unsigned int assoclen = req->assoclen;
- unsigned int totallen;
- int ret;
-
- totallen = rctx->cryptlen + assoclen;
- rctx->dst_nents = sg_nents_for_len(req->dst, totallen);
- if (rctx->dst_nents < 0) {
- dev_err(qce->dev, "Invalid numbers of dst SG.\n");
- return ERR_PTR(-EINVAL);
- }
- if (IS_CCM(rctx->flags))
- rctx->dst_nents += 2;
- else
- rctx->dst_nents += 1;
-
- gfp = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
- GFP_KERNEL : GFP_ATOMIC;
- ret = sg_alloc_table(&rctx->dst_tbl, rctx->dst_nents, gfp);
- if (ret)
- return ERR_PTR(ret);
-
- if (IS_CCM(rctx->flags) && assoclen) {
- /* Get the dst buffer */
- msg_sg = scatterwalk_ffwd(__sg, req->dst, assoclen);
-
- sg = qce_sgtable_add(&rctx->dst_tbl, &rctx->adata_sg,
- rctx->assoclen);
- if (IS_ERR(sg))
- goto dst_tbl_free;
- /* dst buffer */
- sg = qce_sgtable_add(&rctx->dst_tbl, msg_sg, rctx->cryptlen);
- if (IS_ERR(sg))
- goto dst_tbl_free;
- totallen = rctx->cryptlen + rctx->assoclen;
- } else {
- if (totallen) {
- sg = qce_sgtable_add(&rctx->dst_tbl, req->dst, totallen);
- if (IS_ERR(sg))
- goto dst_tbl_free;
- }
- }
- if (IS_CCM(rctx->flags))
- sg = qce_aead_prepare_ccm_result_buf(&rctx->dst_tbl, req);
- else
- sg = qce_aead_prepare_result_buf(&rctx->dst_tbl, req);
-
- if (IS_ERR(sg))
- goto dst_tbl_free;
-
- sg_mark_end(sg);
- rctx->dst_sg = rctx->dst_tbl.sgl;
- rctx->dst_nents = sg_nents_for_len(rctx->dst_sg, totallen) + 1;
-
- return sg;
-
-dst_tbl_free:
- sg_free_table(&rctx->dst_tbl);
- return sg;
-}
-
-static int
-qce_aead_ccm_prepare_buf_assoclen(struct aead_request *req)
-{
- struct scatterlist *sg, *msg_sg, __sg[2];
- struct crypto_aead *tfm = crypto_aead_reqtfm(req);
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
- unsigned int assoclen = rctx->assoclen;
- unsigned int adata_header_len, cryptlen, totallen;
- gfp_t gfp;
- bool diff_dst;
- int ret;
-
- if (IS_DECRYPT(rctx->flags))
- cryptlen = rctx->cryptlen + ctx->authsize;
- else
- cryptlen = rctx->cryptlen;
- totallen = cryptlen + req->assoclen;
-
- /* Get the msg */
- msg_sg = scatterwalk_ffwd(__sg, req->src, req->assoclen);
-
- rctx->adata = kzalloc((ALIGN(assoclen, 16) + MAX_CCM_ADATA_HEADER_LEN) *
- sizeof(unsigned char), GFP_ATOMIC);
- if (!rctx->adata)
- return -ENOMEM;
-
- /*
- * Format associated data (RFC3610 and NIST 800-38C)
- * Even though specification allows for AAD to be up to 2^64 - 1 bytes,
- * the assoclen field in aead_request is unsigned int and thus limits
- * the AAD to be up to 2^32 - 1 bytes. So we handle only two scenarios
- * while forming the header for AAD.
- */
- if (assoclen < 0xff00) {
- adata_header_len = 2;
- *(__be16 *)rctx->adata = cpu_to_be16(assoclen);
- } else {
- adata_header_len = 6;
- *(__be16 *)rctx->adata = cpu_to_be16(0xfffe);
- *(__be32 *)(rctx->adata + 2) = cpu_to_be32(assoclen);
- }
-
- /* Copy the associated data */
- if (sg_copy_to_buffer(req->src, sg_nents_for_len(req->src, assoclen),
- rctx->adata + adata_header_len,
- assoclen) != assoclen)
- return -EINVAL;
-
- /* Pad associated data to block size */
- rctx->assoclen = ALIGN(assoclen + adata_header_len, 16);
-
- diff_dst = (req->src != req->dst) ? true : false;
-
- if (diff_dst)
- rctx->src_nents = sg_nents_for_len(req->src, totallen) + 1;
- else
- rctx->src_nents = sg_nents_for_len(req->src, totallen) + 2;
-
- gfp = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ? GFP_KERNEL : GFP_ATOMIC;
- ret = sg_alloc_table(&rctx->src_tbl, rctx->src_nents, gfp);
- if (ret)
- return ret;
-
- /* Associated Data */
- sg_init_one(&rctx->adata_sg, rctx->adata, rctx->assoclen);
- sg = qce_sgtable_add(&rctx->src_tbl, &rctx->adata_sg,
- rctx->assoclen);
- if (IS_ERR(sg)) {
- ret = PTR_ERR(sg);
- goto err_free;
- }
- /* src msg */
- sg = qce_sgtable_add(&rctx->src_tbl, msg_sg, cryptlen);
- if (IS_ERR(sg)) {
- ret = PTR_ERR(sg);
- goto err_free;
- }
- if (!diff_dst) {
- /*
- * For decrypt, when src and dst buffers are same, there is already space
- * in the buffer for padded 0's which is output in lieu of
- * the MAC that is input. So skip the below.
- */
- if (!IS_DECRYPT(rctx->flags)) {
- sg = qce_aead_prepare_ccm_result_buf(&rctx->src_tbl, req);
- if (IS_ERR(sg)) {
- ret = PTR_ERR(sg);
- goto err_free;
- }
- }
- }
- sg_mark_end(sg);
- rctx->src_sg = rctx->src_tbl.sgl;
- totallen = cryptlen + rctx->assoclen;
- rctx->src_nents = sg_nents_for_len(rctx->src_sg, totallen);
-
- if (diff_dst) {
- sg = qce_aead_prepare_dst_buf(req);
- if (IS_ERR(sg)) {
- ret = PTR_ERR(sg);
- goto err_free;
- }
- } else {
- if (IS_ENCRYPT(rctx->flags))
- rctx->dst_nents = rctx->src_nents + 1;
- else
- rctx->dst_nents = rctx->src_nents;
- rctx->dst_sg = rctx->src_sg;
- }
-
- return 0;
-err_free:
- sg_free_table(&rctx->src_tbl);
- return ret;
-}
-
-static int qce_aead_prepare_buf(struct aead_request *req)
-{
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_aead_tmpl(crypto_aead_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- struct scatterlist *sg;
- bool diff_dst = (req->src != req->dst) ? true : false;
- unsigned int totallen;
-
- totallen = rctx->cryptlen + rctx->assoclen;
-
- sg = qce_aead_prepare_dst_buf(req);
- if (IS_ERR(sg))
- return PTR_ERR(sg);
- if (diff_dst) {
- rctx->src_nents = sg_nents_for_len(req->src, totallen);
- if (rctx->src_nents < 0) {
- dev_err(qce->dev, "Invalid numbers of src SG.\n");
- return -EINVAL;
- }
- rctx->src_sg = req->src;
- } else {
- rctx->src_nents = rctx->dst_nents - 1;
- rctx->src_sg = rctx->dst_sg;
- }
- return 0;
-}
-
-static int qce_aead_ccm_prepare_buf(struct aead_request *req)
-{
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct crypto_aead *tfm = crypto_aead_reqtfm(req);
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
- struct scatterlist *sg;
- bool diff_dst = (req->src != req->dst) ? true : false;
- unsigned int cryptlen;
-
- if (rctx->assoclen)
- return qce_aead_ccm_prepare_buf_assoclen(req);
-
- if (IS_ENCRYPT(rctx->flags))
- return qce_aead_prepare_buf(req);
-
- cryptlen = rctx->cryptlen + ctx->authsize;
- if (diff_dst) {
- rctx->src_nents = sg_nents_for_len(req->src, cryptlen);
- rctx->src_sg = req->src;
- sg = qce_aead_prepare_dst_buf(req);
- if (IS_ERR(sg))
- return PTR_ERR(sg);
- } else {
- rctx->src_nents = sg_nents_for_len(req->src, cryptlen);
- rctx->src_sg = req->src;
- rctx->dst_nents = rctx->src_nents;
- rctx->dst_sg = rctx->src_sg;
- }
-
- return 0;
-}
-
-static int qce_aead_create_ccm_nonce(struct qce_aead_reqctx *rctx, struct qce_aead_ctx *ctx)
-{
- unsigned int msglen_size, ivsize;
- u8 msg_len[4];
- int i;
-
- if (!rctx || !rctx->iv)
- return -EINVAL;
-
- msglen_size = rctx->iv[0] + 1;
-
- /* Verify that msg len size is valid */
- if (msglen_size < 2 || msglen_size > 8)
- return -EINVAL;
-
- ivsize = rctx->ivsize;
-
- /*
- * Clear the msglen bytes in IV.
- * Else the h/w engine and nonce will use any stray value pending there.
- */
- if (!IS_CCM_RFC4309(rctx->flags)) {
- for (i = 0; i < msglen_size; i++)
- rctx->iv[ivsize - i - 1] = 0;
- }
-
- /*
- * The crypto framework encodes cryptlen as unsigned int. Thus, even though
- * spec allows for upto 8 bytes to encode msg_len only 4 bytes are needed.
- */
- if (msglen_size > 4)
- msglen_size = 4;
-
- memcpy(&msg_len[0], &rctx->cryptlen, 4);
-
- memcpy(&rctx->ccm_nonce[0], rctx->iv, rctx->ivsize);
- if (rctx->assoclen)
- rctx->ccm_nonce[0] |= 1 << CCM_NONCE_ADATA_SHIFT;
- rctx->ccm_nonce[0] |= ((ctx->authsize - 2) / 2) <<
- CCM_NONCE_AUTHSIZE_SHIFT;
- for (i = 0; i < msglen_size; i++)
- rctx->ccm_nonce[QCE_MAX_NONCE - i - 1] = msg_len[i];
-
- return 0;
-}
-
-static int
-qce_aead_async_req_handle(struct crypto_async_request *async_req)
-{
- struct aead_request *req = aead_request_cast(async_req);
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct crypto_aead *tfm = crypto_aead_reqtfm(req);
- struct qce_aead_ctx *ctx = crypto_tfm_ctx(async_req->tfm);
- struct qce_alg_template *tmpl = to_aead_tmpl(crypto_aead_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- enum dma_data_direction dir_src, dir_dst;
- bool diff_dst;
- int dst_nents, src_nents, ret;
-
- if (IS_CCM_RFC4309(rctx->flags)) {
- memset(rctx->ccm_rfc4309_iv, 0, QCE_MAX_IV_SIZE);
- rctx->ccm_rfc4309_iv[0] = 3;
- memcpy(&rctx->ccm_rfc4309_iv[1], ctx->ccm4309_salt, QCE_CCM4309_SALT_SIZE);
- memcpy(&rctx->ccm_rfc4309_iv[4], req->iv, 8);
- rctx->iv = rctx->ccm_rfc4309_iv;
- rctx->ivsize = AES_BLOCK_SIZE;
- } else {
- rctx->iv = req->iv;
- rctx->ivsize = crypto_aead_ivsize(tfm);
- }
- if (IS_CCM_RFC4309(rctx->flags))
- rctx->assoclen = req->assoclen - 8;
- else
- rctx->assoclen = req->assoclen;
-
- diff_dst = (req->src != req->dst) ? true : false;
- dir_src = diff_dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL;
- dir_dst = diff_dst ? DMA_FROM_DEVICE : DMA_BIDIRECTIONAL;
-
- if (IS_CCM(rctx->flags)) {
- ret = qce_aead_create_ccm_nonce(rctx, ctx);
- if (ret)
- return ret;
- }
- if (IS_CCM(rctx->flags))
- ret = qce_aead_ccm_prepare_buf(req);
- else
- ret = qce_aead_prepare_buf(req);
-
- if (ret)
- return ret;
- dst_nents = dma_map_sg(qce->dev, rctx->dst_sg, rctx->dst_nents, dir_dst);
- if (!dst_nents) {
- ret = -EIO;
- goto error_free;
- }
-
- if (diff_dst) {
- src_nents = dma_map_sg(qce->dev, rctx->src_sg, rctx->src_nents, dir_src);
- if (src_nents < 0) {
- ret = src_nents;
- goto error_unmap_dst;
- }
- } else {
- if (IS_CCM(rctx->flags) && IS_DECRYPT(rctx->flags))
- src_nents = dst_nents;
- else
- src_nents = dst_nents - 1;
- }
-
- ret = qce_dma_prep_sgs(&qce->dma, rctx->src_sg, src_nents, rctx->dst_sg, dst_nents,
- qce_aead_done, async_req);
- if (ret)
- goto error_unmap_src;
-
- qce_dma_issue_pending(&qce->dma);
-
- ret = qce_start(async_req, tmpl->crypto_alg_type);
- if (ret)
- goto error_terminate;
-
- return 0;
-
-error_terminate:
- qce_dma_terminate_all(&qce->dma);
-error_unmap_src:
- if (diff_dst)
- dma_unmap_sg(qce->dev, req->src, rctx->src_nents, dir_src);
-error_unmap_dst:
- dma_unmap_sg(qce->dev, rctx->dst_sg, rctx->dst_nents, dir_dst);
-error_free:
- if (IS_CCM(rctx->flags) && rctx->assoclen) {
- sg_free_table(&rctx->src_tbl);
- if (diff_dst)
- sg_free_table(&rctx->dst_tbl);
- } else {
- sg_free_table(&rctx->dst_tbl);
- }
- return ret;
-}
-
-static int qce_aead_crypt(struct aead_request *req, int encrypt)
-{
- struct crypto_aead *tfm = crypto_aead_reqtfm(req);
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
- struct qce_alg_template *tmpl = to_aead_tmpl(tfm);
- unsigned int blocksize = crypto_aead_blocksize(tfm);
-
- rctx->flags = tmpl->alg_flags;
- rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
-
- if (encrypt)
- rctx->cryptlen = req->cryptlen;
- else
- rctx->cryptlen = req->cryptlen - ctx->authsize;
-
- /* CE does not handle 0 length messages */
- if (!rctx->cryptlen) {
- if (!(IS_CCM(rctx->flags) && IS_DECRYPT(rctx->flags)))
- ctx->need_fallback = true;
- }
-
- /* If fallback is needed, schedule and exit */
- if (ctx->need_fallback) {
- /* Reset need_fallback in case the same ctx is used for another transaction */
- ctx->need_fallback = false;
-
- aead_request_set_tfm(&rctx->fallback_req, ctx->fallback);
- aead_request_set_callback(&rctx->fallback_req, req->base.flags,
- req->base.complete, req->base.data);
- aead_request_set_crypt(&rctx->fallback_req, req->src,
- req->dst, req->cryptlen, req->iv);
- aead_request_set_ad(&rctx->fallback_req, req->assoclen);
-
- return encrypt ? crypto_aead_encrypt(&rctx->fallback_req) :
- crypto_aead_decrypt(&rctx->fallback_req);
- }
-
- /*
- * CBC algorithms require message lengths to be
- * multiples of block size.
- */
- if (IS_CBC(rctx->flags) && !IS_ALIGNED(rctx->cryptlen, blocksize))
- return -EINVAL;
-
- /* RFC4309 supported AAD size 16 bytes/20 bytes */
- if (IS_CCM_RFC4309(rctx->flags))
- if (crypto_ipsec_check_assoclen(req->assoclen))
- return -EINVAL;
-
- return tmpl->qce->async_req_enqueue(tmpl->qce, &req->base);
-}
-
-static int qce_aead_encrypt(struct aead_request *req)
-{
- return qce_aead_crypt(req, 1);
-}
-
-static int qce_aead_decrypt(struct aead_request *req)
-{
- return qce_aead_crypt(req, 0);
-}
-
-static int qce_aead_ccm_setkey(struct crypto_aead *tfm, const u8 *key,
- unsigned int keylen)
-{
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
- unsigned long flags = to_aead_tmpl(tfm)->alg_flags;
-
- if (IS_CCM_RFC4309(flags)) {
- if (keylen < QCE_CCM4309_SALT_SIZE)
- return -EINVAL;
- keylen -= QCE_CCM4309_SALT_SIZE;
- memcpy(ctx->ccm4309_salt, key + keylen, QCE_CCM4309_SALT_SIZE);
- }
-
- if (keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256 && keylen != AES_KEYSIZE_192)
- return -EINVAL;
-
- ctx->enc_keylen = keylen;
- ctx->auth_keylen = keylen;
-
- memcpy(ctx->enc_key, key, keylen);
- memcpy(ctx->auth_key, key, keylen);
-
- if (keylen == AES_KEYSIZE_192)
- ctx->need_fallback = true;
-
- return IS_CCM_RFC4309(flags) ?
- crypto_aead_setkey(ctx->fallback, key, keylen + QCE_CCM4309_SALT_SIZE) :
- crypto_aead_setkey(ctx->fallback, key, keylen);
-}
-
-static int qce_aead_setkey(struct crypto_aead *tfm, const u8 *key, unsigned int keylen)
-{
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
- struct crypto_authenc_keys authenc_keys;
- unsigned long flags = to_aead_tmpl(tfm)->alg_flags;
- u32 _key[6];
- int err;
-
- err = crypto_authenc_extractkeys(&authenc_keys, key, keylen);
- if (err)
- return err;
-
- if (authenc_keys.enckeylen > QCE_MAX_KEY_SIZE ||
- authenc_keys.authkeylen > QCE_MAX_KEY_SIZE)
- return -EINVAL;
-
- if (IS_DES(flags)) {
- err = verify_aead_des_key(tfm, authenc_keys.enckey, authenc_keys.enckeylen);
- if (err)
- return err;
- } else if (IS_3DES(flags)) {
- err = verify_aead_des3_key(tfm, authenc_keys.enckey, authenc_keys.enckeylen);
- if (err)
- return err;
- /*
- * The crypto engine does not support any two keys
- * being the same for triple des algorithms. The
- * verify_skcipher_des3_key does not check for all the
- * below conditions. Schedule fallback in this case.
- */
- memcpy(_key, authenc_keys.enckey, DES3_EDE_KEY_SIZE);
- if (!((_key[0] ^ _key[2]) | (_key[1] ^ _key[3])) ||
- !((_key[2] ^ _key[4]) | (_key[3] ^ _key[5])) ||
- !((_key[0] ^ _key[4]) | (_key[1] ^ _key[5])))
- ctx->need_fallback = true;
- } else if (IS_AES(flags)) {
- /* No random key sizes */
- if (authenc_keys.enckeylen != AES_KEYSIZE_128 &&
- authenc_keys.enckeylen != AES_KEYSIZE_192 &&
- authenc_keys.enckeylen != AES_KEYSIZE_256)
- return -EINVAL;
- if (authenc_keys.enckeylen == AES_KEYSIZE_192)
- ctx->need_fallback = true;
- }
-
- ctx->enc_keylen = authenc_keys.enckeylen;
- ctx->auth_keylen = authenc_keys.authkeylen;
-
- memcpy(ctx->enc_key, authenc_keys.enckey, authenc_keys.enckeylen);
-
- memcpy_and_pad(ctx->auth_key, sizeof(ctx->auth_key),
- authenc_keys.authkey, authenc_keys.authkeylen, 0);
-
- return crypto_aead_setkey(ctx->fallback, key, keylen);
-}
-
-static int qce_aead_setauthsize(struct crypto_aead *tfm, unsigned int authsize)
-{
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
- unsigned long flags = to_aead_tmpl(tfm)->alg_flags;
-
- if (IS_CCM(flags)) {
- if (authsize < 4 || authsize > 16 || authsize % 2)
- return -EINVAL;
- if (IS_CCM_RFC4309(flags) && (authsize < 8 || authsize % 4))
- return -EINVAL;
- }
- ctx->authsize = authsize;
-
- return crypto_aead_setauthsize(ctx->fallback, authsize);
-}
-
-static int qce_aead_init(struct crypto_aead *tfm)
-{
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
-
- ctx->need_fallback = false;
- ctx->fallback = crypto_alloc_aead(crypto_tfm_alg_name(&tfm->base),
- 0, CRYPTO_ALG_NEED_FALLBACK);
-
- if (IS_ERR(ctx->fallback))
- return PTR_ERR(ctx->fallback);
-
- crypto_aead_set_reqsize_dma(tfm, sizeof(struct qce_aead_reqctx) +
- crypto_aead_reqsize(ctx->fallback));
- return 0;
-}
-
-static void qce_aead_exit(struct crypto_aead *tfm)
-{
- struct qce_aead_ctx *ctx = crypto_aead_ctx(tfm);
-
- crypto_free_aead(ctx->fallback);
-}
-
-struct qce_aead_def {
- unsigned long flags;
- const char *name;
- const char *drv_name;
- unsigned int blocksize;
- unsigned int chunksize;
- unsigned int ivsize;
- unsigned int maxauthsize;
-};
-
-static const struct qce_aead_def aead_def[] = {
- {
- .flags = QCE_ALG_DES | QCE_MODE_CBC | QCE_HASH_SHA1_HMAC,
- .name = "authenc(hmac(sha1),cbc(des))",
- .drv_name = "authenc-hmac-sha1-cbc-des-qce",
- .blocksize = DES_BLOCK_SIZE,
- .ivsize = DES_BLOCK_SIZE,
- .maxauthsize = SHA1_DIGEST_SIZE,
- },
- {
- .flags = QCE_ALG_3DES | QCE_MODE_CBC | QCE_HASH_SHA1_HMAC,
- .name = "authenc(hmac(sha1),cbc(des3_ede))",
- .drv_name = "authenc-hmac-sha1-cbc-3des-qce",
- .blocksize = DES3_EDE_BLOCK_SIZE,
- .ivsize = DES3_EDE_BLOCK_SIZE,
- .maxauthsize = SHA1_DIGEST_SIZE,
- },
- {
- .flags = QCE_ALG_DES | QCE_MODE_CBC | QCE_HASH_SHA256_HMAC,
- .name = "authenc(hmac(sha256),cbc(des))",
- .drv_name = "authenc-hmac-sha256-cbc-des-qce",
- .blocksize = DES_BLOCK_SIZE,
- .ivsize = DES_BLOCK_SIZE,
- .maxauthsize = SHA256_DIGEST_SIZE,
- },
- {
- .flags = QCE_ALG_3DES | QCE_MODE_CBC | QCE_HASH_SHA256_HMAC,
- .name = "authenc(hmac(sha256),cbc(des3_ede))",
- .drv_name = "authenc-hmac-sha256-cbc-3des-qce",
- .blocksize = DES3_EDE_BLOCK_SIZE,
- .ivsize = DES3_EDE_BLOCK_SIZE,
- .maxauthsize = SHA256_DIGEST_SIZE,
- },
- {
- .flags = QCE_ALG_AES | QCE_MODE_CBC | QCE_HASH_SHA256_HMAC,
- .name = "authenc(hmac(sha256),cbc(aes))",
- .drv_name = "authenc-hmac-sha256-cbc-aes-qce",
- .blocksize = AES_BLOCK_SIZE,
- .ivsize = AES_BLOCK_SIZE,
- .maxauthsize = SHA256_DIGEST_SIZE,
- },
- {
- .flags = QCE_ALG_AES | QCE_MODE_CCM,
- .name = "ccm(aes)",
- .drv_name = "ccm-aes-qce",
- .blocksize = 1,
- .ivsize = AES_BLOCK_SIZE,
- .maxauthsize = AES_BLOCK_SIZE,
- },
- {
- .flags = QCE_ALG_AES | QCE_MODE_CCM | QCE_MODE_CCM_RFC4309,
- .name = "rfc4309(ccm(aes))",
- .drv_name = "rfc4309-ccm-aes-qce",
- .blocksize = 1,
- .ivsize = 8,
- .maxauthsize = AES_BLOCK_SIZE,
- },
-};
-
-static int qce_aead_register_one(const struct qce_aead_def *def, struct qce_device *qce)
-{
- struct qce_alg_template *tmpl;
- struct aead_alg *alg;
- int ret;
-
- tmpl = kzalloc_obj(*tmpl);
- if (!tmpl)
- return -ENOMEM;
-
- alg = &tmpl->alg.aead;
-
- strscpy(alg->base.cra_name, def->name);
- strscpy(alg->base.cra_driver_name, def->drv_name);
-
- alg->base.cra_blocksize = def->blocksize;
- alg->chunksize = def->chunksize;
- alg->ivsize = def->ivsize;
- alg->maxauthsize = def->maxauthsize;
- if (IS_CCM(def->flags))
- alg->setkey = qce_aead_ccm_setkey;
- else
- alg->setkey = qce_aead_setkey;
- alg->setauthsize = qce_aead_setauthsize;
- alg->encrypt = qce_aead_encrypt;
- alg->decrypt = qce_aead_decrypt;
- alg->init = qce_aead_init;
- alg->exit = qce_aead_exit;
-
- alg->base.cra_priority = 275;
- alg->base.cra_flags = CRYPTO_ALG_ASYNC |
- CRYPTO_ALG_ALLOCATES_MEMORY |
- CRYPTO_ALG_KERN_DRIVER_ONLY |
- CRYPTO_ALG_NEED_FALLBACK;
- alg->base.cra_ctxsize = sizeof(struct qce_aead_ctx);
- alg->base.cra_alignmask = 0;
- alg->base.cra_module = THIS_MODULE;
-
- INIT_LIST_HEAD(&tmpl->entry);
- tmpl->crypto_alg_type = CRYPTO_ALG_TYPE_AEAD;
- tmpl->alg_flags = def->flags;
- tmpl->qce = qce;
-
- ret = crypto_register_aead(alg);
- if (ret) {
- dev_err(qce->dev, "%s registration failed\n", alg->base.cra_name);
- kfree(tmpl);
- return ret;
- }
-
- list_add_tail(&tmpl->entry, &aead_algs);
- dev_dbg(qce->dev, "%s is registered\n", alg->base.cra_name);
- return 0;
-}
-
-static void qce_aead_unregister(struct qce_device *qce)
-{
- struct qce_alg_template *tmpl, *n;
-
- list_for_each_entry_safe(tmpl, n, &aead_algs, entry) {
- crypto_unregister_aead(&tmpl->alg.aead);
- list_del(&tmpl->entry);
- kfree(tmpl);
- }
-}
-
-static int qce_aead_register(struct qce_device *qce)
-{
- int ret, i;
-
- for (i = 0; i < ARRAY_SIZE(aead_def); i++) {
- ret = qce_aead_register_one(&aead_def[i], qce);
- if (ret)
- goto err;
- }
-
- return 0;
-err:
- qce_aead_unregister(qce);
- return ret;
-}
-
-const struct qce_algo_ops aead_ops = {
- .type = CRYPTO_ALG_TYPE_AEAD,
- .register_algs = qce_aead_register,
- .unregister_algs = qce_aead_unregister,
- .async_req_handle = qce_aead_async_req_handle,
-};
diff --git a/drivers/crypto/qce/aead.h b/drivers/crypto/qce/aead.h
deleted file mode 100644
index efb8477cc088712fe73bf7e700c206b6b9bee8ae..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/aead.h
+++ /dev/null
@@ -1,56 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2021, Linaro Limited. All rights reserved.
- */
-
-#ifndef _AEAD_H_
-#define _AEAD_H_
-
-#include "common.h"
-#include "core.h"
-
-#define QCE_MAX_KEY_SIZE 64
-#define QCE_CCM4309_SALT_SIZE 3
-
-struct qce_aead_ctx {
- u8 enc_key[QCE_MAX_KEY_SIZE];
- u8 auth_key[QCE_MAX_KEY_SIZE];
- u8 ccm4309_salt[QCE_CCM4309_SALT_SIZE];
- unsigned int enc_keylen;
- unsigned int auth_keylen;
- unsigned int authsize;
- bool need_fallback;
- struct crypto_aead *fallback;
-};
-
-struct qce_aead_reqctx {
- unsigned long flags;
- u8 *iv;
- unsigned int ivsize;
- int src_nents;
- int dst_nents;
- struct scatterlist result_sg;
- struct scatterlist adata_sg;
- struct sg_table dst_tbl;
- struct sg_table src_tbl;
- struct scatterlist *dst_sg;
- struct scatterlist *src_sg;
- unsigned int cryptlen;
- unsigned int assoclen;
- unsigned char *adata;
- u8 ccm_nonce[QCE_MAX_NONCE];
- u8 ccmresult_buf[QCE_BAM_BURST_SIZE];
- u8 ccm_rfc4309_iv[QCE_MAX_IV_SIZE];
- struct aead_request fallback_req;
-};
-
-static inline struct qce_alg_template *to_aead_tmpl(struct crypto_aead *tfm)
-{
- struct aead_alg *alg = crypto_aead_alg(tfm);
-
- return container_of(alg, struct qce_alg_template, alg.aead);
-}
-
-extern const struct qce_algo_ops aead_ops;
-
-#endif /* _AEAD_H_ */
diff --git a/drivers/crypto/qce/cipher.h b/drivers/crypto/qce/cipher.h
deleted file mode 100644
index 850f257d00f3aca0397adc1f703aea690c754d60..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/cipher.h
+++ /dev/null
@@ -1,56 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _CIPHER_H_
-#define _CIPHER_H_
-
-#include "common.h"
-#include "core.h"
-
-#define QCE_MAX_KEY_SIZE 64
-
-struct qce_cipher_ctx {
- u8 enc_key[QCE_MAX_KEY_SIZE];
- unsigned int enc_keylen;
- struct crypto_skcipher *fallback;
-};
-
-/**
- * struct qce_cipher_reqctx - holds private cipher objects per request
- * @flags: operation flags
- * @iv: pointer to the IV
- * @ivsize: IV size
- * @src_nents: source entries
- * @dst_nents: destination entries
- * @result_sg: scatterlist used for result buffer
- * @dst_tbl: destination sg table
- * @dst_sg: destination sg pointer table beginning
- * @src_tbl: source sg table
- * @src_sg: source sg pointer table beginning;
- * @cryptlen: crypto length
- */
-struct qce_cipher_reqctx {
- unsigned long flags;
- u8 *iv;
- unsigned int ivsize;
- int src_nents;
- int dst_nents;
- struct scatterlist result_sg;
- struct sg_table dst_tbl;
- struct scatterlist *dst_sg;
- struct scatterlist *src_sg;
- unsigned int cryptlen;
- struct skcipher_request fallback_req; // keep at the end
-};
-
-static inline struct qce_alg_template *to_cipher_tmpl(struct crypto_skcipher *tfm)
-{
- struct skcipher_alg *alg = crypto_skcipher_alg(tfm);
- return container_of(alg, struct qce_alg_template, alg.skcipher);
-}
-
-extern const struct qce_algo_ops skcipher_ops;
-
-#endif /* _CIPHER_H_ */
diff --git a/drivers/crypto/qce/common.c b/drivers/crypto/qce/common.c
deleted file mode 100644
index 54a78a57f63028f01870a3edeb8e390f523bb190..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/common.c
+++ /dev/null
@@ -1,595 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (c) 2012-2014, The Linux Foundation. All rights reserved.
- */
-
-#include <crypto/internal/hash.h>
-#include <linux/err.h>
-#include <linux/interrupt.h>
-#include <linux/types.h>
-#include <crypto/scatterwalk.h>
-#include <crypto/sha1.h>
-#include <crypto/sha2.h>
-
-#include "cipher.h"
-#include "common.h"
-#include "core.h"
-#include "regs-v5.h"
-#include "sha.h"
-#include "aead.h"
-
-static inline u32 qce_read(struct qce_device *qce, u32 offset)
-{
- return readl(qce->base + offset);
-}
-
-static inline void qce_write(struct qce_device *qce, u32 offset, u32 val)
-{
- writel(val, qce->base + offset);
-}
-
-static inline void qce_write_array(struct qce_device *qce, u32 offset,
- const u32 *val, unsigned int len)
-{
- int i;
-
- for (i = 0; i < len; i++)
- qce_write(qce, offset + i * sizeof(u32), val[i]);
-}
-
-static inline void
-qce_clear_array(struct qce_device *qce, u32 offset, unsigned int len)
-{
- int i;
-
- for (i = 0; i < len; i++)
- qce_write(qce, offset + i * sizeof(u32), 0);
-}
-
-static u32 qce_config_reg(struct qce_device *qce, int little)
-{
- u32 beats = (qce->burst_size >> 3) - 1;
- u32 pipe_pair = qce->pipe_pair_id;
- u32 config;
-
- config = (beats << REQ_SIZE_SHIFT) & REQ_SIZE_MASK;
- config |= BIT(MASK_DOUT_INTR_SHIFT) | BIT(MASK_DIN_INTR_SHIFT) |
- BIT(MASK_OP_DONE_INTR_SHIFT) | BIT(MASK_ERR_INTR_SHIFT);
- config |= (pipe_pair << PIPE_SET_SELECT_SHIFT) & PIPE_SET_SELECT_MASK;
- config &= ~HIGH_SPD_EN_N_SHIFT;
-
- if (little)
- config |= BIT(LITTLE_ENDIAN_MODE_SHIFT);
-
- return config;
-}
-
-void qce_cpu_to_be32p_array(__be32 *dst, const u8 *src, unsigned int len)
-{
- __be32 *d = dst;
- const u8 *s = src;
- unsigned int n;
-
- n = len / sizeof(u32);
- for (; n > 0; n--) {
- *d = cpu_to_be32p((const __u32 *) s);
- s += sizeof(__u32);
- d++;
- }
-}
-
-static void qce_setup_config(struct qce_device *qce)
-{
- u32 config;
-
- /* get big endianness */
- config = qce_config_reg(qce, 0);
-
- /* clear status */
- qce_write(qce, REG_STATUS, 0);
- qce_write(qce, REG_CONFIG, config);
-}
-
-static inline void qce_crypto_go(struct qce_device *qce, bool result_dump)
-{
- if (result_dump)
- qce_write(qce, REG_GOPROC, BIT(GO_SHIFT) | BIT(RESULTS_DUMP_SHIFT));
- else
- qce_write(qce, REG_GOPROC, BIT(GO_SHIFT));
-}
-
-#if defined(CONFIG_CRYPTO_DEV_QCE_SHA) || defined(CONFIG_CRYPTO_DEV_QCE_AEAD)
-static u32 qce_auth_cfg(unsigned long flags, u32 key_size, u32 auth_size)
-{
- u32 cfg = 0;
-
- if (IS_CCM(flags) || IS_CMAC(flags))
- cfg |= AUTH_ALG_AES << AUTH_ALG_SHIFT;
- else
- cfg |= AUTH_ALG_SHA << AUTH_ALG_SHIFT;
-
- if (IS_CCM(flags) || IS_CMAC(flags)) {
- if (key_size == AES_KEYSIZE_128)
- cfg |= AUTH_KEY_SZ_AES128 << AUTH_KEY_SIZE_SHIFT;
- else if (key_size == AES_KEYSIZE_256)
- cfg |= AUTH_KEY_SZ_AES256 << AUTH_KEY_SIZE_SHIFT;
- }
-
- if (IS_SHA1(flags) || IS_SHA1_HMAC(flags))
- cfg |= AUTH_SIZE_SHA1 << AUTH_SIZE_SHIFT;
- else if (IS_SHA256(flags) || IS_SHA256_HMAC(flags))
- cfg |= AUTH_SIZE_SHA256 << AUTH_SIZE_SHIFT;
- else if (IS_CMAC(flags))
- cfg |= AUTH_SIZE_ENUM_16_BYTES << AUTH_SIZE_SHIFT;
- else if (IS_CCM(flags))
- cfg |= (auth_size - 1) << AUTH_SIZE_SHIFT;
-
- if (IS_SHA1(flags) || IS_SHA256(flags))
- cfg |= AUTH_MODE_HASH << AUTH_MODE_SHIFT;
- else if (IS_SHA1_HMAC(flags) || IS_SHA256_HMAC(flags))
- cfg |= AUTH_MODE_HMAC << AUTH_MODE_SHIFT;
- else if (IS_CCM(flags))
- cfg |= AUTH_MODE_CCM << AUTH_MODE_SHIFT;
- else if (IS_CMAC(flags))
- cfg |= AUTH_MODE_CMAC << AUTH_MODE_SHIFT;
-
- if (IS_SHA(flags) || IS_SHA_HMAC(flags))
- cfg |= AUTH_POS_BEFORE << AUTH_POS_SHIFT;
-
- if (IS_CCM(flags))
- cfg |= QCE_MAX_NONCE_WORDS << AUTH_NONCE_NUM_WORDS_SHIFT;
-
- return cfg;
-}
-#endif
-
-#ifdef CONFIG_CRYPTO_DEV_QCE_SHA
-static int qce_setup_regs_ahash(struct crypto_async_request *async_req)
-{
- struct ahash_request *req = ahash_request_cast(async_req);
- struct crypto_ahash *ahash = __crypto_ahash_cast(async_req->tfm);
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_ahash_tmpl(async_req->tfm);
- struct qce_device *qce = tmpl->qce;
- unsigned int digestsize = crypto_ahash_digestsize(ahash);
- unsigned int blocksize = crypto_tfm_alg_blocksize(async_req->tfm);
- __be32 auth[SHA256_DIGEST_SIZE / sizeof(__be32)] = {0};
- __be32 mackey[QCE_SHA_HMAC_KEY_SIZE / sizeof(__be32)] = {0};
- u32 auth_cfg = 0, config;
- unsigned int iv_words;
-
- /* if not the last, the size has to be on the block boundary */
- if (!rctx->last_blk && req->nbytes % blocksize)
- return -EINVAL;
-
- qce_setup_config(qce);
-
- if (IS_CMAC(rctx->flags)) {
- qce_write(qce, REG_AUTH_SEG_CFG, 0);
- qce_write(qce, REG_ENCR_SEG_CFG, 0);
- qce_write(qce, REG_ENCR_SEG_SIZE, 0);
- qce_clear_array(qce, REG_AUTH_IV0, 16);
- qce_clear_array(qce, REG_AUTH_KEY0, 16);
- qce_clear_array(qce, REG_AUTH_BYTECNT0, 4);
-
- auth_cfg = qce_auth_cfg(rctx->flags, rctx->authklen, digestsize);
- }
-
- if (IS_SHA_HMAC(rctx->flags) || IS_CMAC(rctx->flags)) {
- u32 authkey_words = rctx->authklen / sizeof(u32);
-
- qce_cpu_to_be32p_array(mackey, rctx->authkey, rctx->authklen);
- qce_write_array(qce, REG_AUTH_KEY0, (u32 *)mackey,
- authkey_words);
- }
-
- if (IS_CMAC(rctx->flags))
- goto go_proc;
-
- if (rctx->first_blk)
- memcpy(auth, rctx->digest, digestsize);
- else
- qce_cpu_to_be32p_array(auth, rctx->digest, digestsize);
-
- iv_words = (IS_SHA1(rctx->flags) || IS_SHA1_HMAC(rctx->flags)) ? 5 : 8;
- qce_write_array(qce, REG_AUTH_IV0, (u32 *)auth, iv_words);
-
- if (rctx->first_blk)
- qce_clear_array(qce, REG_AUTH_BYTECNT0, 4);
- else
- qce_write_array(qce, REG_AUTH_BYTECNT0,
- (u32 *)rctx->byte_count, 2);
-
- auth_cfg = qce_auth_cfg(rctx->flags, 0, digestsize);
-
- if (rctx->last_blk)
- auth_cfg |= BIT(AUTH_LAST_SHIFT);
- else
- auth_cfg &= ~BIT(AUTH_LAST_SHIFT);
-
- if (rctx->first_blk)
- auth_cfg |= BIT(AUTH_FIRST_SHIFT);
- else
- auth_cfg &= ~BIT(AUTH_FIRST_SHIFT);
-
-go_proc:
- qce_write(qce, REG_AUTH_SEG_CFG, auth_cfg);
- qce_write(qce, REG_AUTH_SEG_SIZE, req->nbytes);
- qce_write(qce, REG_AUTH_SEG_START, 0);
- qce_write(qce, REG_ENCR_SEG_CFG, 0);
- qce_write(qce, REG_SEG_SIZE, req->nbytes);
-
- /* get little endianness */
- config = qce_config_reg(qce, 1);
- qce_write(qce, REG_CONFIG, config);
-
- qce_crypto_go(qce, true);
-
- return 0;
-}
-#endif
-
-#if defined(CONFIG_CRYPTO_DEV_QCE_SKCIPHER) || defined(CONFIG_CRYPTO_DEV_QCE_AEAD)
-static u32 qce_encr_cfg(unsigned long flags, u32 aes_key_size)
-{
- u32 cfg = 0;
-
- if (IS_AES(flags)) {
- if (aes_key_size == AES_KEYSIZE_128)
- cfg |= ENCR_KEY_SZ_AES128 << ENCR_KEY_SZ_SHIFT;
- else if (aes_key_size == AES_KEYSIZE_256)
- cfg |= ENCR_KEY_SZ_AES256 << ENCR_KEY_SZ_SHIFT;
- }
-
- if (IS_AES(flags))
- cfg |= ENCR_ALG_AES << ENCR_ALG_SHIFT;
- else if (IS_DES(flags) || IS_3DES(flags))
- cfg |= ENCR_ALG_DES << ENCR_ALG_SHIFT;
-
- if (IS_DES(flags))
- cfg |= ENCR_KEY_SZ_DES << ENCR_KEY_SZ_SHIFT;
-
- if (IS_3DES(flags))
- cfg |= ENCR_KEY_SZ_3DES << ENCR_KEY_SZ_SHIFT;
-
- switch (flags & QCE_MODE_MASK) {
- case QCE_MODE_ECB:
- cfg |= ENCR_MODE_ECB << ENCR_MODE_SHIFT;
- break;
- case QCE_MODE_CBC:
- cfg |= ENCR_MODE_CBC << ENCR_MODE_SHIFT;
- break;
- case QCE_MODE_CTR:
- cfg |= ENCR_MODE_CTR << ENCR_MODE_SHIFT;
- break;
- case QCE_MODE_XTS:
- cfg |= ENCR_MODE_XTS << ENCR_MODE_SHIFT;
- break;
- case QCE_MODE_CCM:
- cfg |= ENCR_MODE_CCM << ENCR_MODE_SHIFT;
- cfg |= LAST_CCM_XFR << LAST_CCM_SHIFT;
- break;
- default:
- return ~0;
- }
-
- return cfg;
-}
-#endif
-
-#ifdef CONFIG_CRYPTO_DEV_QCE_SKCIPHER
-static void qce_xts_swapiv(__be32 *dst, const u8 *src, unsigned int ivsize)
-{
- u8 swap[QCE_AES_IV_LENGTH] = {0};
- unsigned int i, offset;
-
- if (ivsize > QCE_AES_IV_LENGTH)
- return;
-
- offset = QCE_AES_IV_LENGTH - ivsize;
-
- /* Reverse and right-align IV bytes. */
- for (i = 0; i < ivsize; i++)
- swap[offset + i] = src[ivsize - 1 - i];
-
- qce_cpu_to_be32p_array(dst, swap, QCE_AES_IV_LENGTH);
-}
-
-static void qce_xtskey(struct qce_device *qce, const u8 *enckey,
- unsigned int enckeylen, unsigned int cryptlen)
-{
- u32 xtskey[QCE_MAX_CIPHER_KEY_SIZE / sizeof(u32)] = {0};
- unsigned int xtsklen = enckeylen / (2 * sizeof(u32));
-
- qce_cpu_to_be32p_array((__be32 *)xtskey, enckey + enckeylen / 2,
- enckeylen / 2);
- qce_write_array(qce, REG_ENCR_XTS_KEY0, xtskey, xtsklen);
-
- /* Set data unit size to cryptlen. Anything else causes
- * crypto engine to return back incorrect results.
- */
- qce_write(qce, REG_ENCR_XTS_DU_SIZE, cryptlen);
-}
-
-static int qce_setup_regs_skcipher(struct crypto_async_request *async_req)
-{
- struct skcipher_request *req = skcipher_request_cast(async_req);
- struct qce_cipher_reqctx *rctx = skcipher_request_ctx(req);
- struct qce_cipher_ctx *ctx = crypto_tfm_ctx(async_req->tfm);
- struct qce_alg_template *tmpl = to_cipher_tmpl(crypto_skcipher_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- __be32 enckey[QCE_MAX_CIPHER_KEY_SIZE / sizeof(__be32)] = {0};
- __be32 enciv[QCE_MAX_IV_SIZE / sizeof(__be32)] = {0};
- unsigned int enckey_words, enciv_words;
- unsigned int keylen;
- u32 encr_cfg = 0, auth_cfg = 0, config;
- unsigned int ivsize = rctx->ivsize;
- unsigned long flags = rctx->flags;
-
- qce_setup_config(qce);
-
- if (IS_XTS(flags))
- keylen = ctx->enc_keylen / 2;
- else
- keylen = ctx->enc_keylen;
-
- qce_cpu_to_be32p_array(enckey, ctx->enc_key, keylen);
- enckey_words = keylen / sizeof(u32);
-
- qce_write(qce, REG_AUTH_SEG_CFG, auth_cfg);
-
- encr_cfg = qce_encr_cfg(flags, keylen);
-
- if (IS_DES(flags)) {
- enciv_words = 2;
- enckey_words = 2;
- } else if (IS_3DES(flags)) {
- enciv_words = 2;
- enckey_words = 6;
- } else if (IS_AES(flags)) {
- if (IS_XTS(flags))
- qce_xtskey(qce, ctx->enc_key, ctx->enc_keylen,
- rctx->cryptlen);
- enciv_words = 4;
- } else {
- return -EINVAL;
- }
-
- qce_write_array(qce, REG_ENCR_KEY0, (u32 *)enckey, enckey_words);
-
- if (!IS_ECB(flags)) {
- if (IS_XTS(flags))
- qce_xts_swapiv(enciv, rctx->iv, ivsize);
- else
- qce_cpu_to_be32p_array(enciv, rctx->iv, ivsize);
-
- qce_write_array(qce, REG_CNTR0_IV0, (u32 *)enciv, enciv_words);
- }
-
- if (IS_ENCRYPT(flags))
- encr_cfg |= BIT(ENCODE_SHIFT);
-
- qce_write(qce, REG_ENCR_SEG_CFG, encr_cfg);
- qce_write(qce, REG_ENCR_SEG_SIZE, rctx->cryptlen);
- qce_write(qce, REG_ENCR_SEG_START, 0);
-
- if (IS_CTR(flags)) {
- qce_write(qce, REG_CNTR_MASK, ~0);
- qce_write(qce, REG_CNTR_MASK0, ~0);
- qce_write(qce, REG_CNTR_MASK1, ~0);
- qce_write(qce, REG_CNTR_MASK2, ~0);
- }
-
- qce_write(qce, REG_SEG_SIZE, rctx->cryptlen);
-
- /* get little endianness */
- config = qce_config_reg(qce, 1);
- qce_write(qce, REG_CONFIG, config);
-
- qce_crypto_go(qce, true);
-
- return 0;
-}
-#endif
-
-#ifdef CONFIG_CRYPTO_DEV_QCE_AEAD
-static const u32 std_iv_sha1[SHA256_DIGEST_SIZE / sizeof(u32)] = {
- SHA1_H0, SHA1_H1, SHA1_H2, SHA1_H3, SHA1_H4, 0, 0, 0
-};
-
-static const u32 std_iv_sha256[SHA256_DIGEST_SIZE / sizeof(u32)] = {
- SHA256_H0, SHA256_H1, SHA256_H2, SHA256_H3,
- SHA256_H4, SHA256_H5, SHA256_H6, SHA256_H7
-};
-
-static unsigned int qce_be32_to_cpu_array(u32 *dst, const u8 *src, unsigned int len)
-{
- u32 *d = dst;
- const u8 *s = src;
- unsigned int n;
-
- n = len / sizeof(u32);
- for (; n > 0; n--) {
- *d = be32_to_cpup((const __be32 *)s);
- s += sizeof(u32);
- d++;
- }
- return DIV_ROUND_UP(len, sizeof(u32));
-}
-
-static int qce_setup_regs_aead(struct crypto_async_request *async_req)
-{
- struct aead_request *req = aead_request_cast(async_req);
- struct qce_aead_reqctx *rctx = aead_request_ctx_dma(req);
- struct qce_aead_ctx *ctx = crypto_tfm_ctx(async_req->tfm);
- struct qce_alg_template *tmpl = to_aead_tmpl(crypto_aead_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- u32 enckey[QCE_MAX_CIPHER_KEY_SIZE / sizeof(u32)] = {0};
- u32 enciv[QCE_MAX_IV_SIZE / sizeof(u32)] = {0};
- u32 authkey[QCE_SHA_HMAC_KEY_SIZE / sizeof(u32)] = {0};
- u32 authiv[SHA256_DIGEST_SIZE / sizeof(u32)] = {0};
- u32 authnonce[QCE_MAX_NONCE / sizeof(u32)] = {0};
- unsigned int enc_keylen = ctx->enc_keylen;
- unsigned int auth_keylen = ctx->auth_keylen;
- unsigned int enc_ivsize = rctx->ivsize;
- unsigned int auth_ivsize = 0;
- unsigned int enckey_words, enciv_words;
- unsigned int authkey_words, authiv_words, authnonce_words;
- unsigned long flags = rctx->flags;
- u32 encr_cfg, auth_cfg, config, totallen;
- u32 iv_last_word;
-
- qce_setup_config(qce);
-
- /* Write encryption key */
- enckey_words = qce_be32_to_cpu_array(enckey, ctx->enc_key, enc_keylen);
- qce_write_array(qce, REG_ENCR_KEY0, enckey, enckey_words);
-
- /* Write encryption iv */
- enciv_words = qce_be32_to_cpu_array(enciv, rctx->iv, enc_ivsize);
- qce_write_array(qce, REG_CNTR0_IV0, enciv, enciv_words);
-
- if (IS_CCM(rctx->flags)) {
- iv_last_word = enciv[enciv_words - 1];
- qce_write(qce, REG_CNTR3_IV3, iv_last_word + 1);
- qce_write_array(qce, REG_ENCR_CCM_INT_CNTR0, (u32 *)enciv, enciv_words);
- qce_write(qce, REG_CNTR_MASK, ~0);
- qce_write(qce, REG_CNTR_MASK0, ~0);
- qce_write(qce, REG_CNTR_MASK1, ~0);
- qce_write(qce, REG_CNTR_MASK2, ~0);
- }
-
- /* Clear authentication IV and KEY registers of previous values */
- qce_clear_array(qce, REG_AUTH_IV0, 16);
- qce_clear_array(qce, REG_AUTH_KEY0, 16);
-
- /* Clear byte count */
- qce_clear_array(qce, REG_AUTH_BYTECNT0, 4);
-
- /* Write authentication key */
- authkey_words = qce_be32_to_cpu_array(authkey, ctx->auth_key, auth_keylen);
- qce_write_array(qce, REG_AUTH_KEY0, (u32 *)authkey, authkey_words);
-
- /* Write initial authentication IV only for HMAC algorithms */
- if (IS_SHA_HMAC(rctx->flags)) {
- /* Write default authentication iv */
- if (IS_SHA1_HMAC(rctx->flags)) {
- auth_ivsize = SHA1_DIGEST_SIZE;
- memcpy(authiv, std_iv_sha1, auth_ivsize);
- } else if (IS_SHA256_HMAC(rctx->flags)) {
- auth_ivsize = SHA256_DIGEST_SIZE;
- memcpy(authiv, std_iv_sha256, auth_ivsize);
- }
- authiv_words = auth_ivsize / sizeof(u32);
- qce_write_array(qce, REG_AUTH_IV0, (u32 *)authiv, authiv_words);
- } else if (IS_CCM(rctx->flags)) {
- /* Write nonce for CCM algorithms */
- authnonce_words = qce_be32_to_cpu_array(authnonce, rctx->ccm_nonce, QCE_MAX_NONCE);
- qce_write_array(qce, REG_AUTH_INFO_NONCE0, authnonce, authnonce_words);
- }
-
- /* Set up ENCR_SEG_CFG */
- encr_cfg = qce_encr_cfg(flags, enc_keylen);
- if (IS_ENCRYPT(flags))
- encr_cfg |= BIT(ENCODE_SHIFT);
- qce_write(qce, REG_ENCR_SEG_CFG, encr_cfg);
-
- /* Set up AUTH_SEG_CFG */
- auth_cfg = qce_auth_cfg(rctx->flags, auth_keylen, ctx->authsize);
- auth_cfg |= BIT(AUTH_LAST_SHIFT);
- auth_cfg |= BIT(AUTH_FIRST_SHIFT);
- if (IS_ENCRYPT(flags)) {
- if (IS_CCM(rctx->flags))
- auth_cfg |= AUTH_POS_BEFORE << AUTH_POS_SHIFT;
- else
- auth_cfg |= AUTH_POS_AFTER << AUTH_POS_SHIFT;
- } else {
- if (IS_CCM(rctx->flags))
- auth_cfg |= AUTH_POS_AFTER << AUTH_POS_SHIFT;
- else
- auth_cfg |= AUTH_POS_BEFORE << AUTH_POS_SHIFT;
- }
- qce_write(qce, REG_AUTH_SEG_CFG, auth_cfg);
-
- totallen = rctx->cryptlen + rctx->assoclen;
-
- /* Set the encryption size and start offset */
- if (IS_CCM(rctx->flags) && IS_DECRYPT(rctx->flags))
- qce_write(qce, REG_ENCR_SEG_SIZE, rctx->cryptlen + ctx->authsize);
- else
- qce_write(qce, REG_ENCR_SEG_SIZE, rctx->cryptlen);
- qce_write(qce, REG_ENCR_SEG_START, rctx->assoclen & 0xffff);
-
- /* Set the authentication size and start offset */
- qce_write(qce, REG_AUTH_SEG_SIZE, totallen);
- qce_write(qce, REG_AUTH_SEG_START, 0);
-
- /* Write total length */
- if (IS_CCM(rctx->flags) && IS_DECRYPT(rctx->flags))
- qce_write(qce, REG_SEG_SIZE, totallen + ctx->authsize);
- else
- qce_write(qce, REG_SEG_SIZE, totallen);
-
- /* get little endianness */
- config = qce_config_reg(qce, 1);
- qce_write(qce, REG_CONFIG, config);
-
- /* Start the process */
- qce_crypto_go(qce, !IS_CCM(flags));
-
- return 0;
-}
-#endif
-
-int qce_start(struct crypto_async_request *async_req, u32 type)
-{
- switch (type) {
-#ifdef CONFIG_CRYPTO_DEV_QCE_SKCIPHER
- case CRYPTO_ALG_TYPE_SKCIPHER:
- return qce_setup_regs_skcipher(async_req);
-#endif
-#ifdef CONFIG_CRYPTO_DEV_QCE_SHA
- case CRYPTO_ALG_TYPE_AHASH:
- return qce_setup_regs_ahash(async_req);
-#endif
-#ifdef CONFIG_CRYPTO_DEV_QCE_AEAD
- case CRYPTO_ALG_TYPE_AEAD:
- return qce_setup_regs_aead(async_req);
-#endif
- default:
- return -EINVAL;
- }
-}
-
-#define STATUS_ERRORS \
- (BIT(SW_ERR_SHIFT) | BIT(AXI_ERR_SHIFT) | BIT(HSD_ERR_SHIFT))
-
-int qce_check_status(struct qce_device *qce, u32 *status)
-{
- int ret = 0;
-
- *status = qce_read(qce, REG_STATUS);
-
- /*
- * Don't use result dump status. The operation may not be complete.
- * Instead, use the status we just read from device. In case, we need to
- * use result_status from result dump the result_status needs to be byte
- * swapped, since we set the device to little endian.
- */
- if (*status & STATUS_ERRORS || !(*status & BIT(OPERATION_DONE_SHIFT)))
- ret = -ENXIO;
- else if (*status & BIT(MAC_FAILED_SHIFT))
- ret = -EBADMSG;
-
- return ret;
-}
-
-void qce_get_version(struct qce_device *qce, u32 *major, u32 *minor, u32 *step)
-{
- u32 val;
-
- val = qce_read(qce, REG_VERSION);
- *major = (val & CORE_MAJOR_REV_MASK) >> CORE_MAJOR_REV_SHIFT;
- *minor = (val & CORE_MINOR_REV_MASK) >> CORE_MINOR_REV_SHIFT;
- *step = (val & CORE_STEP_REV_MASK) >> CORE_STEP_REV_SHIFT;
-}
diff --git a/drivers/crypto/qce/common.h b/drivers/crypto/qce/common.h
deleted file mode 100644
index 02e63ad9f24557c2238caa70b0ec521d49da4f13..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/common.h
+++ /dev/null
@@ -1,104 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _COMMON_H_
-#define _COMMON_H_
-
-#include <linux/crypto.h>
-#include <linux/types.h>
-#include <crypto/aes.h>
-#include <crypto/hash.h>
-#include <crypto/internal/skcipher.h>
-#include <crypto/internal/aead.h>
-
-/* xts du size */
-#define QCE_SECTOR_SIZE 512
-
-/* key size in bytes */
-#define QCE_SHA_HMAC_KEY_SIZE 64
-#define QCE_MAX_CIPHER_KEY_SIZE AES_KEYSIZE_256
-
-/* IV length in bytes */
-#define QCE_AES_IV_LENGTH AES_BLOCK_SIZE
-/* max of AES_BLOCK_SIZE, DES3_EDE_BLOCK_SIZE */
-#define QCE_MAX_IV_SIZE AES_BLOCK_SIZE
-
-/* maximum nonce bytes */
-#define QCE_MAX_NONCE 16
-#define QCE_MAX_NONCE_WORDS (QCE_MAX_NONCE / sizeof(u32))
-
-/* burst size alignment requirement */
-#define QCE_MAX_ALIGN_SIZE 64
-
-/* cipher algorithms */
-#define QCE_ALG_DES BIT(0)
-#define QCE_ALG_3DES BIT(1)
-#define QCE_ALG_AES BIT(2)
-
-/* hash and hmac algorithms */
-#define QCE_HASH_SHA1 BIT(3)
-#define QCE_HASH_SHA256 BIT(4)
-#define QCE_HASH_SHA1_HMAC BIT(5)
-#define QCE_HASH_SHA256_HMAC BIT(6)
-#define QCE_HASH_AES_CMAC BIT(7)
-
-/* cipher modes */
-#define QCE_MODE_CBC BIT(8)
-#define QCE_MODE_ECB BIT(9)
-#define QCE_MODE_CTR BIT(10)
-#define QCE_MODE_XTS BIT(11)
-#define QCE_MODE_CCM BIT(12)
-#define QCE_MODE_MASK GENMASK(12, 8)
-
-#define QCE_MODE_CCM_RFC4309 BIT(13)
-
-/* cipher encryption/decryption operations */
-#define QCE_ENCRYPT BIT(30)
-#define QCE_DECRYPT BIT(31)
-
-#define IS_DES(flags) (flags & QCE_ALG_DES)
-#define IS_3DES(flags) (flags & QCE_ALG_3DES)
-#define IS_AES(flags) (flags & QCE_ALG_AES)
-
-#define IS_SHA1(flags) (flags & QCE_HASH_SHA1)
-#define IS_SHA256(flags) (flags & QCE_HASH_SHA256)
-#define IS_SHA1_HMAC(flags) (flags & QCE_HASH_SHA1_HMAC)
-#define IS_SHA256_HMAC(flags) (flags & QCE_HASH_SHA256_HMAC)
-#define IS_CMAC(flags) (flags & QCE_HASH_AES_CMAC)
-#define IS_SHA(flags) (IS_SHA1(flags) || IS_SHA256(flags))
-#define IS_SHA_HMAC(flags) \
- (IS_SHA1_HMAC(flags) || IS_SHA256_HMAC(flags))
-
-#define IS_CBC(mode) (mode & QCE_MODE_CBC)
-#define IS_ECB(mode) (mode & QCE_MODE_ECB)
-#define IS_CTR(mode) (mode & QCE_MODE_CTR)
-#define IS_XTS(mode) (mode & QCE_MODE_XTS)
-#define IS_CCM(mode) (mode & QCE_MODE_CCM)
-#define IS_CCM_RFC4309(mode) ((mode) & QCE_MODE_CCM_RFC4309)
-
-#define IS_ENCRYPT(dir) (dir & QCE_ENCRYPT)
-#define IS_DECRYPT(dir) (dir & QCE_DECRYPT)
-
-struct qce_alg_template {
- struct list_head entry;
- u32 crypto_alg_type;
- unsigned long alg_flags;
- const u32 *std_iv;
- union {
- struct skcipher_alg skcipher;
- struct ahash_alg ahash;
- struct aead_alg aead;
- } alg;
- struct qce_device *qce;
- const u8 *hash_zero;
- const u32 digest_size;
-};
-
-void qce_cpu_to_be32p_array(__be32 *dst, const u8 *src, unsigned int len);
-int qce_check_status(struct qce_device *qce, u32 *status);
-void qce_get_version(struct qce_device *qce, u32 *major, u32 *minor, u32 *step);
-int qce_start(struct crypto_async_request *async_req, u32 type);
-
-#endif /* _COMMON_H_ */
diff --git a/drivers/crypto/qce/core.c b/drivers/crypto/qce/core.c
deleted file mode 100644
index b966f3365b7de8d2a8f6707397a34aa4facdc4ac..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/core.c
+++ /dev/null
@@ -1,271 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#include <linux/cleanup.h>
-#include <linux/clk.h>
-#include <linux/device.h>
-#include <linux/dma-mapping.h>
-#include <linux/interconnect.h>
-#include <linux/interrupt.h>
-#include <linux/module.h>
-#include <linux/mod_devicetable.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <crypto/algapi.h>
-#include <crypto/internal/hash.h>
-
-#include "core.h"
-#include "cipher.h"
-#include "sha.h"
-#include "aead.h"
-
-#define QCE_QUEUE_LENGTH 1
-
-#define QCE_DEFAULT_MEM_BANDWIDTH 393600
-
-static const struct qce_algo_ops *qce_ops[] = {
-#ifdef CONFIG_CRYPTO_DEV_QCE_SKCIPHER
- &skcipher_ops,
-#endif
-#ifdef CONFIG_CRYPTO_DEV_QCE_SHA
- &ahash_ops,
-#endif
-#ifdef CONFIG_CRYPTO_DEV_QCE_AEAD
- &aead_ops,
-#endif
-};
-
-static void qce_unregister_algs(void *data)
-{
- const struct qce_algo_ops *ops;
- struct qce_device *qce = data;
- int i;
-
- for (i = 0; i < ARRAY_SIZE(qce_ops); i++) {
- ops = qce_ops[i];
- ops->unregister_algs(qce);
- }
-}
-
-static int devm_qce_register_algs(struct qce_device *qce)
-{
- const struct qce_algo_ops *ops;
- int i, j, ret = -ENODEV;
-
- for (i = 0; i < ARRAY_SIZE(qce_ops); i++) {
- ops = qce_ops[i];
- ret = ops->register_algs(qce);
- if (ret) {
- for (j = i - 1; j >= 0; j--)
- ops->unregister_algs(qce);
- return ret;
- }
- }
-
- return devm_add_action_or_reset(qce->dev, qce_unregister_algs, qce);
-}
-
-static int qce_handle_request(struct crypto_async_request *async_req)
-{
- int ret = -EINVAL, i;
- const struct qce_algo_ops *ops;
- u32 type = crypto_tfm_alg_type(async_req->tfm);
-
- for (i = 0; i < ARRAY_SIZE(qce_ops); i++) {
- ops = qce_ops[i];
- if (type != ops->type)
- continue;
- ret = ops->async_req_handle(async_req);
- break;
- }
-
- return ret;
-}
-
-static int qce_handle_queue(struct qce_device *qce,
- struct crypto_async_request *req)
-{
- struct crypto_async_request *async_req, *backlog;
- int ret = 0, err;
-
- scoped_guard(mutex, &qce->lock) {
- if (req)
- ret = crypto_enqueue_request(&qce->queue, req);
-
- /* busy, do not dequeue request */
- if (qce->req)
- return ret;
-
- backlog = crypto_get_backlog(&qce->queue);
- async_req = crypto_dequeue_request(&qce->queue);
- if (async_req)
- qce->req = async_req;
- }
-
- if (!async_req)
- return ret;
-
- if (backlog) {
- scoped_guard(mutex, &qce->lock)
- crypto_request_complete(backlog, -EINPROGRESS);
- }
-
- err = qce_handle_request(async_req);
- if (err) {
- qce->result = err;
- schedule_work(&qce->done_work);
- }
-
- return ret;
-}
-
-static void qce_req_done_work(struct work_struct *work)
-{
- struct qce_device *qce = container_of(work, struct qce_device,
- done_work);
- struct crypto_async_request *req;
-
- scoped_guard(mutex, &qce->lock) {
- req = qce->req;
- qce->req = NULL;
- }
-
- if (req)
- crypto_request_complete(req, qce->result);
-
- qce_handle_queue(qce, NULL);
-}
-
-static int qce_async_request_enqueue(struct qce_device *qce,
- struct crypto_async_request *req)
-{
- return qce_handle_queue(qce, req);
-}
-
-static void qce_async_request_done(struct qce_device *qce, int ret)
-{
- qce->result = ret;
- schedule_work(&qce->done_work);
-}
-
-static int qce_check_version(struct qce_device *qce)
-{
- u32 major, minor, step;
-
- qce_get_version(qce, &major, &minor, &step);
-
- /*
- * the driver does not support v5 with minor 0 because it has special
- * alignment requirements.
- */
- if (major == 5 && minor == 0)
- return -ENODEV;
-
- qce->burst_size = QCE_BAM_BURST_SIZE;
-
- /*
- * Rx and tx pipes are treated as a pair inside CE.
- * Pipe pair number depends on the actual BAM dma pipe
- * that is used for transfers. The BAM dma pipes are passed
- * from the device tree and used to derive the pipe pair
- * id in the CE driver as follows.
- * BAM dma pipes(rx, tx) CE pipe pair id
- * 0,1 0
- * 2,3 1
- * 4,5 2
- * 6,7 3
- * ...
- */
- qce->pipe_pair_id = qce->dma.rxchan->chan_id >> 1;
-
- dev_dbg(qce->dev, "Crypto device found, version %d.%d.%d\n",
- major, minor, step);
-
- return 0;
-}
-
-static int qce_crypto_probe(struct platform_device *pdev)
-{
- struct device *dev = &pdev->dev;
- struct qce_device *qce;
- int ret;
-
- qce = devm_kzalloc(dev, sizeof(*qce), GFP_KERNEL);
- if (!qce)
- return -ENOMEM;
-
- qce->dev = dev;
- platform_set_drvdata(pdev, qce);
-
- qce->base = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(qce->base))
- return PTR_ERR(qce->base);
-
- ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
- if (ret < 0)
- return ret;
-
- qce->core = devm_clk_get_optional_enabled(qce->dev, "core");
- if (IS_ERR(qce->core))
- return PTR_ERR(qce->core);
-
- qce->iface = devm_clk_get_optional_enabled(qce->dev, "iface");
- if (IS_ERR(qce->iface))
- return PTR_ERR(qce->iface);
-
- qce->bus = devm_clk_get_optional_enabled(qce->dev, "bus");
- if (IS_ERR(qce->bus))
- return PTR_ERR(qce->bus);
-
- qce->mem_path = devm_of_icc_get(qce->dev, "memory");
- if (IS_ERR(qce->mem_path))
- return PTR_ERR(qce->mem_path);
-
- ret = icc_set_bw(qce->mem_path, QCE_DEFAULT_MEM_BANDWIDTH, QCE_DEFAULT_MEM_BANDWIDTH);
- if (ret)
- return ret;
-
- ret = devm_qce_dma_request(qce->dev, &qce->dma);
- if (ret)
- return ret;
-
- ret = qce_check_version(qce);
- if (ret)
- return ret;
-
- ret = devm_mutex_init(qce->dev, &qce->lock);
- if (ret)
- return ret;
-
- INIT_WORK(&qce->done_work, qce_req_done_work);
- crypto_init_queue(&qce->queue, QCE_QUEUE_LENGTH);
-
- qce->async_req_enqueue = qce_async_request_enqueue;
- qce->async_req_done = qce_async_request_done;
-
- return devm_qce_register_algs(qce);
-}
-
-static const struct of_device_id qce_crypto_of_match[] = {
- { .compatible = "qcom,crypto-v5.1", },
- { .compatible = "qcom,crypto-v5.4", },
- { .compatible = "qcom,qce", },
- {}
-};
-MODULE_DEVICE_TABLE(of, qce_crypto_of_match);
-
-static struct platform_driver qce_crypto_driver = {
- .probe = qce_crypto_probe,
- .driver = {
- .name = KBUILD_MODNAME,
- .of_match_table = qce_crypto_of_match,
- },
-};
-module_platform_driver(qce_crypto_driver);
-
-MODULE_LICENSE("GPL v2");
-MODULE_DESCRIPTION("Qualcomm crypto engine driver");
-MODULE_ALIAS("platform:" KBUILD_MODNAME);
-MODULE_AUTHOR("The Linux Foundation");
diff --git a/drivers/crypto/qce/core.h b/drivers/crypto/qce/core.h
deleted file mode 100644
index eb6fa7a8b64a81daf9ad5304a3ae4e5e597a70b8..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/core.h
+++ /dev/null
@@ -1,64 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _CORE_H_
-#define _CORE_H_
-
-#include <linux/mutex.h>
-#include <linux/workqueue.h>
-
-#include "dma.h"
-
-/**
- * struct qce_device - crypto engine device structure
- * @queue: crypto request queue
- * @lock: the lock protects queue and req
- * @done_work: workqueue context
- * @req: current active request
- * @result: result of current transform
- * @base: virtual IO base
- * @dev: pointer to device structure
- * @core: core device clock
- * @iface: interface clock
- * @bus: bus clock
- * @dma: pointer to dma data
- * @burst_size: the crypto burst size
- * @pipe_pair_id: which pipe pair id the device using
- * @async_req_enqueue: invoked by every algorithm to enqueue a request
- * @async_req_done: invoked by every algorithm to finish its request
- */
-struct qce_device {
- struct crypto_queue queue;
- struct mutex lock;
- struct work_struct done_work;
- struct crypto_async_request *req;
- int result;
- void __iomem *base;
- struct device *dev;
- struct clk *core, *iface, *bus;
- struct icc_path *mem_path;
- struct qce_dma_data dma;
- int burst_size;
- unsigned int pipe_pair_id;
- int (*async_req_enqueue)(struct qce_device *qce,
- struct crypto_async_request *req);
- void (*async_req_done)(struct qce_device *qce, int ret);
-};
-
-/**
- * struct qce_algo_ops - algorithm operations per crypto type
- * @type: should be CRYPTO_ALG_TYPE_XXX
- * @register_algs: invoked by core to register the algorithms
- * @unregister_algs: invoked by core to unregister the algorithms
- * @async_req_handle: invoked by core to handle enqueued request
- */
-struct qce_algo_ops {
- u32 type;
- int (*register_algs)(struct qce_device *qce);
- void (*unregister_algs)(struct qce_device *qce);
- int (*async_req_handle)(struct crypto_async_request *async_req);
-};
-
-#endif /* _CORE_H_ */
diff --git a/drivers/crypto/qce/dma.c b/drivers/crypto/qce/dma.c
deleted file mode 100644
index 68cafd4741ad3d91906d39e817fc7873b028d498..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/dma.c
+++ /dev/null
@@ -1,135 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (c) 2012-2014, The Linux Foundation. All rights reserved.
- */
-
-#include <linux/device.h>
-#include <linux/dmaengine.h>
-#include <crypto/scatterwalk.h>
-
-#include "dma.h"
-
-static void qce_dma_release(void *data)
-{
- struct qce_dma_data *dma = data;
-
- dma_release_channel(dma->txchan);
- dma_release_channel(dma->rxchan);
- kfree(dma->result_buf);
-}
-
-int devm_qce_dma_request(struct device *dev, struct qce_dma_data *dma)
-{
- int ret;
-
- dma->txchan = dma_request_chan(dev, "tx");
- if (IS_ERR(dma->txchan))
- return dev_err_probe(dev, PTR_ERR(dma->txchan),
- "Failed to get TX DMA channel\n");
-
- dma->rxchan = dma_request_chan(dev, "rx");
- if (IS_ERR(dma->rxchan)) {
- ret = dev_err_probe(dev, PTR_ERR(dma->rxchan),
- "Failed to get RX DMA channel\n");
- goto error_rx;
- }
-
- dma->result_buf = kmalloc(QCE_RESULT_BUF_SZ + QCE_IGNORE_BUF_SZ,
- GFP_KERNEL);
- if (!dma->result_buf) {
- ret = -ENOMEM;
- goto error_nomem;
- }
-
- dma->ignore_buf = dma->result_buf + QCE_RESULT_BUF_SZ;
-
- return devm_add_action_or_reset(dev, qce_dma_release, dma);
-
-error_nomem:
- dma_release_channel(dma->rxchan);
-error_rx:
- dma_release_channel(dma->txchan);
- return ret;
-}
-
-struct scatterlist *
-qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,
- unsigned int max_len)
-{
- struct scatterlist *sg = sgt->sgl, *sg_last = NULL;
- unsigned int new_len;
-
- while (sg) {
- if (!sg_page(sg))
- break;
- sg = sg_next(sg);
- }
-
- if (!sg)
- return ERR_PTR(-EINVAL);
-
- while (new_sgl && sg && max_len) {
- new_len = new_sgl->length > max_len ? max_len : new_sgl->length;
- sg_set_page(sg, sg_page(new_sgl), new_len, new_sgl->offset);
- sg_last = sg;
- sg = sg_next(sg);
- new_sgl = sg_next(new_sgl);
- max_len -= new_len;
- }
-
- return sg_last;
-}
-
-static int qce_dma_prep_sg(struct dma_chan *chan, struct scatterlist *sg,
- int nents, unsigned long flags,
- enum dma_transfer_direction dir,
- dma_async_tx_callback cb, void *cb_param)
-{
- struct dma_async_tx_descriptor *desc;
- dma_cookie_t cookie;
-
- if (!sg || !nents)
- return -EINVAL;
-
- desc = dmaengine_prep_slave_sg(chan, sg, nents, dir, flags);
- if (!desc)
- return -EINVAL;
-
- desc->callback = cb;
- desc->callback_param = cb_param;
- cookie = dmaengine_submit(desc);
-
- return dma_submit_error(cookie);
-}
-
-int qce_dma_prep_sgs(struct qce_dma_data *dma, struct scatterlist *rx_sg,
- int rx_nents, struct scatterlist *tx_sg, int tx_nents,
- dma_async_tx_callback cb, void *cb_param)
-{
- struct dma_chan *rxchan = dma->rxchan;
- struct dma_chan *txchan = dma->txchan;
- unsigned long flags = DMA_PREP_INTERRUPT | DMA_CTRL_ACK;
- int ret;
-
- ret = qce_dma_prep_sg(rxchan, rx_sg, rx_nents, flags, DMA_MEM_TO_DEV,
- NULL, NULL);
- if (ret)
- return ret;
-
- return qce_dma_prep_sg(txchan, tx_sg, tx_nents, flags, DMA_DEV_TO_MEM,
- cb, cb_param);
-}
-
-void qce_dma_issue_pending(struct qce_dma_data *dma)
-{
- dma_async_issue_pending(dma->rxchan);
- dma_async_issue_pending(dma->txchan);
-}
-
-int qce_dma_terminate_all(struct qce_dma_data *dma)
-{
- int ret;
-
- ret = dmaengine_terminate_all(dma->rxchan);
- return ret ?: dmaengine_terminate_all(dma->txchan);
-}
diff --git a/drivers/crypto/qce/dma.h b/drivers/crypto/qce/dma.h
deleted file mode 100644
index 31629185000e12242fa07c2cc08b95fcbd5d4b8c..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/dma.h
+++ /dev/null
@@ -1,47 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2011-2014, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _DMA_H_
-#define _DMA_H_
-
-#include <linux/dmaengine.h>
-
-/* maximum data transfer block size between BAM and CE */
-#define QCE_BAM_BURST_SIZE 64
-
-#define QCE_AUTHIV_REGS_CNT 16
-#define QCE_AUTH_BYTECOUNT_REGS_CNT 4
-#define QCE_CNTRIV_REGS_CNT 4
-
-struct qce_result_dump {
- u32 auth_iv[QCE_AUTHIV_REGS_CNT];
- u32 auth_byte_count[QCE_AUTH_BYTECOUNT_REGS_CNT];
- u32 encr_cntr_iv[QCE_CNTRIV_REGS_CNT];
- u32 status;
- u32 status2;
-};
-
-#define QCE_IGNORE_BUF_SZ (2 * QCE_BAM_BURST_SIZE)
-#define QCE_RESULT_BUF_SZ \
- ALIGN(sizeof(struct qce_result_dump), QCE_BAM_BURST_SIZE)
-
-struct qce_dma_data {
- struct dma_chan *txchan;
- struct dma_chan *rxchan;
- struct qce_result_dump *result_buf;
- void *ignore_buf;
-};
-
-int devm_qce_dma_request(struct device *dev, struct qce_dma_data *dma);
-int qce_dma_prep_sgs(struct qce_dma_data *dma, struct scatterlist *sg_in,
- int in_ents, struct scatterlist *sg_out, int out_ents,
- dma_async_tx_callback cb, void *cb_param);
-void qce_dma_issue_pending(struct qce_dma_data *dma);
-int qce_dma_terminate_all(struct qce_dma_data *dma);
-struct scatterlist *
-qce_sgtable_add(struct sg_table *sgt, struct scatterlist *sg_add,
- unsigned int max_len);
-
-#endif /* _DMA_H_ */
diff --git a/drivers/crypto/qce/regs-v5.h b/drivers/crypto/qce/regs-v5.h
deleted file mode 100644
index d59ed279890621a8e2e6f4cdb20692dbf39f1461..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/regs-v5.h
+++ /dev/null
@@ -1,326 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2012-2014, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _REGS_V5_H_
-#define _REGS_V5_H_
-
-#include <linux/bitops.h>
-
-#define REG_VERSION 0x000
-#define REG_STATUS 0x100
-#define REG_STATUS2 0x104
-#define REG_ENGINES_AVAIL 0x108
-#define REG_FIFO_SIZES 0x10c
-#define REG_SEG_SIZE 0x110
-#define REG_GOPROC 0x120
-#define REG_ENCR_SEG_CFG 0x200
-#define REG_ENCR_SEG_SIZE 0x204
-#define REG_ENCR_SEG_START 0x208
-#define REG_CNTR0_IV0 0x20c
-#define REG_CNTR1_IV1 0x210
-#define REG_CNTR2_IV2 0x214
-#define REG_CNTR3_IV3 0x218
-#define REG_CNTR_MASK 0x21C
-#define REG_ENCR_CCM_INT_CNTR0 0x220
-#define REG_ENCR_CCM_INT_CNTR1 0x224
-#define REG_ENCR_CCM_INT_CNTR2 0x228
-#define REG_ENCR_CCM_INT_CNTR3 0x22c
-#define REG_ENCR_XTS_DU_SIZE 0x230
-#define REG_CNTR_MASK2 0x234
-#define REG_CNTR_MASK1 0x238
-#define REG_CNTR_MASK0 0x23c
-#define REG_AUTH_SEG_CFG 0x300
-#define REG_AUTH_SEG_SIZE 0x304
-#define REG_AUTH_SEG_START 0x308
-#define REG_AUTH_IV0 0x310
-#define REG_AUTH_IV1 0x314
-#define REG_AUTH_IV2 0x318
-#define REG_AUTH_IV3 0x31c
-#define REG_AUTH_IV4 0x320
-#define REG_AUTH_IV5 0x324
-#define REG_AUTH_IV6 0x328
-#define REG_AUTH_IV7 0x32c
-#define REG_AUTH_IV8 0x330
-#define REG_AUTH_IV9 0x334
-#define REG_AUTH_IV10 0x338
-#define REG_AUTH_IV11 0x33c
-#define REG_AUTH_IV12 0x340
-#define REG_AUTH_IV13 0x344
-#define REG_AUTH_IV14 0x348
-#define REG_AUTH_IV15 0x34c
-#define REG_AUTH_INFO_NONCE0 0x350
-#define REG_AUTH_INFO_NONCE1 0x354
-#define REG_AUTH_INFO_NONCE2 0x358
-#define REG_AUTH_INFO_NONCE3 0x35c
-#define REG_AUTH_BYTECNT0 0x390
-#define REG_AUTH_BYTECNT1 0x394
-#define REG_AUTH_BYTECNT2 0x398
-#define REG_AUTH_BYTECNT3 0x39c
-#define REG_AUTH_EXP_MAC0 0x3a0
-#define REG_AUTH_EXP_MAC1 0x3a4
-#define REG_AUTH_EXP_MAC2 0x3a8
-#define REG_AUTH_EXP_MAC3 0x3ac
-#define REG_AUTH_EXP_MAC4 0x3b0
-#define REG_AUTH_EXP_MAC5 0x3b4
-#define REG_AUTH_EXP_MAC6 0x3b8
-#define REG_AUTH_EXP_MAC7 0x3bc
-#define REG_CONFIG 0x400
-#define REG_GOPROC_QC_KEY 0x1000
-#define REG_GOPROC_OEM_KEY 0x2000
-#define REG_ENCR_KEY0 0x3000
-#define REG_ENCR_KEY1 0x3004
-#define REG_ENCR_KEY2 0x3008
-#define REG_ENCR_KEY3 0x300c
-#define REG_ENCR_KEY4 0x3010
-#define REG_ENCR_KEY5 0x3014
-#define REG_ENCR_KEY6 0x3018
-#define REG_ENCR_KEY7 0x301c
-#define REG_ENCR_XTS_KEY0 0x3020
-#define REG_ENCR_XTS_KEY1 0x3024
-#define REG_ENCR_XTS_KEY2 0x3028
-#define REG_ENCR_XTS_KEY3 0x302c
-#define REG_ENCR_XTS_KEY4 0x3030
-#define REG_ENCR_XTS_KEY5 0x3034
-#define REG_ENCR_XTS_KEY6 0x3038
-#define REG_ENCR_XTS_KEY7 0x303c
-#define REG_AUTH_KEY0 0x3040
-#define REG_AUTH_KEY1 0x3044
-#define REG_AUTH_KEY2 0x3048
-#define REG_AUTH_KEY3 0x304c
-#define REG_AUTH_KEY4 0x3050
-#define REG_AUTH_KEY5 0x3054
-#define REG_AUTH_KEY6 0x3058
-#define REG_AUTH_KEY7 0x305c
-#define REG_AUTH_KEY8 0x3060
-#define REG_AUTH_KEY9 0x3064
-#define REG_AUTH_KEY10 0x3068
-#define REG_AUTH_KEY11 0x306c
-#define REG_AUTH_KEY12 0x3070
-#define REG_AUTH_KEY13 0x3074
-#define REG_AUTH_KEY14 0x3078
-#define REG_AUTH_KEY15 0x307c
-
-/* Register bits - REG_VERSION */
-#define CORE_STEP_REV_SHIFT 0
-#define CORE_STEP_REV_MASK GENMASK(15, 0)
-#define CORE_MINOR_REV_SHIFT 16
-#define CORE_MINOR_REV_MASK GENMASK(23, 16)
-#define CORE_MAJOR_REV_SHIFT 24
-#define CORE_MAJOR_REV_MASK GENMASK(31, 24)
-
-/* Register bits - REG_STATUS */
-#define MAC_FAILED_SHIFT 31
-#define DOUT_SIZE_AVAIL_SHIFT 26
-#define DOUT_SIZE_AVAIL_MASK GENMASK(30, 26)
-#define DIN_SIZE_AVAIL_SHIFT 21
-#define DIN_SIZE_AVAIL_MASK GENMASK(25, 21)
-#define HSD_ERR_SHIFT 20
-#define ACCESS_VIOL_SHIFT 19
-#define PIPE_ACTIVE_ERR_SHIFT 18
-#define CFG_CHNG_ERR_SHIFT 17
-#define DOUT_ERR_SHIFT 16
-#define DIN_ERR_SHIFT 15
-#define AXI_ERR_SHIFT 14
-#define CRYPTO_STATE_SHIFT 10
-#define CRYPTO_STATE_MASK GENMASK(13, 10)
-#define ENCR_BUSY_SHIFT 9
-#define AUTH_BUSY_SHIFT 8
-#define DOUT_INTR_SHIFT 7
-#define DIN_INTR_SHIFT 6
-#define OP_DONE_INTR_SHIFT 5
-#define ERR_INTR_SHIFT 4
-#define DOUT_RDY_SHIFT 3
-#define DIN_RDY_SHIFT 2
-#define OPERATION_DONE_SHIFT 1
-#define SW_ERR_SHIFT 0
-
-/* Register bits - REG_STATUS2 */
-#define AXI_EXTRA_SHIFT 1
-#define LOCKED_SHIFT 2
-
-/* Register bits - REG_CONFIG */
-#define REQ_SIZE_SHIFT 17
-#define REQ_SIZE_MASK GENMASK(20, 17)
-#define REQ_SIZE_ENUM_1_BEAT 0
-#define REQ_SIZE_ENUM_2_BEAT 1
-#define REQ_SIZE_ENUM_3_BEAT 2
-#define REQ_SIZE_ENUM_4_BEAT 3
-#define REQ_SIZE_ENUM_5_BEAT 4
-#define REQ_SIZE_ENUM_6_BEAT 5
-#define REQ_SIZE_ENUM_7_BEAT 6
-#define REQ_SIZE_ENUM_8_BEAT 7
-#define REQ_SIZE_ENUM_9_BEAT 8
-#define REQ_SIZE_ENUM_10_BEAT 9
-#define REQ_SIZE_ENUM_11_BEAT 10
-#define REQ_SIZE_ENUM_12_BEAT 11
-#define REQ_SIZE_ENUM_13_BEAT 12
-#define REQ_SIZE_ENUM_14_BEAT 13
-#define REQ_SIZE_ENUM_15_BEAT 14
-#define REQ_SIZE_ENUM_16_BEAT 15
-
-#define MAX_QUEUED_REQ_SHIFT 14
-#define MAX_QUEUED_REQ_MASK GENMASK(24, 16)
-#define ENUM_1_QUEUED_REQS 0
-#define ENUM_2_QUEUED_REQS 1
-#define ENUM_3_QUEUED_REQS 2
-
-#define IRQ_ENABLES_SHIFT 10
-#define IRQ_ENABLES_MASK GENMASK(13, 10)
-
-#define LITTLE_ENDIAN_MODE_SHIFT 9
-#define PIPE_SET_SELECT_SHIFT 5
-#define PIPE_SET_SELECT_MASK GENMASK(8, 5)
-
-#define HIGH_SPD_EN_N_SHIFT 4
-#define MASK_DOUT_INTR_SHIFT 3
-#define MASK_DIN_INTR_SHIFT 2
-#define MASK_OP_DONE_INTR_SHIFT 1
-#define MASK_ERR_INTR_SHIFT 0
-
-/* Register bits - REG_AUTH_SEG_CFG */
-#define COMP_EXP_MAC_SHIFT 24
-#define COMP_EXP_MAC_DISABLED 0
-#define COMP_EXP_MAC_ENABLED 1
-
-#define F9_DIRECTION_SHIFT 23
-#define F9_DIRECTION_UPLINK 0
-#define F9_DIRECTION_DOWNLINK 1
-
-#define AUTH_NONCE_NUM_WORDS_SHIFT 20
-#define AUTH_NONCE_NUM_WORDS_MASK GENMASK(22, 20)
-
-#define USE_PIPE_KEY_AUTH_SHIFT 19
-#define USE_HW_KEY_AUTH_SHIFT 18
-#define AUTH_FIRST_SHIFT 17
-#define AUTH_LAST_SHIFT 16
-
-#define AUTH_POS_SHIFT 14
-#define AUTH_POS_MASK GENMASK(15, 14)
-#define AUTH_POS_BEFORE 0
-#define AUTH_POS_AFTER 1
-
-#define AUTH_SIZE_SHIFT 9
-#define AUTH_SIZE_MASK GENMASK(13, 9)
-#define AUTH_SIZE_SHA1 0
-#define AUTH_SIZE_SHA256 1
-#define AUTH_SIZE_ENUM_1_BYTES 0
-#define AUTH_SIZE_ENUM_2_BYTES 1
-#define AUTH_SIZE_ENUM_3_BYTES 2
-#define AUTH_SIZE_ENUM_4_BYTES 3
-#define AUTH_SIZE_ENUM_5_BYTES 4
-#define AUTH_SIZE_ENUM_6_BYTES 5
-#define AUTH_SIZE_ENUM_7_BYTES 6
-#define AUTH_SIZE_ENUM_8_BYTES 7
-#define AUTH_SIZE_ENUM_9_BYTES 8
-#define AUTH_SIZE_ENUM_10_BYTES 9
-#define AUTH_SIZE_ENUM_11_BYTES 10
-#define AUTH_SIZE_ENUM_12_BYTES 11
-#define AUTH_SIZE_ENUM_13_BYTES 12
-#define AUTH_SIZE_ENUM_14_BYTES 13
-#define AUTH_SIZE_ENUM_15_BYTES 14
-#define AUTH_SIZE_ENUM_16_BYTES 15
-
-#define AUTH_MODE_SHIFT 6
-#define AUTH_MODE_MASK GENMASK(8, 6)
-#define AUTH_MODE_HASH 0
-#define AUTH_MODE_HMAC 1
-#define AUTH_MODE_CCM 0
-#define AUTH_MODE_CMAC 1
-
-#define AUTH_KEY_SIZE_SHIFT 3
-#define AUTH_KEY_SIZE_MASK GENMASK(5, 3)
-#define AUTH_KEY_SZ_AES128 0
-#define AUTH_KEY_SZ_AES256 2
-
-#define AUTH_ALG_SHIFT 0
-#define AUTH_ALG_MASK GENMASK(2, 0)
-#define AUTH_ALG_NONE 0
-#define AUTH_ALG_SHA 1
-#define AUTH_ALG_AES 2
-#define AUTH_ALG_KASUMI 3
-#define AUTH_ALG_SNOW3G 4
-#define AUTH_ALG_ZUC 5
-
-/* Register bits - REG_ENCR_XTS_DU_SIZE */
-#define ENCR_XTS_DU_SIZE_SHIFT 0
-#define ENCR_XTS_DU_SIZE_MASK GENMASK(19, 0)
-
-/* Register bits - REG_ENCR_SEG_CFG */
-#define F8_KEYSTREAM_ENABLE_SHIFT 17
-#define F8_KEYSTREAM_DISABLED 0
-#define F8_KEYSTREAM_ENABLED 1
-
-#define F8_DIRECTION_SHIFT 16
-#define F8_DIRECTION_UPLINK 0
-#define F8_DIRECTION_DOWNLINK 1
-
-#define USE_PIPE_KEY_ENCR_SHIFT 15
-#define USE_PIPE_KEY_ENCR_ENABLED 1
-#define USE_KEY_REGISTERS 0
-
-#define USE_HW_KEY_ENCR_SHIFT 14
-#define USE_KEY_REG 0
-#define USE_HW_KEY 1
-
-#define LAST_CCM_SHIFT 13
-#define LAST_CCM_XFR 1
-#define INTERM_CCM_XFR 0
-
-#define CNTR_ALG_SHIFT 11
-#define CNTR_ALG_MASK GENMASK(12, 11)
-#define CNTR_ALG_NIST 0
-
-#define ENCODE_SHIFT 10
-
-#define ENCR_MODE_SHIFT 6
-#define ENCR_MODE_MASK GENMASK(9, 6)
-#define ENCR_MODE_ECB 0
-#define ENCR_MODE_CBC 1
-#define ENCR_MODE_CTR 2
-#define ENCR_MODE_XTS 3
-#define ENCR_MODE_CCM 4
-
-#define ENCR_KEY_SZ_SHIFT 3
-#define ENCR_KEY_SZ_MASK GENMASK(5, 3)
-#define ENCR_KEY_SZ_DES 0
-#define ENCR_KEY_SZ_3DES 1
-#define ENCR_KEY_SZ_AES128 0
-#define ENCR_KEY_SZ_AES256 2
-
-#define ENCR_ALG_SHIFT 0
-#define ENCR_ALG_MASK GENMASK(2, 0)
-#define ENCR_ALG_NONE 0
-#define ENCR_ALG_DES 1
-#define ENCR_ALG_AES 2
-#define ENCR_ALG_KASUMI 4
-#define ENCR_ALG_SNOW_3G 5
-#define ENCR_ALG_ZUC 6
-
-/* Register bits - REG_GOPROC */
-#define GO_SHIFT 0
-#define CLR_CNTXT_SHIFT 1
-#define RESULTS_DUMP_SHIFT 2
-
-/* Register bits - REG_ENGINES_AVAIL */
-#define ENCR_AES_SEL_SHIFT 0
-#define DES_SEL_SHIFT 1
-#define ENCR_SNOW3G_SEL_SHIFT 2
-#define ENCR_KASUMI_SEL_SHIFT 3
-#define SHA_SEL_SHIFT 4
-#define SHA512_SEL_SHIFT 5
-#define AUTH_AES_SEL_SHIFT 6
-#define AUTH_SNOW3G_SEL_SHIFT 7
-#define AUTH_KASUMI_SEL_SHIFT 8
-#define BAM_PIPE_SETS_SHIFT 9
-#define BAM_PIPE_SETS_MASK GENMASK(12, 9)
-#define AXI_WR_BEATS_SHIFT 13
-#define AXI_WR_BEATS_MASK GENMASK(18, 13)
-#define AXI_RD_BEATS_SHIFT 19
-#define AXI_RD_BEATS_MASK GENMASK(24, 19)
-#define ENCR_ZUC_SEL_SHIFT 26
-#define AUTH_ZUC_SEL_SHIFT 27
-#define ZUC_ENABLE_SHIFT 28
-
-#endif /* _REGS_V5_H_ */
diff --git a/drivers/crypto/qce/sha.c b/drivers/crypto/qce/sha.c
deleted file mode 100644
index 1b37121cbcdcb70ea02cc8b9cbbd6f03abb79851..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/sha.c
+++ /dev/null
@@ -1,545 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#include <linux/device.h>
-#include <linux/dma-mapping.h>
-#include <linux/interrupt.h>
-#include <linux/string.h>
-#include <crypto/internal/hash.h>
-
-#include "common.h"
-#include "core.h"
-#include "sha.h"
-
-struct qce_sha_saved_state {
- u8 pending_buf[QCE_SHA_MAX_BLOCKSIZE];
- u8 partial_digest[QCE_SHA_MAX_DIGESTSIZE];
- __be32 byte_count[2];
- unsigned int pending_buflen;
- unsigned int flags;
- u64 count;
- bool first_blk;
-};
-
-static LIST_HEAD(ahash_algs);
-
-static const u32 std_iv_sha1[SHA256_DIGEST_SIZE / sizeof(u32)] = {
- SHA1_H0, SHA1_H1, SHA1_H2, SHA1_H3, SHA1_H4, 0, 0, 0
-};
-
-static const u32 std_iv_sha256[SHA256_DIGEST_SIZE / sizeof(u32)] = {
- SHA256_H0, SHA256_H1, SHA256_H2, SHA256_H3,
- SHA256_H4, SHA256_H5, SHA256_H6, SHA256_H7
-};
-
-static void qce_ahash_done(void *data)
-{
- struct crypto_async_request *async_req = data;
- struct ahash_request *req = ahash_request_cast(async_req);
- struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_ahash_tmpl(async_req->tfm);
- struct qce_device *qce = tmpl->qce;
- struct qce_result_dump *result = qce->dma.result_buf;
- unsigned int digestsize = crypto_ahash_digestsize(ahash);
- int error;
- u32 status;
-
- error = qce_dma_terminate_all(&qce->dma);
- if (error)
- dev_dbg(qce->dev, "ahash dma termination error (%d)\n", error);
-
- dma_unmap_sg(qce->dev, req->src, rctx->src_nents, DMA_TO_DEVICE);
- dma_unmap_sg(qce->dev, &rctx->result_sg, 1, DMA_FROM_DEVICE);
-
- memcpy(rctx->digest, result->auth_iv, digestsize);
- if (req->result && rctx->last_blk)
- memcpy(req->result, result->auth_iv, digestsize);
-
- rctx->byte_count[0] = cpu_to_be32(result->auth_byte_count[0]);
- rctx->byte_count[1] = cpu_to_be32(result->auth_byte_count[1]);
-
- error = qce_check_status(qce, &status);
- if (error < 0)
- dev_dbg(qce->dev, "ahash operation error (%x)\n", status);
-
- req->src = rctx->src_orig;
- req->nbytes = rctx->nbytes_orig;
- rctx->last_blk = false;
- rctx->first_blk = false;
-
- qce->async_req_done(tmpl->qce, error);
-}
-
-static int qce_ahash_async_req_handle(struct crypto_async_request *async_req)
-{
- struct ahash_request *req = ahash_request_cast(async_req);
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_sha_ctx *ctx = crypto_tfm_ctx(async_req->tfm);
- struct qce_alg_template *tmpl = to_ahash_tmpl(async_req->tfm);
- struct qce_device *qce = tmpl->qce;
- unsigned long flags = rctx->flags;
- int ret;
-
- if (IS_SHA_HMAC(flags)) {
- rctx->authkey = ctx->authkey;
- rctx->authklen = QCE_SHA_HMAC_KEY_SIZE;
- } else if (IS_CMAC(flags)) {
- rctx->authkey = ctx->authkey;
- rctx->authklen = AES_KEYSIZE_128;
- }
-
- rctx->src_nents = sg_nents_for_len(req->src, req->nbytes);
- if (rctx->src_nents < 0) {
- dev_err(qce->dev, "Invalid numbers of src SG.\n");
- return rctx->src_nents;
- }
-
- ret = dma_map_sg(qce->dev, req->src, rctx->src_nents, DMA_TO_DEVICE);
- if (!ret)
- return -EIO;
-
- sg_init_one(&rctx->result_sg, qce->dma.result_buf, QCE_RESULT_BUF_SZ);
-
- ret = dma_map_sg(qce->dev, &rctx->result_sg, 1, DMA_FROM_DEVICE);
- if (!ret) {
- ret = -EIO;
- goto error_unmap_src;
- }
-
- ret = qce_dma_prep_sgs(&qce->dma, req->src, rctx->src_nents,
- &rctx->result_sg, 1, qce_ahash_done, async_req);
- if (ret)
- goto error_unmap_dst;
-
- qce_dma_issue_pending(&qce->dma);
-
- ret = qce_start(async_req, tmpl->crypto_alg_type);
- if (ret)
- goto error_terminate;
-
- return 0;
-
-error_terminate:
- qce_dma_terminate_all(&qce->dma);
-error_unmap_dst:
- dma_unmap_sg(qce->dev, &rctx->result_sg, 1, DMA_FROM_DEVICE);
-error_unmap_src:
- dma_unmap_sg(qce->dev, req->src, rctx->src_nents, DMA_TO_DEVICE);
- return ret;
-}
-
-static int qce_ahash_init(struct ahash_request *req)
-{
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_ahash_tmpl(req->base.tfm);
- const u32 *std_iv = tmpl->std_iv;
-
- memset(rctx, 0, sizeof(*rctx));
- rctx->first_blk = true;
- rctx->last_blk = false;
- rctx->flags = tmpl->alg_flags;
- memcpy(rctx->digest, std_iv, sizeof(rctx->digest));
-
- return 0;
-}
-
-static int qce_ahash_export(struct ahash_request *req, void *out)
-{
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_sha_saved_state *export_state = out;
-
- memcpy(export_state->pending_buf, rctx->buf, rctx->buflen);
- memcpy(export_state->partial_digest, rctx->digest, sizeof(rctx->digest));
- export_state->byte_count[0] = rctx->byte_count[0];
- export_state->byte_count[1] = rctx->byte_count[1];
- export_state->pending_buflen = rctx->buflen;
- export_state->count = rctx->count;
- export_state->first_blk = rctx->first_blk;
- export_state->flags = rctx->flags;
-
- return 0;
-}
-
-static int qce_ahash_import(struct ahash_request *req, const void *in)
-{
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- const struct qce_sha_saved_state *import_state = in;
-
- memset(rctx, 0, sizeof(*rctx));
- rctx->count = import_state->count;
- rctx->buflen = import_state->pending_buflen;
- rctx->first_blk = import_state->first_blk;
- rctx->flags = import_state->flags;
- rctx->byte_count[0] = import_state->byte_count[0];
- rctx->byte_count[1] = import_state->byte_count[1];
- memcpy(rctx->buf, import_state->pending_buf, rctx->buflen);
- memcpy(rctx->digest, import_state->partial_digest, sizeof(rctx->digest));
-
- return 0;
-}
-
-static int qce_ahash_update(struct ahash_request *req)
-{
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_ahash_tmpl(req->base.tfm);
- struct qce_device *qce = tmpl->qce;
- struct scatterlist *sg_last, *sg;
- unsigned int total, len;
- unsigned int hash_later;
- unsigned int nbytes;
- unsigned int blocksize;
-
- blocksize = crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm));
- rctx->count += req->nbytes;
-
- /* check for buffer from previous updates and append it */
- total = req->nbytes + rctx->buflen;
-
- if (total <= blocksize) {
- scatterwalk_map_and_copy(rctx->buf + rctx->buflen, req->src,
- 0, req->nbytes, 0);
- rctx->buflen += req->nbytes;
- return 0;
- }
-
- /* save the original req structure fields */
- rctx->src_orig = req->src;
- rctx->nbytes_orig = req->nbytes;
-
- /*
- * if we have data from previous update copy them on buffer. The old
- * data will be combined with current request bytes.
- */
- if (rctx->buflen)
- memcpy(rctx->tmpbuf, rctx->buf, rctx->buflen);
-
- /* calculate how many bytes will be hashed later */
- hash_later = total % blocksize;
-
- /*
- * At this point, there is more than one block size of data. If
- * the available data to transfer is exactly a multiple of block
- * size, save the last block to be transferred in qce_ahash_final
- * (with the last block bit set) if this is indeed the end of data
- * stream. If not this saved block will be transferred as part of
- * next update. If this block is not held back and if this is
- * indeed the end of data stream, the digest obtained will be wrong
- * since qce_ahash_final will see that rctx->buflen is 0 and return
- * doing nothing which in turn means that a digest will not be
- * copied to the destination result buffer. qce_ahash_final cannot
- * be made to alter this behavior and allowed to proceed if
- * rctx->buflen is 0 because the crypto engine BAM does not allow
- * for zero length transfers.
- */
- if (!hash_later)
- hash_later = blocksize;
-
- if (hash_later) {
- unsigned int src_offset = req->nbytes - hash_later;
- scatterwalk_map_and_copy(rctx->buf, req->src, src_offset,
- hash_later, 0);
- }
-
- /* here nbytes is multiple of blocksize */
- nbytes = total - hash_later;
-
- len = rctx->buflen;
- sg = sg_last = req->src;
-
- while (len < nbytes && sg) {
- if (len + sg_dma_len(sg) > nbytes)
- break;
- len += sg_dma_len(sg);
- sg_last = sg;
- sg = sg_next(sg);
- }
-
- if (!sg_last)
- return -EINVAL;
-
- if (rctx->buflen) {
- sg_init_table(rctx->sg, 2);
- sg_set_buf(rctx->sg, rctx->tmpbuf, rctx->buflen);
- sg_chain(rctx->sg, 2, req->src);
- req->src = rctx->sg;
- }
-
- req->nbytes = nbytes;
- rctx->buflen = hash_later;
-
- return qce->async_req_enqueue(tmpl->qce, &req->base);
-}
-
-static int qce_ahash_final(struct ahash_request *req)
-{
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_ahash_tmpl(req->base.tfm);
- struct qce_device *qce = tmpl->qce;
-
- if (!rctx->buflen) {
- if (tmpl->hash_zero)
- memcpy(req->result, tmpl->hash_zero,
- tmpl->alg.ahash.halg.digestsize);
- return 0;
- }
-
- rctx->last_blk = true;
-
- rctx->src_orig = req->src;
- rctx->nbytes_orig = req->nbytes;
-
- memcpy(rctx->tmpbuf, rctx->buf, rctx->buflen);
- sg_init_one(rctx->sg, rctx->tmpbuf, rctx->buflen);
-
- req->src = rctx->sg;
- req->nbytes = rctx->buflen;
-
- return qce->async_req_enqueue(tmpl->qce, &req->base);
-}
-
-static int qce_ahash_digest(struct ahash_request *req)
-{
- struct qce_sha_reqctx *rctx = ahash_request_ctx_dma(req);
- struct qce_alg_template *tmpl = to_ahash_tmpl(req->base.tfm);
- struct qce_device *qce = tmpl->qce;
- int ret;
-
- ret = qce_ahash_init(req);
- if (ret)
- return ret;
-
- rctx->src_orig = req->src;
- rctx->nbytes_orig = req->nbytes;
- rctx->first_blk = true;
- rctx->last_blk = true;
-
- if (!rctx->nbytes_orig) {
- if (tmpl->hash_zero)
- memcpy(req->result, tmpl->hash_zero,
- tmpl->alg.ahash.halg.digestsize);
- return 0;
- }
-
- return qce->async_req_enqueue(tmpl->qce, &req->base);
-}
-
-static int qce_ahash_hmac_setkey(struct crypto_ahash *tfm, const u8 *key,
- unsigned int keylen)
-{
- unsigned int digestsize = crypto_ahash_digestsize(tfm);
- struct qce_sha_ctx *ctx = crypto_tfm_ctx(&tfm->base);
- struct crypto_wait wait;
- struct ahash_request *req;
- struct scatterlist sg;
- unsigned int blocksize;
- struct crypto_ahash *ahash_tfm;
- u8 *buf;
- int ret;
- const char *alg_name;
-
- blocksize = crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm));
- memset(ctx->authkey, 0, sizeof(ctx->authkey));
-
- if (keylen <= blocksize) {
- memcpy(ctx->authkey, key, keylen);
- return 0;
- }
-
- if (digestsize == SHA1_DIGEST_SIZE)
- alg_name = "sha1-qce";
- else if (digestsize == SHA256_DIGEST_SIZE)
- alg_name = "sha256-qce";
- else
- return -EINVAL;
-
- ahash_tfm = crypto_alloc_ahash(alg_name, 0, 0);
- if (IS_ERR(ahash_tfm))
- return PTR_ERR(ahash_tfm);
-
- req = ahash_request_alloc(ahash_tfm, GFP_KERNEL);
- if (!req) {
- ret = -ENOMEM;
- goto err_free_ahash;
- }
-
- crypto_init_wait(&wait);
- ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- crypto_req_done, &wait);
- crypto_ahash_clear_flags(ahash_tfm, ~0);
-
- buf = kzalloc(keylen + QCE_MAX_ALIGN_SIZE, GFP_KERNEL);
- if (!buf) {
- ret = -ENOMEM;
- goto err_free_req;
- }
-
- memcpy(buf, key, keylen);
- sg_init_one(&sg, buf, keylen);
- ahash_request_set_crypt(req, &sg, ctx->authkey, keylen);
-
- ret = crypto_wait_req(crypto_ahash_digest(req), &wait);
-
- kfree(buf);
-err_free_req:
- ahash_request_free(req);
-err_free_ahash:
- crypto_free_ahash(ahash_tfm);
- return ret;
-}
-
-static int qce_ahash_cra_init(struct crypto_tfm *tfm)
-{
- struct crypto_ahash *ahash = __crypto_ahash_cast(tfm);
- struct qce_sha_ctx *ctx = crypto_tfm_ctx(tfm);
-
- crypto_ahash_set_reqsize_dma(ahash, sizeof(struct qce_sha_reqctx));
- memset(ctx, 0, sizeof(*ctx));
- return 0;
-}
-
-struct qce_ahash_def {
- unsigned long flags;
- const char *name;
- const char *drv_name;
- unsigned int digestsize;
- unsigned int blocksize;
- unsigned int statesize;
- const u32 *std_iv;
-};
-
-static const struct qce_ahash_def ahash_def[] = {
- {
- .flags = QCE_HASH_SHA1,
- .name = "sha1",
- .drv_name = "sha1-qce",
- .digestsize = SHA1_DIGEST_SIZE,
- .blocksize = SHA1_BLOCK_SIZE,
- .statesize = sizeof(struct qce_sha_saved_state),
- .std_iv = std_iv_sha1,
- },
- {
- .flags = QCE_HASH_SHA256,
- .name = "sha256",
- .drv_name = "sha256-qce",
- .digestsize = SHA256_DIGEST_SIZE,
- .blocksize = SHA256_BLOCK_SIZE,
- .statesize = sizeof(struct qce_sha_saved_state),
- .std_iv = std_iv_sha256,
- },
- {
- .flags = QCE_HASH_SHA1_HMAC,
- .name = "hmac(sha1)",
- .drv_name = "hmac-sha1-qce",
- .digestsize = SHA1_DIGEST_SIZE,
- .blocksize = SHA1_BLOCK_SIZE,
- .statesize = sizeof(struct qce_sha_saved_state),
- .std_iv = std_iv_sha1,
- },
- {
- .flags = QCE_HASH_SHA256_HMAC,
- .name = "hmac(sha256)",
- .drv_name = "hmac-sha256-qce",
- .digestsize = SHA256_DIGEST_SIZE,
- .blocksize = SHA256_BLOCK_SIZE,
- .statesize = sizeof(struct qce_sha_saved_state),
- .std_iv = std_iv_sha256,
- },
-};
-
-static int qce_ahash_register_one(const struct qce_ahash_def *def,
- struct qce_device *qce)
-{
- struct qce_alg_template *tmpl;
- struct ahash_alg *alg;
- struct crypto_alg *base;
- int ret;
-
- tmpl = kzalloc_obj(*tmpl);
- if (!tmpl)
- return -ENOMEM;
-
- tmpl->std_iv = def->std_iv;
-
- alg = &tmpl->alg.ahash;
- alg->init = qce_ahash_init;
- alg->update = qce_ahash_update;
- alg->final = qce_ahash_final;
- alg->digest = qce_ahash_digest;
- alg->export = qce_ahash_export;
- alg->import = qce_ahash_import;
- if (IS_SHA_HMAC(def->flags))
- alg->setkey = qce_ahash_hmac_setkey;
- alg->halg.digestsize = def->digestsize;
- alg->halg.statesize = def->statesize;
-
- if (IS_SHA1(def->flags))
- tmpl->hash_zero = sha1_zero_message_hash;
- else if (IS_SHA256(def->flags))
- tmpl->hash_zero = sha256_zero_message_hash;
-
- base = &alg->halg.base;
- base->cra_blocksize = def->blocksize;
- base->cra_priority = 175;
- base->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY;
- base->cra_ctxsize = sizeof(struct qce_sha_ctx);
- base->cra_alignmask = 0;
- base->cra_module = THIS_MODULE;
- base->cra_init = qce_ahash_cra_init;
-
- strscpy(base->cra_name, def->name);
- strscpy(base->cra_driver_name, def->drv_name);
-
- INIT_LIST_HEAD(&tmpl->entry);
- tmpl->crypto_alg_type = CRYPTO_ALG_TYPE_AHASH;
- tmpl->alg_flags = def->flags;
- tmpl->qce = qce;
-
- ret = crypto_register_ahash(alg);
- if (ret) {
- dev_err(qce->dev, "%s registration failed\n", base->cra_name);
- kfree(tmpl);
- return ret;
- }
-
- list_add_tail(&tmpl->entry, &ahash_algs);
- dev_dbg(qce->dev, "%s is registered\n", base->cra_name);
- return 0;
-}
-
-static void qce_ahash_unregister(struct qce_device *qce)
-{
- struct qce_alg_template *tmpl, *n;
-
- list_for_each_entry_safe(tmpl, n, &ahash_algs, entry) {
- crypto_unregister_ahash(&tmpl->alg.ahash);
- list_del(&tmpl->entry);
- kfree(tmpl);
- }
-}
-
-static int qce_ahash_register(struct qce_device *qce)
-{
- int ret, i;
-
- for (i = 0; i < ARRAY_SIZE(ahash_def); i++) {
- ret = qce_ahash_register_one(&ahash_def[i], qce);
- if (ret)
- goto err;
- }
-
- return 0;
-err:
- qce_ahash_unregister(qce);
- return ret;
-}
-
-const struct qce_algo_ops ahash_ops = {
- .type = CRYPTO_ALG_TYPE_AHASH,
- .register_algs = qce_ahash_register,
- .unregister_algs = qce_ahash_unregister,
- .async_req_handle = qce_ahash_async_req_handle,
-};
diff --git a/drivers/crypto/qce/sha.h b/drivers/crypto/qce/sha.h
deleted file mode 100644
index a22695361f1654cc94325ec5d886a158fa4bfb9c..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/sha.h
+++ /dev/null
@@ -1,72 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _SHA_H_
-#define _SHA_H_
-
-#include <crypto/scatterwalk.h>
-#include <crypto/sha1.h>
-#include <crypto/sha2.h>
-
-#include "common.h"
-#include "core.h"
-
-#define QCE_SHA_MAX_BLOCKSIZE SHA256_BLOCK_SIZE
-#define QCE_SHA_MAX_DIGESTSIZE SHA256_DIGEST_SIZE
-
-struct qce_sha_ctx {
- u8 authkey[QCE_SHA_MAX_BLOCKSIZE];
-};
-
-/**
- * struct qce_sha_reqctx - holds private ahash objects per request
- * @buf: used during update, import and export
- * @tmpbuf: buffer for internal use
- * @digest: calculated digest buffer
- * @buflen: length of the buffer
- * @flags: operation flags
- * @src_orig: original request sg list
- * @nbytes_orig: original request number of bytes
- * @src_nents: source number of entries
- * @byte_count: byte count
- * @count: save count in states during update, import and export
- * @first_blk: is it the first block
- * @last_blk: is it the last block
- * @sg: used to chain sg lists
- * @authkey: pointer to auth key in sha ctx
- * @authklen: auth key length
- * @result_sg: scatterlist used for result buffer
- */
-struct qce_sha_reqctx {
- u8 buf[QCE_SHA_MAX_BLOCKSIZE];
- u8 tmpbuf[QCE_SHA_MAX_BLOCKSIZE];
- u8 digest[QCE_SHA_MAX_DIGESTSIZE];
- unsigned int buflen;
- unsigned long flags;
- struct scatterlist *src_orig;
- unsigned int nbytes_orig;
- int src_nents;
- __be32 byte_count[2];
- u64 count;
- bool first_blk;
- bool last_blk;
- struct scatterlist sg[2];
- u8 *authkey;
- unsigned int authklen;
- struct scatterlist result_sg;
-};
-
-static inline struct qce_alg_template *to_ahash_tmpl(struct crypto_tfm *tfm)
-{
- struct crypto_ahash *ahash = __crypto_ahash_cast(tfm);
- struct ahash_alg *alg = container_of(crypto_hash_alg_common(ahash),
- struct ahash_alg, halg);
-
- return container_of(alg, struct qce_alg_template, alg.ahash);
-}
-
-extern const struct qce_algo_ops ahash_ops;
-
-#endif /* _SHA_H_ */
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
deleted file mode 100644
index db0b648a56eb1adb2e50285468112d051649aa9e..0000000000000000000000000000000000000000
--- a/drivers/crypto/qce/skcipher.c
+++ /dev/null
@@ -1,529 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (c) 2010-2014, The Linux Foundation. All rights reserved.
- */
-
-#include <linux/device.h>
-#include <linux/dma-mapping.h>
-#include <linux/interrupt.h>
-#include <linux/moduleparam.h>
-#include <linux/string.h>
-#include <linux/types.h>
-#include <linux/errno.h>
-#include <crypto/aes.h>
-#include <crypto/internal/des.h>
-#include <crypto/internal/skcipher.h>
-
-#include "cipher.h"
-
-static unsigned int aes_sw_max_len = CONFIG_CRYPTO_DEV_QCE_SW_MAX_LEN;
-module_param(aes_sw_max_len, uint, 0644);
-MODULE_PARM_DESC(aes_sw_max_len,
- "Only use hardware for AES requests larger than this "
- "[0=always use hardware; anything <16 breaks AES-GCM; default="
- __stringify(CONFIG_CRYPTO_DEV_QCE_SW_MAX_LEN)"]");
-
-static LIST_HEAD(skcipher_algs);
-
-static void qce_skcipher_done(void *data)
-{
- struct crypto_async_request *async_req = data;
- struct skcipher_request *req = skcipher_request_cast(async_req);
- struct qce_cipher_reqctx *rctx = skcipher_request_ctx(req);
- struct qce_alg_template *tmpl = to_cipher_tmpl(crypto_skcipher_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- struct qce_result_dump *result_buf = qce->dma.result_buf;
- enum dma_data_direction dir_src, dir_dst;
- u32 status;
- int error;
- bool diff_dst;
-
- diff_dst = (req->src != req->dst) ? true : false;
- dir_src = diff_dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL;
- dir_dst = diff_dst ? DMA_FROM_DEVICE : DMA_BIDIRECTIONAL;
-
- error = qce_dma_terminate_all(&qce->dma);
- if (error)
- dev_dbg(qce->dev, "skcipher dma termination error (%d)\n",
- error);
-
- if (diff_dst)
- dma_unmap_sg(qce->dev, rctx->src_sg, rctx->src_nents, dir_src);
- dma_unmap_sg(qce->dev, rctx->dst_sg, rctx->dst_nents, dir_dst);
-
- sg_free_table(&rctx->dst_tbl);
-
- error = qce_check_status(qce, &status);
- if (error < 0)
- dev_dbg(qce->dev, "skcipher operation error (%x)\n", status);
-
- memcpy(rctx->iv, result_buf->encr_cntr_iv, rctx->ivsize);
- qce->async_req_done(tmpl->qce, error);
-}
-
-static int
-qce_skcipher_async_req_handle(struct crypto_async_request *async_req)
-{
- struct skcipher_request *req = skcipher_request_cast(async_req);
- struct qce_cipher_reqctx *rctx = skcipher_request_ctx(req);
- struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req);
- struct qce_alg_template *tmpl = to_cipher_tmpl(crypto_skcipher_reqtfm(req));
- struct qce_device *qce = tmpl->qce;
- enum dma_data_direction dir_src, dir_dst;
- struct scatterlist *sg;
- bool diff_dst;
- gfp_t gfp;
- int dst_nents, src_nents, ret;
-
- rctx->iv = req->iv;
- rctx->ivsize = crypto_skcipher_ivsize(skcipher);
- rctx->cryptlen = req->cryptlen;
-
- diff_dst = (req->src != req->dst) ? true : false;
- dir_src = diff_dst ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL;
- dir_dst = diff_dst ? DMA_FROM_DEVICE : DMA_BIDIRECTIONAL;
-
- rctx->src_nents = sg_nents_for_len(req->src, req->cryptlen);
- if (diff_dst)
- rctx->dst_nents = sg_nents_for_len(req->dst, req->cryptlen);
- else
- rctx->dst_nents = rctx->src_nents;
- if (rctx->src_nents < 0) {
- dev_err(qce->dev, "Invalid numbers of src SG.\n");
- return rctx->src_nents;
- }
- if (rctx->dst_nents < 0) {
- dev_err(qce->dev, "Invalid numbers of dst SG.\n");
- return -rctx->dst_nents;
- }
-
- rctx->dst_nents += 1;
-
- gfp = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
- GFP_KERNEL : GFP_ATOMIC;
-
- ret = sg_alloc_table(&rctx->dst_tbl, rctx->dst_nents, gfp);
- if (ret)
- return ret;
-
- sg_init_one(&rctx->result_sg, qce->dma.result_buf, QCE_RESULT_BUF_SZ);
-
- sg = qce_sgtable_add(&rctx->dst_tbl, req->dst, req->cryptlen);
- if (IS_ERR(sg)) {
- ret = PTR_ERR(sg);
- goto error_free;
- }
-
- sg = qce_sgtable_add(&rctx->dst_tbl, &rctx->result_sg,
- QCE_RESULT_BUF_SZ);
- if (IS_ERR(sg)) {
- ret = PTR_ERR(sg);
- goto error_free;
- }
-
- sg_mark_end(sg);
- rctx->dst_sg = rctx->dst_tbl.sgl;
-
- dst_nents = dma_map_sg(qce->dev, rctx->dst_sg, rctx->dst_nents, dir_dst);
- if (!dst_nents) {
- ret = -EIO;
- goto error_free;
- }
-
- if (diff_dst) {
- src_nents = dma_map_sg(qce->dev, req->src, rctx->src_nents, dir_src);
- if (!src_nents) {
- ret = -EIO;
- goto error_unmap_dst;
- }
- rctx->src_sg = req->src;
- } else {
- rctx->src_sg = rctx->dst_sg;
- src_nents = dst_nents - 1;
- }
-
- ret = qce_dma_prep_sgs(&qce->dma, rctx->src_sg, src_nents,
- rctx->dst_sg, dst_nents,
- qce_skcipher_done, async_req);
- if (ret)
- goto error_unmap_src;
-
- qce_dma_issue_pending(&qce->dma);
-
- ret = qce_start(async_req, tmpl->crypto_alg_type);
- if (ret)
- goto error_terminate;
-
- return 0;
-
-error_terminate:
- qce_dma_terminate_all(&qce->dma);
-error_unmap_src:
- if (diff_dst)
- dma_unmap_sg(qce->dev, req->src, rctx->src_nents, dir_src);
-error_unmap_dst:
- dma_unmap_sg(qce->dev, rctx->dst_sg, rctx->dst_nents, dir_dst);
-error_free:
- sg_free_table(&rctx->dst_tbl);
- return ret;
-}
-
-static int qce_skcipher_setkey(struct crypto_skcipher *ablk, const u8 *key,
- unsigned int keylen)
-{
- struct crypto_tfm *tfm = crypto_skcipher_tfm(ablk);
- struct qce_cipher_ctx *ctx = crypto_tfm_ctx(tfm);
- unsigned long flags = to_cipher_tmpl(ablk)->alg_flags;
- unsigned int __keylen;
- int ret;
-
- if (!key || !keylen)
- return -EINVAL;
-
- /*
- * AES XTS key1 = key2 not supported by crypto engine.
- * Revisit to request a fallback cipher in this case.
- */
- if (IS_XTS(flags)) {
- __keylen = keylen >> 1;
- if (!memcmp(key, key + __keylen, __keylen))
- return -ENOKEY;
- } else {
- __keylen = keylen;
- }
-
- switch (__keylen) {
- case AES_KEYSIZE_128:
- case AES_KEYSIZE_256:
- memcpy(ctx->enc_key, key, keylen);
- break;
- case AES_KEYSIZE_192:
- break;
- default:
- return -EINVAL;
- }
-
- ret = crypto_skcipher_setkey(ctx->fallback, key, keylen);
- if (!ret)
- ctx->enc_keylen = keylen;
- return ret;
-}
-
-static int qce_des_setkey(struct crypto_skcipher *ablk, const u8 *key,
- unsigned int keylen)
-{
- struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(ablk);
- int err;
-
- err = verify_skcipher_des_key(ablk, key);
- if (err)
- return err;
-
- ctx->enc_keylen = keylen;
- memcpy(ctx->enc_key, key, keylen);
- return 0;
-}
-
-static int qce_des3_setkey(struct crypto_skcipher *ablk, const u8 *key,
- unsigned int keylen)
-{
- struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(ablk);
- u32 _key[6];
- int err;
-
- err = verify_skcipher_des3_key(ablk, key);
- if (err)
- return err;
-
- /*
- * The crypto engine does not support any two keys
- * being the same for triple des algorithms. The
- * verify_skcipher_des3_key does not check for all the
- * below conditions. Return -ENOKEY in case any two keys
- * are the same. Revisit to see if a fallback cipher
- * is needed to handle this condition.
- */
- memcpy(_key, key, DES3_EDE_KEY_SIZE);
- if (!((_key[0] ^ _key[2]) | (_key[1] ^ _key[3])) ||
- !((_key[2] ^ _key[4]) | (_key[3] ^ _key[5])) ||
- !((_key[0] ^ _key[4]) | (_key[1] ^ _key[5])))
- return -ENOKEY;
-
- ctx->enc_keylen = keylen;
- memcpy(ctx->enc_key, key, keylen);
- return 0;
-}
-
-static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
-{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(tfm);
- struct qce_cipher_reqctx *rctx = skcipher_request_ctx(req);
- struct qce_alg_template *tmpl = to_cipher_tmpl(tfm);
- unsigned int blocksize = crypto_skcipher_blocksize(tfm);
- int keylen;
- int ret;
-
- rctx->flags = tmpl->alg_flags;
- rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
- keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
-
- /* CE does not handle 0 length messages */
- if (!req->cryptlen)
- return 0;
-
- /*
- * ECB and CBC algorithms require message lengths to be
- * multiples of block size.
- */
- if (IS_ECB(rctx->flags) || IS_CBC(rctx->flags))
- if (!IS_ALIGNED(req->cryptlen, blocksize))
- return -EINVAL;
-
- /*
- * Conditions for requesting a fallback cipher
- * AES-192 (not supported by crypto engine (CE))
- * AES-XTS request with len <= 512 byte (not recommended to use CE)
- * AES-XTS request with len > QCE_SECTOR_SIZE and
- * is not a multiple of it.(Revisit this condition to check if it is
- * needed in all versions of CE)
- */
- if (IS_AES(rctx->flags) &&
- ((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256) ||
- (IS_XTS(rctx->flags) && ((req->cryptlen <= aes_sw_max_len) ||
- (req->cryptlen > QCE_SECTOR_SIZE &&
- req->cryptlen % QCE_SECTOR_SIZE))))) {
- skcipher_request_set_tfm(&rctx->fallback_req, ctx->fallback);
- skcipher_request_set_callback(&rctx->fallback_req,
- req->base.flags,
- req->base.complete,
- req->base.data);
- skcipher_request_set_crypt(&rctx->fallback_req, req->src,
- req->dst, req->cryptlen, req->iv);
- ret = encrypt ? crypto_skcipher_encrypt(&rctx->fallback_req) :
- crypto_skcipher_decrypt(&rctx->fallback_req);
- return ret;
- }
-
- return tmpl->qce->async_req_enqueue(tmpl->qce, &req->base);
-}
-
-static int qce_skcipher_encrypt(struct skcipher_request *req)
-{
- return qce_skcipher_crypt(req, 1);
-}
-
-static int qce_skcipher_decrypt(struct skcipher_request *req)
-{
- return qce_skcipher_crypt(req, 0);
-}
-
-static int qce_skcipher_init(struct crypto_skcipher *tfm)
-{
- /* take the size without the fallback skcipher_request at the end */
- crypto_skcipher_set_reqsize(tfm, offsetof(struct qce_cipher_reqctx,
- fallback_req));
- return 0;
-}
-
-static int qce_skcipher_init_fallback(struct crypto_skcipher *tfm)
-{
- struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(tfm);
-
- ctx->fallback = crypto_alloc_skcipher(crypto_tfm_alg_name(&tfm->base),
- 0, CRYPTO_ALG_NEED_FALLBACK);
- if (IS_ERR(ctx->fallback))
- return PTR_ERR(ctx->fallback);
-
- crypto_skcipher_set_reqsize(tfm, sizeof(struct qce_cipher_reqctx) +
- crypto_skcipher_reqsize(ctx->fallback));
- return 0;
-}
-
-static void qce_skcipher_exit(struct crypto_skcipher *tfm)
-{
- struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(tfm);
-
- crypto_free_skcipher(ctx->fallback);
-}
-
-struct qce_skcipher_def {
- unsigned long flags;
- const char *name;
- const char *drv_name;
- unsigned int blocksize;
- unsigned int chunksize;
- unsigned int ivsize;
- unsigned int min_keysize;
- unsigned int max_keysize;
-};
-
-static const struct qce_skcipher_def skcipher_def[] = {
- {
- .flags = QCE_ALG_AES | QCE_MODE_ECB,
- .name = "ecb(aes)",
- .drv_name = "ecb-aes-qce",
- .blocksize = AES_BLOCK_SIZE,
- .ivsize = 0,
- .min_keysize = AES_MIN_KEY_SIZE,
- .max_keysize = AES_MAX_KEY_SIZE,
- },
- {
- .flags = QCE_ALG_AES | QCE_MODE_CBC,
- .name = "cbc(aes)",
- .drv_name = "cbc-aes-qce",
- .blocksize = AES_BLOCK_SIZE,
- .ivsize = AES_BLOCK_SIZE,
- .min_keysize = AES_MIN_KEY_SIZE,
- .max_keysize = AES_MAX_KEY_SIZE,
- },
- {
- .flags = QCE_ALG_AES | QCE_MODE_CTR,
- .name = "ctr(aes)",
- .drv_name = "ctr-aes-qce",
- .blocksize = 1,
- .chunksize = AES_BLOCK_SIZE,
- .ivsize = AES_BLOCK_SIZE,
- .min_keysize = AES_MIN_KEY_SIZE,
- .max_keysize = AES_MAX_KEY_SIZE,
- },
- {
- .flags = QCE_ALG_AES | QCE_MODE_XTS,
- .name = "xts(aes)",
- .drv_name = "xts-aes-qce",
- .blocksize = AES_BLOCK_SIZE,
- .ivsize = AES_BLOCK_SIZE,
- .min_keysize = AES_MIN_KEY_SIZE * 2,
- .max_keysize = AES_MAX_KEY_SIZE * 2,
- },
- {
- .flags = QCE_ALG_DES | QCE_MODE_ECB,
- .name = "ecb(des)",
- .drv_name = "ecb-des-qce",
- .blocksize = DES_BLOCK_SIZE,
- .ivsize = 0,
- .min_keysize = DES_KEY_SIZE,
- .max_keysize = DES_KEY_SIZE,
- },
- {
- .flags = QCE_ALG_DES | QCE_MODE_CBC,
- .name = "cbc(des)",
- .drv_name = "cbc-des-qce",
- .blocksize = DES_BLOCK_SIZE,
- .ivsize = DES_BLOCK_SIZE,
- .min_keysize = DES_KEY_SIZE,
- .max_keysize = DES_KEY_SIZE,
- },
- {
- .flags = QCE_ALG_3DES | QCE_MODE_ECB,
- .name = "ecb(des3_ede)",
- .drv_name = "ecb-3des-qce",
- .blocksize = DES3_EDE_BLOCK_SIZE,
- .ivsize = 0,
- .min_keysize = DES3_EDE_KEY_SIZE,
- .max_keysize = DES3_EDE_KEY_SIZE,
- },
- {
- .flags = QCE_ALG_3DES | QCE_MODE_CBC,
- .name = "cbc(des3_ede)",
- .drv_name = "cbc-3des-qce",
- .blocksize = DES3_EDE_BLOCK_SIZE,
- .ivsize = DES3_EDE_BLOCK_SIZE,
- .min_keysize = DES3_EDE_KEY_SIZE,
- .max_keysize = DES3_EDE_KEY_SIZE,
- },
-};
-
-static int qce_skcipher_register_one(const struct qce_skcipher_def *def,
- struct qce_device *qce)
-{
- struct qce_alg_template *tmpl;
- struct skcipher_alg *alg;
- int ret;
-
- tmpl = kzalloc_obj(*tmpl);
- if (!tmpl)
- return -ENOMEM;
-
- alg = &tmpl->alg.skcipher;
-
- strscpy(alg->base.cra_name, def->name);
- strscpy(alg->base.cra_driver_name, def->drv_name);
-
- alg->base.cra_blocksize = def->blocksize;
- alg->chunksize = def->chunksize;
- alg->ivsize = def->ivsize;
- alg->min_keysize = def->min_keysize;
- alg->max_keysize = def->max_keysize;
- alg->setkey = IS_3DES(def->flags) ? qce_des3_setkey :
- IS_DES(def->flags) ? qce_des_setkey :
- qce_skcipher_setkey;
- alg->encrypt = qce_skcipher_encrypt;
- alg->decrypt = qce_skcipher_decrypt;
-
- alg->base.cra_priority = 275;
- alg->base.cra_flags = CRYPTO_ALG_ASYNC |
- CRYPTO_ALG_ALLOCATES_MEMORY |
- CRYPTO_ALG_KERN_DRIVER_ONLY;
- alg->base.cra_ctxsize = sizeof(struct qce_cipher_ctx);
- alg->base.cra_alignmask = 0;
- alg->base.cra_module = THIS_MODULE;
-
- if (IS_AES(def->flags)) {
- alg->base.cra_flags |= CRYPTO_ALG_NEED_FALLBACK;
- alg->init = qce_skcipher_init_fallback;
- alg->exit = qce_skcipher_exit;
- } else {
- alg->init = qce_skcipher_init;
- }
-
- INIT_LIST_HEAD(&tmpl->entry);
- tmpl->crypto_alg_type = CRYPTO_ALG_TYPE_SKCIPHER;
- tmpl->alg_flags = def->flags;
- tmpl->qce = qce;
-
- ret = crypto_register_skcipher(alg);
- if (ret) {
- dev_err(qce->dev, "%s registration failed\n", alg->base.cra_name);
- kfree(tmpl);
- return ret;
- }
-
- list_add_tail(&tmpl->entry, &skcipher_algs);
- dev_dbg(qce->dev, "%s is registered\n", alg->base.cra_name);
- return 0;
-}
-
-static void qce_skcipher_unregister(struct qce_device *qce)
-{
- struct qce_alg_template *tmpl, *n;
-
- list_for_each_entry_safe(tmpl, n, &skcipher_algs, entry) {
- crypto_unregister_skcipher(&tmpl->alg.skcipher);
- list_del(&tmpl->entry);
- kfree(tmpl);
- }
-}
-
-static int qce_skcipher_register(struct qce_device *qce)
-{
- int ret, i;
-
- for (i = 0; i < ARRAY_SIZE(skcipher_def); i++) {
- ret = qce_skcipher_register_one(&skcipher_def[i], qce);
- if (ret)
- goto err;
- }
-
- return 0;
-err:
- qce_skcipher_unregister(qce);
- return ret;
-}
-
-const struct qce_algo_ops skcipher_ops = {
- .type = CRYPTO_ALG_TYPE_SKCIPHER,
- .register_algs = qce_skcipher_register,
- .unregister_algs = qce_skcipher_unregister,
- .async_req_handle = qce_skcipher_async_req_handle,
-};
--
2.54.0
^ permalink raw reply related
* [PATCH 2/2] devicetree: Mark QCE bindings as deprecated
From: Demi Marie Obenour via B4 Relay @ 2026-05-23 19:03 UTC (permalink / raw)
To: Herbert Xu, David S. Miller, Thara Gopinath, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
Russell King
Cc: linux-kernel, linux-crypto, linux-arm-msm, Eric Biggers,
Ard Biesheuvel, devicetree, linux-arm-kernel, Demi Marie Obenour
In-Reply-To: <20260523-delete-qce-v1-0-86105cd7f406@gmail.com>
From: Demi Marie Obenour <demiobenour@gmail.com>
They are no longer used by the kernel. Keep them to avoid unnecessary
churn and because I know next to nothing about devicetree.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
Documentation/devicetree/bindings/crypto/qcom-qce.yaml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/devicetree/bindings/crypto/qcom-qce.yaml b/Documentation/devicetree/bindings/crypto/qcom-qce.yaml
index 08febd66c22ba8220860f1a59403782d12f8531f..0f378073ddf550ff5954fbe169d5d262a4e46dcf 100644
--- a/Documentation/devicetree/bindings/crypto/qcom-qce.yaml
+++ b/Documentation/devicetree/bindings/crypto/qcom-qce.yaml
@@ -14,6 +14,9 @@ description:
This document defines the binding for the QCE crypto
controller found on Qualcomm parts.
+ This driver is no longer used and so this binding only exists
+ for backwards compatibility.
+
properties:
compatible:
oneOf:
--
2.54.0
^ permalink raw reply related
* [PATCH 0/2] Delete the Qualcomm crypto engine
From: Demi Marie Obenour via B4 Relay @ 2026-05-23 19:03 UTC (permalink / raw)
To: Herbert Xu, David S. Miller, Thara Gopinath, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
Russell King
Cc: linux-kernel, linux-crypto, linux-arm-msm, Eric Biggers,
Ard Biesheuvel, devicetree, linux-arm-kernel, Demi Marie Obenour
The only realistic uses I can think of are:
1. Very weak devices where QCE is actually faster.
2. Devices without bitsliced NEON.
Do any such devices exist in the wild? I have no idea.
Not even compile-tested, but should be trivial as it just deletes code.
I didn't change the device tree beyond marking the bindings as
deprecated.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
---
Demi Marie Obenour (2):
crypto: Delete Qualcomm crypto engine driver
devicetree: Mark QCE bindings as deprecated
.../devicetree/bindings/crypto/qcom-qce.yaml | 3 +
MAINTAINERS | 8 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm64/configs/defconfig | 1 -
drivers/crypto/Kconfig | 111 ---
drivers/crypto/Makefile | 1 -
drivers/crypto/qce/Makefile | 9 -
drivers/crypto/qce/aead.c | 841 ---------------------
drivers/crypto/qce/aead.h | 56 --
drivers/crypto/qce/cipher.h | 56 --
drivers/crypto/qce/common.c | 595 ---------------
drivers/crypto/qce/common.h | 104 ---
drivers/crypto/qce/core.c | 271 -------
drivers/crypto/qce/core.h | 64 --
drivers/crypto/qce/dma.c | 135 ----
drivers/crypto/qce/dma.h | 47 --
drivers/crypto/qce/regs-v5.h | 326 --------
drivers/crypto/qce/sha.c | 545 -------------
drivers/crypto/qce/sha.h | 72 --
drivers/crypto/qce/skcipher.c | 529 -------------
20 files changed, 3 insertions(+), 3772 deletions(-)
---
base-commit: 49e05bb00f2e8168695f7af4d694c39e1423e8a2
change-id: 20260523-delete-qce-0363d22a8596
Best regards,
--
Demi Marie Obenour <demiobenour@gmail.com>
^ permalink raw reply
* [PATCH 5/5] crypto: talitos - rename first/last to first_desc/last_desc
From: Goetz Goerisch @ 2026-05-23 15:10 UTC (permalink / raw)
To: gregkh
Cc: ggoerisch, herbert, herve.codina, linux-crypto, miquel.raynal,
paul.louvel, sashal, stable, thomas.petazzoni
In-Reply-To: <20260523151048.14914-1-ggoerisch@gmail.com>
From: Paul Louvel <paul.louvel@bootlin.com>
commit a1b80018b8cec27fc06a8b04a7f8b5f6cfe86eae upstream.
Previous commit introduces a new last_request variable in the context
structure.
Renaming the first/last existing member variable in the context
structure to improve readability.
Cc: stable@vger.kernel.org
Signed-off-by: Paul Louvel <paul.louvel@bootlin.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
drivers/crypto/talitos.c | 46 ++++++++++++++++++++--------------------
1 file changed, 23 insertions(+), 23 deletions(-)
diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c
index ea6ae72c71ad..fb1adc2956b8 100644
--- a/drivers/crypto/talitos.c
+++ b/drivers/crypto/talitos.c
@@ -869,8 +869,8 @@ struct talitos_ahash_req_ctx {
u8 buf[2][HASH_MAX_BLOCK_SIZE];
int buf_idx;
unsigned int swinit;
- unsigned int first;
- unsigned int last;
+ unsigned int first_desc;
+ unsigned int last_desc;
unsigned int last_request;
unsigned int to_hash_later;
unsigned int nbuf;
@@ -889,8 +889,8 @@ struct talitos_export_state {
u32 hw_context[TALITOS_MDEU_MAX_CONTEXT_SIZE / sizeof(u32)];
u8 buf[HASH_MAX_BLOCK_SIZE];
unsigned int swinit;
- unsigned int first;
- unsigned int last;
+ unsigned int first_desc;
+ unsigned int last_desc;
unsigned int to_hash_later;
unsigned int nbuf;
};
@@ -1722,7 +1722,7 @@ static void common_nonsnoop_hash_unmap(struct device *dev,
if (desc->next_desc &&
desc->ptr[5].ptr != desc2->ptr[5].ptr)
unmap_single_talitos_ptr(dev, &desc2->ptr[5], DMA_FROM_DEVICE);
- if (req_ctx->last)
+ if (req_ctx->last_desc)
memcpy(areq->result, req_ctx->hw_context,
crypto_ahash_digestsize(tfm));
@@ -1759,7 +1759,7 @@ static void ahash_done(struct device *dev,
container_of(desc, struct talitos_edesc, desc);
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- if (!req_ctx->last && req_ctx->to_hash_later) {
+ if (!req_ctx->last_desc && req_ctx->to_hash_later) {
/* Position any partial block for next update/final/finup */
req_ctx->buf_idx = (req_ctx->buf_idx + 1) & 1;
req_ctx->nbuf = req_ctx->to_hash_later;
@@ -1825,7 +1825,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
/* first DWORD empty */
/* hash context in */
- if (!req_ctx->first || req_ctx->swinit) {
+ if (!req_ctx->first_desc || req_ctx->swinit) {
map_single_talitos_ptr_nosync(dev, &desc->ptr[1],
req_ctx->hw_context_size,
req_ctx->hw_context,
@@ -1833,7 +1833,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
req_ctx->swinit = 0;
}
/* Indicate next op is not the first. */
- req_ctx->first = 0;
+ req_ctx->first_desc = 0;
/* HMAC key */
if (ctx->keylen)
@@ -1866,7 +1866,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
/* fifth DWORD empty */
/* hash/HMAC out -or- hash context out */
- if (req_ctx->last)
+ if (req_ctx->last_desc)
map_single_talitos_ptr(dev, &desc->ptr[5],
crypto_ahash_digestsize(tfm),
req_ctx->hw_context, DMA_FROM_DEVICE);
@@ -1908,7 +1908,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
if (sg_count > 1)
sync_needed = true;
copy_talitos_ptr(&desc2->ptr[5], &desc->ptr[5], is_sec1);
- if (req_ctx->last)
+ if (req_ctx->last_desc)
map_single_talitos_ptr_nosync(dev, &desc->ptr[5],
req_ctx->hw_context_size,
req_ctx->hw_context,
@@ -1964,7 +1964,7 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
bool is_sec1 = has_ftr_sec1(priv);
u8 *ctx_buf = req_ctx->buf[req_ctx->buf_idx];
- if (!req_ctx->last && (nbytes + req_ctx->nbuf <= blocksize)) {
+ if (!req_ctx->last_desc && (nbytes + req_ctx->nbuf <= blocksize)) {
/* Buffer up to one whole block */
nents = sg_nents_for_len(req_ctx->request_sl, nbytes);
if (nents < 0) {
@@ -1981,7 +1981,7 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
nbytes_to_hash = nbytes + req_ctx->nbuf;
to_hash_later = nbytes_to_hash & (blocksize - 1);
- if (req_ctx->last)
+ if (req_ctx->last_desc)
to_hash_later = 0;
else if (to_hash_later)
/* There is a partial block. Hash the full block(s) now */
@@ -2041,19 +2041,19 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
edesc->desc.hdr = ctx->desc_hdr_template;
/* On last one, request SEC to pad; otherwise continue */
- if (req_ctx->last)
+ if (req_ctx->last_desc)
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_PAD;
else
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_CONT;
/* request SEC to INIT hash. */
- if (req_ctx->first && !req_ctx->swinit)
+ if (req_ctx->first_desc && !req_ctx->swinit)
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_INIT;
/* When the tfm context has a keylen, it's an HMAC.
* A first or last (ie. not middle) descriptor must request HMAC.
*/
- if (ctx->keylen && (req_ctx->first || req_ctx->last))
+ if (ctx->keylen && (req_ctx->first_desc || req_ctx->last_desc))
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_HMAC;
return common_nonsnoop_hash(edesc, req_ctx->areq, nbytes_to_hash, ahash_done);
@@ -2076,7 +2076,7 @@ static void sec1_ahash_process_remaining(struct work_struct *work)
req_ctx->remaining_ahash_request_bytes;
if (req_ctx->last_request)
- req_ctx->last = 1;
+ req_ctx->last_desc = 1;
}
err = ahash_process_req_one(req_ctx->areq,
@@ -2103,7 +2103,7 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
if (nbytes > TALITOS1_MAX_DATA_LEN)
nbytes = TALITOS1_MAX_DATA_LEN;
else if (req_ctx->last_request)
- req_ctx->last = 1;
+ req_ctx->last_desc = 1;
}
req_ctx->current_ahash_request_bytes = nbytes;
@@ -2124,14 +2124,14 @@ static int ahash_init(struct ahash_request *areq)
/* Initialize the context */
req_ctx->buf_idx = 0;
req_ctx->nbuf = 0;
- req_ctx->first = 1; /* first indicates h/w must init its context */
+ req_ctx->first_desc = 1; /* first_desc indicates h/w must init its context */
req_ctx->swinit = 0; /* assume h/w init of context */
size = (crypto_ahash_digestsize(tfm) <= SHA256_DIGEST_SIZE)
? TALITOS_MDEU_CONTEXT_SIZE_MD5_SHA1_SHA256
: TALITOS_MDEU_CONTEXT_SIZE_SHA384_SHA512;
req_ctx->hw_context_size = size;
req_ctx->last_request = 0;
- req_ctx->last = 0;
+ req_ctx->last_desc = 0;
INIT_WORK(&req_ctx->sec1_ahash_process_remaining, sec1_ahash_process_remaining);
dma = dma_map_single(dev, req_ctx->hw_context, req_ctx->hw_context_size,
@@ -2224,8 +2224,8 @@ static int ahash_export(struct ahash_request *areq, void *out)
req_ctx->hw_context_size);
memcpy(export->buf, req_ctx->buf[req_ctx->buf_idx], req_ctx->nbuf);
export->swinit = req_ctx->swinit;
- export->first = req_ctx->first;
- export->last = req_ctx->last;
+ export->first_desc = req_ctx->first_desc;
+ export->last_desc = req_ctx->last_desc;
export->to_hash_later = req_ctx->to_hash_later;
export->nbuf = req_ctx->nbuf;
@@ -2250,8 +2250,8 @@ static int ahash_import(struct ahash_request *areq, const void *in)
memcpy(req_ctx->hw_context, export->hw_context, size);
memcpy(req_ctx->buf[0], export->buf, export->nbuf);
req_ctx->swinit = export->swinit;
- req_ctx->first = export->first;
- req_ctx->last = export->last;
+ req_ctx->first_desc = export->first_desc;
+ req_ctx->last_desc = export->last_desc;
req_ctx->to_hash_later = export->to_hash_later;
req_ctx->nbuf = export->nbuf;
--
2.54.0
^ permalink raw reply related
* [PATCH 4/5] crypto: talitos - fix SEC1 32k ahash request limitation
From: Goetz Goerisch @ 2026-05-23 15:10 UTC (permalink / raw)
To: gregkh
Cc: ggoerisch, herbert, herve.codina, linux-crypto, miquel.raynal,
paul.louvel, sashal, stable, thomas.petazzoni
In-Reply-To: <20260523151048.14914-1-ggoerisch@gmail.com>
From: Paul Louvel <paul.louvel@bootlin.com>
commit 655ef638a2bc3cd0a9eff99a02f83cab94a3a917 upstream.
Since commit c662b043cdca ("crypto: af_alg/hash: Support
MSG_SPLICE_PAGES"), the crypto core may pass large scatterlists spanning
multiple pages to drivers supporting ahash operations. As a result, a
driver can now receive large ahash requests.
The SEC1 engine has a limitation where a single descriptor cannot
process more than 32k of data. The current implementation attempts to
handle the entire request within a single descriptor, which leads to
failures raised by the driver:
"length exceeds h/w max limit"
Address this limitation by splitting large ahash requests into multiple
descriptors, each respecting the 32k hardware limit. This allows
processing arbitrarily large requests.
Cc: stable@vger.kernel.org
Fixes: c662b043cdca ("crypto: af_alg/hash: Support MSG_SPLICE_PAGES")
Signed-off-by: Paul Louvel <paul.louvel@bootlin.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
drivers/crypto/talitos.c | 216 ++++++++++++++++++++++++++-------------
1 file changed, 147 insertions(+), 69 deletions(-)
diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c
index a941ec08817e..ea6ae72c71ad 100644
--- a/drivers/crypto/talitos.c
+++ b/drivers/crypto/talitos.c
@@ -12,6 +12,7 @@
* All rights reserved.
*/
+#include <linux/workqueue.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/mod_devicetable.h>
@@ -870,10 +871,18 @@ struct talitos_ahash_req_ctx {
unsigned int swinit;
unsigned int first;
unsigned int last;
+ unsigned int last_request;
unsigned int to_hash_later;
unsigned int nbuf;
struct scatterlist bufsl[2];
struct scatterlist *psrc;
+
+ struct scatterlist request_bufsl[2];
+ struct ahash_request *areq;
+ struct scatterlist *request_sl;
+ unsigned int remaining_ahash_request_bytes;
+ unsigned int current_ahash_request_bytes;
+ struct work_struct sec1_ahash_process_remaining;
};
struct talitos_export_state {
@@ -1759,7 +1768,20 @@ static void ahash_done(struct device *dev,
kfree(edesc);
- ahash_request_complete(areq, err);
+ if (err) {
+ ahash_request_complete(areq, err);
+ return;
+ }
+
+ req_ctx->remaining_ahash_request_bytes -=
+ req_ctx->current_ahash_request_bytes;
+
+ if (!req_ctx->remaining_ahash_request_bytes) {
+ ahash_request_complete(areq, 0);
+ return;
+ }
+
+ schedule_work(&req_ctx->sec1_ahash_process_remaining);
}
/*
@@ -1925,60 +1947,7 @@ static struct talitos_edesc *ahash_edesc_alloc(struct ahash_request *areq,
nbytes, 0, 0, 0, areq->base.flags, false);
}
-static int ahash_init(struct ahash_request *areq)
-{
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
- struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
- struct device *dev = ctx->dev;
- struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- unsigned int size;
- dma_addr_t dma;
-
- /* Initialize the context */
- req_ctx->buf_idx = 0;
- req_ctx->nbuf = 0;
- req_ctx->first = 1; /* first indicates h/w must init its context */
- req_ctx->swinit = 0; /* assume h/w init of context */
- size = (crypto_ahash_digestsize(tfm) <= SHA256_DIGEST_SIZE)
- ? TALITOS_MDEU_CONTEXT_SIZE_MD5_SHA1_SHA256
- : TALITOS_MDEU_CONTEXT_SIZE_SHA384_SHA512;
- req_ctx->hw_context_size = size;
-
- dma = dma_map_single(dev, req_ctx->hw_context, req_ctx->hw_context_size,
- DMA_TO_DEVICE);
- dma_unmap_single(dev, dma, req_ctx->hw_context_size, DMA_TO_DEVICE);
-
- return 0;
-}
-
-/*
- * on h/w without explicit sha224 support, we initialize h/w context
- * manually with sha224 constants, and tell it to run sha256.
- */
-static int ahash_init_sha224_swinit(struct ahash_request *areq)
-{
- struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
-
- req_ctx->hw_context[0] = SHA224_H0;
- req_ctx->hw_context[1] = SHA224_H1;
- req_ctx->hw_context[2] = SHA224_H2;
- req_ctx->hw_context[3] = SHA224_H3;
- req_ctx->hw_context[4] = SHA224_H4;
- req_ctx->hw_context[5] = SHA224_H5;
- req_ctx->hw_context[6] = SHA224_H6;
- req_ctx->hw_context[7] = SHA224_H7;
-
- /* init 64-bit count */
- req_ctx->hw_context[8] = 0;
- req_ctx->hw_context[9] = 0;
-
- ahash_init(areq);
- req_ctx->swinit = 1;/* prevent h/w initting context with sha256 values*/
-
- return 0;
-}
-
-static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
+static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes)
{
struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
@@ -1997,12 +1966,12 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
if (!req_ctx->last && (nbytes + req_ctx->nbuf <= blocksize)) {
/* Buffer up to one whole block */
- nents = sg_nents_for_len(areq->src, nbytes);
+ nents = sg_nents_for_len(req_ctx->request_sl, nbytes);
if (nents < 0) {
dev_err(dev, "Invalid number of src SG.\n");
return nents;
}
- sg_copy_to_buffer(areq->src, nents,
+ sg_copy_to_buffer(req_ctx->request_sl, nents,
ctx_buf + req_ctx->nbuf, nbytes);
req_ctx->nbuf += nbytes;
return 0;
@@ -2029,7 +1998,7 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
sg_init_table(req_ctx->bufsl, nsg);
sg_set_buf(req_ctx->bufsl, ctx_buf, req_ctx->nbuf);
if (nsg > 1)
- sg_chain(req_ctx->bufsl, 2, areq->src);
+ sg_chain(req_ctx->bufsl, 2, req_ctx->request_sl);
req_ctx->psrc = req_ctx->bufsl;
} else if (is_sec1 && req_ctx->nbuf && req_ctx->nbuf < blocksize) {
int offset;
@@ -2038,26 +2007,26 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
offset = blocksize - req_ctx->nbuf;
else
offset = nbytes_to_hash - req_ctx->nbuf;
- nents = sg_nents_for_len(areq->src, offset);
+ nents = sg_nents_for_len(req_ctx->request_sl, offset);
if (nents < 0) {
dev_err(dev, "Invalid number of src SG.\n");
return nents;
}
- sg_copy_to_buffer(areq->src, nents,
+ sg_copy_to_buffer(req_ctx->request_sl, nents,
ctx_buf + req_ctx->nbuf, offset);
req_ctx->nbuf += offset;
- req_ctx->psrc = scatterwalk_ffwd(req_ctx->bufsl, areq->src,
+ req_ctx->psrc = scatterwalk_ffwd(req_ctx->bufsl, req_ctx->request_sl,
offset);
} else
- req_ctx->psrc = areq->src;
+ req_ctx->psrc = req_ctx->request_sl;
if (to_hash_later) {
- nents = sg_nents_for_len(areq->src, nbytes);
+ nents = sg_nents_for_len(req_ctx->request_sl, nbytes);
if (nents < 0) {
dev_err(dev, "Invalid number of src SG.\n");
return nents;
}
- sg_pcopy_to_buffer(areq->src, nents,
+ sg_pcopy_to_buffer(req_ctx->request_sl, nents,
req_ctx->buf[(req_ctx->buf_idx + 1) & 1],
to_hash_later,
nbytes - to_hash_later);
@@ -2065,7 +2034,7 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
req_ctx->to_hash_later = to_hash_later;
/* Allocate extended descriptor */
- edesc = ahash_edesc_alloc(areq, nbytes_to_hash);
+ edesc = ahash_edesc_alloc(req_ctx->areq, nbytes_to_hash);
if (IS_ERR(edesc))
return PTR_ERR(edesc);
@@ -2087,14 +2056,123 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
if (ctx->keylen && (req_ctx->first || req_ctx->last))
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_HMAC;
- return common_nonsnoop_hash(edesc, areq, nbytes_to_hash, ahash_done);
+ return common_nonsnoop_hash(edesc, req_ctx->areq, nbytes_to_hash, ahash_done);
}
-static int ahash_update(struct ahash_request *areq)
+static void sec1_ahash_process_remaining(struct work_struct *work)
+{
+ struct talitos_ahash_req_ctx *req_ctx =
+ container_of(work, struct talitos_ahash_req_ctx,
+ sec1_ahash_process_remaining);
+ int err = 0;
+
+ req_ctx->request_sl = scatterwalk_ffwd(req_ctx->request_bufsl,
+ req_ctx->request_sl, TALITOS1_MAX_DATA_LEN);
+
+ if (req_ctx->remaining_ahash_request_bytes > TALITOS1_MAX_DATA_LEN)
+ req_ctx->current_ahash_request_bytes = TALITOS1_MAX_DATA_LEN;
+ else {
+ req_ctx->current_ahash_request_bytes =
+ req_ctx->remaining_ahash_request_bytes;
+
+ if (req_ctx->last_request)
+ req_ctx->last = 1;
+ }
+
+ err = ahash_process_req_one(req_ctx->areq,
+ req_ctx->current_ahash_request_bytes);
+
+ if (err != -EINPROGRESS)
+ ahash_request_complete(req_ctx->areq, err);
+}
+
+static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
+ struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct device *dev = ctx->dev;
+ struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
+ struct talitos_private *priv = dev_get_drvdata(dev);
+ bool is_sec1 = has_ftr_sec1(priv);
+
+ req_ctx->areq = areq;
+ req_ctx->request_sl = areq->src;
+ req_ctx->remaining_ahash_request_bytes = nbytes;
+
+ if (is_sec1) {
+ if (nbytes > TALITOS1_MAX_DATA_LEN)
+ nbytes = TALITOS1_MAX_DATA_LEN;
+ else if (req_ctx->last_request)
+ req_ctx->last = 1;
+ }
+
+ req_ctx->current_ahash_request_bytes = nbytes;
+
+ return ahash_process_req_one(req_ctx->areq,
+ req_ctx->current_ahash_request_bytes);
+}
+
+static int ahash_init(struct ahash_request *areq)
{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
+ struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct device *dev = ctx->dev;
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
+ unsigned int size;
+ dma_addr_t dma;
+ /* Initialize the context */
+ req_ctx->buf_idx = 0;
+ req_ctx->nbuf = 0;
+ req_ctx->first = 1; /* first indicates h/w must init its context */
+ req_ctx->swinit = 0; /* assume h/w init of context */
+ size = (crypto_ahash_digestsize(tfm) <= SHA256_DIGEST_SIZE)
+ ? TALITOS_MDEU_CONTEXT_SIZE_MD5_SHA1_SHA256
+ : TALITOS_MDEU_CONTEXT_SIZE_SHA384_SHA512;
+ req_ctx->hw_context_size = size;
+ req_ctx->last_request = 0;
req_ctx->last = 0;
+ INIT_WORK(&req_ctx->sec1_ahash_process_remaining, sec1_ahash_process_remaining);
+
+ dma = dma_map_single(dev, req_ctx->hw_context, req_ctx->hw_context_size,
+ DMA_TO_DEVICE);
+ dma_unmap_single(dev, dma, req_ctx->hw_context_size, DMA_TO_DEVICE);
+
+ return 0;
+}
+
+/*
+ * on h/w without explicit sha224 support, we initialize h/w context
+ * manually with sha224 constants, and tell it to run sha256.
+ */
+static int ahash_init_sha224_swinit(struct ahash_request *areq)
+{
+ struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
+
+ req_ctx->hw_context[0] = SHA224_H0;
+ req_ctx->hw_context[1] = SHA224_H1;
+ req_ctx->hw_context[2] = SHA224_H2;
+ req_ctx->hw_context[3] = SHA224_H3;
+ req_ctx->hw_context[4] = SHA224_H4;
+ req_ctx->hw_context[5] = SHA224_H5;
+ req_ctx->hw_context[6] = SHA224_H6;
+ req_ctx->hw_context[7] = SHA224_H7;
+
+ /* init 64-bit count */
+ req_ctx->hw_context[8] = 0;
+ req_ctx->hw_context[9] = 0;
+
+ ahash_init(areq);
+ req_ctx->swinit = 1;/* prevent h/w initting context with sha256 values*/
+
+ return 0;
+}
+
+static int ahash_update(struct ahash_request *areq)
+{
+ struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
+
+ req_ctx->last_request = 0;
return ahash_process_req(areq, areq->nbytes);
}
@@ -2103,7 +2181,7 @@ static int ahash_final(struct ahash_request *areq)
{
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- req_ctx->last = 1;
+ req_ctx->last_request = 1;
return ahash_process_req(areq, 0);
}
@@ -2112,7 +2190,7 @@ static int ahash_finup(struct ahash_request *areq)
{
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- req_ctx->last = 1;
+ req_ctx->last_request = 1;
return ahash_process_req(areq, areq->nbytes);
}
--
2.54.0
^ permalink raw reply related
* [PATCH 3/5] crypto: talitos - stop using crypto_ahash::init
From: Goetz Goerisch @ 2026-05-23 15:10 UTC (permalink / raw)
To: gregkh
Cc: ggoerisch, herbert, herve.codina, linux-crypto, miquel.raynal,
paul.louvel, sashal, stable, thomas.petazzoni, Eric Biggers
In-Reply-To: <20260523151048.14914-1-ggoerisch@gmail.com>
From: Eric Biggers <ebiggers@google.com>
commit 9826d1d6ed5f86cb3d61610b3b1fe31e96a40418 upstream.
The function pointer crypto_ahash::init is an internal implementation
detail of the ahash API that exists to help it support both ahash and
shash algorithms. With an upcoming refactoring of how the ahash API
supports shash algorithms, this field will be removed.
Some drivers are invoking crypto_ahash::init to call into their own
code, which is unnecessary and inefficient. The talitos driver is one
of those drivers. Make it just call its own code directly.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
drivers/crypto/talitos.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c
index 4ca4fbd227bc..a941ec08817e 100644
--- a/drivers/crypto/talitos.c
+++ b/drivers/crypto/talitos.c
@@ -2119,13 +2119,14 @@ static int ahash_finup(struct ahash_request *areq)
static int ahash_digest(struct ahash_request *areq)
{
- struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
-
- ahash->init(areq);
- req_ctx->last = 1;
+ ahash_init(areq);
+ return ahash_finup(areq);
+}
- return ahash_process_req(areq, areq->nbytes);
+static int ahash_digest_sha224_swinit(struct ahash_request *areq)
+{
+ ahash_init_sha224_swinit(areq);
+ return ahash_finup(areq);
}
static int ahash_export(struct ahash_request *areq, void *out)
@@ -3242,6 +3243,8 @@ static struct talitos_crypto_alg *talitos_alg_alloc(struct device *dev,
(!strcmp(alg->cra_name, "sha224") ||
!strcmp(alg->cra_name, "hmac(sha224)"))) {
t_alg->algt.alg.hash.init = ahash_init_sha224_swinit;
+ t_alg->algt.alg.hash.digest =
+ ahash_digest_sha224_swinit;
t_alg->algt.desc_hdr_template =
DESC_HDR_TYPE_COMMON_NONSNOOP_NO_AFEU |
DESC_HDR_SEL0_MDEUA |
--
2.54.0
^ permalink raw reply related
* [PATCH 2/5] Revert "crypto: talitos - fix SEC1 32k ahash request limitation"
From: Goetz Goerisch @ 2026-05-23 15:10 UTC (permalink / raw)
To: gregkh
Cc: ggoerisch, herbert, herve.codina, linux-crypto, miquel.raynal,
paul.louvel, sashal, stable, thomas.petazzoni
In-Reply-To: <20260523151048.14914-1-ggoerisch@gmail.com>
This reverts commit 00463d5f864ae28b7938d5acd0ddd800d5457e8b.
---
drivers/crypto/talitos.c | 216 +++++++++++++--------------------------
1 file changed, 69 insertions(+), 147 deletions(-)
diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c
index f78a44f99101..4ca4fbd227bc 100644
--- a/drivers/crypto/talitos.c
+++ b/drivers/crypto/talitos.c
@@ -12,7 +12,6 @@
* All rights reserved.
*/
-#include <linux/workqueue.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/mod_devicetable.h>
@@ -871,18 +870,10 @@ struct talitos_ahash_req_ctx {
unsigned int swinit;
unsigned int first;
unsigned int last;
- unsigned int last_request;
unsigned int to_hash_later;
unsigned int nbuf;
struct scatterlist bufsl[2];
struct scatterlist *psrc;
-
- struct scatterlist request_bufsl[2];
- struct ahash_request *areq;
- struct scatterlist *request_sl;
- unsigned int remaining_ahash_request_bytes;
- unsigned int current_ahash_request_bytes;
- struct work_struct sec1_ahash_process_remaining;
};
struct talitos_export_state {
@@ -1768,20 +1759,7 @@ static void ahash_done(struct device *dev,
kfree(edesc);
- if (err) {
- ahash_request_complete(areq, err);
- return;
- }
-
- req_ctx->remaining_ahash_request_bytes -=
- req_ctx->current_ahash_request_bytes;
-
- if (!req_ctx->remaining_ahash_request_bytes) {
- ahash_request_complete(areq, 0);
- return;
- }
-
- schedule_work(&req_ctx->sec1_ahash_process_remaining);
+ ahash_request_complete(areq, err);
}
/*
@@ -1947,7 +1925,60 @@ static struct talitos_edesc *ahash_edesc_alloc(struct ahash_request *areq,
nbytes, 0, 0, 0, areq->base.flags, false);
}
-static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes)
+static int ahash_init(struct ahash_request *areq)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
+ struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct device *dev = ctx->dev;
+ struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
+ unsigned int size;
+ dma_addr_t dma;
+
+ /* Initialize the context */
+ req_ctx->buf_idx = 0;
+ req_ctx->nbuf = 0;
+ req_ctx->first = 1; /* first indicates h/w must init its context */
+ req_ctx->swinit = 0; /* assume h/w init of context */
+ size = (crypto_ahash_digestsize(tfm) <= SHA256_DIGEST_SIZE)
+ ? TALITOS_MDEU_CONTEXT_SIZE_MD5_SHA1_SHA256
+ : TALITOS_MDEU_CONTEXT_SIZE_SHA384_SHA512;
+ req_ctx->hw_context_size = size;
+
+ dma = dma_map_single(dev, req_ctx->hw_context, req_ctx->hw_context_size,
+ DMA_TO_DEVICE);
+ dma_unmap_single(dev, dma, req_ctx->hw_context_size, DMA_TO_DEVICE);
+
+ return 0;
+}
+
+/*
+ * on h/w without explicit sha224 support, we initialize h/w context
+ * manually with sha224 constants, and tell it to run sha256.
+ */
+static int ahash_init_sha224_swinit(struct ahash_request *areq)
+{
+ struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
+
+ req_ctx->hw_context[0] = SHA224_H0;
+ req_ctx->hw_context[1] = SHA224_H1;
+ req_ctx->hw_context[2] = SHA224_H2;
+ req_ctx->hw_context[3] = SHA224_H3;
+ req_ctx->hw_context[4] = SHA224_H4;
+ req_ctx->hw_context[5] = SHA224_H5;
+ req_ctx->hw_context[6] = SHA224_H6;
+ req_ctx->hw_context[7] = SHA224_H7;
+
+ /* init 64-bit count */
+ req_ctx->hw_context[8] = 0;
+ req_ctx->hw_context[9] = 0;
+
+ ahash_init(areq);
+ req_ctx->swinit = 1;/* prevent h/w initting context with sha256 values*/
+
+ return 0;
+}
+
+static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
{
struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
@@ -1966,12 +1997,12 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
if (!req_ctx->last && (nbytes + req_ctx->nbuf <= blocksize)) {
/* Buffer up to one whole block */
- nents = sg_nents_for_len(req_ctx->request_sl, nbytes);
+ nents = sg_nents_for_len(areq->src, nbytes);
if (nents < 0) {
dev_err(dev, "Invalid number of src SG.\n");
return nents;
}
- sg_copy_to_buffer(req_ctx->request_sl, nents,
+ sg_copy_to_buffer(areq->src, nents,
ctx_buf + req_ctx->nbuf, nbytes);
req_ctx->nbuf += nbytes;
return 0;
@@ -1998,7 +2029,7 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
sg_init_table(req_ctx->bufsl, nsg);
sg_set_buf(req_ctx->bufsl, ctx_buf, req_ctx->nbuf);
if (nsg > 1)
- sg_chain(req_ctx->bufsl, 2, req_ctx->request_sl);
+ sg_chain(req_ctx->bufsl, 2, areq->src);
req_ctx->psrc = req_ctx->bufsl;
} else if (is_sec1 && req_ctx->nbuf && req_ctx->nbuf < blocksize) {
int offset;
@@ -2007,26 +2038,26 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
offset = blocksize - req_ctx->nbuf;
else
offset = nbytes_to_hash - req_ctx->nbuf;
- nents = sg_nents_for_len(req_ctx->request_sl, offset);
+ nents = sg_nents_for_len(areq->src, offset);
if (nents < 0) {
dev_err(dev, "Invalid number of src SG.\n");
return nents;
}
- sg_copy_to_buffer(req_ctx->request_sl, nents,
+ sg_copy_to_buffer(areq->src, nents,
ctx_buf + req_ctx->nbuf, offset);
req_ctx->nbuf += offset;
- req_ctx->psrc = scatterwalk_ffwd(req_ctx->bufsl, req_ctx->request_sl,
+ req_ctx->psrc = scatterwalk_ffwd(req_ctx->bufsl, areq->src,
offset);
} else
- req_ctx->psrc = req_ctx->request_sl;
+ req_ctx->psrc = areq->src;
if (to_hash_later) {
- nents = sg_nents_for_len(req_ctx->request_sl, nbytes);
+ nents = sg_nents_for_len(areq->src, nbytes);
if (nents < 0) {
dev_err(dev, "Invalid number of src SG.\n");
return nents;
}
- sg_pcopy_to_buffer(req_ctx->request_sl, nents,
+ sg_pcopy_to_buffer(areq->src, nents,
req_ctx->buf[(req_ctx->buf_idx + 1) & 1],
to_hash_later,
nbytes - to_hash_later);
@@ -2034,7 +2065,7 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
req_ctx->to_hash_later = to_hash_later;
/* Allocate extended descriptor */
- edesc = ahash_edesc_alloc(req_ctx->areq, nbytes_to_hash);
+ edesc = ahash_edesc_alloc(areq, nbytes_to_hash);
if (IS_ERR(edesc))
return PTR_ERR(edesc);
@@ -2056,123 +2087,14 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
if (ctx->keylen && (req_ctx->first || req_ctx->last))
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_HMAC;
- return common_nonsnoop_hash(edesc, req_ctx->areq, nbytes_to_hash, ahash_done);
-}
-
-static void sec1_ahash_process_remaining(struct work_struct *work)
-{
- struct talitos_ahash_req_ctx *req_ctx =
- container_of(work, struct talitos_ahash_req_ctx,
- sec1_ahash_process_remaining);
- int err = 0;
-
- req_ctx->request_sl = scatterwalk_ffwd(req_ctx->request_bufsl,
- req_ctx->request_sl, TALITOS1_MAX_DATA_LEN);
-
- if (req_ctx->remaining_ahash_request_bytes > TALITOS1_MAX_DATA_LEN)
- req_ctx->current_ahash_request_bytes = TALITOS1_MAX_DATA_LEN;
- else {
- req_ctx->current_ahash_request_bytes =
- req_ctx->remaining_ahash_request_bytes;
-
- if (req_ctx->last_request)
- req_ctx->last = 1;
- }
-
- err = ahash_process_req_one(req_ctx->areq,
- req_ctx->current_ahash_request_bytes);
-
- if (err != -EINPROGRESS)
- ahash_request_complete(req_ctx->areq, err);
-}
-
-static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
-{
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
- struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
- struct device *dev = ctx->dev;
- struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- struct talitos_private *priv = dev_get_drvdata(dev);
- bool is_sec1 = has_ftr_sec1(priv);
-
- req_ctx->areq = areq;
- req_ctx->request_sl = areq->src;
- req_ctx->remaining_ahash_request_bytes = nbytes;
-
- if (is_sec1) {
- if (nbytes > TALITOS1_MAX_DATA_LEN)
- nbytes = TALITOS1_MAX_DATA_LEN;
- else if (req_ctx->last_request)
- req_ctx->last = 1;
- }
-
- req_ctx->current_ahash_request_bytes = nbytes;
-
- return ahash_process_req_one(req_ctx->areq,
- req_ctx->current_ahash_request_bytes);
-}
-
-static int ahash_init(struct ahash_request *areq)
-{
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq);
- struct talitos_ctx *ctx = crypto_ahash_ctx(tfm);
- struct device *dev = ctx->dev;
- struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- unsigned int size;
- dma_addr_t dma;
-
- /* Initialize the context */
- req_ctx->buf_idx = 0;
- req_ctx->nbuf = 0;
- req_ctx->first = 1; /* first indicates h/w must init its context */
- req_ctx->swinit = 0; /* assume h/w init of context */
- size = (crypto_ahash_digestsize(tfm) <= SHA256_DIGEST_SIZE)
- ? TALITOS_MDEU_CONTEXT_SIZE_MD5_SHA1_SHA256
- : TALITOS_MDEU_CONTEXT_SIZE_SHA384_SHA512;
- req_ctx->hw_context_size = size;
- req_ctx->last_request = 0;
- req_ctx->last = 0;
- INIT_WORK(&req_ctx->sec1_ahash_process_remaining, sec1_ahash_process_remaining);
-
- dma = dma_map_single(dev, req_ctx->hw_context, req_ctx->hw_context_size,
- DMA_TO_DEVICE);
- dma_unmap_single(dev, dma, req_ctx->hw_context_size, DMA_TO_DEVICE);
-
- return 0;
-}
-
-/*
- * on h/w without explicit sha224 support, we initialize h/w context
- * manually with sha224 constants, and tell it to run sha256.
- */
-static int ahash_init_sha224_swinit(struct ahash_request *areq)
-{
- struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
-
- req_ctx->hw_context[0] = SHA224_H0;
- req_ctx->hw_context[1] = SHA224_H1;
- req_ctx->hw_context[2] = SHA224_H2;
- req_ctx->hw_context[3] = SHA224_H3;
- req_ctx->hw_context[4] = SHA224_H4;
- req_ctx->hw_context[5] = SHA224_H5;
- req_ctx->hw_context[6] = SHA224_H6;
- req_ctx->hw_context[7] = SHA224_H7;
-
- /* init 64-bit count */
- req_ctx->hw_context[8] = 0;
- req_ctx->hw_context[9] = 0;
-
- ahash_init(areq);
- req_ctx->swinit = 1;/* prevent h/w initting context with sha256 values*/
-
- return 0;
+ return common_nonsnoop_hash(edesc, areq, nbytes_to_hash, ahash_done);
}
static int ahash_update(struct ahash_request *areq)
{
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- req_ctx->last_request = 0;
+ req_ctx->last = 0;
return ahash_process_req(areq, areq->nbytes);
}
@@ -2181,7 +2103,7 @@ static int ahash_final(struct ahash_request *areq)
{
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- req_ctx->last_request = 1;
+ req_ctx->last = 1;
return ahash_process_req(areq, 0);
}
@@ -2190,7 +2112,7 @@ static int ahash_finup(struct ahash_request *areq)
{
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- req_ctx->last_request = 1;
+ req_ctx->last = 1;
return ahash_process_req(areq, areq->nbytes);
}
--
2.54.0
^ permalink raw reply related
* [PATCH 1/5] Revert "crypto: talitos - rename first/last to first_desc/last_desc"
From: Goetz Goerisch @ 2026-05-23 15:10 UTC (permalink / raw)
To: gregkh
Cc: ggoerisch, herbert, herve.codina, linux-crypto, miquel.raynal,
paul.louvel, sashal, stable, thomas.petazzoni
In-Reply-To: <20260523151048.14914-1-ggoerisch@gmail.com>
This reverts commit a866e2b1c65edaee2e1bb1024ee2c761ced335f8.
---
drivers/crypto/talitos.c | 46 ++++++++++++++++++++--------------------
1 file changed, 23 insertions(+), 23 deletions(-)
diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c
index 347483f6fc5d..f78a44f99101 100644
--- a/drivers/crypto/talitos.c
+++ b/drivers/crypto/talitos.c
@@ -869,8 +869,8 @@ struct talitos_ahash_req_ctx {
u8 buf[2][HASH_MAX_BLOCK_SIZE];
int buf_idx;
unsigned int swinit;
- unsigned int first_desc;
- unsigned int last_desc;
+ unsigned int first;
+ unsigned int last;
unsigned int last_request;
unsigned int to_hash_later;
unsigned int nbuf;
@@ -889,8 +889,8 @@ struct talitos_export_state {
u32 hw_context[TALITOS_MDEU_MAX_CONTEXT_SIZE / sizeof(u32)];
u8 buf[HASH_MAX_BLOCK_SIZE];
unsigned int swinit;
- unsigned int first_desc;
- unsigned int last_desc;
+ unsigned int first;
+ unsigned int last;
unsigned int to_hash_later;
unsigned int nbuf;
};
@@ -1722,7 +1722,7 @@ static void common_nonsnoop_hash_unmap(struct device *dev,
if (desc->next_desc &&
desc->ptr[5].ptr != desc2->ptr[5].ptr)
unmap_single_talitos_ptr(dev, &desc2->ptr[5], DMA_FROM_DEVICE);
- if (req_ctx->last_desc)
+ if (req_ctx->last)
memcpy(areq->result, req_ctx->hw_context,
crypto_ahash_digestsize(tfm));
@@ -1759,7 +1759,7 @@ static void ahash_done(struct device *dev,
container_of(desc, struct talitos_edesc, desc);
struct talitos_ahash_req_ctx *req_ctx = ahash_request_ctx(areq);
- if (!req_ctx->last_desc && req_ctx->to_hash_later) {
+ if (!req_ctx->last && req_ctx->to_hash_later) {
/* Position any partial block for next update/final/finup */
req_ctx->buf_idx = (req_ctx->buf_idx + 1) & 1;
req_ctx->nbuf = req_ctx->to_hash_later;
@@ -1825,7 +1825,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
/* first DWORD empty */
/* hash context in */
- if (!req_ctx->first_desc || req_ctx->swinit) {
+ if (!req_ctx->first || req_ctx->swinit) {
map_single_talitos_ptr_nosync(dev, &desc->ptr[1],
req_ctx->hw_context_size,
req_ctx->hw_context,
@@ -1833,7 +1833,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
req_ctx->swinit = 0;
}
/* Indicate next op is not the first. */
- req_ctx->first_desc = 0;
+ req_ctx->first = 0;
/* HMAC key */
if (ctx->keylen)
@@ -1866,7 +1866,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
/* fifth DWORD empty */
/* hash/HMAC out -or- hash context out */
- if (req_ctx->last_desc)
+ if (req_ctx->last)
map_single_talitos_ptr(dev, &desc->ptr[5],
crypto_ahash_digestsize(tfm),
req_ctx->hw_context, DMA_FROM_DEVICE);
@@ -1908,7 +1908,7 @@ static int common_nonsnoop_hash(struct talitos_edesc *edesc,
if (sg_count > 1)
sync_needed = true;
copy_talitos_ptr(&desc2->ptr[5], &desc->ptr[5], is_sec1);
- if (req_ctx->last_desc)
+ if (req_ctx->last)
map_single_talitos_ptr_nosync(dev, &desc->ptr[5],
req_ctx->hw_context_size,
req_ctx->hw_context,
@@ -1964,7 +1964,7 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
bool is_sec1 = has_ftr_sec1(priv);
u8 *ctx_buf = req_ctx->buf[req_ctx->buf_idx];
- if (!req_ctx->last_desc && (nbytes + req_ctx->nbuf <= blocksize)) {
+ if (!req_ctx->last && (nbytes + req_ctx->nbuf <= blocksize)) {
/* Buffer up to one whole block */
nents = sg_nents_for_len(req_ctx->request_sl, nbytes);
if (nents < 0) {
@@ -1981,7 +1981,7 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
nbytes_to_hash = nbytes + req_ctx->nbuf;
to_hash_later = nbytes_to_hash & (blocksize - 1);
- if (req_ctx->last_desc)
+ if (req_ctx->last)
to_hash_later = 0;
else if (to_hash_later)
/* There is a partial block. Hash the full block(s) now */
@@ -2041,19 +2041,19 @@ static int ahash_process_req_one(struct ahash_request *areq, unsigned int nbytes
edesc->desc.hdr = ctx->desc_hdr_template;
/* On last one, request SEC to pad; otherwise continue */
- if (req_ctx->last_desc)
+ if (req_ctx->last)
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_PAD;
else
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_CONT;
/* request SEC to INIT hash. */
- if (req_ctx->first_desc && !req_ctx->swinit)
+ if (req_ctx->first && !req_ctx->swinit)
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_INIT;
/* When the tfm context has a keylen, it's an HMAC.
* A first or last (ie. not middle) descriptor must request HMAC.
*/
- if (ctx->keylen && (req_ctx->first_desc || req_ctx->last_desc))
+ if (ctx->keylen && (req_ctx->first || req_ctx->last))
edesc->desc.hdr |= DESC_HDR_MODE0_MDEU_HMAC;
return common_nonsnoop_hash(edesc, req_ctx->areq, nbytes_to_hash, ahash_done);
@@ -2076,7 +2076,7 @@ static void sec1_ahash_process_remaining(struct work_struct *work)
req_ctx->remaining_ahash_request_bytes;
if (req_ctx->last_request)
- req_ctx->last_desc = 1;
+ req_ctx->last = 1;
}
err = ahash_process_req_one(req_ctx->areq,
@@ -2103,7 +2103,7 @@ static int ahash_process_req(struct ahash_request *areq, unsigned int nbytes)
if (nbytes > TALITOS1_MAX_DATA_LEN)
nbytes = TALITOS1_MAX_DATA_LEN;
else if (req_ctx->last_request)
- req_ctx->last_desc = 1;
+ req_ctx->last = 1;
}
req_ctx->current_ahash_request_bytes = nbytes;
@@ -2124,14 +2124,14 @@ static int ahash_init(struct ahash_request *areq)
/* Initialize the context */
req_ctx->buf_idx = 0;
req_ctx->nbuf = 0;
- req_ctx->first_desc = 1; /* first_desc indicates h/w must init its context */
+ req_ctx->first = 1; /* first indicates h/w must init its context */
req_ctx->swinit = 0; /* assume h/w init of context */
size = (crypto_ahash_digestsize(tfm) <= SHA256_DIGEST_SIZE)
? TALITOS_MDEU_CONTEXT_SIZE_MD5_SHA1_SHA256
: TALITOS_MDEU_CONTEXT_SIZE_SHA384_SHA512;
req_ctx->hw_context_size = size;
req_ctx->last_request = 0;
- req_ctx->last_desc = 0;
+ req_ctx->last = 0;
INIT_WORK(&req_ctx->sec1_ahash_process_remaining, sec1_ahash_process_remaining);
dma = dma_map_single(dev, req_ctx->hw_context, req_ctx->hw_context_size,
@@ -2223,8 +2223,8 @@ static int ahash_export(struct ahash_request *areq, void *out)
req_ctx->hw_context_size);
memcpy(export->buf, req_ctx->buf[req_ctx->buf_idx], req_ctx->nbuf);
export->swinit = req_ctx->swinit;
- export->first_desc = req_ctx->first_desc;
- export->last_desc = req_ctx->last_desc;
+ export->first = req_ctx->first;
+ export->last = req_ctx->last;
export->to_hash_later = req_ctx->to_hash_later;
export->nbuf = req_ctx->nbuf;
@@ -2249,8 +2249,8 @@ static int ahash_import(struct ahash_request *areq, const void *in)
memcpy(req_ctx->hw_context, export->hw_context, size);
memcpy(req_ctx->buf[0], export->buf, export->nbuf);
req_ctx->swinit = export->swinit;
- req_ctx->first_desc = export->first_desc;
- req_ctx->last_desc = export->last_desc;
+ req_ctx->first = export->first;
+ req_ctx->last = export->last;
req_ctx->to_hash_later = export->to_hash_later;
req_ctx->nbuf = export->nbuf;
--
2.54.0
^ permalink raw reply related
* [PATCH 0/5] crypto: talitos - fix rename first/last to first_desc/last_desc
From: Goetz Goerisch @ 2026-05-23 15:10 UTC (permalink / raw)
To: gregkh
Cc: ggoerisch, herbert, herve.codina, linux-crypto, miquel.raynal,
paul.louvel, sashal, stable, thomas.petazzoni
In-Reply-To: <2026052212-aged-amply-7bd8@gregkh>
Commit a1b80018b8cec27fc06a8b04a7f8b5f6cfe86eae
was backported to 6.6.y with a866e2b1c65edaee2e1bb1024ee2c761ced335f8
It renames last to last_desc but misses one occurrence which leads to compile errors on mpc85xx
drivers/crypto/talitos.c: In function 'ahash_digest':
drivers/crypto/talitos.c:2204:16: error: 'struct talitos_ahash_req_ctx' has no member named 'last'
2204 | req_ctx->last = 1;
| ^~~~
Instead of renaming req_ctx->last, commit 9826d1d6ed5f8 ("crypto: talitos - stop
using crypto_ahash::init") should be applied.
Ideally before commit 00463d5f864a ("crypto: talitos - fix SEC1 32k ahash
request limitation") to avoid any compilation breakage and ensure correctness of
the code.
> > Greg could you please backport the mentioned commit to 6.6.y in the correct order for the next update?
> Can you send a series of backported patches in the correct order for us
> to apply, so we know to get them correct? Trying to dig out from an
> email like this is usually quite easy to get wrong :)
Hope this is correct.
Goetz
Eric Biggers (1):
crypto: talitos - stop using crypto_ahash::init
Goetz Goerisch (2):
Revert "crypto: talitos - rename first/last to first_desc/last_desc"
Revert "crypto: talitos - fix SEC1 32k ahash request limitation"
Paul Louvel (2):
crypto: talitos - fix SEC1 32k ahash request limitation
crypto: talitos - rename first/last to first_desc/last_desc
drivers/crypto/talitos.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
--
2.54.0
^ permalink raw reply
* Re: [PATCH 0/3] Add packet-mode ESP offload for Airoha/EIP93
From: Jihong Min @ 2026-05-23 12:24 UTC (permalink / raw)
To: Christian Marangi, Antoine Tenart, Herbert Xu, David S . Miller,
Lorenzo Bianconi, Andrew Lunn, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Steffen Klassert
Cc: linux-kernel, linux-crypto, linux-arm-kernel, linux-mediatek,
netdev
In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com>
On 5/23/26 21:15, Jihong Min wrote:
> This series adds the missing plumbing for ESP offload engines that
> operate on whole ESP packets instead of only exposing AES/HMAC through
> the crypto API AEAD interface.
>
> The normal ESP software path can already call into accelerated AEAD
> algorithms, but packet-mode engines such as EIP93 can also generate and
> consume ESP packet framing: padding, pad length, next header and ICV.
> That needs a slightly different XFRM offload contract so the netdev
> driver can hand the skb to a packet backend rather than trying to make
> hardware fit the software trailer layout.
>
> Patch 1 extends the ESP offload infrastructure for packet engines while
> preserving the existing behavior for drivers that do not opt in.
> Patch 2 exposes an EIP93 ESP packet backend for encapsulation and
> decapsulation.
> Patch 3 wires Airoha Ethernet GDM netdevs and DSA user ports to that
> backend through xfrmdev_ops. ESP GSO and ESP TX checksum offload remain
> disabled.
>
> Runtime testing was done on a Gemtek W1700K2 running OpenWrt with the
> same changes applied on top of a 6.18.31-based kernel.
>
> Test parameters:
>
> - Static IPv4 transport-mode XFRM SAs between the AP and host.
> - ESP transform: auth hmac(sha1), enc cbc(aes) with a 128-bit AES key.
> - iperf3 TCP test, AP as client and host as server:
> iperf3 -c <host_ip> -P 4 -t 10
> - The host always used normal Linux XFRM software processing.
> - With AP ESP offload disabled, the AP also used the Linux XFRM
> software path; in this setup, EIP93-backed AEAD crypto was still
> available to that path.
>
> Network-relevant test setup:
>
> - AP: Gemtek W1700K2, Airoha AN7581/EN7581, 4x Arm Cortex-A53 at
> 1.4 GHz, 2 GiB RAM, airoha_eth wan (GDM2) netdev, 10Gb/s full-duplex,
> MTU 9200, EIP93 crypto and IPsec packet engine present.
> - Host: AMD Ryzen 9 9950X3D, 16 cores/32 threads, Open vSwitch,
> MTU 9978, backed by a ConnectX-6 Dx 10Gb/s full-duplex link.
>
> AP to host iperf3 result:
>
> AP offload Sender Receiver Retransmits
> on 918.2 Mbit/s 913.6 Mbit/s 0
> off 782.4 Mbit/s 778.6 Mbit/s 3569
>
> This is a 17.3% receiver-side throughput improvement for the AP TX ESP
> path in this setup, with retransmits eliminated in the offloaded run.
>
> Jihong Min (3):
> xfrm: extend ESP offload infrastructure for packet engines
> crypto: inside-secure: add EIP93 ESP packet backend
> net: airoha: add EIP93-backed ESP XFRM offload
>
> MAINTAINERS | 1 +
> drivers/crypto/inside-secure/eip93/Kconfig | 10 +
> drivers/crypto/inside-secure/eip93/Makefile | 1 +
> .../crypto/inside-secure/eip93/eip93-ipsec.c | 1413 ++++++++++++++++
> .../crypto/inside-secure/eip93/eip93-main.c | 69 +-
> .../crypto/inside-secure/eip93/eip93-main.h | 38 +-
> drivers/net/ethernet/airoha/Kconfig | 11 +
> drivers/net/ethernet/airoha/Makefile | 1 +
> drivers/net/ethernet/airoha/airoha_eth.c | 51 +-
> drivers/net/ethernet/airoha/airoha_eth.h | 69 +
> drivers/net/ethernet/airoha/airoha_xfrm.c | 1474 +++++++++++++++++
> include/crypto/eip93-ipsec.h | 132 ++
> include/linux/netdevice.h | 3 +
> include/net/xfrm.h | 8 +-
> net/ipv4/esp4.c | 6 +-
> net/ipv4/esp4_offload.c | 29 +-
> net/ipv6/esp6.c | 6 +-
> net/ipv6/esp6_offload.c | 29 +-
> 18 files changed, 3324 insertions(+), 27 deletions(-)
> create mode 100644 drivers/crypto/inside-secure/eip93/eip93-ipsec.c
> create mode 100644 drivers/net/ethernet/airoha/airoha_xfrm.c
> create mode 100644 include/crypto/eip93-ipsec.h
>
One note I should have included in the cover letter:
The hardware behavior used by this series was studied from the out-of-tree
IPsec branch of the mtk-eip93 driver:
https://github.com/vschagen/mtk-eip93/tree/ipsec
That code was useful for understanding the EIP93 packet-mode ESP descriptor
programming and SA record values.
This series is not a direct import of that driver. The implementation was
rewritten around the current upstream driver layout and the Linux XFRM
netdev offload model, with EIP93 exposed as a packet-mode ESP backend used
by the Airoha netdev driver.
Sincerely,
Jihong Min
^ permalink raw reply
* [PATCH 3/3] net: airoha: add EIP93-backed ESP XFRM offload
From: Jihong Min @ 2026-05-23 12:15 UTC (permalink / raw)
To: Christian Marangi, Antoine Tenart, Herbert Xu, David S . Miller,
Lorenzo Bianconi, Andrew Lunn, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Steffen Klassert
Cc: linux-kernel, linux-crypto, linux-arm-kernel, linux-mediatek,
netdev, Jihong Min
In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com>
Wire Airoha GDM netdevs and DSA user ports to the EIP93 ESP packet
backend through xfrmdev_ops.
Gate netdev feature advertisement on backend capability, add TX and RX
submit paths, preserve opt-out builds, and handle SA lifetime across
feature changes, DSA detach, and EIP93 provider loss.
Assisted-by: Codex:gpt-5.5
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
drivers/net/ethernet/airoha/Kconfig | 11 +
drivers/net/ethernet/airoha/Makefile | 1 +
drivers/net/ethernet/airoha/airoha_eth.c | 51 +-
drivers/net/ethernet/airoha/airoha_eth.h | 69 +
drivers/net/ethernet/airoha/airoha_xfrm.c | 1474 +++++++++++++++++++++
5 files changed, 1605 insertions(+), 1 deletion(-)
create mode 100644 drivers/net/ethernet/airoha/airoha_xfrm.c
diff --git a/drivers/net/ethernet/airoha/Kconfig b/drivers/net/ethernet/airoha/Kconfig
index ad3ce501e7a5..302534c89fdd 100644
--- a/drivers/net/ethernet/airoha/Kconfig
+++ b/drivers/net/ethernet/airoha/Kconfig
@@ -31,4 +31,15 @@ config NET_AIROHA_FLOW_STATS
help
Enable Aiorha flowtable statistic counters.
+config NET_AIROHA_XFRM
+ bool "Airoha ESP XFRM offload support"
+ depends on NET_AIROHA
+ default y
+ help
+ Enable ESP XFRM offload support for Airoha Ethernet netdevs.
+
+ If unsure, say Y. Say N to opt out of advertising ESP hardware
+ offload from the Airoha Ethernet driver even when the EIP93 IPsec
+ packet backend and XFRM offload support are available.
+
endif #NET_VENDOR_AIROHA
diff --git a/drivers/net/ethernet/airoha/Makefile b/drivers/net/ethernet/airoha/Makefile
index 94468053e34b..15386665bb27 100644
--- a/drivers/net/ethernet/airoha/Makefile
+++ b/drivers/net/ethernet/airoha/Makefile
@@ -5,5 +5,6 @@
obj-$(CONFIG_NET_AIROHA) += airoha-eth.o
airoha-eth-y := airoha_eth.o airoha_ppe.o
+airoha-eth-$(CONFIG_NET_AIROHA_XFRM) += airoha_xfrm.o
airoha-eth-$(CONFIG_DEBUG_FS) += airoha_ppe_debugfs.o
obj-$(CONFIG_NET_AIROHA_NPU) += airoha_npu.o
diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c
index cecd66251dba..877002c03738 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.c
+++ b/drivers/net/ethernet/airoha/airoha_eth.c
@@ -684,6 +684,14 @@ static int airoha_qdma_rx_process(struct airoha_queue *q, int budget)
false);
done++;
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM)
+ if (airoha_xfrm_in_active(port) &&
+ airoha_xfrm_rx_skb(port, q->skb)) {
+ q->skb = NULL;
+ continue;
+ }
+#endif
+
napi_gro_receive(&q->napi, q->skb);
q->skb = NULL;
continue;
@@ -2010,6 +2018,19 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
void *data;
u16 index;
u8 fport;
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM)
+ int err;
+
+ if (airoha_xfrm_out_active(port)) {
+ err = airoha_xfrm_encrypt_skb(port, skb);
+ if (err == -EINPROGRESS)
+ return NETDEV_TX_OK;
+ if (err == -EBUSY)
+ return NETDEV_TX_BUSY;
+ if (err)
+ goto error;
+ }
+#endif
qid = airoha_qdma_get_txq(qdma, skb_get_queue_mapping(skb));
tag = airoha_get_dsa_tag(skb, dev);
@@ -2895,6 +2916,8 @@ static const struct net_device_ops airoha_netdev_ops = {
.ndo_stop = airoha_dev_stop,
.ndo_change_mtu = airoha_dev_change_mtu,
.ndo_select_queue = airoha_dev_select_queue,
+ .ndo_fix_features = airoha_xfrm_fix_features,
+ .ndo_set_features = airoha_xfrm_set_features,
.ndo_start_xmit = airoha_dev_xmit,
.ndo_get_stats64 = airoha_dev_get_stats64,
.ndo_set_mac_address = airoha_dev_set_macaddr,
@@ -3025,6 +3048,7 @@ static int airoha_alloc_gdm_port(struct airoha_eth *eth,
/* XXX: Read nbq from DTS */
port->nbq = id == AIROHA_GDM3_IDX && airoha_is_7581(eth) ? 4 : 0;
eth->ports[p] = port;
+ airoha_xfrm_build_netdev(dev);
return airoha_metadata_dst_alloc(port);
}
@@ -3155,6 +3179,7 @@ static int airoha_probe(struct platform_device *pdev)
if (port->dev->reg_state == NETREG_REGISTERED)
unregister_netdev(port->dev);
+ airoha_xfrm_teardown_netdev(port->dev);
airoha_metadata_dst_free(port);
}
airoha_hw_cleanup(eth);
@@ -3180,6 +3205,7 @@ static void airoha_remove(struct platform_device *pdev)
continue;
unregister_netdev(port->dev);
+ airoha_xfrm_teardown_netdev(port->dev);
airoha_metadata_dst_free(port);
}
airoha_hw_cleanup(eth);
@@ -3328,7 +3354,30 @@ static struct platform_driver airoha_driver = {
.of_match_table = of_airoha_match,
},
};
-module_platform_driver(airoha_driver);
+
+static int __init airoha_init(void)
+{
+ int err;
+
+ err = airoha_xfrm_register_notifier();
+ if (err)
+ return err;
+
+ err = platform_driver_register(&airoha_driver);
+ if (err)
+ airoha_xfrm_unregister_notifier();
+
+ return err;
+}
+
+static void __exit airoha_exit(void)
+{
+ platform_driver_unregister(&airoha_driver);
+ airoha_xfrm_unregister_notifier();
+}
+
+module_init(airoha_init);
+module_exit(airoha_exit);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Lorenzo Bianconi <lorenzo@kernel.org>");
diff --git a/drivers/net/ethernet/airoha/airoha_eth.h b/drivers/net/ethernet/airoha/airoha_eth.h
index 4fad3acc3ccf..4fe04c763271 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.h
+++ b/drivers/net/ethernet/airoha/airoha_eth.h
@@ -11,6 +11,8 @@
#include <linux/etherdevice.h>
#include <linux/iopoll.h>
#include <linux/kernel.h>
+#include <linux/kconfig.h>
+#include <linux/jump_label.h>
#include <linux/netdevice.h>
#include <linux/reset.h>
#include <linux/soc/airoha/airoha_offload.h>
@@ -533,6 +535,12 @@ struct airoha_qdma {
struct airoha_queue q_rx[AIROHA_NUM_RX_RING];
};
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM)
+struct eip93_ipsec;
+DECLARE_STATIC_KEY_FALSE(airoha_xfrm_in_state_key);
+DECLARE_STATIC_KEY_FALSE(airoha_xfrm_out_state_key);
+#endif
+
struct airoha_gdm_port {
struct airoha_qdma *qdma;
struct airoha_eth *eth;
@@ -549,6 +557,13 @@ struct airoha_gdm_port {
u64 fwd_tx_packets;
struct metadata_dst *dsa_meta[AIROHA_MAX_DSA_PORTS];
+
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM)
+ struct eip93_ipsec *xfrm_ipsec;
+ atomic_t xfrm_state_count;
+ atomic_t xfrm_out_state_count;
+ atomic_t xfrm_in_state_count;
+#endif
};
#define AIROHA_RXD4_PPE_CPU_REASON GENMASK(20, 16)
@@ -683,4 +698,58 @@ static inline int airoha_ppe_debugfs_init(struct airoha_ppe *ppe)
}
#endif
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM)
+static inline bool airoha_xfrm_in_active(struct airoha_gdm_port *port)
+{
+ return static_branch_unlikely(&airoha_xfrm_in_state_key) &&
+ atomic_read(&port->xfrm_in_state_count);
+}
+
+static inline bool airoha_xfrm_out_active(struct airoha_gdm_port *port)
+{
+ return static_branch_unlikely(&airoha_xfrm_out_state_key) &&
+ atomic_read(&port->xfrm_out_state_count);
+}
+
+void airoha_xfrm_build_netdev(struct net_device *dev);
+void airoha_xfrm_teardown_netdev(struct net_device *dev);
+netdev_features_t airoha_xfrm_fix_features(struct net_device *dev,
+ netdev_features_t features);
+int airoha_xfrm_set_features(struct net_device *dev,
+ netdev_features_t features);
+bool airoha_xfrm_rx_skb(struct airoha_gdm_port *port, struct sk_buff *skb);
+int airoha_xfrm_encrypt_skb(struct airoha_gdm_port *port, struct sk_buff *skb);
+int airoha_xfrm_register_notifier(void);
+void airoha_xfrm_unregister_notifier(void);
+#else
+static inline void airoha_xfrm_build_netdev(struct net_device *dev)
+{
+}
+
+static inline void airoha_xfrm_teardown_netdev(struct net_device *dev)
+{
+}
+
+static inline netdev_features_t
+airoha_xfrm_fix_features(struct net_device *dev, netdev_features_t features)
+{
+ return features;
+}
+
+static inline int airoha_xfrm_set_features(struct net_device *dev,
+ netdev_features_t features)
+{
+ return 0;
+}
+
+static inline int airoha_xfrm_register_notifier(void)
+{
+ return 0;
+}
+
+static inline void airoha_xfrm_unregister_notifier(void)
+{
+}
+#endif
+
#endif /* AIROHA_ETH_H */
diff --git a/drivers/net/ethernet/airoha/airoha_xfrm.c b/drivers/net/ethernet/airoha/airoha_xfrm.c
new file mode 100644
index 000000000000..58461954d098
--- /dev/null
+++ b/drivers/net/ethernet/airoha/airoha_xfrm.c
@@ -0,0 +1,1474 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026 Jihong Min <hurryman2212@gmail.com>
+ */
+#include <crypto/eip93-ipsec.h>
+#include <linux/err.h>
+#include <linux/kmod.h>
+#include <linux/rtnetlink.h>
+#include <linux/slab.h>
+#include <linux/udp.h>
+#include <net/dst_metadata.h>
+#include <net/esp.h>
+#include <net/ip.h>
+#include <net/ip6_checksum.h>
+#include <net/ipv6.h>
+#include <net/net_namespace.h>
+#include <net/xfrm.h>
+
+#include "airoha_eth.h"
+
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM)
+DEFINE_STATIC_KEY_FALSE(airoha_xfrm_in_state_key);
+DEFINE_STATIC_KEY_FALSE(airoha_xfrm_out_state_key);
+#endif
+
+#if IS_ENABLED(CONFIG_NET_AIROHA_XFRM) && \
+ IS_REACHABLE(CONFIG_CRYPTO_DEV_EIP93) && \
+ IS_ENABLED(CONFIG_CRYPTO_DEV_EIP93_IPSEC) && \
+ IS_REACHABLE(CONFIG_INET_ESP) && \
+ IS_REACHABLE(CONFIG_INET_ESP_OFFLOAD) && \
+ IS_ENABLED(CONFIG_XFRM_OFFLOAD)
+#define AIROHA_XFRM_FEATURES \
+ (NETIF_F_HW_ESP | NETIF_F_HW_ESP_TX_CSUM | NETIF_F_GSO_ESP)
+
+struct airoha_xfrm_state {
+ struct airoha_gdm_port *port;
+ struct eip93_ipsec_sa *sa;
+};
+
+static netdev_features_t airoha_xfrm_ipsec_features(struct eip93_ipsec *ipsec)
+{
+ netdev_features_t features = 0;
+ u32 ipsec_features;
+
+ ipsec_features = eip93_ipsec_features(ipsec);
+ if (ipsec_features & EIP93_IPSEC_FEATURE_ESP)
+ features |= NETIF_F_HW_ESP;
+ if (ipsec_features & EIP93_IPSEC_FEATURE_HW_ESP_TX_CSUM)
+ features |= NETIF_F_HW_ESP_TX_CSUM;
+ if (ipsec_features & EIP93_IPSEC_FEATURE_GSO_ESP)
+ features |= NETIF_F_GSO_ESP;
+
+ return features;
+}
+
+static int airoha_xfrm_request_module(struct net_device *dev,
+ const char *module_name)
+{
+ int err;
+
+ err = request_module("%s", module_name);
+ if (err) {
+ netdev_err(dev, "failed requesting module %s: %d\n",
+ module_name, err);
+ return err < 0 ? err : -ENOENT;
+ }
+
+ return 0;
+}
+
+static int airoha_xfrm_request_modules(struct net_device *dev)
+{
+ int err;
+
+ if (IS_MODULE(CONFIG_INET_ESP)) {
+ err = airoha_xfrm_request_module(dev, "esp4");
+ if (err)
+ return err;
+ }
+
+ if (IS_MODULE(CONFIG_INET_ESP_OFFLOAD)) {
+ err = airoha_xfrm_request_module(dev, "esp4_offload");
+ if (err)
+ return err;
+ }
+
+#if IS_REACHABLE(CONFIG_INET6_ESP)
+ if (IS_MODULE(CONFIG_INET6_ESP)) {
+ err = airoha_xfrm_request_module(dev, "esp6");
+ if (err)
+ return err;
+ }
+#endif
+
+#if IS_REACHABLE(CONFIG_INET6_ESP_OFFLOAD)
+ if (IS_MODULE(CONFIG_INET6_ESP_OFFLOAD)) {
+ err = airoha_xfrm_request_module(dev, "esp6_offload");
+ if (err)
+ return err;
+ }
+#endif
+
+ if (IS_MODULE(CONFIG_CRYPTO_DEV_EIP93)) {
+ err = airoha_xfrm_request_module(dev, "crypto-hw-eip93");
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static int airoha_xfrm_prepare_ipsec(struct net_device *dev)
+{
+ struct airoha_gdm_port *port = netdev_priv(dev);
+ struct eip93_ipsec *ipsec;
+ int err;
+
+ if (port->xfrm_ipsec)
+ return eip93_ipsec_available(port->xfrm_ipsec) ? 0 : -ENODEV;
+
+ err = airoha_xfrm_request_modules(dev);
+ if (err)
+ return err;
+
+ ipsec = eip93_ipsec_get(port->eth->dev);
+ if (IS_ERR(ipsec)) {
+ netdev_dbg(dev,
+ "EIP93 ESP packet backend is unavailable: %ld\n",
+ PTR_ERR(ipsec));
+ return PTR_ERR(ipsec);
+ }
+
+ port->xfrm_ipsec = ipsec;
+ netdev_info(dev, "ESP HW offload available via EIP93 packet backend\n");
+
+ return 0;
+}
+
+static bool airoha_xfrm_state_supported(struct xfrm_state *x,
+ struct netlink_ext_ack *extack)
+{
+ if (x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "only XFRM crypto offload is supported");
+ return false;
+ }
+
+ switch (x->xso.dir) {
+ case XFRM_DEV_OFFLOAD_OUT:
+ case XFRM_DEV_OFFLOAD_IN:
+ break;
+ default:
+ NL_SET_ERR_MSG_MOD(extack, "only in/out SAs are supported");
+ return false;
+ }
+
+ switch (x->props.family) {
+ case AF_INET:
+ break;
+#if IS_REACHABLE(CONFIG_INET6_ESP) && IS_REACHABLE(CONFIG_INET6_ESP_OFFLOAD)
+ case AF_INET6:
+ break;
+#endif
+ default:
+ NL_SET_ERR_MSG_MOD(extack,
+ "only IPv4/IPv6 ESP offload is supported");
+ return false;
+ }
+
+ if (x->outer_mode.family != x->props.family) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "only same-family ESP offload is supported");
+ return false;
+ }
+
+ if (x->id.proto != IPPROTO_ESP) {
+ NL_SET_ERR_MSG_MOD(extack, "only ESP offload is supported");
+ return false;
+ }
+
+ switch (x->props.mode) {
+ case XFRM_MODE_TUNNEL:
+ case XFRM_MODE_TRANSPORT:
+ break;
+ default:
+ NL_SET_ERR_MSG_MOD(extack,
+ "only tunnel/transport modes are supported");
+ return false;
+ }
+
+ if (x->outer_mode.encap != x->props.mode) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "outer ESP mode does not match state mode");
+ return false;
+ }
+
+ if (x->encap) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "NAT-T is unsupported by EIP93 packet ESP");
+ return false;
+ }
+
+ if (x->tfcpad) {
+ NL_SET_ERR_MSG_MOD(extack, "TFC padding is not supported");
+ return false;
+ }
+
+ if (x->aead) {
+ NL_SET_ERR_MSG_MOD(extack, "AEAD SAs are unsupported");
+ return false;
+ }
+
+ if (!x->ealg || !x->aalg) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "encryption/authentication required");
+ return false;
+ }
+
+ return true;
+}
+
+static const struct xfrmdev_ops airoha_xfrmdev_ops;
+
+#if IS_ENABLED(CONFIG_NET_DSA)
+static struct airoha_gdm_port *airoha_xfrm_dsa_dev_port(struct net_device *dev)
+{
+ struct net_device *conduit;
+ struct dsa_port *dp;
+
+ if (!dsa_user_dev_check(dev))
+ return NULL;
+
+ dp = dsa_port_from_netdev(dev);
+ if (IS_ERR(dp))
+ return NULL;
+
+ conduit = dsa_port_to_conduit(dp);
+ if (!conduit || conduit->xfrmdev_ops != &airoha_xfrmdev_ops)
+ return NULL;
+
+ return netdev_priv(conduit);
+}
+
+static struct net_device *airoha_xfrm_dsa_rx_dev(struct airoha_gdm_port *port,
+ struct sk_buff *skb)
+{
+ struct metadata_dst *md_dst = skb_metadata_dst(skb);
+ struct dsa_port *cpu_dp = port->dev->dsa_ptr;
+ struct dsa_port *dp;
+ u32 source_port;
+
+ if (!md_dst || md_dst->type != METADATA_HW_PORT_MUX)
+ return port->dev;
+
+ if (!cpu_dp || !cpu_dp->dst)
+ return NULL;
+
+ source_port = md_dst->u.port_info.port_id;
+ list_for_each_entry(dp, &cpu_dp->dst->ports, list) {
+ if (dp->type != DSA_PORT_TYPE_USER ||
+ dp->index != source_port || dp->cpu_dp != cpu_dp ||
+ dsa_port_to_conduit(dp) != port->dev || !dp->user)
+ continue;
+
+ return dp->user;
+ }
+
+ return NULL;
+}
+
+static bool airoha_xfrm_dsa_user_matches_port(struct net_device *user,
+ struct net_device *conduit)
+{
+ struct dsa_port *dp;
+
+ if (!dsa_user_dev_check(user))
+ return false;
+
+ dp = dsa_port_from_netdev(user);
+ if (IS_ERR(dp))
+ return false;
+
+ return dsa_port_to_conduit(dp) == conduit;
+}
+#else
+static struct airoha_gdm_port *airoha_xfrm_dsa_dev_port(struct net_device *dev)
+{
+ return NULL;
+}
+
+static struct net_device *airoha_xfrm_dsa_rx_dev(struct airoha_gdm_port *port,
+ struct sk_buff *skb)
+{
+ return port->dev;
+}
+#endif
+
+static struct airoha_gdm_port *airoha_xfrm_dev_port(struct net_device *dev)
+{
+ struct airoha_gdm_port *port;
+
+ if (dev->xfrmdev_ops != &airoha_xfrmdev_ops)
+ return NULL;
+
+ port = airoha_xfrm_dsa_dev_port(dev);
+ if (port)
+ return port;
+
+ return netdev_priv(dev);
+}
+
+static netdev_features_t airoha_xfrm_dev_features(struct net_device *dev)
+{
+ struct airoha_gdm_port *port = airoha_xfrm_dev_port(dev);
+
+ if (!port || !port->xfrm_ipsec)
+ return 0;
+
+ return airoha_xfrm_ipsec_features(port->xfrm_ipsec);
+}
+
+static struct net_device *airoha_xfrm_rx_dev(struct airoha_gdm_port *port,
+ struct sk_buff *skb)
+{
+ if (!netdev_uses_dsa(port->dev))
+ return port->dev;
+
+ return airoha_xfrm_dsa_rx_dev(port, skb);
+}
+
+static void airoha_xfrm_state_advance_esn(struct xfrm_state *x)
+{
+ struct airoha_xfrm_state *state;
+
+ state = (struct airoha_xfrm_state *)x->xso.offload_handle;
+ if (state)
+ eip93_ipsec_state_advance_esn(state->sa, x);
+}
+
+static int airoha_xfrm_state_add(struct net_device *dev, struct xfrm_state *x,
+ struct netlink_ext_ack *extack)
+{
+ struct airoha_gdm_port *port = airoha_xfrm_dev_port(dev);
+ struct airoha_xfrm_state *state;
+ int err;
+
+ if (!port) {
+ NL_SET_ERR_MSG_MOD(extack, "device lacks Airoha ESP offload");
+ return -EOPNOTSUPP;
+ }
+
+ if (!(dev->features & NETIF_F_HW_ESP)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "ESP HW offload is disabled on device");
+ return -EOPNOTSUPP;
+ }
+
+ if (!port->xfrm_ipsec || !eip93_ipsec_available(port->xfrm_ipsec)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "EIP93 packet backend is unavailable");
+ return -EOPNOTSUPP;
+ }
+
+ if (!airoha_xfrm_state_supported(x, extack))
+ return -EOPNOTSUPP;
+
+ state = kzalloc(sizeof(*state), GFP_KERNEL);
+ if (!state)
+ return -ENOMEM;
+
+ state->port = port;
+ err = eip93_ipsec_state_add(port->xfrm_ipsec, x, extack, &state->sa);
+ if (err) {
+ kfree(state);
+ return err;
+ }
+
+ x->xso.offload_handle = (unsigned long)state;
+ atomic_inc(&port->xfrm_state_count);
+ if (x->xso.dir == XFRM_DEV_OFFLOAD_OUT) {
+ atomic_inc(&port->xfrm_out_state_count);
+ static_branch_inc(&airoha_xfrm_out_state_key);
+ } else {
+ atomic_inc(&port->xfrm_in_state_count);
+ static_branch_inc(&airoha_xfrm_in_state_key);
+ }
+
+ return 0;
+}
+
+static void airoha_xfrm_state_delete(struct net_device *dev,
+ struct xfrm_state *x)
+{
+ struct airoha_xfrm_state *state;
+ struct airoha_gdm_port *port;
+
+ state = (struct airoha_xfrm_state *)x->xso.offload_handle;
+ if (!state)
+ return;
+
+ port = state->port;
+ x->xso.offload_handle = 0;
+ atomic_dec(&port->xfrm_state_count);
+ if (x->xso.dir == XFRM_DEV_OFFLOAD_OUT) {
+ atomic_dec(&port->xfrm_out_state_count);
+ static_branch_dec(&airoha_xfrm_out_state_key);
+ } else if (x->xso.dir == XFRM_DEV_OFFLOAD_IN) {
+ atomic_dec(&port->xfrm_in_state_count);
+ static_branch_dec(&airoha_xfrm_in_state_key);
+ }
+
+ eip93_ipsec_state_delete(state->sa);
+ kfree(state);
+}
+
+static bool airoha_xfrm_offload_ok(struct sk_buff *skb, struct xfrm_state *x)
+{
+ struct net_device *dev = skb->dev;
+ struct airoha_xfrm_state *state;
+ struct airoha_gdm_port *port;
+
+ if (!dev)
+ return false;
+
+ port = airoha_xfrm_dev_port(dev);
+ if (!port)
+ return false;
+
+ if (unlikely(x->xso.dir != XFRM_DEV_OFFLOAD_OUT ||
+ x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO ||
+ !(dev->features & NETIF_F_HW_ESP) || x->xso.dev != dev))
+ return false;
+
+ state = (struct airoha_xfrm_state *)x->xso.offload_handle;
+ if (!state || state->port != port)
+ return false;
+
+ if (unlikely(skb_is_gso(skb)))
+ return false;
+
+ return true;
+}
+
+/*
+ * EIP93 packet-out mode creates ESP padding, trailer and ICV. The generic ESP
+ * xmit path should reserve tailroom only for plain, non-GSO ESP packets.
+ */
+static bool airoha_xfrm_esp_tx_hw_trailer(struct sk_buff *skb,
+ struct xfrm_state *x)
+{
+ return x->xso.dir == XFRM_DEV_OFFLOAD_OUT &&
+ x->xso.type == XFRM_DEV_OFFLOAD_CRYPTO && !x->encap &&
+ !skb_is_gso(skb);
+}
+
+static const struct xfrmdev_ops airoha_xfrmdev_ops = {
+ .xdo_dev_state_add = airoha_xfrm_state_add,
+ .xdo_dev_state_delete = airoha_xfrm_state_delete,
+ .xdo_dev_state_free = airoha_xfrm_state_delete,
+ .xdo_dev_offload_ok = airoha_xfrm_offload_ok,
+ .xdo_dev_esp_tx_hw_trailer = airoha_xfrm_esp_tx_hw_trailer,
+ .xdo_dev_state_advance_esn = airoha_xfrm_state_advance_esn,
+};
+
+void airoha_xfrm_build_netdev(struct net_device *dev)
+{
+ struct airoha_gdm_port *port = netdev_priv(dev);
+ netdev_features_t features;
+
+ atomic_set(&port->xfrm_state_count, 0);
+ atomic_set(&port->xfrm_out_state_count, 0);
+ atomic_set(&port->xfrm_in_state_count, 0);
+ if (airoha_xfrm_prepare_ipsec(dev))
+ return;
+
+ features = airoha_xfrm_ipsec_features(port->xfrm_ipsec);
+ if (!(features & NETIF_F_HW_ESP)) {
+ eip93_ipsec_put(port->xfrm_ipsec);
+ port->xfrm_ipsec = NULL;
+ return;
+ }
+
+ dev->xfrmdev_ops = &airoha_xfrmdev_ops;
+ dev->hw_features |= features;
+ dev->hw_enc_features |= features;
+ dev->gso_partial_features |= features & NETIF_F_GSO_ESP;
+}
+
+void airoha_xfrm_teardown_netdev(struct net_device *dev)
+{
+ struct airoha_gdm_port *port = netdev_priv(dev);
+
+ if (port->xfrm_ipsec) {
+ eip93_ipsec_put(port->xfrm_ipsec);
+ port->xfrm_ipsec = NULL;
+ }
+}
+
+/* Airoha TX checksum/GSO offloads run after EIP93 has encrypted the skb, so
+ * they cannot operate on plaintext ESP payloads or build per-segment ESP data.
+ */
+netdev_features_t airoha_xfrm_fix_features(struct net_device *dev,
+ netdev_features_t features)
+{
+ netdev_features_t supported = airoha_xfrm_dev_features(dev);
+ netdev_features_t unsupported = AIROHA_XFRM_FEATURES & ~supported;
+
+ if (features & unsupported)
+ features &= ~unsupported;
+
+ if (!(features & NETIF_F_HW_ESP))
+ features &= ~(NETIF_F_HW_ESP_TX_CSUM | NETIF_F_GSO_ESP);
+
+ return features;
+}
+
+int airoha_xfrm_set_features(struct net_device *dev, netdev_features_t features)
+{
+ netdev_features_t changed = (dev->features ^ features) &
+ AIROHA_XFRM_FEATURES;
+ netdev_features_t requested = features & AIROHA_XFRM_FEATURES;
+ struct airoha_gdm_port *port = netdev_priv(dev);
+ netdev_features_t supported;
+ int err;
+
+ if (!changed)
+ return 0;
+
+ if (requested & NETIF_F_HW_ESP) {
+ err = airoha_xfrm_prepare_ipsec(dev);
+ if (err)
+ return err;
+ }
+
+ supported = airoha_xfrm_dev_features(dev);
+ if (requested & ~supported)
+ return -EOPNOTSUPP;
+
+ if (atomic_read(&port->xfrm_state_count)) {
+ netdev_err(dev, "cannot change ESP features with active SAs\n");
+ return -EBUSY;
+ }
+
+ if (!(features & NETIF_F_HW_ESP))
+ netdev_info(dev, "ESP HW offload disabled\n");
+
+ return 0;
+}
+
+struct airoha_xfrm_rx_info {
+ unsigned short family;
+ int encap_type;
+ int esp_offset;
+ int packet_len;
+ __be32 spi;
+ __be32 seq;
+};
+
+struct airoha_xfrm_rx_ctx {
+ struct sk_buff *skb;
+ struct net_device *dev;
+};
+
+static bool airoha_xfrm_parse_rx_ipv4(struct sk_buff *skb,
+ struct airoha_xfrm_rx_info *info)
+{
+ struct ip_esp_hdr *esph;
+ struct iphdr *iph;
+ int packet_len;
+ int iphlen;
+
+ if (!pskb_may_pull(skb, sizeof(*iph)))
+ return false;
+
+ iph = ip_hdr(skb);
+ if (iph->version != 4)
+ return false;
+
+ iphlen = iph->ihl * 4;
+ if (iphlen < sizeof(*iph) || !pskb_may_pull(skb, iphlen))
+ return false;
+
+ if (ip_is_fragment(iph))
+ return false;
+
+ packet_len = ntohs(iph->tot_len);
+ if (packet_len < iphlen || packet_len > skb->len)
+ return false;
+
+ switch (iph->protocol) {
+ case IPPROTO_ESP:
+ info->encap_type = 0;
+ info->esp_offset = iphlen;
+ info->packet_len = packet_len;
+ break;
+ case IPPROTO_UDP: {
+ struct udphdr *uh;
+ int udp_len;
+ __be32 marker;
+
+ if (!pskb_may_pull(skb, iphlen + sizeof(*uh) + sizeof(*esph)))
+ return false;
+
+ uh = (struct udphdr *)(skb->data + iphlen);
+ udp_len = ntohs(uh->len);
+ if (udp_len <= sizeof(*uh) + sizeof(*esph) ||
+ iphlen + udp_len > packet_len)
+ return false;
+
+ memcpy(&marker, skb->data + iphlen + sizeof(*uh),
+ sizeof(marker));
+ if (!marker)
+ return false;
+
+ info->encap_type = UDP_ENCAP_ESPINUDP;
+ info->esp_offset = iphlen + sizeof(*uh);
+ info->packet_len = iphlen + udp_len;
+ break;
+ }
+ default:
+ return false;
+ }
+
+ if (info->esp_offset + sizeof(*esph) > info->packet_len ||
+ !pskb_may_pull(skb, info->esp_offset + sizeof(*esph)))
+ return false;
+
+ esph = (struct ip_esp_hdr *)(skb->data + info->esp_offset);
+ info->family = AF_INET;
+ info->spi = esph->spi;
+ info->seq = esph->seq_no;
+
+ return !!info->spi;
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static bool airoha_xfrm_parse_rx_ipv6(struct sk_buff *skb,
+ struct airoha_xfrm_rx_info *info)
+{
+ struct ip_esp_hdr *esph;
+ struct ipv6hdr *ip6h;
+ __be16 frag_off;
+ int packet_len;
+ int offset;
+ u8 nexthdr;
+
+ if (!pskb_may_pull(skb, sizeof(*ip6h)))
+ return false;
+
+ ip6h = ipv6_hdr(skb);
+ if (ip6h->version != 6)
+ return false;
+
+ if (!ip6h->payload_len)
+ return false;
+
+ packet_len = sizeof(*ip6h) + ntohs(ip6h->payload_len);
+ if (packet_len < sizeof(*ip6h) || packet_len > skb->len)
+ return false;
+
+ nexthdr = ip6h->nexthdr;
+ offset = ipv6_skip_exthdr(skb, sizeof(*ip6h), &nexthdr, &frag_off);
+ if (offset < 0 || frag_off)
+ return false;
+
+ switch (nexthdr) {
+ case NEXTHDR_ESP:
+ info->encap_type = 0;
+ info->esp_offset = offset;
+ info->packet_len = packet_len;
+ break;
+ case NEXTHDR_UDP: {
+ struct udphdr *uh;
+ int udp_len;
+ __be32 marker;
+
+ if (!pskb_may_pull(skb, offset + sizeof(*uh) + sizeof(*esph)))
+ return false;
+
+ uh = (struct udphdr *)(skb->data + offset);
+ udp_len = ntohs(uh->len);
+ if (udp_len <= sizeof(*uh) + sizeof(*esph) ||
+ offset + udp_len > packet_len)
+ return false;
+
+ memcpy(&marker, skb->data + offset + sizeof(*uh),
+ sizeof(marker));
+ if (!marker)
+ return false;
+
+ info->encap_type = UDP_ENCAP_ESPINUDP;
+ info->esp_offset = offset + sizeof(*uh);
+ info->packet_len = offset + udp_len;
+ break;
+ }
+ default:
+ return false;
+ }
+
+ if (info->esp_offset + sizeof(*esph) > info->packet_len ||
+ !pskb_may_pull(skb, info->esp_offset + sizeof(*esph)))
+ return false;
+
+ esph = (struct ip_esp_hdr *)(skb->data + info->esp_offset);
+ info->family = AF_INET6;
+ info->spi = esph->spi;
+ info->seq = esph->seq_no;
+
+ return !!info->spi;
+}
+#else
+static bool airoha_xfrm_parse_rx_ipv6(struct sk_buff *skb,
+ struct airoha_xfrm_rx_info *info)
+{
+ return false;
+}
+#endif
+
+static bool airoha_xfrm_parse_rx_skb(struct sk_buff *skb,
+ struct airoha_xfrm_rx_info *info)
+{
+ switch (skb->protocol) {
+ case htons(ETH_P_IP):
+ return airoha_xfrm_parse_rx_ipv4(skb, info);
+ case htons(ETH_P_IPV6):
+ return airoha_xfrm_parse_rx_ipv6(skb, info);
+ default:
+ return false;
+ }
+}
+
+static struct xfrm_state *
+airoha_xfrm_rx_state_lookup(struct airoha_gdm_port *port, struct sk_buff *skb,
+ const struct airoha_xfrm_rx_info *info)
+{
+ struct airoha_xfrm_state *state;
+ xfrm_address_t daddr = {};
+ struct net_device *dev;
+ struct xfrm_state *x;
+
+ dev = airoha_xfrm_rx_dev(port, skb);
+ if (!dev)
+ return NULL;
+
+ switch (info->family) {
+ case AF_INET:
+ daddr.a4 = ip_hdr(skb)->daddr;
+ break;
+ case AF_INET6:
+ daddr.in6 = ipv6_hdr(skb)->daddr;
+ break;
+ default:
+ return NULL;
+ }
+
+ x = xfrm_input_state_lookup(dev_net(dev), skb->mark, &daddr, info->spi,
+ IPPROTO_ESP, info->family);
+ if (!x)
+ return NULL;
+
+ if (x->dir && x->dir != XFRM_SA_DIR_IN)
+ goto err_put;
+
+ if (x->xso.dir != XFRM_DEV_OFFLOAD_IN ||
+ x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO || x->xso.dev != dev ||
+ !(dev->features & NETIF_F_HW_ESP) || !x->type_offload ||
+ !x->type_offload->input_tail)
+ goto err_put;
+
+ state = (struct airoha_xfrm_state *)x->xso.offload_handle;
+ if (!state || state->port != port)
+ goto err_put;
+
+ if ((x->encap ? x->encap->encap_type : 0) != info->encap_type)
+ goto err_put;
+
+ return x;
+
+err_put:
+ xfrm_state_put(x);
+ return NULL;
+}
+
+static u32 airoha_xfrm_rx_status(int err, struct xfrm_state *x)
+{
+ if (!err)
+ return CRYPTO_SUCCESS;
+
+ if (err == -EBADMSG) {
+ if (x->props.mode == XFRM_MODE_TUNNEL)
+ return CRYPTO_TUNNEL_ESP_AUTH_FAILED;
+
+ return CRYPTO_TRANSPORT_ESP_AUTH_FAILED;
+ }
+
+ if (err == -EINVAL)
+ return CRYPTO_INVALID_PACKET_SYNTAX;
+
+ return CRYPTO_GENERIC_ERROR;
+}
+
+static int airoha_xfrm_rx_apply_result(struct sk_buff *skb,
+ struct xfrm_state *x,
+ struct eip93_ipsec_result result)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (!x || !result.packet_len || result.packet_len > skb->len || !xo)
+ return -EINVAL;
+
+ /*
+ * EIP93 inbound ESP mode removes the ESP pad/trailer/ICV and reports
+ * the decapsulated outer packet length plus the recovered next-header.
+ */
+ xo->proto = result.nexthdr;
+ xo->flags |= XFRM_ESP_NO_TRAILER;
+ if (pskb_trim(skb, result.packet_len))
+ return -EINVAL;
+
+ if (x->props.family == AF_INET) {
+ ip_hdr(skb)->tot_len = htons(skb->len);
+ ip_send_check(ip_hdr(skb));
+ } else if (x->props.family == AF_INET6) {
+ int len = skb->len - skb_network_offset(skb) -
+ sizeof(struct ipv6hdr);
+
+ if (len < 0)
+ return -EINVAL;
+
+ ipv6_hdr(skb)->payload_len = len > IPV6_MAXPLEN ? 0 :
+ htons(len);
+ }
+
+ return 0;
+}
+
+static void airoha_xfrm_rx_free_ctx(struct airoha_xfrm_rx_ctx *ctx)
+{
+ kfree(ctx);
+}
+
+static void airoha_xfrm_rx_finish(void *data, int err,
+ struct eip93_ipsec_result result)
+{
+ struct airoha_xfrm_rx_ctx *ctx = data;
+ struct net_device *dev = ctx->dev;
+ struct sk_buff *skb = ctx->skb;
+ struct xfrm_offload *xo;
+ struct xfrm_state *x;
+
+ x = xfrm_input_state(skb);
+ xo = xfrm_offload(skb);
+ if (!err)
+ err = airoha_xfrm_rx_apply_result(skb, x, result);
+ if (xo) {
+ xo->flags |= CRYPTO_DONE;
+ xo->status = airoha_xfrm_rx_status(err, x);
+ }
+
+ airoha_xfrm_rx_free_ctx(ctx);
+ netif_receive_skb(skb);
+ dev_put(dev);
+}
+
+static bool airoha_xfrm_tx_esp_offset(struct sk_buff *skb, struct xfrm_state *x,
+ unsigned int *esp_offset)
+{
+ u8 *esph = (u8 *)ip_esp_hdr(skb);
+
+ if (x->encap)
+ esph += sizeof(struct udphdr);
+
+ if (esph < skb->data ||
+ esph + sizeof(struct ip_esp_hdr) > skb_tail_pointer(skb))
+ return false;
+
+ *esp_offset = esph - skb->data;
+
+ return true;
+}
+
+static void airoha_xfrm_tx_update_outer_len(struct sk_buff *skb)
+{
+ struct iphdr *iph = ip_hdr(skb);
+
+ if (iph->version == 4) {
+ iph->tot_len = htons(skb->len - skb_network_offset(skb));
+ ip_send_check(iph);
+ } else if (iph->version == 6) {
+ int len = skb->len - skb_network_offset(skb) -
+ sizeof(struct ipv6hdr);
+
+ if (len < 0)
+ return;
+
+ ipv6_hdr(skb)->payload_len = len > IPV6_MAXPLEN ? 0 :
+ htons(len);
+ }
+}
+
+static void airoha_xfrm_tx_udp6_csum(struct sk_buff *skb,
+ struct xfrm_state *x)
+{
+#if IS_ENABLED(CONFIG_IPV6)
+ struct udphdr *uh;
+ struct ipv6hdr *ip6h;
+ unsigned int offset;
+ __wsum csum;
+ int len;
+
+ if (x->props.family != AF_INET6 || !x->encap ||
+ x->encap->encap_type != UDP_ENCAP_ESPINUDP)
+ return;
+
+ offset = skb_transport_offset(skb);
+ if (offset + sizeof(*uh) > skb->len)
+ return;
+
+ uh = udp_hdr(skb);
+ ip6h = ipv6_hdr(skb);
+ len = ntohs(uh->len);
+ if (len < sizeof(*uh) || len > skb->len - offset)
+ return;
+
+ uh->check = 0;
+ csum = skb_checksum(skb, offset, len, 0);
+ uh->check = csum_ipv6_magic(&ip6h->saddr, &ip6h->daddr, len,
+ IPPROTO_UDP, csum);
+ if (!uh->check)
+ uh->check = CSUM_MANGLED_0;
+ #endif
+}
+
+static int airoha_xfrm_tx_apply_result(struct sk_buff *skb,
+ struct xfrm_state *x,
+ struct eip93_ipsec_result result)
+{
+ unsigned int current_esp_len;
+ unsigned int esp_offset;
+ unsigned int new_len;
+
+ if (!result.packet_len ||
+ !airoha_xfrm_tx_esp_offset(skb, x, &esp_offset))
+ return -EINVAL;
+
+ current_esp_len = skb->len - esp_offset;
+ if (result.packet_len == current_esp_len)
+ return 0;
+
+ new_len = esp_offset + result.packet_len;
+ if (new_len < esp_offset)
+ return -EINVAL;
+
+ /*
+ * EIP93 outbound ESP mode reports the generated ESP packet length.
+ * Reflect it in skb->len before the packet resumes into the Ethernet
+ * TX path, because generic ESP left hardware-generated trailer bytes
+ * outside skb->len.
+ */
+ if (new_len > skb->len) {
+ unsigned int delta = new_len - skb->len;
+
+ if (delta > skb_tailroom(skb))
+ return -ENOMEM;
+ skb_put(skb, delta);
+
+ return 0;
+ }
+
+ return pskb_trim(skb, new_len);
+}
+
+bool airoha_xfrm_rx_skb(struct airoha_gdm_port *port, struct sk_buff *skb)
+{
+ struct airoha_xfrm_rx_info info;
+ struct airoha_xfrm_state *state;
+ struct airoha_xfrm_rx_ctx *ctx;
+ struct sk_buff *trailer;
+ struct xfrm_offload *xo;
+ struct xfrm_state *x;
+ struct sec_path *sp;
+ int err;
+ u32 mark = skb->mark;
+
+ if (!airoha_xfrm_parse_rx_skb(skb, &info))
+ return false;
+
+ x = airoha_xfrm_rx_state_lookup(port, skb, &info);
+ if (!x)
+ return false;
+
+ sp = secpath_set(skb);
+ if (!sp)
+ goto err_put_state;
+
+ if (sp->len == XFRM_MAX_DEPTH) {
+ secpath_reset(skb);
+ goto err_put_state;
+ }
+
+ skb->mark = xfrm_smark_get(mark, x);
+ sp->xvec[sp->len++] = x;
+ sp->olen++;
+ XFRM_SKB_CB(skb)->seq.input.low = info.seq;
+ XFRM_SKB_CB(skb)->seq.input.hi = htonl(xfrm_replay_seqhi(x, info.seq));
+ XFRM_SPI_SKB_CB(skb)->family = info.family;
+ XFRM_SPI_SKB_CB(skb)->seq = info.seq;
+ if (info.family == AF_INET) {
+ XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct iphdr, daddr);
+ XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = NULL;
+ } else {
+ XFRM_SPI_SKB_CB(skb)->daddroff =
+ offsetof(struct ipv6hdr, daddr);
+ XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip6 = NULL;
+ }
+
+ xo = xfrm_offload(skb);
+ if (!xo)
+ goto err_reset;
+
+ state = (struct airoha_xfrm_state *)x->xso.offload_handle;
+ if (!state || state->port != port)
+ goto err_reset;
+
+ if (skb_cloned(skb) || skb_is_nonlinear(skb)) {
+ err = skb_cow_data(skb, 0, &trailer);
+ if (err < 0)
+ goto err_reset;
+
+ if (skb_is_nonlinear(skb)) {
+ err = skb_linearize(skb);
+ if (err)
+ goto err_reset;
+ }
+ }
+
+ ctx = kmalloc(sizeof(*ctx), GFP_ATOMIC);
+ if (!ctx)
+ goto err_reset;
+
+ if (!skb->dev)
+ goto err_free_ctx;
+
+ ctx->skb = skb;
+ ctx->dev = skb->dev;
+ skb->ip_summed = CHECKSUM_NONE;
+
+ dev_hold(ctx->dev);
+ err = eip93_ipsec_receive(state->sa, skb, info.packet_len,
+ airoha_xfrm_rx_finish, ctx);
+ if (err == -EINPROGRESS)
+ return true;
+
+ dev_put(ctx->dev);
+ airoha_xfrm_rx_free_ctx(ctx);
+ skb->mark = mark;
+ secpath_reset(skb);
+
+ return false;
+
+err_free_ctx:
+ airoha_xfrm_rx_free_ctx(ctx);
+err_reset:
+ skb->mark = mark;
+ secpath_reset(skb);
+ return false;
+
+err_put_state:
+ xfrm_state_put(x);
+ return false;
+}
+
+static void airoha_xfrm_tx_done(void *data, int err,
+ struct eip93_ipsec_result result)
+{
+ struct sk_buff *skb = data;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ struct sec_path *sp = skb_sec_path(skb);
+ struct xfrm_state *x;
+
+ if (!xo || !sp || !sp->len) {
+ kfree_skb(skb);
+ return;
+ }
+
+ x = sp->xvec[sp->len - 1];
+ if (!err)
+ err = airoha_xfrm_tx_apply_result(skb, x, result);
+ if (err) {
+ XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTSTATEPROTOERROR);
+ kfree_skb(skb);
+ return;
+ }
+
+ airoha_xfrm_tx_update_outer_len(skb);
+ airoha_xfrm_tx_udp6_csum(skb, x);
+ xo->flags |= CRYPTO_DONE;
+ xo->status = CRYPTO_SUCCESS;
+ skb_push(skb, skb->data - skb_mac_header(skb));
+ secpath_reset(skb);
+ xfrm_dev_resume(skb);
+}
+
+int airoha_xfrm_encrypt_skb(struct airoha_gdm_port *port, struct sk_buff *skb)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ struct airoha_xfrm_state *state;
+ struct net_device *dev;
+ struct xfrm_state *x;
+ struct sec_path *sp;
+ struct ip_esp_hdr *esph;
+ struct sk_buff *trailer;
+ unsigned int esp_offset;
+ unsigned int tailen;
+ int err;
+
+ if (!xo || !(xo->flags & XFRM_XMIT) || (xo->flags & CRYPTO_DONE))
+ return 0;
+
+ sp = skb_sec_path(skb);
+ if (!sp || !sp->len)
+ return -EINVAL;
+
+ x = sp->xvec[sp->len - 1];
+ dev = x->xso.dev;
+ if (unlikely(x->xso.dir != XFRM_DEV_OFFLOAD_OUT ||
+ x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO || !dev ||
+ !(dev->features & NETIF_F_HW_ESP)))
+ return -EOPNOTSUPP;
+
+ state = (struct airoha_xfrm_state *)x->xso.offload_handle;
+ if (!state || state->port != port)
+ return -EOPNOTSUPP;
+
+ if (unlikely(skb_is_gso(skb)))
+ return -EOPNOTSUPP;
+
+ if (unlikely(skb->ip_summed == CHECKSUM_PARTIAL)) {
+ err = skb_checksum_help(skb);
+ if (err)
+ return err;
+ }
+
+ tailen = xo->esp_tx_tailen;
+ if (skb_cloned(skb) || skb_is_nonlinear(skb)) {
+ err = skb_cow_data(skb, tailen, &trailer);
+ if (err < 0)
+ return err;
+
+ if (skb_is_nonlinear(skb)) {
+ err = skb_linearize(skb);
+ if (err)
+ return err;
+ }
+ }
+ /*
+ * Generic ESP reserves this tailroom before the skb reaches us. Keep a
+ * small guard here because COW/linearization can replace the skb head.
+ */
+ if (tailen && skb_tailroom(skb) < tailen) {
+ err = pskb_expand_head(skb, 0, tailen - skb_tailroom(skb),
+ GFP_ATOMIC);
+ if (err)
+ return err;
+ }
+
+ if (!airoha_xfrm_tx_esp_offset(skb, x, &esp_offset))
+ return -EINVAL;
+
+ esph = (struct ip_esp_hdr *)(skb->data + esp_offset);
+ esph->seq_no = htonl(xo->seq.low);
+
+ return eip93_ipsec_xmit(state->sa, skb, esp_offset, airoha_xfrm_tx_done,
+ skb);
+}
+
+static void airoha_xfrm_flush_dev(struct net_device *dev)
+{
+ xfrm_dev_state_flush(dev_net(dev), dev, true);
+ xfrm_dev_policy_flush(dev_net(dev), dev, true);
+}
+
+static void airoha_xfrm_link_change(struct net_device *dev)
+{
+ struct airoha_gdm_port *port = airoha_xfrm_dev_port(dev);
+
+ if (!port || !(dev->hw_features & NETIF_F_HW_ESP) ||
+ !atomic_read(&port->xfrm_state_count))
+ return;
+
+ netdev_dbg(dev, "carrier %s, preserving ESP HW offload SAs\n",
+ netif_carrier_ok(dev) ? "up" : "down");
+}
+
+#if IS_ENABLED(CONFIG_NET_DSA)
+static void airoha_xfrm_dsa_attach_user(struct net_device *conduit,
+ struct net_device *user)
+{
+ netdev_features_t features = airoha_xfrm_dev_features(conduit);
+
+ if (conduit->xfrmdev_ops != &airoha_xfrmdev_ops ||
+ !airoha_xfrm_dsa_user_matches_port(user, conduit))
+ return;
+
+ if (!(features & NETIF_F_HW_ESP))
+ return;
+
+ if (user->xfrmdev_ops && user->xfrmdev_ops != &airoha_xfrmdev_ops) {
+ netdev_dbg(conduit,
+ "DSA user %s already has XFRM offload ops\n",
+ user->name);
+ return;
+ }
+
+ user->xfrmdev_ops = &airoha_xfrmdev_ops;
+ user->hw_features |= features;
+ user->hw_enc_features |= features;
+ user->gso_partial_features |= features & NETIF_F_GSO_ESP;
+ netdev_dbg(user, "ESP HW offload available via %s\n", conduit->name);
+}
+
+static void airoha_xfrm_dsa_detach_user(struct net_device *user)
+{
+ struct airoha_gdm_port *port;
+ bool active = false;
+ bool enabled;
+
+ if (user->xfrmdev_ops != &airoha_xfrmdev_ops ||
+ !dsa_user_dev_check(user))
+ return;
+
+ enabled = user->features & NETIF_F_HW_ESP;
+ port = airoha_xfrm_dsa_dev_port(user);
+ if (port)
+ active = atomic_read(&port->xfrm_state_count);
+
+ if (active) {
+ netdev_warn(user, "DSA detach with active ESP SAs, flushing\n");
+ airoha_xfrm_flush_dev(user);
+ }
+
+ user->wanted_features &= ~AIROHA_XFRM_FEATURES;
+ user->features &= ~AIROHA_XFRM_FEATURES;
+ user->hw_features &= ~AIROHA_XFRM_FEATURES;
+ user->hw_enc_features &= ~AIROHA_XFRM_FEATURES;
+ user->gso_partial_features &= ~NETIF_F_GSO_ESP;
+ user->xfrmdev_ops = NULL;
+
+ if (active || enabled)
+ netdev_features_change(user);
+}
+
+static void airoha_xfrm_dsa_feature_change(struct net_device *dev)
+{
+ struct airoha_gdm_port *port;
+
+ if (dev->xfrmdev_ops != &airoha_xfrmdev_ops ||
+ !dsa_user_dev_check(dev) || (dev->features & NETIF_F_HW_ESP))
+ return;
+
+ port = airoha_xfrm_dsa_dev_port(dev);
+ if (port && atomic_read(&port->xfrm_state_count)) {
+ netdev_warn(dev, "DSA feature lost ESP SAs, flushing\n");
+ airoha_xfrm_flush_dev(dev);
+ }
+}
+#endif
+
+static int airoha_xfrm_netdevice_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+
+ switch (event) {
+ case NETDEV_CHANGE:
+ airoha_xfrm_link_change(dev);
+ break;
+#if IS_ENABLED(CONFIG_NET_DSA)
+ case NETDEV_CHANGEUPPER: {
+ struct netdev_notifier_changeupper_info *info = ptr;
+
+ if (info->linking)
+ airoha_xfrm_dsa_attach_user(dev, info->upper_dev);
+ else
+ airoha_xfrm_dsa_detach_user(info->upper_dev);
+ break;
+ }
+ case NETDEV_FEAT_CHANGE:
+ airoha_xfrm_dsa_feature_change(dev);
+ break;
+ case NETDEV_UNREGISTER:
+ airoha_xfrm_dsa_detach_user(dev);
+ break;
+#endif
+ default:
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block airoha_xfrm_netdev_notifier = {
+ .notifier_call = airoha_xfrm_netdevice_event,
+};
+
+static int airoha_xfrm_register_netdev_notifier(void)
+{
+ return register_netdevice_notifier(&airoha_xfrm_netdev_notifier);
+}
+
+static void airoha_xfrm_unregister_netdev_notifier(void)
+{
+ unregister_netdevice_notifier(&airoha_xfrm_netdev_notifier);
+}
+
+static void airoha_xfrm_drop_dev(struct net_device *dev, const char *reason)
+{
+ struct airoha_gdm_port *port = airoha_xfrm_dev_port(dev);
+ bool advertised = dev->hw_features & AIROHA_XFRM_FEATURES;
+ bool enabled = dev->features & NETIF_F_HW_ESP;
+ bool active = false;
+
+ if (port)
+ active = atomic_read(&port->xfrm_state_count);
+
+ if (active) {
+ netdev_warn(dev, "%s, flushing ESP HW offload SAs\n", reason);
+ airoha_xfrm_flush_dev(dev);
+ }
+
+ dev->wanted_features &= ~AIROHA_XFRM_FEATURES;
+ dev->features &= ~AIROHA_XFRM_FEATURES;
+ dev->hw_features &= ~AIROHA_XFRM_FEATURES;
+ dev->hw_enc_features &= ~AIROHA_XFRM_FEATURES;
+ dev->gso_partial_features &= ~NETIF_F_GSO_ESP;
+
+ if (active || enabled || advertised)
+ netdev_features_change(dev);
+}
+
+static void airoha_xfrm_drop_ipsec(struct eip93_ipsec *ipsec,
+ const char *reason)
+{
+ struct net_device *dev;
+ struct net *net;
+
+ rtnl_lock();
+ for_each_net(net) {
+ for_each_netdev(net, dev) {
+ struct airoha_gdm_port *port;
+
+ port = airoha_xfrm_dev_port(dev);
+ if (!port || port->xfrm_ipsec != ipsec)
+ continue;
+
+ airoha_xfrm_drop_dev(dev, reason);
+ }
+ }
+
+ for_each_net(net) {
+ for_each_netdev(net, dev) {
+ struct airoha_gdm_port *port;
+
+ if (dev->xfrmdev_ops != &airoha_xfrmdev_ops)
+ continue;
+
+ if (airoha_xfrm_dsa_dev_port(dev))
+ continue;
+
+ port = netdev_priv(dev);
+ if (dev == port->dev && port->xfrm_ipsec == ipsec) {
+ eip93_ipsec_put(port->xfrm_ipsec);
+ port->xfrm_ipsec = NULL;
+ }
+ }
+ }
+
+ for_each_net(net) {
+ for_each_netdev(net, dev) {
+ if (dev->xfrmdev_ops == &airoha_xfrmdev_ops &&
+ !(dev->hw_features & NETIF_F_HW_ESP))
+ dev->xfrmdev_ops = NULL;
+ }
+ }
+ rtnl_unlock();
+}
+
+static int airoha_xfrm_ipsec_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ switch (event) {
+ case EIP93_IPSEC_EVENT_REMOVE:
+ airoha_xfrm_drop_ipsec(ptr, "EIP93 provider removed");
+ break;
+ case EIP93_IPSEC_EVENT_RESET:
+ airoha_xfrm_drop_ipsec(ptr, "EIP93 provider reset");
+ break;
+ case EIP93_IPSEC_EVENT_DMA_ERROR:
+ airoha_xfrm_drop_ipsec(ptr, "EIP93 DMA error");
+ break;
+ case EIP93_IPSEC_EVENT_CAPABILITY_LOSS:
+ airoha_xfrm_drop_ipsec(ptr, "EIP93 capability loss");
+ break;
+ default:
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block airoha_xfrm_ipsec_notifier = {
+ .notifier_call = airoha_xfrm_ipsec_event,
+};
+
+int airoha_xfrm_register_notifier(void)
+{
+ int err;
+
+ err = airoha_xfrm_register_netdev_notifier();
+ if (err)
+ return err;
+
+ err = eip93_ipsec_register_notifier(&airoha_xfrm_ipsec_notifier);
+ if (err)
+ airoha_xfrm_unregister_netdev_notifier();
+
+ return err;
+}
+
+void airoha_xfrm_unregister_notifier(void)
+{
+ eip93_ipsec_unregister_notifier(&airoha_xfrm_ipsec_notifier);
+ airoha_xfrm_unregister_netdev_notifier();
+}
+#else
+void airoha_xfrm_build_netdev(struct net_device *dev)
+{
+}
+
+void airoha_xfrm_teardown_netdev(struct net_device *dev)
+{
+}
+
+netdev_features_t airoha_xfrm_fix_features(struct net_device *dev,
+ netdev_features_t features)
+{
+ return features & ~(NETIF_F_HW_ESP_TX_CSUM | NETIF_F_GSO_ESP);
+}
+
+int airoha_xfrm_set_features(struct net_device *dev, netdev_features_t features)
+{
+ return 0;
+}
+
+bool airoha_xfrm_rx_skb(struct airoha_gdm_port *port, struct sk_buff *skb)
+{
+ return false;
+}
+
+int airoha_xfrm_encrypt_skb(struct airoha_gdm_port *port, struct sk_buff *skb)
+{
+ return 0;
+}
+
+int airoha_xfrm_register_notifier(void)
+{
+ return 0;
+}
+
+void airoha_xfrm_unregister_notifier(void)
+{
+}
+
+#endif
--
2.53.0
^ permalink raw reply related
* [PATCH 2/3] crypto: inside-secure: add EIP93 ESP packet backend
From: Jihong Min @ 2026-05-23 12:15 UTC (permalink / raw)
To: Christian Marangi, Antoine Tenart, Herbert Xu, David S . Miller,
Lorenzo Bianconi, Andrew Lunn, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Steffen Klassert
Cc: linux-kernel, linux-crypto, linux-arm-kernel, linux-mediatek,
netdev, Jihong Min
In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com>
Expose an EIP93 packet-mode IPsec backend for netdev drivers that need
ESP encapsulation and decapsulation offload without advertising EIP93
itself as a netdev.
Add provider selection, capability reporting, SA lifecycle management,
IPsec request completion, and provider fault notification around the
existing EIP93 descriptor path.
Assisted-by: Codex:gpt-5.5
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
MAINTAINERS | 1 +
drivers/crypto/inside-secure/eip93/Kconfig | 10 +
drivers/crypto/inside-secure/eip93/Makefile | 1 +
.../crypto/inside-secure/eip93/eip93-ipsec.c | 1413 +++++++++++++++++
.../crypto/inside-secure/eip93/eip93-main.c | 69 +-
.../crypto/inside-secure/eip93/eip93-main.h | 38 +-
include/crypto/eip93-ipsec.h | 132 ++
7 files changed, 1643 insertions(+), 21 deletions(-)
create mode 100644 drivers/crypto/inside-secure/eip93/eip93-ipsec.c
create mode 100644 include/crypto/eip93-ipsec.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f1e5e4258e7b..08cfede333e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12743,6 +12743,7 @@ L: linux-crypto@vger.kernel.org
S: Maintained
F: Documentation/devicetree/bindings/crypto/inside-secure,safexcel-eip93.yaml
F: drivers/crypto/inside-secure/eip93/
+F: include/crypto/eip93-ipsec.h
INTEGRITY MEASUREMENT ARCHITECTURE (IMA)
M: Mimi Zohar <zohar@linux.ibm.com>
diff --git a/drivers/crypto/inside-secure/eip93/Kconfig b/drivers/crypto/inside-secure/eip93/Kconfig
index 29523f6927dd..1a33ab6f04da 100644
--- a/drivers/crypto/inside-secure/eip93/Kconfig
+++ b/drivers/crypto/inside-secure/eip93/Kconfig
@@ -18,3 +18,13 @@ config CRYPTO_DEV_EIP93
CTR crypto. Also provide DES and 3DES ECB and CBC.
Also provide AEAD authenc(hmac(x), cipher(y)) for supported algo.
+
+config CRYPTO_DEV_EIP93_IPSEC
+ bool
+ depends on CRYPTO_DEV_EIP93
+ depends on XFRM_OFFLOAD
+ default y
+ help
+ Select this if a netdev driver should be allowed to use EIP93 for
+ ESP packet encapsulation and decapsulation rather than only the
+ crypto transform.
diff --git a/drivers/crypto/inside-secure/eip93/Makefile b/drivers/crypto/inside-secure/eip93/Makefile
index a3d3d3677cdc..a5bb98370ff0 100644
--- a/drivers/crypto/inside-secure/eip93/Makefile
+++ b/drivers/crypto/inside-secure/eip93/Makefile
@@ -3,3 +3,4 @@ obj-$(CONFIG_CRYPTO_DEV_EIP93) += crypto-hw-eip93.o
crypto-hw-eip93-y += eip93-main.o eip93-common.o
crypto-hw-eip93-y += eip93-cipher.o eip93-aead.o
crypto-hw-eip93-y += eip93-hash.o
+crypto-hw-eip93-$(CONFIG_CRYPTO_DEV_EIP93_IPSEC) += eip93-ipsec.o
diff --git a/drivers/crypto/inside-secure/eip93/eip93-ipsec.c b/drivers/crypto/inside-secure/eip93/eip93-ipsec.c
new file mode 100644
index 000000000000..7338f4c7e24a
--- /dev/null
+++ b/drivers/crypto/inside-secure/eip93/eip93-ipsec.c
@@ -0,0 +1,1413 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026
+ *
+ * Jihong Min <hurryman2212@gmail.com>
+ */
+
+#include <crypto/aes.h>
+#include <crypto/eip93-ipsec.h>
+#include <crypto/hash.h>
+#include <crypto/hmac.h>
+#include <crypto/sha1.h>
+#include <crypto/sha2.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/ip.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/netlink.h>
+#include <linux/notifier.h>
+#include <linux/of.h>
+#include <linux/refcount.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/udp.h>
+#include <net/esp.h>
+#include <net/xfrm.h>
+#include <uapi/linux/pfkeyv2.h>
+
+#include "eip93-main.h"
+#include "eip93-regs.h"
+#include "eip93-common.h"
+
+#define EIP93_IPSEC_PAD_ALIGN 2
+#define EIP93_IPSEC_IDR_MIN 0
+#define EIP93_IPSEC_IDR_MAX (EIP93_RING_NUM - 1)
+#define EIP93_IPSEC_DIGEST_WORD_BITS (BITS_PER_BYTE * sizeof(u32))
+#define EIP93_IPSEC_DIGEST_WORDS(bits) ((bits) / EIP93_IPSEC_DIGEST_WORD_BITS)
+#define EIP93_IPSEC_HMAC_STATE_SIZE SHA256_DIGEST_SIZE
+#define EIP93_IPSEC_PRNG_BUF_SIZE 4080
+#define EIP93_IPSEC_PRNG_RESET_MODE 1
+#define EIP93_IPSEC_PRNG_POLL_US 10000
+#define EIP93_IPSEC_PRNG_POLL_STEP_US 10
+
+struct eip93_ipsec {
+ struct eip93_device *eip93;
+ struct list_head node;
+ struct list_head sa_list;
+ struct work_struct fault_work;
+ spinlock_t lock; /* protects dead/refcount admission */
+ refcount_t refcnt;
+ struct completion done;
+ enum eip93_ipsec_event fault_event;
+ u32 algo_flags;
+ bool dead;
+};
+
+struct eip93_ipsec_sa {
+ struct eip93_ipsec *ipsec;
+ struct sa_record *sa_record;
+ dma_addr_t sa_record_base;
+ struct list_head node;
+ struct list_head requests;
+ spinlock_t lock; /* protects dead/refcount admission */
+ refcount_t refcnt;
+ struct completion done;
+ u32 flags;
+ u16 family;
+ u8 authsize;
+ u8 blocksize;
+ u8 ivsize;
+ u8 encap_type;
+ bool esn;
+ bool dead;
+ bool aborting;
+};
+
+struct eip93_ipsec_request {
+ struct eip93_ipsec_sa *sa;
+ struct sk_buff *skb;
+ struct list_head node;
+ refcount_t refcnt;
+ eip93_ipsec_complete_t complete;
+ void *data;
+ dma_addr_t dma;
+ unsigned int dma_len;
+ enum dma_data_direction dma_dir;
+ int idr;
+};
+
+static DEFINE_MUTEX(eip93_ipsec_devices_lock);
+static LIST_HEAD(eip93_ipsec_devices);
+static BLOCKING_NOTIFIER_HEAD(eip93_ipsec_notifier);
+
+static bool eip93_ipsec_get_ref(struct eip93_ipsec *ipsec)
+{
+ bool ret = false;
+
+ spin_lock_bh(&ipsec->lock);
+ if (!ipsec->dead)
+ ret = refcount_inc_not_zero(&ipsec->refcnt);
+ spin_unlock_bh(&ipsec->lock);
+
+ return ret;
+}
+
+void eip93_ipsec_put(struct eip93_ipsec *ipsec)
+{
+ if (ipsec && refcount_dec_and_test(&ipsec->refcnt))
+ complete(&ipsec->done);
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_put);
+
+static bool eip93_ipsec_same_subsystem(struct device *consumer,
+ struct eip93_ipsec *ipsec)
+{
+ struct device_node *consumer_parent;
+ struct device_node *eip93_parent;
+ struct device_node *consumer_np;
+ struct device_node *eip93_np;
+ bool match;
+
+ consumer_np = dev_of_node(consumer);
+ eip93_np = dev_of_node(ipsec->eip93->dev);
+ if (!consumer_np || !eip93_np)
+ return false;
+
+ consumer_parent = of_get_parent(consumer_np);
+ eip93_parent = of_get_parent(eip93_np);
+ match = consumer_parent && consumer_parent == eip93_parent;
+ of_node_put(consumer_parent);
+ of_node_put(eip93_parent);
+
+ return match;
+}
+
+static bool eip93_ipsec_hw_available(u32 flags)
+{
+ if (!(flags & EIP93_PE_OPTION_AES))
+ return false;
+
+ if (!(flags & (EIP93_PE_OPTION_AES_KEY128 | EIP93_PE_OPTION_AES_KEY192 |
+ EIP93_PE_OPTION_AES_KEY256)))
+ return false;
+
+ return flags & (EIP93_PE_OPTION_SHA_1 | EIP93_PE_OPTION_SHA_256);
+}
+
+static bool eip93_ipsec_mark_dead(struct eip93_ipsec *ipsec)
+{
+ bool marked = false;
+
+ spin_lock_bh(&ipsec->lock);
+ if (!ipsec->dead) {
+ ipsec->dead = true;
+ marked = true;
+ }
+ spin_unlock_bh(&ipsec->lock);
+
+ return marked;
+}
+
+static bool eip93_ipsec_mark_dead_async(struct eip93_ipsec *ipsec,
+ enum eip93_ipsec_event event)
+{
+ bool marked = false;
+
+ spin_lock_bh(&ipsec->lock);
+ if (!ipsec->dead && refcount_inc_not_zero(&ipsec->refcnt)) {
+ ipsec->dead = true;
+ ipsec->fault_event = event;
+ marked = true;
+ }
+ spin_unlock_bh(&ipsec->lock);
+
+ if (marked)
+ schedule_work(&ipsec->fault_work);
+
+ return marked;
+}
+
+static bool eip93_ipsec_live_hw_available(struct eip93_ipsec *ipsec)
+{
+ u32 flags = readl(ipsec->eip93->base + EIP93_REG_PE_OPTION_1);
+
+ spin_lock_bh(&ipsec->lock);
+ ipsec->algo_flags = flags;
+ spin_unlock_bh(&ipsec->lock);
+
+ return eip93_ipsec_hw_available(flags);
+}
+
+struct eip93_ipsec *eip93_ipsec_get(struct device *consumer)
+{
+ struct eip93_ipsec *ipsec;
+ int err = -ENODEV;
+
+ if (!consumer)
+ return ERR_PTR(-EINVAL);
+
+ mutex_lock(&eip93_ipsec_devices_lock);
+ list_for_each_entry(ipsec, &eip93_ipsec_devices, node) {
+ if (!eip93_ipsec_same_subsystem(consumer, ipsec))
+ continue;
+
+ if (!eip93_ipsec_live_hw_available(ipsec)) {
+ enum eip93_ipsec_event event;
+
+ event = EIP93_IPSEC_EVENT_CAPABILITY_LOSS;
+ eip93_ipsec_mark_dead_async(ipsec, event);
+ err = -EOPNOTSUPP;
+ continue;
+ }
+
+ if (!eip93_ipsec_get_ref(ipsec)) {
+ err = -ENODEV;
+ continue;
+ }
+
+ mutex_unlock(&eip93_ipsec_devices_lock);
+ return ipsec;
+ }
+ mutex_unlock(&eip93_ipsec_devices_lock);
+
+ return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_get);
+
+bool eip93_ipsec_available(struct eip93_ipsec *ipsec)
+{
+ bool available;
+
+ if (!ipsec)
+ return false;
+
+ spin_lock_bh(&ipsec->lock);
+ available = !ipsec->dead;
+ spin_unlock_bh(&ipsec->lock);
+ if (!available)
+ return false;
+
+ available = eip93_ipsec_live_hw_available(ipsec);
+ if (!available)
+ eip93_ipsec_mark_dead_async(ipsec,
+ EIP93_IPSEC_EVENT_CAPABILITY_LOSS);
+
+ return available;
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_available);
+
+u32 eip93_ipsec_features(struct eip93_ipsec *ipsec)
+{
+ if (!eip93_ipsec_available(ipsec))
+ return 0;
+
+ return EIP93_IPSEC_FEATURE_ESP;
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_features);
+
+int eip93_ipsec_register_notifier(struct notifier_block *nb)
+{
+ return blocking_notifier_chain_register(&eip93_ipsec_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_register_notifier);
+
+void eip93_ipsec_unregister_notifier(struct notifier_block *nb)
+{
+ blocking_notifier_chain_unregister(&eip93_ipsec_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_unregister_notifier);
+
+static bool eip93_ipsec_sa_get(struct eip93_ipsec_sa *sa)
+{
+ bool ret = false;
+
+ spin_lock_bh(&sa->ipsec->lock);
+ spin_lock(&sa->lock);
+ if (!sa->ipsec->dead && !sa->dead)
+ ret = refcount_inc_not_zero(&sa->refcnt);
+ spin_unlock(&sa->lock);
+ spin_unlock_bh(&sa->ipsec->lock);
+
+ return ret;
+}
+
+static void eip93_ipsec_sa_put(struct eip93_ipsec_sa *sa)
+{
+ if (refcount_dec_and_test(&sa->refcnt))
+ complete(&sa->done);
+}
+
+static bool eip93_ipsec_request_get(struct eip93_ipsec_request *req)
+{
+ return refcount_inc_not_zero(&req->refcnt);
+}
+
+static void eip93_ipsec_request_put(struct eip93_ipsec_request *req)
+{
+ if (refcount_dec_and_test(&req->refcnt))
+ kfree(req);
+}
+
+static int eip93_ipsec_parse_flags(struct xfrm_state *x, u32 *flags)
+{
+ switch (x->props.ealgo) {
+ case SADB_X_EALG_AESCBC:
+ *flags |= EIP93_ALG_AES | EIP93_MODE_CBC;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ switch (x->props.aalgo) {
+ case SADB_AALG_SHA1HMAC:
+ *flags |= EIP93_HASH_HMAC | EIP93_HASH_SHA1;
+ break;
+ case SADB_X_AALG_SHA2_256HMAC:
+ *flags |= EIP93_HASH_HMAC | EIP93_HASH_SHA256;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (x->xso.dir == XFRM_DEV_OFFLOAD_IN)
+ *flags |= EIP93_DECRYPT;
+ else
+ *flags |= EIP93_ENCRYPT;
+
+ return 0;
+}
+
+static unsigned int eip93_ipsec_auth_digest_size(struct xfrm_state *x)
+{
+ switch (x->props.aalgo) {
+ case SADB_AALG_SHA1HMAC:
+ return SHA1_DIGEST_SIZE;
+ case SADB_X_AALG_SHA2_256HMAC:
+ return SHA256_DIGEST_SIZE;
+ default:
+ return 0;
+ }
+}
+
+static int eip93_ipsec_hmac_setkey(u32 flags, const u8 *key,
+ unsigned int keylen, u8 *dest_ipad,
+ u8 *dest_opad)
+{
+ u8 *ipad, *opad;
+ struct crypto_shash *tfm;
+ const char *alg_name;
+ unsigned int blocksize;
+ unsigned int digestsize;
+ unsigned int statesize;
+ unsigned int alloc_size;
+ unsigned int i;
+ int err;
+
+ switch (flags & EIP93_HASH_MASK) {
+ case EIP93_HASH_SHA1:
+ alg_name = "sha1";
+ digestsize = SHA1_DIGEST_SIZE;
+ break;
+ case EIP93_HASH_SHA256:
+ alg_name = "sha256";
+ digestsize = SHA256_DIGEST_SIZE;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ tfm = crypto_alloc_shash(alg_name, 0, CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ blocksize = crypto_shash_blocksize(tfm);
+ statesize = crypto_shash_statesize(tfm);
+ if (statesize < EIP93_IPSEC_HMAC_STATE_SIZE) {
+ err = -EINVAL;
+ goto free_tfm;
+ }
+
+ alloc_size = 2 * (blocksize + statesize);
+ ipad = kzalloc(alloc_size, GFP_KERNEL);
+ if (!ipad) {
+ err = -ENOMEM;
+ goto free_tfm;
+ }
+ opad = ipad + blocksize + statesize;
+
+ if (keylen > blocksize) {
+ SHASH_DESC_ON_STACK(desc, tfm);
+
+ desc->tfm = tfm;
+ err = crypto_shash_digest(desc, key, keylen, ipad);
+ shash_desc_zero(desc);
+ if (err)
+ goto free_pad;
+
+ keylen = digestsize;
+ } else {
+ memcpy(ipad, key, keylen);
+ }
+
+ memcpy(opad, ipad, blocksize);
+ for (i = 0; i < blocksize; i++) {
+ ipad[i] ^= HMAC_IPAD_VALUE;
+ opad[i] ^= HMAC_OPAD_VALUE;
+ }
+
+ {
+ SHASH_DESC_ON_STACK(desc, tfm);
+
+ desc->tfm = tfm;
+ err = crypto_shash_init(desc) ?:
+ crypto_shash_update(desc, ipad, blocksize) ?:
+ crypto_shash_export(desc, ipad) ?:
+ crypto_shash_init(desc) ?:
+ crypto_shash_update(desc, opad, blocksize) ?:
+ crypto_shash_export(desc, opad);
+ shash_desc_zero(desc);
+ }
+ if (err)
+ goto free_pad;
+
+ /*
+ * EIP93 ESP protocol mode consumes the raw exported HMAC ipad/opad
+ * state. The crypto API AEAD helper byteswaps this state for its basic
+ * authenc path, but packet ESP mode matches mtk-eip93 with native
+ * exported bytes in the SA record.
+ */
+ memcpy(dest_ipad, ipad, EIP93_IPSEC_HMAC_STATE_SIZE);
+ memcpy(dest_opad, opad, EIP93_IPSEC_HMAC_STATE_SIZE);
+
+free_pad:
+ kfree_sensitive(ipad);
+free_tfm:
+ crypto_free_shash(tfm);
+ return err;
+}
+
+static int eip93_ipsec_validate_algo(struct xfrm_state *x,
+ struct netlink_ext_ack *extack)
+{
+ unsigned int authsize;
+ unsigned int keylen;
+
+ if (x->aead) {
+ NL_SET_ERR_MSG_MOD(extack, "AEAD SAs are unsupported");
+ return -EOPNOTSUPP;
+ }
+
+ if (!x->ealg || !x->aalg) {
+ NL_SET_ERR_MSG_MOD(extack, "encryption/auth required");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->props.ealgo != SADB_X_EALG_AESCBC) {
+ NL_SET_ERR_MSG_MOD(extack, "only AES-CBC is supported");
+ return -EOPNOTSUPP;
+ }
+
+ keylen = x->ealg->alg_key_len / BITS_PER_BYTE;
+ if (keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_192 &&
+ keylen != AES_KEYSIZE_256) {
+ NL_SET_ERR_MSG_MOD(extack, "unsupported AES-CBC key length");
+ return -EOPNOTSUPP;
+ }
+
+ authsize = eip93_ipsec_auth_digest_size(x);
+ if (!authsize) {
+ NL_SET_ERR_MSG_MOD(extack, "only SHA1/SHA256 HMAC");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->aalg->alg_trunc_len % EIP93_IPSEC_DIGEST_WORD_BITS ||
+ x->aalg->alg_trunc_len < EIP93_IPSEC_DIGEST_WORD_BITS ||
+ x->aalg->alg_trunc_len > authsize * BITS_PER_BYTE) {
+ NL_SET_ERR_MSG_MOD(extack, "bad auth truncation length");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int eip93_ipsec_validate_state(struct xfrm_state *x,
+ struct netlink_ext_ack *extack)
+{
+ switch (x->xso.dir) {
+ case XFRM_DEV_OFFLOAD_OUT:
+ case XFRM_DEV_OFFLOAD_IN:
+ break;
+ default:
+ NL_SET_ERR_MSG_MOD(extack, "only in/out SAs are supported");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) {
+ NL_SET_ERR_MSG_MOD(extack, "only crypto offload is supported");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->id.proto != IPPROTO_ESP) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "EIP93 packet backend supports ESP only");
+ return -EOPNOTSUPP;
+ }
+
+ switch (x->props.family) {
+ case AF_INET:
+ break;
+#if IS_ENABLED(CONFIG_IPV6)
+ case AF_INET6:
+ break;
+#endif
+ default:
+ NL_SET_ERR_MSG_MOD(extack, "only IPv4/IPv6 is supported");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->outer_mode.family != x->props.family) {
+ NL_SET_ERR_MSG_MOD(extack, "only same-family ESP is supported");
+ return -EOPNOTSUPP;
+ }
+
+ switch (x->props.mode) {
+ case XFRM_MODE_TUNNEL:
+ case XFRM_MODE_TRANSPORT:
+ break;
+ default:
+ NL_SET_ERR_MSG_MOD(extack, "only tunnel/transport");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->outer_mode.encap != x->props.mode) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "outer ESP mode does not match state mode");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->encap) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "NAT-T is unsupported by EIP93 packet ESP");
+ return -EOPNOTSUPP;
+ }
+
+ if (x->tfcpad) {
+ NL_SET_ERR_MSG_MOD(extack, "TFC padding is unsupported");
+ return -EOPNOTSUPP;
+ }
+
+ return eip93_ipsec_validate_algo(x, extack);
+}
+
+static int eip93_ipsec_validate_hw(struct xfrm_state *x, u32 flags,
+ struct netlink_ext_ack *extack)
+{
+ unsigned int keylen = x->ealg->alg_key_len / BITS_PER_BYTE;
+ u32 required;
+
+ if (!(flags & EIP93_PE_OPTION_AES)) {
+ NL_SET_ERR_MSG_MOD(extack, "EIP93 AES engine is unavailable");
+ return -EOPNOTSUPP;
+ }
+
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ required = EIP93_PE_OPTION_AES_KEY128;
+ break;
+ case AES_KEYSIZE_192:
+ required = EIP93_PE_OPTION_AES_KEY192;
+ break;
+ case AES_KEYSIZE_256:
+ required = EIP93_PE_OPTION_AES_KEY256;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (!(flags & required)) {
+ NL_SET_ERR_MSG_MOD(extack, "unsupported AES key length");
+ return -EOPNOTSUPP;
+ }
+
+ switch (x->props.aalgo) {
+ case SADB_AALG_SHA1HMAC:
+ required = EIP93_PE_OPTION_SHA_1;
+ break;
+ case SADB_X_AALG_SHA2_256HMAC:
+ required = EIP93_PE_OPTION_SHA_256;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (!(flags & required)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "EIP93 does not support this HMAC hash");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static void eip93_ipsec_init_sa_record(struct eip93_ipsec_sa *sa,
+ struct xfrm_state *x)
+{
+ struct sa_record *record = sa->sa_record;
+ unsigned int auth_words;
+ unsigned int enckeylen;
+
+ enckeylen = x->ealg->alg_key_len / BITS_PER_BYTE;
+ auth_words = EIP93_IPSEC_DIGEST_WORDS(x->aalg->alg_trunc_len);
+
+ eip93_set_sa_record(record, enckeylen, sa->flags);
+
+ record->sa_cmd0_word &=
+ ~(EIP93_SA_CMD_OPGROUP | EIP93_SA_CMD_OPCODE |
+ EIP93_SA_CMD_DIGEST_LENGTH | EIP93_SA_CMD_PAD_TYPE |
+ EIP93_SA_CMD_IV_SOURCE | EIP93_SA_CMD_SAVE_IV);
+ record->sa_cmd0_word |=
+ EIP93_SA_CMD_OP_PROTOCOL | EIP93_SA_CMD_HDR_PROC |
+ EIP93_SA_CMD_PAD_IPSEC | EIP93_SA_CMD_SCPAD |
+ FIELD_PREP(EIP93_SA_CMD_OPCODE,
+ EIP93_SA_CMD_OPCODE_PROTOCOL_OUT_ESP) |
+ FIELD_PREP(EIP93_SA_CMD_DIGEST_LENGTH, auth_words);
+
+ /*
+ * ESP packet mode authenticates from the ESP header when the hash
+ * crypt offset is zero. This is intentionally different from the AEAD
+ * authenc path, whose AAD starts after the ESP header.
+ */
+ record->sa_cmd1_word &=
+ ~(EIP93_SA_CMD_HASH_CRYPT_OFFSET | EIP93_SA_CMD_BYTE_OFFSET |
+ EIP93_SA_CMD_COPY_PAD | EIP93_SA_CMD_COPY_HEADER |
+ EIP93_SA_CMD_COPY_DIGEST | EIP93_SA_CMD_COPY_PAYLOAD |
+ EIP93_SA_CMD_EN_SEQNUM_CHK);
+ record->sa_cmd1_word |= EIP93_SA_CMD_HMAC | EIP93_SA_CMD_EN_SEQNUM_CHK;
+
+ if (x->xso.dir == XFRM_DEV_OFFLOAD_IN) {
+ record->sa_cmd0_word |= EIP93_SA_CMD_DIRECTION_IN |
+ EIP93_SA_CMD_IV_FROM_INPUT;
+ /*
+ * Inbound ESP decapsulation keeps the outer header for XFRM and
+ * lets hardware remove ESP pad/trailer/ICV bytes.
+ */
+ record->sa_cmd1_word |= EIP93_SA_CMD_COPY_HEADER;
+ } else {
+ record->sa_cmd0_word |= EIP93_SA_CMD_IV_FROM_PRNG;
+ record->sa_cmd1_word |= EIP93_SA_CMD_COPY_DIGEST;
+ }
+
+ record->sa_spi = ntohl(x->id.spi);
+ if (sa->esn && x->replay_esn) {
+ if (x->xso.dir == XFRM_DEV_OFFLOAD_IN)
+ record->sa_seqnum[1] = x->replay_esn->seq_hi;
+ else
+ record->sa_seqnum[1] = x->replay_esn->oseq_hi;
+ } else {
+ record->sa_seqnum[1] = 0;
+ }
+ record->sa_seqnum[0] = 0;
+ record->sa_seqmum_mask[0] = 0xffffffff;
+ record->sa_seqmum_mask[1] = sa->esn ? 0xffffffff : 0;
+}
+
+static int eip93_ipsec_poll_result(struct eip93_device *eip93,
+ struct eip93_descriptor **rdesc)
+{
+ struct eip93_descriptor *desc;
+ unsigned int i;
+ u32 pe_ctrl_stat;
+ u32 pe_length;
+
+ for (i = 0; i < EIP93_IPSEC_PRNG_POLL_US;
+ i += EIP93_IPSEC_PRNG_POLL_STEP_US) {
+ if (readl(eip93->base + EIP93_REG_PE_RD_COUNT) &
+ EIP93_PE_RD_COUNT)
+ break;
+ udelay(EIP93_IPSEC_PRNG_POLL_STEP_US);
+ }
+ if (i >= EIP93_IPSEC_PRNG_POLL_US)
+ return -ETIMEDOUT;
+
+ scoped_guard(spinlock_irqsave, &eip93->ring->read_lock)
+ desc = eip93_get_descriptor(eip93);
+ if (IS_ERR(desc))
+ return PTR_ERR(desc);
+ *rdesc = desc;
+
+ for (i = 0; i < EIP93_IPSEC_PRNG_POLL_US;
+ i += EIP93_IPSEC_PRNG_POLL_STEP_US) {
+ pe_ctrl_stat = READ_ONCE((*rdesc)->pe_ctrl_stat_word);
+ pe_length = READ_ONCE((*rdesc)->pe_length_word);
+ if (FIELD_GET(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN,
+ pe_ctrl_stat) == EIP93_PE_CTRL_PE_READY &&
+ FIELD_GET(EIP93_PE_LENGTH_HOST_PE_READY, pe_length) ==
+ EIP93_PE_LENGTH_PE_READY)
+ break;
+ udelay(EIP93_IPSEC_PRNG_POLL_STEP_US);
+ }
+
+ writel(1, eip93->base + EIP93_REG_PE_RD_COUNT);
+ writel(EIP93_INT_RDR_THRESH, eip93->base + EIP93_REG_INT_CLR);
+
+ if (i >= EIP93_IPSEC_PRNG_POLL_US)
+ return -ETIMEDOUT;
+
+ return 0;
+}
+
+static int eip93_ipsec_init_prng(struct eip93_device *eip93)
+{
+ static const u32 prng_dt[4] = {};
+ static const u32 prng_key[4] = {
+ 0xe0fc631d, 0xcbb9fb9a, 0x869285cb, 0xcbb9fb9a
+ };
+ static const u32 prng_seed[4] = {
+ 0x758bac03, 0xf20ab39e, 0xa569f104, 0x95dfaea6
+ };
+ struct eip93_descriptor cdesc = {};
+ struct eip93_descriptor *rdesc;
+ struct sa_record *record;
+ dma_addr_t record_dma;
+ dma_addr_t buf_dma;
+ void *buf;
+ int err;
+
+ record = dma_alloc_coherent(eip93->dev, sizeof(*record), &record_dma,
+ GFP_KERNEL);
+ if (!record)
+ return -ENOMEM;
+
+ buf = dma_alloc_coherent(eip93->dev, EIP93_IPSEC_PRNG_BUF_SIZE,
+ &buf_dma, GFP_KERNEL);
+ if (!buf) {
+ err = -ENOMEM;
+ goto free_record;
+ }
+
+ memset(record, 0, sizeof(*record));
+ record->sa_cmd0_word =
+ EIP93_SA_CMD_OP_BASIC |
+ FIELD_PREP(EIP93_SA_CMD_OPCODE,
+ EIP93_SA_CMD_OPCODE_BASIC_OUT_PRNG) |
+ EIP93_SA_CMD_CIPHER_AES | EIP93_SA_CMD_HASH_SHA1;
+ record->sa_cmd1_word = EIP93_SA_CMD_AES_KEY_128BIT;
+ memcpy(record->sa_key, prng_key, sizeof(prng_key));
+ memcpy(record->sa_i_digest, prng_seed, sizeof(prng_seed));
+ memcpy(record->sa_o_digest, prng_dt, sizeof(prng_dt));
+
+ cdesc.pe_ctrl_stat_word =
+ FIELD_PREP(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN,
+ EIP93_PE_CTRL_HOST_READY) |
+ FIELD_PREP(EIP93_PE_CTRL_PE_PRNG_MODE,
+ EIP93_IPSEC_PRNG_RESET_MODE);
+ cdesc.dst_addr = (u32 __force)buf_dma;
+ cdesc.sa_addr = record_dma;
+ cdesc.user_id = FIELD_PREP(EIP93_PE_USER_ID_DESC_FLAGS,
+ EIP93_DESC_PRNG | EIP93_DESC_LAST);
+ cdesc.pe_length_word =
+ FIELD_PREP(EIP93_PE_LENGTH_HOST_PE_READY,
+ EIP93_PE_LENGTH_HOST_READY) |
+ FIELD_PREP(EIP93_PE_LENGTH_LENGTH,
+ EIP93_IPSEC_PRNG_BUF_SIZE);
+
+ /*
+ * Outbound ESP SAs use IV_FROM_PRNG. Initialize the EIP93 PRNG before
+ * exposing the packet backend, otherwise the first ESP encrypt can
+ * fail or emit unusable IV material.
+ */
+ scoped_guard(spinlock_irqsave, &eip93->ring->write_lock)
+ err = eip93_put_descriptor(eip93, &cdesc);
+ if (err)
+ goto free_buf;
+
+ writel(1, eip93->base + EIP93_REG_PE_CD_COUNT);
+ err = eip93_ipsec_poll_result(eip93, &rdesc);
+ if (err)
+ goto free_buf;
+
+ err = rdesc->pe_ctrl_stat_word &
+ (EIP93_PE_CTRL_PE_EXT_ERR_CODE | EIP93_PE_CTRL_PE_EXT_ERR |
+ EIP93_PE_CTRL_PE_SEQNUM_ERR | EIP93_PE_CTRL_PE_PAD_ERR |
+ EIP93_PE_CTRL_PE_AUTH_ERR);
+ err = eip93_parse_ctrl_stat_err(eip93, err);
+ if (err)
+ dev_err(eip93->dev, "IPsec PRNG init failed: %d\n", err);
+
+free_buf:
+ dma_free_coherent(eip93->dev, EIP93_IPSEC_PRNG_BUF_SIZE, buf, buf_dma);
+free_record:
+ dma_free_coherent(eip93->dev, sizeof(*record), record, record_dma);
+
+ return err;
+}
+
+static int eip93_ipsec_submit(struct eip93_ipsec_request *req,
+ struct eip93_descriptor *cdesc)
+{
+ struct eip93_device *eip93 = req->sa->ipsec->eip93;
+ struct eip93_ipsec_sa *sa = req->sa;
+ struct eip93_ipsec *ipsec = sa->ipsec;
+ int err;
+
+ spin_lock_bh(&ipsec->lock);
+ if (ipsec->dead) {
+ err = -EOPNOTSUPP;
+ goto unlock_ipsec;
+ }
+
+ scoped_guard(spinlock_bh, &eip93->ring->idr_lock) req->idr =
+ idr_alloc(&eip93->ring->crypto_async_idr, req,
+ EIP93_IPSEC_IDR_MIN, EIP93_IPSEC_IDR_MAX, GFP_ATOMIC);
+ if (req->idr < 0) {
+ err = req->idr == -ENOSPC ? -EBUSY : req->idr;
+ goto unlock_ipsec;
+ }
+
+ spin_lock(&sa->lock);
+ if (sa->dead) {
+ spin_unlock(&sa->lock);
+ err = -EOPNOTSUPP;
+ goto remove_idr;
+ }
+ list_add_tail(&req->node, &sa->requests);
+ spin_unlock(&sa->lock);
+
+ cdesc->user_id =
+ FIELD_PREP(EIP93_PE_USER_ID_CRYPTO_IDR, (u16)req->idr) |
+ FIELD_PREP(EIP93_PE_USER_ID_DESC_FLAGS,
+ EIP93_DESC_IPSEC | EIP93_DESC_LAST);
+
+ scoped_guard(spinlock_irqsave, &eip93->ring->write_lock)
+ err = eip93_put_descriptor(eip93, cdesc);
+ if (err)
+ goto unlink_request;
+
+ writel(1, eip93->base + EIP93_REG_PE_CD_COUNT);
+ spin_unlock_bh(&ipsec->lock);
+
+ return -EINPROGRESS;
+
+unlink_request:
+ spin_lock(&sa->lock);
+ list_del_init(&req->node);
+ spin_unlock(&sa->lock);
+remove_idr:
+ scoped_guard(spinlock_bh, &eip93->ring->idr_lock)
+ idr_remove(&eip93->ring->crypto_async_idr, req->idr);
+ err = err == -ENOENT ? -EBUSY : err;
+unlock_ipsec:
+ spin_unlock_bh(&ipsec->lock);
+ return err;
+}
+
+static void eip93_ipsec_unlink_request(struct eip93_ipsec_request *req)
+{
+ struct eip93_ipsec_sa *sa = req->sa;
+
+ spin_lock_bh(&sa->lock);
+ if (!list_empty(&req->node))
+ list_del_init(&req->node);
+ spin_unlock_bh(&sa->lock);
+}
+
+static void eip93_ipsec_complete_request(struct eip93_ipsec_request *req,
+ int err,
+ struct eip93_ipsec_result result)
+{
+ struct eip93_ipsec_sa *sa = req->sa;
+ eip93_ipsec_complete_t complete = req->complete;
+ void *data = req->data;
+
+ dma_unmap_single(sa->ipsec->eip93->dev, req->dma, req->dma_len,
+ req->dma_dir);
+ eip93_ipsec_unlink_request(req);
+ eip93_ipsec_sa_put(sa);
+ complete(data, err, result);
+ eip93_ipsec_request_put(req);
+}
+
+static void eip93_ipsec_abort_sa(struct eip93_ipsec_sa *sa, int err)
+{
+ struct eip93_device *eip93 = sa->ipsec->eip93;
+ struct eip93_ipsec_request *req;
+ bool claimed;
+
+ while (true) {
+ spin_lock_bh(&sa->lock);
+ if (list_empty(&sa->requests)) {
+ spin_unlock_bh(&sa->lock);
+ return;
+ }
+
+ req = list_first_entry(&sa->requests,
+ struct eip93_ipsec_request, node);
+ if (!eip93_ipsec_request_get(req)) {
+ list_del_init(&req->node);
+ spin_unlock_bh(&sa->lock);
+ continue;
+ }
+ list_del_init(&req->node);
+ spin_unlock_bh(&sa->lock);
+
+ claimed = false;
+ scoped_guard(spinlock_bh, &eip93->ring->idr_lock) {
+ if (idr_find(&eip93->ring->crypto_async_idr,
+ req->idr) == req) {
+ idr_remove(&eip93->ring->crypto_async_idr,
+ req->idr);
+ claimed = true;
+ }
+ }
+
+ if (claimed) {
+ struct eip93_ipsec_result result = {};
+
+ eip93_ipsec_complete_request(req, err, result);
+ }
+ eip93_ipsec_request_put(req);
+ }
+}
+
+static void eip93_ipsec_abort_requests(struct eip93_ipsec *ipsec, int err)
+{
+ struct eip93_ipsec_sa *sa;
+
+ while (true) {
+ bool found = false;
+
+ spin_lock_bh(&ipsec->lock);
+ list_for_each_entry(sa, &ipsec->sa_list, node) {
+ spin_lock(&sa->lock);
+ if (sa->aborting) {
+ spin_unlock(&sa->lock);
+ continue;
+ }
+
+ sa->aborting = true;
+ found = refcount_inc_not_zero(&sa->refcnt);
+ spin_unlock(&sa->lock);
+ if (found)
+ break;
+ }
+ spin_unlock_bh(&ipsec->lock);
+ if (!found)
+ return;
+
+ eip93_ipsec_abort_sa(sa, err);
+ eip93_ipsec_sa_put(sa);
+ }
+}
+
+static void eip93_ipsec_fault_work(struct work_struct *work)
+{
+ struct eip93_ipsec *ipsec =
+ container_of(work, struct eip93_ipsec, fault_work);
+ enum eip93_ipsec_event event;
+
+ spin_lock_bh(&ipsec->lock);
+ event = ipsec->fault_event;
+ spin_unlock_bh(&ipsec->lock);
+
+ eip93_ipsec_abort_requests(ipsec, -EIO);
+ blocking_notifier_call_chain(&eip93_ipsec_notifier, event, ipsec);
+ eip93_ipsec_put(ipsec);
+}
+
+void eip93_ipsec_handle_result(struct eip93_ipsec_request *req, int err,
+ u32 pe_ctrl_stat, u32 pe_length)
+{
+ struct eip93_ipsec_result result = {};
+
+ if (!req)
+ return;
+
+ if (err == -EIO || err == -EACCES)
+ eip93_ipsec_mark_dead_async(req->sa->ipsec,
+ EIP93_IPSEC_EVENT_DMA_ERROR);
+
+ if (!err) {
+ result.packet_len = FIELD_GET(EIP93_PE_LENGTH_LENGTH, pe_length);
+ result.nexthdr = FIELD_GET(EIP93_PE_CTRL_PE_PAD_VALUE,
+ pe_ctrl_stat);
+ }
+
+ eip93_ipsec_complete_request(req, err, result);
+}
+
+void eip93_ipsec_report_irq(struct eip93_device *eip93, u32 irq_status)
+{
+ struct eip93_ipsec *ipsec = eip93->ipsec;
+
+ if (!ipsec)
+ return;
+
+ if (irq_status & EIP93_INT_HALT) {
+ eip93_ipsec_mark_dead_async(ipsec, EIP93_IPSEC_EVENT_RESET);
+ return;
+ }
+
+ if (irq_status & (EIP93_INT_INTERFACE_ERR | EIP93_INT_RPOC_ERR |
+ EIP93_INT_PE_RING_ERR))
+ eip93_ipsec_mark_dead_async(ipsec, EIP93_IPSEC_EVENT_DMA_ERROR);
+}
+
+int eip93_ipsec_register(struct eip93_device *eip93)
+{
+ struct eip93_ipsec *ipsec;
+ int err;
+
+ ipsec = kzalloc(sizeof(*ipsec), GFP_KERNEL);
+ if (!ipsec)
+ return -ENOMEM;
+
+ err = eip93_ipsec_init_prng(eip93);
+ if (err) {
+ kfree(ipsec);
+ return err;
+ }
+
+ ipsec->eip93 = eip93;
+ ipsec->algo_flags = readl(eip93->base + EIP93_REG_PE_OPTION_1);
+ ipsec->fault_event = EIP93_IPSEC_EVENT_REMOVE;
+ INIT_WORK(&ipsec->fault_work, eip93_ipsec_fault_work);
+ spin_lock_init(&ipsec->lock);
+ refcount_set(&ipsec->refcnt, 1);
+ init_completion(&ipsec->done);
+ INIT_LIST_HEAD(&ipsec->node);
+ INIT_LIST_HEAD(&ipsec->sa_list);
+
+ mutex_lock(&eip93_ipsec_devices_lock);
+ eip93->ipsec = ipsec;
+ list_add_tail(&ipsec->node, &eip93_ipsec_devices);
+ mutex_unlock(&eip93_ipsec_devices_lock);
+
+ return 0;
+}
+
+void eip93_ipsec_unregister(struct eip93_device *eip93)
+{
+ struct eip93_ipsec *ipsec = eip93->ipsec;
+ bool notify_remove;
+
+ if (!ipsec)
+ return;
+
+ mutex_lock(&eip93_ipsec_devices_lock);
+ notify_remove = eip93_ipsec_mark_dead(ipsec);
+ list_del_init(&ipsec->node);
+ eip93->ipsec = NULL;
+ mutex_unlock(&eip93_ipsec_devices_lock);
+
+ eip93_ipsec_abort_requests(ipsec, -ENODEV);
+ if (notify_remove)
+ blocking_notifier_call_chain(&eip93_ipsec_notifier,
+ EIP93_IPSEC_EVENT_REMOVE, ipsec);
+
+ eip93_ipsec_put(ipsec);
+ wait_for_completion(&ipsec->done);
+ cancel_work_sync(&ipsec->fault_work);
+ kfree(ipsec);
+}
+
+int eip93_ipsec_state_add(struct eip93_ipsec *ipsec, struct xfrm_state *x,
+ struct netlink_ext_ack *extack,
+ struct eip93_ipsec_sa **sa)
+{
+ struct eip93_device *eip93;
+ struct eip93_ipsec_sa *new_sa;
+ unsigned int authkeylen;
+ unsigned int enckeylen;
+ int err;
+
+ if (!ipsec || !eip93_ipsec_get_ref(ipsec)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "EIP93 packet backend is unavailable");
+ return -EOPNOTSUPP;
+ }
+
+ err = eip93_ipsec_validate_state(x, extack);
+ if (err)
+ goto put_ipsec;
+
+ err = eip93_ipsec_validate_hw(x, ipsec->algo_flags, extack);
+ if (err)
+ goto put_ipsec;
+
+ eip93 = ipsec->eip93;
+ new_sa = kzalloc(sizeof(*new_sa), GFP_KERNEL);
+ if (!new_sa) {
+ err = -ENOMEM;
+ goto put_ipsec;
+ }
+
+ new_sa->ipsec = ipsec;
+ new_sa->family = x->props.family;
+ new_sa->ivsize = AES_BLOCK_SIZE;
+ new_sa->authsize = x->aalg->alg_trunc_len / BITS_PER_BYTE;
+ new_sa->blocksize = AES_BLOCK_SIZE;
+ new_sa->encap_type = x->encap ? x->encap->encap_type : 0;
+ new_sa->esn = x->props.flags & XFRM_STATE_ESN;
+ INIT_LIST_HEAD(&new_sa->node);
+ INIT_LIST_HEAD(&new_sa->requests);
+ spin_lock_init(&new_sa->lock);
+ refcount_set(&new_sa->refcnt, 1);
+ init_completion(&new_sa->done);
+
+ err = eip93_ipsec_parse_flags(x, &new_sa->flags);
+ if (err)
+ goto free_sa;
+
+ new_sa->sa_record = kzalloc(sizeof(*new_sa->sa_record), GFP_KERNEL);
+ if (!new_sa->sa_record) {
+ err = -ENOMEM;
+ goto free_sa;
+ }
+
+ eip93_ipsec_init_sa_record(new_sa, x);
+
+ enckeylen = x->ealg->alg_key_len / BITS_PER_BYTE;
+ memcpy(new_sa->sa_record->sa_key, x->ealg->alg_key, enckeylen);
+
+ authkeylen = x->aalg->alg_key_len / BITS_PER_BYTE;
+ err = eip93_ipsec_hmac_setkey(new_sa->flags, x->aalg->alg_key,
+ authkeylen,
+ new_sa->sa_record->sa_i_digest,
+ new_sa->sa_record->sa_o_digest);
+ if (err)
+ goto free_record;
+
+ new_sa->sa_record_base = dma_map_single(eip93->dev, new_sa->sa_record,
+ sizeof(*new_sa->sa_record),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(eip93->dev, new_sa->sa_record_base)) {
+ err = -ENOMEM;
+ goto free_record;
+ }
+
+ spin_lock_bh(&ipsec->lock);
+ if (ipsec->dead) {
+ spin_unlock_bh(&ipsec->lock);
+ err = -EOPNOTSUPP;
+ goto unmap_record;
+ }
+ list_add_tail(&new_sa->node, &ipsec->sa_list);
+ spin_unlock_bh(&ipsec->lock);
+
+ *sa = new_sa;
+
+ return 0;
+
+unmap_record:
+ dma_unmap_single(eip93->dev, new_sa->sa_record_base,
+ sizeof(*new_sa->sa_record), DMA_TO_DEVICE);
+free_record:
+ kfree_sensitive(new_sa->sa_record);
+free_sa:
+ kfree(new_sa);
+put_ipsec:
+ eip93_ipsec_put(ipsec);
+ return err;
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_state_add);
+
+void eip93_ipsec_state_delete(struct eip93_ipsec_sa *sa)
+{
+ if (!sa)
+ return;
+
+ spin_lock_bh(&sa->ipsec->lock);
+ spin_lock(&sa->lock);
+ sa->dead = true;
+ list_del_init(&sa->node);
+ spin_unlock(&sa->lock);
+ spin_unlock_bh(&sa->ipsec->lock);
+
+ eip93_ipsec_sa_put(sa);
+ wait_for_completion(&sa->done);
+
+ dma_unmap_single(sa->ipsec->eip93->dev, sa->sa_record_base,
+ sizeof(*sa->sa_record), DMA_TO_DEVICE);
+ kfree_sensitive(sa->sa_record);
+ eip93_ipsec_put(sa->ipsec);
+ kfree(sa);
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_state_delete);
+
+void eip93_ipsec_state_advance_esn(struct eip93_ipsec_sa *sa,
+ struct xfrm_state *x)
+{
+ u32 seq_hi = 0;
+
+ if (!sa || !x || !sa->esn || !x->replay_esn)
+ return;
+
+ if (x->xso.dir == XFRM_DEV_OFFLOAD_IN)
+ seq_hi = x->replay_esn->seq_hi;
+ else if (x->xso.dir == XFRM_DEV_OFFLOAD_OUT)
+ seq_hi = x->replay_esn->oseq_hi;
+
+ spin_lock_bh(&sa->lock);
+ if (!sa->dead) {
+ sa->sa_record->sa_seqnum[1] = seq_hi;
+ dma_sync_single_for_device(sa->ipsec->eip93->dev,
+ sa->sa_record_base,
+ sizeof(*sa->sa_record),
+ DMA_TO_DEVICE);
+ }
+ spin_unlock_bh(&sa->lock);
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_state_advance_esn);
+
+int eip93_ipsec_xmit(struct eip93_ipsec_sa *sa, struct sk_buff *skb,
+ unsigned int esp_offset, eip93_ipsec_complete_t complete,
+ void *data)
+{
+ struct eip93_descriptor cdesc = {};
+ struct eip93_ipsec_request *req;
+ struct xfrm_offload *xo;
+ unsigned int payload_len;
+ unsigned int crypt_len;
+ unsigned int dma_len;
+ unsigned int tailen;
+ int err;
+
+ if (!sa || !complete || !eip93_ipsec_sa_get(sa))
+ return -EOPNOTSUPP;
+
+ if (skb_is_nonlinear(skb)) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ if (skb->len <= esp_offset + sizeof(struct ip_esp_hdr) + sa->ivsize) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ xo = xfrm_offload(skb);
+ if (!xo) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ tailen = xo->esp_tx_tailen;
+ if (tailen) {
+ payload_len = skb->len - esp_offset - sizeof(struct ip_esp_hdr) -
+ sa->ivsize;
+ dma_len = skb->len + tailen;
+ if (tailen > skb_tailroom(skb) || dma_len < skb->len) {
+ err = -ENOMEM;
+ goto put_sa;
+ }
+ } else {
+ u8 *trail;
+ u8 padlen;
+
+ if (skb->len <= esp_offset + sizeof(struct ip_esp_hdr) +
+ sa->ivsize + sa->authsize) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ crypt_len = skb->len - esp_offset - sizeof(struct ip_esp_hdr) -
+ sa->ivsize - sa->authsize;
+ if (crypt_len < 2) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ trail = skb_tail_pointer(skb) - sa->authsize - 2;
+ padlen = trail[0];
+ if (crypt_len < padlen + 2) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ payload_len = crypt_len - padlen - 2;
+ dma_len = skb->len;
+ }
+ if (payload_len > FIELD_MAX(EIP93_PE_LENGTH_LENGTH)) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ req = kmalloc(sizeof(*req), GFP_ATOMIC);
+ if (!req) {
+ err = -ENOMEM;
+ goto put_sa;
+ }
+
+ req->sa = sa;
+ req->skb = skb;
+ INIT_LIST_HEAD(&req->node);
+ refcount_set(&req->refcnt, 1);
+ req->complete = complete;
+ req->data = data;
+ req->dma_len = dma_len;
+ req->dma_dir = DMA_BIDIRECTIONAL;
+ req->dma = dma_map_single(sa->ipsec->eip93->dev, skb->data,
+ req->dma_len, req->dma_dir);
+ if (dma_mapping_error(sa->ipsec->eip93->dev, req->dma)) {
+ err = -ENOMEM;
+ goto free_req;
+ }
+
+ cdesc.pe_ctrl_stat_word =
+ FIELD_PREP(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN,
+ EIP93_PE_CTRL_HOST_READY) |
+ FIELD_PREP(EIP93_PE_CTRL_PE_PAD_CTRL_STAT,
+ EIP93_IPSEC_PAD_ALIGN) |
+ FIELD_PREP(EIP93_PE_CTRL_PE_PAD_VALUE, xo->proto) |
+ EIP93_PE_CTRL_PE_HASH_FINAL;
+ cdesc.src_addr = (u32 __force)req->dma + esp_offset +
+ sizeof(struct ip_esp_hdr) + sa->ivsize;
+ cdesc.dst_addr = (u32 __force)req->dma + esp_offset;
+ cdesc.sa_addr = sa->sa_record_base;
+ /*
+ * EIP93 ESP protocol-out mode wants the plaintext payload length. It
+ * generates ESP padding, next-header and ICV itself when tailroom was
+ * reserved instead of filled by the generic ESP path.
+ */
+ cdesc.pe_length_word = FIELD_PREP(EIP93_PE_LENGTH_HOST_PE_READY,
+ EIP93_PE_LENGTH_HOST_READY) |
+ FIELD_PREP(EIP93_PE_LENGTH_LENGTH, payload_len);
+
+ err = eip93_ipsec_submit(req, &cdesc);
+ if (err == -EINPROGRESS)
+ return err;
+
+ dma_unmap_single(sa->ipsec->eip93->dev, req->dma, req->dma_len,
+ req->dma_dir);
+free_req:
+ eip93_ipsec_request_put(req);
+put_sa:
+ eip93_ipsec_sa_put(sa);
+ return err;
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_xmit);
+
+int eip93_ipsec_receive(struct eip93_ipsec_sa *sa, struct sk_buff *skb,
+ unsigned int packet_len,
+ eip93_ipsec_complete_t complete, void *data)
+{
+ struct eip93_descriptor cdesc = {};
+ struct eip93_ipsec_request *req;
+ int err;
+
+ if (!sa || !complete || !eip93_ipsec_sa_get(sa))
+ return -EOPNOTSUPP;
+
+ if (skb_is_nonlinear(skb)) {
+ err = -EINVAL;
+ goto put_sa;
+ }
+
+ req = kmalloc(sizeof(*req), GFP_ATOMIC);
+ if (!req) {
+ err = -ENOMEM;
+ goto put_sa;
+ }
+
+ req->sa = sa;
+ req->skb = skb;
+ INIT_LIST_HEAD(&req->node);
+ refcount_set(&req->refcnt, 1);
+ req->complete = complete;
+ req->data = data;
+ if (!packet_len || packet_len > skb->len ||
+ packet_len > FIELD_MAX(EIP93_PE_LENGTH_LENGTH)) {
+ err = -EINVAL;
+ goto free_req;
+ }
+
+ req->dma_len = packet_len;
+ req->dma_dir = DMA_BIDIRECTIONAL;
+ req->dma = dma_map_single(sa->ipsec->eip93->dev, skb->data,
+ req->dma_len, req->dma_dir);
+ if (dma_mapping_error(sa->ipsec->eip93->dev, req->dma)) {
+ err = -ENOMEM;
+ goto free_req;
+ }
+
+ cdesc.pe_ctrl_stat_word =
+ FIELD_PREP(EIP93_PE_CTRL_PE_READY_DES_TRING_OWN,
+ EIP93_PE_CTRL_HOST_READY) |
+ FIELD_PREP(EIP93_PE_CTRL_PE_PAD_CTRL_STAT,
+ EIP93_IPSEC_PAD_ALIGN) |
+ EIP93_PE_CTRL_PE_HASH_FINAL;
+ cdesc.src_addr = (u32 __force)req->dma;
+ cdesc.dst_addr = (u32 __force)req->dma;
+ cdesc.sa_addr = sa->sa_record_base;
+ cdesc.pe_length_word = FIELD_PREP(EIP93_PE_LENGTH_HOST_PE_READY,
+ EIP93_PE_LENGTH_HOST_READY) |
+ FIELD_PREP(EIP93_PE_LENGTH_LENGTH, req->dma_len);
+
+ err = eip93_ipsec_submit(req, &cdesc);
+ if (err == -EINPROGRESS)
+ return err;
+
+ dma_unmap_single(sa->ipsec->eip93->dev, req->dma, req->dma_len,
+ req->dma_dir);
+free_req:
+ eip93_ipsec_request_put(req);
+put_sa:
+ eip93_ipsec_sa_put(sa);
+ return err;
+}
+EXPORT_SYMBOL_GPL(eip93_ipsec_receive);
diff --git a/drivers/crypto/inside-secure/eip93/eip93-main.c b/drivers/crypto/inside-secure/eip93/eip93-main.c
index 7dccfdeb7b11..1505e33d62bf 100644
--- a/drivers/crypto/inside-secure/eip93/eip93-main.c
+++ b/drivers/crypto/inside-secure/eip93/eip93-main.c
@@ -185,7 +185,9 @@ static int eip93_register_algs(struct eip93_device *eip93, u32 supported_algo_fl
static void eip93_handle_result_descriptor(struct eip93_device *eip93)
{
- struct crypto_async_request *async;
+ struct crypto_async_request *async = NULL;
+ struct eip93_ipsec_request *ipsec = NULL;
+ void *request;
struct eip93_descriptor *rdesc;
u16 desc_flags, crypto_idr;
bool last_entry;
@@ -224,11 +226,11 @@ static void eip93_handle_result_descriptor(struct eip93_device *eip93)
FIELD_GET(EIP93_PE_LENGTH_HOST_PE_READY, pe_length) !=
EIP93_PE_LENGTH_PE_READY);
- err = rdesc->pe_ctrl_stat_word & (EIP93_PE_CTRL_PE_EXT_ERR_CODE |
- EIP93_PE_CTRL_PE_EXT_ERR |
- EIP93_PE_CTRL_PE_SEQNUM_ERR |
- EIP93_PE_CTRL_PE_PAD_ERR |
- EIP93_PE_CTRL_PE_AUTH_ERR);
+ err = pe_ctrl_stat & (EIP93_PE_CTRL_PE_EXT_ERR_CODE |
+ EIP93_PE_CTRL_PE_EXT_ERR |
+ EIP93_PE_CTRL_PE_SEQNUM_ERR |
+ EIP93_PE_CTRL_PE_PAD_ERR |
+ EIP93_PE_CTRL_PE_AUTH_ERR);
desc_flags = FIELD_GET(EIP93_PE_USER_ID_DESC_FLAGS, rdesc->user_id);
crypto_idr = FIELD_GET(EIP93_PE_USER_ID_CRYPTO_IDR, rdesc->user_id);
@@ -248,23 +250,37 @@ static void eip93_handle_result_descriptor(struct eip93_device *eip93)
if (!last_entry)
goto get_more;
- /* Get crypto async ref only for last descriptor */
+ /* Get request ref only for last descriptor */
scoped_guard(spinlock_bh, &eip93->ring->idr_lock) {
- async = idr_find(&eip93->ring->crypto_async_idr, crypto_idr);
+ request = idr_find(&eip93->ring->crypto_async_idr, crypto_idr);
idr_remove(&eip93->ring->crypto_async_idr, crypto_idr);
}
+ if (!request) {
+ dev_warn_ratelimited(eip93->dev, "missing request id %u\n",
+ crypto_idr);
+ goto get_more;
+ }
/* Parse error in ctrl stat word */
err = eip93_parse_ctrl_stat_err(eip93, err);
+ if (desc_flags & EIP93_DESC_IPSEC) {
+ ipsec = request;
+ eip93_ipsec_handle_result(ipsec, err, pe_ctrl_stat, pe_length);
+ goto get_more;
+ }
+
+ async = request;
+
if (desc_flags & EIP93_DESC_SKCIPHER)
eip93_skcipher_handle_result(async, err);
-
- if (desc_flags & EIP93_DESC_AEAD)
+ else if (desc_flags & EIP93_DESC_AEAD)
eip93_aead_handle_result(async, err);
-
- if (desc_flags & EIP93_DESC_HASH)
+ else if (desc_flags & EIP93_DESC_HASH)
eip93_hash_handle_result(async, err);
+ else
+ dev_warn_ratelimited(eip93->dev, "unknown descriptor flags %#x\n",
+ desc_flags);
goto get_more;
}
@@ -279,21 +295,26 @@ static void eip93_done_task(unsigned long data)
static irqreturn_t eip93_irq_handler(int irq, void *data)
{
struct eip93_device *eip93 = data;
+ bool handled = false;
u32 irq_status;
irq_status = readl(eip93->base + EIP93_REG_INT_MASK_STAT);
if (FIELD_GET(EIP93_INT_RDR_THRESH, irq_status)) {
eip93_irq_disable(eip93, EIP93_INT_RDR_THRESH);
tasklet_schedule(&eip93->ring->done_task);
- return IRQ_HANDLED;
+ irq_status &= ~EIP93_INT_RDR_THRESH;
+ handled = true;
}
- /* Ignore errors in AUTO mode, handled by the RDR */
+ if (!irq_status)
+ return handled ? IRQ_HANDLED : IRQ_NONE;
+
+ eip93_ipsec_report_irq(eip93, irq_status);
+
eip93_irq_clear(eip93, irq_status);
- if (irq_status)
- eip93_irq_disable(eip93, irq_status);
+ eip93_irq_disable(eip93, irq_status);
- return IRQ_NONE;
+ return IRQ_HANDLED;
}
static void eip93_initialize(struct eip93_device *eip93, u32 supported_algo_flags)
@@ -455,15 +476,24 @@ static int eip93_crypto_probe(struct platform_device *pdev)
eip93_initialize(eip93, algo_flags);
- /* Init finished, enable RDR interrupt */
- eip93_irq_enable(eip93, EIP93_INT_RDR_THRESH);
+ ret = eip93_ipsec_register(eip93);
+ if (ret) {
+ eip93_cleanup(eip93);
+ return ret;
+ }
ret = eip93_register_algs(eip93, algo_flags);
if (ret) {
+ eip93_ipsec_unregister(eip93);
eip93_cleanup(eip93);
return ret;
}
+ /* Init finished, enable RDR and fatal error interrupts */
+ eip93_irq_enable(eip93, EIP93_INT_RDR_THRESH | EIP93_INT_INTERFACE_ERR |
+ EIP93_INT_RPOC_ERR | EIP93_INT_PE_RING_ERR |
+ EIP93_INT_HALT);
+
ver = readl(eip93->base + EIP93_REG_PE_REVISION);
/* EIP_EIP_NO:MAJOR_HW_REV:MINOR_HW_REV:HW_PATCH,PE(ALGO_FLAGS) */
dev_info(eip93->dev, "EIP%lu:%lx:%lx:%lx,PE(0x%x:0x%x)\n",
@@ -484,6 +514,7 @@ static void eip93_crypto_remove(struct platform_device *pdev)
algo_flags = readl(eip93->base + EIP93_REG_PE_OPTION_1);
+ eip93_ipsec_unregister(eip93);
eip93_unregister_algs(algo_flags, ARRAY_SIZE(eip93_algs));
eip93_cleanup(eip93);
}
diff --git a/drivers/crypto/inside-secure/eip93/eip93-main.h b/drivers/crypto/inside-secure/eip93/eip93-main.h
index 990c2401b7ce..ca1bda5b2ac0 100644
--- a/drivers/crypto/inside-secure/eip93/eip93-main.h
+++ b/drivers/crypto/inside-secure/eip93/eip93-main.h
@@ -13,6 +13,7 @@
#include <crypto/internal/skcipher.h>
#include <linux/bitfield.h>
#include <linux/interrupt.h>
+#include <linux/kconfig.h>
#define EIP93_RING_BUSY_DELAY 500
@@ -92,6 +93,8 @@
EIP93_HASH_SHA224 | \
EIP93_HASH_SHA256))
+struct eip93_ipsec;
+
/**
* struct eip93_device - crypto engine device structure
*/
@@ -101,6 +104,7 @@ struct eip93_device {
struct clk *clk;
int irq;
struct eip93_ring *ring;
+ struct eip93_ipsec *ipsec;
};
struct eip93_desc_ring {
@@ -124,8 +128,8 @@ struct eip93_ring {
/* command/result rings */
struct eip93_desc_ring cdr;
struct eip93_desc_ring rdr;
- spinlock_t write_lock;
- spinlock_t read_lock;
+ spinlock_t write_lock; /* command descriptor enqueue */
+ spinlock_t read_lock; /* result descriptor dequeue */
/* aync idr */
spinlock_t idr_lock;
struct idr crypto_async_idr;
@@ -148,4 +152,34 @@ struct eip93_alg_template {
} alg;
};
+struct eip93_ipsec_request;
+
+#if IS_ENABLED(CONFIG_CRYPTO_DEV_EIP93_IPSEC)
+int eip93_ipsec_register(struct eip93_device *eip93);
+void eip93_ipsec_unregister(struct eip93_device *eip93);
+void eip93_ipsec_handle_result(struct eip93_ipsec_request *req, int err,
+ u32 pe_ctrl_stat, u32 pe_length);
+void eip93_ipsec_report_irq(struct eip93_device *eip93, u32 irq_status);
+#else
+static inline int eip93_ipsec_register(struct eip93_device *eip93)
+{
+ return 0;
+}
+
+static inline void eip93_ipsec_unregister(struct eip93_device *eip93)
+{
+}
+
+static inline void eip93_ipsec_handle_result(struct eip93_ipsec_request *req,
+ int err, u32 pe_ctrl_stat,
+ u32 pe_length)
+{
+}
+
+static inline void eip93_ipsec_report_irq(struct eip93_device *eip93,
+ u32 irq_status)
+{
+}
+#endif
+
#endif /* _EIP93_MAIN_H_ */
diff --git a/include/crypto/eip93-ipsec.h b/include/crypto/eip93-ipsec.h
new file mode 100644
index 000000000000..bc0ba8f4f84e
--- /dev/null
+++ b/include/crypto/eip93-ipsec.h
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * EIP93 IPsec offload API
+ *
+ * Copyright (c) 2026 Jihong Min <hurryman2212@gmail.com>
+ */
+#ifndef _CRYPTO_EIP93_IPSEC_H
+#define _CRYPTO_EIP93_IPSEC_H
+
+#include <linux/bits.h>
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/kconfig.h>
+#include <linux/types.h>
+
+struct device;
+struct netlink_ext_ack;
+struct notifier_block;
+struct sk_buff;
+struct xfrm_state;
+
+struct eip93_ipsec;
+struct eip93_ipsec_sa;
+
+struct eip93_ipsec_result {
+ unsigned int packet_len;
+ u8 nexthdr;
+};
+
+enum eip93_ipsec_feature {
+ EIP93_IPSEC_FEATURE_ESP = BIT(0),
+ EIP93_IPSEC_FEATURE_GSO_ESP = BIT(1),
+ EIP93_IPSEC_FEATURE_HW_ESP_TX_CSUM = BIT(2),
+};
+
+enum eip93_ipsec_event {
+ EIP93_IPSEC_EVENT_REMOVE,
+ EIP93_IPSEC_EVENT_RESET,
+ EIP93_IPSEC_EVENT_DMA_ERROR,
+ EIP93_IPSEC_EVENT_CAPABILITY_LOSS,
+};
+
+typedef void (*eip93_ipsec_complete_t)(void *data, int err,
+ struct eip93_ipsec_result result);
+
+#if IS_REACHABLE(CONFIG_CRYPTO_DEV_EIP93) && \
+ IS_ENABLED(CONFIG_CRYPTO_DEV_EIP93_IPSEC)
+struct eip93_ipsec *eip93_ipsec_get(struct device *consumer);
+void eip93_ipsec_put(struct eip93_ipsec *ipsec);
+bool eip93_ipsec_available(struct eip93_ipsec *ipsec);
+u32 eip93_ipsec_features(struct eip93_ipsec *ipsec);
+int eip93_ipsec_register_notifier(struct notifier_block *nb);
+void eip93_ipsec_unregister_notifier(struct notifier_block *nb);
+int eip93_ipsec_state_add(struct eip93_ipsec *ipsec, struct xfrm_state *x,
+ struct netlink_ext_ack *extack,
+ struct eip93_ipsec_sa **sa);
+void eip93_ipsec_state_delete(struct eip93_ipsec_sa *sa);
+void eip93_ipsec_state_advance_esn(struct eip93_ipsec_sa *sa,
+ struct xfrm_state *x);
+int eip93_ipsec_xmit(struct eip93_ipsec_sa *sa, struct sk_buff *skb,
+ unsigned int esp_offset, eip93_ipsec_complete_t complete,
+ void *data);
+int eip93_ipsec_receive(struct eip93_ipsec_sa *sa, struct sk_buff *skb,
+ unsigned int packet_len,
+ eip93_ipsec_complete_t complete, void *data);
+#else
+static inline struct eip93_ipsec *eip93_ipsec_get(struct device *consumer)
+{
+ return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline void eip93_ipsec_put(struct eip93_ipsec *ipsec)
+{
+}
+
+static inline bool eip93_ipsec_available(struct eip93_ipsec *ipsec)
+{
+ return false;
+}
+
+static inline u32 eip93_ipsec_features(struct eip93_ipsec *ipsec)
+{
+ return 0;
+}
+
+static inline int eip93_ipsec_register_notifier(struct notifier_block *nb)
+{
+ return 0;
+}
+
+static inline void eip93_ipsec_unregister_notifier(struct notifier_block *nb)
+{
+}
+
+static inline int eip93_ipsec_state_add(struct eip93_ipsec *ipsec,
+ struct xfrm_state *x,
+ struct netlink_ext_ack *extack,
+ struct eip93_ipsec_sa **sa)
+{
+ if (sa)
+ *sa = NULL;
+
+ return -EOPNOTSUPP;
+}
+
+static inline void eip93_ipsec_state_delete(struct eip93_ipsec_sa *sa)
+{
+}
+
+static inline void eip93_ipsec_state_advance_esn(struct eip93_ipsec_sa *sa,
+ struct xfrm_state *x)
+{
+}
+
+static inline int eip93_ipsec_xmit(struct eip93_ipsec_sa *sa,
+ struct sk_buff *skb, unsigned int esp_offset,
+ eip93_ipsec_complete_t complete, void *data)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline int eip93_ipsec_receive(struct eip93_ipsec_sa *sa,
+ struct sk_buff *skb,
+ unsigned int packet_len,
+ eip93_ipsec_complete_t complete,
+ void *data)
+{
+ return -EOPNOTSUPP;
+}
+#endif
+
+#endif /* _CRYPTO_EIP93_IPSEC_H */
--
2.53.0
^ permalink raw reply related
* [PATCH 1/3] xfrm: extend ESP offload infrastructure for packet engines
From: Jihong Min @ 2026-05-23 12:15 UTC (permalink / raw)
To: Christian Marangi, Antoine Tenart, Herbert Xu, David S . Miller,
Lorenzo Bianconi, Andrew Lunn, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Steffen Klassert
Cc: linux-kernel, linux-crypto, linux-arm-kernel, linux-mediatek,
netdev, Jihong Min
In-Reply-To: <20260523121522.3023992-1-hurryman2212@gmail.com>
Some ESP offload engines operate on whole ESP packets rather than the
generic software trailer layout. They can generate outbound ESP padding,
next-header and ICV bytes in hardware, and inbound decapsulation can
return an already-trimmed packet with the recovered next-header value.
Add a netdev offload callback for drivers to opt into hardware-generated
ESP TX trailers, carry the reserved ESP TX tail length in xfrm_offload,
and let ESP input skip software trailer removal when hardware has already
done it.
This keeps the default ESP offload behavior unchanged for existing devices
while providing the infrastructure needed by packet-mode ESP engines.
Assisted-by: Codex:gpt-5.5
Signed-off-by: Jihong Min <hurryman2212@gmail.com>
---
include/linux/netdevice.h | 3 +++
include/net/xfrm.h | 8 +++++++-
net/ipv4/esp4.c | 6 +++++-
net/ipv4/esp4_offload.c | 29 ++++++++++++++++++++++++++++-
net/ipv6/esp6.c | 6 +++++-
net/ipv6/esp6_offload.c | 29 ++++++++++++++++++++++++++++-
6 files changed, 76 insertions(+), 5 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0e1e581efc5a..b6ff04c3df78 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1043,6 +1043,9 @@ struct xfrmdev_ops {
struct xfrm_state *x);
bool (*xdo_dev_offload_ok) (struct sk_buff *skb,
struct xfrm_state *x);
+ /* Return true when the device generates the ESP trailer/ICV itself. */
+ bool (*xdo_dev_esp_tx_hw_trailer)(struct sk_buff *skb,
+ struct xfrm_state *x);
void (*xdo_dev_state_advance_esn) (struct xfrm_state *x);
void (*xdo_dev_state_update_stats) (struct xfrm_state *x);
int (*xdo_dev_policy_add) (struct xfrm_policy *x, struct netlink_ext_ack *extack);
diff --git a/include/net/xfrm.h b/include/net/xfrm.h
index 10d3edde6b2f..160069901e0a 100644
--- a/include/net/xfrm.h
+++ b/include/net/xfrm.h
@@ -1141,7 +1141,7 @@ struct xfrm_offload {
#define CRYPTO_FALLBACK 8
#define XFRM_GSO_SEGMENT 16
#define XFRM_GRO 32
-/* 64 is free */
+#define XFRM_ESP_NO_TRAILER 64
#define XFRM_DEV_RESUME 128
#define XFRM_XMIT 256
@@ -1158,6 +1158,12 @@ struct xfrm_offload {
/* Used to keep whole l2 header for transport mode GRO */
__u16 orig_mac_len;
+ /*
+ * ESP packet engines can reserve tailroom in the generic ESP path and
+ * generate padding, next-header and ICV bytes during device TX.
+ */
+ __u16 esp_tx_tailen;
+
__u8 proto;
__u8 inner_ipproto;
};
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index 6a5febbdbee4..f21c8f2e60f7 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -720,7 +720,11 @@ int esp_input_done2(struct sk_buff *skb, int err)
if (unlikely(err))
goto out;
- err = esp_remove_trailer(skb);
+ /* Hardware ESP decapsulation can already remove pad/trailer/ICV. */
+ if (xo && (xo->flags & XFRM_ESP_NO_TRAILER))
+ err = xo->proto;
+ else
+ err = esp_remove_trailer(skb);
if (unlikely(err < 0))
goto out;
diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c
index abd77162f5e7..f00fff98b69f 100644
--- a/net/ipv4/esp4_offload.c
+++ b/net/ipv4/esp4_offload.c
@@ -270,8 +270,10 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_
struct xfrm_offload *xo;
struct ip_esp_hdr *esph;
struct crypto_aead *aead;
+ struct sk_buff *trailer;
struct esp_info esp;
bool hw_offload = true;
+ bool hw_trailer = false;
__u32 seq;
int encap_type = 0;
@@ -281,6 +283,7 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_
if (!xo)
return -EINVAL;
+ xo->esp_tx_tailen = 0;
if ((!(features & NETIF_F_HW_ESP) &&
!(skb->dev->gso_partial_features & NETIF_F_HW_ESP)) ||
@@ -303,13 +306,37 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_
esp.clen = ALIGN(skb->len + 2 + esp.tfclen, blksize);
esp.plen = esp.clen - skb->len - esp.tfclen;
esp.tailen = esp.tfclen + esp.plen + alen;
+ if (esp.tailen > U16_MAX)
+ return -EINVAL;
esp.esph = ip_esp_hdr(skb);
if (x->encap)
encap_type = x->encap->encap_type;
- if (!hw_offload || !skb_is_gso(skb) || (hw_offload && encap_type == UDP_ENCAP_ESPINUDP)) {
+ if (hw_offload && !skb_is_gso(skb) && !encap_type && x->xso.dev &&
+ x->xso.dev->xfrmdev_ops &&
+ x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer)
+ hw_trailer =
+ x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer(skb, x);
+
+ if (hw_trailer) {
+ int esph_offset;
+
+ /*
+ * The device packet engine will write ESP padding, next-header
+ * and ICV bytes. Keep skb->len unchanged here, but make sure the
+ * later DMA writer owns enough linear tailroom.
+ */
+ esph_offset = (unsigned char *)esp.esph - skb_transport_header(skb);
+ esp.nfrags = skb_cow_data(skb, esp.tailen, &trailer);
+ if (esp.nfrags < 0)
+ return esp.nfrags;
+ esp.esph = (struct ip_esp_hdr *)(skb_transport_header(skb) +
+ esph_offset);
+ xo->esp_tx_tailen = esp.tailen;
+ } else if (!hw_offload || !skb_is_gso(skb) ||
+ (hw_offload && encap_type == UDP_ENCAP_ESPINUDP)) {
esp.nfrags = esp_output_head(x, skb, &esp);
if (esp.nfrags < 0)
return esp.nfrags;
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index 9c06c5a1419d..730588f8eaba 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -751,7 +751,11 @@ int esp6_input_done2(struct sk_buff *skb, int err)
if (unlikely(err))
goto out;
- err = esp_remove_trailer(skb);
+ /* Hardware ESP decapsulation can already remove pad/trailer/ICV. */
+ if (xo && (xo->flags & XFRM_ESP_NO_TRAILER))
+ err = xo->proto;
+ else
+ err = esp_remove_trailer(skb);
if (unlikely(err < 0))
goto out;
diff --git a/net/ipv6/esp6_offload.c b/net/ipv6/esp6_offload.c
index 22895521a57d..d124493da40b 100644
--- a/net/ipv6/esp6_offload.c
+++ b/net/ipv6/esp6_offload.c
@@ -308,8 +308,10 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features
int blksize;
struct xfrm_offload *xo;
struct crypto_aead *aead;
+ struct sk_buff *trailer;
struct esp_info esp;
bool hw_offload = true;
+ bool hw_trailer = false;
__u32 seq;
esp.inplace = true;
@@ -318,6 +320,7 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features
if (!xo)
return -EINVAL;
+ xo->esp_tx_tailen = 0;
if (!(features & NETIF_F_HW_ESP) || x->xso.dev != skb->dev) {
xo->flags |= CRYPTO_FALLBACK;
@@ -338,8 +341,32 @@ static int esp6_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features
esp.clen = ALIGN(skb->len + 2 + esp.tfclen, blksize);
esp.plen = esp.clen - skb->len - esp.tfclen;
esp.tailen = esp.tfclen + esp.plen + alen;
+ if (esp.tailen > U16_MAX)
+ return -EINVAL;
- if (!hw_offload || !skb_is_gso(skb)) {
+ if (hw_offload && !skb_is_gso(skb) && !x->encap && x->xso.dev &&
+ x->xso.dev->xfrmdev_ops &&
+ x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer)
+ hw_trailer =
+ x->xso.dev->xfrmdev_ops->xdo_dev_esp_tx_hw_trailer(skb, x);
+
+ if (hw_trailer) {
+ int esph_offset;
+
+ /*
+ * The device packet engine will write ESP padding, next-header
+ * and ICV bytes. Keep skb->len unchanged here, but make sure the
+ * later DMA writer owns enough linear tailroom.
+ */
+ esp.esph = ip_esp_hdr(skb);
+ esph_offset = (unsigned char *)esp.esph - skb_transport_header(skb);
+ esp.nfrags = skb_cow_data(skb, esp.tailen, &trailer);
+ if (esp.nfrags < 0)
+ return esp.nfrags;
+ esp.esph = (struct ip_esp_hdr *)(skb_transport_header(skb) +
+ esph_offset);
+ xo->esp_tx_tailen = esp.tailen;
+ } else if (!hw_offload || !skb_is_gso(skb)) {
esp.nfrags = esp6_output_head(x, skb, &esp);
if (esp.nfrags < 0)
return esp.nfrags;
--
2.53.0
^ permalink raw reply related
* [PATCH 0/3] Add packet-mode ESP offload for Airoha/EIP93
From: Jihong Min @ 2026-05-23 12:15 UTC (permalink / raw)
To: Christian Marangi, Antoine Tenart, Herbert Xu, David S . Miller,
Lorenzo Bianconi, Andrew Lunn, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Steffen Klassert
Cc: linux-kernel, linux-crypto, linux-arm-kernel, linux-mediatek,
netdev, Jihong Min
This series adds the missing plumbing for ESP offload engines that
operate on whole ESP packets instead of only exposing AES/HMAC through
the crypto API AEAD interface.
The normal ESP software path can already call into accelerated AEAD
algorithms, but packet-mode engines such as EIP93 can also generate and
consume ESP packet framing: padding, pad length, next header and ICV.
That needs a slightly different XFRM offload contract so the netdev
driver can hand the skb to a packet backend rather than trying to make
hardware fit the software trailer layout.
Patch 1 extends the ESP offload infrastructure for packet engines while
preserving the existing behavior for drivers that do not opt in.
Patch 2 exposes an EIP93 ESP packet backend for encapsulation and
decapsulation.
Patch 3 wires Airoha Ethernet GDM netdevs and DSA user ports to that
backend through xfrmdev_ops. ESP GSO and ESP TX checksum offload remain
disabled.
Runtime testing was done on a Gemtek W1700K2 running OpenWrt with the
same changes applied on top of a 6.18.31-based kernel.
Test parameters:
- Static IPv4 transport-mode XFRM SAs between the AP and host.
- ESP transform: auth hmac(sha1), enc cbc(aes) with a 128-bit AES key.
- iperf3 TCP test, AP as client and host as server:
iperf3 -c <host_ip> -P 4 -t 10
- The host always used normal Linux XFRM software processing.
- With AP ESP offload disabled, the AP also used the Linux XFRM
software path; in this setup, EIP93-backed AEAD crypto was still
available to that path.
Network-relevant test setup:
- AP: Gemtek W1700K2, Airoha AN7581/EN7581, 4x Arm Cortex-A53 at
1.4 GHz, 2 GiB RAM, airoha_eth wan (GDM2) netdev, 10Gb/s full-duplex,
MTU 9200, EIP93 crypto and IPsec packet engine present.
- Host: AMD Ryzen 9 9950X3D, 16 cores/32 threads, Open vSwitch,
MTU 9978, backed by a ConnectX-6 Dx 10Gb/s full-duplex link.
AP to host iperf3 result:
AP offload Sender Receiver Retransmits
on 918.2 Mbit/s 913.6 Mbit/s 0
off 782.4 Mbit/s 778.6 Mbit/s 3569
This is a 17.3% receiver-side throughput improvement for the AP TX ESP
path in this setup, with retransmits eliminated in the offloaded run.
Jihong Min (3):
xfrm: extend ESP offload infrastructure for packet engines
crypto: inside-secure: add EIP93 ESP packet backend
net: airoha: add EIP93-backed ESP XFRM offload
MAINTAINERS | 1 +
drivers/crypto/inside-secure/eip93/Kconfig | 10 +
drivers/crypto/inside-secure/eip93/Makefile | 1 +
.../crypto/inside-secure/eip93/eip93-ipsec.c | 1413 ++++++++++++++++
.../crypto/inside-secure/eip93/eip93-main.c | 69 +-
.../crypto/inside-secure/eip93/eip93-main.h | 38 +-
drivers/net/ethernet/airoha/Kconfig | 11 +
drivers/net/ethernet/airoha/Makefile | 1 +
drivers/net/ethernet/airoha/airoha_eth.c | 51 +-
drivers/net/ethernet/airoha/airoha_eth.h | 69 +
drivers/net/ethernet/airoha/airoha_xfrm.c | 1474 +++++++++++++++++
include/crypto/eip93-ipsec.h | 132 ++
include/linux/netdevice.h | 3 +
include/net/xfrm.h | 8 +-
net/ipv4/esp4.c | 6 +-
net/ipv4/esp4_offload.c | 29 +-
net/ipv6/esp6.c | 6 +-
net/ipv6/esp6_offload.c | 29 +-
18 files changed, 3324 insertions(+), 27 deletions(-)
create mode 100644 drivers/crypto/inside-secure/eip93/eip93-ipsec.c
create mode 100644 drivers/net/ethernet/airoha/airoha_xfrm.c
create mode 100644 include/crypto/eip93-ipsec.h
--
2.53.0
^ permalink raw reply
* Re: [PATCH] crypto: nx: fix nx_crypto_ctx_exit argument
From: Simon Richter @ 2026-05-23 6:30 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto, linuxppc-dev
In-Reply-To: <20260522184403.GA35544@quark>
[-- Attachment #1.1: Type: text/plain, Size: 3739 bytes --]
Hi,
On 5/23/26 03:44, Eric Biggers wrote:
> Otherwise this looks good. Really there's a good chance this driver is
> no longer useful (if it ever was) and should just be deleted, but that
> would be a separate effort.
I happen to have one (well, two) of these, so this is relevant to my
interests.
tl;dr: the crypto drivers are most likely unused, the hardware is great,
but the crypto subsystem cannot use it efficiently.
Below drivers/crypto/nx, there are three drivers in a trenchcoat:
- an NX crypto driver that is not endian safe, can therefore only be
used on big endian systems, and that implements a bunch of AES modes
plus SHA256/SHA512, all of them synchronous.
- an scomp driver with an IBM specific compression algorithm
- a gzip driver that does not integrate with the crypto subsystem and
provides its own userspace interface.
The "big endian only" thing is a massive restriction, this is how IBM
separates enterprise and hobbyist customers, so if there are users of
this module, then they both have enterprise support contracts.
The gzip mode is really useful, with 4 GB of random data I get
$ time ./nx_gzip test.bin
real 0m2.989s
user 0m1.317s
sys 0m1.665s
$ time gzip -9k test.bin
real 2m57.468s
user 2m55.325s
sys 0m1.682s
so 3 GB/s vs 22 MB/s. Even if I had a workload where I could use all the
CPU cores in parallel, offloading is still faster, 120W cheaper and
leaves the CPU free as a bonus, so I think that's a no-brainer.
The "842" compression is mainly designed to be fast, the marketing
material claims > 25 GB/s, which makes sense, this unit sits on a 128
bit wide bus clocked at 2 GHz, and the algorithm is designed around
that. On the other hand it is fairly niche.
I couldn't find numbers for the AES and SHA units, I'd expect them to be
in the same ballpark, but I cannot measure them easily. CPU is ~500 MB/s
for SHA1 and SHA512, ~300 MB/s for SHA256, that should be easy to beat
(even a primitive 2-way SHA256 would be at 4 GB/s, and I doubt IBM left
it at that).
POWER11 introduces new opcodes, which will shake things up, but these
machines are on a fairly long replacement cycle.
The main problem with getting the advertised performance is feeding
requests fast enough. Large requests are easy, but the optimum strategy
for feeding small requests is just to start submitting, poll old
requests for completion inbetween, and start requesting interrupts only
if nothing is complete and it looks like the unit will be busy for a while.
That's not what is currently implemented, and I doubt it could be
implemented with the current kernel interfaces, so getting decent
performance inside the kernel would require some redesign.
I suppose that also explains the synchronous implementation: we are
submitting the request and polling for completion, so overhead is fairly
minimal and should break even at a few hundred bytes, but obviously that
is not the ideal way to run this thing.
The endianness issues are trivial to fix (really just needs a sprinkle
of cpu_to_beXX/beXX_to_cpu when putting the job control blocks together,
like nx-842 does); if you have a definition of what you would consider a
"real world" workload for AES I could run that to gather some numbers.
So far however, no one bothered fixing this, and I'm pretty meh about it
myself since I don't have SHA/AES workloads in the kernel, only in
userspace.
Other than that, if you decide to remove the driver from the crypto
subsystem, then nx-gzip should be kept (and probably moved somewhere
else), because it is not a crypto driver, it just shares a bunch of
headers with them.
Simon
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* [PATCH v2] crypto: nx: fix nx_crypto_ctx_exit argument
From: Sam James @ 2026-05-23 4:08 UTC (permalink / raw)
To: Breno Leitão, Nayna Jain, Paulo Flabiano Smorigo,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Herbert Xu, David S. Miller,
Ard Biesheuvel, Eric Biggers
Cc: Sam James, Eric Biggers, stable, Calvin Buckley, Brad Spengler,
linux-crypto, linuxppc-dev, linux-kernel
In-Reply-To: <20260522184403.GA35544@quark>
nx_crypto_ctx_shash_exit calls nx_crypto_ctx_exit with crypto_shash_ctx(...)
but crypto_shash_ctx gives a nx_crypto_ctx *, not a crypto_tfm *.
Fix the type in nx_crypto_ctx_exit and drop the bogus crypto_tfm_ctx
call.
This fixes the following oops:
BUG: Unable to handle kernel data access at 0xc0403effffffffc8
Faulting instruction address: 0xc000000000396cb4
Oops: Kernel access of bad area, sig: 11 [#15]
Call Trace:
nx_crypto_ctx_shash_exit+0x24/0x60
crypto_shash_exit_tfm+0x28/0x40
crypto_destroy_tfm+0x98/0x140
crypto_exit_ahash_using_shash+0x20/0x40
crypto_destroy_tfm+0x98/0x140
hash_release+0x1c/0x30
alg_sock_destruct+0x38/0x60
__sk_destruct+0x48/0x2b0
af_alg_release+0x58/0xb0
__sock_release+0x68/0x150
sock_close+0x20/0x40
__fput+0x110/0x3a0
sys_close+0x48/0xa0
system_call_exception+0x140/0x2d0
system_call_common+0xf4/0x258
.. which came from hardlink(1) opportunistically using AF_ALG.
The same problem exists with nx_crypto_ctx_skcipher_exit getting a context
it wasn't expecting, but apparently nobody hit that for years.
Cc: Eric Biggers <ebiggers@kernel.org>
Cc: stable@vger.kernel.org
Fixes: bfd9efddf990 ("crypto: nx - convert AES-ECB to skcipher API")
Fixes: 9420e628e7d8 ("crypto: nx - Use API partial block handling")
Reviewed-by: Eric Biggers <ebiggers@kernel.org>
Reported-by: Calvin Buckley <calvin@cmpct.info>
Tested-by: Calvin Buckley <calvin@cmpct.info>
Suggested-by: Brad Spengler <brad.spengler@opensrcsec.com>
Signed-off-by: Sam James <sam@gentoo.org>
---
v2: Add stable cc, fix doc for tfm param.
drivers/crypto/nx/nx.c | 6 ++----
drivers/crypto/nx/nx.h | 2 +-
2 files changed, 3 insertions(+), 5 deletions(-)
diff --git a/drivers/crypto/nx/nx.c b/drivers/crypto/nx/nx.c
index 78135fb13f5c..f4bc947086f8 100644
--- a/drivers/crypto/nx/nx.c
+++ b/drivers/crypto/nx/nx.c
@@ -714,15 +714,13 @@ int nx_crypto_ctx_aes_xcbc_init(struct crypto_shash *tfm)
/**
* nx_crypto_ctx_exit - destroy a crypto api context
*
- * @tfm: the crypto transform pointer for the context
+ * @tfm: the crypto api context
*
* As crypto API contexts are destroyed, this exit hook is called to free the
* memory associated with it.
*/
-void nx_crypto_ctx_exit(struct crypto_tfm *tfm)
+void nx_crypto_ctx_exit(struct nx_crypto_ctx *nx_ctx)
{
- struct nx_crypto_ctx *nx_ctx = crypto_tfm_ctx(tfm);
-
kfree_sensitive(nx_ctx->kmem);
nx_ctx->csbcpb = NULL;
nx_ctx->csbcpb_aead = NULL;
diff --git a/drivers/crypto/nx/nx.h b/drivers/crypto/nx/nx.h
index 36974f08490a..6dfabfbf8192 100644
--- a/drivers/crypto/nx/nx.h
+++ b/drivers/crypto/nx/nx.h
@@ -153,7 +153,7 @@ int nx_crypto_ctx_aes_ctr_init(struct crypto_skcipher *tfm);
int nx_crypto_ctx_aes_cbc_init(struct crypto_skcipher *tfm);
int nx_crypto_ctx_aes_ecb_init(struct crypto_skcipher *tfm);
int nx_crypto_ctx_sha_init(struct crypto_shash *tfm);
-void nx_crypto_ctx_exit(struct crypto_tfm *tfm);
+void nx_crypto_ctx_exit(struct nx_crypto_ctx *nx_ctx);
void nx_crypto_ctx_skcipher_exit(struct crypto_skcipher *tfm);
void nx_crypto_ctx_aead_exit(struct crypto_aead *tfm);
void nx_crypto_ctx_shash_exit(struct crypto_shash *tfm);
base-commit: 758c807bb943138f887d42d986b645e12446ba9c
--
2.54.0
^ permalink raw reply related
* Re: (subset) [PATCH v5 00/13] Add explicit clock vote and enable power-domain for QCOM-ICE
From: Bjorn Andersson @ 2026-05-23 2:19 UTC (permalink / raw)
To: Herbert Xu, David S. Miller, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Abel Vesa, Manivannan Sadhasivam,
cros-qcom-dts-watchers, Eric Biggers, Dmitry Baryshkov,
Jingyi Wang, Tengfei Fan, Bartosz Golaszewski, David Wronek,
Luca Weiss, Neil Armstrong, Melody Olvera, Alexander Koskovich,
Abel Vesa, Harshal Dev
Cc: Brian Masney, Neeraj Soni, Gaurav Kashyap, linux-arm-msm,
linux-crypto, devicetree, linux-kernel, Krzysztof Kozlowski,
Konrad Dybcio, Kuldeep Singh, Krzysztof Kozlowski,
Manivannan Sadhasivam
In-Reply-To: <20260416-qcom_ice_power_and_clk_vote-v5-0-5ccf5d7e2846@oss.qualcomm.com>
On Thu, 16 Apr 2026 17:29:17 +0530, Harshal Dev wrote:
> When the kernel is booted without the 'clk_ignore_unused' and
> 'pd_ignore_unused' command‑line flags, votes for unused clocks and power
> domains are dropped by the kernel post late_init and deferred probe
> timeout. Depending on the relative timing between the ICE probe and the
> kernel disabling the unused clocks and power domains occasional unclocked
> register accesses or 'stuck' clocks are observed during QCOM‑ICE probe.
> When the 'iface' clock is not voted on, unclocked register access would
> be observed. On the other hand, if the associated power-domain for ICE
> is not enabled, a 'stuck' clock is observed.
>
> [...]
Applied, thanks!
[03/13] arm64: dts: qcom: kaanapali: Add power-domain and iface clk for ice node
commit: 11b48f6d5ed505ced9cd3645d6615279198a7a54
[04/13] arm64: dts: qcom: lemans: Add power-domain and iface clk for ice node
commit: 04566e287b35fde9fd129db5fdf6a96e336af55c
[05/13] arm64: dts: qcom: monaco: Add power-domain and iface clk for ice node
commit: 68d5d9701a7ab1b1f9c76feaa3a24ca716f03f0b
[06/13] arm64: dts: qcom: sc7180: Add power-domain and iface clk for ice node
commit: 7cd7271ac525e4eadd22734f418219f247638f43
[07/13] arm64: dts: qcom: kodiak: Add power-domain and iface clk for ice node
commit: cca53c338ad87edc4b46d2d82730fd8ca01a164f
[08/13] arm64: dts: qcom: sm8450: Add power-domain and iface clk for ice node
commit: 3a5cb1ccbfb3141862b28f24cd5050083233aae7
[09/13] arm64: dts: qcom: sm8550: Add power-domain and iface clk for ice node
commit: 52696dbbe7bbe0c8fc8c17133ffb5133b8cf37a6
[10/13] arm64: dts: qcom: sm8650: Add power-domain and iface clk for ice node
commit: c62b084d5d1564f808408a2f7d4c514e57cd4106
[11/13] arm64: dts: qcom: sm8750: Add power-domain and iface clk for ice node
commit: 081ac792f0ea6d27a4b130c70cfd7544efee8137
Best regards,
--
Bjorn Andersson <andersson@kernel.org>
^ permalink raw reply
* [PATCH v4 12/12] crypto: atmel-sha204a - switch to module_i2c_driver
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Replace explicit module init and exit boilerplate functions with the
module_i2c_driver() macro helper to simplify the driver registration
path.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-sha204a.c | 13 +------------
1 file changed, 1 insertion(+), 12 deletions(-)
diff --git a/drivers/crypto/atmel-sha204a.c b/drivers/crypto/atmel-sha204a.c
index 86a68f2a27e0..74f91e176713 100644
--- a/drivers/crypto/atmel-sha204a.c
+++ b/drivers/crypto/atmel-sha204a.c
@@ -257,18 +257,7 @@ static struct i2c_driver atmel_sha204a_driver = {
.driver.of_match_table = atmel_sha204a_dt_ids,
};
-static int __init atmel_sha204a_init(void)
-{
- return i2c_add_driver(&atmel_sha204a_driver);
-}
-
-static void __exit atmel_sha204a_exit(void)
-{
- i2c_del_driver(&atmel_sha204a_driver);
-}
-
-module_init(atmel_sha204a_init);
-module_exit(atmel_sha204a_exit);
+module_i2c_driver(atmel_sha204a_driver);
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
MODULE_DESCRIPTION("Microchip / Atmel SHA204A (I2C) driver");
--
2.39.5
^ permalink raw reply related
* [PATCH v4 11/12] crypto: atmel-sha204a - integrate into core management tracking
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Register the SHA204A I2C device instance into the shared atmel_i2c client
management tracking list during the probe phase. This allows the driver to
participate in the central hardware selection infrastructure.
Rework the error-unwind paths inside atmel_sha204a_probe() to prevent stale
entries from remaining in the global tracking structures if a partial
initialization failure occurs. If sysfs group creation fails, explicitly
trigger devm_hwrng_unregister() to preserve the strict lifecycle ordering
introduced in previous stability fixes.
Convert the removal path to use the core teardown helpers. Ensure the
device readiness state is deactivated using atmel_i2c_deactivate_client()
and the tracking node is removed via atmel_i2c_unregister_client() before
local memory resources are freed. This guarantees that any in-flight work
queue items are unconditionally flushed, eliminating a potential
Use-After-Free (UAF) window during device removal.
No functional change intended beyond improved lifecycle handling.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-sha204a.c | 26 ++++++++++++++++++++++----
1 file changed, 22 insertions(+), 4 deletions(-)
diff --git a/drivers/crypto/atmel-sha204a.c b/drivers/crypto/atmel-sha204a.c
index 0c5b5cdbfcbc..86a68f2a27e0 100644
--- a/drivers/crypto/atmel-sha204a.c
+++ b/drivers/crypto/atmel-sha204a.c
@@ -177,9 +177,15 @@ static int atmel_sha204a_probe(struct i2c_client *client)
return ret;
i2c_priv = i2c_get_clientdata(client);
+ i2c_priv->ready = false;
i2c_priv->caps = 0;
+ spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
+ list_add_tail(&i2c_priv->i2c_client_list_node,
+ &atmel_i2c_mgmt.i2c_client_list);
+ spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+
memset(&i2c_priv->hwrng, 0, sizeof(i2c_priv->hwrng));
i2c_priv->hwrng.name = dev_name(&client->dev);
@@ -192,15 +198,26 @@ static int atmel_sha204a_probe(struct i2c_client *client)
ret = devm_hwrng_register(&client->dev, &i2c_priv->hwrng);
if (ret) {
dev_err(&client->dev, "failed to register RNG (%d)\n", ret);
- return ret;
+ goto err_list_del;
}
ret = sysfs_create_group(&client->dev.kobj, &atmel_sha204a_groups);
if (ret) {
dev_err(&client->dev, "failed to create sysfs group (%d)\n", ret);
- return ret;
+ goto err_hwrng_unregister;
}
+ spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
+ i2c_priv->ready = true;
+ spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+
+ return 0;
+
+err_hwrng_unregister:
+ devm_hwrng_unregister(&client->dev, &i2c_priv->hwrng);
+err_list_del:
+ atmel_i2c_unregister_client(i2c_priv);
+
return ret;
}
@@ -208,9 +225,11 @@ static void atmel_sha204a_remove(struct i2c_client *client)
{
struct atmel_i2c_client_priv *i2c_priv = i2c_get_clientdata(client);
+ atmel_i2c_deactivate_client(i2c_priv);
+
sysfs_remove_group(&client->dev.kobj, &atmel_sha204a_groups);
devm_hwrng_unregister(&client->dev, &i2c_priv->hwrng);
- atmel_i2c_flush_queue();
+ atmel_i2c_unregister_client(i2c_priv);
kfree((void *)i2c_priv->hwrng.priv);
}
@@ -245,7 +264,6 @@ static int __init atmel_sha204a_init(void)
static void __exit atmel_sha204a_exit(void)
{
- atmel_i2c_flush_queue();
i2c_del_driver(&atmel_sha204a_driver);
}
--
2.39.5
^ permalink raw reply related
* [PATCH v4 10/12] crypto: atmel-i2c - implement capability-based client selection
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Extend the shared I2C client allocation interface to support feature-aware
hardware selection by introducing capability filtering.
Add a 'caps' mask to 'struct atmel_i2c_client_priv' alongside an
'atmel_i2c_capability' enum. The allocator now explicitly filters hardware
nodes by a requested capability bit while retaining the least-loaded device
load-balancing scheme.
Update the ECC driver to advertise ATMEL_CAP_ECDH configuration capability
during probe, and adapt the tfm context setup execution path to request
this specific capability variant. Initialize the bitmask field to zero
inside the SHA204A driver context for now.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-ecc.c | 4 +++-
drivers/crypto/atmel-i2c.c | 6 +++++-
drivers/crypto/atmel-i2c.h | 8 +++++++-
drivers/crypto/atmel-sha204a.c | 2 ++
4 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c
index 16e607cd06c4..76b8f9e7c2e1 100644
--- a/drivers/crypto/atmel-ecc.c
+++ b/drivers/crypto/atmel-ecc.c
@@ -212,7 +212,7 @@ static int atmel_ecdh_init_tfm(struct crypto_kpp *tfm)
struct atmel_ecdh_ctx *ctx = kpp_tfm_ctx(tfm);
ctx->curve_id = ECC_CURVE_NIST_P256;
- ctx->client = atmel_i2c_client_alloc();
+ ctx->client = atmel_i2c_client_alloc(ATMEL_CAP_ECDH);
if (IS_ERR(ctx->client)) {
pr_err("tfm - i2c_client binding failed\n");
return PTR_ERR(ctx->client);
@@ -287,6 +287,8 @@ static int atmel_ecc_probe(struct i2c_client *client)
i2c_priv = i2c_get_clientdata(client);
i2c_priv->ready = false;
+ i2c_priv->caps = BIT(ATMEL_CAP_ECDH);
+
spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
list_add_tail(&i2c_priv->i2c_client_list_node,
&atmel_i2c_mgmt.i2c_client_list);
diff --git a/drivers/crypto/atmel-i2c.c b/drivers/crypto/atmel-i2c.c
index 92d3e28f9d9a..4953b8fcb02d 100644
--- a/drivers/crypto/atmel-i2c.c
+++ b/drivers/crypto/atmel-i2c.c
@@ -57,7 +57,7 @@ static void atmel_i2c_checksum(struct atmel_i2c_cmd *cmd)
*__crc16 = cpu_to_le16(bitrev16(crc16(0, data, len)));
}
-struct i2c_client *atmel_i2c_client_alloc(void)
+struct i2c_client *atmel_i2c_client_alloc(enum atmel_i2c_capability cap)
{
struct atmel_i2c_client_priv *i2c_priv, *min_i2c_priv = NULL;
struct i2c_client *client = ERR_PTR(-ENODEV);
@@ -75,6 +75,10 @@ struct i2c_client *atmel_i2c_client_alloc(void)
i2c_client_list_node) {
if (!i2c_priv->ready)
continue;
+
+ if (!(i2c_priv->caps & BIT(cap)))
+ continue;
+
tfm_cnt = atomic_read(&i2c_priv->tfm_count);
if (tfm_cnt < min_tfm_cnt) {
min_tfm_cnt = tfm_cnt;
diff --git a/drivers/crypto/atmel-i2c.h b/drivers/crypto/atmel-i2c.h
index ddab80bc1a72..af2e49332ab6 100644
--- a/drivers/crypto/atmel-i2c.h
+++ b/drivers/crypto/atmel-i2c.h
@@ -115,6 +115,10 @@ struct atmel_i2c_cmd {
#define ECDH_PREFIX_MODE 0x00
/* Used for binding tfm objects to i2c clients. */
+enum atmel_i2c_capability {
+ ATMEL_CAP_ECDH = 0,
+};
+
struct atmel_i2c_client_mgmt {
struct list_head i2c_client_list;
spinlock_t i2c_list_lock;
@@ -131,6 +135,7 @@ extern struct atmel_i2c_client_mgmt atmel_i2c_mgmt;
* @tfm_count : number of active crypto transformations on i2c client
* @hwrng : hold the hardware generated rng
* @ready : hw client is ready to use
+ * @caps : feature capability of the particular driver
*
* Reads and writes from/to the i2c client are sequential. The first byte
* transmitted to the device is treated as the byte size. Any attempt to send
@@ -149,6 +154,7 @@ struct atmel_i2c_client_priv {
struct hwrng hwrng;
struct completion remove_done;
bool ready;
+ u32 caps;
};
/**
@@ -193,7 +199,7 @@ void atmel_i2c_init_genkey_cmd(struct atmel_i2c_cmd *cmd, u16 keyid);
int atmel_i2c_init_ecdh_cmd(struct atmel_i2c_cmd *cmd,
struct scatterlist *pubkey);
-struct i2c_client *atmel_i2c_client_alloc(void);
+struct i2c_client *atmel_i2c_client_alloc(enum atmel_i2c_capability cap);
void atmel_i2c_client_free(struct i2c_client *client);
void atmel_i2c_deactivate_client(struct atmel_i2c_client_priv *i2c_priv);
diff --git a/drivers/crypto/atmel-sha204a.c b/drivers/crypto/atmel-sha204a.c
index 33e5a66b843c..0c5b5cdbfcbc 100644
--- a/drivers/crypto/atmel-sha204a.c
+++ b/drivers/crypto/atmel-sha204a.c
@@ -178,6 +178,8 @@ static int atmel_sha204a_probe(struct i2c_client *client)
i2c_priv = i2c_get_clientdata(client);
+ i2c_priv->caps = 0;
+
memset(&i2c_priv->hwrng, 0, sizeof(i2c_priv->hwrng));
i2c_priv->hwrng.name = dev_name(&client->dev);
--
2.39.5
^ permalink raw reply related
* [PATCH v4 09/12] crypto: atmel-i2c - move shared client allocation logic to core
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Migrate the I2C client allocation and runtime load-balancing routines out
of the ECC driver code and into the central atmel-i2c core library module.
Export the symmetric lifecycle helper interfaces atmel_i2c_client_alloc()
and atmel_i2c_client_free() using EXPORT_SYMBOL_GPL() to expose a unified
client management API. This consolidation enables the dynamic selection
subsystem (which chooses the least-loaded client device based on the active
transformation count) to be shared by both the ECC driver and upcoming
Atmel crypto modules.
Refactor the ECC driver's transformation context initialization (init_tfm)
and teardown (exit_tfm) paths to use this centralized core API.
No functional change is intended.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-ecc.c | 51 +++-----------------------------------
drivers/crypto/atmel-i2c.c | 47 +++++++++++++++++++++++++++++++++++
drivers/crypto/atmel-i2c.h | 3 +++
3 files changed, 53 insertions(+), 48 deletions(-)
diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c
index d2490693a198..16e607cd06c4 100644
--- a/drivers/crypto/atmel-ecc.c
+++ b/drivers/crypto/atmel-ecc.c
@@ -205,51 +205,6 @@ static int atmel_ecdh_compute_shared_secret(struct kpp_request *req)
return ret;
}
-static struct i2c_client *atmel_ecc_i2c_client_alloc(void)
-{
- struct atmel_i2c_client_priv *i2c_priv, *min_i2c_priv = NULL;
- struct i2c_client *client = ERR_PTR(-ENODEV);
- int min_tfm_cnt = INT_MAX;
- int tfm_cnt;
-
- spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
-
- if (list_empty(&atmel_i2c_mgmt.i2c_client_list)) {
- spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
- return ERR_PTR(-ENODEV);
- }
-
- list_for_each_entry(i2c_priv, &atmel_i2c_mgmt.i2c_client_list,
- i2c_client_list_node) {
- if (!i2c_priv->ready)
- continue;
- tfm_cnt = atomic_read(&i2c_priv->tfm_count);
- if (tfm_cnt < min_tfm_cnt) {
- min_tfm_cnt = tfm_cnt;
- min_i2c_priv = i2c_priv;
- }
- if (!min_tfm_cnt)
- break;
- }
-
- if (min_i2c_priv) {
- atomic_inc(&min_i2c_priv->tfm_count);
- client = min_i2c_priv->client;
- }
-
- spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
-
- return client;
-}
-
-static void atmel_ecc_i2c_client_free(struct i2c_client *client)
-{
- struct atmel_i2c_client_priv *i2c_priv = i2c_get_clientdata(client);
-
- if (atomic_dec_and_test(&i2c_priv->tfm_count))
- complete(&i2c_priv->remove_done);
-}
-
static int atmel_ecdh_init_tfm(struct crypto_kpp *tfm)
{
const char *alg = kpp_alg_name(tfm);
@@ -257,7 +212,7 @@ static int atmel_ecdh_init_tfm(struct crypto_kpp *tfm)
struct atmel_ecdh_ctx *ctx = kpp_tfm_ctx(tfm);
ctx->curve_id = ECC_CURVE_NIST_P256;
- ctx->client = atmel_ecc_i2c_client_alloc();
+ ctx->client = atmel_i2c_client_alloc();
if (IS_ERR(ctx->client)) {
pr_err("tfm - i2c_client binding failed\n");
return PTR_ERR(ctx->client);
@@ -267,7 +222,7 @@ static int atmel_ecdh_init_tfm(struct crypto_kpp *tfm)
if (IS_ERR(fallback)) {
dev_err(&ctx->client->dev, "Failed to allocate transformation for '%s': %ld\n",
alg, PTR_ERR(fallback));
- atmel_ecc_i2c_client_free(ctx->client);
+ atmel_i2c_client_free(ctx->client);
return PTR_ERR(fallback);
}
@@ -284,7 +239,7 @@ static void atmel_ecdh_exit_tfm(struct crypto_kpp *tfm)
kfree(ctx->public_key);
if (ctx->fallback)
crypto_free_kpp(ctx->fallback);
- atmel_ecc_i2c_client_free(ctx->client);
+ atmel_i2c_client_free(ctx->client);
}
static unsigned int atmel_ecdh_max_size(struct crypto_kpp *tfm)
diff --git a/drivers/crypto/atmel-i2c.c b/drivers/crypto/atmel-i2c.c
index db818ce55033..92d3e28f9d9a 100644
--- a/drivers/crypto/atmel-i2c.c
+++ b/drivers/crypto/atmel-i2c.c
@@ -57,6 +57,53 @@ static void atmel_i2c_checksum(struct atmel_i2c_cmd *cmd)
*__crc16 = cpu_to_le16(bitrev16(crc16(0, data, len)));
}
+struct i2c_client *atmel_i2c_client_alloc(void)
+{
+ struct atmel_i2c_client_priv *i2c_priv, *min_i2c_priv = NULL;
+ struct i2c_client *client = ERR_PTR(-ENODEV);
+ int min_tfm_cnt = INT_MAX;
+ int tfm_cnt;
+
+ spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
+
+ if (list_empty(&atmel_i2c_mgmt.i2c_client_list)) {
+ spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+ return ERR_PTR(-ENODEV);
+ }
+
+ list_for_each_entry(i2c_priv, &atmel_i2c_mgmt.i2c_client_list,
+ i2c_client_list_node) {
+ if (!i2c_priv->ready)
+ continue;
+ tfm_cnt = atomic_read(&i2c_priv->tfm_count);
+ if (tfm_cnt < min_tfm_cnt) {
+ min_tfm_cnt = tfm_cnt;
+ min_i2c_priv = i2c_priv;
+ }
+ if (!min_tfm_cnt)
+ break;
+ }
+
+ if (min_i2c_priv) {
+ atomic_inc(&min_i2c_priv->tfm_count);
+ client = min_i2c_priv->client;
+ }
+
+ spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+
+ return client;
+}
+EXPORT_SYMBOL_GPL(atmel_i2c_client_alloc);
+
+void atmel_i2c_client_free(struct i2c_client *client)
+{
+ struct atmel_i2c_client_priv *i2c_priv = i2c_get_clientdata(client);
+
+ if (atomic_dec_and_test(&i2c_priv->tfm_count))
+ complete(&i2c_priv->remove_done);
+}
+EXPORT_SYMBOL_GPL(atmel_i2c_client_free);
+
void atmel_i2c_init_read_config_cmd(struct atmel_i2c_cmd *cmd)
{
cmd->word_addr = COMMAND;
diff --git a/drivers/crypto/atmel-i2c.h b/drivers/crypto/atmel-i2c.h
index 07fd2248e20b..ddab80bc1a72 100644
--- a/drivers/crypto/atmel-i2c.h
+++ b/drivers/crypto/atmel-i2c.h
@@ -193,6 +193,9 @@ void atmel_i2c_init_genkey_cmd(struct atmel_i2c_cmd *cmd, u16 keyid);
int atmel_i2c_init_ecdh_cmd(struct atmel_i2c_cmd *cmd,
struct scatterlist *pubkey);
+struct i2c_client *atmel_i2c_client_alloc(void);
+void atmel_i2c_client_free(struct i2c_client *client);
+
void atmel_i2c_deactivate_client(struct atmel_i2c_client_priv *i2c_priv);
void atmel_i2c_unregister_client(struct atmel_i2c_client_priv *i2c_priv);
--
2.39.5
^ permalink raw reply related
* [PATCH v4 07/12] crypto: atmel-i2c - introduce shared teardown helpers and fix queue flush
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Introduce atmel_i2c_deactivate_client() and atmel_i2c_unregister_client()
helpers in the atmel-i2c core library to modularize client teardown. This
encapsulates common client state tracking and list manipulation operations.
Convert the ECC driver's error recovery and device removal paths to utilize
these new helpers, ensuring consistent execution ordering when modifying
device-readiness states and deleting linked-list nodes.
Additionally, migrate the atmel_i2c_flush_queue() call out of the module
exit path. It now runs inside the core unregistration helper. Export both
new tracking symbols via EXPORT_SYMBOL_GPL() to match the existing core
driver licensing standard.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-ecc.c | 25 ++++++-------------------
drivers/crypto/atmel-i2c.c | 20 ++++++++++++++++++++
drivers/crypto/atmel-i2c.h | 3 +++
3 files changed, 29 insertions(+), 19 deletions(-)
diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c
index 1ae9c52812df..e6d3e6574251 100644
--- a/drivers/crypto/atmel-ecc.c
+++ b/drivers/crypto/atmel-ecc.c
@@ -351,12 +351,8 @@ static int atmel_ecc_probe(struct i2c_client *client)
msecs_to_jiffies(2000));
mutex_lock(&atmel_ecc_kpp_lock);
if (timeout == 0) {
- spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
- i2c_priv->ready = false;
- list_del(&i2c_priv->i2c_client_list_node);
- spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
- mutex_unlock(&atmel_ecc_kpp_lock);
-
+ atmel_i2c_deactivate_client(i2c_priv);
+ atmel_i2c_unregister_client(i2c_priv);
dev_err(&client->dev, "probe timed out, former driver instance not fully deregistered\n");
return -ETIMEDOUT;
}
@@ -365,12 +361,8 @@ static int atmel_ecc_probe(struct i2c_client *client)
if (atmel_ecc_kpp_refcnt == 0) {
ret = crypto_register_kpp(&atmel_ecdh_nist_p256);
if (ret) {
- spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
- i2c_priv->ready = false;
- list_del(&i2c_priv->i2c_client_list_node);
- spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
- mutex_unlock(&atmel_ecc_kpp_lock);
-
+ atmel_i2c_deactivate_client(i2c_priv);
+ atmel_i2c_unregister_client(i2c_priv);
dev_err(&client->dev, "%s alg registration failed\n",
atmel_ecdh_nist_p256.base.cra_driver_name);
return ret;
@@ -388,9 +380,7 @@ static void atmel_ecc_remove(struct i2c_client *client)
struct atmel_i2c_client_priv *i2c_priv = i2c_get_clientdata(client);
bool trigger_unreg = false;
- spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
- i2c_priv->ready = false;
- spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+ atmel_i2c_deactivate_client(i2c_priv);
/*
* The Linux crypto core automatically blocks until all active
@@ -410,9 +400,7 @@ static void atmel_ecc_remove(struct i2c_client *client)
if (atomic_read(&i2c_priv->tfm_count))
wait_for_completion(&i2c_priv->remove_done);
- spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
- list_del(&i2c_priv->i2c_client_list_node);
- spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+ atmel_i2c_unregister_client(i2c_priv);
/*
* The driver registers once an algorithm, but maintains a list of
@@ -461,7 +449,6 @@ static int __init atmel_ecc_init(void)
static void __exit atmel_ecc_exit(void)
{
- atmel_i2c_flush_queue();
i2c_del_driver(&atmel_ecc_driver);
}
diff --git a/drivers/crypto/atmel-i2c.c b/drivers/crypto/atmel-i2c.c
index a42b0ea30033..db818ce55033 100644
--- a/drivers/crypto/atmel-i2c.c
+++ b/drivers/crypto/atmel-i2c.c
@@ -354,6 +354,26 @@ static int device_sanity_check(struct i2c_client *client)
return ret;
}
+void atmel_i2c_deactivate_client(struct atmel_i2c_client_priv *i2c_priv)
+{
+ spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
+ i2c_priv->ready = false;
+ spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+}
+EXPORT_SYMBOL_GPL(atmel_i2c_deactivate_client);
+
+void atmel_i2c_unregister_client(struct atmel_i2c_client_priv *i2c_priv)
+{
+ spin_lock(&atmel_i2c_mgmt.i2c_list_lock);
+ if (!list_empty(&i2c_priv->i2c_client_list_node))
+ list_del_init(&i2c_priv->i2c_client_list_node);
+ spin_unlock(&atmel_i2c_mgmt.i2c_list_lock);
+
+ /* don't sleep inside spin locks */
+ atmel_i2c_flush_queue();
+}
+EXPORT_SYMBOL_GPL(atmel_i2c_unregister_client);
+
int atmel_i2c_probe(struct i2c_client *client)
{
struct atmel_i2c_client_priv *i2c_priv;
diff --git a/drivers/crypto/atmel-i2c.h b/drivers/crypto/atmel-i2c.h
index 82321c35c21f..07fd2248e20b 100644
--- a/drivers/crypto/atmel-i2c.h
+++ b/drivers/crypto/atmel-i2c.h
@@ -193,4 +193,7 @@ void atmel_i2c_init_genkey_cmd(struct atmel_i2c_cmd *cmd, u16 keyid);
int atmel_i2c_init_ecdh_cmd(struct atmel_i2c_cmd *cmd,
struct scatterlist *pubkey);
+void atmel_i2c_deactivate_client(struct atmel_i2c_client_priv *i2c_priv);
+void atmel_i2c_unregister_client(struct atmel_i2c_client_priv *i2c_priv);
+
#endif /* __ATMEL_I2C_H__ */
--
2.39.5
^ permalink raw reply related
* [PATCH v4 08/12] crypto: atmel-ecc - switch to module_i2c_driver
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Remove custom boilerplate module configuration code and convert the module
init/exit paths to use the modern module_i2c_driver() helper macro.
This shortens and simplifies driver initialization. Custom structure setup
is no longer required here since management tracking context initialization
was already safely moved into the atmel-i2c core library module.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-ecc.c | 13 +------------
1 file changed, 1 insertion(+), 12 deletions(-)
diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c
index e6d3e6574251..d2490693a198 100644
--- a/drivers/crypto/atmel-ecc.c
+++ b/drivers/crypto/atmel-ecc.c
@@ -442,18 +442,7 @@ static struct i2c_driver atmel_ecc_driver = {
.id_table = atmel_ecc_id,
};
-static int __init atmel_ecc_init(void)
-{
- return i2c_add_driver(&atmel_ecc_driver);
-}
-
-static void __exit atmel_ecc_exit(void)
-{
- i2c_del_driver(&atmel_ecc_driver);
-}
-
-module_init(atmel_ecc_init);
-module_exit(atmel_ecc_exit);
+module_i2c_driver(atmel_ecc_driver);
MODULE_AUTHOR("Tudor Ambarus");
MODULE_DESCRIPTION("Microchip / Atmel ECC (I2C) driver");
--
2.39.5
^ permalink raw reply related
* [PATCH v4 06/12] crypto: atmel-i2c - move client management instance into core
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Move the global 'atmel_i2c_mgmt' tracking instance out of the ECC driver
and into the atmel-i2c core library.
This change consolidates the shared I2C client infrastructure into a
central core driver. This centralization allows both the ECC and
upcoming SHA204A driver modules to access and reference a unified,
common device-management context.
As part of this relocation, replace the explicit runtime initialization
calls inside the module init block with static, compile-time macros
(__SPIN_LOCK_UNLOCKED and LIST_HEAD_INIT). Export the tracking structure
via EXPORT_SYMBOL_GPL() to make it available to dependent sub-modules.
No functional change intended.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-ecc.c | 4 ----
drivers/crypto/atmel-i2c.c | 6 ++++++
drivers/crypto/atmel-i2c.h | 1 +
3 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c
index d6ae113c45df..1ae9c52812df 100644
--- a/drivers/crypto/atmel-ecc.c
+++ b/drivers/crypto/atmel-ecc.c
@@ -28,8 +28,6 @@ static int atmel_ecc_kpp_refcnt;
DECLARE_COMPLETION(atmel_ecc_unreg_done);
static bool atmel_ecc_unreg_active;
-static struct atmel_i2c_client_mgmt atmel_i2c_mgmt;
-
/**
* struct atmel_ecdh_ctx - transformation context
* @client : pointer to i2c client device
@@ -458,8 +456,6 @@ static struct i2c_driver atmel_ecc_driver = {
static int __init atmel_ecc_init(void)
{
- spin_lock_init(&atmel_i2c_mgmt.i2c_list_lock);
- INIT_LIST_HEAD(&atmel_i2c_mgmt.i2c_client_list);
return i2c_add_driver(&atmel_ecc_driver);
}
diff --git a/drivers/crypto/atmel-i2c.c b/drivers/crypto/atmel-i2c.c
index ff19857894d0..a42b0ea30033 100644
--- a/drivers/crypto/atmel-i2c.c
+++ b/drivers/crypto/atmel-i2c.c
@@ -21,6 +21,12 @@
#include <linux/workqueue.h>
#include "atmel-i2c.h"
+struct atmel_i2c_client_mgmt atmel_i2c_mgmt = {
+ .i2c_list_lock = __SPIN_LOCK_UNLOCKED(atmel_i2c_mgmt.i2c_list_lock),
+ .i2c_client_list = LIST_HEAD_INIT(atmel_i2c_mgmt.i2c_client_list),
+};
+EXPORT_SYMBOL_GPL(atmel_i2c_mgmt);
+
static const struct {
u8 value;
const char *error_text;
diff --git a/drivers/crypto/atmel-i2c.h b/drivers/crypto/atmel-i2c.h
index 660ca861b705..82321c35c21f 100644
--- a/drivers/crypto/atmel-i2c.h
+++ b/drivers/crypto/atmel-i2c.h
@@ -119,6 +119,7 @@ struct atmel_i2c_client_mgmt {
struct list_head i2c_client_list;
spinlock_t i2c_list_lock;
} ____cacheline_aligned;
+extern struct atmel_i2c_client_mgmt atmel_i2c_mgmt;
/**
* atmel_i2c_client_priv - i2c_client private data
--
2.39.5
^ permalink raw reply related
* [PATCH v4 05/12] crypto: atmel - rename atmel_ecc_driver_data to atmel_i2c_client_mgmt
From: Lothar Rubusch @ 2026-05-22 23:01 UTC (permalink / raw)
To: thorsten.blum, herbert, davem, nicolas.ferre, alexandre.belloni,
claudiu.beznea, tudor.ambarus, ardb, linusw, krzk+dt
Cc: linux-crypto, linux-arm-kernel, linux-kernel, l.rubusch
In-Reply-To: <20260522230134.32414-1-l.rubusch@gmail.com>
Rename struct atmel_ecc_driver_data to atmel_i2c_client_mgmt to reflect its
generic role in shared I2C client tracking and locking. A subsequent change
will move the client management infrastructure into the atmel-i2c core
driver.
No functional changes intended.
Signed-off-by: Lothar Rubusch <l.rubusch@gmail.com>
---
drivers/crypto/atmel-ecc.c | 2 +-
drivers/crypto/atmel-i2c.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c
index d12a9dbe45a7..d6ae113c45df 100644
--- a/drivers/crypto/atmel-ecc.c
+++ b/drivers/crypto/atmel-ecc.c
@@ -28,7 +28,7 @@ static int atmel_ecc_kpp_refcnt;
DECLARE_COMPLETION(atmel_ecc_unreg_done);
static bool atmel_ecc_unreg_active;
-static struct atmel_ecc_driver_data atmel_i2c_mgmt;
+static struct atmel_i2c_client_mgmt atmel_i2c_mgmt;
/**
* struct atmel_ecdh_ctx - transformation context
diff --git a/drivers/crypto/atmel-i2c.h b/drivers/crypto/atmel-i2c.h
index b320559e50eb..660ca861b705 100644
--- a/drivers/crypto/atmel-i2c.h
+++ b/drivers/crypto/atmel-i2c.h
@@ -115,7 +115,7 @@ struct atmel_i2c_cmd {
#define ECDH_PREFIX_MODE 0x00
/* Used for binding tfm objects to i2c clients. */
-struct atmel_ecc_driver_data {
+struct atmel_i2c_client_mgmt {
struct list_head i2c_client_list;
spinlock_t i2c_list_lock;
} ____cacheline_aligned;
--
2.39.5
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox