* Re: [RFC PATCH 2/2] arm64: add support for AES using ARMv8 Crypto Extensions
From: Jussi Kivilinna @ 2013-09-14 14:11 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-arm, linux-crypto, Nicolas Pitre, Catalin Marinas,
Steve Capper
In-Reply-To: <CAKv+Gu9fCaaTVyCt-r-ijpqYDo4LTfvzrorzjWWCYMM17QUOQA@mail.gmail.com>
On 14.09.2013 16:30, Ard Biesheuvel wrote:
> On 14 September 2013 10:08, Jussi Kivilinna <jussi.kivilinna@iki.fi> wrote:
>> On 13.09.2013 18:08, Ard Biesheuvel wrote:
>>> This adds ARMv8 Crypto Extensions based implemenations of
>>> AES in CBC, CTR and XTS mode.
>>>
>>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> ---
>> ..snip..
>>> +static int xts_set_key(struct crypto_tfm *tfm, const u8 *in_key,
>>> + unsigned int key_len)
>>> +{
>>> + struct crypto_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
>>> + u32 *flags = &tfm->crt_flags;
>>> + int ret;
>>> +
>>> + ret = crypto_aes_expand_key(&ctx->key1, in_key, key_len/2);
>>> + if (!ret)
>>> + ret = crypto_aes_expand_key(&ctx->key2, &in_key[key_len/2],
>>> + key_len/2);
>>
>> Use checkpatch.
>>
>
> Um, I did get a bunch of errors and warnings from checkpatch.pl tbh,
> put not in this particular location. Care to elaborate?
>
Well, the checkpatch.pl I had stored to brain had become corrupted and kept
saying that you need spaces around all operators. But apparently spaces are
just required for assignment operators.
>>> +}, {
>>> + .cra_name = "__ctr-aes-aesce",
>>> + .cra_driver_name = "__driver-ctr-aes-aesce",
>>> + .cra_priority = 0,
>>> + .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
>>> + .cra_blocksize = AES_BLOCK_SIZE,
>>
>> CTR mode is stream cipher, cra_blocksize must be set to 1.
>>
>> This should have been picked up by in-kernel run-time tests, check
>> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS (and CONFIG_CRYPTO_TEST/tcrypt
>> module).
>>
>
> Well, run-time implies access to hardware :-) As I indicated in the
> cover letter, these bits are only compile tested.
Ok, I read the cover letter too hastily :)
-Jussi
^ permalink raw reply
* Re: [RFC PATCH 2/2] arm64: add support for AES using ARMv8 Crypto Extensions
From: Ard Biesheuvel @ 2013-09-14 13:30 UTC (permalink / raw)
To: Jussi Kivilinna
Cc: linux-arm, linux-crypto, Nicolas Pitre, Catalin Marinas,
Steve Capper
In-Reply-To: <5234196D.9050101@iki.fi>
On 14 September 2013 10:08, Jussi Kivilinna <jussi.kivilinna@iki.fi> wrote:
> On 13.09.2013 18:08, Ard Biesheuvel wrote:
>> This adds ARMv8 Crypto Extensions based implemenations of
>> AES in CBC, CTR and XTS mode.
>>
>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> ---
> ..snip..
>> +static int xts_set_key(struct crypto_tfm *tfm, const u8 *in_key,
>> + unsigned int key_len)
>> +{
>> + struct crypto_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
>> + u32 *flags = &tfm->crt_flags;
>> + int ret;
>> +
>> + ret = crypto_aes_expand_key(&ctx->key1, in_key, key_len/2);
>> + if (!ret)
>> + ret = crypto_aes_expand_key(&ctx->key2, &in_key[key_len/2],
>> + key_len/2);
>
> Use checkpatch.
>
Um, I did get a bunch of errors and warnings from checkpatch.pl tbh,
put not in this particular location. Care to elaborate?
>> + if (!ret)
>> + return 0;
>> +
>> + *flags |= CRYPTO_TFM_RES_BAD_KEY_LEN;
>> + return -EINVAL;
>> +}
>> +
>> +static int cbc_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
>> + struct scatterlist *src, unsigned int nbytes)
>> +{
>> + struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
>> + int err, first, rounds = 6 + ctx->key_length/4;
>> + struct blkcipher_walk walk;
>> + unsigned int blocks;
>> +
>> + blkcipher_walk_init(&walk, dst, src, nbytes);
>> + err = blkcipher_walk_virt(desc, &walk);
>> +
>> + kernel_neon_begin();
>
> Is sleeping allowed within kernel_neon_begin/end block? If not, you need to
> clear CRYPTO_TFM_REQ_MAY_SLEEP on desc->flags. Otherwise blkcipher_walk_done
> might sleep.
>
Good point. No, sleeping is not allowed in this case, so I should
clear the flag.
>> + for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) {
>> + aesce_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
>> + (u8*)ctx->key_enc, rounds, blocks, walk.iv,
>> + first);
>> +
>> + err = blkcipher_walk_done(desc, &walk, blocks * AES_BLOCK_SIZE);
>> + }
>> + kernel_neon_end();
>> +
>> + /* non-integral sizes are not supported in CBC */
>> + if (unlikely(walk.nbytes))
>> + err = -EINVAL;
>
> I think blkcipher_walk_done already does this check by comparing against
> alg.cra_blocksize.
>
You're right, it does. I will remove it.
>> +
>> + return err;
>> +}
> ..snip..
>> +
>> +static int ctr_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
>> + struct scatterlist *src, unsigned int nbytes)
>> +{
>> + struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
>> + int err, first, rounds = 6 + ctx->key_length/4;
>> + struct blkcipher_walk walk;
>> + u8 ctr[AES_BLOCK_SIZE];
>> +
>> + blkcipher_walk_init(&walk, dst, src, nbytes);
>> + err = blkcipher_walk_virt(desc, &walk);
>> +
>> + memcpy(ctr, walk.iv, AES_BLOCK_SIZE);
>> +
>> + kernel_neon_begin();
>> + for (first = 1; (nbytes = walk.nbytes); first = 0) {
>> + aesce_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
>> + (u8*)ctx->key_enc, rounds, nbytes, ctr, first);
>> +
>> + err = blkcipher_walk_done(desc, &walk, 0);
>> +
>> + /* non-integral block *must* be the last one */
>> + if (unlikely(walk.nbytes && (nbytes & (AES_BLOCK_SIZE-1)))) {
>> + err = -EINVAL;
>
> Other CTR implementations do not have this.. not needed?
>
I should be using blkcipher_walk_virt_block() here with a block size
of AES_BLOCK_SIZE. In that case, all calls but the last one will be at
least AES_BLOCK_SIZE long. But that still means I need to take care to
only consume multiples of AES_BLOCK_SIZE in all iterations except the
last one.
>> + break;
>> + }
>> + }
> ..snip..
>> +static struct crypto_alg aesce_cbc_algs[] = { {
>> + .cra_name = "__cbc-aes-aesce",
>> + .cra_driver_name = "__driver-cbc-aes-aesce",
>> + .cra_priority = 0,
>> + .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
>> + .cra_blocksize = AES_BLOCK_SIZE,
>> + .cra_ctxsize = sizeof(struct crypto_aes_ctx),
>> + .cra_alignmask = 0,
>> + .cra_type = &crypto_blkcipher_type,
>> + .cra_module = THIS_MODULE,
>> + .cra_u = {
>> + .blkcipher = {
>> + .min_keysize = AES_MIN_KEY_SIZE,
>> + .max_keysize = AES_MAX_KEY_SIZE,
>> + .ivsize = AES_BLOCK_SIZE,
>> + .setkey = crypto_aes_set_key,
>> + .encrypt = cbc_encrypt,
>> + .decrypt = cbc_decrypt,
>> + },
>> + },
>> +}, {
>> + .cra_name = "__ctr-aes-aesce",
>> + .cra_driver_name = "__driver-ctr-aes-aesce",
>> + .cra_priority = 0,
>> + .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
>> + .cra_blocksize = AES_BLOCK_SIZE,
>
> CTR mode is stream cipher, cra_blocksize must be set to 1.
>
> This should have been picked up by in-kernel run-time tests, check
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS (and CONFIG_CRYPTO_TEST/tcrypt
> module).
>
Well, run-time implies access to hardware :-) As I indicated in the
cover letter, these bits are only compile tested.
[...]
>> +.Lencsteal:
>> + ldrb w6, [x1], #1
>> + ldrb w7, [x2, #-16]
>> + strb w6, [x2, #-16]
>> + strb w7, [x2], #1
>> + subs x5, x5, #1
>> + bne .Lencsteal
>
> Cipher text stealing here is dead-code, since alg.cra_blocksize is set
> to 16 bytes.
>
> Currently other XTS implementations do not have CTS either so this is
> not really needed anyway atm (crypto/xts.c: "sector sizes which are not
> a multiple of 16 bytes are, however currently unsupported").
>
I'll remove it then, if we can't test it anyway.
Cheers,
Ard.
^ permalink raw reply
* [PATCH] crypto: move ablk_helper out of arch/x86
From: Ard Biesheuvel @ 2013-09-14 10:24 UTC (permalink / raw)
To: linux-crypto, x86, linux-arch
Cc: herbert, mingo, arnd, Ard Biesheuvel, jussi.kivilinna
Move the ablk_helper code out of arch/x86 so it can be reused
by other architectures. The only x86 specific dependency is
a call to irq_fpu_usable(), in the generic case we use
!in_interrupt() instead.
Cc: jussi.kivilinna@iki.fi
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
Any need to split this up between generic/crypto and x86?
arch/x86/crypto/Makefile | 1 -
arch/x86/crypto/ablk_helper.c | 149 ----------------------------
arch/x86/crypto/aesni-intel_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx2_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx_glue.c | 2 +-
arch/x86/crypto/cast5_avx_glue.c | 2 +-
arch/x86/crypto/cast6_avx_glue.c | 2 +-
arch/x86/crypto/serpent_avx2_glue.c | 2 +-
arch/x86/crypto/serpent_avx_glue.c | 2 +-
arch/x86/crypto/serpent_sse2_glue.c | 2 +-
arch/x86/crypto/twofish_avx_glue.c | 2 +-
arch/x86/include/asm/crypto/ablk_helper.h | 38 ++------
crypto/Kconfig | 23 +++--
crypto/Makefile | 1 +
crypto/ablk_helper.c | 150 +++++++++++++++++++++++++++++
include/asm-generic/crypto/ablk_helper.h | 13 +++
include/crypto/ablk_helper.h | 31 ++++++
17 files changed, 224 insertions(+), 200 deletions(-)
delete mode 100644 arch/x86/crypto/ablk_helper.c
create mode 100644 crypto/ablk_helper.c
create mode 100644 include/asm-generic/crypto/ablk_helper.h
create mode 100644 include/crypto/ablk_helper.h
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 7d6ba9d..18fda50 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -4,7 +4,6 @@
avx_supported := $(call as-instr,vpxor %xmm0$(comma)%xmm0$(comma)%xmm0,yes,no)
-obj-$(CONFIG_CRYPTO_ABLK_HELPER_X86) += ablk_helper.o
obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o
obj-$(CONFIG_CRYPTO_AES_586) += aes-i586.o
diff --git a/arch/x86/crypto/ablk_helper.c b/arch/x86/crypto/ablk_helper.c
deleted file mode 100644
index 43282fe..0000000
--- a/arch/x86/crypto/ablk_helper.c
+++ /dev/null
@@ -1,149 +0,0 @@
-/*
- * Shared async block cipher helpers
- *
- * Copyright (c) 2012 Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
- *
- * Based on aesni-intel_glue.c by:
- * Copyright (C) 2008, Intel Corp.
- * Author: Huang Ying <ying.huang@intel.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
- * USA
- *
- */
-
-#include <linux/kernel.h>
-#include <linux/crypto.h>
-#include <linux/init.h>
-#include <linux/module.h>
-#include <crypto/algapi.h>
-#include <crypto/cryptd.h>
-#include <asm/i387.h>
-#include <asm/crypto/ablk_helper.h>
-
-int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
- unsigned int key_len)
-{
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- struct crypto_ablkcipher *child = &ctx->cryptd_tfm->base;
- int err;
-
- crypto_ablkcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
- crypto_ablkcipher_set_flags(child, crypto_ablkcipher_get_flags(tfm)
- & CRYPTO_TFM_REQ_MASK);
- err = crypto_ablkcipher_setkey(child, key, key_len);
- crypto_ablkcipher_set_flags(tfm, crypto_ablkcipher_get_flags(child)
- & CRYPTO_TFM_RES_MASK);
- return err;
-}
-EXPORT_SYMBOL_GPL(ablk_set_key);
-
-int __ablk_encrypt(struct ablkcipher_request *req)
-{
- struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- struct blkcipher_desc desc;
-
- desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
- desc.info = req->info;
- desc.flags = 0;
-
- return crypto_blkcipher_crt(desc.tfm)->encrypt(
- &desc, req->dst, req->src, req->nbytes);
-}
-EXPORT_SYMBOL_GPL(__ablk_encrypt);
-
-int ablk_encrypt(struct ablkcipher_request *req)
-{
- struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
-
- if (!irq_fpu_usable()) {
- struct ablkcipher_request *cryptd_req =
- ablkcipher_request_ctx(req);
-
- memcpy(cryptd_req, req, sizeof(*req));
- ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
-
- return crypto_ablkcipher_encrypt(cryptd_req);
- } else {
- return __ablk_encrypt(req);
- }
-}
-EXPORT_SYMBOL_GPL(ablk_encrypt);
-
-int ablk_decrypt(struct ablkcipher_request *req)
-{
- struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
-
- if (!irq_fpu_usable()) {
- struct ablkcipher_request *cryptd_req =
- ablkcipher_request_ctx(req);
-
- memcpy(cryptd_req, req, sizeof(*req));
- ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
-
- return crypto_ablkcipher_decrypt(cryptd_req);
- } else {
- struct blkcipher_desc desc;
-
- desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
- desc.info = req->info;
- desc.flags = 0;
-
- return crypto_blkcipher_crt(desc.tfm)->decrypt(
- &desc, req->dst, req->src, req->nbytes);
- }
-}
-EXPORT_SYMBOL_GPL(ablk_decrypt);
-
-void ablk_exit(struct crypto_tfm *tfm)
-{
- struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
-
- cryptd_free_ablkcipher(ctx->cryptd_tfm);
-}
-EXPORT_SYMBOL_GPL(ablk_exit);
-
-int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name)
-{
- struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
- struct cryptd_ablkcipher *cryptd_tfm;
-
- cryptd_tfm = cryptd_alloc_ablkcipher(drv_name, 0, 0);
- if (IS_ERR(cryptd_tfm))
- return PTR_ERR(cryptd_tfm);
-
- ctx->cryptd_tfm = cryptd_tfm;
- tfm->crt_ablkcipher.reqsize = sizeof(struct ablkcipher_request) +
- crypto_ablkcipher_reqsize(&cryptd_tfm->base);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(ablk_init_common);
-
-int ablk_init(struct crypto_tfm *tfm)
-{
- char drv_name[CRYPTO_MAX_ALG_NAME];
-
- snprintf(drv_name, sizeof(drv_name), "__driver-%s",
- crypto_tfm_alg_driver_name(tfm));
-
- return ablk_init_common(tfm, drv_name);
-}
-EXPORT_SYMBOL_GPL(ablk_init);
-
-MODULE_LICENSE("GPL");
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index f80e668..835488b 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -34,7 +34,7 @@
#include <asm/cpu_device_id.h>
#include <asm/i387.h>
#include <asm/crypto/aes.h>
-#include <asm/crypto/ablk_helper.h>
+#include <crypto/ablk_helper.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/aead.h>
#include <linux/workqueue.h>
diff --git a/arch/x86/crypto/camellia_aesni_avx2_glue.c b/arch/x86/crypto/camellia_aesni_avx2_glue.c
index 414fe5d..4209a76 100644
--- a/arch/x86/crypto/camellia_aesni_avx2_glue.c
+++ b/arch/x86/crypto/camellia_aesni_avx2_glue.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/ctr.h>
#include <crypto/lrw.h>
@@ -21,7 +22,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/camellia.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAMELLIA_AESNI_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/camellia_aesni_avx_glue.c b/arch/x86/crypto/camellia_aesni_avx_glue.c
index 37fd0c0..87a041a 100644
--- a/arch/x86/crypto/camellia_aesni_avx_glue.c
+++ b/arch/x86/crypto/camellia_aesni_avx_glue.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/ctr.h>
#include <crypto/lrw.h>
@@ -21,7 +22,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/camellia.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAMELLIA_AESNI_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/cast5_avx_glue.c b/arch/x86/crypto/cast5_avx_glue.c
index c663181..e6a3700 100644
--- a/arch/x86/crypto/cast5_avx_glue.c
+++ b/arch/x86/crypto/cast5_avx_glue.c
@@ -26,13 +26,13 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/cast5.h>
#include <crypto/cryptd.h>
#include <crypto/ctr.h>
#include <asm/xcr.h>
#include <asm/xsave.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAST5_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/cast6_avx_glue.c b/arch/x86/crypto/cast6_avx_glue.c
index 8d0dfb8..09f3677 100644
--- a/arch/x86/crypto/cast6_avx_glue.c
+++ b/arch/x86/crypto/cast6_avx_glue.c
@@ -28,6 +28,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/cast6.h>
#include <crypto/cryptd.h>
@@ -37,7 +38,6 @@
#include <crypto/xts.h>
#include <asm/xcr.h>
#include <asm/xsave.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAST6_PARALLEL_BLOCKS 8
diff --git a/arch/x86/crypto/serpent_avx2_glue.c b/arch/x86/crypto/serpent_avx2_glue.c
index 23aabc6..2fae489 100644
--- a/arch/x86/crypto/serpent_avx2_glue.c
+++ b/arch/x86/crypto/serpent_avx2_glue.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/ctr.h>
#include <crypto/lrw.h>
@@ -22,7 +23,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/serpent-avx.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define SERPENT_AVX2_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/serpent_avx_glue.c b/arch/x86/crypto/serpent_avx_glue.c
index 9ae83cf..ff48708 100644
--- a/arch/x86/crypto/serpent_avx_glue.c
+++ b/arch/x86/crypto/serpent_avx_glue.c
@@ -28,6 +28,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/serpent.h>
#include <crypto/cryptd.h>
@@ -38,7 +39,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/serpent-avx.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
/* 8-way parallel cipher functions */
diff --git a/arch/x86/crypto/serpent_sse2_glue.c b/arch/x86/crypto/serpent_sse2_glue.c
index 97a356e..8c95f86 100644
--- a/arch/x86/crypto/serpent_sse2_glue.c
+++ b/arch/x86/crypto/serpent_sse2_glue.c
@@ -34,6 +34,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/serpent.h>
#include <crypto/cryptd.h>
@@ -42,7 +43,6 @@
#include <crypto/lrw.h>
#include <crypto/xts.h>
#include <asm/crypto/serpent-sse2.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
static void serpent_decrypt_cbc_xway(void *ctx, u128 *dst, const u128 *src)
diff --git a/arch/x86/crypto/twofish_avx_glue.c b/arch/x86/crypto/twofish_avx_glue.c
index a62ba54..4e3c665 100644
--- a/arch/x86/crypto/twofish_avx_glue.c
+++ b/arch/x86/crypto/twofish_avx_glue.c
@@ -28,6 +28,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/twofish.h>
#include <crypto/cryptd.h>
@@ -39,7 +40,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/twofish.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#include <crypto/scatterwalk.h>
#include <linux/workqueue.h>
diff --git a/arch/x86/include/asm/crypto/ablk_helper.h b/arch/x86/include/asm/crypto/ablk_helper.h
index 4f93df5..0f68111 100644
--- a/arch/x86/include/asm/crypto/ablk_helper.h
+++ b/arch/x86/include/asm/crypto/ablk_helper.h
@@ -1,31 +1,11 @@
-/*
- * Shared async block cipher helpers
- */
-
-#ifndef _CRYPTO_ABLK_HELPER_H
-#define _CRYPTO_ABLK_HELPER_H
-
-#include <linux/crypto.h>
-#include <linux/kernel.h>
-#include <crypto/cryptd.h>
-
-struct async_helper_ctx {
- struct cryptd_ablkcipher *cryptd_tfm;
-};
-
-extern int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
- unsigned int key_len);
-
-extern int __ablk_encrypt(struct ablkcipher_request *req);
-extern int ablk_encrypt(struct ablkcipher_request *req);
+#include <asm/i387.h>
-extern int ablk_decrypt(struct ablkcipher_request *req);
-
-extern void ablk_exit(struct crypto_tfm *tfm);
-
-extern int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name);
-
-extern int ablk_init(struct crypto_tfm *tfm);
-
-#endif /* _CRYPTO_ABLK_HELPER_H */
+/*
+ * ablk_can_run_sync - used by crypto/ablk_helper to decide whether a request
+ * can be handled synchronously or needs to be queued up
+ */
+static inline int ablk_can_run_sync(void)
+{
+ return irq_fpu_usable();
+}
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 69ce573..7c0a4c5 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -174,9 +174,8 @@ config CRYPTO_TEST
help
Quick & dirty crypto test module.
-config CRYPTO_ABLK_HELPER_X86
+config CRYPTO_ABLK_HELPER
tristate
- depends on X86
select CRYPTO_CRYPTD
config CRYPTO_GLUE_HELPER_X86
@@ -695,7 +694,7 @@ config CRYPTO_AES_NI_INTEL
select CRYPTO_AES_X86_64 if 64BIT
select CRYPTO_AES_586 if !64BIT
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_ALGAPI
select CRYPTO_GLUE_HELPER_X86 if 64BIT
select CRYPTO_LRW
@@ -879,7 +878,7 @@ config CRYPTO_CAMELLIA_AESNI_AVX_X86_64
depends on CRYPTO
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_CAMELLIA_X86_64
select CRYPTO_LRW
@@ -901,7 +900,7 @@ config CRYPTO_CAMELLIA_AESNI_AVX2_X86_64
depends on CRYPTO
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_CAMELLIA_X86_64
select CRYPTO_CAMELLIA_AESNI_AVX_X86_64
@@ -953,7 +952,7 @@ config CRYPTO_CAST5_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_CAST_COMMON
select CRYPTO_CAST5
help
@@ -976,7 +975,7 @@ config CRYPTO_CAST6_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_CAST_COMMON
select CRYPTO_CAST6
@@ -1094,7 +1093,7 @@ config CRYPTO_SERPENT_SSE2_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_LRW
@@ -1116,7 +1115,7 @@ config CRYPTO_SERPENT_SSE2_586
depends on X86 && !64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_LRW
@@ -1138,7 +1137,7 @@ config CRYPTO_SERPENT_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_LRW
@@ -1160,7 +1159,7 @@ config CRYPTO_SERPENT_AVX2_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_SERPENT_AVX_X86_64
@@ -1276,7 +1275,7 @@ config CRYPTO_TWOFISH_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_TWOFISH_COMMON
select CRYPTO_TWOFISH_X86_64
diff --git a/crypto/Makefile b/crypto/Makefile
index 2d5ed08..580af97 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -104,3 +104,4 @@ obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o
obj-$(CONFIG_XOR_BLOCKS) += xor.o
obj-$(CONFIG_ASYNC_CORE) += async_tx/
obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/
+obj-$(CONFIG_CRYPTO_ABLK_HELPER) += ablk_helper.o
diff --git a/crypto/ablk_helper.c b/crypto/ablk_helper.c
new file mode 100644
index 0000000..63ee760
--- /dev/null
+++ b/crypto/ablk_helper.c
@@ -0,0 +1,150 @@
+/*
+ * Shared async block cipher helpers
+ *
+ * Copyright (c) 2012 Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
+ *
+ * Based on aesni-intel_glue.c by:
+ * Copyright (C) 2008, Intel Corp.
+ * Author: Huang Ying <ying.huang@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+ * USA
+ *
+ */
+
+#include <linux/kernel.h>
+#include <linux/crypto.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/hardirq.h>
+#include <crypto/algapi.h>
+#include <crypto/cryptd.h>
+#include <crypto/ablk_helper.h>
+#include <asm/crypto/ablk_helper.h>
+
+int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
+ unsigned int key_len)
+{
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+ struct crypto_ablkcipher *child = &ctx->cryptd_tfm->base;
+ int err;
+
+ crypto_ablkcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+ crypto_ablkcipher_set_flags(child, crypto_ablkcipher_get_flags(tfm)
+ & CRYPTO_TFM_REQ_MASK);
+ err = crypto_ablkcipher_setkey(child, key, key_len);
+ crypto_ablkcipher_set_flags(tfm, crypto_ablkcipher_get_flags(child)
+ & CRYPTO_TFM_RES_MASK);
+ return err;
+}
+EXPORT_SYMBOL_GPL(ablk_set_key);
+
+int __ablk_encrypt(struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+ struct blkcipher_desc desc;
+
+ desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
+ desc.info = req->info;
+ desc.flags = 0;
+
+ return crypto_blkcipher_crt(desc.tfm)->encrypt(
+ &desc, req->dst, req->src, req->nbytes);
+}
+EXPORT_SYMBOL_GPL(__ablk_encrypt);
+
+int ablk_encrypt(struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+
+ if (!ablk_can_run_sync()) {
+ struct ablkcipher_request *cryptd_req =
+ ablkcipher_request_ctx(req);
+
+ memcpy(cryptd_req, req, sizeof(*req));
+ ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
+
+ return crypto_ablkcipher_encrypt(cryptd_req);
+ } else {
+ return __ablk_encrypt(req);
+ }
+}
+EXPORT_SYMBOL_GPL(ablk_encrypt);
+
+int ablk_decrypt(struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+
+ if (!ablk_can_run_sync()) {
+ struct ablkcipher_request *cryptd_req =
+ ablkcipher_request_ctx(req);
+
+ memcpy(cryptd_req, req, sizeof(*req));
+ ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
+
+ return crypto_ablkcipher_decrypt(cryptd_req);
+ } else {
+ struct blkcipher_desc desc;
+
+ desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
+ desc.info = req->info;
+ desc.flags = 0;
+
+ return crypto_blkcipher_crt(desc.tfm)->decrypt(
+ &desc, req->dst, req->src, req->nbytes);
+ }
+}
+EXPORT_SYMBOL_GPL(ablk_decrypt);
+
+void ablk_exit(struct crypto_tfm *tfm)
+{
+ struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ cryptd_free_ablkcipher(ctx->cryptd_tfm);
+}
+EXPORT_SYMBOL_GPL(ablk_exit);
+
+int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name)
+{
+ struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct cryptd_ablkcipher *cryptd_tfm;
+
+ cryptd_tfm = cryptd_alloc_ablkcipher(drv_name, 0, 0);
+ if (IS_ERR(cryptd_tfm))
+ return PTR_ERR(cryptd_tfm);
+
+ ctx->cryptd_tfm = cryptd_tfm;
+ tfm->crt_ablkcipher.reqsize = sizeof(struct ablkcipher_request) +
+ crypto_ablkcipher_reqsize(&cryptd_tfm->base);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ablk_init_common);
+
+int ablk_init(struct crypto_tfm *tfm)
+{
+ char drv_name[CRYPTO_MAX_ALG_NAME];
+
+ snprintf(drv_name, sizeof(drv_name), "__driver-%s",
+ crypto_tfm_alg_driver_name(tfm));
+
+ return ablk_init_common(tfm, drv_name);
+}
+EXPORT_SYMBOL_GPL(ablk_init);
+
+MODULE_LICENSE("GPL");
diff --git a/include/asm-generic/crypto/ablk_helper.h b/include/asm-generic/crypto/ablk_helper.h
new file mode 100644
index 0000000..ede807f
--- /dev/null
+++ b/include/asm-generic/crypto/ablk_helper.h
@@ -0,0 +1,13 @@
+
+#include <linux/hardirq.h>
+
+/*
+ * ablk_can_run_sync - used by crypto/ablk_helper to decide whether a request
+ * can be handled synchronously or needs to be queued up.
+ *
+ * Choose in_interrupt() as a reasonable default
+ */
+static inline int ablk_can_run_sync(void)
+{
+ return !in_interrupt();
+}
diff --git a/include/crypto/ablk_helper.h b/include/crypto/ablk_helper.h
new file mode 100644
index 0000000..4f93df5
--- /dev/null
+++ b/include/crypto/ablk_helper.h
@@ -0,0 +1,31 @@
+/*
+ * Shared async block cipher helpers
+ */
+
+#ifndef _CRYPTO_ABLK_HELPER_H
+#define _CRYPTO_ABLK_HELPER_H
+
+#include <linux/crypto.h>
+#include <linux/kernel.h>
+#include <crypto/cryptd.h>
+
+struct async_helper_ctx {
+ struct cryptd_ablkcipher *cryptd_tfm;
+};
+
+extern int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
+ unsigned int key_len);
+
+extern int __ablk_encrypt(struct ablkcipher_request *req);
+
+extern int ablk_encrypt(struct ablkcipher_request *req);
+
+extern int ablk_decrypt(struct ablkcipher_request *req);
+
+extern void ablk_exit(struct crypto_tfm *tfm);
+
+extern int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name);
+
+extern int ablk_init(struct crypto_tfm *tfm);
+
+#endif /* _CRYPTO_ABLK_HELPER_H */
--
1.8.1.2
^ permalink raw reply related
* Re: [RFC PATCH 2/2] arm64: add support for AES using ARMv8 Crypto Extensions
From: Jussi Kivilinna @ 2013-09-14 8:08 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-arm, linux-crypto, nico, catalin.marinas, steve.capper
In-Reply-To: <1379084886-1178-3-git-send-email-ard.biesheuvel@linaro.org>
On 13.09.2013 18:08, Ard Biesheuvel wrote:
> This adds ARMv8 Crypto Extensions based implemenations of
> AES in CBC, CTR and XTS mode.
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
..snip..
> +static int xts_set_key(struct crypto_tfm *tfm, const u8 *in_key,
> + unsigned int key_len)
> +{
> + struct crypto_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
> + u32 *flags = &tfm->crt_flags;
> + int ret;
> +
> + ret = crypto_aes_expand_key(&ctx->key1, in_key, key_len/2);
> + if (!ret)
> + ret = crypto_aes_expand_key(&ctx->key2, &in_key[key_len/2],
> + key_len/2);
Use checkpatch.
> + if (!ret)
> + return 0;
> +
> + *flags |= CRYPTO_TFM_RES_BAD_KEY_LEN;
> + return -EINVAL;
> +}
> +
> +static int cbc_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
> + struct scatterlist *src, unsigned int nbytes)
> +{
> + struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
> + int err, first, rounds = 6 + ctx->key_length/4;
> + struct blkcipher_walk walk;
> + unsigned int blocks;
> +
> + blkcipher_walk_init(&walk, dst, src, nbytes);
> + err = blkcipher_walk_virt(desc, &walk);
> +
> + kernel_neon_begin();
Is sleeping allowed within kernel_neon_begin/end block? If not, you need to
clear CRYPTO_TFM_REQ_MAY_SLEEP on desc->flags. Otherwise blkcipher_walk_done
might sleep.
> + for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) {
> + aesce_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
> + (u8*)ctx->key_enc, rounds, blocks, walk.iv,
> + first);
> +
> + err = blkcipher_walk_done(desc, &walk, blocks * AES_BLOCK_SIZE);
> + }
> + kernel_neon_end();
> +
> + /* non-integral sizes are not supported in CBC */
> + if (unlikely(walk.nbytes))
> + err = -EINVAL;
I think blkcipher_walk_done already does this check by comparing against
alg.cra_blocksize.
> +
> + return err;
> +}
..snip..
> +
> +static int ctr_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
> + struct scatterlist *src, unsigned int nbytes)
> +{
> + struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
> + int err, first, rounds = 6 + ctx->key_length/4;
> + struct blkcipher_walk walk;
> + u8 ctr[AES_BLOCK_SIZE];
> +
> + blkcipher_walk_init(&walk, dst, src, nbytes);
> + err = blkcipher_walk_virt(desc, &walk);
> +
> + memcpy(ctr, walk.iv, AES_BLOCK_SIZE);
> +
> + kernel_neon_begin();
> + for (first = 1; (nbytes = walk.nbytes); first = 0) {
> + aesce_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
> + (u8*)ctx->key_enc, rounds, nbytes, ctr, first);
> +
> + err = blkcipher_walk_done(desc, &walk, 0);
> +
> + /* non-integral block *must* be the last one */
> + if (unlikely(walk.nbytes && (nbytes & (AES_BLOCK_SIZE-1)))) {
> + err = -EINVAL;
Other CTR implementations do not have this.. not needed?
> + break;
> + }
> + }
..snip..
> +static struct crypto_alg aesce_cbc_algs[] = { {
> + .cra_name = "__cbc-aes-aesce",
> + .cra_driver_name = "__driver-cbc-aes-aesce",
> + .cra_priority = 0,
> + .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
> + .cra_blocksize = AES_BLOCK_SIZE,
> + .cra_ctxsize = sizeof(struct crypto_aes_ctx),
> + .cra_alignmask = 0,
> + .cra_type = &crypto_blkcipher_type,
> + .cra_module = THIS_MODULE,
> + .cra_u = {
> + .blkcipher = {
> + .min_keysize = AES_MIN_KEY_SIZE,
> + .max_keysize = AES_MAX_KEY_SIZE,
> + .ivsize = AES_BLOCK_SIZE,
> + .setkey = crypto_aes_set_key,
> + .encrypt = cbc_encrypt,
> + .decrypt = cbc_decrypt,
> + },
> + },
> +}, {
> + .cra_name = "__ctr-aes-aesce",
> + .cra_driver_name = "__driver-ctr-aes-aesce",
> + .cra_priority = 0,
> + .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
> + .cra_blocksize = AES_BLOCK_SIZE,
CTR mode is stream cipher, cra_blocksize must be set to 1.
This should have been picked up by in-kernel run-time tests, check
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS (and CONFIG_CRYPTO_TEST/tcrypt
module).
> + .cra_ctxsize = sizeof(struct crypto_aes_ctx),
> + .cra_alignmask = 0,
> + .cra_type = &crypto_blkcipher_type,
> + .cra_module = THIS_MODULE,
> + .cra_u = {
> + .blkcipher = {
> + .min_keysize = AES_MIN_KEY_SIZE,
> + .max_keysize = AES_MAX_KEY_SIZE,
> + .ivsize = AES_BLOCK_SIZE,
> + .setkey = crypto_aes_set_key,
> + .encrypt = ctr_encrypt,
> + .decrypt = ctr_encrypt,
> + },
> + },
> +}, {
> + .cra_name = "__xts-aes-aesce",
> + .cra_driver_name = "__driver-xts-aes-aesce",
> + .cra_priority = 0,
> + .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
> + .cra_blocksize = AES_BLOCK_SIZE,
> + .cra_ctxsize = sizeof(struct crypto_aes_xts_ctx),
> + .cra_alignmask = 0,
> + .cra_type = &crypto_blkcipher_type,
> + .cra_module = THIS_MODULE,
> + .cra_u = {
> + .blkcipher = {
> + .min_keysize = 2*AES_MIN_KEY_SIZE,
> + .max_keysize = 2*AES_MAX_KEY_SIZE,
> + .ivsize = AES_BLOCK_SIZE,
> + .setkey = xts_set_key,
> + .encrypt = xts_encrypt,
> + .decrypt = xts_decrypt,
> + },
> + },
> +}, {
> + .cra_name = "cbc(aes)",
> + .cra_driver_name = "cbc-aes-aesce",
> + .cra_priority = 200,
> + .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER|CRYPTO_ALG_ASYNC,
> + .cra_blocksize = AES_BLOCK_SIZE,
> + .cra_ctxsize = sizeof(struct async_helper_ctx),
> + .cra_alignmask = 0,
> + .cra_type = &crypto_ablkcipher_type,
> + .cra_module = THIS_MODULE,
> + .cra_init = ablk_init,
> + .cra_exit = ablk_exit,
> + .cra_u = {
> + .ablkcipher = {
> + .min_keysize = AES_MIN_KEY_SIZE,
> + .max_keysize = AES_MAX_KEY_SIZE,
> + .ivsize = AES_BLOCK_SIZE,
> + .setkey = ablk_set_key,
> + .encrypt = ablk_encrypt,
> + .decrypt = ablk_decrypt,
> + }
> + }
> +}, {
> + .cra_name = "ctr(aes)",
> + .cra_driver_name = "ctr-aes-aesce",
> + .cra_priority = 200,
> + .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER|CRYPTO_ALG_ASYNC,
> + .cra_blocksize = AES_BLOCK_SIZE,
cra_blocksize = 1
> + .cra_ctxsize = sizeof(struct async_helper_ctx),
> + .cra_alignmask = 0,
..snip..
> +ENTRY(aesce_xts_encrypt)
> + tst w7, #1 // first call?
> + beq .Lencmore
> +
> + ld1 {v0.16b}, [x6]
> + load_round_keys w4, x2
> + encrypt_block v3.16b, v0.16b, w4 // first tweak
> + load_round_keys w4, x3
> + ldr q4, .Lxts_mul_x
> + b .Lencfirst
> +.Lencmore:
> + NEXT_TWEAK (v3, v4, v8)
> +.Lencfirst:
> + subs x5, x5, #16
> +.Lencloop:
> + ld1 {v1.16b}, [x1], #16
> + eor v0.16b, v1.16b, v3.16b
> + encrypt_block v2.16b, v0.16b, w4
> + eor v2.16b, v2.16b, v3.16b
> + st1 {v2.16b}, [x0], #16
> + beq .Lencout
> +
> + NEXT_TWEAK (v3, v4, v8)
> + subs x5, x5, #16
> + bpl .Lencloop
> +
> + sub x0, x0, #16
> + add x5, x5, #16
> + mov x2, x0
> +.Lencsteal:
> + ldrb w6, [x1], #1
> + ldrb w7, [x2, #-16]
> + strb w6, [x2, #-16]
> + strb w7, [x2], #1
> + subs x5, x5, #1
> + bne .Lencsteal
Cipher text stealing here is dead-code, since alg.cra_blocksize is set
to 16 bytes.
Currently other XTS implementations do not have CTS either so this is
not really needed anyway atm (crypto/xts.c: "sector sizes which are not
a multiple of 16 bytes are, however currently unsupported").
-Jussi
^ permalink raw reply
* Re: [RFC PATCH 1/2] crypto: move ablk_helper out of arch/x86
From: Jussi Kivilinna @ 2013-09-14 7:26 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-arm, linux-crypto, nico, catalin.marinas, steve.capper
In-Reply-To: <1379084886-1178-2-git-send-email-ard.biesheuvel@linaro.org>
On 13.09.2013 18:08, Ard Biesheuvel wrote:
> Move the ablk_helper code out of arch/x86 so it can be reused
> by other architectures. The only x86 specific dependency was
> a call to irq_fpu_usable(), this has been factored out and moved
> to crypto/ablk_helper_x86.c
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
..snip..
> diff --git a/crypto/ablk_helper_generic.c b/crypto/ablk_helper_generic.c
> new file mode 100644
> index 0000000..b63b800
> --- /dev/null
> +++ b/crypto/ablk_helper_generic.c
> @@ -0,0 +1,155 @@
> +/*
> + * Shared async block cipher helpers
> + *
> + * Copyright (c) 2012 Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
> + *
> + * Based on aesni-intel_glue.c by:
> + * Copyright (C) 2008, Intel Corp.
> + * Author: Huang Ying <ying.huang@intel.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
> + * USA
> + *
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/crypto.h>
> +#include <linux/init.h>
> +#include <linux/module.h>
> +#include <linux/hardirq.h>
> +#include <crypto/algapi.h>
> +#include <crypto/cryptd.h>
> +#include <crypto/ablk_helper.h>
> +
> +/* can be overridden by the architecture if desired */
> +bool __weak ablk_can_run_sync(void)
> +{
> + return !in_interrupt();
> +}
Why not have architecture specific header file that provides this function?
With architecture using in_interrupt for this, you would avoid extra function
call.
-Jussi
^ permalink raw reply
* Re: [PATCH] crypto: tegra: use kernel entropy instead of ad-hoc
From: Herbert Xu @ 2013-09-13 23:39 UTC (permalink / raw)
To: Stephen Warren
Cc: Linus Walleij, Varun Wadekar, linux-crypto-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Neil Horman,
linux-tegra-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <52333974.4040404-3lzwWm7+Weoh9ZMKESR00Q@public.gmane.org>
On Fri, Sep 13, 2013 at 10:12:36AM -0600, Stephen Warren wrote:
>
> I'm curious which kernel version it was merged for; it'd be nice to
> remove tegra_chip_uid() from the Tegra tree now since it's unused, but
> that obviously requires this patch in the history.
This is in cryptodev which will be pushed upstream for 3.13.
Cheers,
--
Email: Herbert Xu <herbert-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: tegra: use kernel entropy instead of ad-hoc
From: Stephen Warren @ 2013-09-13 16:12 UTC (permalink / raw)
To: Herbert Xu
Cc: Linus Walleij, Varun Wadekar, linux-crypto-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Neil Horman,
linux-tegra-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20130913122318.GD8579-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org>
On 09/13/2013 06:23 AM, Herbert Xu wrote:
> On Mon, Sep 09, 2013 at 10:02:04AM -0600, Stephen Warren wrote:
>> On 09/09/2013 01:35 AM, Linus Walleij wrote:
>>> The way I read the Tegra AES RNG is that it has a homebrew
>>> algorithm for initializing the 128bit RNG using timespec and
>>> the unique chip ID. This looks like reinventing the (square)
>>> wheel, instead just grab 128bits from the kernel entropy pool
>>> where the time and (after another patch) chip unique ID is
>>> already mixed in.
>>>
>>> Incidentally this also gets rid of a rather ugly
>>> cross-dependence on the machine using an extern declaration.
>>
>> This sounds reasonable to me, although I know little about the driver.
>> Varun, can you please comment?
>>
>> Acked-by: Stephen Warren <swarren-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
>
> Patch applied. Thanks.
I'm curious which kernel version it was merged for; it'd be nice to
remove tegra_chip_uid() from the Tegra tree now since it's unused, but
that obviously requires this patch in the history.
^ permalink raw reply
* [RFC PATCH 2/2] arm64: add support for AES using ARMv8 Crypto Extensions
From: Ard Biesheuvel @ 2013-09-13 15:08 UTC (permalink / raw)
To: linux-arm
Cc: linux-crypto, nico, catalin.marinas, steve.capper, Ard Biesheuvel
In-Reply-To: <1379084886-1178-1-git-send-email-ard.biesheuvel@linaro.org>
This adds ARMv8 Crypto Extensions based implemenations of
AES in CBC, CTR and XTS mode.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/Makefile | 8 +-
arch/arm64/crypto/Makefile | 12 ++
arch/arm64/crypto/aesce-cbc.S | 58 +++++++
arch/arm64/crypto/aesce-ctr.S | 83 +++++++++
arch/arm64/crypto/aesce-glue.c | 352 +++++++++++++++++++++++++++++++++++++++
arch/arm64/crypto/aesce-macros.S | 95 +++++++++++
arch/arm64/crypto/aesce-xts.S | 129 ++++++++++++++
crypto/Kconfig | 7 +
8 files changed, 741 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/crypto/Makefile
create mode 100644 arch/arm64/crypto/aesce-cbc.S
create mode 100644 arch/arm64/crypto/aesce-ctr.S
create mode 100644 arch/arm64/crypto/aesce-glue.c
create mode 100644 arch/arm64/crypto/aesce-macros.S
create mode 100644 arch/arm64/crypto/aesce-xts.S
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index d90cf79..c7d4959 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -36,9 +36,11 @@ TEXT_OFFSET := 0x00080000
export TEXT_OFFSET GZFLAGS
-core-y += arch/arm64/kernel/ arch/arm64/mm/
-core-$(CONFIG_KVM) += arch/arm64/kvm/
-core-$(CONFIG_XEN) += arch/arm64/xen/
+core-y += arch/arm64/kernel/ arch/arm64/mm/
+core-$(CONFIG_KVM) += arch/arm64/kvm/
+core-$(CONFIG_XEN) += arch/arm64/xen/
+core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
+
libs-y := arch/arm64/lib/ $(libs-y)
libs-y += $(LIBGCC)
diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile
new file mode 100644
index 0000000..da1a437
--- /dev/null
+++ b/arch/arm64/crypto/Makefile
@@ -0,0 +1,12 @@
+#
+# linux/arch/arm64/crypto/Makefile
+#
+# Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+
+aes-arm64ce-y := aesce-cbc.o aesce-ctr.o aesce-xts.o aesce-glue.o
+obj-$(CONFIG_CRYPTO_AES_ARM64CE) += aes-arm64ce.o
diff --git a/arch/arm64/crypto/aesce-cbc.S b/arch/arm64/crypto/aesce-cbc.S
new file mode 100644
index 0000000..d955bf2
--- /dev/null
+++ b/arch/arm64/crypto/aesce-cbc.S
@@ -0,0 +1,58 @@
+/*
+ * linux/arch/arm64/crypto/aesce-cbc.S - AES-CBC using ARMv8 crypto extensions
+ *
+ * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/linkage.h>
+
+#include "aesce-macros.S"
+
+ .text
+ .arch armv8-a+crypto
+
+ // aesce_cbc_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
+ // int blocks, u8 iv[], int first)
+ // aesce_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
+ // int blocks, u8 iv[], int first)
+
+ENTRY(aesce_cbc_encrypt)
+ tst w6, #1
+ beq .Lencloop
+
+ ld1 {v2.16b}, [x5] // get iv
+ load_round_keys w3, x2
+
+.Lencloop:
+ ld1 {v1.16b}, [x1], #16 // get next pt block
+ eor v0.16b, v1.16b, v2.16b // ... and xor with iv
+ encrypt_block v2.16b, v0.16b, w3
+ st1 {v2.16b}, [x0], #16
+ subs w4, w4, #1
+ bne .Lencloop
+ ret
+ENDPROC(aesce_cbc_encrypt)
+
+
+ENTRY(aesce_cbc_decrypt)
+ tst w6, #1
+ beq .Ldecloop
+
+ ld1 {v3.16b}, [x5] // get iv
+ load_round_keys w3, x2
+
+.Ldecloop:
+ ld1 {v1.16b}, [x1], #16 // get next ct block
+ mov v0.16b, v1.16b // ... and copy to v0
+ decrypt_block v2.16b, v0.16b, w3
+ eor v0.16b, v2.16b, v3.16b // xor with iv to get pt
+ mov v3.16b, v1.16b // ct is next round's iv
+ st1 {v0.16b}, [x0], #16
+ subs w4, w4, #1
+ bne .Ldecloop
+ ret
+ENDPROC(aesce_cbc_decrypt)
diff --git a/arch/arm64/crypto/aesce-ctr.S b/arch/arm64/crypto/aesce-ctr.S
new file mode 100644
index 0000000..5b5f02f
--- /dev/null
+++ b/arch/arm64/crypto/aesce-ctr.S
@@ -0,0 +1,83 @@
+/*
+ * linux/arch/arm64/crypto/aesce-ctr.S - AES-CTR using ARMv8 crypto extensions
+ *
+ * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/linkage.h>
+
+#include "aesce-macros.S"
+
+ .text
+ .arch armv8-a+crypto
+
+ // aesce_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
+ // size_t bytes, u8 ctr[], int first)
+
+ENTRY(aesce_ctr_encrypt)
+ ld1 {v1.16b}, [x5]
+ tst w6, #1 // 1st time around?
+ umov x6, v1.d[1] // keep swabbed ctr
+ rev x6, x6 // ... in x6
+ beq .Linc
+
+ load_round_keys w3, x2
+
+.Lloop:
+ mov v0.16b, v1.16b
+ encrypt_block v2.16b, v0.16b, w3
+ ld1 {v0.16b}, [x1], #16
+ eor v2.16b, v2.16b, v0.16b
+
+ subs x4, x4, #16
+ bmi .Ltail8
+
+ st1 {v2.16b}, [x0], #16
+ beq .Lout
+.Linc:
+ adds x6, x6, #1 // increment BE ctr
+ rev x7, x6
+ ins v1.d[1], x7
+ bne .Lloop // no overflow?
+
+ umov x7, v1.d[0] // load upper word of ctr
+ rev x7, x7 // ... to handle the carry
+ add x7, x7, #1
+ rev x7, x7
+ ins v1.d[0], x7
+ b .Lloop
+.Lout:
+ st1 {v1.16b}, [x5]
+ ret
+
+.Ltail8:
+ adds x5, x4, #8
+ bmi .Ltail4
+ mov x4, x5
+ st1 {v2.8b}, [x0], #8
+ beq .Lout
+ ext v2.16b, v2.16b, v2.16b, #8
+.Ltail4:
+ subs x5, x4, #4
+ bmi .Ltail2
+ mov x4, x5
+ umov w7, v2.s[0]
+ str w7, [x0], #4
+ beq .Lout
+ ext v2.16b, v2.16b, v2.16b, #4
+.Ltail2:
+ subs x5, x4, #2
+ bmi .Ltail1
+ umov w7, v2.h[0]
+ strh w7, [x0], #2
+ beq .Lout
+ ext v2.16b, v2.16b, v2.16b, #2
+.Ltail1:
+ umov w7, v2.b[0]
+ strb w7, [x0]
+ ret
+ENDPROC(aesce_ctr_encrypt)
diff --git a/arch/arm64/crypto/aesce-glue.c b/arch/arm64/crypto/aesce-glue.c
new file mode 100644
index 0000000..dd259b9
--- /dev/null
+++ b/arch/arm64/crypto/aesce-glue.c
@@ -0,0 +1,352 @@
+/*
+ * linux/arch/arm64/crypto/aesce-cbc-glue.c - wrapper code for AES-CBC for ARMv8
+ *
+ * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <asm/neon.h>
+#include <crypto/aes.h>
+#include <crypto/ablk_helper.h>
+#include <crypto/algapi.h>
+#include <linux/module.h>
+
+/* defined in aesce-cbc.S */
+asmlinkage void aesce_cbc_encrypt(u8 out[], u8 const in[], u8 const rk[],
+ int rounds, int blocks, u8 iv[], int first);
+asmlinkage void aesce_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[],
+ int rounds, int blocks, u8 iv[], int first);
+
+/* defined in aesce-ctr.S */
+asmlinkage void aesce_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
+ int rounds, size_t bytes, u8 ctr[], int first);
+
+/* defined in aesce-xts.S */
+asmlinkage void aesce_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[],
+ u8 const rk2[], int rounds, size_t bytes,
+ u8 iv[], int first);
+
+asmlinkage void aesce_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[],
+ u8 const rk2[], int rounds, size_t bytes,
+ u8 iv[], int first);
+
+struct crypto_aes_xts_ctx {
+ struct crypto_aes_ctx key1;
+ struct crypto_aes_ctx key2;
+};
+
+static int xts_set_key(struct crypto_tfm *tfm, const u8 *in_key,
+ unsigned int key_len)
+{
+ struct crypto_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
+ u32 *flags = &tfm->crt_flags;
+ int ret;
+
+ ret = crypto_aes_expand_key(&ctx->key1, in_key, key_len/2);
+ if (!ret)
+ ret = crypto_aes_expand_key(&ctx->key2, &in_key[key_len/2],
+ key_len/2);
+ if (!ret)
+ return 0;
+
+ *flags |= CRYPTO_TFM_RES_BAD_KEY_LEN;
+ return -EINVAL;
+}
+
+static int cbc_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
+ struct scatterlist *src, unsigned int nbytes)
+{
+ struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
+ int err, first, rounds = 6 + ctx->key_length/4;
+ struct blkcipher_walk walk;
+ unsigned int blocks;
+
+ blkcipher_walk_init(&walk, dst, src, nbytes);
+ err = blkcipher_walk_virt(desc, &walk);
+
+ kernel_neon_begin();
+ for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) {
+ aesce_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
+ (u8*)ctx->key_enc, rounds, blocks, walk.iv,
+ first);
+
+ err = blkcipher_walk_done(desc, &walk, blocks * AES_BLOCK_SIZE);
+ }
+ kernel_neon_end();
+
+ /* non-integral sizes are not supported in CBC */
+ if (unlikely(walk.nbytes))
+ err = -EINVAL;
+
+ return err;
+}
+
+static int cbc_decrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
+ struct scatterlist *src, unsigned int nbytes)
+{
+ struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
+ int err, first, rounds = 6 + ctx->key_length/4;
+ struct blkcipher_walk walk;
+ unsigned int blocks;
+
+ blkcipher_walk_init(&walk, dst, src, nbytes);
+ err = blkcipher_walk_virt(desc, &walk);
+
+ kernel_neon_begin();
+ for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) {
+ aesce_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr,
+ (u8*)ctx->key_dec, rounds, blocks, walk.iv,
+ first);
+
+ err = blkcipher_walk_done(desc, &walk, blocks * AES_BLOCK_SIZE);
+ }
+ kernel_neon_end();
+
+ /* non-integral sizes are not supported in CBC */
+ if (unlikely(walk.nbytes))
+ err = -EINVAL;
+
+ return err;
+}
+
+static int ctr_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
+ struct scatterlist *src, unsigned int nbytes)
+{
+ struct crypto_aes_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
+ int err, first, rounds = 6 + ctx->key_length/4;
+ struct blkcipher_walk walk;
+ u8 ctr[AES_BLOCK_SIZE];
+
+ blkcipher_walk_init(&walk, dst, src, nbytes);
+ err = blkcipher_walk_virt(desc, &walk);
+
+ memcpy(ctr, walk.iv, AES_BLOCK_SIZE);
+
+ kernel_neon_begin();
+ for (first = 1; (nbytes = walk.nbytes); first = 0) {
+ aesce_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
+ (u8*)ctx->key_enc, rounds, nbytes, ctr, first);
+
+ err = blkcipher_walk_done(desc, &walk, 0);
+
+ /* non-integral block *must* be the last one */
+ if (unlikely(walk.nbytes && (nbytes & (AES_BLOCK_SIZE-1)))) {
+ err = -EINVAL;
+ break;
+ }
+ }
+ kernel_neon_end();
+
+ return err;
+}
+
+static int xts_encrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
+ struct scatterlist *src, unsigned int nbytes)
+{
+ struct crypto_aes_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
+ int err, first, rounds = 6 + ctx->key1.key_length/4;
+ struct blkcipher_walk walk;
+
+ blkcipher_walk_init(&walk, dst, src, nbytes);
+ err = blkcipher_walk_virt(desc, &walk);
+
+ kernel_neon_begin();
+ for (first = 1; (nbytes = walk.nbytes); first = 0) {
+ aesce_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
+ (u8*)ctx->key1.key_enc,
+ (u8*)ctx->key2.key_enc,
+ rounds, nbytes, walk.iv, first);
+
+ err = blkcipher_walk_done(desc, &walk, 0);
+
+ /* non-integral block *must* be the last one */
+ if (unlikely(walk.nbytes && (nbytes & (AES_BLOCK_SIZE-1)))) {
+ err = -EINVAL;
+ break;
+ }
+ }
+ kernel_neon_end();
+
+ return err;
+}
+
+static int xts_decrypt(struct blkcipher_desc *desc, struct scatterlist *dst,
+ struct scatterlist *src, unsigned int nbytes)
+{
+ struct crypto_aes_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
+ int err, first, rounds = 6 + ctx->key1.key_length/4;
+ struct blkcipher_walk walk;
+
+ blkcipher_walk_init(&walk, dst, src, nbytes);
+ err = blkcipher_walk_virt(desc, &walk);
+
+ kernel_neon_begin();
+ for (first = 1; (nbytes = walk.nbytes); first = 0) {
+ aesce_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr,
+ (u8*)ctx->key1.key_enc,
+ (u8*)ctx->key2.key_dec,
+ rounds, nbytes, walk.iv, first);
+
+ err = blkcipher_walk_done(desc, &walk, 0);
+
+ /* non-integral block *must* be the last one */
+ if (unlikely(walk.nbytes && (nbytes & (AES_BLOCK_SIZE-1)))) {
+ err = -EINVAL;
+ break;
+ }
+ }
+ kernel_neon_end();
+
+ return err;
+}
+
+static struct crypto_alg aesce_cbc_algs[] = { {
+ .cra_name = "__cbc-aes-aesce",
+ .cra_driver_name = "__driver-cbc-aes-aesce",
+ .cra_priority = 0,
+ .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct crypto_aes_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_blkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_u = {
+ .blkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_set_key,
+ .encrypt = cbc_encrypt,
+ .decrypt = cbc_decrypt,
+ },
+ },
+}, {
+ .cra_name = "__ctr-aes-aesce",
+ .cra_driver_name = "__driver-ctr-aes-aesce",
+ .cra_priority = 0,
+ .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct crypto_aes_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_blkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_u = {
+ .blkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_set_key,
+ .encrypt = ctr_encrypt,
+ .decrypt = ctr_encrypt,
+ },
+ },
+}, {
+ .cra_name = "__xts-aes-aesce",
+ .cra_driver_name = "__driver-xts-aes-aesce",
+ .cra_priority = 0,
+ .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct crypto_aes_xts_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_blkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_u = {
+ .blkcipher = {
+ .min_keysize = 2*AES_MIN_KEY_SIZE,
+ .max_keysize = 2*AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = xts_set_key,
+ .encrypt = xts_encrypt,
+ .decrypt = xts_decrypt,
+ },
+ },
+}, {
+ .cra_name = "cbc(aes)",
+ .cra_driver_name = "cbc-aes-aesce",
+ .cra_priority = 200,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER|CRYPTO_ALG_ASYNC,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct async_helper_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_init = ablk_init,
+ .cra_exit = ablk_exit,
+ .cra_u = {
+ .ablkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = ablk_set_key,
+ .encrypt = ablk_encrypt,
+ .decrypt = ablk_decrypt,
+ }
+ }
+}, {
+ .cra_name = "ctr(aes)",
+ .cra_driver_name = "ctr-aes-aesce",
+ .cra_priority = 200,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER|CRYPTO_ALG_ASYNC,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct async_helper_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_init = ablk_init,
+ .cra_exit = ablk_exit,
+ .cra_u = {
+ .ablkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = ablk_set_key,
+ .encrypt = ablk_encrypt,
+ .decrypt = ablk_decrypt,
+ }
+ }
+}, {
+ .cra_name = "xts(aes)",
+ .cra_driver_name = "xts-aes-aesce",
+ .cra_priority = 200,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER|CRYPTO_ALG_ASYNC,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct async_helper_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_init = ablk_init,
+ .cra_exit = ablk_exit,
+ .cra_u = {
+ .ablkcipher = {
+ .min_keysize = 2*AES_MIN_KEY_SIZE,
+ .max_keysize = 2*AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = ablk_set_key,
+ .encrypt = ablk_encrypt,
+ .decrypt = ablk_decrypt,
+ }
+ }
+} };
+
+static int __init aesce_cbc_init(void)
+{
+ if (0) // TODO check for crypto extensions
+ return -ENODEV;
+
+ return crypto_register_algs(aesce_cbc_algs, ARRAY_SIZE(aesce_cbc_algs));
+}
+
+static void __exit aesce_cbc_exit(void)
+{
+ crypto_unregister_algs(aesce_cbc_algs, ARRAY_SIZE(aesce_cbc_algs));
+}
+
+module_init(aesce_cbc_init);
+module_exit(aesce_cbc_exit);
+
+MODULE_DESCRIPTION("AES in CBC/CTR/XTS modes using ARMv8 Crypto Extensions");
+MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
+MODULE_LICENSE("GPL");
diff --git a/arch/arm64/crypto/aesce-macros.S b/arch/arm64/crypto/aesce-macros.S
new file mode 100644
index 0000000..37d78f7
--- /dev/null
+++ b/arch/arm64/crypto/aesce-macros.S
@@ -0,0 +1,95 @@
+/*
+ * linux/arch/arm64/crypto/aesce-macros.s - shared macros for ARMv8 AES
+ *
+ * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+ .macro load_round_keys,rounds,rk
+ cmp \rounds, #12
+ ld1 {v16.16b-v19.16b}, [\rk], #64
+ ld1 {v20.16b-v23.16b}, [\rk], #64
+ ld1 {v24.16b-v26.16b}, [\rk], #48
+ blo 1111f
+ ld1 {v27.16b-v28.16b}, [\rk], #32
+ beq 1111f
+ ld1 {v29.16b-v30.16b}, [\rk]
+1111:
+ .endm
+
+ .macro encrypt_block,out,in,rounds
+ cmp \rounds, #12
+ aese \in, v16.16b // 1
+ aesmc \in, \in
+ aese \in, v17.16b // 2
+ aesmc \in, \in
+ aese \in, v18.16b // 3
+ aesmc \in, \in
+ aese \in, v19.16b // 4
+ aesmc \in, \in
+ aese \in, v20.16b // 5
+ aesmc \in, \in
+ aese \in, v21.16b // 6
+ aesmc \in, \in
+ aese \in, v22.16b // 7
+ aesmc \in, \in
+ aese \in, v23.16b // 8
+ aesmc \in, \in
+ aese \in, v24.16b // 9
+ aesmc \in, \in
+ aese \in, v25.16b // 10
+ eor \out, \in, v26.16b
+ blo 2222f
+ aesmc \in, \in
+ aese \in, v26.16b // 11
+ aesmc \in, \in
+ aese \in, v27.16b // 12
+ eor \out, \in, v28.16b
+ beq 2222f
+ aesmc \in, \in
+ aese \in, v28.16b // 13
+ aesmc \in, \in
+ aese \in, v29.16b // 14
+ eor \out, \in, v30.16b
+2222:
+ .endm
+
+ .macro decrypt_block,out,in,rounds
+ cmp \rounds, #12
+ aesd \in, v16.16b // 1
+ aesimc \in, \in
+ aesd \in, v17.16b // 2
+ aesimc \in, \in
+ aesd \in, v18.16b // 3
+ aesimc \in, \in
+ aesd \in, v19.16b // 4
+ aesimc \in, \in
+ aesd \in, v20.16b // 5
+ aesimc \in, \in
+ aesd \in, v21.16b // 6
+ aesimc \in, \in
+ aesd \in, v22.16b // 7
+ aesimc \in, \in
+ aesd \in, v23.16b // 8
+ aesimc \in, \in
+ aesd \in, v24.16b // 9
+ aesimc \in, \in
+ aesd \in, v25.16b // 10
+ eor \out, \in, v26.16b
+ blo 3333f
+ aesimc \in, \in
+ aesd \in, v26.16b // 11
+ aesimc \in, \in
+ aesd \in, v27.16b // 12
+ eor \out, \in, v28.16b
+ beq 3333f
+ aesimc \in, \in
+ aesd \in, v28.16b // 13
+ aesimc \in, \in
+ aesd \in, v29.16b // 14
+ eor \out, \in, v30.16b
+3333:
+ .endm
diff --git a/arch/arm64/crypto/aesce-xts.S b/arch/arm64/crypto/aesce-xts.S
new file mode 100644
index 0000000..9d4a475
--- /dev/null
+++ b/arch/arm64/crypto/aesce-xts.S
@@ -0,0 +1,129 @@
+/*
+ * linux/arch/arm64/crypto/aesce-xts.S - AES-XTS using ARMv8 crypto extensions
+ *
+ * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/linkage.h>
+
+#include "aesce-macros.S"
+
+#define NEXT_TWEAK(tweak,const,spare) ;\
+ sshr spare ##.2d, tweak ##.2d, #63 ;\
+ and spare ##.16b, spare ##.16b, const ##.16b ;\
+ add tweak ##.2d, tweak ##.2d, tweak ##.2d ;\
+ ext spare ##.16b, spare ##.16b, spare ##.16b, #8 ;\
+ eor tweak ##.16b, tweak ##.16b, spare ##.16b
+
+ .text
+ .arch armv8-a+crypto
+
+ // aesce_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[],
+ // u8 const rk2[], int rounds, size_t bytes, u8 iv[],
+ // int first)
+ // aesce_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[],
+ // u8 const rk2[], int rounds, size_t bytes, u8 iv[],
+ // int first)
+
+.Lxts_mul_x:
+ .word 1, 0, 0x87, 0
+
+ENTRY(aesce_xts_encrypt)
+ tst w7, #1 // first call?
+ beq .Lencmore
+
+ ld1 {v0.16b}, [x6]
+ load_round_keys w4, x2
+ encrypt_block v3.16b, v0.16b, w4 // first tweak
+ load_round_keys w4, x3
+ ldr q4, .Lxts_mul_x
+ b .Lencfirst
+.Lencmore:
+ NEXT_TWEAK (v3, v4, v8)
+.Lencfirst:
+ subs x5, x5, #16
+.Lencloop:
+ ld1 {v1.16b}, [x1], #16
+ eor v0.16b, v1.16b, v3.16b
+ encrypt_block v2.16b, v0.16b, w4
+ eor v2.16b, v2.16b, v3.16b
+ st1 {v2.16b}, [x0], #16
+ beq .Lencout
+
+ NEXT_TWEAK (v3, v4, v8)
+ subs x5, x5, #16
+ bpl .Lencloop
+
+ sub x0, x0, #16
+ add x5, x5, #16
+ mov x2, x0
+.Lencsteal:
+ ldrb w6, [x1], #1
+ ldrb w7, [x2, #-16]
+ strb w6, [x2, #-16]
+ strb w7, [x2], #1
+ subs x5, x5, #1
+ bne .Lencsteal
+ mov x1, x0
+ b .Lencloop
+.Lencout:
+ ret
+ENDPROC(aesce_xts_encrypt)
+
+
+ENTRY(aesce_xts_decrypt)
+ tst w7, #1 // first call?
+ beq .Ldecmore
+
+ ld1 {v0.16b}, [x6]
+ load_round_keys w4, x2
+ encrypt_block v3.16b, v0.16b, w4 // first tweak
+ load_round_keys w4, x3
+ ldr q4, .Lxts_mul_x
+ b .Ldecfirst
+.Ldecmore:
+ NEXT_TWEAK (v3, v4, v8)
+.Ldecfirst:
+ tst x5, #15 // odd size?
+ beq .Ldeceven
+ sub x5, x5, #16
+.Ldeceven:
+ subs x5, x5, #16
+ bmi .Lshort
+.Ldecloop:
+ ld1 {v1.16b}, [x1], #16
+ eor v0.16b, v1.16b, v3.16b
+ decrypt_block v2.16b, v0.16b, w4
+ eor v2.16b, v2.16b, v3.16b
+ st1 {v2.16b}, [x0], #16
+ beq .Ldecout
+ bmi .Ldecstealout
+
+ NEXT_TWEAK (v3, v4, v8)
+ subs x5, x5, #16
+ bpl .Ldecloop
+.Lshort:
+ mov v5.16b, v3.16b
+ NEXT_TWEAK (v3, v4, v8) // last round of tweak
+ b .Ldecloop
+.Ldecstealout:
+ sub x0, x0, #16
+ add x5, x5, #16
+ mov x2, x0
+.Ldecsteal:
+ ldrb w7, [x2]
+ ldrb w6, [x1], #1
+ strb w7, [x2, #16]
+ strb w6, [x2], #1
+ subs x5, x5, #1
+ bne .Ldecsteal
+ mov x1, x0
+ mov v3.16b, v5.16b
+ b .Ldecloop
+.Ldecout:
+ ret
+ENDPROC(aesce_xts_decrypt)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 15750a5..fc0ad85 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -775,6 +775,13 @@ config CRYPTO_AES_ARM
See <http://csrc.nist.gov/encryption/aes/> for more information.
+config CRYPTO_AES_ARM64CE
+ tristate "AES using ARMv8 Crypto Extensions"
+ depends on ARM64
+ select CRYPTO_BLKCIPHER
+ select CRYPTO_AES
+ select CRYPTO_ABLK_HELPER
+
config CRYPTO_ANUBIS
tristate "Anubis cipher algorithm"
select CRYPTO_ALGAPI
--
1.8.1.2
^ permalink raw reply related
* [RFC PATCH 1/2] crypto: move ablk_helper out of arch/x86
From: Ard Biesheuvel @ 2013-09-13 15:08 UTC (permalink / raw)
To: linux-arm
Cc: linux-crypto, nico, catalin.marinas, steve.capper, Ard Biesheuvel
In-Reply-To: <1379084886-1178-1-git-send-email-ard.biesheuvel@linaro.org>
Move the ablk_helper code out of arch/x86 so it can be reused
by other architectures. The only x86 specific dependency was
a call to irq_fpu_usable(), this has been factored out and moved
to crypto/ablk_helper_x86.c
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/x86/crypto/Makefile | 1 -
arch/x86/crypto/ablk_helper.c | 149 ---------------------------
arch/x86/crypto/aesni-intel_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx2_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx_glue.c | 2 +-
arch/x86/crypto/cast5_avx_glue.c | 2 +-
arch/x86/crypto/cast6_avx_glue.c | 2 +-
arch/x86/crypto/serpent_avx2_glue.c | 2 +-
arch/x86/crypto/serpent_avx_glue.c | 2 +-
arch/x86/crypto/serpent_sse2_glue.c | 2 +-
arch/x86/crypto/twofish_avx_glue.c | 2 +-
arch/x86/include/asm/crypto/ablk_helper.h | 31 ------
crypto/Kconfig | 21 ++--
crypto/Makefile | 4 +
crypto/ablk_helper_generic.c | 155 +++++++++++++++++++++++++++++
crypto/ablk_helper_x86.c | 8 ++
include/crypto/ablk_helper.h | 34 +++++++
17 files changed, 220 insertions(+), 201 deletions(-)
delete mode 100644 arch/x86/crypto/ablk_helper.c
delete mode 100644 arch/x86/include/asm/crypto/ablk_helper.h
create mode 100644 crypto/ablk_helper_generic.c
create mode 100644 crypto/ablk_helper_x86.c
create mode 100644 include/crypto/ablk_helper.h
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 7d6ba9d..18fda50 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -4,7 +4,6 @@
avx_supported := $(call as-instr,vpxor %xmm0$(comma)%xmm0$(comma)%xmm0,yes,no)
-obj-$(CONFIG_CRYPTO_ABLK_HELPER_X86) += ablk_helper.o
obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o
obj-$(CONFIG_CRYPTO_AES_586) += aes-i586.o
diff --git a/arch/x86/crypto/ablk_helper.c b/arch/x86/crypto/ablk_helper.c
deleted file mode 100644
index 43282fe..0000000
--- a/arch/x86/crypto/ablk_helper.c
+++ /dev/null
@@ -1,149 +0,0 @@
-/*
- * Shared async block cipher helpers
- *
- * Copyright (c) 2012 Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
- *
- * Based on aesni-intel_glue.c by:
- * Copyright (C) 2008, Intel Corp.
- * Author: Huang Ying <ying.huang@intel.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
- * USA
- *
- */
-
-#include <linux/kernel.h>
-#include <linux/crypto.h>
-#include <linux/init.h>
-#include <linux/module.h>
-#include <crypto/algapi.h>
-#include <crypto/cryptd.h>
-#include <asm/i387.h>
-#include <asm/crypto/ablk_helper.h>
-
-int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
- unsigned int key_len)
-{
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- struct crypto_ablkcipher *child = &ctx->cryptd_tfm->base;
- int err;
-
- crypto_ablkcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
- crypto_ablkcipher_set_flags(child, crypto_ablkcipher_get_flags(tfm)
- & CRYPTO_TFM_REQ_MASK);
- err = crypto_ablkcipher_setkey(child, key, key_len);
- crypto_ablkcipher_set_flags(tfm, crypto_ablkcipher_get_flags(child)
- & CRYPTO_TFM_RES_MASK);
- return err;
-}
-EXPORT_SYMBOL_GPL(ablk_set_key);
-
-int __ablk_encrypt(struct ablkcipher_request *req)
-{
- struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- struct blkcipher_desc desc;
-
- desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
- desc.info = req->info;
- desc.flags = 0;
-
- return crypto_blkcipher_crt(desc.tfm)->encrypt(
- &desc, req->dst, req->src, req->nbytes);
-}
-EXPORT_SYMBOL_GPL(__ablk_encrypt);
-
-int ablk_encrypt(struct ablkcipher_request *req)
-{
- struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
-
- if (!irq_fpu_usable()) {
- struct ablkcipher_request *cryptd_req =
- ablkcipher_request_ctx(req);
-
- memcpy(cryptd_req, req, sizeof(*req));
- ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
-
- return crypto_ablkcipher_encrypt(cryptd_req);
- } else {
- return __ablk_encrypt(req);
- }
-}
-EXPORT_SYMBOL_GPL(ablk_encrypt);
-
-int ablk_decrypt(struct ablkcipher_request *req)
-{
- struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
- struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
-
- if (!irq_fpu_usable()) {
- struct ablkcipher_request *cryptd_req =
- ablkcipher_request_ctx(req);
-
- memcpy(cryptd_req, req, sizeof(*req));
- ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
-
- return crypto_ablkcipher_decrypt(cryptd_req);
- } else {
- struct blkcipher_desc desc;
-
- desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
- desc.info = req->info;
- desc.flags = 0;
-
- return crypto_blkcipher_crt(desc.tfm)->decrypt(
- &desc, req->dst, req->src, req->nbytes);
- }
-}
-EXPORT_SYMBOL_GPL(ablk_decrypt);
-
-void ablk_exit(struct crypto_tfm *tfm)
-{
- struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
-
- cryptd_free_ablkcipher(ctx->cryptd_tfm);
-}
-EXPORT_SYMBOL_GPL(ablk_exit);
-
-int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name)
-{
- struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
- struct cryptd_ablkcipher *cryptd_tfm;
-
- cryptd_tfm = cryptd_alloc_ablkcipher(drv_name, 0, 0);
- if (IS_ERR(cryptd_tfm))
- return PTR_ERR(cryptd_tfm);
-
- ctx->cryptd_tfm = cryptd_tfm;
- tfm->crt_ablkcipher.reqsize = sizeof(struct ablkcipher_request) +
- crypto_ablkcipher_reqsize(&cryptd_tfm->base);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(ablk_init_common);
-
-int ablk_init(struct crypto_tfm *tfm)
-{
- char drv_name[CRYPTO_MAX_ALG_NAME];
-
- snprintf(drv_name, sizeof(drv_name), "__driver-%s",
- crypto_tfm_alg_driver_name(tfm));
-
- return ablk_init_common(tfm, drv_name);
-}
-EXPORT_SYMBOL_GPL(ablk_init);
-
-MODULE_LICENSE("GPL");
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index f80e668..835488b 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -34,7 +34,7 @@
#include <asm/cpu_device_id.h>
#include <asm/i387.h>
#include <asm/crypto/aes.h>
-#include <asm/crypto/ablk_helper.h>
+#include <crypto/ablk_helper.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/aead.h>
#include <linux/workqueue.h>
diff --git a/arch/x86/crypto/camellia_aesni_avx2_glue.c b/arch/x86/crypto/camellia_aesni_avx2_glue.c
index 414fe5d..4209a76 100644
--- a/arch/x86/crypto/camellia_aesni_avx2_glue.c
+++ b/arch/x86/crypto/camellia_aesni_avx2_glue.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/ctr.h>
#include <crypto/lrw.h>
@@ -21,7 +22,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/camellia.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAMELLIA_AESNI_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/camellia_aesni_avx_glue.c b/arch/x86/crypto/camellia_aesni_avx_glue.c
index 37fd0c0..87a041a 100644
--- a/arch/x86/crypto/camellia_aesni_avx_glue.c
+++ b/arch/x86/crypto/camellia_aesni_avx_glue.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/ctr.h>
#include <crypto/lrw.h>
@@ -21,7 +22,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/camellia.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAMELLIA_AESNI_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/cast5_avx_glue.c b/arch/x86/crypto/cast5_avx_glue.c
index c663181..e6a3700 100644
--- a/arch/x86/crypto/cast5_avx_glue.c
+++ b/arch/x86/crypto/cast5_avx_glue.c
@@ -26,13 +26,13 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/cast5.h>
#include <crypto/cryptd.h>
#include <crypto/ctr.h>
#include <asm/xcr.h>
#include <asm/xsave.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAST5_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/cast6_avx_glue.c b/arch/x86/crypto/cast6_avx_glue.c
index 8d0dfb8..09f3677 100644
--- a/arch/x86/crypto/cast6_avx_glue.c
+++ b/arch/x86/crypto/cast6_avx_glue.c
@@ -28,6 +28,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/cast6.h>
#include <crypto/cryptd.h>
@@ -37,7 +38,6 @@
#include <crypto/xts.h>
#include <asm/xcr.h>
#include <asm/xsave.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define CAST6_PARALLEL_BLOCKS 8
diff --git a/arch/x86/crypto/serpent_avx2_glue.c b/arch/x86/crypto/serpent_avx2_glue.c
index 23aabc6..2fae489 100644
--- a/arch/x86/crypto/serpent_avx2_glue.c
+++ b/arch/x86/crypto/serpent_avx2_glue.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/ctr.h>
#include <crypto/lrw.h>
@@ -22,7 +23,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/serpent-avx.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#define SERPENT_AVX2_PARALLEL_BLOCKS 16
diff --git a/arch/x86/crypto/serpent_avx_glue.c b/arch/x86/crypto/serpent_avx_glue.c
index 9ae83cf..ff48708 100644
--- a/arch/x86/crypto/serpent_avx_glue.c
+++ b/arch/x86/crypto/serpent_avx_glue.c
@@ -28,6 +28,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/serpent.h>
#include <crypto/cryptd.h>
@@ -38,7 +39,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/serpent-avx.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
/* 8-way parallel cipher functions */
diff --git a/arch/x86/crypto/serpent_sse2_glue.c b/arch/x86/crypto/serpent_sse2_glue.c
index 97a356e..8c95f86 100644
--- a/arch/x86/crypto/serpent_sse2_glue.c
+++ b/arch/x86/crypto/serpent_sse2_glue.c
@@ -34,6 +34,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/serpent.h>
#include <crypto/cryptd.h>
@@ -42,7 +43,6 @@
#include <crypto/lrw.h>
#include <crypto/xts.h>
#include <asm/crypto/serpent-sse2.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
static void serpent_decrypt_cbc_xway(void *ctx, u128 *dst, const u128 *src)
diff --git a/arch/x86/crypto/twofish_avx_glue.c b/arch/x86/crypto/twofish_avx_glue.c
index a62ba54..4e3c665 100644
--- a/arch/x86/crypto/twofish_avx_glue.c
+++ b/arch/x86/crypto/twofish_avx_glue.c
@@ -28,6 +28,7 @@
#include <linux/types.h>
#include <linux/crypto.h>
#include <linux/err.h>
+#include <crypto/ablk_helper.h>
#include <crypto/algapi.h>
#include <crypto/twofish.h>
#include <crypto/cryptd.h>
@@ -39,7 +40,6 @@
#include <asm/xcr.h>
#include <asm/xsave.h>
#include <asm/crypto/twofish.h>
-#include <asm/crypto/ablk_helper.h>
#include <asm/crypto/glue_helper.h>
#include <crypto/scatterwalk.h>
#include <linux/workqueue.h>
diff --git a/arch/x86/include/asm/crypto/ablk_helper.h b/arch/x86/include/asm/crypto/ablk_helper.h
deleted file mode 100644
index 4f93df5..0000000
--- a/arch/x86/include/asm/crypto/ablk_helper.h
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Shared async block cipher helpers
- */
-
-#ifndef _CRYPTO_ABLK_HELPER_H
-#define _CRYPTO_ABLK_HELPER_H
-
-#include <linux/crypto.h>
-#include <linux/kernel.h>
-#include <crypto/cryptd.h>
-
-struct async_helper_ctx {
- struct cryptd_ablkcipher *cryptd_tfm;
-};
-
-extern int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
- unsigned int key_len);
-
-extern int __ablk_encrypt(struct ablkcipher_request *req);
-
-extern int ablk_encrypt(struct ablkcipher_request *req);
-
-extern int ablk_decrypt(struct ablkcipher_request *req);
-
-extern void ablk_exit(struct crypto_tfm *tfm);
-
-extern int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name);
-
-extern int ablk_init(struct crypto_tfm *tfm);
-
-#endif /* _CRYPTO_ABLK_HELPER_H */
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 69ce573..15750a5 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -174,9 +174,8 @@ config CRYPTO_TEST
help
Quick & dirty crypto test module.
-config CRYPTO_ABLK_HELPER_X86
+config CRYPTO_ABLK_HELPER
tristate
- depends on X86
select CRYPTO_CRYPTD
config CRYPTO_GLUE_HELPER_X86
@@ -879,7 +878,7 @@ config CRYPTO_CAMELLIA_AESNI_AVX_X86_64
depends on CRYPTO
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_CAMELLIA_X86_64
select CRYPTO_LRW
@@ -901,7 +900,7 @@ config CRYPTO_CAMELLIA_AESNI_AVX2_X86_64
depends on CRYPTO
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_CAMELLIA_X86_64
select CRYPTO_CAMELLIA_AESNI_AVX_X86_64
@@ -953,7 +952,7 @@ config CRYPTO_CAST5_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_CAST_COMMON
select CRYPTO_CAST5
help
@@ -976,7 +975,7 @@ config CRYPTO_CAST6_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_CAST_COMMON
select CRYPTO_CAST6
@@ -1094,7 +1093,7 @@ config CRYPTO_SERPENT_SSE2_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_LRW
@@ -1116,7 +1115,7 @@ config CRYPTO_SERPENT_SSE2_586
depends on X86 && !64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_LRW
@@ -1138,7 +1137,7 @@ config CRYPTO_SERPENT_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_LRW
@@ -1160,7 +1159,7 @@ config CRYPTO_SERPENT_AVX2_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_SERPENT
select CRYPTO_SERPENT_AVX_X86_64
@@ -1276,7 +1275,7 @@ config CRYPTO_TWOFISH_AVX_X86_64
depends on X86 && 64BIT
select CRYPTO_ALGAPI
select CRYPTO_CRYPTD
- select CRYPTO_ABLK_HELPER_X86
+ select CRYPTO_ABLK_HELPER
select CRYPTO_GLUE_HELPER_X86
select CRYPTO_TWOFISH_COMMON
select CRYPTO_TWOFISH_X86_64
diff --git a/crypto/Makefile b/crypto/Makefile
index 2d5ed08..1013b4b 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -104,3 +104,7 @@ obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o
obj-$(CONFIG_XOR_BLOCKS) += xor.o
obj-$(CONFIG_ASYNC_CORE) += async_tx/
obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/
+
+obj-$(CONFIG_CRYPTO_ABLK_HELPER) += ablk_helper.o
+ablk_helper-y := ablk_helper_generic.o
+ablk_helper-$(CONFIG_X86) += ablk_helper_x86.o
diff --git a/crypto/ablk_helper_generic.c b/crypto/ablk_helper_generic.c
new file mode 100644
index 0000000..b63b800
--- /dev/null
+++ b/crypto/ablk_helper_generic.c
@@ -0,0 +1,155 @@
+/*
+ * Shared async block cipher helpers
+ *
+ * Copyright (c) 2012 Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
+ *
+ * Based on aesni-intel_glue.c by:
+ * Copyright (C) 2008, Intel Corp.
+ * Author: Huang Ying <ying.huang@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+ * USA
+ *
+ */
+
+#include <linux/kernel.h>
+#include <linux/crypto.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/hardirq.h>
+#include <crypto/algapi.h>
+#include <crypto/cryptd.h>
+#include <crypto/ablk_helper.h>
+
+/* can be overridden by the architecture if desired */
+bool __weak ablk_can_run_sync(void)
+{
+ return !in_interrupt();
+}
+
+int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
+ unsigned int key_len)
+{
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+ struct crypto_ablkcipher *child = &ctx->cryptd_tfm->base;
+ int err;
+
+ crypto_ablkcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+ crypto_ablkcipher_set_flags(child, crypto_ablkcipher_get_flags(tfm)
+ & CRYPTO_TFM_REQ_MASK);
+ err = crypto_ablkcipher_setkey(child, key, key_len);
+ crypto_ablkcipher_set_flags(tfm, crypto_ablkcipher_get_flags(child)
+ & CRYPTO_TFM_RES_MASK);
+ return err;
+}
+EXPORT_SYMBOL_GPL(ablk_set_key);
+
+int __ablk_encrypt(struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+ struct blkcipher_desc desc;
+
+ desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
+ desc.info = req->info;
+ desc.flags = 0;
+
+ return crypto_blkcipher_crt(desc.tfm)->encrypt(
+ &desc, req->dst, req->src, req->nbytes);
+}
+EXPORT_SYMBOL_GPL(__ablk_encrypt);
+
+int ablk_encrypt(struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+
+ if (!ablk_can_run_sync()) {
+ struct ablkcipher_request *cryptd_req =
+ ablkcipher_request_ctx(req);
+
+ memcpy(cryptd_req, req, sizeof(*req));
+ ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
+
+ return crypto_ablkcipher_encrypt(cryptd_req);
+ } else {
+ return __ablk_encrypt(req);
+ }
+}
+EXPORT_SYMBOL_GPL(ablk_encrypt);
+
+int ablk_decrypt(struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
+ struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
+
+ if (!ablk_can_run_sync()) {
+ struct ablkcipher_request *cryptd_req =
+ ablkcipher_request_ctx(req);
+
+ memcpy(cryptd_req, req, sizeof(*req));
+ ablkcipher_request_set_tfm(cryptd_req, &ctx->cryptd_tfm->base);
+
+ return crypto_ablkcipher_decrypt(cryptd_req);
+ } else {
+ struct blkcipher_desc desc;
+
+ desc.tfm = cryptd_ablkcipher_child(ctx->cryptd_tfm);
+ desc.info = req->info;
+ desc.flags = 0;
+
+ return crypto_blkcipher_crt(desc.tfm)->decrypt(
+ &desc, req->dst, req->src, req->nbytes);
+ }
+}
+EXPORT_SYMBOL_GPL(ablk_decrypt);
+
+void ablk_exit(struct crypto_tfm *tfm)
+{
+ struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ cryptd_free_ablkcipher(ctx->cryptd_tfm);
+}
+EXPORT_SYMBOL_GPL(ablk_exit);
+
+int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name)
+{
+ struct async_helper_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct cryptd_ablkcipher *cryptd_tfm;
+
+ cryptd_tfm = cryptd_alloc_ablkcipher(drv_name, 0, 0);
+ if (IS_ERR(cryptd_tfm))
+ return PTR_ERR(cryptd_tfm);
+
+ ctx->cryptd_tfm = cryptd_tfm;
+ tfm->crt_ablkcipher.reqsize = sizeof(struct ablkcipher_request) +
+ crypto_ablkcipher_reqsize(&cryptd_tfm->base);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ablk_init_common);
+
+int ablk_init(struct crypto_tfm *tfm)
+{
+ char drv_name[CRYPTO_MAX_ALG_NAME];
+
+ snprintf(drv_name, sizeof(drv_name), "__driver-%s",
+ crypto_tfm_alg_driver_name(tfm));
+
+ return ablk_init_common(tfm, drv_name);
+}
+EXPORT_SYMBOL_GPL(ablk_init);
+
+MODULE_LICENSE("GPL");
diff --git a/crypto/ablk_helper_x86.c b/crypto/ablk_helper_x86.c
new file mode 100644
index 0000000..5671fcb
--- /dev/null
+++ b/crypto/ablk_helper_x86.c
@@ -0,0 +1,8 @@
+
+#include <linux/types.h>
+#include <asm/i387.h>
+
+bool ablk_can_run_sync(void)
+{
+ return irq_fpu_usable();
+}
diff --git a/include/crypto/ablk_helper.h b/include/crypto/ablk_helper.h
new file mode 100644
index 0000000..f8d855c
--- /dev/null
+++ b/include/crypto/ablk_helper.h
@@ -0,0 +1,34 @@
+/*
+ * Shared async block cipher helpers
+ */
+
+#ifndef _CRYPTO_ABLK_HELPER_H
+#define _CRYPTO_ABLK_HELPER_H
+
+#include <linux/crypto.h>
+#include <linux/kernel.h>
+#include <crypto/cryptd.h>
+
+struct async_helper_ctx {
+ struct cryptd_ablkcipher *cryptd_tfm;
+};
+
+/* to be implemented by the architecture */
+extern bool ablk_arch_sync_allowed(void);
+
+extern int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
+ unsigned int key_len);
+
+extern int __ablk_encrypt(struct ablkcipher_request *req);
+
+extern int ablk_encrypt(struct ablkcipher_request *req);
+
+extern int ablk_decrypt(struct ablkcipher_request *req);
+
+extern void ablk_exit(struct crypto_tfm *tfm);
+
+extern int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name);
+
+extern int ablk_init(struct crypto_tfm *tfm);
+
+#endif /* _CRYPTO_ABLK_HELPER_H */
--
1.8.1.2
^ permalink raw reply related
* [RFC PATCH 0/2] AES in CBC/CTR/XTS modes using ARMv8 Crypto Extensions
From: Ard Biesheuvel @ 2013-09-13 15:08 UTC (permalink / raw)
To: linux-arm
Cc: linux-crypto, nico, catalin.marinas, steve.capper, Ard Biesheuvel
Hello all,
This is a first attempt at getting something implemented that uses the ARMv8
crypto extensions for performing AES encryption in CBC, CTR and XTS modes.
The first patch moves the ablk helper code out of arch/x86. This code is used
to automagically instantiate async blkciphers based on the synchronous ones in
my implementation.
The second patch contains the actual AES code. Note that this is only compile
tested, there are most likely numerous bugs that need to be shaken out before
this will even run, so consider yourselves warned.
Comments highly appreciated,
Regards,
Ard.
Ard Biesheuvel (2):
crypto: move ablk_helper out of arch/x86
arm64: add support for AES using ARMv8 Crypto Extensions
arch/arm64/Makefile | 8 +-
arch/arm64/crypto/Makefile | 12 +
arch/arm64/crypto/aesce-cbc.S | 58 +++++
arch/arm64/crypto/aesce-ctr.S | 83 +++++++
arch/arm64/crypto/aesce-glue.c | 352 +++++++++++++++++++++++++++++
arch/arm64/crypto/aesce-macros.S | 95 ++++++++
arch/arm64/crypto/aesce-xts.S | 129 +++++++++++
arch/x86/crypto/Makefile | 1 -
arch/x86/crypto/ablk_helper.c | 149 ------------
arch/x86/crypto/aesni-intel_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx2_glue.c | 2 +-
arch/x86/crypto/camellia_aesni_avx_glue.c | 2 +-
arch/x86/crypto/cast5_avx_glue.c | 2 +-
arch/x86/crypto/cast6_avx_glue.c | 2 +-
arch/x86/crypto/serpent_avx2_glue.c | 2 +-
arch/x86/crypto/serpent_avx_glue.c | 2 +-
arch/x86/crypto/serpent_sse2_glue.c | 2 +-
arch/x86/crypto/twofish_avx_glue.c | 2 +-
arch/x86/include/asm/crypto/ablk_helper.h | 31 ---
crypto/Kconfig | 28 ++-
crypto/Makefile | 4 +
crypto/ablk_helper_generic.c | 155 +++++++++++++
crypto/ablk_helper_x86.c | 8 +
include/crypto/ablk_helper.h | 34 +++
24 files changed, 961 insertions(+), 204 deletions(-)
create mode 100644 arch/arm64/crypto/Makefile
create mode 100644 arch/arm64/crypto/aesce-cbc.S
create mode 100644 arch/arm64/crypto/aesce-ctr.S
create mode 100644 arch/arm64/crypto/aesce-glue.c
create mode 100644 arch/arm64/crypto/aesce-macros.S
create mode 100644 arch/arm64/crypto/aesce-xts.S
delete mode 100644 arch/x86/crypto/ablk_helper.c
delete mode 100644 arch/x86/include/asm/crypto/ablk_helper.h
create mode 100644 crypto/ablk_helper_generic.c
create mode 100644 crypto/ablk_helper_x86.c
create mode 100644 include/crypto/ablk_helper.h
--
1.8.1.2
^ permalink raw reply
* Re: Crypto Fixes for 3.12
From: Herbert Xu @ 2013-09-13 14:39 UTC (permalink / raw)
To: Linus Torvalds
Cc: David S. Miller, Linux Kernel Mailing List,
Linux Crypto Mailing List
In-Reply-To: <CA+55aFyp8mfqWb2CNyCKgMtAn88a5i+MLshq0DjYEmW7FNoZXA@mail.gmail.com>
On Fri, Sep 13, 2013 at 07:22:55AM -0700, Linus Torvalds wrote:
> On Fri, Sep 13, 2013 at 4:30 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> >
> > Herbert Xu (2):
> > crypto: api - Fix race condition in larval lookup
> > crypto: crct10dif - Add fallback for broken initrds
> >
> > crypto/Makefile | 2 +-
> > crypto/api.c | 7 +-
> > crypto/{crct10dif.c => crct10dif_common.c} | 100 +--------------------------
> > crypto/{crct10dif.c => crct10dif_generic.c} | 53 +-------------
> > lib/crc-t10dif.c | 11 ++-
> > 5 files changed, 20 insertions(+), 153 deletions(-)
>
> Please fix your script. You apparently have it using "-C" to find
> copies, which can be very useful to see what is going on especially
> with "--summary" (which you don't have), but is misleading when
> sending diffstats when people don't expect it.
Sorry, will do for future pushes. FWIW the -M stats are
crypto/Makefile | 2 +-
crypto/api.c | 7 +-
crypto/{crct10dif.c => crct10dif_common.c} | 100 +---------------------
crypto/crct10dif_generic.c | 127 ++++++++++++++++++++++++++++
lib/crc-t10dif.c | 11 ++-
5 files changed, 146 insertions(+), 101 deletions(-)
> With just rename detection (git diff -M --stat --summary)
> crypto/Makefile | 2 +-
> crypto/api.c | 7 +-
> crypto/{crct10dif.c => crct10dif_common.c} | 100 +---------------
> crypto/crct10dif_generic.c | 127 +++++++++++++++++++++
> lib/crc-t10dif.c | 11 +-
> 5 files changed, 146 insertions(+), 101 deletions(-)
> rename crypto/{crct10dif.c => crct10dif_common.c} (63%)
> create mode 100644 crypto/crct10dif_generic.c
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: Crypto Fixes for 3.12
From: Linus Torvalds @ 2013-09-13 14:22 UTC (permalink / raw)
To: Herbert Xu
Cc: David S. Miller, Linux Kernel Mailing List,
Linux Crypto Mailing List
In-Reply-To: <20130913113051.GA8189@gondor.apana.org.au>
On Fri, Sep 13, 2013 at 4:30 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> Herbert Xu (2):
> crypto: api - Fix race condition in larval lookup
> crypto: crct10dif - Add fallback for broken initrds
>
> crypto/Makefile | 2 +-
> crypto/api.c | 7 +-
> crypto/{crct10dif.c => crct10dif_common.c} | 100 +--------------------------
> crypto/{crct10dif.c => crct10dif_generic.c} | 53 +-------------
> lib/crc-t10dif.c | 11 ++-
> 5 files changed, 20 insertions(+), 153 deletions(-)
Please fix your script. You apparently have it using "-C" to find
copies, which can be very useful to see what is going on especially
with "--summary" (which you don't have), but is misleading when
sending diffstats when people don't expect it.
The pull request does not have "20 insertions", it has "146
insertions", and it's just that a fair chunk of them come from a file
being essentially duplicated. See the difference:
With copy detection ("git diff -C --stat --summary")
crypto/Makefile | 2 +-
crypto/api.c | 7 +-
crypto/{crct10dif.c => crct10dif_common.c} | 100 +-------------------
crypto/{crct10dif.c => crct10dif_generic.c} | 53 +----------
lib/crc-t10dif.c | 11 ++-
5 files changed, 20 insertions(+), 153 deletions(-)
copy crypto/{crct10dif.c => crct10dif_common.c} (63%)
rename crypto/{crct10dif.c => crct10dif_generic.c} (55%)
With just rename detection (git diff -M --stat --summary)
crypto/Makefile | 2 +-
crypto/api.c | 7 +-
crypto/{crct10dif.c => crct10dif_common.c} | 100 +---------------
crypto/crct10dif_generic.c | 127 +++++++++++++++++++++
lib/crc-t10dif.c | 11 +-
5 files changed, 146 insertions(+), 101 deletions(-)
rename crypto/{crct10dif.c => crct10dif_common.c} (63%)
create mode 100644 crypto/crct10dif_generic.c
and your pull request looked really misleading because it did "-C" but
didn't have that summary pointing out that one of them was a copy.
So please use "-M --stat --summary". That's what git shows me when I
do a "git pull", so that's what I'm going to compare with..
As mentioned "-C" _is_ useful, but it's useful when you're
specifically looking for "that's a lot of new lines, is it copying old
files" kind of things.
Linus
^ permalink raw reply
* Re: [3.12-rc1] Dependency on module-init-tools >= 3.11 ?
From: Tetsuo Handa @ 2013-09-13 13:27 UTC (permalink / raw)
To: waiman.long
Cc: herbert, andr345, rusty, linux-kernel, arthur.marsh, tim.c.chen,
linux-crypto
In-Reply-To: <5231CF30.8050500@hp.com>
Waiman Long wrote:
> I would like to report that I also have the same boot problem on a
> RHEL6.4 box with the crypto patch. My workaround is to force kernel
> build to have the crc_t10dif code built-in by changing the config file:
>
> 4889c4889
> < CONFIG_CRYPTO_CRCT10DIF=m
> ---
> > CONFIG_CRYPTO_CRCT10DIF=y
> 5002c5002
> < CONFIG_CRC_T10DIF=m
> ---
> > CONFIG_CRC_T10DIF=y
>
> This solved the boot problem without any additional patch. Do you think
> you should consider changing the configuration default to "y" instead of
> "m" or doesn't allow the "m" option at all?
That was proposed but not accepted.
https://lkml.org/lkml/2013/7/17/543
You should choose CONFIG_CRYPTO_CRCT10DIF_PCLMUL=y in your kernel config
if your CPU supports PCLMULQDQ.
^ permalink raw reply
* Re: [PATCH] crypto: tegra: use kernel entropy instead of ad-hoc
From: Herbert Xu @ 2013-09-13 12:23 UTC (permalink / raw)
To: Stephen Warren
Cc: Linus Walleij, Varun Wadekar, linux-crypto, linux-arm-kernel,
Neil Horman, linux-tegra
In-Reply-To: <522DF0FC.5080200@wwwdotorg.org>
On Mon, Sep 09, 2013 at 10:02:04AM -0600, Stephen Warren wrote:
> On 09/09/2013 01:35 AM, Linus Walleij wrote:
> > The way I read the Tegra AES RNG is that it has a homebrew
> > algorithm for initializing the 128bit RNG using timespec and
> > the unique chip ID. This looks like reinventing the (square)
> > wheel, instead just grab 128bits from the kernel entropy pool
> > where the time and (after another patch) chip unique ID is
> > already mixed in.
> >
> > Incidentally this also gets rid of a rather ugly
> > cross-dependence on the machine using an extern declaration.
>
> This sounds reasonable to me, although I know little about the driver.
> Varun, can you please comment?
>
> Acked-by: Stephen Warren <swarren@nvidia.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 0/7] crypto: caam - RNG4 patches and fixes
From: Herbert Xu @ 2013-09-13 12:23 UTC (permalink / raw)
To: Alex Porosanu; +Cc: linux-crypto, horia.geanta, vakul
In-Reply-To: <1378742194-32194-1-git-send-email-alexandru.porosanu@freescale.com>
On Mon, Sep 09, 2013 at 06:56:27PM +0300, Alex Porosanu wrote:
> This patch series attempts to fix some identified issues and add some new
> functionalities regarding the RNG4 block in the CAAM driver:
> o if the CAAM driver isn't properly instantiated (e.g. RNG4 initialization
> fails), then there's an illegal memory access generated by the modules
> depending on it; patch 1 in the patch-set fixes this;
> o if the CAAM module is removed, the state handles are not uninstantiated;
> patch 3 in the patch-set adds the necessary descriptor to uninstantiate
> state handle 0;
> o the RNG4 block in CAAM needs to be 'seeded' first before being used
> for generating pseudo-random data. The 'seeding' is done by getting
> entropy from the TRNG ring oscillator. The RTFRQMAX register controls
> the maximum allowable number of samples that can be acquired during
> an entropy sample. Depending on the clock at which the RNG4 block
> (and for that matter the SEC block) runs, it's possible that a
> hard-coded value for the maximum frequency is inadequate, i.e. more
> samples than needed are taken. This leads to failures on devices
> like BSC913x. Patch number 2 fixes this issue by using a kind of
> a software loop to increase the maximum number of samples taken
> until the state handle can be properly initialized; o there are two
> state handles present in the RNG4 block and only one
> is initialized; patch 5 in the patch-set fixes this issue, also
> adding the necessary code for deinstantiation only the handles that were
> instantiated by the driver.
All applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: x86: restore avx2_supported check
From: Herbert Xu @ 2013-09-13 12:22 UTC (permalink / raw)
To: Jussi Kivilinna; +Cc: linux-crypto
In-Reply-To: <20130903134947.5366.51429.stgit@localhost6.localdomain6>
On Tue, Sep 03, 2013 at 04:49:47PM +0300, Jussi Kivilinna wrote:
> Commit 3d387ef08c4 (Revert "crypto: blowfish - add AVX2/x86_64 implementation
> of blowfish cipher") reverted too much as it removed the 'assembler supports
> AVX2' check and therefore disabled remaining AVX2 implementations of Camellia
> and Serpent. Patch restores the check and enables these implementations.
>
> Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Also applied.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: sha256_ssse3 - use correct module alias for sha224
From: Herbert Xu @ 2013-09-13 12:22 UTC (permalink / raw)
To: Jussi Kivilinna; +Cc: linux-crypto
In-Reply-To: <20130903132649.6239.4068.stgit@localhost6.localdomain6>
On Tue, Sep 03, 2013 at 04:26:49PM +0300, Jussi Kivilinna wrote:
> Commit a710f761f (crypto: sha256_ssse3 - add sha224 support) attempted to add
> MODULE_ALIAS for SHA-224, but it ended up being "sha384", probably because
> mix-up with previous commit 340991e30 (crypto: sha512_ssse3 - add sha384
> support). Patch corrects module alias to "sha224".
>
> Reported-by: Pierre-Mayeul Badaire <pierre-mayeul.badaire@m4x.org>
> Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [PATCH 3/3] crypto: tegra-aes - Staticize tegra_aes_cra_exit
From: Sachin Kamat @ 2013-09-13 11:36 UTC (permalink / raw)
To: linux-crypto; +Cc: herbert, sachin.kamat, Stephen Warren
In-Reply-To: <1379072207-11179-1-git-send-email-sachin.kamat@linaro.org>
'tegra_aes_cra_exit' is used only in this file.
Signed-off-by: Sachin Kamat <sachin.kamat@linaro.org>
Cc: Stephen Warren <swarren@wwwdotorg.org>
---
drivers/crypto/tegra-aes.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/crypto/tegra-aes.c b/drivers/crypto/tegra-aes.c
index 2d58da9..4f5f715 100644
--- a/drivers/crypto/tegra-aes.c
+++ b/drivers/crypto/tegra-aes.c
@@ -804,7 +804,7 @@ static int tegra_aes_cra_init(struct crypto_tfm *tfm)
return 0;
}
-void tegra_aes_cra_exit(struct crypto_tfm *tfm)
+static void tegra_aes_cra_exit(struct crypto_tfm *tfm)
{
struct tegra_aes_ctx *ctx =
crypto_ablkcipher_ctx((struct crypto_ablkcipher *)tfm);
--
1.7.9.5
^ permalink raw reply related
* [PATCH 2/3] crypto: omap-aes - Staticize local symbols
From: Sachin Kamat @ 2013-09-13 11:36 UTC (permalink / raw)
To: linux-crypto; +Cc: herbert, sachin.kamat, Dmitry Kasatkin
In-Reply-To: <1379072207-11179-1-git-send-email-sachin.kamat@linaro.org>
Local symbols used only in this file are made static.
Signed-off-by: Sachin Kamat <sachin.kamat@linaro.org>
Cc: Dmitry Kasatkin <dmitry.kasatkin@nokia.com>
---
drivers/crypto/omap-aes.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c
index ce791c2..e1e58d0 100644
--- a/drivers/crypto/omap-aes.c
+++ b/drivers/crypto/omap-aes.c
@@ -554,7 +554,7 @@ static int omap_aes_crypt_dma_stop(struct omap_aes_dev *dd)
return err;
}
-int omap_aes_check_aligned(struct scatterlist *sg)
+static int omap_aes_check_aligned(struct scatterlist *sg)
{
while (sg) {
if (!IS_ALIGNED(sg->offset, 4))
@@ -566,7 +566,7 @@ int omap_aes_check_aligned(struct scatterlist *sg)
return 0;
}
-int omap_aes_copy_sgs(struct omap_aes_dev *dd)
+static int omap_aes_copy_sgs(struct omap_aes_dev *dd)
{
void *buf_in, *buf_out;
int pages;
--
1.7.9.5
^ permalink raw reply related
* [PATCH 1/3] crypto: mv_cesa - Staticize local symbols
From: Sachin Kamat @ 2013-09-13 11:36 UTC (permalink / raw)
To: linux-crypto; +Cc: herbert, sachin.kamat, Sebastian Andrzej Siewior
Local symbols used only in this file are made static.
Signed-off-by: Sachin Kamat <sachin.kamat@linaro.org>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
---
drivers/crypto/mv_cesa.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/crypto/mv_cesa.c b/drivers/crypto/mv_cesa.c
index 3374a3e..f9b9560 100644
--- a/drivers/crypto/mv_cesa.c
+++ b/drivers/crypto/mv_cesa.c
@@ -907,7 +907,7 @@ static int mv_cra_hash_hmac_sha1_init(struct crypto_tfm *tfm)
return mv_cra_hash_init(tfm, "sha1", COP_HMAC_SHA1, SHA1_BLOCK_SIZE);
}
-irqreturn_t crypto_int(int irq, void *priv)
+static irqreturn_t crypto_int(int irq, void *priv)
{
u32 val;
@@ -928,7 +928,7 @@ irqreturn_t crypto_int(int irq, void *priv)
return IRQ_HANDLED;
}
-struct crypto_alg mv_aes_alg_ecb = {
+static struct crypto_alg mv_aes_alg_ecb = {
.cra_name = "ecb(aes)",
.cra_driver_name = "mv-ecb-aes",
.cra_priority = 300,
@@ -951,7 +951,7 @@ struct crypto_alg mv_aes_alg_ecb = {
},
};
-struct crypto_alg mv_aes_alg_cbc = {
+static struct crypto_alg mv_aes_alg_cbc = {
.cra_name = "cbc(aes)",
.cra_driver_name = "mv-cbc-aes",
.cra_priority = 300,
@@ -975,7 +975,7 @@ struct crypto_alg mv_aes_alg_cbc = {
},
};
-struct ahash_alg mv_sha1_alg = {
+static struct ahash_alg mv_sha1_alg = {
.init = mv_hash_init,
.update = mv_hash_update,
.final = mv_hash_final,
@@ -999,7 +999,7 @@ struct ahash_alg mv_sha1_alg = {
}
};
-struct ahash_alg mv_hmac_sha1_alg = {
+static struct ahash_alg mv_hmac_sha1_alg = {
.init = mv_hash_init,
.update = mv_hash_update,
.final = mv_hash_final,
--
1.7.9.5
^ permalink raw reply related
* Crypto Fixes for 3.12
From: Herbert Xu @ 2013-09-13 11:30 UTC (permalink / raw)
To: Linus Torvalds, David S. Miller, Linux Kernel Mailing List,
Linux Crypto Mailing List
In-Reply-To: <20130328082841.GA7425@gondor.apana.org.au>
Hi Linus:
This push fixes a 7+ year race condition in the crypto API that
causes sporadic crashes when multiple threads load the same
algorithm.
It also fixes the crct10dif algorithm again to prevent boot
failures on systems where the initramfs tool ignores module
softdeps.
Please pull from
git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6.git
or
master.kernel.org:/pub/scm/linux/kernel/git/herbert/crypto-2.6.git
Herbert Xu (2):
crypto: api - Fix race condition in larval lookup
crypto: crct10dif - Add fallback for broken initrds
crypto/Makefile | 2 +-
crypto/api.c | 7 +-
crypto/{crct10dif.c => crct10dif_common.c} | 100 +--------------------------
crypto/{crct10dif.c => crct10dif_generic.c} | 53 +-------------
lib/crc-t10dif.c | 11 ++-
5 files changed, 20 insertions(+), 153 deletions(-)
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: question about rfc404 support.
From: Herbert Xu @ 2013-09-13 11:09 UTC (permalink / raw)
To: Hsieh, Che-Min; +Cc: linux-crypto@vger.kernel.org, Bronstein, Dan
In-Reply-To: <A83C99F8370D4D42BEB2D2B3153A6BBC23D6C796@NASANEXD02C.na.qualcomm.com>
On Wed, Sep 11, 2013 at 07:16:13PM +0000, Hsieh, Che-Min wrote:
> Herbert:
>
> Can you confirm the following. Thanks.
>
> Rfc2404 - The Use of HMAC-SHA-1-96 within ESP and AH
>
> For the support, I can't find any algorithm to be specified in the .craname of ahash_alg for Rfc2404.
>
> From
> http://www.freebsd.org/cgi/man.cgi?query=setkey&sektion=8
> it says the following :
>
> ALGORITHMS
> The following list shows the supported algorithms. The protocol and algorithm are almost completely orthogonal. The following list of authentication algorithms can be used as aalgo in the -A aalgo of the protocol parameter:
>
> algorithm keylen (bits) comment
>
> hmac-sha1 160 ah: rfc2404
>
>
> That leads me to believe, from crypto driver stand point, there is no need to have a new algorithm for hmac-sha-1-96 support. Instead, the agent SW(such as ipsec) should use "hmac(sha1)", and do the truncation of digested data from 160 bits to 96 bits.
>
> I run a quick test. The input file to setkey command is defined as such ----
>
> flush;
> spdflush;
> add 10.2.243.75 10.2.243.29 ah 0x604 -A hmac-sha1 0x8D967D88F6CAA9D714800AB3D48051D63F73A312;
> add 10.2.243.29 10.2.243.75 ah 0x605 -A hmac-sha1 0x8D967D88F6CAA9D714800AB3D48051D63F73A314;
> spdadd 10.2.243.75 10.2.243.29 any -P in ipsec ah/transport//use;
> spdadd 10.2.243.29 10.2.243.75 any -P out ipsec ah/transport//use;
>
>
> I use tcpdump. Clearly I can see 24 bytes of AH header of the ipsec packets. The header has 12 bytes of fixed information - Next (1), Ah len (1) ,Reserved (2), Spi (4) ,Sequence (4), and Auth result. (12 bytes)
>
>
> Can you confirm the above? Thanks.
Yes, see net/xfrm/xfrm_algo.c.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto_memcmp: add constant-time memcmp
From: Daniel Borkmann @ 2013-09-13 8:33 UTC (permalink / raw)
To: James Yonan; +Cc: Marcelo Cerri, linux-crypto, herbert, Florian Weimer
In-Reply-To: <5230A649.5010703@openvpn.net>
On 09/11/2013 07:20 PM, James Yonan wrote:
> On 10/09/2013 12:57, Daniel Borkmann wrote:
>> There was a similar patch posted some time ago [1] on lkml, where
>> Florian (CC) made a good point in [2] that future compiler optimizations
>> could short circuit on this. This issue should probably be addressed in
>> such a patch here as well.
>>
>> [1] https://lkml.org/lkml/2013/2/10/131
>> [2] https://lkml.org/lkml/2013/2/11/381
>
> On 11/09/2013 06:19, Marcelo Cerri wrote:
>> The discussion that Daniel pointed out has another interesting point
>> regarding the function name. I don't think it's a good idea to name it
>> crypto_memcpy since it doesn't have behavior the same way as strcmp.
>>
>> Florian suggested in the thread names such crypto_mem_equal, which I
>> think fits better here.
>
> Ok, here's another stab at this:
>
> * Changed the name to crypto_mem_not_equal. The "not_equal" seems to
> make more sense because the function returns a nonzero "true" value if
> the memory regions are not equal.
Ok, sounds good.
> * Good point that a smart optimizer might add instructions to
> short-circuit the loop if all bits in ret have been set. One way to
> deal with this is to disable optimizations that might increase code
> size, since a short-circuit optimization in this case would require
> adding instructions.
>
> #pragma GCC optimize ("Os")
>
> The nice thing about using #pragma is that older versions of gcc that
> don't recognize it will simply ignore it, and we can probably presume
> that older versions of gcc do not support a short-circuit optimization
> if the latest one does not. I did a quick test using gcc 3.4.6 at -O2,
> and did not see any evidence of a short-circuit optimization.
>
> * Improved performance when CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is
> enabled. This makes the performance roughly on-par with memcmp.
Hm, why don't we take fixed-size versions of Daniel J Bernstein from NaCl
library [1], e.g. for comparing hashes?
E.g. for 16 bytes:
int crypto_verify(const unsigned char *x,const unsigned char *y)
{
unsigned int differentbits = 0;
#define F(i) differentbits |= x[i] ^ y[i];
F(0)
F(1)
F(2)
F(3)
F(4)
F(5)
F(6)
F(7)
F(8)
F(9)
F(10)
F(11)
F(12)
F(13)
F(14)
F(15)
return (1 & ((differentbits - 1) >> 8)) - 1;
}
It will return 0 if x[0], x[1], ..., x[15] are the same as y[0], y[1], ..., y[15],
otherwise it returns -1. That's w/o for loops, so probably more "compiler-proof" ...
[1] http://nacl.cr.yp.to/
> ----------------
>
> #pragma GCC optimize ("Os")
>
> noinline unsigned long crypto_mem_not_equal(const void *a, const void *b, size_t size)
> {
> unsigned long ret = 0;
>
> #ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> #if BITS_PER_LONG == 64
> while (size >= 8) {
> ret |= *(unsigned long *)a ^ *(unsigned long *)b;
> a += 8;
> b += 8;
> size -= 8;
> }
> if (!size)
> return ret;
> #endif /* BITS_PER_LONG == 64 */
> if (sizeof(unsigned int) == 4) {
> while (size >= 4) {
> ret |= *(unsigned int *)a ^ *(unsigned int *)b;
> a += 4;
> b += 4;
> size -= 4;
> }
> if (!size)
> return ret;
> }
> #endif /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */
> while (size > 0) {
> ret |= *(unsigned char *)a ^ *(unsigned char *)b;
> a += 1;
> b += 1;
> size -= 1;
> }
> return ret;
> }
>
> James
^ permalink raw reply
* Re: [PATCH] modules: add support for soft module dependencies
From: Rusty Russell @ 2013-09-13 0:07 UTC (permalink / raw)
To: Lucas De Marchi, Herbert Xu
Cc: Andreas Robinson, Linux Kernel Mailing List, Zhao Hongjiang,
David Miller, tim.c.chen, Andrew Morton,
gregkh@linuxfoundation.org, linux-crypto, linux-modules,
Tom Gundersen
In-Reply-To: <CAKi4VA+=1vOOt0BqdQHQaeF1hP3DrRtCL_ObXY0cPeO7S_MtYw@mail.gmail.com>
Lucas De Marchi <lucas.de.marchi@gmail.com> writes:
> On Wed, Jul 24, 2013 at 11:03 PM, Herbert Xu
> <herbert@gondor.apana.org.au> wrote:
>> On Thu, Jul 25, 2013 at 09:32:02AM +0930, Rusty Russell wrote:
>>> Herbert Xu <herbert@gondor.apana.org.au> writes:
>>> > Hi Rusty:
>>> >
>>> > I don't know why this patch never went into the kernel, even
>>> > though the corresponding features have been added to modprobe
>>> > in most if not all distros.
>>>
>>> Because Andreas never sent me the patch? This is the first I've *heard*
>>> of this feature. Looks like it didn't hit lkml either. And what was
>>> 2/2?
>>
>> 2/2 was the patch to actually use this in crc32c.
>>
>>> It's not how I would have done this: post-deps are more flexibly done at
>>> runtime, because the module may have to do work to figure out what to
>>> pull in. But since it already exists, I'll apply this patch: it doesn't
>>> cost the kernel anything.
>
> But it did cause boot failures. The file modules.softdep file was
> supposed to be informational until now. That's why depmod put a
> comment saying to "copy on user's discretion to /etc/modules.d"
> instead of parsing it directly.
I'm happy to change this macro to create a modinfo line like
"softdep:<modname>"
ie. tools like mkinitrd could pick it up and try to find a matching
module, but depmod would ignore it.
It's really up to Lucas, since this affects him.
Cheers,
Rusty.
^ permalink raw reply
* Re: [3.12-rc1] Dependency on module-init-tools >= 3.11 ?
From: Waiman Long @ 2013-09-12 14:26 UTC (permalink / raw)
To: Herbert Xu
Cc: Tetsuo Handa, andr345, rusty, linux-kernel, arthur.marsh,
tim.c.chen, linux-crypto
In-Reply-To: <20130912102953.GA18916@gondor.apana.org.au>
On 09/12/2013 06:29 AM, Herbert Xu wrote:
> On Thu, Sep 12, 2013 at 07:20:23PM +0900, Tetsuo Handa wrote:
>> Herbert Xu wrote:
>>> The trouble is not all distros will include the softdep modules in
>>> the initramfs. So for now I think we will have to live with a fallback.
>> I see.
>>
>> Herbert Xu wrote:
>>> OK, can you please try this patch on top of the current tree?
>>>
>>> This way at least you'll have a working system until your initramfs
>>> tool is fixed to do the right thing.
>> I tested the patch and confirmed that the boot failure was solved.
>>
>> But arch/x86/crypto/crct10dif-pclmul.ko is not included into initramfs and
>> therefore we cannot benefit from PCLMULQDQ version.
> That is expected and is also the status quo. So once the initrd
> generation tool is fixed to include softdeps it will work properly.
>
> Thanks!
I would like to report that I also have the same boot problem on a
RHEL6.4 box with the crypto patch. My workaround is to force kernel
build to have the crc_t10dif code built-in by changing the config file:
4889c4889
< CONFIG_CRYPTO_CRCT10DIF=m
---
> CONFIG_CRYPTO_CRCT10DIF=y
5002c5002
< CONFIG_CRC_T10DIF=m
---
> CONFIG_CRC_T10DIF=y
This solved the boot problem without any additional patch. Do you think
you should consider changing the configuration default to "y" instead of
"m" or doesn't allow the "m" option at all?
Thanks!
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox