* Re: [PATCH v4 1/4] lib: Update LZ4 compressor module
From: Sven Schmidt @ 2017-01-24 16:48 UTC (permalink / raw)
To: Andrew Morton
Cc: bongkyu.kim, rsalvaterra, sergey.senozhatsky, gregkh,
linux-kernel, herbert, davem, linux-crypto, anton, ccross,
keescook, tony.luck, phillip
In-Reply-To: <20170123162357.69f8b6a8717af086a03b512e@linux-foundation.org>
On Mon, Jan 23, 2017 at 04:23:57PM -0800, Andrew Morton wrote:
> On Sun, 22 Jan 2017 20:35:14 +0100 Sven Schmidt <4sschmid@informatik.uni-hamburg.de> wrote:
>
> > This patch updates LZ4 kernel module to LZ4 v1.7.3 by Yann Collet.
> > The kernel module is inspired by the previous work by Chanho Min.
> > The updated LZ4 module will not break existing code since there were alias
> > methods added to ensure backwards compatibility.
> >
> > API changes:
> >
> > New method LZ4_compress_fast which differs from the variant available
> > in kernel by the new acceleration parameter,
> > allowing to trade compression ratio for more compression speed
> > and vice versa.
> >
> > LZ4_decompress_fast is the respective decompression method, featuring a very
> > fast decoder (multiple GB/s per core), able to reach RAM speed in multi-core
> > systems. The decompressor allows to decompress data compressed with
> > LZ4 fast as well as the LZ4 HC (high compression) algorithm.
> >
> > Also the useful functions LZ4_decompress_safe_partial
> > LZ4_compress_destsize were added. The latter reverses the logic by trying to
> > compress as much data as possible from source to dest while the former aims
> > to decompress partial blocks of data.
> >
> > A bunch of streaming functions were also added
> > which allow compressig/decompressing data in multiple steps
> > (so called "streaming mode").
> >
> > The methods lz4_compress and lz4_decompress_unknownoutputsize
> > are now known as LZ4_compress_default respectivley LZ4_decompress_safe.
> > The old methods are still available for providing backwards compatibility.
> >
> > ...
> >
> > +/*
> > + * For backward compatibility
> > + */
> > +static inline int lz4_compressbound(size_t isize)
> > {
>
> Do we actually need the back-compat wrappers? After your other three
> patches we have no callers, correct? If so, we should go ahead and
> eliminate such back-compatibility interfaces.
>
>
> > - return isize + (isize / 255) + 16;
> > + return LZ4_COMPRESSBOUND(isize);
> > }
>
Hey Andrew,
you're absolutely right. Since I changed the callers to use the new functions, we do not need these wrappers anymore.
I will add a patch removing them and send an update.
Thanks,
Sven
^ permalink raw reply
* [PATCH] dm: switch dm-verity to async hash crypto API
From: Gilad Ben-Yossef @ 2017-01-24 14:38 UTC (permalink / raw)
To: dm-devel, agk, snitzer; +Cc: linux-crypto, gilad.benyossef, ofir.drang
Use of the synchronous digest API limits dm-verity when using pure
CPU based algorithm providers and rules out the use of most off
CPU algorithm providers which are normally asynchronous by nature,
which can speed up performance even further.
This reduced performance is especially expressed during boot when
a typical dm-verity use case makes a lot of file access to the
protected volume.
Move dm-verity to the asynchronous hash API to gain more performance
with potentially less CPU load, showing a speedup of 13% while using
the same underlying shash provider (Arm asm SHA256) and 21% speedup
with an off-CPU cryptographic engine algorithm provider (Arm CryptoCell)
for large files/volumes while showing no meaningful effect for small
files.
The following measurements demonstrate the results:
...
-rwxr-xr-x 1 root root 16.0M Jan 1 1980 fs1.img
-rwxr-xr-x 1 root root 16.0M Jan 1 1980 fs2.img
-rwxr-xr-x 1 root root 817.4M Jan 1 1980 fsb1.img
-rwxr-xr-x 1 root root 39.1M Jan 1 1980 fsb2.img
fs1.img and fsb1.img are small and large file system images, respectfully.
fs2.img and fsb2.img are small and large hash volumes, respectfully.
Before change (Arm asm shash)
32768+0 records in
32768+0 records out
real 0m 0.83s
user 0m 0.02s
sys 0m 0.17s
1674040+0 records in
1674040+0 records out
real 0m 41.22s
user 0m 1.15s
sys 0m 8.74s
1674040+0 records in
1674040+0 records out
real 0m 41.26s
user 0m 1.02s
sys 0m 8.95s
After change (Arm asm ahash):
32768+0 records in
32768+0 records out
real 0m 0.81s
user 0m 0.02s
sys 0m 0.17s
1674040+0 records in
1674040+0 records out
real 0m 38.52s
user 0m 1.13s
sys 0m 8.80s
1674040+0 records in
1674040+0 records out
real 0m 36.38s
user 0m 1.08s
sys 0m 8.18s
After change (Arm CryptoCell ahash):
32768+0 records in
32768+0 records out
real 0m 0.81s
user 0m 0.01s
sys 0m 0.18s
1674040+0 records in
1674040+0 records out
real 0m 35.87s
user 0m 0.95s
sys 0m 8.19s
1674040+0 records in
1674040+0 records out
real 0m 33.96s
user 0m 1.10s
sys 0m 8.00s
All measurements done on a dual core Zynq ZC706 development board
with an Armv7 processors.
Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com>
---
drivers/md/dm-verity-fec.c | 4 +-
drivers/md/dm-verity-target.c | 230 +++++++++++++++++++++++++++++++-----------
drivers/md/dm-verity.h | 26 +++--
3 files changed, 192 insertions(+), 68 deletions(-)
diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c
index 0f0eb8a..dab98fe 100644
--- a/drivers/md/dm-verity-fec.c
+++ b/drivers/md/dm-verity-fec.c
@@ -188,7 +188,7 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_fec_io *fio,
static int fec_is_erasure(struct dm_verity *v, struct dm_verity_io *io,
u8 *want_digest, u8 *data)
{
- if (unlikely(verity_hash(v, verity_io_hash_desc(v, io),
+ if (unlikely(verity_hash(v, verity_io_hash_req(v, io),
data, 1 << v->data_dev_block_bits,
verity_io_real_digest(v, io))))
return 0;
@@ -397,7 +397,7 @@ static int fec_decode_rsb(struct dm_verity *v, struct dm_verity_io *io,
}
/* Always re-validate the corrected block against the expected hash */
- r = verity_hash(v, verity_io_hash_desc(v, io), fio->output,
+ r = verity_hash(v, verity_io_hash_req(v, io), fio->output,
1 << v->data_dev_block_bits,
verity_io_real_digest(v, io));
if (unlikely(r < 0))
diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index 7335d8a..910f083 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -93,81 +93,144 @@ static sector_t verity_position_at_level(struct dm_verity *v, sector_t block,
}
/*
- * Wrapper for crypto_shash_init, which handles verity salting.
+ * Callback function for asynchrnous crypto API completion notification
*/
-static int verity_hash_init(struct dm_verity *v, struct shash_desc *desc)
+static void verity_hash_done(struct crypto_async_request *base, int err)
+{
+ struct verity_result *res = (struct verity_result *)base->data;
+
+ if (err == -EINPROGRESS)
+ return;
+
+ res->err = err;
+ complete(&res->completion);
+}
+
+/*
+ * Wait for async crypto API callback
+ */
+static inline int verity_wait_hash(struct verity_result *res)
{
int r;
- desc->tfm = v->tfm;
- desc->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+ r = wait_for_completion_interruptible(&res->completion);
- r = crypto_shash_init(desc);
+ if (!r)
+ r = res->err;
- if (unlikely(r < 0)) {
- DMERR("crypto_shash_init failed: %d", r);
- return r;
- }
+ reinit_completion(&res->completion);
+ return r;
+}
- if (likely(v->version >= 1)) {
- r = crypto_shash_update(desc, v->salt, v->salt_size);
+static int verity_hash_update(struct dm_verity *v, struct ahash_request *req,
+ const u8 *data, size_t len,
+ struct verity_result *res)
+{
+ int r;
+ struct scatterlist sg;
- if (unlikely(r < 0)) {
- DMERR("crypto_shash_update failed: %d", r);
- return r;
- }
+ sg_init_table(&sg, 1);
+ sg_set_buf(&sg, data, len);
+ ahash_request_set_crypt(req, &sg, NULL, len);
+ r = crypto_ahash_update(req);
+
+ switch (r) {
+ case 0:
+ break;
+
+ case -EINPROGRESS:
+ case -EBUSY:
+ r = verity_wait_hash(res);
+ break;
+
+ default:
+ DMERR("crypto_ahash_update request submission failed: %d", r);
}
- return 0;
+ if (unlikely(r < 0))
+ DMERR("crypto_ahash_update failed: %d", r);
+
+ return r;
}
-static int verity_hash_update(struct dm_verity *v, struct shash_desc *desc,
- const u8 *data, size_t len)
+/*
+ * Wrapper for crypto_ahash_init, which handles verity salting.
+ */
+static int verity_hash_init(struct dm_verity *v, struct ahash_request *req,
+ struct verity_result *res)
{
- int r = crypto_shash_update(desc, data, len);
+ int r;
- if (unlikely(r < 0))
- DMERR("crypto_shash_update failed: %d", r);
+ ahash_request_set_tfm(req, v->tfm);
+ ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP,
+ verity_hash_done, (void *)res);
+ init_completion(&res->completion);
+
+ r = crypto_ahash_init(req);
+
+ if (unlikely(r < 0)) {
+ DMERR("crypto_ahash_init failed: %d", r);
+ return r;
+ }
+
+ if (likely(v->version >= 1))
+ r = verity_hash_update(v, req, v->salt, v->salt_size, res);
return r;
}
-static int verity_hash_final(struct dm_verity *v, struct shash_desc *desc,
- u8 *digest)
+static int verity_hash_final(struct dm_verity *v, struct ahash_request *req,
+ u8 *digest, struct verity_result *res)
{
int r;
if (unlikely(!v->version)) {
- r = crypto_shash_update(desc, v->salt, v->salt_size);
+ r = verity_hash_update(v, req, v->salt, v->salt_size, res);
if (r < 0) {
- DMERR("crypto_shash_update failed: %d", r);
- return r;
+ DMERR("verity_hash_final failed updating salt: %d", r);
+ goto out;
}
}
- r = crypto_shash_final(desc, digest);
+ ahash_request_set_crypt(req, NULL, digest, 0);
+ r = crypto_ahash_final(req);
- if (unlikely(r < 0))
- DMERR("crypto_shash_final failed: %d", r);
+ switch (r) {
+ case 0:
+ break;
+
+ case -EINPROGRESS:
+ case -EBUSY:
+ r = verity_wait_hash(res);
+ break;
+
+ default:
+ DMERR("crypto_ahash_final request submission failed: %d", r);
+ }
+out:
return r;
}
-int verity_hash(struct dm_verity *v, struct shash_desc *desc,
+int verity_hash(struct dm_verity *v, struct ahash_request *req,
const u8 *data, size_t len, u8 *digest)
{
int r;
+ struct verity_result res;
- r = verity_hash_init(v, desc);
+ r = verity_hash_init(v, req, &res);
if (unlikely(r < 0))
- return r;
+ goto out;
- r = verity_hash_update(v, desc, data, len);
+ r = verity_hash_update(v, req, data, len, &res);
if (unlikely(r < 0))
- return r;
+ goto out;
+
+ r = verity_hash_final(v, req, digest, &res);
- return verity_hash_final(v, desc, digest);
+out:
+ return r;
}
static void verity_hash_at_level(struct dm_verity *v, sector_t block, int level,
@@ -275,7 +338,7 @@ static int verity_verify_level(struct dm_verity *v, struct dm_verity_io *io,
goto release_ret_r;
}
- r = verity_hash(v, verity_io_hash_desc(v, io),
+ r = verity_hash(v, verity_io_hash_req(v, io),
data, 1 << v->hash_dev_block_bits,
verity_io_real_digest(v, io));
if (unlikely(r < 0))
@@ -344,6 +407,63 @@ int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io,
}
/*
+ * Calculates the digest for the given bio
+ */
+int verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io,
+ struct bvec_iter *iter)
+{
+ unsigned int todo = 1 << v->data_dev_block_bits;
+ struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size);
+ struct scatterlist sg;
+ struct verity_result res;
+ struct ahash_request *req = verity_io_hash_req(v, io);
+
+ do {
+ int r;
+ unsigned int len;
+ struct bio_vec bv = bio_iter_iovec(bio, *iter);
+
+ sg_init_table(&sg, 1);
+
+ len = bv.bv_len;
+
+ if (likely(len >= todo))
+ len = todo;
+ /*
+ * Operating on a single page at a time looks suboptimal
+ * until you consider the typical block size is 4,096B.
+ * Going through this loops twice should be very rare.
+ */
+ sg_set_page(&sg, bv.bv_page, bv.bv_len, bv.bv_offset);
+ ahash_request_set_crypt(req, &sg, NULL, len);
+ r = crypto_ahash_update(req);
+
+ switch (r) {
+ case 0:
+ break;
+
+ case -EINPROGRESS:
+ case -EBUSY:
+ r = verity_wait_hash(&res);
+ break;
+
+ default:
+ break;
+ }
+
+ if (unlikely(r < 0)) {
+ DMERR("verity_for_io_block failed: %d", r);
+ return r;
+ }
+
+ bio_advance_iter(bio, iter, len);
+ todo -= len;
+ } while (todo);
+
+ return 0;
+}
+
+/*
* Calls function process for 1 << v->data_dev_block_bits bytes in the bio_vec
* starting from iter.
*/
@@ -381,12 +501,6 @@ int verity_for_bv_block(struct dm_verity *v, struct dm_verity_io *io,
return 0;
}
-static int verity_bv_hash_update(struct dm_verity *v, struct dm_verity_io *io,
- u8 *data, size_t len)
-{
- return verity_hash_update(v, verity_io_hash_desc(v, io), data, len);
-}
-
static int verity_bv_zero(struct dm_verity *v, struct dm_verity_io *io,
u8 *data, size_t len)
{
@@ -403,10 +517,11 @@ static int verity_verify_io(struct dm_verity_io *io)
struct dm_verity *v = io->v;
struct bvec_iter start;
unsigned b;
+ struct verity_result res;
for (b = 0; b < io->n_blocks; b++) {
int r;
- struct shash_desc *desc = verity_io_hash_desc(v, io);
+ struct ahash_request *req = verity_io_hash_req(v, io);
r = verity_hash_for_block(v, io, io->block + b,
verity_io_want_digest(v, io),
@@ -427,16 +542,17 @@ static int verity_verify_io(struct dm_verity_io *io)
continue;
}
- r = verity_hash_init(v, desc);
+ r = verity_hash_init(v, req, &res);
if (unlikely(r < 0))
return r;
start = io->iter;
- r = verity_for_bv_block(v, io, &io->iter, verity_bv_hash_update);
+ r = verity_for_io_block(v, io, &io->iter);
if (unlikely(r < 0))
return r;
- r = verity_hash_final(v, desc, verity_io_real_digest(v, io));
+ r = verity_hash_final(v, req, verity_io_real_digest(v, io),
+ &res);
if (unlikely(r < 0))
return r;
@@ -705,7 +821,7 @@ static void verity_dtr(struct dm_target *ti)
kfree(v->zero_digest);
if (v->tfm)
- crypto_free_shash(v->tfm);
+ crypto_free_ahash(v->tfm);
kfree(v->alg_name);
@@ -723,7 +839,7 @@ static void verity_dtr(struct dm_target *ti)
static int verity_alloc_zero_digest(struct dm_verity *v)
{
int r = -ENOMEM;
- struct shash_desc *desc;
+ struct ahash_request *req;
u8 *zero_data;
v->zero_digest = kmalloc(v->digest_size, GFP_KERNEL);
@@ -731,9 +847,9 @@ static int verity_alloc_zero_digest(struct dm_verity *v)
if (!v->zero_digest)
return r;
- desc = kmalloc(v->shash_descsize, GFP_KERNEL);
+ req = kmalloc(v->ahash_reqsize, GFP_KERNEL);
- if (!desc)
+ if (!req)
return r; /* verity_dtr will free zero_digest */
zero_data = kzalloc(1 << v->data_dev_block_bits, GFP_KERNEL);
@@ -741,11 +857,11 @@ static int verity_alloc_zero_digest(struct dm_verity *v)
if (!zero_data)
goto out;
- r = verity_hash(v, desc, zero_data, 1 << v->data_dev_block_bits,
+ r = verity_hash(v, req, zero_data, 1 << v->data_dev_block_bits,
v->zero_digest);
out:
- kfree(desc);
+ kfree(req);
kfree(zero_data);
return r;
@@ -923,21 +1039,21 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
goto bad;
}
- v->tfm = crypto_alloc_shash(v->alg_name, 0, 0);
+ v->tfm = crypto_alloc_ahash(v->alg_name, 0, CRYPTO_ALG_ASYNC);
if (IS_ERR(v->tfm)) {
ti->error = "Cannot initialize hash function";
r = PTR_ERR(v->tfm);
v->tfm = NULL;
goto bad;
}
- v->digest_size = crypto_shash_digestsize(v->tfm);
+ v->digest_size = crypto_ahash_digestsize(v->tfm);
if ((1 << v->hash_dev_block_bits) < v->digest_size * 2) {
ti->error = "Digest size too big";
r = -EINVAL;
goto bad;
}
- v->shash_descsize =
- sizeof(struct shash_desc) + crypto_shash_descsize(v->tfm);
+ v->ahash_reqsize = sizeof(struct ahash_request) +
+ crypto_ahash_reqsize(v->tfm);
v->root_digest = kmalloc(v->digest_size, GFP_KERNEL);
if (!v->root_digest) {
@@ -1037,7 +1153,7 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
}
ti->per_io_data_size = sizeof(struct dm_verity_io) +
- v->shash_descsize + v->digest_size * 2;
+ v->ahash_reqsize + v->digest_size * 2;
r = verity_fec_ctr(v);
if (r)
diff --git a/drivers/md/dm-verity.h b/drivers/md/dm-verity.h
index fb419f4..aed710a 100644
--- a/drivers/md/dm-verity.h
+++ b/drivers/md/dm-verity.h
@@ -37,7 +37,7 @@ struct dm_verity {
struct dm_target *ti;
struct dm_bufio_client *bufio;
char *alg_name;
- struct crypto_shash *tfm;
+ struct crypto_ahash *tfm;
u8 *root_digest; /* digest of the root block */
u8 *salt; /* salt: its size is salt_size */
u8 *zero_digest; /* digest for a zero block */
@@ -52,7 +52,7 @@ struct dm_verity {
unsigned char levels; /* the number of tree levels */
unsigned char version;
unsigned digest_size; /* digest size for the current hash algorithm */
- unsigned shash_descsize;/* the size of temporary space for crypto */
+ unsigned int ahash_reqsize;/* the size of temporary space for crypto */
int hash_failed; /* set to 1 if hash of any block failed */
enum verity_mode mode; /* mode for handling verification errors */
unsigned corrupted_errs;/* Number of errors for corrupted blocks */
@@ -81,31 +81,36 @@ struct dm_verity_io {
/*
* Three variably-size fields follow this struct:
*
- * u8 hash_desc[v->shash_descsize];
+ * u8 hash_req[v->ahash_reqsize];
* u8 real_digest[v->digest_size];
* u8 want_digest[v->digest_size];
*
- * To access them use: verity_io_hash_desc(), verity_io_real_digest()
+ * To access them use: verity_io_hash_req(), verity_io_real_digest()
* and verity_io_want_digest().
*/
};
-static inline struct shash_desc *verity_io_hash_desc(struct dm_verity *v,
+struct verity_result {
+ struct completion completion;
+ int err;
+};
+
+static inline struct ahash_request *verity_io_hash_req(struct dm_verity *v,
struct dm_verity_io *io)
{
- return (struct shash_desc *)(io + 1);
+ return (struct ahash_request *)(io + 1);
}
static inline u8 *verity_io_real_digest(struct dm_verity *v,
struct dm_verity_io *io)
{
- return (u8 *)(io + 1) + v->shash_descsize;
+ return (u8 *)(io + 1) + v->ahash_reqsize;
}
static inline u8 *verity_io_want_digest(struct dm_verity *v,
struct dm_verity_io *io)
{
- return (u8 *)(io + 1) + v->shash_descsize + v->digest_size;
+ return (u8 *)(io + 1) + v->ahash_reqsize + v->digest_size;
}
static inline u8 *verity_io_digest_end(struct dm_verity *v,
@@ -114,13 +119,16 @@ static inline u8 *verity_io_digest_end(struct dm_verity *v,
return verity_io_want_digest(v, io) + v->digest_size;
}
+extern int verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io,
+ struct bvec_iter *iter);
+
extern int verity_for_bv_block(struct dm_verity *v, struct dm_verity_io *io,
struct bvec_iter *iter,
int (*process)(struct dm_verity *v,
struct dm_verity_io *io,
u8 *data, size_t len));
-extern int verity_hash(struct dm_verity *v, struct shash_desc *desc,
+extern int verity_hash(struct dm_verity *v, struct ahash_request *req,
const u8 *data, size_t len, u8 *digest);
extern int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io,
--
2.1.4
^ permalink raw reply related
* Re: [bug report] crypto: chcr - Add AEAD algos.
From: Dan Carpenter @ 2017-01-24 10:43 UTC (permalink / raw)
To: harsh; +Cc: linux-crypto
In-Reply-To: <20161201204858.GA10660@mwanda>
Ping?
regards,
dan carpenter
On Thu, Dec 01, 2016 at 11:48:58PM +0300, Dan Carpenter wrote:
> Hello Harsh Jain,
>
> This is a semi-automatic email about new static checker warnings.
>
> The patch 2debd3325e55: "crypto: chcr - Add AEAD algos." from Nov 29,
> 2016, leads to the following Smatch complaint:
>
> drivers/crypto/chelsio/chcr_algo.c:2460 chcr_aead_op()
> warn: variable dereferenced before check 'ctx' (see line 2457)
>
> drivers/crypto/chelsio/chcr_algo.c:2460 chcr_aead_op()
> warn: variable dereferenced before check 'ctx->dev' (see line 2457)
>
> drivers/crypto/chelsio/chcr_algo.c
> 2456 struct chcr_context *ctx = crypto_aead_ctx(tfm);
> 2457 struct uld_ctx *u_ctx = ULD_CTX(ctx);
> ^^^^^^^^^^^^
> Derefences both.
>
> 2458 struct sk_buff *skb;
> 2459
> 2460 if (ctx && !ctx->dev) {
> ^^^^^^^^^^^^^^^^
> Check is too late.
>
> 2461 pr_err("chcr : %s : No crypto device.\n", __func__);
> 2462 return -ENXIO;
>
> regards,
> dan carpenter
^ permalink raw reply
* [PATCH v2 0/2 ] Bug Fixes for 4.10
From: Harsh Jain @ 2017-01-24 5:04 UTC (permalink / raw)
To: herbert, linux-crypto; +Cc: Harsh Jain
This patch series includes critical bug fixes
Atul Gupta (2):
crypto:chcr- Fix panic on dma_unmap_sg
crypto:chcr- Check device is allocated before use
drivers/crypto/chelsio/chcr_algo.c | 49 +++++++++++++++++++-----------------
drivers/crypto/chelsio/chcr_core.c | 18 ++++++-------
drivers/crypto/chelsio/chcr_crypto.h | 3 +++
3 files changed, 37 insertions(+), 33 deletions(-)
--
1.8.2.3
^ permalink raw reply
* [PATCH v2 1/2] crypto:chcr- Fix panic on dma_unmap_sg
From: Harsh Jain @ 2017-01-24 5:04 UTC (permalink / raw)
To: herbert, linux-crypto; +Cc: Harsh Jain, Atul Gupta
In-Reply-To: <cover.1485233822.git.harsh@chelsio.com>
Save DMA mapped sg list addresses to request context buffer.
Signed-off-by: Atul Gupta <atul.gupta@chelsio.com>
---
drivers/crypto/chelsio/chcr_algo.c | 49 +++++++++++++++++++-----------------
drivers/crypto/chelsio/chcr_crypto.h | 3 +++
2 files changed, 29 insertions(+), 23 deletions(-)
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 2ed1e24..d29c2b4 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -158,7 +158,7 @@ int chcr_handle_resp(struct crypto_async_request *req, unsigned char *input,
case CRYPTO_ALG_TYPE_AEAD:
ctx_req.req.aead_req = (struct aead_request *)req;
ctx_req.ctx.reqctx = aead_request_ctx(ctx_req.req.aead_req);
- dma_unmap_sg(&u_ctx->lldi.pdev->dev, ctx_req.req.aead_req->dst,
+ dma_unmap_sg(&u_ctx->lldi.pdev->dev, ctx_req.ctx.reqctx->dst,
ctx_req.ctx.reqctx->dst_nents, DMA_FROM_DEVICE);
if (ctx_req.ctx.reqctx->skb) {
kfree_skb(ctx_req.ctx.reqctx->skb);
@@ -1362,8 +1362,7 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req,
struct chcr_wr *chcr_req;
struct cpl_rx_phys_dsgl *phys_cpl;
struct phys_sge_parm sg_param;
- struct scatterlist *src, *dst;
- struct scatterlist src_sg[2], dst_sg[2];
+ struct scatterlist *src;
unsigned int frags = 0, transhdr_len;
unsigned int ivsize = crypto_aead_ivsize(tfm), dst_size = 0;
unsigned int kctx_len = 0;
@@ -1383,19 +1382,21 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req,
if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
goto err;
- src = scatterwalk_ffwd(src_sg, req->src, req->assoclen);
- dst = src;
+ src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
+ reqctx->dst = src;
+
if (req->src != req->dst) {
err = chcr_copy_assoc(req, aeadctx);
if (err)
return ERR_PTR(err);
- dst = scatterwalk_ffwd(dst_sg, req->dst, req->assoclen);
+ reqctx->dst = scatterwalk_ffwd(reqctx->dstffwd, req->dst,
+ req->assoclen);
}
if (get_aead_subtype(tfm) == CRYPTO_ALG_SUB_TYPE_AEAD_NULL) {
null = 1;
assoclen = 0;
}
- reqctx->dst_nents = sg_nents_for_len(dst, req->cryptlen +
+ reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
if (reqctx->dst_nents <= 0) {
pr_err("AUTHENC:Invalid Destination sg entries\n");
@@ -1460,7 +1461,7 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req,
sg_param.obsize = req->cryptlen + (op_type ? -authsize : authsize);
sg_param.qid = qid;
sg_param.align = 0;
- if (map_writesg_phys_cpl(&u_ctx->lldi.pdev->dev, phys_cpl, dst,
+ if (map_writesg_phys_cpl(&u_ctx->lldi.pdev->dev, phys_cpl, reqctx->dst,
&sg_param))
goto dstmap_fail;
@@ -1711,8 +1712,7 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req,
struct chcr_wr *chcr_req;
struct cpl_rx_phys_dsgl *phys_cpl;
struct phys_sge_parm sg_param;
- struct scatterlist *src, *dst;
- struct scatterlist src_sg[2], dst_sg[2];
+ struct scatterlist *src;
unsigned int frags = 0, transhdr_len, ivsize = AES_BLOCK_SIZE;
unsigned int dst_size = 0, kctx_len;
unsigned int sub_type;
@@ -1728,17 +1728,19 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req,
if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
goto err;
sub_type = get_aead_subtype(tfm);
- src = scatterwalk_ffwd(src_sg, req->src, req->assoclen);
- dst = src;
+ src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
+ reqctx->dst = src;
+
if (req->src != req->dst) {
err = chcr_copy_assoc(req, aeadctx);
if (err) {
pr_err("AAD copy to destination buffer fails\n");
return ERR_PTR(err);
}
- dst = scatterwalk_ffwd(dst_sg, req->dst, req->assoclen);
+ reqctx->dst = scatterwalk_ffwd(reqctx->dstffwd, req->dst,
+ req->assoclen);
}
- reqctx->dst_nents = sg_nents_for_len(dst, req->cryptlen +
+ reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
if (reqctx->dst_nents <= 0) {
pr_err("CCM:Invalid Destination sg entries\n");
@@ -1777,7 +1779,7 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req,
sg_param.obsize = req->cryptlen + (op_type ? -authsize : authsize);
sg_param.qid = qid;
sg_param.align = 0;
- if (map_writesg_phys_cpl(&u_ctx->lldi.pdev->dev, phys_cpl, dst,
+ if (map_writesg_phys_cpl(&u_ctx->lldi.pdev->dev, phys_cpl, reqctx->dst,
&sg_param))
goto dstmap_fail;
@@ -1809,8 +1811,7 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req,
struct chcr_wr *chcr_req;
struct cpl_rx_phys_dsgl *phys_cpl;
struct phys_sge_parm sg_param;
- struct scatterlist *src, *dst;
- struct scatterlist src_sg[2], dst_sg[2];
+ struct scatterlist *src;
unsigned int frags = 0, transhdr_len;
unsigned int ivsize = AES_BLOCK_SIZE;
unsigned int dst_size = 0, kctx_len;
@@ -1832,13 +1833,14 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req,
if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
goto err;
- src = scatterwalk_ffwd(src_sg, req->src, req->assoclen);
- dst = src;
+ src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
+ reqctx->dst = src;
if (req->src != req->dst) {
err = chcr_copy_assoc(req, aeadctx);
if (err)
return ERR_PTR(err);
- dst = scatterwalk_ffwd(dst_sg, req->dst, req->assoclen);
+ reqctx->dst = scatterwalk_ffwd(reqctx->dstffwd, req->dst,
+ req->assoclen);
}
if (!req->cryptlen)
@@ -1848,7 +1850,7 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req,
crypt_len = AES_BLOCK_SIZE;
else
crypt_len = req->cryptlen;
- reqctx->dst_nents = sg_nents_for_len(dst, req->cryptlen +
+ reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
if (reqctx->dst_nents <= 0) {
pr_err("GCM:Invalid Destination sg entries\n");
@@ -1923,7 +1925,7 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req,
sg_param.obsize = req->cryptlen + (op_type ? -authsize : authsize);
sg_param.qid = qid;
sg_param.align = 0;
- if (map_writesg_phys_cpl(&u_ctx->lldi.pdev->dev, phys_cpl, dst,
+ if (map_writesg_phys_cpl(&u_ctx->lldi.pdev->dev, phys_cpl, reqctx->dst,
&sg_param))
goto dstmap_fail;
@@ -1937,7 +1939,8 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req,
write_sg_to_skb(skb, &frags, src, req->cryptlen);
} else {
aes_gcm_empty_pld_pad(req->dst, authsize - 1);
- write_sg_to_skb(skb, &frags, dst, crypt_len);
+ write_sg_to_skb(skb, &frags, reqctx->dst, crypt_len);
+
}
create_wreq(ctx, chcr_req, req, skb, kctx_len, size, 1,
diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h
index d5af7d6..7ec0a8f 100644
--- a/drivers/crypto/chelsio/chcr_crypto.h
+++ b/drivers/crypto/chelsio/chcr_crypto.h
@@ -158,6 +158,9 @@ struct ablk_ctx {
};
struct chcr_aead_reqctx {
struct sk_buff *skb;
+ struct scatterlist *dst;
+ struct scatterlist srcffwd[2];
+ struct scatterlist dstffwd[2];
short int dst_nents;
u16 verify;
u8 iv[CHCR_MAX_CRYPTO_IV_LEN];
--
1.8.2.3
^ permalink raw reply related
* [PATCH v2 2/2] crypto:chcr- Check device is allocated before use
From: Harsh Jain @ 2017-01-24 5:04 UTC (permalink / raw)
To: herbert, linux-crypto; +Cc: Harsh Jain, Atul Gupta
In-Reply-To: <cover.1485233822.git.harsh@chelsio.com>
Ensure dev is allocated for crypto uld context before using the device
for crypto operations.
Signed-off-by: Atul Gupta <atul.gupta@chelsio.com>
---
drivers/crypto/chelsio/chcr_core.c | 18 ++++++++----------
1 file changed, 8 insertions(+), 10 deletions(-)
diff --git a/drivers/crypto/chelsio/chcr_core.c b/drivers/crypto/chelsio/chcr_core.c
index 918da8e..1c65f07 100644
--- a/drivers/crypto/chelsio/chcr_core.c
+++ b/drivers/crypto/chelsio/chcr_core.c
@@ -52,6 +52,7 @@
int assign_chcr_device(struct chcr_dev **dev)
{
struct uld_ctx *u_ctx;
+ int ret = -ENXIO;
/*
* Which device to use if multiple devices are available TODO
@@ -59,15 +60,14 @@ int assign_chcr_device(struct chcr_dev **dev)
* must go to the same device to maintain the ordering.
*/
mutex_lock(&dev_mutex); /* TODO ? */
- u_ctx = list_first_entry(&uld_ctx_list, struct uld_ctx, entry);
- if (!u_ctx) {
- mutex_unlock(&dev_mutex);
- return -ENXIO;
+ list_for_each_entry(u_ctx, &uld_ctx_list, entry)
+ if (u_ctx && u_ctx->dev) {
+ *dev = u_ctx->dev;
+ ret = 0;
+ break;
}
-
- *dev = u_ctx->dev;
mutex_unlock(&dev_mutex);
- return 0;
+ return ret;
}
static int chcr_dev_add(struct uld_ctx *u_ctx)
@@ -202,10 +202,8 @@ static int chcr_uld_state_change(void *handle, enum cxgb4_state state)
static int __init chcr_crypto_init(void)
{
- if (cxgb4_register_uld(CXGB4_ULD_CRYPTO, &chcr_uld_info)) {
+ if (cxgb4_register_uld(CXGB4_ULD_CRYPTO, &chcr_uld_info))
pr_err("ULD register fail: No chcr crypto support in cxgb4");
- return -1;
- }
return 0;
}
--
1.8.2.3
^ permalink raw reply related
* Re: [PATCH v4 1/4] lib: Update LZ4 compressor module
From: Andrew Morton @ 2017-01-24 0:23 UTC (permalink / raw)
To: Sven Schmidt
Cc: bongkyu.kim, rsalvaterra, sergey.senozhatsky, gregkh,
linux-kernel, herbert, davem, linux-crypto, anton, ccross,
keescook, tony.luck, phillip
In-Reply-To: <1485113717-29261-2-git-send-email-4sschmid@informatik.uni-hamburg.de>
On Sun, 22 Jan 2017 20:35:14 +0100 Sven Schmidt <4sschmid@informatik.uni-hamburg.de> wrote:
> This patch updates LZ4 kernel module to LZ4 v1.7.3 by Yann Collet.
> The kernel module is inspired by the previous work by Chanho Min.
> The updated LZ4 module will not break existing code since there were alias
> methods added to ensure backwards compatibility.
>
> API changes:
>
> New method LZ4_compress_fast which differs from the variant available
> in kernel by the new acceleration parameter,
> allowing to trade compression ratio for more compression speed
> and vice versa.
>
> LZ4_decompress_fast is the respective decompression method, featuring a very
> fast decoder (multiple GB/s per core), able to reach RAM speed in multi-core
> systems. The decompressor allows to decompress data compressed with
> LZ4 fast as well as the LZ4 HC (high compression) algorithm.
>
> Also the useful functions LZ4_decompress_safe_partial
> LZ4_compress_destsize were added. The latter reverses the logic by trying to
> compress as much data as possible from source to dest while the former aims
> to decompress partial blocks of data.
>
> A bunch of streaming functions were also added
> which allow compressig/decompressing data in multiple steps
> (so called "streaming mode").
>
> The methods lz4_compress and lz4_decompress_unknownoutputsize
> are now known as LZ4_compress_default respectivley LZ4_decompress_safe.
> The old methods are still available for providing backwards compatibility.
>
> ...
>
> +/*
> + * For backward compatibility
> + */
> +static inline int lz4_compressbound(size_t isize)
> {
Do we actually need the back-compat wrappers? After your other three
patches we have no callers, correct? If so, we should go ahead and
eliminate such back-compatibility interfaces.
> - return isize + (isize / 255) + 16;
> + return LZ4_COMPRESSBOUND(isize);
> }
^ permalink raw reply
* RE: [PATCH] crypto: api - Clear CRYPTO_ALG_DEAD bit before registering an alg
From: Benedetto, Salvatore @ 2017-01-23 17:06 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto@vger.kernel.org, Benedetto, Salvatore
In-Reply-To: <20170123145735.GA20806@gondor.apana.org.au>
Hi Herbert,
> -----Original Message-----
> From: linux-crypto-owner@vger.kernel.org [mailto:linux-crypto-
> owner@vger.kernel.org] On Behalf Of Herbert Xu
> Sent: Monday, January 23, 2017 2:58 PM
> To: Benedetto, Salvatore <salvatore.benedetto@intel.com>
> Cc: linux-crypto@vger.kernel.org
> Subject: Re: [PATCH] crypto: api - Clear CRYPTO_ALG_DEAD bit before
> registering an alg
>
> On Fri, Jan 13, 2017 at 11:54:08AM +0000, Salvatore Benedetto wrote:
> > Make sure CRYPTO_ALG_DEAD bit is cleared before proceeding with the
> > algorithm registration. This fixes qat-dh registration when driver is
> > restarted
> >
> > Signed-off-by: Salvatore Benedetto <salvatore.benedetto@intel.com>
>
> Patch applied. Thanks.
I forgot to add CC stable to it.
This error was introduced in 4.8 and so I think it should go into stable 4.8 and 4.9.
Should I resend or can you add that?
Regards,
Salvatore
> --
> Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page:
> http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-crypto" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] crypto: vmx -- disable preemption to enable vsx in aes_ctr.c
From: Paulo Flabiano Smorigo @ 2017-01-23 16:07 UTC (permalink / raw)
To: Li Zhong; +Cc: linux-crypto, herbert
In-Reply-To: <1484901333.8207.11.camel@TP420>
On Fri, 20 Jan 2017 16:35:33 +0800
Li Zhong <zhong@linux.vnet.ibm.com> wrote:
> Some preemptible check warnings were reported from
> enable_kernel_vsx(). This patch disables preemption in aes_ctr.c
> before enabling vsx, and they are now consistent with other files in
> the same directory.
>
> Signed-off-by: Li Zhong <zhong@linux.vnet.ibm.com>
> ---
> drivers/crypto/vmx/aes_ctr.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/drivers/crypto/vmx/aes_ctr.c
> b/drivers/crypto/vmx/aes_ctr.c index 38ed10d..7cf6d31 100644
> --- a/drivers/crypto/vmx/aes_ctr.c
> +++ b/drivers/crypto/vmx/aes_ctr.c
> @@ -80,11 +80,13 @@ static int p8_aes_ctr_setkey(struct crypto_tfm
> *tfm, const u8 *key, int ret;
> struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
>
> + preempt_disable();
> pagefault_disable();
> enable_kernel_vsx();
> ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
> disable_kernel_vsx();
> pagefault_enable();
> + preempt_enable();
>
> ret += crypto_blkcipher_setkey(ctx->fallback, key, keylen);
> return ret;
> @@ -99,11 +101,13 @@ static void p8_aes_ctr_final(struct
> p8_aes_ctr_ctx *ctx, u8 *dst = walk->dst.virt.addr;
> unsigned int nbytes = walk->nbytes;
>
> + preempt_disable();
> pagefault_disable();
> enable_kernel_vsx();
> aes_p8_encrypt(ctrblk, keystream, &ctx->enc_key);
> disable_kernel_vsx();
> pagefault_enable();
> + preempt_enable();
>
> crypto_xor(keystream, src, nbytes);
> memcpy(dst, keystream, nbytes);
> @@ -132,6 +136,7 @@ static int p8_aes_ctr_crypt(struct blkcipher_desc
> *desc, blkcipher_walk_init(&walk, dst, src, nbytes);
> ret = blkcipher_walk_virt_block(desc, &walk,
> AES_BLOCK_SIZE); while ((nbytes = walk.nbytes) >= AES_BLOCK_SIZE) {
> + preempt_disable();
> pagefault_disable();
> enable_kernel_vsx();
> aes_p8_ctr32_encrypt_blocks(walk.src.virt.addr,
> @@ -143,6 +148,7 @@ static int p8_aes_ctr_crypt(struct blkcipher_desc
> *desc, walk.iv);
> disable_kernel_vsx();
> pagefault_enable();
> + preempt_enable();
>
> /* We need to update IV mostly for last
> bytes/round */ inc = (nbytes & AES_BLOCK_MASK) / AES_BLOCK_SIZE;
>
>
Well observed, Thanks.
^ permalink raw reply
* Re: [PATCH] x86/crypto: make constants readonly, allow linker to merge them
From: Denys Vlasenko @ 2017-01-23 15:21 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Herbert Xu, Josh Poimboeuf, Xiaodong Liu, Megha Dey, linux-crypto,
x86, linux-kernel
In-Reply-To: <alpine.DEB.2.20.1701200006100.5358@nanos>
On 01/20/2017 12:09 AM, Thomas Gleixner wrote:
> On Thu, 19 Jan 2017, Denys Vlasenko wrote:
>
>> A lot of asm-optimized routines in arch/x86/crypto/ keep its
>> constants in .data. This is wrong, they should be on .rodata.
>>
>> Mnay of these constants are the same in different modules.
>> For example, 128-bit shuffle mask 0x000102030405060708090A0B0C0D0E0F
>> exists in at least half a dozen places.
>>
>> There is a way to let linker merge them and use just one copy.
>> The rules are as follows: mergeable objects of different sizes
>> should not share sections. You can't put them all in one .rodata
>> section, they will lose "mergeability".
>>
>> GCC puts its mergeable constants in ".rodata.cstSIZE" sections,
>> or ".rodata.cstSIZE.<object_name>" if -fdata-sections is used.
>> This patch does the same:
>>
>> .section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
>>
>> It is important that all data in such section consists of
>> 16-byte elements, not larger ones, and there are no implicit
>> use of one element from another.
>>
>> When this is not the case, use non-mergeable section:
>>
>> .section .rodata[.VAR_NAME], "a", @progbits
>>
>> This reduces .data by ~15 kbytes:
>>
>> text data bss dec hex filename
>> 11097415 2705840 2630712 16433967 fac32f vmlinux-prev.o
>> 11112095 2690672 2630712 16433479 fac147 vmlinux.o
>
> And at the same time it increases text size by ~15k. The overall change in
> total size is 488 byte reduction. Weird.
Of course: the constants do need to go somewhere. They migrate to
.rodata.blabla sections, which eventually got appended to read-only .text
Without merging, if I would just move constants to .rodata,
there would be no net size win at all.
I did not look deepper why the overall change is smaller than expected.
It may be affected by changed padding. The linker is not too clever
about it, and also IIRC we don't actually giving it the best options
to sort sections during link time to pack them better.
^ permalink raw reply
* Re: [PATCH] crypto: tcrypt - Add mode to test specified algs
From: Rabin Vincent @ 2017-01-23 15:13 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto
In-Reply-To: <20170123141403.GC19957@gondor.apana.org.au>
On Mon, Jan 23, 2017 at 10:14:03PM +0800, Herbert Xu wrote:
> On Wed, Jan 18, 2017 at 05:25:00PM +0100, Rabin Vincent wrote:
> > From: Rabin Vincent <rabinv@axis.com>
> > tcrypt offers a bunch of mode= values to test various (groups of)
> > algorithms, but there is no way provided to test a subset of the
> > algorithms. This adds a new mode=2000 which interprets alg= as a
> > colon-separated list of algorithms to test with alg_test(). Colon is
> > used since the names may contain commas.
> >
> > This is useful during driver development and also for regression testing
> > to avoid the errors that are otherwise generated when attempting to test
> > non-enabled algorithms.
> >
> > # insmod tcrypt.ko dyndbg mode=2000 alg="cbc(aes):ecb(aes):hmac(sha256):sha256:xts(aes)"
> > [ 649.418569] tcrypt: testing cbc(aes)
> > [ 649.420809] tcrypt: testing ecb(aes)
> > [ 649.422627] tcrypt: testing hmac(sha256)
> > [ 649.424861] tcrypt: testing sha256
> > [ 649.426368] tcrypt: testing xts(aes)
> > [ 649.430014] tcrypt: all tests passed
> >
> > Signed-off-by: Rabin Vincent <rabinv@axis.com>
>
> You can already do this with the existing mode=0 setting, no?
That's what I thought so too, but that doesn't seem to be the case. The
mode=0 handling is this:
switch (m) {
case 0:
if (alg) {
if (!crypto_has_alg(alg, type,
mask ?: CRYPTO_ALG_TYPE_MASK))
ret = -ENOENT;
break;
}
for (i = 1; i < 200; i++)
ret += do_test(NULL, 0, 0, i);
break;
So, if alg= is specified, after first checking if the specified alg is
present, it just goes ahead and runs all the tests. I'm not sure what
mode=0 alg=foo is meant to be used for.
^ permalink raw reply
* Re: [PATCH] crypto: vmx -- disable preemption to enable vsx in aes_ctr.c
From: Herbert Xu @ 2017-01-23 15:03 UTC (permalink / raw)
To: Li Zhong; +Cc: linux-crypto, leosilva, pfsmorigo
In-Reply-To: <1484901333.8207.11.camel@TP420>
Li Zhong <zhong@linux.vnet.ibm.com> wrote:
> Some preemptible check warnings were reported from enable_kernel_vsx(). This
> patch disables preemption in aes_ctr.c before enabling vsx, and they are now
> consistent with other files in the same directory.
>
> Signed-off-by: Li Zhong <zhong@linux.vnet.ibm.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 0/8] update mediatek crypto driver
From: Herbert Xu @ 2017-01-23 15:01 UTC (permalink / raw)
To: Ryder Lee
Cc: David S. Miller, linux-mediatek, linux-kernel, linux-crypto,
linux-arm-kernel
In-Reply-To: <1484890875-57105-1-git-send-email-ryder.lee@mediatek.com>
On Fri, Jan 20, 2017 at 01:41:07PM +0800, Ryder Lee wrote:
> Hi,
>
> This series of patches is a global rework of the mtk driver.
> Fix bug - incomplete DMA data transfer when SG buffer dst.len != src.len
>
> It also updates some part of the code to make them more generic. For
> instance the crypto request queue management supports both async block
> cipher and AEAD requests, which allows us to add support the the GCM mode.
> GMAC mode is not supported yet.
>
> Current implementation was validated using the tcrypt
> module running modes:
> - 10: ecb(aes), cbc(aes), ctr(aes), rfc3686(ctr(aes))
> - 35: gcm(aes)
> - 2,6,11,12: sha1, sha2 family
>
> tcrypt speed test was run with modes:
> - 211: rfc4106(gcm(aes)), gcm(aes)
> - 500: ecb(aes), cbc(aes), ctr(aes), rfc3686(ctr(aes))
> - 403 ~ 406: sha1, sha2 family
>
> IxChariot multiple pairs throughput 24 hours test:
> - IPSec VPN
> - MACSec
>
> Ryder Lee (8):
> crypto: mediatek - move HW control data to transformation context
> crypto: mediatek - fix incorrect data transfer result
> crypto: mediatek - make crypto request queue management more generic
> crypto: mediatek - rework crypto request completion
> crypto: mediatek - regroup functions by usage
> crypto: mediatek - fix typo and indentation
> crypto: mediatek - add support to CTR mode
> crypto: mediatek - add support to GCM mode
All applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] x86/crypto: make constants readonly, allow linker to merge them
From: Herbert Xu @ 2017-01-23 15:00 UTC (permalink / raw)
To: Denys Vlasenko
Cc: Josh Poimboeuf, Xiaodong Liu, Megha Dey, linux-crypto, x86,
linux-kernel
In-Reply-To: <20170119213304.18140-1-dvlasenk@redhat.com>
On Thu, Jan 19, 2017 at 10:33:04PM +0100, Denys Vlasenko wrote:
> A lot of asm-optimized routines in arch/x86/crypto/ keep its
> constants in .data. This is wrong, they should be on .rodata.
>
> Mnay of these constants are the same in different modules.
> For example, 128-bit shuffle mask 0x000102030405060708090A0B0C0D0E0F
> exists in at least half a dozen places.
>
> There is a way to let linker merge them and use just one copy.
> The rules are as follows: mergeable objects of different sizes
> should not share sections. You can't put them all in one .rodata
> section, they will lose "mergeability".
>
> GCC puts its mergeable constants in ".rodata.cstSIZE" sections,
> or ".rodata.cstSIZE.<object_name>" if -fdata-sections is used.
> This patch does the same:
>
> .section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
>
> It is important that all data in such section consists of
> 16-byte elements, not larger ones, and there are no implicit
> use of one element from another.
>
> When this is not the case, use non-mergeable section:
>
> .section .rodata[.VAR_NAME], "a", @progbits
>
> This reduces .data by ~15 kbytes:
>
> text data bss dec hex filename
> 11097415 2705840 2630712 16433967 fac32f vmlinux-prev.o
> 11112095 2690672 2630712 16433479 fac147 vmlinux.o
>
> Merged objects are visible in System.map:
>
> ffffffff81a28810 r POLY
> ffffffff81a28810 r POLY
> ffffffff81a28820 r TWOONE
> ffffffff81a28820 r TWOONE
> ffffffff81a28830 r PSHUFFLE_BYTE_FLIP_MASK <- merged regardless of
> ffffffff81a28830 r SHUF_MASK <------------- the name difference
> ffffffff81a28830 r SHUF_MASK
> ffffffff81a28830 r SHUF_MASK
> ..
> ffffffff81a28d00 r K512 <- merged three identical 640-byte tables
> ffffffff81a28d00 r K512
> ffffffff81a28d00 r K512
>
> Use of object names in section name suffixes is not strictly necessary,
> but might help if someday link stage will use garbage collection
> to eliminate unused sections (ld --gc-sections).
>
> Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] x86/crypto: fix %progbits -> @progbits
From: Herbert Xu @ 2017-01-23 15:00 UTC (permalink / raw)
To: Denys Vlasenko
Cc: Josh Poimboeuf, Xiaodong Liu, Megha Dey, George Spelvin,
linux-crypto, x86, linux-kernel
In-Reply-To: <20170119212805.18049-1-dvlasenk@redhat.com>
On Thu, Jan 19, 2017 at 10:28:05PM +0100, Denys Vlasenko wrote:
> %progbits form is used on ARM (where @ is a comment char).
>
> x86 consistently uses @progbits everywhere else.
>
> Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: tcrypt - Add mode to test specified algs
From: Herbert Xu @ 2017-01-23 14:14 UTC (permalink / raw)
To: Rabin Vincent; +Cc: linux-crypto, Rabin Vincent
In-Reply-To: <1484756700-10019-1-git-send-email-rabin.vincent@axis.com>
On Wed, Jan 18, 2017 at 05:25:00PM +0100, Rabin Vincent wrote:
> From: Rabin Vincent <rabinv@axis.com>
>
> tcrypt offers a bunch of mode= values to test various (groups of)
> algorithms, but there is no way provided to test a subset of the
> algorithms. This adds a new mode=2000 which interprets alg= as a
> colon-separated list of algorithms to test with alg_test(). Colon is
> used since the names may contain commas.
>
> This is useful during driver development and also for regression testing
> to avoid the errors that are otherwise generated when attempting to test
> non-enabled algorithms.
>
> # insmod tcrypt.ko dyndbg mode=2000 alg="cbc(aes):ecb(aes):hmac(sha256):sha256:xts(aes)"
> [ 649.418569] tcrypt: testing cbc(aes)
> [ 649.420809] tcrypt: testing ecb(aes)
> [ 649.422627] tcrypt: testing hmac(sha256)
> [ 649.424861] tcrypt: testing sha256
> [ 649.426368] tcrypt: testing xts(aes)
> [ 649.430014] tcrypt: all tests passed
>
> Signed-off-by: Rabin Vincent <rabinv@axis.com>
You can already do this with the existing mode=0 setting, no?
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: arm/aes-neonbs - fix issue with v2.22 and older assembler
From: Herbert Xu @ 2017-01-23 14:59 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, arnd, paul.gortmaker
In-Reply-To: <1484828612-18956-1-git-send-email-ard.biesheuvel@linaro.org>
On Thu, Jan 19, 2017 at 12:23:32PM +0000, Ard Biesheuvel wrote:
> The GNU assembler for ARM version 2.22 or older fails to infer the
> element size from the vmov instructions, and aborts the build in
> the following way;
>
> .../aes-neonbs-core.S: Assembler messages:
> .../aes-neonbs-core.S:817: Error: bad type for scalar -- `vmov q1h[1],r10'
> .../aes-neonbs-core.S:817: Error: bad type for scalar -- `vmov q1h[0],r9'
> .../aes-neonbs-core.S:817: Error: bad type for scalar -- `vmov q1l[1],r8'
> .../aes-neonbs-core.S:817: Error: bad type for scalar -- `vmov q1l[0],r7'
> .../aes-neonbs-core.S:818: Error: bad type for scalar -- `vmov q2h[1],r10'
> .../aes-neonbs-core.S:818: Error: bad type for scalar -- `vmov q2h[0],r9'
> .../aes-neonbs-core.S:818: Error: bad type for scalar -- `vmov q2l[1],r8'
> .../aes-neonbs-core.S:818: Error: bad type for scalar -- `vmov q2l[0],r7'
>
> Fix this by setting the element size explicitly, by replacing vmov with
> vmov.32.
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: tcrypt - Add debug prints
From: Herbert Xu @ 2017-01-23 14:59 UTC (permalink / raw)
To: Rabin Vincent; +Cc: linux-crypto, Rabin Vincent
In-Reply-To: <1484747645-17463-1-git-send-email-rabin.vincent@axis.com>
On Wed, Jan 18, 2017 at 02:54:05PM +0100, Rabin Vincent wrote:
> From: Rabin Vincent <rabinv@axis.com>
>
> tcrypt is very tight-lipped when it succeeds, but a bit more feedback
> would be useful when developing or debugging crypto drivers, especially
> since even a successful run ends with the module failing to insert. Add
> a couple of debug prints, which can be enabled with dynamic debug:
>
> Before:
>
> # insmod tcrypt.ko mode=10
> insmod: can't insert 'tcrypt.ko': Resource temporarily unavailable
>
> After:
>
> # insmod tcrypt.ko mode=10 dyndbg
> tcrypt: testing ecb(aes)
> tcrypt: testing cbc(aes)
> tcrypt: testing lrw(aes)
> tcrypt: testing xts(aes)
> tcrypt: testing ctr(aes)
> tcrypt: testing rfc3686(ctr(aes))
> tcrypt: all tests passed
> insmod: can't insert 'tcrypt.ko': Resource temporarily unavailable
>
> Signed-off-by: Rabin Vincent <rabinv@axis.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: arm64/aes-blk - honour iv_out requirement in CBC and CTR modes
From: Herbert Xu @ 2017-01-23 14:58 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel
In-Reply-To: <1484660789-7036-1-git-send-email-ard.biesheuvel@linaro.org>
On Tue, Jan 17, 2017 at 01:46:29PM +0000, Ard Biesheuvel wrote:
> Update the ARMv8 Crypto Extensions and the plain NEON AES implementations
> in CBC and CTR modes to return the next IV back to the skcipher API client.
> This is necessary for chaining to work correctly.
>
> Note that for CTR, this is only done if the request is a round multiple of
> the block size, since otherwise, chaining is impossible anyway.
>
> Cc: <stable@vger.kernel.org> # v3.16+
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 1/1] crypto: img-hash - use dma_data_direction when calling dma_map_sg
From: Herbert Xu @ 2017-01-23 14:58 UTC (permalink / raw)
To: Nicolas Iooss; +Cc: David S. Miller, linux-crypto, linux-kernel
In-Reply-To: <20170115123750.3010-1-nicolas.iooss_linux@m4x.org>
On Sun, Jan 15, 2017 at 01:37:50PM +0100, Nicolas Iooss wrote:
> The fourth argument of dma_map_sg() and dma_unmap_sg() is an item of
> dma_data_direction enum. Function img_hash_xmit_dma() wrongly used
> DMA_MEM_TO_DEV, which is an item of dma_transfer_direction enum.
>
> Replace DMA_MEM_TO_DEV (which value is 1) with DMA_TO_DEVICE (which
> value is fortunately also 1) when calling dma_map_sg() and
> dma_unmap_sg().
>
> Signed-off-by: Nicolas Iooss <nicolas.iooss_linux@m4x.org>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: api - Clear CRYPTO_ALG_DEAD bit before registering an alg
From: Herbert Xu @ 2017-01-23 14:57 UTC (permalink / raw)
To: Salvatore Benedetto; +Cc: linux-crypto
In-Reply-To: <1484308448-3225-1-git-send-email-salvatore.benedetto@intel.com>
On Fri, Jan 13, 2017 at 11:54:08AM +0000, Salvatore Benedetto wrote:
> Make sure CRYPTO_ALG_DEAD bit is cleared before proceeding with
> the algorithm registration. This fixes qat-dh registration when
> driver is restarted
>
> Signed-off-by: Salvatore Benedetto <salvatore.benedetto@intel.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 0/6] Add support for ECDSA algorithm
From: Herbert Xu @ 2017-01-23 14:24 UTC (permalink / raw)
To: Nitin Kumbhar; +Cc: davem, linux-crypto
In-Reply-To: <1484912161-5932-1-git-send-email-nkumbhar@nvidia.com>
On Fri, Jan 20, 2017 at 05:05:55PM +0530, Nitin Kumbhar wrote:
> Hello,
>
> This patch series adds support for Elliptic Curve Digital Signature
> Algorithm (ECDSA). To reuse existing ECC functionality, which is
> added as part of ECDH, it separates out ECC and ECDH so that
> only ECC functionality is available for ECDSA even when ECDH is in
> a disabled state.
>
> Patch #1 restructures ECC and ECDH code such that ECC is not
> dependent on ECDH config.
>
> Patches #2 & #3 add vli and ecc functions which are required
> for other Elliptic curve algorithms like ECDSA and ECIES.
>
> Patch #4 adds support for ECDSA. This has been validated for P192
> and P256 elliptic curves.
>
> Patches #5 and #6 add ECDSA tests to validate ECDSA functionality
> and measure ECDSA performance.
Who is going to use this in the kernel?
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Urgent Please,,
From: Joyes Dadi @ 2017-01-23 14:07 UTC (permalink / raw)
Good Day Dear,
My name is Ms. Joyes Dadi, I am glad you are reading this letter and I hope
we will start our communication and I know that this message will look strange,
surprising and probably unbelievable to you, but it is the reality. I want to
make a donation of money to you.
I contact you by the will of God. I am a firm German woman specialized in
mining gold and diamonds in Africa. But now, I'm very sick of a cancer. My
husband died in an accident two years ago with our two children and now I have
cancer of the esophagus that damaged almost all the cells in my system/agencies
and I'll die soon according to my doctor.
My most concern now is, we grew up in the orphanage and were married in
orphanage. If I die this deposited fund will soon be left alone in the hand of
the bank, and I do want to it that way. Please, if you can be reliable and
sincere to accept my humble proposal; I have (10.5Millions Euro) in a fixed
deposit account; I will order the Bank to transfer the money into your account
in your country immediately, and then you will take the fund to your country
and invest it to the orphanage homes Please, answer as quickly as possible.
God bless you.
Ms. Joyes Dadi
Email: joyesdadi767@citromail.hu
^ permalink raw reply
* [PATCH v2 10/10] crypto: arm64/aes - replace scalar fallback with plain NEON fallback
From: Ard Biesheuvel @ 2017-01-23 14:05 UTC (permalink / raw)
To: linux-crypto, herbert; +Cc: Ard Biesheuvel
In-Reply-To: <1485180326-25612-1-git-send-email-ard.biesheuvel@linaro.org>
The new bitsliced NEON implementation of AES uses a fallback in two
places: CBC encryption (which is strictly sequential, whereas this
driver can only operate efficiently on 8 blocks at a time), and the
XTS tweak generation, which involves encrypting a single AES block
with a different key schedule.
The plain (i.e., non-bitsliced) NEON code is more suitable as a fallback,
given that it is faster than scalar on low end cores (which is what
the NEON implementations target, since high end cores have dedicated
instructions for AES), and shows similar behavior in terms of D-cache
footprint and sensitivity to cache timing attacks. So switch the fallback
handling to the plain NEON driver.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/crypto/Kconfig | 2 +-
arch/arm64/crypto/aes-neonbs-glue.c | 38 ++++++++++++++------
2 files changed, 29 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig
index 5de75c3dcbd4..bed7feddfeed 100644
--- a/arch/arm64/crypto/Kconfig
+++ b/arch/arm64/crypto/Kconfig
@@ -86,7 +86,7 @@ config CRYPTO_AES_ARM64_BS
tristate "AES in ECB/CBC/CTR/XTS modes using bit-sliced NEON algorithm"
depends on KERNEL_MODE_NEON
select CRYPTO_BLKCIPHER
- select CRYPTO_AES_ARM64
+ select CRYPTO_AES_ARM64_NEON_BLK
select CRYPTO_SIMD
endif
diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c
index 323dd76ae5f0..863e436ecf89 100644
--- a/arch/arm64/crypto/aes-neonbs-glue.c
+++ b/arch/arm64/crypto/aes-neonbs-glue.c
@@ -10,7 +10,6 @@
#include <asm/neon.h>
#include <crypto/aes.h>
-#include <crypto/cbc.h>
#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <crypto/xts.h>
@@ -42,7 +41,12 @@ asmlinkage void aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[],
asmlinkage void aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
-asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds);
+/* borrowed from aes-neon-blk.ko */
+asmlinkage void neon_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[],
+ int rounds, int blocks, int first);
+asmlinkage void neon_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[],
+ int rounds, int blocks, u8 iv[],
+ int first);
struct aesbs_ctx {
u8 rk[13 * (8 * AES_BLOCK_SIZE) + 32];
@@ -140,16 +144,28 @@ static int aesbs_cbc_setkey(struct crypto_skcipher *tfm, const u8 *in_key,
return 0;
}
-static void cbc_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst)
+static int cbc_encrypt(struct skcipher_request *req)
{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct aesbs_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
+ struct skcipher_walk walk;
+ int err, first = 1;
- __aes_arm64_encrypt(ctx->enc, dst, src, ctx->key.rounds);
-}
+ err = skcipher_walk_virt(&walk, req, true);
-static int cbc_encrypt(struct skcipher_request *req)
-{
- return crypto_cbc_encrypt_walk(req, cbc_encrypt_one);
+ kernel_neon_begin();
+ while (walk.nbytes >= AES_BLOCK_SIZE) {
+ unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE;
+
+ /* fall back to the non-bitsliced NEON implementation */
+ neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
+ ctx->enc, ctx->key.rounds, blocks, walk.iv,
+ first);
+ err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE);
+ first = 0;
+ }
+ kernel_neon_end();
+ return err;
}
static int cbc_decrypt(struct skcipher_request *req)
@@ -254,9 +270,11 @@ static int __xts_crypt(struct skcipher_request *req,
err = skcipher_walk_virt(&walk, req, true);
- __aes_arm64_encrypt(ctx->twkey, walk.iv, walk.iv, ctx->key.rounds);
-
kernel_neon_begin();
+
+ neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey,
+ ctx->key.rounds, 1, 1);
+
while (walk.nbytes >= AES_BLOCK_SIZE) {
unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE;
--
2.7.4
^ permalink raw reply related
* [PATCH v2 09/10] crypto: arm64/aes-neon-blk - tweak performance for low end cores
From: Ard Biesheuvel @ 2017-01-23 14:05 UTC (permalink / raw)
To: linux-crypto, herbert; +Cc: Ard Biesheuvel
In-Reply-To: <1485180326-25612-1-git-send-email-ard.biesheuvel@linaro.org>
The non-bitsliced AES implementation using the NEON is highly sensitive
to micro-architectural details, and, as it turns out, the Cortex-A53 on
the Raspberry Pi 3 is a core that can benefit from this code, given that
its scalar AES performance is abysmal (32.9 cycles per byte).
The new bitsliced AES code manages 19.8 cycles per byte on this core,
but can only operate on 8 blocks at a time, which is not supported by
all chaining modes. With a bit of tweaking, we can get the plain NEON
code to run at 22.0 cycles per byte, making it useful for sequential
modes like CBC encryption. (Like bitsliced NEON, the plain NEON
implementation does not use any lookup tables, which makes it easy on
the D-cache, and invulnerable to cache timing attacks)
So tweak the plain NEON AES code to use tbl instructions rather than
shl/sri pairs, and to avoid the need to reload permutation vectors or
other constants from memory in every round.
To allow the ECB and CBC encrypt routines to be reused by the bitsliced
NEON code in a subsequent patch, export them from the module.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/crypto/aes-glue.c | 2 +
arch/arm64/crypto/aes-neon.S | 210 ++++++++------------
2 files changed, 81 insertions(+), 131 deletions(-)
diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c
index 8ee1fb7aaa4f..055bc3f61138 100644
--- a/arch/arm64/crypto/aes-glue.c
+++ b/arch/arm64/crypto/aes-glue.c
@@ -409,5 +409,7 @@ static int __init aes_init(void)
module_cpu_feature_match(AES, aes_init);
#else
module_init(aes_init);
+EXPORT_SYMBOL(neon_aes_ecb_encrypt);
+EXPORT_SYMBOL(neon_aes_cbc_encrypt);
#endif
module_exit(aes_exit);
diff --git a/arch/arm64/crypto/aes-neon.S b/arch/arm64/crypto/aes-neon.S
index 85f07ead7c5c..0d111a6fc229 100644
--- a/arch/arm64/crypto/aes-neon.S
+++ b/arch/arm64/crypto/aes-neon.S
@@ -1,7 +1,7 @@
/*
* linux/arch/arm64/crypto/aes-neon.S - AES cipher for ARMv8 NEON
*
- * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ * Copyright (C) 2013 - 2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -25,9 +25,9 @@
/* preload the entire Sbox */
.macro prepare, sbox, shiftrows, temp
adr \temp, \sbox
- movi v12.16b, #0x40
+ movi v12.16b, #0x1b
ldr q13, \shiftrows
- movi v14.16b, #0x1b
+ ldr q14, .Lror32by8
ld1 {v16.16b-v19.16b}, [\temp], #64
ld1 {v20.16b-v23.16b}, [\temp], #64
ld1 {v24.16b-v27.16b}, [\temp], #64
@@ -50,37 +50,32 @@
/* apply SubBytes transformation using the the preloaded Sbox */
.macro sub_bytes, in
- sub v9.16b, \in\().16b, v12.16b
+ sub v9.16b, \in\().16b, v15.16b
tbl \in\().16b, {v16.16b-v19.16b}, \in\().16b
- sub v10.16b, v9.16b, v12.16b
+ sub v10.16b, v9.16b, v15.16b
tbx \in\().16b, {v20.16b-v23.16b}, v9.16b
- sub v11.16b, v10.16b, v12.16b
+ sub v11.16b, v10.16b, v15.16b
tbx \in\().16b, {v24.16b-v27.16b}, v10.16b
tbx \in\().16b, {v28.16b-v31.16b}, v11.16b
.endm
/* apply MixColumns transformation */
- .macro mix_columns, in
- mul_by_x v10.16b, \in\().16b, v9.16b, v14.16b
- rev32 v8.8h, \in\().8h
- eor \in\().16b, v10.16b, \in\().16b
- shl v9.4s, v8.4s, #24
- shl v11.4s, \in\().4s, #24
- sri v9.4s, v8.4s, #8
- sri v11.4s, \in\().4s, #8
- eor v9.16b, v9.16b, v8.16b
- eor v10.16b, v10.16b, v9.16b
- eor \in\().16b, v10.16b, v11.16b
- .endm
-
+ .macro mix_columns, in, enc
+ .if \enc == 0
/* Inverse MixColumns: pre-multiply by { 5, 0, 4, 0 } */
- .macro inv_mix_columns, in
- mul_by_x v11.16b, \in\().16b, v10.16b, v14.16b
- mul_by_x v11.16b, v11.16b, v10.16b, v14.16b
- eor \in\().16b, \in\().16b, v11.16b
- rev32 v11.8h, v11.8h
- eor \in\().16b, \in\().16b, v11.16b
- mix_columns \in
+ mul_by_x v8.16b, \in\().16b, v9.16b, v12.16b
+ mul_by_x v8.16b, v8.16b, v9.16b, v12.16b
+ eor \in\().16b, \in\().16b, v8.16b
+ rev32 v8.8h, v8.8h
+ eor \in\().16b, \in\().16b, v8.16b
+ .endif
+
+ mul_by_x v9.16b, \in\().16b, v8.16b, v12.16b
+ rev32 v8.8h, \in\().8h
+ eor v8.16b, v8.16b, v9.16b
+ eor \in\().16b, \in\().16b, v8.16b
+ tbl \in\().16b, {\in\().16b}, v14.16b
+ eor \in\().16b, \in\().16b, v8.16b
.endm
.macro do_block, enc, in, rounds, rk, rkp, i
@@ -88,16 +83,13 @@
add \rkp, \rk, #16
mov \i, \rounds
1111: eor \in\().16b, \in\().16b, v15.16b /* ^round key */
+ movi v15.16b, #0x40
tbl \in\().16b, {\in\().16b}, v13.16b /* ShiftRows */
sub_bytes \in
- ld1 {v15.4s}, [\rkp], #16
subs \i, \i, #1
+ ld1 {v15.4s}, [\rkp], #16
beq 2222f
- .if \enc == 1
- mix_columns \in
- .else
- inv_mix_columns \in
- .endif
+ mix_columns \in, \enc
b 1111b
2222: eor \in\().16b, \in\().16b, v15.16b /* ^round key */
.endm
@@ -116,48 +108,48 @@
*/
.macro sub_bytes_2x, in0, in1
- sub v8.16b, \in0\().16b, v12.16b
- sub v9.16b, \in1\().16b, v12.16b
+ sub v8.16b, \in0\().16b, v15.16b
tbl \in0\().16b, {v16.16b-v19.16b}, \in0\().16b
+ sub v9.16b, \in1\().16b, v15.16b
tbl \in1\().16b, {v16.16b-v19.16b}, \in1\().16b
- sub v10.16b, v8.16b, v12.16b
- sub v11.16b, v9.16b, v12.16b
+ sub v10.16b, v8.16b, v15.16b
tbx \in0\().16b, {v20.16b-v23.16b}, v8.16b
+ sub v11.16b, v9.16b, v15.16b
tbx \in1\().16b, {v20.16b-v23.16b}, v9.16b
- sub v8.16b, v10.16b, v12.16b
- sub v9.16b, v11.16b, v12.16b
+ sub v8.16b, v10.16b, v15.16b
tbx \in0\().16b, {v24.16b-v27.16b}, v10.16b
+ sub v9.16b, v11.16b, v15.16b
tbx \in1\().16b, {v24.16b-v27.16b}, v11.16b
tbx \in0\().16b, {v28.16b-v31.16b}, v8.16b
tbx \in1\().16b, {v28.16b-v31.16b}, v9.16b
.endm
.macro sub_bytes_4x, in0, in1, in2, in3
- sub v8.16b, \in0\().16b, v12.16b
+ sub v8.16b, \in0\().16b, v15.16b
tbl \in0\().16b, {v16.16b-v19.16b}, \in0\().16b
- sub v9.16b, \in1\().16b, v12.16b
+ sub v9.16b, \in1\().16b, v15.16b
tbl \in1\().16b, {v16.16b-v19.16b}, \in1\().16b
- sub v10.16b, \in2\().16b, v12.16b
+ sub v10.16b, \in2\().16b, v15.16b
tbl \in2\().16b, {v16.16b-v19.16b}, \in2\().16b
- sub v11.16b, \in3\().16b, v12.16b
+ sub v11.16b, \in3\().16b, v15.16b
tbl \in3\().16b, {v16.16b-v19.16b}, \in3\().16b
tbx \in0\().16b, {v20.16b-v23.16b}, v8.16b
tbx \in1\().16b, {v20.16b-v23.16b}, v9.16b
- sub v8.16b, v8.16b, v12.16b
+ sub v8.16b, v8.16b, v15.16b
tbx \in2\().16b, {v20.16b-v23.16b}, v10.16b
- sub v9.16b, v9.16b, v12.16b
+ sub v9.16b, v9.16b, v15.16b
tbx \in3\().16b, {v20.16b-v23.16b}, v11.16b
- sub v10.16b, v10.16b, v12.16b
+ sub v10.16b, v10.16b, v15.16b
tbx \in0\().16b, {v24.16b-v27.16b}, v8.16b
- sub v11.16b, v11.16b, v12.16b
+ sub v11.16b, v11.16b, v15.16b
tbx \in1\().16b, {v24.16b-v27.16b}, v9.16b
- sub v8.16b, v8.16b, v12.16b
+ sub v8.16b, v8.16b, v15.16b
tbx \in2\().16b, {v24.16b-v27.16b}, v10.16b
- sub v9.16b, v9.16b, v12.16b
+ sub v9.16b, v9.16b, v15.16b
tbx \in3\().16b, {v24.16b-v27.16b}, v11.16b
- sub v10.16b, v10.16b, v12.16b
+ sub v10.16b, v10.16b, v15.16b
tbx \in0\().16b, {v28.16b-v31.16b}, v8.16b
- sub v11.16b, v11.16b, v12.16b
+ sub v11.16b, v11.16b, v15.16b
tbx \in1\().16b, {v28.16b-v31.16b}, v9.16b
tbx \in2\().16b, {v28.16b-v31.16b}, v10.16b
tbx \in3\().16b, {v28.16b-v31.16b}, v11.16b
@@ -174,59 +166,30 @@
eor \out1\().16b, \out1\().16b, \tmp1\().16b
.endm
- .macro mix_columns_2x, in0, in1
- mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v14
- rev32 v10.8h, \in0\().8h
- rev32 v11.8h, \in1\().8h
- eor \in0\().16b, v8.16b, \in0\().16b
- eor \in1\().16b, v9.16b, \in1\().16b
- shl v12.4s, v10.4s, #24
- shl v13.4s, v11.4s, #24
- eor v8.16b, v8.16b, v10.16b
- sri v12.4s, v10.4s, #8
- shl v10.4s, \in0\().4s, #24
- eor v9.16b, v9.16b, v11.16b
- sri v13.4s, v11.4s, #8
- shl v11.4s, \in1\().4s, #24
- sri v10.4s, \in0\().4s, #8
- eor \in0\().16b, v8.16b, v12.16b
- sri v11.4s, \in1\().4s, #8
- eor \in1\().16b, v9.16b, v13.16b
- eor \in0\().16b, v10.16b, \in0\().16b
- eor \in1\().16b, v11.16b, \in1\().16b
- .endm
-
- .macro inv_mix_cols_2x, in0, in1
- mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v14
- mul_by_x_2x v8, v9, v8, v9, v10, v11, v14
+ .macro mix_columns_2x, in0, in1, enc
+ .if \enc == 0
+ /* Inverse MixColumns: pre-multiply by { 5, 0, 4, 0 } */
+ mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v12
+ mul_by_x_2x v8, v9, v8, v9, v10, v11, v12
eor \in0\().16b, \in0\().16b, v8.16b
- eor \in1\().16b, \in1\().16b, v9.16b
rev32 v8.8h, v8.8h
- rev32 v9.8h, v9.8h
- eor \in0\().16b, \in0\().16b, v8.16b
eor \in1\().16b, \in1\().16b, v9.16b
- mix_columns_2x \in0, \in1
- .endm
-
- .macro inv_mix_cols_4x, in0, in1, in2, in3
- mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v14
- mul_by_x_2x v10, v11, \in2, \in3, v12, v13, v14
- mul_by_x_2x v8, v9, v8, v9, v12, v13, v14
- mul_by_x_2x v10, v11, v10, v11, v12, v13, v14
- eor \in0\().16b, \in0\().16b, v8.16b
- eor \in1\().16b, \in1\().16b, v9.16b
- eor \in2\().16b, \in2\().16b, v10.16b
- eor \in3\().16b, \in3\().16b, v11.16b
- rev32 v8.8h, v8.8h
rev32 v9.8h, v9.8h
- rev32 v10.8h, v10.8h
- rev32 v11.8h, v11.8h
eor \in0\().16b, \in0\().16b, v8.16b
eor \in1\().16b, \in1\().16b, v9.16b
- eor \in2\().16b, \in2\().16b, v10.16b
- eor \in3\().16b, \in3\().16b, v11.16b
- mix_columns_2x \in0, \in1
- mix_columns_2x \in2, \in3
+ .endif
+
+ mul_by_x_2x v8, v9, \in0, \in1, v10, v11, v12
+ rev32 v10.8h, \in0\().8h
+ rev32 v11.8h, \in1\().8h
+ eor v10.16b, v10.16b, v8.16b
+ eor v11.16b, v11.16b, v9.16b
+ eor \in0\().16b, \in0\().16b, v10.16b
+ eor \in1\().16b, \in1\().16b, v11.16b
+ tbl \in0\().16b, {\in0\().16b}, v14.16b
+ tbl \in1\().16b, {\in1\().16b}, v14.16b
+ eor \in0\().16b, \in0\().16b, v10.16b
+ eor \in1\().16b, \in1\().16b, v11.16b
.endm
.macro do_block_2x, enc, in0, in1 rounds, rk, rkp, i
@@ -235,20 +198,14 @@
mov \i, \rounds
1111: eor \in0\().16b, \in0\().16b, v15.16b /* ^round key */
eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */
- sub_bytes_2x \in0, \in1
+ movi v15.16b, #0x40
tbl \in0\().16b, {\in0\().16b}, v13.16b /* ShiftRows */
tbl \in1\().16b, {\in1\().16b}, v13.16b /* ShiftRows */
- ld1 {v15.4s}, [\rkp], #16
+ sub_bytes_2x \in0, \in1
subs \i, \i, #1
+ ld1 {v15.4s}, [\rkp], #16
beq 2222f
- .if \enc == 1
- mix_columns_2x \in0, \in1
- ldr q13, .LForward_ShiftRows
- .else
- inv_mix_cols_2x \in0, \in1
- ldr q13, .LReverse_ShiftRows
- .endif
- movi v12.16b, #0x40
+ mix_columns_2x \in0, \in1, \enc
b 1111b
2222: eor \in0\().16b, \in0\().16b, v15.16b /* ^round key */
eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */
@@ -262,23 +219,17 @@
eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */
eor \in2\().16b, \in2\().16b, v15.16b /* ^round key */
eor \in3\().16b, \in3\().16b, v15.16b /* ^round key */
- sub_bytes_4x \in0, \in1, \in2, \in3
+ movi v15.16b, #0x40
tbl \in0\().16b, {\in0\().16b}, v13.16b /* ShiftRows */
tbl \in1\().16b, {\in1\().16b}, v13.16b /* ShiftRows */
tbl \in2\().16b, {\in2\().16b}, v13.16b /* ShiftRows */
tbl \in3\().16b, {\in3\().16b}, v13.16b /* ShiftRows */
- ld1 {v15.4s}, [\rkp], #16
+ sub_bytes_4x \in0, \in1, \in2, \in3
subs \i, \i, #1
+ ld1 {v15.4s}, [\rkp], #16
beq 2222f
- .if \enc == 1
- mix_columns_2x \in0, \in1
- mix_columns_2x \in2, \in3
- ldr q13, .LForward_ShiftRows
- .else
- inv_mix_cols_4x \in0, \in1, \in2, \in3
- ldr q13, .LReverse_ShiftRows
- .endif
- movi v12.16b, #0x40
+ mix_columns_2x \in0, \in1, \enc
+ mix_columns_2x \in2, \in3, \enc
b 1111b
2222: eor \in0\().16b, \in0\().16b, v15.16b /* ^round key */
eor \in1\().16b, \in1\().16b, v15.16b /* ^round key */
@@ -305,19 +256,7 @@
#include "aes-modes.S"
.text
- .align 4
-.LForward_ShiftRows:
-CPU_LE( .byte 0x0, 0x5, 0xa, 0xf, 0x4, 0x9, 0xe, 0x3 )
-CPU_LE( .byte 0x8, 0xd, 0x2, 0x7, 0xc, 0x1, 0x6, 0xb )
-CPU_BE( .byte 0xb, 0x6, 0x1, 0xc, 0x7, 0x2, 0xd, 0x8 )
-CPU_BE( .byte 0x3, 0xe, 0x9, 0x4, 0xf, 0xa, 0x5, 0x0 )
-
-.LReverse_ShiftRows:
-CPU_LE( .byte 0x0, 0xd, 0xa, 0x7, 0x4, 0x1, 0xe, 0xb )
-CPU_LE( .byte 0x8, 0x5, 0x2, 0xf, 0xc, 0x9, 0x6, 0x3 )
-CPU_BE( .byte 0x3, 0x6, 0x9, 0xc, 0xf, 0x2, 0x5, 0x8 )
-CPU_BE( .byte 0xb, 0xe, 0x1, 0x4, 0x7, 0xa, 0xd, 0x0 )
-
+ .align 6
.LForward_Sbox:
.byte 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5
.byte 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76
@@ -385,3 +324,12 @@ CPU_BE( .byte 0xb, 0xe, 0x1, 0x4, 0x7, 0xa, 0xd, 0x0 )
.byte 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61
.byte 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26
.byte 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d
+
+.LForward_ShiftRows:
+ .octa 0x0b06010c07020d08030e09040f0a0500
+
+.LReverse_ShiftRows:
+ .octa 0x0306090c0f0205080b0e0104070a0d00
+
+.Lror32by8:
+ .octa 0x0c0f0e0d080b0a090407060500030201
--
2.7.4
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox