* Re: [PATCH] crypto: aesni-intel - RFC4106 can zero copy when !PageHighMem
From: Herbert Xu @ 2016-12-27 10:05 UTC (permalink / raw)
To: Ilya Lesokhin; +Cc: linux-crypto, tadeusz.struk, davejwatson, tls-fpga-sw-dev
In-Reply-To: <1481639526-71743-1-git-send-email-ilyal@mellanox.com>
On Tue, Dec 13, 2016 at 04:32:06PM +0200, Ilya Lesokhin wrote:
> In the common case of !PageHighMem we can do zero copy crypto
> even if sg crosses a pages boundary.
>
> Signed-off-by: Ilya Lesokhin <ilyal@mellanox.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: bring back alphabetical order of Makefile
From: Herbert Xu @ 2016-12-27 10:05 UTC (permalink / raw)
To: Corentin Labbe; +Cc: davem, linux-crypto, linux-kernel
In-Reply-To: <20161213143059.13543-1-clabbe.montjoie@gmail.com>
On Tue, Dec 13, 2016 at 03:30:59PM +0100, Corentin Labbe wrote:
> THe major content of drivers/crypto/Makefile is sorted, only recent
> addition break this sort.
>
> This patch bring back this alphabetical sorting.
>
> Signed-off-by: Corentin Labbe <clabbe.montjoie@gmail.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v2 1/3] crypto: chacha20 - convert generic and x86 versions to skcipher
From: Herbert Xu @ 2016-12-27 10:04 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel
In-Reply-To: <1481294033-23508-2-git-send-email-ard.biesheuvel@linaro.org>
On Fri, Dec 09, 2016 at 02:33:51PM +0000, Ard Biesheuvel wrote:
> This converts the ChaCha20 code from a blkcipher to a skcipher, which
> is now the preferred way to implement symmetric block and stream ciphers.
>
> This ports the generic and x86 versions at the same time because the
> latter reuses routines of the former.
>
> Note that the skcipher_walk() API guarantees that all presented blocks
> except the final one are a multiple of the chunk size, so we can simplify
> the encrypt() routine somewhat.
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 0/2] crypto: arm64/ARM: NEON accelerated ChaCha20
From: Herbert Xu @ 2016-12-27 10:04 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel
In-Reply-To: <1481207339-17332-1-git-send-email-ard.biesheuvel@linaro.org>
On Thu, Dec 08, 2016 at 02:28:57PM +0000, Ard Biesheuvel wrote:
> Another port of existing x86 SSE code to NEON, again both for arm64 and ARM.
>
> ChaCha20 is a stream cipher described in RFC 7539, and is intended to be
> an efficient software implementable 'standby cipher', in case AES cannot
> be used.
>
> This NEON implementation is almost 2x as fast as the generic C code
> (measured on Cortex-A57 using the arm64 version)
>
> I'm aware that blkciphers are deprecated in favor of skciphers, but this
> code (like the x86 version) uses the init and setkey routines of the generic
> version, so it is probably better to port all implementations at once.
>
> Ard Biesheuvel (2):
> crypto: arm64/chacha20 - implement NEON version based on SSE3 code
> crypto: arm/chacha20 - implement NEON version based on SSE3 code
Both patches applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC PATCH 4.10 1/6] crypto/sha256: Refactor the API so it can be used without shash
From: Herbert Xu @ 2016-12-27 9:58 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Ard Biesheuvel, Andy Lutomirski, Daniel Borkmann, Netdev, LKML,
Linux Crypto Mailing List, Jason A. Donenfeld,
Hannes Frederic Sowa, Alexei Starovoitov, Eric Dumazet,
Eric Biggers, Tom Herbert, David S. Miller
In-Reply-To: <CALCETrXFTQ2AXgbQzPnRcDRHK81=FS1R0zqfoTqGjUsqZy2PvQ@mail.gmail.com>
On Mon, Dec 26, 2016 at 10:08:48AM -0800, Andy Lutomirski wrote:
>
> According to Daniel, the networking folks want to let embedded systems
> include BPF without requiring the crypto core.
Last I checked the IPv4 stack depended on the crypto API so this
sounds bogus.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [sparc64] cryptomgr_test OOPS kernel 4.9.0+
From: Herbert Xu @ 2016-12-27 9:52 UTC (permalink / raw)
To: David Miller; +Cc: matorola, linux-crypto, sparclinux, giovanni.cabiddu
In-Reply-To: <20161226.172619.1321397093716823099.davem@davemloft.net>
On Mon, Dec 26, 2016 at 05:26:19PM -0500, David Miller wrote:
> From: Anatoly Pugachev <matorola@gmail.com>
> Date: Sun, 25 Dec 2016 20:56:08 +0300
>
> > Disabling kernel config option
> > CRYPTO_MANAGER_DISABLE_TESTS
> > i.e. enable run-time self tests, makes kernel unbootable:
> >
> > tested with git kernels v4.9-8648-g5cc60aeedf31 and v4.9-12259-g7c0f6ba682b9
>
> I think the testing code for the new synchronous compression module is
> putting kernel image pointers into scatterlists, which in turn we
> attempt to transform to and from page structs.
>
> That doesn't work.
>
> It's coming from the test input buffers:
>
> static int test_acomp(struct crypto_acomp *tfm, struct comp_testvec *ctemplate,
> struct comp_testvec *dtemplate, int ctcount, int dtcount)
> {
> ...
> sg_init_one(&src, ctemplate[i].input, ilen);
>
> These have to be copied into kmalloc() buffers or similar, just like
> the skchiper tests do.
>
> The crash on sparc64 shows that we try to dereference a page struct at
> a bogus vmemmap address for a page that doesn't exist.
>
> I hacked up the following and this makes the crashes go away:
Thanks Dave. I've just applied the patch
https://patchwork.kernel.org/patch/9483763/
which should fix this.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Crypto Fixes for 4.10
From: Herbert Xu @ 2016-12-27 9:45 UTC (permalink / raw)
To: Linus Torvalds, David S. Miller, Linux Kernel Mailing List,
Linux Crypto Mailing List
In-Reply-To: <20161215160732.GA16580@gondor.apana.org.au>
Hi Linus:
This push fixes a hash corruption bug in the marvell driver.
Please pull from
git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6.git linus
Romain Perier (1):
crypto: marvell - Copy IVDIG before launching partial DMA ahash requests
drivers/crypto/marvell/cesa.h | 3 ++-
drivers/crypto/marvell/hash.c | 34 +++++++++++++++++++++++++++++++++-
drivers/crypto/marvell/tdma.c | 9 ++++++++-
3 files changed, 43 insertions(+), 3 deletions(-)
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC PATCH 0/3] Cavium ThunderX ZIP driver
From: Herbert Xu @ 2016-12-27 9:02 UTC (permalink / raw)
To: Jan Glauber
Cc: linux-crypto, linux-kernel, David S . Miller, Mahipal Challa,
Vishnu Nair
In-Reply-To: <20161212150439.18627-1-jglauber@cavium.com>
Hi Jan:
On Mon, Dec 12, 2016 at 04:04:36PM +0100, Jan Glauber wrote:
>
> this series adds support for hardware accelerated compression & decompression
> as found on ThunderX (arm64) SOCs. I've been reviewing this driver internally
> for some time and would like to get feedback on the RFC to see if this goes
> into the right direction and to see if there are any concerns.
>
> We've discussed switching to the new acomp algorithm but for the time being
> decided against acomp because our test cases are not yet supported with it.
OK. Do you see any major problems in converting this over to acomp?
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v2 7/7] hwrng: core: Remove linux/sched.h from includes
From: Herbert Xu @ 2016-12-27 9:14 UTC (permalink / raw)
To: Corentin Labbe; +Cc: mpm, arnd, gregkh, linux-crypto, linux-kernel
In-Reply-To: <20161213145115.30082-7-clabbe.montjoie@gmail.com>
On Tue, Dec 13, 2016 at 03:51:15PM +0100, Corentin Labbe wrote:
> linux/sched.h is useless for hw_random/core.c.
> This patch remove it.
I see a schedule_timeout_interruptible call in core.c.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: arm64/aes: reimplement bit-sliced ARM/NEON implementation for arm64
From: Herbert Xu @ 2016-12-27 9:03 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel, nico, will.deacon
In-Reply-To: <1481564758-7275-1-git-send-email-ard.biesheuvel@linaro.org>
On Mon, Dec 12, 2016 at 05:45:58PM +0000, Ard Biesheuvel wrote:
>
> + .chunksize = 8 * AES_BLOCK_SIZE,
Same comment as to the previous patches regarding chunksize.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v2 2/3] crypto: arm64/chacha20 - implement NEON version based on SSE3 code
From: Herbert Xu @ 2016-12-27 9:00 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel
In-Reply-To: <1481294033-23508-3-git-send-email-ard.biesheuvel@linaro.org>
On Fri, Dec 09, 2016 at 02:33:52PM +0000, Ard Biesheuvel wrote:
>
> + .chunksize = 4 * CHACHA20_BLOCK_SIZE,
This should use a new attribute specific to the walk interface.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: arm/aes-neonbs - process 8 blocks in parallel if we can
From: Herbert Xu @ 2016-12-27 8:57 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel
In-Reply-To: <1481291246-20216-1-git-send-email-ard.biesheuvel@linaro.org>
On Fri, Dec 09, 2016 at 01:47:26PM +0000, Ard Biesheuvel wrote:
> The bit-sliced NEON implementation of AES only performs optimally if
> it can process 8 blocks of input in parallel. This is due to the nature
> of bit slicing, where the n-th bit of each byte of AES state of each input
> block is collected into NEON register 'n', for registers q0 - q7.
>
> This implies that the amount of work for the transform is fixed,
> regardless of whether we are handling just one block or 8 in parallel.
>
> So let's try a bit harder to iterate over the input in suitably sized
> chunks, by increasing the chunksize to 8 * AES_BLOCK_SIZE, and tweaking
> the loops to only process multiples of the chunk size, unless we are
> handling the last chunk in the input stream.
>
> Note that the skcipher walk API guarantees that a step in the walk never
> returns less that 'chunksize' bytes if there are at least that many bytes
> of input still available. However, it does *not* guarantee that those steps
> produce an exact multiple of the chunk size.
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
I like this patch. However, I had different plans for the chunksize
attribute. It's primarily meant to be a hint to the upper layer
in case it does partial updates. It's meant to provide the minimum
number of bytes a partial update can carry without screwing up
subsequent updates.
It just happens to be the same value that we were using during
an skcipher walk.
So I think for your case we should add a new attribute, perhaps
walk_chunksize or walksize, which doesn't need to be exported to
the outside at all and can then be used by the walk interface.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [PATCH] virtio-crypto: support crypto engine framework
From: Gonglei @ 2016-12-27 6:49 UTC (permalink / raw)
To: linux-kernel, virtualization, linux-crypto
Cc: Herbert Xu, Michael S . Tsirkin, Baolin Wang, longpeng2, wu.wubin
crypto engine was introduced since 'commit 735d37b5424b ("crypto: engine
- Introduce the block request crypto engine framework")' which uses work
queue to realize the asynchronous processing for ablk_cipher and ahash.
For virtio-crypto device, I register an engine for each
data virtqueue so that we can use the capability of
multiple data queues in future.
Cc: Baolin Wang <baolin.wang@linaro.org>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
drivers/crypto/virtio/Kconfig | 1 +
drivers/crypto/virtio/virtio_crypto_algs.c | 52 ++++++++++++-------
drivers/crypto/virtio/virtio_crypto_common.h | 16 ++++++
drivers/crypto/virtio/virtio_crypto_core.c | 74 ++++++++++++++++++++++++++--
4 files changed, 121 insertions(+), 22 deletions(-)
diff --git a/drivers/crypto/virtio/Kconfig b/drivers/crypto/virtio/Kconfig
index d80f733..5db0749 100644
--- a/drivers/crypto/virtio/Kconfig
+++ b/drivers/crypto/virtio/Kconfig
@@ -4,6 +4,7 @@ config CRYPTO_DEV_VIRTIO
select CRYPTO_AEAD
select CRYPTO_AUTHENC
select CRYPTO_BLKCIPHER
+ select CRYPTO_ENGINE
default m
help
This driver provides support for virtio crypto device. If you
diff --git a/drivers/crypto/virtio/virtio_crypto_algs.c b/drivers/crypto/virtio/virtio_crypto_algs.c
index c2374df..970d0ca 100644
--- a/drivers/crypto/virtio/virtio_crypto_algs.c
+++ b/drivers/crypto/virtio/virtio_crypto_algs.c
@@ -288,8 +288,7 @@ static int virtio_crypto_ablkcipher_setkey(struct crypto_ablkcipher *tfm,
static int
__virtio_crypto_ablkcipher_do_req(struct virtio_crypto_request *vc_req,
struct ablkcipher_request *req,
- struct data_queue *data_vq,
- __u8 op)
+ struct data_queue *data_vq)
{
struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
unsigned int ivsize = crypto_ablkcipher_ivsize(tfm);
@@ -329,7 +328,7 @@ static int virtio_crypto_ablkcipher_setkey(struct crypto_ablkcipher *tfm,
vc_req->req_data = req_data;
vc_req->type = VIRTIO_CRYPTO_SYM_OP_CIPHER;
/* Head of operation */
- if (op) {
+ if (vc_req->encrypt) {
req_data->header.session_id =
cpu_to_le64(ctx->enc_sess_info.session_id);
req_data->header.opcode =
@@ -424,19 +423,15 @@ static int virtio_crypto_ablkcipher_encrypt(struct ablkcipher_request *req)
struct virtio_crypto_ablkcipher_ctx *ctx = crypto_ablkcipher_ctx(atfm);
struct virtio_crypto_request *vc_req = ablkcipher_request_ctx(req);
struct virtio_crypto *vcrypto = ctx->vcrypto;
- int ret;
/* Use the first data virtqueue as default */
struct data_queue *data_vq = &vcrypto->data_vq[0];
vc_req->ablkcipher_ctx = ctx;
vc_req->ablkcipher_req = req;
- ret = __virtio_crypto_ablkcipher_do_req(vc_req, req, data_vq, 1);
- if (ret < 0) {
- pr_err("virtio_crypto: Encryption failed!\n");
- return ret;
- }
+ vc_req->encrypt = true;
+ vc_req->dataq = data_vq;
- return -EINPROGRESS;
+ return crypto_transfer_cipher_request_to_engine(data_vq->engine, req);
}
static int virtio_crypto_ablkcipher_decrypt(struct ablkcipher_request *req)
@@ -445,20 +440,16 @@ static int virtio_crypto_ablkcipher_decrypt(struct ablkcipher_request *req)
struct virtio_crypto_ablkcipher_ctx *ctx = crypto_ablkcipher_ctx(atfm);
struct virtio_crypto_request *vc_req = ablkcipher_request_ctx(req);
struct virtio_crypto *vcrypto = ctx->vcrypto;
- int ret;
/* Use the first data virtqueue as default */
struct data_queue *data_vq = &vcrypto->data_vq[0];
vc_req->ablkcipher_ctx = ctx;
vc_req->ablkcipher_req = req;
- ret = __virtio_crypto_ablkcipher_do_req(vc_req, req, data_vq, 0);
- if (ret < 0) {
- pr_err("virtio_crypto: Decryption failed!\n");
- return ret;
- }
+ vc_req->encrypt = false;
+ vc_req->dataq = data_vq;
- return -EINPROGRESS;
+ return crypto_transfer_cipher_request_to_engine(data_vq->engine, req);
}
static int virtio_crypto_ablkcipher_init(struct crypto_tfm *tfm)
@@ -484,6 +475,33 @@ static void virtio_crypto_ablkcipher_exit(struct crypto_tfm *tfm)
ctx->vcrypto = NULL;
}
+int virtio_crypto_ablkcipher_crypt_req(
+ struct crypto_engine *engine,
+ struct ablkcipher_request *req)
+{
+ struct virtio_crypto_request *vc_req = ablkcipher_request_ctx(req);
+ struct data_queue *data_vq = vc_req->dataq;
+ int ret;
+
+ ret = __virtio_crypto_ablkcipher_do_req(vc_req, req, data_vq);
+ if (ret < 0)
+ return ret;
+
+ virtqueue_kick(data_vq->vq);
+
+ return 0;
+}
+
+void virtio_crypto_ablkcipher_finalize_req(
+ struct virtio_crypto_request *vc_req,
+ struct ablkcipher_request *req,
+ int err)
+{
+ crypto_finalize_cipher_request(vc_req->dataq->engine, req, err);
+
+ virtcrypto_clear_request(vc_req);
+}
+
static struct crypto_alg virtio_crypto_algs[] = { {
.cra_name = "cbc(aes)",
.cra_driver_name = "virtio_crypto_aes_cbc",
diff --git a/drivers/crypto/virtio/virtio_crypto_common.h b/drivers/crypto/virtio/virtio_crypto_common.h
index 3d6566b..da6d8c0 100644
--- a/drivers/crypto/virtio/virtio_crypto_common.h
+++ b/drivers/crypto/virtio/virtio_crypto_common.h
@@ -25,6 +25,7 @@
#include <crypto/aead.h>
#include <crypto/aes.h>
#include <crypto/authenc.h>
+#include <crypto/engine.h>
/* Internal representation of a data virtqueue */
@@ -37,6 +38,8 @@ struct data_queue {
/* Name of the tx queue: dataq.$index */
char name[32];
+
+ struct crypto_engine *engine;
};
struct virtio_crypto {
@@ -97,6 +100,9 @@ struct virtio_crypto_request {
struct virtio_crypto_op_data_req *req_data;
struct scatterlist **sgs;
uint8_t *iv;
+ /* Encryption? */
+ bool encrypt;
+ struct data_queue *dataq;
};
int virtcrypto_devmgr_add_dev(struct virtio_crypto *vcrypto_dev);
@@ -110,6 +116,16 @@ struct virtio_crypto_request {
struct virtio_crypto *virtcrypto_get_dev_node(int node);
int virtcrypto_dev_start(struct virtio_crypto *vcrypto);
void virtcrypto_dev_stop(struct virtio_crypto *vcrypto);
+int virtio_crypto_ablkcipher_crypt_req(
+ struct crypto_engine *engine,
+ struct ablkcipher_request *req);
+void virtio_crypto_ablkcipher_finalize_req(
+ struct virtio_crypto_request *vc_req,
+ struct ablkcipher_request *req,
+ int err);
+
+void
+virtcrypto_clear_request(struct virtio_crypto_request *vc_req);
static inline int virtio_crypto_get_current_node(void)
{
diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
index fe70ec8..b5b1533 100644
--- a/drivers/crypto/virtio/virtio_crypto_core.c
+++ b/drivers/crypto/virtio/virtio_crypto_core.c
@@ -25,7 +25,7 @@
#include "virtio_crypto_common.h"
-static void
+void
virtcrypto_clear_request(struct virtio_crypto_request *vc_req)
{
if (vc_req) {
@@ -66,12 +66,12 @@ static void virtcrypto_dataq_callback(struct virtqueue *vq)
break;
}
ablk_req = vc_req->ablkcipher_req;
- virtcrypto_clear_request(vc_req);
spin_unlock_irqrestore(
&vcrypto->data_vq[qid].lock, flags);
/* Finish the encrypt or decrypt process */
- ablk_req->base.complete(&ablk_req->base, error);
+ virtio_crypto_ablkcipher_finalize_req(vc_req,
+ ablk_req, error);
spin_lock_irqsave(
&vcrypto->data_vq[qid].lock, flags);
}
@@ -87,6 +87,7 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi)
int ret = -ENOMEM;
int i, total_vqs;
const char **names;
+ struct device *dev = &vi->vdev->dev;
/*
* We expect 1 data virtqueue, followed by
@@ -128,6 +129,15 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi)
for (i = 0; i < vi->max_data_queues; i++) {
spin_lock_init(&vi->data_vq[i].lock);
vi->data_vq[i].vq = vqs[i];
+ /* Initialize crypto engine */
+ vi->data_vq[i].engine = crypto_engine_alloc_init(dev, 1);
+ if (!vi->data_vq[i].engine) {
+ ret = -ENOMEM;
+ goto err_engine;
+ }
+
+ vi->data_vq[i].engine->cipher_one_request =
+ virtio_crypto_ablkcipher_crypt_req;
}
kfree(names);
@@ -136,6 +146,7 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi)
return 0;
+err_engine:
err_find:
kfree(names);
err_names:
@@ -269,6 +280,38 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
return 0;
}
+static int virtcrypto_start_crypto_engines(struct virtio_crypto *vcrypto)
+{
+ int32_t i;
+ int ret;
+
+ for (i = 0; i < vcrypto->max_data_queues; i++) {
+ if (vcrypto->data_vq[i].engine) {
+ ret = crypto_engine_start(vcrypto->data_vq[i].engine);
+ if (ret)
+ goto err;
+ }
+ }
+
+ return 0;
+
+err:
+ while (--i >= 0)
+ if (vcrypto->data_vq[i].engine)
+ crypto_engine_exit(vcrypto->data_vq[i].engine);
+
+ return ret;
+}
+
+static void virtcrypto_clear_crypto_engines(struct virtio_crypto *vcrypto)
+{
+ u32 i;
+
+ for (i = 0; i < vcrypto->max_data_queues; i++)
+ if (vcrypto->data_vq[i].engine)
+ crypto_engine_exit(vcrypto->data_vq[i].engine);
+}
+
static void virtcrypto_del_vqs(struct virtio_crypto *vcrypto)
{
struct virtio_device *vdev = vcrypto->vdev;
@@ -355,14 +398,21 @@ static int virtcrypto_probe(struct virtio_device *vdev)
dev_err(&vdev->dev, "Failed to initialize vqs.\n");
goto free_dev;
}
+
+ err = virtcrypto_start_crypto_engines(vcrypto);
+ if (err)
+ goto free_vqs;
+
virtio_device_ready(vdev);
err = virtcrypto_update_status(vcrypto);
if (err)
- goto free_vqs;
+ goto free_engines;
return 0;
+free_engines:
+ virtcrypto_clear_crypto_engines(vcrypto);
free_vqs:
vcrypto->vdev->config->reset(vdev);
virtcrypto_del_vqs(vcrypto);
@@ -398,6 +448,7 @@ static void virtcrypto_remove(struct virtio_device *vdev)
virtcrypto_dev_stop(vcrypto);
vdev->config->reset(vdev);
virtcrypto_free_unused_reqs(vcrypto);
+ virtcrypto_clear_crypto_engines(vcrypto);
virtcrypto_del_vqs(vcrypto);
virtcrypto_devmgr_rm_dev(vcrypto);
kfree(vcrypto);
@@ -420,6 +471,7 @@ static int virtcrypto_freeze(struct virtio_device *vdev)
if (virtcrypto_dev_started(vcrypto))
virtcrypto_dev_stop(vcrypto);
+ virtcrypto_clear_crypto_engines(vcrypto);
virtcrypto_del_vqs(vcrypto);
return 0;
}
@@ -433,14 +485,26 @@ static int virtcrypto_restore(struct virtio_device *vdev)
if (err)
return err;
+ err = virtcrypto_start_crypto_engines(vcrypto);
+ if (err)
+ goto free_vqs;
+
virtio_device_ready(vdev);
+
err = virtcrypto_dev_start(vcrypto);
if (err) {
dev_err(&vdev->dev, "Failed to start virtio crypto device.\n");
- return -EFAULT;
+ goto free_engines;
}
return 0;
+
+free_engines:
+ virtcrypto_clear_crypto_engines(vcrypto);
+free_vqs:
+ vcrypto->vdev->config->reset(vdev);
+ virtcrypto_del_vqs(vcrypto);
+ return err;
}
#endif
--
1.8.3.1
^ permalink raw reply related
* Re: [RFC PATCH 4.10 3/6] bpf: Use SHA256 instead of SHA1 for bpf digests
From: Andy Lutomirski @ 2016-12-27 2:08 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Daniel Borkmann, Andy Lutomirski, Netdev, LKML,
Linux Crypto Mailing List, Jason A. Donenfeld,
Hannes Frederic Sowa, Eric Dumazet, Eric Biggers, Tom Herbert,
David S. Miller, Alexei Starovoitov
In-Reply-To: <20161227013644.GA96815@ast-mbp.thefacebook.com>
On Mon, Dec 26, 2016 at 5:36 PM, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
> On Sat, Dec 24, 2016 at 08:59:53PM +0100, Daniel Borkmann wrote:
>> On 12/24/2016 03:22 AM, Andy Lutomirski wrote:
>> >BPF digests are intended to be used to avoid reloading programs that
>> >are already loaded. For use cases (CRIU?) where untrusted programs
>> >are involved, intentional hash collisions could cause the wrong BPF
>> >program to execute. Additionally, if BPF digests are ever used
>> >in-kernel to skip verification, a hash collision could give privilege
>> >escalation directly.
>>
>> Just for the record, digests will never ever be used to skip the
>> verification step, so I don't know why this idea even comes up
>> here (?) or is part of the changelog? As this will never be done
>> anyway, rather drop that part so we can avoid confusion on this?
>
> +1 to what Daniel said above.
>
> For the others let me explain what this patch set is actually
> trying to accomplish.
The patch:
a) cleans up the code and
b) uses a cryptographic hash that is actually believed to satisfy the
definition of a cryptographic hash.
There's no excuse for not doing b.
> and I have an obvious NACK for bpf related patches 3,4,5,6.
Did you *read* the ones that were pure cleanups?
>
> sha1 is 20 bytes which is already a bit long to print and copy paste by humans.
> whereas 4 byte jhash is a bit too short, since collisions are not that rare
> and may lead to incorrect assumptions from the users that develop the programs.
> I would prefer something in 6-10 byte range that prevents collisions most of
> the time and short to print as hex, but I don't know of anything like this
> in the existing kernel and inventing bpf specific hash is not great.
> Another requirement for debugging (and prog_digest) that user space
> should be able to produce the same hash without asking kernel, so
> sha1 fits that as well, since it's well known and easy to put into library.
Then truncate them in user space.
^ permalink raw reply
* Re: [RFC PATCH 4.10 3/6] bpf: Use SHA256 instead of SHA1 for bpf digests
From: Alexei Starovoitov @ 2016-12-27 1:36 UTC (permalink / raw)
To: Daniel Borkmann
Cc: Andy Lutomirski, Netdev, LKML, Linux Crypto Mailing List,
Jason A. Donenfeld, Hannes Frederic Sowa, Eric Dumazet,
Eric Biggers, Tom Herbert, David S. Miller, Alexei Starovoitov
In-Reply-To: <585ED3B9.6020407@iogearbox.net>
On Sat, Dec 24, 2016 at 08:59:53PM +0100, Daniel Borkmann wrote:
> On 12/24/2016 03:22 AM, Andy Lutomirski wrote:
> >BPF digests are intended to be used to avoid reloading programs that
> >are already loaded. For use cases (CRIU?) where untrusted programs
> >are involved, intentional hash collisions could cause the wrong BPF
> >program to execute. Additionally, if BPF digests are ever used
> >in-kernel to skip verification, a hash collision could give privilege
> >escalation directly.
>
> Just for the record, digests will never ever be used to skip the
> verification step, so I don't know why this idea even comes up
> here (?) or is part of the changelog? As this will never be done
> anyway, rather drop that part so we can avoid confusion on this?
+1 to what Daniel said above.
For the others let me explain what this patch set is actually
trying to accomplish.
Andy had an idea that sha256 of the program can somehow be used
to bypass kernel verifier during program loading. Furthemore
he thinks that such 'bypass' would be useful for criu of bpf programs,
hence see vigorously attacking existing prog_digest (sha1) because
it's not as secure as sha256 and hence cannot be used for such 'bypass'.
The problem with criu of bpf programs is same as criu of kernel modules.
For the main tracing and networking use cases, we cannot stop the kernel,
so criu is out of question already.
Even if we could stop all the events that trigger bpf program execution,
the sha256 or memcmp() of the full program is not enough to guarantee
that two programs are the same.
Ex. bpf_map_lookup() may be safe for one program and not for another
depending on how map was created. Two programs of different types
are not comparable either. Etc, etc.
Therefore the idea of using sha256 for such purpose is bogus,
and I have an obvious NACK for bpf related patches 3,4,5,6.
For the questions raised in other threads:
I'm not ok with making BPF depend on CRYPTO, since it's the same as
requiring CRYPTO to select BPF for no good reason.
And 0/6 commit log:
> Since there are plenty of uses for the new-in-4.10 BPF digest feature
> that would be problematic if malicious users could produce collisions,
> the BPF digest should be collision-resistant.
This statement is also bogus. The only reason we added prog_digest is
to improve debuggability and introspection of bpf programs.
As I said in the previous thread "collisions are ok" and we could have
used jhash here to avoid patches like this ever appearing
and wasting everyones time.
sha1 is 20 bytes which is already a bit long to print and copy paste by humans.
whereas 4 byte jhash is a bit too short, since collisions are not that rare
and may lead to incorrect assumptions from the users that develop the programs.
I would prefer something in 6-10 byte range that prevents collisions most of
the time and short to print as hex, but I don't know of anything like this
in the existing kernel and inventing bpf specific hash is not great.
Another requirement for debugging (and prog_digest) that user space
should be able to produce the same hash without asking kernel, so
sha1 fits that as well, since it's well known and easy to put into library.
sha256 doesn't fit either of these requirements. 32-bytes are too long to print
and when we use it as a substitue for the prog name for jited ksym, looking
at long function names will screw up all tools like perf, which we don't
want. sha256 is equally not easy for user space app like iproute2,
so not an acceptable choice from that pov either.
^ permalink raw reply
* Re: [sparc64] cryptomgr_test OOPS kernel 4.9.0+
From: David Miller @ 2016-12-26 22:26 UTC (permalink / raw)
To: matorola; +Cc: linux-crypto, sparclinux, herbert, giovanni.cabiddu
In-Reply-To: <CADxRZqwxU+XjTe+jwfrDdsp7Vs47YsJHww3=bsENhrUsYgHvkg@mail.gmail.com>
From: Anatoly Pugachev <matorola@gmail.com>
Date: Sun, 25 Dec 2016 20:56:08 +0300
> Disabling kernel config option
> CRYPTO_MANAGER_DISABLE_TESTS
> i.e. enable run-time self tests, makes kernel unbootable:
>
> tested with git kernels v4.9-8648-g5cc60aeedf31 and v4.9-12259-g7c0f6ba682b9
I think the testing code for the new synchronous compression module is
putting kernel image pointers into scatterlists, which in turn we
attempt to transform to and from page structs.
That doesn't work.
It's coming from the test input buffers:
static int test_acomp(struct crypto_acomp *tfm, struct comp_testvec *ctemplate,
struct comp_testvec *dtemplate, int ctcount, int dtcount)
{
...
sg_init_one(&src, ctemplate[i].input, ilen);
These have to be copied into kmalloc() buffers or similar, just like
the skchiper tests do.
The crash on sparc64 shows that we try to dereference a page struct at
a bogus vmemmap address for a page that doesn't exist.
I hacked up the following and this makes the crashes go away:
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index f616ad7..117bb33 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -1449,22 +1449,31 @@ static int test_acomp(struct crypto_acomp *tfm, struct comp_testvec *ctemplate,
const char *algo = crypto_tfm_alg_driver_name(crypto_acomp_tfm(tfm));
unsigned int i;
char *output;
+ char *input;
int ret;
struct scatterlist src, dst;
struct acomp_req *req;
struct tcrypt_result result;
+ pr_info("test_acomp: COMP_BUF_SIZE %d\n", (int) COMP_BUF_SIZE);
+
output = kmalloc(COMP_BUF_SIZE, GFP_KERNEL);
if (!output)
return -ENOMEM;
+ input = kmalloc(COMP_BUF_SIZE, GFP_KERNEL);
+ if (!input) {
+ kfree(output);
+ return -ENOMEM;
+ }
for (i = 0; i < ctcount; i++) {
unsigned int dlen = COMP_BUF_SIZE;
int ilen = ctemplate[i].inlen;
memset(output, 0, dlen);
+ memcpy(input, ctemplate[i].input, ilen);
init_completion(&result.completion);
- sg_init_one(&src, ctemplate[i].input, ilen);
+ sg_init_one(&src, input, ilen);
sg_init_one(&dst, output, dlen);
req = acomp_request_alloc(tfm);
@@ -1512,8 +1521,9 @@ static int test_acomp(struct crypto_acomp *tfm, struct comp_testvec *ctemplate,
int ilen = dtemplate[i].inlen;
memset(output, 0, dlen);
+ memcpy(input, dtemplate[i].input, ilen);
init_completion(&result.completion);
- sg_init_one(&src, dtemplate[i].input, ilen);
+ sg_init_one(&src, input, ilen);
sg_init_one(&dst, output, dlen);
req = acomp_request_alloc(tfm);
@@ -1559,6 +1569,7 @@ static int test_acomp(struct crypto_acomp *tfm, struct comp_testvec *ctemplate,
ret = 0;
out:
+ kfree(input);
kfree(output);
return ret;
}
^ permalink raw reply related
* Re: [sparc64] cryptomgr_test OOPS kernel 4.9.0+
From: David Miller @ 2016-12-26 20:59 UTC (permalink / raw)
To: matorola; +Cc: linux-crypto, sparclinux
In-Reply-To: <CADxRZqwxU+XjTe+jwfrDdsp7Vs47YsJHww3=bsENhrUsYgHvkg@mail.gmail.com>
From: Anatoly Pugachev <matorola@gmail.com>
Date: Sun, 25 Dec 2016 20:56:08 +0300
> Disabling kernel config option
> CRYPTO_MANAGER_DISABLE_TESTS
> i.e. enable run-time self tests, makes kernel unbootable:
>
> tested with git kernels v4.9-8648-g5cc60aeedf31 and v4.9-12259-g7c0f6ba682b9
I'm getting this with the current GIT tree too, will try to see
what's going wrong.
^ permalink raw reply
* Re: [RFC PATCH 4.10 1/6] crypto/sha256: Refactor the API so it can be used without shash
From: Andy Lutomirski @ 2016-12-26 18:08 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Herbert Xu, Andy Lutomirski, Daniel Borkmann, Netdev, LKML,
Linux Crypto Mailing List, Jason A. Donenfeld,
Hannes Frederic Sowa, Alexei Starovoitov, Eric Dumazet,
Eric Biggers, Tom Herbert, David S. Miller
In-Reply-To: <CAKv+Gu-NMNCEs61f4k7JSf9iSJ1A_Gy1r=kZRGqtbDsEDz7--Q@mail.gmail.com>
On Mon, Dec 26, 2016 at 9:51 AM, Ard Biesheuvel
<ard.biesheuvel@linaro.org> wrote:
> On 26 December 2016 at 07:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>> On Sat, Dec 24, 2016 at 09:57:53AM -0800, Andy Lutomirski wrote:
>>>
>>> I actually do use incremental hashing later on. BPF currently
>>> vmallocs() a big temporary buffer just so it can fill it and hash it.
>>> I change it to hash as it goes.
>>
>> How much data is this supposed to hash on average? If it's a large
>> amount then perhaps using the existing crypto API would be a better
>> option than adding this.
>>
>
> This is a good point actually: you didn't explain *why* BPF shouldn't
> depend on the crypto API.
According to Daniel, the networking folks want to let embedded systems
include BPF without requiring the crypto core.
At some point, I'd also like to use modern hash functions for module
verification. If doing so would require the crypto core to be
available when modules are loaded, then the crypto core couldn't be
modular. (Although it occurs to me that my patches get that wrong --
if this change happens, I need to split the code so that the library
functions can be built in even if CRYPTO=m.)
Daniel, would you be okay with BPF selecting CRYPTO and CRYPTO_HASH?
Also, as a bikeshed thought: I could call the functions
sha256_init_direct(), etc. Then there wouldn't be namespace
collisions and the fact that they bypass accelerated drivers would be
more obvious.
--Andy
^ permalink raw reply
* Re: [RFC PATCH 4.10 1/6] crypto/sha256: Refactor the API so it can be used without shash
From: Ard Biesheuvel @ 2016-12-26 17:51 UTC (permalink / raw)
To: Herbert Xu
Cc: Andy Lutomirski, Andy Lutomirski, Daniel Borkmann, Netdev, LKML,
Linux Crypto Mailing List, Jason A. Donenfeld,
Hannes Frederic Sowa, Alexei Starovoitov, Eric Dumazet,
Eric Biggers, Tom Herbert, David S. Miller
In-Reply-To: <20161226075757.GA8916@gondor.apana.org.au>
On 26 December 2016 at 07:57, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Sat, Dec 24, 2016 at 09:57:53AM -0800, Andy Lutomirski wrote:
>>
>> I actually do use incremental hashing later on. BPF currently
>> vmallocs() a big temporary buffer just so it can fill it and hash it.
>> I change it to hash as it goes.
>
> How much data is this supposed to hash on average? If it's a large
> amount then perhaps using the existing crypto API would be a better
> option than adding this.
>
This is a good point actually: you didn't explain *why* BPF shouldn't
depend on the crypto API.
^ permalink raw reply
* Re: [RFC PATCH 4.10 0/6] Switch BPF's digest to SHA256
From: Herbert Xu @ 2016-12-26 8:20 UTC (permalink / raw)
To: Andy Lutomirski
Cc: daniel, netdev, linux-kernel, linux-crypto, Jason, hannes,
alexei.starovoitov, edumazet, ebiggers3, tom, davem, luto
In-Reply-To: <cover.1482545792.git.luto@kernel.org>
Andy Lutomirski <luto@kernel.org> wrote:
> Since there are plenty of uses for the new-in-4.10 BPF digest feature
> that would be problematic if malicious users could produce collisions,
> the BPF digest should be collision-resistant. SHA-1 is no longer
> considered collision-resistant, so switch it to SHA-256.
>
> The actual switchover is trivial. Most of this series consists of
> cleanups to the SHA256 code to make it usable as a standalone library
> (since BPF should not depend on crypto).
>
> The cleaned up library is much more user-friendly than the SHA-1 code,
> so this also significantly tidies up the BPF digest code.
>
> This is intended for 4.10. If this series misses 4.10 and nothing
> takes its place, then we'll have an unpleasant ABI stability
> situation.
Can you please explain why BPF needs to be able to use SHA directly
rather than through the crypto API?
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC PATCH 4.10 1/6] crypto/sha256: Refactor the API so it can be used without shash
From: Herbert Xu @ 2016-12-26 7:57 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Ard Biesheuvel, Andy Lutomirski, Daniel Borkmann, Netdev, LKML,
Linux Crypto Mailing List, Jason A. Donenfeld,
Hannes Frederic Sowa, Alexei Starovoitov, Eric Dumazet,
Eric Biggers, Tom Herbert, David S. Miller
In-Reply-To: <CALCETrVbT_1=cdU1+a-+KbhoFqeT3XvbHjY0s_U7C5JVgiPx_Q@mail.gmail.com>
On Sat, Dec 24, 2016 at 09:57:53AM -0800, Andy Lutomirski wrote:
>
> I actually do use incremental hashing later on. BPF currently
> vmallocs() a big temporary buffer just so it can fill it and hash it.
> I change it to hash as it goes.
How much data is this supposed to hash on average? If it's a large
amount then perhaps using the existing crypto API would be a better
option than adding this.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [sparc64] cryptomgr_test OOPS kernel 4.9.0+
From: Anatoly Pugachev @ 2016-12-25 17:56 UTC (permalink / raw)
To: linux-crypto; +Cc: Sparc kernel list
Hello!
Disabling kernel config option
CRYPTO_MANAGER_DISABLE_TESTS
i.e. enable run-time self tests, makes kernel unbootable:
tested with git kernels v4.9-8648-g5cc60aeedf31 and v4.9-12259-g7c0f6ba682b9
SILO Version 1.4.14
boot:
Allocated 64 Megs of memory at 0x40000000 for kernel
Uncompressing image...
Loaded kernel version 4.9.0
Loading initial ramdisk (14000758 bytes at 0x74000000 phys, 0x40C00000 virt)...
/
[ 0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 4.38.5 2016/06/22 19:36'
[ 0.000000] PROMLIB: Root node compatible: sun4v
[ 0.000000] Linux version 4.9.0+ (mator@ttip) (gcc version 6.2.1
20161215 (Debian 6.2.1-7) ) #38 SMP Sun Dec 25 13:35:48 MSK 2016
[ 0.000000] debug: skip boot console de-registration.
[ 0.000000] bootconsole [earlyprom0] enabled
[ 0.000000] ARCH: SUN4V
[ 0.000000] Ethernet address: 00:14:4f:fa:06:f2
[ 0.000000] MM: PAGE_OFFSET is 0xfff8000000000000 (max_phys_bits == 47)
[ 0.000000] MM: VMALLOC [0x0000000100000000 --> 0x0006000000000000]
[ 0.000000] MM: VMEMMAP [0x0006000000000000 --> 0x000c000000000000]
[ 0.000000] Kernel: Using 5 locked TLB entries for main kernel image.
[ 0.000000] Remapping the kernel...
[ 0.000000] done.
[ 0.000000] kmemleak: Kernel memory leak detector disabled
[ 0.000000] OF stdout device is: /virtual-devices@100/console@1
[ 0.000000] PROM: Built device tree with 85327 bytes of memory.
[ 0.000000] MDESC: Size is 35552 bytes.
[ 0.000000] PLATFORM: banner-name [SPARC T5-2]
[ 0.000000] PLATFORM: name [ORCL,SPARC-T5-2]
[ 0.000000] PLATFORM: hostid [84fa06f2]
[ 0.000000] PLATFORM: serial# [0035260e]
[ 0.000000] PLATFORM: stick-frequency [3b9aca00]
[ 0.000000] PLATFORM: mac-address [144ffa06f2]
[ 0.000000] PLATFORM: watchdog-resolution [1000 ms]
[ 0.000000] PLATFORM: watchdog-max-timeout [31536000000 ms]
[ 0.000000] PLATFORM: max-cpus [1024]
[ 0.000000] Top of RAM: 0x82f93a000, Total RAM: 0x7ff350000
[ 0.000000] Memory hole size: 773MB
[ 0.000000] Allocated 24576 bytes for kernel page tables.
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000030400000-0x000000082f939fff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000030400000-0x000000006fe7ffff]
[ 0.000000] node 0: [mem 0x000000006ff00000-0x000000006ff25fff]
[ 0.000000] node 0: [mem 0x0000000070000000-0x000000082f87ffff]
[ 0.000000] node 0: [mem 0x000000082f900000-0x000000082f921fff]
[ 0.000000] node 0: [mem 0x000000082f932000-0x000000082f939fff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000030400000-0x000000082f939fff]
[ 0.000000] Booting Linux...
[ 0.000000] CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32]
[ 0.000000] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit,fmaf,vis3]
[ 0.000000] CPU CAPS: [hpc,ima,pause,cbcond,aes,des,kasumi,camellia]
[ 0.000000] CPU CAPS: [md5,sha1,sha256,sha512,mpmul,montmul,montsqr,crc32c]
[ 0.000000] percpu: Embedded 11 pages/cpu @fff800082d000000 s46024
r8192 d35896 u131072
[ 0.000000] SUN4V: Mondo queue sizes [cpu(131072) dev(16384) r(8192) nr(256)]
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on.
Total pages: 4155828
[ 0.000000] Kernel command line: root=/dev/vdiska2 ro
zswap.enabled=1 keep_bootcon noresume
[ 0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[ 0.000000] log_buf_len total cpu_extra contributions: 1044480 bytes
[ 0.000000] log_buf_len min size: 131072 bytes
[ 0.000000] log_buf_len: 2097152 bytes
[ 0.000000] early log buf free: 126208(96%)
[ 0.000000] PID hash table entries: 4096 (order: 2, 32768 bytes)
[ 0.000000] Dentry cache hash table entries: 4194304 (order: 12,
33554432 bytes)
[ 0.000000] Inode-cache hash table entries: 2097152 (order: 11,
16777216 bytes)
[ 0.000000] Sorting __ex_table...
[ 0.000000] Memory: 33114224K/33541440K available (6603K kernel
code, 894K rwdata, 1824K rodata, 608K init, 9985K bss, 427216K
reserved, 0K cma-reserved)
[ 0.000000] Running RCU self tests
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU lockdep checking is enabled.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] NR_IRQS:2048 nr_irqs:2048 1
[ 0.000000] SUN4V: Using IRQ API major 3, cookie only virqs enabled
[11059882.082988] clocksource: stick: mask: 0xffffffffffffffff
max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[11059882.083097] clocksource: mult[800000] shift[23]
[11059882.083148] clockevent: mult[80000000] shift[31]
[11059882.085633] Console: colour dummy device 80x25
[11059882.085696] console [tty0] enabled
[11059882.085740] Lock dependency validator: Copyright (c) 2006 Red
Hat, Inc., Ingo Molnar
[11059882.085819] ... MAX_LOCKDEP_SUBCLASSES: 8
[11059882.085866] ... MAX_LOCK_DEPTH: 48
[11059882.085912] ... MAX_LOCKDEP_KEYS: 8191
[11059882.085962] ... CLASSHASH_SIZE: 4096
[11059882.086011] ... MAX_LOCKDEP_ENTRIES: 16384
[11059882.086061] ... MAX_LOCKDEP_CHAINS: 32768
[11059882.086110] ... CHAINHASH_SIZE: 16384
[11059882.086160] memory used by lock dependency info: 5855 kB
[11059882.086221] per task-struct memory footprint: 1920 bytes
[11059882.086971] kmemleak: Early log buffer exceeded (26295), please
increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE
[11059884.094525] Calibrating delay using timer specific routine..
2006.23 BogoMIPS (lpj=4012467)
[11059884.094652] pid_max: default: 262144 minimum: 2048
[11059884.095661] Security Framework initialized
[11059884.095713] Yama: becoming mindful.
[11059884.095776] AppArmor: AppArmor disabled by boot time parameter
[11059884.096179] Mount-cache hash table entries: 65536 (order: 6, 524288 bytes)
[11059884.096261] Mountpoint-cache hash table entries: 65536 (order:
6, 524288 bytes)
[11059884.098289] ftrace: allocating 19365 entries in 38 pages
[11059884.119485] smp: Bringing up secondary CPUs ...
[11059884.173617] smp: Brought up 1 node, 32 CPUs
[11059884.179296] devtmpfs: initialized
[11059884.185352] Performance events:
[11059884.185387] Testing NMI watchdog ...
[11059884.265513] OK.
[11059884.265632] Supported PMU type is 'niagara5'
[11059884.299456] ldc.c:v1.1 (July 22, 2008)
[11059884.301371] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns
[11059884.306042] NET: Registered protocol family 16
[11059884.316862] VIO: Adding device channel-devices
[11059884.317163] VIO: Adding device vnet-port-0-0
[11059884.317437] VIO: Adding device vnet-port-0-1
[11059884.317694] VIO: Adding device vnet-port-0-2
[11059884.317961] VIO: Adding device vnet-port-0-3
[11059884.318852] VIO: Adding device vnet-port-0-4
[11059884.319757] VIO: Adding device vdc-port-0-0
[11059884.320640] VIO: Adding device vdc-port-1-0
[11059884.321523] VIO: Adding device vdc-port-2-0
[11059884.322420] VIO: Adding device vdc-port-3-0
[11059884.323303] VIO: Adding device vlds-port-0-0
[11059884.324207] VIO: Adding device ds-0
[11059884.393536] HugeTLB registered 8 MB page size, pre-allocated 0 pages
[11059884.403418] vgaarb: loaded
[11059884.407260] SUN4V: Reboot data supported (maj=1,min=0).
[11059884.407477] ds.c:v1.0 (Jul 11, 2007)
[11059884.407815] ds-0: ds_conn_reset() from send_events
[11059884.407968] ds-0: Registered md-update service.
[11059884.408021] ds-0: Registered domain-shutdown service.
[11059884.408090] ds-0: Registered domain-panic service.
[11059884.408155] ds-0: Registered dr-cpu service.
[11059884.408211] ds-0: Registered pri service.
[11059884.408266] ds-0: Registered var-config service.
[11059884.408413] clocksource: Switched to clocksource stick
[11059884.509681] VFS: Disk quotas dquot_6.6.0
[11059884.511529] VFS: Dquot-cache hash table entries: 1024 (order 0,
8192 bytes)
[11059884.531075] NET: Registered protocol family 2
[11059884.538440] TCP established hash table entries: 262144 (order:
8, 2097152 bytes)
[11059884.539519] TCP bind hash table entries: 65536 (order: 9, 4194304 bytes)
[11059884.550419] TCP: Hash tables configured (established 262144 bind 65536)
[11059884.550753] UDP hash table entries: 16384 (order: 8, 2621440 bytes)
[11059884.557183] UDP-Lite hash table entries: 16384 (order: 8, 2621440 bytes)
[11059884.568838] NET: Registered protocol family 1
[11059884.569346] Unpacking initramfs...
[11059884.902162] Freeing initrd memory: 13672K
[11059884.944211] futex hash table entries: 65536 (order: 10, 8388608 bytes)
[11059884.955799] audit: initializing netlink subsys (disabled)
[11059884.956191] audit: type=2000 audit(0.936:1): initialized
[11059884.958931] workingset: timestamp_bits=46 max_order=22 bucket_order=0
[11059884.960142] zbud: loaded
[11059884.979364] Unable to handle kernel paging request at virtual
address 000612000001c000
[11059884.979448] tsk->{mm,active_mm}->context = 0000000000000000
[11059884.979502] tsk->{mm,active_mm}->pgd = fff8000070002000
[11059884.979555] \|/ ____ \|/
[11059884.979555] "@'/ .. \`@"
[11059884.979555] /_| \__/ |_\
[11059884.979555] \__U_/
[11059884.979695] cryptomgr_test(229): Oops [#1]
[11059884.979743] CPU: 27 PID: 229 Comm: cryptomgr_test Not tainted 4.9.0+ #38
[11059884.979808] task: fff800080c42c040 task.stack: fff8000808c74000
[11059884.979866] TSTATE: 0000009980001602 TPC: 0000000000745208 TNPC:
000000000074520c Y: 000000d1 Not tainted
[11059884.979966] TPC: <scatterwalk_copychunks+0xa8/0x1e0>
[11059884.980015] g0: 0000000000000000 g1: 000612000001cc20 g2:
00000000000000b8 g3: 0000000000000002
[11059884.980096] g4: fff800080c42c040 g5: fff800082c5ac000 g6:
fff8000808c74000 g7: 0000000000001b00
[11059884.980177] o0: fff8000808c77b08 o1: 00000001003a4000 o2:
0000000000000046 o3: 00000000024002c0
[11059884.980258] o4: fff8000030404920 o5: 0000000000cdec00 sp:
fff8000808c76f11 ret_pc: 0000000000000000
[11059884.980341] RPC: < (null)>
[11059884.980405] l0: 0000000000001fff l1: 0000000000000000 l2:
0000000000baddf0 l3: 00000000000011e8
[11059884.980492] l4: 00000fff00000000 l5: 0006000000000000 l6:
0000000000d0afc0 l7: 0000000000dc13c8
[11059884.980573] i0: 00000001003a4000 i1: fff8000808c77870 i2:
0000000000000046 i3: 0000000000000000
[11059884.980651] i4: 0000000000000046 i5: 0000000000000046 i6:
fff8000808c76fc1 i7: 000000000074537c
[11059884.980734] I7: <scatterwalk_map_and_copy+0x3c/0xc0>
[11059884.980784] Call Trace:
[11059884.980814] [000000000074537c] scatterwalk_map_and_copy+0x3c/0xc0
[11059884.980879] [000000000074ba94] scomp_acomp_comp_decomp+0xb4/0x260
[11059884.980940] [000000000074bc70] scomp_acomp_compress+0x10/0x20
[11059884.981000] [0000000000751080] test_acomp+0x160/0x4c0
[11059884.981052] [0000000000751474] alg_test_comp+0x94/0x100
[11059884.981104] [000000000074f6fc] alg_test+0x15c/0x300
[11059884.981157] [000000000074c748] cryptomgr_test+0x48/0x60
[11059884.982214] [00000000004927cc] kthread+0xec/0x140
[11059884.982265] [0000000000406084] ret_from_fork+0x1c/0x2c
[11059884.982317] [0000000000000000] (null)
[11059884.982364] Disabling lock debugging due to kernel taint
[11059884.982416] Caller[000000000074537c]: scatterwalk_map_and_copy+0x3c/0xc0
[11059884.982479] Caller[000000000074ba94]: scomp_acomp_comp_decomp+0xb4/0x260
[11059884.982544] Caller[000000000074bc70]: scomp_acomp_compress+0x10/0x20
[11059884.982606] Caller[0000000000751080]: test_acomp+0x160/0x4c0
[11059884.982661] Caller[0000000000751474]: alg_test_comp+0x94/0x100
[11059884.982717] Caller[000000000074f6fc]: alg_test+0x15c/0x300
[11059884.982773] Caller[000000000074c748]: cryptomgr_test+0x48/0x60
[11059884.982829] Caller[00000000004927cc]: kthread+0xec/0x140
[11059884.982881] Caller[0000000000406084]: ret_from_fork+0x1c/0x2c
[11059884.982937] Caller[0000000000000000]: (null)
[11059884.982987] Instruction DUMP:
[11059884.982989] c4066008
[11059884.983021] 92100018
[11059884.983050] 9410001c
[11059884.983079] <d0586040>
[11059884.983106] 82088010
[11059884.983135] 90020001
[11059884.983162] 937ec408
[11059884.983190] 40019e7a
[11059884.983219] 917ec418
[11059884.983247]
[11059884.983298] note: cryptomgr_test[229] exited with preempt_count 2
^ permalink raw reply
* [PATCH 2/2] crypto: skcipher AF_ALG - overhaul memory management
From: Stephan Müller @ 2016-12-25 17:15 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
In-Reply-To: <1486189.x0AQ4O6r2j@positron.chronox.de>
The updated memory management is described in the top part of the code.
As one benefit of the changed memory management, the AIO and synchronous
operation is now implemented in one common function. The AF_ALG
operation uses the async kernel crypto API interface for each cipher
operation. Thus, the only difference between the AIO and sync operation
types visible from user space is:
1. the callback function to be invoked when the asynchronous operation
is completed
2. whether to wait for the completion of the kernel crypto API operation
or not
In addition, the code structure is adjusted to match the structure of
algif_aead for easier code assessment.
The user space interface changed slightly as follows: the old AIO
operation returned zero upon success and < 0 in case of an error to user
space. As all other AF_ALG interfaces (including the sync skcipher
interface) returned the number of processed bytes upon success and < 0
in case of an error, the new skcipher interface (regardless of AIO or
sync) returns the number of processed bytes in case of success.
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_skcipher.c | 438 +++++++++++++++++++-----------------------------
1 file changed, 174 insertions(+), 264 deletions(-)
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index a9e79d8..437e60a 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -10,6 +10,24 @@
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*
+ * The following concept of the memory management is used:
+ *
+ * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
+ * filled by user space with the data submitted via sendpage/sendmsg. Filling
+ * up the TX SGL does not cause a crypto operation -- the data will only be
+ * tracked by the kernel. Upon receipt of one recvmsg call, the caller must
+ * provide a buffer which is tracked with the RX SGL.
+ *
+ * During the processing of the recvmsg operation, the cipher request is
+ * allocated and prepared. To support multiple recvmsg operations operating
+ * on one TX SGL, an offset pointer into the TX SGL is maintained. The TX SGL
+ * that is used for the crypto request is scatterwalk_ffwd by the offset
+ * pointer to obtain the start address the crypto operation shall use for
+ * the input data.
+ *
+ * After the completion of the crypto operation, the RX SGL and the cipher
+ * request is released. The TX SGL is released once all parts of it are
+ * processed.
*/
#include <crypto/scatterwalk.h>
@@ -31,6 +49,22 @@ struct skcipher_sg_list {
struct scatterlist sg[0];
};
+struct skcipher_async_rsgl {
+ struct af_alg_sgl sgl;
+ struct list_head list;
+};
+
+struct skcipher_async_req {
+ struct kiocb *iocb;
+ struct sock *sk;
+
+ struct skcipher_async_rsgl first_sgl;
+ struct list_head list;
+
+ unsigned int areqlen;
+ struct skcipher_request req;
+};
+
struct skcipher_tfm {
struct crypto_skcipher *skcipher;
bool has_key;
@@ -44,65 +78,22 @@ struct skcipher_ctx {
struct af_alg_completion completion;
- atomic_t inflight;
+ unsigned int inflight;
size_t used;
+ size_t processed;
- unsigned int len;
bool more;
bool merge;
bool enc;
- struct skcipher_request req;
-};
-
-struct skcipher_async_rsgl {
- struct af_alg_sgl sgl;
- struct list_head list;
+ unsigned int len;
};
-struct skcipher_async_req {
- struct kiocb *iocb;
- struct skcipher_async_rsgl first_sgl;
- struct list_head list;
- struct scatterlist *tsg;
- atomic_t *inflight;
- struct skcipher_request req;
-};
+static DECLARE_WAIT_QUEUE_HEAD(skcipher_aio_finish_wait);
#define MAX_SGL_ENTS ((4096 - sizeof(struct skcipher_sg_list)) / \
sizeof(struct scatterlist) - 1)
-static void skcipher_free_async_sgls(struct skcipher_async_req *sreq)
-{
- struct skcipher_async_rsgl *rsgl, *tmp;
- struct scatterlist *sgl;
- struct scatterlist *sg;
- int i, n;
-
- list_for_each_entry_safe(rsgl, tmp, &sreq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &sreq->first_sgl)
- kfree(rsgl);
- }
- sgl = sreq->tsg;
- n = sg_nents(sgl);
- for_each_sg(sgl, sg, n, i)
- put_page(sg_page(sg));
-
- kfree(sreq->tsg);
-}
-
-static void skcipher_async_cb(struct crypto_async_request *req, int err)
-{
- struct skcipher_async_req *sreq = req->data;
- struct kiocb *iocb = sreq->iocb;
-
- atomic_dec(sreq->inflight);
- skcipher_free_async_sgls(sreq);
- kzfree(sreq);
- iocb->ki_complete(iocb, err, err);
-}
-
static inline int skcipher_sndbuf(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
@@ -117,7 +108,7 @@ static inline bool skcipher_writable(struct sock *sk)
return PAGE_SIZE <= skcipher_sndbuf(sk);
}
-static int skcipher_alloc_sgl(struct sock *sk)
+static int skcipher_alloc_tsgl(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
@@ -147,7 +138,7 @@ static int skcipher_alloc_sgl(struct sock *sk)
return 0;
}
-static void skcipher_pull_sgl(struct sock *sk, size_t used, int put)
+static void skcipher_pull_tsgl(struct sock *sk, size_t used)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
@@ -171,30 +162,35 @@ static void skcipher_pull_sgl(struct sock *sk, size_t used, int put)
used -= plen;
ctx->used -= plen;
+ ctx->processed -= plen;
if (sg[i].length)
return;
- if (put)
- put_page(sg_page(sg + i));
+
+ put_page(sg_page(sg + i));
sg_assign_page(sg + i, NULL);
}
list_del(&sgl->list);
- sock_kfree_s(sk, sgl,
- sizeof(*sgl) + sizeof(sgl->sg[0]) *
- (MAX_SGL_ENTS + 1));
+ sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) *
+ (MAX_SGL_ENTS + 1));
}
if (!ctx->used)
ctx->merge = 0;
}
-static void skcipher_free_sgl(struct sock *sk)
+static void skcipher_free_rsgl(struct skcipher_async_req *areq)
{
- struct alg_sock *ask = alg_sk(sk);
- struct skcipher_ctx *ctx = ask->private;
+ struct sock *sk = areq->sk;
+ struct skcipher_async_rsgl *rsgl, *tmp;
- skcipher_pull_sgl(sk, ctx->used, 1);
+ list_for_each_entry_safe(rsgl, tmp, &areq->list, list) {
+ af_alg_free_sg(&rsgl->sgl);
+ if (rsgl != &areq->first_sgl)
+ sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ list_del(&rsgl->list);
+ }
}
static int skcipher_wait_for_wmem(struct sock *sk, unsigned flags)
@@ -378,7 +374,7 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
len = min_t(unsigned long, len, skcipher_sndbuf(sk));
- err = skcipher_alloc_sgl(sk);
+ err = skcipher_alloc_tsgl(sk);
if (err)
goto unlock;
@@ -453,7 +449,7 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
goto unlock;
}
- err = skcipher_alloc_sgl(sk);
+ err = skcipher_alloc_tsgl(sk);
if (err)
goto unlock;
@@ -479,25 +475,33 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
return err ?: size;
}
-static int skcipher_all_sg_nents(struct skcipher_ctx *ctx)
+static void skcipher_async_cb(struct crypto_async_request *req, int err)
{
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- int nents = 0;
+ struct skcipher_async_req *areq = req->data;
+ struct sock *sk = areq->sk;
+ struct alg_sock *ask = alg_sk(sk);
+ struct skcipher_ctx *ctx = ask->private;
+ struct kiocb *iocb = areq->iocb;
- list_for_each_entry(sgl, &ctx->tsgl, list) {
- sg = sgl->sg;
+ lock_sock(sk);
- while (!sg->length)
- sg++;
+ BUG_ON(!ctx->inflight);
- nents += sg_nents(sg);
- }
- return nents;
+ skcipher_free_rsgl(areq);
+ sock_kfree_s(sk, areq, areq->areqlen);
+ skcipher_pull_tsgl(sk, areq->req.cryptlen);
+ __sock_put(sk);
+ ctx->inflight--;
+
+ iocb->ki_complete(iocb, err, err);
+
+ release_sock(sk);
+
+ wake_up_interruptible(&skcipher_aio_finish_wait);
}
-static int skcipher_recvmsg_async(struct socket *sock, struct msghdr *msg,
- int flags)
+static int skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
@@ -506,215 +510,131 @@ static int skcipher_recvmsg_async(struct socket *sock, struct msghdr *msg,
struct skcipher_ctx *ctx = ask->private;
struct skcipher_tfm *skc = pask->private;
struct crypto_skcipher *tfm = skc->skcipher;
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- struct skcipher_async_req *sreq;
- struct skcipher_request *req;
+ unsigned bs = crypto_skcipher_blocksize(tfm);
+ unsigned int areqlen = sizeof(struct skcipher_async_req) +
+ crypto_skcipher_reqsize(tfm);
+ struct skcipher_sg_list *tsgl;
+ struct skcipher_async_req *areq;
struct skcipher_async_rsgl *last_rsgl = NULL;
- unsigned int txbufs = 0, len = 0, tx_nents;
- unsigned int reqsize = crypto_skcipher_reqsize(tfm);
- unsigned int ivsize = crypto_skcipher_ivsize(tfm);
+ struct scatterlist tsgl_head[2], *tsgl_head_p;
int err = -ENOMEM;
- bool mark = false;
- char *iv;
-
- sreq = kzalloc(sizeof(*sreq) + reqsize + ivsize, GFP_KERNEL);
- if (unlikely(!sreq))
- goto out;
-
- req = &sreq->req;
- iv = (char *)(req + 1) + reqsize;
- sreq->iocb = msg->msg_iocb;
- INIT_LIST_HEAD(&sreq->list);
- sreq->inflight = &ctx->inflight;
+ size_t len = 0;
lock_sock(sk);
- tx_nents = skcipher_all_sg_nents(ctx);
- sreq->tsg = kcalloc(tx_nents, sizeof(*sg), GFP_KERNEL);
- if (unlikely(!sreq->tsg))
+
+ /* Allocate cipher request for current operation. */
+ areq = sock_kmalloc(sk, areqlen, GFP_KERNEL);
+ if (unlikely(!areq))
goto unlock;
- sg_init_table(sreq->tsg, tx_nents);
- memcpy(iv, ctx->iv, ivsize);
- skcipher_request_set_tfm(req, tfm);
- skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP,
- skcipher_async_cb, sreq);
+ areq->areqlen = areqlen;
+ areq->sk = sk;
+ INIT_LIST_HEAD(&areq->list);
- while (iov_iter_count(&msg->msg_iter)) {
+ /* convert iovecs of output buffers into RX SGL */
+ while (len < ctx->used && iov_iter_count(&msg->msg_iter)) {
struct skcipher_async_rsgl *rsgl;
- int used;
+ size_t seglen;
if (!ctx->used) {
err = skcipher_wait_for_data(sk, flags);
if (err)
goto free;
}
- sgl = list_first_entry(&ctx->tsgl,
- struct skcipher_sg_list, list);
- sg = sgl->sg;
- while (!sg->length)
- sg++;
-
- used = min_t(unsigned long, ctx->used,
- iov_iter_count(&msg->msg_iter));
- used = min_t(unsigned long, used, sg->length);
-
- if (txbufs == tx_nents) {
- struct scatterlist *tmp;
- int x;
- /* Ran out of tx slots in async request
- * need to expand */
- tmp = kcalloc(tx_nents * 2, sizeof(*tmp),
- GFP_KERNEL);
- if (!tmp) {
- err = -ENOMEM;
- goto free;
- }
+ seglen = min_t(size_t, ctx->used,
+ iov_iter_count(&msg->msg_iter));
- sg_init_table(tmp, tx_nents * 2);
- for (x = 0; x < tx_nents; x++)
- sg_set_page(&tmp[x], sg_page(&sreq->tsg[x]),
- sreq->tsg[x].length,
- sreq->tsg[x].offset);
- kfree(sreq->tsg);
- sreq->tsg = tmp;
- tx_nents *= 2;
- mark = true;
- }
- /* Need to take over the tx sgl from ctx
- * to the asynch req - these sgls will be freed later */
- sg_set_page(sreq->tsg + txbufs++, sg_page(sg), sg->length,
- sg->offset);
-
- if (list_empty(&sreq->list)) {
- rsgl = &sreq->first_sgl;
- list_add_tail(&rsgl->list, &sreq->list);
+ if (list_empty(&areq->list)) {
+ rsgl = &areq->first_sgl;
} else {
- rsgl = kmalloc(sizeof(*rsgl), GFP_KERNEL);
+ rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
if (!rsgl) {
err = -ENOMEM;
goto free;
}
- list_add_tail(&rsgl->list, &sreq->list);
}
- used = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, used);
- err = used;
- if (used < 0)
+ rsgl->sgl.npages = 0;
+ list_add_tail(&rsgl->list, &areq->list);
+
+ /* make one iovec available as scatterlist */
+ err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
+ if (err < 0)
goto free;
+
+ /* chain the new scatterlist with previous one */
if (last_rsgl)
af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
last_rsgl = rsgl;
- len += used;
- skcipher_pull_sgl(sk, used, 0);
- iov_iter_advance(&msg->msg_iter, used);
+ len += err;
+ iov_iter_advance(&msg->msg_iter, err);
}
- if (mark)
- sg_mark_end(sreq->tsg + txbufs - 1);
+ /* Process only as much RX buffers for which we have TX data */
+ if (len > ctx->used)
+ len = ctx->used;
+
+ /*
+ * If more buffers are to be expected to be processed, process only
+ * full block size buffers.
+ */
+ if (ctx->more || len < ctx->used)
+ len -= len % bs;
+
+ tsgl = list_first_entry(&ctx->tsgl, struct skcipher_sg_list, list);
+ /* Get the head of the SGL we want to process */
+ tsgl_head_p = scatterwalk_ffwd(tsgl_head, tsgl->sg, ctx->processed);
+ BUG_ON(!tsgl_head_p);
+
+ /* Initialize the crypto operation */
+ skcipher_request_set_tfm(&areq->req, tfm);
+ skcipher_request_set_crypt(&areq->req, tsgl_head_p,
+ areq->first_sgl.sgl.sg, len, ctx->iv);
+
+ if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
+ /* AIO operation */
+ areq->iocb = msg->msg_iocb;
+ skcipher_request_set_callback(&areq->req,
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ skcipher_async_cb, areq);
+ err = ctx->enc ? crypto_skcipher_encrypt(&areq->req) :
+ crypto_skcipher_decrypt(&areq->req);
+ } else {
+ /* Synchronous operation */
+ skcipher_request_set_callback(&areq->req,
+ CRYPTO_TFM_REQ_MAY_SLEEP |
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ af_alg_complete,
+ &ctx->completion);
+ err = af_alg_wait_for_completion(ctx->enc ?
+ crypto_skcipher_encrypt(&areq->req) :
+ crypto_skcipher_decrypt(&areq->req),
+ &ctx->completion);
+ }
- skcipher_request_set_crypt(req, sreq->tsg, sreq->first_sgl.sgl.sg,
- len, iv);
- err = ctx->enc ? crypto_skcipher_encrypt(req) :
- crypto_skcipher_decrypt(req);
+ /* AIO operation in progress */
if (err == -EINPROGRESS) {
- atomic_inc(&ctx->inflight);
+ sock_hold(sk);
err = -EIOCBQUEUED;
- sreq = NULL;
+ ctx->inflight++;
+ /* Remember the TX bytes that were processed. */
+ ctx->processed += len;
goto unlock;
- }
-free:
- skcipher_free_async_sgls(sreq);
-unlock:
- skcipher_wmem_wakeup(sk);
- release_sock(sk);
- kzfree(sreq);
-out:
- return err;
-}
-
-static int skcipher_recvmsg_sync(struct socket *sock, struct msghdr *msg,
- int flags)
-{
- struct sock *sk = sock->sk;
- struct alg_sock *ask = alg_sk(sk);
- struct sock *psk = ask->parent;
- struct alg_sock *pask = alg_sk(psk);
- struct skcipher_ctx *ctx = ask->private;
- struct skcipher_tfm *skc = pask->private;
- struct crypto_skcipher *tfm = skc->skcipher;
- unsigned bs = crypto_skcipher_blocksize(tfm);
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- int err = -EAGAIN;
- int used;
- long copied = 0;
-
- lock_sock(sk);
- while (msg_data_left(msg)) {
- if (!ctx->used) {
- err = skcipher_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- used = min_t(unsigned long, ctx->used, msg_data_left(msg));
-
- used = af_alg_make_sg(&ctx->rsgl, &msg->msg_iter, used);
- err = used;
- if (err < 0)
- goto unlock;
-
- if (ctx->more || used < ctx->used)
- used -= used % bs;
-
- err = -EINVAL;
- if (!used)
- goto free;
-
- sgl = list_first_entry(&ctx->tsgl,
- struct skcipher_sg_list, list);
- sg = sgl->sg;
-
- while (!sg->length)
- sg++;
-
- skcipher_request_set_crypt(&ctx->req, sg, ctx->rsgl.sg, used,
- ctx->iv);
-
- err = af_alg_wait_for_completion(
- ctx->enc ?
- crypto_skcipher_encrypt(&ctx->req) :
- crypto_skcipher_decrypt(&ctx->req),
- &ctx->completion);
+ } else if (!err)
+ /* Remember the TX bytes that were processed. */
+ ctx->processed += len;
free:
- af_alg_free_sg(&ctx->rsgl);
-
- if (err)
- goto unlock;
-
- copied += used;
- skcipher_pull_sgl(sk, used, 1);
- iov_iter_advance(&msg->msg_iter, used);
- }
-
- err = 0;
+ skcipher_free_rsgl(areq);
+ if (areq)
+ sock_kfree_s(sk, areq, areqlen);
+ skcipher_pull_tsgl(sk, len);
unlock:
skcipher_wmem_wakeup(sk);
release_sock(sk);
-
- return copied ?: err;
-}
-
-static int skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
- size_t ignored, int flags)
-{
- return (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) ?
- skcipher_recvmsg_async(sock, msg, flags) :
- skcipher_recvmsg_sync(sock, msg, flags);
+ return err ? err : len;
}
static unsigned int skcipher_poll(struct file *file, struct socket *sock,
@@ -894,26 +814,20 @@ static int skcipher_setkey(void *private, const u8 *key, unsigned int keylen)
return err;
}
-static void skcipher_wait(struct sock *sk)
-{
- struct alg_sock *ask = alg_sk(sk);
- struct skcipher_ctx *ctx = ask->private;
- int ctr = 0;
-
- while (atomic_read(&ctx->inflight) && ctr++ < 100)
- msleep(100);
-}
-
static void skcipher_sock_destruct(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(&ctx->req);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
+ struct skcipher_tfm *skc = pask->private;
+ struct crypto_skcipher *tfm = skc->skcipher;
- if (atomic_read(&ctx->inflight))
- skcipher_wait(sk);
+ /* Suspend caller if AIO operations are in flight. */
+ wait_event_interruptible(skcipher_aio_finish_wait,
+ (ctx->inflight == 0));
- skcipher_free_sgl(sk);
+ skcipher_pull_tsgl(sk, ctx->used);
sock_kzfree_s(sk, ctx->iv, crypto_skcipher_ivsize(tfm));
sock_kfree_s(sk, ctx, ctx->len);
af_alg_release_parent(sk);
@@ -925,7 +839,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct skcipher_tfm *tfm = private;
struct crypto_skcipher *skcipher = tfm->skcipher;
- unsigned int len = sizeof(*ctx) + crypto_skcipher_reqsize(skcipher);
+ unsigned int len = sizeof(*ctx);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
@@ -943,19 +857,15 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
INIT_LIST_HEAD(&ctx->tsgl);
ctx->len = len;
ctx->used = 0;
+ ctx->processed = 0;
+ ctx->inflight = 0;
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
- atomic_set(&ctx->inflight, 0);
af_alg_init_completion(&ctx->completion);
ask->private = ctx;
- skcipher_request_set_tfm(&ctx->req, skcipher);
- skcipher_request_set_callback(&ctx->req, CRYPTO_TFM_REQ_MAY_SLEEP |
- CRYPTO_TFM_REQ_MAY_BACKLOG,
- af_alg_complete, &ctx->completion);
-
sk->sk_destruct = skcipher_sock_destruct;
return 0;
--
2.9.3
^ permalink raw reply related
* [PATCH 1/2] crypto: aead AF_ALG - overhaul memory management
From: Stephan Müller @ 2016-12-25 17:15 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
In-Reply-To: <1486189.x0AQ4O6r2j@positron.chronox.de>
The updated memory management is described in the top part of the code.
As one benefit of the changed memory management, the AIO and synchronous
operation is now implemented in one common function. The AF_ALG
operation uses the async kernel crypto API interface for each cipher
operation. Thus, the only difference between the AIO and sync operation
types visible from user space is:
1. the callback function to be invoked when the asynchronous operation
is completed
2. whether to wait for the completion of the kernel crypto API operation
or not
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_aead.c | 439 +++++++++++++++++++++++-----------------------------
1 file changed, 195 insertions(+), 244 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index f849311..3ca68fb 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -11,6 +11,25 @@
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
+ *
+ * The following concept of the memory management is used:
+ *
+ * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
+ * filled by user space with the data submitted via sendpage/sendmsg. Filling
+ * up the TX SGL does not cause a crypto operation -- the data will only be
+ * tracked by the kernel. Upon receipt of one recvmsg call, the caller must
+ * provide a buffer which is tracked with the RX SGL.
+ *
+ * During the processing of the recvmsg operation, the cipher request is
+ * allocated and prepared. To support multiple recvmsg operations operating
+ * on one TX SGL, an offset pointer into the TX SGL is maintained. The TX SGL
+ * that is used for the crypto request is scatterwalk_ffwd by the offset
+ * pointer to obtain the start address the crypto operation shall use for
+ * the input data.
+ *
+ * After the completion of the crypto operation, the RX SGL and the cipher
+ * request is released. The TX SGL is released once all parts of it are
+ * processed.
*/
#include <crypto/internal/aead.h>
@@ -24,7 +43,7 @@
#include <linux/net.h>
#include <net/sock.h>
-struct aead_sg_list {
+struct aead_async_tsgl {
unsigned int cur;
struct scatterlist sg[ALG_MAX_PAGES];
};
@@ -35,34 +54,38 @@ struct aead_async_rsgl {
};
struct aead_async_req {
- struct scatterlist *tsgl;
- struct aead_async_rsgl first_rsgl;
- struct list_head list;
struct kiocb *iocb;
- unsigned int tsgls;
- char iv[];
+ struct sock *sk;
+
+ struct aead_async_rsgl first_rsgl; /* First RX SG */
+ struct list_head list; /* Track RX SGs */
+
+ unsigned int areqlen; /* Length of this data struct */
+ struct aead_request aead_req; /* req ctx trails this struct */
};
struct aead_ctx {
- struct aead_sg_list tsgl;
- struct aead_async_rsgl first_rsgl;
- struct list_head list;
+ struct aead_async_tsgl tsgl;
void *iv;
+ size_t aead_assoclen;
- struct af_alg_completion completion;
+ struct af_alg_completion completion; /* sync work queue */
- unsigned long used;
+ unsigned int inflight; /* outstanding AIO ops */
+ size_t used; /* TX bytes sent to kernel */
+ size_t processed; /* number of bytes processed */
- unsigned int len;
bool more;
bool merge;
bool enc;
- size_t aead_assoclen;
- struct aead_request aead_req;
+ unsigned int len; /* length data structure */
+ struct crypto_aead *aead_tfm;
};
+static DECLARE_WAIT_QUEUE_HEAD(aead_aio_finish_wait);
+
static inline int aead_sndbuf(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
@@ -79,34 +102,43 @@ static inline bool aead_writable(struct sock *sk)
static inline bool aead_sufficient_data(struct aead_ctx *ctx)
{
- unsigned as = crypto_aead_authsize(crypto_aead_reqtfm(&ctx->aead_req));
+ unsigned as = crypto_aead_authsize(ctx->aead_tfm);
/*
* The minimum amount of memory needed for an AEAD cipher is
* the AAD and in case of decryption the tag.
+ *
+ * Also, sufficient data must be available after disregarding the
+ * already processed data.
*/
- return ctx->used >= ctx->aead_assoclen + (ctx->enc ? 0 : as);
+ return ctx->used >= ctx->aead_assoclen + (ctx->enc ? 0 : as) +
+ ctx->processed;
}
static void aead_reset_ctx(struct aead_ctx *ctx)
{
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct aead_async_tsgl *sgl = &ctx->tsgl;
sg_init_table(sgl->sg, ALG_MAX_PAGES);
sgl->cur = 0;
ctx->used = 0;
+ ctx->processed = 0;
ctx->more = 0;
ctx->merge = 0;
}
-static void aead_put_sgl(struct sock *sk)
+static void aead_pull_tsgl(struct sock *sk, bool force)
{
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct aead_async_tsgl *sgl = &ctx->tsgl;
struct scatterlist *sg = sgl->sg;
unsigned int i;
+ /* leave provided data in tact if we did not finish AIO work */
+ if (!force && ctx->used > ctx->processed)
+ return;
+
for (i = 0; i < sgl->cur; i++) {
if (!sg_page(sg + i))
continue;
@@ -117,6 +149,19 @@ static void aead_put_sgl(struct sock *sk)
aead_reset_ctx(ctx);
}
+static void aead_free_rsgl(struct aead_async_req *areq)
+{
+ struct sock *sk = areq->sk;
+ struct aead_async_rsgl *rsgl, *tmp;
+
+ list_for_each_entry_safe(rsgl, tmp, &areq->list, list) {
+ af_alg_free_sg(&rsgl->sgl);
+ if (rsgl != &areq->first_rsgl)
+ sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ list_del(&rsgl->list);
+ }
+}
+
static void aead_wmem_wakeup(struct sock *sk)
{
struct socket_wq *wq;
@@ -189,9 +234,8 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned ivsize =
- crypto_aead_ivsize(crypto_aead_reqtfm(&ctx->aead_req));
- struct aead_sg_list *sgl = &ctx->tsgl;
+ unsigned ivsize = crypto_aead_ivsize(ctx->aead_tfm);
+ struct aead_async_tsgl *sgl = &ctx->tsgl;
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
@@ -258,7 +302,7 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
if (!aead_writable(sk)) {
/* user space sent too much data */
- aead_put_sgl(sk);
+ aead_pull_tsgl(sk, 1);
err = -EMSGSIZE;
goto unlock;
}
@@ -269,7 +313,7 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
size_t plen = 0;
if (sgl->cur >= ALG_MAX_PAGES) {
- aead_put_sgl(sk);
+ aead_pull_tsgl(sk, 1);
err = -E2BIG;
goto unlock;
}
@@ -305,7 +349,7 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
ctx->more = msg->msg_flags & MSG_MORE;
if (!ctx->more && !aead_sufficient_data(ctx)) {
- aead_put_sgl(sk);
+ aead_pull_tsgl(sk, 1);
err = -EMSGSIZE;
}
@@ -322,7 +366,7 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct aead_async_tsgl *sgl = &ctx->tsgl;
int err = -EINVAL;
if (flags & MSG_SENDPAGE_NOTLAST)
@@ -340,7 +384,7 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
if (!aead_writable(sk)) {
/* user space sent too much data */
- aead_put_sgl(sk);
+ aead_pull_tsgl(sk, 1);
err = -EMSGSIZE;
goto unlock;
}
@@ -357,7 +401,7 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
done:
ctx->more = flags & MSG_MORE;
if (!ctx->more && !aead_sufficient_data(ctx)) {
- aead_put_sgl(sk);
+ aead_pull_tsgl(sk, 1);
err = -EMSGSIZE;
}
@@ -368,190 +412,48 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
return err ?: size;
}
-#define GET_ASYM_REQ(req, tfm) (struct aead_async_req *) \
- ((char *)req + sizeof(struct aead_request) + \
- crypto_aead_reqsize(tfm))
-
- #define GET_REQ_SIZE(tfm) sizeof(struct aead_async_req) + \
- crypto_aead_reqsize(tfm) + crypto_aead_ivsize(tfm) + \
- sizeof(struct aead_request)
-
static void aead_async_cb(struct crypto_async_request *_req, int err)
{
- struct sock *sk = _req->data;
+ struct aead_async_req *areq = _req->data;
+ struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct crypto_aead *tfm = crypto_aead_reqtfm(&ctx->aead_req);
- struct aead_request *req = aead_request_cast(_req);
- struct aead_async_req *areq = GET_ASYM_REQ(req, tfm);
- struct scatterlist *sg = areq->tsgl;
- struct aead_async_rsgl *rsgl;
struct kiocb *iocb = areq->iocb;
- unsigned int i, reqlen = GET_REQ_SIZE(tfm);
- list_for_each_entry(rsgl, &areq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &areq->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
+ lock_sock(sk);
- for (i = 0; i < areq->tsgls; i++)
- put_page(sg_page(sg + i));
+ BUG_ON(!ctx->inflight);
- sock_kfree_s(sk, areq->tsgl, sizeof(*areq->tsgl) * areq->tsgls);
- sock_kfree_s(sk, req, reqlen);
+ aead_free_rsgl(areq);
+ sock_kfree_s(sk, areq, areq->areqlen);
+ aead_pull_tsgl(sk, 0);
__sock_put(sk);
+ ctx->inflight--;
iocb->ki_complete(iocb, err, err);
-}
-
-static int aead_recvmsg_async(struct socket *sock, struct msghdr *msg,
- int flags)
-{
- struct sock *sk = sock->sk;
- struct alg_sock *ask = alg_sk(sk);
- struct aead_ctx *ctx = ask->private;
- struct crypto_aead *tfm = crypto_aead_reqtfm(&ctx->aead_req);
- struct aead_async_req *areq;
- struct aead_request *req = NULL;
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct aead_async_rsgl *last_rsgl = NULL, *rsgl;
- unsigned int as = crypto_aead_authsize(tfm);
- unsigned int i, reqlen = GET_REQ_SIZE(tfm);
- int err = -ENOMEM;
- unsigned long used;
- size_t outlen = 0;
- size_t usedpages = 0;
- lock_sock(sk);
- if (ctx->more) {
- err = aead_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- if (!aead_sufficient_data(ctx))
- goto unlock;
-
- used = ctx->used;
- if (ctx->enc)
- outlen = used + as;
- else
- outlen = used - as;
-
- req = sock_kmalloc(sk, reqlen, GFP_KERNEL);
- if (unlikely(!req))
- goto unlock;
-
- areq = GET_ASYM_REQ(req, tfm);
- memset(&areq->first_rsgl, '\0', sizeof(areq->first_rsgl));
- INIT_LIST_HEAD(&areq->list);
- areq->iocb = msg->msg_iocb;
- memcpy(areq->iv, ctx->iv, crypto_aead_ivsize(tfm));
- aead_request_set_tfm(req, tfm);
- aead_request_set_ad(req, ctx->aead_assoclen);
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- aead_async_cb, sk);
- used -= ctx->aead_assoclen;
-
- /* take over all tx sgls from ctx */
- areq->tsgl = sock_kmalloc(sk,
- sizeof(*areq->tsgl) * max_t(u32, sgl->cur, 1),
- GFP_KERNEL);
- if (unlikely(!areq->tsgl))
- goto free;
-
- sg_init_table(areq->tsgl, max_t(u32, sgl->cur, 1));
- for (i = 0; i < sgl->cur; i++)
- sg_set_page(&areq->tsgl[i], sg_page(&sgl->sg[i]),
- sgl->sg[i].length, sgl->sg[i].offset);
-
- areq->tsgls = sgl->cur;
-
- /* create rx sgls */
- while (outlen > usedpages && iov_iter_count(&msg->msg_iter)) {
- size_t seglen = min_t(size_t, iov_iter_count(&msg->msg_iter),
- (outlen - usedpages));
-
- if (list_empty(&areq->list)) {
- rsgl = &areq->first_rsgl;
-
- } else {
- rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
- if (unlikely(!rsgl)) {
- err = -ENOMEM;
- goto free;
- }
- }
- rsgl->sgl.npages = 0;
- list_add_tail(&rsgl->list, &areq->list);
-
- /* make one iovec available as scatterlist */
- err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
- if (err < 0)
- goto free;
-
- usedpages += err;
-
- /* chain the new scatterlist with previous one */
- if (last_rsgl)
- af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
-
- last_rsgl = rsgl;
-
- iov_iter_advance(&msg->msg_iter, err);
- }
-
- /* ensure output buffer is sufficiently large */
- if (usedpages < outlen) {
- err = -EINVAL;
- goto unlock;
- }
-
- aead_request_set_crypt(req, areq->tsgl, areq->first_rsgl.sgl.sg, used,
- areq->iv);
- err = ctx->enc ? crypto_aead_encrypt(req) : crypto_aead_decrypt(req);
- if (err) {
- if (err == -EINPROGRESS) {
- sock_hold(sk);
- err = -EIOCBQUEUED;
- aead_reset_ctx(ctx);
- goto unlock;
- } else if (err == -EBADMSG) {
- aead_put_sgl(sk);
- }
- goto free;
- }
- aead_put_sgl(sk);
-
-free:
- list_for_each_entry(rsgl, &areq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &areq->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
- if (areq->tsgl)
- sock_kfree_s(sk, areq->tsgl, sizeof(*areq->tsgl) * areq->tsgls);
- if (req)
- sock_kfree_s(sk, req, reqlen);
-unlock:
- aead_wmem_wakeup(sk);
release_sock(sk);
- return err ? err : outlen;
+
+ wake_up_interruptible(&aead_aio_finish_wait);
}
-static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
+static int aead_recvmsg(struct socket *sock, struct msghdr *msg, size_t ignored,
+ int flags)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned as = crypto_aead_authsize(crypto_aead_reqtfm(&ctx->aead_req));
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct crypto_aead *tfm = ctx->aead_tfm;
+ unsigned int as = crypto_aead_authsize(tfm);
+ unsigned int areqlen =
+ sizeof(struct aead_async_req) + crypto_aead_reqsize(tfm);
+ struct aead_async_tsgl *tsgl = &ctx->tsgl;
+ struct aead_async_req *areq;
struct aead_async_rsgl *last_rsgl = NULL;
- struct aead_async_rsgl *rsgl, *tmp;
+ struct scatterlist tsgl_head[2], *tsgl_head_p;
int err = -EINVAL;
- unsigned long used = 0;
- size_t outlen = 0;
- size_t usedpages = 0;
+ size_t used = 0; /* [in] TX bufs to be en/decrypted */
+ size_t outlen = 0; /* [out] RX bufs produced by kernel */
+ size_t usedpages = 0; /* [in] RX bufs to be used from user */
lock_sock(sk);
@@ -566,8 +468,11 @@ static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
goto unlock;
}
- /* data length provided by caller via sendmsg/sendpage */
- used = ctx->used;
+ /*
+ * Data length provided by caller via sendmsg/sendpage that has not
+ * yet been processed.
+ */
+ used = ctx->used - ctx->processed;
/*
* Make sure sufficient data is present -- note, the same check is
@@ -600,86 +505,131 @@ static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
*/
used -= ctx->aead_assoclen;
- /* convert iovecs of output buffers into scatterlists */
+ /* Allocate cipher request for current operation. */
+ areq = sock_kmalloc(sk, areqlen, GFP_KERNEL);
+ if (unlikely(!areq)) {
+ err = -ENOMEM;
+ goto unlock;
+ }
+ areq->areqlen = areqlen;
+ areq->sk = sk;
+ INIT_LIST_HEAD(&areq->list);
+
+ /* convert iovecs of output buffers into RX SGL */
while (outlen > usedpages && iov_iter_count(&msg->msg_iter)) {
+ struct aead_async_rsgl *rsgl;
size_t seglen = min_t(size_t, iov_iter_count(&msg->msg_iter),
(outlen - usedpages));
- if (list_empty(&ctx->list)) {
- rsgl = &ctx->first_rsgl;
+ if (list_empty(&areq->list)) {
+ rsgl = &areq->first_rsgl;
} else {
rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
if (unlikely(!rsgl)) {
err = -ENOMEM;
- goto unlock;
+ goto free;
}
}
+
rsgl->sgl.npages = 0;
- list_add_tail(&rsgl->list, &ctx->list);
+ list_add_tail(&rsgl->list, &areq->list);
/* make one iovec available as scatterlist */
err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
if (err < 0)
- goto unlock;
- usedpages += err;
+ goto free;
+
/* chain the new scatterlist with previous one */
if (last_rsgl)
af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
last_rsgl = rsgl;
-
+ usedpages += err;
iov_iter_advance(&msg->msg_iter, err);
}
- /* ensure output buffer is sufficiently large */
+ /*
+ * Ensure output buffer is sufficiently large. If the caller provides
+ * less buffer space, only use the relative required input size. This
+ * allows AIO operation where the caller sent all data to be processed
+ * and the AIO operation performs the operation on the different chunks
+ * of the input data.
+ */
if (usedpages < outlen) {
- err = -EINVAL;
- goto unlock;
- }
+ size_t less = outlen - usedpages;
- sg_mark_end(sgl->sg + sgl->cur - 1);
- aead_request_set_crypt(&ctx->aead_req, sgl->sg, ctx->first_rsgl.sgl.sg,
- used, ctx->iv);
- aead_request_set_ad(&ctx->aead_req, ctx->aead_assoclen);
+ if (used < less) {
+ err = -EINVAL;
+ goto free;
+ }
+ used -= less;
+ outlen -= less;
+ }
- err = af_alg_wait_for_completion(ctx->enc ?
- crypto_aead_encrypt(&ctx->aead_req) :
- crypto_aead_decrypt(&ctx->aead_req),
+ sg_mark_end(tsgl->sg + tsgl->cur - 1);
+
+ /* Get the head of the SGL we want to process */
+ tsgl_head_p = scatterwalk_ffwd(tsgl_head, tsgl->sg, ctx->processed);
+ BUG_ON(!tsgl_head_p);
+
+ /* Initialize the crypto operation */
+ aead_request_set_crypt(&areq->aead_req, tsgl_head_p,
+ areq->first_rsgl.sgl.sg, used, ctx->iv);
+ aead_request_set_ad(&areq->aead_req, ctx->aead_assoclen);
+ aead_request_set_tfm(&areq->aead_req, tfm);
+
+ if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
+ /* AIO operation */
+ areq->iocb = msg->msg_iocb;
+ aead_request_set_callback(&areq->aead_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ aead_async_cb, areq);
+ err = ctx->enc ? crypto_aead_encrypt(&areq->aead_req) :
+ crypto_aead_decrypt(&areq->aead_req);
+ } else {
+ /* Synchronous operation */
+ aead_request_set_callback(&areq->aead_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ af_alg_complete, &ctx->completion);
+ err = af_alg_wait_for_completion(ctx->enc ?
+ crypto_aead_encrypt(&areq->aead_req) :
+ crypto_aead_decrypt(&areq->aead_req),
&ctx->completion);
+ }
if (err) {
- /* EBADMSG implies a valid cipher operation took place */
- if (err == -EBADMSG)
- aead_put_sgl(sk);
+ /* AIO operation in progress */
+ if (err == -EINPROGRESS) {
+ sock_hold(sk);
+ err = -EIOCBQUEUED;
- goto unlock;
+ /* Remember the TX bytes that were processed. */
+ ctx->processed += ctx->enc ? (outlen - as) :
+ (outlen + as);
+ ctx->inflight++;
+
+ goto unlock;
+ }
+ /* EBADMSG implies a valid cipher operation took place */
+ else if (err != -EBADMSG)
+ goto free;
}
- aead_put_sgl(sk);
- err = 0;
+ /* Remember the TX bytes that were processed. */
+ ctx->processed += ctx->enc ? (outlen - as) : (outlen + as);
+
+free:
+ aead_free_rsgl(areq);
+ if (areq)
+ sock_kfree_s(sk, areq, areqlen);
+ aead_pull_tsgl(sk, 0);
unlock:
- list_for_each_entry_safe(rsgl, tmp, &ctx->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &ctx->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- list_del(&rsgl->list);
- }
- INIT_LIST_HEAD(&ctx->list);
aead_wmem_wakeup(sk);
release_sock(sk);
-
return err ? err : outlen;
}
-static int aead_recvmsg(struct socket *sock, struct msghdr *msg, size_t ignored,
- int flags)
-{
- return (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) ?
- aead_recvmsg_async(sock, msg, flags) :
- aead_recvmsg_sync(sock, msg, flags);
-}
-
static unsigned int aead_poll(struct file *file, struct socket *sock,
poll_table *wait)
{
@@ -746,11 +696,14 @@ static void aead_sock_destruct(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned int ivlen = crypto_aead_ivsize(
- crypto_aead_reqtfm(&ctx->aead_req));
+ unsigned int ivlen = crypto_aead_ivsize(ctx->aead_tfm);
WARN_ON(atomic_read(&sk->sk_refcnt) != 0);
- aead_put_sgl(sk);
+
+ /* Suspend caller if AIO operations are in flight. */
+ wait_event_interruptible(aead_aio_finish_wait, (ctx->inflight == 0));
+
+ aead_pull_tsgl(sk, 1);
sock_kzfree_s(sk, ctx->iv, ivlen);
sock_kfree_s(sk, ctx, ctx->len);
af_alg_release_parent(sk);
@@ -760,7 +713,7 @@ static int aead_accept_parent(void *private, struct sock *sk)
{
struct aead_ctx *ctx;
struct alg_sock *ask = alg_sk(sk);
- unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(private);
+ unsigned int len = sizeof(*ctx);
unsigned int ivlen = crypto_aead_ivsize(private);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
@@ -777,20 +730,18 @@ static int aead_accept_parent(void *private, struct sock *sk)
ctx->len = len;
ctx->used = 0;
+ ctx->processed = 0;
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->inflight = 0;
ctx->tsgl.cur = 0;
ctx->aead_assoclen = 0;
af_alg_init_completion(&ctx->completion);
sg_init_table(ctx->tsgl.sg, ALG_MAX_PAGES);
- INIT_LIST_HEAD(&ctx->list);
ask->private = ctx;
-
- aead_request_set_tfm(&ctx->aead_req, private);
- aead_request_set_callback(&ctx->aead_req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- af_alg_complete, &ctx->completion);
+ ctx->aead_tfm = private;
sk->sk_destruct = aead_sock_destruct;
--
2.9.3
^ permalink raw reply related
* [PATCH 0/2] crypto: AF_ALG memory management fix
From: Stephan Müller @ 2016-12-25 17:13 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
Hi Herbert,
please find attached memory management updates to
- simplify the code: the old AIO memory management is very
complex and seemingly very fragile -- the update now
eliminates all reported bugs in the skcipher and AEAD
interfaces which allowed the kernel to be crashed by
an unprivileged user
- streamline the code: there is one code path for AIO and sync
operation; the code between algif_skcipher and algif_aead
is very similar (if that patch set is accepted, I volunteer
to reduce code duplication by moving service operations
into af_alg.c and to further unify the TX SGL handling)
- unify the AIO and sync operation which only differ in the
kernel crypto API callback and whether to wait for the
crypto operation or not
- fix all reported bugs regarding the handling of multiple
IOCBs.
The following testing was performed:
- stress testing to verify that no memleaks exist
- testing using Tadeusz Struck AIO test tool (see
https://github.com/tstruk/afalg_async_test) -- the AEAD test
is not applicable any more due to the changed user space
interface; the skcipher test works once the user space
interface change is honored in the test code
- using the libkcapi test suite, all tests including the
originally failing ones (AIO with multiple IOCBs) work now --
the current libkcapi code artificially limits the AEAD
operation to one IOCB. After altering the libkcapi code
to allow multiple IOCBs, the testing works flawless.
Stephan Mueller (2):
crypto: aead AF_ALG - overhaul memory management
crypto: skcipher AF_ALG - overhaul memory management
crypto/algif_aead.c | 439 ++++++++++++++++++++
+---------------------------
crypto/algif_skcipher.c | 438 +++++++++++++++++++----------------------------
2 files changed, 369 insertions(+), 508 deletions(-)
--
2.9.3
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox