* Re: [PATCH 5/7] crypto: marvell: Adding a complete operation for async requests
From: Boris Brezillon @ 2016-06-15 20:55 UTC (permalink / raw)
To: Romain Perier
Cc: Arnaud Ebalard, Gregory Clement, Thomas Petazzoni,
David S. Miller, Russell King, linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-6-git-send-email-romain.perier@free-electrons.com>
On Wed, 15 Jun 2016 21:15:32 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> So far, the 'process' operation was used to check if the current request
> was correctly handled by the engine, if it was the case it copied
> information from the SRAM to the main memory. Now, we split this
> operation. We keep the 'process' operation, which still checks if the
> request was correctly handled by the engine or not, then we add a new
> operation for completion. The 'complete' method copies the content of
> the SRAM to memory. This will soon become useful if we want to call
> the process and the complete operations from different locations
> depending on the type of the request (different cleanup logic).
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
> ---
> drivers/crypto/marvell/cesa.c | 1 +
> drivers/crypto/marvell/cesa.h | 3 +++
> drivers/crypto/marvell/cipher.c | 47 ++++++++++++++++++++++++-----------------
> drivers/crypto/marvell/hash.c | 22 ++++++++++---------
> 4 files changed, 44 insertions(+), 29 deletions(-)
>
> diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
> index fe04d1b..af96426 100644
> --- a/drivers/crypto/marvell/cesa.c
> +++ b/drivers/crypto/marvell/cesa.c
> @@ -98,6 +98,7 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
> engine->req = NULL;
> mv_cesa_dequeue_req_unlocked(engine);
> spin_unlock_bh(&engine->lock);
> + ctx->ops->complete(req);
> ctx->ops->cleanup(req);
> local_bh_disable();
> req->complete(req, res);
> diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
> index 158ff82..32de08b 100644
> --- a/drivers/crypto/marvell/cesa.h
> +++ b/drivers/crypto/marvell/cesa.h
> @@ -456,6 +456,8 @@ struct mv_cesa_engine {
> * code)
> * @step: launch the crypto operation on the next chunk
> * @cleanup: cleanup the crypto request (release associated data)
> + * @complete: complete the request, i.e copy result from sram or contexts
> + * when it is needed.
> */
> struct mv_cesa_req_ops {
> void (*prepare)(struct crypto_async_request *req,
> @@ -463,6 +465,7 @@ struct mv_cesa_req_ops {
> int (*process)(struct crypto_async_request *req, u32 status);
> void (*step)(struct crypto_async_request *req);
> void (*cleanup)(struct crypto_async_request *req);
> + void (*complete)(struct crypto_async_request *req);
> };
>
> /**
> diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
> index 15d2c5a..fbaae2f 100644
> --- a/drivers/crypto/marvell/cipher.c
> +++ b/drivers/crypto/marvell/cipher.c
> @@ -118,7 +118,6 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
> struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
> struct mv_cesa_engine *engine = sreq->base.engine;
> size_t len;
> - unsigned int ivsize;
>
> len = sg_pcopy_from_buffer(req->dst, creq->dst_nents,
> engine->sram + CESA_SA_DATA_SRAM_OFFSET,
> @@ -128,10 +127,6 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
> if (sreq->offset < req->nbytes)
> return -EINPROGRESS;
>
> - ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
> - memcpy_fromio(req->info,
> - engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET, ivsize);
> -
> return 0;
> }
>
> @@ -141,21 +136,9 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
> struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
>
> - if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
> - int ret;
> - struct mv_cesa_req *basereq;
> - unsigned int ivsize;
> -
> - ret = mv_cesa_dma_process(&creq->req.base, status);
> - if (ret)
> - return ret;
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> + return mv_cesa_dma_process(&creq->req.base, status);
>
> - basereq = &creq->req.base;
> - ivsize = crypto_ablkcipher_ivsize(
> - crypto_ablkcipher_reqtfm(ablkreq));
> - memcpy_fromio(ablkreq->info, basereq->chain.last->data, ivsize);
> - return ret;
> - }
> return mv_cesa_ablkcipher_std_process(ablkreq, status);
> }
>
> @@ -197,6 +180,7 @@ static inline void mv_cesa_ablkcipher_prepare(struct crypto_async_request *req,
> {
> struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
> +
Nit: not sure you should mix this cosmetic change with the other
changes.
> creq->req.base.engine = engine;
>
> if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> @@ -213,11 +197,36 @@ mv_cesa_ablkcipher_req_cleanup(struct crypto_async_request *req)
> mv_cesa_ablkcipher_cleanup(ablkreq);
> }
>
> +static void
> +mv_cesa_ablkcipher_complete(struct crypto_async_request *req)
> +{
> + struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> + struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
> + struct mv_cesa_engine *engine = creq->req.base.engine;
> + unsigned int ivsize;
> +
> + ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq));
> +
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
> + struct mv_cesa_req *basereq;
> +
> + basereq = &creq->req.base;
> + ivsize = crypto_ablkcipher_ivsize(
> + crypto_ablkcipher_reqtfm(ablkreq));
You already have ivsize initialized.
> + memcpy_fromio(ablkreq->info, basereq->chain.last->data, ivsize);
Use memcpy() here.
> + } else {
> + memcpy_fromio(ablkreq->info,
> + engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET,
> + ivsize);
> + }
> +}
> +
> static const struct mv_cesa_req_ops mv_cesa_ablkcipher_req_ops = {
> .step = mv_cesa_ablkcipher_step,
> .process = mv_cesa_ablkcipher_process,
> .prepare = mv_cesa_ablkcipher_prepare,
> .cleanup = mv_cesa_ablkcipher_req_cleanup,
> + .complete = mv_cesa_ablkcipher_complete,
> };
>
> static int mv_cesa_ablkcipher_cra_init(struct crypto_tfm *tfm)
> diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
> index cc7c5b0..f7f84cc 100644
> --- a/drivers/crypto/marvell/hash.c
> +++ b/drivers/crypto/marvell/hash.c
> @@ -287,17 +287,20 @@ static int mv_cesa_ahash_process(struct crypto_async_request *req, u32 status)
> {
> struct ahash_request *ahashreq = ahash_request_cast(req);
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
> - struct mv_cesa_engine *engine = creq->req.base.engine;
> - unsigned int digsize;
> - int ret, i;
>
> if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> - ret = mv_cesa_dma_process(&creq->req.base, status);
> - else
> - ret = mv_cesa_ahash_std_process(ahashreq, status);
> + return mv_cesa_dma_process(&creq->req.base, status);
>
> - if (ret == -EINPROGRESS)
> - return ret;
> + return mv_cesa_ahash_std_process(ahashreq, status);
> +}
> +
> +static void mv_cesa_ahash_complete(struct crypto_async_request *req)
> +{
> + struct ahash_request *ahashreq = ahash_request_cast(req);
> + struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
> + struct mv_cesa_engine *engine = creq->req.base.engine;
> + unsigned int digsize;
> + int i;
>
> digsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(ahashreq));
> for (i = 0; i < digsize / 4; i++)
> @@ -326,8 +329,6 @@ static int mv_cesa_ahash_process(struct crypto_async_request *req, u32 status)
> result[i] = cpu_to_be32(creq->state[i]);
> }
> }
> -
> - return ret;
> }
>
> static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
> @@ -366,6 +367,7 @@ static const struct mv_cesa_req_ops mv_cesa_ahash_req_ops = {
> .process = mv_cesa_ahash_process,
> .prepare = mv_cesa_ahash_prepare,
> .cleanup = mv_cesa_ahash_req_cleanup,
> + .complete = mv_cesa_ahash_complete,
> };
>
> static int mv_cesa_ahash_init(struct ahash_request *req,
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH 3/7] crypto: marvell: Copy IV vectors by DMA transfers for acipher requests
From: Boris Brezillon @ 2016-06-15 20:48 UTC (permalink / raw)
To: Romain Perier
Cc: Arnaud Ebalard, Gregory Clement, Thomas Petazzoni,
David S. Miller, Russell King, linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-4-git-send-email-romain.perier@free-electrons.com>
On Wed, 15 Jun 2016 21:15:30 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> @@ -135,23 +140,23 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
> {
> struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
> - struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
> - struct mv_cesa_engine *engine = sreq->base.engine;
> - int ret;
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (creq->req.base.type == CESA_DMA_REQ) {
> + int ret;
> + struct mv_cesa_tdma_req *dreq;
> + unsigned int ivsize;
> +
> ret = mv_cesa_dma_process(&creq->req.dma, status);
> - else
> - ret = mv_cesa_ablkcipher_std_process(ablkreq, status);
> + if (ret)
> + return ret;
>
> - if (ret)
> + dreq = &creq->req.dma;
> + ivsize = crypto_ablkcipher_ivsize(
> + crypto_ablkcipher_reqtfm(ablkreq));
> + memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
Just use memcpy() here: you're not copying from an iomem region here.
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH 4/7] crypto: marvell: Moving the tdma chain out of mv_cesa_tdma_req
From: Boris Brezillon @ 2016-06-15 20:42 UTC (permalink / raw)
To: Romain Perier
Cc: Arnaud Ebalard, Gregory Clement, Thomas Petazzoni,
David S. Miller, Russell King, linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-5-git-send-email-romain.perier@free-electrons.com>
On Wed, 15 Jun 2016 21:15:31 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> Actually the only way to access the tdma chain is to use the 'req' union
Currently, ...
> from a mv_cesa_{ablkcipher,ahash}. This will soon become a problem if we
> want to handle the TDMA chaining vs standard/non-DMA processing in a
> generic way (with generic functions at the cesa.c level detecting
> whether the request should be queued at the DMA level or not). Hence the
> decision to move the chain field a the mv_cesa_req level at the expense
at
> of adding 2 void * fields to all request contexts (including non-DMA
> ones). To limit the overhead, we get rid of the type field, which can
> now be deduced from the req->chain.first value.
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
> ---
> drivers/crypto/marvell/cesa.c | 3 ++-
> drivers/crypto/marvell/cesa.h | 31 +++++++++++++------------------
> drivers/crypto/marvell/cipher.c | 40 ++++++++++++++++++++++------------------
> drivers/crypto/marvell/hash.c | 36 +++++++++++++++---------------------
> drivers/crypto/marvell/tdma.c | 8 ++++----
> 5 files changed, 56 insertions(+), 62 deletions(-)
>
> diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
> index 93700cd..fe04d1b 100644
> --- a/drivers/crypto/marvell/cesa.c
> +++ b/drivers/crypto/marvell/cesa.c
> @@ -111,7 +111,8 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
> return ret;
> }
>
> -int mv_cesa_queue_req(struct crypto_async_request *req)
> +int mv_cesa_queue_req(struct crypto_async_request *req,
> + struct mv_cesa_req *creq)
> {
> int ret;
> int i;
> diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
> index 74b84bd..158ff82 100644
> --- a/drivers/crypto/marvell/cesa.h
> +++ b/drivers/crypto/marvell/cesa.h
> @@ -509,21 +509,11 @@ enum mv_cesa_req_type {
>
> /**
> * struct mv_cesa_req - CESA request
> - * @type: request type
> * @engine: engine associated with this request
> + * @chain: list of tdma descriptors associated with this request
^ extra white space.
> */
> struct mv_cesa_req {
> - enum mv_cesa_req_type type;
> struct mv_cesa_engine *engine;
> -};
> -
> -/**
> - * struct mv_cesa_tdma_req - CESA TDMA request
> - * @base: base information
> - * @chain: TDMA chain
> - */
> -struct mv_cesa_tdma_req {
> - struct mv_cesa_req base;
> struct mv_cesa_tdma_chain chain;
> };
>
> @@ -562,7 +552,6 @@ struct mv_cesa_ablkcipher_std_req {
> struct mv_cesa_ablkcipher_req {
> union {
> struct mv_cesa_req base;
> - struct mv_cesa_tdma_req dma;
> struct mv_cesa_ablkcipher_std_req std;
Now that the dma specific fields are part of the base request there's no
reason to keep this union.
You can just put struct mv_cesa_req base; directly under struct
mv_cesa_ablkcipher_req, and move mv_cesa_ablkcipher_std_req fields in
mv_cesa_ablkcipher_req.
> } req;
> int src_nents;
> @@ -587,7 +576,6 @@ struct mv_cesa_ahash_std_req {
> * @cache_dma: DMA address of the cache buffer
> */
> struct mv_cesa_ahash_dma_req {
> - struct mv_cesa_tdma_req base;
> u8 *padding;
> dma_addr_t padding_dma;
> u8 *cache;
> @@ -625,6 +613,12 @@ struct mv_cesa_ahash_req {
>
> extern struct mv_cesa_dev *cesa_dev;
>
> +static inline enum mv_cesa_req_type
> +mv_cesa_req_get_type(struct mv_cesa_req *req)
> +{
> + return req->chain.first ? CESA_DMA_REQ : CESA_STD_REQ;
> +}
> +
> static inline void mv_cesa_update_op_cfg(struct mv_cesa_op_ctx *op,
> u32 cfg, u32 mask)
> {
> @@ -697,7 +691,8 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
> CESA_SA_DESC_CFG_FIRST_FRAG;
> }
>
> -int mv_cesa_queue_req(struct crypto_async_request *req);
> +int mv_cesa_queue_req(struct crypto_async_request *req,
> + struct mv_cesa_req *creq);
>
> /*
> * Helper function that indicates whether a crypto request needs to be
> @@ -767,9 +762,9 @@ static inline bool mv_cesa_req_dma_iter_next_op(struct mv_cesa_dma_iter *iter)
> return iter->op_len;
> }
>
> -void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq);
> +void mv_cesa_dma_step(struct mv_cesa_req *dreq);
>
> -static inline int mv_cesa_dma_process(struct mv_cesa_tdma_req *dreq,
> +static inline int mv_cesa_dma_process(struct mv_cesa_req *dreq,
> u32 status)
> {
> if (!(status & CESA_SA_INT_ACC0_IDMA_DONE))
> @@ -781,10 +776,10 @@ static inline int mv_cesa_dma_process(struct mv_cesa_tdma_req *dreq,
> return 0;
> }
>
> -void mv_cesa_dma_prepare(struct mv_cesa_tdma_req *dreq,
> +void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
> struct mv_cesa_engine *engine);
> +void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq);
>
> -void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq);
>
> static inline void
> mv_cesa_tdma_desc_iter_init(struct mv_cesa_tdma_chain *chain)
> diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
> index f42620e..15d2c5a 100644
> --- a/drivers/crypto/marvell/cipher.c
> +++ b/drivers/crypto/marvell/cipher.c
> @@ -70,14 +70,14 @@ mv_cesa_ablkcipher_dma_cleanup(struct ablkcipher_request *req)
> dma_unmap_sg(cesa_dev->dev, req->src, creq->src_nents,
> DMA_BIDIRECTIONAL);
> }
> - mv_cesa_dma_cleanup(&creq->req.dma);
> + mv_cesa_dma_cleanup(&creq->req.base);
> }
>
> static inline void mv_cesa_ablkcipher_cleanup(struct ablkcipher_request *req)
> {
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> mv_cesa_ablkcipher_dma_cleanup(req);
> }
>
> @@ -141,19 +141,19 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
> struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
>
> - if (creq->req.base.type == CESA_DMA_REQ) {
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
> int ret;
> - struct mv_cesa_tdma_req *dreq;
> + struct mv_cesa_req *basereq;
> unsigned int ivsize;
>
> - ret = mv_cesa_dma_process(&creq->req.dma, status);
> + ret = mv_cesa_dma_process(&creq->req.base, status);
Initialize basereq earlier and pass it as the first argument of
mv_cesa_dma_process().
> if (ret)
> return ret;
>
> - dreq = &creq->req.dma;
> + basereq = &creq->req.base;
> ivsize = crypto_ablkcipher_ivsize(
> crypto_ablkcipher_reqtfm(ablkreq));
> - memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
> + memcpy_fromio(ablkreq->info, basereq->chain.last->data, ivsize);
> return ret;
> }
> return mv_cesa_ablkcipher_std_process(ablkreq, status);
> @@ -164,8 +164,8 @@ static void mv_cesa_ablkcipher_step(struct crypto_async_request *req)
> struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> - mv_cesa_dma_step(&creq->req.dma);
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> + mv_cesa_dma_step(&creq->req.base);
> else
> mv_cesa_ablkcipher_std_step(ablkreq);
> }
> @@ -174,9 +174,9 @@ static inline void
> mv_cesa_ablkcipher_dma_prepare(struct ablkcipher_request *req)
> {
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> - struct mv_cesa_tdma_req *dreq = &creq->req.dma;
> + struct mv_cesa_req *dreq = &creq->req.base;
You named it basereq in mv_cesa_ablkcipher_step(). Try to be
consistent, no matter the name.
>
> - mv_cesa_dma_prepare(dreq, dreq->base.engine);
> + mv_cesa_dma_prepare(dreq, dreq->engine);
> }
>
> static inline void
> @@ -199,7 +199,7 @@ static inline void mv_cesa_ablkcipher_prepare(struct crypto_async_request *req,
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
> creq->req.base.engine = engine;
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> mv_cesa_ablkcipher_dma_prepare(ablkreq);
> else
> mv_cesa_ablkcipher_std_prepare(ablkreq);
> @@ -302,14 +302,13 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
> GFP_KERNEL : GFP_ATOMIC;
> - struct mv_cesa_tdma_req *dreq = &creq->req.dma;
> + struct mv_cesa_req *dreq = &creq->req.base;
Ditto.
> struct mv_cesa_ablkcipher_dma_iter iter;
> struct mv_cesa_tdma_chain chain;
> bool skip_ctx = false;
> int ret;
> unsigned int ivsize;
>
> - dreq->base.type = CESA_DMA_REQ;
> dreq->chain.first = NULL;
> dreq->chain.last = NULL;
>
> @@ -397,10 +396,12 @@ mv_cesa_ablkcipher_std_req_init(struct ablkcipher_request *req,
> {
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
> + struct mv_cesa_req *basereq = &creq->req.base;
>
> - sreq->base.type = CESA_STD_REQ;
> sreq->op = *op_templ;
> sreq->skip_ctx = false;
> + basereq->chain.first = NULL;
> + basereq->chain.last = NULL;
>
> return 0;
> }
> @@ -442,6 +443,7 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
> static int mv_cesa_des_op(struct ablkcipher_request *req,
> struct mv_cesa_op_ctx *tmpl)
> {
> + struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> struct mv_cesa_des_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> int ret;
>
> @@ -454,7 +456,7 @@ static int mv_cesa_des_op(struct ablkcipher_request *req,
> if (ret)
> return ret;
>
> - ret = mv_cesa_queue_req(&req->base);
> + ret = mv_cesa_queue_req(&req->base, &creq->req.base);
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ablkcipher_cleanup(req);
>
> @@ -562,6 +564,7 @@ struct crypto_alg mv_cesa_cbc_des_alg = {
> static int mv_cesa_des3_op(struct ablkcipher_request *req,
> struct mv_cesa_op_ctx *tmpl)
> {
> + struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> struct mv_cesa_des3_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> int ret;
>
> @@ -574,7 +577,7 @@ static int mv_cesa_des3_op(struct ablkcipher_request *req,
> if (ret)
> return ret;
>
> - ret = mv_cesa_queue_req(&req->base);
> + ret = mv_cesa_queue_req(&req->base, &creq->req.base);
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ablkcipher_cleanup(req);
>
> @@ -688,6 +691,7 @@ struct crypto_alg mv_cesa_cbc_des3_ede_alg = {
> static int mv_cesa_aes_op(struct ablkcipher_request *req,
> struct mv_cesa_op_ctx *tmpl)
> {
> + struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> struct mv_cesa_aes_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> int ret, i;
> u32 *key;
> @@ -716,7 +720,7 @@ static int mv_cesa_aes_op(struct ablkcipher_request *req,
> if (ret)
> return ret;
>
> - ret = mv_cesa_queue_req(&req->base);
> + ret = mv_cesa_queue_req(&req->base, &creq->req.base);
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ablkcipher_cleanup(req);
>
> diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
> index 0fae351..cc7c5b0 100644
> --- a/drivers/crypto/marvell/hash.c
> +++ b/drivers/crypto/marvell/hash.c
> @@ -103,14 +103,14 @@ static inline void mv_cesa_ahash_dma_cleanup(struct ahash_request *req)
>
> dma_unmap_sg(cesa_dev->dev, req->src, creq->src_nents, DMA_TO_DEVICE);
> mv_cesa_ahash_dma_free_cache(&creq->req.dma);
> - mv_cesa_dma_cleanup(&creq->req.dma.base);
> + mv_cesa_dma_cleanup(&creq->req.base);
> }
>
> static inline void mv_cesa_ahash_cleanup(struct ahash_request *req)
> {
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> mv_cesa_ahash_dma_cleanup(req);
> }
>
> @@ -118,7 +118,7 @@ static void mv_cesa_ahash_last_cleanup(struct ahash_request *req)
> {
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> mv_cesa_ahash_dma_last_cleanup(req);
> }
>
> @@ -256,9 +256,9 @@ static int mv_cesa_ahash_std_process(struct ahash_request *req, u32 status)
> static inline void mv_cesa_ahash_dma_prepare(struct ahash_request *req)
> {
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> - struct mv_cesa_tdma_req *dreq = &creq->req.dma.base;
> + struct mv_cesa_req *dreq = &creq->req.base;
Ditto.
>
> - mv_cesa_dma_prepare(dreq, dreq->base.engine);
> + mv_cesa_dma_prepare(dreq, dreq->engine);
> }
>
> static void mv_cesa_ahash_std_prepare(struct ahash_request *req)
> @@ -277,8 +277,8 @@ static void mv_cesa_ahash_step(struct crypto_async_request *req)
> struct ahash_request *ahashreq = ahash_request_cast(req);
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> - mv_cesa_dma_step(&creq->req.dma.base);
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> + mv_cesa_dma_step(&creq->req.base);
> else
> mv_cesa_ahash_std_step(ahashreq);
> }
> @@ -291,8 +291,8 @@ static int mv_cesa_ahash_process(struct crypto_async_request *req, u32 status)
> unsigned int digsize;
> int ret, i;
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> - ret = mv_cesa_dma_process(&creq->req.dma.base, status);
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> + ret = mv_cesa_dma_process(&creq->req.base, status);
> else
> ret = mv_cesa_ahash_std_process(ahashreq, status);
>
> @@ -340,7 +340,7 @@ static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
>
> creq->req.base.engine = engine;
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
> mv_cesa_ahash_dma_prepare(ahashreq);
> else
> mv_cesa_ahash_std_prepare(ahashreq);
> @@ -555,8 +555,7 @@ static int mv_cesa_ahash_dma_req_init(struct ahash_request *req)
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
> GFP_KERNEL : GFP_ATOMIC;
> - struct mv_cesa_ahash_dma_req *ahashdreq = &creq->req.dma;
> - struct mv_cesa_tdma_req *dreq = &ahashdreq->base;
> + struct mv_cesa_req *dreq = &creq->req.base;
Ditto.
> struct mv_cesa_ahash_dma_iter iter;
> struct mv_cesa_op_ctx *op = NULL;
> unsigned int frag_len;
> @@ -662,11 +661,6 @@ static int mv_cesa_ahash_req_init(struct ahash_request *req, bool *cached)
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> int ret;
>
> - if (cesa_dev->caps->has_tdma)
> - creq->req.base.type = CESA_DMA_REQ;
> - else
> - creq->req.base.type = CESA_STD_REQ;
> -
Hm, where is it decided now? I mean, I don't see this test anywhere
else in your patch, which means you're now always using standard mode.
> creq->src_nents = sg_nents_for_len(req->src, req->nbytes);
> if (creq->src_nents < 0) {
> dev_err(cesa_dev->dev, "Invalid number of src SG");
> @@ -680,7 +674,7 @@ static int mv_cesa_ahash_req_init(struct ahash_request *req, bool *cached)
> if (*cached)
> return 0;
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
Should be
if (cesa_dev->caps->has_tdma)
> ret = mv_cesa_ahash_dma_req_init(req);
>
> return ret;
> @@ -700,7 +694,7 @@ static int mv_cesa_ahash_update(struct ahash_request *req)
> if (cached)
> return 0;
>
> - ret = mv_cesa_queue_req(&req->base);
> + ret = mv_cesa_queue_req(&req->base, &creq->req.base);
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ahash_cleanup(req);
>
> @@ -725,7 +719,7 @@ static int mv_cesa_ahash_final(struct ahash_request *req)
> if (cached)
> return 0;
>
> - ret = mv_cesa_queue_req(&req->base);
> + ret = mv_cesa_queue_req(&req->base, &creq->req.base);
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ahash_cleanup(req);
>
> @@ -750,7 +744,7 @@ static int mv_cesa_ahash_finup(struct ahash_request *req)
> if (cached)
> return 0;
>
> - ret = mv_cesa_queue_req(&req->base);
> + ret = mv_cesa_queue_req(&req->base, &creq->req.base);
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ahash_cleanup(req);
>
> diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
> index 88c87be..9a424f9 100644
> --- a/drivers/crypto/marvell/tdma.c
> +++ b/drivers/crypto/marvell/tdma.c
> @@ -37,9 +37,9 @@ bool mv_cesa_req_dma_iter_next_transfer(struct mv_cesa_dma_iter *iter,
> return true;
> }
>
> -void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
> +void mv_cesa_dma_step(struct mv_cesa_req *dreq)
> {
> - struct mv_cesa_engine *engine = dreq->base.engine;
> + struct mv_cesa_engine *engine = dreq->engine;
>
> writel_relaxed(0, engine->regs + CESA_SA_CFG);
>
> @@ -58,7 +58,7 @@ void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
> writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
> }
>
> -void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
> +void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq)
> {
> struct mv_cesa_tdma_desc *tdma;
>
> @@ -81,7 +81,7 @@ void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
> dreq->chain.last = NULL;
> }
>
> -void mv_cesa_dma_prepare(struct mv_cesa_tdma_req *dreq,
> +void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
> struct mv_cesa_engine *engine)
> {
> struct mv_cesa_tdma_desc *tdma;
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH 3/7] crypto: marvell: Copy IV vectors by DMA transfers for acipher requests
From: Boris Brezillon @ 2016-06-15 20:07 UTC (permalink / raw)
To: Romain Perier
Cc: Thomas Petazzoni, Russell King, Arnaud Ebalard, linux-crypto,
Gregory Clement, David S. Miller, linux-arm-kernel
In-Reply-To: <1466018134-10779-4-git-send-email-romain.perier@free-electrons.com>
On Wed, 15 Jun 2016 21:15:30 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> Adding a TDMA descriptor at the end of the request for copying the
> output IV vector via a DMA transfer. This is required for processing
> cipher requests asynchroniously in chained mode, otherwise the content
asynchronously
> of the IV vector will be overwriten for each new finished request.
BTW, Not sure the term 'asynchronously' is appropriate here. The
standard (AKA non-DMA) processing is also asynchronous. The real reason
here is that you want to chain the requests and offload as much
processing as possible to the DMA and crypto engine. And as you
explained, this is only possible if we retrieve the updated IV using
DMA.
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
> ---
> drivers/crypto/marvell/cesa.c | 4 ++++
> drivers/crypto/marvell/cesa.h | 5 +++++
> drivers/crypto/marvell/cipher.c | 40 +++++++++++++++++++++++++++-------------
> drivers/crypto/marvell/tdma.c | 29 +++++++++++++++++++++++++++++
> 4 files changed, 65 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
> index fb403e1..93700cd 100644
> --- a/drivers/crypto/marvell/cesa.c
> +++ b/drivers/crypto/marvell/cesa.c
> @@ -312,6 +312,10 @@ static int mv_cesa_dev_dma_init(struct mv_cesa_dev *cesa)
> if (!dma->padding_pool)
> return -ENOMEM;
>
> + dma->iv_pool = dmam_pool_create("cesa_iv", dev, 16, 1, 0);
> + if (!dma->iv_pool)
> + return -ENOMEM;
> +
> cesa->dma = dma;
>
> return 0;
> diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
> index 74071e4..74b84bd 100644
> --- a/drivers/crypto/marvell/cesa.h
> +++ b/drivers/crypto/marvell/cesa.h
> @@ -275,6 +275,7 @@ struct mv_cesa_op_ctx {
> #define CESA_TDMA_DUMMY 0
> #define CESA_TDMA_DATA 1
> #define CESA_TDMA_OP 2
> +#define CESA_TDMA_IV 4
Should be 3 and not 4: TDMA_TYPE is an enum, not a bit field.
>
> /**
> * struct mv_cesa_tdma_desc - TDMA descriptor
> @@ -390,6 +391,7 @@ struct mv_cesa_dev_dma {
> struct dma_pool *op_pool;
> struct dma_pool *cache_pool;
> struct dma_pool *padding_pool;
> + struct dma_pool *iv_pool;
> };
>
> /**
> @@ -790,6 +792,9 @@ mv_cesa_tdma_desc_iter_init(struct mv_cesa_tdma_chain *chain)
> memset(chain, 0, sizeof(*chain));
> }
>
> +int mv_cesa_dma_add_iv_op(struct mv_cesa_tdma_chain *chain, dma_addr_t src,
> + u32 size, u32 flags, gfp_t gfp_flags);
> +
> struct mv_cesa_op_ctx *mv_cesa_dma_add_op(struct mv_cesa_tdma_chain *chain,
> const struct mv_cesa_op_ctx *op_templ,
> bool skip_ctx,
> diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
> index 8d0fabb..f42620e 100644
> --- a/drivers/crypto/marvell/cipher.c
> +++ b/drivers/crypto/marvell/cipher.c
> @@ -118,6 +118,7 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
> struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
> struct mv_cesa_engine *engine = sreq->base.engine;
> size_t len;
> + unsigned int ivsize;
>
> len = sg_pcopy_from_buffer(req->dst, creq->dst_nents,
> engine->sram + CESA_SA_DATA_SRAM_OFFSET,
> @@ -127,6 +128,10 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
> if (sreq->offset < req->nbytes)
> return -EINPROGRESS;
>
> + ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
> + memcpy_fromio(req->info,
> + engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET, ivsize);
> +
> return 0;
> }
>
> @@ -135,23 +140,23 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
> {
> struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
> struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
> - struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
> - struct mv_cesa_engine *engine = sreq->base.engine;
> - int ret;
>
> - if (creq->req.base.type == CESA_DMA_REQ)
> + if (creq->req.base.type == CESA_DMA_REQ) {
> + int ret;
> + struct mv_cesa_tdma_req *dreq;
> + unsigned int ivsize;
> +
> ret = mv_cesa_dma_process(&creq->req.dma, status);
> - else
> - ret = mv_cesa_ablkcipher_std_process(ablkreq, status);
> + if (ret)
> + return ret;
>
> - if (ret)
> + dreq = &creq->req.dma;
> + ivsize = crypto_ablkcipher_ivsize(
> + crypto_ablkcipher_reqtfm(ablkreq));
Sometime it's better to offend the < 80 characters rule than doing
funky stuff ;).
> + memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
> return ret;
> -
> - memcpy_fromio(ablkreq->info,
> - engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET,
> - crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq)));
> -
> - return 0;
> + }
Missing blank line.
> + return mv_cesa_ablkcipher_std_process(ablkreq, status);
This version is more readable IMHO:
struct mv_cesa_tdma_req *dreq;
unsigned int ivsize;
int ret;
if (creq->req.base.type == CESA_STD_REQ)
return mv_cesa_ablkcipher_std_process(ablkreq, status);
ret = mv_cesa_dma_process(&creq->req.dma, status);
if (ret)
return ret;
dreq = &creq->req.dma;
ivsize =
crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq));
memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
return 0;
>
> static void mv_cesa_ablkcipher_step(struct crypto_async_request *req)
> @@ -302,6 +307,7 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
> struct mv_cesa_tdma_chain chain;
> bool skip_ctx = false;
> int ret;
> + unsigned int ivsize;
>
> dreq->base.type = CESA_DMA_REQ;
> dreq->chain.first = NULL;
> @@ -360,6 +366,14 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
>
> } while (mv_cesa_ablkcipher_req_iter_next_op(&iter));
>
> + /* Add output data for IV */
> + ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
> + ret = mv_cesa_dma_add_iv_op(&chain, CESA_SA_CRYPT_IV_SRAM_OFFSET,
> + ivsize, CESA_TDMA_SRC_IN_SRAM, flags);
> +
> + if (ret)
> + goto err_free_tdma;
> +
> dreq->chain = chain;
>
> return 0;
> diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
> index d493714..88c87be 100644
> --- a/drivers/crypto/marvell/tdma.c
> +++ b/drivers/crypto/marvell/tdma.c
> @@ -68,6 +68,9 @@ void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
> if (tdma->flags & CESA_TDMA_OP)
I realize this test is wrong.
It should be
type = tdma->flags & CESA_TDMA_TYPE_MSK;
if (type == CESA_TDMA_OP)
> dma_pool_free(cesa_dev->dma->op_pool, tdma->op,
> le32_to_cpu(tdma->src));
> + else if (tdma->flags & CESA_TDMA_IV)
and here
else if (type == CESA_TDMA_IV)
> + dma_pool_free(cesa_dev->dma->iv_pool, tdma->data,
> + le32_to_cpu(tdma->dst));
>
> tdma = tdma->next;
> dma_pool_free(cesa_dev->dma->tdma_desc_pool, old_tdma,
> @@ -120,6 +123,32 @@ mv_cesa_dma_add_desc(struct mv_cesa_tdma_chain *chain, gfp_t flags)
> return new_tdma;
> }
>
> +int mv_cesa_dma_add_iv_op(struct mv_cesa_tdma_chain *chain, dma_addr_t src,
> + u32 size, u32 flags, gfp_t gfp_flags)
> +{
> +
> + struct mv_cesa_tdma_desc *tdma;
> + u8 *cache;
Why do you name that one cache? iv would be a better name.
> + dma_addr_t dma_handle;
> +
> + tdma = mv_cesa_dma_add_desc(chain, gfp_flags);
> + if (IS_ERR(tdma))
> + return PTR_ERR(tdma);
> +
> + cache = dma_pool_alloc(cesa_dev->dma->iv_pool, flags, &dma_handle);
> + if (!cache)
> + return -ENOMEM;
> +
> + tdma->byte_cnt = cpu_to_le32(size | BIT(31));
> + tdma->src = src;
> + tdma->dst = cpu_to_le32(dma_handle);
> + tdma->data = cache;
> +
> + flags &= (CESA_TDMA_DST_IN_SRAM | CESA_TDMA_SRC_IN_SRAM);
> + tdma->flags = flags | CESA_TDMA_DATA | CESA_TDMA_IV;
You should not mix 2 different types, it's either CESA_TDMA_DATA or
CESA_TDMA_IV, and in this case it should be CESA_TDMA_IV.
> + return 0;
> +}
> +
> struct mv_cesa_op_ctx *mv_cesa_dma_add_op(struct mv_cesa_tdma_chain *chain,
> const struct mv_cesa_op_ctx *op_templ,
> bool skip_ctx,
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH 2/7] crypto: marvell: Check engine is not already running when enabling a req
From: Boris Brezillon @ 2016-06-15 19:37 UTC (permalink / raw)
To: Romain Perier
Cc: Arnaud Ebalard, Gregory Clement, Thomas Petazzoni,
David S. Miller, Russell King, linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-3-git-send-email-romain.perier@free-electrons.com>
On Wed, 15 Jun 2016 21:15:29 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> Adding BUG_ON() macro to be sure that the step operation is not about
> to activate a request on the engine if the corresponding engine is
> already processing a crypto request. This is helpful when the support
> for chaining crypto requests will be added. Instead of hanging the
> system when the engine is in an incoherent state, we add this macro
You don't add the macro, you use it.
> which throws an understandable error.
How about rewording the commit message this way:
"
Add a BUG_ON() call when the driver tries to launch a crypto request
while the engine is still processing the previous one. This replaces
a silent system hang by a verbose kernel panic with the associated
backtrace to let the user know that something went wrong in the CESA
driver.
"
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Apart from the coding style issue mentioned below,
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
> ---
> drivers/crypto/marvell/cipher.c | 2 ++
> drivers/crypto/marvell/hash.c | 2 ++
> drivers/crypto/marvell/tdma.c | 2 ++
> 3 files changed, 6 insertions(+)
>
> diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
> index dcf1fce..8d0fabb 100644
> --- a/drivers/crypto/marvell/cipher.c
> +++ b/drivers/crypto/marvell/cipher.c
> @@ -106,6 +106,8 @@ static void mv_cesa_ablkcipher_std_step(struct ablkcipher_request *req)
>
> mv_cesa_set_int_mask(engine, CESA_SA_INT_ACCEL0_DONE);
> writel_relaxed(CESA_SA_CFG_PARA_DIS, engine->regs + CESA_SA_CFG);
> + BUG_ON(readl(engine->regs + CESA_SA_CMD)
> + & CESA_SA_CMD_EN_CESA_SA_ACCL0);
Nit: please put the '&' operator at the end of the first line and
align CESA_SA_CMD_EN_CESA_SA_ACCL0 on the open parenthesis.
BUG_ON(readl(engine->regs + CESA_SA_CMD) &
CESA_SA_CMD_EN_CESA_SA_ACCL0);
> writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
> }
>
> diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
> index 7ca2e0f..0fae351 100644
> --- a/drivers/crypto/marvell/hash.c
> +++ b/drivers/crypto/marvell/hash.c
> @@ -237,6 +237,8 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
>
> mv_cesa_set_int_mask(engine, CESA_SA_INT_ACCEL0_DONE);
> writel_relaxed(CESA_SA_CFG_PARA_DIS, engine->regs + CESA_SA_CFG);
> + BUG_ON(readl(engine->regs + CESA_SA_CMD)
> + & CESA_SA_CMD_EN_CESA_SA_ACCL0);
Ditto.
> writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
> }
>
> diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
> index 7642798..d493714 100644
> --- a/drivers/crypto/marvell/tdma.c
> +++ b/drivers/crypto/marvell/tdma.c
> @@ -53,6 +53,8 @@ void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
> engine->regs + CESA_SA_CFG);
> writel_relaxed(dreq->chain.first->cur_dma,
> engine->regs + CESA_TDMA_NEXT_ADDR);
> + BUG_ON(readl(engine->regs + CESA_SA_CMD)
> + & CESA_SA_CMD_EN_CESA_SA_ACCL0);
Ditto.
> writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
> }
>
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH 1/7] crypto: marvell: Add a macro constant for the size of the crypto queue
From: Boris Brezillon @ 2016-06-15 19:20 UTC (permalink / raw)
To: Romain Perier
Cc: Arnaud Ebalard, Gregory Clement, Thomas Petazzoni,
David S. Miller, Russell King, linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-2-git-send-email-romain.perier@free-electrons.com>
On Wed, 15 Jun 2016 21:15:28 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> Adding a macro constant to be used for the size of the crypto queue,
> instead of using a numeric value directly. It will be easier to
> maintain in case we add more than one crypto queue of the same size.
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
> ---
> drivers/crypto/marvell/cesa.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
> index 056a754..fb403e1 100644
> --- a/drivers/crypto/marvell/cesa.c
> +++ b/drivers/crypto/marvell/cesa.c
> @@ -31,6 +31,9 @@
>
> #include "cesa.h"
>
> +/* Limit of the crypto queue before reaching the backlog */
> +#define CESA_CRYPTO_DEFAULT_MAX_QLEN 50
> +
> static int allhwsupport = !IS_ENABLED(CONFIG_CRYPTO_DEV_MV_CESA);
> module_param_named(allhwsupport, allhwsupport, int, 0444);
> MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if overlaps with the mv_cesa driver)");
> @@ -416,7 +419,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
> return -ENOMEM;
>
> spin_lock_init(&cesa->lock);
> - crypto_init_queue(&cesa->queue, 50);
> + crypto_init_queue(&cesa->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
> res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
> cesa->regs = devm_ioremap_resource(dev, res);
> if (IS_ERR(cesa->regs))
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* [PATCH 7/7] crypto: marvell: Add support for chaining crypto requests in TDMA mode
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
The Cryptographic Engines and Security Accelerators (CESA) supports the
Multi-Packet Chain Mode. With this mode enabled, multiple tdma requests
can be chained and processed by the hardware without software
interferences. This mode was already activated, however the crypto
requests were not chained together. By doing so, we reduce significantly
the number of IRQs. Instead of being interrupted at the end of each
crypto request, we are interrupted at the end of the last cryptographic
request processed by the engine.
This commits re-factorizes the code, changes the code architecture and
adds the required data structures to chain cryptographic requests
together before sending them to an engine.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 117 +++++++++++++++++++++++++++++++---------
drivers/crypto/marvell/cesa.h | 38 ++++++++++++-
drivers/crypto/marvell/cipher.c | 3 +-
drivers/crypto/marvell/hash.c | 9 +++-
drivers/crypto/marvell/tdma.c | 81 ++++++++++++++++++++++++++++
5 files changed, 218 insertions(+), 30 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index f9e6688..33411f6 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -32,7 +32,7 @@
#include "cesa.h"
/* Limit of the crypto queue before reaching the backlog */
-#define CESA_CRYPTO_DEFAULT_MAX_QLEN 50
+#define CESA_CRYPTO_DEFAULT_MAX_QLEN 128
static int allhwsupport = !IS_ENABLED(CONFIG_CRYPTO_DEV_MV_CESA);
module_param_named(allhwsupport, allhwsupport, int, 0444);
@@ -40,23 +40,83 @@ MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if over
struct mv_cesa_dev *cesa_dev;
-static void mv_cesa_dequeue_req_unlocked(struct mv_cesa_engine *engine)
+struct crypto_async_request *mv_cesa_dequeue_req_locked(
+ struct mv_cesa_engine *engine, struct crypto_async_request **backlog)
+{
+ struct crypto_async_request *req;
+
+ *backlog = crypto_get_backlog(&engine->queue);
+ req = crypto_dequeue_request(&engine->queue);
+
+ if (!req)
+ return NULL;
+
+ return req;
+}
+
+static void mv_cesa_rearm_engine(struct mv_cesa_engine *engine)
{
struct crypto_async_request *req, *backlog;
struct mv_cesa_ctx *ctx;
- backlog = crypto_get_backlog(&engine->queue);
- req = crypto_dequeue_request(&engine->queue);
- engine->req = req;
+ spin_lock_bh(&engine->lock);
+ if (engine->req)
+ goto out_unlock;
+
+ req = mv_cesa_dequeue_req_locked(engine, &backlog);
if (!req)
- return;
+ goto out_unlock;
+
+ engine->req = req;
+ spin_unlock_bh(&engine->lock);
if (backlog)
backlog->complete(backlog, -EINPROGRESS);
ctx = crypto_tfm_ctx(req->tfm);
ctx->ops->step(req);
+ return;
+out_unlock:
+ spin_unlock_bh(&engine->lock);
+}
+
+static int mv_cesa_std_process(struct mv_cesa_engine *engine, u32 status)
+{
+ struct crypto_async_request *req;
+ struct mv_cesa_ctx *ctx;
+ int res;
+
+ req = engine->req;
+ ctx = crypto_tfm_ctx(req->tfm);
+ res = ctx->ops->process(req, status);
+
+ if (res == 0) {
+ ctx->ops->complete(req);
+ mv_cesa_engine_enqueue_complete_request(engine, req);
+ } else if (res == -EINPROGRESS) {
+ ctx->ops->step(req);
+ } else {
+ ctx->ops->complete(req);
+ }
+
+ return res;
+}
+
+static int mv_cesa_int_process(struct mv_cesa_engine *engine, u32 status)
+{
+ if (engine->chain.first && engine->chain.last)
+ return mv_cesa_tdma_process(engine, status);
+ return mv_cesa_std_process(engine, status);
+}
+
+static inline void mv_cesa_complete_req(struct mv_cesa_ctx *ctx,
+ struct crypto_async_request *req, int res)
+{
+ ctx->ops->cleanup(req);
+ local_bh_disable();
+ req->complete(req, res);
+ local_bh_enable();
}
static irqreturn_t mv_cesa_int(int irq, void *priv)
@@ -83,26 +143,31 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
writel(~status, engine->regs + CESA_SA_FPGA_INT_STATUS);
writel(~status, engine->regs + CESA_SA_INT_STATUS);
+ /* Process fetched requests */
+ res = mv_cesa_int_process(engine, status & mask);
ret = IRQ_HANDLED;
+
spin_lock_bh(&engine->lock);
req = engine->req;
+ if (res != -EINPROGRESS)
+ engine->req = NULL;
spin_unlock_bh(&engine->lock);
- if (req) {
- ctx = crypto_tfm_ctx(req->tfm);
- res = ctx->ops->process(req, status & mask);
- if (res != -EINPROGRESS) {
- spin_lock_bh(&engine->lock);
- engine->req = NULL;
- mv_cesa_dequeue_req_unlocked(engine);
- spin_unlock_bh(&engine->lock);
- ctx->ops->complete(req);
- ctx->ops->cleanup(req);
- local_bh_disable();
- req->complete(req, res);
- local_bh_enable();
- } else {
- ctx->ops->step(req);
- }
+
+ ctx = crypto_tfm_ctx(req->tfm);
+
+ if (res && res != -EINPROGRESS)
+ mv_cesa_complete_req(ctx, req, res);
+
+ /* Launch the next pending request */
+ mv_cesa_rearm_engine(engine);
+
+ /* Iterate over the complete queue */
+ while (true) {
+ req = mv_cesa_engine_dequeue_complete_request(engine);
+ if (!req)
+ break;
+
+ mv_cesa_complete_req(ctx, req, 0);
}
}
@@ -116,16 +181,15 @@ int mv_cesa_queue_req(struct crypto_async_request *req,
struct mv_cesa_engine *engine = creq->engine;
spin_lock_bh(&engine->lock);
+ if (mv_cesa_req_get_type(creq) == CESA_DMA_REQ)
+ mv_cesa_tdma_chain(engine, creq);
ret = crypto_enqueue_request(&engine->queue, req);
spin_unlock_bh(&engine->lock);
if (ret != -EINPROGRESS)
return ret;
- spin_lock_bh(&engine->lock);
- if (!engine->req)
- mv_cesa_dequeue_req_unlocked(engine);
- spin_unlock_bh(&engine->lock);
+ mv_cesa_rearm_engine(engine);
return -EINPROGRESS;
}
@@ -496,6 +560,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
crypto_init_queue(&engine->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
atomic_set(&engine->load, 0);
+ INIT_LIST_HEAD(&engine->complete_queue);
}
cesa_dev = cesa;
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 5626aa7..e0fee1f 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -271,7 +271,9 @@ struct mv_cesa_op_ctx {
/* TDMA descriptor flags */
#define CESA_TDMA_DST_IN_SRAM BIT(31)
#define CESA_TDMA_SRC_IN_SRAM BIT(30)
-#define CESA_TDMA_TYPE_MSK GENMASK(29, 0)
+#define CESA_TDMA_END_OF_REQ BIT(29)
+#define CESA_TDMA_NOT_CHAIN BIT(28)
+#define CESA_TDMA_TYPE_MSK GENMASK(27, 0)
#define CESA_TDMA_DUMMY 0
#define CESA_TDMA_DATA 1
#define CESA_TDMA_OP 2
@@ -431,6 +433,9 @@ struct mv_cesa_dev {
* SRAM
* @queue: fifo of the pending crypto requests
* @load: engine load counter, useful for load balancing
+ * @chain: list of the current tdma descriptors being processed
+ * by this engine.
+ * @complete_queue: fifo of the processed requests by the engine
*
* Structure storing CESA engine information.
*/
@@ -448,6 +453,8 @@ struct mv_cesa_engine {
struct gen_pool *pool;
struct crypto_queue queue;
atomic_t load;
+ struct mv_cesa_tdma_chain chain;
+ struct list_head complete_queue;
};
/**
@@ -618,6 +625,28 @@ struct mv_cesa_ahash_req {
extern struct mv_cesa_dev *cesa_dev;
+
+static inline void mv_cesa_engine_enqueue_complete_request(
+ struct mv_cesa_engine *engine, struct crypto_async_request *req)
+{
+ list_add_tail(&req->list, &engine->complete_queue);
+}
+
+static inline struct crypto_async_request *
+mv_cesa_engine_dequeue_complete_request(struct mv_cesa_engine *engine)
+{
+ struct crypto_async_request *req;
+
+ req = list_first_entry_or_null(&engine->complete_queue,
+ struct crypto_async_request,
+ list);
+ if (req)
+ list_del(&req->list);
+
+ return req;
+}
+
+
static inline enum mv_cesa_req_type
mv_cesa_req_get_type(struct mv_cesa_req *req)
{
@@ -699,6 +728,10 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
int mv_cesa_queue_req(struct crypto_async_request *req,
struct mv_cesa_req *creq);
+struct crypto_async_request *mv_cesa_dequeue_req_locked(
+ struct mv_cesa_engine *engine,
+ struct crypto_async_request **backlog);
+
static inline struct mv_cesa_engine *mv_cesa_select_engine(int weight)
{
int i;
@@ -804,6 +837,9 @@ static inline int mv_cesa_dma_process(struct mv_cesa_req *dreq,
void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
struct mv_cesa_engine *engine);
void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq);
+void mv_cesa_tdma_chain(struct mv_cesa_engine *engine,
+ struct mv_cesa_req *dreq);
+int mv_cesa_tdma_process(struct mv_cesa_engine *engine, u32 status);
static inline void
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index 02aa38f..9033191 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -225,7 +225,6 @@ mv_cesa_ablkcipher_complete(struct crypto_async_request *req)
static const struct mv_cesa_req_ops mv_cesa_ablkcipher_req_ops = {
.step = mv_cesa_ablkcipher_step,
.process = mv_cesa_ablkcipher_process,
- .prepare = mv_cesa_ablkcipher_prepare,
.cleanup = mv_cesa_ablkcipher_req_cleanup,
.complete = mv_cesa_ablkcipher_complete,
};
@@ -384,6 +383,7 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
goto err_free_tdma;
dreq->chain = chain;
+ dreq->chain.last->flags |= CESA_TDMA_END_OF_REQ;
return 0;
@@ -441,7 +441,6 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_OP_CRYPT_ONLY,
CESA_SA_DESC_CFG_OP_MSK);
- /* TODO: add a threshold for DMA usage */
if (cesa_dev->caps->has_tdma)
ret = mv_cesa_ablkcipher_dma_req_init(req, tmpl);
else
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index 5946a69..c2ff353 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -172,6 +172,9 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
for (i = 0; i < digsize / 4; i++)
writel_relaxed(creq->state[i], engine->regs + CESA_IVDIG(i));
+ mv_cesa_adjust_op(engine, &creq->op_tmpl);
+ memcpy_toio(engine->sram, &creq->op_tmpl, sizeof(creq->op_tmpl));
+
if (creq->cache_ptr)
memcpy_toio(engine->sram + CESA_SA_DATA_SRAM_OFFSET,
creq->cache, creq->cache_ptr);
@@ -282,6 +285,9 @@ static void mv_cesa_ahash_step(struct crypto_async_request *req)
{
struct ahash_request *ahashreq = ahash_request_cast(req);
struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
+ struct mv_cesa_engine *engine = creq->req.base.engine;
+ unsigned int digsize;
+ int i;
if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
mv_cesa_dma_step(&creq->req.base);
@@ -367,7 +373,6 @@ static void mv_cesa_ahash_req_cleanup(struct crypto_async_request *req)
static const struct mv_cesa_req_ops mv_cesa_ahash_req_ops = {
.step = mv_cesa_ahash_step,
.process = mv_cesa_ahash_process,
- .prepare = mv_cesa_ahash_prepare,
.cleanup = mv_cesa_ahash_req_cleanup,
.complete = mv_cesa_ahash_complete,
};
@@ -648,6 +653,8 @@ static int mv_cesa_ahash_dma_req_init(struct ahash_request *req)
else
creq->cache_ptr = 0;
+ dreq->chain.last->flags |= (CESA_TDMA_END_OF_REQ | CESA_TDMA_NOT_CHAIN);
+
return 0;
err_free_tdma:
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index 9a424f9..ae50545 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -98,6 +98,87 @@ void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
}
}
+void
+mv_cesa_tdma_chain(struct mv_cesa_engine *engine, struct mv_cesa_req *dreq)
+{
+ if (engine->chain.first == NULL && engine->chain.last == NULL) {
+ engine->chain.first = dreq->chain.first;
+ engine->chain.last = dreq->chain.last;
+ } else {
+ struct mv_cesa_tdma_desc *last;
+
+ last = engine->chain.last;
+ last->next = dreq->chain.first;
+ engine->chain.last = dreq->chain.last;
+ if (!(last->flags & CESA_TDMA_NOT_CHAIN))
+ last->next_dma = dreq->chain.first->cur_dma;
+ }
+}
+
+int
+mv_cesa_tdma_process(struct mv_cesa_engine *engine, u32 status)
+{
+ struct crypto_async_request *req = NULL;
+ struct mv_cesa_tdma_desc *tdma = NULL, *next = NULL;
+ dma_addr_t tdma_cur;
+ int res = 0;
+
+ tdma_cur = readl(engine->regs + CESA_TDMA_CUR);
+
+ for (tdma = engine->chain.first; tdma; tdma = next) {
+ spin_lock_bh(&engine->lock);
+ next = tdma->next;
+ spin_unlock_bh(&engine->lock);
+
+ if (tdma->flags & CESA_TDMA_END_OF_REQ) {
+ struct crypto_async_request *backlog = NULL;
+ struct mv_cesa_ctx *ctx;
+
+ spin_lock_bh(&engine->lock);
+ /*
+ * if req is NULL, this means we're processing the
+ * request in engine->req.
+ */
+ if (!req)
+ req = engine->req;
+ else
+ req = mv_cesa_dequeue_req_locked(engine,
+ &backlog);
+
+ /* Re-chaining to the next request */
+ engine->chain.first = tdma->next;
+ tdma->next = NULL;
+
+ /* If this is the last request, clear the chain */
+ if (engine->chain.first == NULL)
+ engine->chain.last = NULL;
+ spin_unlock_bh(&engine->lock);
+
+ ctx = crypto_tfm_ctx(req->tfm);
+ res = ctx->ops->process(req, status);
+ ctx->ops->complete(req);
+
+ if (res == 0)
+ mv_cesa_engine_enqueue_complete_request(engine,
+ req);
+
+ if (backlog)
+ backlog->complete(backlog, -EINPROGRESS);
+ }
+ if (res || tdma->cur_dma == tdma_cur)
+ break;
+ }
+
+ if (res) {
+ spin_lock_bh(&engine->lock);
+ engine->req = req;
+ spin_unlock_bh(&engine->lock);
+ }
+
+ return res;
+}
+
+
static struct mv_cesa_tdma_desc *
mv_cesa_dma_add_desc(struct mv_cesa_tdma_chain *chain, gfp_t flags)
{
--
2.7.4
^ permalink raw reply related
* [PATCH 6/7] crypto: marvell: Adding load balancing between engines
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
This commits adds support for fine grained load balancing on
multi-engine IPs. The engine is pre-selected based on its current load
and on the weight of the crypto request that is about to be processed.
The global crypto queue is also moved to each engine. These changes are
useful for preparing the code to support TDMA chaining between crypto
requests, because each tdma chain will be handled per engine. By using
a crypto queue per engine, we make sure that we keep the state of the
tdma chain synchronized with the crypto queue. We also reduce contention
on 'cesa_dev->lock' and improve parallelism.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 30 +++++++++----------
drivers/crypto/marvell/cesa.h | 26 +++++++++++++++--
drivers/crypto/marvell/cipher.c | 59 ++++++++++++++++++-------------------
drivers/crypto/marvell/hash.c | 65 +++++++++++++++++++----------------------
4 files changed, 97 insertions(+), 83 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index af96426..f9e6688 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -45,11 +45,9 @@ static void mv_cesa_dequeue_req_unlocked(struct mv_cesa_engine *engine)
struct crypto_async_request *req, *backlog;
struct mv_cesa_ctx *ctx;
- spin_lock_bh(&cesa_dev->lock);
- backlog = crypto_get_backlog(&cesa_dev->queue);
- req = crypto_dequeue_request(&cesa_dev->queue);
+ backlog = crypto_get_backlog(&engine->queue);
+ req = crypto_dequeue_request(&engine->queue);
engine->req = req;
- spin_unlock_bh(&cesa_dev->lock);
if (!req)
return;
@@ -58,7 +56,6 @@ static void mv_cesa_dequeue_req_unlocked(struct mv_cesa_engine *engine)
backlog->complete(backlog, -EINPROGRESS);
ctx = crypto_tfm_ctx(req->tfm);
- ctx->ops->prepare(req, engine);
ctx->ops->step(req);
}
@@ -116,21 +113,19 @@ int mv_cesa_queue_req(struct crypto_async_request *req,
struct mv_cesa_req *creq)
{
int ret;
- int i;
+ struct mv_cesa_engine *engine = creq->engine;
- spin_lock_bh(&cesa_dev->lock);
- ret = crypto_enqueue_request(&cesa_dev->queue, req);
- spin_unlock_bh(&cesa_dev->lock);
+ spin_lock_bh(&engine->lock);
+ ret = crypto_enqueue_request(&engine->queue, req);
+ spin_unlock_bh(&engine->lock);
if (ret != -EINPROGRESS)
return ret;
- for (i = 0; i < cesa_dev->caps->nengines; i++) {
- spin_lock_bh(&cesa_dev->engines[i].lock);
- if (!cesa_dev->engines[i].req)
- mv_cesa_dequeue_req_unlocked(&cesa_dev->engines[i]);
- spin_unlock_bh(&cesa_dev->engines[i].lock);
- }
+ spin_lock_bh(&engine->lock);
+ if (!engine->req)
+ mv_cesa_dequeue_req_unlocked(engine);
+ spin_unlock_bh(&engine->lock);
return -EINPROGRESS;
}
@@ -425,7 +420,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
return -ENOMEM;
spin_lock_init(&cesa->lock);
- crypto_init_queue(&cesa->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
+
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
cesa->regs = devm_ioremap_resource(dev, res);
if (IS_ERR(cesa->regs))
@@ -498,6 +493,9 @@ static int mv_cesa_probe(struct platform_device *pdev)
engine);
if (ret)
goto err_cleanup;
+
+ crypto_init_queue(&engine->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
+ atomic_set(&engine->load, 0);
}
cesa_dev = cesa;
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 32de08b..5626aa7 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -400,7 +400,6 @@ struct mv_cesa_dev_dma {
* @regs: device registers
* @sram_size: usable SRAM size
* @lock: device lock
- * @queue: crypto request queue
* @engines: array of engines
* @dma: dma pools
*
@@ -412,7 +411,6 @@ struct mv_cesa_dev {
struct device *dev;
unsigned int sram_size;
spinlock_t lock;
- struct crypto_queue queue;
struct mv_cesa_engine *engines;
struct mv_cesa_dev_dma *dma;
};
@@ -431,6 +429,8 @@ struct mv_cesa_dev {
* @int_mask: interrupt mask cache
* @pool: memory pool pointing to the memory region reserved in
* SRAM
+ * @queue: fifo of the pending crypto requests
+ * @load: engine load counter, useful for load balancing
*
* Structure storing CESA engine information.
*/
@@ -446,6 +446,8 @@ struct mv_cesa_engine {
size_t max_req_len;
u32 int_mask;
struct gen_pool *pool;
+ struct crypto_queue queue;
+ atomic_t load;
};
/**
@@ -697,6 +699,26 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
int mv_cesa_queue_req(struct crypto_async_request *req,
struct mv_cesa_req *creq);
+static inline struct mv_cesa_engine *mv_cesa_select_engine(int weight)
+{
+ int i;
+ u32 min_load = U32_MAX;
+ struct mv_cesa_engine *selected = NULL;
+
+ for (i = 0; i < cesa_dev->caps->nengines; i++) {
+ struct mv_cesa_engine *engine = cesa_dev->engines + i;
+ u32 load = atomic_read(&engine->load);
+ if (load < min_load) {
+ min_load = load;
+ selected = engine;
+ }
+ }
+
+ atomic_add(weight, &selected->load);
+
+ return selected;
+}
+
/*
* Helper function that indicates whether a crypto request needs to be
* cleaned up or not after being enqueued using mv_cesa_queue_req().
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index fbaae2f..02aa38f 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -89,6 +89,9 @@ static void mv_cesa_ablkcipher_std_step(struct ablkcipher_request *req)
size_t len = min_t(size_t, req->nbytes - sreq->offset,
CESA_SA_SRAM_PAYLOAD_SIZE);
+ mv_cesa_adjust_op(engine, &sreq->op);
+ memcpy_toio(engine->sram, &sreq->op, sizeof(sreq->op));
+
len = sg_pcopy_to_buffer(req->src, creq->src_nents,
engine->sram + CESA_SA_DATA_SRAM_OFFSET,
len, sreq->offset);
@@ -167,12 +170,9 @@ mv_cesa_ablkcipher_std_prepare(struct ablkcipher_request *req)
{
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
- struct mv_cesa_engine *engine = sreq->base.engine;
sreq->size = 0;
sreq->offset = 0;
- mv_cesa_adjust_op(engine, &sreq->op);
- memcpy_toio(engine->sram, &sreq->op, sizeof(sreq->op));
}
static inline void mv_cesa_ablkcipher_prepare(struct crypto_async_request *req,
@@ -205,6 +205,7 @@ mv_cesa_ablkcipher_complete(struct crypto_async_request *req)
struct mv_cesa_engine *engine = creq->req.base.engine;
unsigned int ivsize;
+ atomic_sub(ablkreq->nbytes, &engine->load);
ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq));
if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
@@ -449,29 +450,43 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
return ret;
}
-static int mv_cesa_des_op(struct ablkcipher_request *req,
- struct mv_cesa_op_ctx *tmpl)
+static int mv_cesa_ablkcipher_queue_req(struct ablkcipher_request *req,
+ struct mv_cesa_op_ctx *tmpl)
{
- struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
- struct mv_cesa_des_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
int ret;
-
- mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_CRYPTM_DES,
- CESA_SA_DESC_CFG_CRYPTM_MSK);
-
- memcpy(tmpl->ctx.blkcipher.key, ctx->key, DES_KEY_SIZE);
+ struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
+ struct mv_cesa_engine *engine;
ret = mv_cesa_ablkcipher_req_init(req, tmpl);
if (ret)
return ret;
+ engine = mv_cesa_select_engine(req->nbytes);
+ mv_cesa_ablkcipher_prepare(&req->base, engine);
+
ret = mv_cesa_queue_req(&req->base, &creq->req.base);
+
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ablkcipher_cleanup(req);
return ret;
}
+static int mv_cesa_des_op(struct ablkcipher_request *req,
+ struct mv_cesa_op_ctx *tmpl)
+{
+ struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
+ struct mv_cesa_des_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ int ret;
+
+ mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_CRYPTM_DES,
+ CESA_SA_DESC_CFG_CRYPTM_MSK);
+
+ memcpy(tmpl->ctx.blkcipher.key, ctx->key, DES_KEY_SIZE);
+
+ return mv_cesa_ablkcipher_queue_req(req, tmpl);
+}
+
static int mv_cesa_ecb_des_encrypt(struct ablkcipher_request *req)
{
struct mv_cesa_op_ctx tmpl;
@@ -582,15 +597,7 @@ static int mv_cesa_des3_op(struct ablkcipher_request *req,
memcpy(tmpl->ctx.blkcipher.key, ctx->key, DES3_EDE_KEY_SIZE);
- ret = mv_cesa_ablkcipher_req_init(req, tmpl);
- if (ret)
- return ret;
-
- ret = mv_cesa_queue_req(&req->base, &creq->req.base);
- if (mv_cesa_req_needs_cleanup(&req->base, ret))
- mv_cesa_ablkcipher_cleanup(req);
-
- return ret;
+ return mv_cesa_ablkcipher_queue_req(req, tmpl);
}
static int mv_cesa_ecb_des3_ede_encrypt(struct ablkcipher_request *req)
@@ -725,15 +732,7 @@ static int mv_cesa_aes_op(struct ablkcipher_request *req,
CESA_SA_DESC_CFG_CRYPTM_MSK |
CESA_SA_DESC_CFG_AES_LEN_MSK);
- ret = mv_cesa_ablkcipher_req_init(req, tmpl);
- if (ret)
- return ret;
-
- ret = mv_cesa_queue_req(&req->base, &creq->req.base);
- if (mv_cesa_req_needs_cleanup(&req->base, ret))
- mv_cesa_ablkcipher_cleanup(req);
-
- return ret;
+ return mv_cesa_ablkcipher_queue_req(req, tmpl);
}
static int mv_cesa_ecb_aes_encrypt(struct ablkcipher_request *req)
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index f7f84cc..5946a69 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -162,6 +162,15 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
unsigned int new_cache_ptr = 0;
u32 frag_mode;
size_t len;
+ unsigned int digsize;
+ int i;
+
+ mv_cesa_adjust_op(engine, &creq->op_tmpl);
+ memcpy_toio(engine->sram, &creq->op_tmpl, sizeof(creq->op_tmpl));
+
+ digsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(req));
+ for (i = 0; i < digsize / 4; i++)
+ writel_relaxed(creq->state[i], engine->regs + CESA_IVDIG(i));
if (creq->cache_ptr)
memcpy_toio(engine->sram + CESA_SA_DATA_SRAM_OFFSET,
@@ -265,11 +274,8 @@ static void mv_cesa_ahash_std_prepare(struct ahash_request *req)
{
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
struct mv_cesa_ahash_std_req *sreq = &creq->req.std;
- struct mv_cesa_engine *engine = sreq->base.engine;
sreq->offset = 0;
- mv_cesa_adjust_op(engine, &creq->op_tmpl);
- memcpy_toio(engine->sram, &creq->op_tmpl, sizeof(creq->op_tmpl));
}
static void mv_cesa_ahash_step(struct crypto_async_request *req)
@@ -329,6 +335,8 @@ static void mv_cesa_ahash_complete(struct crypto_async_request *req)
result[i] = cpu_to_be32(creq->state[i]);
}
}
+
+ atomic_sub(ahashreq->nbytes, &engine->load);
}
static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
@@ -336,8 +344,6 @@ static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
{
struct ahash_request *ahashreq = ahash_request_cast(req);
struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
- unsigned int digsize;
- int i;
creq->req.base.engine = engine;
@@ -345,10 +351,6 @@ static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
mv_cesa_ahash_dma_prepare(ahashreq);
else
mv_cesa_ahash_std_prepare(ahashreq);
-
- digsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(ahashreq));
- for (i = 0; i < digsize / 4; i++)
- writel_relaxed(creq->state[i], engine->regs + CESA_IVDIG(i));
}
static void mv_cesa_ahash_req_cleanup(struct crypto_async_request *req)
@@ -682,13 +684,13 @@ static int mv_cesa_ahash_req_init(struct ahash_request *req, bool *cached)
return ret;
}
-static int mv_cesa_ahash_update(struct ahash_request *req)
+static int mv_cesa_ahash_queue_req(struct ahash_request *req)
{
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
+ struct mv_cesa_engine *engine;
bool cached = false;
int ret;
- creq->len += req->nbytes;
ret = mv_cesa_ahash_req_init(req, &cached);
if (ret)
return ret;
@@ -696,13 +698,28 @@ static int mv_cesa_ahash_update(struct ahash_request *req)
if (cached)
return 0;
+ engine = mv_cesa_select_engine(req->nbytes);
+ mv_cesa_ahash_prepare(&req->base, engine);
+
ret = mv_cesa_queue_req(&req->base, &creq->req.base);
+
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ahash_cleanup(req);
return ret;
}
+static int mv_cesa_ahash_update(struct ahash_request *req)
+{
+ struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
+ bool cached = false;
+ int ret;
+
+ creq->len += req->nbytes;
+
+ return mv_cesa_ahash_queue_req(req);
+}
+
static int mv_cesa_ahash_final(struct ahash_request *req)
{
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
@@ -714,18 +731,7 @@ static int mv_cesa_ahash_final(struct ahash_request *req)
creq->last_req = true;
req->nbytes = 0;
- ret = mv_cesa_ahash_req_init(req, &cached);
- if (ret)
- return ret;
-
- if (cached)
- return 0;
-
- ret = mv_cesa_queue_req(&req->base, &creq->req.base);
- if (mv_cesa_req_needs_cleanup(&req->base, ret))
- mv_cesa_ahash_cleanup(req);
-
- return ret;
+ return mv_cesa_ahash_queue_req(req);
}
static int mv_cesa_ahash_finup(struct ahash_request *req)
@@ -739,18 +745,7 @@ static int mv_cesa_ahash_finup(struct ahash_request *req)
mv_cesa_set_mac_op_total_len(tmpl, creq->len);
creq->last_req = true;
- ret = mv_cesa_ahash_req_init(req, &cached);
- if (ret)
- return ret;
-
- if (cached)
- return 0;
-
- ret = mv_cesa_queue_req(&req->base, &creq->req.base);
- if (mv_cesa_req_needs_cleanup(&req->base, ret))
- mv_cesa_ahash_cleanup(req);
-
- return ret;
+ return mv_cesa_ahash_queue_req(req);
}
static int mv_cesa_ahash_export(struct ahash_request *req, void *hash,
--
2.7.4
^ permalink raw reply related
* [PATCH 5/7] crypto: marvell: Adding a complete operation for async requests
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
So far, the 'process' operation was used to check if the current request
was correctly handled by the engine, if it was the case it copied
information from the SRAM to the main memory. Now, we split this
operation. We keep the 'process' operation, which still checks if the
request was correctly handled by the engine or not, then we add a new
operation for completion. The 'complete' method copies the content of
the SRAM to memory. This will soon become useful if we want to call
the process and the complete operations from different locations
depending on the type of the request (different cleanup logic).
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 1 +
drivers/crypto/marvell/cesa.h | 3 +++
drivers/crypto/marvell/cipher.c | 47 ++++++++++++++++++++++++-----------------
drivers/crypto/marvell/hash.c | 22 ++++++++++---------
4 files changed, 44 insertions(+), 29 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index fe04d1b..af96426 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -98,6 +98,7 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
engine->req = NULL;
mv_cesa_dequeue_req_unlocked(engine);
spin_unlock_bh(&engine->lock);
+ ctx->ops->complete(req);
ctx->ops->cleanup(req);
local_bh_disable();
req->complete(req, res);
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 158ff82..32de08b 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -456,6 +456,8 @@ struct mv_cesa_engine {
* code)
* @step: launch the crypto operation on the next chunk
* @cleanup: cleanup the crypto request (release associated data)
+ * @complete: complete the request, i.e copy result from sram or contexts
+ * when it is needed.
*/
struct mv_cesa_req_ops {
void (*prepare)(struct crypto_async_request *req,
@@ -463,6 +465,7 @@ struct mv_cesa_req_ops {
int (*process)(struct crypto_async_request *req, u32 status);
void (*step)(struct crypto_async_request *req);
void (*cleanup)(struct crypto_async_request *req);
+ void (*complete)(struct crypto_async_request *req);
};
/**
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index 15d2c5a..fbaae2f 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -118,7 +118,6 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
struct mv_cesa_engine *engine = sreq->base.engine;
size_t len;
- unsigned int ivsize;
len = sg_pcopy_from_buffer(req->dst, creq->dst_nents,
engine->sram + CESA_SA_DATA_SRAM_OFFSET,
@@ -128,10 +127,6 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
if (sreq->offset < req->nbytes)
return -EINPROGRESS;
- ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
- memcpy_fromio(req->info,
- engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET, ivsize);
-
return 0;
}
@@ -141,21 +136,9 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
- if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
- int ret;
- struct mv_cesa_req *basereq;
- unsigned int ivsize;
-
- ret = mv_cesa_dma_process(&creq->req.base, status);
- if (ret)
- return ret;
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
+ return mv_cesa_dma_process(&creq->req.base, status);
- basereq = &creq->req.base;
- ivsize = crypto_ablkcipher_ivsize(
- crypto_ablkcipher_reqtfm(ablkreq));
- memcpy_fromio(ablkreq->info, basereq->chain.last->data, ivsize);
- return ret;
- }
return mv_cesa_ablkcipher_std_process(ablkreq, status);
}
@@ -197,6 +180,7 @@ static inline void mv_cesa_ablkcipher_prepare(struct crypto_async_request *req,
{
struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
+
creq->req.base.engine = engine;
if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
@@ -213,11 +197,36 @@ mv_cesa_ablkcipher_req_cleanup(struct crypto_async_request *req)
mv_cesa_ablkcipher_cleanup(ablkreq);
}
+static void
+mv_cesa_ablkcipher_complete(struct crypto_async_request *req)
+{
+ struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
+ struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
+ struct mv_cesa_engine *engine = creq->req.base.engine;
+ unsigned int ivsize;
+
+ ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq));
+
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
+ struct mv_cesa_req *basereq;
+
+ basereq = &creq->req.base;
+ ivsize = crypto_ablkcipher_ivsize(
+ crypto_ablkcipher_reqtfm(ablkreq));
+ memcpy_fromio(ablkreq->info, basereq->chain.last->data, ivsize);
+ } else {
+ memcpy_fromio(ablkreq->info,
+ engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET,
+ ivsize);
+ }
+}
+
static const struct mv_cesa_req_ops mv_cesa_ablkcipher_req_ops = {
.step = mv_cesa_ablkcipher_step,
.process = mv_cesa_ablkcipher_process,
.prepare = mv_cesa_ablkcipher_prepare,
.cleanup = mv_cesa_ablkcipher_req_cleanup,
+ .complete = mv_cesa_ablkcipher_complete,
};
static int mv_cesa_ablkcipher_cra_init(struct crypto_tfm *tfm)
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index cc7c5b0..f7f84cc 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -287,17 +287,20 @@ static int mv_cesa_ahash_process(struct crypto_async_request *req, u32 status)
{
struct ahash_request *ahashreq = ahash_request_cast(req);
struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
- struct mv_cesa_engine *engine = creq->req.base.engine;
- unsigned int digsize;
- int ret, i;
if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
- ret = mv_cesa_dma_process(&creq->req.base, status);
- else
- ret = mv_cesa_ahash_std_process(ahashreq, status);
+ return mv_cesa_dma_process(&creq->req.base, status);
- if (ret == -EINPROGRESS)
- return ret;
+ return mv_cesa_ahash_std_process(ahashreq, status);
+}
+
+static void mv_cesa_ahash_complete(struct crypto_async_request *req)
+{
+ struct ahash_request *ahashreq = ahash_request_cast(req);
+ struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
+ struct mv_cesa_engine *engine = creq->req.base.engine;
+ unsigned int digsize;
+ int i;
digsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(ahashreq));
for (i = 0; i < digsize / 4; i++)
@@ -326,8 +329,6 @@ static int mv_cesa_ahash_process(struct crypto_async_request *req, u32 status)
result[i] = cpu_to_be32(creq->state[i]);
}
}
-
- return ret;
}
static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
@@ -366,6 +367,7 @@ static const struct mv_cesa_req_ops mv_cesa_ahash_req_ops = {
.process = mv_cesa_ahash_process,
.prepare = mv_cesa_ahash_prepare,
.cleanup = mv_cesa_ahash_req_cleanup,
+ .complete = mv_cesa_ahash_complete,
};
static int mv_cesa_ahash_init(struct ahash_request *req,
--
2.7.4
^ permalink raw reply related
* [PATCH 4/7] crypto: marvell: Moving the tdma chain out of mv_cesa_tdma_req
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
Actually the only way to access the tdma chain is to use the 'req' union
from a mv_cesa_{ablkcipher,ahash}. This will soon become a problem if we
want to handle the TDMA chaining vs standard/non-DMA processing in a
generic way (with generic functions at the cesa.c level detecting
whether the request should be queued at the DMA level or not). Hence the
decision to move the chain field a the mv_cesa_req level at the expense
of adding 2 void * fields to all request contexts (including non-DMA
ones). To limit the overhead, we get rid of the type field, which can
now be deduced from the req->chain.first value.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 3 ++-
drivers/crypto/marvell/cesa.h | 31 +++++++++++++------------------
drivers/crypto/marvell/cipher.c | 40 ++++++++++++++++++++++------------------
drivers/crypto/marvell/hash.c | 36 +++++++++++++++---------------------
drivers/crypto/marvell/tdma.c | 8 ++++----
5 files changed, 56 insertions(+), 62 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index 93700cd..fe04d1b 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -111,7 +111,8 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
return ret;
}
-int mv_cesa_queue_req(struct crypto_async_request *req)
+int mv_cesa_queue_req(struct crypto_async_request *req,
+ struct mv_cesa_req *creq)
{
int ret;
int i;
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 74b84bd..158ff82 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -509,21 +509,11 @@ enum mv_cesa_req_type {
/**
* struct mv_cesa_req - CESA request
- * @type: request type
* @engine: engine associated with this request
+ * @chain: list of tdma descriptors associated with this request
*/
struct mv_cesa_req {
- enum mv_cesa_req_type type;
struct mv_cesa_engine *engine;
-};
-
-/**
- * struct mv_cesa_tdma_req - CESA TDMA request
- * @base: base information
- * @chain: TDMA chain
- */
-struct mv_cesa_tdma_req {
- struct mv_cesa_req base;
struct mv_cesa_tdma_chain chain;
};
@@ -562,7 +552,6 @@ struct mv_cesa_ablkcipher_std_req {
struct mv_cesa_ablkcipher_req {
union {
struct mv_cesa_req base;
- struct mv_cesa_tdma_req dma;
struct mv_cesa_ablkcipher_std_req std;
} req;
int src_nents;
@@ -587,7 +576,6 @@ struct mv_cesa_ahash_std_req {
* @cache_dma: DMA address of the cache buffer
*/
struct mv_cesa_ahash_dma_req {
- struct mv_cesa_tdma_req base;
u8 *padding;
dma_addr_t padding_dma;
u8 *cache;
@@ -625,6 +613,12 @@ struct mv_cesa_ahash_req {
extern struct mv_cesa_dev *cesa_dev;
+static inline enum mv_cesa_req_type
+mv_cesa_req_get_type(struct mv_cesa_req *req)
+{
+ return req->chain.first ? CESA_DMA_REQ : CESA_STD_REQ;
+}
+
static inline void mv_cesa_update_op_cfg(struct mv_cesa_op_ctx *op,
u32 cfg, u32 mask)
{
@@ -697,7 +691,8 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
CESA_SA_DESC_CFG_FIRST_FRAG;
}
-int mv_cesa_queue_req(struct crypto_async_request *req);
+int mv_cesa_queue_req(struct crypto_async_request *req,
+ struct mv_cesa_req *creq);
/*
* Helper function that indicates whether a crypto request needs to be
@@ -767,9 +762,9 @@ static inline bool mv_cesa_req_dma_iter_next_op(struct mv_cesa_dma_iter *iter)
return iter->op_len;
}
-void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq);
+void mv_cesa_dma_step(struct mv_cesa_req *dreq);
-static inline int mv_cesa_dma_process(struct mv_cesa_tdma_req *dreq,
+static inline int mv_cesa_dma_process(struct mv_cesa_req *dreq,
u32 status)
{
if (!(status & CESA_SA_INT_ACC0_IDMA_DONE))
@@ -781,10 +776,10 @@ static inline int mv_cesa_dma_process(struct mv_cesa_tdma_req *dreq,
return 0;
}
-void mv_cesa_dma_prepare(struct mv_cesa_tdma_req *dreq,
+void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
struct mv_cesa_engine *engine);
+void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq);
-void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq);
static inline void
mv_cesa_tdma_desc_iter_init(struct mv_cesa_tdma_chain *chain)
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index f42620e..15d2c5a 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -70,14 +70,14 @@ mv_cesa_ablkcipher_dma_cleanup(struct ablkcipher_request *req)
dma_unmap_sg(cesa_dev->dev, req->src, creq->src_nents,
DMA_BIDIRECTIONAL);
}
- mv_cesa_dma_cleanup(&creq->req.dma);
+ mv_cesa_dma_cleanup(&creq->req.base);
}
static inline void mv_cesa_ablkcipher_cleanup(struct ablkcipher_request *req)
{
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
mv_cesa_ablkcipher_dma_cleanup(req);
}
@@ -141,19 +141,19 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
- if (creq->req.base.type == CESA_DMA_REQ) {
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ) {
int ret;
- struct mv_cesa_tdma_req *dreq;
+ struct mv_cesa_req *basereq;
unsigned int ivsize;
- ret = mv_cesa_dma_process(&creq->req.dma, status);
+ ret = mv_cesa_dma_process(&creq->req.base, status);
if (ret)
return ret;
- dreq = &creq->req.dma;
+ basereq = &creq->req.base;
ivsize = crypto_ablkcipher_ivsize(
crypto_ablkcipher_reqtfm(ablkreq));
- memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
+ memcpy_fromio(ablkreq->info, basereq->chain.last->data, ivsize);
return ret;
}
return mv_cesa_ablkcipher_std_process(ablkreq, status);
@@ -164,8 +164,8 @@ static void mv_cesa_ablkcipher_step(struct crypto_async_request *req)
struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
- if (creq->req.base.type == CESA_DMA_REQ)
- mv_cesa_dma_step(&creq->req.dma);
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
+ mv_cesa_dma_step(&creq->req.base);
else
mv_cesa_ablkcipher_std_step(ablkreq);
}
@@ -174,9 +174,9 @@ static inline void
mv_cesa_ablkcipher_dma_prepare(struct ablkcipher_request *req)
{
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
- struct mv_cesa_tdma_req *dreq = &creq->req.dma;
+ struct mv_cesa_req *dreq = &creq->req.base;
- mv_cesa_dma_prepare(dreq, dreq->base.engine);
+ mv_cesa_dma_prepare(dreq, dreq->engine);
}
static inline void
@@ -199,7 +199,7 @@ static inline void mv_cesa_ablkcipher_prepare(struct crypto_async_request *req,
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
creq->req.base.engine = engine;
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
mv_cesa_ablkcipher_dma_prepare(ablkreq);
else
mv_cesa_ablkcipher_std_prepare(ablkreq);
@@ -302,14 +302,13 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
GFP_KERNEL : GFP_ATOMIC;
- struct mv_cesa_tdma_req *dreq = &creq->req.dma;
+ struct mv_cesa_req *dreq = &creq->req.base;
struct mv_cesa_ablkcipher_dma_iter iter;
struct mv_cesa_tdma_chain chain;
bool skip_ctx = false;
int ret;
unsigned int ivsize;
- dreq->base.type = CESA_DMA_REQ;
dreq->chain.first = NULL;
dreq->chain.last = NULL;
@@ -397,10 +396,12 @@ mv_cesa_ablkcipher_std_req_init(struct ablkcipher_request *req,
{
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
+ struct mv_cesa_req *basereq = &creq->req.base;
- sreq->base.type = CESA_STD_REQ;
sreq->op = *op_templ;
sreq->skip_ctx = false;
+ basereq->chain.first = NULL;
+ basereq->chain.last = NULL;
return 0;
}
@@ -442,6 +443,7 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
static int mv_cesa_des_op(struct ablkcipher_request *req,
struct mv_cesa_op_ctx *tmpl)
{
+ struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
struct mv_cesa_des_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
int ret;
@@ -454,7 +456,7 @@ static int mv_cesa_des_op(struct ablkcipher_request *req,
if (ret)
return ret;
- ret = mv_cesa_queue_req(&req->base);
+ ret = mv_cesa_queue_req(&req->base, &creq->req.base);
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ablkcipher_cleanup(req);
@@ -562,6 +564,7 @@ struct crypto_alg mv_cesa_cbc_des_alg = {
static int mv_cesa_des3_op(struct ablkcipher_request *req,
struct mv_cesa_op_ctx *tmpl)
{
+ struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
struct mv_cesa_des3_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
int ret;
@@ -574,7 +577,7 @@ static int mv_cesa_des3_op(struct ablkcipher_request *req,
if (ret)
return ret;
- ret = mv_cesa_queue_req(&req->base);
+ ret = mv_cesa_queue_req(&req->base, &creq->req.base);
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ablkcipher_cleanup(req);
@@ -688,6 +691,7 @@ struct crypto_alg mv_cesa_cbc_des3_ede_alg = {
static int mv_cesa_aes_op(struct ablkcipher_request *req,
struct mv_cesa_op_ctx *tmpl)
{
+ struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
struct mv_cesa_aes_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
int ret, i;
u32 *key;
@@ -716,7 +720,7 @@ static int mv_cesa_aes_op(struct ablkcipher_request *req,
if (ret)
return ret;
- ret = mv_cesa_queue_req(&req->base);
+ ret = mv_cesa_queue_req(&req->base, &creq->req.base);
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ablkcipher_cleanup(req);
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index 0fae351..cc7c5b0 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -103,14 +103,14 @@ static inline void mv_cesa_ahash_dma_cleanup(struct ahash_request *req)
dma_unmap_sg(cesa_dev->dev, req->src, creq->src_nents, DMA_TO_DEVICE);
mv_cesa_ahash_dma_free_cache(&creq->req.dma);
- mv_cesa_dma_cleanup(&creq->req.dma.base);
+ mv_cesa_dma_cleanup(&creq->req.base);
}
static inline void mv_cesa_ahash_cleanup(struct ahash_request *req)
{
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
mv_cesa_ahash_dma_cleanup(req);
}
@@ -118,7 +118,7 @@ static void mv_cesa_ahash_last_cleanup(struct ahash_request *req)
{
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
mv_cesa_ahash_dma_last_cleanup(req);
}
@@ -256,9 +256,9 @@ static int mv_cesa_ahash_std_process(struct ahash_request *req, u32 status)
static inline void mv_cesa_ahash_dma_prepare(struct ahash_request *req)
{
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
- struct mv_cesa_tdma_req *dreq = &creq->req.dma.base;
+ struct mv_cesa_req *dreq = &creq->req.base;
- mv_cesa_dma_prepare(dreq, dreq->base.engine);
+ mv_cesa_dma_prepare(dreq, dreq->engine);
}
static void mv_cesa_ahash_std_prepare(struct ahash_request *req)
@@ -277,8 +277,8 @@ static void mv_cesa_ahash_step(struct crypto_async_request *req)
struct ahash_request *ahashreq = ahash_request_cast(req);
struct mv_cesa_ahash_req *creq = ahash_request_ctx(ahashreq);
- if (creq->req.base.type == CESA_DMA_REQ)
- mv_cesa_dma_step(&creq->req.dma.base);
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
+ mv_cesa_dma_step(&creq->req.base);
else
mv_cesa_ahash_std_step(ahashreq);
}
@@ -291,8 +291,8 @@ static int mv_cesa_ahash_process(struct crypto_async_request *req, u32 status)
unsigned int digsize;
int ret, i;
- if (creq->req.base.type == CESA_DMA_REQ)
- ret = mv_cesa_dma_process(&creq->req.dma.base, status);
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
+ ret = mv_cesa_dma_process(&creq->req.base, status);
else
ret = mv_cesa_ahash_std_process(ahashreq, status);
@@ -340,7 +340,7 @@ static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
creq->req.base.engine = engine;
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
mv_cesa_ahash_dma_prepare(ahashreq);
else
mv_cesa_ahash_std_prepare(ahashreq);
@@ -555,8 +555,7 @@ static int mv_cesa_ahash_dma_req_init(struct ahash_request *req)
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ?
GFP_KERNEL : GFP_ATOMIC;
- struct mv_cesa_ahash_dma_req *ahashdreq = &creq->req.dma;
- struct mv_cesa_tdma_req *dreq = &ahashdreq->base;
+ struct mv_cesa_req *dreq = &creq->req.base;
struct mv_cesa_ahash_dma_iter iter;
struct mv_cesa_op_ctx *op = NULL;
unsigned int frag_len;
@@ -662,11 +661,6 @@ static int mv_cesa_ahash_req_init(struct ahash_request *req, bool *cached)
struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
int ret;
- if (cesa_dev->caps->has_tdma)
- creq->req.base.type = CESA_DMA_REQ;
- else
- creq->req.base.type = CESA_STD_REQ;
-
creq->src_nents = sg_nents_for_len(req->src, req->nbytes);
if (creq->src_nents < 0) {
dev_err(cesa_dev->dev, "Invalid number of src SG");
@@ -680,7 +674,7 @@ static int mv_cesa_ahash_req_init(struct ahash_request *req, bool *cached)
if (*cached)
return 0;
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (mv_cesa_req_get_type(&creq->req.base) == CESA_DMA_REQ)
ret = mv_cesa_ahash_dma_req_init(req);
return ret;
@@ -700,7 +694,7 @@ static int mv_cesa_ahash_update(struct ahash_request *req)
if (cached)
return 0;
- ret = mv_cesa_queue_req(&req->base);
+ ret = mv_cesa_queue_req(&req->base, &creq->req.base);
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ahash_cleanup(req);
@@ -725,7 +719,7 @@ static int mv_cesa_ahash_final(struct ahash_request *req)
if (cached)
return 0;
- ret = mv_cesa_queue_req(&req->base);
+ ret = mv_cesa_queue_req(&req->base, &creq->req.base);
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ahash_cleanup(req);
@@ -750,7 +744,7 @@ static int mv_cesa_ahash_finup(struct ahash_request *req)
if (cached)
return 0;
- ret = mv_cesa_queue_req(&req->base);
+ ret = mv_cesa_queue_req(&req->base, &creq->req.base);
if (mv_cesa_req_needs_cleanup(&req->base, ret))
mv_cesa_ahash_cleanup(req);
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index 88c87be..9a424f9 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -37,9 +37,9 @@ bool mv_cesa_req_dma_iter_next_transfer(struct mv_cesa_dma_iter *iter,
return true;
}
-void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
+void mv_cesa_dma_step(struct mv_cesa_req *dreq)
{
- struct mv_cesa_engine *engine = dreq->base.engine;
+ struct mv_cesa_engine *engine = dreq->engine;
writel_relaxed(0, engine->regs + CESA_SA_CFG);
@@ -58,7 +58,7 @@ void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
-void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
+void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq)
{
struct mv_cesa_tdma_desc *tdma;
@@ -81,7 +81,7 @@ void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
dreq->chain.last = NULL;
}
-void mv_cesa_dma_prepare(struct mv_cesa_tdma_req *dreq,
+void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
struct mv_cesa_engine *engine)
{
struct mv_cesa_tdma_desc *tdma;
--
2.7.4
^ permalink raw reply related
* [PATCH 3/7] crypto: marvell: Copy IV vectors by DMA transfers for acipher requests
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
Adding a TDMA descriptor at the end of the request for copying the
output IV vector via a DMA transfer. This is required for processing
cipher requests asynchroniously in chained mode, otherwise the content
of the IV vector will be overwriten for each new finished request.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 4 ++++
drivers/crypto/marvell/cesa.h | 5 +++++
drivers/crypto/marvell/cipher.c | 40 +++++++++++++++++++++++++++-------------
drivers/crypto/marvell/tdma.c | 29 +++++++++++++++++++++++++++++
4 files changed, 65 insertions(+), 13 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index fb403e1..93700cd 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -312,6 +312,10 @@ static int mv_cesa_dev_dma_init(struct mv_cesa_dev *cesa)
if (!dma->padding_pool)
return -ENOMEM;
+ dma->iv_pool = dmam_pool_create("cesa_iv", dev, 16, 1, 0);
+ if (!dma->iv_pool)
+ return -ENOMEM;
+
cesa->dma = dma;
return 0;
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 74071e4..74b84bd 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -275,6 +275,7 @@ struct mv_cesa_op_ctx {
#define CESA_TDMA_DUMMY 0
#define CESA_TDMA_DATA 1
#define CESA_TDMA_OP 2
+#define CESA_TDMA_IV 4
/**
* struct mv_cesa_tdma_desc - TDMA descriptor
@@ -390,6 +391,7 @@ struct mv_cesa_dev_dma {
struct dma_pool *op_pool;
struct dma_pool *cache_pool;
struct dma_pool *padding_pool;
+ struct dma_pool *iv_pool;
};
/**
@@ -790,6 +792,9 @@ mv_cesa_tdma_desc_iter_init(struct mv_cesa_tdma_chain *chain)
memset(chain, 0, sizeof(*chain));
}
+int mv_cesa_dma_add_iv_op(struct mv_cesa_tdma_chain *chain, dma_addr_t src,
+ u32 size, u32 flags, gfp_t gfp_flags);
+
struct mv_cesa_op_ctx *mv_cesa_dma_add_op(struct mv_cesa_tdma_chain *chain,
const struct mv_cesa_op_ctx *op_templ,
bool skip_ctx,
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index 8d0fabb..f42620e 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -118,6 +118,7 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
struct mv_cesa_engine *engine = sreq->base.engine;
size_t len;
+ unsigned int ivsize;
len = sg_pcopy_from_buffer(req->dst, creq->dst_nents,
engine->sram + CESA_SA_DATA_SRAM_OFFSET,
@@ -127,6 +128,10 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
if (sreq->offset < req->nbytes)
return -EINPROGRESS;
+ ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
+ memcpy_fromio(req->info,
+ engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET, ivsize);
+
return 0;
}
@@ -135,23 +140,23 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
{
struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
- struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
- struct mv_cesa_engine *engine = sreq->base.engine;
- int ret;
- if (creq->req.base.type == CESA_DMA_REQ)
+ if (creq->req.base.type == CESA_DMA_REQ) {
+ int ret;
+ struct mv_cesa_tdma_req *dreq;
+ unsigned int ivsize;
+
ret = mv_cesa_dma_process(&creq->req.dma, status);
- else
- ret = mv_cesa_ablkcipher_std_process(ablkreq, status);
+ if (ret)
+ return ret;
- if (ret)
+ dreq = &creq->req.dma;
+ ivsize = crypto_ablkcipher_ivsize(
+ crypto_ablkcipher_reqtfm(ablkreq));
+ memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
return ret;
-
- memcpy_fromio(ablkreq->info,
- engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET,
- crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq)));
-
- return 0;
+ }
+ return mv_cesa_ablkcipher_std_process(ablkreq, status);
}
static void mv_cesa_ablkcipher_step(struct crypto_async_request *req)
@@ -302,6 +307,7 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
struct mv_cesa_tdma_chain chain;
bool skip_ctx = false;
int ret;
+ unsigned int ivsize;
dreq->base.type = CESA_DMA_REQ;
dreq->chain.first = NULL;
@@ -360,6 +366,14 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
} while (mv_cesa_ablkcipher_req_iter_next_op(&iter));
+ /* Add output data for IV */
+ ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
+ ret = mv_cesa_dma_add_iv_op(&chain, CESA_SA_CRYPT_IV_SRAM_OFFSET,
+ ivsize, CESA_TDMA_SRC_IN_SRAM, flags);
+
+ if (ret)
+ goto err_free_tdma;
+
dreq->chain = chain;
return 0;
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index d493714..88c87be 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -68,6 +68,9 @@ void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
if (tdma->flags & CESA_TDMA_OP)
dma_pool_free(cesa_dev->dma->op_pool, tdma->op,
le32_to_cpu(tdma->src));
+ else if (tdma->flags & CESA_TDMA_IV)
+ dma_pool_free(cesa_dev->dma->iv_pool, tdma->data,
+ le32_to_cpu(tdma->dst));
tdma = tdma->next;
dma_pool_free(cesa_dev->dma->tdma_desc_pool, old_tdma,
@@ -120,6 +123,32 @@ mv_cesa_dma_add_desc(struct mv_cesa_tdma_chain *chain, gfp_t flags)
return new_tdma;
}
+int mv_cesa_dma_add_iv_op(struct mv_cesa_tdma_chain *chain, dma_addr_t src,
+ u32 size, u32 flags, gfp_t gfp_flags)
+{
+
+ struct mv_cesa_tdma_desc *tdma;
+ u8 *cache;
+ dma_addr_t dma_handle;
+
+ tdma = mv_cesa_dma_add_desc(chain, gfp_flags);
+ if (IS_ERR(tdma))
+ return PTR_ERR(tdma);
+
+ cache = dma_pool_alloc(cesa_dev->dma->iv_pool, flags, &dma_handle);
+ if (!cache)
+ return -ENOMEM;
+
+ tdma->byte_cnt = cpu_to_le32(size | BIT(31));
+ tdma->src = src;
+ tdma->dst = cpu_to_le32(dma_handle);
+ tdma->data = cache;
+
+ flags &= (CESA_TDMA_DST_IN_SRAM | CESA_TDMA_SRC_IN_SRAM);
+ tdma->flags = flags | CESA_TDMA_DATA | CESA_TDMA_IV;
+ return 0;
+}
+
struct mv_cesa_op_ctx *mv_cesa_dma_add_op(struct mv_cesa_tdma_chain *chain,
const struct mv_cesa_op_ctx *op_templ,
bool skip_ctx,
--
2.7.4
^ permalink raw reply related
* [PATCH 2/7] crypto: marvell: Check engine is not already running when enabling a req
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
Adding BUG_ON() macro to be sure that the step operation is not about
to activate a request on the engine if the corresponding engine is
already processing a crypto request. This is helpful when the support
for chaining crypto requests will be added. Instead of hanging the
system when the engine is in an incoherent state, we add this macro
which throws an understandable error.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cipher.c | 2 ++
drivers/crypto/marvell/hash.c | 2 ++
drivers/crypto/marvell/tdma.c | 2 ++
3 files changed, 6 insertions(+)
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index dcf1fce..8d0fabb 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -106,6 +106,8 @@ static void mv_cesa_ablkcipher_std_step(struct ablkcipher_request *req)
mv_cesa_set_int_mask(engine, CESA_SA_INT_ACCEL0_DONE);
writel_relaxed(CESA_SA_CFG_PARA_DIS, engine->regs + CESA_SA_CFG);
+ BUG_ON(readl(engine->regs + CESA_SA_CMD)
+ & CESA_SA_CMD_EN_CESA_SA_ACCL0);
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index 7ca2e0f..0fae351 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -237,6 +237,8 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
mv_cesa_set_int_mask(engine, CESA_SA_INT_ACCEL0_DONE);
writel_relaxed(CESA_SA_CFG_PARA_DIS, engine->regs + CESA_SA_CFG);
+ BUG_ON(readl(engine->regs + CESA_SA_CMD)
+ & CESA_SA_CMD_EN_CESA_SA_ACCL0);
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index 7642798..d493714 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -53,6 +53,8 @@ void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
engine->regs + CESA_SA_CFG);
writel_relaxed(dreq->chain.first->cur_dma,
engine->regs + CESA_TDMA_NEXT_ADDR);
+ BUG_ON(readl(engine->regs + CESA_SA_CMD)
+ & CESA_SA_CMD_EN_CESA_SA_ACCL0);
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
--
2.7.4
^ permalink raw reply related
* [PATCH 1/7] crypto: marvell: Add a macro constant for the size of the crypto queue
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466018134-10779-1-git-send-email-romain.perier@free-electrons.com>
Adding a macro constant to be used for the size of the crypto queue,
instead of using a numeric value directly. It will be easier to
maintain in case we add more than one crypto queue of the same size.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index 056a754..fb403e1 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -31,6 +31,9 @@
#include "cesa.h"
+/* Limit of the crypto queue before reaching the backlog */
+#define CESA_CRYPTO_DEFAULT_MAX_QLEN 50
+
static int allhwsupport = !IS_ENABLED(CONFIG_CRYPTO_DEV_MV_CESA);
module_param_named(allhwsupport, allhwsupport, int, 0444);
MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if overlaps with the mv_cesa driver)");
@@ -416,7 +419,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
return -ENOMEM;
spin_lock_init(&cesa->lock);
- crypto_init_queue(&cesa->queue, 50);
+ crypto_init_queue(&cesa->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
cesa->regs = devm_ioremap_resource(dev, res);
if (IS_ERR(cesa->regs))
--
2.7.4
^ permalink raw reply related
* [PATCH 0/7] Chain crypto requests together at the DMA level
From: Romain Perier @ 2016-06-15 19:15 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
The Cryptographic Engines and Security Accelerators (CESA) supports
the TDMA chained mode support. When this mode is enabled and crypto
requests are chained at the DMA level, multiple crypto requests can be
handled by the hardware engine without requiring any software
intervention. This approach limits the number of interrupts generated
by the engines thus improving its throughput and making the whole system
behave nicely under heavy crypto load.
Benchmarking results with dmcrypt
=================================
I/O read I/O write
Before 81.7 MB/s 31.7 MB/s
After 129 MB/s 39.8 MB/s
Improvement +57.8 % +25.5 %
Romain Perier (7):
crypto: marvell: Add a macro constant for the size of the crypto queue
crypto: marvell: Check engine is not already running when enabling a
req
crypto: marvell: Copy IV vectors by DMA transfers for acipher requests
crypto: marvell: Moving the tdma chain out of mv_cesa_tdma_req
crypto: marvell: Adding a complete operation for async requests
crypto: marvell: Adding load balancing between engines
crypto: marvell: Add support for chaining crypto requests in TDMA mode
drivers/crypto/marvell/cesa.c | 142 ++++++++++++++++++++++++++++++----------
drivers/crypto/marvell/cesa.h | 103 +++++++++++++++++++++++------
drivers/crypto/marvell/cipher.c | 141 +++++++++++++++++++++++----------------
drivers/crypto/marvell/hash.c | 126 +++++++++++++++++------------------
drivers/crypto/marvell/tdma.c | 120 +++++++++++++++++++++++++++++++--
5 files changed, 452 insertions(+), 180 deletions(-)
--
2.7.4
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: Stephan Mueller @ 2016-06-15 16:58 UTC (permalink / raw)
To: David Jaša
Cc: Andi Kleen, sandyinchina, Jason Cooper, John Denker,
H. Peter Anvin, Joe Perches, Pavel Machek, George Spelvin,
linux-crypto, linux-kernel
In-Reply-To: <1466007463.20087.11.camel@redhat.com>
Am Mittwoch, 15. Juni 2016, 18:17:43 schrieb David Jaša:
Hi David,
> Hello Stephan,
>
> Did you consider blocking urandom output or returning error until
> initialized? Given the speed of initialization you report, it shouldn't
> break any userspace apps while making sure that nobody uses predictable
> pseudoranom numbers.
My LRNG will definitely touch the beginning of the initramfs booting until it
is fully seeded. As these days the initramfs is driven by systemd which always
pulls from /dev/urandom, we cannot block as this would block systemd. In Ted's
last patch, he mentioned that he tried to make /dev/urandom block which caused
user space pain.
But if you use the getrandom system call, it works like /dev/urandom but
blocks until the DRBG behind /dev/urandom is fully initialized.
>
> I was considering asking for patch (or even trying to write it myself)
> to make current urandom block/fail when not initialized but that would
> surely have to be off by default over "never break userspace" rule (even
> if it means way too easy security problem with both random and urandom).
> Properties of your urandom implementation makes this point moot and it
> could make the random/urandom wars over.
That patch unfortunately will not work. But if you are interested in that
blocking /dev/urandom behavior for your application, use getrandom.
>
> Best Regards,
>
> David Jaša
Ciao
Stephan
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: David Jaša @ 2016-06-15 16:17 UTC (permalink / raw)
To: Stephan Mueller
Cc: Andi Kleen, sandyinchina, Jason Cooper, John Denker,
H. Peter Anvin, Joe Perches, Pavel Machek, George Spelvin,
linux-crypto, linux-kernel
In-Reply-To: <1668650.acZVSyjHlL@positron.chronox.de>
Hello Stephan,
Did you consider blocking urandom output or returning error until
initialized? Given the speed of initialization you report, it shouldn't
break any userspace apps while making sure that nobody uses predictable
pseudoranom numbers.
I was considering asking for patch (or even trying to write it myself)
to make current urandom block/fail when not initialized but that would
surely have to be off by default over "never break userspace" rule (even
if it means way too easy security problem with both random and urandom).
Properties of your urandom implementation makes this point moot and it
could make the random/urandom wars over.
Best Regards,
David Jaša
^ permalink raw reply
* Re: [RESEND PATCH 0/2] hw_random: Add Amlogic Meson SoCs Random Generator driver
From: Kevin Hilman @ 2016-06-15 15:57 UTC (permalink / raw)
To: Neil Armstrong
Cc: Matt Mackall, Herbert Xu, linux-arm-kernel, linux-amlogic,
linux-kernel, linux-crypto
In-Reply-To: <1465984906-17840-1-git-send-email-narmstrong@baylibre.com>
Neil Armstrong <narmstrong@baylibre.com> writes:
> NOTE: This is a resent of the DT Bindings and DTSI patches based on the Amlogic DT 64bit
> GIT pull request from Kevin Hilman at [1].
>
> Changes since v2 at http://lkml.kernel.org/r/1465546915-24229-1-git-send-email-narmstrong@baylibre.com :
> - Move rng peripheral node into periphs simple-bus node
Thanks for the update.
Applied to the dt64 branch of the amlogic tree.
Kevin
^ permalink raw reply
* [PATCH v8 0/3] crypto: caam - add support for RSA algorithm
From: Tudor Ambarus @ 2016-06-15 14:52 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto, horia.geanta, Tudor Ambarus
Changes in v8:
- store raw keys on stack
- use d_sz instead of n_sz for RSA private exponent
- add caam_read_raw_data function for reading RSA modulus raw byte stream
as a positive integer. The function updates the n_sz byte length too.
Needed because the decryption descriptor uses the RSA modulus length as
decryption output length. The accelerator will try to write n_sz bytes
to output SGT, resulting a SGT overfflow if RSA modulus contains leading
zeros.
- add caam_rsa_check_key_length function. Maximum supported modulus size is
4096 bits. Fallback mechanism to be added after removing
the (same) key length constraint from software implementation.
Changes in v7:
- sync with ASN.1 parser
Changes in v6:
- write descriptor PDB fields with inline append
- move Protocol Data Block (pdb) structures to pdb.h
- move setting of PDB fields in new functions
- unmap sec4_sg_dma on done callback
- remove redundant clean code on error path
- fix doc typos
Changes in v5:
- sync with ASN.1 parser
Changes in v4:
- sync with ASN.1 parser
Changes in v3:
- sync with ASN.1 parser
Changes in v2:
- fix memory leaks on error path
- rename struct akcipher_alg rsa to caam_rsa
Tudor Ambarus (3):
crypto: scatterwak - Add scatterwalk_sg_copychunks
crypto: scatterwalk - export scatterwalk_pagedone
crypto: caam - add support for RSA algorithm
crypto/scatterwalk.c | 31 +-
drivers/crypto/caam/Kconfig | 12 +
drivers/crypto/caam/Makefile | 4 +
drivers/crypto/caam/caampkc.c | 693 ++++++++++++++++++++++++++++++++++++++
drivers/crypto/caam/caampkc.h | 70 ++++
drivers/crypto/caam/compat.h | 3 +
drivers/crypto/caam/desc.h | 2 +
drivers/crypto/caam/desc_constr.h | 7 +
drivers/crypto/caam/pdb.h | 51 ++-
drivers/crypto/caam/pkc_desc.c | 36 ++
include/crypto/scatterwalk.h | 4 +
11 files changed, 910 insertions(+), 3 deletions(-)
create mode 100644 drivers/crypto/caam/caampkc.c
create mode 100644 drivers/crypto/caam/caampkc.h
create mode 100644 drivers/crypto/caam/pkc_desc.c
--
1.8.3.1
^ permalink raw reply
* Re: [PATCH 5/7] random: replace non-blocking pool with a Chacha20-based CRNG
From: Herbert Xu @ 2016-06-15 14:59 UTC (permalink / raw)
To: Theodore Ts'o
Cc: Linux Kernel Developers List, linux-crypto, smueller, andi,
sandyinchina, jsd, hpa
In-Reply-To: <1465832919-11316-6-git-send-email-tytso@mit.edu>
On Mon, Jun 13, 2016 at 11:48:37AM -0400, Theodore Ts'o wrote:
> The CRNG is faster, and we don't pretend to track entropy usage in the
> CRNG any more.
>
> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
> ---
> crypto/chacha20_generic.c | 61 --------
> drivers/char/random.c | 374 +++++++++++++++++++++++++++++++++-------------
> include/crypto/chacha20.h | 1 +
> lib/Makefile | 2 +-
> lib/chacha20.c | 79 ++++++++++
> 5 files changed, 353 insertions(+), 164 deletions(-)
> create mode 100644 lib/chacha20.c
>
> diff --git a/crypto/chacha20_generic.c b/crypto/chacha20_generic.c
> index da9c899..1cab831 100644
> --- a/crypto/chacha20_generic.c
> +++ b/crypto/chacha20_generic.c
I think you should be accessing this through the crypto API rather
than going direct. We already have at least one accelerated
implementation of chacha20 and there may well be more of them
in future. Going through the crypto API means that you will
automatically pick up the best implementation for the platform.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [PATCH v8 3/3] crypto: caam - add support for RSA algorithm
From: Tudor Ambarus @ 2016-06-15 14:52 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto, horia.geanta, Tudor Ambarus
In-Reply-To: <1466002364-16945-1-git-send-email-tudor-dan.ambarus@nxp.com>
Add RSA support to caam driver.
Initial author is Yashpal Dutta <yashpal.dutta@freescale.com>.
Signed-off-by: Tudor Ambarus <tudor-dan.ambarus@nxp.com>
---
drivers/crypto/caam/Kconfig | 12 +
drivers/crypto/caam/Makefile | 4 +
drivers/crypto/caam/caampkc.c | 693 ++++++++++++++++++++++++++++++++++++++
drivers/crypto/caam/caampkc.h | 70 ++++
drivers/crypto/caam/compat.h | 3 +
drivers/crypto/caam/desc.h | 2 +
drivers/crypto/caam/desc_constr.h | 7 +
drivers/crypto/caam/pdb.h | 51 ++-
drivers/crypto/caam/pkc_desc.c | 36 ++
9 files changed, 877 insertions(+), 1 deletion(-)
create mode 100644 drivers/crypto/caam/caampkc.c
create mode 100644 drivers/crypto/caam/caampkc.h
create mode 100644 drivers/crypto/caam/pkc_desc.c
diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index ff54c42..64bf302 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -99,6 +99,18 @@ config CRYPTO_DEV_FSL_CAAM_AHASH_API
To compile this as a module, choose M here: the module
will be called caamhash.
+config CRYPTO_DEV_FSL_CAAM_PKC_API
+ tristate "Register public key cryptography implementations with Crypto API"
+ depends on CRYPTO_DEV_FSL_CAAM && CRYPTO_DEV_FSL_CAAM_JR
+ default y
+ select CRYPTO_RSA
+ help
+ Selecting this will allow SEC Public key support for RSA.
+ Supported cryptographic primitives: encryption, decryption,
+ signature and verification.
+ To compile this as a module, choose M here: the module
+ will be called caam_pkc.
+
config CRYPTO_DEV_FSL_CAAM_RNG_API
tristate "Register caam device for hwrng API"
depends on CRYPTO_DEV_FSL_CAAM && CRYPTO_DEV_FSL_CAAM_JR
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index 550758a..399ad55 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -5,11 +5,15 @@ ifeq ($(CONFIG_CRYPTO_DEV_FSL_CAAM_DEBUG), y)
EXTRA_CFLAGS := -DDEBUG
endif
+ccflags-y += -I$(srctree)/crypto
+
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_JR) += caam_jr.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API) += caamrng.o
+obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API) += caam_pkc.o
caam-objs := ctrl.o
caam_jr-objs := jr.o key_gen.o error.o
+caam_pkc-y := caampkc.o pkc_desc.o
diff --git a/drivers/crypto/caam/caampkc.c b/drivers/crypto/caam/caampkc.c
new file mode 100644
index 0000000..36d3419
--- /dev/null
+++ b/drivers/crypto/caam/caampkc.c
@@ -0,0 +1,693 @@
+/*
+ * caam - Freescale FSL CAAM support for Public Key Cryptography
+ *
+ * Copyright 2016 Freescale Semiconductor, Inc.
+ *
+ * There is no Shared Descriptor for PKC so that the Job Descriptor must carry
+ * all the desired key parameters, input and output pointers.
+ */
+#include "compat.h"
+#include "regs.h"
+#include "intern.h"
+#include "jr.h"
+#include "error.h"
+#include "desc_constr.h"
+#include "sg_sw_sec4.h"
+#include "caampkc.h"
+#include "rsapubkey-asn1.h"
+#include "rsaprivkey-asn1.h"
+
+#define DESC_RSA_PUB_LEN (2 * CAAM_CMD_SZ + sizeof(struct rsa_pub_pdb))
+#define DESC_RSA_PRIV_F1_LEN (2 * CAAM_CMD_SZ + \
+ sizeof(struct rsa_priv_f1_pdb))
+
+static void rsa_io_unmap(struct device *dev, struct rsa_edesc *edesc,
+ struct akcipher_request *req)
+{
+ dma_unmap_sg(dev, req->dst, edesc->dst_nents, DMA_FROM_DEVICE);
+ dma_unmap_sg(dev, req->src, edesc->src_nents, DMA_TO_DEVICE);
+
+ if (edesc->sec4_sg_bytes)
+ dma_unmap_single(dev, edesc->sec4_sg_dma, edesc->sec4_sg_bytes,
+ DMA_TO_DEVICE);
+}
+
+static void rsa_pub_unmap(struct device *dev, struct rsa_edesc *edesc,
+ struct akcipher_request *req)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+ struct rsa_pub_pdb *pdb = &edesc->pdb.pub;
+
+ dma_unmap_single(dev, pdb->n_dma, key->n_sz, DMA_TO_DEVICE);
+ dma_unmap_single(dev, pdb->e_dma, key->e_sz, DMA_TO_DEVICE);
+}
+
+static void rsa_priv_f1_unmap(struct device *dev, struct rsa_edesc *edesc,
+ struct akcipher_request *req)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+ struct rsa_priv_f1_pdb *pdb = &edesc->pdb.priv_f1;
+
+ dma_unmap_single(dev, pdb->n_dma, key->n_sz, DMA_TO_DEVICE);
+ dma_unmap_single(dev, pdb->d_dma, key->d_sz, DMA_TO_DEVICE);
+}
+
+static size_t caam_count_leading_zeros(u8 *ptr, size_t nbytes)
+{
+ size_t nr_zeros = 0;
+
+ while (!(*ptr) && nbytes) {
+ nbytes--;
+ ptr++;
+ nr_zeros++;
+ }
+
+ return nr_zeros;
+}
+
+static size_t scatterwalk_skip_leading_zeros(struct scatter_walk *walk,
+ size_t nbytes)
+{
+ size_t len_this_page, nr_zeros, cnt = 0;
+ u8 *vaddr;
+
+ for (;;) {
+ nr_zeros = 0;
+ len_this_page = scatterwalk_pagelen(walk);
+
+ if (len_this_page > nbytes)
+ len_this_page = nbytes;
+
+ vaddr = scatterwalk_map(walk);
+ nr_zeros = caam_count_leading_zeros(vaddr, len_this_page);
+ scatterwalk_unmap(vaddr);
+
+ /* count total number of zeros */
+ cnt += nr_zeros;
+
+ /* advance scatterwalk to the nonzero data */
+ scatterwalk_advance(walk, nr_zeros);
+
+ if (nr_zeros < len_this_page || nbytes == len_this_page)
+ break;
+
+ nbytes -= len_this_page;
+
+ scatterwalk_pagedone(walk, 0, 1);
+ }
+
+ return cnt;
+}
+
+/*
+ * This function drops the leading zeros and copies the data to the initial
+ * pointer so that it can be freed later on. Returns the updated data length.
+ */
+static size_t drop_leading_zeros(struct scatterlist *sg, size_t nbytes)
+{
+ struct scatter_walk walk_src, walk_dst;
+ size_t nr_zeros = 0;
+
+ scatterwalk_start(&walk_src, sg);
+ nr_zeros = scatterwalk_skip_leading_zeros(&walk_src, nbytes);
+
+ if (nr_zeros) {
+ nbytes = nbytes - nr_zeros;
+
+ scatterwalk_start(&walk_dst, sg);
+ scatterwalk_sg_copychunks(&walk_dst, &walk_src, nbytes);
+ scatterwalk_done(&walk_dst, 0, 0);
+ }
+
+ scatterwalk_done(&walk_src, 0, 0);
+
+ return nbytes;
+}
+
+/* RSA Job Completion handler */
+static void rsa_pub_done(struct device *dev, u32 *desc, u32 err, void *context)
+{
+ struct akcipher_request *req = context;
+ struct rsa_edesc *edesc;
+
+ if (err)
+ caam_jr_strstatus(dev, err);
+
+ /*
+ * RSA's output is expected to be a big integer. Drop the leading
+ * zeros since they are not meaningful in the world of numbers.
+ */
+ req->dst_len = drop_leading_zeros(req->dst, req->dst_len);
+
+ edesc = container_of(desc, struct rsa_edesc, hw_desc[0]);
+
+ rsa_pub_unmap(dev, edesc, req);
+ rsa_io_unmap(dev, edesc, req);
+ kfree(edesc);
+
+ akcipher_request_complete(req, err);
+}
+
+static void rsa_priv_f1_done(struct device *dev, u32 *desc, u32 err,
+ void *context)
+{
+ struct akcipher_request *req = context;
+ struct rsa_edesc *edesc;
+
+ if (err)
+ caam_jr_strstatus(dev, err);
+
+ /*
+ * RSA's output is expected to be a big integer. Drop the leading
+ * zeros since they are not meaningful in the world of numbers.
+ */
+ req->dst_len = drop_leading_zeros(req->dst, req->dst_len);
+
+ edesc = container_of(desc, struct rsa_edesc, hw_desc[0]);
+
+ rsa_priv_f1_unmap(dev, edesc, req);
+ rsa_io_unmap(dev, edesc, req);
+ kfree(edesc);
+
+ akcipher_request_complete(req, err);
+}
+
+static struct rsa_edesc *rsa_edesc_alloc(struct akcipher_request *req,
+ size_t desclen)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct device *dev = ctx->dev;
+ struct rsa_edesc *edesc;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ int sgc;
+ int sec4_sg_index, sec4_sg_len = 0, sec4_sg_bytes;
+ int src_nents, dst_nents;
+
+ src_nents = sg_nents_for_len(req->src, req->src_len);
+ dst_nents = sg_nents_for_len(req->dst, req->dst_len);
+
+ if (src_nents > 1)
+ sec4_sg_len = src_nents;
+ if (dst_nents > 1)
+ sec4_sg_len += dst_nents;
+
+ sec4_sg_bytes = sec4_sg_len * sizeof(struct sec4_sg_entry);
+
+ /* allocate space for base edesc, hw desc commands and link tables */
+ edesc = kzalloc(sizeof(*edesc) + desclen + sec4_sg_bytes,
+ GFP_DMA | flags);
+ if (!edesc)
+ return ERR_PTR(-ENOMEM);
+
+ sgc = dma_map_sg(dev, req->src, src_nents, DMA_TO_DEVICE);
+ if (unlikely(!sgc)) {
+ dev_err(dev, "unable to map source\n");
+ goto src_fail;
+ }
+
+ sgc = dma_map_sg(dev, req->dst, dst_nents, DMA_FROM_DEVICE);
+ if (unlikely(!sgc)) {
+ dev_err(dev, "unable to map destination\n");
+ goto dst_fail;
+ }
+
+ edesc->sec4_sg = (void *)edesc + sizeof(*edesc) + desclen;
+
+ sec4_sg_index = 0;
+ if (src_nents > 1) {
+ sg_to_sec4_sg_last(req->src, src_nents, edesc->sec4_sg, 0);
+ sec4_sg_index += src_nents;
+ }
+ if (dst_nents > 1)
+ sg_to_sec4_sg_last(req->dst, dst_nents,
+ edesc->sec4_sg + sec4_sg_index, 0);
+
+ /* Save nents for later use in Job Descriptor */
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+
+ if (!sec4_sg_bytes)
+ return edesc;
+
+ edesc->sec4_sg_dma = dma_map_single(dev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, edesc->sec4_sg_dma)) {
+ dev_err(dev, "unable to map S/G table\n");
+ goto sec4_sg_fail;
+ }
+
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+
+ return edesc;
+
+sec4_sg_fail:
+ dma_unmap_sg(dev, req->dst, dst_nents, DMA_FROM_DEVICE);
+dst_fail:
+ dma_unmap_sg(dev, req->src, src_nents, DMA_TO_DEVICE);
+src_fail:
+ kfree(edesc);
+ return ERR_PTR(-ENOMEM);
+}
+
+static int set_rsa_pub_pdb(struct akcipher_request *req,
+ struct rsa_edesc *edesc)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+ struct device *dev = ctx->dev;
+ struct rsa_pub_pdb *pdb = &edesc->pdb.pub;
+ int sec4_sg_index = 0;
+
+ pdb->n_dma = dma_map_single(dev, key->n, key->n_sz, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, pdb->n_dma)) {
+ dev_err(dev, "Unable to map RSA modulus memory\n");
+ return -ENOMEM;
+ }
+
+ pdb->e_dma = dma_map_single(dev, key->e, key->e_sz, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, pdb->e_dma)) {
+ dev_err(dev, "Unable to map RSA public exponent memory\n");
+ dma_unmap_single(dev, pdb->n_dma, key->n_sz, DMA_TO_DEVICE);
+ return -ENOMEM;
+ }
+
+ if (edesc->src_nents > 1) {
+ pdb->sgf |= RSA_PDB_SGF_F;
+ pdb->f_dma = edesc->sec4_sg_dma;
+ sec4_sg_index += edesc->src_nents;
+ } else {
+ pdb->f_dma = sg_dma_address(req->src);
+ }
+
+ if (edesc->dst_nents > 1) {
+ pdb->sgf |= RSA_PDB_SGF_G;
+ pdb->g_dma = edesc->sec4_sg_dma +
+ sec4_sg_index * sizeof(struct sec4_sg_entry);
+ } else {
+ pdb->g_dma = sg_dma_address(req->dst);
+ }
+
+ pdb->sgf |= (key->e_sz << RSA_PDB_E_SHIFT) | key->n_sz;
+ pdb->f_len = req->src_len;
+
+ return 0;
+}
+
+static int set_rsa_priv_f1_pdb(struct akcipher_request *req,
+ struct rsa_edesc *edesc)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+ struct device *dev = ctx->dev;
+ struct rsa_priv_f1_pdb *pdb = &edesc->pdb.priv_f1;
+ int sec4_sg_index = 0;
+
+ pdb->n_dma = dma_map_single(dev, key->n, key->n_sz, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, pdb->n_dma)) {
+ dev_err(dev, "Unable to map modulus memory\n");
+ return -ENOMEM;
+ }
+
+ pdb->d_dma = dma_map_single(dev, key->d, key->d_sz, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, pdb->d_dma)) {
+ dev_err(dev, "Unable to map RSA private exponent memory\n");
+ dma_unmap_single(dev, pdb->n_dma, key->n_sz, DMA_TO_DEVICE);
+ return -ENOMEM;
+ }
+
+ if (edesc->src_nents > 1) {
+ pdb->sgf |= RSA_PRIV_PDB_SGF_G;
+ pdb->g_dma = edesc->sec4_sg_dma;
+ sec4_sg_index += edesc->src_nents;
+ } else {
+ pdb->g_dma = sg_dma_address(req->src);
+ }
+
+ if (edesc->dst_nents > 1) {
+ pdb->sgf |= RSA_PRIV_PDB_SGF_F;
+ pdb->f_dma = edesc->sec4_sg_dma +
+ sec4_sg_index * sizeof(struct sec4_sg_entry);
+ } else {
+ pdb->f_dma = sg_dma_address(req->dst);
+ }
+
+ pdb->sgf |= (key->d_sz << RSA_PDB_D_SHIFT) | key->n_sz;
+
+ return 0;
+}
+
+static int caam_rsa_enc(struct akcipher_request *req)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+ struct device *jrdev = ctx->dev;
+ struct rsa_edesc *edesc;
+ int ret;
+
+ if (unlikely(!key->n || !key->e))
+ return -EINVAL;
+
+ if (req->dst_len < key->n_sz) {
+ req->dst_len = key->n_sz;
+ dev_err(jrdev, "Output buffer length less than parameter n\n");
+ return -EOVERFLOW;
+ }
+
+ /* Allocate extended descriptor */
+ edesc = rsa_edesc_alloc(req, DESC_RSA_PUB_LEN);
+ if (IS_ERR(edesc))
+ return PTR_ERR(edesc);
+
+ /* Set RSA Encrypt Protocol Data Block */
+ ret = set_rsa_pub_pdb(req, edesc);
+ if (ret)
+ goto init_fail;
+
+ /* Initialize Job Descriptor */
+ init_rsa_pub_desc(edesc->hw_desc, &edesc->pdb.pub);
+
+ ret = caam_jr_enqueue(jrdev, edesc->hw_desc, rsa_pub_done, req);
+ if (!ret)
+ return -EINPROGRESS;
+
+ rsa_pub_unmap(jrdev, edesc, req);
+
+init_fail:
+ rsa_io_unmap(jrdev, edesc, req);
+ kfree(edesc);
+ return ret;
+}
+
+static int caam_rsa_dec(struct akcipher_request *req)
+{
+ struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req);
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+ struct device *jrdev = ctx->dev;
+ struct rsa_edesc *edesc;
+ int ret;
+
+ if (unlikely(!key->n || !key->d))
+ return -EINVAL;
+
+ if (req->dst_len < key->n_sz) {
+ req->dst_len = key->n_sz;
+ dev_err(jrdev, "Output buffer length less than parameter n\n");
+ return -EOVERFLOW;
+ }
+
+ /* Allocate extended descriptor */
+ edesc = rsa_edesc_alloc(req, DESC_RSA_PRIV_F1_LEN);
+ if (IS_ERR(edesc))
+ return PTR_ERR(edesc);
+
+ /* Set RSA Decrypt Protocol Data Block - Private Key Form #1 */
+ ret = set_rsa_priv_f1_pdb(req, edesc);
+ if (ret)
+ goto init_fail;
+
+ /* Initialize Job Descriptor */
+ init_rsa_priv_f1_desc(edesc->hw_desc, &edesc->pdb.priv_f1);
+
+ ret = caam_jr_enqueue(jrdev, edesc->hw_desc, rsa_priv_f1_done, req);
+ if (!ret)
+ return -EINPROGRESS;
+
+ rsa_priv_f1_unmap(jrdev, edesc, req);
+
+init_fail:
+ rsa_io_unmap(jrdev, edesc, req);
+ kfree(edesc);
+ return ret;
+}
+
+static void caam_rsa_free_key(struct caam_rsa_key *key)
+{
+ kzfree(key->d);
+ kfree(key->e);
+ kfree(key->n);
+ key->d = NULL;
+ key->e = NULL;
+ key->n = NULL;
+ key->d_sz = 0;
+ key->e_sz = 0;
+ key->n_sz = 0;
+}
+
+/**
+ * caam_read_raw_data - Read a raw byte stream as a positive integer.
+ * The function skips buffer's leading zeros, copies the remained data
+ * to a buffer allocated in the GFP_DMA | GFP_KERNEL zone and returns
+ * the address of the new buffer.
+ *
+ * @buf : The data to read
+ * @nbytes: The amount of data to read
+ */
+static inline u8 *caam_read_raw_data(const u8 *buf, size_t *nbytes)
+{
+ u8 *val;
+
+ while (!*buf && *nbytes) {
+ buf++;
+ (*nbytes)--;
+ }
+
+ val = kzalloc(*nbytes, GFP_DMA | GFP_KERNEL);
+ if (!val)
+ return NULL;
+
+ memcpy(val, buf, *nbytes);
+
+ return val;
+}
+
+static int caam_rsa_check_key_length(unsigned int len)
+{
+ if (len > 4096)
+ return -EINVAL;
+ return 0;
+}
+
+static int caam_rsa_set_pub_key(struct crypto_akcipher *tfm, const void *key,
+ unsigned int keylen)
+{
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct rsa_key raw_key = {0};
+ struct caam_rsa_key *rsa_key = &ctx->key;
+ int ret;
+
+ /* Free the old RSA key if any */
+ caam_rsa_free_key(rsa_key);
+
+ ret = rsa_parse_pub_key(&raw_key, key, keylen);
+ if (ret)
+ return ret;
+
+ /* Copy key in DMA zone */
+ rsa_key->e = kzalloc(raw_key.e_sz, GFP_DMA | GFP_KERNEL);
+ if (!rsa_key->e)
+ goto err;
+
+ /*
+ * Skip leading zeros and copy the positive integer to a buffer
+ * allocated in the GFP_DMA | GFP_KERNEL zone. The decryption descriptor
+ * expects a positive integer for the RSA modulus and uses its length as
+ * decryption output length.
+ */
+ rsa_key->n = caam_read_raw_data(raw_key.n, &raw_key.n_sz);
+ if (!rsa_key->n)
+ goto err;
+
+ if (caam_rsa_check_key_length(raw_key.n_sz << 3)) {
+ caam_rsa_free_key(rsa_key);
+ return -EINVAL;
+ }
+
+ rsa_key->e_sz = raw_key.e_sz;
+ rsa_key->n_sz = raw_key.n_sz;
+
+ memcpy(rsa_key->e, raw_key.e, raw_key.e_sz);
+
+ return 0;
+err:
+ caam_rsa_free_key(rsa_key);
+ return -ENOMEM;
+}
+
+static int caam_rsa_set_priv_key(struct crypto_akcipher *tfm, const void *key,
+ unsigned int keylen)
+{
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct rsa_key raw_key = {0};
+ struct caam_rsa_key *rsa_key = &ctx->key;
+ int ret;
+
+ /* Free the old RSA key if any */
+ caam_rsa_free_key(rsa_key);
+
+ ret = rsa_parse_priv_key(&raw_key, key, keylen);
+ if (ret)
+ return ret;
+
+ /* Copy key in DMA zone */
+ rsa_key->d = kzalloc(raw_key.d_sz, GFP_DMA | GFP_KERNEL);
+ if (!rsa_key->d)
+ goto err;
+
+ rsa_key->e = kzalloc(raw_key.e_sz, GFP_DMA | GFP_KERNEL);
+ if (!rsa_key->e)
+ goto err;
+
+ /*
+ * Skip leading zeros and copy the positive integer to a buffer
+ * allocated in the GFP_DMA | GFP_KERNEL zone. The decryption descriptor
+ * expects a positive integer for the RSA modulus and uses its length as
+ * decryption output length.
+ */
+ rsa_key->n = caam_read_raw_data(raw_key.n, &raw_key.n_sz);
+ if (!rsa_key->n)
+ goto err;
+
+ if (caam_rsa_check_key_length(raw_key.n_sz << 3)) {
+ caam_rsa_free_key(rsa_key);
+ return -EINVAL;
+ }
+
+ rsa_key->d_sz = raw_key.d_sz;
+ rsa_key->e_sz = raw_key.e_sz;
+ rsa_key->n_sz = raw_key.n_sz;
+
+ memcpy(rsa_key->d, raw_key.d, raw_key.d_sz);
+ memcpy(rsa_key->e, raw_key.e, raw_key.e_sz);
+
+ return 0;
+
+err:
+ caam_rsa_free_key(rsa_key);
+ return -ENOMEM;
+}
+
+static int caam_rsa_max_size(struct crypto_akcipher *tfm)
+{
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+
+ return (key->n) ? key->n_sz : -EINVAL;
+}
+
+/* Per session pkc's driver context creation function */
+static int caam_rsa_init_tfm(struct crypto_akcipher *tfm)
+{
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+
+ ctx->dev = caam_jr_alloc();
+
+ if (IS_ERR(ctx->dev)) {
+ dev_err(ctx->dev, "Job Ring Device allocation for transform failed\n");
+ return PTR_ERR(ctx->dev);
+ }
+
+ return 0;
+}
+
+/* Per session pkc's driver context cleanup function */
+static void caam_rsa_exit_tfm(struct crypto_akcipher *tfm)
+{
+ struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
+ struct caam_rsa_key *key = &ctx->key;
+
+ caam_rsa_free_key(key);
+ caam_jr_free(ctx->dev);
+}
+
+static struct akcipher_alg caam_rsa = {
+ .encrypt = caam_rsa_enc,
+ .decrypt = caam_rsa_dec,
+ .sign = caam_rsa_dec,
+ .verify = caam_rsa_enc,
+ .set_pub_key = caam_rsa_set_pub_key,
+ .set_priv_key = caam_rsa_set_priv_key,
+ .max_size = caam_rsa_max_size,
+ .init = caam_rsa_init_tfm,
+ .exit = caam_rsa_exit_tfm,
+ .base = {
+ .cra_name = "rsa",
+ .cra_driver_name = "rsa-caam",
+ .cra_priority = 3000,
+ .cra_module = THIS_MODULE,
+ .cra_ctxsize = sizeof(struct caam_rsa_ctx),
+ },
+};
+
+/* Public Key Cryptography module initialization handler */
+static int __init caam_pkc_init(void)
+{
+ struct device_node *dev_node;
+ struct platform_device *pdev;
+ struct device *ctrldev;
+ struct caam_drv_private *priv;
+ u32 cha_inst, pk_inst;
+ int err;
+
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
+ if (!dev_node) {
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec4.0");
+ if (!dev_node)
+ return -ENODEV;
+ }
+
+ pdev = of_find_device_by_node(dev_node);
+ if (!pdev) {
+ of_node_put(dev_node);
+ return -ENODEV;
+ }
+
+ ctrldev = &pdev->dev;
+ priv = dev_get_drvdata(ctrldev);
+ of_node_put(dev_node);
+
+ /*
+ * If priv is NULL, it's probably because the caam driver wasn't
+ * properly initialized (e.g. RNG4 init failed). Thus, bail out here.
+ */
+ if (!priv)
+ return -ENODEV;
+
+ /* Determine public key hardware accelerator presence. */
+ cha_inst = rd_reg32(&priv->ctrl->perfmon.cha_num_ls);
+ pk_inst = (cha_inst & CHA_ID_LS_PK_MASK) >> CHA_ID_LS_PK_SHIFT;
+
+ /* Do not register algorithms if PKHA is not present. */
+ if (!pk_inst)
+ return -ENODEV;
+
+ err = crypto_register_akcipher(&caam_rsa);
+ if (err)
+ dev_warn(ctrldev, "%s alg registration failed\n",
+ caam_rsa.base.cra_driver_name);
+ else
+ dev_info(ctrldev, "caam pkc algorithms registered in /proc/crypto\n");
+
+ return err;
+}
+
+static void __exit caam_pkc_exit(void)
+{
+ crypto_unregister_akcipher(&caam_rsa);
+}
+
+module_init(caam_pkc_init);
+module_exit(caam_pkc_exit);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("FSL CAAM support for PKC functions of crypto API");
+MODULE_AUTHOR("Freescale Semiconductor");
diff --git a/drivers/crypto/caam/caampkc.h b/drivers/crypto/caam/caampkc.h
new file mode 100644
index 0000000..f595d15
--- /dev/null
+++ b/drivers/crypto/caam/caampkc.h
@@ -0,0 +1,70 @@
+/*
+ * caam - Freescale FSL CAAM support for Public Key Cryptography descriptors
+ *
+ * Copyright 2016 Freescale Semiconductor, Inc.
+ *
+ * There is no Shared Descriptor for PKC so that the Job Descriptor must carry
+ * all the desired key parameters, input and output pointers.
+ */
+
+#ifndef _PKC_DESC_H_
+#define _PKC_DESC_H_
+#include "compat.h"
+#include "pdb.h"
+
+/**
+ * caam_rsa_key - CAAM RSA key structure. Keys are allocated in DMA zone.
+ * @n : RSA modulus raw byte stream
+ * @e : RSA public exponent raw byte stream
+ * @d : RSA private exponent raw byte stream
+ * @n_sz : length in bytes of RSA modulus n
+ * @e_sz : length in bytes of RSA public exponent
+ * @d_sz : length in bytes of RSA private exponent
+ */
+struct caam_rsa_key {
+ u8 *n;
+ u8 *e;
+ u8 *d;
+ size_t n_sz;
+ size_t e_sz;
+ size_t d_sz;
+};
+
+/**
+ * caam_rsa_ctx - per session context.
+ * @key : RSA key in DMA zone
+ * @dev : device structure
+ */
+struct caam_rsa_ctx {
+ struct caam_rsa_key key;
+ struct device *dev;
+};
+
+/**
+ * rsa_edesc - s/w-extended rsa descriptor
+ * @src_nents : number of segments in input scatterlist
+ * @dst_nents : number of segments in output scatterlist
+ * @sec4_sg_bytes : length of h/w link table
+ * @sec4_sg_dma : dma address of h/w link table
+ * @sec4_sg : pointer to h/w link table
+ * @pdb : specific RSA Protocol Data Block (PDB)
+ * @hw_desc : descriptor followed by link tables if any
+ */
+struct rsa_edesc {
+ int src_nents;
+ int dst_nents;
+ int sec4_sg_bytes;
+ dma_addr_t sec4_sg_dma;
+ struct sec4_sg_entry *sec4_sg;
+ union {
+ struct rsa_pub_pdb pub;
+ struct rsa_priv_f1_pdb priv_f1;
+ } pdb;
+ u32 hw_desc[];
+};
+
+/* Descriptor construction primitives. */
+void init_rsa_pub_desc(u32 *desc, struct rsa_pub_pdb *pdb);
+void init_rsa_priv_f1_desc(u32 *desc, struct rsa_priv_f1_pdb *pdb);
+
+#endif
diff --git a/drivers/crypto/caam/compat.h b/drivers/crypto/caam/compat.h
index b6955ec..7149cd2 100644
--- a/drivers/crypto/caam/compat.h
+++ b/drivers/crypto/caam/compat.h
@@ -35,8 +35,11 @@
#include <crypto/md5.h>
#include <crypto/internal/aead.h>
#include <crypto/authenc.h>
+#include <crypto/akcipher.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/skcipher.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/rsa.h>
+#include <crypto/internal/akcipher.h>
#endif /* !defined(CAAM_COMPAT_H) */
diff --git a/drivers/crypto/caam/desc.h b/drivers/crypto/caam/desc.h
index d8d5584..26427c1 100644
--- a/drivers/crypto/caam/desc.h
+++ b/drivers/crypto/caam/desc.h
@@ -453,6 +453,8 @@ struct sec4_sg_entry {
#define OP_PCLID_PUBLICKEYPAIR (0x14 << OP_PCLID_SHIFT)
#define OP_PCLID_DSASIGN (0x15 << OP_PCLID_SHIFT)
#define OP_PCLID_DSAVERIFY (0x16 << OP_PCLID_SHIFT)
+#define OP_PCLID_RSAENC_PUBKEY (0x18 << OP_PCLID_SHIFT)
+#define OP_PCLID_RSADEC_PRVKEY (0x19 << OP_PCLID_SHIFT)
/* Assuming OP_TYPE = OP_TYPE_DECAP_PROTOCOL/ENCAP_PROTOCOL */
#define OP_PCLID_IPSEC (0x01 << OP_PCLID_SHIFT)
diff --git a/drivers/crypto/caam/desc_constr.h b/drivers/crypto/caam/desc_constr.h
index ae3aef6..d3869b9 100644
--- a/drivers/crypto/caam/desc_constr.h
+++ b/drivers/crypto/caam/desc_constr.h
@@ -77,6 +77,13 @@ static inline void init_job_desc(u32 *desc, u32 options)
init_desc(desc, CMD_DESC_HDR | options);
}
+static inline void init_job_desc_pdb(u32 *desc, u32 options, size_t pdb_bytes)
+{
+ u32 pdb_len = (pdb_bytes + CAAM_CMD_SZ - 1) / CAAM_CMD_SZ;
+
+ init_job_desc(desc, (((pdb_len + 1) << HDR_START_IDX_SHIFT)) | options);
+}
+
static inline void append_ptr(u32 *desc, dma_addr_t ptr)
{
dma_addr_t *offset = (dma_addr_t *)desc_end(desc);
diff --git a/drivers/crypto/caam/pdb.h b/drivers/crypto/caam/pdb.h
index d383573..aaa00dd 100644
--- a/drivers/crypto/caam/pdb.h
+++ b/drivers/crypto/caam/pdb.h
@@ -1,12 +1,13 @@
/*
* CAAM Protocol Data Block (PDB) definition header file
*
- * Copyright 2008-2012 Freescale Semiconductor, Inc.
+ * Copyright 2008-2016 Freescale Semiconductor, Inc.
*
*/
#ifndef CAAM_PDB_H
#define CAAM_PDB_H
+#include "compat.h"
/*
* PDB- IPSec ESP Header Modification Options
@@ -476,4 +477,52 @@ struct dsa_verify_pdb {
u8 *ab; /* only used if ECC processing */
};
+/* RSA Protocol Data Block */
+#define RSA_PDB_SGF_SHIFT 28
+#define RSA_PDB_E_SHIFT 12
+#define RSA_PDB_E_MASK (0xFFF << RSA_PDB_E_SHIFT)
+#define RSA_PDB_D_SHIFT 12
+#define RSA_PDB_D_MASK (0xFFF << RSA_PDB_D_SHIFT)
+
+#define RSA_PDB_SGF_F (0x8 << RSA_PDB_SGF_SHIFT)
+#define RSA_PDB_SGF_G (0x4 << RSA_PDB_SGF_SHIFT)
+#define RSA_PRIV_PDB_SGF_F (0x4 << RSA_PDB_SGF_SHIFT)
+#define RSA_PRIV_PDB_SGF_G (0x8 << RSA_PDB_SGF_SHIFT)
+
+#define RSA_PRIV_KEY_FRM_1 0
+
+/**
+ * RSA Encrypt Protocol Data Block
+ * @sgf: scatter-gather field
+ * @f_dma: dma address of input data
+ * @g_dma: dma address of encrypted output data
+ * @n_dma: dma address of RSA modulus
+ * @e_dma: dma address of RSA public exponent
+ * @f_len: length in octets of the input data
+ */
+struct rsa_pub_pdb {
+ u32 sgf;
+ dma_addr_t f_dma;
+ dma_addr_t g_dma;
+ dma_addr_t n_dma;
+ dma_addr_t e_dma;
+ u32 f_len;
+} __packed;
+
+/**
+ * RSA Decrypt PDB - Private Key Form #1
+ * @sgf: scatter-gather field
+ * @g_dma: dma address of encrypted input data
+ * @f_dma: dma address of output data
+ * @n_dma: dma address of RSA modulus
+ * @d_dma: dma address of RSA private exponent
+ */
+struct rsa_priv_f1_pdb {
+ u32 sgf;
+ dma_addr_t g_dma;
+ dma_addr_t f_dma;
+ dma_addr_t n_dma;
+ dma_addr_t d_dma;
+} __packed;
+
#endif
diff --git a/drivers/crypto/caam/pkc_desc.c b/drivers/crypto/caam/pkc_desc.c
new file mode 100644
index 0000000..4e4183e
--- /dev/null
+++ b/drivers/crypto/caam/pkc_desc.c
@@ -0,0 +1,36 @@
+/*
+ * caam - Freescale FSL CAAM support for Public Key Cryptography descriptors
+ *
+ * Copyright 2016 Freescale Semiconductor, Inc.
+ *
+ * There is no Shared Descriptor for PKC so that the Job Descriptor must carry
+ * all the desired key parameters, input and output pointers.
+ */
+#include "caampkc.h"
+#include "desc_constr.h"
+
+/* Descriptor for RSA Public operation */
+void init_rsa_pub_desc(u32 *desc, struct rsa_pub_pdb *pdb)
+{
+ init_job_desc_pdb(desc, 0, sizeof(*pdb));
+ append_cmd(desc, pdb->sgf);
+ append_ptr(desc, pdb->f_dma);
+ append_ptr(desc, pdb->g_dma);
+ append_ptr(desc, pdb->n_dma);
+ append_ptr(desc, pdb->e_dma);
+ append_cmd(desc, pdb->f_len);
+ append_operation(desc, OP_TYPE_UNI_PROTOCOL | OP_PCLID_RSAENC_PUBKEY);
+}
+
+/* Descriptor for RSA Private operation - Private Key Form #1 */
+void init_rsa_priv_f1_desc(u32 *desc, struct rsa_priv_f1_pdb *pdb)
+{
+ init_job_desc_pdb(desc, 0, sizeof(*pdb));
+ append_cmd(desc, pdb->sgf);
+ append_ptr(desc, pdb->g_dma);
+ append_ptr(desc, pdb->f_dma);
+ append_ptr(desc, pdb->n_dma);
+ append_ptr(desc, pdb->d_dma);
+ append_operation(desc, OP_TYPE_UNI_PROTOCOL | OP_PCLID_RSADEC_PRVKEY |
+ RSA_PRIV_KEY_FRM_1);
+}
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 2/3] crypto: scatterwalk - export scatterwalk_pagedone
From: Tudor Ambarus @ 2016-06-15 14:52 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto, horia.geanta, Tudor Ambarus
In-Reply-To: <1466002364-16945-1-git-send-email-tudor-dan.ambarus@nxp.com>
Used in caam driver. Export the symbol since the caam driver
can be built as a module.
Signed-off-by: Tudor Ambarus <tudor-dan.ambarus@nxp.com>
---
crypto/scatterwalk.c | 5 +++--
include/crypto/scatterwalk.h | 2 ++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/crypto/scatterwalk.c b/crypto/scatterwalk.c
index bc3222d..03d34f9 100644
--- a/crypto/scatterwalk.c
+++ b/crypto/scatterwalk.c
@@ -47,8 +47,8 @@ void *scatterwalk_map(struct scatter_walk *walk)
}
EXPORT_SYMBOL_GPL(scatterwalk_map);
-static void scatterwalk_pagedone(struct scatter_walk *walk, int out,
- unsigned int more)
+void scatterwalk_pagedone(struct scatter_walk *walk, int out,
+ unsigned int more)
{
if (out) {
struct page *page;
@@ -69,6 +69,7 @@ static void scatterwalk_pagedone(struct scatter_walk *walk, int out,
scatterwalk_start(walk, sg_next(walk->sg));
}
}
+EXPORT_SYMBOL_GPL(scatterwalk_pagedone);
void scatterwalk_done(struct scatter_walk *walk, int out, int more)
{
diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h
index 8b799c5..6535a20 100644
--- a/include/crypto/scatterwalk.h
+++ b/include/crypto/scatterwalk.h
@@ -89,6 +89,8 @@ void scatterwalk_copychunks(void *buf, struct scatter_walk *walk,
void scatterwalk_sg_copychunks(struct scatter_walk *dest,
struct scatter_walk *src, size_t nbytes);
void *scatterwalk_map(struct scatter_walk *walk);
+void scatterwalk_pagedone(struct scatter_walk *walk, int out,
+ unsigned int more);
void scatterwalk_done(struct scatter_walk *walk, int out, int more);
void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg,
--
1.8.3.1
^ permalink raw reply related
* [PATCH v8 1/3] crypto: scatterwak - Add scatterwalk_sg_copychunks
From: Tudor Ambarus @ 2016-06-15 14:52 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto, horia.geanta, Tudor Ambarus
In-Reply-To: <1466002364-16945-1-git-send-email-tudor-dan.ambarus@nxp.com>
This patch adds the function scatterwalk_sg_copychunks which writes
a chunk of data from a scatterwalk to another scatterwalk.
It will be used by caam driver to remove the leading zeros
for the output data of the RSA algorithm, after the computation completes.
Signed-off-by: Tudor Ambarus <tudor-dan.ambarus@nxp.com>
---
crypto/scatterwalk.c | 26 ++++++++++++++++++++++++++
include/crypto/scatterwalk.h | 2 ++
2 files changed, 28 insertions(+)
diff --git a/crypto/scatterwalk.c b/crypto/scatterwalk.c
index ea5815c..bc3222d 100644
--- a/crypto/scatterwalk.c
+++ b/crypto/scatterwalk.c
@@ -125,6 +125,32 @@ void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg,
}
EXPORT_SYMBOL_GPL(scatterwalk_map_and_copy);
+void scatterwalk_sg_copychunks(struct scatter_walk *dest,
+ struct scatter_walk *src, size_t nbytes)
+{
+ for (;;) {
+ unsigned int len_this_page = scatterwalk_pagelen(dest);
+ u8 *vaddr;
+
+ if (len_this_page > nbytes)
+ len_this_page = nbytes;
+
+ vaddr = scatterwalk_map(dest);
+ scatterwalk_copychunks(vaddr, src, len_this_page, 0);
+ scatterwalk_unmap(vaddr);
+
+ scatterwalk_advance(dest, len_this_page);
+
+ if (nbytes == len_this_page)
+ break;
+
+ nbytes -= len_this_page;
+
+ scatterwalk_pagedone(dest, 0, 1);
+ }
+}
+EXPORT_SYMBOL_GPL(scatterwalk_sg_copychunks);
+
int scatterwalk_bytes_sglen(struct scatterlist *sg, int num_bytes)
{
int offset = 0, n = 0;
diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h
index 35f99b6..8b799c5 100644
--- a/include/crypto/scatterwalk.h
+++ b/include/crypto/scatterwalk.h
@@ -86,6 +86,8 @@ static inline void scatterwalk_unmap(void *vaddr)
void scatterwalk_start(struct scatter_walk *walk, struct scatterlist *sg);
void scatterwalk_copychunks(void *buf, struct scatter_walk *walk,
size_t nbytes, int out);
+void scatterwalk_sg_copychunks(struct scatter_walk *dest,
+ struct scatter_walk *src, size_t nbytes);
void *scatterwalk_map(struct scatter_walk *walk);
void scatterwalk_done(struct scatter_walk *walk, int out, int more);
--
1.8.3.1
^ permalink raw reply related
* Re: select on non-existing Kconfig option CRC32C
From: Hendrik Brueckner @ 2016-06-15 14:41 UTC (permalink / raw)
To: Andreas Ziegler
Cc: Hendrik Brueckner, Martin Schwidefsky, Herbert Xu,
David S. Miller, linux-crypto, linux-kernel
In-Reply-To: <5761275B.4010508@fau.de>
Hi Andreas,
On Wed, Jun 15, 2016 at 12:00:59PM +0200, Andreas Ziegler wrote:
>
> your patch "s390/crc32-vx: add crypto API module for optimized CRC-32
> algorithms" showed up in linux-next today (next-20160615) as commit 364148e0b195.
>
> The patch defines the Kconfig option CRYPTO_CRC32_S390 which 'select's CRC32C.
> However, this should probably have been CRYPTO_CRC32C, as CRC32C does not exist.
Thanks for informing me. Actually, the crc32-vx driver requires the
__crc32c_le() function which is available by selecting CONFIG_CRC32. There is
no need for CRYPTO_CRC32C. So this can be safely removed.
> Should I prepare a trivial patch to fix this up or would you like to do that on
> your side?
Martin have already corrected the patch.
Thanks and kind regards,
Hendrik
^ permalink raw reply
* Re: select on non-existing Kconfig option CRC32C
From: Randy Dunlap @ 2016-06-15 14:40 UTC (permalink / raw)
To: Andreas Ziegler, Hendrik Brueckner
Cc: Martin Schwidefsky, Herbert Xu, David S. Miller, linux-crypto,
linux-kernel
In-Reply-To: <5761275B.4010508@fau.de>
On 06/15/16 03:00, Andreas Ziegler wrote:
> Hi Hendrik,
>
> your patch "s390/crc32-vx: add crypto API module for optimized CRC-32
> algorithms" showed up in linux-next today (next-20160615) as commit 364148e0b195.
>
> The patch defines the Kconfig option CRYPTO_CRC32_S390 which 'select's CRC32C.
> However, this should probably have been CRYPTO_CRC32C, as CRC32C does not exist.
> Should I prepare a trivial patch to fix this up or would you like to do that on
> your side?
>
> I found this issue by comparing yesterday's tree and today's tree using
> 'scripts/checkkconfigsymbols -f -d next-20160614..next-20160615'.
or should it select CRC32 or LIBCRC32C? (probably not the LIB... one)
--
~Randy
^ permalink raw reply
* crypto: gcm - Filter out async ghash if necessary
From: Herbert Xu @ 2016-06-15 14:27 UTC (permalink / raw)
To: Linux Crypto Mailing List
As it is if you ask for a sync gcm you may actually end up with
an async one because it does not filter out async implementations
of ghash.
This patch fixes this by adding the necessary filter when looking
for ghash.
Cc: stable@vger.kernel.org
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
diff --git a/crypto/gcm.c b/crypto/gcm.c
index bec329b..917b8fb 100644
--- a/crypto/gcm.c
+++ b/crypto/gcm.c
@@ -639,7 +639,9 @@ static int crypto_gcm_create_common(struct crypto_template *tmpl,
ghash_alg = crypto_find_alg(ghash_name, &crypto_ahash_type,
CRYPTO_ALG_TYPE_HASH,
- CRYPTO_ALG_TYPE_AHASH_MASK);
+ CRYPTO_ALG_TYPE_AHASH_MASK |
+ crypto_requires_sync(algt->type,
+ algt->mask));
if (IS_ERR(ghash_alg))
return PTR_ERR(ghash_alg);
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox