* [PATCH] crypto: s5p-sss - Fix spinlock recursion on LRW(AES)
From: Krzysztof Kozlowski @ 2017-03-08 21:14 UTC (permalink / raw)
To: Herbert Xu, davem, Vladimir Zapolskiy, Marek Szyprowski,
linux-crypto, linux-kernel
Cc: Nathan Royce, Krzysztof Kozlowski,
# v4 . 10 . x : 07de4bc88c crypto : s5p-sss - Fix completing
Running TCRYPT with LRW compiled causes spinlock recursion:
testing speed of async lrw(aes) (lrw(ecb-aes-s5p)) encryption
tcrypt: test 0 (256 bit key, 16 byte blocks): 19007 operations in 1 seconds (304112 bytes)
tcrypt: test 1 (256 bit key, 64 byte blocks): 15753 operations in 1 seconds (1008192 bytes)
tcrypt: test 2 (256 bit key, 256 byte blocks): 14293 operations in 1 seconds (3659008 bytes)
tcrypt: test 3 (256 bit key, 1024 byte blocks): 11906 operations in 1 seconds (12191744 bytes)
tcrypt: test 4 (256 bit key, 8192 byte blocks):
BUG: spinlock recursion on CPU#1, irq/84-10830000/89
lock: 0xeea99a68, .magic: dead4ead, .owner: irq/84-10830000/89, .owner_cpu: 1
CPU: 1 PID: 89 Comm: irq/84-10830000 Not tainted 4.11.0-rc1-00001-g897ca6d0800d #559
Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[<c010e1ec>] (unwind_backtrace) from [<c010ae1c>] (show_stack+0x10/0x14)
[<c010ae1c>] (show_stack) from [<c03449c0>] (dump_stack+0x78/0x8c)
[<c03449c0>] (dump_stack) from [<c015de68>] (do_raw_spin_lock+0x11c/0x120)
[<c015de68>] (do_raw_spin_lock) from [<c0720110>] (_raw_spin_lock_irqsave+0x20/0x28)
[<c0720110>] (_raw_spin_lock_irqsave) from [<c0572ca0>] (s5p_aes_crypt+0x2c/0xb4)
[<c0572ca0>] (s5p_aes_crypt) from [<bf1d8aa4>] (do_encrypt+0x78/0xb0 [lrw])
[<bf1d8aa4>] (do_encrypt [lrw]) from [<bf1d8b00>] (encrypt_done+0x24/0x54 [lrw])
[<bf1d8b00>] (encrypt_done [lrw]) from [<c05732a0>] (s5p_aes_complete+0x60/0xcc)
[<c05732a0>] (s5p_aes_complete) from [<c0573440>] (s5p_aes_interrupt+0x134/0x1a0)
[<c0573440>] (s5p_aes_interrupt) from [<c01667c4>] (irq_thread_fn+0x1c/0x54)
[<c01667c4>] (irq_thread_fn) from [<c0166a98>] (irq_thread+0x12c/0x1e0)
[<c0166a98>] (irq_thread) from [<c0136a28>] (kthread+0x108/0x138)
[<c0136a28>] (kthread) from [<c0107778>] (ret_from_fork+0x14/0x3c)
Interrupt handling routine was calling req->base.complete() under
spinlock. In most cases this wasn't fatal but when combined with some
of the cipher modes (like LRW) this caused recursion - starting the new
encryption (s5p_aes_crypt()) while still holding the spinlock from
previous round (s5p_aes_complete()).
Beside that, the s5p_aes_interrupt() error handling path could execute
two completions in case of error for RX and TX blocks.
Rewrite the interrupt handling routine and the completion by:
1. Splitting the operations on scatterlist copies from
s5p_aes_complete() into separate s5p_sg_done(). This still should be
done under lock.
The s5p_aes_complete() now only calls req->base.complete() and it has
to be called outside of lock.
2. Moving the s5p_aes_complete() out of spinlock critical sections.
In interrupt service routine s5p_aes_interrupts(), it appeared in few
places, including error paths inside other functions called from ISR.
This code was not so obvious to read so simplify it by putting the
s5p_aes_complete() only within ISR level.
Reported-by: Nathan Royce <nroycea+kernel@gmail.com>
Cc: <stable@vger.kernel.org> # v4.10.x: 07de4bc88c crypto: s5p-sss - Fix completing
Cc: <stable@vger.kernel.org> # v4.10.x
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
---
I think, along with 07de4bc88c this should be backported to v4.10 as it
happens there.
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
---
drivers/crypto/s5p-sss.c | 127 ++++++++++++++++++++++++++++++-----------------
1 file changed, 82 insertions(+), 45 deletions(-)
diff --git a/drivers/crypto/s5p-sss.c b/drivers/crypto/s5p-sss.c
index a668286d62cb..1b9da3dc799b 100644
--- a/drivers/crypto/s5p-sss.c
+++ b/drivers/crypto/s5p-sss.c
@@ -270,7 +270,7 @@ static void s5p_sg_copy_buf(void *buf, struct scatterlist *sg,
scatterwalk_done(&walk, out, 0);
}
-static void s5p_aes_complete(struct s5p_aes_dev *dev, int err)
+static void s5p_sg_done(struct s5p_aes_dev *dev)
{
if (dev->sg_dst_cpy) {
dev_dbg(dev->dev,
@@ -281,8 +281,11 @@ static void s5p_aes_complete(struct s5p_aes_dev *dev, int err)
}
s5p_free_sg_cpy(dev, &dev->sg_src_cpy);
s5p_free_sg_cpy(dev, &dev->sg_dst_cpy);
+}
- /* holding a lock outside */
+/* Calls the completion. Cannot be called with dev->lock hold. */
+static void s5p_aes_complete(struct s5p_aes_dev *dev, int err)
+{
dev->req->base.complete(&dev->req->base, err);
dev->busy = false;
}
@@ -368,51 +371,44 @@ static int s5p_set_indata(struct s5p_aes_dev *dev, struct scatterlist *sg)
}
/*
- * Returns true if new transmitting (output) data is ready and its
- * address+length have to be written to device (by calling
- * s5p_set_dma_outdata()). False otherwise.
+ * Returns -ERRNO on error (mapping of new data failed).
+ * On success returns:
+ * - 0 if there is no more data,
+ * - 1 if new transmitting (output) data is ready and its address+length
+ * have to be written to device (by calling s5p_set_dma_outdata()).
*/
-static bool s5p_aes_tx(struct s5p_aes_dev *dev)
+static int s5p_aes_tx(struct s5p_aes_dev *dev)
{
- int err = 0;
- bool ret = false;
+ int ret = 0;
s5p_unset_outdata(dev);
if (!sg_is_last(dev->sg_dst)) {
- err = s5p_set_outdata(dev, sg_next(dev->sg_dst));
- if (err)
- s5p_aes_complete(dev, err);
- else
- ret = true;
- } else {
- s5p_aes_complete(dev, err);
-
- dev->busy = true;
- tasklet_schedule(&dev->tasklet);
+ ret = s5p_set_outdata(dev, sg_next(dev->sg_dst));
+ if (!ret)
+ ret = 1;
}
return ret;
}
/*
- * Returns true if new receiving (input) data is ready and its
- * address+length have to be written to device (by calling
- * s5p_set_dma_indata()). False otherwise.
+ * Returns -ERRNO on error (mapping of new data failed).
+ * On success returns:
+ * - 0 if there is no more data,
+ * - 1 if new receiving (input) data is ready and its address+length
+ * have to be written to device (by calling s5p_set_dma_indata()).
*/
-static bool s5p_aes_rx(struct s5p_aes_dev *dev)
+static int s5p_aes_rx(struct s5p_aes_dev *dev/*, bool *set_dma*/)
{
- int err;
- bool ret = false;
+ int ret = 0;
s5p_unset_indata(dev);
if (!sg_is_last(dev->sg_src)) {
- err = s5p_set_indata(dev, sg_next(dev->sg_src));
- if (err)
- s5p_aes_complete(dev, err);
- else
- ret = true;
+ ret = s5p_set_indata(dev, sg_next(dev->sg_src));
+ if (!ret)
+ ret = 1;
}
return ret;
@@ -422,33 +418,73 @@ static irqreturn_t s5p_aes_interrupt(int irq, void *dev_id)
{
struct platform_device *pdev = dev_id;
struct s5p_aes_dev *dev = platform_get_drvdata(pdev);
- bool set_dma_tx = false;
- bool set_dma_rx = false;
+ int err_dma_tx = 0;
+ int err_dma_rx = 0;
+ bool tx_end = false;
unsigned long flags;
uint32_t status;
+ int err;
spin_lock_irqsave(&dev->lock, flags);
+ /*
+ * Handle rx or tx interrupt. If there is still data (scatterlist did not
+ * reach end), then map next scatterlist entry.
+ * In case of such mapping error, s5p_aes_complete() should be called.
+ *
+ * If there is no more data in tx scatter list, call s5p_aes_complete()
+ * and schedule new tasklet.
+ */
status = SSS_READ(dev, FCINTSTAT);
if (status & SSS_FCINTSTAT_BRDMAINT)
- set_dma_rx = s5p_aes_rx(dev);
- if (status & SSS_FCINTSTAT_BTDMAINT)
- set_dma_tx = s5p_aes_tx(dev);
+ err_dma_rx = s5p_aes_rx(dev);
+
+ if (status & SSS_FCINTSTAT_BTDMAINT) {
+ if (sg_is_last(dev->sg_dst))
+ tx_end = true;
+ err_dma_tx = s5p_aes_tx(dev);
+ }
SSS_WRITE(dev, FCINTPEND, status);
- /*
- * Writing length of DMA block (either receiving or transmitting)
- * will start the operation immediately, so this should be done
- * at the end (even after clearing pending interrupts to not miss the
- * interrupt).
- */
- if (set_dma_tx)
- s5p_set_dma_outdata(dev, dev->sg_dst);
- if (set_dma_rx)
- s5p_set_dma_indata(dev, dev->sg_src);
+ if (err_dma_rx < 0) {
+ err = err_dma_rx;
+ goto error;
+ }
+ if (err_dma_tx < 0) {
+ err = err_dma_tx;
+ goto error;
+ }
+
+ if (tx_end) {
+ s5p_sg_done(dev);
+
+ spin_unlock_irqrestore(&dev->lock, flags);
+
+ s5p_aes_complete(dev, 0);
+ dev->busy = true;
+ tasklet_schedule(&dev->tasklet);
+ } else {
+ /*
+ * Writing length of DMA block (either receiving or
+ * transmitting) will start the operation immediately, so this
+ * should be done at the end (even after clearing pending
+ * interrupts to not miss the interrupt).
+ */
+ if (err_dma_tx == 1)
+ s5p_set_dma_outdata(dev, dev->sg_dst);
+ if (err_dma_rx == 1)
+ s5p_set_dma_indata(dev, dev->sg_src);
+ spin_unlock_irqrestore(&dev->lock, flags);
+ }
+
+ return IRQ_HANDLED;
+
+error:
+ s5p_sg_done(dev);
spin_unlock_irqrestore(&dev->lock, flags);
+ s5p_aes_complete(dev, err);
return IRQ_HANDLED;
}
@@ -597,8 +633,9 @@ static void s5p_aes_crypt_start(struct s5p_aes_dev *dev, unsigned long mode)
s5p_unset_indata(dev);
indata_error:
- s5p_aes_complete(dev, err);
+ s5p_sg_done(dev);
spin_unlock_irqrestore(&dev->lock, flags);
+ s5p_aes_complete(dev, err);
}
static void s5p_tasklet_cb(unsigned long data)
--
2.9.3
^ permalink raw reply related
* Re: XTS Crypto Not Found In /proc/crypto Even After Compiled for 4.10.1.
From: Krzysztof Kozlowski @ 2017-03-08 21:15 UTC (permalink / raw)
To: Nathan Royce
Cc: Herbert Xu, davem, linux-crypto, linux-kernel, Marek Szyprowski
In-Reply-To: <20170308174542.2rydwxmrb3oehyrc@kozik-lap>
On Wed, Mar 08, 2017 at 07:45:42PM +0200, Krzysztof Kozlowski wrote:
> On Mon, Mar 06, 2017 at 03:29:48PM -0600, Nathan Royce wrote:
> > OK, I just tried 4.10.0 and the output is looking the same.
> >
> > I can't say my setup is all that odd. The cryptographic use is only
> > with the swap partition found in my original email (seen in Herbert's
> > reply).
>
> You have quite specific/customized config and compilation flags.
> I doubt that it causes the issue but at least the config is unusable on
> anything other then XU3/XU4.
>
> Anyway I reproduced some of the issues with tcrypt on Odroid U3 on v4.10
> and v4.11-rc1, with regular exynos defconfig plus some more crypto
> algorithms:
> 1. Without my patch: the original warning.
> 2. With my patch:
I sent a fix. At least for spin lock recursion in tcrypt.
Could you give it a try?
Best regards,
Krzysztof
^ permalink raw reply
* [PATCH v1 0/8] improve performances on mediatek crypto driver
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
Hi all,
Some patches of this series improve the performances whereas others
clean up code and refine data structure to make it more efficient
Changes since v1:
- drop OFB and CFB patch
Ryder Lee (8):
crypto: mediatek - rework interrupt handler
crypto: mediatek - add MTK_* prefix and correct annotations.
crypto: mediatek - make mtk_sha_xmit() more generic
crypto: mediatek - simplify descriptor ring management
crypto: mediatek - add queue_task tasklet
crypto: mediatek - fix error handling in mtk_aes_complete()
crypto: mediatek - add mtk_aes_gcm_tag_verify()
crypto: mediatek - make hardware operation flow more efficient
drivers/crypto/mediatek/mtk-aes.c | 421 +++++++++++++++++----------------
drivers/crypto/mediatek/mtk-platform.c | 15 +-
drivers/crypto/mediatek/mtk-platform.h | 56 +++--
drivers/crypto/mediatek/mtk-sha.c | 309 +++++++++---------------
4 files changed, 369 insertions(+), 432 deletions(-)
--
1.9.1
^ permalink raw reply
* [PATCH v1 1/8] crypto: mediatek - rework interrupt handler
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This patch removes redundant task that used to handle interrupt
from ring manager, so that the same task/handler can be shared.
It also uses aes->id and sha-id to distinguish interrupt sources.
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 73 +++++++++++-----------------------
drivers/crypto/mediatek/mtk-platform.h | 4 ++
drivers/crypto/mediatek/mtk-sha.c | 73 ++++++++++------------------------
3 files changed, 49 insertions(+), 101 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 3a47cdb..e67881f 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -1092,55 +1092,26 @@ static void mtk_aes_gcm_exit(struct crypto_aead *aead)
},
};
-static void mtk_aes_enc_task(unsigned long data)
+static void mtk_aes_done_task(unsigned long data)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_aes_rec *aes = cryp->aes[0];
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
+ struct mtk_cryp *cryp = aes->cryp;
mtk_aes_unmap(cryp, aes);
aes->resume(cryp, aes);
}
-static void mtk_aes_dec_task(unsigned long data)
+static irqreturn_t mtk_aes_irq(int irq, void *dev_id)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_aes_rec *aes = cryp->aes[1];
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)dev_id;
+ struct mtk_cryp *cryp = aes->cryp;
+ u32 val = mtk_aes_read(cryp, RDR_STAT(aes->id));
- mtk_aes_unmap(cryp, aes);
- aes->resume(cryp, aes);
-}
-
-static irqreturn_t mtk_aes_enc_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_aes_rec *aes = cryp->aes[0];
- u32 val = mtk_aes_read(cryp, RDR_STAT(RING0));
-
- mtk_aes_write(cryp, RDR_STAT(RING0), val);
-
- if (likely(AES_FLAGS_BUSY & aes->flags)) {
- mtk_aes_write(cryp, RDR_PROC_COUNT(RING0), MTK_CNT_RST);
- mtk_aes_write(cryp, RDR_THRESH(RING0),
- MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
-
- tasklet_schedule(&aes->task);
- } else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
- }
- return IRQ_HANDLED;
-}
-
-static irqreturn_t mtk_aes_dec_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_aes_rec *aes = cryp->aes[1];
- u32 val = mtk_aes_read(cryp, RDR_STAT(RING1));
-
- mtk_aes_write(cryp, RDR_STAT(RING1), val);
+ mtk_aes_write(cryp, RDR_STAT(aes->id), val);
if (likely(AES_FLAGS_BUSY & aes->flags)) {
- mtk_aes_write(cryp, RDR_PROC_COUNT(RING1), MTK_CNT_RST);
- mtk_aes_write(cryp, RDR_THRESH(RING1),
+ mtk_aes_write(cryp, RDR_PROC_COUNT(aes->id), MTK_CNT_RST);
+ mtk_aes_write(cryp, RDR_THRESH(aes->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
tasklet_schedule(&aes->task);
@@ -1171,14 +1142,18 @@ static int mtk_aes_record_init(struct mtk_cryp *cryp)
if (!aes[i]->buf)
goto err_cleanup;
- aes[i]->id = i;
+ aes[i]->cryp = cryp;
spin_lock_init(&aes[i]->lock);
crypto_init_queue(&aes[i]->queue, AES_QUEUE_SIZE);
+
+ tasklet_init(&aes[i]->task, mtk_aes_done_task,
+ (unsigned long)aes[i]);
}
- tasklet_init(&aes[0]->task, mtk_aes_enc_task, (unsigned long)cryp);
- tasklet_init(&aes[1]->task, mtk_aes_dec_task, (unsigned long)cryp);
+ /* Link to ring0 and ring1 respectively */
+ aes[0]->id = RING0;
+ aes[1]->id = RING1;
return 0;
@@ -1246,19 +1221,17 @@ int mtk_cipher_alg_register(struct mtk_cryp *cryp)
if (ret)
goto err_record;
- /* Ring0 is use by encryption record */
- ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_enc_irq,
- IRQF_TRIGGER_LOW, "mtk-aes", cryp);
+ ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_irq,
+ 0, "mtk-aes", cryp->aes[0]);
if (ret) {
- dev_err(cryp->dev, "unable to request AES encryption irq.\n");
+ dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
- /* Ring1 is use by decryption record */
- ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_dec_irq,
- IRQF_TRIGGER_LOW, "mtk-aes", cryp);
+ ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_irq,
+ 0, "mtk-aes", cryp->aes[1]);
if (ret) {
- dev_err(cryp->dev, "unable to request AES decryption irq.\n");
+ dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index ed6d871..d2a1db6 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -125,6 +125,7 @@ struct mtk_aes_dma {
/**
* struct mtk_aes_rec - AES operation record
+ * @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @areq: pointer to async request
* @task: the tasklet is use in AES interrupt
@@ -143,6 +144,7 @@ struct mtk_aes_dma {
* Structure used to record AES execution state.
*/
struct mtk_aes_rec {
+ struct mtk_cryp *cryp;
struct crypto_queue queue;
struct crypto_async_request *areq;
struct tasklet_struct task;
@@ -166,6 +168,7 @@ struct mtk_aes_rec {
/**
* struct mtk_sha_rec - SHA operation record
+ * @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @req: pointer to ahash request
* @task: the tasklet is use in SHA interrupt
@@ -176,6 +179,7 @@ struct mtk_aes_rec {
* Structure used to record SHA execution state.
*/
struct mtk_sha_rec {
+ struct mtk_cryp *cryp;
struct crypto_queue queue;
struct ahash_request *req;
struct tasklet_struct task;
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 55e3805..1fb98f6 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -1216,60 +1216,31 @@ static void mtk_sha_cra_exit(struct crypto_tfm *tfm)
},
};
-static void mtk_sha_task0(unsigned long data)
+static void mtk_sha_done_task(unsigned long data)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_sha_rec *sha = cryp->sha[0];
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
+ struct mtk_cryp *cryp = sha->cryp;
mtk_sha_unmap(cryp, sha);
mtk_sha_complete(cryp, sha);
}
-static void mtk_sha_task1(unsigned long data)
+static irqreturn_t mtk_sha_irq(int irq, void *dev_id)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_sha_rec *sha = cryp->sha[1];
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)dev_id;
+ struct mtk_cryp *cryp = sha->cryp;
+ u32 val = mtk_sha_read(cryp, RDR_STAT(sha->id));
- mtk_sha_unmap(cryp, sha);
- mtk_sha_complete(cryp, sha);
-}
-
-static irqreturn_t mtk_sha_ring2_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_sha_rec *sha = cryp->sha[0];
- u32 val = mtk_sha_read(cryp, RDR_STAT(RING2));
-
- mtk_sha_write(cryp, RDR_STAT(RING2), val);
+ mtk_sha_write(cryp, RDR_STAT(sha->id), val);
if (likely((SHA_FLAGS_BUSY & sha->flags))) {
- mtk_sha_write(cryp, RDR_PROC_COUNT(RING2), MTK_CNT_RST);
- mtk_sha_write(cryp, RDR_THRESH(RING2),
+ mtk_sha_write(cryp, RDR_PROC_COUNT(sha->id), MTK_CNT_RST);
+ mtk_sha_write(cryp, RDR_THRESH(sha->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
tasklet_schedule(&sha->task);
} else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
- }
- return IRQ_HANDLED;
-}
-
-static irqreturn_t mtk_sha_ring3_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_sha_rec *sha = cryp->sha[1];
- u32 val = mtk_sha_read(cryp, RDR_STAT(RING3));
-
- mtk_sha_write(cryp, RDR_STAT(RING3), val);
-
- if (likely((SHA_FLAGS_BUSY & sha->flags))) {
- mtk_sha_write(cryp, RDR_PROC_COUNT(RING3), MTK_CNT_RST);
- mtk_sha_write(cryp, RDR_THRESH(RING3),
- MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
-
- tasklet_schedule(&sha->task);
- } else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
+ dev_warn(cryp->dev, "SHA interrupt when no active requests.\n");
}
return IRQ_HANDLED;
}
@@ -1288,14 +1259,18 @@ static int mtk_sha_record_init(struct mtk_cryp *cryp)
if (!sha[i])
goto err_cleanup;
- sha[i]->id = i + RING2;
+ sha[i]->cryp = cryp;
spin_lock_init(&sha[i]->lock);
crypto_init_queue(&sha[i]->queue, SHA_QUEUE_SIZE);
+
+ tasklet_init(&sha[i]->task, mtk_sha_done_task,
+ (unsigned long)sha[i]);
}
- tasklet_init(&sha[0]->task, mtk_sha_task0, (unsigned long)cryp);
- tasklet_init(&sha[1]->task, mtk_sha_task1, (unsigned long)cryp);
+ /* Link to ring2 and ring3 respectively */
+ sha[0]->id = RING2;
+ sha[1]->id = RING3;
cryp->rec = 1;
@@ -1368,19 +1343,15 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
if (err)
goto err_record;
- /* Ring2 is use by SHA record0 */
- err = devm_request_irq(cryp->dev, cryp->irq[RING2],
- mtk_sha_ring2_irq, IRQF_TRIGGER_LOW,
- "mtk-sha", cryp);
+ err = devm_request_irq(cryp->dev, cryp->irq[RING2], mtk_sha_irq,
+ 0, "mtk-sha", cryp->sha[0]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq0.\n");
goto err_res;
}
- /* Ring3 is use by SHA record1 */
- err = devm_request_irq(cryp->dev, cryp->irq[RING3],
- mtk_sha_ring3_irq, IRQF_TRIGGER_LOW,
- "mtk-sha", cryp);
+ err = devm_request_irq(cryp->dev, cryp->irq[RING3], mtk_sha_irq,
+ 0, "mtk-sha", cryp->sha[1]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq1.\n");
goto err_res;
--
1.9.1
^ permalink raw reply related
* [PATCH v1 2/8] crypto: mediatek - add MTK_* prefix and correct annotations.
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
Dummy patch to add MTK_* prefix to ring enum and fix incorrect annotations.
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 12 ++++++------
drivers/crypto/mediatek/mtk-platform.c | 12 ++++++------
drivers/crypto/mediatek/mtk-platform.h | 26 +++++++++++++-------------
drivers/crypto/mediatek/mtk-sha.c | 14 +++++++-------
4 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index e67881f..b57b68f 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -1152,8 +1152,8 @@ static int mtk_aes_record_init(struct mtk_cryp *cryp)
}
/* Link to ring0 and ring1 respectively */
- aes[0]->id = RING0;
- aes[1]->id = RING1;
+ aes[0]->id = MTK_RING0;
+ aes[1]->id = MTK_RING1;
return 0;
@@ -1221,14 +1221,14 @@ int mtk_cipher_alg_register(struct mtk_cryp *cryp)
if (ret)
goto err_record;
- ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_irq,
+ ret = devm_request_irq(cryp->dev, cryp->irq[MTK_RING0], mtk_aes_irq,
0, "mtk-aes", cryp->aes[0]);
if (ret) {
dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
- ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_irq,
+ ret = devm_request_irq(cryp->dev, cryp->irq[MTK_RING1], mtk_aes_irq,
0, "mtk-aes", cryp->aes[1]);
if (ret) {
dev_err(cryp->dev, "unable to request AES irq.\n");
@@ -1236,8 +1236,8 @@ int mtk_cipher_alg_register(struct mtk_cryp *cryp)
}
/* Enable ring0 and ring1 interrupt */
- mtk_aes_write(cryp, AIC_ENABLE_SET(RING0), MTK_IRQ_RDR0);
- mtk_aes_write(cryp, AIC_ENABLE_SET(RING1), MTK_IRQ_RDR1);
+ mtk_aes_write(cryp, AIC_ENABLE_SET(MTK_RING0), MTK_IRQ_RDR0);
+ mtk_aes_write(cryp, AIC_ENABLE_SET(MTK_RING1), MTK_IRQ_RDR1);
spin_lock(&mtk_aes.lock);
list_add_tail(&cryp->aes_list, &mtk_aes.dev_list);
diff --git a/drivers/crypto/mediatek/mtk-platform.c b/drivers/crypto/mediatek/mtk-platform.c
index a9c713d..50de335 100644
--- a/drivers/crypto/mediatek/mtk-platform.c
+++ b/drivers/crypto/mediatek/mtk-platform.c
@@ -334,7 +334,7 @@ static int mtk_packet_engine_setup(struct mtk_cryp *cryp)
/* Enable the 4 rings for the packet engines. */
mtk_desc_ring_link(cryp, 0xf);
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
mtk_cmd_desc_ring_setup(cryp, i, &cap);
mtk_res_desc_ring_setup(cryp, i, &cap);
}
@@ -359,7 +359,7 @@ static int mtk_aic_cap_check(struct mtk_cryp *cryp, int hw)
{
u32 val;
- if (hw == RING_MAX)
+ if (hw == MTK_RING_MAX)
val = readl(cryp->base + AIC_G_VERSION);
else
val = readl(cryp->base + AIC_VERSION(hw));
@@ -368,7 +368,7 @@ static int mtk_aic_cap_check(struct mtk_cryp *cryp, int hw)
if (val != MTK_AIC_VER11 && val != MTK_AIC_VER12)
return -ENXIO;
- if (hw == RING_MAX)
+ if (hw == MTK_RING_MAX)
val = readl(cryp->base + AIC_G_OPTIONS);
else
val = readl(cryp->base + AIC_OPTIONS(hw));
@@ -389,7 +389,7 @@ static int mtk_aic_init(struct mtk_cryp *cryp, int hw)
return err;
/* Disable all interrupts and set initial configuration */
- if (hw == RING_MAX) {
+ if (hw == MTK_RING_MAX) {
writel(0, cryp->base + AIC_G_ENABLE_CTRL);
writel(0, cryp->base + AIC_G_POL_CTRL);
writel(0, cryp->base + AIC_G_TYPE_CTRL);
@@ -431,7 +431,7 @@ static void mtk_desc_dma_free(struct mtk_cryp *cryp)
{
int i;
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
dma_free_coherent(cryp->dev, MTK_DESC_RING_SZ,
cryp->ring[i]->res_base,
cryp->ring[i]->res_dma);
@@ -447,7 +447,7 @@ static int mtk_desc_ring_alloc(struct mtk_cryp *cryp)
struct mtk_ring **ring = cryp->ring;
int i, err = ENOMEM;
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
ring[i] = kzalloc(sizeof(**ring), GFP_KERNEL);
if (!ring[i])
goto err_cleanup;
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index d2a1db6..3bbe9b0 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -38,14 +38,14 @@
* Ring 2/3 are used by SHA.
*/
enum {
- RING0 = 0,
- RING1,
- RING2,
- RING3,
- RING_MAX,
+ MTK_RING0,
+ MTK_RING1,
+ MTK_RING2,
+ MTK_RING3,
+ MTK_RING_MAX
};
-#define MTK_REC_NUM (RING_MAX / 2)
+#define MTK_REC_NUM (MTK_RING_MAX / 2)
#define MTK_IRQ_NUM 5
/**
@@ -137,7 +137,7 @@ struct mtk_aes_dma {
* @resume: pointer to resume function
* @total: request buffer length
* @buf: pointer to page buffer
- * @id: record identification
+ * @id: the current use of ring
* @flags: it's describing AES operation state
* @lock: the async queue lock
*
@@ -172,9 +172,9 @@ struct mtk_aes_rec {
* @queue: crypto request queue
* @req: pointer to ahash request
* @task: the tasklet is use in SHA interrupt
- * @id: record identification
+ * @id: the current use of ring
* @flags: it's describing SHA operation state
- * @lock: the ablkcipher queue lock
+ * @lock: the async queue lock
*
* Structure used to record SHA execution state.
*/
@@ -197,9 +197,9 @@ struct mtk_sha_rec {
* @clk_ethif: pointer to ethif clock
* @clk_cryp: pointer to crypto clock
* @irq: global system and rings IRQ
- * @ring: pointer to execution state of AES
- * @aes: pointer to execution state of SHA
- * @sha: each execution record map to a ring
+ * @ring: pointer to descriptor rings
+ * @aes: pointer to operation record of AES
+ * @sha: pointer to operation record of SHA
* @aes_list: device list of AES
* @sha_list: device list of SHA
* @tmp: pointer to temporary buffer for internal use
@@ -215,7 +215,7 @@ struct mtk_cryp {
struct clk *clk_cryp;
int irq[MTK_IRQ_NUM];
- struct mtk_ring *ring[RING_MAX];
+ struct mtk_ring *ring[MTK_RING_MAX];
struct mtk_aes_rec *aes[MTK_REC_NUM];
struct mtk_sha_rec *sha[MTK_REC_NUM];
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 1fb98f6..b4f0c62 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -694,7 +694,7 @@ static void mtk_sha_finish_req(struct mtk_cryp *cryp,
sha->req->base.complete(&sha->req->base, err);
/* Handle new request */
- mtk_sha_handle_queue(cryp, sha->id - RING2, NULL);
+ mtk_sha_handle_queue(cryp, sha->id - MTK_RING2, NULL);
}
static int mtk_sha_handle_queue(struct mtk_cryp *cryp, u8 id,
@@ -1269,8 +1269,8 @@ static int mtk_sha_record_init(struct mtk_cryp *cryp)
}
/* Link to ring2 and ring3 respectively */
- sha[0]->id = RING2;
- sha[1]->id = RING3;
+ sha[0]->id = MTK_RING2;
+ sha[1]->id = MTK_RING3;
cryp->rec = 1;
@@ -1343,14 +1343,14 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
if (err)
goto err_record;
- err = devm_request_irq(cryp->dev, cryp->irq[RING2], mtk_sha_irq,
+ err = devm_request_irq(cryp->dev, cryp->irq[MTK_RING2], mtk_sha_irq,
0, "mtk-sha", cryp->sha[0]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq0.\n");
goto err_res;
}
- err = devm_request_irq(cryp->dev, cryp->irq[RING3], mtk_sha_irq,
+ err = devm_request_irq(cryp->dev, cryp->irq[MTK_RING3], mtk_sha_irq,
0, "mtk-sha", cryp->sha[1]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq1.\n");
@@ -1358,8 +1358,8 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
}
/* Enable ring2 and ring3 interrupt for hash */
- mtk_sha_write(cryp, AIC_ENABLE_SET(RING2), MTK_IRQ_RDR2);
- mtk_sha_write(cryp, AIC_ENABLE_SET(RING3), MTK_IRQ_RDR3);
+ mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING2), MTK_IRQ_RDR2);
+ mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING3), MTK_IRQ_RDR3);
cryp->tmp = dma_alloc_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
&cryp->tmp_dma, GFP_KERNEL);
--
1.9.1
^ permalink raw reply related
* [PATCH v1 3/8] crypto: mediatek - make mtk_sha_xmit() more generic
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This is a transitional patch. It merges mtk_sha_xmit() and mtk_sha_xmit2()
to make transmit function more generic.
In addition, res->buf and cryp->tmp_dma in mtk_sha_xmit() are useless, since
crypto engine writes the result digests into ctx->tfm.digest instead of
res->buf. It's better to remove it.
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-platform.h | 4 --
drivers/crypto/mediatek/mtk-sha.c | 116 ++++++++++-----------------------
2 files changed, 34 insertions(+), 86 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index 3bbe9b0..78ce54e 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -202,8 +202,6 @@ struct mtk_sha_rec {
* @sha: pointer to operation record of SHA
* @aes_list: device list of AES
* @sha_list: device list of SHA
- * @tmp: pointer to temporary buffer for internal use
- * @tmp_dma: DMA address of temporary buffer
* @rec: it's used to select SHA record for tfm
*
* Structure storing cryptographic device information.
@@ -222,8 +220,6 @@ struct mtk_cryp {
struct list_head aes_list;
struct list_head sha_list;
- void *tmp;
- dma_addr_t tmp_dma;
bool rec;
};
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index b4f0c62..ef6fb20 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -17,7 +17,6 @@
#define SHA_ALIGN_MSK (sizeof(u32) - 1)
#define SHA_QUEUE_SIZE 512
-#define SHA_TMP_BUF_SIZE 512
#define SHA_BUF_SIZE ((u32)PAGE_SIZE)
#define SHA_OP_UPDATE 1
@@ -319,7 +318,7 @@ static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
*/
static int mtk_sha_info_update(struct mtk_cryp *cryp,
struct mtk_sha_rec *sha,
- size_t len)
+ size_t len1, size_t len2)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_sha_info *info = &ctx->info;
@@ -331,11 +330,11 @@ static int mtk_sha_info_update(struct mtk_cryp *cryp,
ct->ctrl[0] &= ~SHA_TFM_START;
ctx->ct_hdr &= ~SHA_DATA_LEN_MSK;
- ctx->ct_hdr |= cpu_to_le32(len);
+ ctx->ct_hdr |= cpu_to_le32(len1 + len2);
ct->cmd[0] &= ~SHA_DATA_LEN_MSK;
- ct->cmd[0] |= cpu_to_le32(len);
+ ct->cmd[0] |= cpu_to_le32(len1 + len2);
- ctx->digcnt += len;
+ ctx->digcnt += len1;
ctx->ct_dma = dma_map_single(cryp->dev, info, sizeof(*info),
DMA_BIDIRECTIONAL);
@@ -422,67 +421,24 @@ static int mtk_sha_init(struct ahash_request *req)
}
static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
- dma_addr_t addr, size_t len)
+ dma_addr_t addr1, size_t len1,
+ dma_addr_t addr2, size_t len2)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_ring *ring = cryp->ring[sha->id];
struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
struct mtk_desc *res = ring->res_base + ring->res_pos;
- int err;
-
- err = mtk_sha_info_update(cryp, sha, len);
- if (err)
- return err;
-
- /* Fill in the command/result descriptors */
- res->hdr = MTK_DESC_FIRST | MTK_DESC_LAST | MTK_DESC_BUF_LEN(len);
- res->buf = cpu_to_le32(cryp->tmp_dma);
-
- cmd->hdr = MTK_DESC_FIRST | MTK_DESC_LAST | MTK_DESC_BUF_LEN(len) |
- MTK_DESC_CT_LEN(ctx->ct_size);
-
- cmd->buf = cpu_to_le32(addr);
- cmd->ct = cpu_to_le32(ctx->ct_dma);
- cmd->ct_hdr = ctx->ct_hdr;
- cmd->tfm = cpu_to_le32(ctx->tfm_dma);
-
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
- /*
- * Make sure that all changes to the DMA ring are done before we
- * start engine.
- */
- wmb();
- /* Start DMA transfer */
- mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(1));
- mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(1));
-
- return -EINPROGRESS;
-}
-
-static int mtk_sha_xmit2(struct mtk_cryp *cryp,
- struct mtk_sha_rec *sha,
- struct mtk_sha_reqctx *ctx,
- size_t len1, size_t len2)
-{
- struct mtk_ring *ring = cryp->ring[sha->id];
- struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
- struct mtk_desc *res = ring->res_base + ring->res_pos;
- int err;
+ int err, count = 0;
- err = mtk_sha_info_update(cryp, sha, len1 + len2);
+ err = mtk_sha_info_update(cryp, sha, len1, len2);
if (err)
return err;
/* Fill in the command/result descriptors */
- res->hdr = MTK_DESC_BUF_LEN(len1) | MTK_DESC_FIRST;
- res->buf = cpu_to_le32(cryp->tmp_dma);
-
- cmd->hdr = MTK_DESC_BUF_LEN(len1) | MTK_DESC_FIRST |
+ res->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1);
+ cmd->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1) |
MTK_DESC_CT_LEN(ctx->ct_size);
- cmd->buf = cpu_to_le32(sg_dma_address(ctx->sg));
+ cmd->buf = cpu_to_le32(addr1);
cmd->ct = cpu_to_le32(ctx->ct_dma);
cmd->ct_hdr = ctx->ct_hdr;
cmd->tfm = cpu_to_le32(ctx->tfm_dma);
@@ -491,20 +447,25 @@ static int mtk_sha_xmit2(struct mtk_cryp *cryp,
ring->cmd_pos = 0;
ring->res_pos = ring->cmd_pos;
+ count++;
- cmd = ring->cmd_base + ring->cmd_pos;
- res = ring->res_base + ring->res_pos;
+ if (len2) {
+ cmd = ring->cmd_base + ring->cmd_pos;
+ res = ring->res_base + ring->res_pos;
- res->hdr = MTK_DESC_BUF_LEN(len2) | MTK_DESC_LAST;
- res->buf = cpu_to_le32(cryp->tmp_dma);
+ res->hdr = MTK_DESC_BUF_LEN(len2);
+ cmd->hdr = MTK_DESC_BUF_LEN(len2);
+ cmd->buf = cpu_to_le32(addr2);
- cmd->hdr = MTK_DESC_BUF_LEN(len2) | MTK_DESC_LAST;
- cmd->buf = cpu_to_le32(ctx->dma_addr);
+ if (++ring->cmd_pos == MTK_DESC_NUM)
+ ring->cmd_pos = 0;
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
+ ring->res_pos = ring->cmd_pos;
+ count++;
+ }
- ring->res_pos = ring->cmd_pos;
+ cmd->hdr |= MTK_DESC_LAST;
+ res->hdr |= MTK_DESC_LAST;
/*
* Make sure that all changes to the DMA ring are done before we
@@ -512,8 +473,8 @@ static int mtk_sha_xmit2(struct mtk_cryp *cryp,
*/
wmb();
/* Start DMA transfer */
- mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(2));
- mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(2));
+ mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(count));
+ mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(count));
return -EINPROGRESS;
}
@@ -532,7 +493,7 @@ static int mtk_sha_dma_map(struct mtk_cryp *cryp,
ctx->flags &= ~SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count);
+ return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count, 0, 0);
}
static int mtk_sha_update_slow(struct mtk_cryp *cryp,
@@ -625,7 +586,8 @@ static int mtk_sha_update_start(struct mtk_cryp *cryp,
if (len == 0) {
ctx->flags &= ~SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count);
+ return mtk_sha_xmit(cryp, sha, ctx->dma_addr,
+ count, 0, 0);
} else {
ctx->sg = sg;
@@ -635,7 +597,8 @@ static int mtk_sha_update_start(struct mtk_cryp *cryp,
}
ctx->flags |= SHA_FLAGS_SG;
- return mtk_sha_xmit2(cryp, sha, ctx, len, count);
+ return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg),
+ len, ctx->dma_addr, count);
}
}
@@ -646,7 +609,8 @@ static int mtk_sha_update_start(struct mtk_cryp *cryp,
ctx->flags |= SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg), len);
+ return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg),
+ len, 0, 0);
}
static int mtk_sha_final_req(struct mtk_cryp *cryp,
@@ -1361,14 +1325,6 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING2), MTK_IRQ_RDR2);
mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING3), MTK_IRQ_RDR3);
- cryp->tmp = dma_alloc_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- &cryp->tmp_dma, GFP_KERNEL);
- if (!cryp->tmp) {
- dev_err(cryp->dev, "unable to allocate tmp buffer.\n");
- err = -EINVAL;
- goto err_res;
- }
-
spin_lock(&mtk_sha.lock);
list_add_tail(&cryp->sha_list, &mtk_sha.dev_list);
spin_unlock(&mtk_sha.lock);
@@ -1383,8 +1339,6 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
spin_lock(&mtk_sha.lock);
list_del(&cryp->sha_list);
spin_unlock(&mtk_sha.lock);
- dma_free_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- cryp->tmp, cryp->tmp_dma);
err_res:
mtk_sha_record_free(cryp);
err_record:
@@ -1400,7 +1354,5 @@ void mtk_hash_alg_release(struct mtk_cryp *cryp)
spin_unlock(&mtk_sha.lock);
mtk_sha_unregister_algs();
- dma_free_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- cryp->tmp, cryp->tmp_dma);
mtk_sha_record_free(cryp);
}
--
1.9.1
^ permalink raw reply related
* [PATCH v1 4/8] crypto: mediatek - simplify descriptor ring management
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This patch replaces cmd_pos/res_pos with pointer cmd_next/res_next.
In old code, we must to add one to shift ring to the next segment, and
then use this value to caculate current offset from ring base for each
DMA operation. Now these pointers helps us to simplify flow, so we just
need to move pointers and check the boundaries of ring.
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 14 ++++++++------
drivers/crypto/mediatek/mtk-platform.c | 3 +++
drivers/crypto/mediatek/mtk-platform.h | 8 ++++----
drivers/crypto/mediatek/mtk-sha.c | 35 ++++++++++++++++++----------------
4 files changed, 34 insertions(+), 26 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index b57b68f..21f3e59 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -262,7 +262,7 @@ static int mtk_aes_xmit(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
/* Write command descriptors */
for (nents = 0; nents < slen; ++nents, ssg = sg_next(ssg)) {
- cmd = ring->cmd_base + ring->cmd_pos;
+ cmd = ring->cmd_next;
cmd->hdr = MTK_DESC_BUF_LEN(ssg->length);
cmd->buf = cpu_to_le32(sg_dma_address(ssg));
@@ -274,22 +274,24 @@ static int mtk_aes_xmit(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
cmd->tfm = cpu_to_le32(aes->ctx->tfm_dma);
}
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
+ /* Shift ring buffer and check boundary */
+ if (++ring->cmd_next == ring->cmd_base + MTK_DESC_NUM)
+ ring->cmd_next = ring->cmd_base;
}
cmd->hdr |= MTK_DESC_LAST;
/* Prepare result descriptors */
for (nents = 0; nents < dlen; ++nents, dsg = sg_next(dsg)) {
- res = ring->res_base + ring->res_pos;
+ res = ring->res_next;
res->hdr = MTK_DESC_BUF_LEN(dsg->length);
res->buf = cpu_to_le32(sg_dma_address(dsg));
if (nents == 0)
res->hdr |= MTK_DESC_FIRST;
- if (++ring->res_pos == MTK_DESC_NUM)
- ring->res_pos = 0;
+ /* Shift ring buffer and check boundary */
+ if (++ring->res_next == ring->res_base + MTK_DESC_NUM)
+ ring->res_next = ring->res_base;
}
res->hdr |= MTK_DESC_LAST;
diff --git a/drivers/crypto/mediatek/mtk-platform.c b/drivers/crypto/mediatek/mtk-platform.c
index 50de335..b6ecc28 100644
--- a/drivers/crypto/mediatek/mtk-platform.c
+++ b/drivers/crypto/mediatek/mtk-platform.c
@@ -465,6 +465,9 @@ static int mtk_desc_ring_alloc(struct mtk_cryp *cryp)
GFP_KERNEL);
if (!ring[i]->res_base)
goto err_cleanup;
+
+ ring[i]->cmd_next = ring[i]->cmd_base;
+ ring[i]->res_next = ring[i]->res_base;
}
return 0;
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index 78ce54e..218e30d 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -84,11 +84,11 @@ struct mtk_desc {
/**
* struct mtk_ring - Descriptor ring
* @cmd_base: pointer to command descriptor ring base
+ * @cmd_next: pointer to the next command descriptor
* @cmd_dma: DMA address of command descriptor ring
- * @cmd_pos: current position in the command descriptor ring
* @res_base: pointer to result descriptor ring base
+ * @res_next: pointer to the next result descriptor
* @res_dma: DMA address of result descriptor ring
- * @res_pos: current position in the result descriptor ring
*
* A descriptor ring is a circular buffer that is used to manage
* one or more descriptors. There are two type of descriptor rings;
@@ -96,11 +96,11 @@ struct mtk_desc {
*/
struct mtk_ring {
struct mtk_desc *cmd_base;
+ struct mtk_desc *cmd_next;
dma_addr_t cmd_dma;
- u32 cmd_pos;
struct mtk_desc *res_base;
+ struct mtk_desc *res_next;
dma_addr_t res_dma;
- u32 res_pos;
};
/**
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index ef6fb20..0884d62 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -152,6 +152,21 @@ static inline void mtk_sha_write(struct mtk_cryp *cryp,
writel_relaxed(value, cryp->base + offset);
}
+static inline void mtk_sha_ring_shift(struct mtk_ring *ring,
+ struct mtk_desc **cmd_curr,
+ struct mtk_desc **res_curr,
+ int *count)
+{
+ *cmd_curr = ring->cmd_next++;
+ *res_curr = ring->res_next++;
+ (*count)++;
+
+ if (ring->cmd_next == ring->cmd_base + MTK_DESC_NUM) {
+ ring->cmd_next = ring->cmd_base;
+ ring->res_next = ring->res_base;
+ }
+}
+
static struct mtk_cryp *mtk_sha_find_dev(struct mtk_sha_ctx *tctx)
{
struct mtk_cryp *cryp = NULL;
@@ -426,8 +441,7 @@ static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_ring *ring = cryp->ring[sha->id];
- struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
- struct mtk_desc *res = ring->res_base + ring->res_pos;
+ struct mtk_desc *cmd, *res;
int err, count = 0;
err = mtk_sha_info_update(cryp, sha, len1, len2);
@@ -435,6 +449,8 @@ static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
return err;
/* Fill in the command/result descriptors */
+ mtk_sha_ring_shift(ring, &cmd, &res, &count);
+
res->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1);
cmd->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1) |
MTK_DESC_CT_LEN(ctx->ct_size);
@@ -443,25 +459,12 @@ static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
cmd->ct_hdr = ctx->ct_hdr;
cmd->tfm = cpu_to_le32(ctx->tfm_dma);
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
- count++;
-
if (len2) {
- cmd = ring->cmd_base + ring->cmd_pos;
- res = ring->res_base + ring->res_pos;
+ mtk_sha_ring_shift(ring, &cmd, &res, &count);
res->hdr = MTK_DESC_BUF_LEN(len2);
cmd->hdr = MTK_DESC_BUF_LEN(len2);
cmd->buf = cpu_to_le32(addr2);
-
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
- count++;
}
cmd->hdr |= MTK_DESC_LAST;
--
1.9.1
^ permalink raw reply related
* [PATCH v1 5/8] crypto: mediatek - add queue_task tasklet
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This patch adds 'queue_task' to dequeue crypto requset. This will help to
avoid directly calling mtk_aes_handle_queue() / mtk_sha_handle_queue()
from done tasklet or error handler.
In order to avoid confusion, the new code properly renames DMA completion
"task" to "done_task".
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 20 ++++++++++++++++----
drivers/crypto/mediatek/mtk-platform.h | 12 ++++++++----
drivers/crypto/mediatek/mtk-sha.c | 19 +++++++++++++++----
3 files changed, 39 insertions(+), 12 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 21f3e59..140e9a3 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -533,7 +533,8 @@ static int mtk_aes_complete(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
aes->areq->complete(aes->areq, 0);
/* Handle new request */
- return mtk_aes_handle_queue(cryp, aes->id, NULL);
+ tasklet_schedule(&aes->queue_task);
+ return 0;
}
static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
@@ -1094,6 +1095,13 @@ static void mtk_aes_gcm_exit(struct crypto_aead *aead)
},
};
+static void mtk_aes_queue_task(unsigned long data)
+{
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
+
+ mtk_aes_handle_queue(aes->cryp, aes->id, NULL);
+}
+
static void mtk_aes_done_task(unsigned long data)
{
struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
@@ -1116,7 +1124,7 @@ static irqreturn_t mtk_aes_irq(int irq, void *dev_id)
mtk_aes_write(cryp, RDR_THRESH(aes->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
- tasklet_schedule(&aes->task);
+ tasklet_schedule(&aes->done_task);
} else {
dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
}
@@ -1149,7 +1157,9 @@ static int mtk_aes_record_init(struct mtk_cryp *cryp)
spin_lock_init(&aes[i]->lock);
crypto_init_queue(&aes[i]->queue, AES_QUEUE_SIZE);
- tasklet_init(&aes[i]->task, mtk_aes_done_task,
+ tasklet_init(&aes[i]->queue_task, mtk_aes_queue_task,
+ (unsigned long)aes[i]);
+ tasklet_init(&aes[i]->done_task, mtk_aes_done_task,
(unsigned long)aes[i]);
}
@@ -1173,7 +1183,9 @@ static void mtk_aes_record_free(struct mtk_cryp *cryp)
int i;
for (i = 0; i < MTK_REC_NUM; i++) {
- tasklet_kill(&cryp->aes[i]->task);
+ tasklet_kill(&cryp->aes[i]->done_task);
+ tasklet_kill(&cryp->aes[i]->queue_task);
+
free_page((unsigned long)cryp->aes[i]->buf);
kfree(cryp->aes[i]);
}
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index 218e30d..cc98c2c 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -128,7 +128,8 @@ struct mtk_aes_dma {
* @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @areq: pointer to async request
- * @task: the tasklet is use in AES interrupt
+ * @done_task: the tasklet is use in AES interrupt
+ * @queue_task: the tasklet is used to dequeue request
* @ctx: pointer to current context
* @src: the structure that holds source sg list info
* @dst: the structure that holds destination sg list info
@@ -147,7 +148,8 @@ struct mtk_aes_rec {
struct mtk_cryp *cryp;
struct crypto_queue queue;
struct crypto_async_request *areq;
- struct tasklet_struct task;
+ struct tasklet_struct done_task;
+ struct tasklet_struct queue_task;
struct mtk_aes_base_ctx *ctx;
struct mtk_aes_dma src;
struct mtk_aes_dma dst;
@@ -171,7 +173,8 @@ struct mtk_aes_rec {
* @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @req: pointer to ahash request
- * @task: the tasklet is use in SHA interrupt
+ * @done_task: the tasklet is use in SHA interrupt
+ * @queue_task: the tasklet is used to dequeue request
* @id: the current use of ring
* @flags: it's describing SHA operation state
* @lock: the async queue lock
@@ -182,7 +185,8 @@ struct mtk_sha_rec {
struct mtk_cryp *cryp;
struct crypto_queue queue;
struct ahash_request *req;
- struct tasklet_struct task;
+ struct tasklet_struct done_task;
+ struct tasklet_struct queue_task;
u8 id;
unsigned long flags;
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 0884d62..dd3582b 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -661,7 +661,7 @@ static void mtk_sha_finish_req(struct mtk_cryp *cryp,
sha->req->base.complete(&sha->req->base, err);
/* Handle new request */
- mtk_sha_handle_queue(cryp, sha->id - MTK_RING2, NULL);
+ tasklet_schedule(&sha->queue_task);
}
static int mtk_sha_handle_queue(struct mtk_cryp *cryp, u8 id,
@@ -1183,6 +1183,13 @@ static void mtk_sha_cra_exit(struct crypto_tfm *tfm)
},
};
+static void mtk_sha_queue_task(unsigned long data)
+{
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
+
+ mtk_sha_handle_queue(sha->cryp, sha->id - MTK_RING2, NULL);
+}
+
static void mtk_sha_done_task(unsigned long data)
{
struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
@@ -1205,7 +1212,7 @@ static irqreturn_t mtk_sha_irq(int irq, void *dev_id)
mtk_sha_write(cryp, RDR_THRESH(sha->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
- tasklet_schedule(&sha->task);
+ tasklet_schedule(&sha->done_task);
} else {
dev_warn(cryp->dev, "SHA interrupt when no active requests.\n");
}
@@ -1231,7 +1238,9 @@ static int mtk_sha_record_init(struct mtk_cryp *cryp)
spin_lock_init(&sha[i]->lock);
crypto_init_queue(&sha[i]->queue, SHA_QUEUE_SIZE);
- tasklet_init(&sha[i]->task, mtk_sha_done_task,
+ tasklet_init(&sha[i]->queue_task, mtk_sha_queue_task,
+ (unsigned long)sha[i]);
+ tasklet_init(&sha[i]->done_task, mtk_sha_done_task,
(unsigned long)sha[i]);
}
@@ -1254,7 +1263,9 @@ static void mtk_sha_record_free(struct mtk_cryp *cryp)
int i;
for (i = 0; i < MTK_REC_NUM; i++) {
- tasklet_kill(&cryp->sha[i]->task);
+ tasklet_kill(&cryp->sha[i]->done_task);
+ tasklet_kill(&cryp->sha[i]->queue_task);
+
kfree(cryp->sha[i]);
}
}
--
1.9.1
^ permalink raw reply related
* [PATCH v1 6/8] crypto: mediatek - fix error handling in mtk_aes_complete()
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This patch fixes how errors should be handled by mtk_aes_complete().
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 39 +++++++++++++++++++++++----------------
1 file changed, 23 insertions(+), 16 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 140e9a3..6a0180d 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -248,6 +248,17 @@ static inline void mtk_aes_restore_sg(const struct mtk_aes_dma *dma)
sg->length += dma->remainder;
}
+static inline int mtk_aes_complete(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes,
+ int err)
+{
+ aes->flags &= ~AES_FLAGS_BUSY;
+ aes->areq->complete(aes->areq, err);
+ /* Handle new request */
+ tasklet_schedule(&aes->queue_task);
+ return err;
+}
+
/*
* Write descriptors for processing. This will configure the engine, load
* the transform information and then start the packet processing.
@@ -352,7 +363,7 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
ctx->ct_dma = dma_map_single(cryp->dev, &ctx->ct, sizeof(ctx->ct),
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(cryp->dev, ctx->ct_dma)))
- return -EINVAL;
+ goto exit;
ctx->tfm_dma = dma_map_single(cryp->dev, &ctx->tfm, sizeof(ctx->tfm),
DMA_TO_DEVICE);
@@ -389,8 +400,8 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
tfm_map_err:
dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
DMA_TO_DEVICE);
-
- return -EINVAL;
+exit:
+ return mtk_aes_complete(cryp, aes, -EINVAL);
}
/* Initialize transform information of CBC/ECB/CTR mode */
@@ -467,7 +478,7 @@ static int mtk_aes_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
padlen = mtk_aes_padlen(len);
if (len + padlen > AES_BUF_SIZE)
- return -ENOMEM;
+ return mtk_aes_complete(cryp, aes, -ENOMEM);
if (!src_aligned) {
sg_copy_to_buffer(src, sg_nents(src), aes->buf, len);
@@ -527,14 +538,10 @@ static int mtk_aes_handle_queue(struct mtk_cryp *cryp, u8 id,
return ctx->start(cryp, aes);
}
-static int mtk_aes_complete(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
+static int mtk_aes_transfer_complete(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes)
{
- aes->flags &= ~AES_FLAGS_BUSY;
- aes->areq->complete(aes->areq, 0);
-
- /* Handle new request */
- tasklet_schedule(&aes->queue_task);
- return 0;
+ return mtk_aes_complete(cryp, aes, 0);
}
static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
@@ -543,7 +550,7 @@ static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
struct mtk_aes_reqctx *rctx = ablkcipher_request_ctx(req);
mtk_aes_set_mode(aes, rctx);
- aes->resume = mtk_aes_complete;
+ aes->resume = mtk_aes_transfer_complete;
return mtk_aes_dma(cryp, aes, req->src, req->dst, req->nbytes);
}
@@ -568,7 +575,7 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
/* Check for transfer completion. */
cctx->offset += aes->total;
if (cctx->offset >= req->nbytes)
- return mtk_aes_complete(cryp, aes);
+ return mtk_aes_transfer_complete(cryp, aes);
/* Compute data length. */
datalen = req->nbytes - cctx->offset;
@@ -602,7 +609,6 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
cctx->iv[3] = cpu_to_be32(ctr);
crypto_inc((u8 *)cctx->iv, AES_BLOCK_SIZE);
}
- aes->resume = mtk_aes_ctr_transfer;
return mtk_aes_dma(cryp, aes, src, dst, datalen);
}
@@ -618,6 +624,7 @@ static int mtk_aes_ctr_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
memcpy(cctx->iv, req->info, AES_BLOCK_SIZE);
cctx->offset = 0;
aes->total = 0;
+ aes->resume = mtk_aes_ctr_transfer;
return mtk_aes_ctr_transfer(cryp, aes);
}
@@ -859,7 +866,7 @@ static int mtk_aes_gcm_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
if (!src_aligned || !dst_aligned) {
if (aes->total > AES_BUF_SIZE)
- return -ENOMEM;
+ return mtk_aes_complete(cryp, aes, -ENOMEM);
if (!src_aligned) {
sg_copy_to_buffer(src, sg_nents(src), aes->buf, len);
@@ -905,7 +912,7 @@ static int mtk_aes_gcm_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
aes->total = len;
gctx->textlen = req->cryptlen - gctx->authsize;
}
- aes->resume = mtk_aes_complete;
+ aes->resume = mtk_aes_transfer_complete;
return mtk_aes_gcm_dma(cryp, aes, req->src, req->dst, len);
}
--
1.9.1
^ permalink raw reply related
* [PATCH v1 7/8] crypto: mediatek - add mtk_aes_gcm_tag_verify()
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This patch adds mtk_aes_gcm_tag_verify() which is used to compare
authenticated tag.
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 24 ++++++++++++++++++++++--
drivers/crypto/mediatek/mtk-platform.h | 2 ++
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 6a0180d..8f3efa5 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -70,6 +70,8 @@
#define AES_FLAGS_ENCRYPT BIT(4)
#define AES_FLAGS_BUSY BIT(5)
+#define AES_AUTH_TAG_ERR cpu_to_le32(BIT(26))
+
/**
* Command token(CT) is a set of hardware instructions that
* are used to control engine's processing flow of AES.
@@ -306,6 +308,9 @@ static int mtk_aes_xmit(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
}
res->hdr |= MTK_DESC_LAST;
+ /* Pointer to current result descriptor */
+ ring->res_prev = res;
+
/* Prepare enough space for authenticated tag */
if (aes->flags & AES_FLAGS_GCM)
res->hdr += AES_BLOCK_SIZE;
@@ -799,6 +804,19 @@ static int mtk_aes_ctr_cra_init(struct crypto_tfm *tfm)
return container_of(ctx, struct mtk_aes_gcm_ctx, base);
}
+/*
+ * Engine will verify and compare tag automatically, so we just need
+ * to check returned status which stored in the result descriptor.
+ */
+static int mtk_aes_gcm_tag_verify(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes)
+{
+ u32 status = cryp->ring[aes->id]->res_prev->ct;
+
+ return mtk_aes_complete(cryp, aes, (status & AES_AUTH_TAG_ERR) ?
+ -EBADMSG : 0);
+}
+
/* Initialize transform information of GCM mode */
static void mtk_aes_gcm_info_init(struct mtk_cryp *cryp,
struct mtk_aes_rec *aes,
@@ -902,6 +920,8 @@ static int mtk_aes_gcm_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
if (aes->flags & AES_FLAGS_ENCRYPT) {
u32 tag[4];
+
+ aes->resume = mtk_aes_transfer_complete;
/* Compute total process length. */
aes->total = len + gctx->authsize;
/* Compute text length. */
@@ -909,10 +929,10 @@ static int mtk_aes_gcm_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
/* Hardware will append authenticated tag to output buffer */
scatterwalk_map_and_copy(tag, req->dst, len, gctx->authsize, 1);
} else {
+ aes->resume = mtk_aes_gcm_tag_verify;
aes->total = len;
gctx->textlen = req->cryptlen - gctx->authsize;
}
- aes->resume = mtk_aes_transfer_complete;
return mtk_aes_gcm_dma(cryp, aes, req->src, req->dst, len);
}
@@ -925,7 +945,7 @@ static int mtk_aes_gcm_crypt(struct aead_request *req, u64 mode)
rctx->mode = AES_FLAGS_GCM | mode;
return mtk_aes_handle_queue(ctx->cryp, !!(mode & AES_FLAGS_ENCRYPT),
- &req->base);
+ &req->base);
}
static void mtk_gcm_setkey_done(struct crypto_async_request *req, int err)
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index cc98c2c..303c152 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -88,6 +88,7 @@ struct mtk_desc {
* @cmd_dma: DMA address of command descriptor ring
* @res_base: pointer to result descriptor ring base
* @res_next: pointer to the next result descriptor
+ * @res_prev: pointer to the previous result descriptor
* @res_dma: DMA address of result descriptor ring
*
* A descriptor ring is a circular buffer that is used to manage
@@ -100,6 +101,7 @@ struct mtk_ring {
dma_addr_t cmd_dma;
struct mtk_desc *res_base;
struct mtk_desc *res_next;
+ struct mtk_desc *res_prev;
dma_addr_t res_dma;
};
--
1.9.1
^ permalink raw reply related
* [PATCH v1 8/8] crypto: mediatek - make hardware operation flow more efficient
From: Ryder Lee @ 2017-03-09 2:11 UTC (permalink / raw)
To: Herbert Xu
Cc: Ryder Lee, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-crypto-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1489025479-48036-1-git-send-email-ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
This patch refines data structures, which are used to control engine's
data path, to make it more efficient. Hence current change are:
- gathers the broken pieces of structures 'mtk_aes_ct''mtk_aes_tfm'
into struct mtk_aes_info hence avoiding additional DMA-mapping.
- adds 'keymode' in struct mtk_aes_base_ctx. When .setkey() callback is
called, we store keybit setting in keymode. Doing so, there is no need
to check keylen second time in mtk_aes_info_init() / mtk_aes_gcm_info_init().
Besides, this patch also removes unused macro definitions and adds helper
inline function to write security information(key, IV,...) to info->state.
Signed-off-by: Ryder Lee <ryder.lee-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
---
drivers/crypto/mediatek/mtk-aes.c | 263 ++++++++++++++++++--------------------
drivers/crypto/mediatek/mtk-sha.c | 90 ++++++-------
2 files changed, 165 insertions(+), 188 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 8f3efa5..9e845e8 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -19,13 +19,10 @@
#define AES_BUF_ORDER 2
#define AES_BUF_SIZE ((PAGE_SIZE << AES_BUF_ORDER) \
& ~(AES_BLOCK_SIZE - 1))
+#define AES_MAX_STATE_BUF_SIZE SIZE_IN_WORDS(AES_KEYSIZE_256 + \
+ AES_BLOCK_SIZE * 2)
+#define AES_MAX_CT_SIZE 6
-/* AES command token size */
-#define AES_CT_SIZE_ECB 2
-#define AES_CT_SIZE_CBC 3
-#define AES_CT_SIZE_CTR 3
-#define AES_CT_SIZE_GCM_OUT 5
-#define AES_CT_SIZE_GCM_IN 6
#define AES_CT_CTRL_HDR cpu_to_le32(0x00220000)
/* AES-CBC/ECB/CTR command token */
@@ -50,6 +47,8 @@
#define AES_TFM_128BITS cpu_to_le32(0xb << 16)
#define AES_TFM_192BITS cpu_to_le32(0xd << 16)
#define AES_TFM_256BITS cpu_to_le32(0xf << 16)
+#define AES_TFM_GHASH_DIGEST cpu_to_le32(0x2 << 21)
+#define AES_TFM_GHASH cpu_to_le32(0x4 << 23)
/* AES transform information word 1 fields */
#define AES_TFM_ECB cpu_to_le32(0x0 << 0)
#define AES_TFM_CBC cpu_to_le32(0x1 << 0)
@@ -59,10 +58,9 @@
#define AES_TFM_FULL_IV cpu_to_le32(0xf << 5) /* using IV 0-3 */
#define AES_TFM_IV_CTR_MODE cpu_to_le32(0x1 << 10)
#define AES_TFM_ENC_HASH cpu_to_le32(0x1 << 17)
-#define AES_TFM_GHASH_DIG cpu_to_le32(0x2 << 21)
-#define AES_TFM_GHASH cpu_to_le32(0x4 << 23)
/* AES flags */
+#define AES_FLAGS_CIPHER_MSK GENMASK(2, 0)
#define AES_FLAGS_ECB BIT(0)
#define AES_FLAGS_CBC BIT(1)
#define AES_FLAGS_CTR BIT(2)
@@ -73,18 +71,12 @@
#define AES_AUTH_TAG_ERR cpu_to_le32(BIT(26))
/**
- * Command token(CT) is a set of hardware instructions that
- * are used to control engine's processing flow of AES.
- *
- * Transform information(TFM) is used to define AES state and
- * contains all keys and initial vectors.
- *
- * The engine requires CT and TFM to do:
- * - Commands decoding and control of the engine's data path.
- * - Coordinating hardware data fetch and store operations.
- * - Result token construction and output.
+ * mtk_aes_info - hardware information of AES
+ * @cmd: command token, hardware instruction
+ * @tfm: transform state of cipher algorithm.
+ * @state: contains keys and initial vectors.
*
- * Memory map of GCM's TFM:
+ * Memory layout of GCM buffer:
* /-----------\
* | AES KEY | 128/196/256 bits
* |-----------|
@@ -92,14 +84,16 @@
* |-----------|
* | IVs | 4 * 4 bytes
* \-----------/
+ *
+ * The engine requires all these info to do:
+ * - Commands decoding and control of the engine's data path.
+ * - Coordinating hardware data fetch and store operations.
+ * - Result token construction and output.
*/
-struct mtk_aes_ct {
- __le32 cmd[AES_CT_SIZE_GCM_IN];
-};
-
-struct mtk_aes_tfm {
- __le32 ctrl[2];
- __le32 state[SIZE_IN_WORDS(AES_KEYSIZE_256 + AES_BLOCK_SIZE * 2)];
+struct mtk_aes_info {
+ __le32 cmd[AES_MAX_CT_SIZE];
+ __le32 tfm[2];
+ __le32 state[AES_MAX_STATE_BUF_SIZE];
};
struct mtk_aes_reqctx {
@@ -109,11 +103,12 @@ struct mtk_aes_reqctx {
struct mtk_aes_base_ctx {
struct mtk_cryp *cryp;
u32 keylen;
+ __le32 keymode;
+
mtk_aes_fn start;
- struct mtk_aes_ct ct;
+ struct mtk_aes_info info;
dma_addr_t ct_dma;
- struct mtk_aes_tfm tfm;
dma_addr_t tfm_dma;
__le32 ct_hdr;
@@ -250,6 +245,22 @@ static inline void mtk_aes_restore_sg(const struct mtk_aes_dma *dma)
sg->length += dma->remainder;
}
+static inline void mtk_aes_write_state_le(__le32 *dst, const u32 *src, u32 size)
+{
+ int i;
+
+ for (i = 0; i < SIZE_IN_WORDS(size); i++)
+ dst[i] = cpu_to_le32(src[i]);
+}
+
+static inline void mtk_aes_write_state_be(__be32 *dst, const u32 *src, u32 size)
+{
+ int i;
+
+ for (i = 0; i < SIZE_IN_WORDS(size); i++)
+ dst[i] = cpu_to_be32(src[i]);
+}
+
static inline int mtk_aes_complete(struct mtk_cryp *cryp,
struct mtk_aes_rec *aes,
int err)
@@ -331,9 +342,7 @@ static void mtk_aes_unmap(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
{
struct mtk_aes_base_ctx *ctx = aes->ctx;
- dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
- DMA_TO_DEVICE);
- dma_unmap_single(cryp->dev, ctx->tfm_dma, sizeof(ctx->tfm),
+ dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->info),
DMA_TO_DEVICE);
if (aes->src.sg == aes->dst.sg) {
@@ -364,16 +373,14 @@ static void mtk_aes_unmap(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
{
struct mtk_aes_base_ctx *ctx = aes->ctx;
+ struct mtk_aes_info *info = &ctx->info;
- ctx->ct_dma = dma_map_single(cryp->dev, &ctx->ct, sizeof(ctx->ct),
+ ctx->ct_dma = dma_map_single(cryp->dev, info, sizeof(*info),
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(cryp->dev, ctx->ct_dma)))
goto exit;
- ctx->tfm_dma = dma_map_single(cryp->dev, &ctx->tfm, sizeof(ctx->tfm),
- DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(cryp->dev, ctx->tfm_dma)))
- goto tfm_map_err;
+ ctx->tfm_dma = ctx->ct_dma + sizeof(info->cmd);
if (aes->src.sg == aes->dst.sg) {
aes->src.sg_len = dma_map_sg(cryp->dev, aes->src.sg,
@@ -400,11 +407,7 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
return mtk_aes_xmit(cryp, aes);
sg_map_err:
- dma_unmap_single(cryp->dev, ctx->tfm_dma, sizeof(ctx->tfm),
- DMA_TO_DEVICE);
-tfm_map_err:
- dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
- DMA_TO_DEVICE);
+ dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(*info), DMA_TO_DEVICE);
exit:
return mtk_aes_complete(cryp, aes, -EINVAL);
}
@@ -415,50 +418,43 @@ static void mtk_aes_info_init(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
{
struct ablkcipher_request *req = ablkcipher_request_cast(aes->areq);
struct mtk_aes_base_ctx *ctx = aes->ctx;
+ struct mtk_aes_info *info = &ctx->info;
+ u32 cnt = 0;
ctx->ct_hdr = AES_CT_CTRL_HDR | cpu_to_le32(len);
- ctx->ct.cmd[0] = AES_CMD0 | cpu_to_le32(len);
- ctx->ct.cmd[1] = AES_CMD1;
+ info->cmd[cnt++] = AES_CMD0 | cpu_to_le32(len);
+ info->cmd[cnt++] = AES_CMD1;
+ info->tfm[0] = AES_TFM_SIZE(ctx->keylen) | ctx->keymode;
if (aes->flags & AES_FLAGS_ENCRYPT)
- ctx->tfm.ctrl[0] = AES_TFM_BASIC_OUT;
+ info->tfm[0] |= AES_TFM_BASIC_OUT;
else
- ctx->tfm.ctrl[0] = AES_TFM_BASIC_IN;
+ info->tfm[0] |= AES_TFM_BASIC_IN;
- if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_128))
- ctx->tfm.ctrl[0] |= AES_TFM_128BITS;
- else if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_256))
- ctx->tfm.ctrl[0] |= AES_TFM_256BITS;
- else
- ctx->tfm.ctrl[0] |= AES_TFM_192BITS;
-
- if (aes->flags & AES_FLAGS_CBC) {
- const u32 *iv = (const u32 *)req->info;
- u32 *iv_state = ctx->tfm.state + ctx->keylen;
- int i;
-
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE));
- ctx->tfm.ctrl[1] = AES_TFM_CBC | AES_TFM_FULL_IV;
-
- for (i = 0; i < SIZE_IN_WORDS(AES_BLOCK_SIZE); i++)
- iv_state[i] = cpu_to_le32(iv[i]);
-
- ctx->ct.cmd[2] = AES_CMD2;
- ctx->ct_size = AES_CT_SIZE_CBC;
- } else if (aes->flags & AES_FLAGS_ECB) {
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen);
- ctx->tfm.ctrl[1] = AES_TFM_ECB;
-
- ctx->ct_size = AES_CT_SIZE_ECB;
- } else if (aes->flags & AES_FLAGS_CTR) {
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE));
- ctx->tfm.ctrl[1] = AES_TFM_CTR_LOAD | AES_TFM_FULL_IV;
-
- ctx->ct.cmd[2] = AES_CMD2;
- ctx->ct_size = AES_CT_SIZE_CTR;
+ switch (aes->flags & AES_FLAGS_CIPHER_MSK) {
+ case AES_FLAGS_CBC:
+ info->tfm[1] = AES_TFM_CBC;
+ break;
+ case AES_FLAGS_ECB:
+ info->tfm[1] = AES_TFM_ECB;
+ goto ecb;
+ case AES_FLAGS_CTR:
+ info->tfm[1] = AES_TFM_CTR_LOAD;
+ goto ctr;
+
+ default:
+ /* Should not happen... */
+ return;
}
+
+ mtk_aes_write_state_le(info->state + ctx->keylen, req->info,
+ AES_BLOCK_SIZE);
+ctr:
+ info->tfm[0] += AES_TFM_SIZE(SIZE_IN_WORDS(AES_BLOCK_SIZE));
+ info->tfm[1] |= AES_TFM_FULL_IV;
+ info->cmd[cnt++] = AES_CMD2;
+ecb:
+ ctx->ct_size = cnt;
}
static int mtk_aes_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
@@ -572,8 +568,7 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
struct mtk_aes_ctr_ctx *cctx = mtk_aes_ctr_ctx_cast(ctx);
struct ablkcipher_request *req = ablkcipher_request_cast(aes->areq);
struct scatterlist *src, *dst;
- int i;
- u32 start, end, ctr, blocks, *iv_state;
+ u32 start, end, ctr, blocks;
size_t datalen;
bool fragmented = false;
@@ -602,9 +597,8 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
scatterwalk_ffwd(cctx->dst, req->dst, cctx->offset));
/* Write IVs into transform state buffer. */
- iv_state = ctx->tfm.state + ctx->keylen;
- for (i = 0; i < SIZE_IN_WORDS(AES_BLOCK_SIZE); i++)
- iv_state[i] = cpu_to_le32(cctx->iv[i]);
+ mtk_aes_write_state_le(ctx->info.state + ctx->keylen, cctx->iv,
+ AES_BLOCK_SIZE);
if (unlikely(fragmented)) {
/*
@@ -639,21 +633,25 @@ static int mtk_aes_setkey(struct crypto_ablkcipher *tfm,
const u8 *key, u32 keylen)
{
struct mtk_aes_base_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- const u32 *aes_key = (const u32 *)key;
- u32 *key_state = ctx->tfm.state;
- int i;
- if (keylen != AES_KEYSIZE_128 &&
- keylen != AES_KEYSIZE_192 &&
- keylen != AES_KEYSIZE_256) {
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ ctx->keymode = AES_TFM_128BITS;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->keymode = AES_TFM_192BITS;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->keymode = AES_TFM_256BITS;
+ break;
+
+ default:
crypto_ablkcipher_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
ctx->keylen = SIZE_IN_WORDS(keylen);
-
- for (i = 0; i < ctx->keylen; i++)
- key_state[i] = cpu_to_le32(aes_key[i]);
+ mtk_aes_write_state_le(ctx->info.state, (const u32 *)key, keylen);
return 0;
}
@@ -825,45 +823,35 @@ static void mtk_aes_gcm_info_init(struct mtk_cryp *cryp,
struct aead_request *req = aead_request_cast(aes->areq);
struct mtk_aes_base_ctx *ctx = aes->ctx;
struct mtk_aes_gcm_ctx *gctx = mtk_aes_gcm_ctx_cast(ctx);
- const u32 *iv = (const u32 *)req->iv;
- u32 *iv_state = ctx->tfm.state + ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE);
+ struct mtk_aes_info *info = &ctx->info;
u32 ivsize = crypto_aead_ivsize(crypto_aead_reqtfm(req));
- int i;
+ u32 cnt = 0;
ctx->ct_hdr = AES_CT_CTRL_HDR | len;
- ctx->ct.cmd[0] = AES_GCM_CMD0 | cpu_to_le32(req->assoclen);
- ctx->ct.cmd[1] = AES_GCM_CMD1 | cpu_to_le32(req->assoclen);
- ctx->ct.cmd[2] = AES_GCM_CMD2;
- ctx->ct.cmd[3] = AES_GCM_CMD3 | cpu_to_le32(gctx->textlen);
+ info->cmd[cnt++] = AES_GCM_CMD0 | cpu_to_le32(req->assoclen);
+ info->cmd[cnt++] = AES_GCM_CMD1 | cpu_to_le32(req->assoclen);
+ info->cmd[cnt++] = AES_GCM_CMD2;
+ info->cmd[cnt++] = AES_GCM_CMD3 | cpu_to_le32(gctx->textlen);
if (aes->flags & AES_FLAGS_ENCRYPT) {
- ctx->ct.cmd[4] = AES_GCM_CMD4 | cpu_to_le32(gctx->authsize);
- ctx->ct_size = AES_CT_SIZE_GCM_OUT;
- ctx->tfm.ctrl[0] = AES_TFM_GCM_OUT;
+ info->cmd[cnt++] = AES_GCM_CMD4 | cpu_to_le32(gctx->authsize);
+ info->tfm[0] = AES_TFM_GCM_OUT;
} else {
- ctx->ct.cmd[4] = AES_GCM_CMD5 | cpu_to_le32(gctx->authsize);
- ctx->ct.cmd[5] = AES_GCM_CMD6 | cpu_to_le32(gctx->authsize);
- ctx->ct_size = AES_CT_SIZE_GCM_IN;
- ctx->tfm.ctrl[0] = AES_TFM_GCM_IN;
+ info->cmd[cnt++] = AES_GCM_CMD5 | cpu_to_le32(gctx->authsize);
+ info->cmd[cnt++] = AES_GCM_CMD6 | cpu_to_le32(gctx->authsize);
+ info->tfm[0] = AES_TFM_GCM_IN;
}
+ ctx->ct_size = cnt;
- if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_128))
- ctx->tfm.ctrl[0] |= AES_TFM_128BITS;
- else if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_256))
- ctx->tfm.ctrl[0] |= AES_TFM_256BITS;
- else
- ctx->tfm.ctrl[0] |= AES_TFM_192BITS;
-
- ctx->tfm.ctrl[0] |= AES_TFM_GHASH_DIG | AES_TFM_GHASH |
- AES_TFM_SIZE(ctx->keylen + SIZE_IN_WORDS(
- AES_BLOCK_SIZE + ivsize));
- ctx->tfm.ctrl[1] = AES_TFM_CTR_INIT | AES_TFM_IV_CTR_MODE |
- AES_TFM_3IV | AES_TFM_ENC_HASH;
+ info->tfm[0] |= AES_TFM_GHASH_DIGEST | AES_TFM_GHASH | AES_TFM_SIZE(
+ ctx->keylen + SIZE_IN_WORDS(AES_BLOCK_SIZE + ivsize)) |
+ ctx->keymode;
+ info->tfm[1] = AES_TFM_CTR_INIT | AES_TFM_IV_CTR_MODE | AES_TFM_3IV |
+ AES_TFM_ENC_HASH;
- for (i = 0; i < SIZE_IN_WORDS(ivsize); i++)
- iv_state[i] = cpu_to_le32(iv[i]);
+ mtk_aes_write_state_le(info->state + ctx->keylen + SIZE_IN_WORDS(
+ AES_BLOCK_SIZE), (const u32 *)req->iv, ivsize);
}
static int mtk_aes_gcm_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
@@ -979,24 +967,26 @@ static int mtk_aes_gcm_setkey(struct crypto_aead *aead, const u8 *key,
struct scatterlist sg[1];
struct skcipher_request req;
} *data;
- const u32 *aes_key;
- u32 *key_state, *hash_state;
- int err, i;
+ int err;
- if (keylen != AES_KEYSIZE_256 &&
- keylen != AES_KEYSIZE_192 &&
- keylen != AES_KEYSIZE_128) {
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ ctx->keymode = AES_TFM_128BITS;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->keymode = AES_TFM_192BITS;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->keymode = AES_TFM_256BITS;
+ break;
+
+ default:
crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- key_state = ctx->tfm.state;
- aes_key = (u32 *)key;
ctx->keylen = SIZE_IN_WORDS(keylen);
- for (i = 0; i < ctx->keylen; i++)
- ctx->tfm.state[i] = cpu_to_le32(aes_key[i]);
-
/* Same as crypto_gcm_setkey() from crypto/gcm.c */
crypto_skcipher_clear_flags(ctr, CRYPTO_TFM_REQ_MASK);
crypto_skcipher_set_flags(ctr, crypto_aead_get_flags(aead) &
@@ -1031,10 +1021,11 @@ static int mtk_aes_gcm_setkey(struct crypto_aead *aead, const u8 *key,
if (err)
goto out;
- hash_state = key_state + ctx->keylen;
-
- for (i = 0; i < 4; i++)
- hash_state[i] = cpu_to_be32(data->hash[i]);
+ /* Write key into state buffer */
+ mtk_aes_write_state_le(ctx->info.state, (const u32 *)key, keylen);
+ /* Write key(H) into state buffer */
+ mtk_aes_write_state_be(ctx->info.state + ctx->keylen, data->hash,
+ AES_BLOCK_SIZE);
out:
kzfree(data);
return err;
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index dd3582b..2226f12 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -23,6 +23,7 @@
#define SHA_OP_FINAL 2
#define SHA_DATA_LEN_MSK cpu_to_le32(GENMASK(16, 0))
+#define SHA_MAX_DIGEST_BUF_SIZE 32
/* SHA command token */
#define SHA_CT_SIZE 5
@@ -33,7 +34,6 @@
/* SHA transform information */
#define SHA_TFM_HASH cpu_to_le32(0x2 << 0)
-#define SHA_TFM_INNER_DIG cpu_to_le32(0x1 << 21)
#define SHA_TFM_SIZE(x) cpu_to_le32((x) << 8)
#define SHA_TFM_START cpu_to_le32(0x1 << 4)
#define SHA_TFM_CONTINUE cpu_to_le32(0x1 << 5)
@@ -60,31 +60,17 @@
#define SHA_FLAGS_PAD BIT(10)
/**
- * mtk_sha_ct is a set of hardware instructions(command token)
- * that are used to control engine's processing flow of SHA,
- * and it contains the first two words of transform state.
+ * mtk_sha_info - hardware information of AES
+ * @cmd: command token, hardware instruction
+ * @tfm: transform state of cipher algorithm.
+ * @state: contains keys and initial vectors.
+ *
*/
-struct mtk_sha_ct {
+struct mtk_sha_info {
__le32 ctrl[2];
__le32 cmd[3];
-};
-
-/**
- * mtk_sha_tfm is used to define SHA transform state
- * and store result digest that produced by engine.
- */
-struct mtk_sha_tfm {
- __le32 ctrl[2];
- __le32 digest[SIZE_IN_WORDS(SHA512_DIGEST_SIZE)];
-};
-
-/**
- * mtk_sha_info consists of command token and transform state
- * of SHA, its role is similar to mtk_aes_info.
- */
-struct mtk_sha_info {
- struct mtk_sha_ct ct;
- struct mtk_sha_tfm tfm;
+ __le32 tfm[2];
+ __le32 digest[SHA_MAX_DIGEST_BUF_SIZE];
};
struct mtk_sha_reqctx {
@@ -93,7 +79,6 @@ struct mtk_sha_reqctx {
unsigned long op;
u64 digcnt;
- bool start;
size_t bufcnt;
dma_addr_t dma_addr;
@@ -265,7 +250,9 @@ static void mtk_sha_fill_padding(struct mtk_sha_reqctx *ctx, u32 len)
bits[1] = cpu_to_be64(size << 3);
bits[0] = cpu_to_be64(size >> 61);
- if (ctx->flags & (SHA_FLAGS_SHA384 | SHA_FLAGS_SHA512)) {
+ switch (ctx->flags & SHA_FLAGS_ALGO_MSK) {
+ case SHA_FLAGS_SHA384:
+ case SHA_FLAGS_SHA512:
index = ctx->bufcnt & 0x7f;
padlen = (index < 112) ? (112 - index) : ((128 + 112) - index);
*(ctx->buffer + ctx->bufcnt) = 0x80;
@@ -273,7 +260,9 @@ static void mtk_sha_fill_padding(struct mtk_sha_reqctx *ctx, u32 len)
memcpy(ctx->buffer + ctx->bufcnt + padlen, bits, 16);
ctx->bufcnt += padlen + 16;
ctx->flags |= SHA_FLAGS_PAD;
- } else {
+ break;
+
+ default:
index = ctx->bufcnt & 0x3f;
padlen = (index < 56) ? (56 - index) : ((64 + 56) - index);
*(ctx->buffer + ctx->bufcnt) = 0x80;
@@ -281,36 +270,35 @@ static void mtk_sha_fill_padding(struct mtk_sha_reqctx *ctx, u32 len)
memcpy(ctx->buffer + ctx->bufcnt + padlen, &bits[1], 8);
ctx->bufcnt += padlen + 8;
ctx->flags |= SHA_FLAGS_PAD;
+ break;
}
}
/* Initialize basic transform information of SHA */
static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
{
- struct mtk_sha_ct *ct = &ctx->info.ct;
- struct mtk_sha_tfm *tfm = &ctx->info.tfm;
+ struct mtk_sha_info *info = &ctx->info;
ctx->ct_hdr = SHA_CT_CTRL_HDR;
ctx->ct_size = SHA_CT_SIZE;
- tfm->ctrl[0] = SHA_TFM_HASH | SHA_TFM_INNER_DIG |
- SHA_TFM_SIZE(SIZE_IN_WORDS(ctx->ds));
+ info->tfm[0] = SHA_TFM_HASH | SHA_TFM_SIZE(SIZE_IN_WORDS(ctx->ds));
switch (ctx->flags & SHA_FLAGS_ALGO_MSK) {
case SHA_FLAGS_SHA1:
- tfm->ctrl[0] |= SHA_TFM_SHA1;
+ info->tfm[0] |= SHA_TFM_SHA1;
break;
case SHA_FLAGS_SHA224:
- tfm->ctrl[0] |= SHA_TFM_SHA224;
+ info->tfm[0] |= SHA_TFM_SHA224;
break;
case SHA_FLAGS_SHA256:
- tfm->ctrl[0] |= SHA_TFM_SHA256;
+ info->tfm[0] |= SHA_TFM_SHA256;
break;
case SHA_FLAGS_SHA384:
- tfm->ctrl[0] |= SHA_TFM_SHA384;
+ info->tfm[0] |= SHA_TFM_SHA384;
break;
case SHA_FLAGS_SHA512:
- tfm->ctrl[0] |= SHA_TFM_SHA512;
+ info->tfm[0] |= SHA_TFM_SHA512;
break;
default:
@@ -318,13 +306,13 @@ static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
return;
}
- tfm->ctrl[1] = SHA_TFM_HASH_STORE;
- ct->ctrl[0] = tfm->ctrl[0] | SHA_TFM_CONTINUE | SHA_TFM_START;
- ct->ctrl[1] = tfm->ctrl[1];
+ info->tfm[1] = SHA_TFM_HASH_STORE;
+ info->ctrl[0] = info->tfm[0] | SHA_TFM_CONTINUE | SHA_TFM_START;
+ info->ctrl[1] = info->tfm[1];
- ct->cmd[0] = SHA_CMD0;
- ct->cmd[1] = SHA_CMD1;
- ct->cmd[2] = SHA_CMD2 | SHA_TFM_DIGEST(SIZE_IN_WORDS(ctx->ds));
+ info->cmd[0] = SHA_CMD0;
+ info->cmd[1] = SHA_CMD1;
+ info->cmd[2] = SHA_CMD2 | SHA_TFM_DIGEST(SIZE_IN_WORDS(ctx->ds));
}
/*
@@ -337,17 +325,15 @@ static int mtk_sha_info_update(struct mtk_cryp *cryp,
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_sha_info *info = &ctx->info;
- struct mtk_sha_ct *ct = &info->ct;
-
- if (ctx->start)
- ctx->start = false;
- else
- ct->ctrl[0] &= ~SHA_TFM_START;
ctx->ct_hdr &= ~SHA_DATA_LEN_MSK;
ctx->ct_hdr |= cpu_to_le32(len1 + len2);
- ct->cmd[0] &= ~SHA_DATA_LEN_MSK;
- ct->cmd[0] |= cpu_to_le32(len1 + len2);
+ info->cmd[0] &= ~SHA_DATA_LEN_MSK;
+ info->cmd[0] |= cpu_to_le32(len1 + len2);
+
+ /* Setting SHA_TFM_START only for the first iteration */
+ if (ctx->digcnt)
+ info->ctrl[0] &= ~SHA_TFM_START;
ctx->digcnt += len1;
@@ -357,7 +343,8 @@ static int mtk_sha_info_update(struct mtk_cryp *cryp,
dev_err(cryp->dev, "dma %zu bytes error\n", sizeof(*info));
return -EINVAL;
}
- ctx->tfm_dma = ctx->ct_dma + sizeof(*ct);
+
+ ctx->tfm_dma = ctx->ct_dma + sizeof(info->ctrl) + sizeof(info->cmd);
return 0;
}
@@ -422,7 +409,6 @@ static int mtk_sha_init(struct ahash_request *req)
ctx->bufcnt = 0;
ctx->digcnt = 0;
ctx->buffer = tctx->buf;
- ctx->start = true;
if (tctx->flags & SHA_FLAGS_HMAC) {
struct mtk_sha_hmac_ctx *bctx = tctx->base;
@@ -635,7 +621,7 @@ static int mtk_sha_final_req(struct mtk_cryp *cryp,
static int mtk_sha_finish(struct ahash_request *req)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(req);
- u32 *digest = ctx->info.tfm.digest;
+ __le32 *digest = ctx->info.digest;
u32 *result = (u32 *)req->result;
int i;
--
1.9.1
^ permalink raw reply related
* Re: [PATCH] crypto: testmgr - don't DMA map IV from stack in test_skcipher()
From: Herbert Xu @ 2017-03-09 4:18 UTC (permalink / raw)
To: Horia Geantă; +Cc: David S. Miller, linux-crypto@vger.kernel.org
In-Reply-To: <DB4PR04MB08477C25A41104FA6D879D07982E0@DB4PR04MB0847.eurprd04.prod.outlook.com>
On Wed, Mar 08, 2017 at 01:04:14PM +0000, Horia Geantă wrote:
>
> Since changing the API and converting the users looks pretty lengthy,
> would it be acceptable to fix tcrypt for now?
> Indeed, I've missed updating test_skcipher_speed, I can add this in v2.
Might as well leave it there because at least it reminds us to
fix the problem. With the API as it is any new kernel user could
potentially provide us with an IV off the stack.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [PATCH] crypto: doc - fix typo (struct sdesc)
From: Fabien Dessenne @ 2017-03-09 9:40 UTC (permalink / raw)
To: linux-crypto
Add missing " " in api-samples.rst
Signed-off-by: Fabien Dessenne <fabien.dessenne@st.com>
---
Documentation/crypto/api-samples.rst | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/crypto/api-samples.rst b/Documentation/crypto/api-samples.rst
index 0a10819..d021fd9 100644
--- a/Documentation/crypto/api-samples.rst
+++ b/Documentation/crypto/api-samples.rst
@@ -155,9 +155,9 @@ Code Example For Use of Operational State Memory With SHASH
char ctx[];
};
- static struct sdescinit_sdesc(struct crypto_shash *alg)
+ static struct sdesc init_sdesc(struct crypto_shash *alg)
{
- struct sdescsdesc;
+ struct sdesc sdesc;
int size;
size = sizeof(struct shash_desc) + crypto_shash_descsize(alg);
@@ -172,7 +172,7 @@ Code Example For Use of Operational State Memory With SHASH
static int calc_hash(struct crypto_shashalg,
const unsigned chardata, unsigned int datalen,
unsigned chardigest) {
- struct sdescsdesc;
+ struct sdesc sdesc;
int ret;
sdesc = init_sdesc(alg);
--
2.7.4
^ permalink raw reply related
* Re: [RFC] how to handle AAD copy operation for algif_aead
From: Herbert Xu @ 2017-03-09 10:00 UTC (permalink / raw)
To: Stephan Müller; +Cc: linux-crypto
In-Reply-To: <13406914.zeok8basum@positron.chronox.de>
On Fri, Feb 24, 2017 at 02:24:47PM +0100, Stephan Müller wrote:
> Hi Herbert et al,
>
> attached are two patches where each patch has a different approach to copy the
> AAD in the algif_aead operation. I would like to hear your opinion which
> approach should be taken.
>
> The patch 0001-crypto-algif_aead-copy-AAD-from-src-to-dst_separate.patch
> simply copies the AAD over from TX SGL to RX SGL. The pro is that the patch is
> small. The con is that this approach does *not* provide an in-place crypto
> operation.
I prefer this patch with the proviso that it copy the whole thing
instead of just the AD. That way you can just feed the dst memory
to crypto_aead for in-place operation. Of course you have to mangle
the tag data onto the dst SG list for decryption but it shouldn't
be too hard.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC] how to handle AAD copy operation for algif_aead
From: Stephan Müller @ 2017-03-09 10:02 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto
In-Reply-To: <20170309100011.GA6844@gondor.apana.org.au>
Am Donnerstag, 9. März 2017, 11:00:11 CET schrieb Herbert Xu:
Hi Herbert,
> On Fri, Feb 24, 2017 at 02:24:47PM +0100, Stephan Müller wrote:
> > Hi Herbert et al,
> >
> > attached are two patches where each patch has a different approach to copy
> > the AAD in the algif_aead operation. I would like to hear your opinion
> > which approach should be taken.
> >
> > The patch 0001-crypto-algif_aead-copy-AAD-from-src-to-dst_separate.patch
> > simply copies the AAD over from TX SGL to RX SGL. The pro is that the
> > patch is small. The con is that this approach does *not* provide an
> > in-place crypto operation.
>
> I prefer this patch with the proviso that it copy the whole thing
> instead of just the AD. That way you can just feed the dst memory
> to crypto_aead for in-place operation. Of course you have to mangle
> the tag data onto the dst SG list for decryption but it shouldn't
> be too hard.
I thought that is exactly the second patch. It copies the entire data to the
dst SGL and extends the SGL with the tag in case of decryption.
Ciao
Stephan
^ permalink raw reply
* Re: [RFC] how to handle AAD copy operation for algif_aead
From: Herbert Xu @ 2017-03-09 10:05 UTC (permalink / raw)
To: Stephan Müller; +Cc: linux-crypto
In-Reply-To: <78373946.kaJU5cUr5f@tauon.atsec.com>
On Thu, Mar 09, 2017 at 11:02:41AM +0100, Stephan Müller wrote:
>
> > > The patch 0001-crypto-algif_aead-copy-AAD-from-src-to-dst_separate.patch
> > > simply copies the AAD over from TX SGL to RX SGL. The pro is that the
> > > patch is small. The con is that this approach does *not* provide an
> > > in-place crypto operation.
> >
> > I prefer this patch with the proviso that it copy the whole thing
> > instead of just the AD. That way you can just feed the dst memory
> > to crypto_aead for in-place operation. Of course you have to mangle
> > the tag data onto the dst SG list for decryption but it shouldn't
> > be too hard.
>
> I thought that is exactly the second patch. It copies the entire data to the
> dst SGL and extends the SGL with the tag in case of decryption.
Are you sure? The patch says:
+ /* copy AAD from src to dst */
+ err = crypto_aead_copy_sgl(ctx->null, areq->tsgl,
+ areq->first_rsgl.sgl.sg, ctx->aead_assoclen);
Which seems to only copy the AD.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC] how to handle AAD copy operation for algif_aead
From: Stephan Müller @ 2017-03-09 10:08 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto
In-Reply-To: <20170309100554.GA6945@gondor.apana.org.au>
Am Donnerstag, 9. März 2017, 11:05:54 CET schrieb Herbert Xu:
Hi Herbert,
> On Thu, Mar 09, 2017 at 11:02:41AM +0100, Stephan Müller wrote:
> > > > The patch
> > > > 0001-crypto-algif_aead-copy-AAD-from-src-to-dst_separate.patch
> > > > simply copies the AAD over from TX SGL to RX SGL. The pro is that the
> > > > patch is small. The con is that this approach does *not* provide an
> > > > in-place crypto operation.
> > >
> > > I prefer this patch with the proviso that it copy the whole thing
> > > instead of just the AD. That way you can just feed the dst memory
> > > to crypto_aead for in-place operation. Of course you have to mangle
> > > the tag data onto the dst SG list for decryption but it shouldn't
> > > be too hard.
> >
> > I thought that is exactly the second patch. It copies the entire data to
> > the dst SGL and extends the SGL with the tag in case of decryption.
>
> Are you sure? The patch says:
>
> + /* copy AAD from src to dst */
> + err = crypto_aead_copy_sgl(ctx->null, areq->tsgl,
> + areq->first_rsgl.sgl.sg,
> ctx->aead_assoclen);
>
> Which seems to only copy the AD.
This is the first patch (0001-crypto-algif_aead-copy-AAD-from-src-to-
dst_separate.patch).
The second alternative patch (0001-crypto-algif_aead-copy-AAD-from-src-to-
dst_inplace.patch) does:
+ if (ctx->enc) {
+ /* Copy AAD || PT to RX SGL buffer for in-place operation. */
+ err = crypto_aead_copy_sgl(ctx->null, tsgl->sg,
+ areq->first_rsgl.sgl.sg, processed);
+ if (err)
+ goto free;
+ aead_pull_tsgl(sk, processed, NULL, 0);
+ } else {
+ /* Copy AAD || CT to RX SGL buffer for in-place operation. */
+ err = crypto_aead_copy_sgl(ctx->null, tsgl->sg,
+ areq->first_rsgl.sgl.sg, outlen);
+ if (err)
+ goto free;
+
+ /* Create TX SGL for tag and chain it to RX SGL. */
+ areq->tsgl_entries = aead_count_tsgl(sk, processed);
+ if (!areq->tsgl_entries)
+ areq->tsgl_entries = 1;
+ areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) *
+ areq->tsgl_entries,
+ GFP_KERNEL);
+ if (!areq->tsgl) {
+ err = -ENOMEM;
+ goto free;
+ }
+ sg_init_table(areq->tsgl, areq->tsgl_entries);
+
+ /* Release TX SGL, except for tag data. */
+ aead_pull_tsgl(sk, processed, areq->tsgl, processed - as);
+
+ /* chain the areq TX SGL holding the tag with RX SGL */
+ if (!last_rsgl) {
+ /* no RX SGL present (e.g. only authentication) */
+ sg_init_table(areq->first_rsgl.sgl.sg, 2);
+ sg_chain(areq->first_rsgl.sgl.sg, 2, areq->tsgl);
+ } else {
+ /* RX SGL present */
+ struct af_alg_sgl *sgl_prev = &last_rsgl->sgl;
+
+ sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1);
+ sg_chain(sgl_prev->sg, sgl_prev->npages + 1, areq->tsgl);
+ }
}
Ciao
Stephan
^ permalink raw reply
* Re: [RFC] how to handle AAD copy operation for algif_aead
From: Herbert Xu @ 2017-03-09 10:23 UTC (permalink / raw)
To: Stephan Müller; +Cc: linux-crypto
In-Reply-To: <9194388.iDFJIB4SCo@tauon.atsec.com>
On Thu, Mar 09, 2017 at 11:08:32AM +0100, Stephan Müller wrote:
>
> This is the first patch (0001-crypto-algif_aead-copy-AAD-from-src-to-
> dst_separate.patch).
>
> The second alternative patch (0001-crypto-algif_aead-copy-AAD-from-src-to-
> dst_inplace.patch) does:
I see. Yes that looks good.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC] how to handle AAD copy operation for algif_aead
From: Stephan Müller @ 2017-03-09 10:27 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto
In-Reply-To: <20170309102316.GA7027@gondor.apana.org.au>
Am Donnerstag, 9. März 2017, 11:23:16 CET schrieb Herbert Xu:
Hi Herbert,
> I see. Yes that looks good.
Thank you. I will provide an official patch after the discussion about the
memory handling is completed.
As the memory handling patch should be considered for stable whereas the AAD
copy patch should not, I would therefore recommend completing the memory
handling discussion first.
Ciao
Stephan
^ permalink raw reply
* Re: [PATCH V2 2/3] crypto: ccp - Enable support for AES GCM on v5 CCPs
From: Herbert Xu @ 2017-03-09 10:32 UTC (permalink / raw)
To: Gary R Hook; +Cc: linux-crypto, thomas.lendacky, davem
In-Reply-To: <20170302212654.30959.31.stgit@taos>
On Thu, Mar 02, 2017 at 03:26:54PM -0600, Gary R Hook wrote:
> A version 5 device provides the primitive commands
> required for AES GCM. This patch adds support for
> en/decryption.
>
> Signed-off-by: Gary R Hook <gary.hook@amd.com>
This patch doesn't apply to the current cryptodev tree.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC PATCH v2 0/3] Cavium ThunderX ZIP driver
From: Herbert Xu @ 2017-03-09 10:46 UTC (permalink / raw)
To: Mahipal Challa
Cc: davem, linux-crypto, linux-kernel, Jan.Glauber, Vishnu.Nair,
pathreya, Mahipal.Challa
In-Reply-To: <1487135710-9692-1-git-send-email-Mahipal.Challa@cavium.com>
On Wed, Feb 15, 2017 at 10:45:07AM +0530, Mahipal Challa wrote:
> Hi Herbert,
>
> This series adds support for hardware accelerated compression & decompression
> as found on ThunderX (arm64) SOCs.
>
> As per your suggestion, we've switched to the new crypto acomp/scomp interface.
>
> To test the ZIP driver, we modified the kernel's ZSWAP to use acomp API's.
>
> Performance numbers from ZSWAP look promising.
> The "average time" for compressing a 4KB page:
>
> Compression Software : 278 usec
> Compression HW deflate : 17 usec
> Compression HW LZS : 11 usec
>
> Decompression Software : 20 usec
> Decompression HW deflate: 8 usec
> Decompression HW LZS : 6 usec
>
> Addressed the review comments from the RFC v1.
> - Added acomp/scomp crypto interface support.
> - Improved the error handling.
> - ZIP command completion codes are renamed suitably.
> - Removed some un-used code.
> - raw_smp_processor_id() replaced with smp_processor_id().
> - Some more miscellaneous changes.
>
> Patches are on top of "kernel/git/herbert/crypto-2.6.git" repository
>
> Please provide your feedback.
All applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: brcm: fix spelling mistake: "genereate" -> "generate"
From: Herbert Xu @ 2017-03-09 10:47 UTC (permalink / raw)
To: Colin King
Cc: David S . Miller, Rob Rice, Steve Lin, linux-crypto, linux-kernel
In-Reply-To: <20170222235138.8531-1-colin.king@canonical.com>
On Wed, Feb 22, 2017 at 11:51:38PM +0000, Colin King wrote:
> From: Colin Ian King <colin.king@canonical.com>
>
> trivial fix to spelling mistake in pr_err message
>
> Signed-off-by: Colin Ian King <colin.king@canonical.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v2 0/2] Propagate fallback bit for cbc and ctr
From: Herbert Xu @ 2017-03-09 10:47 UTC (permalink / raw)
To: Marcelo Henrique Cerri; +Cc: David S. Miller, linux-crypto, linux-kernel
In-Reply-To: <1488199106-16061-1-git-send-email-marcelo.cerri@canonical.com>
On Mon, Feb 27, 2017 at 09:38:24AM -0300, Marcelo Henrique Cerri wrote:
> Hi Hebert,
>
> For v2:
>
> - fixed the memory leakage in cbc.
> - included crypto/algapi.h in crypto/cbc.c for crypto_requires_off();
> - ERR_CAST instead PTR_ERR in ctr.
> - Also propagated the fallback bit for rfc3686.
>
> Marcelo Henrique Cerri (2):
> crypto: cbc - Propagate NEED_FALLBACK bit
> crypto: ctr - Propagate NEED_FALLBACK bit
>
> crypto/cbc.c | 15 +++++++++++++--
> crypto/ctr.c | 23 ++++++++++++++++++-----
> 2 files changed, 31 insertions(+), 7 deletions(-)
All applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: s5p-sss - Fix spinlock recursion on LRW(AES)
From: Herbert Xu @ 2017-03-09 10:48 UTC (permalink / raw)
To: Krzysztof Kozlowski
Cc: davem, Vladimir Zapolskiy, Marek Szyprowski, linux-crypto,
linux-kernel, Nathan Royce,
# v4 . 10 . x : 07de4bc88c crypto : s5p-sss - Fix completing
In-Reply-To: <20170308211420.5372-1-krzk@kernel.org>
On Wed, Mar 08, 2017 at 11:14:20PM +0200, Krzysztof Kozlowski wrote:
> Running TCRYPT with LRW compiled causes spinlock recursion:
>
> testing speed of async lrw(aes) (lrw(ecb-aes-s5p)) encryption
> tcrypt: test 0 (256 bit key, 16 byte blocks): 19007 operations in 1 seconds (304112 bytes)
> tcrypt: test 1 (256 bit key, 64 byte blocks): 15753 operations in 1 seconds (1008192 bytes)
> tcrypt: test 2 (256 bit key, 256 byte blocks): 14293 operations in 1 seconds (3659008 bytes)
> tcrypt: test 3 (256 bit key, 1024 byte blocks): 11906 operations in 1 seconds (12191744 bytes)
> tcrypt: test 4 (256 bit key, 8192 byte blocks):
> BUG: spinlock recursion on CPU#1, irq/84-10830000/89
> lock: 0xeea99a68, .magic: dead4ead, .owner: irq/84-10830000/89, .owner_cpu: 1
> CPU: 1 PID: 89 Comm: irq/84-10830000 Not tainted 4.11.0-rc1-00001-g897ca6d0800d #559
> Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
> [<c010e1ec>] (unwind_backtrace) from [<c010ae1c>] (show_stack+0x10/0x14)
> [<c010ae1c>] (show_stack) from [<c03449c0>] (dump_stack+0x78/0x8c)
> [<c03449c0>] (dump_stack) from [<c015de68>] (do_raw_spin_lock+0x11c/0x120)
> [<c015de68>] (do_raw_spin_lock) from [<c0720110>] (_raw_spin_lock_irqsave+0x20/0x28)
> [<c0720110>] (_raw_spin_lock_irqsave) from [<c0572ca0>] (s5p_aes_crypt+0x2c/0xb4)
> [<c0572ca0>] (s5p_aes_crypt) from [<bf1d8aa4>] (do_encrypt+0x78/0xb0 [lrw])
> [<bf1d8aa4>] (do_encrypt [lrw]) from [<bf1d8b00>] (encrypt_done+0x24/0x54 [lrw])
> [<bf1d8b00>] (encrypt_done [lrw]) from [<c05732a0>] (s5p_aes_complete+0x60/0xcc)
> [<c05732a0>] (s5p_aes_complete) from [<c0573440>] (s5p_aes_interrupt+0x134/0x1a0)
> [<c0573440>] (s5p_aes_interrupt) from [<c01667c4>] (irq_thread_fn+0x1c/0x54)
> [<c01667c4>] (irq_thread_fn) from [<c0166a98>] (irq_thread+0x12c/0x1e0)
> [<c0166a98>] (irq_thread) from [<c0136a28>] (kthread+0x108/0x138)
> [<c0136a28>] (kthread) from [<c0107778>] (ret_from_fork+0x14/0x3c)
>
> Interrupt handling routine was calling req->base.complete() under
> spinlock. In most cases this wasn't fatal but when combined with some
> of the cipher modes (like LRW) this caused recursion - starting the new
> encryption (s5p_aes_crypt()) while still holding the spinlock from
> previous round (s5p_aes_complete()).
>
> Beside that, the s5p_aes_interrupt() error handling path could execute
> two completions in case of error for RX and TX blocks.
>
> Rewrite the interrupt handling routine and the completion by:
>
> 1. Splitting the operations on scatterlist copies from
> s5p_aes_complete() into separate s5p_sg_done(). This still should be
> done under lock.
> The s5p_aes_complete() now only calls req->base.complete() and it has
> to be called outside of lock.
>
> 2. Moving the s5p_aes_complete() out of spinlock critical sections.
> In interrupt service routine s5p_aes_interrupts(), it appeared in few
> places, including error paths inside other functions called from ISR.
> This code was not so obvious to read so simplify it by putting the
> s5p_aes_complete() only within ISR level.
>
> Reported-by: Nathan Royce <nroycea+kernel@gmail.com>
> Cc: <stable@vger.kernel.org> # v4.10.x: 07de4bc88c crypto: s5p-sss - Fix completing
> Cc: <stable@vger.kernel.org> # v4.10.x
> Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: XTS Crypto Not Found In /proc/crypto Even After Compiled for 4.10.1.
From: Nathan Royce @ 2017-03-09 11:16 UTC (permalink / raw)
To: Krzysztof Kozlowski
Cc: Herbert Xu, davem, linux-crypto, linux-kernel, Marek Szyprowski
In-Reply-To: <20170308211543.euqexxlhdgpfcdjk@kozik-lap>
Gave it a try on 4.10.1, but still to no avail:
*****
[ 8.516138] raid6: using intx1 recovery algorithm
[ [0;32m OK [0m] Started Flush Journal to Persistent Storage.
[ 9.692091] Unable to handle kernel NULL pointer dereference at
virtual address 00000004
[ 9.698896] pgd = c0004000
[ 9.701489] [00000004] *pgd=00000000
[ 9.705055] Internal error: Oops: 17 [#1] SMP ARM
[ 9.709677] Modules linked in: xor_neon zlib_deflate aes_arm
raid6_pq nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
ip_tables x_tables
[ 9.719177] xor: measuring software checksum speed
[ 9.727455] CPU: 2 PID: 121 Comm: irq/69-10830000 Not tainted 4.10.1-dirty #1
[ 9.728911] arm4regs : 304.000 MB/sec
[ 9.738707] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[ 9.738913] 8regs : 224.000 MB/sec
[ 9.748924] 32regs : 208.000 MB/sec
[ 9.753095] task: edc80b00 task.stack: edd08000
[ 9.757626] PC is at post_crypt+0x1b4/0x1c4
[ 9.758914] neon : 316.000 MB/sec
[ 9.758927] xor: using function: neon (316.000 MB/sec)
[ 9.771040] LR is at post_crypt+0x1a8/0x1c4
[ 9.775197] pc : [<c0335c68>] lr : [<c0335c5c>] psr: 200c0013
[ 9.775197] sp : edd09e90 ip : edcd64f4 fp : 02cfca75
[ 9.786670] r10: 3df4074e r9 : c0c0540c r8 : edcd6400
[ 9.791831] r7 : 00000000 r6 : 00000400 r5 : 00000000 r4 : 00000000
[ 9.798333] r3 : ef4a775a r2 : 00000200 r1 : 00000200 r0 : 00000000
[ 9.804834] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 9.811901] Control: 10c5387d Table: 6c61c06a DAC: 00000051
[ 9.817618] Process irq/69-10830000 (pid: 121, stack limit = 0xedd08218)
[ 9.824291] Stack: (0xedd09e90 to 0xedd0a000)
[ 9.828624] 9e80: ef4a7758
ecca6200 ef4a7758 ecca6200
[ 9.836781] 9ea0: edcd65dc 00000400 00000000 00000000 00000400
00000000 eea8f810 00000002
[ 9.844926] 9ec0: 00000000 00000000 00000000 00000000 00000000
00000000 00000010 00000010
[ 9.853072] 9ee0: 0000000f 00040a01 ee958390 edcd6400 ee9583bc
0000000c ee9583e8 00000000
[ 9.861217] 9f00: 00000000 600c0013 ee889d20 c033608c ee958390
c05a7ea8 00000000 00000001
[ 9.869363] 9f20: ee957b40 eea8a400 eea8a400 ee957b40 c016ee68
c0c0540c 00000000 c016ee84
[ 9.877508] 9f40: edd08000 ee957b64 eea8a400 c016f198 ee957b80
00000000 c016ef7c 00040a01
[ 9.885653] 9f60: 00000000 eea21380 edd08000 00000000 ee957b80
ee957b40 c016f04c eea213a8
[ 9.893800] 9f80: ee889d20 c0138710 edd08000 ee957b80 c0138608
00000000 00000000 00000000
[ 9.901944] 9fa0: 00000000 00000000 00000000 c0107a38 00000000
00000000 00000000 00000000
[ 9.910089] 9fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[ 9.918235] 9fe0: 00000000 00000000 00000000 00000000 00000013
00000000 00000000 00000000
[ 9.926399] [<c0335c68>] (post_crypt) from [<c033608c>]
(decrypt_done+0x4c/0x54)
[ 9.933761] [<c033608c>] (decrypt_done) from [<c05a7ea8>]
(s5p_aes_interrupt+0x1bc/0x208)
[ 9.941908] [<c05a7ea8>] (s5p_aes_interrupt) from [<c016ee84>]
(irq_thread_fn+0x1c/0x54)
[ 9.949956] [<c016ee84>] (irq_thread_fn) from [<c016f198>]
(irq_thread+0x14c/0x204)
[ 9.957585] [<c016f198>] (irq_thread) from [<c0138710>] (kthread+0x108/0x138)
[ 9.964681] [<c0138710>] (kthread) from [<c0107a38>]
(ret_from_fork+0x14/0x3c)
[ 9.971871] Code: eb0114aa e598c118 e58d001c e1a04000 (e5906004)
[ 9.977963] ---[ end trace 8c160bf6676cfe1c ]---
[ 9.982560] genirq: exiting task "irq/69-10830000" (121) is an
active IRQ thread (irq 69)
[ 11.715339] Btrfs loaded, crc32c=crc32c-generic
*****
Also for the sake of testing, I did not add any FLAGS for compilation this time.
On Wed, Mar 8, 2017 at 3:15 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> On Wed, Mar 08, 2017 at 07:45:42PM +0200, Krzysztof Kozlowski wrote:
> I sent a fix. At least for spin lock recursion in tcrypt.
>
> Could you give it a try?
>
> Best regards,
> Krzysztof
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox