* [PATCH v2 1/3] clk: meson-gxbb: expose clock CLKID_RNG0
From: Heiner Kallweit @ 2017-02-21 20:11 UTC (permalink / raw)
To: Jerome Brunet, Kevin Hilman, Herbert Xu, Neil Armstrong,
linux-amlogic, linux-crypto, Stephen Boyd, Michael Turquette
Cc: linux-clk, devicetree
Expose clock CLKID_RNG0 which is needed for the HW random number generator.
Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
---
v2:
- added clk and DT maintainers
- split exposing the clock and using it in DT into two patches
- comment out clock definition in drivers/clk/meson/gxbb.h
- silently move CLKID_SPI to the right place
---
drivers/clk/meson/gxbb.h | 2 +-
include/dt-bindings/clock/gxbb-clkc.h | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/clk/meson/gxbb.h b/drivers/clk/meson/gxbb.h
index 8ee2022c..cbd62e46 100644
--- a/drivers/clk/meson/gxbb.h
+++ b/drivers/clk/meson/gxbb.h
@@ -193,7 +193,7 @@
/* CLKID_I2C */
/* #define CLKID_SAR_ADC */
#define CLKID_SMART_CARD 24
-#define CLKID_RNG0 25
+/* CLKID_RNG0 */
#define CLKID_UART0 26
#define CLKID_SDHC 27
#define CLKID_STREAM 28
diff --git a/include/dt-bindings/clock/gxbb-clkc.h b/include/dt-bindings/clock/gxbb-clkc.h
index 692846c7..63f4c2c4 100644
--- a/include/dt-bindings/clock/gxbb-clkc.h
+++ b/include/dt-bindings/clock/gxbb-clkc.h
@@ -12,9 +12,10 @@
#define CLKID_FCLK_DIV4 6
#define CLKID_CLK81 12
#define CLKID_MPLL2 15
-#define CLKID_SPI 34
#define CLKID_I2C 22
#define CLKID_SAR_ADC 23
+#define CLKID_RNG0 25
+#define CLKID_SPI 34
#define CLKID_ETH 36
#define CLKID_USB0 50
#define CLKID_USB1 51
--
2.11.1
^ permalink raw reply related
* Re: crypto/cavium MSI-X fixups
From: David Daney @ 2017-02-21 17:36 UTC (permalink / raw)
To: Christoph Hellwig, George Cherian
Cc: George Cherian, David Daney, Herbert Xu, linux-crypto,
linux-kernel
In-Reply-To: <20170219173223.GA4682@lst.de>
On 02/19/2017 09:32 AM, Christoph Hellwig wrote:
> Herbert,
>
> any comment? I'd really like to avoid introducing new pci_enable_msix
> users in this merge window..
Hi Cristoph,
With respect to pci_enable_msix(), what do you recommend as a
replacement? For the crypto/cavium driver, you recommend
pci_alloc_irq_vectors(), which works well if the required MSI-X indexes
are contiguous starting at zero. What would be used for a device that
has 184 MSI-X, but only a sparse subset (fewer than half) of these are
required for the driver operation. It would waste system resources to
use an API that forces us to allocate 184 when only 80 are required.
Currently pci_enable_msix() allows an arbitrary set of MSI-X to be
requested, which exactly fits the requirements of our (non
crypto/cavium) hardware.
Thanks in advance for any insight you can provide,
David Daney
>
> On Wed, Feb 15, 2017 at 02:47:09PM +0530, George Cherian wrote:
>> Hi Christoph,
>>
>>
>> On 02/15/2017 12:48 PM, Christoph Hellwig wrote:
>>> Hi George,
>>>
>>> your commit "crypto: cavium - Add Support for Octeon-tx CPT Engine"
>>> add a new caller to pci_enable_msix. This API has long been deprecated
>>> so this series switches it to use pci_alloc_irq_vectors instead.
>>>
>>> Can you please test it and make sure it goes in before the end of the
>>> merge window so that no more users of the old API hit mainline?
>>
>> Yes the changes works well.
>> Acked-by: George Cherian <george.cherian@cavium.com>
>>
>> for the series.
>>>
> ---end quoted text---
>
^ permalink raw reply
* Re: [PATCH 0/2] hwrng: meson: add clock handling
From: Neil Armstrong @ 2017-02-21 1:58 UTC (permalink / raw)
To: Heiner Kallweit, Kevin Hilman, Herbert Xu; +Cc: linux-amlogic, linux-crypto
In-Reply-To: <00b2a4cc-686e-0cc0-43e8-552b70a1e868@gmail.com>
On 02/19/2017 09:30 PM, Heiner Kallweit wrote:
> The HW randon number generator requires a clock and we shouldn't rely
> on the boot loader to enable it.
>
> Heiner Kallweit (2):
> hwrng: meson: expose RNG0 clock via DT
> hwrng: meson: add clock handling to driver
>
> arch/arm64/boot/dts/amlogic/meson-gx.dtsi | 2 +-
> arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi | 5 +++++
> drivers/char/hw_random/meson-rng.c | 27 ++++++++++++++++++++++++++-
> include/dt-bindings/clock/gxbb-clkc.h | 1 +
> 4 files changed, 33 insertions(+), 2 deletions(-)
>
Hi Heiner,
Thanks for the fix, but can you also update the dt-bindings document ?
Documentation/devicetree/bindings/rng/amlogic,meson-rng.txt
I also think this clock should be optional, to keep driver working using old bindings, set the core_clk to NULL if not available, clock_prepare_enable and clock_disable_unprepare won't fail with a NULL clock
Neil
^ permalink raw reply
* Re: [PATCH v3] dm: switch dm-verity to async hash crypto API
From: Milan Broz @ 2017-02-20 12:51 UTC (permalink / raw)
To: Gilad Ben-Yossef, dm-devel, Alasdair Kergon, MIKE Snitzer
Cc: linux-crypto, gilad.benyossef, Ofir Drang, Eric Biggers,
Ondrej Mosnáček
In-Reply-To: <1487508367-16036-1-git-send-email-gilad@benyossef.com>
On 02/19/2017 01:46 PM, Gilad Ben-Yossef wrote:
> Use of the synchronous digest API limits dm-verity to using pure
> CPU based algorithm providers and rules out the use of off CPU
> algorithm providers which are normally asynchronous by nature,
> potentially freeing CPU cycles.
>
> This can reduce performance per Watt in situations such as during
> boot time when a lot of concurrent file accesses are made to the
> protected volume.
>
> Move DM_VERITY to the asynchronous hash API.
>
> Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com>
> CC: Eric Biggers <ebiggers3@gmail.com>
> CC: Ondrej Mosnáček <omosnacek+linux-crypto@gmail.com>
> CC: Milan Broz <gmazyland@gmail.com>
> ---
>
> The patch was tested on an Armv7 based dual core Zynq ZC706 development
> board with SHA256-asm, SHA256-neon synchronous providers with no visible
> degradation of performance, with cryptd based asynchronous versions of the
> same and with an off tree Arm CryptoCell asynchronous provider.
>
> Changes from v2:
>
> - Use completion to potentially wait also on crypto_ahash_init() as it
> may finish asynchronously as well in some drivers, such as cryptd, as
> discovered by Milan Broz.
This version survived my simple veritysetup test (in cryptsetup package)
with forced async path so I think it is ok for 4.12 queue...
Thanks,
Milan
> - Use sg_init_one() where apropriate as pointed out by Milan Broz.
>
> Changes from v1:
>
> - Use a 0 mask to allocate crypto alg indicating we welcome async algo
> providers, as suggested by Ondrej Mosnáček.
> - Fix use of un-initialized completion when using async provider for IO
> blocks hashing
> - Pass flag indicating we are OK with crypto provider backlog
> - Re-factor checking for need to wait into a common function
>
^ permalink raw reply
* Re: [PATCH 2/2] hwrng: meson: add clock handling to driver
From: PrasannaKumar Muralidharan @ 2017-02-20 11:37 UTC (permalink / raw)
To: Heiner Kallweit
Cc: Kevin Hilman, Herbert Xu, Neil Armstrong, linux-amlogic,
linux-crypto
In-Reply-To: <eff6bfbe-3257-5d74-35c7-226a29c61fcb@gmail.com>
On 20 February 2017 at 02:05, Heiner Kallweit <hkallweit1@gmail.com> wrote:
> Add handling of RNG0 clock to the driver.
>
> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
> ---
> drivers/char/hw_random/meson-rng.c | 27 ++++++++++++++++++++++++++-
> 1 file changed, 26 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/char/hw_random/meson-rng.c b/drivers/char/hw_random/meson-rng.c
> index 119d6984..1f586e48 100644
> --- a/drivers/char/hw_random/meson-rng.c
> +++ b/drivers/char/hw_random/meson-rng.c
> @@ -62,6 +62,7 @@
> #include <linux/slab.h>
> #include <linux/types.h>
> #include <linux/of.h>
> +#include <linux/clk.h>
>
> #define RNG_DATA 0x00
>
> @@ -69,6 +70,7 @@ struct meson_rng_data {
> void __iomem *base;
> struct platform_device *pdev;
> struct hwrng rng;
> + struct clk *core_clk;
> };
>
> static int meson_rng_read(struct hwrng *rng, void *buf, size_t max, bool wait)
> @@ -86,6 +88,7 @@ static int meson_rng_probe(struct platform_device *pdev)
> struct device *dev = &pdev->dev;
> struct meson_rng_data *data;
> struct resource *res;
> + int ret;
Variable ret is not used. It can be removed.
> data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
> if (!data)
> @@ -98,12 +101,33 @@ static int meson_rng_probe(struct platform_device *pdev)
> if (IS_ERR(data->base))
> return PTR_ERR(data->base);
>
> + data->core_clk = devm_clk_get(dev, "core");
> + if (IS_ERR(data->core_clk))
> + return PTR_ERR(data->core_clk);
> +
> + ret = clk_prepare_enable(data->core_clk);
> + if (ret)
> + return ret;
> +
> data->rng.name = pdev->name;
> data->rng.read = meson_rng_read;
>
> platform_set_drvdata(pdev, data);
>
> - return devm_hwrng_register(dev, &data->rng);
> + ret = devm_hwrng_register(dev, &data->rng);
> + if (ret)
> + clk_disable_unprepare(data->core_clk);
> +
> + return ret;
> +}
> +
> +static int meson_rng_remove(struct platform_device *pdev)
> +{
> + struct meson_rng_data *data = platform_get_drvdata(pdev);
> +
> + clk_disable_unprepare(data->core_clk);
> +
> + return 0;
> }
In .remove clock gets disabled before the hwrng_unregister is called.
The device node '/dev/hwrng' could be accessed while meson_rng_remove
is called which could lead to problems. Instead of devm_hwrng_register
use hwrng_register in .probe and call hwrng_unregister in .remove.
> static const struct of_device_id meson_rng_of_match[] = {
> @@ -114,6 +138,7 @@ MODULE_DEVICE_TABLE(of, meson_rng_of_match);
>
> static struct platform_driver meson_rng_driver = {
> .probe = meson_rng_probe,
> + .remove = meson_rng_remove,
> .driver = {
> .name = "meson-rng",
> .of_match_table = meson_rng_of_match,
> --
> 2.11.1
>
>
^ permalink raw reply
* Re: [PATCH 1/2] hwrng: meson: expose RNG0 clock via DT
From: Jerome Brunet @ 2017-02-20 10:07 UTC (permalink / raw)
To: Heiner Kallweit, Kevin Hilman, Herbert Xu
Cc: linux-amlogic, linux-crypto, Neil Armstrong
In-Reply-To: <a02b078d-9171-cf66-d5d8-84259f9f9a88@gmail.com>
On Sun, 2017-02-19 at 21:33 +0100, Heiner Kallweit wrote:
> Expose the RNG0 clock via DT.
>
> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
> ---
> arch/arm64/boot/dts/amlogic/meson-gx.dtsi | 2 +-
> arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi | 5 +++++
> include/dt-bindings/clock/gxbb-clkc.h | 1 +
> 3 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
> b/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
> index 5d995f77..620495a4 100644
> --- a/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
> +++ b/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
> @@ -380,7 +380,7 @@
> #size-cells = <2>;
> ranges = <0x0 0x0 0x0 0xc8834000 0x0
> 0x2000>;
>
> - rng {
> + hwrng: rng {
> compatible = "amlogic,meson-rng";
> reg = <0x0 0x0 0x0 0x4>;
> };
> diff --git a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
> b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
> index 04b3324b..a375cb21 100644
> --- a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
> +++ b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
> @@ -524,3 +524,8 @@
> &vpu {
> compatible = "amlogic,meson-gxbb-vpu", "amlogic,meson-gx-
> vpu";
> };
> +
> +&hwrng {
> + clocks = <&clkc CLKID_RNG0>;
> + clock-names = "core";
> +};
> diff --git a/include/dt-bindings/clock/gxbb-clkc.h b/include/dt-
> bindings/clock/gxbb-clkc.h
> index 692846c7..473676b1 100644
> --- a/include/dt-bindings/clock/gxbb-clkc.h
> +++ b/include/dt-bindings/clock/gxbb-clkc.h
> @@ -15,6 +15,7 @@
> #define CLKID_SPI 34
> #define CLKID_I2C 22
> #define CLKID_SAR_ADC 23
> +#define CLKID_RNG0 25
Shouldn't you comment out the corresponding "define" in
drivers/clk/meson/gxbb.h ?
I don't know if it is rule, but we usually "expose" the clock (moving
the define from drivers/clk/meson/gxbb.h to include/dt-
bindings/clock/gxbb-clkc.h) in a dedicated patch, with the subject
starting with "clk: gxbb: expose foobar clock". It helps the
maintainers of the subsystem to give their ack.
For this to happen, please remember the patch should be sent to them as
well. Here you are missing the clock and dt guys:
* Stephen Boyd <sboyd@codeaurora.org>
* Michael Turquette <mturquette@baylibre.com>
* linux-clk@vger.kernel.org
* devicetree@vger.kernel.org
scripts/get_maintainer.pl can help you finding all people your patch
should be sent to.
The remaining part of this patch involves DT, so the subject should
start with "ARM64: dts: meson-gx:"
You also need to add devicetree@vger.kernel.org in CC.
Apart from these minor remarks, the patchset looks good to me.
Thx !
> #define CLKID_ETH 36
> #define CLKID_USB0 50
> #define CLKID_USB1 51
^ permalink raw reply
* Re: [RFC PATCH v1 1/1] mm: zswap - Add crypto acomp/scomp framework support
From: Nair, Vishnu @ 2017-02-20 9:52 UTC (permalink / raw)
To: Giovanni Cabiddu, Narayana, Prasad Athreya
Cc: Seth Jennings, Mahipal Challa, herbert@gondor.apana.org.au,
davem@davemloft.net, linux-crypto@vger.kernel.org, LKML, Linux-MM,
Challa, Mahipal
In-Reply-To: <20170215221208.GA820@silv-gc1.ir.intel.com>
[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]
>This assumption is not correct. An asynchronous implementation, when
>it finishes processing a request, will call acomp_request_complete() which
>in turn calls the callback.
>If the callback is set to NULL, this function will dereference a NULL
>pointer.
This would leave us with the option of waiting in zswap until completion. Here we had a doubt.
If we go ahead with an implementation similar to the one found in crypto/testmgr.c, the private data(result) which is registered via 'acomp_request_set_callback()' is coming from stack. Do you see this as a potential problem for an acutal asynchronus algorithm due to the context from which callback is called? Do we have to use per-cpu dynamic allocation?
Thanks,
Vishnu
________________________________
From: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Sent: Thursday, February 16, 2017 3:42 AM
To: Narayana, Prasad Athreya
Cc: Seth Jennings; Mahipal Challa; herbert@gondor.apana.org.au; davem@davemloft.net; linux-crypto@vger.kernel.org; LKML; Linux-MM; Narayana, Prasad Athreya; Nair, Vishnu; Challa, Mahipal; Nair, Vishnu
Subject: Re: [RFC PATCH v1 1/1] mm: zswap - Add crypto acomp/scomp framework support
On Wed, Feb 15, 2017 at 07:27:30PM +0530, Narayana Prasad Athreya wrote:
> > I assume all of these crypto_acomp_[compress|decompress] calls are
> > actually synchronous,
> > not asynchronous as the name suggests. Otherwise, this would blow up
> > quite spectacularly
> > since all the resources we use in the call get derefed/unmapped below.
> >
> > Could an async algorithm be implement/used that would break this assumption?
>
> The callback is set to NULL using acomp_request_set_callback(). This implies
> synchronous mode of operation. So the underlying implementation must
> complete the operation synchronously.
This assumption is not correct. An asynchronous implementation, when
it finishes processing a request, will call acomp_request_complete() which
in turn calls the callback.
If the callback is set to NULL, this function will dereference a NULL
pointer.
Regards,
--
Giovanni
[-- Attachment #2: Type: text/html, Size: 3609 bytes --]
^ permalink raw reply
* [PATCH 9/9] crypto: mediatek - add support to OFB mode and CFB128 mode
From: Ryder Lee @ 2017-02-20 9:27 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch adds support to OFB mode and CFB128 mode.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 84 ++++++++++++++++++++++++++++++++++++---
1 file changed, 78 insertions(+), 6 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 9e845e8..c2bdfb9 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -25,7 +25,7 @@
#define AES_CT_CTRL_HDR cpu_to_le32(0x00220000)
-/* AES-CBC/ECB/CTR command token */
+/* AES-CBC/ECB/CTR/OFB/CFB command token */
#define AES_CMD0 cpu_to_le32(0x05000000)
#define AES_CMD1 cpu_to_le32(0x2d060000)
#define AES_CMD2 cpu_to_le32(0xe4a63806)
@@ -52,6 +52,8 @@
/* AES transform information word 1 fields */
#define AES_TFM_ECB cpu_to_le32(0x0 << 0)
#define AES_TFM_CBC cpu_to_le32(0x1 << 0)
+#define AES_TFM_OFB cpu_to_le32(0x4 << 0)
+#define AES_TFM_CFB128 cpu_to_le32(0x5 << 0)
#define AES_TFM_CTR_INIT cpu_to_le32(0x2 << 0) /* init counter to 1 */
#define AES_TFM_CTR_LOAD cpu_to_le32(0x6 << 0) /* load/reuse counter */
#define AES_TFM_3IV cpu_to_le32(0x7 << 5) /* using IV 0-2 */
@@ -60,13 +62,15 @@
#define AES_TFM_ENC_HASH cpu_to_le32(0x1 << 17)
/* AES flags */
-#define AES_FLAGS_CIPHER_MSK GENMASK(2, 0)
+#define AES_FLAGS_CIPHER_MSK GENMASK(4, 0)
#define AES_FLAGS_ECB BIT(0)
#define AES_FLAGS_CBC BIT(1)
#define AES_FLAGS_CTR BIT(2)
-#define AES_FLAGS_GCM BIT(3)
-#define AES_FLAGS_ENCRYPT BIT(4)
-#define AES_FLAGS_BUSY BIT(5)
+#define AES_FLAGS_OFB BIT(3)
+#define AES_FLAGS_CFB128 BIT(4)
+#define AES_FLAGS_GCM BIT(5)
+#define AES_FLAGS_ENCRYPT BIT(6)
+#define AES_FLAGS_BUSY BIT(7)
#define AES_AUTH_TAG_ERR cpu_to_le32(BIT(26))
@@ -412,7 +416,7 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
return mtk_aes_complete(cryp, aes, -EINVAL);
}
-/* Initialize transform information of CBC/ECB/CTR mode */
+/* Initialize transform information of CBC/ECB/CTR/OFB/CFB mode */
static void mtk_aes_info_init(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
size_t len)
{
@@ -441,6 +445,12 @@ static void mtk_aes_info_init(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
case AES_FLAGS_CTR:
info->tfm[1] = AES_TFM_CTR_LOAD;
goto ctr;
+ case AES_FLAGS_OFB:
+ info->tfm[1] = AES_TFM_OFB;
+ break;
+ case AES_FLAGS_CFB128:
+ info->tfm[1] = AES_TFM_CFB128;
+ break;
default:
/* Should not happen... */
@@ -699,6 +709,26 @@ static int mtk_aes_ctr_decrypt(struct ablkcipher_request *req)
return mtk_aes_crypt(req, AES_FLAGS_CTR);
}
+static int mtk_aes_ofb_encrypt(struct ablkcipher_request *req)
+{
+ return mtk_aes_crypt(req, AES_FLAGS_ENCRYPT | AES_FLAGS_OFB);
+}
+
+static int mtk_aes_ofb_decrypt(struct ablkcipher_request *req)
+{
+ return mtk_aes_crypt(req, AES_FLAGS_OFB);
+}
+
+static int mtk_aes_cfb_encrypt(struct ablkcipher_request *req)
+{
+ return mtk_aes_crypt(req, AES_FLAGS_ENCRYPT | AES_FLAGS_CFB128);
+}
+
+static int mtk_aes_cfb_decrypt(struct ablkcipher_request *req)
+{
+ return mtk_aes_crypt(req, AES_FLAGS_CFB128);
+}
+
static int mtk_aes_cra_init(struct crypto_tfm *tfm)
{
struct mtk_aes_ctx *ctx = crypto_tfm_ctx(tfm);
@@ -794,6 +824,48 @@ static int mtk_aes_ctr_cra_init(struct crypto_tfm *tfm)
.decrypt = mtk_aes_ctr_decrypt,
}
},
+{
+ .cra_name = "ofb(aes)",
+ .cra_driver_name = "ofb-aes-mtk",
+ .cra_priority = 400,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER |
+ CRYPTO_ALG_ASYNC,
+ .cra_init = mtk_aes_cra_init,
+ .cra_blocksize = 1,
+ .cra_ctxsize = sizeof(struct mtk_aes_ctx),
+ .cra_alignmask = 0xf,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_u.ablkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = mtk_aes_setkey,
+ .encrypt = mtk_aes_ofb_encrypt,
+ .decrypt = mtk_aes_ofb_decrypt,
+ }
+},
+{
+ .cra_name = "cfb(aes)",
+ .cra_driver_name = "cfb-aes-mtk",
+ .cra_priority = 400,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER |
+ CRYPTO_ALG_ASYNC,
+ .cra_init = mtk_aes_cra_init,
+ .cra_blocksize = 1,
+ .cra_ctxsize = sizeof(struct mtk_aes_ctx),
+ .cra_alignmask = 0xf,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_u.ablkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = mtk_aes_setkey,
+ .encrypt = mtk_aes_cfb_encrypt,
+ .decrypt = mtk_aes_cfb_decrypt,
+ }
+},
};
static inline struct mtk_aes_gcm_ctx *
--
1.9.1
^ permalink raw reply related
* [PATCH 1/9] crypto: mediatek - rework interrupt handler
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch removes redundant task that used to handle interrupt
from ring manager, so that the same task/handler can be shared.
It also uses aes->id and sha-id to distinguish interrupt sources.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 73 +++++++++++-----------------------
drivers/crypto/mediatek/mtk-platform.h | 4 ++
drivers/crypto/mediatek/mtk-sha.c | 73 ++++++++++------------------------
3 files changed, 49 insertions(+), 101 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 3a47cdb..e67881f 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -1092,55 +1092,26 @@ static void mtk_aes_gcm_exit(struct crypto_aead *aead)
},
};
-static void mtk_aes_enc_task(unsigned long data)
+static void mtk_aes_done_task(unsigned long data)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_aes_rec *aes = cryp->aes[0];
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
+ struct mtk_cryp *cryp = aes->cryp;
mtk_aes_unmap(cryp, aes);
aes->resume(cryp, aes);
}
-static void mtk_aes_dec_task(unsigned long data)
+static irqreturn_t mtk_aes_irq(int irq, void *dev_id)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_aes_rec *aes = cryp->aes[1];
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)dev_id;
+ struct mtk_cryp *cryp = aes->cryp;
+ u32 val = mtk_aes_read(cryp, RDR_STAT(aes->id));
- mtk_aes_unmap(cryp, aes);
- aes->resume(cryp, aes);
-}
-
-static irqreturn_t mtk_aes_enc_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_aes_rec *aes = cryp->aes[0];
- u32 val = mtk_aes_read(cryp, RDR_STAT(RING0));
-
- mtk_aes_write(cryp, RDR_STAT(RING0), val);
-
- if (likely(AES_FLAGS_BUSY & aes->flags)) {
- mtk_aes_write(cryp, RDR_PROC_COUNT(RING0), MTK_CNT_RST);
- mtk_aes_write(cryp, RDR_THRESH(RING0),
- MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
-
- tasklet_schedule(&aes->task);
- } else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
- }
- return IRQ_HANDLED;
-}
-
-static irqreturn_t mtk_aes_dec_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_aes_rec *aes = cryp->aes[1];
- u32 val = mtk_aes_read(cryp, RDR_STAT(RING1));
-
- mtk_aes_write(cryp, RDR_STAT(RING1), val);
+ mtk_aes_write(cryp, RDR_STAT(aes->id), val);
if (likely(AES_FLAGS_BUSY & aes->flags)) {
- mtk_aes_write(cryp, RDR_PROC_COUNT(RING1), MTK_CNT_RST);
- mtk_aes_write(cryp, RDR_THRESH(RING1),
+ mtk_aes_write(cryp, RDR_PROC_COUNT(aes->id), MTK_CNT_RST);
+ mtk_aes_write(cryp, RDR_THRESH(aes->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
tasklet_schedule(&aes->task);
@@ -1171,14 +1142,18 @@ static int mtk_aes_record_init(struct mtk_cryp *cryp)
if (!aes[i]->buf)
goto err_cleanup;
- aes[i]->id = i;
+ aes[i]->cryp = cryp;
spin_lock_init(&aes[i]->lock);
crypto_init_queue(&aes[i]->queue, AES_QUEUE_SIZE);
+
+ tasklet_init(&aes[i]->task, mtk_aes_done_task,
+ (unsigned long)aes[i]);
}
- tasklet_init(&aes[0]->task, mtk_aes_enc_task, (unsigned long)cryp);
- tasklet_init(&aes[1]->task, mtk_aes_dec_task, (unsigned long)cryp);
+ /* Link to ring0 and ring1 respectively */
+ aes[0]->id = RING0;
+ aes[1]->id = RING1;
return 0;
@@ -1246,19 +1221,17 @@ int mtk_cipher_alg_register(struct mtk_cryp *cryp)
if (ret)
goto err_record;
- /* Ring0 is use by encryption record */
- ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_enc_irq,
- IRQF_TRIGGER_LOW, "mtk-aes", cryp);
+ ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_irq,
+ 0, "mtk-aes", cryp->aes[0]);
if (ret) {
- dev_err(cryp->dev, "unable to request AES encryption irq.\n");
+ dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
- /* Ring1 is use by decryption record */
- ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_dec_irq,
- IRQF_TRIGGER_LOW, "mtk-aes", cryp);
+ ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_irq,
+ 0, "mtk-aes", cryp->aes[1]);
if (ret) {
- dev_err(cryp->dev, "unable to request AES decryption irq.\n");
+ dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index ed6d871..d2a1db6 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -125,6 +125,7 @@ struct mtk_aes_dma {
/**
* struct mtk_aes_rec - AES operation record
+ * @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @areq: pointer to async request
* @task: the tasklet is use in AES interrupt
@@ -143,6 +144,7 @@ struct mtk_aes_dma {
* Structure used to record AES execution state.
*/
struct mtk_aes_rec {
+ struct mtk_cryp *cryp;
struct crypto_queue queue;
struct crypto_async_request *areq;
struct tasklet_struct task;
@@ -166,6 +168,7 @@ struct mtk_aes_rec {
/**
* struct mtk_sha_rec - SHA operation record
+ * @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @req: pointer to ahash request
* @task: the tasklet is use in SHA interrupt
@@ -176,6 +179,7 @@ struct mtk_aes_rec {
* Structure used to record SHA execution state.
*/
struct mtk_sha_rec {
+ struct mtk_cryp *cryp;
struct crypto_queue queue;
struct ahash_request *req;
struct tasklet_struct task;
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 55e3805..1fb98f6 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -1216,60 +1216,31 @@ static void mtk_sha_cra_exit(struct crypto_tfm *tfm)
},
};
-static void mtk_sha_task0(unsigned long data)
+static void mtk_sha_done_task(unsigned long data)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_sha_rec *sha = cryp->sha[0];
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
+ struct mtk_cryp *cryp = sha->cryp;
mtk_sha_unmap(cryp, sha);
mtk_sha_complete(cryp, sha);
}
-static void mtk_sha_task1(unsigned long data)
+static irqreturn_t mtk_sha_irq(int irq, void *dev_id)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_sha_rec *sha = cryp->sha[1];
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)dev_id;
+ struct mtk_cryp *cryp = sha->cryp;
+ u32 val = mtk_sha_read(cryp, RDR_STAT(sha->id));
- mtk_sha_unmap(cryp, sha);
- mtk_sha_complete(cryp, sha);
-}
-
-static irqreturn_t mtk_sha_ring2_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_sha_rec *sha = cryp->sha[0];
- u32 val = mtk_sha_read(cryp, RDR_STAT(RING2));
-
- mtk_sha_write(cryp, RDR_STAT(RING2), val);
+ mtk_sha_write(cryp, RDR_STAT(sha->id), val);
if (likely((SHA_FLAGS_BUSY & sha->flags))) {
- mtk_sha_write(cryp, RDR_PROC_COUNT(RING2), MTK_CNT_RST);
- mtk_sha_write(cryp, RDR_THRESH(RING2),
+ mtk_sha_write(cryp, RDR_PROC_COUNT(sha->id), MTK_CNT_RST);
+ mtk_sha_write(cryp, RDR_THRESH(sha->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
tasklet_schedule(&sha->task);
} else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
- }
- return IRQ_HANDLED;
-}
-
-static irqreturn_t mtk_sha_ring3_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_sha_rec *sha = cryp->sha[1];
- u32 val = mtk_sha_read(cryp, RDR_STAT(RING3));
-
- mtk_sha_write(cryp, RDR_STAT(RING3), val);
-
- if (likely((SHA_FLAGS_BUSY & sha->flags))) {
- mtk_sha_write(cryp, RDR_PROC_COUNT(RING3), MTK_CNT_RST);
- mtk_sha_write(cryp, RDR_THRESH(RING3),
- MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
-
- tasklet_schedule(&sha->task);
- } else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
+ dev_warn(cryp->dev, "SHA interrupt when no active requests.\n");
}
return IRQ_HANDLED;
}
@@ -1288,14 +1259,18 @@ static int mtk_sha_record_init(struct mtk_cryp *cryp)
if (!sha[i])
goto err_cleanup;
- sha[i]->id = i + RING2;
+ sha[i]->cryp = cryp;
spin_lock_init(&sha[i]->lock);
crypto_init_queue(&sha[i]->queue, SHA_QUEUE_SIZE);
+
+ tasklet_init(&sha[i]->task, mtk_sha_done_task,
+ (unsigned long)sha[i]);
}
- tasklet_init(&sha[0]->task, mtk_sha_task0, (unsigned long)cryp);
- tasklet_init(&sha[1]->task, mtk_sha_task1, (unsigned long)cryp);
+ /* Link to ring2 and ring3 respectively */
+ sha[0]->id = RING2;
+ sha[1]->id = RING3;
cryp->rec = 1;
@@ -1368,19 +1343,15 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
if (err)
goto err_record;
- /* Ring2 is use by SHA record0 */
- err = devm_request_irq(cryp->dev, cryp->irq[RING2],
- mtk_sha_ring2_irq, IRQF_TRIGGER_LOW,
- "mtk-sha", cryp);
+ err = devm_request_irq(cryp->dev, cryp->irq[RING2], mtk_sha_irq,
+ 0, "mtk-sha", cryp->sha[0]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq0.\n");
goto err_res;
}
- /* Ring3 is use by SHA record1 */
- err = devm_request_irq(cryp->dev, cryp->irq[RING3],
- mtk_sha_ring3_irq, IRQF_TRIGGER_LOW,
- "mtk-sha", cryp);
+ err = devm_request_irq(cryp->dev, cryp->irq[RING3], mtk_sha_irq,
+ 0, "mtk-sha", cryp->sha[1]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq1.\n");
goto err_res;
--
1.9.1
^ permalink raw reply related
* [PATCH 8/9] crypto: mediatek - make hardware operation flow more efficient
From: Ryder Lee @ 2017-02-20 9:27 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch refines data structures, which are used to control engine's
data path, to make it more efficient. Hence current change are:
- gathers the broken pieces of structures 'mtk_aes_ct''mtk_aes_tfm'
into struct mtk_aes_info hence avoiding additional DMA-mapping.
- adds 'keymode' in struct mtk_aes_base_ctx. When .setkey() callback is
called, we store keybit setting in keymode. Doing so, there is no need
to check keylen second time in mtk_aes_info_init() / mtk_aes_gcm_info_init().
Besides, this patch also removes unused macro definitions and adds helper
inline function to write security information(key, IV,...) to info->state.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 263 ++++++++++++++++++--------------------
drivers/crypto/mediatek/mtk-sha.c | 90 ++++++-------
2 files changed, 165 insertions(+), 188 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 8f3efa5..9e845e8 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -19,13 +19,10 @@
#define AES_BUF_ORDER 2
#define AES_BUF_SIZE ((PAGE_SIZE << AES_BUF_ORDER) \
& ~(AES_BLOCK_SIZE - 1))
+#define AES_MAX_STATE_BUF_SIZE SIZE_IN_WORDS(AES_KEYSIZE_256 + \
+ AES_BLOCK_SIZE * 2)
+#define AES_MAX_CT_SIZE 6
-/* AES command token size */
-#define AES_CT_SIZE_ECB 2
-#define AES_CT_SIZE_CBC 3
-#define AES_CT_SIZE_CTR 3
-#define AES_CT_SIZE_GCM_OUT 5
-#define AES_CT_SIZE_GCM_IN 6
#define AES_CT_CTRL_HDR cpu_to_le32(0x00220000)
/* AES-CBC/ECB/CTR command token */
@@ -50,6 +47,8 @@
#define AES_TFM_128BITS cpu_to_le32(0xb << 16)
#define AES_TFM_192BITS cpu_to_le32(0xd << 16)
#define AES_TFM_256BITS cpu_to_le32(0xf << 16)
+#define AES_TFM_GHASH_DIGEST cpu_to_le32(0x2 << 21)
+#define AES_TFM_GHASH cpu_to_le32(0x4 << 23)
/* AES transform information word 1 fields */
#define AES_TFM_ECB cpu_to_le32(0x0 << 0)
#define AES_TFM_CBC cpu_to_le32(0x1 << 0)
@@ -59,10 +58,9 @@
#define AES_TFM_FULL_IV cpu_to_le32(0xf << 5) /* using IV 0-3 */
#define AES_TFM_IV_CTR_MODE cpu_to_le32(0x1 << 10)
#define AES_TFM_ENC_HASH cpu_to_le32(0x1 << 17)
-#define AES_TFM_GHASH_DIG cpu_to_le32(0x2 << 21)
-#define AES_TFM_GHASH cpu_to_le32(0x4 << 23)
/* AES flags */
+#define AES_FLAGS_CIPHER_MSK GENMASK(2, 0)
#define AES_FLAGS_ECB BIT(0)
#define AES_FLAGS_CBC BIT(1)
#define AES_FLAGS_CTR BIT(2)
@@ -73,18 +71,12 @@
#define AES_AUTH_TAG_ERR cpu_to_le32(BIT(26))
/**
- * Command token(CT) is a set of hardware instructions that
- * are used to control engine's processing flow of AES.
- *
- * Transform information(TFM) is used to define AES state and
- * contains all keys and initial vectors.
- *
- * The engine requires CT and TFM to do:
- * - Commands decoding and control of the engine's data path.
- * - Coordinating hardware data fetch and store operations.
- * - Result token construction and output.
+ * mtk_aes_info - hardware information of AES
+ * @cmd: command token, hardware instruction
+ * @tfm: transform state of cipher algorithm.
+ * @state: contains keys and initial vectors.
*
- * Memory map of GCM's TFM:
+ * Memory layout of GCM buffer:
* /-----------\
* | AES KEY | 128/196/256 bits
* |-----------|
@@ -92,14 +84,16 @@
* |-----------|
* | IVs | 4 * 4 bytes
* \-----------/
+ *
+ * The engine requires all these info to do:
+ * - Commands decoding and control of the engine's data path.
+ * - Coordinating hardware data fetch and store operations.
+ * - Result token construction and output.
*/
-struct mtk_aes_ct {
- __le32 cmd[AES_CT_SIZE_GCM_IN];
-};
-
-struct mtk_aes_tfm {
- __le32 ctrl[2];
- __le32 state[SIZE_IN_WORDS(AES_KEYSIZE_256 + AES_BLOCK_SIZE * 2)];
+struct mtk_aes_info {
+ __le32 cmd[AES_MAX_CT_SIZE];
+ __le32 tfm[2];
+ __le32 state[AES_MAX_STATE_BUF_SIZE];
};
struct mtk_aes_reqctx {
@@ -109,11 +103,12 @@ struct mtk_aes_reqctx {
struct mtk_aes_base_ctx {
struct mtk_cryp *cryp;
u32 keylen;
+ __le32 keymode;
+
mtk_aes_fn start;
- struct mtk_aes_ct ct;
+ struct mtk_aes_info info;
dma_addr_t ct_dma;
- struct mtk_aes_tfm tfm;
dma_addr_t tfm_dma;
__le32 ct_hdr;
@@ -250,6 +245,22 @@ static inline void mtk_aes_restore_sg(const struct mtk_aes_dma *dma)
sg->length += dma->remainder;
}
+static inline void mtk_aes_write_state_le(__le32 *dst, const u32 *src, u32 size)
+{
+ int i;
+
+ for (i = 0; i < SIZE_IN_WORDS(size); i++)
+ dst[i] = cpu_to_le32(src[i]);
+}
+
+static inline void mtk_aes_write_state_be(__be32 *dst, const u32 *src, u32 size)
+{
+ int i;
+
+ for (i = 0; i < SIZE_IN_WORDS(size); i++)
+ dst[i] = cpu_to_be32(src[i]);
+}
+
static inline int mtk_aes_complete(struct mtk_cryp *cryp,
struct mtk_aes_rec *aes,
int err)
@@ -331,9 +342,7 @@ static void mtk_aes_unmap(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
{
struct mtk_aes_base_ctx *ctx = aes->ctx;
- dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
- DMA_TO_DEVICE);
- dma_unmap_single(cryp->dev, ctx->tfm_dma, sizeof(ctx->tfm),
+ dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->info),
DMA_TO_DEVICE);
if (aes->src.sg == aes->dst.sg) {
@@ -364,16 +373,14 @@ static void mtk_aes_unmap(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
{
struct mtk_aes_base_ctx *ctx = aes->ctx;
+ struct mtk_aes_info *info = &ctx->info;
- ctx->ct_dma = dma_map_single(cryp->dev, &ctx->ct, sizeof(ctx->ct),
+ ctx->ct_dma = dma_map_single(cryp->dev, info, sizeof(*info),
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(cryp->dev, ctx->ct_dma)))
goto exit;
- ctx->tfm_dma = dma_map_single(cryp->dev, &ctx->tfm, sizeof(ctx->tfm),
- DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(cryp->dev, ctx->tfm_dma)))
- goto tfm_map_err;
+ ctx->tfm_dma = ctx->ct_dma + sizeof(info->cmd);
if (aes->src.sg == aes->dst.sg) {
aes->src.sg_len = dma_map_sg(cryp->dev, aes->src.sg,
@@ -400,11 +407,7 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
return mtk_aes_xmit(cryp, aes);
sg_map_err:
- dma_unmap_single(cryp->dev, ctx->tfm_dma, sizeof(ctx->tfm),
- DMA_TO_DEVICE);
-tfm_map_err:
- dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
- DMA_TO_DEVICE);
+ dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(*info), DMA_TO_DEVICE);
exit:
return mtk_aes_complete(cryp, aes, -EINVAL);
}
@@ -415,50 +418,43 @@ static void mtk_aes_info_init(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
{
struct ablkcipher_request *req = ablkcipher_request_cast(aes->areq);
struct mtk_aes_base_ctx *ctx = aes->ctx;
+ struct mtk_aes_info *info = &ctx->info;
+ u32 cnt = 0;
ctx->ct_hdr = AES_CT_CTRL_HDR | cpu_to_le32(len);
- ctx->ct.cmd[0] = AES_CMD0 | cpu_to_le32(len);
- ctx->ct.cmd[1] = AES_CMD1;
+ info->cmd[cnt++] = AES_CMD0 | cpu_to_le32(len);
+ info->cmd[cnt++] = AES_CMD1;
+ info->tfm[0] = AES_TFM_SIZE(ctx->keylen) | ctx->keymode;
if (aes->flags & AES_FLAGS_ENCRYPT)
- ctx->tfm.ctrl[0] = AES_TFM_BASIC_OUT;
+ info->tfm[0] |= AES_TFM_BASIC_OUT;
else
- ctx->tfm.ctrl[0] = AES_TFM_BASIC_IN;
+ info->tfm[0] |= AES_TFM_BASIC_IN;
- if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_128))
- ctx->tfm.ctrl[0] |= AES_TFM_128BITS;
- else if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_256))
- ctx->tfm.ctrl[0] |= AES_TFM_256BITS;
- else
- ctx->tfm.ctrl[0] |= AES_TFM_192BITS;
-
- if (aes->flags & AES_FLAGS_CBC) {
- const u32 *iv = (const u32 *)req->info;
- u32 *iv_state = ctx->tfm.state + ctx->keylen;
- int i;
-
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE));
- ctx->tfm.ctrl[1] = AES_TFM_CBC | AES_TFM_FULL_IV;
-
- for (i = 0; i < SIZE_IN_WORDS(AES_BLOCK_SIZE); i++)
- iv_state[i] = cpu_to_le32(iv[i]);
-
- ctx->ct.cmd[2] = AES_CMD2;
- ctx->ct_size = AES_CT_SIZE_CBC;
- } else if (aes->flags & AES_FLAGS_ECB) {
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen);
- ctx->tfm.ctrl[1] = AES_TFM_ECB;
-
- ctx->ct_size = AES_CT_SIZE_ECB;
- } else if (aes->flags & AES_FLAGS_CTR) {
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE));
- ctx->tfm.ctrl[1] = AES_TFM_CTR_LOAD | AES_TFM_FULL_IV;
-
- ctx->ct.cmd[2] = AES_CMD2;
- ctx->ct_size = AES_CT_SIZE_CTR;
+ switch (aes->flags & AES_FLAGS_CIPHER_MSK) {
+ case AES_FLAGS_CBC:
+ info->tfm[1] = AES_TFM_CBC;
+ break;
+ case AES_FLAGS_ECB:
+ info->tfm[1] = AES_TFM_ECB;
+ goto ecb;
+ case AES_FLAGS_CTR:
+ info->tfm[1] = AES_TFM_CTR_LOAD;
+ goto ctr;
+
+ default:
+ /* Should not happen... */
+ return;
}
+
+ mtk_aes_write_state_le(info->state + ctx->keylen, req->info,
+ AES_BLOCK_SIZE);
+ctr:
+ info->tfm[0] += AES_TFM_SIZE(SIZE_IN_WORDS(AES_BLOCK_SIZE));
+ info->tfm[1] |= AES_TFM_FULL_IV;
+ info->cmd[cnt++] = AES_CMD2;
+ecb:
+ ctx->ct_size = cnt;
}
static int mtk_aes_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
@@ -572,8 +568,7 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
struct mtk_aes_ctr_ctx *cctx = mtk_aes_ctr_ctx_cast(ctx);
struct ablkcipher_request *req = ablkcipher_request_cast(aes->areq);
struct scatterlist *src, *dst;
- int i;
- u32 start, end, ctr, blocks, *iv_state;
+ u32 start, end, ctr, blocks;
size_t datalen;
bool fragmented = false;
@@ -602,9 +597,8 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
scatterwalk_ffwd(cctx->dst, req->dst, cctx->offset));
/* Write IVs into transform state buffer. */
- iv_state = ctx->tfm.state + ctx->keylen;
- for (i = 0; i < SIZE_IN_WORDS(AES_BLOCK_SIZE); i++)
- iv_state[i] = cpu_to_le32(cctx->iv[i]);
+ mtk_aes_write_state_le(ctx->info.state + ctx->keylen, cctx->iv,
+ AES_BLOCK_SIZE);
if (unlikely(fragmented)) {
/*
@@ -639,21 +633,25 @@ static int mtk_aes_setkey(struct crypto_ablkcipher *tfm,
const u8 *key, u32 keylen)
{
struct mtk_aes_base_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- const u32 *aes_key = (const u32 *)key;
- u32 *key_state = ctx->tfm.state;
- int i;
- if (keylen != AES_KEYSIZE_128 &&
- keylen != AES_KEYSIZE_192 &&
- keylen != AES_KEYSIZE_256) {
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ ctx->keymode = AES_TFM_128BITS;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->keymode = AES_TFM_192BITS;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->keymode = AES_TFM_256BITS;
+ break;
+
+ default:
crypto_ablkcipher_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
ctx->keylen = SIZE_IN_WORDS(keylen);
-
- for (i = 0; i < ctx->keylen; i++)
- key_state[i] = cpu_to_le32(aes_key[i]);
+ mtk_aes_write_state_le(ctx->info.state, (const u32 *)key, keylen);
return 0;
}
@@ -825,45 +823,35 @@ static void mtk_aes_gcm_info_init(struct mtk_cryp *cryp,
struct aead_request *req = aead_request_cast(aes->areq);
struct mtk_aes_base_ctx *ctx = aes->ctx;
struct mtk_aes_gcm_ctx *gctx = mtk_aes_gcm_ctx_cast(ctx);
- const u32 *iv = (const u32 *)req->iv;
- u32 *iv_state = ctx->tfm.state + ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE);
+ struct mtk_aes_info *info = &ctx->info;
u32 ivsize = crypto_aead_ivsize(crypto_aead_reqtfm(req));
- int i;
+ u32 cnt = 0;
ctx->ct_hdr = AES_CT_CTRL_HDR | len;
- ctx->ct.cmd[0] = AES_GCM_CMD0 | cpu_to_le32(req->assoclen);
- ctx->ct.cmd[1] = AES_GCM_CMD1 | cpu_to_le32(req->assoclen);
- ctx->ct.cmd[2] = AES_GCM_CMD2;
- ctx->ct.cmd[3] = AES_GCM_CMD3 | cpu_to_le32(gctx->textlen);
+ info->cmd[cnt++] = AES_GCM_CMD0 | cpu_to_le32(req->assoclen);
+ info->cmd[cnt++] = AES_GCM_CMD1 | cpu_to_le32(req->assoclen);
+ info->cmd[cnt++] = AES_GCM_CMD2;
+ info->cmd[cnt++] = AES_GCM_CMD3 | cpu_to_le32(gctx->textlen);
if (aes->flags & AES_FLAGS_ENCRYPT) {
- ctx->ct.cmd[4] = AES_GCM_CMD4 | cpu_to_le32(gctx->authsize);
- ctx->ct_size = AES_CT_SIZE_GCM_OUT;
- ctx->tfm.ctrl[0] = AES_TFM_GCM_OUT;
+ info->cmd[cnt++] = AES_GCM_CMD4 | cpu_to_le32(gctx->authsize);
+ info->tfm[0] = AES_TFM_GCM_OUT;
} else {
- ctx->ct.cmd[4] = AES_GCM_CMD5 | cpu_to_le32(gctx->authsize);
- ctx->ct.cmd[5] = AES_GCM_CMD6 | cpu_to_le32(gctx->authsize);
- ctx->ct_size = AES_CT_SIZE_GCM_IN;
- ctx->tfm.ctrl[0] = AES_TFM_GCM_IN;
+ info->cmd[cnt++] = AES_GCM_CMD5 | cpu_to_le32(gctx->authsize);
+ info->cmd[cnt++] = AES_GCM_CMD6 | cpu_to_le32(gctx->authsize);
+ info->tfm[0] = AES_TFM_GCM_IN;
}
+ ctx->ct_size = cnt;
- if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_128))
- ctx->tfm.ctrl[0] |= AES_TFM_128BITS;
- else if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_256))
- ctx->tfm.ctrl[0] |= AES_TFM_256BITS;
- else
- ctx->tfm.ctrl[0] |= AES_TFM_192BITS;
-
- ctx->tfm.ctrl[0] |= AES_TFM_GHASH_DIG | AES_TFM_GHASH |
- AES_TFM_SIZE(ctx->keylen + SIZE_IN_WORDS(
- AES_BLOCK_SIZE + ivsize));
- ctx->tfm.ctrl[1] = AES_TFM_CTR_INIT | AES_TFM_IV_CTR_MODE |
- AES_TFM_3IV | AES_TFM_ENC_HASH;
+ info->tfm[0] |= AES_TFM_GHASH_DIGEST | AES_TFM_GHASH | AES_TFM_SIZE(
+ ctx->keylen + SIZE_IN_WORDS(AES_BLOCK_SIZE + ivsize)) |
+ ctx->keymode;
+ info->tfm[1] = AES_TFM_CTR_INIT | AES_TFM_IV_CTR_MODE | AES_TFM_3IV |
+ AES_TFM_ENC_HASH;
- for (i = 0; i < SIZE_IN_WORDS(ivsize); i++)
- iv_state[i] = cpu_to_le32(iv[i]);
+ mtk_aes_write_state_le(info->state + ctx->keylen + SIZE_IN_WORDS(
+ AES_BLOCK_SIZE), (const u32 *)req->iv, ivsize);
}
static int mtk_aes_gcm_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
@@ -979,24 +967,26 @@ static int mtk_aes_gcm_setkey(struct crypto_aead *aead, const u8 *key,
struct scatterlist sg[1];
struct skcipher_request req;
} *data;
- const u32 *aes_key;
- u32 *key_state, *hash_state;
- int err, i;
+ int err;
- if (keylen != AES_KEYSIZE_256 &&
- keylen != AES_KEYSIZE_192 &&
- keylen != AES_KEYSIZE_128) {
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ ctx->keymode = AES_TFM_128BITS;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->keymode = AES_TFM_192BITS;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->keymode = AES_TFM_256BITS;
+ break;
+
+ default:
crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- key_state = ctx->tfm.state;
- aes_key = (u32 *)key;
ctx->keylen = SIZE_IN_WORDS(keylen);
- for (i = 0; i < ctx->keylen; i++)
- ctx->tfm.state[i] = cpu_to_le32(aes_key[i]);
-
/* Same as crypto_gcm_setkey() from crypto/gcm.c */
crypto_skcipher_clear_flags(ctr, CRYPTO_TFM_REQ_MASK);
crypto_skcipher_set_flags(ctr, crypto_aead_get_flags(aead) &
@@ -1031,10 +1021,11 @@ static int mtk_aes_gcm_setkey(struct crypto_aead *aead, const u8 *key,
if (err)
goto out;
- hash_state = key_state + ctx->keylen;
-
- for (i = 0; i < 4; i++)
- hash_state[i] = cpu_to_be32(data->hash[i]);
+ /* Write key into state buffer */
+ mtk_aes_write_state_le(ctx->info.state, (const u32 *)key, keylen);
+ /* Write key(H) into state buffer */
+ mtk_aes_write_state_be(ctx->info.state + ctx->keylen, data->hash,
+ AES_BLOCK_SIZE);
out:
kzfree(data);
return err;
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index dd3582b..2226f12 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -23,6 +23,7 @@
#define SHA_OP_FINAL 2
#define SHA_DATA_LEN_MSK cpu_to_le32(GENMASK(16, 0))
+#define SHA_MAX_DIGEST_BUF_SIZE 32
/* SHA command token */
#define SHA_CT_SIZE 5
@@ -33,7 +34,6 @@
/* SHA transform information */
#define SHA_TFM_HASH cpu_to_le32(0x2 << 0)
-#define SHA_TFM_INNER_DIG cpu_to_le32(0x1 << 21)
#define SHA_TFM_SIZE(x) cpu_to_le32((x) << 8)
#define SHA_TFM_START cpu_to_le32(0x1 << 4)
#define SHA_TFM_CONTINUE cpu_to_le32(0x1 << 5)
@@ -60,31 +60,17 @@
#define SHA_FLAGS_PAD BIT(10)
/**
- * mtk_sha_ct is a set of hardware instructions(command token)
- * that are used to control engine's processing flow of SHA,
- * and it contains the first two words of transform state.
+ * mtk_sha_info - hardware information of AES
+ * @cmd: command token, hardware instruction
+ * @tfm: transform state of cipher algorithm.
+ * @state: contains keys and initial vectors.
+ *
*/
-struct mtk_sha_ct {
+struct mtk_sha_info {
__le32 ctrl[2];
__le32 cmd[3];
-};
-
-/**
- * mtk_sha_tfm is used to define SHA transform state
- * and store result digest that produced by engine.
- */
-struct mtk_sha_tfm {
- __le32 ctrl[2];
- __le32 digest[SIZE_IN_WORDS(SHA512_DIGEST_SIZE)];
-};
-
-/**
- * mtk_sha_info consists of command token and transform state
- * of SHA, its role is similar to mtk_aes_info.
- */
-struct mtk_sha_info {
- struct mtk_sha_ct ct;
- struct mtk_sha_tfm tfm;
+ __le32 tfm[2];
+ __le32 digest[SHA_MAX_DIGEST_BUF_SIZE];
};
struct mtk_sha_reqctx {
@@ -93,7 +79,6 @@ struct mtk_sha_reqctx {
unsigned long op;
u64 digcnt;
- bool start;
size_t bufcnt;
dma_addr_t dma_addr;
@@ -265,7 +250,9 @@ static void mtk_sha_fill_padding(struct mtk_sha_reqctx *ctx, u32 len)
bits[1] = cpu_to_be64(size << 3);
bits[0] = cpu_to_be64(size >> 61);
- if (ctx->flags & (SHA_FLAGS_SHA384 | SHA_FLAGS_SHA512)) {
+ switch (ctx->flags & SHA_FLAGS_ALGO_MSK) {
+ case SHA_FLAGS_SHA384:
+ case SHA_FLAGS_SHA512:
index = ctx->bufcnt & 0x7f;
padlen = (index < 112) ? (112 - index) : ((128 + 112) - index);
*(ctx->buffer + ctx->bufcnt) = 0x80;
@@ -273,7 +260,9 @@ static void mtk_sha_fill_padding(struct mtk_sha_reqctx *ctx, u32 len)
memcpy(ctx->buffer + ctx->bufcnt + padlen, bits, 16);
ctx->bufcnt += padlen + 16;
ctx->flags |= SHA_FLAGS_PAD;
- } else {
+ break;
+
+ default:
index = ctx->bufcnt & 0x3f;
padlen = (index < 56) ? (56 - index) : ((64 + 56) - index);
*(ctx->buffer + ctx->bufcnt) = 0x80;
@@ -281,36 +270,35 @@ static void mtk_sha_fill_padding(struct mtk_sha_reqctx *ctx, u32 len)
memcpy(ctx->buffer + ctx->bufcnt + padlen, &bits[1], 8);
ctx->bufcnt += padlen + 8;
ctx->flags |= SHA_FLAGS_PAD;
+ break;
}
}
/* Initialize basic transform information of SHA */
static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
{
- struct mtk_sha_ct *ct = &ctx->info.ct;
- struct mtk_sha_tfm *tfm = &ctx->info.tfm;
+ struct mtk_sha_info *info = &ctx->info;
ctx->ct_hdr = SHA_CT_CTRL_HDR;
ctx->ct_size = SHA_CT_SIZE;
- tfm->ctrl[0] = SHA_TFM_HASH | SHA_TFM_INNER_DIG |
- SHA_TFM_SIZE(SIZE_IN_WORDS(ctx->ds));
+ info->tfm[0] = SHA_TFM_HASH | SHA_TFM_SIZE(SIZE_IN_WORDS(ctx->ds));
switch (ctx->flags & SHA_FLAGS_ALGO_MSK) {
case SHA_FLAGS_SHA1:
- tfm->ctrl[0] |= SHA_TFM_SHA1;
+ info->tfm[0] |= SHA_TFM_SHA1;
break;
case SHA_FLAGS_SHA224:
- tfm->ctrl[0] |= SHA_TFM_SHA224;
+ info->tfm[0] |= SHA_TFM_SHA224;
break;
case SHA_FLAGS_SHA256:
- tfm->ctrl[0] |= SHA_TFM_SHA256;
+ info->tfm[0] |= SHA_TFM_SHA256;
break;
case SHA_FLAGS_SHA384:
- tfm->ctrl[0] |= SHA_TFM_SHA384;
+ info->tfm[0] |= SHA_TFM_SHA384;
break;
case SHA_FLAGS_SHA512:
- tfm->ctrl[0] |= SHA_TFM_SHA512;
+ info->tfm[0] |= SHA_TFM_SHA512;
break;
default:
@@ -318,13 +306,13 @@ static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
return;
}
- tfm->ctrl[1] = SHA_TFM_HASH_STORE;
- ct->ctrl[0] = tfm->ctrl[0] | SHA_TFM_CONTINUE | SHA_TFM_START;
- ct->ctrl[1] = tfm->ctrl[1];
+ info->tfm[1] = SHA_TFM_HASH_STORE;
+ info->ctrl[0] = info->tfm[0] | SHA_TFM_CONTINUE | SHA_TFM_START;
+ info->ctrl[1] = info->tfm[1];
- ct->cmd[0] = SHA_CMD0;
- ct->cmd[1] = SHA_CMD1;
- ct->cmd[2] = SHA_CMD2 | SHA_TFM_DIGEST(SIZE_IN_WORDS(ctx->ds));
+ info->cmd[0] = SHA_CMD0;
+ info->cmd[1] = SHA_CMD1;
+ info->cmd[2] = SHA_CMD2 | SHA_TFM_DIGEST(SIZE_IN_WORDS(ctx->ds));
}
/*
@@ -337,17 +325,15 @@ static int mtk_sha_info_update(struct mtk_cryp *cryp,
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_sha_info *info = &ctx->info;
- struct mtk_sha_ct *ct = &info->ct;
-
- if (ctx->start)
- ctx->start = false;
- else
- ct->ctrl[0] &= ~SHA_TFM_START;
ctx->ct_hdr &= ~SHA_DATA_LEN_MSK;
ctx->ct_hdr |= cpu_to_le32(len1 + len2);
- ct->cmd[0] &= ~SHA_DATA_LEN_MSK;
- ct->cmd[0] |= cpu_to_le32(len1 + len2);
+ info->cmd[0] &= ~SHA_DATA_LEN_MSK;
+ info->cmd[0] |= cpu_to_le32(len1 + len2);
+
+ /* Setting SHA_TFM_START only for the first iteration */
+ if (ctx->digcnt)
+ info->ctrl[0] &= ~SHA_TFM_START;
ctx->digcnt += len1;
@@ -357,7 +343,8 @@ static int mtk_sha_info_update(struct mtk_cryp *cryp,
dev_err(cryp->dev, "dma %zu bytes error\n", sizeof(*info));
return -EINVAL;
}
- ctx->tfm_dma = ctx->ct_dma + sizeof(*ct);
+
+ ctx->tfm_dma = ctx->ct_dma + sizeof(info->ctrl) + sizeof(info->cmd);
return 0;
}
@@ -422,7 +409,6 @@ static int mtk_sha_init(struct ahash_request *req)
ctx->bufcnt = 0;
ctx->digcnt = 0;
ctx->buffer = tctx->buf;
- ctx->start = true;
if (tctx->flags & SHA_FLAGS_HMAC) {
struct mtk_sha_hmac_ctx *bctx = tctx->base;
@@ -635,7 +621,7 @@ static int mtk_sha_final_req(struct mtk_cryp *cryp,
static int mtk_sha_finish(struct ahash_request *req)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(req);
- u32 *digest = ctx->info.tfm.digest;
+ __le32 *digest = ctx->info.digest;
u32 *result = (u32 *)req->result;
int i;
--
1.9.1
^ permalink raw reply related
* [PATCH 6/9] crypto: mediatek - fix error handling in mtk_aes_complete()
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch fixes how errors should be handled by mtk_aes_complete().
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 39 +++++++++++++++++++++++----------------
1 file changed, 23 insertions(+), 16 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 140e9a3..6a0180d 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -248,6 +248,17 @@ static inline void mtk_aes_restore_sg(const struct mtk_aes_dma *dma)
sg->length += dma->remainder;
}
+static inline int mtk_aes_complete(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes,
+ int err)
+{
+ aes->flags &= ~AES_FLAGS_BUSY;
+ aes->areq->complete(aes->areq, err);
+ /* Handle new request */
+ tasklet_schedule(&aes->queue_task);
+ return err;
+}
+
/*
* Write descriptors for processing. This will configure the engine, load
* the transform information and then start the packet processing.
@@ -352,7 +363,7 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
ctx->ct_dma = dma_map_single(cryp->dev, &ctx->ct, sizeof(ctx->ct),
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(cryp->dev, ctx->ct_dma)))
- return -EINVAL;
+ goto exit;
ctx->tfm_dma = dma_map_single(cryp->dev, &ctx->tfm, sizeof(ctx->tfm),
DMA_TO_DEVICE);
@@ -389,8 +400,8 @@ static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
tfm_map_err:
dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
DMA_TO_DEVICE);
-
- return -EINVAL;
+exit:
+ return mtk_aes_complete(cryp, aes, -EINVAL);
}
/* Initialize transform information of CBC/ECB/CTR mode */
@@ -467,7 +478,7 @@ static int mtk_aes_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
padlen = mtk_aes_padlen(len);
if (len + padlen > AES_BUF_SIZE)
- return -ENOMEM;
+ return mtk_aes_complete(cryp, aes, -ENOMEM);
if (!src_aligned) {
sg_copy_to_buffer(src, sg_nents(src), aes->buf, len);
@@ -527,14 +538,10 @@ static int mtk_aes_handle_queue(struct mtk_cryp *cryp, u8 id,
return ctx->start(cryp, aes);
}
-static int mtk_aes_complete(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
+static int mtk_aes_transfer_complete(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes)
{
- aes->flags &= ~AES_FLAGS_BUSY;
- aes->areq->complete(aes->areq, 0);
-
- /* Handle new request */
- tasklet_schedule(&aes->queue_task);
- return 0;
+ return mtk_aes_complete(cryp, aes, 0);
}
static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
@@ -543,7 +550,7 @@ static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
struct mtk_aes_reqctx *rctx = ablkcipher_request_ctx(req);
mtk_aes_set_mode(aes, rctx);
- aes->resume = mtk_aes_complete;
+ aes->resume = mtk_aes_transfer_complete;
return mtk_aes_dma(cryp, aes, req->src, req->dst, req->nbytes);
}
@@ -568,7 +575,7 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
/* Check for transfer completion. */
cctx->offset += aes->total;
if (cctx->offset >= req->nbytes)
- return mtk_aes_complete(cryp, aes);
+ return mtk_aes_transfer_complete(cryp, aes);
/* Compute data length. */
datalen = req->nbytes - cctx->offset;
@@ -602,7 +609,6 @@ static int mtk_aes_ctr_transfer(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
cctx->iv[3] = cpu_to_be32(ctr);
crypto_inc((u8 *)cctx->iv, AES_BLOCK_SIZE);
}
- aes->resume = mtk_aes_ctr_transfer;
return mtk_aes_dma(cryp, aes, src, dst, datalen);
}
@@ -618,6 +624,7 @@ static int mtk_aes_ctr_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
memcpy(cctx->iv, req->info, AES_BLOCK_SIZE);
cctx->offset = 0;
aes->total = 0;
+ aes->resume = mtk_aes_ctr_transfer;
return mtk_aes_ctr_transfer(cryp, aes);
}
@@ -859,7 +866,7 @@ static int mtk_aes_gcm_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
if (!src_aligned || !dst_aligned) {
if (aes->total > AES_BUF_SIZE)
- return -ENOMEM;
+ return mtk_aes_complete(cryp, aes, -ENOMEM);
if (!src_aligned) {
sg_copy_to_buffer(src, sg_nents(src), aes->buf, len);
@@ -905,7 +912,7 @@ static int mtk_aes_gcm_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
aes->total = len;
gctx->textlen = req->cryptlen - gctx->authsize;
}
- aes->resume = mtk_aes_complete;
+ aes->resume = mtk_aes_transfer_complete;
return mtk_aes_gcm_dma(cryp, aes, req->src, req->dst, len);
}
--
1.9.1
^ permalink raw reply related
* [PATCH 7/9] crypto: mediatek - add mtk_aes_gcm_tag_verify()
From: Ryder Lee @ 2017-02-20 9:27 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch adds mtk_aes_gcm_tag_verify() which is used to compare
authenticated tag.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 24 ++++++++++++++++++++++--
drivers/crypto/mediatek/mtk-platform.h | 2 ++
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 6a0180d..8f3efa5 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -70,6 +70,8 @@
#define AES_FLAGS_ENCRYPT BIT(4)
#define AES_FLAGS_BUSY BIT(5)
+#define AES_AUTH_TAG_ERR cpu_to_le32(BIT(26))
+
/**
* Command token(CT) is a set of hardware instructions that
* are used to control engine's processing flow of AES.
@@ -306,6 +308,9 @@ static int mtk_aes_xmit(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
}
res->hdr |= MTK_DESC_LAST;
+ /* Pointer to current result descriptor */
+ ring->res_prev = res;
+
/* Prepare enough space for authenticated tag */
if (aes->flags & AES_FLAGS_GCM)
res->hdr += AES_BLOCK_SIZE;
@@ -799,6 +804,19 @@ static int mtk_aes_ctr_cra_init(struct crypto_tfm *tfm)
return container_of(ctx, struct mtk_aes_gcm_ctx, base);
}
+/*
+ * Engine will verify and compare tag automatically, so we just need
+ * to check returned status which stored in the result descriptor.
+ */
+static int mtk_aes_gcm_tag_verify(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes)
+{
+ u32 status = cryp->ring[aes->id]->res_prev->ct;
+
+ return mtk_aes_complete(cryp, aes, (status & AES_AUTH_TAG_ERR) ?
+ -EBADMSG : 0);
+}
+
/* Initialize transform information of GCM mode */
static void mtk_aes_gcm_info_init(struct mtk_cryp *cryp,
struct mtk_aes_rec *aes,
@@ -902,6 +920,8 @@ static int mtk_aes_gcm_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
if (aes->flags & AES_FLAGS_ENCRYPT) {
u32 tag[4];
+
+ aes->resume = mtk_aes_transfer_complete;
/* Compute total process length. */
aes->total = len + gctx->authsize;
/* Compute text length. */
@@ -909,10 +929,10 @@ static int mtk_aes_gcm_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
/* Hardware will append authenticated tag to output buffer */
scatterwalk_map_and_copy(tag, req->dst, len, gctx->authsize, 1);
} else {
+ aes->resume = mtk_aes_gcm_tag_verify;
aes->total = len;
gctx->textlen = req->cryptlen - gctx->authsize;
}
- aes->resume = mtk_aes_transfer_complete;
return mtk_aes_gcm_dma(cryp, aes, req->src, req->dst, len);
}
@@ -925,7 +945,7 @@ static int mtk_aes_gcm_crypt(struct aead_request *req, u64 mode)
rctx->mode = AES_FLAGS_GCM | mode;
return mtk_aes_handle_queue(ctx->cryp, !!(mode & AES_FLAGS_ENCRYPT),
- &req->base);
+ &req->base);
}
static void mtk_gcm_setkey_done(struct crypto_async_request *req, int err)
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index cc98c2c..303c152 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -88,6 +88,7 @@ struct mtk_desc {
* @cmd_dma: DMA address of command descriptor ring
* @res_base: pointer to result descriptor ring base
* @res_next: pointer to the next result descriptor
+ * @res_prev: pointer to the previous result descriptor
* @res_dma: DMA address of result descriptor ring
*
* A descriptor ring is a circular buffer that is used to manage
@@ -100,6 +101,7 @@ struct mtk_ring {
dma_addr_t cmd_dma;
struct mtk_desc *res_base;
struct mtk_desc *res_next;
+ struct mtk_desc *res_prev;
dma_addr_t res_dma;
};
--
1.9.1
^ permalink raw reply related
* [PATCH 5/9] crypto: mediatek - add queue_task tasklet
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch adds 'queue_task' to dequeue crypto requset. This will help to
avoid directly calling mtk_aes_handle_queue() / mtk_sha_handle_queue()
from done tasklet or error handler.
In order to avoid confusion, the new code properly renames DMA completion
"task" to "done_task".
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 20 ++++++++++++++++----
drivers/crypto/mediatek/mtk-platform.h | 12 ++++++++----
drivers/crypto/mediatek/mtk-sha.c | 19 +++++++++++++++----
3 files changed, 39 insertions(+), 12 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 21f3e59..140e9a3 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -533,7 +533,8 @@ static int mtk_aes_complete(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
aes->areq->complete(aes->areq, 0);
/* Handle new request */
- return mtk_aes_handle_queue(cryp, aes->id, NULL);
+ tasklet_schedule(&aes->queue_task);
+ return 0;
}
static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
@@ -1094,6 +1095,13 @@ static void mtk_aes_gcm_exit(struct crypto_aead *aead)
},
};
+static void mtk_aes_queue_task(unsigned long data)
+{
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
+
+ mtk_aes_handle_queue(aes->cryp, aes->id, NULL);
+}
+
static void mtk_aes_done_task(unsigned long data)
{
struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
@@ -1116,7 +1124,7 @@ static irqreturn_t mtk_aes_irq(int irq, void *dev_id)
mtk_aes_write(cryp, RDR_THRESH(aes->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
- tasklet_schedule(&aes->task);
+ tasklet_schedule(&aes->done_task);
} else {
dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
}
@@ -1149,7 +1157,9 @@ static int mtk_aes_record_init(struct mtk_cryp *cryp)
spin_lock_init(&aes[i]->lock);
crypto_init_queue(&aes[i]->queue, AES_QUEUE_SIZE);
- tasklet_init(&aes[i]->task, mtk_aes_done_task,
+ tasklet_init(&aes[i]->queue_task, mtk_aes_queue_task,
+ (unsigned long)aes[i]);
+ tasklet_init(&aes[i]->done_task, mtk_aes_done_task,
(unsigned long)aes[i]);
}
@@ -1173,7 +1183,9 @@ static void mtk_aes_record_free(struct mtk_cryp *cryp)
int i;
for (i = 0; i < MTK_REC_NUM; i++) {
- tasklet_kill(&cryp->aes[i]->task);
+ tasklet_kill(&cryp->aes[i]->done_task);
+ tasklet_kill(&cryp->aes[i]->queue_task);
+
free_page((unsigned long)cryp->aes[i]->buf);
kfree(cryp->aes[i]);
}
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index 218e30d..cc98c2c 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -128,7 +128,8 @@ struct mtk_aes_dma {
* @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @areq: pointer to async request
- * @task: the tasklet is use in AES interrupt
+ * @done_task: the tasklet is use in AES interrupt
+ * @queue_task: the tasklet is used to dequeue request
* @ctx: pointer to current context
* @src: the structure that holds source sg list info
* @dst: the structure that holds destination sg list info
@@ -147,7 +148,8 @@ struct mtk_aes_rec {
struct mtk_cryp *cryp;
struct crypto_queue queue;
struct crypto_async_request *areq;
- struct tasklet_struct task;
+ struct tasklet_struct done_task;
+ struct tasklet_struct queue_task;
struct mtk_aes_base_ctx *ctx;
struct mtk_aes_dma src;
struct mtk_aes_dma dst;
@@ -171,7 +173,8 @@ struct mtk_aes_rec {
* @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @req: pointer to ahash request
- * @task: the tasklet is use in SHA interrupt
+ * @done_task: the tasklet is use in SHA interrupt
+ * @queue_task: the tasklet is used to dequeue request
* @id: the current use of ring
* @flags: it's describing SHA operation state
* @lock: the async queue lock
@@ -182,7 +185,8 @@ struct mtk_sha_rec {
struct mtk_cryp *cryp;
struct crypto_queue queue;
struct ahash_request *req;
- struct tasklet_struct task;
+ struct tasklet_struct done_task;
+ struct tasklet_struct queue_task;
u8 id;
unsigned long flags;
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 0884d62..dd3582b 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -661,7 +661,7 @@ static void mtk_sha_finish_req(struct mtk_cryp *cryp,
sha->req->base.complete(&sha->req->base, err);
/* Handle new request */
- mtk_sha_handle_queue(cryp, sha->id - MTK_RING2, NULL);
+ tasklet_schedule(&sha->queue_task);
}
static int mtk_sha_handle_queue(struct mtk_cryp *cryp, u8 id,
@@ -1183,6 +1183,13 @@ static void mtk_sha_cra_exit(struct crypto_tfm *tfm)
},
};
+static void mtk_sha_queue_task(unsigned long data)
+{
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
+
+ mtk_sha_handle_queue(sha->cryp, sha->id - MTK_RING2, NULL);
+}
+
static void mtk_sha_done_task(unsigned long data)
{
struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
@@ -1205,7 +1212,7 @@ static irqreturn_t mtk_sha_irq(int irq, void *dev_id)
mtk_sha_write(cryp, RDR_THRESH(sha->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
- tasklet_schedule(&sha->task);
+ tasklet_schedule(&sha->done_task);
} else {
dev_warn(cryp->dev, "SHA interrupt when no active requests.\n");
}
@@ -1231,7 +1238,9 @@ static int mtk_sha_record_init(struct mtk_cryp *cryp)
spin_lock_init(&sha[i]->lock);
crypto_init_queue(&sha[i]->queue, SHA_QUEUE_SIZE);
- tasklet_init(&sha[i]->task, mtk_sha_done_task,
+ tasklet_init(&sha[i]->queue_task, mtk_sha_queue_task,
+ (unsigned long)sha[i]);
+ tasklet_init(&sha[i]->done_task, mtk_sha_done_task,
(unsigned long)sha[i]);
}
@@ -1254,7 +1263,9 @@ static void mtk_sha_record_free(struct mtk_cryp *cryp)
int i;
for (i = 0; i < MTK_REC_NUM; i++) {
- tasklet_kill(&cryp->sha[i]->task);
+ tasklet_kill(&cryp->sha[i]->done_task);
+ tasklet_kill(&cryp->sha[i]->queue_task);
+
kfree(cryp->sha[i]);
}
}
--
1.9.1
^ permalink raw reply related
* [PATCH 2/9] crypto: mediatek - add MTK_* prefix and correct annotations.
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
Dummy patch to add MTK_* prefix to ring enum and fix incorrect annotations.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 12 ++++++------
drivers/crypto/mediatek/mtk-platform.c | 12 ++++++------
drivers/crypto/mediatek/mtk-platform.h | 26 +++++++++++++-------------
drivers/crypto/mediatek/mtk-sha.c | 14 +++++++-------
4 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index e67881f..b57b68f 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -1152,8 +1152,8 @@ static int mtk_aes_record_init(struct mtk_cryp *cryp)
}
/* Link to ring0 and ring1 respectively */
- aes[0]->id = RING0;
- aes[1]->id = RING1;
+ aes[0]->id = MTK_RING0;
+ aes[1]->id = MTK_RING1;
return 0;
@@ -1221,14 +1221,14 @@ int mtk_cipher_alg_register(struct mtk_cryp *cryp)
if (ret)
goto err_record;
- ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_irq,
+ ret = devm_request_irq(cryp->dev, cryp->irq[MTK_RING0], mtk_aes_irq,
0, "mtk-aes", cryp->aes[0]);
if (ret) {
dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
- ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_irq,
+ ret = devm_request_irq(cryp->dev, cryp->irq[MTK_RING1], mtk_aes_irq,
0, "mtk-aes", cryp->aes[1]);
if (ret) {
dev_err(cryp->dev, "unable to request AES irq.\n");
@@ -1236,8 +1236,8 @@ int mtk_cipher_alg_register(struct mtk_cryp *cryp)
}
/* Enable ring0 and ring1 interrupt */
- mtk_aes_write(cryp, AIC_ENABLE_SET(RING0), MTK_IRQ_RDR0);
- mtk_aes_write(cryp, AIC_ENABLE_SET(RING1), MTK_IRQ_RDR1);
+ mtk_aes_write(cryp, AIC_ENABLE_SET(MTK_RING0), MTK_IRQ_RDR0);
+ mtk_aes_write(cryp, AIC_ENABLE_SET(MTK_RING1), MTK_IRQ_RDR1);
spin_lock(&mtk_aes.lock);
list_add_tail(&cryp->aes_list, &mtk_aes.dev_list);
diff --git a/drivers/crypto/mediatek/mtk-platform.c b/drivers/crypto/mediatek/mtk-platform.c
index a9c713d..50de335 100644
--- a/drivers/crypto/mediatek/mtk-platform.c
+++ b/drivers/crypto/mediatek/mtk-platform.c
@@ -334,7 +334,7 @@ static int mtk_packet_engine_setup(struct mtk_cryp *cryp)
/* Enable the 4 rings for the packet engines. */
mtk_desc_ring_link(cryp, 0xf);
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
mtk_cmd_desc_ring_setup(cryp, i, &cap);
mtk_res_desc_ring_setup(cryp, i, &cap);
}
@@ -359,7 +359,7 @@ static int mtk_aic_cap_check(struct mtk_cryp *cryp, int hw)
{
u32 val;
- if (hw == RING_MAX)
+ if (hw == MTK_RING_MAX)
val = readl(cryp->base + AIC_G_VERSION);
else
val = readl(cryp->base + AIC_VERSION(hw));
@@ -368,7 +368,7 @@ static int mtk_aic_cap_check(struct mtk_cryp *cryp, int hw)
if (val != MTK_AIC_VER11 && val != MTK_AIC_VER12)
return -ENXIO;
- if (hw == RING_MAX)
+ if (hw == MTK_RING_MAX)
val = readl(cryp->base + AIC_G_OPTIONS);
else
val = readl(cryp->base + AIC_OPTIONS(hw));
@@ -389,7 +389,7 @@ static int mtk_aic_init(struct mtk_cryp *cryp, int hw)
return err;
/* Disable all interrupts and set initial configuration */
- if (hw == RING_MAX) {
+ if (hw == MTK_RING_MAX) {
writel(0, cryp->base + AIC_G_ENABLE_CTRL);
writel(0, cryp->base + AIC_G_POL_CTRL);
writel(0, cryp->base + AIC_G_TYPE_CTRL);
@@ -431,7 +431,7 @@ static void mtk_desc_dma_free(struct mtk_cryp *cryp)
{
int i;
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
dma_free_coherent(cryp->dev, MTK_DESC_RING_SZ,
cryp->ring[i]->res_base,
cryp->ring[i]->res_dma);
@@ -447,7 +447,7 @@ static int mtk_desc_ring_alloc(struct mtk_cryp *cryp)
struct mtk_ring **ring = cryp->ring;
int i, err = ENOMEM;
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
ring[i] = kzalloc(sizeof(**ring), GFP_KERNEL);
if (!ring[i])
goto err_cleanup;
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index d2a1db6..3bbe9b0 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -38,14 +38,14 @@
* Ring 2/3 are used by SHA.
*/
enum {
- RING0 = 0,
- RING1,
- RING2,
- RING3,
- RING_MAX,
+ MTK_RING0,
+ MTK_RING1,
+ MTK_RING2,
+ MTK_RING3,
+ MTK_RING_MAX
};
-#define MTK_REC_NUM (RING_MAX / 2)
+#define MTK_REC_NUM (MTK_RING_MAX / 2)
#define MTK_IRQ_NUM 5
/**
@@ -137,7 +137,7 @@ struct mtk_aes_dma {
* @resume: pointer to resume function
* @total: request buffer length
* @buf: pointer to page buffer
- * @id: record identification
+ * @id: the current use of ring
* @flags: it's describing AES operation state
* @lock: the async queue lock
*
@@ -172,9 +172,9 @@ struct mtk_aes_rec {
* @queue: crypto request queue
* @req: pointer to ahash request
* @task: the tasklet is use in SHA interrupt
- * @id: record identification
+ * @id: the current use of ring
* @flags: it's describing SHA operation state
- * @lock: the ablkcipher queue lock
+ * @lock: the async queue lock
*
* Structure used to record SHA execution state.
*/
@@ -197,9 +197,9 @@ struct mtk_sha_rec {
* @clk_ethif: pointer to ethif clock
* @clk_cryp: pointer to crypto clock
* @irq: global system and rings IRQ
- * @ring: pointer to execution state of AES
- * @aes: pointer to execution state of SHA
- * @sha: each execution record map to a ring
+ * @ring: pointer to descriptor rings
+ * @aes: pointer to operation record of AES
+ * @sha: pointer to operation record of SHA
* @aes_list: device list of AES
* @sha_list: device list of SHA
* @tmp: pointer to temporary buffer for internal use
@@ -215,7 +215,7 @@ struct mtk_cryp {
struct clk *clk_cryp;
int irq[MTK_IRQ_NUM];
- struct mtk_ring *ring[RING_MAX];
+ struct mtk_ring *ring[MTK_RING_MAX];
struct mtk_aes_rec *aes[MTK_REC_NUM];
struct mtk_sha_rec *sha[MTK_REC_NUM];
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 1fb98f6..b4f0c62 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -694,7 +694,7 @@ static void mtk_sha_finish_req(struct mtk_cryp *cryp,
sha->req->base.complete(&sha->req->base, err);
/* Handle new request */
- mtk_sha_handle_queue(cryp, sha->id - RING2, NULL);
+ mtk_sha_handle_queue(cryp, sha->id - MTK_RING2, NULL);
}
static int mtk_sha_handle_queue(struct mtk_cryp *cryp, u8 id,
@@ -1269,8 +1269,8 @@ static int mtk_sha_record_init(struct mtk_cryp *cryp)
}
/* Link to ring2 and ring3 respectively */
- sha[0]->id = RING2;
- sha[1]->id = RING3;
+ sha[0]->id = MTK_RING2;
+ sha[1]->id = MTK_RING3;
cryp->rec = 1;
@@ -1343,14 +1343,14 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
if (err)
goto err_record;
- err = devm_request_irq(cryp->dev, cryp->irq[RING2], mtk_sha_irq,
+ err = devm_request_irq(cryp->dev, cryp->irq[MTK_RING2], mtk_sha_irq,
0, "mtk-sha", cryp->sha[0]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq0.\n");
goto err_res;
}
- err = devm_request_irq(cryp->dev, cryp->irq[RING3], mtk_sha_irq,
+ err = devm_request_irq(cryp->dev, cryp->irq[MTK_RING3], mtk_sha_irq,
0, "mtk-sha", cryp->sha[1]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq1.\n");
@@ -1358,8 +1358,8 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
}
/* Enable ring2 and ring3 interrupt for hash */
- mtk_sha_write(cryp, AIC_ENABLE_SET(RING2), MTK_IRQ_RDR2);
- mtk_sha_write(cryp, AIC_ENABLE_SET(RING3), MTK_IRQ_RDR3);
+ mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING2), MTK_IRQ_RDR2);
+ mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING3), MTK_IRQ_RDR3);
cryp->tmp = dma_alloc_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
&cryp->tmp_dma, GFP_KERNEL);
--
1.9.1
^ permalink raw reply related
* [PATCH 3/9] crypto: mediatek - make mtk_sha_xmit() more generic
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This is a transitional patch. It merges mtk_sha_xmit() and mtk_sha_xmit2()
to make transmit function more generic.
In addition, res->buf and cryp->tmp_dma in mtk_sha_xmit() are useless, since
crypto engine writes the result digests into ctx->tfm.digest instead of
res->buf. It's better to remove it.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-platform.h | 4 --
drivers/crypto/mediatek/mtk-sha.c | 116 ++++++++++-----------------------
2 files changed, 34 insertions(+), 86 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index 3bbe9b0..78ce54e 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -202,8 +202,6 @@ struct mtk_sha_rec {
* @sha: pointer to operation record of SHA
* @aes_list: device list of AES
* @sha_list: device list of SHA
- * @tmp: pointer to temporary buffer for internal use
- * @tmp_dma: DMA address of temporary buffer
* @rec: it's used to select SHA record for tfm
*
* Structure storing cryptographic device information.
@@ -222,8 +220,6 @@ struct mtk_cryp {
struct list_head aes_list;
struct list_head sha_list;
- void *tmp;
- dma_addr_t tmp_dma;
bool rec;
};
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index b4f0c62..ef6fb20 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -17,7 +17,6 @@
#define SHA_ALIGN_MSK (sizeof(u32) - 1)
#define SHA_QUEUE_SIZE 512
-#define SHA_TMP_BUF_SIZE 512
#define SHA_BUF_SIZE ((u32)PAGE_SIZE)
#define SHA_OP_UPDATE 1
@@ -319,7 +318,7 @@ static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
*/
static int mtk_sha_info_update(struct mtk_cryp *cryp,
struct mtk_sha_rec *sha,
- size_t len)
+ size_t len1, size_t len2)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_sha_info *info = &ctx->info;
@@ -331,11 +330,11 @@ static int mtk_sha_info_update(struct mtk_cryp *cryp,
ct->ctrl[0] &= ~SHA_TFM_START;
ctx->ct_hdr &= ~SHA_DATA_LEN_MSK;
- ctx->ct_hdr |= cpu_to_le32(len);
+ ctx->ct_hdr |= cpu_to_le32(len1 + len2);
ct->cmd[0] &= ~SHA_DATA_LEN_MSK;
- ct->cmd[0] |= cpu_to_le32(len);
+ ct->cmd[0] |= cpu_to_le32(len1 + len2);
- ctx->digcnt += len;
+ ctx->digcnt += len1;
ctx->ct_dma = dma_map_single(cryp->dev, info, sizeof(*info),
DMA_BIDIRECTIONAL);
@@ -422,67 +421,24 @@ static int mtk_sha_init(struct ahash_request *req)
}
static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
- dma_addr_t addr, size_t len)
+ dma_addr_t addr1, size_t len1,
+ dma_addr_t addr2, size_t len2)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_ring *ring = cryp->ring[sha->id];
struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
struct mtk_desc *res = ring->res_base + ring->res_pos;
- int err;
-
- err = mtk_sha_info_update(cryp, sha, len);
- if (err)
- return err;
-
- /* Fill in the command/result descriptors */
- res->hdr = MTK_DESC_FIRST | MTK_DESC_LAST | MTK_DESC_BUF_LEN(len);
- res->buf = cpu_to_le32(cryp->tmp_dma);
-
- cmd->hdr = MTK_DESC_FIRST | MTK_DESC_LAST | MTK_DESC_BUF_LEN(len) |
- MTK_DESC_CT_LEN(ctx->ct_size);
-
- cmd->buf = cpu_to_le32(addr);
- cmd->ct = cpu_to_le32(ctx->ct_dma);
- cmd->ct_hdr = ctx->ct_hdr;
- cmd->tfm = cpu_to_le32(ctx->tfm_dma);
-
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
- /*
- * Make sure that all changes to the DMA ring are done before we
- * start engine.
- */
- wmb();
- /* Start DMA transfer */
- mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(1));
- mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(1));
-
- return -EINPROGRESS;
-}
-
-static int mtk_sha_xmit2(struct mtk_cryp *cryp,
- struct mtk_sha_rec *sha,
- struct mtk_sha_reqctx *ctx,
- size_t len1, size_t len2)
-{
- struct mtk_ring *ring = cryp->ring[sha->id];
- struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
- struct mtk_desc *res = ring->res_base + ring->res_pos;
- int err;
+ int err, count = 0;
- err = mtk_sha_info_update(cryp, sha, len1 + len2);
+ err = mtk_sha_info_update(cryp, sha, len1, len2);
if (err)
return err;
/* Fill in the command/result descriptors */
- res->hdr = MTK_DESC_BUF_LEN(len1) | MTK_DESC_FIRST;
- res->buf = cpu_to_le32(cryp->tmp_dma);
-
- cmd->hdr = MTK_DESC_BUF_LEN(len1) | MTK_DESC_FIRST |
+ res->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1);
+ cmd->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1) |
MTK_DESC_CT_LEN(ctx->ct_size);
- cmd->buf = cpu_to_le32(sg_dma_address(ctx->sg));
+ cmd->buf = cpu_to_le32(addr1);
cmd->ct = cpu_to_le32(ctx->ct_dma);
cmd->ct_hdr = ctx->ct_hdr;
cmd->tfm = cpu_to_le32(ctx->tfm_dma);
@@ -491,20 +447,25 @@ static int mtk_sha_xmit2(struct mtk_cryp *cryp,
ring->cmd_pos = 0;
ring->res_pos = ring->cmd_pos;
+ count++;
- cmd = ring->cmd_base + ring->cmd_pos;
- res = ring->res_base + ring->res_pos;
+ if (len2) {
+ cmd = ring->cmd_base + ring->cmd_pos;
+ res = ring->res_base + ring->res_pos;
- res->hdr = MTK_DESC_BUF_LEN(len2) | MTK_DESC_LAST;
- res->buf = cpu_to_le32(cryp->tmp_dma);
+ res->hdr = MTK_DESC_BUF_LEN(len2);
+ cmd->hdr = MTK_DESC_BUF_LEN(len2);
+ cmd->buf = cpu_to_le32(addr2);
- cmd->hdr = MTK_DESC_BUF_LEN(len2) | MTK_DESC_LAST;
- cmd->buf = cpu_to_le32(ctx->dma_addr);
+ if (++ring->cmd_pos == MTK_DESC_NUM)
+ ring->cmd_pos = 0;
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
+ ring->res_pos = ring->cmd_pos;
+ count++;
+ }
- ring->res_pos = ring->cmd_pos;
+ cmd->hdr |= MTK_DESC_LAST;
+ res->hdr |= MTK_DESC_LAST;
/*
* Make sure that all changes to the DMA ring are done before we
@@ -512,8 +473,8 @@ static int mtk_sha_xmit2(struct mtk_cryp *cryp,
*/
wmb();
/* Start DMA transfer */
- mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(2));
- mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(2));
+ mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(count));
+ mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(count));
return -EINPROGRESS;
}
@@ -532,7 +493,7 @@ static int mtk_sha_dma_map(struct mtk_cryp *cryp,
ctx->flags &= ~SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count);
+ return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count, 0, 0);
}
static int mtk_sha_update_slow(struct mtk_cryp *cryp,
@@ -625,7 +586,8 @@ static int mtk_sha_update_start(struct mtk_cryp *cryp,
if (len == 0) {
ctx->flags &= ~SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count);
+ return mtk_sha_xmit(cryp, sha, ctx->dma_addr,
+ count, 0, 0);
} else {
ctx->sg = sg;
@@ -635,7 +597,8 @@ static int mtk_sha_update_start(struct mtk_cryp *cryp,
}
ctx->flags |= SHA_FLAGS_SG;
- return mtk_sha_xmit2(cryp, sha, ctx, len, count);
+ return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg),
+ len, ctx->dma_addr, count);
}
}
@@ -646,7 +609,8 @@ static int mtk_sha_update_start(struct mtk_cryp *cryp,
ctx->flags |= SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg), len);
+ return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg),
+ len, 0, 0);
}
static int mtk_sha_final_req(struct mtk_cryp *cryp,
@@ -1361,14 +1325,6 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING2), MTK_IRQ_RDR2);
mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING3), MTK_IRQ_RDR3);
- cryp->tmp = dma_alloc_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- &cryp->tmp_dma, GFP_KERNEL);
- if (!cryp->tmp) {
- dev_err(cryp->dev, "unable to allocate tmp buffer.\n");
- err = -EINVAL;
- goto err_res;
- }
-
spin_lock(&mtk_sha.lock);
list_add_tail(&cryp->sha_list, &mtk_sha.dev_list);
spin_unlock(&mtk_sha.lock);
@@ -1383,8 +1339,6 @@ int mtk_hash_alg_register(struct mtk_cryp *cryp)
spin_lock(&mtk_sha.lock);
list_del(&cryp->sha_list);
spin_unlock(&mtk_sha.lock);
- dma_free_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- cryp->tmp, cryp->tmp_dma);
err_res:
mtk_sha_record_free(cryp);
err_record:
@@ -1400,7 +1354,5 @@ void mtk_hash_alg_release(struct mtk_cryp *cryp)
spin_unlock(&mtk_sha.lock);
mtk_sha_unregister_algs();
- dma_free_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- cryp->tmp, cryp->tmp_dma);
mtk_sha_record_free(cryp);
}
--
1.9.1
^ permalink raw reply related
* [PATCH 4/9] crypto: mediatek - simplify descriptor ring management
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
In-Reply-To: <1487582822-30393-1-git-send-email-ryder.lee@mediatek.com>
This patch replaces cmd_pos/res_pos with pointer cmd_next/res_next.
In old code, we must to add one to shift ring to the next segment, and
then use this value to caculate current offset from ring base for each
DMA operation. Now these pointers helps us to simplify flow, so we just
need to move pointers and check the boundaries of ring.
Signed-off-by: Ryder Lee <ryder.lee@mediatek.com>
---
drivers/crypto/mediatek/mtk-aes.c | 14 ++++++++------
drivers/crypto/mediatek/mtk-platform.c | 3 +++
drivers/crypto/mediatek/mtk-platform.h | 8 ++++----
drivers/crypto/mediatek/mtk-sha.c | 35 ++++++++++++++++++----------------
4 files changed, 34 insertions(+), 26 deletions(-)
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index b57b68f..21f3e59 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -262,7 +262,7 @@ static int mtk_aes_xmit(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
/* Write command descriptors */
for (nents = 0; nents < slen; ++nents, ssg = sg_next(ssg)) {
- cmd = ring->cmd_base + ring->cmd_pos;
+ cmd = ring->cmd_next;
cmd->hdr = MTK_DESC_BUF_LEN(ssg->length);
cmd->buf = cpu_to_le32(sg_dma_address(ssg));
@@ -274,22 +274,24 @@ static int mtk_aes_xmit(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
cmd->tfm = cpu_to_le32(aes->ctx->tfm_dma);
}
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
+ /* Shift ring buffer and check boundary */
+ if (++ring->cmd_next == ring->cmd_base + MTK_DESC_NUM)
+ ring->cmd_next = ring->cmd_base;
}
cmd->hdr |= MTK_DESC_LAST;
/* Prepare result descriptors */
for (nents = 0; nents < dlen; ++nents, dsg = sg_next(dsg)) {
- res = ring->res_base + ring->res_pos;
+ res = ring->res_next;
res->hdr = MTK_DESC_BUF_LEN(dsg->length);
res->buf = cpu_to_le32(sg_dma_address(dsg));
if (nents == 0)
res->hdr |= MTK_DESC_FIRST;
- if (++ring->res_pos == MTK_DESC_NUM)
- ring->res_pos = 0;
+ /* Shift ring buffer and check boundary */
+ if (++ring->res_next == ring->res_base + MTK_DESC_NUM)
+ ring->res_next = ring->res_base;
}
res->hdr |= MTK_DESC_LAST;
diff --git a/drivers/crypto/mediatek/mtk-platform.c b/drivers/crypto/mediatek/mtk-platform.c
index 50de335..b6ecc28 100644
--- a/drivers/crypto/mediatek/mtk-platform.c
+++ b/drivers/crypto/mediatek/mtk-platform.c
@@ -465,6 +465,9 @@ static int mtk_desc_ring_alloc(struct mtk_cryp *cryp)
GFP_KERNEL);
if (!ring[i]->res_base)
goto err_cleanup;
+
+ ring[i]->cmd_next = ring[i]->cmd_base;
+ ring[i]->res_next = ring[i]->res_base;
}
return 0;
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index 78ce54e..218e30d 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -84,11 +84,11 @@ struct mtk_desc {
/**
* struct mtk_ring - Descriptor ring
* @cmd_base: pointer to command descriptor ring base
+ * @cmd_next: pointer to the next command descriptor
* @cmd_dma: DMA address of command descriptor ring
- * @cmd_pos: current position in the command descriptor ring
* @res_base: pointer to result descriptor ring base
+ * @res_next: pointer to the next result descriptor
* @res_dma: DMA address of result descriptor ring
- * @res_pos: current position in the result descriptor ring
*
* A descriptor ring is a circular buffer that is used to manage
* one or more descriptors. There are two type of descriptor rings;
@@ -96,11 +96,11 @@ struct mtk_desc {
*/
struct mtk_ring {
struct mtk_desc *cmd_base;
+ struct mtk_desc *cmd_next;
dma_addr_t cmd_dma;
- u32 cmd_pos;
struct mtk_desc *res_base;
+ struct mtk_desc *res_next;
dma_addr_t res_dma;
- u32 res_pos;
};
/**
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index ef6fb20..0884d62 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -152,6 +152,21 @@ static inline void mtk_sha_write(struct mtk_cryp *cryp,
writel_relaxed(value, cryp->base + offset);
}
+static inline void mtk_sha_ring_shift(struct mtk_ring *ring,
+ struct mtk_desc **cmd_curr,
+ struct mtk_desc **res_curr,
+ int *count)
+{
+ *cmd_curr = ring->cmd_next++;
+ *res_curr = ring->res_next++;
+ (*count)++;
+
+ if (ring->cmd_next == ring->cmd_base + MTK_DESC_NUM) {
+ ring->cmd_next = ring->cmd_base;
+ ring->res_next = ring->res_base;
+ }
+}
+
static struct mtk_cryp *mtk_sha_find_dev(struct mtk_sha_ctx *tctx)
{
struct mtk_cryp *cryp = NULL;
@@ -426,8 +441,7 @@ static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_ring *ring = cryp->ring[sha->id];
- struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
- struct mtk_desc *res = ring->res_base + ring->res_pos;
+ struct mtk_desc *cmd, *res;
int err, count = 0;
err = mtk_sha_info_update(cryp, sha, len1, len2);
@@ -435,6 +449,8 @@ static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
return err;
/* Fill in the command/result descriptors */
+ mtk_sha_ring_shift(ring, &cmd, &res, &count);
+
res->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1);
cmd->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1) |
MTK_DESC_CT_LEN(ctx->ct_size);
@@ -443,25 +459,12 @@ static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
cmd->ct_hdr = ctx->ct_hdr;
cmd->tfm = cpu_to_le32(ctx->tfm_dma);
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
- count++;
-
if (len2) {
- cmd = ring->cmd_base + ring->cmd_pos;
- res = ring->res_base + ring->res_pos;
+ mtk_sha_ring_shift(ring, &cmd, &res, &count);
res->hdr = MTK_DESC_BUF_LEN(len2);
cmd->hdr = MTK_DESC_BUF_LEN(len2);
cmd->buf = cpu_to_le32(addr2);
-
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
- count++;
}
cmd->hdr |= MTK_DESC_LAST;
--
1.9.1
^ permalink raw reply related
* [PATCH 0/9] improve performances on mediatek crypto driver
From: Ryder Lee @ 2017-02-20 9:26 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-mediatek, linux-crypto, Ryder Lee
Hi all,
Some patches of this series improve the performances whereas others
clean up code and refine data structure to make it more efficient.
It also adds support to AES-OFB and AES-CFB128. The new mode has been
tested by using the tcrypt
Ryder Lee (9):
crypto: mediatek - rework interrupt handler
crypto: mediatek - add MTK_* prefix and correct annotations.
crypto: mediatek - make mtk_sha_xmit() more generic
crypto: mediatek - simplify descriptor ring management
crypto: mediatek - add queue_task tasklet
crypto: mediatek - fix error handling in mtk_aes_complete()
crypto: mediatek - add mtk_aes_gcm_tag_verify()
crypto: mediatek - make hardware operation flow more efficient
crypto: mediatek - add support to OFB mode and CFB128 mode
drivers/crypto/mediatek/mtk-aes.c | 503 +++++++++++++++++++--------------
drivers/crypto/mediatek/mtk-platform.c | 15 +-
drivers/crypto/mediatek/mtk-platform.h | 56 ++--
drivers/crypto/mediatek/mtk-sha.c | 309 ++++++++------------
4 files changed, 446 insertions(+), 437 deletions(-)
--
1.9.1
^ permalink raw reply
* [PATCH 2/2] hwrng: meson: add clock handling to driver
From: Heiner Kallweit @ 2017-02-19 20:35 UTC (permalink / raw)
To: Kevin Hilman, Herbert Xu; +Cc: Neil Armstrong, linux-amlogic, linux-crypto
In-Reply-To: <00b2a4cc-686e-0cc0-43e8-552b70a1e868@gmail.com>
Add handling of RNG0 clock to the driver.
Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
---
drivers/char/hw_random/meson-rng.c | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/drivers/char/hw_random/meson-rng.c b/drivers/char/hw_random/meson-rng.c
index 119d6984..1f586e48 100644
--- a/drivers/char/hw_random/meson-rng.c
+++ b/drivers/char/hw_random/meson-rng.c
@@ -62,6 +62,7 @@
#include <linux/slab.h>
#include <linux/types.h>
#include <linux/of.h>
+#include <linux/clk.h>
#define RNG_DATA 0x00
@@ -69,6 +70,7 @@ struct meson_rng_data {
void __iomem *base;
struct platform_device *pdev;
struct hwrng rng;
+ struct clk *core_clk;
};
static int meson_rng_read(struct hwrng *rng, void *buf, size_t max, bool wait)
@@ -86,6 +88,7 @@ static int meson_rng_probe(struct platform_device *pdev)
struct device *dev = &pdev->dev;
struct meson_rng_data *data;
struct resource *res;
+ int ret;
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
if (!data)
@@ -98,12 +101,33 @@ static int meson_rng_probe(struct platform_device *pdev)
if (IS_ERR(data->base))
return PTR_ERR(data->base);
+ data->core_clk = devm_clk_get(dev, "core");
+ if (IS_ERR(data->core_clk))
+ return PTR_ERR(data->core_clk);
+
+ ret = clk_prepare_enable(data->core_clk);
+ if (ret)
+ return ret;
+
data->rng.name = pdev->name;
data->rng.read = meson_rng_read;
platform_set_drvdata(pdev, data);
- return devm_hwrng_register(dev, &data->rng);
+ ret = devm_hwrng_register(dev, &data->rng);
+ if (ret)
+ clk_disable_unprepare(data->core_clk);
+
+ return ret;
+}
+
+static int meson_rng_remove(struct platform_device *pdev)
+{
+ struct meson_rng_data *data = platform_get_drvdata(pdev);
+
+ clk_disable_unprepare(data->core_clk);
+
+ return 0;
}
static const struct of_device_id meson_rng_of_match[] = {
@@ -114,6 +138,7 @@ MODULE_DEVICE_TABLE(of, meson_rng_of_match);
static struct platform_driver meson_rng_driver = {
.probe = meson_rng_probe,
+ .remove = meson_rng_remove,
.driver = {
.name = "meson-rng",
.of_match_table = meson_rng_of_match,
--
2.11.1
^ permalink raw reply related
* [PATCH 1/2] hwrng: meson: expose RNG0 clock via DT
From: Heiner Kallweit @ 2017-02-19 20:33 UTC (permalink / raw)
To: Kevin Hilman, Herbert Xu; +Cc: Neil Armstrong, linux-amlogic, linux-crypto
In-Reply-To: <00b2a4cc-686e-0cc0-43e8-552b70a1e868@gmail.com>
Expose the RNG0 clock via DT.
Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
---
arch/arm64/boot/dts/amlogic/meson-gx.dtsi | 2 +-
arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi | 5 +++++
include/dt-bindings/clock/gxbb-clkc.h | 1 +
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/boot/dts/amlogic/meson-gx.dtsi b/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
index 5d995f77..620495a4 100644
--- a/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
+++ b/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
@@ -380,7 +380,7 @@
#size-cells = <2>;
ranges = <0x0 0x0 0x0 0xc8834000 0x0 0x2000>;
- rng {
+ hwrng: rng {
compatible = "amlogic,meson-rng";
reg = <0x0 0x0 0x0 0x4>;
};
diff --git a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
index 04b3324b..a375cb21 100644
--- a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
+++ b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
@@ -524,3 +524,8 @@
&vpu {
compatible = "amlogic,meson-gxbb-vpu", "amlogic,meson-gx-vpu";
};
+
+&hwrng {
+ clocks = <&clkc CLKID_RNG0>;
+ clock-names = "core";
+};
diff --git a/include/dt-bindings/clock/gxbb-clkc.h b/include/dt-bindings/clock/gxbb-clkc.h
index 692846c7..473676b1 100644
--- a/include/dt-bindings/clock/gxbb-clkc.h
+++ b/include/dt-bindings/clock/gxbb-clkc.h
@@ -15,6 +15,7 @@
#define CLKID_SPI 34
#define CLKID_I2C 22
#define CLKID_SAR_ADC 23
+#define CLKID_RNG0 25
#define CLKID_ETH 36
#define CLKID_USB0 50
#define CLKID_USB1 51
--
2.11.1
^ permalink raw reply related
* [PATCH 0/2] hwrng: meson: add clock handling
From: Heiner Kallweit @ 2017-02-19 20:30 UTC (permalink / raw)
To: Kevin Hilman, Herbert Xu; +Cc: Neil Armstrong, linux-amlogic, linux-crypto
The HW randon number generator requires a clock and we shouldn't rely
on the boot loader to enable it.
Heiner Kallweit (2):
hwrng: meson: expose RNG0 clock via DT
hwrng: meson: add clock handling to driver
arch/arm64/boot/dts/amlogic/meson-gx.dtsi | 2 +-
arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi | 5 +++++
drivers/char/hw_random/meson-rng.c | 27 ++++++++++++++++++++++++++-
include/dt-bindings/clock/gxbb-clkc.h | 1 +
4 files changed, 33 insertions(+), 2 deletions(-)
--
2.11.1
^ permalink raw reply
* Re: crypto/cavium MSI-X fixups
From: Christoph Hellwig @ 2017-02-19 17:32 UTC (permalink / raw)
To: George Cherian
Cc: Christoph Hellwig, George Cherian, David Daney, Herbert Xu,
linux-crypto, linux-kernel
In-Reply-To: <58A41C95.2090408@caviumnetworks.com>
Herbert,
any comment? I'd really like to avoid introducing new pci_enable_msix
users in this merge window..
On Wed, Feb 15, 2017 at 02:47:09PM +0530, George Cherian wrote:
> Hi Christoph,
>
>
> On 02/15/2017 12:48 PM, Christoph Hellwig wrote:
>> Hi George,
>>
>> your commit "crypto: cavium - Add Support for Octeon-tx CPT Engine"
>> add a new caller to pci_enable_msix. This API has long been deprecated
>> so this series switches it to use pci_alloc_irq_vectors instead.
>>
>> Can you please test it and make sure it goes in before the end of the
>> merge window so that no more users of the old API hit mainline?
>
> Yes the changes works well.
> Acked-by: George Cherian <george.cherian@cavium.com>
>
> for the series.
>>
---end quoted text---
^ permalink raw reply
* [PATCH v3] dm: switch dm-verity to async hash crypto API
From: Gilad Ben-Yossef @ 2017-02-19 12:46 UTC (permalink / raw)
To: dm-devel, Alasdair Kergon, MIKE Snitzer
Cc: gilad.benyossef, Eric Biggers, linux-crypto,
Ondrej Mosnáček, Milan Broz, Ofir Drang
Use of the synchronous digest API limits dm-verity to using pure
CPU based algorithm providers and rules out the use of off CPU
algorithm providers which are normally asynchronous by nature,
potentially freeing CPU cycles.
This can reduce performance per Watt in situations such as during
boot time when a lot of concurrent file accesses are made to the
protected volume.
Move DM_VERITY to the asynchronous hash API.
Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com>
CC: Eric Biggers <ebiggers3@gmail.com>
CC: Ondrej Mosnáček <omosnacek+linux-crypto@gmail.com>
CC: Milan Broz <gmazyland@gmail.com>
---
The patch was tested on an Armv7 based dual core Zynq ZC706 development
board with SHA256-asm, SHA256-neon synchronous providers with no visible
degradation of performance, with cryptd based asynchronous versions of the
same and with an off tree Arm CryptoCell asynchronous provider.
Changes from v2:
- Use completion to potentially wait also on crypto_ahash_init() as it
may finish asynchronously as well in some drivers, such as cryptd, as
discovered by Milan Broz.
- Use sg_init_one() where apropriate as pointed out by Milan Broz.
Changes from v1:
- Use a 0 mask to allocate crypto alg indicating we welcome async algo
providers, as suggested by Ondrej Mosnáček.
- Fix use of un-initialized completion when using async provider for IO
blocks hashing
- Pass flag indicating we are OK with crypto provider backlog
- Re-factor checking for need to wait into a common function
drivers/md/dm-verity-fec.c | 4 +-
drivers/md/dm-verity-target.c | 202 +++++++++++++++++++++++++++++-------------
drivers/md/dm-verity.h | 23 +++--
3 files changed, 158 insertions(+), 71 deletions(-)
drivers/md/dm-verity-fec.c | 4 +-
drivers/md/dm-verity-target.c | 201 +++++++++++++++++++++++++++++-------------
drivers/md/dm-verity.h | 23 +++--
3 files changed, 157 insertions(+), 71 deletions(-)
diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c
index 0f0eb8a..dab98fe 100644
--- a/drivers/md/dm-verity-fec.c
+++ b/drivers/md/dm-verity-fec.c
@@ -188,7 +188,7 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_fec_io *fio,
static int fec_is_erasure(struct dm_verity *v, struct dm_verity_io *io,
u8 *want_digest, u8 *data)
{
- if (unlikely(verity_hash(v, verity_io_hash_desc(v, io),
+ if (unlikely(verity_hash(v, verity_io_hash_req(v, io),
data, 1 << v->data_dev_block_bits,
verity_io_real_digest(v, io))))
return 0;
@@ -397,7 +397,7 @@ static int fec_decode_rsb(struct dm_verity *v, struct dm_verity_io *io,
}
/* Always re-validate the corrected block against the expected hash */
- r = verity_hash(v, verity_io_hash_desc(v, io), fio->output,
+ r = verity_hash(v, verity_io_hash_req(v, io), fio->output,
1 << v->data_dev_block_bits,
verity_io_real_digest(v, io));
if (unlikely(r < 0))
diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index 7335d8a..97de961 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -93,81 +93,123 @@ static sector_t verity_position_at_level(struct dm_verity *v, sector_t block,
}
/*
- * Wrapper for crypto_shash_init, which handles verity salting.
+ * Callback function for asynchrnous crypto API completion notification
*/
-static int verity_hash_init(struct dm_verity *v, struct shash_desc *desc)
+static void verity_op_done(struct crypto_async_request *base, int err)
{
- int r;
+ struct verity_result *res = (struct verity_result *)base->data;
- desc->tfm = v->tfm;
- desc->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+ if (err == -EINPROGRESS)
+ return;
- r = crypto_shash_init(desc);
+ res->err = err;
+ complete(&res->completion);
+}
- if (unlikely(r < 0)) {
- DMERR("crypto_shash_init failed: %d", r);
- return r;
- }
+/*
+ * Wait for async crypto API callback
+ */
+static inline int verity_complete_op(struct verity_result *res, int ret)
+{
+ switch (ret) {
+ case 0:
+ break;
- if (likely(v->version >= 1)) {
- r = crypto_shash_update(desc, v->salt, v->salt_size);
+ case -EINPROGRESS:
+ case -EBUSY:
+ ret = wait_for_completion_interruptible(&res->completion);
+ if (!ret)
+ ret = res->err;
+ reinit_completion(&res->completion);
+ break;
- if (unlikely(r < 0)) {
- DMERR("crypto_shash_update failed: %d", r);
- return r;
- }
+ default:
+ DMERR("verity_wait_hash: crypto op submission failed: %d", ret);
}
- return 0;
+ if (unlikely(ret < 0))
+ DMERR("verity_wait_hash: crypto op failed: %d", ret);
+
+ return ret;
}
-static int verity_hash_update(struct dm_verity *v, struct shash_desc *desc,
- const u8 *data, size_t len)
+static int verity_hash_update(struct dm_verity *v, struct ahash_request *req,
+ const u8 *data, size_t len,
+ struct verity_result *res)
{
- int r = crypto_shash_update(desc, data, len);
+ struct scatterlist sg;
- if (unlikely(r < 0))
- DMERR("crypto_shash_update failed: %d", r);
+ sg_init_one(&sg, data, len);
+ ahash_request_set_crypt(req, &sg, NULL, len);
+
+ return verity_complete_op(res, crypto_ahash_update(req));
+}
+
+/*
+ * Wrapper for crypto_ahash_init, which handles verity salting.
+ */
+static int verity_hash_init(struct dm_verity *v, struct ahash_request *req,
+ struct verity_result *res)
+{
+ int r;
+
+ ahash_request_set_tfm(req, v->tfm);
+ ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP |
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ verity_op_done, (void *)res);
+ init_completion(&res->completion);
+
+ r = verity_complete_op(res, crypto_ahash_init(req));
+
+ if (unlikely(r < 0)) {
+ DMERR("crypto_ahash_init failed: %d", r);
+ return r;
+ }
+
+ if (likely(v->version >= 1))
+ r = verity_hash_update(v, req, v->salt, v->salt_size, res);
return r;
}
-static int verity_hash_final(struct dm_verity *v, struct shash_desc *desc,
- u8 *digest)
+static int verity_hash_final(struct dm_verity *v, struct ahash_request *req,
+ u8 *digest, struct verity_result *res)
{
int r;
if (unlikely(!v->version)) {
- r = crypto_shash_update(desc, v->salt, v->salt_size);
+ r = verity_hash_update(v, req, v->salt, v->salt_size, res);
if (r < 0) {
- DMERR("crypto_shash_update failed: %d", r);
- return r;
+ DMERR("verity_hash_final failed updating salt: %d", r);
+ goto out;
}
}
- r = crypto_shash_final(desc, digest);
-
- if (unlikely(r < 0))
- DMERR("crypto_shash_final failed: %d", r);
-
+ ahash_request_set_crypt(req, NULL, digest, 0);
+ r = verity_complete_op(res, crypto_ahash_final(req));
+out:
return r;
}
-int verity_hash(struct dm_verity *v, struct shash_desc *desc,
+int verity_hash(struct dm_verity *v, struct ahash_request *req,
const u8 *data, size_t len, u8 *digest)
{
int r;
+ struct verity_result res;
- r = verity_hash_init(v, desc);
+ r = verity_hash_init(v, req, &res);
if (unlikely(r < 0))
- return r;
+ goto out;
- r = verity_hash_update(v, desc, data, len);
+ r = verity_hash_update(v, req, data, len, &res);
if (unlikely(r < 0))
- return r;
+ goto out;
+
+ r = verity_hash_final(v, req, digest, &res);
- return verity_hash_final(v, desc, digest);
+out:
+ return r;
}
static void verity_hash_at_level(struct dm_verity *v, sector_t block, int level,
@@ -275,7 +317,7 @@ static int verity_verify_level(struct dm_verity *v, struct dm_verity_io *io,
goto release_ret_r;
}
- r = verity_hash(v, verity_io_hash_desc(v, io),
+ r = verity_hash(v, verity_io_hash_req(v, io),
data, 1 << v->hash_dev_block_bits,
verity_io_real_digest(v, io));
if (unlikely(r < 0))
@@ -344,6 +386,49 @@ int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io,
}
/*
+ * Calculates the digest for the given bio
+ */
+int verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io,
+ struct bvec_iter *iter, struct verity_result *res)
+{
+ unsigned int todo = 1 << v->data_dev_block_bits;
+ struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size);
+ struct scatterlist sg;
+ struct ahash_request *req = verity_io_hash_req(v, io);
+
+ do {
+ int r;
+ unsigned int len;
+ struct bio_vec bv = bio_iter_iovec(bio, *iter);
+
+ sg_init_table(&sg, 1);
+
+ len = bv.bv_len;
+
+ if (likely(len >= todo))
+ len = todo;
+ /*
+ * Operating on a single page at a time looks suboptimal
+ * until you consider the typical block size is 4,096B.
+ * Going through this loops twice should be very rare.
+ */
+ sg_set_page(&sg, bv.bv_page, len, bv.bv_offset);
+ ahash_request_set_crypt(req, &sg, NULL, len);
+ r = verity_complete_op(res, crypto_ahash_update(req));
+
+ if (unlikely(r < 0)) {
+ DMERR("verity_for_io_block crypto op failed: %d", r);
+ return r;
+ }
+
+ bio_advance_iter(bio, iter, len);
+ todo -= len;
+ } while (todo);
+
+ return 0;
+}
+
+/*
* Calls function process for 1 << v->data_dev_block_bits bytes in the bio_vec
* starting from iter.
*/
@@ -381,12 +466,6 @@ int verity_for_bv_block(struct dm_verity *v, struct dm_verity_io *io,
return 0;
}
-static int verity_bv_hash_update(struct dm_verity *v, struct dm_verity_io *io,
- u8 *data, size_t len)
-{
- return verity_hash_update(v, verity_io_hash_desc(v, io), data, len);
-}
-
static int verity_bv_zero(struct dm_verity *v, struct dm_verity_io *io,
u8 *data, size_t len)
{
@@ -403,10 +482,11 @@ static int verity_verify_io(struct dm_verity_io *io)
struct dm_verity *v = io->v;
struct bvec_iter start;
unsigned b;
+ struct verity_result res;
for (b = 0; b < io->n_blocks; b++) {
int r;
- struct shash_desc *desc = verity_io_hash_desc(v, io);
+ struct ahash_request *req = verity_io_hash_req(v, io);
r = verity_hash_for_block(v, io, io->block + b,
verity_io_want_digest(v, io),
@@ -427,16 +507,17 @@ static int verity_verify_io(struct dm_verity_io *io)
continue;
}
- r = verity_hash_init(v, desc);
+ r = verity_hash_init(v, req, &res);
if (unlikely(r < 0))
return r;
start = io->iter;
- r = verity_for_bv_block(v, io, &io->iter, verity_bv_hash_update);
+ r = verity_for_io_block(v, io, &io->iter, &res);
if (unlikely(r < 0))
return r;
- r = verity_hash_final(v, desc, verity_io_real_digest(v, io));
+ r = verity_hash_final(v, req, verity_io_real_digest(v, io),
+ &res);
if (unlikely(r < 0))
return r;
@@ -705,7 +786,7 @@ static void verity_dtr(struct dm_target *ti)
kfree(v->zero_digest);
if (v->tfm)
- crypto_free_shash(v->tfm);
+ crypto_free_ahash(v->tfm);
kfree(v->alg_name);
@@ -723,7 +804,7 @@ static void verity_dtr(struct dm_target *ti)
static int verity_alloc_zero_digest(struct dm_verity *v)
{
int r = -ENOMEM;
- struct shash_desc *desc;
+ struct ahash_request *req;
u8 *zero_data;
v->zero_digest = kmalloc(v->digest_size, GFP_KERNEL);
@@ -731,9 +812,9 @@ static int verity_alloc_zero_digest(struct dm_verity *v)
if (!v->zero_digest)
return r;
- desc = kmalloc(v->shash_descsize, GFP_KERNEL);
+ req = kmalloc(v->ahash_reqsize, GFP_KERNEL);
- if (!desc)
+ if (!req)
return r; /* verity_dtr will free zero_digest */
zero_data = kzalloc(1 << v->data_dev_block_bits, GFP_KERNEL);
@@ -741,11 +822,11 @@ static int verity_alloc_zero_digest(struct dm_verity *v)
if (!zero_data)
goto out;
- r = verity_hash(v, desc, zero_data, 1 << v->data_dev_block_bits,
+ r = verity_hash(v, req, zero_data, 1 << v->data_dev_block_bits,
v->zero_digest);
out:
- kfree(desc);
+ kfree(req);
kfree(zero_data);
return r;
@@ -923,21 +1004,21 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
goto bad;
}
- v->tfm = crypto_alloc_shash(v->alg_name, 0, 0);
+ v->tfm = crypto_alloc_ahash(v->alg_name, 0, 0);
if (IS_ERR(v->tfm)) {
ti->error = "Cannot initialize hash function";
r = PTR_ERR(v->tfm);
v->tfm = NULL;
goto bad;
}
- v->digest_size = crypto_shash_digestsize(v->tfm);
+ v->digest_size = crypto_ahash_digestsize(v->tfm);
if ((1 << v->hash_dev_block_bits) < v->digest_size * 2) {
ti->error = "Digest size too big";
r = -EINVAL;
goto bad;
}
- v->shash_descsize =
- sizeof(struct shash_desc) + crypto_shash_descsize(v->tfm);
+ v->ahash_reqsize = sizeof(struct ahash_request) +
+ crypto_ahash_reqsize(v->tfm);
v->root_digest = kmalloc(v->digest_size, GFP_KERNEL);
if (!v->root_digest) {
@@ -1037,7 +1118,7 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
}
ti->per_io_data_size = sizeof(struct dm_verity_io) +
- v->shash_descsize + v->digest_size * 2;
+ v->ahash_reqsize + v->digest_size * 2;
r = verity_fec_ctr(v);
if (r)
diff --git a/drivers/md/dm-verity.h b/drivers/md/dm-verity.h
index fb419f4..a59e0ad 100644
--- a/drivers/md/dm-verity.h
+++ b/drivers/md/dm-verity.h
@@ -37,7 +37,7 @@ struct dm_verity {
struct dm_target *ti;
struct dm_bufio_client *bufio;
char *alg_name;
- struct crypto_shash *tfm;
+ struct crypto_ahash *tfm;
u8 *root_digest; /* digest of the root block */
u8 *salt; /* salt: its size is salt_size */
u8 *zero_digest; /* digest for a zero block */
@@ -52,7 +52,7 @@ struct dm_verity {
unsigned char levels; /* the number of tree levels */
unsigned char version;
unsigned digest_size; /* digest size for the current hash algorithm */
- unsigned shash_descsize;/* the size of temporary space for crypto */
+ unsigned int ahash_reqsize;/* the size of temporary space for crypto */
int hash_failed; /* set to 1 if hash of any block failed */
enum verity_mode mode; /* mode for handling verification errors */
unsigned corrupted_errs;/* Number of errors for corrupted blocks */
@@ -81,31 +81,36 @@ struct dm_verity_io {
/*
* Three variably-size fields follow this struct:
*
- * u8 hash_desc[v->shash_descsize];
+ * u8 hash_req[v->ahash_reqsize];
* u8 real_digest[v->digest_size];
* u8 want_digest[v->digest_size];
*
- * To access them use: verity_io_hash_desc(), verity_io_real_digest()
+ * To access them use: verity_io_hash_req(), verity_io_real_digest()
* and verity_io_want_digest().
*/
};
-static inline struct shash_desc *verity_io_hash_desc(struct dm_verity *v,
+struct verity_result {
+ struct completion completion;
+ int err;
+};
+
+static inline struct ahash_request *verity_io_hash_req(struct dm_verity *v,
struct dm_verity_io *io)
{
- return (struct shash_desc *)(io + 1);
+ return (struct ahash_request *)(io + 1);
}
static inline u8 *verity_io_real_digest(struct dm_verity *v,
struct dm_verity_io *io)
{
- return (u8 *)(io + 1) + v->shash_descsize;
+ return (u8 *)(io + 1) + v->ahash_reqsize;
}
static inline u8 *verity_io_want_digest(struct dm_verity *v,
struct dm_verity_io *io)
{
- return (u8 *)(io + 1) + v->shash_descsize + v->digest_size;
+ return (u8 *)(io + 1) + v->ahash_reqsize + v->digest_size;
}
static inline u8 *verity_io_digest_end(struct dm_verity *v,
@@ -120,7 +125,7 @@ extern int verity_for_bv_block(struct dm_verity *v, struct dm_verity_io *io,
struct dm_verity_io *io,
u8 *data, size_t len));
-extern int verity_hash(struct dm_verity *v, struct shash_desc *desc,
+extern int verity_hash(struct dm_verity *v, struct ahash_request *req,
const u8 *data, size_t len, u8 *digest);
extern int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io,
--
2.1.4
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply related
* Re: [PATCH v2] dm: switch dm-verity to async hash crypto API
From: Gilad Ben-Yossef @ 2017-02-19 12:37 UTC (permalink / raw)
To: Milan Broz
Cc: dm-devel, Alasdair Kergon, Mike Snitzer, linux-crypto,
gilad.benyossef, Ofir Drang, Eric Biggers,
Ondrej Mosnáček
In-Reply-To: <CAOtvUMc7s-ymufSiEty-vifcp12_TMnVMq6=mLcuz60T7RgnNQ@mail.gmail.com>
On Fri, Feb 17, 2017 at 4:47 PM, Gilad Ben-Yossef <gilad@benyossef.com> wrote:
> Hi Milan,
>
> Thank you for the review and testing.
>
> On Fri, Feb 17, 2017 at 3:00 PM, Milan Broz <gmazyland@gmail.com> wrote:
>>
>> On 02/06/2017 02:58 PM, Gilad Ben-Yossef wrote:
>> > Use of the synchronous digest API limits dm-verity to using pure
>> > CPU based algorithm providers and rules out the use of off CPU
>> > algorithm providers which are normally asynchronous by nature,
>> > potentially freeing CPU cycles.
>> >
>> > This can reduce performance per Watt in situations such as during
>> > boot time when a lot of concurrent file accesses are made to the
>> > protected volume.
>> >
>> > Move DM_VERITY to the asynchronous hash API.
>>
>> Did you test that async hash completion path?
>>
> Yes, I did - with the Arm TrustZone CryptoCell hardware accelerator.
> I did not try with cryptd though.
>
>
>>
>> I just tried to force async crypto by replacing "sha256"
>> in mapping table with "cryptd(sha256-generic)" and it kills
>> kernel immediately.
>> https://mbroz.fedorapeople.org/tmp/verity-fail.png
>>
>> (I hope this trick should cause to use cryptd and use async processing.
>> In previous version the parameter is properly rejected, because
>> unsupported
>> by shash api.)
>>
>>
>
> Thanks for this trick. I was not aware you can invoke cryptd it like that.
>
> I will recreate the issue and fix it.
I found out what the problem was and why I haven't seen it - I
thought crypto_ahash_init() only
sets up internal data structures and such and doesn't talk to the
device and therefore never
goes in the asynchronous completion path, which is true for the
hardware based provider
I was using for testing and probably for other HW based providers as well.
BUT, it isn't true for cryptd, which I guess uses the init function to
spawn kernel threads and does
use the asynchronous code path there.
Thanks for the review and testing. The next revision is coming up in a
jiffie. I'd be delighted if you
can take it out for a spin.
Gilad
--
Gilad Ben-Yossef
Chief Coffee Drinker
"If you take a class in large-scale robotics, can you end up in a
situation where the homework eats your dog?"
-- Jean-Baptiste Queru
^ permalink raw reply
* [PATCH v5 2/2] crypto: aead AF_ALG - overhaul memory management
From: Stephan Müller @ 2017-02-17 22:32 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
In-Reply-To: <2523592.mde6d2a8Lg@positron.chronox.de>
The updated memory management is described in the top part of the code.
As one benefit of the changed memory management, the AIO and synchronous
operation is now implemented in one common function. The AF_ALG
operation uses the async kernel crypto API interface for each cipher
operation. Thus, the only difference between the AIO and sync operation
types visible from user space is:
1. the callback function to be invoked when the asynchronous operation
is completed
2. whether to wait for the completion of the kernel crypto API operation
or not
The change includes the overhaul of the TX and RX SGL handling. The TX
SGL holding the data sent from user space to the kernel is now dynamic
similar to algif_skcipher. This dynamic nature allows a continuous
operation of a thread sending data and a second thread receiving the
data. These threads do not need to synchronize as the kernel processes
as much data from the TX SGL to fill the RX SGL.
The caller reading the data from the kernel defines the amount of data
to be processed. Considering that the interface covers AEAD
authenticating ciphers, the reader must provide the buffer in the
correct size. Thus the reader defines the encryption size.
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_aead.c | 702 +++++++++++++++++++++++++++++-----------------------
1 file changed, 386 insertions(+), 316 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 533265f..050a866 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -5,12 +5,26 @@
*
* This file provides the user-space API for AEAD ciphers.
*
- * This file is derived from algif_skcipher.c.
- *
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
+ *
+ * The following concept of the memory management is used:
+ *
+ * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
+ * filled by user space with the data submitted via sendpage/sendmsg. Filling
+ * up the TX SGL does not cause a crypto operation -- the data will only be
+ * tracked by the kernel. Upon receipt of one recvmsg call, the caller must
+ * provide a buffer which is tracked with the RX SGL.
+ *
+ * During the processing of the recvmsg operation, the cipher request is
+ * allocated and prepared. As part of the recvmsg operation, the processed
+ * TX buffers are extracted from the TX SGL into a separate SGL.
+ *
+ * After the completion of the crypto operation, the RX SGL and the cipher
+ * request is released. The extracted TX SGL parts are released together with
+ * the RX SGL release.
*/
#include <crypto/internal/aead.h>
@@ -24,45 +38,57 @@
#include <linux/net.h>
#include <net/sock.h>
-struct aead_sg_list {
- unsigned int cur;
- struct scatterlist sg[ALG_MAX_PAGES];
+struct aead_tsgl {
+ struct list_head list;
+ unsigned int cur; /* Last processed SG entry */
+ struct scatterlist sg[0]; /* Array of SGs forming the SGL */
};
-struct aead_async_rsgl {
+struct aead_rsgl {
struct af_alg_sgl sgl;
struct list_head list;
};
struct aead_async_req {
- struct scatterlist *tsgl;
- struct aead_async_rsgl first_rsgl;
- struct list_head list;
struct kiocb *iocb;
- unsigned int tsgls;
- char iv[];
+ struct sock *sk;
+
+ struct aead_rsgl first_rsgl; /* First RX SG */
+ struct list_head rsgl_list; /* Track RX SGs */
+
+ struct scatterlist *tsgl; /* priv. TX SGL of buffers to process */
+ unsigned int tsgl_entries; /* number of entries in priv. TX SGL */
+
+ unsigned int outlen; /* Filled output buf length */
+
+ unsigned int areqlen; /* Length of this data struct */
+ struct aead_request aead_req; /* req ctx trails this struct */
};
struct aead_ctx {
- struct aead_sg_list tsgl;
- struct aead_async_rsgl first_rsgl;
- struct list_head list;
+ struct list_head tsgl_list; /* Link to TX SGL */
void *iv;
+ size_t aead_assoclen;
- struct af_alg_completion completion;
+ struct af_alg_completion completion; /* sync work queue */
- unsigned long used;
+ unsigned int inflight; /* Outstanding AIO ops */
+ size_t used; /* TX bytes sent to kernel */
- unsigned int len;
- bool more;
- bool merge;
- bool enc;
+ bool more; /* More data to be expected? */
+ bool merge; /* Merge new data into existing SG */
+ bool enc; /* Crypto operation: enc, dec */
- size_t aead_assoclen;
- struct aead_request aead_req;
+ unsigned int len; /* Length of allocated memory for this struct */
+ struct crypto_aead *aead_tfm;
};
+static DECLARE_WAIT_QUEUE_HEAD(aead_aio_finish_wait);
+
+#define MAX_SGL_ENTS ((4096 - sizeof(struct aead_tsgl)) / \
+ sizeof(struct scatterlist) - 1)
+
static inline int aead_sndbuf(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
@@ -79,7 +105,7 @@ static inline bool aead_writable(struct sock *sk)
static inline bool aead_sufficient_data(struct aead_ctx *ctx)
{
- unsigned as = crypto_aead_authsize(crypto_aead_reqtfm(&ctx->aead_req));
+ unsigned int as = crypto_aead_authsize(ctx->aead_tfm);
/*
* The minimum amount of memory needed for an AEAD cipher is
@@ -88,33 +114,163 @@ static inline bool aead_sufficient_data(struct aead_ctx *ctx)
return ctx->used >= ctx->aead_assoclen + (ctx->enc ? 0 : as);
}
-static void aead_reset_ctx(struct aead_ctx *ctx)
+static int aead_alloc_tsgl(struct sock *sk)
{
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct alg_sock *ask = alg_sk(sk);
+ struct aead_ctx *ctx = ask->private;
+ struct aead_tsgl *sgl;
+ struct scatterlist *sg = NULL;
- sg_init_table(sgl->sg, ALG_MAX_PAGES);
- sgl->cur = 0;
- ctx->used = 0;
- ctx->more = 0;
- ctx->merge = 0;
+ sgl = list_entry(ctx->tsgl_list.prev, struct aead_tsgl, list);
+ if (!list_empty(&ctx->tsgl_list))
+ sg = sgl->sg;
+
+ if (!sg || sgl->cur >= MAX_SGL_ENTS) {
+ sgl = sock_kmalloc(sk, sizeof(*sgl) +
+ sizeof(sgl->sg[0]) * (MAX_SGL_ENTS + 1),
+ GFP_KERNEL);
+ if (!sgl)
+ return -ENOMEM;
+
+ sg_init_table(sgl->sg, MAX_SGL_ENTS + 1);
+ sgl->cur = 0;
+
+ if (sg)
+ sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg);
+
+ list_add_tail(&sgl->list, &ctx->tsgl_list);
+ }
+
+ return 0;
+}
+
+static unsigned int aead_count_tsgl(struct sock *sk, size_t bytes)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct aead_ctx *ctx = ask->private;
+ struct aead_tsgl *sgl, *tmp;
+ unsigned int i;
+ unsigned int sgl_count = 0;
+
+ if (!bytes)
+ return 0;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ struct scatterlist *sg = sgl->sg;
+
+ for (i = 0; i < sgl->cur; i++) {
+ sgl_count++;
+ if (sg[i].length >= bytes)
+ return sgl_count;
+
+ bytes -= sg[i].length;
+ }
+ }
+
+ return sgl_count;
}
-static void aead_put_sgl(struct sock *sk)
+static void aead_pull_tsgl(struct sock *sk, size_t used,
+ struct scatterlist *dst)
{
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct scatterlist *sg = sgl->sg;
+ struct aead_tsgl *sgl;
+ struct scatterlist *sg;
unsigned int i;
- for (i = 0; i < sgl->cur; i++) {
- if (!sg_page(sg + i))
+ while (!list_empty(&ctx->tsgl_list)) {
+ sgl = list_first_entry(&ctx->tsgl_list, struct aead_tsgl,
+ list);
+ sg = sgl->sg;
+
+ for (i = 0; i < sgl->cur; i++) {
+ size_t plen = min_t(size_t, used, sg[i].length);
+ struct page *page = sg_page(sg + i);
+
+ if (!page)
+ continue;
+
+ /*
+ * Assumption: caller created aead_count_tsgl(len)
+ * SG entries in dst.
+ */
+ if (dst)
+ sg_set_page(dst + i, page, plen, sg[i].offset);
+
+ sg[i].length -= plen;
+ sg[i].offset += plen;
+
+ used -= plen;
+ ctx->used -= plen;
+
+ if (sg[i].length)
+ return;
+
+ if (!dst)
+ put_page(page);
+ sg_assign_page(sg + i, NULL);
+ }
+
+ list_del(&sgl->list);
+ sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) *
+ (MAX_SGL_ENTS + 1));
+ }
+
+ if (!ctx->used)
+ ctx->merge = 0;
+}
+
+static void aead_free_areq_sgls(struct aead_async_req *areq)
+{
+ struct sock *sk = areq->sk;
+ struct aead_rsgl *rsgl, *tmp;
+ struct scatterlist *tsgl;
+ struct scatterlist *sg;
+ unsigned int i;
+
+ list_for_each_entry_safe(rsgl, tmp, &areq->rsgl_list, list) {
+ af_alg_free_sg(&rsgl->sgl);
+ list_del(&rsgl->list);
+ if (rsgl != &areq->first_rsgl)
+ sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ }
+
+ tsgl = areq->tsgl;
+ for_each_sg(tsgl, sg, areq->tsgl_entries, i) {
+ if (!sg_page(sg))
continue;
+ put_page(sg_page(sg));
+ }
+
+ if (areq->tsgl && areq->tsgl_entries)
+ sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl));
+}
- put_page(sg_page(sg + i));
- sg_assign_page(sg + i, NULL);
+static int aead_wait_for_wmem(struct sock *sk, unsigned flags)
+{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
+ int err = -ERESTARTSYS;
+ long timeout;
+
+ if (flags & MSG_DONTWAIT)
+ return -EAGAIN;
+
+ sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
+
+ add_wait_queue(sk_sleep(sk), &wait);
+ for (;;) {
+ if (signal_pending(current))
+ break;
+ timeout = MAX_SCHEDULE_TIMEOUT;
+ if (sk_wait_event(sk, &timeout, aead_writable(sk), &wait)) {
+ err = 0;
+ break;
+ }
}
- aead_reset_ctx(ctx);
+ remove_wait_queue(sk_sleep(sk), &wait);
+
+ return err;
}
static void aead_wmem_wakeup(struct sock *sk)
@@ -146,6 +302,7 @@ static int aead_wait_for_data(struct sock *sk, unsigned flags)
return -EAGAIN;
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
+
add_wait_queue(sk_sleep(sk), &wait);
for (;;) {
if (signal_pending(current))
@@ -169,8 +326,6 @@ static void aead_data_wakeup(struct sock *sk)
struct aead_ctx *ctx = ask->private;
struct socket_wq *wq;
- if (ctx->more)
- return;
if (!ctx->used)
return;
@@ -189,14 +344,13 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned ivsize =
- crypto_aead_ivsize(crypto_aead_reqtfm(&ctx->aead_req));
- struct aead_sg_list *sgl = &ctx->tsgl;
+ unsigned int ivsize = crypto_aead_ivsize(ctx->aead_tfm);
+ struct aead_tsgl *sgl;
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
bool init = 0;
- int err = -EINVAL;
+ int err = 0;
if (msg->msg_controllen) {
err = af_alg_cmsg_send(msg, &con);
@@ -220,8 +374,10 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
}
lock_sock(sk);
- if (!ctx->more && ctx->used)
+ if (!ctx->more && ctx->used) {
+ err = -EINVAL;
goto unlock;
+ }
if (init) {
ctx->enc = enc;
@@ -232,11 +388,14 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
}
while (size) {
+ struct scatterlist *sg;
size_t len = size;
- struct scatterlist *sg = NULL;
+ size_t plen;
/* use the existing memory in an allocated page */
if (ctx->merge) {
+ sgl = list_entry(ctx->tsgl_list.prev,
+ struct aead_tsgl, list);
sg = sgl->sg + sgl->cur - 1;
len = min_t(unsigned long, len,
PAGE_SIZE - sg->offset - sg->length);
@@ -257,57 +416,60 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
}
if (!aead_writable(sk)) {
- /* user space sent too much data */
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- goto unlock;
+ err = aead_wait_for_wmem(sk, msg->msg_flags);
+ if (err)
+ goto unlock;
}
/* allocate a new page */
len = min_t(unsigned long, size, aead_sndbuf(sk));
- while (len) {
- size_t plen = 0;
- if (sgl->cur >= ALG_MAX_PAGES) {
- aead_put_sgl(sk);
- err = -E2BIG;
- goto unlock;
- }
+ err = aead_alloc_tsgl(sk);
+ if (err)
+ goto unlock;
+
+ sgl = list_entry(ctx->tsgl_list.prev, struct aead_tsgl,
+ list);
+ sg = sgl->sg;
+ if (sgl->cur)
+ sg_unmark_end(sg + sgl->cur - 1);
+
+ do {
+ unsigned int i = sgl->cur;
- sg = sgl->sg + sgl->cur;
plen = min_t(size_t, len, PAGE_SIZE);
- sg_assign_page(sg, alloc_page(GFP_KERNEL));
- err = -ENOMEM;
- if (!sg_page(sg))
+ sg_assign_page(sg + i, alloc_page(GFP_KERNEL));
+ if (!sg_page(sg + i)) {
+ err = -ENOMEM;
goto unlock;
+ }
- err = memcpy_from_msg(page_address(sg_page(sg)),
+ err = memcpy_from_msg(page_address(sg_page(sg + i)),
msg, plen);
if (err) {
- __free_page(sg_page(sg));
- sg_assign_page(sg, NULL);
+ __free_page(sg_page(sg + i));
+ sg_assign_page(sg + i, NULL);
goto unlock;
}
- sg->offset = 0;
- sg->length = plen;
+ sg[i].length = plen;
len -= plen;
ctx->used += plen;
copied += plen;
- sgl->cur++;
size -= plen;
- ctx->merge = plen & (PAGE_SIZE - 1);
- }
+ sgl->cur++;
+ } while (len && sgl->cur < MAX_SGL_ENTS);
+
+ if (!size)
+ sg_mark_end(sg + sgl->cur - 1);
+
+ ctx->merge = plen & (PAGE_SIZE - 1);
}
err = 0;
ctx->more = msg->msg_flags & MSG_MORE;
- if (!ctx->more && !aead_sufficient_data(ctx)) {
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- }
unlock:
aead_data_wakeup(sk);
@@ -322,15 +484,12 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct aead_tsgl *sgl;
int err = -EINVAL;
if (flags & MSG_SENDPAGE_NOTLAST)
flags |= MSG_MORE;
- if (sgl->cur >= ALG_MAX_PAGES)
- return -E2BIG;
-
lock_sock(sk);
if (!ctx->more && ctx->used)
goto unlock;
@@ -339,13 +498,22 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
goto done;
if (!aead_writable(sk)) {
- /* user space sent too much data */
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- goto unlock;
+ err = aead_wait_for_wmem(sk, flags);
+ if (err)
+ goto unlock;
}
+ err = aead_alloc_tsgl(sk);
+ if (err)
+ goto unlock;
+
ctx->merge = 0;
+ sgl = list_entry(ctx->tsgl_list.prev, struct aead_tsgl, list);
+
+ if (sgl->cur)
+ sg_unmark_end(sgl->sg + sgl->cur - 1);
+
+ sg_mark_end(sgl->sg + sgl->cur);
get_page(page);
sg_set_page(sgl->sg + sgl->cur, page, size, offset);
@@ -356,11 +524,6 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
done:
ctx->more = flags & MSG_MORE;
- if (!ctx->more && !aead_sufficient_data(ctx)) {
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- }
-
unlock:
aead_data_wakeup(sk);
release_sock(sk);
@@ -368,205 +531,58 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
return err ?: size;
}
-#define GET_ASYM_REQ(req, tfm) (struct aead_async_req *) \
- ((char *)req + sizeof(struct aead_request) + \
- crypto_aead_reqsize(tfm))
-
- #define GET_REQ_SIZE(tfm) sizeof(struct aead_async_req) + \
- crypto_aead_reqsize(tfm) + crypto_aead_ivsize(tfm) + \
- sizeof(struct aead_request)
-
static void aead_async_cb(struct crypto_async_request *_req, int err)
{
- struct sock *sk = _req->data;
+ struct aead_async_req *areq = _req->data;
+ struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct crypto_aead *tfm = crypto_aead_reqtfm(&ctx->aead_req);
- struct aead_request *req = aead_request_cast(_req);
- struct aead_async_req *areq = GET_ASYM_REQ(req, tfm);
- struct scatterlist *sg = areq->tsgl;
- struct aead_async_rsgl *rsgl;
struct kiocb *iocb = areq->iocb;
- unsigned int i, reqlen = GET_REQ_SIZE(tfm);
-
- list_for_each_entry(rsgl, &areq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &areq->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
-
- for (i = 0; i < areq->tsgls; i++)
- put_page(sg_page(sg + i));
-
- sock_kfree_s(sk, areq->tsgl, sizeof(*areq->tsgl) * areq->tsgls);
- sock_kfree_s(sk, req, reqlen);
- __sock_put(sk);
- iocb->ki_complete(iocb, err, err);
-}
-
-static int aead_recvmsg_async(struct socket *sock, struct msghdr *msg,
- int flags)
-{
- struct sock *sk = sock->sk;
- struct alg_sock *ask = alg_sk(sk);
- struct aead_ctx *ctx = ask->private;
- struct crypto_aead *tfm = crypto_aead_reqtfm(&ctx->aead_req);
- struct aead_async_req *areq;
- struct aead_request *req = NULL;
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct aead_async_rsgl *last_rsgl = NULL, *rsgl;
- unsigned int as = crypto_aead_authsize(tfm);
- unsigned int i, reqlen = GET_REQ_SIZE(tfm);
- int err = -ENOMEM;
- unsigned long used;
- size_t outlen = 0;
- size_t usedpages = 0;
+ unsigned int resultlen;
lock_sock(sk);
- if (ctx->more) {
- err = aead_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- if (!aead_sufficient_data(ctx))
- goto unlock;
-
- used = ctx->used;
- if (ctx->enc)
- outlen = used + as;
- else
- outlen = used - as;
-
- req = sock_kmalloc(sk, reqlen, GFP_KERNEL);
- if (unlikely(!req))
- goto unlock;
-
- areq = GET_ASYM_REQ(req, tfm);
- memset(&areq->first_rsgl, '\0', sizeof(areq->first_rsgl));
- INIT_LIST_HEAD(&areq->list);
- areq->iocb = msg->msg_iocb;
- memcpy(areq->iv, ctx->iv, crypto_aead_ivsize(tfm));
- aead_request_set_tfm(req, tfm);
- aead_request_set_ad(req, ctx->aead_assoclen);
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- aead_async_cb, sk);
- used -= ctx->aead_assoclen;
-
- /* take over all tx sgls from ctx */
- areq->tsgl = sock_kmalloc(sk,
- sizeof(*areq->tsgl) * max_t(u32, sgl->cur, 1),
- GFP_KERNEL);
- if (unlikely(!areq->tsgl))
- goto free;
-
- sg_init_table(areq->tsgl, max_t(u32, sgl->cur, 1));
- for (i = 0; i < sgl->cur; i++)
- sg_set_page(&areq->tsgl[i], sg_page(&sgl->sg[i]),
- sgl->sg[i].length, sgl->sg[i].offset);
-
- areq->tsgls = sgl->cur;
-
- /* create rx sgls */
- while (outlen > usedpages && iov_iter_count(&msg->msg_iter)) {
- size_t seglen = min_t(size_t, iov_iter_count(&msg->msg_iter),
- (outlen - usedpages));
-
- if (list_empty(&areq->list)) {
- rsgl = &areq->first_rsgl;
- } else {
- rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
- if (unlikely(!rsgl)) {
- err = -ENOMEM;
- goto free;
- }
- }
- rsgl->sgl.npages = 0;
- list_add_tail(&rsgl->list, &areq->list);
+ BUG_ON(!ctx->inflight);
- /* make one iovec available as scatterlist */
- err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
- if (err < 0)
- goto free;
-
- usedpages += err;
+ /* Buffer size written by crypto operation. */
+ resultlen = areq->outlen;
- /* chain the new scatterlist with previous one */
- if (last_rsgl)
- af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
-
- last_rsgl = rsgl;
-
- iov_iter_advance(&msg->msg_iter, err);
- }
-
- /* ensure output buffer is sufficiently large */
- if (usedpages < outlen) {
- err = -EINVAL;
- goto unlock;
- }
+ aead_free_areq_sgls(areq);
+ sock_kfree_s(sk, areq, areq->areqlen);
+ __sock_put(sk);
+ ctx->inflight--;
- aead_request_set_crypt(req, areq->tsgl, areq->first_rsgl.sgl.sg, used,
- areq->iv);
- err = ctx->enc ? crypto_aead_encrypt(req) : crypto_aead_decrypt(req);
- if (err) {
- if (err == -EINPROGRESS) {
- sock_hold(sk);
- err = -EIOCBQUEUED;
- aead_reset_ctx(ctx);
- goto unlock;
- } else if (err == -EBADMSG) {
- aead_put_sgl(sk);
- }
- goto free;
- }
- aead_put_sgl(sk);
+ iocb->ki_complete(iocb, err ? err : resultlen, 0);
-free:
- list_for_each_entry(rsgl, &areq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &areq->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
- if (areq->tsgl)
- sock_kfree_s(sk, areq->tsgl, sizeof(*areq->tsgl) * areq->tsgls);
- if (req)
- sock_kfree_s(sk, req, reqlen);
-unlock:
- aead_wmem_wakeup(sk);
release_sock(sk);
- return err ? err : outlen;
+
+ wake_up_interruptible(&aead_aio_finish_wait);
}
-static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
+static int aead_recvmsg(struct socket *sock, struct msghdr *msg, size_t ignored,
+ int flags)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned as = crypto_aead_authsize(crypto_aead_reqtfm(&ctx->aead_req));
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct aead_async_rsgl *last_rsgl = NULL;
- struct aead_async_rsgl *rsgl, *tmp;
+ struct crypto_aead *tfm = ctx->aead_tfm;
+ unsigned int as = crypto_aead_authsize(tfm);
+ unsigned int areqlen =
+ sizeof(struct aead_async_req) + crypto_aead_reqsize(tfm);
+ struct aead_async_req *areq;
+ struct aead_rsgl *last_rsgl = NULL;
int err = -EINVAL;
- unsigned long used = 0;
- size_t outlen = 0;
- size_t usedpages = 0;
+ size_t used = 0; /* [in] TX bufs to be en/decrypted */
+ size_t outlen = 0; /* [out] RX bufs produced by kernel */
+ size_t usedpages = 0; /* [in] RX bufs to be used from user */
+ size_t processed = 0; /* [in] TX bufs to be consumed */
lock_sock(sk);
/*
- * Please see documentation of aead_request_set_crypt for the
- * description of the AEAD memory structure expected from the caller.
+ * Data length provided by caller via sendmsg/sendpage that has not
+ * yet been processed.
*/
-
- if (ctx->more) {
- err = aead_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- /* data length provided by caller via sendmsg/sendpage */
used = ctx->used;
/*
@@ -600,96 +616,151 @@ static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
*/
used -= ctx->aead_assoclen;
- /* convert iovecs of output buffers into scatterlists */
+ /* Allocate cipher request for current operation. */
+ areq = sock_kmalloc(sk, areqlen, GFP_KERNEL);
+ if (unlikely(!areq)) {
+ err = -ENOMEM;
+ goto unlock;
+ }
+ areq->areqlen = areqlen;
+ areq->sk = sk;
+ INIT_LIST_HEAD(&areq->rsgl_list);
+ areq->tsgl = NULL;
+ areq->tsgl_entries = 0;
+
+ /* convert iovecs of output buffers into RX SGL */
while (outlen > usedpages && iov_iter_count(&msg->msg_iter)) {
- size_t seglen = min_t(size_t, iov_iter_count(&msg->msg_iter),
- (outlen - usedpages));
+ struct aead_rsgl *rsgl;
+ size_t seglen;
+
+ if (!ctx->used) {
+ err = aead_wait_for_data(sk, flags);
+ if (err)
+ goto free;
+ }
+
+ seglen = min_t(size_t, (outlen - usedpages),
+ iov_iter_count(&msg->msg_iter));
- if (list_empty(&ctx->list)) {
- rsgl = &ctx->first_rsgl;
+ if (list_empty(&areq->rsgl_list)) {
+ rsgl = &areq->first_rsgl;
} else {
rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
if (unlikely(!rsgl)) {
err = -ENOMEM;
- goto unlock;
+ goto free;
}
}
+
rsgl->sgl.npages = 0;
- list_add_tail(&rsgl->list, &ctx->list);
+ list_add_tail(&rsgl->list, &areq->rsgl_list);
/* make one iovec available as scatterlist */
err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
if (err < 0)
- goto unlock;
- usedpages += err;
+ goto free;
+
/* chain the new scatterlist with previous one */
if (last_rsgl)
af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
last_rsgl = rsgl;
-
+ usedpages += err;
iov_iter_advance(&msg->msg_iter, err);
}
- /* ensure output buffer is sufficiently large */
+ /*
+ * Ensure output buffer is sufficiently large. If the caller provides
+ * less buffer space, only use the relative required input size. This
+ * allows AIO operation where the caller sent all data to be processed
+ * and the AIO operation performs the operation on the different chunks
+ * of the input data.
+ */
if (usedpages < outlen) {
- err = -EINVAL;
- goto unlock;
- }
+ size_t less = outlen - usedpages;
- sg_mark_end(sgl->sg + sgl->cur - 1);
- aead_request_set_crypt(&ctx->aead_req, sgl->sg, ctx->first_rsgl.sgl.sg,
- used, ctx->iv);
- aead_request_set_ad(&ctx->aead_req, ctx->aead_assoclen);
+ if (used < less) {
+ err = -EINVAL;
+ goto free;
+ }
+ used -= less;
+ outlen -= less;
+ }
- err = af_alg_wait_for_completion(ctx->enc ?
- crypto_aead_encrypt(&ctx->aead_req) :
- crypto_aead_decrypt(&ctx->aead_req),
+ /*
+ * Create a per request TX SGL for this request which tracks the
+ * SG entries from the global TX SGL.
+ */
+ processed = used + ctx->aead_assoclen;
+ areq->tsgl_entries = aead_count_tsgl(sk, processed);
+ if (!areq->tsgl_entries)
+ areq->tsgl_entries = 1;
+ areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) * areq->tsgl_entries,
+ GFP_KERNEL);
+ if (!areq->tsgl) {
+ err = -ENOMEM;
+ goto free;
+ }
+ sg_init_table(areq->tsgl, areq->tsgl_entries);
+ aead_pull_tsgl(sk, processed, areq->tsgl);
+
+ /* Initialize the crypto operation */
+ aead_request_set_crypt(&areq->aead_req, areq->tsgl,
+ areq->first_rsgl.sgl.sg, used, ctx->iv);
+ aead_request_set_ad(&areq->aead_req, ctx->aead_assoclen);
+ aead_request_set_tfm(&areq->aead_req, tfm);
+
+ if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
+ /* AIO operation */
+ areq->iocb = msg->msg_iocb;
+ aead_request_set_callback(&areq->aead_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ aead_async_cb, areq);
+ err = ctx->enc ? crypto_aead_encrypt(&areq->aead_req) :
+ crypto_aead_decrypt(&areq->aead_req);
+ } else {
+ /* Synchronous operation */
+ aead_request_set_callback(&areq->aead_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ af_alg_complete, &ctx->completion);
+ err = af_alg_wait_for_completion(ctx->enc ?
+ crypto_aead_encrypt(&areq->aead_req) :
+ crypto_aead_decrypt(&areq->aead_req),
&ctx->completion);
+ }
- if (err) {
- /* EBADMSG implies a valid cipher operation took place */
- if (err == -EBADMSG)
- aead_put_sgl(sk);
+ /* AIO operation in progress */
+ if (err == -EINPROGRESS) {
+ sock_hold(sk);
+ err = -EIOCBQUEUED;
+ ctx->inflight++;
+
+ /* Remember output size that will be generated. */
+ areq->outlen = outlen;
goto unlock;
}
- aead_put_sgl(sk);
- err = 0;
+free:
+ aead_free_areq_sgls(areq);
+ if (areq)
+ sock_kfree_s(sk, areq, areqlen);
unlock:
- list_for_each_entry_safe(rsgl, tmp, &ctx->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- list_del(&rsgl->list);
- if (rsgl != &ctx->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
- INIT_LIST_HEAD(&ctx->list);
aead_wmem_wakeup(sk);
release_sock(sk);
-
return err ? err : outlen;
}
-static int aead_recvmsg(struct socket *sock, struct msghdr *msg, size_t ignored,
- int flags)
-{
- return (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) ?
- aead_recvmsg_async(sock, msg, flags) :
- aead_recvmsg_sync(sock, msg, flags);
-}
-
static unsigned int aead_poll(struct file *file, struct socket *sock,
poll_table *wait)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned int mask;
+ unsigned int mask = 0;
sock_poll_wait(file, sk_sleep(sk), wait);
- mask = 0;
if (!ctx->more)
mask |= POLLIN | POLLRDNORM;
@@ -746,11 +817,14 @@ static void aead_sock_destruct(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned int ivlen = crypto_aead_ivsize(
- crypto_aead_reqtfm(&ctx->aead_req));
+ unsigned int ivlen = crypto_aead_ivsize(ctx->aead_tfm);
WARN_ON(atomic_read(&sk->sk_refcnt) != 0);
- aead_put_sgl(sk);
+
+ /* Suspend caller if AIO operations are in flight. */
+ wait_event_interruptible(aead_aio_finish_wait, (ctx->inflight == 0));
+
+ aead_pull_tsgl(sk, ctx->used, NULL);
sock_kzfree_s(sk, ctx->iv, ivlen);
sock_kfree_s(sk, ctx, ctx->len);
af_alg_release_parent(sk);
@@ -760,7 +834,7 @@ static int aead_accept_parent(void *private, struct sock *sk)
{
struct aead_ctx *ctx;
struct alg_sock *ask = alg_sk(sk);
- unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(private);
+ unsigned int len = sizeof(*ctx);
unsigned int ivlen = crypto_aead_ivsize(private);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
@@ -775,23 +849,19 @@ static int aead_accept_parent(void *private, struct sock *sk)
}
memset(ctx->iv, 0, ivlen);
+ INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
ctx->used = 0;
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
- ctx->tsgl.cur = 0;
+ ctx->inflight = 0;
ctx->aead_assoclen = 0;
af_alg_init_completion(&ctx->completion);
- sg_init_table(ctx->tsgl.sg, ALG_MAX_PAGES);
- INIT_LIST_HEAD(&ctx->list);
+ ctx->aead_tfm = private;
ask->private = ctx;
- aead_request_set_tfm(&ctx->aead_req, private);
- aead_request_set_callback(&ctx->aead_req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- af_alg_complete, &ctx->completion);
-
sk->sk_destruct = aead_sock_destruct;
return 0;
--
2.9.3
^ permalink raw reply related
* [PATCH v5 1/2] crypto: skcipher AF_ALG - overhaul memory management
From: Stephan Müller @ 2017-02-17 22:31 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
In-Reply-To: <2523592.mde6d2a8Lg@positron.chronox.de>
The updated memory management is described in the top part of the code.
As one benefit of the changed memory management, the AIO and synchronous
operation is now implemented in one common function. The AF_ALG
operation uses the async kernel crypto API interface for each cipher
operation. Thus, the only difference between the AIO and sync operation
types visible from user space is:
1. the callback function to be invoked when the asynchronous operation
is completed
2. whether to wait for the completion of the kernel crypto API operation
or not
In addition, the code structure is adjusted to match the structure of
algif_aead for easier code assessment.
The user space interface changed slightly as follows: the old AIO
operation returned zero upon success and < 0 in case of an error to user
space. As all other AF_ALG interfaces (including the sync skcipher
interface) returned the number of processed bytes upon success and < 0
in case of an error, the new skcipher interface (regardless of AIO or
sync) returns the number of processed bytes in case of success.
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_skcipher.c | 540 ++++++++++++++++++++++--------------------------
1 file changed, 251 insertions(+), 289 deletions(-)
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index a9e79d8..d3e0226 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -10,6 +10,21 @@
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*
+ * The following concept of the memory management is used:
+ *
+ * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
+ * filled by user space with the data submitted via sendpage/sendmsg. Filling
+ * up the TX SGL does not cause a crypto operation -- the data will only be
+ * tracked by the kernel. Upon receipt of one recvmsg call, the caller must
+ * provide a buffer which is tracked with the RX SGL.
+ *
+ * During the processing of the recvmsg operation, the cipher request is
+ * allocated and prepared. As part of the recvmsg operation, the processed
+ * TX buffers are extracted from the TX SGL into a separate SGL.
+ *
+ * After the completion of the crypto operation, the RX SGL and the cipher
+ * request is released. The extracted TX SGL parts are released together with
+ * the RX SGL release.
*/
#include <crypto/scatterwalk.h>
@@ -23,86 +38,58 @@
#include <linux/net.h>
#include <net/sock.h>
-struct skcipher_sg_list {
+struct skcipher_tsgl {
struct list_head list;
-
int cur;
-
struct scatterlist sg[0];
};
+struct skcipher_rsgl {
+ struct af_alg_sgl sgl;
+ struct list_head list;
+};
+
+struct skcipher_async_req {
+ struct kiocb *iocb;
+ struct sock *sk;
+
+ struct skcipher_rsgl first_sgl;
+ struct list_head rsgl_list;
+
+ struct scatterlist *tsgl;
+ unsigned int tsgl_entries;
+
+ unsigned int areqlen;
+ struct skcipher_request req;
+};
+
struct skcipher_tfm {
struct crypto_skcipher *skcipher;
bool has_key;
};
struct skcipher_ctx {
- struct list_head tsgl;
- struct af_alg_sgl rsgl;
+ struct list_head tsgl_list;
void *iv;
struct af_alg_completion completion;
- atomic_t inflight;
+ unsigned int inflight;
size_t used;
- unsigned int len;
bool more;
bool merge;
bool enc;
- struct skcipher_request req;
-};
-
-struct skcipher_async_rsgl {
- struct af_alg_sgl sgl;
- struct list_head list;
+ unsigned int len;
};
-struct skcipher_async_req {
- struct kiocb *iocb;
- struct skcipher_async_rsgl first_sgl;
- struct list_head list;
- struct scatterlist *tsg;
- atomic_t *inflight;
- struct skcipher_request req;
-};
+static DECLARE_WAIT_QUEUE_HEAD(skcipher_aio_finish_wait);
-#define MAX_SGL_ENTS ((4096 - sizeof(struct skcipher_sg_list)) / \
+#define MAX_SGL_ENTS ((4096 - sizeof(struct skcipher_tsgl)) / \
sizeof(struct scatterlist) - 1)
-static void skcipher_free_async_sgls(struct skcipher_async_req *sreq)
-{
- struct skcipher_async_rsgl *rsgl, *tmp;
- struct scatterlist *sgl;
- struct scatterlist *sg;
- int i, n;
-
- list_for_each_entry_safe(rsgl, tmp, &sreq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &sreq->first_sgl)
- kfree(rsgl);
- }
- sgl = sreq->tsg;
- n = sg_nents(sgl);
- for_each_sg(sgl, sg, n, i)
- put_page(sg_page(sg));
-
- kfree(sreq->tsg);
-}
-
-static void skcipher_async_cb(struct crypto_async_request *req, int err)
-{
- struct skcipher_async_req *sreq = req->data;
- struct kiocb *iocb = sreq->iocb;
-
- atomic_dec(sreq->inflight);
- skcipher_free_async_sgls(sreq);
- kzfree(sreq);
- iocb->ki_complete(iocb, err, err);
-}
-
static inline int skcipher_sndbuf(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
@@ -117,15 +104,15 @@ static inline bool skcipher_writable(struct sock *sk)
return PAGE_SIZE <= skcipher_sndbuf(sk);
}
-static int skcipher_alloc_sgl(struct sock *sk)
+static int skcipher_alloc_tsgl(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
struct scatterlist *sg = NULL;
- sgl = list_entry(ctx->tsgl.prev, struct skcipher_sg_list, list);
- if (!list_empty(&ctx->tsgl))
+ sgl = list_entry(ctx->tsgl_list.prev, struct skcipher_tsgl, list);
+ if (!list_empty(&ctx->tsgl_list))
sg = sgl->sg;
if (!sg || sgl->cur >= MAX_SGL_ENTS) {
@@ -141,31 +128,66 @@ static int skcipher_alloc_sgl(struct sock *sk)
if (sg)
sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg);
- list_add_tail(&sgl->list, &ctx->tsgl);
+ list_add_tail(&sgl->list, &ctx->tsgl_list);
}
return 0;
}
-static void skcipher_pull_sgl(struct sock *sk, size_t used, int put)
+static unsigned int skcipher_count_tsgl(struct sock *sk, size_t bytes)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct skcipher_ctx *ctx = ask->private;
+ struct skcipher_tsgl *sgl, *tmp;
+ unsigned int i;
+ unsigned int sgl_count = 0;
+
+ if (!bytes)
+ return 0;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ struct scatterlist *sg = sgl->sg;
+
+ for (i = 0; i < sgl->cur; i++) {
+ sgl_count++;
+ if (sg[i].length >= bytes)
+ return sgl_count;
+
+ bytes -= sg[i].length;
+ }
+ }
+
+ return sgl_count;
+}
+
+static void skcipher_pull_tsgl(struct sock *sk, size_t used,
+ struct scatterlist *dst)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
struct scatterlist *sg;
- int i;
+ unsigned int i;
- while (!list_empty(&ctx->tsgl)) {
- sgl = list_first_entry(&ctx->tsgl, struct skcipher_sg_list,
+ while (!list_empty(&ctx->tsgl_list)) {
+ sgl = list_first_entry(&ctx->tsgl_list, struct skcipher_tsgl,
list);
sg = sgl->sg;
for (i = 0; i < sgl->cur; i++) {
size_t plen = min_t(size_t, used, sg[i].length);
+ struct page *page = sg_page(sg + i);
- if (!sg_page(sg + i))
+ if (!page)
continue;
+ /*
+ * Assumption: caller created skcipher_count_tsgl(len)
+ * SG entries in dst.
+ */
+ if (dst)
+ sg_set_page(dst + i, page, plen, sg[i].offset);
+
sg[i].length -= plen;
sg[i].offset += plen;
@@ -174,27 +196,45 @@ static void skcipher_pull_sgl(struct sock *sk, size_t used, int put)
if (sg[i].length)
return;
- if (put)
- put_page(sg_page(sg + i));
+
+ if (!dst)
+ put_page(page);
sg_assign_page(sg + i, NULL);
}
list_del(&sgl->list);
- sock_kfree_s(sk, sgl,
- sizeof(*sgl) + sizeof(sgl->sg[0]) *
- (MAX_SGL_ENTS + 1));
+ sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) *
+ (MAX_SGL_ENTS + 1));
}
if (!ctx->used)
ctx->merge = 0;
}
-static void skcipher_free_sgl(struct sock *sk)
+static void skcipher_free_areq_sgls(struct skcipher_async_req *areq)
{
- struct alg_sock *ask = alg_sk(sk);
- struct skcipher_ctx *ctx = ask->private;
+ struct sock *sk = areq->sk;
+ struct skcipher_rsgl *rsgl, *tmp;
+ struct scatterlist *tsgl;
+ struct scatterlist *sg;
+ unsigned int i;
+
+ list_for_each_entry_safe(rsgl, tmp, &areq->rsgl_list, list) {
+ af_alg_free_sg(&rsgl->sgl);
+ list_del(&rsgl->list);
+ if (rsgl != &areq->first_sgl)
+ sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ }
+
+ tsgl = areq->tsgl;
+ for_each_sg(tsgl, sg, areq->tsgl_entries, i) {
+ if (!sg_page(sg))
+ continue;
+ put_page(sg_page(sg));
+ }
- skcipher_pull_sgl(sk, ctx->used, 1);
+ if (areq->tsgl && areq->tsgl_entries)
+ sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl));
}
static int skcipher_wait_for_wmem(struct sock *sk, unsigned flags)
@@ -301,7 +341,7 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
struct skcipher_tfm *skc = pask->private;
struct crypto_skcipher *tfm = skc->skcipher;
unsigned ivsize = crypto_skcipher_ivsize(tfm);
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
@@ -348,8 +388,8 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
size_t plen;
if (ctx->merge) {
- sgl = list_entry(ctx->tsgl.prev,
- struct skcipher_sg_list, list);
+ sgl = list_entry(ctx->tsgl_list.prev,
+ struct skcipher_tsgl, list);
sg = sgl->sg + sgl->cur - 1;
len = min_t(unsigned long, len,
PAGE_SIZE - sg->offset - sg->length);
@@ -378,11 +418,12 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
len = min_t(unsigned long, len, skcipher_sndbuf(sk));
- err = skcipher_alloc_sgl(sk);
+ err = skcipher_alloc_tsgl(sk);
if (err)
goto unlock;
- sgl = list_entry(ctx->tsgl.prev, struct skcipher_sg_list, list);
+ sgl = list_entry(ctx->tsgl_list.prev, struct skcipher_tsgl,
+ list);
sg = sgl->sg;
if (sgl->cur)
sg_unmark_end(sg + sgl->cur - 1);
@@ -434,7 +475,7 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
int err = -EINVAL;
if (flags & MSG_SENDPAGE_NOTLAST)
@@ -453,12 +494,12 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
goto unlock;
}
- err = skcipher_alloc_sgl(sk);
+ err = skcipher_alloc_tsgl(sk);
if (err)
goto unlock;
ctx->merge = 0;
- sgl = list_entry(ctx->tsgl.prev, struct skcipher_sg_list, list);
+ sgl = list_entry(ctx->tsgl_list.prev, struct skcipher_tsgl, list);
if (sgl->cur)
sg_unmark_end(sgl->sg + sgl->cur - 1);
@@ -479,25 +520,36 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
return err ?: size;
}
-static int skcipher_all_sg_nents(struct skcipher_ctx *ctx)
+static void skcipher_async_cb(struct crypto_async_request *req, int err)
{
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- int nents = 0;
+ struct skcipher_async_req *areq = req->data;
+ struct sock *sk = areq->sk;
+ struct alg_sock *ask = alg_sk(sk);
+ struct skcipher_ctx *ctx = ask->private;
+ struct kiocb *iocb = areq->iocb;
+ unsigned int resultlen;
- list_for_each_entry(sgl, &ctx->tsgl, list) {
- sg = sgl->sg;
+ lock_sock(sk);
- while (!sg->length)
- sg++;
+ BUG_ON(!ctx->inflight);
- nents += sg_nents(sg);
- }
- return nents;
+ /* Buffer size written by crypto operation. */
+ resultlen = areq->req.cryptlen;
+
+ skcipher_free_areq_sgls(areq);
+ sock_kfree_s(sk, areq, areq->areqlen);
+ __sock_put(sk);
+ ctx->inflight--;
+
+ iocb->ki_complete(iocb, err ? err : resultlen, 0);
+
+ release_sock(sk);
+
+ wake_up_interruptible(&skcipher_aio_finish_wait);
}
-static int skcipher_recvmsg_async(struct socket *sock, struct msghdr *msg,
- int flags)
+static int skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
@@ -506,215 +558,137 @@ static int skcipher_recvmsg_async(struct socket *sock, struct msghdr *msg,
struct skcipher_ctx *ctx = ask->private;
struct skcipher_tfm *skc = pask->private;
struct crypto_skcipher *tfm = skc->skcipher;
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- struct skcipher_async_req *sreq;
- struct skcipher_request *req;
- struct skcipher_async_rsgl *last_rsgl = NULL;
- unsigned int txbufs = 0, len = 0, tx_nents;
- unsigned int reqsize = crypto_skcipher_reqsize(tfm);
- unsigned int ivsize = crypto_skcipher_ivsize(tfm);
+ unsigned int bs = crypto_skcipher_blocksize(tfm);
+ unsigned int areqlen = sizeof(struct skcipher_async_req) +
+ crypto_skcipher_reqsize(tfm);
+ struct skcipher_async_req *areq;
+ struct skcipher_rsgl *last_rsgl = NULL;
int err = -ENOMEM;
- bool mark = false;
- char *iv;
-
- sreq = kzalloc(sizeof(*sreq) + reqsize + ivsize, GFP_KERNEL);
- if (unlikely(!sreq))
- goto out;
-
- req = &sreq->req;
- iv = (char *)(req + 1) + reqsize;
- sreq->iocb = msg->msg_iocb;
- INIT_LIST_HEAD(&sreq->list);
- sreq->inflight = &ctx->inflight;
+ size_t len = 0;
lock_sock(sk);
- tx_nents = skcipher_all_sg_nents(ctx);
- sreq->tsg = kcalloc(tx_nents, sizeof(*sg), GFP_KERNEL);
- if (unlikely(!sreq->tsg))
+
+ /* Allocate cipher request for current operation. */
+ areq = sock_kmalloc(sk, areqlen, GFP_KERNEL);
+ if (unlikely(!areq))
goto unlock;
- sg_init_table(sreq->tsg, tx_nents);
- memcpy(iv, ctx->iv, ivsize);
- skcipher_request_set_tfm(req, tfm);
- skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP,
- skcipher_async_cb, sreq);
+ areq->areqlen = areqlen;
+ areq->sk = sk;
+ INIT_LIST_HEAD(&areq->rsgl_list);
+ areq->tsgl = NULL;
+ areq->tsgl_entries = 0;
- while (iov_iter_count(&msg->msg_iter)) {
- struct skcipher_async_rsgl *rsgl;
- int used;
+ /* convert iovecs of output buffers into RX SGL */
+ while (len < ctx->used && iov_iter_count(&msg->msg_iter)) {
+ struct skcipher_rsgl *rsgl;
+ size_t seglen;
if (!ctx->used) {
err = skcipher_wait_for_data(sk, flags);
if (err)
goto free;
}
- sgl = list_first_entry(&ctx->tsgl,
- struct skcipher_sg_list, list);
- sg = sgl->sg;
- while (!sg->length)
- sg++;
-
- used = min_t(unsigned long, ctx->used,
- iov_iter_count(&msg->msg_iter));
- used = min_t(unsigned long, used, sg->length);
-
- if (txbufs == tx_nents) {
- struct scatterlist *tmp;
- int x;
- /* Ran out of tx slots in async request
- * need to expand */
- tmp = kcalloc(tx_nents * 2, sizeof(*tmp),
- GFP_KERNEL);
- if (!tmp) {
- err = -ENOMEM;
- goto free;
- }
+ seglen = min_t(size_t, ctx->used,
+ iov_iter_count(&msg->msg_iter));
- sg_init_table(tmp, tx_nents * 2);
- for (x = 0; x < tx_nents; x++)
- sg_set_page(&tmp[x], sg_page(&sreq->tsg[x]),
- sreq->tsg[x].length,
- sreq->tsg[x].offset);
- kfree(sreq->tsg);
- sreq->tsg = tmp;
- tx_nents *= 2;
- mark = true;
- }
- /* Need to take over the tx sgl from ctx
- * to the asynch req - these sgls will be freed later */
- sg_set_page(sreq->tsg + txbufs++, sg_page(sg), sg->length,
- sg->offset);
-
- if (list_empty(&sreq->list)) {
- rsgl = &sreq->first_sgl;
- list_add_tail(&rsgl->list, &sreq->list);
+ if (list_empty(&areq->rsgl_list)) {
+ rsgl = &areq->first_sgl;
} else {
- rsgl = kmalloc(sizeof(*rsgl), GFP_KERNEL);
+ rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
if (!rsgl) {
err = -ENOMEM;
goto free;
}
- list_add_tail(&rsgl->list, &sreq->list);
}
- used = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, used);
- err = used;
- if (used < 0)
+ rsgl->sgl.npages = 0;
+ list_add_tail(&rsgl->list, &areq->rsgl_list);
+
+ /* make one iovec available as scatterlist */
+ err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
+ if (err < 0)
goto free;
+
+ /* chain the new scatterlist with previous one */
if (last_rsgl)
af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
last_rsgl = rsgl;
- len += used;
- skcipher_pull_sgl(sk, used, 0);
- iov_iter_advance(&msg->msg_iter, used);
+ len += err;
+ iov_iter_advance(&msg->msg_iter, err);
}
- if (mark)
- sg_mark_end(sreq->tsg + txbufs - 1);
+ /* Process only as much RX buffers for which we have TX data */
+ if (len > ctx->used)
+ len = ctx->used;
+
+ /*
+ * If more buffers are to be expected to be processed, process only
+ * full block size buffers.
+ */
+ if (ctx->more || len < ctx->used)
+ len -= len % bs;
+
+ /*
+ * Create a per request TX SGL for this request which tracks the
+ * SG entries from the global TX SGL.
+ */
+ areq->tsgl_entries = skcipher_count_tsgl(sk, len);
+ if (!areq->tsgl_entries)
+ areq->tsgl_entries = 1;
+ areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) * areq->tsgl_entries,
+ GFP_KERNEL);
+ if (!areq->tsgl) {
+ err = -ENOMEM;
+ goto free;
+ }
+ sg_init_table(areq->tsgl, areq->tsgl_entries);
+ skcipher_pull_tsgl(sk, len, areq->tsgl);
+
+ /* Initialize the crypto operation */
+ skcipher_request_set_tfm(&areq->req, tfm);
+ skcipher_request_set_crypt(&areq->req, areq->tsgl,
+ areq->first_sgl.sgl.sg, len, ctx->iv);
+
+ if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
+ /* AIO operation */
+ areq->iocb = msg->msg_iocb;
+ skcipher_request_set_callback(&areq->req,
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ skcipher_async_cb, areq);
+ err = ctx->enc ? crypto_skcipher_encrypt(&areq->req) :
+ crypto_skcipher_decrypt(&areq->req);
+ } else {
+ /* Synchronous operation */
+ skcipher_request_set_callback(&areq->req,
+ CRYPTO_TFM_REQ_MAY_SLEEP |
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ af_alg_complete,
+ &ctx->completion);
+ err = af_alg_wait_for_completion(ctx->enc ?
+ crypto_skcipher_encrypt(&areq->req) :
+ crypto_skcipher_decrypt(&areq->req),
+ &ctx->completion);
+ }
- skcipher_request_set_crypt(req, sreq->tsg, sreq->first_sgl.sgl.sg,
- len, iv);
- err = ctx->enc ? crypto_skcipher_encrypt(req) :
- crypto_skcipher_decrypt(req);
+ /* AIO operation in progress */
if (err == -EINPROGRESS) {
- atomic_inc(&ctx->inflight);
+ sock_hold(sk);
err = -EIOCBQUEUED;
- sreq = NULL;
+ ctx->inflight++;
goto unlock;
}
-free:
- skcipher_free_async_sgls(sreq);
-unlock:
- skcipher_wmem_wakeup(sk);
- release_sock(sk);
- kzfree(sreq);
-out:
- return err;
-}
-
-static int skcipher_recvmsg_sync(struct socket *sock, struct msghdr *msg,
- int flags)
-{
- struct sock *sk = sock->sk;
- struct alg_sock *ask = alg_sk(sk);
- struct sock *psk = ask->parent;
- struct alg_sock *pask = alg_sk(psk);
- struct skcipher_ctx *ctx = ask->private;
- struct skcipher_tfm *skc = pask->private;
- struct crypto_skcipher *tfm = skc->skcipher;
- unsigned bs = crypto_skcipher_blocksize(tfm);
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- int err = -EAGAIN;
- int used;
- long copied = 0;
-
- lock_sock(sk);
- while (msg_data_left(msg)) {
- if (!ctx->used) {
- err = skcipher_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- used = min_t(unsigned long, ctx->used, msg_data_left(msg));
-
- used = af_alg_make_sg(&ctx->rsgl, &msg->msg_iter, used);
- err = used;
- if (err < 0)
- goto unlock;
-
- if (ctx->more || used < ctx->used)
- used -= used % bs;
-
- err = -EINVAL;
- if (!used)
- goto free;
-
- sgl = list_first_entry(&ctx->tsgl,
- struct skcipher_sg_list, list);
- sg = sgl->sg;
-
- while (!sg->length)
- sg++;
-
- skcipher_request_set_crypt(&ctx->req, sg, ctx->rsgl.sg, used,
- ctx->iv);
-
- err = af_alg_wait_for_completion(
- ctx->enc ?
- crypto_skcipher_encrypt(&ctx->req) :
- crypto_skcipher_decrypt(&ctx->req),
- &ctx->completion);
free:
- af_alg_free_sg(&ctx->rsgl);
-
- if (err)
- goto unlock;
-
- copied += used;
- skcipher_pull_sgl(sk, used, 1);
- iov_iter_advance(&msg->msg_iter, used);
- }
-
- err = 0;
+ skcipher_free_areq_sgls(areq);
+ if (areq)
+ sock_kfree_s(sk, areq, areqlen);
unlock:
skcipher_wmem_wakeup(sk);
release_sock(sk);
-
- return copied ?: err;
-}
-
-static int skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
- size_t ignored, int flags)
-{
- return (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) ?
- skcipher_recvmsg_async(sock, msg, flags) :
- skcipher_recvmsg_sync(sock, msg, flags);
+ return err ? err : len;
}
static unsigned int skcipher_poll(struct file *file, struct socket *sock,
@@ -723,10 +697,9 @@ static unsigned int skcipher_poll(struct file *file, struct socket *sock,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- unsigned int mask;
+ unsigned int mask = 0;
sock_poll_wait(file, sk_sleep(sk), wait);
- mask = 0;
if (ctx->used)
mask |= POLLIN | POLLRDNORM;
@@ -894,26 +867,20 @@ static int skcipher_setkey(void *private, const u8 *key, unsigned int keylen)
return err;
}
-static void skcipher_wait(struct sock *sk)
-{
- struct alg_sock *ask = alg_sk(sk);
- struct skcipher_ctx *ctx = ask->private;
- int ctr = 0;
-
- while (atomic_read(&ctx->inflight) && ctr++ < 100)
- msleep(100);
-}
-
static void skcipher_sock_destruct(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(&ctx->req);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
+ struct skcipher_tfm *skc = pask->private;
+ struct crypto_skcipher *tfm = skc->skcipher;
- if (atomic_read(&ctx->inflight))
- skcipher_wait(sk);
+ /* Suspend caller if AIO operations are in flight. */
+ wait_event_interruptible(skcipher_aio_finish_wait,
+ (ctx->inflight == 0));
- skcipher_free_sgl(sk);
+ skcipher_pull_tsgl(sk, ctx->used, NULL);
sock_kzfree_s(sk, ctx->iv, crypto_skcipher_ivsize(tfm));
sock_kfree_s(sk, ctx, ctx->len);
af_alg_release_parent(sk);
@@ -925,7 +892,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct skcipher_tfm *tfm = private;
struct crypto_skcipher *skcipher = tfm->skcipher;
- unsigned int len = sizeof(*ctx) + crypto_skcipher_reqsize(skcipher);
+ unsigned int len = sizeof(*ctx);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
@@ -940,22 +907,17 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
memset(ctx->iv, 0, crypto_skcipher_ivsize(skcipher));
- INIT_LIST_HEAD(&ctx->tsgl);
+ INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
ctx->used = 0;
+ ctx->inflight = 0;
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
- atomic_set(&ctx->inflight, 0);
af_alg_init_completion(&ctx->completion);
ask->private = ctx;
- skcipher_request_set_tfm(&ctx->req, skcipher);
- skcipher_request_set_callback(&ctx->req, CRYPTO_TFM_REQ_MAY_SLEEP |
- CRYPTO_TFM_REQ_MAY_BACKLOG,
- af_alg_complete, &ctx->completion);
-
sk->sk_destruct = skcipher_sock_destruct;
return 0;
--
2.9.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox