* Re: [PATCH 0/7] arm64: marvell: add cryptographic engine support for 7k/8k
From: Herbert Xu @ 2017-04-12 6:11 UTC (permalink / raw)
To: Gregory CLEMENT
Cc: Antoine Tenart, davem, jason, andrew, sebastian.hesselbarth,
linux-crypto, linux-arm-kernel, thomas.petazzoni, boris.brezillon,
oferh, igall, nadavh
In-Reply-To: <87efwyj53q.fsf@free-electrons.com>
On Tue, Apr 11, 2017 at 06:12:25PM +0200, Gregory CLEMENT wrote:
>
> > I have no problems with the crypto bits.
>
> Does it means that you agree that I apply the arm-soc related patches
> (from 4 to 7) to my mvebu trees?
Yes sure.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [PATCH v5 2/2] crypto: hw_random - Add new Exynos RNG driver
From: Krzysztof Kozlowski @ 2017-04-11 18:08 UTC (permalink / raw)
To: James Hogan, Matt Mackall, Herbert Xu, Kukjin Kim,
Krzysztof Kozlowski, Javier Martinez Canillas, David S. Miller,
Dave Airlie, David Airlie, Bernie Thompson,
Bartlomiej Zolnierkiewicz, Greg Kroah-Hartman,
PrasannaKumar Muralidharan, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-metag-u79uwXL29TY76Z2rM5mHXA,
linux-crypto-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-samsung-soc-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-fbdev-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20170411180835.8928-1-krzk-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Replace existing hw_ranndom/exynos-rng driver with a new, reworked one.
This is a driver for pseudo random number generator block which on
Exynos4 chipsets must be seeded with some value. On newer Exynos5420
chipsets it might seed itself from true random number generator block
but this is not implemented yet.
New driver is a complete rework to use the crypto ALGAPI instead of
hw_random API. Rationale for the change:
1. hw_random interface is for true RNG devices.
2. The old driver was seeding itself with jiffies which is not a
reliable source for randomness.
3. Device generates five random 32-bit numbers in each pass but old
driver was returning only one 32-bit number thus its performance was
reduced.
Compatibility with DeviceTree bindings is preserved.
New driver does not use runtime power management but manually enables
and disables the clock when needed. This is preferred approach because
using runtime PM just to toggle clock is huge overhead.
Another difference is reseeding itself with generated random data
periodically and during resuming from system suspend (previously driver
was re-seeding itself again with jiffies).
Signed-off-by: Krzysztof Kozlowski <krzk-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Reviewed-by: Stephan Müller <smueller-T9tCv8IpfcWELgA04lAiVw@public.gmane.org>
Reviewed-by: PrasannaKumar Muralidharan <prasannatsmkumar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Reviewed-by: Bartlomiej Zolnierkiewicz <b.zolnierkie-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
---
MAINTAINERS | 8 +
drivers/char/hw_random/Kconfig | 14 --
drivers/char/hw_random/Makefile | 1 -
drivers/char/hw_random/exynos-rng.c | 231 ---------------------
drivers/crypto/Kconfig | 15 ++
drivers/crypto/Makefile | 1 +
drivers/crypto/exynos-rng.c | 389 ++++++++++++++++++++++++++++++++++++
7 files changed, 413 insertions(+), 246 deletions(-)
delete mode 100644 drivers/char/hw_random/exynos-rng.c
create mode 100644 drivers/crypto/exynos-rng.c
diff --git a/MAINTAINERS b/MAINTAINERS
index affecc6d59f4..371fda859d43 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -10977,6 +10977,14 @@ L: alsa-devel-K7yf7f+aM1XWsZ/bQMPhNw@public.gmane.org (moderated for non-subscribers)
S: Supported
F: sound/soc/samsung/
+SAMSUNG EXYNOS PSEUDO RANDOM NUMBER GENERATOR (RNG) DRIVER
+M: Krzysztof Kozlowski <krzk-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
+L: linux-crypto-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
+L: linux-samsung-soc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
+S: Maintained
+F: drivers/crypto/exynos-rng.c
+F: Documentation/devicetree/bindings/rng/samsung,exynos-rng4.txt
+
SAMSUNG FRAMEBUFFER DRIVER
M: Jingoo Han <jingoohan1-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
L: linux-fbdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
diff --git a/drivers/char/hw_random/Kconfig b/drivers/char/hw_random/Kconfig
index 0cafe08919c9..bdae802e7154 100644
--- a/drivers/char/hw_random/Kconfig
+++ b/drivers/char/hw_random/Kconfig
@@ -294,20 +294,6 @@ config HW_RANDOM_POWERNV
If unsure, say Y.
-config HW_RANDOM_EXYNOS
- tristate "EXYNOS HW random number generator support"
- depends on ARCH_EXYNOS || COMPILE_TEST
- depends on HAS_IOMEM
- default HW_RANDOM
- ---help---
- This driver provides kernel-side support for the Random Number
- Generator hardware found on EXYNOS SOCs.
-
- To compile this driver as a module, choose M here: the
- module will be called exynos-rng.
-
- If unsure, say Y.
-
config HW_RANDOM_TPM
tristate "TPM HW Random Number Generator support"
depends on TCG_TPM
diff --git a/drivers/char/hw_random/Makefile b/drivers/char/hw_random/Makefile
index 5f52b1e4e7be..6f1eecc2045c 100644
--- a/drivers/char/hw_random/Makefile
+++ b/drivers/char/hw_random/Makefile
@@ -24,7 +24,6 @@ obj-$(CONFIG_HW_RANDOM_OCTEON) += octeon-rng.o
obj-$(CONFIG_HW_RANDOM_NOMADIK) += nomadik-rng.o
obj-$(CONFIG_HW_RANDOM_PSERIES) += pseries-rng.o
obj-$(CONFIG_HW_RANDOM_POWERNV) += powernv-rng.o
-obj-$(CONFIG_HW_RANDOM_EXYNOS) += exynos-rng.o
obj-$(CONFIG_HW_RANDOM_HISI) += hisi-rng.o
obj-$(CONFIG_HW_RANDOM_TPM) += tpm-rng.o
obj-$(CONFIG_HW_RANDOM_BCM2835) += bcm2835-rng.o
diff --git a/drivers/char/hw_random/exynos-rng.c b/drivers/char/hw_random/exynos-rng.c
deleted file mode 100644
index 23d358553b21..000000000000
--- a/drivers/char/hw_random/exynos-rng.c
+++ /dev/null
@@ -1,231 +0,0 @@
-/*
- * exynos-rng.c - Random Number Generator driver for the exynos
- *
- * Copyright (C) 2012 Samsung Electronics
- * Jonghwa Lee <jonghwa3.lee-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation;
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- *
- */
-
-#include <linux/hw_random.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/io.h>
-#include <linux/platform_device.h>
-#include <linux/clk.h>
-#include <linux/pm_runtime.h>
-#include <linux/err.h>
-
-#define EXYNOS_PRNG_STATUS_OFFSET 0x10
-#define EXYNOS_PRNG_SEED_OFFSET 0x140
-#define EXYNOS_PRNG_OUT1_OFFSET 0x160
-#define SEED_SETTING_DONE BIT(1)
-#define PRNG_START 0x18
-#define PRNG_DONE BIT(5)
-#define EXYNOS_AUTOSUSPEND_DELAY 100
-
-struct exynos_rng {
- struct device *dev;
- struct hwrng rng;
- void __iomem *mem;
- struct clk *clk;
-};
-
-static u32 exynos_rng_readl(struct exynos_rng *rng, u32 offset)
-{
- return readl_relaxed(rng->mem + offset);
-}
-
-static void exynos_rng_writel(struct exynos_rng *rng, u32 val, u32 offset)
-{
- writel_relaxed(val, rng->mem + offset);
-}
-
-static int exynos_rng_configure(struct exynos_rng *exynos_rng)
-{
- int i;
- int ret = 0;
-
- for (i = 0 ; i < 5 ; i++)
- exynos_rng_writel(exynos_rng, jiffies,
- EXYNOS_PRNG_SEED_OFFSET + 4*i);
-
- if (!(exynos_rng_readl(exynos_rng, EXYNOS_PRNG_STATUS_OFFSET)
- & SEED_SETTING_DONE))
- ret = -EIO;
-
- return ret;
-}
-
-static int exynos_init(struct hwrng *rng)
-{
- struct exynos_rng *exynos_rng = container_of(rng,
- struct exynos_rng, rng);
- int ret = 0;
-
- pm_runtime_get_sync(exynos_rng->dev);
- ret = exynos_rng_configure(exynos_rng);
- pm_runtime_mark_last_busy(exynos_rng->dev);
- pm_runtime_put_autosuspend(exynos_rng->dev);
-
- return ret;
-}
-
-static int exynos_read(struct hwrng *rng, void *buf,
- size_t max, bool wait)
-{
- struct exynos_rng *exynos_rng = container_of(rng,
- struct exynos_rng, rng);
- u32 *data = buf;
- int retry = 100;
- int ret = 4;
-
- pm_runtime_get_sync(exynos_rng->dev);
-
- exynos_rng_writel(exynos_rng, PRNG_START, 0);
-
- while (!(exynos_rng_readl(exynos_rng,
- EXYNOS_PRNG_STATUS_OFFSET) & PRNG_DONE) && --retry)
- cpu_relax();
- if (!retry) {
- ret = -ETIMEDOUT;
- goto out;
- }
-
- exynos_rng_writel(exynos_rng, PRNG_DONE, EXYNOS_PRNG_STATUS_OFFSET);
-
- *data = exynos_rng_readl(exynos_rng, EXYNOS_PRNG_OUT1_OFFSET);
-
-out:
- pm_runtime_mark_last_busy(exynos_rng->dev);
- pm_runtime_put_sync_autosuspend(exynos_rng->dev);
-
- return ret;
-}
-
-static int exynos_rng_probe(struct platform_device *pdev)
-{
- struct exynos_rng *exynos_rng;
- struct resource *res;
- int ret;
-
- exynos_rng = devm_kzalloc(&pdev->dev, sizeof(struct exynos_rng),
- GFP_KERNEL);
- if (!exynos_rng)
- return -ENOMEM;
-
- exynos_rng->dev = &pdev->dev;
- exynos_rng->rng.name = "exynos";
- exynos_rng->rng.init = exynos_init;
- exynos_rng->rng.read = exynos_read;
- exynos_rng->clk = devm_clk_get(&pdev->dev, "secss");
- if (IS_ERR(exynos_rng->clk)) {
- dev_err(&pdev->dev, "Couldn't get clock.\n");
- return -ENOENT;
- }
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- exynos_rng->mem = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(exynos_rng->mem))
- return PTR_ERR(exynos_rng->mem);
-
- platform_set_drvdata(pdev, exynos_rng);
-
- pm_runtime_set_autosuspend_delay(&pdev->dev, EXYNOS_AUTOSUSPEND_DELAY);
- pm_runtime_use_autosuspend(&pdev->dev);
- pm_runtime_enable(&pdev->dev);
-
- ret = devm_hwrng_register(&pdev->dev, &exynos_rng->rng);
- if (ret) {
- pm_runtime_dont_use_autosuspend(&pdev->dev);
- pm_runtime_disable(&pdev->dev);
- }
-
- return ret;
-}
-
-static int exynos_rng_remove(struct platform_device *pdev)
-{
- pm_runtime_dont_use_autosuspend(&pdev->dev);
- pm_runtime_disable(&pdev->dev);
-
- return 0;
-}
-
-static int __maybe_unused exynos_rng_runtime_suspend(struct device *dev)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct exynos_rng *exynos_rng = platform_get_drvdata(pdev);
-
- clk_disable_unprepare(exynos_rng->clk);
-
- return 0;
-}
-
-static int __maybe_unused exynos_rng_runtime_resume(struct device *dev)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct exynos_rng *exynos_rng = platform_get_drvdata(pdev);
-
- return clk_prepare_enable(exynos_rng->clk);
-}
-
-static int __maybe_unused exynos_rng_suspend(struct device *dev)
-{
- return pm_runtime_force_suspend(dev);
-}
-
-static int __maybe_unused exynos_rng_resume(struct device *dev)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct exynos_rng *exynos_rng = platform_get_drvdata(pdev);
- int ret;
-
- ret = pm_runtime_force_resume(dev);
- if (ret)
- return ret;
-
- return exynos_rng_configure(exynos_rng);
-}
-
-static const struct dev_pm_ops exynos_rng_pm_ops = {
- SET_SYSTEM_SLEEP_PM_OPS(exynos_rng_suspend, exynos_rng_resume)
- SET_RUNTIME_PM_OPS(exynos_rng_runtime_suspend,
- exynos_rng_runtime_resume, NULL)
-};
-
-static const struct of_device_id exynos_rng_dt_match[] = {
- {
- .compatible = "samsung,exynos4-rng",
- },
- { },
-};
-MODULE_DEVICE_TABLE(of, exynos_rng_dt_match);
-
-static struct platform_driver exynos_rng_driver = {
- .driver = {
- .name = "exynos-rng",
- .pm = &exynos_rng_pm_ops,
- .of_match_table = exynos_rng_dt_match,
- },
- .probe = exynos_rng_probe,
- .remove = exynos_rng_remove,
-};
-
-module_platform_driver(exynos_rng_driver);
-
-MODULE_DESCRIPTION("EXYNOS 4 H/W Random Number Generator driver");
-MODULE_AUTHOR("Jonghwa Lee <jonghwa3.lee-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>");
-MODULE_LICENSE("GPL");
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 1a60626937e4..7d1fdf6a751c 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -388,6 +388,21 @@ config CRYPTO_DEV_MXC_SCC
This option enables support for the Security Controller (SCC)
found in Freescale i.MX25 chips.
+config CRYPTO_DEV_EXYNOS_RNG
+ tristate "EXYNOS HW pseudo random number generator support"
+ depends on ARCH_EXYNOS || COMPILE_TEST
+ depends on HAS_IOMEM
+ select CRYPTO_RNG
+ ---help---
+ This driver provides kernel-side support through the
+ cryptographic API for the pseudo random number generator hardware
+ found on Exynos SoCs.
+
+ To compile this driver as a module, choose M here: the
+ module will be called exynos-rng.
+
+ If unsure, say Y.
+
config CRYPTO_DEV_S5P
tristate "Support for Samsung S5PV210/Exynos crypto accelerator"
depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index 41ca339e89d3..9603f1862b30 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += cavium/
obj-$(CONFIG_CRYPTO_DEV_CCP) += ccp/
obj-$(CONFIG_CRYPTO_DEV_CHELSIO) += chelsio/
obj-$(CONFIG_CRYPTO_DEV_CPT) += cavium/cpt/
+obj-$(CONFIG_CRYPTO_DEV_EXYNOS_RNG) += exynos-rng.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam/
obj-$(CONFIG_CRYPTO_DEV_GEODE) += geode-aes.o
obj-$(CONFIG_CRYPTO_DEV_HIFN_795X) += hifn_795x.o
diff --git a/drivers/crypto/exynos-rng.c b/drivers/crypto/exynos-rng.c
new file mode 100644
index 000000000000..451620b475a0
--- /dev/null
+++ b/drivers/crypto/exynos-rng.c
@@ -0,0 +1,389 @@
+/*
+ * exynos-rng.c - Random Number Generator driver for the Exynos
+ *
+ * Copyright (c) 2017 Krzysztof Kozlowski <krzk-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
+ *
+ * Loosely based on old driver from drivers/char/hw_random/exynos-rng.c:
+ * Copyright (C) 2012 Samsung Electronics
+ * Jonghwa Lee <jonghwa3.lee-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation;
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/clk.h>
+#include <linux/crypto.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include <crypto/internal/rng.h>
+
+#define EXYNOS_RNG_CONTROL 0x0
+#define EXYNOS_RNG_STATUS 0x10
+#define EXYNOS_RNG_SEED_BASE 0x140
+#define EXYNOS_RNG_SEED(n) (EXYNOS_RNG_SEED_BASE + (n * 0x4))
+#define EXYNOS_RNG_OUT_BASE 0x160
+#define EXYNOS_RNG_OUT(n) (EXYNOS_RNG_OUT_BASE + (n * 0x4))
+
+/* EXYNOS_RNG_CONTROL bit fields */
+#define EXYNOS_RNG_CONTROL_START 0x18
+/* EXYNOS_RNG_STATUS bit fields */
+#define EXYNOS_RNG_STATUS_SEED_SETTING_DONE BIT(1)
+#define EXYNOS_RNG_STATUS_RNG_DONE BIT(5)
+
+/* Five seed and output registers, each 4 bytes */
+#define EXYNOS_RNG_SEED_REGS 5
+#define EXYNOS_RNG_SEED_SIZE (EXYNOS_RNG_SEED_REGS * 4)
+
+/*
+ * Driver re-seeds itself with generated random numbers to increase
+ * the randomness.
+ *
+ * Time for next re-seed in ms.
+ */
+#define EXYNOS_RNG_RESEED_TIME 100
+/*
+ * In polling mode, do not wait infinitely for the engine to finish the work.
+ */
+#define EXYNOS_RNG_WAIT_RETRIES 100
+
+/* Context for crypto */
+struct exynos_rng_ctx {
+ struct exynos_rng_dev *rng;
+};
+
+/* Device associated memory */
+struct exynos_rng_dev {
+ struct device *dev;
+ void __iomem *mem;
+ struct clk *clk;
+ /* Generated numbers stored for seeding during resume */
+ u8 seed_save[EXYNOS_RNG_SEED_SIZE];
+ unsigned int seed_save_len;
+ /* Time of last seeding in jiffies */
+ unsigned long last_seeding;
+};
+
+static struct exynos_rng_dev *exynos_rng_dev;
+
+static u32 exynos_rng_readl(struct exynos_rng_dev *rng, u32 offset)
+{
+ return readl_relaxed(rng->mem + offset);
+}
+
+static void exynos_rng_writel(struct exynos_rng_dev *rng, u32 val, u32 offset)
+{
+ writel_relaxed(val, rng->mem + offset);
+}
+
+static int exynos_rng_set_seed(struct exynos_rng_dev *rng,
+ const u8 *seed, unsigned int slen)
+{
+ u32 val;
+ int i;
+
+ /* Round seed length because loop iterates over full register size */
+ slen = ALIGN_DOWN(slen, 4);
+
+ if (slen < EXYNOS_RNG_SEED_SIZE)
+ return -EINVAL;
+
+ for (i = 0; i < slen ; i += 4) {
+ unsigned int seed_reg = (i / 4) % EXYNOS_RNG_SEED_REGS;
+
+ val = seed[i] << 24;
+ val |= seed[i + 1] << 16;
+ val |= seed[i + 2] << 8;
+ val |= seed[i + 3] << 0;
+
+ exynos_rng_writel(rng, val, EXYNOS_RNG_SEED(seed_reg));
+ }
+
+ val = exynos_rng_readl(rng, EXYNOS_RNG_STATUS);
+ if (!(val & EXYNOS_RNG_STATUS_SEED_SETTING_DONE)) {
+ dev_warn(rng->dev, "Seed setting not finished\n");
+ return -EIO;
+ }
+
+ rng->last_seeding = jiffies;
+
+ return 0;
+}
+
+/*
+ * Read from output registers and put the data under 'dst' array,
+ * up to dlen bytes.
+ *
+ * Returns number of bytes actually stored in 'dst' (dlen
+ * or EXYNOS_RNG_SEED_SIZE).
+ */
+static unsigned int exynos_rng_copy_random(struct exynos_rng_dev *rng,
+ u8 *dst, unsigned int dlen)
+{
+ unsigned int cnt = 0;
+ int i, j;
+ u32 val;
+
+ for (j = 0; j < EXYNOS_RNG_SEED_REGS; j++) {
+ val = exynos_rng_readl(rng, EXYNOS_RNG_OUT(j));
+
+ for (i = 0; i < 4; i++) {
+ dst[cnt] = val & 0xff;
+ val >>= 8;
+ if (++cnt >= dlen)
+ return cnt;
+ }
+ }
+
+ return cnt;
+}
+
+/*
+ * Start the engine and poll for finish. Then read from output registers
+ * filling the 'dst' buffer up to 'dlen' bytes or up to size of generated
+ * random data (EXYNOS_RNG_SEED_SIZE).
+ *
+ * On success: return 0 and store number of read bytes under 'read' address.
+ * On error: return -ERRNO.
+ */
+static int exynos_rng_get_random(struct exynos_rng_dev *rng,
+ u8 *dst, unsigned int dlen,
+ unsigned int *read)
+{
+ int retry = EXYNOS_RNG_WAIT_RETRIES;
+
+ exynos_rng_writel(rng, EXYNOS_RNG_CONTROL_START,
+ EXYNOS_RNG_CONTROL);
+
+ while (!(exynos_rng_readl(rng,
+ EXYNOS_RNG_STATUS) & EXYNOS_RNG_STATUS_RNG_DONE) && --retry)
+ cpu_relax();
+
+ if (!retry)
+ return -ETIMEDOUT;
+
+ /* Clear status bit */
+ exynos_rng_writel(rng, EXYNOS_RNG_STATUS_RNG_DONE,
+ EXYNOS_RNG_STATUS);
+ *read = exynos_rng_copy_random(rng, dst, dlen);
+
+ return 0;
+}
+
+/* Re-seed itself from time to time */
+static void exynos_rng_reseed(struct exynos_rng_dev *rng)
+{
+ unsigned long next_seeding = rng->last_seeding + \
+ msecs_to_jiffies(EXYNOS_RNG_RESEED_TIME);
+ unsigned long now = jiffies;
+ unsigned int read = 0;
+ u8 seed[EXYNOS_RNG_SEED_SIZE];
+
+ if (time_before(now, next_seeding))
+ return;
+
+ if (exynos_rng_get_random(rng, seed, sizeof(seed), &read))
+ return;
+
+ exynos_rng_set_seed(rng, seed, read);
+}
+
+static int exynos_rng_generate(struct crypto_rng *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int dlen)
+{
+ struct exynos_rng_ctx *ctx = crypto_rng_ctx(tfm);
+ struct exynos_rng_dev *rng = ctx->rng;
+ unsigned int read = 0;
+ int ret;
+
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ do {
+ ret = exynos_rng_get_random(rng, dst, dlen, &read);
+ if (ret)
+ break;
+
+ dlen -= read;
+ dst += read;
+
+ exynos_rng_reseed(rng);
+ } while (dlen > 0);
+
+ clk_disable_unprepare(rng->clk);
+
+ return ret;
+}
+
+static int exynos_rng_seed(struct crypto_rng *tfm, const u8 *seed,
+ unsigned int slen)
+{
+ struct exynos_rng_ctx *ctx = crypto_rng_ctx(tfm);
+ struct exynos_rng_dev *rng = ctx->rng;
+ int ret;
+
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ ret = exynos_rng_set_seed(ctx->rng, seed, slen);
+
+ clk_disable_unprepare(rng->clk);
+
+ return ret;
+}
+
+static int exynos_rng_kcapi_init(struct crypto_tfm *tfm)
+{
+ struct exynos_rng_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ ctx->rng = exynos_rng_dev;
+
+ return 0;
+}
+
+static struct rng_alg exynos_rng_alg = {
+ .generate = exynos_rng_generate,
+ .seed = exynos_rng_seed,
+ .seedsize = EXYNOS_RNG_SEED_SIZE,
+ .base = {
+ .cra_name = "stdrng",
+ .cra_driver_name = "exynos_rng",
+ .cra_priority = 100,
+ .cra_ctxsize = sizeof(struct exynos_rng_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_init = exynos_rng_kcapi_init,
+ }
+};
+
+static int exynos_rng_probe(struct platform_device *pdev)
+{
+ struct exynos_rng_dev *rng;
+ struct resource *res;
+ int ret;
+
+ if (exynos_rng_dev)
+ return -EEXIST;
+
+ rng = devm_kzalloc(&pdev->dev, sizeof(*rng), GFP_KERNEL);
+ if (!rng)
+ return -ENOMEM;
+
+ rng->dev = &pdev->dev;
+ rng->clk = devm_clk_get(&pdev->dev, "secss");
+ if (IS_ERR(rng->clk)) {
+ dev_err(&pdev->dev, "Couldn't get clock.\n");
+ return PTR_ERR(rng->clk);
+ }
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ rng->mem = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(rng->mem))
+ return PTR_ERR(rng->mem);
+
+ platform_set_drvdata(pdev, rng);
+
+ exynos_rng_dev = rng;
+
+ ret = crypto_register_rng(&exynos_rng_alg);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "Couldn't register rng crypto alg: %d\n", ret);
+ exynos_rng_dev = NULL;
+ }
+
+ return ret;
+}
+
+static int exynos_rng_remove(struct platform_device *pdev)
+{
+ crypto_unregister_rng(&exynos_rng_alg);
+
+ exynos_rng_dev = NULL;
+
+ return 0;
+}
+
+static int __maybe_unused exynos_rng_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct exynos_rng_dev *rng = platform_get_drvdata(pdev);
+ int ret;
+
+ /* If we were never seeded then after resume it will be the same */
+ if (!rng->last_seeding)
+ return 0;
+
+ rng->seed_save_len = 0;
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ /* Get new random numbers and store them for seeding on resume. */
+ exynos_rng_get_random(rng, rng->seed_save, sizeof(rng->seed_save),
+ &(rng->seed_save_len));
+ dev_dbg(rng->dev, "Stored %u bytes for seeding on system resume\n",
+ rng->seed_save_len);
+
+ clk_disable_unprepare(rng->clk);
+
+ return 0;
+}
+
+static int __maybe_unused exynos_rng_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct exynos_rng_dev *rng = platform_get_drvdata(pdev);
+ int ret;
+
+ /* Never seeded so nothing to do */
+ if (!rng->last_seeding)
+ return 0;
+
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ ret = exynos_rng_set_seed(rng, rng->seed_save, rng->seed_save_len);
+
+ clk_disable_unprepare(rng->clk);
+
+ return ret;
+}
+
+static SIMPLE_DEV_PM_OPS(exynos_rng_pm_ops, exynos_rng_suspend,
+ exynos_rng_resume);
+
+static const struct of_device_id exynos_rng_dt_match[] = {
+ {
+ .compatible = "samsung,exynos4-rng",
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, exynos_rng_dt_match);
+
+static struct platform_driver exynos_rng_driver = {
+ .driver = {
+ .name = "exynos-rng",
+ .pm = &exynos_rng_pm_ops,
+ .of_match_table = exynos_rng_dt_match,
+ },
+ .probe = exynos_rng_probe,
+ .remove = exynos_rng_remove,
+};
+
+module_platform_driver(exynos_rng_driver);
+
+MODULE_DESCRIPTION("Exynos H/W Random Number Generator driver");
+MODULE_AUTHOR("Krzysztof Kozlowski <krzk-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>");
+MODULE_LICENSE("GPL");
--
2.9.3
^ permalink raw reply related
* [PATCH v5 1/2] linux/kernel.h: Add ALIGN_DOWN macro
From: Krzysztof Kozlowski @ 2017-04-11 18:08 UTC (permalink / raw)
To: James Hogan, Matt Mackall, Herbert Xu, Kukjin Kim,
Krzysztof Kozlowski, Javier Martinez Canillas, David S. Miller,
Dave Airlie, David Airlie, Bernie Thompson,
Bartlomiej Zolnierkiewicz, Greg Kroah-Hartman,
PrasannaKumar Muralidharan, linux-kernel, linux-metag,
linux-crypto, linux-arm-kernel, linux-samsung-soc, dri-devel,
linux-fbdev
In-Reply-To: <20170411180835.8928-1-krzk@kernel.org>
Few parts of kernel define their own macro for aligning down so provide
a common define for this, with the same usage and assumptions as existing
ALIGN.
Convert also three existing implementations to this one.
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
---
The metag change was not even compiled due to lack of cross compiler.
---
arch/metag/kernel/stacktrace.c | 2 --
drivers/gpu/drm/udl/udl_fb.c | 2 +-
include/linux/kernel.h | 1 +
include/video/udlfb.h | 2 +-
4 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/metag/kernel/stacktrace.c b/arch/metag/kernel/stacktrace.c
index 91ffc4b75c33..09d67b7f51ca 100644
--- a/arch/metag/kernel/stacktrace.c
+++ b/arch/metag/kernel/stacktrace.c
@@ -31,8 +31,6 @@ static void tbi_boing_init(void)
}
#endif
-#define ALIGN_DOWN(addr, size) ((addr)&(~((size)-1)))
-
/*
* Unwind the current stack frame and store the new register values in the
* structure passed as argument. Unwinding is equivalent to a function return,
diff --git a/drivers/gpu/drm/udl/udl_fb.c b/drivers/gpu/drm/udl/udl_fb.c
index d05abc69e305..4a6500362564 100644
--- a/drivers/gpu/drm/udl/udl_fb.c
+++ b/drivers/gpu/drm/udl/udl_fb.c
@@ -37,7 +37,7 @@ struct udl_fbdev {
};
#define DL_ALIGN_UP(x, a) ALIGN(x, a)
-#define DL_ALIGN_DOWN(x, a) ALIGN(x-(a-1), a)
+#define DL_ALIGN_DOWN(x, a) ALIGN_DOWN(x, a)
/** Read the red component (0..255) of a 32 bpp colour. */
#define DLO_RGB_GETRED(col) (uint8_t)((col) & 0xFF)
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 4c26dc3a8295..3d9f8420f973 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -47,6 +47,7 @@
/* @a is a power of 2 value */
#define ALIGN(x, a) __ALIGN_KERNEL((x), (a))
+#define ALIGN_DOWN(x, a) __ALIGN_KERNEL((x) - ((a) - 1), (a))
#define __ALIGN_MASK(x, mask) __ALIGN_KERNEL_MASK((x), (mask))
#define PTR_ALIGN(p, a) ((typeof(p))ALIGN((unsigned long)(p), (a)))
#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0)
diff --git a/include/video/udlfb.h b/include/video/udlfb.h
index f9466fa54ba4..3ea90aea5617 100644
--- a/include/video/udlfb.h
+++ b/include/video/udlfb.h
@@ -92,6 +92,6 @@ struct dlfb_data {
/* remove these once align.h patch is taken into kernel */
#define DL_ALIGN_UP(x, a) ALIGN(x, a)
-#define DL_ALIGN_DOWN(x, a) ALIGN(x-(a-1), a)
+#define DL_ALIGN_DOWN(x, a) ALIGN_DOWN(x, a)
#endif
--
2.9.3
^ permalink raw reply related
* [PATCH v5 0/2] crypto: hw_random - Add new Exynos RNG driver
From: Krzysztof Kozlowski @ 2017-04-11 18:08 UTC (permalink / raw)
To: James Hogan, Matt Mackall, Herbert Xu, Kukjin Kim,
Krzysztof Kozlowski, Javier Martinez Canillas, David S. Miller,
Dave Airlie, David Airlie, Bernie Thompson,
Bartlomiej Zolnierkiewicz, Greg Kroah-Hartman,
PrasannaKumar Muralidharan, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-metag-u79uwXL29TY76Z2rM5mHXA,
linux-crypto-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-samsung-soc-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-fbdev-u79uwXL29TY76Z2rM5mHXA
Hi,
This is a follow up of my questions around exynos-rng [1].
Changes since v4:
=================
1. Patch 2/2: Use "stdrng" name, as suggested by Herbert.
2. Patch 2/2: Add Bartlomiej's reviewed-by.
Changes since v3:
=================
1. New patch: 1/2 for ALIGN_DOWN macro. The change in metag architecture
was not compiled. Please test it.
2. Dropped patches touching ARM defconfig as they are not changing and
they pollute this submission.
3. Utilize all seed provided by kcapi (suggested by Stephan Müller).
4. Drop dev->ctx (suggested by PrasannaKumar Muralidharan).
5. Remove any printks from set_seed callback as this might be a way
for unprivileged user to pollute the log (suggested by Stephan).
6. Minor cleanups: initialize 'read' variable in exynos_rng_reseed()
for readability (it is not strictly required).
7. Add review tags from Stephen and PrasannaKumar.
Changes since v2:
=================
1. Do not re-use random numbers for re-seed (neither for system resume
nor for periodic re-seed). Instead the driver will just generate new
random numbers (suggested by Stephan Müller).
Suspend path tested with suspend-to-freeze, not real suspend. Testing
on Trats2 would be welcomed.
Changes since v1:
=================
1. Re-work the code for seeding after system resume, following suggestions
and review by Stephan Müller.
2. Re-seed itself from time to time (every 100 ms), suggested by Stephan
Müller.
3. Use a define for retries (Bartlomiej Zolnierkiewicz).
4. Add some docs.
Description:
============
The existing exynos-rng has many issues. The most important one is that
it is a pseudo RNG device but uses hw_random interface which does not allow
proper seeding.
The RNG module on Exynos4 requires seeding. On newer SoCs (like Exynos5420)
it can seed itself from a true RNG. Converting the existing driver
to use TRNG would effectively drop support for Exynos4 and break
compatibility with existing users.
Instead I decided to convert it to crypto API. In the future I hope
to add support for seeding from TRNG module.
Tested with app [2].
Patches are independent. I will take the defconfig changes (2/3 and 3/3)
through samsung-soc tree.
Best regards,
Krzysztof
[1] https://www.spinics.net/lists/arm-kernel/msg569641.html
[2] https://www.spinics.net/lists/arm-kernel/msg571184.html
Krzysztof Kozlowski (2):
linux/kernel.h: Add ALIGN_DOWN macro
crypto: hw_random - Add new Exynos RNG driver
MAINTAINERS | 8 +
arch/metag/kernel/stacktrace.c | 2 -
drivers/char/hw_random/Kconfig | 14 --
drivers/char/hw_random/Makefile | 1 -
drivers/char/hw_random/exynos-rng.c | 231 ---------------------
drivers/crypto/Kconfig | 15 ++
drivers/crypto/Makefile | 1 +
drivers/crypto/exynos-rng.c | 389 ++++++++++++++++++++++++++++++++++++
drivers/gpu/drm/udl/udl_fb.c | 2 +-
include/linux/kernel.h | 1 +
include/video/udlfb.h | 2 +-
11 files changed, 416 insertions(+), 250 deletions(-)
delete mode 100644 drivers/char/hw_random/exynos-rng.c
create mode 100644 drivers/crypto/exynos-rng.c
--
2.9.3
^ permalink raw reply
* Re: [PATCH 0/7] arm64: marvell: add cryptographic engine support for 7k/8k
From: Gregory CLEMENT @ 2017-04-11 16:12 UTC (permalink / raw)
To: Herbert Xu
Cc: Antoine Tenart, davem, jason, andrew, sebastian.hesselbarth,
linux-crypto, linux-arm-kernel, thomas.petazzoni, boris.brezillon,
oferh, igall, nadavh
In-Reply-To: <20170410093903.GA8035@gondor.apana.org.au>
Hi Herbert,
On lun., avril 10 2017, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Wed, Mar 29, 2017 at 02:44:25PM +0200, Antoine Tenart wrote:
>> Hi all,
>>
>> This series adds support for the Inside Secure SafeXcel EIP197
>> cryptographic engine which can be found on Marvell Armada 7k and 8k
>> boards. A new cryptographic engine driver is added, as well as the
>> relevant device tree definition for the Armada 7040 DB and 8040 DB
>> boards.
>>
>> This driver needs two firmwares to work correctly. These firmware are
>> usually used for more advanced operations than the ones supported (as of
>> now), but we still need them to pass the data to the internal
>> cryptographic engine.
>>
>> This series was tested in various ways on both the Armada 7040 DB and
>> the Armada 8040 DB: using the crypto framework self tests, using tcrypt
>> and while performing various transfers with iperf on top of IPsec.
>>
>> This series is based on top of v4.11-rc1, and is available on a branch
>> (which also contains the PPv2 support for 7k/8k, to ease the process of
>> testing IPsec): https://github.com/atenart/linux v4.11-rc1/7k8k-crypto
>> I can rebase if needed.
>>
>> Thanks,
>> Antoine
>>
>>
>> Antoine Tenart (7):
>> Documentation/bindings: Document the SafeXel cryptographic engine
>> driver
>> crypto: inside-secure: add SafeXcel EIP197 crypto engine driver
>> MAINTAINERS: add a maintainer for the Inside Secure crypto driver
>> arm64: marvell: dts: add crypto engine description for 7k/8k
>> arm64: marvell: dts: enable the crypto engine on the Armada 7040 DB
>> arm64: marvell: dts: enable the crypto engine on the Armada 8040 DB
>> arm64: defconfig: enable the Safexcel crypto engine as a module
>
> I have no problems with the crypto bits.
Does it means that you agree that I apply the arm-soc related patches
(from 4 to 7) to my mvebu trees?
Thanks,
Gregory
>
> Thanks,
> --
> Email: Herbert Xu <herbert@gondor.apana.org.au>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply
* [PATCH -next] crypto: caam - fix error return code in caam_qi_init()
From: Wei Yongjun @ 2017-04-11 16:04 UTC (permalink / raw)
To: horia.geanta, dan.douglass, herbert, davem; +Cc: Wei Yongjun, linux-crypto
From: Wei Yongjun <weiyongjun1@huawei.com>
Fix to return error code -ENOMEM from the kmem_cache_create() error
handling case instead of 0(err is 0 here), as done elsewhere in this
function.
Fixes: 67c2315def06 ("crypto: caam - add Queue Interface (QI) backend support")
Signed-off-by: Wei Yongjun <weiyongjun1@huawei.com>
---
drivers/crypto/caam/qi.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/crypto/caam/qi.c b/drivers/crypto/caam/qi.c
index 45de8fd..1990ed4 100644
--- a/drivers/crypto/caam/qi.c
+++ b/drivers/crypto/caam/qi.c
@@ -789,7 +789,7 @@ int caam_qi_init(struct platform_device *caam_pdev)
dev_err(qidev, "Can't allocate CAAM cache\n");
free_rsp_fqs();
platform_device_unregister(qi_pdev);
- return err;
+ return -ENOMEM;
}
/* Done with the CGRs; restore the cpus allowed mask */
^ permalink raw reply related
* Re: Fwd: [linux-next:master 7274/8070] undefined reference to `crypto_alloc_shash'
From: Stephan Müller @ 2017-04-11 10:56 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto, David Howells
In-Reply-To: <20170411105000.GA12081@gondor.apana.org.au>
Am Dienstag, 11. April 2017, 12:50:00 CEST schrieb Herbert Xu:
Hi Herbert,
> I see the problem. When you select CRYPTO_HASH you must also
> select CRYPTO.
Thank you very much.
I will prepare a patch.
Ciao
Stephan
^ permalink raw reply
* Re: Fwd: [linux-next:master 7274/8070] undefined reference to `crypto_alloc_shash'
From: Herbert Xu @ 2017-04-11 10:50 UTC (permalink / raw)
To: Stephan Müller; +Cc: linux-crypto, David Howells
In-Reply-To: <1883619.lA7a8XQHvy@tauon.chronox.de>
On Mon, Apr 10, 2017 at 12:46:22PM +0200, Stephan Müller wrote:
> Am Montag, 10. April 2017, 12:22:23 CEST schrieb Herbert Xu:
>
> Hi Herbert,
>
> > Hmm that's weird because CRYPTO_HASH selects CRYPTO_HASH2. Do
> > you have a kconfig file that reproduces this?
>
> Please see attached.
I see the problem. When you select CRYPTO_HASH you must also
select CRYPTO.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [bug report] padata: simplify serialization mechanism
From: Dan Carpenter @ 2017-04-11 9:16 UTC (permalink / raw)
To: steffen.klassert; +Cc: linux-crypto
Hello Steffen Klassert,
The patch 5f1a8c1bc724: "padata: simplify serialization mechanism"
from Jul 7, 2010, leads to the following static checker warning:
kernel/padata.c:243 padata_reorder()
warn: 'padata' is an error pointer or valid
kernel/padata.c
231 if (!spin_trylock_bh(&pd->lock))
232 return;
233
234 while (1) {
235 padata = padata_get_next(pd);
236
237 /*
238 * All reorder queues are empty, or the next object that needs
239 * serialization is parallel processed by another cpu and is
240 * still on it's way to the cpu's reorder queue, nothing to
241 * do for now.
242 */
243 if (!padata || PTR_ERR(padata) == -EINPROGRESS)
^^^^^^
The comments still say that we sometimes return NULL but we removed that
in 2010.
244 break;
245
246 /*
247 * This cpu has to do the parallel processing of the next
248 * object. It's waiting in the cpu's parallelization queue,
249 * so exit immediately.
250 */
251 if (PTR_ERR(padata) == -ENODATA) {
252 del_timer(&pd->timer);
253 spin_unlock_bh(&pd->lock);
254 return;
255 }
regards,
dan carpenter
^ permalink raw reply
* Re: [PATCH v4 2/2] crypto: hw_random - Add new Exynos RNG driver
From: Krzysztof Kozlowski @ 2017-04-11 8:09 UTC (permalink / raw)
To: Herbert Xu
Cc: James Hogan, Matt Mackall, Kukjin Kim, Javier Martinez Canillas,
David S. Miller, Dave Airlie, David Airlie, Bernie Thompson,
Bartlomiej Zolnierkiewicz, Greg Kroah-Hartman,
PrasannaKumar Muralidharan, linux-kernel, linux-metag,
linux-crypto, linux-arm-kernel, linux-samsung-soc, dri-devel,
linux-fbdev, Stephan Müller
In-Reply-To: <20170410105547.GA8615@gondor.apana.org.au>
On Mon, Apr 10, 2017 at 12:55 PM, Herbert Xu
<herbert@gondor.apana.org.au> wrote:
> On Sat, Apr 08, 2017 at 03:32:45PM +0200, Krzysztof Kozlowski wrote:
>>
>> +static struct rng_alg exynos_rng_alg = {
>> + .generate = exynos_rng_generate,
>> + .seed = exynos_rng_seed,
>> + .seedsize = EXYNOS_RNG_SEED_SIZE,
>> + .base = {
>> + .cra_name = "exynos_rng",
>
> Please use stdrng. Or is there a reason why this can't be used
> by the crypto layer itself?
I think there is no reason against, I'll fix it.
Best regards,
Krzysztof
^ permalink raw reply
* Darlehen angebot 3 %
From: Frau SCHMIDT @ 2017-04-10 19:01 UTC (permalink / raw)
Sehr geehrte Damen und Herren,
Haben Sie Interesse über einer finanziellen Darlehen zu 3%?
kontaktieren Sie mich für mehr Details und Bedingungen. ich kann all
jenen helfen, wer ein Darlehen benötigen.
Ich kann Ihnen biete ein darlehen in hohe von 10.000.000 EUR
Meine mail: info@rschmidt.online
Mit freundlichen Grüßen
Frau SCHMIDT
^ permalink raw reply
* Re: [PATCH 2/7] Makefile, x86, LLVM: disable unsupported optimization flags
From: Masahiro Yamada @ 2017-04-10 14:54 UTC (permalink / raw)
To: Michael Davidson
Cc: Matthias Kaehlcke, Michal Marek, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Herbert Xu, David S. Miller, Shaohua Li,
Alexander Potapenko, Dmitry Vyukov, X86 ML,
Linux Kbuild mailing list, Linux Kernel Mailing List,
linux-crypto, linux-raid, Arnd Bergmann
In-Reply-To: <CA+=D-XVbZfgaQ+ZrjctAF5Ek3kaF3XqTLsSu==hK9savDztKsw@mail.gmail.com>
Hi.
2017-04-06 4:11 GMT+09:00 Michael Davidson <md@google.com>:
> It "works" for the cases that I currently care about but I have to say
> that I am uneasy about adding -Werror to the cc-option test in this
> way.
>
> Suppose that one of the *other* flags that is implicitly passed to the
> compiler by cc-option - eg something that was explicitly specified in
> $(KBUILD_CFLAGS) - triggers a warning. In that case all calls to
> cc-option will silently fail because of the -Werror and valid options
> will not be detected correctly.
Theoretically, options explicitly specified in KBUILD_CFLAGS
should be always valid.
Options that may not be supported in some cases
should be wrapped with $(call cc-option ).
> If everyone is OK with that because "it shouldn't normally ever
> happen" then that is fine, but if does result in a subtle change from
> existing behavior (and a trap that I almost immediately fell into
> after applying a similar patch).
There is a rare case where a particular combination fails
(such as the conflict between -pg and -ffunction-sections
as reported in https://patchwork.kernel.org/patch/9624573/).
In a such case, we may end up with swapping the order,
but this should not happen quite often.
> On Wed, Apr 5, 2017 at 12:01 PM, Matthias Kaehlcke <mka@chromium.org> wrote:
>> Hi Masahiro,
>>
>> El Thu, Apr 06, 2017 at 03:08:26AM +0900 Masahiro Yamada ha dit:
>>
>>> 2017-03-17 9:15 GMT+09:00 Michael Davidson <md@google.com>:
>>> > Unfortunately, while clang generates a warning about these flags
>>> > being unsupported it still exits with a status of 0 so we have
>>> > to explicitly disable them instead of just using a cc-option check.
>>> >
>>> > Signed-off-by: Michael Davidson <md@google.com>
>>>
>>>
>>> Instead, does the following work for you?
>>> https://patchwork.kernel.org/patch/9657285/
>>
>> Thanks for the pointer, I was about to give this change (or rather its
>> ancestor) a rework myself :)
>>
>>> You need to use
>>> $(call cc-option, ...)
>>> for -falign-jumps=1 and -falign-loops=1
>>
>> I can confirm that this works.
>>
>> Thanks
>>
>> Matthias
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Best Regards
Masahiro Yamada
^ permalink raw reply
* Re: [RFC PATCH v5] IV Generation algorithms for dm-crypt
From: Milan Broz @ 2017-04-10 14:00 UTC (permalink / raw)
To: Binoy Jayan, Oded, Ofir
Cc: Herbert Xu, David S. Miller, linux-crypto, Mark Brown,
Arnd Bergmann, linux-kernel, Alasdair Kergon, Mike Snitzer,
dm-devel, Shaohua Li, linux-raid, Rajendra, Milan Broz, Gilad
In-Reply-To: <1491562064-23591-1-git-send-email-binoy.jayan@linaro.org>
On 04/07/2017 12:47 PM, Binoy Jayan wrote:
> ===============================================================================
> dm-crypt optimization for larger block sizes
> ===============================================================================
...
> Tests with dd [direct i/o]
>
> Sequential read -0.134 %
> Sequential Write +0.091 %
>
> Tests with fio [Aggregate bandwidth - aggrb]
>
> Random Read +0.358 %
> Random Write +0.010 %
>
> Tests with bonnie++ [768 MiB File, 384 MiB Ram]
> after mounting dm-crypt target as ext4
>
> Sequential o/p [per-char] -2.876 %
> Sequential o/p [per-blk] +0.992 %
> Sequential o/p [re-write] +4.465 %
>
> Sequential i/p [per-char] -0.453 %
> Sequential i/p [per-blk] -0.740 %
>
> Sequential create -0.255 %
> Sequential delete +0.042 %
> Random create -0.007 %
> Random delete +0.454 %
>
> NB: The '+' sign shows improvement and '-' shows degradation.
> The tests were performed with minimal cpu load.
> Tests with higher cpu load to be done
Well, it is good that there is no performance degradation but it
would be nice to have some user of it that proves it is really
working for your hw.
FYI - with patch that increases dmcrypt sector size to 4k
I can see improvement in speed usually in 5-15% with sync AES-NI
(depends on access pattern), with dmcrypt mapped to memory
it is even close to 20% speed up (but such a configuration is
completely artificial).
I wonder why increased dmcrypt sector size does not work for your hw,
it should help as well (and can be combiuned later with this IV approach).
(For native 4k drives this should be used in future anyway...)
Milan
^ permalink raw reply
* [PATCH 3/4] chcr:Fix txq ids.
From: Harsh Jain @ 2017-04-10 12:54 UTC (permalink / raw)
To: herbert, linux-crypto, netdev, atul.gupta, ganeshgr; +Cc: Harsh Jain
In-Reply-To: <cover.1491807106.git.harsh@chelsio.com>
The patch fixes a critical issue to map txqid with flows on the hardware appropriately,
if tx queues created are more than flows configured then txqid shall map within
the range of hardware flows configured. This ensure that un-mapped txqid does not remain un-handled.
The patch also segregated the rxqid and txqid for clarity.
Signed-off-by: Atul Gupta <atul.gupta@chelsio.com>
Reviewed-by: Ganesh Goudar <ganeshgr@chelsio.com>
---
drivers/crypto/chelsio/chcr_algo.c | 47 +++++++++++++---------
drivers/crypto/chelsio/chcr_core.h | 2 +
drivers/crypto/chelsio/chcr_crypto.h | 3 +-
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 9 ++++
drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h | 1 +
drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h | 3 +-
6 files changed, 44 insertions(+), 21 deletions(-)
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 2d61043..5470e4e 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -522,7 +522,7 @@ static inline void create_wreq(struct chcr_context *ctx,
{
struct uld_ctx *u_ctx = ULD_CTX(ctx);
int iv_loc = IV_DSGL;
- int qid = u_ctx->lldi.rxq_ids[ctx->tx_channel_id];
+ int qid = u_ctx->lldi.rxq_ids[ctx->rx_qidx];
unsigned int immdatalen = 0, nr_frags = 0;
if (is_ofld_imm(skb)) {
@@ -543,7 +543,7 @@ static inline void create_wreq(struct chcr_context *ctx,
chcr_req->wreq.cookie = cpu_to_be64((uintptr_t)req);
chcr_req->wreq.rx_chid_to_rx_q_id =
FILL_WR_RX_Q_ID(ctx->dev->rx_channel_id, qid,
- is_iv ? iv_loc : IV_NOP, ctx->tx_channel_id);
+ is_iv ? iv_loc : IV_NOP, ctx->tx_qidx);
chcr_req->ulptx.cmd_dest = FILL_ULPTX_CMD_DEST(ctx->dev->tx_channel_id,
qid);
@@ -721,19 +721,19 @@ static int chcr_aes_encrypt(struct ablkcipher_request *req)
struct sk_buff *skb;
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
- skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[ctx->tx_channel_id],
+ skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx],
CHCR_ENCRYPT_OP);
if (IS_ERR(skb)) {
pr_err("chcr : %s : Failed to form WR. No memory\n", __func__);
return PTR_ERR(skb);
}
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -746,19 +746,19 @@ static int chcr_aes_decrypt(struct ablkcipher_request *req)
struct sk_buff *skb;
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
- skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[0],
+ skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx],
CHCR_DECRYPT_OP);
if (IS_ERR(skb)) {
pr_err("chcr : %s : Failed to form WR. No memory\n", __func__);
return PTR_ERR(skb);
}
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -766,7 +766,9 @@ static int chcr_aes_decrypt(struct ablkcipher_request *req)
static int chcr_device_init(struct chcr_context *ctx)
{
struct uld_ctx *u_ctx;
+ struct adapter *adap;
unsigned int id;
+ int txq_perchan, txq_idx, ntxq;
int err = 0, rxq_perchan, rxq_idx;
id = smp_processor_id();
@@ -777,11 +779,18 @@ static int chcr_device_init(struct chcr_context *ctx)
goto out;
}
u_ctx = ULD_CTX(ctx);
+ adap = padap(ctx->dev);
+ ntxq = min_not_zero((unsigned int)u_ctx->lldi.nrxq,
+ adap->vres.ncrypto_fc);
rxq_perchan = u_ctx->lldi.nrxq / u_ctx->lldi.nchan;
+ txq_perchan = ntxq / u_ctx->lldi.nchan;
rxq_idx = ctx->dev->tx_channel_id * rxq_perchan;
rxq_idx += id % rxq_perchan;
+ txq_idx = ctx->dev->tx_channel_id * txq_perchan;
+ txq_idx += id % txq_perchan;
spin_lock(&ctx->dev->lock_chcr_dev);
- ctx->tx_channel_id = rxq_idx;
+ ctx->rx_qidx = rxq_idx;
+ ctx->tx_qidx = txq_idx;
ctx->dev->tx_channel_id = !ctx->dev->tx_channel_id;
ctx->dev->rx_channel_id = 0;
spin_unlock(&ctx->dev->lock_chcr_dev);
@@ -935,7 +944,7 @@ static int chcr_ahash_update(struct ahash_request *req)
u_ctx = ULD_CTX(ctx);
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
@@ -975,7 +984,7 @@ static int chcr_ahash_update(struct ahash_request *req)
}
req_ctx->reqlen = remainder;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
@@ -1028,7 +1037,7 @@ static int chcr_ahash_final(struct ahash_request *req)
return -ENOMEM;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -1047,7 +1056,7 @@ static int chcr_ahash_finup(struct ahash_request *req)
u_ctx = ULD_CTX(ctx);
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
@@ -1079,7 +1088,7 @@ static int chcr_ahash_finup(struct ahash_request *req)
return -ENOMEM;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
@@ -1100,7 +1109,7 @@ static int chcr_ahash_digest(struct ahash_request *req)
u_ctx = ULD_CTX(ctx);
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
@@ -1130,7 +1139,7 @@ static int chcr_ahash_digest(struct ahash_request *req)
return -ENOMEM;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -2451,13 +2460,13 @@ static int chcr_aead_op(struct aead_request *req,
}
u_ctx = ULD_CTX(ctx);
if (cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id)) {
+ ctx->tx_qidx)) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
/* Form a WR from req */
- skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[ctx->tx_channel_id], size,
+ skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx], size,
op_type);
if (IS_ERR(skb) || skb == NULL) {
@@ -2466,7 +2475,7 @@ static int chcr_aead_op(struct aead_request *req,
}
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
diff --git a/drivers/crypto/chelsio/chcr_core.h b/drivers/crypto/chelsio/chcr_core.h
index 79da22b..cd0c35a 100644
--- a/drivers/crypto/chelsio/chcr_core.h
+++ b/drivers/crypto/chelsio/chcr_core.h
@@ -54,6 +54,8 @@
#define CHK_MAC_ERR_BIT(x) (((x) >> MAC_ERROR_BIT) & 1)
#define MAX_SALT 4
+#define padap(dev) pci_get_drvdata(dev->u_ctx->lldi.pdev)
+
struct uld_ctx;
struct _key_ctx {
diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h
index 4469fea..c5673f0 100644
--- a/drivers/crypto/chelsio/chcr_crypto.h
+++ b/drivers/crypto/chelsio/chcr_crypto.h
@@ -211,7 +211,8 @@ struct __crypto_ctx {
struct chcr_context {
struct chcr_dev *dev;
- unsigned char tx_channel_id;
+ unsigned char tx_qidx;
+ unsigned char rx_qidx;
struct __crypto_ctx crypto_ctx[0];
};
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
index afb0967..6faaca1 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
@@ -3809,6 +3809,15 @@ static int adap_init0(struct adapter *adap)
}
if (caps_cmd.cryptocaps) {
/* Should query params here...TODO */
+ params[0] = FW_PARAM_PFVF(NCRYPTO_LOOKASIDE);
+ ret = t4_query_params(adap, adap->mbox, adap->pf, 0, 2,
+ params, val);
+ if (ret < 0) {
+ if (ret != -EINVAL)
+ goto bye;
+ } else {
+ adap->vres.ncrypto_fc = val[0];
+ }
adap->params.crypto |= ULP_CRYPTO_LOOKASIDE;
adap->num_uld += 1;
}
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
index 4c85660..6e74040 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
@@ -272,6 +272,7 @@ struct cxgb4_virt_res { /* virtualized HW resources */
struct cxgb4_range qp;
struct cxgb4_range cq;
struct cxgb4_range ocq;
+ unsigned int ncrypto_fc;
};
#define OCQ_WIN_OFFSET(pdev, vres) \
diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h b/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
index ccc05f8..8f8c079 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
@@ -1167,7 +1167,8 @@ enum fw_params_param_pfvf {
FW_PARAMS_PARAM_PFVF_ACTIVE_FILTER_START = 0x2D,
FW_PARAMS_PARAM_PFVF_ACTIVE_FILTER_END = 0x2E,
FW_PARAMS_PARAM_PFVF_ETHOFLD_END = 0x30,
- FW_PARAMS_PARAM_PFVF_CPLFW4MSG_ENCAP = 0x31
+ FW_PARAMS_PARAM_PFVF_CPLFW4MSG_ENCAP = 0x31,
+ FW_PARAMS_PARAM_PFVF_NCRYPTO_LOOKASIDE = 0x32
};
/*
--
1.7.1
^ permalink raw reply related
* [PATCH 1/4] chcr: Increase priority of AEAD algos.
From: Harsh Jain @ 2017-04-10 12:53 UTC (permalink / raw)
To: herbert, linux-crypto, netdev, atul.gupta, ganeshgr; +Cc: Harsh Jain
In-Reply-To: <cover.1491807106.git.harsh@chelsio.com>
templates(gcm,ccm etc) inherit priority value of driver to
calculate its priority. In some cases template priority becomes
more than driver priority for same algo.
Without this patch we will not be able to use driver authenc algos. It will
be good if it pushed in stable kernel.
Signed-off-by: Harsh Jain <harsh@chelsio.com>
---
drivers/crypto/chelsio/chcr_algo.c | 12 ++++++++++--
drivers/crypto/chelsio/chcr_crypto.h | 4 ++--
2 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 41bc7f4..7d59591 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -2673,6 +2673,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_name = "gcm(aes)",
.cra_driver_name = "gcm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_gcm_ctx),
@@ -2691,6 +2692,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_name = "rfc4106(gcm(aes))",
.cra_driver_name = "rfc4106-gcm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY + 1,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_gcm_ctx),
@@ -2710,6 +2712,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_name = "ccm(aes)",
.cra_driver_name = "ccm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx),
@@ -2728,6 +2731,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_name = "rfc4309(ccm(aes))",
.cra_driver_name = "rfc4309-ccm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY + 1,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx),
@@ -2747,6 +2751,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_driver_name =
"authenc-hmac-sha1-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2768,6 +2773,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_driver_name =
"authenc-hmac-sha256-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2788,6 +2794,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_driver_name =
"authenc-hmac-sha224-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2807,6 +2814,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_driver_name =
"authenc-hmac-sha384-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2827,6 +2835,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_driver_name =
"authenc-hmac-sha512-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2847,6 +2856,7 @@ static int chcr_aead_op(struct aead_request *req,
.cra_driver_name =
"authenc-digest_null-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2915,8 +2925,6 @@ static int chcr_register_alg(void)
name = driver_algs[i].alg.crypto.cra_driver_name;
break;
case CRYPTO_ALG_TYPE_AEAD:
- driver_algs[i].alg.aead.base.cra_priority =
- CHCR_CRA_PRIORITY;
driver_algs[i].alg.aead.base.cra_flags =
CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC;
driver_algs[i].alg.aead.encrypt = chcr_aead_encrypt;
diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h
index 81cfd0b..4469fea 100644
--- a/drivers/crypto/chelsio/chcr_crypto.h
+++ b/drivers/crypto/chelsio/chcr_crypto.h
@@ -48,8 +48,8 @@
* giving the processed data
*/
-#define CHCR_CRA_PRIORITY 3000
-
+#define CHCR_CRA_PRIORITY 500
+#define CHCR_AEAD_PRIORITY 6000
#define CHCR_AES_MAX_KEY_LEN (2 * (AES_MAX_KEY_SIZE)) /* consider xts */
#define CHCR_MAX_CRYPTO_IV_LEN 16 /* AES IV len */
--
1.7.1
^ permalink raw reply related
* [PATCH 0/4] Bug fixes and fallback for AEAD
From: Harsh Jain @ 2017-04-10 12:53 UTC (permalink / raw)
To: herbert, linux-crypto, netdev, atul.gupta, ganeshgr; +Cc: Harsh Jain
This series based on Herbert cryptodev-2.6.
It includes bug fixes and fallback for AEAD algos.
Harsh Jain (3):
chcr: Increase priority of AEAD algos.
chcr:Set hmac_ctrl bit to use HW register HMAC_CFG[456].
chcr: Add fallback for AEAD algos
Atul Gupta (1):
chcr: Fix txq ids
drivers/crypto/chelsio/chcr_algo.c | 298 ++++++++++++++---------
drivers/crypto/chelsio/chcr_algo.h | 4 +
drivers/crypto/chelsio/chcr_core.h | 2 +
drivers/crypto/chelsio/chcr_crypto.h | 10 +-
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 9 +
drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h | 1 +
drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h | 3 +-
7 files changed, 210 insertions(+), 117 deletions(-)
^ permalink raw reply
* [PATCH 4/4] chcr: Add fallback for AEAD algos
From: Harsh Jain @ 2017-04-10 12:54 UTC (permalink / raw)
To: herbert, linux-crypto, netdev, atul.gupta, ganeshgr; +Cc: Harsh Jain
In-Reply-To: <cover.1491807106.git.harsh@chelsio.com>
Fallback to sw when
I AAD length greater than 511
II Zero length payload
II No of sg entries exceeds Request size.
Signed-off-by: Harsh Jain <harsh@chelsio.com>
---
drivers/crypto/chelsio/chcr_algo.c | 219 ++++++++++++++++++++++------------
drivers/crypto/chelsio/chcr_algo.h | 4 +
drivers/crypto/chelsio/chcr_crypto.h | 3 +-
3 files changed, 151 insertions(+), 75 deletions(-)
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 5470e4e..53d9ce4 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -1343,7 +1343,36 @@ static int chcr_copy_assoc(struct aead_request *req,
return crypto_skcipher_encrypt(skreq);
}
+static int chcr_aead_need_fallback(struct aead_request *req, int src_nent,
+ int aadmax, int wrlen,
+ unsigned short op_type)
+{
+ unsigned int authsize = crypto_aead_authsize(crypto_aead_reqtfm(req));
+
+ if (((req->cryptlen - (op_type ? authsize : 0)) == 0) ||
+ (req->assoclen > aadmax) ||
+ (src_nent > MAX_SKB_FRAGS) ||
+ (wrlen > MAX_WR_SIZE))
+ return 1;
+ return 0;
+}
+static int chcr_aead_fallback(struct aead_request *req, unsigned short op_type)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ struct chcr_context *ctx = crypto_aead_ctx(tfm);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+ struct aead_request *subreq = aead_request_ctx(req);
+
+ aead_request_set_tfm(subreq, aeadctx->sw_cipher);
+ aead_request_set_callback(subreq, req->base.flags,
+ req->base.complete, req->base.data);
+ aead_request_set_crypt(subreq, req->src, req->dst, req->cryptlen,
+ req->iv);
+ aead_request_set_ad(subreq, req->assoclen);
+ return op_type ? crypto_aead_decrypt(subreq) :
+ crypto_aead_encrypt(subreq);
+}
static struct sk_buff *create_authenc_wr(struct aead_request *req,
unsigned short qid,
@@ -1367,7 +1396,7 @@ static int chcr_copy_assoc(struct aead_request *req,
unsigned short stop_offset = 0;
unsigned int assoclen = req->assoclen;
unsigned int authsize = crypto_aead_authsize(tfm);
- int err = 0;
+ int err = -EINVAL, src_nent;
int null = 0;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
@@ -1377,8 +1406,8 @@ static int chcr_copy_assoc(struct aead_request *req,
if (op_type && req->cryptlen < crypto_aead_authsize(tfm))
goto err;
-
- if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
+ src_nent = sg_nents_for_len(req->src, req->assoclen + req->cryptlen);
+ if (src_nent < 0)
goto err;
src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
reqctx->dst = src;
@@ -1396,7 +1425,7 @@ static int chcr_copy_assoc(struct aead_request *req,
}
reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
- if (reqctx->dst_nents <= 0) {
+ if (reqctx->dst_nents < 0) {
pr_err("AUTHENC:Invalid Destination sg entries\n");
goto err;
}
@@ -1404,6 +1433,12 @@ static int chcr_copy_assoc(struct aead_request *req,
kctx_len = (ntohl(KEY_CONTEXT_CTX_LEN_V(aeadctx->key_ctx_hdr)) << 4)
- sizeof(chcr_req->key_ctx);
transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size);
+ if (chcr_aead_need_fallback(req, src_nent + MIN_AUTH_SG,
+ T6_MAX_AAD_SIZE,
+ transhdr_len + (sgl_len(src_nent + MIN_AUTH_SG) * 8),
+ op_type)) {
+ return ERR_PTR(chcr_aead_fallback(req, op_type));
+ }
skb = alloc_skb((transhdr_len + sizeof(struct sge_opaque_hdr)), flags);
if (!skb)
goto err;
@@ -1485,24 +1520,6 @@ static int chcr_copy_assoc(struct aead_request *req,
return ERR_PTR(-EINVAL);
}
-static void aes_gcm_empty_pld_pad(struct scatterlist *sg,
- unsigned short offset)
-{
- struct page *spage;
- unsigned char *addr;
-
- spage = sg_page(sg);
- get_page(spage); /* so that it is not freed by NIC */
-#ifdef KMAP_ATOMIC_ARGS
- addr = kmap_atomic(spage, KM_SOFTIRQ0);
-#else
- addr = kmap_atomic(spage);
-#endif
- memset(addr + sg->offset, 0, offset + 1);
-
- kunmap_atomic(addr);
-}
-
static int set_msg_len(u8 *block, unsigned int msglen, int csize)
{
__be32 data;
@@ -1566,11 +1583,6 @@ static int ccm_format_packet(struct aead_request *req,
struct chcr_aead_reqctx *reqctx = aead_request_ctx(req);
int rc = 0;
- if (req->assoclen > T5_MAX_AAD_SIZE) {
- pr_err("CCM: Unsupported AAD data. It should be < %d\n",
- T5_MAX_AAD_SIZE);
- return -EINVAL;
- }
if (sub_type == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309) {
reqctx->iv[0] = 3;
memcpy(reqctx->iv + 1, &aeadctx->salt[0], 3);
@@ -1715,16 +1727,17 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
unsigned int dst_size = 0, kctx_len;
unsigned int sub_type;
unsigned int authsize = crypto_aead_authsize(tfm);
- int err = 0;
+ int err = -EINVAL, src_nent;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
if (op_type && req->cryptlen < crypto_aead_authsize(tfm))
goto err;
-
- if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
+ src_nent = sg_nents_for_len(req->src, req->assoclen + req->cryptlen);
+ if (src_nent < 0)
goto err;
+
sub_type = get_aead_subtype(tfm);
src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
reqctx->dst = src;
@@ -1740,7 +1753,7 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
}
reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
- if (reqctx->dst_nents <= 0) {
+ if (reqctx->dst_nents < 0) {
pr_err("CCM:Invalid Destination sg entries\n");
goto err;
}
@@ -1752,6 +1765,13 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
dst_size = get_space_for_phys_dsgl(reqctx->dst_nents);
kctx_len = ((DIV_ROUND_UP(aeadctx->enckey_len, 16)) << 4) * 2;
transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size);
+ if (chcr_aead_need_fallback(req, src_nent + MIN_CCM_SG,
+ T6_MAX_AAD_SIZE - 18,
+ transhdr_len + (sgl_len(src_nent + MIN_CCM_SG) * 8),
+ op_type)) {
+ return ERR_PTR(chcr_aead_fallback(req, op_type));
+ }
+
skb = alloc_skb((transhdr_len + sizeof(struct sge_opaque_hdr)), flags);
if (!skb)
@@ -1816,7 +1836,7 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
unsigned char tag_offset = 0;
unsigned int crypt_len = 0;
unsigned int authsize = crypto_aead_authsize(tfm);
- int err = 0;
+ int err = -EINVAL, src_nent;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
@@ -1826,8 +1846,8 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
if (op_type && req->cryptlen < crypto_aead_authsize(tfm))
goto err;
-
- if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
+ src_nent = sg_nents_for_len(req->src, req->assoclen + req->cryptlen);
+ if (src_nent < 0)
goto err;
src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
@@ -1849,7 +1869,7 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
crypt_len = req->cryptlen;
reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
- if (reqctx->dst_nents <= 0) {
+ if (reqctx->dst_nents < 0) {
pr_err("GCM:Invalid Destination sg entries\n");
goto err;
}
@@ -1859,6 +1879,12 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
kctx_len = ((DIV_ROUND_UP(aeadctx->enckey_len, 16)) << 4) +
AEAD_H_SIZE;
transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size);
+ if (chcr_aead_need_fallback(req, src_nent + MIN_GCM_SG,
+ T6_MAX_AAD_SIZE,
+ transhdr_len + (sgl_len(src_nent + MIN_GCM_SG) * 8),
+ op_type)) {
+ return ERR_PTR(chcr_aead_fallback(req, op_type));
+ }
skb = alloc_skb((transhdr_len + sizeof(struct sge_opaque_hdr)), flags);
if (!skb)
goto err;
@@ -1876,11 +1902,11 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
chcr_req->sec_cpl.op_ivinsrtofst = FILL_SEC_CPL_OP_IVINSR(
ctx->dev->rx_channel_id, 2, (ivsize ?
(req->assoclen + 1) : 0));
- chcr_req->sec_cpl.pldlen = htonl(req->assoclen + ivsize + crypt_len);
+ chcr_req->sec_cpl.pldlen =
+ htonl(req->assoclen + ivsize + req->cryptlen);
chcr_req->sec_cpl.aadstart_cipherstop_hi = FILL_SEC_CPL_CIPHERSTOP_HI(
req->assoclen ? 1 : 0, req->assoclen,
req->assoclen + ivsize + 1, 0);
- if (req->cryptlen) {
chcr_req->sec_cpl.cipherstop_lo_authinsert =
FILL_SEC_CPL_AUTHINSERT(0, req->assoclen + ivsize + 1,
tag_offset, tag_offset);
@@ -1890,15 +1916,6 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
CHCR_SCMD_CIPHER_MODE_AES_GCM,
CHCR_SCMD_AUTH_MODE_GHASH,
aeadctx->hmac_ctrl, ivsize >> 1);
- } else {
- chcr_req->sec_cpl.cipherstop_lo_authinsert =
- FILL_SEC_CPL_AUTHINSERT(0, 0, 0, 0);
- chcr_req->sec_cpl.seqno_numivs =
- FILL_SEC_CPL_SCMD0_SEQNO(op_type,
- (op_type == CHCR_ENCRYPT_OP) ?
- 1 : 0, CHCR_SCMD_CIPHER_MODE_AES_CBC,
- 0, 0, ivsize >> 1);
- }
chcr_req->sec_cpl.ivgen_hdrlen = FILL_SEC_CPL_IVGEN_HDRLEN(0, 0, 1,
0, 1, dst_size);
chcr_req->key_ctx.ctx_hdr = aeadctx->key_ctx_hdr;
@@ -1931,15 +1948,7 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
write_sg_to_skb(skb, &frags, req->src, req->assoclen);
write_buffer_to_skb(skb, &frags, reqctx->iv, ivsize);
-
- if (req->cryptlen) {
- write_sg_to_skb(skb, &frags, src, req->cryptlen);
- } else {
- aes_gcm_empty_pld_pad(req->dst, authsize - 1);
- write_sg_to_skb(skb, &frags, reqctx->dst, crypt_len);
-
- }
-
+ write_sg_to_skb(skb, &frags, src, req->cryptlen);
create_wreq(ctx, chcr_req, req, skb, kctx_len, size, 1,
sizeof(struct cpl_rx_phys_dsgl) + dst_size);
reqctx->skb = skb;
@@ -1960,8 +1969,15 @@ static int chcr_aead_cra_init(struct crypto_aead *tfm)
{
struct chcr_context *ctx = crypto_aead_ctx(tfm);
struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
-
- crypto_aead_set_reqsize(tfm, sizeof(struct chcr_aead_reqctx));
+ struct aead_alg *alg = crypto_aead_alg(tfm);
+
+ aeadctx->sw_cipher = crypto_alloc_aead(alg->base.cra_name, 0,
+ CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(aeadctx->sw_cipher))
+ return PTR_ERR(aeadctx->sw_cipher);
+ crypto_aead_set_reqsize(tfm, max(sizeof(struct chcr_aead_reqctx),
+ sizeof(struct aead_request) +
+ crypto_aead_reqsize(aeadctx->sw_cipher)));
aeadctx->null = crypto_get_default_null_skcipher();
if (IS_ERR(aeadctx->null))
return PTR_ERR(aeadctx->null);
@@ -1970,7 +1986,11 @@ static int chcr_aead_cra_init(struct crypto_aead *tfm)
static void chcr_aead_cra_exit(struct crypto_aead *tfm)
{
+ struct chcr_context *ctx = crypto_aead_ctx(tfm);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+
crypto_put_default_null_skcipher();
+ crypto_free_aead(aeadctx->sw_cipher);
}
static int chcr_authenc_null_setauthsize(struct crypto_aead *tfm,
@@ -1980,7 +2000,7 @@ static int chcr_authenc_null_setauthsize(struct crypto_aead *tfm,
aeadctx->hmac_ctrl = CHCR_SCMD_HMAC_CTRL_NOP;
aeadctx->mayverify = VERIFY_HW;
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
static int chcr_authenc_setauthsize(struct crypto_aead *tfm,
unsigned int authsize)
@@ -2017,7 +2037,7 @@ static int chcr_authenc_setauthsize(struct crypto_aead *tfm,
aeadctx->hmac_ctrl = CHCR_SCMD_HMAC_CTRL_NO_TRUNC;
aeadctx->mayverify = VERIFY_SW;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
@@ -2057,7 +2077,7 @@ static int chcr_gcm_setauthsize(struct crypto_aead *tfm, unsigned int authsize)
CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
static int chcr_4106_4309_setauthsize(struct crypto_aead *tfm,
@@ -2083,7 +2103,7 @@ static int chcr_4106_4309_setauthsize(struct crypto_aead *tfm,
CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
static int chcr_ccm_setauthsize(struct crypto_aead *tfm,
@@ -2125,10 +2145,10 @@ static int chcr_ccm_setauthsize(struct crypto_aead *tfm,
CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
-static int chcr_aead_ccm_setkey(struct crypto_aead *aead,
+static int chcr_ccm_common_setkey(struct crypto_aead *aead,
const u8 *key,
unsigned int keylen)
{
@@ -2137,8 +2157,6 @@ static int chcr_aead_ccm_setkey(struct crypto_aead *aead,
unsigned char ck_size, mk_size;
int key_ctx_size = 0;
- memcpy(aeadctx->key, key, keylen);
- aeadctx->enckey_len = keylen;
key_ctx_size = sizeof(struct _key_ctx) +
((DIV_ROUND_UP(keylen, 16)) << 4) * 2;
if (keylen == AES_KEYSIZE_128) {
@@ -2158,9 +2176,32 @@ static int chcr_aead_ccm_setkey(struct crypto_aead *aead,
}
aeadctx->key_ctx_hdr = FILL_KEY_CTX_HDR(ck_size, mk_size, 0, 0,
key_ctx_size >> 4);
+ memcpy(aeadctx->key, key, keylen);
+ aeadctx->enckey_len = keylen;
+
return 0;
}
+static int chcr_aead_ccm_setkey(struct crypto_aead *aead,
+ const u8 *key,
+ unsigned int keylen)
+{
+ struct chcr_context *ctx = crypto_aead_ctx(aead);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+ int error;
+
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(aead) &
+ CRYPTO_TFM_REQ_MASK);
+ error = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(aead, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(aead, crypto_aead_get_flags(aeadctx->sw_cipher) &
+ CRYPTO_TFM_RES_MASK);
+ if (error)
+ return error;
+ return chcr_ccm_common_setkey(aead, key, keylen);
+}
+
static int chcr_aead_rfc4309_setkey(struct crypto_aead *aead, const u8 *key,
unsigned int keylen)
{
@@ -2175,7 +2216,7 @@ static int chcr_aead_rfc4309_setkey(struct crypto_aead *aead, const u8 *key,
}
keylen -= 3;
memcpy(aeadctx->salt, key + keylen, 3);
- return chcr_aead_ccm_setkey(aead, key, keylen);
+ return chcr_ccm_common_setkey(aead, key, keylen);
}
static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key,
@@ -2188,6 +2229,17 @@ static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key,
unsigned int ck_size;
int ret = 0, key_ctx_size = 0;
+ aeadctx->enckey_len = 0;
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(aead)
+ & CRYPTO_TFM_REQ_MASK);
+ ret = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(aead, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(aead, crypto_aead_get_flags(aeadctx->sw_cipher) &
+ CRYPTO_TFM_RES_MASK);
+ if (ret)
+ goto out;
+
if (get_aead_subtype(aead) == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4106 &&
keylen > 3) {
keylen -= 4; /* nonce/salt is present in the last 4 bytes */
@@ -2202,8 +2254,7 @@ static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key,
} else {
crypto_tfm_set_flags((struct crypto_tfm *)aead,
CRYPTO_TFM_RES_BAD_KEY_LEN);
- aeadctx->enckey_len = 0;
- pr_err("GCM: Invalid key length %d", keylen);
+ pr_err("GCM: Invalid key length %d\n", keylen);
ret = -EINVAL;
goto out;
}
@@ -2259,6 +2310,16 @@ static int chcr_authenc_setkey(struct crypto_aead *authenc, const u8 *key,
int align;
u8 *o_ptr = NULL;
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(authenc)
+ & CRYPTO_TFM_REQ_MASK);
+ err = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(authenc, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(authenc, crypto_aead_get_flags(aeadctx->sw_cipher)
+ & CRYPTO_TFM_RES_MASK);
+ if (err)
+ goto out;
+
if (crypto_authenc_extractkeys(&keys, key, keylen) != 0) {
crypto_aead_set_flags(authenc, CRYPTO_TFM_RES_BAD_KEY_LEN);
goto out;
@@ -2291,7 +2352,8 @@ static int chcr_authenc_setkey(struct crypto_aead *authenc, const u8 *key,
base_hash = chcr_alloc_shash(max_authsize);
if (IS_ERR(base_hash)) {
pr_err("chcr : Base driver cannot be loaded\n");
- goto out;
+ aeadctx->enckey_len = 0;
+ return -EINVAL;
}
{
SHASH_DESC_ON_STACK(shash, base_hash);
@@ -2358,11 +2420,21 @@ static int chcr_aead_digest_null_setkey(struct crypto_aead *authenc,
struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
struct chcr_authenc_ctx *actx = AUTHENC_CTX(aeadctx);
struct crypto_authenc_keys keys;
-
+ int err;
/* it contains auth and cipher key both*/
int key_ctx_len = 0;
unsigned char ck_size = 0;
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(authenc)
+ & CRYPTO_TFM_REQ_MASK);
+ err = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(authenc, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(authenc, crypto_aead_get_flags(aeadctx->sw_cipher)
+ & CRYPTO_TFM_RES_MASK);
+ if (err)
+ goto out;
+
if (crypto_authenc_extractkeys(&keys, key, keylen) != 0) {
crypto_aead_set_flags(authenc, CRYPTO_TFM_RES_BAD_KEY_LEN);
goto out;
@@ -2469,10 +2541,8 @@ static int chcr_aead_op(struct aead_request *req,
skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx], size,
op_type);
- if (IS_ERR(skb) || skb == NULL) {
- pr_err("chcr : %s : failed to form WR. No memory\n", __func__);
+ if (IS_ERR(skb) || !skb)
return PTR_ERR(skb);
- }
skb->dev = u_ctx->lldi.ports[0];
set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
@@ -2921,7 +2991,8 @@ static int chcr_register_alg(void)
break;
case CRYPTO_ALG_TYPE_AEAD:
driver_algs[i].alg.aead.base.cra_flags =
- CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC;
+ CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK;
driver_algs[i].alg.aead.encrypt = chcr_aead_encrypt;
driver_algs[i].alg.aead.decrypt = chcr_aead_decrypt;
driver_algs[i].alg.aead.init = chcr_aead_cra_init;
diff --git a/drivers/crypto/chelsio/chcr_algo.h b/drivers/crypto/chelsio/chcr_algo.h
index ba38bae..751d06a 100644
--- a/drivers/crypto/chelsio/chcr_algo.h
+++ b/drivers/crypto/chelsio/chcr_algo.h
@@ -218,6 +218,10 @@
#define MAX_NK 8
#define CRYPTO_MAX_IMM_TX_PKT_LEN 256
+#define MAX_WR_SIZE 512
+#define MIN_AUTH_SG 2 /*IV + AAD*/
+#define MIN_GCM_SG 2 /* IV + AAD*/
+#define MIN_CCM_SG 3 /*IV+AAD+B0*/
struct algo_param {
unsigned int auth_mode;
diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h
index c5673f0..5b2fabb 100644
--- a/drivers/crypto/chelsio/chcr_crypto.h
+++ b/drivers/crypto/chelsio/chcr_crypto.h
@@ -41,7 +41,7 @@
#define CCM_B0_SIZE 16
#define CCM_AAD_FIELD_SIZE 2
-#define T5_MAX_AAD_SIZE 512
+#define T6_MAX_AAD_SIZE 511
/* Define following if h/w is not dropping the AAD and IV data before
@@ -188,6 +188,7 @@ struct chcr_aead_ctx {
__be32 key_ctx_hdr;
unsigned int enckey_len;
struct crypto_skcipher *null;
+ struct crypto_aead *sw_cipher;
u8 salt[MAX_SALT];
u8 key[CHCR_AES_MAX_KEY_LEN];
u16 hmac_ctrl;
--
1.7.1
^ permalink raw reply related
* [PATCH 2/4] chcr:Set hmac_ctrl bit to use HW register HMAC_CFG[456]
From: Harsh Jain @ 2017-04-10 12:53 UTC (permalink / raw)
To: herbert, linux-crypto, netdev, atul.gupta, ganeshgr; +Cc: Harsh Jain
In-Reply-To: <cover.1491807106.git.harsh@chelsio.com>
Use hmac_ctrl bit value saved in setauthsize callback.
Signed-off-by: Harsh Jain <harsh@chelsio.com>
---
drivers/crypto/chelsio/chcr_algo.c | 24 +++++-------------------
1 files changed, 5 insertions(+), 19 deletions(-)
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 7d59591..2d61043 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -1335,19 +1335,6 @@ static int chcr_copy_assoc(struct aead_request *req,
return crypto_skcipher_encrypt(skreq);
}
-static unsigned char get_hmac(unsigned int authsize)
-{
- switch (authsize) {
- case ICV_8:
- return CHCR_SCMD_HMAC_CTRL_PL1;
- case ICV_10:
- return CHCR_SCMD_HMAC_CTRL_TRUNC_RFC4366;
- case ICV_12:
- return CHCR_SCMD_HMAC_CTRL_IPSEC_96BIT;
- }
- return CHCR_SCMD_HMAC_CTRL_NO_TRUNC;
-}
-
static struct sk_buff *create_authenc_wr(struct aead_request *req,
unsigned short qid,
@@ -1600,13 +1587,13 @@ static void fill_sec_cpl_for_aead(struct cpl_tx_sec_pdu *sec_cpl,
struct chcr_context *chcrctx)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(crypto_aead_ctx(tfm));
unsigned int ivsize = AES_BLOCK_SIZE;
unsigned int cipher_mode = CHCR_SCMD_CIPHER_MODE_AES_CCM;
unsigned int mac_mode = CHCR_SCMD_AUTH_MODE_CBCMAC;
unsigned int c_id = chcrctx->dev->rx_channel_id;
unsigned int ccm_xtra;
unsigned char tag_offset = 0, auth_offset = 0;
- unsigned char hmac_ctrl = get_hmac(crypto_aead_authsize(tfm));
unsigned int assoclen;
if (get_aead_subtype(tfm) == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309)
@@ -1642,8 +1629,8 @@ static void fill_sec_cpl_for_aead(struct cpl_tx_sec_pdu *sec_cpl,
crypto_aead_authsize(tfm));
sec_cpl->seqno_numivs = FILL_SEC_CPL_SCMD0_SEQNO(op_type,
(op_type == CHCR_ENCRYPT_OP) ? 0 : 1,
- cipher_mode, mac_mode, hmac_ctrl,
- ivsize >> 1);
+ cipher_mode, mac_mode,
+ aeadctx->hmac_ctrl, ivsize >> 1);
sec_cpl->ivgen_hdrlen = FILL_SEC_CPL_IVGEN_HDRLEN(0, 0, 1, 0,
1, dst_size);
@@ -1820,7 +1807,6 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
unsigned char tag_offset = 0;
unsigned int crypt_len = 0;
unsigned int authsize = crypto_aead_authsize(tfm);
- unsigned char hmac_ctrl = get_hmac(authsize);
int err = 0;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
@@ -1893,8 +1879,8 @@ unsigned int fill_aead_req_fields(struct sk_buff *skb,
FILL_SEC_CPL_SCMD0_SEQNO(op_type, (op_type ==
CHCR_ENCRYPT_OP) ? 1 : 0,
CHCR_SCMD_CIPHER_MODE_AES_GCM,
- CHCR_SCMD_AUTH_MODE_GHASH, hmac_ctrl,
- ivsize >> 1);
+ CHCR_SCMD_AUTH_MODE_GHASH,
+ aeadctx->hmac_ctrl, ivsize >> 1);
} else {
chcr_req->sec_cpl.cipherstop_lo_authinsert =
FILL_SEC_CPL_AUTHINSERT(0, 0, 0, 0);
--
1.7.1
^ permalink raw reply related
* [PATCH v7 1/2] crypto: skcipher AF_ALG - overhaul memory management
From: Stephan Müller @ 2017-04-10 12:11 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
In-Reply-To: <5852545.bu9l2NtIoo@positron.chronox.de>
The updated memory management is described in the top part of the code.
As one benefit of the changed memory management, the AIO and synchronous
operation is now implemented in one common function. The AF_ALG
operation uses the async kernel crypto API interface for each cipher
operation. Thus, the only difference between the AIO and sync operation
types visible from user space is:
1. the callback function to be invoked when the asynchronous operation
is completed
2. whether to wait for the completion of the kernel crypto API operation
or not
In addition, the code structure is adjusted to match the structure of
algif_aead for easier code assessment.
The user space interface changed slightly as follows: the old AIO
operation returned zero upon success and < 0 in case of an error to user
space. As all other AF_ALG interfaces (including the sync skcipher
interface) returned the number of processed bytes upon success and < 0
in case of an error, the new skcipher interface (regardless of AIO or
sync) returns the number of processed bytes in case of success.
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_skcipher.c | 579 +++++++++++++++++++++++++-----------------------
1 file changed, 296 insertions(+), 283 deletions(-)
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index 43839b0..5107016 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -10,6 +10,21 @@
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*
+ * The following concept of the memory management is used:
+ *
+ * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
+ * filled by user space with the data submitted via sendpage/sendmsg. Filling
+ * up the TX SGL does not cause a crypto operation -- the data will only be
+ * tracked by the kernel. Upon receipt of one recvmsg call, the caller must
+ * provide a buffer which is tracked with the RX SGL.
+ *
+ * During the processing of the recvmsg operation, the cipher request is
+ * allocated and prepared. As part of the recvmsg operation, the processed
+ * TX buffers are extracted from the TX SGL into a separate SGL.
+ *
+ * After the completion of the crypto operation, the RX SGL and the cipher
+ * request is released. The extracted TX SGL parts are released together with
+ * the RX SGL release.
*/
#include <crypto/scatterwalk.h>
@@ -24,109 +39,97 @@
#include <linux/net.h>
#include <net/sock.h>
-struct skcipher_sg_list {
+struct skcipher_tsgl {
struct list_head list;
-
int cur;
-
struct scatterlist sg[0];
};
+struct skcipher_rsgl {
+ struct af_alg_sgl sgl;
+ struct list_head list;
+ size_t sg_num_bytes;
+};
+
+struct skcipher_async_req {
+ struct kiocb *iocb;
+ struct sock *sk;
+
+ struct skcipher_rsgl first_sgl;
+ struct list_head rsgl_list;
+
+ struct scatterlist *tsgl;
+ unsigned int tsgl_entries;
+
+ unsigned int areqlen;
+ struct skcipher_request req;
+};
+
struct skcipher_tfm {
struct crypto_skcipher *skcipher;
bool has_key;
};
struct skcipher_ctx {
- struct list_head tsgl;
- struct af_alg_sgl rsgl;
+ struct list_head tsgl_list;
void *iv;
struct af_alg_completion completion;
- atomic_t inflight;
+ unsigned int inflight;
size_t used;
+ size_t rcvused;
- unsigned int len;
bool more;
bool merge;
bool enc;
- struct skcipher_request req;
-};
-
-struct skcipher_async_rsgl {
- struct af_alg_sgl sgl;
- struct list_head list;
+ unsigned int len;
};
-struct skcipher_async_req {
- struct kiocb *iocb;
- struct skcipher_async_rsgl first_sgl;
- struct list_head list;
- struct scatterlist *tsg;
- atomic_t *inflight;
- struct skcipher_request req;
-};
+static DECLARE_WAIT_QUEUE_HEAD(skcipher_aio_finish_wait);
-#define MAX_SGL_ENTS ((4096 - sizeof(struct skcipher_sg_list)) / \
+#define MAX_SGL_ENTS ((4096 - sizeof(struct skcipher_tsgl)) / \
sizeof(struct scatterlist) - 1)
-static void skcipher_free_async_sgls(struct skcipher_async_req *sreq)
+static inline int skcipher_sndbuf(struct sock *sk)
{
- struct skcipher_async_rsgl *rsgl, *tmp;
- struct scatterlist *sgl;
- struct scatterlist *sg;
- int i, n;
-
- list_for_each_entry_safe(rsgl, tmp, &sreq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &sreq->first_sgl)
- kfree(rsgl);
- }
- sgl = sreq->tsg;
- n = sg_nents(sgl);
- for_each_sg(sgl, sg, n, i)
- put_page(sg_page(sg));
+ struct alg_sock *ask = alg_sk(sk);
+ struct skcipher_ctx *ctx = ask->private;
- kfree(sreq->tsg);
+ return max_t(int, max_t(int, sk->sk_sndbuf & PAGE_MASK, PAGE_SIZE) -
+ ctx->used, 0);
}
-static void skcipher_async_cb(struct crypto_async_request *req, int err)
+static inline bool skcipher_writable(struct sock *sk)
{
- struct skcipher_async_req *sreq = req->data;
- struct kiocb *iocb = sreq->iocb;
-
- atomic_dec(sreq->inflight);
- skcipher_free_async_sgls(sreq);
- kzfree(sreq);
- iocb->ki_complete(iocb, err, err);
+ return PAGE_SIZE <= skcipher_sndbuf(sk);
}
-static inline int skcipher_sndbuf(struct sock *sk)
+static inline int skcipher_rcvbuf(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- return max_t(int, max_t(int, sk->sk_sndbuf & PAGE_MASK, PAGE_SIZE) -
- ctx->used, 0);
+ return max_t(int, max_t(int, sk->sk_rcvbuf & PAGE_MASK, PAGE_SIZE) -
+ ctx->rcvused, 0);
}
-static inline bool skcipher_writable(struct sock *sk)
+static inline bool skcipher_readable(struct sock *sk)
{
- return PAGE_SIZE <= skcipher_sndbuf(sk);
+ return PAGE_SIZE <= skcipher_rcvbuf(sk);
}
-static int skcipher_alloc_sgl(struct sock *sk)
+static int skcipher_alloc_tsgl(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
struct scatterlist *sg = NULL;
- sgl = list_entry(ctx->tsgl.prev, struct skcipher_sg_list, list);
- if (!list_empty(&ctx->tsgl))
+ sgl = list_entry(ctx->tsgl_list.prev, struct skcipher_tsgl, list);
+ if (!list_empty(&ctx->tsgl_list))
sg = sgl->sg;
if (!sg || sgl->cur >= MAX_SGL_ENTS) {
@@ -142,31 +145,66 @@ static int skcipher_alloc_sgl(struct sock *sk)
if (sg)
sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg);
- list_add_tail(&sgl->list, &ctx->tsgl);
+ list_add_tail(&sgl->list, &ctx->tsgl_list);
}
return 0;
}
-static void skcipher_pull_sgl(struct sock *sk, size_t used, int put)
+static unsigned int skcipher_count_tsgl(struct sock *sk, size_t bytes)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl, *tmp;
+ unsigned int i;
+ unsigned int sgl_count = 0;
+
+ if (!bytes)
+ return 0;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ struct scatterlist *sg = sgl->sg;
+
+ for (i = 0; i < sgl->cur; i++) {
+ sgl_count++;
+ if (sg[i].length >= bytes)
+ return sgl_count;
+
+ bytes -= sg[i].length;
+ }
+ }
+
+ return sgl_count;
+}
+
+static void skcipher_pull_tsgl(struct sock *sk, size_t used,
+ struct scatterlist *dst)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct skcipher_ctx *ctx = ask->private;
+ struct skcipher_tsgl *sgl;
struct scatterlist *sg;
- int i;
+ unsigned int i;
- while (!list_empty(&ctx->tsgl)) {
- sgl = list_first_entry(&ctx->tsgl, struct skcipher_sg_list,
+ while (!list_empty(&ctx->tsgl_list)) {
+ sgl = list_first_entry(&ctx->tsgl_list, struct skcipher_tsgl,
list);
sg = sgl->sg;
for (i = 0; i < sgl->cur; i++) {
size_t plen = min_t(size_t, used, sg[i].length);
+ struct page *page = sg_page(sg + i);
- if (!sg_page(sg + i))
+ if (!page)
continue;
+ /*
+ * Assumption: caller created skcipher_count_tsgl(len)
+ * SG entries in dst.
+ */
+ if (dst)
+ sg_set_page(dst + i, page, plen, sg[i].offset);
+
sg[i].length -= plen;
sg[i].offset += plen;
@@ -175,27 +213,48 @@ static void skcipher_pull_sgl(struct sock *sk, size_t used, int put)
if (sg[i].length)
return;
- if (put)
- put_page(sg_page(sg + i));
+
+ if (!dst)
+ put_page(page);
sg_assign_page(sg + i, NULL);
}
list_del(&sgl->list);
- sock_kfree_s(sk, sgl,
- sizeof(*sgl) + sizeof(sgl->sg[0]) *
- (MAX_SGL_ENTS + 1));
+ sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) *
+ (MAX_SGL_ENTS + 1));
}
if (!ctx->used)
ctx->merge = 0;
}
-static void skcipher_free_sgl(struct sock *sk)
+static void skcipher_free_areq_sgls(struct skcipher_async_req *areq)
{
+ struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
+ struct skcipher_rsgl *rsgl, *tmp;
+ struct scatterlist *tsgl;
+ struct scatterlist *sg;
+ unsigned int i;
+
+ list_for_each_entry_safe(rsgl, tmp, &areq->rsgl_list, list) {
+ ctx->rcvused -= rsgl->sg_num_bytes;
+ af_alg_free_sg(&rsgl->sgl);
+ list_del(&rsgl->list);
+ if (rsgl != &areq->first_sgl)
+ sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ }
+
+ tsgl = areq->tsgl;
+ for_each_sg(tsgl, sg, areq->tsgl_entries, i) {
+ if (!sg_page(sg))
+ continue;
+ put_page(sg_page(sg));
+ }
- skcipher_pull_sgl(sk, ctx->used, 1);
+ if (areq->tsgl && areq->tsgl_entries)
+ sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl));
}
static int skcipher_wait_for_wmem(struct sock *sk, unsigned flags)
@@ -302,7 +361,7 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
struct skcipher_tfm *skc = pask->private;
struct crypto_skcipher *tfm = skc->skcipher;
unsigned ivsize = crypto_skcipher_ivsize(tfm);
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
@@ -349,8 +408,8 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
size_t plen;
if (ctx->merge) {
- sgl = list_entry(ctx->tsgl.prev,
- struct skcipher_sg_list, list);
+ sgl = list_entry(ctx->tsgl_list.prev,
+ struct skcipher_tsgl, list);
sg = sgl->sg + sgl->cur - 1;
len = min_t(unsigned long, len,
PAGE_SIZE - sg->offset - sg->length);
@@ -379,11 +438,12 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
len = min_t(unsigned long, len, skcipher_sndbuf(sk));
- err = skcipher_alloc_sgl(sk);
+ err = skcipher_alloc_tsgl(sk);
if (err)
goto unlock;
- sgl = list_entry(ctx->tsgl.prev, struct skcipher_sg_list, list);
+ sgl = list_entry(ctx->tsgl_list.prev, struct skcipher_tsgl,
+ list);
sg = sgl->sg;
if (sgl->cur)
sg_unmark_end(sg + sgl->cur - 1);
@@ -435,7 +495,7 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct skcipher_sg_list *sgl;
+ struct skcipher_tsgl *sgl;
int err = -EINVAL;
if (flags & MSG_SENDPAGE_NOTLAST)
@@ -454,12 +514,12 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
goto unlock;
}
- err = skcipher_alloc_sgl(sk);
+ err = skcipher_alloc_tsgl(sk);
if (err)
goto unlock;
ctx->merge = 0;
- sgl = list_entry(ctx->tsgl.prev, struct skcipher_sg_list, list);
+ sgl = list_entry(ctx->tsgl_list.prev, struct skcipher_tsgl, list);
if (sgl->cur)
sg_unmark_end(sgl->sg + sgl->cur - 1);
@@ -480,25 +540,36 @@ static ssize_t skcipher_sendpage(struct socket *sock, struct page *page,
return err ?: size;
}
-static int skcipher_all_sg_nents(struct skcipher_ctx *ctx)
+static void skcipher_async_cb(struct crypto_async_request *req, int err)
{
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- int nents = 0;
+ struct skcipher_async_req *areq = req->data;
+ struct sock *sk = areq->sk;
+ struct alg_sock *ask = alg_sk(sk);
+ struct skcipher_ctx *ctx = ask->private;
+ struct kiocb *iocb = areq->iocb;
+ unsigned int resultlen;
- list_for_each_entry(sgl, &ctx->tsgl, list) {
- sg = sgl->sg;
+ lock_sock(sk);
- while (!sg->length)
- sg++;
+ BUG_ON(!ctx->inflight);
- nents += sg_nents(sg);
- }
- return nents;
+ /* Buffer size written by crypto operation. */
+ resultlen = areq->req.cryptlen;
+
+ skcipher_free_areq_sgls(areq);
+ sock_kfree_s(sk, areq, areq->areqlen);
+ __sock_put(sk);
+ ctx->inflight--;
+
+ iocb->ki_complete(iocb, err ? err : resultlen, 0);
+
+ release_sock(sk);
+
+ wake_up_interruptible(&skcipher_aio_finish_wait);
}
-static int skcipher_recvmsg_async(struct socket *sock, struct msghdr *msg,
- int flags)
+static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
@@ -507,215 +578,168 @@ static int skcipher_recvmsg_async(struct socket *sock, struct msghdr *msg,
struct skcipher_ctx *ctx = ask->private;
struct skcipher_tfm *skc = pask->private;
struct crypto_skcipher *tfm = skc->skcipher;
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- struct skcipher_async_req *sreq;
- struct skcipher_request *req;
- struct skcipher_async_rsgl *last_rsgl = NULL;
- unsigned int txbufs = 0, len = 0, tx_nents;
- unsigned int reqsize = crypto_skcipher_reqsize(tfm);
- unsigned int ivsize = crypto_skcipher_ivsize(tfm);
- int err = -ENOMEM;
- bool mark = false;
- char *iv;
-
- sreq = kzalloc(sizeof(*sreq) + reqsize + ivsize, GFP_KERNEL);
- if (unlikely(!sreq))
- goto out;
-
- req = &sreq->req;
- iv = (char *)(req + 1) + reqsize;
- sreq->iocb = msg->msg_iocb;
- INIT_LIST_HEAD(&sreq->list);
- sreq->inflight = &ctx->inflight;
-
- lock_sock(sk);
- tx_nents = skcipher_all_sg_nents(ctx);
- sreq->tsg = kcalloc(tx_nents, sizeof(*sg), GFP_KERNEL);
- if (unlikely(!sreq->tsg))
- goto unlock;
- sg_init_table(sreq->tsg, tx_nents);
- memcpy(iv, ctx->iv, ivsize);
- skcipher_request_set_tfm(req, tfm);
- skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP,
- skcipher_async_cb, sreq);
+ unsigned int bs = crypto_skcipher_blocksize(tfm);
+ unsigned int areqlen = sizeof(struct skcipher_async_req) +
+ crypto_skcipher_reqsize(tfm);
+ struct skcipher_async_req *areq;
+ struct skcipher_rsgl *last_rsgl = NULL;
+ int err = 0;
+ size_t len = 0;
- while (iov_iter_count(&msg->msg_iter)) {
- struct skcipher_async_rsgl *rsgl;
- int used;
+ /* Allocate cipher request for current operation. */
+ areq = sock_kmalloc(sk, areqlen, GFP_KERNEL);
+ if (unlikely(!areq))
+ return -ENOMEM;
+ areq->areqlen = areqlen;
+ areq->sk = sk;
+ INIT_LIST_HEAD(&areq->rsgl_list);
+ areq->tsgl = NULL;
+ areq->tsgl_entries = 0;
+
+ /* convert iovecs of output buffers into RX SGL */
+ while (len < ctx->used && msg_data_left(msg)) {
+ struct skcipher_rsgl *rsgl;
+ size_t seglen;
+
+ /* limit the amount of readable buffers */
+ if (!skcipher_readable(sk))
+ break;
if (!ctx->used) {
err = skcipher_wait_for_data(sk, flags);
if (err)
goto free;
}
- sgl = list_first_entry(&ctx->tsgl,
- struct skcipher_sg_list, list);
- sg = sgl->sg;
- while (!sg->length)
- sg++;
-
- used = min_t(unsigned long, ctx->used,
- iov_iter_count(&msg->msg_iter));
- used = min_t(unsigned long, used, sg->length);
-
- if (txbufs == tx_nents) {
- struct scatterlist *tmp;
- int x;
- /* Ran out of tx slots in async request
- * need to expand */
- tmp = kcalloc(tx_nents * 2, sizeof(*tmp),
- GFP_KERNEL);
- if (!tmp) {
- err = -ENOMEM;
- goto free;
- }
+ seglen = min_t(size_t, ctx->used, msg_data_left(msg));
- sg_init_table(tmp, tx_nents * 2);
- for (x = 0; x < tx_nents; x++)
- sg_set_page(&tmp[x], sg_page(&sreq->tsg[x]),
- sreq->tsg[x].length,
- sreq->tsg[x].offset);
- kfree(sreq->tsg);
- sreq->tsg = tmp;
- tx_nents *= 2;
- mark = true;
- }
- /* Need to take over the tx sgl from ctx
- * to the asynch req - these sgls will be freed later */
- sg_set_page(sreq->tsg + txbufs++, sg_page(sg), sg->length,
- sg->offset);
-
- if (list_empty(&sreq->list)) {
- rsgl = &sreq->first_sgl;
- list_add_tail(&rsgl->list, &sreq->list);
+ if (list_empty(&areq->rsgl_list)) {
+ rsgl = &areq->first_sgl;
} else {
- rsgl = kmalloc(sizeof(*rsgl), GFP_KERNEL);
+ rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
if (!rsgl) {
err = -ENOMEM;
goto free;
}
- list_add_tail(&rsgl->list, &sreq->list);
}
- used = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, used);
- err = used;
- if (used < 0)
+ rsgl->sgl.npages = 0;
+ list_add_tail(&rsgl->list, &areq->rsgl_list);
+
+ /* make one iovec available as scatterlist */
+ err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
+ if (err < 0)
goto free;
+
+ /* chain the new scatterlist with previous one */
if (last_rsgl)
af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
last_rsgl = rsgl;
- len += used;
- skcipher_pull_sgl(sk, used, 0);
- iov_iter_advance(&msg->msg_iter, used);
+ len += err;
+ ctx->rcvused =+ err;
+ rsgl->sg_num_bytes = err;
+ iov_iter_advance(&msg->msg_iter, err);
}
- if (mark)
- sg_mark_end(sreq->tsg + txbufs - 1);
+ /* Process only as much RX buffers for which we have TX data */
+ if (len > ctx->used)
+ len = ctx->used;
+
+ /*
+ * If more buffers are to be expected to be processed, process only
+ * full block size buffers.
+ */
+ if (ctx->more || len < ctx->used)
+ len -= len % bs;
+
+ /*
+ * Create a per request TX SGL for this request which tracks the
+ * SG entries from the global TX SGL.
+ */
+ areq->tsgl_entries = skcipher_count_tsgl(sk, len);
+ if (!areq->tsgl_entries)
+ areq->tsgl_entries = 1;
+ areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) * areq->tsgl_entries,
+ GFP_KERNEL);
+ if (!areq->tsgl) {
+ err = -ENOMEM;
+ goto free;
+ }
+ sg_init_table(areq->tsgl, areq->tsgl_entries);
+ skcipher_pull_tsgl(sk, len, areq->tsgl);
+
+ /* Initialize the crypto operation */
+ skcipher_request_set_tfm(&areq->req, tfm);
+ skcipher_request_set_crypt(&areq->req, areq->tsgl,
+ areq->first_sgl.sgl.sg, len, ctx->iv);
+
+ if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
+ /* AIO operation */
+ areq->iocb = msg->msg_iocb;
+ skcipher_request_set_callback(&areq->req,
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ skcipher_async_cb, areq);
+ err = ctx->enc ? crypto_skcipher_encrypt(&areq->req) :
+ crypto_skcipher_decrypt(&areq->req);
+ } else {
+ /* Synchronous operation */
+ skcipher_request_set_callback(&areq->req,
+ CRYPTO_TFM_REQ_MAY_SLEEP |
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ af_alg_complete,
+ &ctx->completion);
+ err = af_alg_wait_for_completion(ctx->enc ?
+ crypto_skcipher_encrypt(&areq->req) :
+ crypto_skcipher_decrypt(&areq->req),
+ &ctx->completion);
+ }
- skcipher_request_set_crypt(req, sreq->tsg, sreq->first_sgl.sgl.sg,
- len, iv);
- err = ctx->enc ? crypto_skcipher_encrypt(req) :
- crypto_skcipher_decrypt(req);
+ /* AIO operation in progress */
if (err == -EINPROGRESS) {
- atomic_inc(&ctx->inflight);
- err = -EIOCBQUEUED;
- sreq = NULL;
- goto unlock;
+ sock_hold(sk);
+ ctx->inflight++;
+ return -EIOCBQUEUED;
}
+
free:
- skcipher_free_async_sgls(sreq);
-unlock:
- skcipher_wmem_wakeup(sk);
- release_sock(sk);
- kzfree(sreq);
-out:
- return err;
+ skcipher_free_areq_sgls(areq);
+ if (areq)
+ sock_kfree_s(sk, areq, areqlen);
+
+ return err ? err : len;
}
-static int skcipher_recvmsg_sync(struct socket *sock, struct msghdr *msg,
- int flags)
+static int skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
{
struct sock *sk = sock->sk;
- struct alg_sock *ask = alg_sk(sk);
- struct sock *psk = ask->parent;
- struct alg_sock *pask = alg_sk(psk);
- struct skcipher_ctx *ctx = ask->private;
- struct skcipher_tfm *skc = pask->private;
- struct crypto_skcipher *tfm = skc->skcipher;
- unsigned bs = crypto_skcipher_blocksize(tfm);
- struct skcipher_sg_list *sgl;
- struct scatterlist *sg;
- int err = -EAGAIN;
- int used;
- long copied = 0;
+ int ret = 0;
lock_sock(sk);
while (msg_data_left(msg)) {
- if (!ctx->used) {
- err = skcipher_wait_for_data(sk, flags);
- if (err)
- goto unlock;
+ int err = _skcipher_recvmsg(sock, msg, ignored, flags);
+
+ /*
+ * This error covers -EIOCBQUEUED which implies that we can
+ * only handle one AIO request. If the caller wants to have
+ * multiple AIO requests in parallel, he must make multiple
+ * separate AIO calls.
+ */
+ if (err < 0) {
+ ret = err;
+ goto out;
}
+ if (!err)
+ goto out;
- used = min_t(unsigned long, ctx->used, msg_data_left(msg));
-
- used = af_alg_make_sg(&ctx->rsgl, &msg->msg_iter, used);
- err = used;
- if (err < 0)
- goto unlock;
-
- if (ctx->more || used < ctx->used)
- used -= used % bs;
-
- err = -EINVAL;
- if (!used)
- goto free;
-
- sgl = list_first_entry(&ctx->tsgl,
- struct skcipher_sg_list, list);
- sg = sgl->sg;
-
- while (!sg->length)
- sg++;
-
- skcipher_request_set_crypt(&ctx->req, sg, ctx->rsgl.sg, used,
- ctx->iv);
-
- err = af_alg_wait_for_completion(
- ctx->enc ?
- crypto_skcipher_encrypt(&ctx->req) :
- crypto_skcipher_decrypt(&ctx->req),
- &ctx->completion);
-
-free:
- af_alg_free_sg(&ctx->rsgl);
-
- if (err)
- goto unlock;
-
- copied += used;
- skcipher_pull_sgl(sk, used, 1);
- iov_iter_advance(&msg->msg_iter, used);
+ ret += err;
}
- err = 0;
-
-unlock:
+out:
skcipher_wmem_wakeup(sk);
release_sock(sk);
-
- return copied ?: err;
-}
-
-static int skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
- size_t ignored, int flags)
-{
- return (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) ?
- skcipher_recvmsg_async(sock, msg, flags) :
- skcipher_recvmsg_sync(sock, msg, flags);
+ return ret;
}
static unsigned int skcipher_poll(struct file *file, struct socket *sock,
@@ -724,10 +748,9 @@ static unsigned int skcipher_poll(struct file *file, struct socket *sock,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- unsigned int mask;
+ unsigned int mask = 0;
sock_poll_wait(file, sk_sleep(sk), wait);
- mask = 0;
if (ctx->used)
mask |= POLLIN | POLLRDNORM;
@@ -895,26 +918,20 @@ static int skcipher_setkey(void *private, const u8 *key, unsigned int keylen)
return err;
}
-static void skcipher_wait(struct sock *sk)
-{
- struct alg_sock *ask = alg_sk(sk);
- struct skcipher_ctx *ctx = ask->private;
- int ctr = 0;
-
- while (atomic_read(&ctx->inflight) && ctr++ < 100)
- msleep(100);
-}
-
static void skcipher_sock_destruct(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct skcipher_ctx *ctx = ask->private;
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(&ctx->req);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
+ struct skcipher_tfm *skc = pask->private;
+ struct crypto_skcipher *tfm = skc->skcipher;
- if (atomic_read(&ctx->inflight))
- skcipher_wait(sk);
+ /* Suspend caller if AIO operations are in flight. */
+ wait_event_interruptible(skcipher_aio_finish_wait,
+ (ctx->inflight == 0));
- skcipher_free_sgl(sk);
+ skcipher_pull_tsgl(sk, ctx->used, NULL);
sock_kzfree_s(sk, ctx->iv, crypto_skcipher_ivsize(tfm));
sock_kfree_s(sk, ctx, ctx->len);
af_alg_release_parent(sk);
@@ -926,7 +943,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct skcipher_tfm *tfm = private;
struct crypto_skcipher *skcipher = tfm->skcipher;
- unsigned int len = sizeof(*ctx) + crypto_skcipher_reqsize(skcipher);
+ unsigned int len = sizeof(*ctx);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
@@ -941,22 +958,18 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
memset(ctx->iv, 0, crypto_skcipher_ivsize(skcipher));
- INIT_LIST_HEAD(&ctx->tsgl);
+ INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
ctx->used = 0;
+ ctx->rcvused = 0;
+ ctx->inflight = 0;
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
- atomic_set(&ctx->inflight, 0);
af_alg_init_completion(&ctx->completion);
ask->private = ctx;
- skcipher_request_set_tfm(&ctx->req, skcipher);
- skcipher_request_set_callback(&ctx->req, CRYPTO_TFM_REQ_MAY_SLEEP |
- CRYPTO_TFM_REQ_MAY_BACKLOG,
- af_alg_complete, &ctx->completion);
-
sk->sk_destruct = skcipher_sock_destruct;
return 0;
--
2.9.3
^ permalink raw reply related
* [PATCH v7 2/2] crypto: aead AF_ALG - overhaul memory management
From: Stephan Müller @ 2017-04-10 12:12 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
In-Reply-To: <5852545.bu9l2NtIoo@positron.chronox.de>
The updated memory management is described in the top part of the code.
As one benefit of the changed memory management, the AIO and synchronous
operation is now implemented in one common function. The AF_ALG
operation uses the async kernel crypto API interface for each cipher
operation. Thus, the only difference between the AIO and sync operation
types visible from user space is:
1. the callback function to be invoked when the asynchronous operation
is completed
2. whether to wait for the completion of the kernel crypto API operation
or not
The change includes the overhaul of the TX and RX SGL handling. The TX
SGL holding the data sent from user space to the kernel is now dynamic
similar to algif_skcipher. This dynamic nature allows a continuous
operation of a thread sending data and a second thread receiving the
data. These threads do not need to synchronize as the kernel processes
as much data from the TX SGL to fill the RX SGL.
The caller reading the data from the kernel defines the amount of data
to be processed. Considering that the interface covers AEAD
authenticating ciphers, the reader must provide the buffer in the
correct size. Thus the reader defines the encryption size.
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_aead.c | 776 ++++++++++++++++++++++++++++++----------------------
1 file changed, 454 insertions(+), 322 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 1f77a6f..1061340 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -5,12 +5,26 @@
*
* This file provides the user-space API for AEAD ciphers.
*
- * This file is derived from algif_skcipher.c.
- *
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
+ *
+ * The following concept of the memory management is used:
+ *
+ * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
+ * filled by user space with the data submitted via sendpage/sendmsg. Filling
+ * up the TX SGL does not cause a crypto operation -- the data will only be
+ * tracked by the kernel. Upon receipt of one recvmsg call, the caller must
+ * provide a buffer which is tracked with the RX SGL.
+ *
+ * During the processing of the recvmsg operation, the cipher request is
+ * allocated and prepared. As part of the recvmsg operation, the processed
+ * TX buffers are extracted from the TX SGL into a separate SGL.
+ *
+ * After the completion of the crypto operation, the RX SGL and the cipher
+ * request is released. The extracted TX SGL parts are released together with
+ * the RX SGL release.
*/
#include <crypto/internal/aead.h>
@@ -25,23 +39,32 @@
#include <linux/net.h>
#include <net/sock.h>
-struct aead_sg_list {
- unsigned int cur;
- struct scatterlist sg[ALG_MAX_PAGES];
+struct aead_tsgl {
+ struct list_head list;
+ unsigned int cur; /* Last processed SG entry */
+ struct scatterlist sg[0]; /* Array of SGs forming the SGL */
};
-struct aead_async_rsgl {
+struct aead_rsgl {
struct af_alg_sgl sgl;
struct list_head list;
+ size_t sg_num_bytes; /* Bytes of data in that SGL */
};
struct aead_async_req {
- struct scatterlist *tsgl;
- struct aead_async_rsgl first_rsgl;
- struct list_head list;
struct kiocb *iocb;
- unsigned int tsgls;
- char iv[];
+ struct sock *sk;
+
+ struct aead_rsgl first_rsgl; /* First RX SG */
+ struct list_head rsgl_list; /* Track RX SGs */
+
+ struct scatterlist *tsgl; /* priv. TX SGL of buffers to process */
+ unsigned int tsgl_entries; /* number of entries in priv. TX SGL */
+
+ unsigned int outlen; /* Filled output buf length */
+
+ unsigned int areqlen; /* Length of this data struct */
+ struct aead_request aead_req; /* req ctx trails this struct */
};
struct aead_tfm {
@@ -50,25 +73,29 @@ struct aead_tfm {
};
struct aead_ctx {
- struct aead_sg_list tsgl;
- struct aead_async_rsgl first_rsgl;
- struct list_head list;
+ struct list_head tsgl_list; /* Link to TX SGL */
void *iv;
+ size_t aead_assoclen;
- struct af_alg_completion completion;
+ struct af_alg_completion completion; /* sync work queue */
- unsigned long used;
+ unsigned int inflight; /* Outstanding AIO ops */
+ size_t used; /* TX bytes sent to kernel */
+ size_t rcvused; /* total RX bytes to be processed by kernel */
- unsigned int len;
- bool more;
- bool merge;
- bool enc;
+ bool more; /* More data to be expected? */
+ bool merge; /* Merge new data into existing SG */
+ bool enc; /* Crypto operation: enc, dec */
- size_t aead_assoclen;
- struct aead_request aead_req;
+ unsigned int len; /* Length of allocated memory for this struct */
};
+static DECLARE_WAIT_QUEUE_HEAD(aead_aio_finish_wait);
+
+#define MAX_SGL_ENTS ((4096 - sizeof(struct aead_tsgl)) / \
+ sizeof(struct scatterlist) - 1)
+
static inline int aead_sndbuf(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
@@ -83,9 +110,29 @@ static inline bool aead_writable(struct sock *sk)
return PAGE_SIZE <= aead_sndbuf(sk);
}
-static inline bool aead_sufficient_data(struct aead_ctx *ctx)
+static inline int aead_rcvbuf(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct aead_ctx *ctx = ask->private;
+
+ return max_t(int, max_t(int, sk->sk_rcvbuf & PAGE_MASK, PAGE_SIZE) -
+ ctx->rcvused, 0);
+}
+
+static inline bool aead_readable(struct sock *sk)
{
- unsigned as = crypto_aead_authsize(crypto_aead_reqtfm(&ctx->aead_req));
+ return PAGE_SIZE <= aead_rcvbuf(sk);
+}
+
+static inline bool aead_sufficient_data(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
+ struct aead_ctx *ctx = ask->private;
+ struct aead_tfm *aeadc = pask->private;
+ struct crypto_aead *tfm = aeadc->aead;
+ unsigned int as = crypto_aead_authsize(tfm);
/*
* The minimum amount of memory needed for an AEAD cipher is
@@ -94,33 +141,166 @@ static inline bool aead_sufficient_data(struct aead_ctx *ctx)
return ctx->used >= ctx->aead_assoclen + (ctx->enc ? 0 : as);
}
-static void aead_reset_ctx(struct aead_ctx *ctx)
+static int aead_alloc_tsgl(struct sock *sk)
{
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct alg_sock *ask = alg_sk(sk);
+ struct aead_ctx *ctx = ask->private;
+ struct aead_tsgl *sgl;
+ struct scatterlist *sg = NULL;
- sg_init_table(sgl->sg, ALG_MAX_PAGES);
- sgl->cur = 0;
- ctx->used = 0;
- ctx->more = 0;
- ctx->merge = 0;
+ sgl = list_entry(ctx->tsgl_list.prev, struct aead_tsgl, list);
+ if (!list_empty(&ctx->tsgl_list))
+ sg = sgl->sg;
+
+ if (!sg || sgl->cur >= MAX_SGL_ENTS) {
+ sgl = sock_kmalloc(sk, sizeof(*sgl) +
+ sizeof(sgl->sg[0]) * (MAX_SGL_ENTS + 1),
+ GFP_KERNEL);
+ if (!sgl)
+ return -ENOMEM;
+
+ sg_init_table(sgl->sg, MAX_SGL_ENTS + 1);
+ sgl->cur = 0;
+
+ if (sg)
+ sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg);
+
+ list_add_tail(&sgl->list, &ctx->tsgl_list);
+ }
+
+ return 0;
+}
+
+static unsigned int aead_count_tsgl(struct sock *sk, size_t bytes)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct aead_ctx *ctx = ask->private;
+ struct aead_tsgl *sgl, *tmp;
+ unsigned int i;
+ unsigned int sgl_count = 0;
+
+ if (!bytes)
+ return 0;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ struct scatterlist *sg = sgl->sg;
+
+ for (i = 0; i < sgl->cur; i++) {
+ sgl_count++;
+ if (sg[i].length >= bytes)
+ return sgl_count;
+
+ bytes -= sg[i].length;
+ }
+ }
+
+ return sgl_count;
+}
+
+static void aead_pull_tsgl(struct sock *sk, size_t used,
+ struct scatterlist *dst)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct aead_ctx *ctx = ask->private;
+ struct aead_tsgl *sgl;
+ struct scatterlist *sg;
+ unsigned int i;
+
+ while (!list_empty(&ctx->tsgl_list)) {
+ sgl = list_first_entry(&ctx->tsgl_list, struct aead_tsgl,
+ list);
+ sg = sgl->sg;
+
+ for (i = 0; i < sgl->cur; i++) {
+ size_t plen = min_t(size_t, used, sg[i].length);
+ struct page *page = sg_page(sg + i);
+
+ if (!page)
+ continue;
+
+ /*
+ * Assumption: caller created aead_count_tsgl(len)
+ * SG entries in dst.
+ */
+ if (dst)
+ sg_set_page(dst + i, page, plen, sg[i].offset);
+
+ sg[i].length -= plen;
+ sg[i].offset += plen;
+
+ used -= plen;
+ ctx->used -= plen;
+
+ if (sg[i].length)
+ return;
+
+ if (!dst)
+ put_page(page);
+ sg_assign_page(sg + i, NULL);
+ }
+
+ list_del(&sgl->list);
+ sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) *
+ (MAX_SGL_ENTS + 1));
+ }
+
+ if (!ctx->used)
+ ctx->merge = 0;
}
-static void aead_put_sgl(struct sock *sk)
+static void aead_free_areq_sgls(struct aead_async_req *areq)
{
+ struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct scatterlist *sg = sgl->sg;
+ struct aead_rsgl *rsgl, *tmp;
+ struct scatterlist *tsgl;
+ struct scatterlist *sg;
unsigned int i;
- for (i = 0; i < sgl->cur; i++) {
- if (!sg_page(sg + i))
+ list_for_each_entry_safe(rsgl, tmp, &areq->rsgl_list, list) {
+ ctx->rcvused -= rsgl->sg_num_bytes;
+ af_alg_free_sg(&rsgl->sgl);
+ list_del(&rsgl->list);
+ if (rsgl != &areq->first_rsgl)
+ sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ }
+
+ tsgl = areq->tsgl;
+ for_each_sg(tsgl, sg, areq->tsgl_entries, i) {
+ if (!sg_page(sg))
continue;
+ put_page(sg_page(sg));
+ }
+
+ if (areq->tsgl && areq->tsgl_entries)
+ sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl));
+}
+
+static int aead_wait_for_wmem(struct sock *sk, unsigned flags)
+{
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
+ int err = -ERESTARTSYS;
+ long timeout;
+
+ if (flags & MSG_DONTWAIT)
+ return -EAGAIN;
+
+ sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
- put_page(sg_page(sg + i));
- sg_assign_page(sg + i, NULL);
+ add_wait_queue(sk_sleep(sk), &wait);
+ for (;;) {
+ if (signal_pending(current))
+ break;
+ timeout = MAX_SCHEDULE_TIMEOUT;
+ if (sk_wait_event(sk, &timeout, aead_writable(sk), &wait)) {
+ err = 0;
+ break;
+ }
}
- aead_reset_ctx(ctx);
+ remove_wait_queue(sk_sleep(sk), &wait);
+
+ return err;
}
static void aead_wmem_wakeup(struct sock *sk)
@@ -152,6 +332,7 @@ static int aead_wait_for_data(struct sock *sk, unsigned flags)
return -EAGAIN;
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
+
add_wait_queue(sk_sleep(sk), &wait);
for (;;) {
if (signal_pending(current))
@@ -175,8 +356,6 @@ static void aead_data_wakeup(struct sock *sk)
struct aead_ctx *ctx = ask->private;
struct socket_wq *wq;
- if (ctx->more)
- return;
if (!ctx->used)
return;
@@ -194,15 +373,18 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
struct aead_ctx *ctx = ask->private;
- unsigned ivsize =
- crypto_aead_ivsize(crypto_aead_reqtfm(&ctx->aead_req));
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct aead_tfm *aeadc = pask->private;
+ struct crypto_aead *tfm = aeadc->aead;
+ unsigned int ivsize = crypto_aead_ivsize(tfm);
+ struct aead_tsgl *sgl;
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
bool init = 0;
- int err = -EINVAL;
+ int err = 0;
if (msg->msg_controllen) {
err = af_alg_cmsg_send(msg, &con);
@@ -226,8 +408,10 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
}
lock_sock(sk);
- if (!ctx->more && ctx->used)
+ if (!ctx->more && ctx->used) {
+ err = -EINVAL;
goto unlock;
+ }
if (init) {
ctx->enc = enc;
@@ -238,11 +422,14 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
}
while (size) {
+ struct scatterlist *sg;
size_t len = size;
- struct scatterlist *sg = NULL;
+ size_t plen;
/* use the existing memory in an allocated page */
if (ctx->merge) {
+ sgl = list_entry(ctx->tsgl_list.prev,
+ struct aead_tsgl, list);
sg = sgl->sg + sgl->cur - 1;
len = min_t(unsigned long, len,
PAGE_SIZE - sg->offset - sg->length);
@@ -263,57 +450,60 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
}
if (!aead_writable(sk)) {
- /* user space sent too much data */
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- goto unlock;
+ err = aead_wait_for_wmem(sk, msg->msg_flags);
+ if (err)
+ goto unlock;
}
/* allocate a new page */
len = min_t(unsigned long, size, aead_sndbuf(sk));
- while (len) {
- size_t plen = 0;
- if (sgl->cur >= ALG_MAX_PAGES) {
- aead_put_sgl(sk);
- err = -E2BIG;
- goto unlock;
- }
+ err = aead_alloc_tsgl(sk);
+ if (err)
+ goto unlock;
+
+ sgl = list_entry(ctx->tsgl_list.prev, struct aead_tsgl,
+ list);
+ sg = sgl->sg;
+ if (sgl->cur)
+ sg_unmark_end(sg + sgl->cur - 1);
+
+ do {
+ unsigned int i = sgl->cur;
- sg = sgl->sg + sgl->cur;
plen = min_t(size_t, len, PAGE_SIZE);
- sg_assign_page(sg, alloc_page(GFP_KERNEL));
- err = -ENOMEM;
- if (!sg_page(sg))
+ sg_assign_page(sg + i, alloc_page(GFP_KERNEL));
+ if (!sg_page(sg + i)) {
+ err = -ENOMEM;
goto unlock;
+ }
- err = memcpy_from_msg(page_address(sg_page(sg)),
+ err = memcpy_from_msg(page_address(sg_page(sg + i)),
msg, plen);
if (err) {
- __free_page(sg_page(sg));
- sg_assign_page(sg, NULL);
+ __free_page(sg_page(sg + i));
+ sg_assign_page(sg + i, NULL);
goto unlock;
}
- sg->offset = 0;
- sg->length = plen;
+ sg[i].length = plen;
len -= plen;
ctx->used += plen;
copied += plen;
- sgl->cur++;
size -= plen;
- ctx->merge = plen & (PAGE_SIZE - 1);
- }
+ sgl->cur++;
+ } while (len && sgl->cur < MAX_SGL_ENTS);
+
+ if (!size)
+ sg_mark_end(sg + sgl->cur - 1);
+
+ ctx->merge = plen & (PAGE_SIZE - 1);
}
err = 0;
ctx->more = msg->msg_flags & MSG_MORE;
- if (!ctx->more && !aead_sufficient_data(ctx)) {
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- }
unlock:
aead_data_wakeup(sk);
@@ -328,15 +518,12 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct aead_sg_list *sgl = &ctx->tsgl;
+ struct aead_tsgl *sgl;
int err = -EINVAL;
if (flags & MSG_SENDPAGE_NOTLAST)
flags |= MSG_MORE;
- if (sgl->cur >= ALG_MAX_PAGES)
- return -E2BIG;
-
lock_sock(sk);
if (!ctx->more && ctx->used)
goto unlock;
@@ -345,13 +532,22 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
goto done;
if (!aead_writable(sk)) {
- /* user space sent too much data */
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- goto unlock;
+ err = aead_wait_for_wmem(sk, flags);
+ if (err)
+ goto unlock;
}
+ err = aead_alloc_tsgl(sk);
+ if (err)
+ goto unlock;
+
ctx->merge = 0;
+ sgl = list_entry(ctx->tsgl_list.prev, struct aead_tsgl, list);
+
+ if (sgl->cur)
+ sg_unmark_end(sgl->sg + sgl->cur - 1);
+
+ sg_mark_end(sgl->sg + sgl->cur);
get_page(page);
sg_set_page(sgl->sg + sgl->cur, page, size, offset);
@@ -362,11 +558,6 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
done:
ctx->more = flags & MSG_MORE;
- if (!ctx->more && !aead_sufficient_data(ctx)) {
- aead_put_sgl(sk);
- err = -EMSGSIZE;
- }
-
unlock:
aead_data_wakeup(sk);
release_sock(sk);
@@ -374,205 +565,59 @@ static ssize_t aead_sendpage(struct socket *sock, struct page *page,
return err ?: size;
}
-#define GET_ASYM_REQ(req, tfm) (struct aead_async_req *) \
- ((char *)req + sizeof(struct aead_request) + \
- crypto_aead_reqsize(tfm))
-
- #define GET_REQ_SIZE(tfm) sizeof(struct aead_async_req) + \
- crypto_aead_reqsize(tfm) + crypto_aead_ivsize(tfm) + \
- sizeof(struct aead_request)
-
static void aead_async_cb(struct crypto_async_request *_req, int err)
{
- struct sock *sk = _req->data;
+ struct aead_async_req *areq = _req->data;
+ struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- struct crypto_aead *tfm = crypto_aead_reqtfm(&ctx->aead_req);
- struct aead_request *req = aead_request_cast(_req);
- struct aead_async_req *areq = GET_ASYM_REQ(req, tfm);
- struct scatterlist *sg = areq->tsgl;
- struct aead_async_rsgl *rsgl;
struct kiocb *iocb = areq->iocb;
- unsigned int i, reqlen = GET_REQ_SIZE(tfm);
-
- list_for_each_entry(rsgl, &areq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &areq->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
-
- for (i = 0; i < areq->tsgls; i++)
- put_page(sg_page(sg + i));
-
- sock_kfree_s(sk, areq->tsgl, sizeof(*areq->tsgl) * areq->tsgls);
- sock_kfree_s(sk, req, reqlen);
- __sock_put(sk);
- iocb->ki_complete(iocb, err, err);
-}
-
-static int aead_recvmsg_async(struct socket *sock, struct msghdr *msg,
- int flags)
-{
- struct sock *sk = sock->sk;
- struct alg_sock *ask = alg_sk(sk);
- struct aead_ctx *ctx = ask->private;
- struct crypto_aead *tfm = crypto_aead_reqtfm(&ctx->aead_req);
- struct aead_async_req *areq;
- struct aead_request *req = NULL;
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct aead_async_rsgl *last_rsgl = NULL, *rsgl;
- unsigned int as = crypto_aead_authsize(tfm);
- unsigned int i, reqlen = GET_REQ_SIZE(tfm);
- int err = -ENOMEM;
- unsigned long used;
- size_t outlen = 0;
- size_t usedpages = 0;
+ unsigned int resultlen;
lock_sock(sk);
- if (ctx->more) {
- err = aead_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- if (!aead_sufficient_data(ctx))
- goto unlock;
- used = ctx->used;
- if (ctx->enc)
- outlen = used + as;
- else
- outlen = used - as;
+ BUG_ON(!ctx->inflight);
- req = sock_kmalloc(sk, reqlen, GFP_KERNEL);
- if (unlikely(!req))
- goto unlock;
-
- areq = GET_ASYM_REQ(req, tfm);
- memset(&areq->first_rsgl, '\0', sizeof(areq->first_rsgl));
- INIT_LIST_HEAD(&areq->list);
- areq->iocb = msg->msg_iocb;
- memcpy(areq->iv, ctx->iv, crypto_aead_ivsize(tfm));
- aead_request_set_tfm(req, tfm);
- aead_request_set_ad(req, ctx->aead_assoclen);
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- aead_async_cb, sk);
- used -= ctx->aead_assoclen;
+ /* Buffer size written by crypto operation. */
+ resultlen = areq->outlen;
- /* take over all tx sgls from ctx */
- areq->tsgl = sock_kmalloc(sk,
- sizeof(*areq->tsgl) * max_t(u32, sgl->cur, 1),
- GFP_KERNEL);
- if (unlikely(!areq->tsgl))
- goto free;
-
- sg_init_table(areq->tsgl, max_t(u32, sgl->cur, 1));
- for (i = 0; i < sgl->cur; i++)
- sg_set_page(&areq->tsgl[i], sg_page(&sgl->sg[i]),
- sgl->sg[i].length, sgl->sg[i].offset);
-
- areq->tsgls = sgl->cur;
-
- /* create rx sgls */
- while (outlen > usedpages && iov_iter_count(&msg->msg_iter)) {
- size_t seglen = min_t(size_t, iov_iter_count(&msg->msg_iter),
- (outlen - usedpages));
-
- if (list_empty(&areq->list)) {
- rsgl = &areq->first_rsgl;
-
- } else {
- rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
- if (unlikely(!rsgl)) {
- err = -ENOMEM;
- goto free;
- }
- }
- rsgl->sgl.npages = 0;
- list_add_tail(&rsgl->list, &areq->list);
-
- /* make one iovec available as scatterlist */
- err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
- if (err < 0)
- goto free;
-
- usedpages += err;
-
- /* chain the new scatterlist with previous one */
- if (last_rsgl)
- af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
-
- last_rsgl = rsgl;
-
- iov_iter_advance(&msg->msg_iter, err);
- }
-
- /* ensure output buffer is sufficiently large */
- if (usedpages < outlen) {
- err = -EINVAL;
- goto unlock;
- }
+ aead_free_areq_sgls(areq);
+ sock_kfree_s(sk, areq, areq->areqlen);
+ __sock_put(sk);
+ ctx->inflight--;
- aead_request_set_crypt(req, areq->tsgl, areq->first_rsgl.sgl.sg, used,
- areq->iv);
- err = ctx->enc ? crypto_aead_encrypt(req) : crypto_aead_decrypt(req);
- if (err) {
- if (err == -EINPROGRESS) {
- sock_hold(sk);
- err = -EIOCBQUEUED;
- aead_reset_ctx(ctx);
- goto unlock;
- } else if (err == -EBADMSG) {
- aead_put_sgl(sk);
- }
- goto free;
- }
- aead_put_sgl(sk);
+ iocb->ki_complete(iocb, err ? err : resultlen, 0);
-free:
- list_for_each_entry(rsgl, &areq->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- if (rsgl != &areq->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
- }
- if (areq->tsgl)
- sock_kfree_s(sk, areq->tsgl, sizeof(*areq->tsgl) * areq->tsgls);
- if (req)
- sock_kfree_s(sk, req, reqlen);
-unlock:
- aead_wmem_wakeup(sk);
release_sock(sk);
- return err ? err : outlen;
+
+ wake_up_interruptible(&aead_aio_finish_wait);
}
-static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
+static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
struct aead_ctx *ctx = ask->private;
- unsigned as = crypto_aead_authsize(crypto_aead_reqtfm(&ctx->aead_req));
- struct aead_sg_list *sgl = &ctx->tsgl;
- struct aead_async_rsgl *last_rsgl = NULL;
- struct aead_async_rsgl *rsgl, *tmp;
- int err = -EINVAL;
- unsigned long used = 0;
- size_t outlen = 0;
- size_t usedpages = 0;
-
- lock_sock(sk);
+ struct aead_tfm *aeadc = pask->private;
+ struct crypto_aead *tfm = aeadc->aead;
+ unsigned int as = crypto_aead_authsize(tfm);
+ unsigned int areqlen =
+ sizeof(struct aead_async_req) + crypto_aead_reqsize(tfm);
+ struct aead_async_req *areq;
+ struct aead_rsgl *last_rsgl = NULL;
+ int err = 0;
+ size_t used = 0; /* [in] TX bufs to be en/decrypted */
+ size_t outlen = 0; /* [out] RX bufs produced by kernel */
+ size_t usedpages = 0; /* [in] RX bufs to be used from user */
+ size_t processed = 0; /* [in] TX bufs to be consumed */
/*
- * Please see documentation of aead_request_set_crypt for the
- * description of the AEAD memory structure expected from the caller.
+ * Data length provided by caller via sendmsg/sendpage that has not
+ * yet been processed.
*/
-
- if (ctx->more) {
- err = aead_wait_for_data(sk, flags);
- if (err)
- goto unlock;
- }
-
- /* data length provided by caller via sendmsg/sendpage */
used = ctx->used;
/*
@@ -584,8 +629,8 @@ static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
* the error message in sendmsg/sendpage and still call recvmsg. This
* check here protects the kernel integrity.
*/
- if (!aead_sufficient_data(ctx))
- goto unlock;
+ if (!aead_sufficient_data(sk))
+ return -EINVAL;
/*
* Calculate the minimum output buffer size holding the result of the
@@ -606,84 +651,172 @@ static int aead_recvmsg_sync(struct socket *sock, struct msghdr *msg, int flags)
*/
used -= ctx->aead_assoclen;
- /* convert iovecs of output buffers into scatterlists */
- while (outlen > usedpages && iov_iter_count(&msg->msg_iter)) {
- size_t seglen = min_t(size_t, iov_iter_count(&msg->msg_iter),
- (outlen - usedpages));
+ /* Allocate cipher request for current operation. */
+ areq = sock_kmalloc(sk, areqlen, GFP_KERNEL);
+ if (unlikely(!areq))
+ return -ENOMEM;
+ areq->areqlen = areqlen;
+ areq->sk = sk;
+ INIT_LIST_HEAD(&areq->rsgl_list);
+ areq->tsgl = NULL;
+ areq->tsgl_entries = 0;
+
+ /* convert iovecs of output buffers into RX SGL */
+ while (outlen > usedpages && msg_data_left(msg)) {
+ struct aead_rsgl *rsgl;
+ size_t seglen;
+
+ /* limit the amount of readable buffers */
+ if (!aead_readable(sk))
+ break;
- if (list_empty(&ctx->list)) {
- rsgl = &ctx->first_rsgl;
+ if (!ctx->used) {
+ err = aead_wait_for_data(sk, flags);
+ if (err)
+ goto free;
+ }
+
+ seglen = min_t(size_t, (outlen - usedpages),
+ msg_data_left(msg));
+
+ if (list_empty(&areq->rsgl_list)) {
+ rsgl = &areq->first_rsgl;
} else {
rsgl = sock_kmalloc(sk, sizeof(*rsgl), GFP_KERNEL);
if (unlikely(!rsgl)) {
err = -ENOMEM;
- goto unlock;
+ goto free;
}
}
+
rsgl->sgl.npages = 0;
- list_add_tail(&rsgl->list, &ctx->list);
+ list_add_tail(&rsgl->list, &areq->rsgl_list);
/* make one iovec available as scatterlist */
err = af_alg_make_sg(&rsgl->sgl, &msg->msg_iter, seglen);
if (err < 0)
- goto unlock;
- usedpages += err;
+ goto free;
+
/* chain the new scatterlist with previous one */
if (last_rsgl)
af_alg_link_sg(&last_rsgl->sgl, &rsgl->sgl);
last_rsgl = rsgl;
-
+ usedpages += err;
+ ctx->rcvused += err;
+ rsgl->sg_num_bytes = err;
iov_iter_advance(&msg->msg_iter, err);
}
- /* ensure output buffer is sufficiently large */
+ /*
+ * Ensure output buffer is sufficiently large. If the caller provides
+ * less buffer space, only use the relative required input size. This
+ * allows AIO operation where the caller sent all data to be processed
+ * and the AIO operation performs the operation on the different chunks
+ * of the input data.
+ */
if (usedpages < outlen) {
- err = -EINVAL;
- goto unlock;
- }
+ size_t less = outlen - usedpages;
- sg_mark_end(sgl->sg + sgl->cur - 1);
- aead_request_set_crypt(&ctx->aead_req, sgl->sg, ctx->first_rsgl.sgl.sg,
- used, ctx->iv);
- aead_request_set_ad(&ctx->aead_req, ctx->aead_assoclen);
+ if (used < less) {
+ err = -EINVAL;
+ goto free;
+ }
+ used -= less;
+ outlen -= less;
+ }
- err = af_alg_wait_for_completion(ctx->enc ?
- crypto_aead_encrypt(&ctx->aead_req) :
- crypto_aead_decrypt(&ctx->aead_req),
+ /*
+ * Create a per request TX SGL for this request which tracks the
+ * SG entries from the global TX SGL.
+ */
+ processed = used + ctx->aead_assoclen;
+ areq->tsgl_entries = aead_count_tsgl(sk, processed);
+ if (!areq->tsgl_entries)
+ areq->tsgl_entries = 1;
+ areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) * areq->tsgl_entries,
+ GFP_KERNEL);
+ if (!areq->tsgl) {
+ err = -ENOMEM;
+ goto free;
+ }
+ sg_init_table(areq->tsgl, areq->tsgl_entries);
+ aead_pull_tsgl(sk, processed, areq->tsgl);
+
+ /* Initialize the crypto operation */
+ aead_request_set_crypt(&areq->aead_req, areq->tsgl,
+ areq->first_rsgl.sgl.sg, used, ctx->iv);
+ aead_request_set_ad(&areq->aead_req, ctx->aead_assoclen);
+ aead_request_set_tfm(&areq->aead_req, tfm);
+
+ if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
+ /* AIO operation */
+ areq->iocb = msg->msg_iocb;
+ aead_request_set_callback(&areq->aead_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ aead_async_cb, areq);
+ err = ctx->enc ? crypto_aead_encrypt(&areq->aead_req) :
+ crypto_aead_decrypt(&areq->aead_req);
+ } else {
+ /* Synchronous operation */
+ aead_request_set_callback(&areq->aead_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG,
+ af_alg_complete, &ctx->completion);
+ err = af_alg_wait_for_completion(ctx->enc ?
+ crypto_aead_encrypt(&areq->aead_req) :
+ crypto_aead_decrypt(&areq->aead_req),
&ctx->completion);
-
- if (err) {
- /* EBADMSG implies a valid cipher operation took place */
- if (err == -EBADMSG)
- aead_put_sgl(sk);
-
- goto unlock;
}
- aead_put_sgl(sk);
- err = 0;
+ /* AIO operation in progress */
+ if (err == -EINPROGRESS) {
+ sock_hold(sk);
+ ctx->inflight++;
-unlock:
- list_for_each_entry_safe(rsgl, tmp, &ctx->list, list) {
- af_alg_free_sg(&rsgl->sgl);
- list_del(&rsgl->list);
- if (rsgl != &ctx->first_rsgl)
- sock_kfree_s(sk, rsgl, sizeof(*rsgl));
+ /* Remember output size that will be generated. */
+ areq->outlen = outlen;
+
+ return -EIOCBQUEUED;
}
- INIT_LIST_HEAD(&ctx->list);
- aead_wmem_wakeup(sk);
- release_sock(sk);
+
+free:
+ aead_free_areq_sgls(areq);
+ if (areq)
+ sock_kfree_s(sk, areq, areqlen);
return err ? err : outlen;
}
-static int aead_recvmsg(struct socket *sock, struct msghdr *msg, size_t ignored,
- int flags)
+static int aead_recvmsg(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
{
- return (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) ?
- aead_recvmsg_async(sock, msg, flags) :
- aead_recvmsg_sync(sock, msg, flags);
+ struct sock *sk = sock->sk;
+ int ret = 0;
+
+ lock_sock(sk);
+ while (msg_data_left(msg)) {
+ int err = _aead_recvmsg(sock, msg, ignored, flags);
+
+ /*
+ * This error covers -EIOCBQUEUED which implies that we can
+ * only handle one AIO request. If the caller wants to have
+ * multiple AIO requests in parallel, he must make multiple
+ * separate AIO calls.
+ */
+ if (err < 0) {
+ ret = err;
+ goto out;
+ }
+ if (!err)
+ goto out;
+
+ ret += err;
+ }
+
+out:
+ aead_wmem_wakeup(sk);
+ release_sock(sk);
+ return ret;
}
static unsigned int aead_poll(struct file *file, struct socket *sock,
@@ -692,10 +825,9 @@ static unsigned int aead_poll(struct file *file, struct socket *sock,
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned int mask;
+ unsigned int mask = 0;
sock_poll_wait(file, sk_sleep(sk), wait);
- mask = 0;
if (!ctx->more)
mask |= POLLIN | POLLRDNORM;
@@ -881,7 +1013,11 @@ static void aead_sock_destruct(struct sock *sk)
unsigned int ivlen = crypto_aead_ivsize(tfm);
WARN_ON(atomic_read(&sk->sk_refcnt) != 0);
- aead_put_sgl(sk);
+
+ /* Suspend caller if AIO operations are in flight. */
+ wait_event_interruptible(aead_aio_finish_wait, (ctx->inflight == 0));
+
+ aead_pull_tsgl(sk, ctx->used, NULL);
sock_kzfree_s(sk, ctx->iv, ivlen);
sock_kfree_s(sk, ctx, ctx->len);
af_alg_release_parent(sk);
@@ -893,7 +1029,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct aead_tfm *tfm = private;
struct crypto_aead *aead = tfm->aead;
- unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(aead);
+ unsigned int len = sizeof(*ctx);
unsigned int ivlen = crypto_aead_ivsize(aead);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
@@ -908,23 +1044,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
}
memset(ctx->iv, 0, ivlen);
+ INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
ctx->used = 0;
+ ctx->rcvused = 0;
+ ctx->inflight = 0;
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
- ctx->tsgl.cur = 0;
ctx->aead_assoclen = 0;
af_alg_init_completion(&ctx->completion);
- sg_init_table(ctx->tsgl.sg, ALG_MAX_PAGES);
- INIT_LIST_HEAD(&ctx->list);
ask->private = ctx;
- aead_request_set_tfm(&ctx->aead_req, aead);
- aead_request_set_callback(&ctx->aead_req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- af_alg_complete, &ctx->completion);
-
sk->sk_destruct = aead_sock_destruct;
return 0;
--
2.9.3
^ permalink raw reply related
* [PATCH v7 0/2] crypto: AF_ALG memory management fix
From: Stephan Müller @ 2017-04-10 12:11 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
Hi Herbert,
Changes v7:
* use msg_data_left instead of iov_iter_count
* rebase algif_aead patch on top of the require setkey before accept(2) (i.e.
the *nokey functions are included)
Changes v6:
* port to 4.11-rc1 (a header file was included from upstream)
* Limit the RX SGL in an identical fashion as the TX SGL size is limited by
adding [skcipher|aead]_readable which is an equivalent to the existing
[skcipher|aead]_writable. To ensure that the limit is enforced, ctx->rcvused
is added which tracks the RX SGL memory currently in use for the context
(i.e. covering all requests that are currently processed).
* Add an outer loop to the recvmsg operation to allow continuous recvmsg
operation in parallel to a continuous sendmsg operation.
Changes v5:
* use list_for_each_entry_safe in aead_count_tsgl
Changes v4:
* replace ctx->processed with a maintenance of a private recvmsg
TX SGL
* algif_skcipher: rename skcipher_sg_list into skcipher_tsgl to make
it consistent with algif_aead to prepare for a code consolidation
Changes v3:
* in *_pull_tsgl: make sure ctx->processed cannot be less than zero
* perform fuzzing of all input parameters with bogus values
Changes v2:
* import fix from Harsh Jain <harsh@chelsio.com> to remove SG
from list before freeing
* fix return code used for ki_complete to match AIO behavior
with sync behavior
* rename variable list -> tsgl_list
* update the algif_aead patch to include a dynamic TX SGL
allocation similar to what algif_skcipher does. This allows
concurrent continuous read/write operations to the extent
you requested. Although I have not implemented "pairs of
TX/RX SGLs" as I think that is even more overhead, the
implementation conceptually defines such pairs. The recvmsg
call defines how much from the input data is processed.
The caller can have arbitrary number of sendmsg calls
where the data is added to the TX SGL before an recvmsg
asks the kernel to process a given amount (or all) of the
TX SGL.
With the changes, you will see a lot of code duplication now
as I deliberately tried to use the same struct and variable names,
the same function names and even the same oder of functions.
If you agree to this patch, I volunteer to provide a followup
patch that will extract the code duplication into common
functions.
Please find attached memory management updates to
- simplify the code: the old AIO memory management is very
complex and seemingly very fragile -- the update now
eliminates all reported bugs in the skcipher and AEAD
interfaces which allowed the kernel to be crashed by
an unprivileged user
- streamline the code: there is one code path for AIO and sync
operation; the code between algif_skcipher and algif_aead
is very similar (if that patch set is accepted, I volunteer
to reduce code duplication by moving service operations
into af_alg.c and to further unify the TX SGL handling)
- unify the AIO and sync operation which only differ in the
kernel crypto API callback and whether to wait for the
crypto operation or not
- fix all reported bugs regarding the handling of multiple
IOCBs.
The following testing was performed:
- stress testing to verify that no memleaks exist
- testing using Tadeusz Struck AIO test tool (see
https://github.com/tstruk/afalg_async_test) -- the AEAD test
is not applicable any more due to the changed user space
interface; the skcipher test works once the user space
interface change is honored in the test code
- using the libkcapi test suite, all tests including the
originally failing ones (AIO with multiple IOCBs) work now --
the current libkcapi code artificially limits the AEAD
operation to one IOCB. After altering the libkcapi code
to allow multiple IOCBs, the testing works flawless.
Stephan Mueller (2):
crypto: skcipher AF_ALG - overhaul memory management
crypto: aead AF_ALG - overhaul memory management
crypto/algif_aead.c | 776 ++++++++++++++++++++++++++++--------------------
crypto/algif_skcipher.c | 579 ++++++++++++++++++------------------
2 files changed, 750 insertions(+), 605 deletions(-)
--
2.9.3
^ permalink raw reply
* [PATCH] crypto: algif_aead - Require setkey before accept(2)
From: Stephan Müller @ 2017-04-10 11:59 UTC (permalink / raw)
To: herbert; +Cc: linux-crypto
Some cipher implementations will crash if you try to use them
without calling setkey first. This patch adds a check so that
the accept(2) call will fail with -ENOKEY if setkey hasn't been
done on the socket yet.
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/algif_aead.c | 164 ++++++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 154 insertions(+), 10 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 5a80537..1f77a6f 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -44,6 +44,11 @@ struct aead_async_req {
char iv[];
};
+struct aead_tfm {
+ struct crypto_aead *aead;
+ bool has_key;
+};
+
struct aead_ctx {
struct aead_sg_list tsgl;
struct aead_async_rsgl first_rsgl;
@@ -723,32 +728,157 @@ static struct proto_ops algif_aead_ops = {
.poll = aead_poll,
};
+static int aead_check_key(struct socket *sock)
+{
+ int err = 0;
+ struct sock *psk;
+ struct alg_sock *pask;
+ struct aead_tfm *tfm;
+ struct sock *sk = sock->sk;
+ struct alg_sock *ask = alg_sk(sk);
+
+ lock_sock(sk);
+ if (ask->refcnt)
+ goto unlock_child;
+
+ psk = ask->parent;
+ pask = alg_sk(ask->parent);
+ tfm = pask->private;
+
+ err = -ENOKEY;
+ lock_sock_nested(psk, SINGLE_DEPTH_NESTING);
+ if (!tfm->has_key)
+ goto unlock;
+
+ if (!pask->refcnt++)
+ sock_hold(psk);
+
+ ask->refcnt = 1;
+ sock_put(psk);
+
+ err = 0;
+
+unlock:
+ release_sock(psk);
+unlock_child:
+ release_sock(sk);
+
+ return err;
+}
+
+static int aead_sendmsg_nokey(struct socket *sock, struct msghdr *msg,
+ size_t size)
+{
+ int err;
+
+ err = aead_check_key(sock);
+ if (err)
+ return err;
+
+ return aead_sendmsg(sock, msg, size);
+}
+
+static ssize_t aead_sendpage_nokey(struct socket *sock, struct page *page,
+ int offset, size_t size, int flags)
+{
+ int err;
+
+ err = aead_check_key(sock);
+ if (err)
+ return err;
+
+ return aead_sendpage(sock, page, offset, size, flags);
+}
+
+static int aead_recvmsg_nokey(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
+{
+ int err;
+
+ err = aead_check_key(sock);
+ if (err)
+ return err;
+
+ return aead_recvmsg(sock, msg, ignored, flags);
+}
+
+static struct proto_ops algif_aead_ops_nokey = {
+ .family = PF_ALG,
+
+ .connect = sock_no_connect,
+ .socketpair = sock_no_socketpair,
+ .getname = sock_no_getname,
+ .ioctl = sock_no_ioctl,
+ .listen = sock_no_listen,
+ .shutdown = sock_no_shutdown,
+ .getsockopt = sock_no_getsockopt,
+ .mmap = sock_no_mmap,
+ .bind = sock_no_bind,
+ .accept = sock_no_accept,
+ .setsockopt = sock_no_setsockopt,
+
+ .release = af_alg_release,
+ .sendmsg = aead_sendmsg_nokey,
+ .sendpage = aead_sendpage_nokey,
+ .recvmsg = aead_recvmsg_nokey,
+ .poll = aead_poll,
+};
+
static void *aead_bind(const char *name, u32 type, u32 mask)
{
- return crypto_alloc_aead(name, type, mask);
+ struct aead_tfm *tfm;
+ struct crypto_aead *aead;
+
+ tfm = kzalloc(sizeof(*tfm), GFP_KERNEL);
+ if (!tfm)
+ return ERR_PTR(-ENOMEM);
+
+ aead = crypto_alloc_aead(name, type, mask);
+ if (IS_ERR(aead)) {
+ kfree(tfm);
+ return ERR_CAST(aead);
+ }
+
+ tfm->aead = aead;
+
+ return tfm;
}
static void aead_release(void *private)
{
- crypto_free_aead(private);
+ struct aead_tfm *tfm = private;
+
+ crypto_free_aead(tfm->aead);
+ kfree(tfm);
}
static int aead_setauthsize(void *private, unsigned int authsize)
{
- return crypto_aead_setauthsize(private, authsize);
+ struct aead_tfm *tfm = private;
+
+ return crypto_aead_setauthsize(tfm->aead, authsize);
}
static int aead_setkey(void *private, const u8 *key, unsigned int keylen)
{
- return crypto_aead_setkey(private, key, keylen);
+ struct aead_tfm *tfm = private;
+ int err;
+
+ err = crypto_aead_setkey(tfm->aead, key, keylen);
+ tfm->has_key = !err;
+
+ return err;
}
static void aead_sock_destruct(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct aead_ctx *ctx = ask->private;
- unsigned int ivlen = crypto_aead_ivsize(
- crypto_aead_reqtfm(&ctx->aead_req));
+ struct sock *psk = ask->parent;
+ struct alg_sock *pask = alg_sk(psk);
+ struct aead_tfm *aeadc = pask->private;
+ struct crypto_aead *tfm = aeadc->aead;
+ unsigned int ivlen = crypto_aead_ivsize(tfm);
WARN_ON(atomic_read(&sk->sk_refcnt) != 0);
aead_put_sgl(sk);
@@ -757,12 +887,14 @@ static void aead_sock_destruct(struct sock *sk)
af_alg_release_parent(sk);
}
-static int aead_accept_parent(void *private, struct sock *sk)
+static int aead_accept_parent_nokey(void *private, struct sock *sk)
{
struct aead_ctx *ctx;
struct alg_sock *ask = alg_sk(sk);
- unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(private);
- unsigned int ivlen = crypto_aead_ivsize(private);
+ struct aead_tfm *tfm = private;
+ struct crypto_aead *aead = tfm->aead;
+ unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(aead);
+ unsigned int ivlen = crypto_aead_ivsize(aead);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
@@ -789,7 +921,7 @@ static int aead_accept_parent(void *private, struct sock *sk)
ask->private = ctx;
- aead_request_set_tfm(&ctx->aead_req, private);
+ aead_request_set_tfm(&ctx->aead_req, aead);
aead_request_set_callback(&ctx->aead_req, CRYPTO_TFM_REQ_MAY_BACKLOG,
af_alg_complete, &ctx->completion);
@@ -798,13 +930,25 @@ static int aead_accept_parent(void *private, struct sock *sk)
return 0;
}
+static int aead_accept_parent(void *private, struct sock *sk)
+{
+ struct aead_tfm *tfm = private;
+
+ if (!tfm->has_key)
+ return -ENOKEY;
+
+ return aead_accept_parent_nokey(private, sk);
+}
+
static const struct af_alg_type algif_type_aead = {
.bind = aead_bind,
.release = aead_release,
.setkey = aead_setkey,
.setauthsize = aead_setauthsize,
.accept = aead_accept_parent,
+ .accept_nokey = aead_accept_parent_nokey,
.ops = &algif_aead_ops,
+ .ops_nokey = &algif_aead_ops_nokey,
.name = "aead",
.owner = THIS_MODULE
};
--
2.9.3
^ permalink raw reply related
* Re: What should be the algo priority
From: Herbert Xu @ 2017-04-10 11:25 UTC (permalink / raw)
To: Harsh Jain; +Cc: smueller, linux-crypto
In-Reply-To: <CAFXBA=mHp_LB9tufr-aRGXkfoULGbMn5RxRV1ajSozx88Er0CQ@mail.gmail.com>
Harsh Jain <harshjain.prof@gmail.com> wrote:
>
> It means cbc, hmac should have smaller(nearly 10 times less) priority
> than their authenc implementation otherwise request will not offload
> to driver because sw authenc priority is (aes * 10 + hmac).
I think you should look at it the other way. The priority of
your hardware authenc should be greater than (aes * 10 + hmac)
where aes is the priority of your hardware aes and hmac is the
priority of your hardware hmac.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] padata: free correct variable
From: Herbert Xu @ 2017-04-10 11:25 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: linux-crypto, steffen.klassert, linux-kernel, Jason, stable
In-Reply-To: <20170407003330.6552-1-Jason@zx2c4.com>
Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> The author meant to free the variable that was just allocated, instead
> of the one that failed to be allocated, but made a simple typo. This
> patch rectifies that.
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Cc: stable@vger.kernel.org
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: lz4: fixed decompress function to return error code
From: Herbert Xu @ 2017-04-10 11:23 UTC (permalink / raw)
To: Myungho Jung; +Cc: linux-crypto
In-Reply-To: <1491784462-31424-1-git-send-email-mhjungk@gmail.com>
On Sun, Apr 09, 2017 at 05:34:22PM -0700, Myungho Jung wrote:
> Decompress function in LZ4 library is supposed to return an error code or
> negative result. But, it returns -1 when any error is detected. Return
> error code when the library returns negative value.
>
> Signed-off-by: Myungho Jung <mhjungk@gmail.com>
Patch applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox