* Re: [PATCH v6 0/4] Broadcom SBA RAID support
From: Anup Patel @ 2017-03-29 6:05 UTC (permalink / raw)
To: Vinod Koul
Cc: Jassi Brar, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Dan Williams, Ray Jui, Scott Branden, Jon Mason,
Rob Rice, BCM Kernel Feedback, dmaengine, Device Tree,
Linux ARM Kernel, Linux Kernel, linux-crypto, linux-raid
In-Reply-To: <20170321091829.GZ2843@localhost>
On Tue, Mar 21, 2017 at 2:48 PM, Vinod Koul <vinod.koul@intel.com> wrote:
> On Tue, Mar 21, 2017 at 02:17:21PM +0530, Anup Patel wrote:
>> On Tue, Mar 21, 2017 at 2:00 PM, Vinod Koul <vinod.koul@intel.com> wrote:
>> > On Mon, Mar 06, 2017 at 03:13:24PM +0530, Anup Patel wrote:
>> >> The Broadcom SBA RAID is a stream-based device which provides
>> >> RAID5/6 offload.
>> >>
>> >> It requires a SoC specific ring manager (such as Broadcom FlexRM
>> >> ring manager) to provide ring-based programming interface. Due to
>> >> this, the Broadcom SBA RAID driver (mailbox client) implements
>> >> DMA device having one DMA channel using a set of mailbox channels
>> >> provided by Broadcom SoC specific ring manager driver (mailbox
>> >> controller).
>> >>
>> >> The Broadcom SBA RAID hardware requires PQ disk position instead
>> >> of PQ disk coefficient. To address this, we have added raid_gflog
>> >> table which will help driver to convert PQ disk coefficient to PQ
>> >> disk position.
>> >>
>> >> This patchset is based on Linux-4.11-rc1 and depends on patchset
>> >> "[PATCH v5 0/2] Broadcom FlexRM ring manager support"
>> >
>> > Okay I applied and was about to push when I noticed this :(
>> >
>> > So what is the status of this..?
>>
>> PATCH2 is Acked but PATCH1 is under-review. Currently, its
>> v6 of that patchset.
>>
>> The only dependency on that patchset is the changes in
>> brcm-message.h which are required by this BCM-SBA-RAID
>> driver.
>>
>> @Jassi,
>> Can you please have a look at PATCH v6?
>
> And I would need an immutable branch/tag once merged. I am going to keep
> this series pending till then.
The Broadcom FlexRM patchset is pickedup by Jassi and
can be found in mailbox-for-next branch of
git://git.linaro.org/landing-teams/working/fujitsu/integration
Both patchset (Broadcom FlexRM patchset and this one) are
also available in sba-raid-v7 branch of
https://github.com/Broadcom/arm64-linux.git
Regards,
Anup
^ permalink raw reply
* [PATCH 1/7] Documentation/bindings: Document the SafeXel cryptographic engine driver
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
The Inside Secure Safexcel cryptographic engine is found on some Marvell
SoCs (7k/8k). Document the bindings used by its driver.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
.../bindings/crypto/inside-secure-safexcel.txt | 27 ++++++++++++++++++++++
1 file changed, 27 insertions(+)
create mode 100644 Documentation/devicetree/bindings/crypto/inside-secure-safexcel.txt
diff --git a/Documentation/devicetree/bindings/crypto/inside-secure-safexcel.txt b/Documentation/devicetree/bindings/crypto/inside-secure-safexcel.txt
new file mode 100644
index 000000000000..ff56b9384fcc
--- /dev/null
+++ b/Documentation/devicetree/bindings/crypto/inside-secure-safexcel.txt
@@ -0,0 +1,27 @@
+Inside Secure SafeXcel cryptographic engine
+
+Required properties:
+- compatible: Should be "inside-secure,safexcel-eip197".
+- reg: Base physical address of the engine and length of memory mapped region.
+- interrupts: Interrupt numbers for the rings and engine.
+- interrupt-names: Should be "ring0", "ring1", "ring2", "ring3", "eip", "mem".
+
+Optional properties:
+- clocks: Reference to the crypto engine clock.
+
+Example:
+
+ crypto: crypto@800000 {
+ compatible = "inside-secure,safexcel-eip197";
+ reg = <0x800000 0x200000>;
+ interrupts = <GIC_SPI 34 (IRQ_TYPE_EDGE_RISING | IRQ_TYPE_LEVEL_HIGH)>,
+ <GIC_SPI 54 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 56 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 57 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 58 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "mem", "ring0", "ring1", "ring2", "ring3",
+ "eip";
+ clocks = <&cpm_syscon0 1 26>;
+ status = "disabled";
+ };
--
2.11.0
^ permalink raw reply related
* [PATCH 5/7] arm64: marvell: dts: enable the crypto engine on the Armada 7040 DB
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
Enable the cryptographic engine available in the CP110 master on the
Armada 7040 DB.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
arch/arm64/boot/dts/marvell/armada-7040-db.dts | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/boot/dts/marvell/armada-7040-db.dts b/arch/arm64/boot/dts/marvell/armada-7040-db.dts
index bb0e1e8e7138..7e9a8a4e89a4 100644
--- a/arch/arm64/boot/dts/marvell/armada-7040-db.dts
+++ b/arch/arm64/boot/dts/marvell/armada-7040-db.dts
@@ -171,3 +171,7 @@
phy = <&phy1>;
phy-mode = "rgmii-id";
};
+
+&cpm_crypto {
+ status = "okay";
+};
--
2.11.0
^ permalink raw reply related
* [PATCH 0/7] arm64: marvell: add cryptographic engine support for 7k/8k
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
Hi all,
This series adds support for the Inside Secure SafeXcel EIP197
cryptographic engine which can be found on Marvell Armada 7k and 8k
boards. A new cryptographic engine driver is added, as well as the
relevant device tree definition for the Armada 7040 DB and 8040 DB
boards.
This driver needs two firmwares to work correctly. These firmware are
usually used for more advanced operations than the ones supported (as of
now), but we still need them to pass the data to the internal
cryptographic engine.
This series was tested in various ways on both the Armada 7040 DB and
the Armada 8040 DB: using the crypto framework self tests, using tcrypt
and while performing various transfers with iperf on top of IPsec.
This series is based on top of v4.11-rc1, and is available on a branch
(which also contains the PPv2 support for 7k/8k, to ease the process of
testing IPsec): https://github.com/atenart/linux v4.11-rc1/7k8k-crypto
I can rebase if needed.
Thanks,
Antoine
Antoine Tenart (7):
Documentation/bindings: Document the SafeXel cryptographic engine
driver
crypto: inside-secure: add SafeXcel EIP197 crypto engine driver
MAINTAINERS: add a maintainer for the Inside Secure crypto driver
arm64: marvell: dts: add crypto engine description for 7k/8k
arm64: marvell: dts: enable the crypto engine on the Armada 7040 DB
arm64: marvell: dts: enable the crypto engine on the Armada 8040 DB
arm64: defconfig: enable the Safexcel crypto engine as a module
.../bindings/crypto/inside-secure-safexcel.txt | 27 +
MAINTAINERS | 6 +
arch/arm64/boot/dts/marvell/armada-7040-db.dts | 4 +
arch/arm64/boot/dts/marvell/armada-8040-db.dts | 4 +
.../boot/dts/marvell/armada-cp110-master.dtsi | 15 +
.../arm64/boot/dts/marvell/armada-cp110-slave.dtsi | 15 +
arch/arm64/configs/defconfig | 1 +
drivers/crypto/Kconfig | 17 +
drivers/crypto/Makefile | 1 +
drivers/crypto/inside-secure/Makefile | 2 +
drivers/crypto/inside-secure/safexcel.c | 939 ++++++++++++++++++
drivers/crypto/inside-secure/safexcel.h | 581 +++++++++++
drivers/crypto/inside-secure/safexcel_cipher.c | 544 ++++++++++
drivers/crypto/inside-secure/safexcel_hash.c | 1045 ++++++++++++++++++++
drivers/crypto/inside-secure/safexcel_ring.c | 173 ++++
15 files changed, 3374 insertions(+)
create mode 100644 Documentation/devicetree/bindings/crypto/inside-secure-safexcel.txt
create mode 100644 drivers/crypto/inside-secure/Makefile
create mode 100644 drivers/crypto/inside-secure/safexcel.c
create mode 100644 drivers/crypto/inside-secure/safexcel.h
create mode 100644 drivers/crypto/inside-secure/safexcel_cipher.c
create mode 100644 drivers/crypto/inside-secure/safexcel_hash.c
create mode 100644 drivers/crypto/inside-secure/safexcel_ring.c
--
2.11.0
^ permalink raw reply
* [PATCH 4/7] arm64: marvell: dts: add crypto engine description for 7k/8k
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
Add the description of the crypto engine hardware block for the Marvell
Armada 7k and Armada 8k processors; for both the CP110 slave and master.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
arch/arm64/boot/dts/marvell/armada-cp110-master.dtsi | 15 +++++++++++++++
arch/arm64/boot/dts/marvell/armada-cp110-slave.dtsi | 15 +++++++++++++++
2 files changed, 30 insertions(+)
diff --git a/arch/arm64/boot/dts/marvell/armada-cp110-master.dtsi b/arch/arm64/boot/dts/marvell/armada-cp110-master.dtsi
index 9a2ce2ae49cd..7530a1b541c5 100644
--- a/arch/arm64/boot/dts/marvell/armada-cp110-master.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-cp110-master.dtsi
@@ -210,6 +210,21 @@
clocks = <&cpm_syscon0 1 25>;
status = "okay";
};
+
+ cpm_crypto: crypto@800000 {
+ compatible = "inside-secure,safexcel-eip197";
+ reg = <0x800000 0x200000>;
+ interrupts = <GIC_SPI 34 (IRQ_TYPE_EDGE_RISING | IRQ_TYPE_LEVEL_HIGH)>,
+ <GIC_SPI 54 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 56 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 57 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 58 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "mem", "ring0", "ring1",
+ "ring2", "ring3", "eip";
+ clocks = <&cpm_syscon0 1 26>;
+ status = "disabled";
+ };
};
cpm_pcie0: pcie@f2600000 {
diff --git a/arch/arm64/boot/dts/marvell/armada-cp110-slave.dtsi b/arch/arm64/boot/dts/marvell/armada-cp110-slave.dtsi
index c9dfa244ebb6..6110d39087a6 100644
--- a/arch/arm64/boot/dts/marvell/armada-cp110-slave.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-cp110-slave.dtsi
@@ -210,6 +210,21 @@
clocks = <&cps_syscon0 1 25>;
status = "okay";
};
+
+ cps_crypto: crypto@800000 {
+ compatible = "inside-secure,safexcel-eip197";
+ reg = <0x800000 0x200000>;
+ interrupts = <GIC_SPI 34 (IRQ_TYPE_EDGE_RISING | IRQ_TYPE_LEVEL_HIGH)>,
+ <GIC_SPI 278 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 279 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 280 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 281 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 282 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "mem", "ring0", "ring1",
+ "ring2", "ring3", "eip";
+ clocks = <&cps_syscon0 1 26>;
+ status = "disabled";
+ };
};
cps_pcie0: pcie@f4600000 {
--
2.11.0
^ permalink raw reply related
* [PATCH 2/7] crypto: inside-secure: add SafeXcel EIP197 crypto engine driver
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
Add support for Inside Secure SafeXcel EIP197 cryptographic engine,
which can be found on Marvell Armada 7k and 8k boards. This driver
currently implements: ecb(aes), cbc(aes), sha1, sha224, sha256 and
hmac(sah1) algorithms.
Two firmwares are needed for this engine to work. Their are mostly used
for more advanced operations than the ones supported (as of now), but we
still need them to pass the data to the internal cryptographic engine.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
drivers/crypto/Kconfig | 17 +
drivers/crypto/Makefile | 1 +
drivers/crypto/inside-secure/Makefile | 2 +
drivers/crypto/inside-secure/safexcel.c | 939 +++++++++++++++++++++
drivers/crypto/inside-secure/safexcel.h | 581 +++++++++++++
drivers/crypto/inside-secure/safexcel_cipher.c | 544 ++++++++++++
drivers/crypto/inside-secure/safexcel_hash.c | 1045 ++++++++++++++++++++++++
drivers/crypto/inside-secure/safexcel_ring.c | 173 ++++
8 files changed, 3302 insertions(+)
create mode 100644 drivers/crypto/inside-secure/Makefile
create mode 100644 drivers/crypto/inside-secure/safexcel.c
create mode 100644 drivers/crypto/inside-secure/safexcel.h
create mode 100644 drivers/crypto/inside-secure/safexcel_cipher.c
create mode 100644 drivers/crypto/inside-secure/safexcel_hash.c
create mode 100644 drivers/crypto/inside-secure/safexcel_ring.c
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 473d31288ad8..4ff39d3132dc 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -619,4 +619,21 @@ config CRYPTO_DEV_BCM_SPU
Secure Processing Unit (SPU). The SPU driver registers ablkcipher,
ahash, and aead algorithms with the kernel cryptographic API.
+config CRYPTO_DEV_SAFEXCEL
+ tristate "Inside Secure's SafeXcel cryptographic engine driver"
+ depends on HAS_DMA && OF
+ depends on (ARM64 && ARCH_MVEBU) || COMPILE_TEST
+ select CRYPTO_AES
+ select CRYPTO_BLKCIPHER
+ select CRYPTO_HASH
+ select CRYPTO_HMAC
+ select CRYPTO_SHA1
+ select CRYPTO_SHA256
+ select CRYPTO_SHA512
+ help
+ This driver interfaces with the SafeXcel EIP-197 cryptographic engine
+ designed by Inside Secure. Select this if you want to use CBC/ECB
+ chain mode, AES cipher mode and SHA1/SHA224/SHA256/SHA512 hash
+ algorithms.
+
endif # CRYPTO_HW
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index 739609471169..7ed3e7940f76 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -36,3 +36,4 @@ obj-$(CONFIG_CRYPTO_DEV_UX500) += ux500/
obj-$(CONFIG_CRYPTO_DEV_VIRTIO) += virtio/
obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/
+obj-$(CONFIG_CRYPTO_DEV_SAFEXCEL) += inside-secure/
diff --git a/drivers/crypto/inside-secure/Makefile b/drivers/crypto/inside-secure/Makefile
new file mode 100644
index 000000000000..302f07dde98c
--- /dev/null
+++ b/drivers/crypto/inside-secure/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_CRYPTO_DEV_SAFEXCEL) += crypto_safexcel.o
+crypto_safexcel-objs := safexcel.o safexcel_ring.o safexcel_cipher.o safexcel_hash.o
diff --git a/drivers/crypto/inside-secure/safexcel.c b/drivers/crypto/inside-secure/safexcel.c
new file mode 100644
index 000000000000..00f3f2c85d05
--- /dev/null
+++ b/drivers/crypto/inside-secure/safexcel.c
@@ -0,0 +1,939 @@
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Antoine Tenart <antoine.tenart@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/clk.h>
+#include <linux/device.h>
+#include <linux/dmapool.h>
+#include <linux/firmware.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/of_irq.h>
+#include <linux/platform_device.h>
+#include <linux/workqueue.h>
+
+#include "safexcel.h"
+
+static u32 max_rings = EIP197_MAX_RINGS;
+module_param(max_rings, uint, 0644);
+MODULE_PARM_DESC(max_rings, "Maximum number of rings to use.");
+
+static void eip197_trc_cache_init(struct safexcel_crypto_priv *priv)
+{
+ u32 val, htable_offset;
+ int i;
+
+ /* Enable the record cache memory access */
+ val = readl(priv->base + EIP197_CS_RAM_CTRL);
+ val &= ~EIP197_TRC_ENABLE_MASK;
+ val |= EIP197_TRC_ENABLE_0;
+ writel(val, priv->base + EIP197_CS_RAM_CTRL);
+
+ /* Clear all ECC errors */
+ writel(0, priv->base + EIP197_TRC_ECCCTRL);
+
+ /*
+ * Make sure the cache memory is accessible by taking record cache into
+ * reset.
+ */
+ val = readl(priv->base + EIP197_TRC_PARAMS);
+ val |= EIP197_TRC_PARAMS_SW_RESET;
+ val &= ~EIP197_TRC_PARAMS_DATA_ACCESS;
+ writel(val, priv->base + EIP197_TRC_PARAMS);
+
+ /* Clear all records */
+ for (i = 0; i < EIP197_CS_RC_MAX; i++) {
+ u32 val, offset = EIP197_CLASSIFICATION_RAMS + i * EIP197_CS_RC_SIZE;
+
+ writel(EIP197_CS_RC_NEXT(EIP197_RC_NULL) |
+ EIP197_CS_RC_PREV(EIP197_RC_NULL),
+ priv->base + offset);
+
+ val = EIP197_CS_RC_NEXT(i+1) | EIP197_CS_RC_PREV(i-1);
+ if (i == 0)
+ val |= EIP197_CS_RC_PREV(EIP197_RC_NULL);
+ else if (i == EIP197_CS_RC_MAX - 1)
+ val |= EIP197_CS_RC_NEXT(EIP197_RC_NULL);
+ writel(val, priv->base + offset + sizeof(u32));
+ }
+
+ /* Clear the hash table entries */
+ htable_offset = EIP197_CS_RC_MAX * EIP197_CS_RC_SIZE;
+ for (i = 0; i < 64; i++)
+ writel(GENMASK(29, 0),
+ priv->base + EIP197_CLASSIFICATION_RAMS + htable_offset + i * sizeof(u32));
+
+ /* Disable the record cache memory access */
+ val = readl(priv->base + EIP197_CS_RAM_CTRL);
+ val &= ~EIP197_TRC_ENABLE_MASK;
+ writel(val, priv->base + EIP197_CS_RAM_CTRL);
+
+ /* Write head and tail pointers of the record free chain */
+ val = EIP197_TRC_FREECHAIN_HEAD_PTR(0) |
+ EIP197_TRC_FREECHAIN_TAIL_PTR(EIP197_CS_RC_MAX - 1);
+ writel(val, priv->base + EIP197_TRC_FREECHAIN);
+
+ /* Configure the record cache #1 */
+ val = EIP197_TRC_PARAMS2_RC_SZ_SMALL(EIP197_CS_TRC_REC_WC) |
+ EIP197_TRC_PARAMS2_HTABLE_PTR(EIP197_CS_RC_MAX);
+ writel(val, priv->base + EIP197_TRC_PARAMS2);
+
+ /* Configure the record cache #2 */
+ val = EIP197_TRC_PARAMS_RC_SZ_LARGE(EIP197_CS_TRC_LG_REC_WC) |
+ EIP197_TRC_PARAMS_BLK_TIMER_SPEED(1) |
+ EIP197_TRC_PARAMS_HTABLE_SZ(2);
+ writel(val, priv->base + EIP197_TRC_PARAMS);
+}
+
+static void eip197_write_firmware(struct safexcel_crypto_priv *priv,
+ const struct firmware *fw, u32 ctrl,
+ u32 prog_en)
+{
+ const u32 *data = (const u32 *)fw->data;
+ u32 val;
+ int i;
+
+ /* Reset the engine to make its program memory accessible */
+ writel(EIP197_PE_ICE_x_CTRL_SW_RESET |
+ EIP197_PE_ICE_x_CTRL_CLR_ECC_CORR |
+ EIP197_PE_ICE_x_CTRL_CLR_ECC_NON_CORR,
+ priv->base + ctrl);
+
+ /* Enable access to the program memory */
+ writel(prog_en, priv->base + EIP197_PE_ICE_RAM_CTRL);
+
+ /* Write the firmware */
+ for (i = 0; i < fw->size / sizeof(u32); i++)
+ writel(be32_to_cpu(data[i]),
+ priv->base + EIP197_CLASSIFICATION_RAMS + i * sizeof(u32));
+
+ /* Disable access to the program memory */
+ writel(0, priv->base + EIP197_PE_ICE_RAM_CTRL);
+
+ /* Release engine from reset */
+ val = readl(priv->base + ctrl);
+ val &= ~EIP197_PE_ICE_x_CTRL_SW_RESET;
+ writel(val, priv->base + ctrl);
+}
+
+static int eip197_load_firmwares(struct safexcel_crypto_priv *priv)
+{
+ const char *fw_name[] = {"ifpp.bin", "ipue.bin"};
+ const struct firmware *fw[FW_NB];
+ int i, ret = 0;
+ u32 val;
+
+ for (i = 0; i < FW_NB; i++) {
+ ret = request_firmware(&fw[i], fw_name[i], priv->dev);
+ if (ret) {
+ dev_err(priv->dev,
+ "Failed to request firmware %s (%d)\n",
+ fw_name[i], ret);
+ goto release_fw;
+ }
+ }
+
+ /* Clear the scratchpad memory */
+ val = readl(priv->base + EIP197_PE_ICE_SCRATCH_CTRL);
+ val |= EIP197_PE_ICE_SCRATCH_CTRL_CHANGE_TIMER |
+ EIP197_PE_ICE_SCRATCH_CTRL_TIMER_EN |
+ EIP197_PE_ICE_SCRATCH_CTRL_SCRATCH_ACCESS |
+ EIP197_PE_ICE_SCRATCH_CTRL_CHANGE_ACCESS;
+ writel(val, priv->base + EIP197_PE_ICE_SCRATCH_CTRL);
+
+ memset(priv->base + EIP197_PE_ICE_SCRATCH_RAM, 0,
+ EIP197_NUM_OF_SCRATCH_BLOCKS * sizeof(u32));
+
+ eip197_write_firmware(priv, fw[FW_IFPP], EIP197_PE_ICE_FPP_CTRL,
+ EIP197_PE_ICE_RAM_CTRL_FPP_PROG_EN);
+
+ eip197_write_firmware(priv, fw[FW_IPUE], EIP197_PE_ICE_PUE_CTRL,
+ EIP197_PE_ICE_RAM_CTRL_PUE_PROG_EN);
+
+release_fw:
+ for (i = 0; i < FW_NB; i++)
+ release_firmware(fw[i]);
+
+ return ret;
+}
+
+static int safexcel_hw_setup_cdesc_rings(struct safexcel_crypto_priv *priv)
+{
+ u32 hdw, cd_size_rnd, val;
+ int i;
+
+ hdw = readl(priv->base + EIP197_HIA_OPTIONS);
+ hdw &= GENMASK(27, 25);
+ hdw >>= 25;
+
+ cd_size_rnd = (priv->config.cd_size + (BIT(hdw) - 1)) >> hdw;
+
+ for (i = 0; i < priv->config.rings; i++) {
+ /* ring base address */
+ writel(lower_32_bits(priv->ring[i].cdr.base_dma),
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_RING_BASE_ADDR_LO);
+ writel(upper_32_bits(priv->ring[i].cdr.base_dma),
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_RING_BASE_ADDR_HI);
+
+ writel(EIP197_xDR_DESC_MODE_64BIT | (priv->config.cd_offset << 16) |
+ priv->config.cd_size,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_DESC_SIZE);
+ writel(((EIP197_FETCH_COUNT * (cd_size_rnd << hdw)) << 16) |
+ (EIP197_FETCH_COUNT * priv->config.cd_offset),
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_CFG);
+
+ /* Configure DMA tx control */
+ val = EIP197_HIA_xDR_CFG_WR_CACHE(WR_CACHE_3BITS);
+ val |= EIP197_HIA_xDR_CFG_RD_CACHE(RD_CACHE_3BITS);
+ writel(val,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_DMA_CFG);
+
+ /* clear any pending interrupt */
+ writel(GENMASK(5, 0),
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_STAT);
+ }
+
+ return 0;
+}
+
+static int safexcel_hw_setup_rdesc_rings(struct safexcel_crypto_priv *priv)
+{
+ u32 hdw, rd_size_rnd, val;
+ int i;
+
+ hdw = readl(priv->base + EIP197_HIA_OPTIONS);
+ hdw &= GENMASK(27, 25);
+ hdw >>= 25;
+
+ rd_size_rnd = (priv->config.rd_size + (BIT(hdw) - 1)) >> hdw;
+
+ for (i = 0; i < priv->config.rings; i++) {
+ /* ring base address */
+ writel(lower_32_bits(priv->ring[i].rdr.base_dma),
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_RING_BASE_ADDR_LO);
+ writel(upper_32_bits(priv->ring[i].rdr.base_dma),
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_RING_BASE_ADDR_HI);
+
+ writel(EIP197_xDR_DESC_MODE_64BIT | (priv->config.rd_offset << 16) |
+ priv->config.rd_size,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_DESC_SIZE);
+
+ writel(((EIP197_FETCH_COUNT * (rd_size_rnd << hdw)) << 16) |
+ (EIP197_FETCH_COUNT * priv->config.rd_offset),
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_CFG);
+
+ /* Configure DMA tx control */
+ val = EIP197_HIA_xDR_CFG_WR_CACHE(WR_CACHE_3BITS);
+ val |= EIP197_HIA_xDR_CFG_RD_CACHE(RD_CACHE_3BITS);
+ val |= EIP197_HIA_xDR_WR_RES_BUF | EIP197_HIA_xDR_WR_CTRL_BUG;
+ writel(val,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_DMA_CFG);
+
+ /* clear any pending interrupt */
+ writel(GENMASK(7, 0),
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_STAT);
+
+ /* enable ring interrupt */
+ val = readl(priv->base + EIP197_HIA_AIC_R_ENABLE_CTRL(i));
+ val |= EIP197_RDR_IRQ(i);
+ writel(val, priv->base + EIP197_HIA_AIC_R_ENABLE_CTRL(i));
+ }
+
+ return 0;
+}
+
+static int safexcel_hw_init(struct safexcel_crypto_priv *priv)
+{
+ u32 version, val;
+ int i, ret;
+
+ /* Determine endianess and configure byte swap */
+ version = readl(priv->base + EIP197_HIA_VERSION);
+ val = readl(priv->base + EIP197_HIA_MST_CTRL);
+
+ if ((version & 0xffff) == EIP197_HIA_VERSION_BE)
+ val |= EIP197_MST_CTRL_BYTE_SWAP;
+ else if (((version >> 16) & 0xffff) == EIP197_HIA_VERSION_LE)
+ val |= (EIP197_MST_CTRL_NO_BYTE_SWAP >> 24);
+
+ writel(val, priv->base + EIP197_HIA_MST_CTRL);
+
+
+ /* configure wr/rd cache values */
+ writel(EIP197_MST_CTRL_RD_CACHE(RD_CACHE_4BITS) |
+ EIP197_MST_CTRL_WD_CACHE(WR_CACHE_4BITS),
+ priv->base + EIP197_MST_CTRL);
+
+ /*
+ * Interrupts reset
+ */
+
+ /* Disable all global interrupts */
+ writel(0, priv->base + EIP197_HIA_AIC_G_ENABLE_CTRL);
+
+ /* Clear any pending interrupt */
+ writel(GENMASK(31, 0), priv->base + EIP197_HIA_AIC_G_ACK);
+
+ /*
+ * Data Fetch Engine configuration
+ */
+
+ /* Reset all DFE threads */
+ writel(EIP197_DxE_THR_CTRL_RESET_PE,
+ priv->base + EIP197_HIA_DFE_THR_CTRL);
+
+ /* Reset HIA input interface arbiter */
+ writel(EIP197_HIA_RA_PE_CTRL_RESET,
+ priv->base + EIP197_HIA_RA_PE_CTRL);
+
+ /* DMA transfer size to use */
+ val = EIP197_HIA_DFE_CFG_DIS_DEBUG;
+ val |= EIP197_HIA_DxE_CFG_MIN_DATA_SIZE(5) | EIP197_HIA_DxE_CFG_MAX_DATA_SIZE(9);
+ val |= EIP197_HIA_DxE_CFG_MIN_CTRL_SIZE(5) | EIP197_HIA_DxE_CFG_MAX_CTRL_SIZE(7);
+ val |= EIP197_HIA_DxE_CFG_DATA_CACHE_CTRL(RD_CACHE_3BITS);
+ val |= EIP197_HIA_DxE_CFG_CTRL_CACHE_CTRL(RD_CACHE_3BITS);
+ writel(val, priv->base + EIP197_HIA_DFE_CFG);
+
+ /* Leave the DFE threads reset state */
+ writel(0, priv->base + EIP197_HIA_DFE_THR_CTRL);
+
+ /* Configure the procesing engine thresholds */
+ writel(EIP197_PE_IN_xBUF_THRES_MIN(5) | EIP197_PE_IN_xBUF_THRES_MAX(9),
+ priv->base + EIP197_PE_IN_DBUF_THRES);
+ writel(EIP197_PE_IN_xBUF_THRES_MIN(5) | EIP197_PE_IN_xBUF_THRES_MAX(7),
+ priv->base + EIP197_PE_IN_TBUF_THRES);
+
+ /* enable HIA input interface arbiter and rings */
+ writel(EIP197_HIA_RA_PE_CTRL_EN | GENMASK(priv->config.rings - 1, 0),
+ priv->base + EIP197_HIA_RA_PE_CTRL);
+
+ /*
+ * Data Store Engine configuration
+ */
+
+ /* Reset all DSE threads */
+ writel(EIP197_DxE_THR_CTRL_RESET_PE,
+ priv->base + EIP197_HIA_DSE_THR_CTRL);
+
+ /* Wait for all DSE threads to complete */
+ while ((readl(priv->base + EIP197_HIA_DSE_THR_STAT) &
+ GENMASK(15, 12)) != GENMASK(15, 12))
+ ;
+
+ /* DMA transfer size to use */
+ val = EIP197_HIA_DSE_CFG_DIS_DEBUG;
+ val |= EIP197_HIA_DxE_CFG_MIN_DATA_SIZE(7) | EIP197_HIA_DxE_CFG_MAX_DATA_SIZE(8);
+ val |= EIP197_HIA_DxE_CFG_DATA_CACHE_CTRL(RD_CACHE_3BITS);
+ writel(val, priv->base + EIP197_HIA_DSE_CFG);
+
+ /* Leave the DSE threads reset state */
+ writel(0, priv->base + EIP197_HIA_DSE_THR_CTRL);
+
+ /* Configure the procesing engine thresholds */
+ writel(EIP197_PE_OUT_DBUF_THRES_MIN(7) | EIP197_PE_OUT_DBUF_THRES_MAX(8),
+ priv->base + EIP197_PE_OUT_DBUF_THRES);
+
+ /*
+ * Processing Engine configuration
+ */
+
+ /* H/W capabilities selection */
+ val = EIP197_FUNCTION_RSVD;
+ val |= EIP197_PROTOCOL_ENCRYPT_ONLY | EIP197_PROTOCOL_HASH_ONLY;
+ val |= EIP197_ALG_AES_ECB | EIP197_ALG_AES_CBC;
+ val |= EIP197_ALG_SHA1 | EIP197_ALG_HMAC_SHA1;
+ val |= EIP197_ALG_SHA2;
+ writel(val, priv->base + EIP197_PE_EIP96_FUNCTION_EN);
+
+ /*
+ * Command Descriptor Rings prepare
+ */
+ for (i = 0; i < priv->config.rings; i++) {
+ /* Clear interrupts for this ring */
+ writel(GENMASK(31, 0),
+ priv->base + EIP197_HIA_AIC_R_ENABLE_CLR(i));
+
+ /* disable external triggering */
+ writel(0, priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_CFG);
+
+ /* Clear the pending prepared counter */
+ writel(EIP197_xDR_PREP_CLR_COUNT,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_PREP_COUNT);
+
+ /* Clear the pending processed counter */
+ writel(EIP197_xDR_PROC_CLR_COUNT,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_PROC_COUNT);
+
+ writel(0,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_PREP_PNTR);
+ writel(0,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_PROC_PNTR);
+
+ writel((EIP197_DEFAULT_RING_SIZE * priv->config.cd_offset) << 2,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_RING_SIZE);
+ }
+
+ /*
+ * Result Descriptor Ring prepare
+ */
+
+ for (i = 0; i < priv->config.rings; i++) {
+ /* disable external triggering*/
+ writel(0, priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_CFG);
+
+ /* Clear the pending prepared counter */
+ writel(EIP197_xDR_PREP_CLR_COUNT,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_PREP_COUNT);
+
+ /* Clear the pending processed counter */
+ writel(EIP197_xDR_PROC_CLR_COUNT,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_PROC_COUNT);
+
+ writel(0,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_PREP_PNTR);
+ writel(0,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_PROC_PNTR);
+
+ /* ring size */
+ writel((EIP197_DEFAULT_RING_SIZE * priv->config.rd_offset) << 2,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_RING_SIZE);
+ }
+
+ /* Enable command descriptor rings */
+ writel(EIP197_DxE_THR_CTRL_EN | GENMASK(priv->config.rings - 1, 0),
+ priv->base + EIP197_HIA_DFE_THR_CTRL);
+
+ /* Enable result descriptor rings */
+ writel(EIP197_DxE_THR_CTRL_EN | GENMASK(priv->config.rings - 1, 0),
+ priv->base + EIP197_HIA_DSE_THR_CTRL);
+
+ /* Clear any HIA interrupt */
+ writel(GENMASK(30, 20), priv->base + EIP197_HIA_AIC_G_ACK);
+
+ eip197_trc_cache_init(priv);
+
+ ret = eip197_load_firmwares(priv);
+ if (ret)
+ return ret;
+
+ safexcel_hw_setup_cdesc_rings(priv);
+ safexcel_hw_setup_rdesc_rings(priv);
+
+ return 0;
+}
+
+void safexcel_dequeue(struct safexcel_crypto_priv *priv)
+{
+ struct crypto_async_request *req, *backlog;
+ struct safexcel_context *ctx;
+ struct safexcel_request *request;
+ int i, ret, n = 0, nreq[EIP197_MAX_RINGS] = {0};
+ int cdesc[EIP197_MAX_RINGS] = {0}, rdesc[EIP197_MAX_RINGS] = {0};
+ int commands, results;
+
+ do {
+ spin_lock_bh(&priv->lock);
+ req = crypto_dequeue_request(&priv->queue);
+ backlog = crypto_get_backlog(&priv->queue);
+ spin_unlock_bh(&priv->lock);
+
+ if (!req)
+ goto finalize;
+
+ request = kzalloc(sizeof(*request), EIP197_GFP_FLAGS(*req));
+ if (!request)
+ goto requeue;
+
+ ctx = crypto_tfm_ctx(req->tfm);
+ ret = ctx->send(req, ctx->ring, request, &commands, &results);
+ if (ret) {
+ kfree(request);
+requeue:
+ spin_lock_bh(&priv->lock);
+ crypto_enqueue_request(&priv->queue, req);
+ spin_unlock_bh(&priv->lock);
+
+ priv->need_dequeue = true;
+ continue;
+ }
+
+ if (backlog)
+ backlog->complete(backlog, -EINPROGRESS);
+
+ spin_lock_bh(&priv->ring[ctx->ring].egress_lock);
+ list_add_tail(&request->list, &priv->ring[ctx->ring].list);
+ spin_unlock_bh(&priv->ring[ctx->ring].egress_lock);
+
+ cdesc[ctx->ring] += commands;
+ rdesc[ctx->ring] += results;
+
+ nreq[ctx->ring]++;
+ } while (n++ < EIP197_MAX_BATCH_SZ);
+
+finalize:
+ if (n == EIP197_MAX_BATCH_SZ)
+ priv->need_dequeue = true;
+ else if (!n)
+ return;
+
+ for (i = 0; i < priv->config.rings; i++) {
+ if (!nreq[i])
+ continue;
+
+ spin_lock_bh(&priv->ring[i].lock);
+
+ /* Configure when we want an interrupt */
+ writel(EIP197_HIA_RDR_THRESH_PKT_MODE |
+ EIP197_HIA_RDR_THRESH_PROC_PKT(nreq[i]),
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_THRESH);
+
+ /* let the RDR know we have pending descriptors */
+ writel((rdesc[i] * priv->config.rd_offset) << 2,
+ priv->base + EIP197_HIA_RDR(i) + EIP197_HIA_xDR_PREP_COUNT);
+
+ /* let the CDR know we have pending descriptors */
+ writel((cdesc[i] * priv->config.cd_offset) << 2,
+ priv->base + EIP197_HIA_CDR(i) + EIP197_HIA_xDR_PREP_COUNT);
+
+ spin_unlock_bh(&priv->ring[i].lock);
+ }
+}
+
+void safexcel_free_context(struct safexcel_crypto_priv *priv,
+ struct crypto_async_request *req,
+ int result_sz)
+{
+ struct safexcel_context *ctx = crypto_tfm_ctx(req->tfm);
+
+ if (ctx->result_dma)
+ dma_unmap_single(priv->dev, ctx->result_dma, result_sz,
+ DMA_FROM_DEVICE);
+
+ if (ctx->cache_dma) {
+ dma_unmap_single(priv->dev, ctx->cache_dma, ctx->cache_sz,
+ DMA_TO_DEVICE);
+ ctx->cache_sz = 0;
+ }
+}
+
+void safexcel_complete(struct safexcel_crypto_priv *priv, int ring)
+{
+ struct safexcel_command_desc *cdesc;
+
+ /* Acknowledge the command descriptors */
+ do {
+ cdesc = safexcel_ring_next_rptr(priv, &priv->ring[ring].cdr);
+ if (IS_ERR(cdesc)) {
+ dev_err(priv->dev,
+ "Could not retrieve the command descriptor\n");
+ return;
+ }
+ } while (!cdesc->last_seg);
+}
+
+void safexcel_inv_complete(struct crypto_async_request *req, int error)
+{
+ struct safexcel_inv_result *result = req->data;
+
+ if (error == -EINPROGRESS)
+ return;
+
+ result->error = error;
+ complete(&result->completion);
+}
+
+int safexcel_invalidate_cache(struct crypto_async_request *async,
+ struct safexcel_context *ctx,
+ struct safexcel_crypto_priv *priv,
+ dma_addr_t ctxr_dma, int ring,
+ struct safexcel_request *request)
+{
+ struct safexcel_command_desc *cdesc;
+ struct safexcel_result_desc *rdesc;
+ phys_addr_t ctxr_phys;
+ int ret = 0;
+
+ ctxr_phys = dma_to_phys(priv->dev, ctxr_dma);
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+
+ /* prepare command descriptor */
+ cdesc = safexcel_add_cdesc(priv, ring, true, true, 0, 0, 0, ctxr_phys);
+ if (IS_ERR(cdesc)) {
+ ret = PTR_ERR(cdesc);
+ goto unlock;
+ }
+
+ cdesc->control_data.type = EIP197_TYPE_EXTENDED;
+ cdesc->control_data.options = 0;
+ cdesc->control_data.refresh = 0;
+ cdesc->control_data.control0 = CONTEXT_CONTROL_INV_TR;
+
+ /* prepare result descriptor */
+ rdesc = safexcel_add_rdesc(priv, ring, true, true, 0, 0);
+
+ if (IS_ERR(rdesc)) {
+ ret = PTR_ERR(rdesc);
+ goto cdesc_rollback;
+ }
+
+ request->req = async;
+ goto unlock;
+
+cdesc_rollback:
+ safexcel_ring_rollback_wptr(priv, &priv->ring[ring].cdr);
+
+unlock:
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+ return ret;
+}
+
+static inline void safexcel_handle_result_descriptor(struct safexcel_crypto_priv *priv,
+ int ring)
+{
+ struct safexcel_request *sreq;
+ struct safexcel_context *ctx;
+ int ret, i, nreq, ndesc = 0;
+ bool should_complete;
+
+ nreq = readl(priv->base + EIP197_HIA_RDR(ring) + EIP197_HIA_xDR_PROC_COUNT);
+ nreq >>= 24;
+ nreq &= GENMASK(6, 0);
+ if (!nreq)
+ return;
+
+ for (i = 0; i < nreq; i++) {
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+ sreq = list_first_entry(&priv->ring[ring].list,
+ struct safexcel_request, list);
+ list_del(&sreq->list);
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ ctx = crypto_tfm_ctx(sreq->req->tfm);
+ ndesc = ctx->handle_result(priv, ring, sreq->req,
+ &should_complete, &ret);
+ if (ndesc < 0) {
+ dev_err(priv->dev, "failed to handle result (%d)", ndesc);
+ return;
+ }
+
+ writel(EIP197_xDR_PROC_xD_PKT(1) |
+ EIP197_xDR_PROC_xD_COUNT(ndesc * priv->config.rd_offset),
+ priv->base + EIP197_HIA_RDR(ring) + EIP197_HIA_xDR_PROC_COUNT);
+
+ if (should_complete) {
+ local_bh_disable();
+ sreq->req->complete(sreq->req, ret);
+ local_bh_enable();
+ }
+
+ kfree(sreq);
+ }
+}
+
+static void safexcel_handle_result_work(struct work_struct *work)
+{
+ struct safexcel_work_data *data =
+ container_of(work, struct safexcel_work_data, work);
+ struct safexcel_crypto_priv *priv = data->priv;
+
+ safexcel_handle_result_descriptor(priv, data->ring);
+
+ if (priv->need_dequeue) {
+ priv->need_dequeue = false;
+ safexcel_dequeue(data->priv);
+ }
+}
+
+struct safexcel_ring_irq_data {
+ struct safexcel_crypto_priv *priv;
+ int ring;
+};
+
+static irqreturn_t safexcel_irq_ring(int irq, void *data)
+{
+ struct safexcel_ring_irq_data *irq_data = data;
+ struct safexcel_crypto_priv *priv = irq_data->priv;
+ int ring = irq_data->ring;
+ u32 status, stat;
+
+ status = readl(priv->base + EIP197_HIA_AIC_R_ENABLED_STAT(ring));
+ if (!status)
+ return IRQ_NONE;
+
+ /* RDR interrupts */
+ if (status & EIP197_RDR_IRQ(ring)) {
+ stat = readl(priv->base + EIP197_HIA_RDR(ring) + EIP197_HIA_xDR_STAT);
+
+ if (unlikely(stat & EIP197_xDR_ERR)) {
+ /*
+ * Fatal error, the RDR is unusable and must be
+ * reinitialized. This should not happen under
+ * normal circumstances.
+ */
+ dev_err(priv->dev, "RDR: fatal error.");
+ } else if (likely(stat & EIP197_xDR_THRESH)) {
+ queue_work(priv->ring[ring].workqueue, &priv->ring[ring].work_data.work);
+ }
+
+ /* ACK the interrupts */
+ writel(stat & 0xff,
+ priv->base + EIP197_HIA_RDR(ring) + EIP197_HIA_xDR_STAT);
+ }
+
+ /* ACK the interrupts */
+ writel(status, priv->base + EIP197_HIA_AIC_R_ACK(ring));
+
+ return IRQ_HANDLED;
+}
+
+static int safexcel_request_ring_irq(struct platform_device *pdev, const char *name,
+ irq_handler_t handler,
+ struct safexcel_ring_irq_data *ring_irq_priv)
+{
+ int ret, irq = platform_get_irq_byname(pdev, name);
+
+ if (irq < 0) {
+ dev_err(&pdev->dev, "unable to get IRQ '%s'\n", name);
+ return irq;
+ }
+
+ ret = devm_request_irq(&pdev->dev, irq, handler, 0,
+ dev_name(&pdev->dev), ring_irq_priv);
+ if (ret) {
+ dev_err(&pdev->dev, "unable to request IRQ %d\n", irq);
+ return ret;
+ }
+
+ return irq;
+}
+
+static struct safexcel_alg_template *safexcel_algs[] = {
+ &safexcel_alg_ecb_aes,
+ &safexcel_alg_cbc_aes,
+ &safexcel_alg_sha1,
+ &safexcel_alg_sha224,
+ &safexcel_alg_sha256,
+ &safexcel_alg_hmac_sha1,
+};
+
+static int safexcel_register_algorithms(struct safexcel_crypto_priv *priv)
+{
+ int i, j, ret = 0;
+
+ for (i = 0; i < ARRAY_SIZE(safexcel_algs); i++) {
+ safexcel_algs[i]->priv = priv;
+
+ if (safexcel_algs[i]->type == SAFEXCEL_ALG_TYPE_CIPHER)
+ ret = crypto_register_alg(&safexcel_algs[i]->alg.crypto);
+ else
+ ret = crypto_register_ahash(&safexcel_algs[i]->alg.ahash);
+
+ if (ret)
+ goto fail;
+ }
+
+ return 0;
+
+fail:
+ for (j = i; j < 0; j--) {
+ if (safexcel_algs[j]->type == SAFEXCEL_ALG_TYPE_CIPHER)
+ crypto_unregister_alg(&safexcel_algs[j]->alg.crypto);
+ else
+ crypto_unregister_ahash(&safexcel_algs[j]->alg.ahash);
+ }
+
+ return ret;
+}
+
+static void safexcel_unregister_algorithms(struct safexcel_crypto_priv *priv)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(safexcel_algs); i++) {
+ if (safexcel_algs[i]->type == SAFEXCEL_ALG_TYPE_CIPHER)
+ crypto_unregister_alg(&safexcel_algs[i]->alg.crypto);
+ else
+ crypto_unregister_ahash(&safexcel_algs[i]->alg.ahash);
+ }
+}
+
+static void safexcel_configure(struct safexcel_crypto_priv *priv)
+{
+ u32 val, mask;
+
+ val = readl(priv->base + EIP197_HIA_OPTIONS);
+ val = (val & GENMASK(27, 25)) >> 25;
+ mask = BIT(val) - 1;
+
+ val = readl(priv->base + EIP197_HIA_OPTIONS);
+ priv->config.rings = min_t(u32, val & GENMASK(3, 0), max_rings);
+
+ priv->config.cd_size = (sizeof(struct safexcel_command_desc) / sizeof(u32));
+ priv->config.cd_offset = (priv->config.cd_size + mask) & ~mask;
+
+ priv->config.rd_size = (sizeof(struct safexcel_result_desc) / sizeof(u32));
+ priv->config.rd_offset = (priv->config.rd_size + mask) & ~mask;
+}
+
+static int safexcel_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct resource *res;
+ struct safexcel_crypto_priv *priv;
+ int i, ret;
+
+ priv = devm_kzalloc(dev, sizeof(struct safexcel_crypto_priv),
+ GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->dev = dev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ priv->base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(priv->base)) {
+ dev_err(dev, "failed to get resource\n");
+ return PTR_ERR(priv->base);
+ }
+
+ priv->clk = of_clk_get(dev->of_node, 0);
+ if (!IS_ERR(priv->clk)) {
+ ret = clk_prepare_enable(priv->clk);
+ if (ret) {
+ dev_err(dev, "unable to enable clk (%d)\n", ret);
+ return ret;
+ }
+ } else {
+ /* The clock isn't mandatory */
+ if (PTR_ERR(priv->clk) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ }
+
+ priv->context_pool = dmam_pool_create("safexcel-context", dev,
+ sizeof(struct safexcel_context_record),
+ 1, 0);
+ if (!priv->context_pool) {
+ ret = -ENOMEM;
+ goto err_clk;
+ }
+
+ safexcel_configure(priv);
+
+ for (i = 0; i < priv->config.rings; i++) {
+ char irq_name[6] = {0}; /* "ringX\0" */
+ char wq_name[9] = {0}; /* "wq_ringX\0" */
+ int irq;
+ struct safexcel_ring_irq_data *ring_irq;
+
+ ret = safexcel_init_ring_descriptors(priv,
+ &priv->ring[i].cdr,
+ &priv->ring[i].rdr);
+ if (ret)
+ goto err_pool;
+
+ ring_irq = devm_kzalloc(dev, sizeof(struct safexcel_ring_irq_data),
+ GFP_KERNEL);
+ if (!ring_irq) {
+ ret = -ENOMEM;
+ goto err_pool;
+ }
+
+ ring_irq->priv = priv;
+ ring_irq->ring = i;
+
+ snprintf(irq_name, 6, "ring%d", i);
+ irq = safexcel_request_ring_irq(pdev, irq_name, safexcel_irq_ring,
+ ring_irq);
+
+ if (irq < 0)
+ goto err_pool;
+
+ priv->ring[i].work_data.priv = priv;
+ priv->ring[i].work_data.ring = i;
+ INIT_WORK(&priv->ring[i].work_data.work, safexcel_handle_result_work);
+
+ snprintf(wq_name, 9, "wq_ring%d", i);
+ priv->ring[i].workqueue = create_singlethread_workqueue(wq_name);
+ if (!priv->ring[i].workqueue) {
+ ret = -ENOMEM;
+ goto err_pool;
+ }
+
+ INIT_LIST_HEAD(&priv->ring[i].list);
+ spin_lock_init(&priv->ring[i].lock);
+ spin_lock_init(&priv->ring[i].egress_lock);
+ }
+
+ platform_set_drvdata(pdev, priv);
+ atomic_set(&priv->ring_used, 0);
+
+ spin_lock_init(&priv->lock);
+ crypto_init_queue(&priv->queue, EIP197_DEFAULT_RING_SIZE);
+
+ ret = safexcel_hw_init(priv);
+ if (ret) {
+ dev_err(dev, "EIP h/w init failed (%d)\n", ret);
+ goto err_pool;
+ }
+
+ ret = safexcel_register_algorithms(priv);
+ if (ret) {
+ dev_err(dev, "Failed to register algorithms (%d)\n", ret);
+ goto err_pool;
+ }
+
+ return 0;
+
+err_pool:
+ dma_pool_destroy(priv->context_pool);
+err_clk:
+ clk_disable_unprepare(priv->clk);
+ return ret;
+}
+
+
+static int safexcel_remove(struct platform_device *pdev)
+{
+ struct safexcel_crypto_priv *priv = platform_get_drvdata(pdev);
+ int i;
+
+ safexcel_unregister_algorithms(priv);
+ clk_disable_unprepare(priv->clk);
+
+ for (i = 0; i < priv->config.rings; i++) {
+ safexcel_free_ring_descriptors(priv, &priv->ring[i].cdr,
+ &priv->ring[i].rdr);
+ destroy_workqueue(priv->ring[i].workqueue);
+ }
+
+ return 0;
+}
+
+static const struct of_device_id safexcel_of_match_table[] = {
+ { .compatible = "inside-secure,safexcel-eip197" },
+};
+
+
+static struct platform_driver crypto_safexcel = {
+ .probe = safexcel_probe,
+ .remove = safexcel_remove,
+ .driver = {
+ .name = "crypto-safexcel",
+ .of_match_table = safexcel_of_match_table,
+ },
+};
+module_platform_driver(crypto_safexcel);
+
+MODULE_AUTHOR("Antoine Tenart <antoine.tenart@free-electrons.com>");
+MODULE_AUTHOR("Ofer Heifetz <oferh@marvell.com>");
+MODULE_AUTHOR("Igal Liberman <igall@marvell.com>");
+MODULE_DESCRIPTION("Support for SafeXcel cryptographic engine EIP197");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/crypto/inside-secure/safexcel.h b/drivers/crypto/inside-secure/safexcel.h
new file mode 100644
index 000000000000..ffe39aaabe1c
--- /dev/null
+++ b/drivers/crypto/inside-secure/safexcel.h
@@ -0,0 +1,581 @@
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Antoine Tenart <antoine.tenart@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#ifndef __SAFEXCEL_H__
+#define __SAFEXCEL_H__
+
+#include <crypto/algapi.h>
+#include <crypto/hash.h>
+#include <crypto/aes.h>
+#include <crypto/internal/hash.h>
+#include <crypto/sha.h>
+#include <linux/device.h>
+#include <linux/clk.h>
+#include <linux/interrupt.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#define EIP197_HIA_VERSION_LE 0xca35
+#define EIP197_HIA_VERSION_BE 0x35ca
+
+/* Static configuration */
+#define EIP197_DEFAULT_RING_SIZE 64
+#define EIP197_MAX_TOKENS 5
+#define EIP197_MAX_RINGS 4
+#define EIP197_FETCH_COUNT 1
+#define EIP197_MAX_BATCH_SZ 8
+
+#define EIP197_GFP_FLAGS(base) ((base).flags & CRYPTO_TFM_REQ_MAY_SLEEP ? \
+ GFP_KERNEL : GFP_ATOMIC)
+
+/* CDR/RDR register offsets */
+#define EIP197_HIA_xDR_OFF(r) (0x80000 + (r) * 0x1000)
+#define EIP197_HIA_CDR(r) (EIP197_HIA_xDR_OFF(r))
+#define EIP197_HIA_RDR(r) (EIP197_HIA_xDR_OFF(r) + 0x800)
+#define EIP197_HIA_xDR_RING_BASE_ADDR_LO 0x0
+#define EIP197_HIA_xDR_RING_BASE_ADDR_HI 0x4
+#define EIP197_HIA_xDR_RING_SIZE 0x18
+#define EIP197_HIA_xDR_DESC_SIZE 0x1c
+#define EIP197_HIA_xDR_CFG 0x20
+#define EIP197_HIA_xDR_DMA_CFG 0x24
+#define EIP197_HIA_xDR_THRESH 0x28
+#define EIP197_HIA_xDR_PREP_COUNT 0x2c
+#define EIP197_HIA_xDR_PROC_COUNT 0x30
+#define EIP197_HIA_xDR_PREP_PNTR 0x34
+#define EIP197_HIA_xDR_PROC_PNTR 0x38
+#define EIP197_HIA_xDR_STAT 0x3c
+
+/* register offsets */
+#define EIP197_HIA_DFE_CFG 0x8c000
+#define EIP197_HIA_DFE_THR_CTRL 0x8c040
+#define EIP197_HIA_DFE_THR_STAT 0x8c044
+#define EIP197_HIA_DSE_CFG 0x8d000
+#define EIP197_HIA_DSE_THR_CTRL 0x8d040
+#define EIP197_HIA_DSE_THR_STAT 0x8d044
+#define EIP197_HIA_RA_PE_CTRL 0x90010
+#define EIP197_HIA_RA_PE_STAT 0x90014
+#define EIP197_HIA_AIC_R_OFF(r) ((r) * 0x1000)
+#define EIP197_HIA_AIC_R_ENABLE_CTRL(r) (0x9e808 - EIP197_HIA_AIC_R_OFF(r))
+#define EIP197_HIA_AIC_R_ENABLED_STAT(r) (0x9e810 - EIP197_HIA_AIC_R_OFF(r))
+#define EIP197_HIA_AIC_R_ACK(r) (0x9e810 - EIP197_HIA_AIC_R_OFF(r))
+#define EIP197_HIA_AIC_R_ENABLE_CLR(r) (0x9e814 - EIP197_HIA_AIC_R_OFF(r))
+#define EIP197_HIA_AIC_G_ENABLE_CTRL 0x9f808
+#define EIP197_HIA_AIC_G_ENABLED_STAT 0x9f810
+#define EIP197_HIA_AIC_G_ACK 0x9f810
+#define EIP197_HIA_MST_CTRL 0x9fff4
+#define EIP197_HIA_OPTIONS 0x9fff8
+#define EIP197_HIA_VERSION 0x9fffc
+#define EIP197_PE_IN_DBUF_THRES 0xa0000
+#define EIP197_PE_IN_TBUF_THRES 0xa0100
+#define EIP197_PE_ICE_SCRATCH_RAM 0xa0800
+#define EIP197_PE_ICE_PUE_CTRL 0xa0c80
+#define EIP197_PE_ICE_SCRATCH_CTRL 0xa0d04
+#define EIP197_PE_ICE_FPP_CTRL 0xa0d80
+#define EIP197_PE_ICE_RAM_CTRL 0xa0ff0
+#define EIP197_PE_EIP96_FUNCTION_EN 0xa1004
+#define EIP197_PE_EIP96_CONTEXT_CTRL 0xa1008
+#define EIP197_PE_EIP96_CONTEXT_STAT 0xa100c
+#define EIP197_PE_OUT_DBUF_THRES 0xa1c00
+#define EIP197_PE_OUT_TBUF_THRES 0xa1d00
+#define EIP197_CLASSIFICATION_RAMS 0xe0000
+#define EIP197_TRC_CTRL 0xf0800
+#define EIP197_TRC_LASTRES 0xf0804
+#define EIP197_TRC_REGINDEX 0xf0808
+#define EIP197_TRC_PARAMS 0xf0820
+#define EIP197_TRC_FREECHAIN 0xf0824
+#define EIP197_TRC_PARAMS2 0xf0828
+#define EIP197_TRC_ECCCTRL 0xf0830
+#define EIP197_TRC_ECCSTAT 0xf0834
+#define EIP197_TRC_ECCADMINSTAT 0xf0838
+#define EIP197_TRC_ECCDATASTAT 0xf083c
+#define EIP197_TRC_ECCDATA 0xf0840
+#define EIP197_CS_RAM_CTRL 0xf7ff0
+#define EIP197_MST_CTRL 0xffff4
+
+/* EIP197_HIA_xDR_DESC_SIZE */
+#define EIP197_xDR_DESC_MODE_64BIT BIT(31)
+
+/* EIP197_HIA_xDR_DMA_CFG */
+#define EIP197_HIA_xDR_WR_RES_BUF BIT(22)
+#define EIP197_HIA_xDR_WR_CTRL_BUG BIT(23)
+#define EIP197_HIA_xDR_WR_OWN_BUF BIT(24)
+#define EIP197_HIA_xDR_CFG_WR_CACHE(n) (((n) & 0x7) << 23)
+#define EIP197_HIA_xDR_CFG_RD_CACHE(n) (((n) & 0x7) << 29)
+
+/* EIP197_HIA_CDR_THRESH */
+#define EIP197_HIA_CDR_THRESH_PROC_PKT(n) (n)
+#define EIP197_HIA_CDR_THRESH_PROC_MODE BIT(22)
+#define EIP197_HIA_CDR_THRESH_PKT_MODE BIT(23)
+#define EIP197_HIA_CDR_THRESH_TIMEOUT(n) ((n) << 24) /* x256 clk cycles */
+
+/* EIP197_HIA_RDR_THRESH */
+#define EIP197_HIA_RDR_THRESH_PROC_PKT(n) (n)
+#define EIP197_HIA_RDR_THRESH_PKT_MODE BIT(23)
+#define EIP197_HIA_RDR_THRESH_TIMEOUT(n) ((n) << 24) /* x256 clk cycles */
+
+/* EIP197_HIA_xDR_PREP_COUNT */
+#define EIP197_xDR_PREP_CLR_COUNT BIT(31)
+
+/* EIP197_HIA_xDR_PROC_COUNT */
+#define EIP197_xDR_PROC_xD_COUNT(n) ((n) << 2)
+#define EIP197_xDR_PROC_xD_PKT(n) ((n) << 24)
+#define EIP197_xDR_PROC_CLR_COUNT BIT(31)
+
+/* EIP197_HIA_xDR_STAT */
+#define EIP197_xDR_DMA_ERR BIT(0)
+#define EIP197_xDR_PREP_CMD_THRES BIT(1)
+#define EIP197_xDR_ERR BIT(2)
+#define EIP197_xDR_THRESH BIT(4)
+#define EIP197_xDR_TIMEOUT BIT(5)
+
+#define EIP197_HIA_RA_PE_CTRL_RESET BIT(31)
+#define EIP197_HIA_RA_PE_CTRL_EN BIT(30)
+
+/* EIP197_HIA_AIC_R_ENABLE_CTRL */
+#define EIP197_CDR_IRQ(n) BIT((n) * 2)
+#define EIP197_RDR_IRQ(n) BIT((n) * 2 + 1)
+
+/* EIP197_HIA_DFE/DSE_CFG */
+#define EIP197_HIA_DxE_CFG_MIN_DATA_SIZE(n) ((n) << 0)
+#define EIP197_HIA_DxE_CFG_DATA_CACHE_CTRL(n) (((n) & 0x7) << 4)
+#define EIP197_HIA_DxE_CFG_MAX_DATA_SIZE(n) ((n) << 8)
+#define EIP197_HIA_DxE_CFG_MIN_CTRL_SIZE(n) ((n) << 16)
+#define EIP197_HIA_DxE_CFG_CTRL_CACHE_CTRL(n) (((n) & 0x7) << 20)
+#define EIP197_HIA_DxE_CFG_MAX_CTRL_SIZE(n) ((n) << 24)
+#define EIP197_HIA_DFE_CFG_DIS_DEBUG (BIT(31) | BIT(29))
+#define EIP197_HIA_DSE_CFG_DIS_DEBUG BIT(31)
+
+/* EIP197_HIA_DFE/DSE_THR_CTRL */
+#define EIP197_DxE_THR_CTRL_EN BIT(30)
+#define EIP197_DxE_THR_CTRL_RESET_PE BIT(31)
+
+/* EIP197_HIA_AIC_G_ENABLED_STAT */
+#define EIP197_G_IRQ_DFE(n) BIT((n) << 1)
+#define EIP197_G_IRQ_DSE(n) BIT(((n) << 1) + 1)
+#define EIP197_G_IRQ_RING BIT(16)
+#define EIP197_G_IRQ_PE(n) BIT((n) + 20)
+
+/* EIP197_HIA_MST_CTRL */
+#define RD_CACHE_3BITS 0x5
+#define WR_CACHE_3BITS 0x3
+#define RD_CACHE_4BITS (RD_CACHE_3BITS << 1 | BIT(0))
+#define WR_CACHE_4BITS (WR_CACHE_3BITS << 1 | BIT(0))
+#define EIP197_MST_CTRL_RD_CACHE(n) (((n) & 0xf) << 0)
+#define EIP197_MST_CTRL_WD_CACHE(n) (((n) & 0xf) << 4)
+#define EIP197_MST_CTRL_BYTE_SWAP BIT(24)
+#define EIP197_MST_CTRL_NO_BYTE_SWAP BIT(25)
+
+/* EIP197_PE_IN_DBUF/TBUF_THRES */
+#define EIP197_PE_IN_xBUF_THRES_MIN(n) ((n) << 8)
+#define EIP197_PE_IN_xBUF_THRES_MAX(n) ((n) << 12)
+
+/* EIP197_PE_OUT_DBUF_THRES */
+#define EIP197_PE_OUT_DBUF_THRES_MIN(n) ((n) << 0)
+#define EIP197_PE_OUT_DBUF_THRES_MAX(n) ((n) << 4)
+
+/* EIP197_PE_ICE_SCRATCH_CTRL */
+#define EIP197_PE_ICE_SCRATCH_CTRL_CHANGE_TIMER BIT(2)
+#define EIP197_PE_ICE_SCRATCH_CTRL_TIMER_EN BIT(3)
+#define EIP197_PE_ICE_SCRATCH_CTRL_CHANGE_ACCESS BIT(24)
+#define EIP197_PE_ICE_SCRATCH_CTRL_SCRATCH_ACCESS BIT(25)
+
+/* EIP197_PE_ICE_SCRATCH_RAM */
+#define EIP197_NUM_OF_SCRATCH_BLOCKS 32
+
+/* EIP197_PE_ICE_PUE/FPP_CTRL */
+#define EIP197_PE_ICE_x_CTRL_SW_RESET BIT(0)
+#define EIP197_PE_ICE_x_CTRL_CLR_ECC_NON_CORR BIT(14)
+#define EIP197_PE_ICE_x_CTRL_CLR_ECC_CORR BIT(15)
+
+/* EIP197_PE_ICE_RAM_CTRL */
+#define EIP197_PE_ICE_RAM_CTRL_PUE_PROG_EN BIT(0)
+#define EIP197_PE_ICE_RAM_CTRL_FPP_PROG_EN BIT(1)
+
+/* EIP197_PE_EIP96_FUNCTION_EN */
+#define EIP197_FUNCTION_RSVD (BIT(6) | BIT(15) | BIT(20) | BIT(23))
+#define EIP197_PROTOCOL_HASH_ONLY BIT(0)
+#define EIP197_PROTOCOL_ENCRYPT_ONLY BIT(1)
+#define EIP197_PROTOCOL_HASH_ENCRYPT BIT(2)
+#define EIP197_PROTOCOL_HASH_DECRYPT BIT(3)
+#define EIP197_PROTOCOL_ENCRYPT_HASH BIT(4)
+#define EIP197_PROTOCOL_DECRYPT_HASH BIT(5)
+#define EIP197_ALG_ARC4 BIT(7)
+#define EIP197_ALG_AES_ECB BIT(8)
+#define EIP197_ALG_AES_CBC BIT(9)
+#define EIP197_ALG_AES_CTR_ICM BIT(10)
+#define EIP197_ALG_AES_OFB BIT(11)
+#define EIP197_ALG_AES_CFB BIT(12)
+#define EIP197_ALG_DES_ECB BIT(13)
+#define EIP197_ALG_DES_CBC BIT(14)
+#define EIP197_ALG_DES_OFB BIT(16)
+#define EIP197_ALG_DES_CFB BIT(17)
+#define EIP197_ALG_3DES_ECB BIT(18)
+#define EIP197_ALG_3DES_CBC BIT(19)
+#define EIP197_ALG_3DES_OFB BIT(21)
+#define EIP197_ALG_3DES_CFB BIT(22)
+#define EIP197_ALG_MD5 BIT(24)
+#define EIP197_ALG_HMAC_MD5 BIT(25)
+#define EIP197_ALG_SHA1 BIT(26)
+#define EIP197_ALG_HMAC_SHA1 BIT(27)
+#define EIP197_ALG_SHA2 BIT(28)
+#define EIP197_ALG_HMAC_SHA2 BIT(29)
+#define EIP197_ALG_AES_XCBC_MAC BIT(30)
+#define EIP197_ALG_GCM_HASH BIT(31)
+
+/* EIP197_PE_EIP96_CONTEXT_CTRL */
+#define EIP197_CONTEXT_SIZE(n) (n)
+#define EIP197_ADDRESS_MODE BIT(8)
+#define EIP197_CONTROL_MODE BIT(9)
+
+/* Context Control */
+struct safexcel_context_record {
+ u32 control0;
+ u32 control1;
+
+ __le32 data[12];
+} __packed;
+
+/* control0 */
+#define CONTEXT_CONTROL_TYPE_NULL_OUT 0x0
+#define CONTEXT_CONTROL_TYPE_NULL_IN 0x1
+#define CONTEXT_CONTROL_TYPE_HASH_OUT 0x2
+#define CONTEXT_CONTROL_TYPE_HASH_IN 0x3
+#define CONTEXT_CONTROL_TYPE_CRYPTO_OUT 0x4
+#define CONTEXT_CONTROL_TYPE_CRYPTO_IN 0x5
+#define CONTEXT_CONTROL_TYPE_ENCRYPT_HASH_OUT 0x6
+#define CONTEXT_CONTROL_TYPE_DECRYPT_HASH_IN 0x7
+#define CONTEXT_CONTROL_TYPE_HASH_ENCRYPT_OUT 0x14
+#define CONTEXT_CONTROL_TYPE_HASH_DECRYPT_OUT 0x15
+#define CONTEXT_CONTROL_RESTART_HASH BIT(4)
+#define CONTEXT_CONTROL_NO_FINISH_HASH BIT(5)
+#define CONTEXT_CONTROL_SIZE(n) ((n) << 8)
+#define CONTEXT_CONTROL_KEY_EN BIT(16)
+#define CONTEXT_CONTROL_CRYPTO_ALG_AES128 (0x5 << 17)
+#define CONTEXT_CONTROL_CRYPTO_ALG_AES192 (0x6 << 17)
+#define CONTEXT_CONTROL_CRYPTO_ALG_AES256 (0x7 << 17)
+#define CONTEXT_CONTROL_DIGEST_PRECOMPUTED (0x1 << 21)
+#define CONTEXT_CONTROL_DIGEST_HMAC (0x3 << 21)
+#define CONTEXT_CONTROL_CRYPTO_ALG_SHA1 (0x2 << 23)
+#define CONTEXT_CONTROL_CRYPTO_ALG_SHA224 (0x4 << 23)
+#define CONTEXT_CONTROL_CRYPTO_ALG_SHA256 (0x3 << 23)
+#define CONTEXT_CONTROL_INV_FR (0x5 << 24)
+#define CONTEXT_CONTROL_INV_TR (0x6 << 24)
+
+/* control1 */
+#define CONTEXT_CONTROL_CRYPTO_MODE_ECB (0 << 0)
+#define CONTEXT_CONTROL_CRYPTO_MODE_CBC (1 << 0)
+#define CONTEXT_CONTROL_IV0 BIT(5)
+#define CONTEXT_CONTROL_IV1 BIT(6)
+#define CONTEXT_CONTROL_IV2 BIT(7)
+#define CONTEXT_CONTROL_IV3 BIT(8)
+#define CONTEXT_CONTROL_DIGEST_CNT BIT(9)
+#define CONTEXT_CONTROL_COUNTER_MODE BIT(10)
+#define CONTEXT_CONTROL_HASH_STORE BIT(19)
+
+/* EIP197_CS_RAM_CTRL */
+#define EIP197_TRC_ENABLE_0 BIT(4)
+#define EIP197_TRC_ENABLE_1 BIT(5)
+#define EIP197_TRC_ENABLE_2 BIT(6)
+#define EIP197_TRC_ENABLE_MASK GENMASK(6, 4)
+
+/* EIP197_TRC_PARAMS */
+#define EIP197_TRC_PARAMS_SW_RESET BIT(0)
+#define EIP197_TRC_PARAMS_DATA_ACCESS BIT(2)
+#define EIP197_TRC_PARAMS_HTABLE_SZ(x) ((x) << 4)
+#define EIP197_TRC_PARAMS_BLK_TIMER_SPEED(x) ((x) << 10)
+#define EIP197_TRC_PARAMS_RC_SZ_LARGE(n) ((n) << 18)
+
+/* EIP197_TRC_FREECHAIN */
+#define EIP197_TRC_FREECHAIN_HEAD_PTR(p) (p)
+#define EIP197_TRC_FREECHAIN_TAIL_PTR(p) ((p) << 16)
+
+/* EIP197_TRC_PARAMS2 */
+#define EIP197_TRC_PARAMS2_HTABLE_PTR(p) (p)
+#define EIP197_TRC_PARAMS2_RC_SZ_SMALL(n) ((n) << 18)
+
+/* Cache helpers */
+#define EIP197_CS_RC_MAX 52
+#define EIP197_CS_RC_SIZE (4 * sizeof(u32))
+#define EIP197_CS_RC_NEXT(x) (x)
+#define EIP197_CS_RC_PREV(x) ((x) << 10)
+#define EIP197_RC_NULL 0x3ff
+#define EIP197_CS_TRC_REC_WC 59
+#define EIP197_CS_TRC_LG_REC_WC 73
+
+/* Result data */
+struct result_data_desc {
+ u32 packet_length:17;
+ u32 error_code:15;
+
+ u8 bypass_length:4;
+ u8 e15:1;
+ u16 rsvd0;
+ u8 hash_bytes:1;
+ u8 hash_length:6;
+ u8 generic_bytes:1;
+ u8 checksum:1;
+ u8 next_header:1;
+ u8 length:1;
+
+ u16 application_id;
+ u16 rsvd1;
+
+ u32 rsvd2;
+} __packed;
+
+
+/* Basic Result Descriptor format */
+struct safexcel_result_desc {
+ u32 particle_size:17;
+ u8 rsvd0:3;
+ u8 descriptor_overflow:1;
+ u8 buffer_overflow:1;
+ u8 last_seg:1;
+ u8 first_seg:1;
+ u16 result_size:8;
+
+ u32 rsvd1;
+
+ u32 data_lo;
+ u32 data_hi;
+
+ struct result_data_desc result_data;
+} __packed;
+
+struct safexcel_token {
+ u32 packet_length:17;
+ u8 stat:2;
+ u16 instructions:9;
+ u8 opcode:4;
+} __packed;
+
+#define EIP197_TOKEN_STAT_LAST_HASH BIT(0)
+#define EIP197_TOKEN_STAT_LAST_PACKET BIT(1)
+#define EIP197_TOKEN_OPCODE_DIRECTION 0x0
+#define EIP197_TOKEN_OPCODE_INSERT 0x2
+#define EIP197_TOKEN_OPCODE_NOOP EIP197_TOKEN_OPCODE_INSERT
+#define EIP197_TOKEN_OPCODE_BYPASS GENMASK(3, 0)
+
+static inline void eip197_noop_token(struct safexcel_token *token)
+{
+ token->opcode = EIP197_TOKEN_OPCODE_NOOP;
+ token->packet_length = BIT(2);
+}
+
+/* Instructions */
+#define EIP197_TOKEN_INS_INSERT_HASH_DIGEST 0x1c
+#define EIP197_TOKEN_INS_TYPE_OUTPUT BIT(5)
+#define EIP197_TOKEN_INS_TYPE_HASH BIT(6)
+#define EIP197_TOKEN_INS_TYPE_CRYTO BIT(7)
+#define EIP197_TOKEN_INS_LAST BIT(8)
+
+/* Processing Engine Control Data */
+struct safexcel_control_data_desc {
+ u32 packet_length:17;
+ u16 options:13;
+ u8 type:2;
+
+ u16 application_id;
+ u16 rsvd;
+
+ u8 refresh:2;
+ u32 context_lo:30;
+ u32 context_hi;
+
+ u32 control0;
+ u32 control1;
+
+ u32 token[EIP197_MAX_TOKENS];
+} __packed;
+
+#define EIP197_OPTION_MAGIC_VALUE BIT(0)
+#define EIP197_OPTION_64BIT_CTX BIT(1)
+#define EIP197_OPTION_CTX_CTRL_IN_CMD BIT(8)
+#define EIP197_OPTION_4_TOKEN_IV_CMD GENMASK(11, 9)
+
+#define EIP197_TYPE_EXTENDED 0x3
+
+/* Basic Command Descriptor format */
+struct safexcel_command_desc {
+ u32 particle_size:17;
+ u8 rsvd0:5;
+ u8 last_seg:1;
+ u8 first_seg:1;
+ u16 additional_cdata_size:8;
+
+ u32 rsvd1;
+
+ u32 data_lo;
+ u32 data_hi;
+
+ struct safexcel_control_data_desc control_data;
+} __packed;
+
+/*
+ * Internal structures & functions
+ */
+
+enum eip197_fw {
+ FW_IFPP = 0,
+ FW_IPUE,
+ FW_NB
+};
+
+struct safexcel_ring {
+ void *base;
+ void *base_end;
+ dma_addr_t base_dma;
+
+ /* write and read pointers */
+ void *write;
+ void *read;
+
+ /* number of elements used in the ring */
+ unsigned nr;
+ unsigned offset;
+};
+
+enum safexcel_alg_type {
+ SAFEXCEL_ALG_TYPE_CIPHER,
+ SAFEXCEL_ALG_TYPE_AHASH,
+};
+
+struct safexcel_request {
+ struct list_head list;
+ struct crypto_async_request *req;
+};
+
+struct safexcel_config {
+ u32 rings;
+
+ u32 cd_size;
+ u32 cd_offset;
+
+ u32 rd_size;
+ u32 rd_offset;
+};
+
+struct safexcel_work_data {
+ struct work_struct work;
+ struct safexcel_crypto_priv *priv;
+ int ring;
+};
+
+struct safexcel_crypto_priv {
+ void __iomem *base;
+ struct device *dev;
+ struct clk *clk;
+ struct safexcel_config config;
+
+ spinlock_t lock;
+ struct crypto_queue queue;
+
+ bool need_dequeue;
+
+ /* context DMA pool */
+ struct dma_pool *context_pool;
+
+ atomic_t ring_used;
+
+ struct {
+ spinlock_t lock;
+ spinlock_t egress_lock;
+
+ struct list_head list;
+ struct workqueue_struct *workqueue;
+ struct safexcel_work_data work_data;
+
+ /* command/result rings */
+ struct safexcel_ring cdr;
+ struct safexcel_ring rdr;
+ } ring[EIP197_MAX_RINGS];
+};
+
+struct safexcel_context {
+ int (*send)(struct crypto_async_request *req, int ring,
+ struct safexcel_request *request, int *commands,
+ int *results);
+ int (*handle_result)(struct safexcel_crypto_priv *priv, int ring,
+ struct crypto_async_request *req, bool *complete,
+ int *ret);
+ struct safexcel_context_record *ctxr;
+ dma_addr_t ctxr_dma;
+
+ int ring;
+ bool needs_inv;
+ bool exit_inv;
+
+ /* Used for ahash requests */
+ dma_addr_t result_dma;
+ dma_addr_t cache_dma;
+ unsigned int cache_sz;
+};
+
+/*
+ * Template structure to describe the algorithms in order to register them.
+ * It also has the purpose to contain our private structure and is actually
+ * the only way I know in this framework to avoid having global pointers...
+ */
+struct safexcel_alg_template {
+ struct safexcel_crypto_priv *priv;
+ enum safexcel_alg_type type;
+ union {
+ struct crypto_alg crypto;
+ struct ahash_alg ahash;
+ } alg;
+};
+
+struct safexcel_inv_result {
+ struct completion completion;
+ int error;
+};
+
+void safexcel_dequeue(struct safexcel_crypto_priv *priv);
+void safexcel_complete(struct safexcel_crypto_priv *priv, int ring);
+void safexcel_free_context(struct safexcel_crypto_priv *priv,
+ struct crypto_async_request *req,
+ int result_sz);
+int safexcel_invalidate_cache(struct crypto_async_request *async,
+ struct safexcel_context *ctx,
+ struct safexcel_crypto_priv *priv,
+ dma_addr_t ctxr_dma,
+ int ring, struct safexcel_request *request);
+int safexcel_init_ring_descriptors(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *cdr,
+ struct safexcel_ring *rdr);
+void safexcel_free_ring_descriptors(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *cdr,
+ struct safexcel_ring *rdr);
+int safexcel_select_ring(struct safexcel_crypto_priv *priv);
+void *safexcel_ring_next_rptr(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *ring);
+void safexcel_ring_rollback_wptr(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *ring);
+struct safexcel_command_desc *safexcel_add_cdesc(struct safexcel_crypto_priv *priv,
+ int ring_id,
+ bool first, bool last,
+ phys_addr_t data, u32 len,
+ u32 full_data_len,
+ phys_addr_t context);
+struct safexcel_result_desc *safexcel_add_rdesc(struct safexcel_crypto_priv *priv,
+ int ring_id,
+ bool first, bool last,
+ phys_addr_t data, u32 len);
+void safexcel_inv_complete(struct crypto_async_request *req, int error);
+
+/* available algorithms */
+extern struct safexcel_alg_template safexcel_alg_ecb_aes;
+extern struct safexcel_alg_template safexcel_alg_cbc_aes;
+extern struct safexcel_alg_template safexcel_alg_sha1;
+extern struct safexcel_alg_template safexcel_alg_sha224;
+extern struct safexcel_alg_template safexcel_alg_sha256;
+extern struct safexcel_alg_template safexcel_alg_hmac_sha1;
+
+#endif
diff --git a/drivers/crypto/inside-secure/safexcel_cipher.c b/drivers/crypto/inside-secure/safexcel_cipher.c
new file mode 100644
index 000000000000..ff42e45ae43e
--- /dev/null
+++ b/drivers/crypto/inside-secure/safexcel_cipher.c
@@ -0,0 +1,544 @@
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Antoine Tenart <antoine.tenart@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <crypto/aes.h>
+#include <linux/dmapool.h>
+
+#include "safexcel.h"
+
+enum safexcel_cipher_direction {
+ SAFEXCEL_ENCRYPT,
+ SAFEXCEL_DECRYPT,
+};
+
+struct safexcel_cipher_ctx {
+ struct safexcel_context base;
+ struct safexcel_crypto_priv *priv;
+
+ enum safexcel_cipher_direction direction;
+ u32 mode;
+
+ __le32 key[8];
+ unsigned int key_len;
+};
+
+static void safexcel_cipher_token(struct safexcel_cipher_ctx *ctx,
+ struct crypto_async_request *async,
+ struct safexcel_command_desc *cdesc,
+ u32 length)
+{
+ struct ablkcipher_request *req = ablkcipher_request_cast(async);
+ struct safexcel_token *token;
+ unsigned offset = 0;
+
+ if (ctx->mode == CONTEXT_CONTROL_CRYPTO_MODE_CBC) {
+ offset = AES_BLOCK_SIZE / sizeof(u32);
+ memcpy(cdesc->control_data.token, req->info, AES_BLOCK_SIZE);
+
+ cdesc->control_data.options |= EIP197_OPTION_4_TOKEN_IV_CMD;
+ }
+
+ token = (struct safexcel_token *)(cdesc->control_data.token + offset);
+
+ token[0].opcode = EIP197_TOKEN_OPCODE_DIRECTION;
+ token[0].packet_length = length;
+ token[0].stat = EIP197_TOKEN_STAT_LAST_PACKET;
+ token[0].instructions = EIP197_TOKEN_INS_LAST |
+ EIP197_TOKEN_INS_TYPE_CRYTO |
+ EIP197_TOKEN_INS_TYPE_OUTPUT;
+}
+
+static int safexcel_aes_setkey(struct crypto_ablkcipher *ctfm, const u8 *key,
+ unsigned int len)
+{
+ struct crypto_tfm *tfm = crypto_ablkcipher_tfm(ctfm);
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct crypto_aes_ctx aes;
+ int ret, i;
+
+ ret = crypto_aes_expand_key(&aes, key, len);
+ if (ret) {
+ crypto_ablkcipher_set_flags(ctfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return ret;
+ }
+
+ for (i = 0; i < len / sizeof(u32); i++) {
+ if (ctx->key[i] != cpu_to_le32(aes.key_enc[i])) {
+ ctx->base.needs_inv = true;
+ break;
+ }
+ }
+
+ for (i = 0; i < len / sizeof(u32); i++)
+ ctx->key[i] = cpu_to_le32(aes.key_enc[i]);
+
+ ctx->key_len = len;
+
+ return 0;
+}
+
+static int safexcel_context_control(struct safexcel_cipher_ctx *ctx,
+ struct safexcel_command_desc *cdesc)
+{
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ int ctrl_size;
+
+ if (ctx->direction == SAFEXCEL_ENCRYPT)
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_CRYPTO_OUT;
+ else
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_CRYPTO_IN;
+
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_KEY_EN;
+ cdesc->control_data.control1 |= ctx->mode;
+
+ switch (ctx->key_len) {
+ case AES_KEYSIZE_128:
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_CRYPTO_ALG_AES128;
+ ctrl_size = 4;
+ break;
+ case AES_KEYSIZE_192:
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_CRYPTO_ALG_AES192;
+ ctrl_size = 6;
+ break;
+ case AES_KEYSIZE_256:
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_CRYPTO_ALG_AES256;
+ ctrl_size = 8;
+ break;
+ default:
+ dev_err(priv->dev, "aes keysize not supported: %u\n",
+ ctx->key_len);
+ return -EINVAL;
+ }
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_SIZE(ctrl_size);
+
+ return 0;
+}
+
+static int safexcel_handle_result(struct safexcel_crypto_priv *priv, int ring,
+ struct crypto_async_request *async,
+ bool *should_complete, int *ret)
+{
+ struct ablkcipher_request *req = ablkcipher_request_cast(async);
+ struct safexcel_result_desc *rdesc;
+ int ndesc = 0;
+
+ *ret = 0;
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+ do {
+ rdesc = safexcel_ring_next_rptr(priv, &priv->ring[ring].rdr);
+ if (IS_ERR(rdesc)) {
+ dev_err(priv->dev,
+ "cipher: result: could not retrieve the result descriptor\n");
+ *ret = PTR_ERR(rdesc);
+ break;
+ }
+
+ if (rdesc->result_data.error_code) {
+ dev_err(priv->dev,
+ "cipher: result: result descriptor error (%d)\n",
+ rdesc->result_data.error_code);
+ *ret = -EIO;
+ }
+
+ ndesc++;
+ } while (!rdesc->last_seg);
+
+ safexcel_complete(priv, ring);
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ if (req->src == req->dst) {
+ dma_unmap_sg(priv->dev, req->src,
+ sg_nents_for_len(req->src, req->nbytes),
+ DMA_BIDIRECTIONAL);
+ } else {
+ dma_unmap_sg(priv->dev, req->src,
+ sg_nents_for_len(req->src, req->nbytes),
+ DMA_TO_DEVICE);
+ dma_unmap_sg(priv->dev, req->dst,
+ sg_nents_for_len(req->dst, req->nbytes),
+ DMA_FROM_DEVICE);
+ }
+
+ *should_complete = true;
+
+ return ndesc;
+}
+
+static int safexcel_aes_send(struct crypto_async_request *async,
+ int ring, struct safexcel_request *request,
+ int *commands, int *results)
+{
+ struct ablkcipher_request *req = ablkcipher_request_cast(async);
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ struct safexcel_command_desc *cdesc;
+ struct safexcel_result_desc *rdesc;
+ struct scatterlist *sg;
+ phys_addr_t ctxr_phys;
+ int nr_src, nr_dst, n_cdesc = 0, n_rdesc = 0, queued = req->nbytes;
+ int i, ret = 0;
+
+ request->req = &req->base;
+
+ if (req->src == req->dst) {
+ nr_src = sg_nents_for_len(req->src, req->nbytes);
+ nr_dst = nr_src;
+
+ if (dma_map_sg(priv->dev, req->src, nr_src, DMA_BIDIRECTIONAL) <= 0)
+ return -EINVAL;
+ } else {
+ nr_src = sg_nents_for_len(req->src, req->nbytes);
+ nr_dst = sg_nents_for_len(req->dst, req->nbytes);
+
+ if (dma_map_sg(priv->dev, req->src, nr_src, DMA_TO_DEVICE) <= 0)
+ return -EINVAL;
+
+ if (dma_map_sg(priv->dev, req->dst, nr_dst, DMA_FROM_DEVICE) <= 0) {
+ dma_unmap_sg(priv->dev, req->src, nr_src,
+ DMA_TO_DEVICE);
+ return -EINVAL;
+ }
+ }
+
+ ctxr_phys = dma_to_phys(priv->dev, ctx->base.ctxr_dma);
+
+ memcpy(ctx->base.ctxr->data, ctx->key, ctx->key_len);
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+
+ /* command descriptors */
+ for_each_sg(req->src, sg, nr_src, i) {
+ phys_addr_t sg_phys = dma_to_phys(priv->dev, sg_dma_address(sg));
+ int len = sg_dma_len(sg);
+
+ /* Do not overflow the request */
+ if (queued - len < 0)
+ len = queued;
+
+ cdesc = safexcel_add_cdesc(priv, ring, !n_cdesc, !(queued - len),
+ sg_phys, len, req->nbytes, ctxr_phys);
+ if (IS_ERR(cdesc)) {
+ /* No space left in the command descriptor ring */
+ ret = PTR_ERR(cdesc);
+ goto cdesc_rollback;
+ }
+ n_cdesc++;
+
+ if (n_cdesc == 1) {
+ safexcel_context_control(ctx, cdesc);
+ safexcel_cipher_token(ctx, async, cdesc, req->nbytes);
+ }
+
+ queued -= len;
+ if (!queued)
+ break;
+ }
+
+ /* result descriptors */
+ for_each_sg(req->dst, sg, nr_dst, i) {
+ bool first = !i, last = (i == nr_dst - 1);
+ phys_addr_t sg_phys = dma_to_phys(priv->dev, sg_dma_address(sg));
+ u32 len = sg_dma_len(sg);
+
+ rdesc = safexcel_add_rdesc(priv, ring, first, last, sg_phys, len);
+ if (IS_ERR(rdesc)) {
+ /* No space left in the result descriptor ring */
+ ret = PTR_ERR(rdesc);
+ goto rdesc_rollback;
+ }
+ n_rdesc++;
+ }
+
+ ctx->base.handle_result = safexcel_handle_result;
+
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ *commands = n_cdesc;
+ *results = nr_dst;
+ return 0;
+
+rdesc_rollback:
+ for (i = 0; i < n_rdesc; i++)
+ safexcel_ring_rollback_wptr(priv, &priv->ring[ring].rdr);
+cdesc_rollback:
+ for (i = 0; i < n_cdesc; i++)
+ safexcel_ring_rollback_wptr(priv, &priv->ring[ring].cdr);
+
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ if (req->src == req->dst) {
+ dma_unmap_sg(priv->dev, req->src, nr_src, DMA_BIDIRECTIONAL);
+ } else {
+ dma_unmap_sg(priv->dev, req->src, nr_src, DMA_TO_DEVICE);
+ dma_unmap_sg(priv->dev, req->dst, nr_dst, DMA_FROM_DEVICE);
+ }
+
+ return ret;
+}
+
+static int safexcel_handle_inv_result(struct safexcel_crypto_priv *priv,
+ int ring,
+ struct crypto_async_request *async,
+ bool *should_complete, int *ret)
+{
+ struct ablkcipher_request *req = ablkcipher_request_cast(async);
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ struct safexcel_result_desc *rdesc;
+ int ndesc = 0, enq_ret;
+
+ *ret = 0;
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+ do {
+ rdesc = safexcel_ring_next_rptr(priv, &priv->ring[ring].rdr);
+ if (IS_ERR(rdesc)) {
+ dev_err(priv->dev,
+ "cipher: invalidate: could not retrieve the result descriptor\n");
+ *ret = PTR_ERR(rdesc);
+ break;
+ }
+
+ if (rdesc->result_data.error_code) {
+ dev_err(priv->dev, "cipher: invalidate: result descriptor error (%d)\n",
+ rdesc->result_data.error_code);
+ *ret = -EIO;
+ }
+
+ ndesc++;
+ } while (!rdesc->last_seg);
+
+ safexcel_complete(priv, ring);
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ if (ctx->base.exit_inv) {
+ dma_pool_free(priv->context_pool, ctx->base.ctxr,
+ ctx->base.ctxr_dma);
+
+ *should_complete = true;
+
+ return ndesc;
+ }
+
+ ctx->base.needs_inv = false;
+ ctx->base.ring = safexcel_select_ring(priv);
+ ctx->base.send = safexcel_aes_send;
+
+ spin_lock_bh(&priv->lock);
+ enq_ret = crypto_enqueue_request(&priv->queue, async);
+ spin_unlock_bh(&priv->lock);
+
+ if (enq_ret != -EINPROGRESS)
+ *ret = enq_ret;
+
+ priv->need_dequeue = true;
+ *should_complete = false;
+
+ return ndesc;
+}
+
+static int safexcel_cipher_send_inv(struct crypto_async_request *async,
+ int ring, struct safexcel_request *request,
+ int *commands, int *results)
+{
+ struct ablkcipher_request *req = ablkcipher_request_cast(async);
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ int ret;
+
+ ctx->base.handle_result = safexcel_handle_inv_result;
+
+ ret = safexcel_invalidate_cache(async, &ctx->base, priv,
+ ctx->base.ctxr_dma, ring, request);
+ if (unlikely(ret))
+ return ret;
+
+ *commands = 1;
+ *results = 1;
+
+ return 0;
+}
+
+static int safexcel_cipher_exit_inv(struct crypto_tfm *tfm)
+{
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ struct ablkcipher_request req;
+ struct safexcel_inv_result result = { 0 };
+
+ memset(&req, 0, sizeof(struct ablkcipher_request));
+
+ /* create invalidation request */
+ init_completion(&result.completion);
+ ablkcipher_request_set_callback(&req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ safexcel_inv_complete, &result);
+
+ ablkcipher_request_set_tfm(&req, __crypto_ablkcipher_cast(tfm));
+ ctx = crypto_tfm_ctx(req.base.tfm);
+ ctx->base.exit_inv = true;
+ ctx->base.send = safexcel_cipher_send_inv;
+
+ spin_lock_bh(&priv->lock);
+ crypto_enqueue_request(&priv->queue, &req.base);
+ spin_unlock_bh(&priv->lock);
+
+ if (!priv->need_dequeue)
+ safexcel_dequeue(priv);
+
+ wait_for_completion_interruptible(&result.completion);
+
+ if (result.error) {
+ dev_warn(priv->dev,
+ "cipher: sync: invalidate: completion error %d\n",
+ result.error);
+ return result.error;
+ }
+
+ return 0;
+}
+
+static int safexcel_aes(struct ablkcipher_request *req,
+ enum safexcel_cipher_direction dir, u32 mode)
+{
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ int ret;
+
+ ctx->direction = dir;
+ ctx->mode = mode;
+
+ if (ctx->base.ctxr) {
+ if (ctx->base.needs_inv)
+ ctx->base.send = safexcel_cipher_send_inv;
+ } else {
+ ctx->base.ring = safexcel_select_ring(priv);
+ ctx->base.send = safexcel_aes_send;
+
+ ctx->base.ctxr = dma_pool_zalloc(priv->context_pool,
+ EIP197_GFP_FLAGS(req->base),
+ &ctx->base.ctxr_dma);
+ if (!ctx->base.ctxr)
+ return -ENOMEM;
+ }
+
+ spin_lock_bh(&priv->lock);
+ ret = crypto_enqueue_request(&priv->queue, &req->base);
+ spin_unlock_bh(&priv->lock);
+
+ if (!priv->need_dequeue)
+ safexcel_dequeue(priv);
+
+ return ret;
+}
+
+static int safexcel_ecb_aes_encrypt(struct ablkcipher_request *req)
+{
+ return safexcel_aes(req, SAFEXCEL_ENCRYPT,
+ CONTEXT_CONTROL_CRYPTO_MODE_ECB);
+}
+
+static int safexcel_ecb_aes_decrypt(struct ablkcipher_request *req)
+{
+ return safexcel_aes(req, SAFEXCEL_DECRYPT,
+ CONTEXT_CONTROL_CRYPTO_MODE_ECB);
+}
+
+static int safexcel_ablkcipher_cra_init(struct crypto_tfm *tfm)
+{
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct safexcel_alg_template *tmpl =
+ container_of(tfm->__crt_alg, struct safexcel_alg_template, alg.crypto);
+
+ ctx->priv = tmpl->priv;
+
+ return 0;
+}
+
+static void safexcel_ablkcipher_cra_exit(struct crypto_tfm *tfm)
+{
+ struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ int ret;
+
+ /* context not allocated, skip invalidation */
+ if (!ctx->base.ctxr)
+ return;
+
+ ret = safexcel_cipher_exit_inv(tfm);
+ if (ret)
+ dev_warn(priv->dev, "cipher: invalidation error %d\n", ret);
+}
+
+struct safexcel_alg_template safexcel_alg_ecb_aes = {
+ .type = SAFEXCEL_ALG_TYPE_CIPHER,
+ .alg.crypto = {
+ .cra_name = "ecb(aes)",
+ .cra_driver_name = "safexcel-ecb-aes",
+ .cra_priority = 300,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct safexcel_cipher_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_init = safexcel_ablkcipher_cra_init,
+ .cra_exit = safexcel_ablkcipher_cra_exit,
+ .cra_u = {
+ .ablkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .setkey = safexcel_aes_setkey,
+ .encrypt = safexcel_ecb_aes_encrypt,
+ .decrypt = safexcel_ecb_aes_decrypt,
+ },
+ },
+ },
+};
+
+static int safexcel_cbc_aes_encrypt(struct ablkcipher_request *req)
+{
+ return safexcel_aes(req, SAFEXCEL_ENCRYPT,
+ CONTEXT_CONTROL_CRYPTO_MODE_CBC);
+}
+
+static int safexcel_cbc_aes_decrypt(struct ablkcipher_request *req)
+{
+ return safexcel_aes(req, SAFEXCEL_DECRYPT,
+ CONTEXT_CONTROL_CRYPTO_MODE_CBC);
+}
+
+struct safexcel_alg_template safexcel_alg_cbc_aes = {
+ .type = SAFEXCEL_ALG_TYPE_CIPHER,
+ .alg.crypto = {
+ .cra_name = "cbc(aes)",
+ .cra_driver_name = "safexcel-cbc-aes",
+ .cra_priority = 300,
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct safexcel_cipher_ctx),
+ .cra_alignmask = 0,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_module = THIS_MODULE,
+ .cra_init = safexcel_ablkcipher_cra_init,
+ .cra_exit = safexcel_ablkcipher_cra_exit,
+ .cra_u = {
+ .ablkcipher = {
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = safexcel_aes_setkey,
+ .encrypt = safexcel_cbc_aes_encrypt,
+ .decrypt = safexcel_cbc_aes_decrypt,
+ },
+ },
+ },
+};
diff --git a/drivers/crypto/inside-secure/safexcel_hash.c b/drivers/crypto/inside-secure/safexcel_hash.c
new file mode 100644
index 000000000000..1f44e0a2ddf1
--- /dev/null
+++ b/drivers/crypto/inside-secure/safexcel_hash.c
@@ -0,0 +1,1045 @@
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Antoine Tenart <antoine.tenart@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <crypto/sha.h>
+#include <linux/dmapool.h>
+
+#include "safexcel.h"
+
+struct safexcel_ahash_ctx {
+ struct safexcel_context base;
+ struct safexcel_crypto_priv *priv;
+
+ u32 alg;
+ u32 digest;
+
+ u32 ipad[SHA1_DIGEST_SIZE / sizeof(u32)];
+ u32 opad[SHA1_DIGEST_SIZE / sizeof(u32)];
+};
+
+struct safexcel_ahash_req {
+ bool last_req;
+ bool finish;
+ bool hmac;
+
+ u8 state_sz; /* expected sate size, only set once */
+ u32 state[SHA256_DIGEST_SIZE / sizeof(u32)];
+
+ u64 len;
+ u64 processed;
+
+ u8 cache[SHA256_BLOCK_SIZE] __aligned(sizeof(u32));
+ u8 cache_next[SHA256_BLOCK_SIZE] __aligned(sizeof(u32));
+};
+
+struct safexcel_ahash_export_state {
+ u64 len;
+ u64 processed;
+
+ u32 state[SHA256_DIGEST_SIZE / sizeof(u32)];
+ u8 cache[SHA256_BLOCK_SIZE];
+};
+
+static const u8 sha1_zero_digest[SHA1_DIGEST_SIZE] = {
+ 0xda, 0x39, 0xa3, 0xee, 0x5e, 0x6b, 0x4b, 0x0d, 0x32, 0x55,
+ 0xbf, 0xef, 0x95, 0x60, 0x18, 0x90, 0xaf, 0xd8, 0x07, 0x09,
+};
+
+static const u8 sha224_zero_digest[SHA224_DIGEST_SIZE] = {
+ 0xd1, 0x4a, 0x02, 0x8c, 0x2a, 0x3a, 0x2b, 0xc9, 0x47, 0x61,
+ 0x02, 0xbb, 0x28, 0x82, 0x34, 0xc4, 0x15, 0xa2, 0xb0, 0x1f,
+ 0x82, 0x8e, 0xa6, 0x2a, 0xc5, 0xb3, 0xe4, 0x2f
+};
+
+static const u8 sha256_zero_digest[SHA256_DIGEST_SIZE] = {
+ 0xe3, 0xb0, 0xc4, 0x42, 0x98, 0xfc, 0x1c, 0x14, 0x9a, 0xfb,
+ 0xf4, 0xc8, 0x99, 0x6f, 0xb9, 0x24, 0x27, 0xae, 0x41, 0xe4,
+ 0x64, 0x9b, 0x93, 0x4c, 0xa4, 0x95, 0x99, 0x1b, 0x78, 0x52,
+ 0xb8, 0x55
+};
+
+static void safexcel_hash_token(struct safexcel_command_desc *cdesc,
+ u32 input_length, u32 result_length)
+{
+ struct safexcel_token *token =
+ (struct safexcel_token *)cdesc->control_data.token;
+
+ token[0].opcode = EIP197_TOKEN_OPCODE_DIRECTION;
+ token[0].packet_length = input_length;
+ token[0].stat = EIP197_TOKEN_STAT_LAST_HASH;
+ token[0].instructions = EIP197_TOKEN_INS_TYPE_HASH;
+
+ token[1].opcode = EIP197_TOKEN_OPCODE_INSERT;
+ token[1].packet_length = result_length;
+ token[1].stat = EIP197_TOKEN_STAT_LAST_HASH |
+ EIP197_TOKEN_STAT_LAST_PACKET;
+ token[1].instructions = EIP197_TOKEN_INS_TYPE_OUTPUT |
+ EIP197_TOKEN_INS_INSERT_HASH_DIGEST;
+}
+
+static void safexcel_context_control(struct safexcel_ahash_ctx *ctx,
+ struct safexcel_ahash_req *req,
+ struct safexcel_command_desc *cdesc,
+ unsigned int digestsize,
+ unsigned int blocksize)
+{
+ int i;
+
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_HASH_OUT;
+ cdesc->control_data.control0 |= ctx->alg;
+ cdesc->control_data.control0 |= ctx->digest;
+
+ if (ctx->digest == CONTEXT_CONTROL_DIGEST_PRECOMPUTED) {
+ if (req->processed) {
+ if (ctx->alg == CONTEXT_CONTROL_CRYPTO_ALG_SHA1)
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_SIZE(6);
+ else if (ctx->alg == CONTEXT_CONTROL_CRYPTO_ALG_SHA224 ||
+ ctx->alg == CONTEXT_CONTROL_CRYPTO_ALG_SHA256)
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_SIZE(9);
+
+ cdesc->control_data.control1 |= CONTEXT_CONTROL_DIGEST_CNT;
+ } else {
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_RESTART_HASH;
+ }
+
+ if (!req->finish)
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_NO_FINISH_HASH;
+
+ /*
+ * Copy the input digest if needed, and setup the context
+ * fields. Do this now as we need it to setup the first command
+ * descriptor.
+ */
+ if (req->processed) {
+ for (i = 0; i < digestsize / sizeof(u32); i++)
+ ctx->base.ctxr->data[i] = cpu_to_le32(req->state[i]);
+
+ if (req->finish)
+ ctx->base.ctxr->data[i] = cpu_to_le32(req->processed / blocksize);
+ }
+ } else if (ctx->digest == CONTEXT_CONTROL_DIGEST_HMAC) {
+ cdesc->control_data.control0 |= CONTEXT_CONTROL_SIZE(10);
+
+ memcpy(ctx->base.ctxr->data, ctx->ipad, digestsize);
+ memcpy(ctx->base.ctxr->data + digestsize / sizeof(u32),
+ ctx->opad, digestsize);
+ }
+}
+
+static int safexcel_handle_result(struct safexcel_crypto_priv *priv, int ring,
+ struct crypto_async_request *async,
+ bool *should_complete, int *ret)
+{
+ struct safexcel_result_desc *rdesc;
+ struct ahash_request *areq = ahash_request_cast(async);
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ struct safexcel_ahash_req *sreq = ahash_request_ctx(areq);
+ int cache_len, result_sz = sreq->state_sz;
+
+ *ret = 0;
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+ rdesc = safexcel_ring_next_rptr(priv, &priv->ring[ring].rdr);
+ if (IS_ERR(rdesc)) {
+ dev_err(priv->dev,
+ "hash: result: could not retrieve the result descriptor\n");
+ *ret = PTR_ERR(rdesc);
+ } else if (rdesc->result_data.error_code) {
+ dev_err(priv->dev,
+ "hash: result: result descriptor error (%d)\n",
+ rdesc->result_data.error_code);
+ *ret = -EINVAL;
+ }
+
+ safexcel_complete(priv, ring);
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ if (sreq->finish)
+ result_sz = crypto_ahash_digestsize(ahash);
+ memcpy(sreq->state, areq->result, result_sz);
+
+ dma_unmap_sg(priv->dev, areq->src,
+ sg_nents_for_len(areq->src, areq->nbytes), DMA_TO_DEVICE);
+
+ safexcel_free_context(priv, async, sreq->state_sz);
+
+ cache_len = sreq->len - sreq->processed;
+ if (cache_len)
+ memcpy(sreq->cache, sreq->cache_next, cache_len);
+
+ *should_complete = true;
+
+ return 1;
+}
+
+static int safexcel_ahash_send(struct crypto_async_request *async, int ring,
+ struct safexcel_request *request, int *commands,
+ int *results)
+{
+ struct ahash_request *areq = ahash_request_cast(async);
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ struct safexcel_command_desc *cdesc, *first_cdesc = NULL;
+ struct safexcel_result_desc *rdesc;
+ struct scatterlist *sg;
+ int i, nents, queued, len, cache_len, extra, n_cdesc = 0, ret = 0;
+ phys_addr_t ctxr_phys = 0;
+
+ queued = len = req->len - req->processed;
+ if (queued < crypto_ahash_blocksize(ahash))
+ cache_len = queued;
+ else
+ cache_len = queued - areq->nbytes;
+
+ /*
+ * If this is not the last request and the queued data does not fit
+ * into full blocks, cache it for the next send() call.
+ */
+ extra = queued & (crypto_ahash_blocksize(ahash) - 1);
+ if (!req->last_req && extra) {
+ sg_pcopy_to_buffer(areq->src, sg_nents(areq->src),
+ req->cache_next, extra, areq->nbytes - extra);
+
+ queued -= extra;
+ len -= extra;
+ }
+
+ request->req = &areq->base;
+ ctx->base.handle_result = safexcel_handle_result;
+ ctxr_phys = dma_to_phys(priv->dev, ctx->base.ctxr_dma);
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+
+ /* Add a command descriptor for the cached data, if any */
+ if (cache_len) {
+ ctx->base.cache_dma = dma_map_single(priv->dev, req->cache,
+ cache_len, DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->dev, ctx->base.cache_dma)) {
+ ret = -EINVAL;
+ goto unlock;
+ }
+
+ ctx->base.cache_sz = cache_len;
+ first_cdesc = safexcel_add_cdesc(priv, ring, 1,
+ (cache_len == len),
+ dma_to_phys(priv->dev, ctx->base.cache_dma),
+ cache_len, len, ctxr_phys);
+ if (IS_ERR(first_cdesc)) {
+ ret = PTR_ERR(first_cdesc);
+ goto free_cache;
+ }
+ n_cdesc++;
+
+ queued -= cache_len;
+ if (!queued)
+ goto send_command;
+ }
+
+ /* Now handle the current ahash request buffer(s) */
+ nents = sg_nents_for_len(areq->src, areq->nbytes);
+ if (dma_map_sg(priv->dev, areq->src, nents, DMA_TO_DEVICE) <= 0) {
+ ret = -ENOMEM;
+ goto cdesc_rollback;
+ }
+
+ for_each_sg(areq->src, sg, nents, i) {
+ phys_addr_t sg_phys = dma_to_phys(priv->dev, sg_dma_address(sg));
+ int sglen = sg_dma_len(sg);
+
+ /* Do not overflow the request */
+ if (queued - sglen < 0)
+ sglen = queued;
+
+ cdesc = safexcel_add_cdesc(priv, ring, !n_cdesc,
+ !(queued - sglen), sg_phys, sglen,
+ len, ctxr_phys);
+ if (IS_ERR(cdesc)) {
+ ret = PTR_ERR(cdesc);
+ goto cdesc_rollback;
+ }
+ n_cdesc++;
+
+ if (n_cdesc == 1)
+ first_cdesc = cdesc;
+
+ queued -= sglen;
+ if (!queued)
+ break;
+ }
+
+send_command:
+ /* Setup the context options */
+ safexcel_context_control(ctx, req, first_cdesc, req->state_sz,
+ crypto_ahash_blocksize(ahash));
+
+ /* Add the token */
+ safexcel_hash_token(first_cdesc, len, req->state_sz);
+
+ ctx->base.result_dma = dma_map_single(priv->dev, areq->result,
+ req->state_sz, DMA_FROM_DEVICE);
+ if (dma_mapping_error(priv->dev, ctx->base.result_dma)) {
+ ret = -EINVAL;
+ goto cdesc_rollback;
+ }
+
+ /* Add a result descriptor */
+ rdesc = safexcel_add_rdesc(priv, ring, 1, 1,
+ dma_to_phys(priv->dev, ctx->base.result_dma),
+ req->state_sz);
+ if (IS_ERR(rdesc)) {
+ ret = PTR_ERR(rdesc);
+ goto cdesc_rollback;
+ }
+
+ req->processed += len;
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ *commands = n_cdesc;
+ *results = 1;
+ return 0;
+
+cdesc_rollback:
+ for (i = 0; i < n_cdesc; i++)
+ safexcel_ring_rollback_wptr(priv, &priv->ring[ring].cdr);
+free_cache:
+ if (ctx->base.cache_dma)
+ dma_unmap_single(priv->dev, ctx->base.cache_dma,
+ ctx->base.cache_sz, DMA_TO_DEVICE);
+
+unlock:
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+ return ret;
+}
+
+static inline bool safexcel_ahash_needs_inv_get(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ unsigned int state_w_sz = req->state_sz / sizeof(u32);
+ int i;
+
+ for (i = 0; i < state_w_sz; i++)
+ if (ctx->base.ctxr->data[i] != cpu_to_le32(req->state[i]))
+ return true;
+
+ if (ctx->base.ctxr->data[state_w_sz] !=
+ cpu_to_le32(req->processed / crypto_ahash_blocksize(ahash)))
+ return true;
+
+ return false;
+}
+
+static int safexcel_handle_inv_result(struct safexcel_crypto_priv *priv,
+ int ring,
+ struct crypto_async_request *async,
+ bool *should_complete, int *ret)
+{
+ struct safexcel_result_desc *rdesc;
+ struct ahash_request *areq = ahash_request_cast(async);
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(ahash);
+ int enq_ret;
+
+ *ret = 0;
+
+ spin_lock_bh(&priv->ring[ring].egress_lock);
+ rdesc = safexcel_ring_next_rptr(priv, &priv->ring[ring].rdr);
+ if (IS_ERR(rdesc)) {
+ dev_err(priv->dev,
+ "hash: invalidate: could not retrieve the result descriptor\n");
+ *ret = PTR_ERR(rdesc);
+ } else if (rdesc->result_data.error_code) {
+ dev_err(priv->dev,
+ "hash: invalidate: result descriptor error (%d)\n",
+ rdesc->result_data.error_code);
+ *ret = -EINVAL;
+ }
+
+ safexcel_complete(priv, ring);
+ spin_unlock_bh(&priv->ring[ring].egress_lock);
+
+ if (ctx->base.exit_inv) {
+ dma_pool_free(priv->context_pool, ctx->base.ctxr,
+ ctx->base.ctxr_dma);
+
+ *should_complete = true;
+ return 1;
+ }
+
+ ctx->base.ring = safexcel_select_ring(priv);
+ ctx->base.needs_inv = false;
+ ctx->base.send = safexcel_ahash_send;
+
+ spin_lock_bh(&priv->lock);
+ enq_ret = crypto_enqueue_request(&priv->queue, async);
+ spin_unlock_bh(&priv->lock);
+
+ if (enq_ret != -EINPROGRESS)
+ *ret = enq_ret;
+
+ priv->need_dequeue = true;
+ *should_complete = false;
+
+ return 1;
+}
+
+static int safexcel_ahash_send_inv(struct crypto_async_request *async,
+ int ring, struct safexcel_request *request,
+ int *commands, int *results)
+{
+ struct ahash_request *areq = ahash_request_cast(async);
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ int ret;
+
+ ctx->base.handle_result = safexcel_handle_inv_result;
+ ret = safexcel_invalidate_cache(async, &ctx->base, ctx->priv,
+ ctx->base.ctxr_dma, ring, request);
+ if (unlikely(ret))
+ return ret;
+
+ *commands = 1;
+ *results = 1;
+
+ return 0;
+}
+
+static int safexcel_ahash_exit_inv(struct crypto_tfm *tfm)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ struct ahash_request req;
+ struct safexcel_inv_result result = { 0 };
+
+ memset(&req, 0, sizeof(struct ahash_request));
+
+ /* create invalidation request */
+ init_completion(&result.completion);
+ ahash_request_set_callback(&req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ safexcel_inv_complete, &result);
+
+ ahash_request_set_tfm(&req, __crypto_ahash_cast(tfm));
+ ctx = crypto_tfm_ctx(req.base.tfm);
+ ctx->base.exit_inv = true;
+ ctx->base.send = safexcel_ahash_send_inv;
+
+ spin_lock_bh(&priv->lock);
+ crypto_enqueue_request(&priv->queue, &req.base);
+ spin_unlock_bh(&priv->lock);
+
+ if (!priv->need_dequeue)
+ safexcel_dequeue(priv);
+
+ wait_for_completion_interruptible(&result.completion);
+
+ if (result.error) {
+ dev_warn(priv->dev, "hash: completion error (%d)\n",
+ result.error);
+ return result.error;
+ }
+
+ return 0;
+}
+
+static int safexcel_ahash_cache(struct ahash_request *areq)
+{
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ int queued, cache_len;
+
+ cache_len = req->len - areq->nbytes - req->processed;
+ queued = req->len - req->processed;
+
+ /*
+ * In case there isn't enough bytes to proceed (less than a
+ * block size), cache the data until we have enough.
+ */
+ if (cache_len + areq->nbytes <= crypto_ahash_blocksize(ahash)) {
+ sg_pcopy_to_buffer(areq->src, sg_nents(areq->src),
+ req->cache + cache_len,
+ areq->nbytes, 0);
+ return areq->nbytes;
+ }
+
+ /* We could'nt cache all the data */
+ return -E2BIG;
+}
+
+static int safexcel_ahash_enqueue(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ int ret;
+
+ ctx->base.send = safexcel_ahash_send;
+
+ if (req->processed && ctx->digest == CONTEXT_CONTROL_DIGEST_PRECOMPUTED)
+ ctx->base.needs_inv = safexcel_ahash_needs_inv_get(areq);
+
+ if (ctx->base.ctxr) {
+ if (ctx->base.needs_inv)
+ ctx->base.send = safexcel_ahash_send_inv;
+ } else {
+ ctx->base.ring = safexcel_select_ring(priv);
+ ctx->base.ctxr = dma_pool_zalloc(priv->context_pool,
+ EIP197_GFP_FLAGS(areq->base),
+ &ctx->base.ctxr_dma);
+ if (!ctx->base.ctxr)
+ return -ENOMEM;
+ }
+
+ spin_lock_bh(&priv->lock);
+ ret = crypto_enqueue_request(&priv->queue, &areq->base);
+ spin_unlock_bh(&priv->lock);
+
+ if (!priv->need_dequeue)
+ safexcel_dequeue(priv);
+
+ return ret;
+}
+
+static int safexcel_ahash_update(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+
+ /* If the request is 0 length, do nothing */
+ if (!areq->nbytes)
+ return 0;
+
+ req->len += areq->nbytes;
+
+ safexcel_ahash_cache(areq);
+
+ /*
+ * We're not doing partial updates when performing an hmac request.
+ * Everything will be handled by the final() call.
+ */
+ if (ctx->digest == CONTEXT_CONTROL_DIGEST_HMAC)
+ return 0;
+
+ if (req->hmac)
+ return safexcel_ahash_enqueue(areq);
+
+ if (!req->last_req &&
+ req->len - req->processed > crypto_ahash_blocksize(ahash))
+ return safexcel_ahash_enqueue(areq);
+
+ return 0;
+}
+
+static int safexcel_ahash_final(struct ahash_request *areq)
+{
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+
+ req->last_req = true;
+ req->finish = true;
+
+ /* If we have an overall 0 length request */
+ if (!(req->len + areq->nbytes)) {
+ if (ctx->alg == CONTEXT_CONTROL_CRYPTO_ALG_SHA1)
+ memcpy(areq->result, sha1_zero_digest, SHA1_DIGEST_SIZE);
+ else if (ctx->alg == CONTEXT_CONTROL_CRYPTO_ALG_SHA224)
+ memcpy(areq->result, sha224_zero_digest, SHA224_DIGEST_SIZE);
+ else if (ctx->alg == CONTEXT_CONTROL_CRYPTO_ALG_SHA256)
+ memcpy(areq->result, sha256_zero_digest, SHA256_DIGEST_SIZE);
+
+ return 0;
+ }
+
+ return safexcel_ahash_enqueue(areq);
+}
+
+static int safexcel_ahash_finup(struct ahash_request *areq)
+{
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+
+ req->last_req = true;
+ req->finish = true;
+
+ safexcel_ahash_update(areq);
+ return safexcel_ahash_final(areq);
+}
+
+static int safexcel_ahash_export(struct ahash_request *areq, void *out)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ struct safexcel_ahash_export_state *export = out;
+
+ export->len = req->len;
+ export->processed = req->processed;
+
+ memcpy(export->state, req->state, req->state_sz);
+ memset(export->cache, 0, crypto_ahash_blocksize(ahash));
+ memcpy(export->cache, req->cache, crypto_ahash_blocksize(ahash));
+
+ return 0;
+}
+
+static int safexcel_ahash_import(struct ahash_request *areq, const void *in)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(areq);
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+ const struct safexcel_ahash_export_state *export = in;
+ int ret;
+
+ ret = crypto_ahash_init(areq);
+ if (ret)
+ return ret;
+
+ req->len = export->len;
+ req->processed = export->processed;
+
+ memcpy(req->cache, export->cache, crypto_ahash_blocksize(ahash));
+ memcpy(req->state, export->state, req->state_sz);
+
+ return 0;
+}
+
+static int safexcel_ahash_cra_init(struct crypto_tfm *tfm)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct safexcel_alg_template *tmpl =
+ container_of(__crypto_ahash_alg(tfm->__crt_alg),
+ struct safexcel_alg_template, alg.ahash);
+
+ ctx->priv = tmpl->priv;
+
+ crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
+ sizeof(struct safexcel_ahash_req));
+ return 0;
+}
+
+static int safexcel_sha1_init(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+
+ memset(req, 0, sizeof(*req));
+
+ req->state[0] = SHA1_H0;
+ req->state[1] = SHA1_H1;
+ req->state[2] = SHA1_H2;
+ req->state[3] = SHA1_H3;
+ req->state[4] = SHA1_H4;
+
+ ctx->alg = CONTEXT_CONTROL_CRYPTO_ALG_SHA1;
+ ctx->digest = CONTEXT_CONTROL_DIGEST_PRECOMPUTED;
+ req->state_sz = SHA1_DIGEST_SIZE;
+
+ return 0;
+}
+
+static int safexcel_sha1_digest(struct ahash_request *areq)
+{
+ int ret = safexcel_sha1_init(areq);
+
+ if (ret)
+ return ret;
+
+ return safexcel_ahash_finup(areq);
+}
+
+static void safexcel_ahash_cra_exit(struct crypto_tfm *tfm)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct safexcel_crypto_priv *priv = ctx->priv;
+ int ret;
+
+ /* context not allocated, skip invalidation */
+ if (!ctx->base.ctxr)
+ return;
+
+ ret = safexcel_ahash_exit_inv(tfm);
+ if (ret)
+ dev_warn(priv->dev, "hash: invalidation error %d\n", ret);
+}
+
+struct safexcel_alg_template safexcel_alg_sha1 = {
+ .type = SAFEXCEL_ALG_TYPE_AHASH,
+ .alg.ahash = {
+ .init = safexcel_sha1_init,
+ .update = safexcel_ahash_update,
+ .final = safexcel_ahash_final,
+ .finup = safexcel_ahash_finup,
+ .digest = safexcel_sha1_digest,
+ .export = safexcel_ahash_export,
+ .import = safexcel_ahash_import,
+ .halg = {
+ .digestsize = SHA1_DIGEST_SIZE,
+ .statesize = sizeof(struct safexcel_ahash_export_state),
+ .base = {
+ .cra_name = "sha1",
+ .cra_driver_name = "safexcel-sha1",
+ .cra_priority = 300,
+ .cra_flags = CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY,
+ .cra_blocksize = SHA1_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct safexcel_ahash_ctx),
+ .cra_init = safexcel_ahash_cra_init,
+ .cra_exit = safexcel_ahash_cra_exit,
+ .cra_module = THIS_MODULE,
+ },
+ },
+ },
+};
+
+static int safexcel_hmac_sha1_init(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+
+ safexcel_sha1_init(areq);
+ ctx->digest = CONTEXT_CONTROL_DIGEST_HMAC;
+ return 0;
+}
+
+static int safexcel_hmac_sha1_digest(struct ahash_request *areq)
+{
+ int ret = safexcel_hmac_sha1_init(areq);
+
+ if (ret)
+ return ret;
+
+ return safexcel_ahash_finup(areq);
+}
+
+struct safexcel_ahash_result {
+ struct completion completion;
+ int error;
+};
+
+static void safexcel_ahash_complete(struct crypto_async_request *req, int error)
+{
+ struct safexcel_ahash_result *result = req->data;
+
+ if (error == -EINPROGRESS)
+ return;
+
+ result->error = error;
+ complete(&result->completion);
+}
+
+static int safexcel_hmac_init_pad(struct ahash_request *areq,
+ unsigned int blocksize, const u8 *key,
+ unsigned int keylen, u8 *ipad, u8 *opad)
+{
+ struct safexcel_ahash_result result;
+ struct scatterlist sg;
+ int ret, i;
+ u8 *keydup;
+
+ if (keylen <= blocksize) {
+ memcpy(ipad, key, keylen);
+ } else {
+ keydup = kmemdup(key, keylen, GFP_KERNEL);
+ if (!keydup)
+ return -ENOMEM;
+
+ ahash_request_set_callback(areq, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ safexcel_ahash_complete, &result);
+ sg_init_one(&sg, keydup, keylen);
+ ahash_request_set_crypt(areq, &sg, ipad, keylen);
+ init_completion(&result.completion);
+
+ ret = crypto_ahash_digest(areq);
+ if (ret == -EINPROGRESS) {
+ wait_for_completion_interruptible(&result.completion);
+ ret = result.error;
+ }
+
+ /* Avoid leaking */
+ memset(keydup, 0, keylen);
+ kfree(keydup);
+
+ if (ret)
+ return ret;
+
+ keylen = crypto_ahash_digestsize(crypto_ahash_reqtfm(areq));
+ }
+
+ memset(ipad + keylen, 0, blocksize - keylen);
+ memcpy(opad, ipad, blocksize);
+
+ for (i = 0; i < blocksize; i++) {
+ ipad[i] ^= 0x36;
+ opad[i] ^= 0x5c;
+ }
+
+ return 0;
+}
+
+static int safexcel_hmac_init_iv(struct ahash_request *areq,
+ unsigned int blocksize, u8 *pad, void *state)
+{
+ struct safexcel_ahash_result result;
+ struct safexcel_ahash_req *req;
+ struct scatterlist sg;
+ int ret;
+
+ ahash_request_set_callback(areq, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ safexcel_ahash_complete, &result);
+ sg_init_one(&sg, pad, blocksize);
+ ahash_request_set_crypt(areq, &sg, pad, blocksize);
+ init_completion(&result.completion);
+
+ ret = crypto_ahash_init(areq);
+ if (ret)
+ return ret;
+
+ req = ahash_request_ctx(areq);
+ req->hmac = true;
+ req->last_req = true;
+
+ ret = crypto_ahash_update(areq);
+ if (ret && ret != -EINPROGRESS)
+ return ret;
+
+ wait_for_completion_interruptible(&result.completion);
+ if (result.error)
+ return result.error;
+
+ return crypto_ahash_export(areq, state);
+}
+
+static int safexcel_hmac_setkey(const char *alg, const u8 *key,
+ unsigned int keylen, void *istate, void *ostate)
+{
+ struct ahash_request *areq;
+ struct crypto_ahash *tfm;
+ unsigned int blocksize;
+ u8 *ipad, *opad;
+ int ret;
+
+ tfm = crypto_alloc_ahash(alg, CRYPTO_ALG_TYPE_AHASH,
+ CRYPTO_ALG_TYPE_AHASH_MASK);
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ areq = ahash_request_alloc(tfm, GFP_KERNEL);
+ if (!areq) {
+ ret = -ENOMEM;
+ goto free_ahash;
+ }
+
+ crypto_ahash_clear_flags(tfm, ~0);
+ blocksize = crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm));
+
+ ipad = kzalloc(2 * blocksize, GFP_KERNEL);
+ if (!ipad) {
+ ret = -ENOMEM;
+ goto free_request;
+ }
+
+ opad = ipad + blocksize;
+
+ ret = safexcel_hmac_init_pad(areq, blocksize, key, keylen, ipad, opad);
+ if (ret)
+ goto free_ipad;
+
+ ret = safexcel_hmac_init_iv(areq, blocksize, ipad, istate);
+ if (ret)
+ goto free_ipad;
+
+ ret = safexcel_hmac_init_iv(areq, blocksize, opad, ostate);
+
+free_ipad:
+ kfree(ipad);
+free_request:
+ ahash_request_free(areq);
+free_ahash:
+ crypto_free_ahash(tfm);
+
+ return ret;
+}
+
+static int safexcel_hmac_sha1_setkey(struct crypto_ahash *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_tfm_ctx(crypto_ahash_tfm(tfm));
+ struct safexcel_ahash_export_state istate, ostate;
+ int ret, i;
+
+ ret = safexcel_hmac_setkey("safexcel-sha1", key, keylen, &istate, &ostate);
+ if (ret)
+ return ret;
+
+ memcpy(ctx->ipad, &istate.state, SHA1_DIGEST_SIZE);
+ memcpy(ctx->opad, &ostate.state, SHA1_DIGEST_SIZE);
+
+ for (i = 0; i < ARRAY_SIZE(istate.state); i++) {
+ if (ctx->ipad[i] != le32_to_cpu(istate.state[i]) ||
+ ctx->opad[i] != le32_to_cpu(ostate.state[i])) {
+ ctx->base.needs_inv = true;
+ break;
+ }
+ }
+
+ return 0;
+}
+
+struct safexcel_alg_template safexcel_alg_hmac_sha1 = {
+ .type = SAFEXCEL_ALG_TYPE_AHASH,
+ .alg.ahash = {
+ .init = safexcel_hmac_sha1_init,
+ .update = safexcel_ahash_update,
+ .final = safexcel_ahash_final,
+ .finup = safexcel_ahash_finup,
+ .digest = safexcel_hmac_sha1_digest,
+ .setkey = safexcel_hmac_sha1_setkey,
+ .export = safexcel_ahash_export,
+ .import = safexcel_ahash_import,
+ .halg = {
+ .digestsize = SHA1_DIGEST_SIZE,
+ .statesize = sizeof(struct safexcel_ahash_export_state),
+ .base = {
+ .cra_name = "hmac(sha1)",
+ .cra_driver_name = "safexcel-hmac-sha1",
+ .cra_priority = 300,
+ .cra_flags = CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY,
+ .cra_blocksize = SHA1_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct safexcel_ahash_ctx),
+ .cra_init = safexcel_ahash_cra_init,
+ .cra_exit = safexcel_ahash_cra_exit,
+ .cra_module = THIS_MODULE,
+ },
+ },
+ },
+};
+
+static int safexcel_sha256_init(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+
+ memset(req, 0, sizeof(*req));
+
+ req->state[0] = SHA256_H0;
+ req->state[1] = SHA256_H1;
+ req->state[2] = SHA256_H2;
+ req->state[3] = SHA256_H3;
+ req->state[4] = SHA256_H4;
+ req->state[5] = SHA256_H5;
+ req->state[6] = SHA256_H6;
+ req->state[7] = SHA256_H7;
+
+ ctx->alg = CONTEXT_CONTROL_CRYPTO_ALG_SHA256;
+ ctx->digest = CONTEXT_CONTROL_DIGEST_PRECOMPUTED;
+ req->state_sz = SHA256_DIGEST_SIZE;
+
+ return 0;
+}
+
+static int safexcel_sha256_digest(struct ahash_request *areq)
+{
+ int ret = safexcel_sha256_init(areq);
+
+ if (ret)
+ return ret;
+
+ return safexcel_ahash_finup(areq);
+}
+
+struct safexcel_alg_template safexcel_alg_sha256 = {
+ .type = SAFEXCEL_ALG_TYPE_AHASH,
+ .alg.ahash = {
+ .init = safexcel_sha256_init,
+ .update = safexcel_ahash_update,
+ .final = safexcel_ahash_final,
+ .finup = safexcel_ahash_finup,
+ .digest = safexcel_sha256_digest,
+ .export = safexcel_ahash_export,
+ .import = safexcel_ahash_import,
+ .halg = {
+ .digestsize = SHA256_DIGEST_SIZE,
+ .statesize = sizeof(struct safexcel_ahash_export_state),
+ .base = {
+ .cra_name = "sha256",
+ .cra_driver_name = "safexcel-sha256",
+ .cra_priority = 300,
+ .cra_flags = CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY,
+ .cra_blocksize = SHA256_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct safexcel_ahash_ctx),
+ .cra_init = safexcel_ahash_cra_init,
+ .cra_exit = safexcel_ahash_cra_exit,
+ .cra_module = THIS_MODULE,
+ },
+ },
+ },
+};
+
+static int safexcel_sha224_init(struct ahash_request *areq)
+{
+ struct safexcel_ahash_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(areq));
+ struct safexcel_ahash_req *req = ahash_request_ctx(areq);
+
+ memset(req, 0, sizeof(*req));
+
+ req->state[0] = SHA224_H0;
+ req->state[1] = SHA224_H1;
+ req->state[2] = SHA224_H2;
+ req->state[3] = SHA224_H3;
+ req->state[4] = SHA224_H4;
+ req->state[5] = SHA224_H5;
+ req->state[6] = SHA224_H6;
+ req->state[7] = SHA224_H7;
+
+ ctx->alg = CONTEXT_CONTROL_CRYPTO_ALG_SHA224;
+ ctx->digest = CONTEXT_CONTROL_DIGEST_PRECOMPUTED;
+ req->state_sz = SHA256_DIGEST_SIZE;
+
+ return 0;
+}
+
+static int safexcel_sha224_digest(struct ahash_request *areq)
+{
+ int ret = safexcel_sha224_init(areq);
+
+ if (ret)
+ return ret;
+
+ return safexcel_ahash_finup(areq);
+}
+
+struct safexcel_alg_template safexcel_alg_sha224 = {
+ .type = SAFEXCEL_ALG_TYPE_AHASH,
+ .alg.ahash = {
+ .init = safexcel_sha224_init,
+ .update = safexcel_ahash_update,
+ .final = safexcel_ahash_final,
+ .finup = safexcel_ahash_finup,
+ .digest = safexcel_sha224_digest,
+ .export = safexcel_ahash_export,
+ .import = safexcel_ahash_import,
+ .halg = {
+ .digestsize = SHA224_DIGEST_SIZE,
+ .statesize = sizeof(struct safexcel_ahash_export_state),
+ .base = {
+ .cra_name = "sha224",
+ .cra_driver_name = "safexcel-sha224",
+ .cra_priority = 300,
+ .cra_flags = CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY,
+ .cra_blocksize = SHA224_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct safexcel_ahash_ctx),
+ .cra_init = safexcel_ahash_cra_init,
+ .cra_exit = safexcel_ahash_cra_exit,
+ .cra_module = THIS_MODULE,
+ },
+ },
+ },
+};
diff --git a/drivers/crypto/inside-secure/safexcel_ring.c b/drivers/crypto/inside-secure/safexcel_ring.c
new file mode 100644
index 000000000000..5b4ba7a8cc0a
--- /dev/null
+++ b/drivers/crypto/inside-secure/safexcel_ring.c
@@ -0,0 +1,173 @@
+/*
+ * Copyright (C) 2017 Marvell
+ *
+ * Antoine Tenart <antoine.tenart@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/spinlock.h>
+
+#include "safexcel.h"
+
+int safexcel_init_ring_descriptors(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *cdr,
+ struct safexcel_ring *rdr)
+{
+ cdr->offset = sizeof(u32) * priv->config.cd_offset;
+ cdr->base = dmam_alloc_coherent(priv->dev,
+ cdr->offset * EIP197_DEFAULT_RING_SIZE,
+ &cdr->base_dma, GFP_KERNEL);
+ if (!cdr->base)
+ return -ENOMEM;
+ cdr->write = cdr->base;
+ cdr->base_end = cdr->base + cdr->offset * EIP197_DEFAULT_RING_SIZE;
+ cdr->read = cdr->base;
+
+ rdr->offset = sizeof(u32) * priv->config.rd_offset;
+ rdr->base = dmam_alloc_coherent(priv->dev,
+ rdr->offset * EIP197_DEFAULT_RING_SIZE,
+ &rdr->base_dma, GFP_KERNEL);
+ if (!rdr->base) {
+ dmam_free_coherent(priv->dev,
+ cdr->offset * EIP197_DEFAULT_RING_SIZE,
+ cdr->base, cdr->base_dma);
+ return -ENOMEM;
+ }
+ rdr->write = rdr->base;
+ rdr->base_end = rdr->base + rdr->offset * EIP197_DEFAULT_RING_SIZE;
+ rdr->read = rdr->base;
+
+ return 0;
+}
+
+void safexcel_free_ring_descriptors(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *cdr,
+ struct safexcel_ring *rdr)
+{
+ dmam_free_coherent(priv->dev,
+ cdr->offset * EIP197_DEFAULT_RING_SIZE,
+ cdr->base, cdr->base_dma);
+ dmam_free_coherent(priv->dev,
+ rdr->offset * EIP197_DEFAULT_RING_SIZE,
+ rdr->base, rdr->base_dma);
+}
+
+inline int safexcel_select_ring(struct safexcel_crypto_priv *priv)
+{
+ return (atomic_inc_return(&priv->ring_used) % priv->config.rings);
+}
+
+static void *safexcel_ring_next_wptr(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *ring)
+{
+ void *ptr = ring->write;
+
+ if (ring->nr == EIP197_DEFAULT_RING_SIZE - 1)
+ return ERR_PTR(-ENOMEM);
+
+ ring->write += ring->offset;
+ if (ring->write == ring->base_end)
+ ring->write = ring->base;
+
+ ring->nr++;
+ return ptr;
+}
+
+void *safexcel_ring_next_rptr(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *ring)
+{
+ void *ptr = ring->read;
+
+ if (!ring->nr)
+ return ERR_PTR(-ENOENT);
+
+ ring->read += ring->offset;
+ if (ring->read == ring->base_end)
+ ring->read = ring->base;
+
+ ring->nr--;
+ return ptr;
+}
+
+void safexcel_ring_rollback_wptr(struct safexcel_crypto_priv *priv,
+ struct safexcel_ring *ring)
+{
+ if (!ring->nr)
+ return;
+
+ if (ring->write == ring->base)
+ ring->write += (EIP197_DEFAULT_RING_SIZE - 1) * ring->offset;
+ else
+ ring->write -= ring->offset;
+
+ ring->nr--;
+}
+
+struct safexcel_command_desc *safexcel_add_cdesc(struct safexcel_crypto_priv *priv,
+ int ring_id,
+ bool first, bool last,
+ phys_addr_t data, u32 data_len,
+ u32 full_data_len,
+ phys_addr_t context) {
+ struct safexcel_command_desc *cdesc;
+ int i;
+
+ cdesc = safexcel_ring_next_wptr(priv, &priv->ring[ring_id].cdr);
+ if (IS_ERR(cdesc))
+ return cdesc;
+
+ memset(cdesc, 0, sizeof(struct safexcel_command_desc));
+
+ cdesc->first_seg = first;
+ cdesc->last_seg = last;
+ cdesc->particle_size = data_len;
+ cdesc->data_lo = lower_32_bits(data);
+ cdesc->data_hi = upper_32_bits(data);
+
+ if (first && context) {
+ struct safexcel_token *token =
+ (struct safexcel_token *)cdesc->control_data.token;
+
+ cdesc->control_data.packet_length = full_data_len;
+ cdesc->control_data.options = EIP197_OPTION_MAGIC_VALUE |
+ EIP197_OPTION_64BIT_CTX |
+ EIP197_OPTION_CTX_CTRL_IN_CMD;
+ cdesc->control_data.context_lo =
+ (lower_32_bits(context) & GENMASK(31, 2)) >> 2;
+ cdesc->control_data.context_hi = upper_32_bits(context);
+
+ /* TODO: large xform HMAC with SHA-384/512 uses refresh = 3 */
+ cdesc->control_data.refresh = 2;
+
+ for (i = 0; i < EIP197_MAX_TOKENS; i++)
+ eip197_noop_token(&token[i]);
+ }
+
+ return cdesc;
+}
+
+struct safexcel_result_desc *safexcel_add_rdesc(struct safexcel_crypto_priv *priv,
+ int ring_id,
+ bool first, bool last,
+ phys_addr_t data, u32 len)
+{
+ struct safexcel_result_desc *rdesc;
+
+ rdesc = safexcel_ring_next_wptr(priv, &priv->ring[ring_id].rdr);
+ if (IS_ERR(rdesc))
+ return rdesc;
+
+ memset(rdesc, 0, sizeof(struct safexcel_result_desc));
+
+ rdesc->first_seg = first;
+ rdesc->last_seg = last;
+ rdesc->particle_size = len;
+ rdesc->data_lo = lower_32_bits(data);
+ rdesc->data_hi = upper_32_bits(data);
+
+ return rdesc;
+}
--
2.11.0
^ permalink raw reply related
* [PATCH 6/7] arm64: marvell: dts: enable the crypto engine on the Armada 8040 DB
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
Enable the cryptographic engine available in the CP110 master on the
Armada 8040 DB. Do not enable the one in the CP110 salve for now, as we
do not support multiple cryptographic engines yet.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
arch/arm64/boot/dts/marvell/armada-8040-db.dts | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/boot/dts/marvell/armada-8040-db.dts b/arch/arm64/boot/dts/marvell/armada-8040-db.dts
index 80e685e4235d..f6d808bced51 100644
--- a/arch/arm64/boot/dts/marvell/armada-8040-db.dts
+++ b/arch/arm64/boot/dts/marvell/armada-8040-db.dts
@@ -140,6 +140,10 @@
phy-mode = "rgmii-id";
};
+&cpm_crypto {
+ status = "okay";
+};
+
/* CON5 on CP1 expansion */
&cps_pcie2 {
status = "okay";
--
2.11.0
^ permalink raw reply related
* [PATCH 7/7] arm64: defconfig: enable the Safexcel crypto engine as a module
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
The Safexcel EIP197 cryptographic engine is used on some Marvell SoCs,
such as Armada 7040 and Armada 8040. Enable this driver as a module in
the ARM64 defconfig.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
arch/arm64/configs/defconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index fece4d3c2095..76308fa2fa7a 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -519,6 +519,7 @@ CONFIG_MEMTEST=y
CONFIG_SECURITY=y
CONFIG_CRYPTO_ECHAINIV=y
CONFIG_CRYPTO_ANSI_CPRNG=y
+CONFIG_CRYPTO_DEV_SAFEXCEL=m
CONFIG_ARM64_CRYPTO=y
CONFIG_CRYPTO_SHA1_ARM64_CE=y
CONFIG_CRYPTO_SHA2_ARM64_CE=y
--
2.11.0
^ permalink raw reply related
* [PATCH 3/7] MAINTAINERS: add a maintainer for the Inside Secure crypto driver
From: Antoine Tenart @ 2017-03-29 12:44 UTC (permalink / raw)
To: herbert, davem, jason, andrew, gregory.clement,
sebastian.hesselbarth
Cc: Antoine Tenart, linux-crypto, linux-arm-kernel, thomas.petazzoni,
boris.brezillon, oferh, igall, nadavh
In-Reply-To: <20170329124432.27457-1-antoine.tenart@free-electrons.com>
A new cryptographic engine driver was added in
drivers/crypto/inside-secure. Add myself as a maintainer for this
driver.
Signed-off-by: Antoine Tenart <antoine.tenart@free-electrons.com>
---
MAINTAINERS | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index c265a5fe4848..7240b9bca638 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6487,6 +6487,12 @@ F: Documentation/input/multi-touch-protocol.txt
F: drivers/input/input-mt.c
K: \b(ABS|SYN)_MT_
+INSIDE SECURE CRYPTO DRIVER
+M: Antoine Tenart <antoine.tenart@free-electrons.com>
+F: drivers/crypto/inside-secure/
+S: Maintained
+L: linux-crypto@vger.kernel.org
+
INTEL ASoC BDW/HSW DRIVERS
M: Jie Yang <yang.jie@linux.intel.com>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
--
2.11.0
^ permalink raw reply related
* [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Michal Suchanek @ 2017-03-29 12:56 UTC (permalink / raw)
To: Herbert Xu, David S. Miller, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman, Greg Kroah-Hartman,
Geert Uytterhoeven, Mauro Carvalho Chehab, linux-kernel,
linux-crypto, linuxppc-dev
Cc: Michal Suchanek
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 148207 bytes --]
While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
subroutines for XTS") which adds the OpenSSL license header to
drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
qestion. The whole license reads:
# Licensed under the OpenSSL license (the "License"). You may not use
# this file except in compliance with the License. You can obtain a
# copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html
#
# ====================================================================
# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
# project. The module is, however, dual licensed under OpenSSL and
# CRYPTOGAMS licenses depending on where you obtain it. For further
# details see http://www.openssl.org/~appro/cryptogams/.
# ====================================================================
After seeking legal advice it is still not clear that this driver can be
legally used in Linux. In particular the "depending on where you obtain
it" part does not make it clear when you can apply the GPL and when the
OpenSSL license.
I tried contacting the author of the code for clarification but did not
hear back. In absence of clear licensing the only solution I see is
removing this code.
Signed-off-by: Michal Suchanek <msuchanek@suse.de>
---
MAINTAINERS | 12 -
drivers/crypto/Kconfig | 8 -
drivers/crypto/Makefile | 1 -
drivers/crypto/vmx/.gitignore | 2 -
drivers/crypto/vmx/Kconfig | 9 -
drivers/crypto/vmx/Makefile | 21 -
drivers/crypto/vmx/aes.c | 150 --
drivers/crypto/vmx/aes_cbc.c | 202 --
drivers/crypto/vmx/aes_ctr.c | 191 --
drivers/crypto/vmx/aes_xts.c | 190 --
drivers/crypto/vmx/aesp8-ppc.h | 25 -
drivers/crypto/vmx/aesp8-ppc.pl | 3789 -------------------------------------
drivers/crypto/vmx/ghash.c | 227 ---
drivers/crypto/vmx/ghashp8-ppc.pl | 234 ---
drivers/crypto/vmx/ppc-xlate.pl | 228 ---
drivers/crypto/vmx/vmx.c | 88 -
16 files changed, 5377 deletions(-)
delete mode 100644 drivers/crypto/vmx/.gitignore
delete mode 100644 drivers/crypto/vmx/Kconfig
delete mode 100644 drivers/crypto/vmx/Makefile
delete mode 100644 drivers/crypto/vmx/aes.c
delete mode 100644 drivers/crypto/vmx/aes_cbc.c
delete mode 100644 drivers/crypto/vmx/aes_ctr.c
delete mode 100644 drivers/crypto/vmx/aes_xts.c
delete mode 100644 drivers/crypto/vmx/aesp8-ppc.h
delete mode 100644 drivers/crypto/vmx/aesp8-ppc.pl
delete mode 100644 drivers/crypto/vmx/ghash.c
delete mode 100644 drivers/crypto/vmx/ghashp8-ppc.pl
delete mode 100644 drivers/crypto/vmx/ppc-xlate.pl
delete mode 100644 drivers/crypto/vmx/vmx.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 1b0a87ffffab..fd4cbf046ab4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6190,18 +6190,6 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
S: Maintained
F: arch/ia64/
-IBM Power VMX Cryptographic instructions
-M: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
-M: Paulo Flabiano Smorigo <pfsmorigo@linux.vnet.ibm.com>
-L: linux-crypto@vger.kernel.org
-S: Supported
-F: drivers/crypto/vmx/Makefile
-F: drivers/crypto/vmx/Kconfig
-F: drivers/crypto/vmx/vmx.c
-F: drivers/crypto/vmx/aes*
-F: drivers/crypto/vmx/ghash*
-F: drivers/crypto/vmx/ppc-xlate.pl
-
IBM Power in-Nest Crypto Acceleration
M: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
M: Paulo Flabiano Smorigo <pfsmorigo@linux.vnet.ibm.com>
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 473d31288ad8..9fcd3af1f2f1 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -530,14 +530,6 @@ config CRYPTO_DEV_QCE
hardware. To compile this driver as a module, choose M here. The
module will be called qcrypto.
-config CRYPTO_DEV_VMX
- bool "Support for VMX cryptographic acceleration instructions"
- depends on PPC64 && VSX
- help
- Support for VMX cryptographic acceleration instructions.
-
-source "drivers/crypto/vmx/Kconfig"
-
config CRYPTO_DEV_IMGTEC_HASH
tristate "Imagination Technologies hardware hash accelerator"
depends on MIPS || COMPILE_TEST
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index 739609471169..486e57e10e7a 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -34,5 +34,4 @@ obj-$(CONFIG_CRYPTO_DEV_SUN4I_SS) += sunxi-ss/
obj-$(CONFIG_CRYPTO_DEV_TALITOS) += talitos.o
obj-$(CONFIG_CRYPTO_DEV_UX500) += ux500/
obj-$(CONFIG_CRYPTO_DEV_VIRTIO) += virtio/
-obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/
diff --git a/drivers/crypto/vmx/.gitignore b/drivers/crypto/vmx/.gitignore
deleted file mode 100644
index af4a7ce4738d..000000000000
--- a/drivers/crypto/vmx/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-aesp8-ppc.S
-ghashp8-ppc.S
diff --git a/drivers/crypto/vmx/Kconfig b/drivers/crypto/vmx/Kconfig
deleted file mode 100644
index c3d524ea6998..000000000000
--- a/drivers/crypto/vmx/Kconfig
+++ /dev/null
@@ -1,9 +0,0 @@
-config CRYPTO_DEV_VMX_ENCRYPT
- tristate "Encryption acceleration support on P8 CPU"
- depends on CRYPTO_DEV_VMX
- select CRYPTO_GHASH
- default m
- help
- Support for VMX cryptographic acceleration instructions on Power8 CPU.
- This module supports acceleration for AES and GHASH in hardware. If you
- choose 'M' here, this module will be called vmx-crypto.
diff --git a/drivers/crypto/vmx/Makefile b/drivers/crypto/vmx/Makefile
deleted file mode 100644
index 55f7c392582f..000000000000
--- a/drivers/crypto/vmx/Makefile
+++ /dev/null
@@ -1,21 +0,0 @@
-obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
-vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
-
-ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
-TARGET := linux-ppc64le
-else
-TARGET := linux-ppc64
-endif
-
-quiet_cmd_perl = PERL $@
- cmd_perl = $(PERL) $(<) $(TARGET) > $(@)
-
-targets += aesp8-ppc.S ghashp8-ppc.S
-
-$(obj)/aesp8-ppc.S: $(src)/aesp8-ppc.pl FORCE
- $(call if_changed,perl)
-
-$(obj)/ghashp8-ppc.S: $(src)/ghashp8-ppc.pl FORCE
- $(call if_changed,perl)
-
-clean-files := aesp8-ppc.S ghashp8-ppc.S
diff --git a/drivers/crypto/vmx/aes.c b/drivers/crypto/vmx/aes.c
deleted file mode 100644
index 022c7ab7351a..000000000000
--- a/drivers/crypto/vmx/aes.c
+++ /dev/null
@@ -1,150 +0,0 @@
-/**
- * AES routines supporting VMX instructions on the Power 8
- *
- * Copyright (C) 2015 International Business Machines Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; version 2 only.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- *
- * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
- */
-
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/crypto.h>
-#include <linux/delay.h>
-#include <linux/hardirq.h>
-#include <asm/switch_to.h>
-#include <crypto/aes.h>
-
-#include "aesp8-ppc.h"
-
-struct p8_aes_ctx {
- struct crypto_cipher *fallback;
- struct aes_key enc_key;
- struct aes_key dec_key;
-};
-
-static int p8_aes_init(struct crypto_tfm *tfm)
-{
- const char *alg;
- struct crypto_cipher *fallback;
- struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (!(alg = crypto_tfm_alg_name(tfm))) {
- printk(KERN_ERR "Failed to get algorithm name.\n");
- return -ENOENT;
- }
-
- fallback = crypto_alloc_cipher(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
- if (IS_ERR(fallback)) {
- printk(KERN_ERR
- "Failed to allocate transformation for '%s': %ld\n",
- alg, PTR_ERR(fallback));
- return PTR_ERR(fallback);
- }
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
-
- crypto_cipher_set_flags(fallback,
- crypto_cipher_get_flags((struct
- crypto_cipher *)
- tfm));
- ctx->fallback = fallback;
-
- return 0;
-}
-
-static void p8_aes_exit(struct crypto_tfm *tfm)
-{
- struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (ctx->fallback) {
- crypto_free_cipher(ctx->fallback);
- ctx->fallback = NULL;
- }
-}
-
-static int p8_aes_setkey(struct crypto_tfm *tfm, const u8 *key,
- unsigned int keylen)
-{
- int ret;
- struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
- ret += aes_p8_set_decrypt_key(key, keylen * 8, &ctx->dec_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
-
- ret += crypto_cipher_setkey(ctx->fallback, key, keylen);
- return ret;
-}
-
-static void p8_aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
-{
- struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (in_interrupt()) {
- crypto_cipher_encrypt_one(ctx->fallback, dst, src);
- } else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- aes_p8_encrypt(src, dst, &ctx->enc_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- }
-}
-
-static void p8_aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
-{
- struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (in_interrupt()) {
- crypto_cipher_decrypt_one(ctx->fallback, dst, src);
- } else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- aes_p8_decrypt(src, dst, &ctx->dec_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- }
-}
-
-struct crypto_alg p8_aes_alg = {
- .cra_name = "aes",
- .cra_driver_name = "p8_aes",
- .cra_module = THIS_MODULE,
- .cra_priority = 1000,
- .cra_type = NULL,
- .cra_flags = CRYPTO_ALG_TYPE_CIPHER | CRYPTO_ALG_NEED_FALLBACK,
- .cra_alignmask = 0,
- .cra_blocksize = AES_BLOCK_SIZE,
- .cra_ctxsize = sizeof(struct p8_aes_ctx),
- .cra_init = p8_aes_init,
- .cra_exit = p8_aes_exit,
- .cra_cipher = {
- .cia_min_keysize = AES_MIN_KEY_SIZE,
- .cia_max_keysize = AES_MAX_KEY_SIZE,
- .cia_setkey = p8_aes_setkey,
- .cia_encrypt = p8_aes_encrypt,
- .cia_decrypt = p8_aes_decrypt,
- },
-};
diff --git a/drivers/crypto/vmx/aes_cbc.c b/drivers/crypto/vmx/aes_cbc.c
deleted file mode 100644
index 72a26eb4e954..000000000000
--- a/drivers/crypto/vmx/aes_cbc.c
+++ /dev/null
@@ -1,202 +0,0 @@
-/**
- * AES CBC routines supporting VMX instructions on the Power 8
- *
- * Copyright (C) 2015 International Business Machines Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; version 2 only.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- *
- * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
- */
-
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/crypto.h>
-#include <linux/delay.h>
-#include <linux/hardirq.h>
-#include <asm/switch_to.h>
-#include <crypto/aes.h>
-#include <crypto/scatterwalk.h>
-#include <crypto/skcipher.h>
-
-#include "aesp8-ppc.h"
-
-struct p8_aes_cbc_ctx {
- struct crypto_skcipher *fallback;
- struct aes_key enc_key;
- struct aes_key dec_key;
-};
-
-static int p8_aes_cbc_init(struct crypto_tfm *tfm)
-{
- const char *alg;
- struct crypto_skcipher *fallback;
- struct p8_aes_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (!(alg = crypto_tfm_alg_name(tfm))) {
- printk(KERN_ERR "Failed to get algorithm name.\n");
- return -ENOENT;
- }
-
- fallback = crypto_alloc_skcipher(alg, 0,
- CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
-
- if (IS_ERR(fallback)) {
- printk(KERN_ERR
- "Failed to allocate transformation for '%s': %ld\n",
- alg, PTR_ERR(fallback));
- return PTR_ERR(fallback);
- }
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_skcipher_driver_name(fallback));
-
-
- crypto_skcipher_set_flags(
- fallback,
- crypto_skcipher_get_flags((struct crypto_skcipher *)tfm));
- ctx->fallback = fallback;
-
- return 0;
-}
-
-static void p8_aes_cbc_exit(struct crypto_tfm *tfm)
-{
- struct p8_aes_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (ctx->fallback) {
- crypto_free_skcipher(ctx->fallback);
- ctx->fallback = NULL;
- }
-}
-
-static int p8_aes_cbc_setkey(struct crypto_tfm *tfm, const u8 *key,
- unsigned int keylen)
-{
- int ret;
- struct p8_aes_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
-
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
- ret += aes_p8_set_decrypt_key(key, keylen * 8, &ctx->dec_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
-
- ret += crypto_skcipher_setkey(ctx->fallback, key, keylen);
- return ret;
-}
-
-static int p8_aes_cbc_encrypt(struct blkcipher_desc *desc,
- struct scatterlist *dst,
- struct scatterlist *src, unsigned int nbytes)
-{
- int ret;
- struct blkcipher_walk walk;
- struct p8_aes_cbc_ctx *ctx =
- crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
-
- if (in_interrupt()) {
- SKCIPHER_REQUEST_ON_STACK(req, ctx->fallback);
- skcipher_request_set_tfm(req, ctx->fallback);
- skcipher_request_set_callback(req, desc->flags, NULL, NULL);
- skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
- ret = crypto_skcipher_encrypt(req);
- skcipher_request_zero(req);
- } else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
-
- blkcipher_walk_init(&walk, dst, src, nbytes);
- ret = blkcipher_walk_virt(desc, &walk);
- while ((nbytes = walk.nbytes)) {
- aes_p8_cbc_encrypt(walk.src.virt.addr,
- walk.dst.virt.addr,
- nbytes & AES_BLOCK_MASK,
- &ctx->enc_key, walk.iv, 1);
- nbytes &= AES_BLOCK_SIZE - 1;
- ret = blkcipher_walk_done(desc, &walk, nbytes);
- }
-
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- }
-
- return ret;
-}
-
-static int p8_aes_cbc_decrypt(struct blkcipher_desc *desc,
- struct scatterlist *dst,
- struct scatterlist *src, unsigned int nbytes)
-{
- int ret;
- struct blkcipher_walk walk;
- struct p8_aes_cbc_ctx *ctx =
- crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
-
- if (in_interrupt()) {
- SKCIPHER_REQUEST_ON_STACK(req, ctx->fallback);
- skcipher_request_set_tfm(req, ctx->fallback);
- skcipher_request_set_callback(req, desc->flags, NULL, NULL);
- skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
- ret = crypto_skcipher_decrypt(req);
- skcipher_request_zero(req);
- } else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
-
- blkcipher_walk_init(&walk, dst, src, nbytes);
- ret = blkcipher_walk_virt(desc, &walk);
- while ((nbytes = walk.nbytes)) {
- aes_p8_cbc_encrypt(walk.src.virt.addr,
- walk.dst.virt.addr,
- nbytes & AES_BLOCK_MASK,
- &ctx->dec_key, walk.iv, 0);
- nbytes &= AES_BLOCK_SIZE - 1;
- ret = blkcipher_walk_done(desc, &walk, nbytes);
- }
-
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- }
-
- return ret;
-}
-
-
-struct crypto_alg p8_aes_cbc_alg = {
- .cra_name = "cbc(aes)",
- .cra_driver_name = "p8_aes_cbc",
- .cra_module = THIS_MODULE,
- .cra_priority = 2000,
- .cra_type = &crypto_blkcipher_type,
- .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER | CRYPTO_ALG_NEED_FALLBACK,
- .cra_alignmask = 0,
- .cra_blocksize = AES_BLOCK_SIZE,
- .cra_ctxsize = sizeof(struct p8_aes_cbc_ctx),
- .cra_init = p8_aes_cbc_init,
- .cra_exit = p8_aes_cbc_exit,
- .cra_blkcipher = {
- .ivsize = AES_BLOCK_SIZE,
- .min_keysize = AES_MIN_KEY_SIZE,
- .max_keysize = AES_MAX_KEY_SIZE,
- .setkey = p8_aes_cbc_setkey,
- .encrypt = p8_aes_cbc_encrypt,
- .decrypt = p8_aes_cbc_decrypt,
- },
-};
diff --git a/drivers/crypto/vmx/aes_ctr.c b/drivers/crypto/vmx/aes_ctr.c
deleted file mode 100644
index 7cf6d31c1123..000000000000
--- a/drivers/crypto/vmx/aes_ctr.c
+++ /dev/null
@@ -1,191 +0,0 @@
-/**
- * AES CTR routines supporting VMX instructions on the Power 8
- *
- * Copyright (C) 2015 International Business Machines Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; version 2 only.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- *
- * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
- */
-
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/crypto.h>
-#include <linux/delay.h>
-#include <linux/hardirq.h>
-#include <asm/switch_to.h>
-#include <crypto/aes.h>
-#include <crypto/scatterwalk.h>
-#include "aesp8-ppc.h"
-
-struct p8_aes_ctr_ctx {
- struct crypto_blkcipher *fallback;
- struct aes_key enc_key;
-};
-
-static int p8_aes_ctr_init(struct crypto_tfm *tfm)
-{
- const char *alg;
- struct crypto_blkcipher *fallback;
- struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (!(alg = crypto_tfm_alg_name(tfm))) {
- printk(KERN_ERR "Failed to get algorithm name.\n");
- return -ENOENT;
- }
-
- fallback =
- crypto_alloc_blkcipher(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
- if (IS_ERR(fallback)) {
- printk(KERN_ERR
- "Failed to allocate transformation for '%s': %ld\n",
- alg, PTR_ERR(fallback));
- return PTR_ERR(fallback);
- }
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
-
- crypto_blkcipher_set_flags(
- fallback,
- crypto_blkcipher_get_flags((struct crypto_blkcipher *)tfm));
- ctx->fallback = fallback;
-
- return 0;
-}
-
-static void p8_aes_ctr_exit(struct crypto_tfm *tfm)
-{
- struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (ctx->fallback) {
- crypto_free_blkcipher(ctx->fallback);
- ctx->fallback = NULL;
- }
-}
-
-static int p8_aes_ctr_setkey(struct crypto_tfm *tfm, const u8 *key,
- unsigned int keylen)
-{
- int ret;
- struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
-
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
-
- ret += crypto_blkcipher_setkey(ctx->fallback, key, keylen);
- return ret;
-}
-
-static void p8_aes_ctr_final(struct p8_aes_ctr_ctx *ctx,
- struct blkcipher_walk *walk)
-{
- u8 *ctrblk = walk->iv;
- u8 keystream[AES_BLOCK_SIZE];
- u8 *src = walk->src.virt.addr;
- u8 *dst = walk->dst.virt.addr;
- unsigned int nbytes = walk->nbytes;
-
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- aes_p8_encrypt(ctrblk, keystream, &ctx->enc_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
-
- crypto_xor(keystream, src, nbytes);
- memcpy(dst, keystream, nbytes);
- crypto_inc(ctrblk, AES_BLOCK_SIZE);
-}
-
-static int p8_aes_ctr_crypt(struct blkcipher_desc *desc,
- struct scatterlist *dst,
- struct scatterlist *src, unsigned int nbytes)
-{
- int ret;
- u64 inc;
- struct blkcipher_walk walk;
- struct p8_aes_ctr_ctx *ctx =
- crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
- struct blkcipher_desc fallback_desc = {
- .tfm = ctx->fallback,
- .info = desc->info,
- .flags = desc->flags
- };
-
- if (in_interrupt()) {
- ret = crypto_blkcipher_encrypt(&fallback_desc, dst, src,
- nbytes);
- } else {
- blkcipher_walk_init(&walk, dst, src, nbytes);
- ret = blkcipher_walk_virt_block(desc, &walk, AES_BLOCK_SIZE);
- while ((nbytes = walk.nbytes) >= AES_BLOCK_SIZE) {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- aes_p8_ctr32_encrypt_blocks(walk.src.virt.addr,
- walk.dst.virt.addr,
- (nbytes &
- AES_BLOCK_MASK) /
- AES_BLOCK_SIZE,
- &ctx->enc_key,
- walk.iv);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
-
- /* We need to update IV mostly for last bytes/round */
- inc = (nbytes & AES_BLOCK_MASK) / AES_BLOCK_SIZE;
- if (inc > 0)
- while (inc--)
- crypto_inc(walk.iv, AES_BLOCK_SIZE);
-
- nbytes &= AES_BLOCK_SIZE - 1;
- ret = blkcipher_walk_done(desc, &walk, nbytes);
- }
- if (walk.nbytes) {
- p8_aes_ctr_final(ctx, &walk);
- ret = blkcipher_walk_done(desc, &walk, 0);
- }
- }
-
- return ret;
-}
-
-struct crypto_alg p8_aes_ctr_alg = {
- .cra_name = "ctr(aes)",
- .cra_driver_name = "p8_aes_ctr",
- .cra_module = THIS_MODULE,
- .cra_priority = 2000,
- .cra_type = &crypto_blkcipher_type,
- .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER | CRYPTO_ALG_NEED_FALLBACK,
- .cra_alignmask = 0,
- .cra_blocksize = 1,
- .cra_ctxsize = sizeof(struct p8_aes_ctr_ctx),
- .cra_init = p8_aes_ctr_init,
- .cra_exit = p8_aes_ctr_exit,
- .cra_blkcipher = {
- .ivsize = AES_BLOCK_SIZE,
- .min_keysize = AES_MIN_KEY_SIZE,
- .max_keysize = AES_MAX_KEY_SIZE,
- .setkey = p8_aes_ctr_setkey,
- .encrypt = p8_aes_ctr_crypt,
- .decrypt = p8_aes_ctr_crypt,
- },
-};
diff --git a/drivers/crypto/vmx/aes_xts.c b/drivers/crypto/vmx/aes_xts.c
deleted file mode 100644
index 6adc9290557a..000000000000
--- a/drivers/crypto/vmx/aes_xts.c
+++ /dev/null
@@ -1,190 +0,0 @@
-/**
- * AES XTS routines supporting VMX In-core instructions on Power 8
- *
- * Copyright (C) 2015 International Business Machines Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundations; version 2 only.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY of FITNESS FOR A PARTICUPAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- *
- * Author: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
- */
-
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/crypto.h>
-#include <linux/delay.h>
-#include <linux/hardirq.h>
-#include <asm/switch_to.h>
-#include <crypto/aes.h>
-#include <crypto/scatterwalk.h>
-#include <crypto/xts.h>
-#include <crypto/skcipher.h>
-
-#include "aesp8-ppc.h"
-
-struct p8_aes_xts_ctx {
- struct crypto_skcipher *fallback;
- struct aes_key enc_key;
- struct aes_key dec_key;
- struct aes_key tweak_key;
-};
-
-static int p8_aes_xts_init(struct crypto_tfm *tfm)
-{
- const char *alg;
- struct crypto_skcipher *fallback;
- struct p8_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (!(alg = crypto_tfm_alg_name(tfm))) {
- printk(KERN_ERR "Failed to get algorithm name.\n");
- return -ENOENT;
- }
-
- fallback = crypto_alloc_skcipher(alg, 0,
- CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
- if (IS_ERR(fallback)) {
- printk(KERN_ERR
- "Failed to allocate transformation for '%s': %ld\n",
- alg, PTR_ERR(fallback));
- return PTR_ERR(fallback);
- }
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_skcipher_driver_name(fallback));
-
- crypto_skcipher_set_flags(
- fallback,
- crypto_skcipher_get_flags((struct crypto_skcipher *)tfm));
- ctx->fallback = fallback;
-
- return 0;
-}
-
-static void p8_aes_xts_exit(struct crypto_tfm *tfm)
-{
- struct p8_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (ctx->fallback) {
- crypto_free_skcipher(ctx->fallback);
- ctx->fallback = NULL;
- }
-}
-
-static int p8_aes_xts_setkey(struct crypto_tfm *tfm, const u8 *key,
- unsigned int keylen)
-{
- int ret;
- struct p8_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
-
- ret = xts_check_key(tfm, key, keylen);
- if (ret)
- return ret;
-
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- ret = aes_p8_set_encrypt_key(key + keylen/2, (keylen/2) * 8, &ctx->tweak_key);
- ret += aes_p8_set_encrypt_key(key, (keylen/2) * 8, &ctx->enc_key);
- ret += aes_p8_set_decrypt_key(key, (keylen/2) * 8, &ctx->dec_key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
-
- ret += crypto_skcipher_setkey(ctx->fallback, key, keylen);
- return ret;
-}
-
-static int p8_aes_xts_crypt(struct blkcipher_desc *desc,
- struct scatterlist *dst,
- struct scatterlist *src,
- unsigned int nbytes, int enc)
-{
- int ret;
- u8 tweak[AES_BLOCK_SIZE];
- u8 *iv;
- struct blkcipher_walk walk;
- struct p8_aes_xts_ctx *ctx =
- crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
-
- if (in_interrupt()) {
- SKCIPHER_REQUEST_ON_STACK(req, ctx->fallback);
- skcipher_request_set_tfm(req, ctx->fallback);
- skcipher_request_set_callback(req, desc->flags, NULL, NULL);
- skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
- ret = enc? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req);
- skcipher_request_zero(req);
- } else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
-
- blkcipher_walk_init(&walk, dst, src, nbytes);
-
- ret = blkcipher_walk_virt(desc, &walk);
- iv = walk.iv;
- memset(tweak, 0, AES_BLOCK_SIZE);
- aes_p8_encrypt(iv, tweak, &ctx->tweak_key);
-
- while ((nbytes = walk.nbytes)) {
- if (enc)
- aes_p8_xts_encrypt(walk.src.virt.addr, walk.dst.virt.addr,
- nbytes & AES_BLOCK_MASK, &ctx->enc_key, NULL, tweak);
- else
- aes_p8_xts_decrypt(walk.src.virt.addr, walk.dst.virt.addr,
- nbytes & AES_BLOCK_MASK, &ctx->dec_key, NULL, tweak);
-
- nbytes &= AES_BLOCK_SIZE - 1;
- ret = blkcipher_walk_done(desc, &walk, nbytes);
- }
-
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- }
- return ret;
-}
-
-static int p8_aes_xts_encrypt(struct blkcipher_desc *desc,
- struct scatterlist *dst,
- struct scatterlist *src, unsigned int nbytes)
-{
- return p8_aes_xts_crypt(desc, dst, src, nbytes, 1);
-}
-
-static int p8_aes_xts_decrypt(struct blkcipher_desc *desc,
- struct scatterlist *dst,
- struct scatterlist *src, unsigned int nbytes)
-{
- return p8_aes_xts_crypt(desc, dst, src, nbytes, 0);
-}
-
-struct crypto_alg p8_aes_xts_alg = {
- .cra_name = "xts(aes)",
- .cra_driver_name = "p8_aes_xts",
- .cra_module = THIS_MODULE,
- .cra_priority = 2000,
- .cra_type = &crypto_blkcipher_type,
- .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER | CRYPTO_ALG_NEED_FALLBACK,
- .cra_alignmask = 0,
- .cra_blocksize = AES_BLOCK_SIZE,
- .cra_ctxsize = sizeof(struct p8_aes_xts_ctx),
- .cra_init = p8_aes_xts_init,
- .cra_exit = p8_aes_xts_exit,
- .cra_blkcipher = {
- .ivsize = AES_BLOCK_SIZE,
- .min_keysize = 2 * AES_MIN_KEY_SIZE,
- .max_keysize = 2 * AES_MAX_KEY_SIZE,
- .setkey = p8_aes_xts_setkey,
- .encrypt = p8_aes_xts_encrypt,
- .decrypt = p8_aes_xts_decrypt,
- }
-};
diff --git a/drivers/crypto/vmx/aesp8-ppc.h b/drivers/crypto/vmx/aesp8-ppc.h
deleted file mode 100644
index 01972e16a6c0..000000000000
--- a/drivers/crypto/vmx/aesp8-ppc.h
+++ /dev/null
@@ -1,25 +0,0 @@
-#include <linux/types.h>
-#include <crypto/aes.h>
-
-#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE-1))
-
-struct aes_key {
- u8 key[AES_MAX_KEYLENGTH];
- int rounds;
-};
-
-int aes_p8_set_encrypt_key(const u8 *userKey, const int bits,
- struct aes_key *key);
-int aes_p8_set_decrypt_key(const u8 *userKey, const int bits,
- struct aes_key *key);
-void aes_p8_encrypt(const u8 *in, u8 *out, const struct aes_key *key);
-void aes_p8_decrypt(const u8 *in, u8 *out, const struct aes_key *key);
-void aes_p8_cbc_encrypt(const u8 *in, u8 *out, size_t len,
- const struct aes_key *key, u8 *iv, const int enc);
-void aes_p8_ctr32_encrypt_blocks(const u8 *in, u8 *out,
- size_t len, const struct aes_key *key,
- const u8 *iv);
-void aes_p8_xts_encrypt(const u8 *in, u8 *out, size_t len,
- const struct aes_key *key1, const struct aes_key *key2, u8 *iv);
-void aes_p8_xts_decrypt(const u8 *in, u8 *out, size_t len,
- const struct aes_key *key1, const struct aes_key *key2, u8 *iv);
diff --git a/drivers/crypto/vmx/aesp8-ppc.pl b/drivers/crypto/vmx/aesp8-ppc.pl
deleted file mode 100644
index 0b4a293b8a1e..000000000000
--- a/drivers/crypto/vmx/aesp8-ppc.pl
+++ /dev/null
@@ -1,3789 +0,0 @@
-#! /usr/bin/env perl
-# Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved.
-#
-# Licensed under the OpenSSL license (the "License"). You may not use
-# this file except in compliance with the License. You can obtain a copy
-# in the file LICENSE in the source distribution or at
-# https://www.openssl.org/source/license.html
-
-#
-# ====================================================================
-# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
-# project. The module is, however, dual licensed under OpenSSL and
-# CRYPTOGAMS licenses depending on where you obtain it. For further
-# details see http://www.openssl.org/~appro/cryptogams/.
-# ====================================================================
-#
-# This module implements support for AES instructions as per PowerISA
-# specification version 2.07, first implemented by POWER8 processor.
-# The module is endian-agnostic in sense that it supports both big-
-# and little-endian cases. Data alignment in parallelizable modes is
-# handled with VSX loads and stores, which implies MSR.VSX flag being
-# set. It should also be noted that ISA specification doesn't prohibit
-# alignment exceptions for these instructions on page boundaries.
-# Initially alignment was handled in pure AltiVec/VMX way [when data
-# is aligned programmatically, which in turn guarantees exception-
-# free execution], but it turned to hamper performance when vcipher
-# instructions are interleaved. It's reckoned that eventual
-# misalignment penalties at page boundaries are in average lower
-# than additional overhead in pure AltiVec approach.
-#
-# May 2016
-#
-# Add XTS subroutine, 9x on little- and 12x improvement on big-endian
-# systems were measured.
-#
-######################################################################
-# Current large-block performance in cycles per byte processed with
-# 128-bit key (less is better).
-#
-# CBC en-/decrypt CTR XTS
-# POWER8[le] 3.96/0.72 0.74 1.1
-# POWER8[be] 3.75/0.65 0.66 1.0
-
-$flavour = shift;
-
-if ($flavour =~ /64/) {
- $SIZE_T =8;
- $LRSAVE =2*$SIZE_T;
- $STU ="stdu";
- $POP ="ld";
- $PUSH ="std";
- $UCMP ="cmpld";
- $SHL ="sldi";
-} elsif ($flavour =~ /32/) {
- $SIZE_T =4;
- $LRSAVE =$SIZE_T;
- $STU ="stwu";
- $POP ="lwz";
- $PUSH ="stw";
- $UCMP ="cmplw";
- $SHL ="slwi";
-} else { die "nonsense $flavour"; }
-
-$LITTLE_ENDIAN = ($flavour=~/le$/) ? $SIZE_T : 0;
-
-$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
-( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or
-( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or
-die "can't locate ppc-xlate.pl";
-
-open STDOUT,"| $^X $xlate $flavour ".shift || die "can't call $xlate: $!";
-
-$FRAME=8*$SIZE_T;
-$prefix="aes_p8";
-
-$sp="r1";
-$vrsave="r12";
-
-#########################################################################
-{{{ # Key setup procedures #
-my ($inp,$bits,$out,$ptr,$cnt,$rounds)=map("r$_",(3..8));
-my ($zero,$in0,$in1,$key,$rcon,$mask,$tmp)=map("v$_",(0..6));
-my ($stage,$outperm,$outmask,$outhead,$outtail)=map("v$_",(7..11));
-
-$code.=<<___;
-.machine "any"
-
-.text
-
-.align 7
-rcon:
-.long 0x01000000, 0x01000000, 0x01000000, 0x01000000 ?rev
-.long 0x1b000000, 0x1b000000, 0x1b000000, 0x1b000000 ?rev
-.long 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c ?rev
-.long 0,0,0,0 ?asis
-Lconsts:
- mflr r0
- bcl 20,31,\$+4
- mflr $ptr #vvvvv "distance between . and rcon
- addi $ptr,$ptr,-0x48
- mtlr r0
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,0,0
-.asciz "AES for PowerISA 2.07, CRYPTOGAMS by <appro\@openssl.org>"
-
-.globl .${prefix}_set_encrypt_key
-Lset_encrypt_key:
- mflr r11
- $PUSH r11,$LRSAVE($sp)
-
- li $ptr,-1
- ${UCMP}i $inp,0
- beq- Lenc_key_abort # if ($inp==0) return -1;
- ${UCMP}i $out,0
- beq- Lenc_key_abort # if ($out==0) return -1;
- li $ptr,-2
- cmpwi $bits,128
- blt- Lenc_key_abort
- cmpwi $bits,256
- bgt- Lenc_key_abort
- andi. r0,$bits,0x3f
- bne- Lenc_key_abort
-
- lis r0,0xfff0
- mfspr $vrsave,256
- mtspr 256,r0
-
- bl Lconsts
- mtlr r11
-
- neg r9,$inp
- lvx $in0,0,$inp
- addi $inp,$inp,15 # 15 is not typo
- lvsr $key,0,r9 # borrow $key
- li r8,0x20
- cmpwi $bits,192
- lvx $in1,0,$inp
- le?vspltisb $mask,0x0f # borrow $mask
- lvx $rcon,0,$ptr
- le?vxor $key,$key,$mask # adjust for byte swap
- lvx $mask,r8,$ptr
- addi $ptr,$ptr,0x10
- vperm $in0,$in0,$in1,$key # align [and byte swap in LE]
- li $cnt,8
- vxor $zero,$zero,$zero
- mtctr $cnt
-
- ?lvsr $outperm,0,$out
- vspltisb $outmask,-1
- lvx $outhead,0,$out
- ?vperm $outmask,$zero,$outmask,$outperm
-
- blt Loop128
- addi $inp,$inp,8
- beq L192
- addi $inp,$inp,8
- b L256
-
-.align 4
-Loop128:
- vperm $key,$in0,$in0,$mask # rotate-n-splat
- vsldoi $tmp,$zero,$in0,12 # >>32
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- vcipherlast $key,$key,$rcon
- stvx $stage,0,$out
- addi $out,$out,16
-
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vadduwm $rcon,$rcon,$rcon
- vxor $in0,$in0,$key
- bdnz Loop128
-
- lvx $rcon,0,$ptr # last two round keys
-
- vperm $key,$in0,$in0,$mask # rotate-n-splat
- vsldoi $tmp,$zero,$in0,12 # >>32
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- vcipherlast $key,$key,$rcon
- stvx $stage,0,$out
- addi $out,$out,16
-
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vadduwm $rcon,$rcon,$rcon
- vxor $in0,$in0,$key
-
- vperm $key,$in0,$in0,$mask # rotate-n-splat
- vsldoi $tmp,$zero,$in0,12 # >>32
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- vcipherlast $key,$key,$rcon
- stvx $stage,0,$out
- addi $out,$out,16
-
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vxor $in0,$in0,$key
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- stvx $stage,0,$out
-
- addi $inp,$out,15 # 15 is not typo
- addi $out,$out,0x50
-
- li $rounds,10
- b Ldone
-
-.align 4
-L192:
- lvx $tmp,0,$inp
- li $cnt,4
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- stvx $stage,0,$out
- addi $out,$out,16
- vperm $in1,$in1,$tmp,$key # align [and byte swap in LE]
- vspltisb $key,8 # borrow $key
- mtctr $cnt
- vsububm $mask,$mask,$key # adjust the mask
-
-Loop192:
- vperm $key,$in1,$in1,$mask # roate-n-splat
- vsldoi $tmp,$zero,$in0,12 # >>32
- vcipherlast $key,$key,$rcon
-
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
-
- vsldoi $stage,$zero,$in1,8
- vspltw $tmp,$in0,3
- vxor $tmp,$tmp,$in1
- vsldoi $in1,$zero,$in1,12 # >>32
- vadduwm $rcon,$rcon,$rcon
- vxor $in1,$in1,$tmp
- vxor $in0,$in0,$key
- vxor $in1,$in1,$key
- vsldoi $stage,$stage,$in0,8
-
- vperm $key,$in1,$in1,$mask # rotate-n-splat
- vsldoi $tmp,$zero,$in0,12 # >>32
- vperm $outtail,$stage,$stage,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- vcipherlast $key,$key,$rcon
- stvx $stage,0,$out
- addi $out,$out,16
-
- vsldoi $stage,$in0,$in1,8
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vperm $outtail,$stage,$stage,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- stvx $stage,0,$out
- addi $out,$out,16
-
- vspltw $tmp,$in0,3
- vxor $tmp,$tmp,$in1
- vsldoi $in1,$zero,$in1,12 # >>32
- vadduwm $rcon,$rcon,$rcon
- vxor $in1,$in1,$tmp
- vxor $in0,$in0,$key
- vxor $in1,$in1,$key
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- stvx $stage,0,$out
- addi $inp,$out,15 # 15 is not typo
- addi $out,$out,16
- bdnz Loop192
-
- li $rounds,12
- addi $out,$out,0x20
- b Ldone
-
-.align 4
-L256:
- lvx $tmp,0,$inp
- li $cnt,7
- li $rounds,14
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- stvx $stage,0,$out
- addi $out,$out,16
- vperm $in1,$in1,$tmp,$key # align [and byte swap in LE]
- mtctr $cnt
-
-Loop256:
- vperm $key,$in1,$in1,$mask # rotate-n-splat
- vsldoi $tmp,$zero,$in0,12 # >>32
- vperm $outtail,$in1,$in1,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- vcipherlast $key,$key,$rcon
- stvx $stage,0,$out
- addi $out,$out,16
-
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in0,$in0,$tmp
- vadduwm $rcon,$rcon,$rcon
- vxor $in0,$in0,$key
- vperm $outtail,$in0,$in0,$outperm # rotate
- vsel $stage,$outhead,$outtail,$outmask
- vmr $outhead,$outtail
- stvx $stage,0,$out
- addi $inp,$out,15 # 15 is not typo
- addi $out,$out,16
- bdz Ldone
-
- vspltw $key,$in0,3 # just splat
- vsldoi $tmp,$zero,$in1,12 # >>32
- vsbox $key,$key
-
- vxor $in1,$in1,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in1,$in1,$tmp
- vsldoi $tmp,$zero,$tmp,12 # >>32
- vxor $in1,$in1,$tmp
-
- vxor $in1,$in1,$key
- b Loop256
-
-.align 4
-Ldone:
- lvx $in1,0,$inp # redundant in aligned case
- vsel $in1,$outhead,$in1,$outmask
- stvx $in1,0,$inp
- li $ptr,0
- mtspr 256,$vrsave
- stw $rounds,0($out)
-
-Lenc_key_abort:
- mr r3,$ptr
- blr
- .long 0
- .byte 0,12,0x14,1,0,0,3,0
- .long 0
-.size .${prefix}_set_encrypt_key,.-.${prefix}_set_encrypt_key
-
-.globl .${prefix}_set_decrypt_key
- $STU $sp,-$FRAME($sp)
- mflr r10
- $PUSH r10,$FRAME+$LRSAVE($sp)
- bl Lset_encrypt_key
- mtlr r10
-
- cmpwi r3,0
- bne- Ldec_key_abort
-
- slwi $cnt,$rounds,4
- subi $inp,$out,240 # first round key
- srwi $rounds,$rounds,1
- add $out,$inp,$cnt # last round key
- mtctr $rounds
-
-Ldeckey:
- lwz r0, 0($inp)
- lwz r6, 4($inp)
- lwz r7, 8($inp)
- lwz r8, 12($inp)
- addi $inp,$inp,16
- lwz r9, 0($out)
- lwz r10,4($out)
- lwz r11,8($out)
- lwz r12,12($out)
- stw r0, 0($out)
- stw r6, 4($out)
- stw r7, 8($out)
- stw r8, 12($out)
- subi $out,$out,16
- stw r9, -16($inp)
- stw r10,-12($inp)
- stw r11,-8($inp)
- stw r12,-4($inp)
- bdnz Ldeckey
-
- xor r3,r3,r3 # return value
-Ldec_key_abort:
- addi $sp,$sp,$FRAME
- blr
- .long 0
- .byte 0,12,4,1,0x80,0,3,0
- .long 0
-.size .${prefix}_set_decrypt_key,.-.${prefix}_set_decrypt_key
-___
-}}}
-#########################################################################
-{{{ # Single block en- and decrypt procedures #
-sub gen_block () {
-my $dir = shift;
-my $n = $dir eq "de" ? "n" : "";
-my ($inp,$out,$key,$rounds,$idx)=map("r$_",(3..7));
-
-$code.=<<___;
-.globl .${prefix}_${dir}crypt
- lwz $rounds,240($key)
- lis r0,0xfc00
- mfspr $vrsave,256
- li $idx,15 # 15 is not typo
- mtspr 256,r0
-
- lvx v0,0,$inp
- neg r11,$out
- lvx v1,$idx,$inp
- lvsl v2,0,$inp # inpperm
- le?vspltisb v4,0x0f
- ?lvsl v3,0,r11 # outperm
- le?vxor v2,v2,v4
- li $idx,16
- vperm v0,v0,v1,v2 # align [and byte swap in LE]
- lvx v1,0,$key
- ?lvsl v5,0,$key # keyperm
- srwi $rounds,$rounds,1
- lvx v2,$idx,$key
- addi $idx,$idx,16
- subi $rounds,$rounds,1
- ?vperm v1,v1,v2,v5 # align round key
-
- vxor v0,v0,v1
- lvx v1,$idx,$key
- addi $idx,$idx,16
- mtctr $rounds
-
-Loop_${dir}c:
- ?vperm v2,v2,v1,v5
- v${n}cipher v0,v0,v2
- lvx v2,$idx,$key
- addi $idx,$idx,16
- ?vperm v1,v1,v2,v5
- v${n}cipher v0,v0,v1
- lvx v1,$idx,$key
- addi $idx,$idx,16
- bdnz Loop_${dir}c
-
- ?vperm v2,v2,v1,v5
- v${n}cipher v0,v0,v2
- lvx v2,$idx,$key
- ?vperm v1,v1,v2,v5
- v${n}cipherlast v0,v0,v1
-
- vspltisb v2,-1
- vxor v1,v1,v1
- li $idx,15 # 15 is not typo
- ?vperm v2,v1,v2,v3 # outmask
- le?vxor v3,v3,v4
- lvx v1,0,$out # outhead
- vperm v0,v0,v0,v3 # rotate [and byte swap in LE]
- vsel v1,v1,v0,v2
- lvx v4,$idx,$out
- stvx v1,0,$out
- vsel v0,v0,v4,v2
- stvx v0,$idx,$out
-
- mtspr 256,$vrsave
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,3,0
- .long 0
-.size .${prefix}_${dir}crypt,.-.${prefix}_${dir}crypt
-___
-}
-&gen_block("en");
-&gen_block("de");
-}}}
-#########################################################################
-{{{ # CBC en- and decrypt procedures #
-my ($inp,$out,$len,$key,$ivp,$enc,$rounds,$idx)=map("r$_",(3..10));
-my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3));
-my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm)=
- map("v$_",(4..10));
-$code.=<<___;
-.globl .${prefix}_cbc_encrypt
- ${UCMP}i $len,16
- bltlr-
-
- cmpwi $enc,0 # test direction
- lis r0,0xffe0
- mfspr $vrsave,256
- mtspr 256,r0
-
- li $idx,15
- vxor $rndkey0,$rndkey0,$rndkey0
- le?vspltisb $tmp,0x0f
-
- lvx $ivec,0,$ivp # load [unaligned] iv
- lvsl $inpperm,0,$ivp
- lvx $inptail,$idx,$ivp
- le?vxor $inpperm,$inpperm,$tmp
- vperm $ivec,$ivec,$inptail,$inpperm
-
- neg r11,$inp
- ?lvsl $keyperm,0,$key # prepare for unaligned key
- lwz $rounds,240($key)
-
- lvsr $inpperm,0,r11 # prepare for unaligned load
- lvx $inptail,0,$inp
- addi $inp,$inp,15 # 15 is not typo
- le?vxor $inpperm,$inpperm,$tmp
-
- ?lvsr $outperm,0,$out # prepare for unaligned store
- vspltisb $outmask,-1
- lvx $outhead,0,$out
- ?vperm $outmask,$rndkey0,$outmask,$outperm
- le?vxor $outperm,$outperm,$tmp
-
- srwi $rounds,$rounds,1
- li $idx,16
- subi $rounds,$rounds,1
- beq Lcbc_dec
-
-Lcbc_enc:
- vmr $inout,$inptail
- lvx $inptail,0,$inp
- addi $inp,$inp,16
- mtctr $rounds
- subi $len,$len,16 # len-=16
-
- lvx $rndkey0,0,$key
- vperm $inout,$inout,$inptail,$inpperm
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
- vxor $inout,$inout,$ivec
-
-Loop_cbc_enc:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipher $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
- bdnz Loop_cbc_enc
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key
- li $idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipherlast $ivec,$inout,$rndkey0
- ${UCMP}i $len,16
-
- vperm $tmp,$ivec,$ivec,$outperm
- vsel $inout,$outhead,$tmp,$outmask
- vmr $outhead,$tmp
- stvx $inout,0,$out
- addi $out,$out,16
- bge Lcbc_enc
-
- b Lcbc_done
-
-.align 4
-Lcbc_dec:
- ${UCMP}i $len,128
- bge _aesp8_cbc_decrypt8x
- vmr $tmp,$inptail
- lvx $inptail,0,$inp
- addi $inp,$inp,16
- mtctr $rounds
- subi $len,$len,16 # len-=16
-
- lvx $rndkey0,0,$key
- vperm $tmp,$tmp,$inptail,$inpperm
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$tmp,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
-
-Loop_cbc_dec:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vncipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vncipher $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
- bdnz Loop_cbc_dec
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vncipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key
- li $idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vncipherlast $inout,$inout,$rndkey0
- ${UCMP}i $len,16
-
- vxor $inout,$inout,$ivec
- vmr $ivec,$tmp
- vperm $tmp,$inout,$inout,$outperm
- vsel $inout,$outhead,$tmp,$outmask
- vmr $outhead,$tmp
- stvx $inout,0,$out
- addi $out,$out,16
- bge Lcbc_dec
-
-Lcbc_done:
- addi $out,$out,-1
- lvx $inout,0,$out # redundant in aligned case
- vsel $inout,$outhead,$inout,$outmask
- stvx $inout,0,$out
-
- neg $enc,$ivp # write [unaligned] iv
- li $idx,15 # 15 is not typo
- vxor $rndkey0,$rndkey0,$rndkey0
- vspltisb $outmask,-1
- le?vspltisb $tmp,0x0f
- ?lvsl $outperm,0,$enc
- ?vperm $outmask,$rndkey0,$outmask,$outperm
- le?vxor $outperm,$outperm,$tmp
- lvx $outhead,0,$ivp
- vperm $ivec,$ivec,$ivec,$outperm
- vsel $inout,$outhead,$ivec,$outmask
- lvx $inptail,$idx,$ivp
- stvx $inout,0,$ivp
- vsel $inout,$ivec,$inptail,$outmask
- stvx $inout,$idx,$ivp
-
- mtspr 256,$vrsave
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,6,0
- .long 0
-___
-#########################################################################
-{{ # Optimized CBC decrypt procedure #
-my $key_="r11";
-my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31));
-my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10..13));
-my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(14..21));
-my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys
- # v26-v31 last 6 round keys
-my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment
-
-$code.=<<___;
-.align 5
-_aesp8_cbc_decrypt8x:
- $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
- li r10,`$FRAME+8*16+15`
- li r11,`$FRAME+8*16+31`
- stvx v20,r10,$sp # ABI says so
- addi r10,r10,32
- stvx v21,r11,$sp
- addi r11,r11,32
- stvx v22,r10,$sp
- addi r10,r10,32
- stvx v23,r11,$sp
- addi r11,r11,32
- stvx v24,r10,$sp
- addi r10,r10,32
- stvx v25,r11,$sp
- addi r11,r11,32
- stvx v26,r10,$sp
- addi r10,r10,32
- stvx v27,r11,$sp
- addi r11,r11,32
- stvx v28,r10,$sp
- addi r10,r10,32
- stvx v29,r11,$sp
- addi r11,r11,32
- stvx v30,r10,$sp
- stvx v31,r11,$sp
- li r0,-1
- stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
- li $x10,0x10
- $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- li $x20,0x20
- $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- li $x30,0x30
- $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- li $x40,0x40
- $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- li $x50,0x50
- $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- li $x60,0x60
- $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- li $x70,0x70
- mtspr 256,r0
-
- subi $rounds,$rounds,3 # -4 in total
- subi $len,$len,128 # bias
-
- lvx $rndkey0,$x00,$key # load key schedule
- lvx v30,$x10,$key
- addi $key,$key,0x20
- lvx v31,$x00,$key
- ?vperm $rndkey0,$rndkey0,v30,$keyperm
- addi $key_,$sp,$FRAME+15
- mtctr $rounds
-
-Load_cbc_dec_key:
- ?vperm v24,v30,v31,$keyperm
- lvx v30,$x10,$key
- addi $key,$key,0x20
- stvx v24,$x00,$key_ # off-load round[1]
- ?vperm v25,v31,v30,$keyperm
- lvx v31,$x00,$key
- stvx v25,$x10,$key_ # off-load round[2]
- addi $key_,$key_,0x20
- bdnz Load_cbc_dec_key
-
- lvx v26,$x10,$key
- ?vperm v24,v30,v31,$keyperm
- lvx v27,$x20,$key
- stvx v24,$x00,$key_ # off-load round[3]
- ?vperm v25,v31,v26,$keyperm
- lvx v28,$x30,$key
- stvx v25,$x10,$key_ # off-load round[4]
- addi $key_,$sp,$FRAME+15 # rewind $key_
- ?vperm v26,v26,v27,$keyperm
- lvx v29,$x40,$key
- ?vperm v27,v27,v28,$keyperm
- lvx v30,$x50,$key
- ?vperm v28,v28,v29,$keyperm
- lvx v31,$x60,$key
- ?vperm v29,v29,v30,$keyperm
- lvx $out0,$x70,$key # borrow $out0
- ?vperm v30,v30,v31,$keyperm
- lvx v24,$x00,$key_ # pre-load round[1]
- ?vperm v31,v31,$out0,$keyperm
- lvx v25,$x10,$key_ # pre-load round[2]
-
- #lvx $inptail,0,$inp # "caller" already did this
- #addi $inp,$inp,15 # 15 is not typo
- subi $inp,$inp,15 # undo "caller"
-
- le?li $idx,8
- lvx_u $in0,$x00,$inp # load first 8 "words"
- le?lvsl $inpperm,0,$idx
- le?vspltisb $tmp,0x0f
- lvx_u $in1,$x10,$inp
- le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u
- lvx_u $in2,$x20,$inp
- le?vperm $in0,$in0,$in0,$inpperm
- lvx_u $in3,$x30,$inp
- le?vperm $in1,$in1,$in1,$inpperm
- lvx_u $in4,$x40,$inp
- le?vperm $in2,$in2,$in2,$inpperm
- vxor $out0,$in0,$rndkey0
- lvx_u $in5,$x50,$inp
- le?vperm $in3,$in3,$in3,$inpperm
- vxor $out1,$in1,$rndkey0
- lvx_u $in6,$x60,$inp
- le?vperm $in4,$in4,$in4,$inpperm
- vxor $out2,$in2,$rndkey0
- lvx_u $in7,$x70,$inp
- addi $inp,$inp,0x80
- le?vperm $in5,$in5,$in5,$inpperm
- vxor $out3,$in3,$rndkey0
- le?vperm $in6,$in6,$in6,$inpperm
- vxor $out4,$in4,$rndkey0
- le?vperm $in7,$in7,$in7,$inpperm
- vxor $out5,$in5,$rndkey0
- vxor $out6,$in6,$rndkey0
- vxor $out7,$in7,$rndkey0
-
- mtctr $rounds
- b Loop_cbc_dec8x
-.align 5
-Loop_cbc_dec8x:
- vncipher $out0,$out0,v24
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
- vncipher $out5,$out5,v24
- vncipher $out6,$out6,v24
- vncipher $out7,$out7,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vncipher $out0,$out0,v25
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- vncipher $out5,$out5,v25
- vncipher $out6,$out6,v25
- vncipher $out7,$out7,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_cbc_dec8x
-
- subic $len,$len,128 # $len-=128
- vncipher $out0,$out0,v24
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
- vncipher $out5,$out5,v24
- vncipher $out6,$out6,v24
- vncipher $out7,$out7,v24
-
- subfe. r0,r0,r0 # borrow?-1:0
- vncipher $out0,$out0,v25
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- vncipher $out5,$out5,v25
- vncipher $out6,$out6,v25
- vncipher $out7,$out7,v25
-
- and r0,r0,$len
- vncipher $out0,$out0,v26
- vncipher $out1,$out1,v26
- vncipher $out2,$out2,v26
- vncipher $out3,$out3,v26
- vncipher $out4,$out4,v26
- vncipher $out5,$out5,v26
- vncipher $out6,$out6,v26
- vncipher $out7,$out7,v26
-
- add $inp,$inp,r0 # $inp is adjusted in such
- # way that at exit from the
- # loop inX-in7 are loaded
- # with last "words"
- vncipher $out0,$out0,v27
- vncipher $out1,$out1,v27
- vncipher $out2,$out2,v27
- vncipher $out3,$out3,v27
- vncipher $out4,$out4,v27
- vncipher $out5,$out5,v27
- vncipher $out6,$out6,v27
- vncipher $out7,$out7,v27
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vncipher $out0,$out0,v28
- vncipher $out1,$out1,v28
- vncipher $out2,$out2,v28
- vncipher $out3,$out3,v28
- vncipher $out4,$out4,v28
- vncipher $out5,$out5,v28
- vncipher $out6,$out6,v28
- vncipher $out7,$out7,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
-
- vncipher $out0,$out0,v29
- vncipher $out1,$out1,v29
- vncipher $out2,$out2,v29
- vncipher $out3,$out3,v29
- vncipher $out4,$out4,v29
- vncipher $out5,$out5,v29
- vncipher $out6,$out6,v29
- vncipher $out7,$out7,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
-
- vncipher $out0,$out0,v30
- vxor $ivec,$ivec,v31 # xor with last round key
- vncipher $out1,$out1,v30
- vxor $in0,$in0,v31
- vncipher $out2,$out2,v30
- vxor $in1,$in1,v31
- vncipher $out3,$out3,v30
- vxor $in2,$in2,v31
- vncipher $out4,$out4,v30
- vxor $in3,$in3,v31
- vncipher $out5,$out5,v30
- vxor $in4,$in4,v31
- vncipher $out6,$out6,v30
- vxor $in5,$in5,v31
- vncipher $out7,$out7,v30
- vxor $in6,$in6,v31
-
- vncipherlast $out0,$out0,$ivec
- vncipherlast $out1,$out1,$in0
- lvx_u $in0,$x00,$inp # load next input block
- vncipherlast $out2,$out2,$in1
- lvx_u $in1,$x10,$inp
- vncipherlast $out3,$out3,$in2
- le?vperm $in0,$in0,$in0,$inpperm
- lvx_u $in2,$x20,$inp
- vncipherlast $out4,$out4,$in3
- le?vperm $in1,$in1,$in1,$inpperm
- lvx_u $in3,$x30,$inp
- vncipherlast $out5,$out5,$in4
- le?vperm $in2,$in2,$in2,$inpperm
- lvx_u $in4,$x40,$inp
- vncipherlast $out6,$out6,$in5
- le?vperm $in3,$in3,$in3,$inpperm
- lvx_u $in5,$x50,$inp
- vncipherlast $out7,$out7,$in6
- le?vperm $in4,$in4,$in4,$inpperm
- lvx_u $in6,$x60,$inp
- vmr $ivec,$in7
- le?vperm $in5,$in5,$in5,$inpperm
- lvx_u $in7,$x70,$inp
- addi $inp,$inp,0x80
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $in6,$in6,$in6,$inpperm
- vxor $out0,$in0,$rndkey0
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- le?vperm $in7,$in7,$in7,$inpperm
- vxor $out1,$in1,$rndkey0
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x20,$out
- vxor $out2,$in2,$rndkey0
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x30,$out
- vxor $out3,$in3,$rndkey0
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x40,$out
- vxor $out4,$in4,$rndkey0
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x50,$out
- vxor $out5,$in5,$rndkey0
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x60,$out
- vxor $out6,$in6,$rndkey0
- stvx_u $out7,$x70,$out
- addi $out,$out,0x80
- vxor $out7,$in7,$rndkey0
-
- mtctr $rounds
- beq Loop_cbc_dec8x # did $len-=128 borrow?
-
- addic. $len,$len,128
- beq Lcbc_dec8x_done
- nop
- nop
-
-Loop_cbc_dec8x_tail: # up to 7 "words" tail...
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
- vncipher $out5,$out5,v24
- vncipher $out6,$out6,v24
- vncipher $out7,$out7,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- vncipher $out5,$out5,v25
- vncipher $out6,$out6,v25
- vncipher $out7,$out7,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_cbc_dec8x_tail
-
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
- vncipher $out5,$out5,v24
- vncipher $out6,$out6,v24
- vncipher $out7,$out7,v24
-
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- vncipher $out5,$out5,v25
- vncipher $out6,$out6,v25
- vncipher $out7,$out7,v25
-
- vncipher $out1,$out1,v26
- vncipher $out2,$out2,v26
- vncipher $out3,$out3,v26
- vncipher $out4,$out4,v26
- vncipher $out5,$out5,v26
- vncipher $out6,$out6,v26
- vncipher $out7,$out7,v26
-
- vncipher $out1,$out1,v27
- vncipher $out2,$out2,v27
- vncipher $out3,$out3,v27
- vncipher $out4,$out4,v27
- vncipher $out5,$out5,v27
- vncipher $out6,$out6,v27
- vncipher $out7,$out7,v27
-
- vncipher $out1,$out1,v28
- vncipher $out2,$out2,v28
- vncipher $out3,$out3,v28
- vncipher $out4,$out4,v28
- vncipher $out5,$out5,v28
- vncipher $out6,$out6,v28
- vncipher $out7,$out7,v28
-
- vncipher $out1,$out1,v29
- vncipher $out2,$out2,v29
- vncipher $out3,$out3,v29
- vncipher $out4,$out4,v29
- vncipher $out5,$out5,v29
- vncipher $out6,$out6,v29
- vncipher $out7,$out7,v29
-
- vncipher $out1,$out1,v30
- vxor $ivec,$ivec,v31 # last round key
- vncipher $out2,$out2,v30
- vxor $in1,$in1,v31
- vncipher $out3,$out3,v30
- vxor $in2,$in2,v31
- vncipher $out4,$out4,v30
- vxor $in3,$in3,v31
- vncipher $out5,$out5,v30
- vxor $in4,$in4,v31
- vncipher $out6,$out6,v30
- vxor $in5,$in5,v31
- vncipher $out7,$out7,v30
- vxor $in6,$in6,v31
-
- cmplwi $len,32 # switch($len)
- blt Lcbc_dec8x_one
- nop
- beq Lcbc_dec8x_two
- cmplwi $len,64
- blt Lcbc_dec8x_three
- nop
- beq Lcbc_dec8x_four
- cmplwi $len,96
- blt Lcbc_dec8x_five
- nop
- beq Lcbc_dec8x_six
-
-Lcbc_dec8x_seven:
- vncipherlast $out1,$out1,$ivec
- vncipherlast $out2,$out2,$in1
- vncipherlast $out3,$out3,$in2
- vncipherlast $out4,$out4,$in3
- vncipherlast $out5,$out5,$in4
- vncipherlast $out6,$out6,$in5
- vncipherlast $out7,$out7,$in6
- vmr $ivec,$in7
-
- le?vperm $out1,$out1,$out1,$inpperm
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x00,$out
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x10,$out
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x20,$out
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x30,$out
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x40,$out
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x50,$out
- stvx_u $out7,$x60,$out
- addi $out,$out,0x70
- b Lcbc_dec8x_done
-
-.align 5
-Lcbc_dec8x_six:
- vncipherlast $out2,$out2,$ivec
- vncipherlast $out3,$out3,$in2
- vncipherlast $out4,$out4,$in3
- vncipherlast $out5,$out5,$in4
- vncipherlast $out6,$out6,$in5
- vncipherlast $out7,$out7,$in6
- vmr $ivec,$in7
-
- le?vperm $out2,$out2,$out2,$inpperm
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x00,$out
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x10,$out
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x20,$out
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x30,$out
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x40,$out
- stvx_u $out7,$x50,$out
- addi $out,$out,0x60
- b Lcbc_dec8x_done
-
-.align 5
-Lcbc_dec8x_five:
- vncipherlast $out3,$out3,$ivec
- vncipherlast $out4,$out4,$in3
- vncipherlast $out5,$out5,$in4
- vncipherlast $out6,$out6,$in5
- vncipherlast $out7,$out7,$in6
- vmr $ivec,$in7
-
- le?vperm $out3,$out3,$out3,$inpperm
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x00,$out
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x10,$out
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x20,$out
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x30,$out
- stvx_u $out7,$x40,$out
- addi $out,$out,0x50
- b Lcbc_dec8x_done
-
-.align 5
-Lcbc_dec8x_four:
- vncipherlast $out4,$out4,$ivec
- vncipherlast $out5,$out5,$in4
- vncipherlast $out6,$out6,$in5
- vncipherlast $out7,$out7,$in6
- vmr $ivec,$in7
-
- le?vperm $out4,$out4,$out4,$inpperm
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x00,$out
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x10,$out
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x20,$out
- stvx_u $out7,$x30,$out
- addi $out,$out,0x40
- b Lcbc_dec8x_done
-
-.align 5
-Lcbc_dec8x_three:
- vncipherlast $out5,$out5,$ivec
- vncipherlast $out6,$out6,$in5
- vncipherlast $out7,$out7,$in6
- vmr $ivec,$in7
-
- le?vperm $out5,$out5,$out5,$inpperm
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x00,$out
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x10,$out
- stvx_u $out7,$x20,$out
- addi $out,$out,0x30
- b Lcbc_dec8x_done
-
-.align 5
-Lcbc_dec8x_two:
- vncipherlast $out6,$out6,$ivec
- vncipherlast $out7,$out7,$in6
- vmr $ivec,$in7
-
- le?vperm $out6,$out6,$out6,$inpperm
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x00,$out
- stvx_u $out7,$x10,$out
- addi $out,$out,0x20
- b Lcbc_dec8x_done
-
-.align 5
-Lcbc_dec8x_one:
- vncipherlast $out7,$out7,$ivec
- vmr $ivec,$in7
-
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out7,0,$out
- addi $out,$out,0x10
-
-Lcbc_dec8x_done:
- le?vperm $ivec,$ivec,$ivec,$inpperm
- stvx_u $ivec,0,$ivp # write [unaligned] iv
-
- li r10,`$FRAME+15`
- li r11,`$FRAME+31`
- stvx $inpperm,r10,$sp # wipe copies of round keys
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
- stvx $inpperm,r10,$sp
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
- stvx $inpperm,r10,$sp
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
- stvx $inpperm,r10,$sp
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
-
- mtspr 256,$vrsave
- lvx v20,r10,$sp # ABI says so
- addi r10,r10,32
- lvx v21,r11,$sp
- addi r11,r11,32
- lvx v22,r10,$sp
- addi r10,r10,32
- lvx v23,r11,$sp
- addi r11,r11,32
- lvx v24,r10,$sp
- addi r10,r10,32
- lvx v25,r11,$sp
- addi r11,r11,32
- lvx v26,r10,$sp
- addi r10,r10,32
- lvx v27,r11,$sp
- addi r11,r11,32
- lvx v28,r10,$sp
- addi r10,r10,32
- lvx v29,r11,$sp
- addi r11,r11,32
- lvx v30,r10,$sp
- lvx v31,r11,$sp
- $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
- blr
- .long 0
- .byte 0,12,0x14,0,0x80,6,6,0
- .long 0
-.size .${prefix}_cbc_encrypt,.-.${prefix}_cbc_encrypt
-___
-}} }}}
-
-#########################################################################
-{{{ # CTR procedure[s] #
-my ($inp,$out,$len,$key,$ivp,$x10,$rounds,$idx)=map("r$_",(3..10));
-my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3));
-my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm,$one)=
- map("v$_",(4..11));
-my $dat=$tmp;
-
-$code.=<<___;
-.globl .${prefix}_ctr32_encrypt_blocks
- ${UCMP}i $len,1
- bltlr-
-
- lis r0,0xfff0
- mfspr $vrsave,256
- mtspr 256,r0
-
- li $idx,15
- vxor $rndkey0,$rndkey0,$rndkey0
- le?vspltisb $tmp,0x0f
-
- lvx $ivec,0,$ivp # load [unaligned] iv
- lvsl $inpperm,0,$ivp
- lvx $inptail,$idx,$ivp
- vspltisb $one,1
- le?vxor $inpperm,$inpperm,$tmp
- vperm $ivec,$ivec,$inptail,$inpperm
- vsldoi $one,$rndkey0,$one,1
-
- neg r11,$inp
- ?lvsl $keyperm,0,$key # prepare for unaligned key
- lwz $rounds,240($key)
-
- lvsr $inpperm,0,r11 # prepare for unaligned load
- lvx $inptail,0,$inp
- addi $inp,$inp,15 # 15 is not typo
- le?vxor $inpperm,$inpperm,$tmp
-
- srwi $rounds,$rounds,1
- li $idx,16
- subi $rounds,$rounds,1
-
- ${UCMP}i $len,8
- bge _aesp8_ctr32_encrypt8x
-
- ?lvsr $outperm,0,$out # prepare for unaligned store
- vspltisb $outmask,-1
- lvx $outhead,0,$out
- ?vperm $outmask,$rndkey0,$outmask,$outperm
- le?vxor $outperm,$outperm,$tmp
-
- lvx $rndkey0,0,$key
- mtctr $rounds
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$ivec,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
- b Loop_ctr32_enc
-
-.align 5
-Loop_ctr32_enc:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipher $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
- bdnz Loop_ctr32_enc
-
- vadduwm $ivec,$ivec,$one
- vmr $dat,$inptail
- lvx $inptail,0,$inp
- addi $inp,$inp,16
- subic. $len,$len,1 # blocks--
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key
- vperm $dat,$dat,$inptail,$inpperm
- li $idx,16
- ?vperm $rndkey1,$rndkey0,$rndkey1,$keyperm
- lvx $rndkey0,0,$key
- vxor $dat,$dat,$rndkey1 # last round key
- vcipherlast $inout,$inout,$dat
-
- lvx $rndkey1,$idx,$key
- addi $idx,$idx,16
- vperm $inout,$inout,$inout,$outperm
- vsel $dat,$outhead,$inout,$outmask
- mtctr $rounds
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vmr $outhead,$inout
- vxor $inout,$ivec,$rndkey0
- lvx $rndkey0,$idx,$key
- addi $idx,$idx,16
- stvx $dat,0,$out
- addi $out,$out,16
- bne Loop_ctr32_enc
-
- addi $out,$out,-1
- lvx $inout,0,$out # redundant in aligned case
- vsel $inout,$outhead,$inout,$outmask
- stvx $inout,0,$out
-
- mtspr 256,$vrsave
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,6,0
- .long 0
-___
-#########################################################################
-{{ # Optimized CTR procedure #
-my $key_="r11";
-my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31));
-my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10,12..14));
-my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(15..22));
-my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys
- # v26-v31 last 6 round keys
-my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment
-my ($two,$three,$four)=($outhead,$outperm,$outmask);
-
-$code.=<<___;
-.align 5
-_aesp8_ctr32_encrypt8x:
- $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
- li r10,`$FRAME+8*16+15`
- li r11,`$FRAME+8*16+31`
- stvx v20,r10,$sp # ABI says so
- addi r10,r10,32
- stvx v21,r11,$sp
- addi r11,r11,32
- stvx v22,r10,$sp
- addi r10,r10,32
- stvx v23,r11,$sp
- addi r11,r11,32
- stvx v24,r10,$sp
- addi r10,r10,32
- stvx v25,r11,$sp
- addi r11,r11,32
- stvx v26,r10,$sp
- addi r10,r10,32
- stvx v27,r11,$sp
- addi r11,r11,32
- stvx v28,r10,$sp
- addi r10,r10,32
- stvx v29,r11,$sp
- addi r11,r11,32
- stvx v30,r10,$sp
- stvx v31,r11,$sp
- li r0,-1
- stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
- li $x10,0x10
- $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- li $x20,0x20
- $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- li $x30,0x30
- $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- li $x40,0x40
- $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- li $x50,0x50
- $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- li $x60,0x60
- $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- li $x70,0x70
- mtspr 256,r0
-
- subi $rounds,$rounds,3 # -4 in total
-
- lvx $rndkey0,$x00,$key # load key schedule
- lvx v30,$x10,$key
- addi $key,$key,0x20
- lvx v31,$x00,$key
- ?vperm $rndkey0,$rndkey0,v30,$keyperm
- addi $key_,$sp,$FRAME+15
- mtctr $rounds
-
-Load_ctr32_enc_key:
- ?vperm v24,v30,v31,$keyperm
- lvx v30,$x10,$key
- addi $key,$key,0x20
- stvx v24,$x00,$key_ # off-load round[1]
- ?vperm v25,v31,v30,$keyperm
- lvx v31,$x00,$key
- stvx v25,$x10,$key_ # off-load round[2]
- addi $key_,$key_,0x20
- bdnz Load_ctr32_enc_key
-
- lvx v26,$x10,$key
- ?vperm v24,v30,v31,$keyperm
- lvx v27,$x20,$key
- stvx v24,$x00,$key_ # off-load round[3]
- ?vperm v25,v31,v26,$keyperm
- lvx v28,$x30,$key
- stvx v25,$x10,$key_ # off-load round[4]
- addi $key_,$sp,$FRAME+15 # rewind $key_
- ?vperm v26,v26,v27,$keyperm
- lvx v29,$x40,$key
- ?vperm v27,v27,v28,$keyperm
- lvx v30,$x50,$key
- ?vperm v28,v28,v29,$keyperm
- lvx v31,$x60,$key
- ?vperm v29,v29,v30,$keyperm
- lvx $out0,$x70,$key # borrow $out0
- ?vperm v30,v30,v31,$keyperm
- lvx v24,$x00,$key_ # pre-load round[1]
- ?vperm v31,v31,$out0,$keyperm
- lvx v25,$x10,$key_ # pre-load round[2]
-
- vadduqm $two,$one,$one
- subi $inp,$inp,15 # undo "caller"
- $SHL $len,$len,4
-
- vadduqm $out1,$ivec,$one # counter values ...
- vadduqm $out2,$ivec,$two
- vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0]
- le?li $idx,8
- vadduqm $out3,$out1,$two
- vxor $out1,$out1,$rndkey0
- le?lvsl $inpperm,0,$idx
- vadduqm $out4,$out2,$two
- vxor $out2,$out2,$rndkey0
- le?vspltisb $tmp,0x0f
- vadduqm $out5,$out3,$two
- vxor $out3,$out3,$rndkey0
- le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u
- vadduqm $out6,$out4,$two
- vxor $out4,$out4,$rndkey0
- vadduqm $out7,$out5,$two
- vxor $out5,$out5,$rndkey0
- vadduqm $ivec,$out6,$two # next counter value
- vxor $out6,$out6,$rndkey0
- vxor $out7,$out7,$rndkey0
-
- mtctr $rounds
- b Loop_ctr32_enc8x
-.align 5
-Loop_ctr32_enc8x:
- vcipher $out0,$out0,v24
- vcipher $out1,$out1,v24
- vcipher $out2,$out2,v24
- vcipher $out3,$out3,v24
- vcipher $out4,$out4,v24
- vcipher $out5,$out5,v24
- vcipher $out6,$out6,v24
- vcipher $out7,$out7,v24
-Loop_ctr32_enc8x_middle:
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vcipher $out0,$out0,v25
- vcipher $out1,$out1,v25
- vcipher $out2,$out2,v25
- vcipher $out3,$out3,v25
- vcipher $out4,$out4,v25
- vcipher $out5,$out5,v25
- vcipher $out6,$out6,v25
- vcipher $out7,$out7,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_ctr32_enc8x
-
- subic r11,$len,256 # $len-256, borrow $key_
- vcipher $out0,$out0,v24
- vcipher $out1,$out1,v24
- vcipher $out2,$out2,v24
- vcipher $out3,$out3,v24
- vcipher $out4,$out4,v24
- vcipher $out5,$out5,v24
- vcipher $out6,$out6,v24
- vcipher $out7,$out7,v24
-
- subfe r0,r0,r0 # borrow?-1:0
- vcipher $out0,$out0,v25
- vcipher $out1,$out1,v25
- vcipher $out2,$out2,v25
- vcipher $out3,$out3,v25
- vcipher $out4,$out4,v25
- vcipher $out5,$out5,v25
- vcipher $out6,$out6,v25
- vcipher $out7,$out7,v25
-
- and r0,r0,r11
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vcipher $out0,$out0,v26
- vcipher $out1,$out1,v26
- vcipher $out2,$out2,v26
- vcipher $out3,$out3,v26
- vcipher $out4,$out4,v26
- vcipher $out5,$out5,v26
- vcipher $out6,$out6,v26
- vcipher $out7,$out7,v26
- lvx v24,$x00,$key_ # re-pre-load round[1]
-
- subic $len,$len,129 # $len-=129
- vcipher $out0,$out0,v27
- addi $len,$len,1 # $len-=128 really
- vcipher $out1,$out1,v27
- vcipher $out2,$out2,v27
- vcipher $out3,$out3,v27
- vcipher $out4,$out4,v27
- vcipher $out5,$out5,v27
- vcipher $out6,$out6,v27
- vcipher $out7,$out7,v27
- lvx v25,$x10,$key_ # re-pre-load round[2]
-
- vcipher $out0,$out0,v28
- lvx_u $in0,$x00,$inp # load input
- vcipher $out1,$out1,v28
- lvx_u $in1,$x10,$inp
- vcipher $out2,$out2,v28
- lvx_u $in2,$x20,$inp
- vcipher $out3,$out3,v28
- lvx_u $in3,$x30,$inp
- vcipher $out4,$out4,v28
- lvx_u $in4,$x40,$inp
- vcipher $out5,$out5,v28
- lvx_u $in5,$x50,$inp
- vcipher $out6,$out6,v28
- lvx_u $in6,$x60,$inp
- vcipher $out7,$out7,v28
- lvx_u $in7,$x70,$inp
- addi $inp,$inp,0x80
-
- vcipher $out0,$out0,v29
- le?vperm $in0,$in0,$in0,$inpperm
- vcipher $out1,$out1,v29
- le?vperm $in1,$in1,$in1,$inpperm
- vcipher $out2,$out2,v29
- le?vperm $in2,$in2,$in2,$inpperm
- vcipher $out3,$out3,v29
- le?vperm $in3,$in3,$in3,$inpperm
- vcipher $out4,$out4,v29
- le?vperm $in4,$in4,$in4,$inpperm
- vcipher $out5,$out5,v29
- le?vperm $in5,$in5,$in5,$inpperm
- vcipher $out6,$out6,v29
- le?vperm $in6,$in6,$in6,$inpperm
- vcipher $out7,$out7,v29
- le?vperm $in7,$in7,$in7,$inpperm
-
- add $inp,$inp,r0 # $inp is adjusted in such
- # way that at exit from the
- # loop inX-in7 are loaded
- # with last "words"
- subfe. r0,r0,r0 # borrow?-1:0
- vcipher $out0,$out0,v30
- vxor $in0,$in0,v31 # xor with last round key
- vcipher $out1,$out1,v30
- vxor $in1,$in1,v31
- vcipher $out2,$out2,v30
- vxor $in2,$in2,v31
- vcipher $out3,$out3,v30
- vxor $in3,$in3,v31
- vcipher $out4,$out4,v30
- vxor $in4,$in4,v31
- vcipher $out5,$out5,v30
- vxor $in5,$in5,v31
- vcipher $out6,$out6,v30
- vxor $in6,$in6,v31
- vcipher $out7,$out7,v30
- vxor $in7,$in7,v31
-
- bne Lctr32_enc8x_break # did $len-129 borrow?
-
- vcipherlast $in0,$out0,$in0
- vcipherlast $in1,$out1,$in1
- vadduqm $out1,$ivec,$one # counter values ...
- vcipherlast $in2,$out2,$in2
- vadduqm $out2,$ivec,$two
- vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0]
- vcipherlast $in3,$out3,$in3
- vadduqm $out3,$out1,$two
- vxor $out1,$out1,$rndkey0
- vcipherlast $in4,$out4,$in4
- vadduqm $out4,$out2,$two
- vxor $out2,$out2,$rndkey0
- vcipherlast $in5,$out5,$in5
- vadduqm $out5,$out3,$two
- vxor $out3,$out3,$rndkey0
- vcipherlast $in6,$out6,$in6
- vadduqm $out6,$out4,$two
- vxor $out4,$out4,$rndkey0
- vcipherlast $in7,$out7,$in7
- vadduqm $out7,$out5,$two
- vxor $out5,$out5,$rndkey0
- le?vperm $in0,$in0,$in0,$inpperm
- vadduqm $ivec,$out6,$two # next counter value
- vxor $out6,$out6,$rndkey0
- le?vperm $in1,$in1,$in1,$inpperm
- vxor $out7,$out7,$rndkey0
- mtctr $rounds
-
- vcipher $out0,$out0,v24
- stvx_u $in0,$x00,$out
- le?vperm $in2,$in2,$in2,$inpperm
- vcipher $out1,$out1,v24
- stvx_u $in1,$x10,$out
- le?vperm $in3,$in3,$in3,$inpperm
- vcipher $out2,$out2,v24
- stvx_u $in2,$x20,$out
- le?vperm $in4,$in4,$in4,$inpperm
- vcipher $out3,$out3,v24
- stvx_u $in3,$x30,$out
- le?vperm $in5,$in5,$in5,$inpperm
- vcipher $out4,$out4,v24
- stvx_u $in4,$x40,$out
- le?vperm $in6,$in6,$in6,$inpperm
- vcipher $out5,$out5,v24
- stvx_u $in5,$x50,$out
- le?vperm $in7,$in7,$in7,$inpperm
- vcipher $out6,$out6,v24
- stvx_u $in6,$x60,$out
- vcipher $out7,$out7,v24
- stvx_u $in7,$x70,$out
- addi $out,$out,0x80
-
- b Loop_ctr32_enc8x_middle
-
-.align 5
-Lctr32_enc8x_break:
- cmpwi $len,-0x60
- blt Lctr32_enc8x_one
- nop
- beq Lctr32_enc8x_two
- cmpwi $len,-0x40
- blt Lctr32_enc8x_three
- nop
- beq Lctr32_enc8x_four
- cmpwi $len,-0x20
- blt Lctr32_enc8x_five
- nop
- beq Lctr32_enc8x_six
- cmpwi $len,0x00
- blt Lctr32_enc8x_seven
-
-Lctr32_enc8x_eight:
- vcipherlast $out0,$out0,$in0
- vcipherlast $out1,$out1,$in1
- vcipherlast $out2,$out2,$in2
- vcipherlast $out3,$out3,$in3
- vcipherlast $out4,$out4,$in4
- vcipherlast $out5,$out5,$in5
- vcipherlast $out6,$out6,$in6
- vcipherlast $out7,$out7,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x20,$out
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x30,$out
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x40,$out
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x50,$out
- le?vperm $out7,$out7,$out7,$inpperm
- stvx_u $out6,$x60,$out
- stvx_u $out7,$x70,$out
- addi $out,$out,0x80
- b Lctr32_enc8x_done
-
-.align 5
-Lctr32_enc8x_seven:
- vcipherlast $out0,$out0,$in1
- vcipherlast $out1,$out1,$in2
- vcipherlast $out2,$out2,$in3
- vcipherlast $out3,$out3,$in4
- vcipherlast $out4,$out4,$in5
- vcipherlast $out5,$out5,$in6
- vcipherlast $out6,$out6,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x20,$out
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x30,$out
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x40,$out
- le?vperm $out6,$out6,$out6,$inpperm
- stvx_u $out5,$x50,$out
- stvx_u $out6,$x60,$out
- addi $out,$out,0x70
- b Lctr32_enc8x_done
-
-.align 5
-Lctr32_enc8x_six:
- vcipherlast $out0,$out0,$in2
- vcipherlast $out1,$out1,$in3
- vcipherlast $out2,$out2,$in4
- vcipherlast $out3,$out3,$in5
- vcipherlast $out4,$out4,$in6
- vcipherlast $out5,$out5,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x20,$out
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x30,$out
- le?vperm $out5,$out5,$out5,$inpperm
- stvx_u $out4,$x40,$out
- stvx_u $out5,$x50,$out
- addi $out,$out,0x60
- b Lctr32_enc8x_done
-
-.align 5
-Lctr32_enc8x_five:
- vcipherlast $out0,$out0,$in3
- vcipherlast $out1,$out1,$in4
- vcipherlast $out2,$out2,$in5
- vcipherlast $out3,$out3,$in6
- vcipherlast $out4,$out4,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x20,$out
- le?vperm $out4,$out4,$out4,$inpperm
- stvx_u $out3,$x30,$out
- stvx_u $out4,$x40,$out
- addi $out,$out,0x50
- b Lctr32_enc8x_done
-
-.align 5
-Lctr32_enc8x_four:
- vcipherlast $out0,$out0,$in4
- vcipherlast $out1,$out1,$in5
- vcipherlast $out2,$out2,$in6
- vcipherlast $out3,$out3,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$inpperm
- stvx_u $out2,$x20,$out
- stvx_u $out3,$x30,$out
- addi $out,$out,0x40
- b Lctr32_enc8x_done
-
-.align 5
-Lctr32_enc8x_three:
- vcipherlast $out0,$out0,$in5
- vcipherlast $out1,$out1,$in6
- vcipherlast $out2,$out2,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- le?vperm $out2,$out2,$out2,$inpperm
- stvx_u $out1,$x10,$out
- stvx_u $out2,$x20,$out
- addi $out,$out,0x30
- b Lcbc_dec8x_done
-
-.align 5
-Lctr32_enc8x_two:
- vcipherlast $out0,$out0,$in6
- vcipherlast $out1,$out1,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- le?vperm $out1,$out1,$out1,$inpperm
- stvx_u $out0,$x00,$out
- stvx_u $out1,$x10,$out
- addi $out,$out,0x20
- b Lcbc_dec8x_done
-
-.align 5
-Lctr32_enc8x_one:
- vcipherlast $out0,$out0,$in7
-
- le?vperm $out0,$out0,$out0,$inpperm
- stvx_u $out0,0,$out
- addi $out,$out,0x10
-
-Lctr32_enc8x_done:
- li r10,`$FRAME+15`
- li r11,`$FRAME+31`
- stvx $inpperm,r10,$sp # wipe copies of round keys
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
- stvx $inpperm,r10,$sp
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
- stvx $inpperm,r10,$sp
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
- stvx $inpperm,r10,$sp
- addi r10,r10,32
- stvx $inpperm,r11,$sp
- addi r11,r11,32
-
- mtspr 256,$vrsave
- lvx v20,r10,$sp # ABI says so
- addi r10,r10,32
- lvx v21,r11,$sp
- addi r11,r11,32
- lvx v22,r10,$sp
- addi r10,r10,32
- lvx v23,r11,$sp
- addi r11,r11,32
- lvx v24,r10,$sp
- addi r10,r10,32
- lvx v25,r11,$sp
- addi r11,r11,32
- lvx v26,r10,$sp
- addi r10,r10,32
- lvx v27,r11,$sp
- addi r11,r11,32
- lvx v28,r10,$sp
- addi r10,r10,32
- lvx v29,r11,$sp
- addi r11,r11,32
- lvx v30,r10,$sp
- lvx v31,r11,$sp
- $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
- blr
- .long 0
- .byte 0,12,0x14,0,0x80,6,6,0
- .long 0
-.size .${prefix}_ctr32_encrypt_blocks,.-.${prefix}_ctr32_encrypt_blocks
-___
-}} }}}
-
-#########################################################################
-{{{ # XTS procedures #
-# int aes_p8_xts_[en|de]crypt(const char *inp, char *out, size_t len, #
-# const AES_KEY *key1, const AES_KEY *key2, #
-# [const] unsigned char iv[16]); #
-# If $key2 is NULL, then a "tweak chaining" mode is engaged, in which #
-# input tweak value is assumed to be encrypted already, and last tweak #
-# value, one suitable for consecutive call on same chunk of data, is #
-# written back to original buffer. In addition, in "tweak chaining" #
-# mode only complete input blocks are processed. #
-
-my ($inp,$out,$len,$key1,$key2,$ivp,$rounds,$idx) = map("r$_",(3..10));
-my ($rndkey0,$rndkey1,$inout) = map("v$_",(0..2));
-my ($output,$inptail,$inpperm,$leperm,$keyperm) = map("v$_",(3..7));
-my ($tweak,$seven,$eighty7,$tmp,$tweak1) = map("v$_",(8..12));
-my $taillen = $key2;
-
- ($inp,$idx) = ($idx,$inp); # reassign
-
-$code.=<<___;
-.globl .${prefix}_xts_encrypt
- mr $inp,r3 # reassign
- li r3,-1
- ${UCMP}i $len,16
- bltlr-
-
- lis r0,0xfff0
- mfspr r12,256 # save vrsave
- li r11,0
- mtspr 256,r0
-
- vspltisb $seven,0x07 # 0x070707..07
- le?lvsl $leperm,r11,r11
- le?vspltisb $tmp,0x0f
- le?vxor $leperm,$leperm,$seven
-
- li $idx,15
- lvx $tweak,0,$ivp # load [unaligned] iv
- lvsl $inpperm,0,$ivp
- lvx $inptail,$idx,$ivp
- le?vxor $inpperm,$inpperm,$tmp
- vperm $tweak,$tweak,$inptail,$inpperm
-
- neg r11,$inp
- lvsr $inpperm,0,r11 # prepare for unaligned load
- lvx $inout,0,$inp
- addi $inp,$inp,15 # 15 is not typo
- le?vxor $inpperm,$inpperm,$tmp
-
- ${UCMP}i $key2,0 # key2==NULL?
- beq Lxts_enc_no_key2
-
- ?lvsl $keyperm,0,$key2 # prepare for unaligned key
- lwz $rounds,240($key2)
- srwi $rounds,$rounds,1
- subi $rounds,$rounds,1
- li $idx,16
-
- lvx $rndkey0,0,$key2
- lvx $rndkey1,$idx,$key2
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $tweak,$tweak,$rndkey0
- lvx $rndkey0,$idx,$key2
- addi $idx,$idx,16
- mtctr $rounds
-
-Ltweak_xts_enc:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $tweak,$tweak,$rndkey1
- lvx $rndkey1,$idx,$key2
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipher $tweak,$tweak,$rndkey0
- lvx $rndkey0,$idx,$key2
- addi $idx,$idx,16
- bdnz Ltweak_xts_enc
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $tweak,$tweak,$rndkey1
- lvx $rndkey1,$idx,$key2
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipherlast $tweak,$tweak,$rndkey0
-
- li $ivp,0 # don't chain the tweak
- b Lxts_enc
-
-Lxts_enc_no_key2:
- li $idx,-16
- and $len,$len,$idx # in "tweak chaining"
- # mode only complete
- # blocks are processed
-Lxts_enc:
- lvx $inptail,0,$inp
- addi $inp,$inp,16
-
- ?lvsl $keyperm,0,$key1 # prepare for unaligned key
- lwz $rounds,240($key1)
- srwi $rounds,$rounds,1
- subi $rounds,$rounds,1
- li $idx,16
-
- vslb $eighty7,$seven,$seven # 0x808080..80
- vor $eighty7,$eighty7,$seven # 0x878787..87
- vspltisb $tmp,1 # 0x010101..01
- vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01
-
- ${UCMP}i $len,96
- bge _aesp8_xts_encrypt6x
-
- andi. $taillen,$len,15
- subic r0,$len,32
- subi $taillen,$taillen,16
- subfe r0,r0,r0
- and r0,r0,$taillen
- add $inp,$inp,r0
-
- lvx $rndkey0,0,$key1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
- vperm $inout,$inout,$inptail,$inpperm
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$inout,$tweak
- vxor $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
- mtctr $rounds
- b Loop_xts_enc
-
-.align 5
-Loop_xts_enc:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipher $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
- bdnz Loop_xts_enc
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key1
- li $idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $rndkey0,$rndkey0,$tweak
- vcipherlast $output,$inout,$rndkey0
-
- le?vperm $tmp,$output,$output,$leperm
- be?nop
- le?stvx_u $tmp,0,$out
- be?stvx_u $output,0,$out
- addi $out,$out,16
-
- subic. $len,$len,16
- beq Lxts_enc_done
-
- vmr $inout,$inptail
- lvx $inptail,0,$inp
- addi $inp,$inp,16
- lvx $rndkey0,0,$key1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
-
- subic r0,$len,32
- subfe r0,r0,r0
- and r0,r0,$taillen
- add $inp,$inp,r0
-
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $tweak,$tweak,$tmp
-
- vperm $inout,$inout,$inptail,$inpperm
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$inout,$tweak
- vxor $output,$output,$rndkey0 # just in case $len<16
- vxor $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
-
- mtctr $rounds
- ${UCMP}i $len,16
- bge Loop_xts_enc
-
- vxor $output,$output,$tweak
- lvsr $inpperm,0,$len # $inpperm is no longer needed
- vxor $inptail,$inptail,$inptail # $inptail is no longer needed
- vspltisb $tmp,-1
- vperm $inptail,$inptail,$tmp,$inpperm
- vsel $inout,$inout,$output,$inptail
-
- subi r11,$out,17
- subi $out,$out,16
- mtctr $len
- li $len,16
-Loop_xts_enc_steal:
- lbzu r0,1(r11)
- stb r0,16(r11)
- bdnz Loop_xts_enc_steal
-
- mtctr $rounds
- b Loop_xts_enc # one more time...
-
-Lxts_enc_done:
- ${UCMP}i $ivp,0
- beq Lxts_enc_ret
-
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $tweak,$tweak,$tmp
-
- le?vperm $tweak,$tweak,$tweak,$leperm
- stvx_u $tweak,0,$ivp
-
-Lxts_enc_ret:
- mtspr 256,r12 # restore vrsave
- li r3,0
- blr
- .long 0
- .byte 0,12,0x04,0,0x80,6,6,0
- .long 0
-.size .${prefix}_xts_encrypt,.-.${prefix}_xts_encrypt
-
-.globl .${prefix}_xts_decrypt
- mr $inp,r3 # reassign
- li r3,-1
- ${UCMP}i $len,16
- bltlr-
-
- lis r0,0xfff8
- mfspr r12,256 # save vrsave
- li r11,0
- mtspr 256,r0
-
- andi. r0,$len,15
- neg r0,r0
- andi. r0,r0,16
- sub $len,$len,r0
-
- vspltisb $seven,0x07 # 0x070707..07
- le?lvsl $leperm,r11,r11
- le?vspltisb $tmp,0x0f
- le?vxor $leperm,$leperm,$seven
-
- li $idx,15
- lvx $tweak,0,$ivp # load [unaligned] iv
- lvsl $inpperm,0,$ivp
- lvx $inptail,$idx,$ivp
- le?vxor $inpperm,$inpperm,$tmp
- vperm $tweak,$tweak,$inptail,$inpperm
-
- neg r11,$inp
- lvsr $inpperm,0,r11 # prepare for unaligned load
- lvx $inout,0,$inp
- addi $inp,$inp,15 # 15 is not typo
- le?vxor $inpperm,$inpperm,$tmp
-
- ${UCMP}i $key2,0 # key2==NULL?
- beq Lxts_dec_no_key2
-
- ?lvsl $keyperm,0,$key2 # prepare for unaligned key
- lwz $rounds,240($key2)
- srwi $rounds,$rounds,1
- subi $rounds,$rounds,1
- li $idx,16
-
- lvx $rndkey0,0,$key2
- lvx $rndkey1,$idx,$key2
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $tweak,$tweak,$rndkey0
- lvx $rndkey0,$idx,$key2
- addi $idx,$idx,16
- mtctr $rounds
-
-Ltweak_xts_dec:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $tweak,$tweak,$rndkey1
- lvx $rndkey1,$idx,$key2
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipher $tweak,$tweak,$rndkey0
- lvx $rndkey0,$idx,$key2
- addi $idx,$idx,16
- bdnz Ltweak_xts_dec
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vcipher $tweak,$tweak,$rndkey1
- lvx $rndkey1,$idx,$key2
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vcipherlast $tweak,$tweak,$rndkey0
-
- li $ivp,0 # don't chain the tweak
- b Lxts_dec
-
-Lxts_dec_no_key2:
- neg $idx,$len
- andi. $idx,$idx,15
- add $len,$len,$idx # in "tweak chaining"
- # mode only complete
- # blocks are processed
-Lxts_dec:
- lvx $inptail,0,$inp
- addi $inp,$inp,16
-
- ?lvsl $keyperm,0,$key1 # prepare for unaligned key
- lwz $rounds,240($key1)
- srwi $rounds,$rounds,1
- subi $rounds,$rounds,1
- li $idx,16
-
- vslb $eighty7,$seven,$seven # 0x808080..80
- vor $eighty7,$eighty7,$seven # 0x878787..87
- vspltisb $tmp,1 # 0x010101..01
- vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01
-
- ${UCMP}i $len,96
- bge _aesp8_xts_decrypt6x
-
- lvx $rndkey0,0,$key1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
- vperm $inout,$inout,$inptail,$inpperm
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$inout,$tweak
- vxor $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
- mtctr $rounds
-
- ${UCMP}i $len,16
- blt Ltail_xts_dec
- be?b Loop_xts_dec
-
-.align 5
-Loop_xts_dec:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vncipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vncipher $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
- bdnz Loop_xts_dec
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vncipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key1
- li $idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $rndkey0,$rndkey0,$tweak
- vncipherlast $output,$inout,$rndkey0
-
- le?vperm $tmp,$output,$output,$leperm
- be?nop
- le?stvx_u $tmp,0,$out
- be?stvx_u $output,0,$out
- addi $out,$out,16
-
- subic. $len,$len,16
- beq Lxts_dec_done
-
- vmr $inout,$inptail
- lvx $inptail,0,$inp
- addi $inp,$inp,16
- lvx $rndkey0,0,$key1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
-
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $tweak,$tweak,$tmp
-
- vperm $inout,$inout,$inptail,$inpperm
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $inout,$inout,$tweak
- vxor $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
-
- mtctr $rounds
- ${UCMP}i $len,16
- bge Loop_xts_dec
-
-Ltail_xts_dec:
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak1,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $tweak1,$tweak1,$tmp
-
- subi $inp,$inp,16
- add $inp,$inp,$len
-
- vxor $inout,$inout,$tweak # :-(
- vxor $inout,$inout,$tweak1 # :-)
-
-Loop_xts_dec_short:
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vncipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vncipher $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
- bdnz Loop_xts_dec_short
-
- ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
- vncipher $inout,$inout,$rndkey1
- lvx $rndkey1,$idx,$key1
- li $idx,16
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
- vxor $rndkey0,$rndkey0,$tweak1
- vncipherlast $output,$inout,$rndkey0
-
- le?vperm $tmp,$output,$output,$leperm
- be?nop
- le?stvx_u $tmp,0,$out
- be?stvx_u $output,0,$out
-
- vmr $inout,$inptail
- lvx $inptail,0,$inp
- #addi $inp,$inp,16
- lvx $rndkey0,0,$key1
- lvx $rndkey1,$idx,$key1
- addi $idx,$idx,16
- vperm $inout,$inout,$inptail,$inpperm
- ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
-
- lvsr $inpperm,0,$len # $inpperm is no longer needed
- vxor $inptail,$inptail,$inptail # $inptail is no longer needed
- vspltisb $tmp,-1
- vperm $inptail,$inptail,$tmp,$inpperm
- vsel $inout,$inout,$output,$inptail
-
- vxor $rndkey0,$rndkey0,$tweak
- vxor $inout,$inout,$rndkey0
- lvx $rndkey0,$idx,$key1
- addi $idx,$idx,16
-
- subi r11,$out,1
- mtctr $len
- li $len,16
-Loop_xts_dec_steal:
- lbzu r0,1(r11)
- stb r0,16(r11)
- bdnz Loop_xts_dec_steal
-
- mtctr $rounds
- b Loop_xts_dec # one more time...
-
-Lxts_dec_done:
- ${UCMP}i $ivp,0
- beq Lxts_dec_ret
-
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $tweak,$tweak,$tmp
-
- le?vperm $tweak,$tweak,$tweak,$leperm
- stvx_u $tweak,0,$ivp
-
-Lxts_dec_ret:
- mtspr 256,r12 # restore vrsave
- li r3,0
- blr
- .long 0
- .byte 0,12,0x04,0,0x80,6,6,0
- .long 0
-.size .${prefix}_xts_decrypt,.-.${prefix}_xts_decrypt
-___
-#########################################################################
-{{ # Optimized XTS procedures #
-my $key_=$key2;
-my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,3,26..31));
- $x00=0 if ($flavour =~ /osx/);
-my ($in0, $in1, $in2, $in3, $in4, $in5 )=map("v$_",(0..5));
-my ($out0, $out1, $out2, $out3, $out4, $out5)=map("v$_",(7,12..16));
-my ($twk0, $twk1, $twk2, $twk3, $twk4, $twk5)=map("v$_",(17..22));
-my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys
- # v26-v31 last 6 round keys
-my ($keyperm)=($out0); # aliases with "caller", redundant assignment
-my $taillen=$x70;
-
-$code.=<<___;
-.align 5
-_aesp8_xts_encrypt6x:
- $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
- mflr r11
- li r7,`$FRAME+8*16+15`
- li r3,`$FRAME+8*16+31`
- $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp)
- stvx v20,r7,$sp # ABI says so
- addi r7,r7,32
- stvx v21,r3,$sp
- addi r3,r3,32
- stvx v22,r7,$sp
- addi r7,r7,32
- stvx v23,r3,$sp
- addi r3,r3,32
- stvx v24,r7,$sp
- addi r7,r7,32
- stvx v25,r3,$sp
- addi r3,r3,32
- stvx v26,r7,$sp
- addi r7,r7,32
- stvx v27,r3,$sp
- addi r3,r3,32
- stvx v28,r7,$sp
- addi r7,r7,32
- stvx v29,r3,$sp
- addi r3,r3,32
- stvx v30,r7,$sp
- stvx v31,r3,$sp
- li r0,-1
- stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
- li $x10,0x10
- $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- li $x20,0x20
- $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- li $x30,0x30
- $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- li $x40,0x40
- $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- li $x50,0x50
- $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- li $x60,0x60
- $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- li $x70,0x70
- mtspr 256,r0
-
- subi $rounds,$rounds,3 # -4 in total
-
- lvx $rndkey0,$x00,$key1 # load key schedule
- lvx v30,$x10,$key1
- addi $key1,$key1,0x20
- lvx v31,$x00,$key1
- ?vperm $rndkey0,$rndkey0,v30,$keyperm
- addi $key_,$sp,$FRAME+15
- mtctr $rounds
-
-Load_xts_enc_key:
- ?vperm v24,v30,v31,$keyperm
- lvx v30,$x10,$key1
- addi $key1,$key1,0x20
- stvx v24,$x00,$key_ # off-load round[1]
- ?vperm v25,v31,v30,$keyperm
- lvx v31,$x00,$key1
- stvx v25,$x10,$key_ # off-load round[2]
- addi $key_,$key_,0x20
- bdnz Load_xts_enc_key
-
- lvx v26,$x10,$key1
- ?vperm v24,v30,v31,$keyperm
- lvx v27,$x20,$key1
- stvx v24,$x00,$key_ # off-load round[3]
- ?vperm v25,v31,v26,$keyperm
- lvx v28,$x30,$key1
- stvx v25,$x10,$key_ # off-load round[4]
- addi $key_,$sp,$FRAME+15 # rewind $key_
- ?vperm v26,v26,v27,$keyperm
- lvx v29,$x40,$key1
- ?vperm v27,v27,v28,$keyperm
- lvx v30,$x50,$key1
- ?vperm v28,v28,v29,$keyperm
- lvx v31,$x60,$key1
- ?vperm v29,v29,v30,$keyperm
- lvx $twk5,$x70,$key1 # borrow $twk5
- ?vperm v30,v30,v31,$keyperm
- lvx v24,$x00,$key_ # pre-load round[1]
- ?vperm v31,v31,$twk5,$keyperm
- lvx v25,$x10,$key_ # pre-load round[2]
-
- vperm $in0,$inout,$inptail,$inpperm
- subi $inp,$inp,31 # undo "caller"
- vxor $twk0,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $out0,$in0,$twk0
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in1,$x10,$inp
- vxor $twk1,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in1,$in1,$in1,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out1,$in1,$twk1
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in2,$x20,$inp
- andi. $taillen,$len,15
- vxor $twk2,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in2,$in2,$in2,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out2,$in2,$twk2
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in3,$x30,$inp
- sub $len,$len,$taillen
- vxor $twk3,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in3,$in3,$in3,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out3,$in3,$twk3
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in4,$x40,$inp
- subi $len,$len,0x60
- vxor $twk4,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in4,$in4,$in4,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out4,$in4,$twk4
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in5,$x50,$inp
- addi $inp,$inp,0x60
- vxor $twk5,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in5,$in5,$in5,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out5,$in5,$twk5
- vxor $tweak,$tweak,$tmp
-
- vxor v31,v31,$rndkey0
- mtctr $rounds
- b Loop_xts_enc6x
-
-.align 5
-Loop_xts_enc6x:
- vcipher $out0,$out0,v24
- vcipher $out1,$out1,v24
- vcipher $out2,$out2,v24
- vcipher $out3,$out3,v24
- vcipher $out4,$out4,v24
- vcipher $out5,$out5,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vcipher $out0,$out0,v25
- vcipher $out1,$out1,v25
- vcipher $out2,$out2,v25
- vcipher $out3,$out3,v25
- vcipher $out4,$out4,v25
- vcipher $out5,$out5,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_xts_enc6x
-
- subic $len,$len,96 # $len-=96
- vxor $in0,$twk0,v31 # xor with last round key
- vcipher $out0,$out0,v24
- vcipher $out1,$out1,v24
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk0,$tweak,$rndkey0
- vaddubm $tweak,$tweak,$tweak
- vcipher $out2,$out2,v24
- vcipher $out3,$out3,v24
- vsldoi $tmp,$tmp,$tmp,15
- vcipher $out4,$out4,v24
- vcipher $out5,$out5,v24
-
- subfe. r0,r0,r0 # borrow?-1:0
- vand $tmp,$tmp,$eighty7
- vcipher $out0,$out0,v25
- vcipher $out1,$out1,v25
- vxor $tweak,$tweak,$tmp
- vcipher $out2,$out2,v25
- vcipher $out3,$out3,v25
- vxor $in1,$twk1,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk1,$tweak,$rndkey0
- vcipher $out4,$out4,v25
- vcipher $out5,$out5,v25
-
- and r0,r0,$len
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vcipher $out0,$out0,v26
- vcipher $out1,$out1,v26
- vand $tmp,$tmp,$eighty7
- vcipher $out2,$out2,v26
- vcipher $out3,$out3,v26
- vxor $tweak,$tweak,$tmp
- vcipher $out4,$out4,v26
- vcipher $out5,$out5,v26
-
- add $inp,$inp,r0 # $inp is adjusted in such
- # way that at exit from the
- # loop inX-in5 are loaded
- # with last "words"
- vxor $in2,$twk2,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk2,$tweak,$rndkey0
- vaddubm $tweak,$tweak,$tweak
- vcipher $out0,$out0,v27
- vcipher $out1,$out1,v27
- vsldoi $tmp,$tmp,$tmp,15
- vcipher $out2,$out2,v27
- vcipher $out3,$out3,v27
- vand $tmp,$tmp,$eighty7
- vcipher $out4,$out4,v27
- vcipher $out5,$out5,v27
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vxor $tweak,$tweak,$tmp
- vcipher $out0,$out0,v28
- vcipher $out1,$out1,v28
- vxor $in3,$twk3,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk3,$tweak,$rndkey0
- vcipher $out2,$out2,v28
- vcipher $out3,$out3,v28
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vcipher $out4,$out4,v28
- vcipher $out5,$out5,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
- vand $tmp,$tmp,$eighty7
-
- vcipher $out0,$out0,v29
- vcipher $out1,$out1,v29
- vxor $tweak,$tweak,$tmp
- vcipher $out2,$out2,v29
- vcipher $out3,$out3,v29
- vxor $in4,$twk4,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk4,$tweak,$rndkey0
- vcipher $out4,$out4,v29
- vcipher $out5,$out5,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
-
- vcipher $out0,$out0,v30
- vcipher $out1,$out1,v30
- vand $tmp,$tmp,$eighty7
- vcipher $out2,$out2,v30
- vcipher $out3,$out3,v30
- vxor $tweak,$tweak,$tmp
- vcipher $out4,$out4,v30
- vcipher $out5,$out5,v30
- vxor $in5,$twk5,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk5,$tweak,$rndkey0
-
- vcipherlast $out0,$out0,$in0
- lvx_u $in0,$x00,$inp # load next input block
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vcipherlast $out1,$out1,$in1
- lvx_u $in1,$x10,$inp
- vcipherlast $out2,$out2,$in2
- le?vperm $in0,$in0,$in0,$leperm
- lvx_u $in2,$x20,$inp
- vand $tmp,$tmp,$eighty7
- vcipherlast $out3,$out3,$in3
- le?vperm $in1,$in1,$in1,$leperm
- lvx_u $in3,$x30,$inp
- vcipherlast $out4,$out4,$in4
- le?vperm $in2,$in2,$in2,$leperm
- lvx_u $in4,$x40,$inp
- vxor $tweak,$tweak,$tmp
- vcipherlast $tmp,$out5,$in5 # last block might be needed
- # in stealing mode
- le?vperm $in3,$in3,$in3,$leperm
- lvx_u $in5,$x50,$inp
- addi $inp,$inp,0x60
- le?vperm $in4,$in4,$in4,$leperm
- le?vperm $in5,$in5,$in5,$leperm
-
- le?vperm $out0,$out0,$out0,$leperm
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $out0,$in0,$twk0
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- vxor $out1,$in1,$twk1
- le?vperm $out3,$out3,$out3,$leperm
- stvx_u $out2,$x20,$out
- vxor $out2,$in2,$twk2
- le?vperm $out4,$out4,$out4,$leperm
- stvx_u $out3,$x30,$out
- vxor $out3,$in3,$twk3
- le?vperm $out5,$tmp,$tmp,$leperm
- stvx_u $out4,$x40,$out
- vxor $out4,$in4,$twk4
- le?stvx_u $out5,$x50,$out
- be?stvx_u $tmp, $x50,$out
- vxor $out5,$in5,$twk5
- addi $out,$out,0x60
-
- mtctr $rounds
- beq Loop_xts_enc6x # did $len-=96 borrow?
-
- addic. $len,$len,0x60
- beq Lxts_enc6x_zero
- cmpwi $len,0x20
- blt Lxts_enc6x_one
- nop
- beq Lxts_enc6x_two
- cmpwi $len,0x40
- blt Lxts_enc6x_three
- nop
- beq Lxts_enc6x_four
-
-Lxts_enc6x_five:
- vxor $out0,$in1,$twk0
- vxor $out1,$in2,$twk1
- vxor $out2,$in3,$twk2
- vxor $out3,$in4,$twk3
- vxor $out4,$in5,$twk4
-
- bl _aesp8_xts_enc5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk5 # unused tweak
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$leperm
- stvx_u $out2,$x20,$out
- vxor $tmp,$out4,$twk5 # last block prep for stealing
- le?vperm $out4,$out4,$out4,$leperm
- stvx_u $out3,$x30,$out
- stvx_u $out4,$x40,$out
- addi $out,$out,0x50
- bne Lxts_enc6x_steal
- b Lxts_enc6x_done
-
-.align 4
-Lxts_enc6x_four:
- vxor $out0,$in2,$twk0
- vxor $out1,$in3,$twk1
- vxor $out2,$in4,$twk2
- vxor $out3,$in5,$twk3
- vxor $out4,$out4,$out4
-
- bl _aesp8_xts_enc5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk4 # unused tweak
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- vxor $tmp,$out3,$twk4 # last block prep for stealing
- le?vperm $out3,$out3,$out3,$leperm
- stvx_u $out2,$x20,$out
- stvx_u $out3,$x30,$out
- addi $out,$out,0x40
- bne Lxts_enc6x_steal
- b Lxts_enc6x_done
-
-.align 4
-Lxts_enc6x_three:
- vxor $out0,$in3,$twk0
- vxor $out1,$in4,$twk1
- vxor $out2,$in5,$twk2
- vxor $out3,$out3,$out3
- vxor $out4,$out4,$out4
-
- bl _aesp8_xts_enc5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk3 # unused tweak
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $tmp,$out2,$twk3 # last block prep for stealing
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- stvx_u $out2,$x20,$out
- addi $out,$out,0x30
- bne Lxts_enc6x_steal
- b Lxts_enc6x_done
-
-.align 4
-Lxts_enc6x_two:
- vxor $out0,$in4,$twk0
- vxor $out1,$in5,$twk1
- vxor $out2,$out2,$out2
- vxor $out3,$out3,$out3
- vxor $out4,$out4,$out4
-
- bl _aesp8_xts_enc5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk2 # unused tweak
- vxor $tmp,$out1,$twk2 # last block prep for stealing
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- stvx_u $out1,$x10,$out
- addi $out,$out,0x20
- bne Lxts_enc6x_steal
- b Lxts_enc6x_done
-
-.align 4
-Lxts_enc6x_one:
- vxor $out0,$in5,$twk0
- nop
-Loop_xts_enc1x:
- vcipher $out0,$out0,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vcipher $out0,$out0,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_xts_enc1x
-
- add $inp,$inp,$taillen
- cmpwi $taillen,0
- vcipher $out0,$out0,v24
-
- subi $inp,$inp,16
- vcipher $out0,$out0,v25
-
- lvsr $inpperm,0,$taillen
- vcipher $out0,$out0,v26
-
- lvx_u $in0,0,$inp
- vcipher $out0,$out0,v27
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vcipher $out0,$out0,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
-
- vcipher $out0,$out0,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vxor $twk0,$twk0,v31
-
- le?vperm $in0,$in0,$in0,$leperm
- vcipher $out0,$out0,v30
-
- vperm $in0,$in0,$in0,$inpperm
- vcipherlast $out0,$out0,$twk0
-
- vmr $twk0,$twk1 # unused tweak
- vxor $tmp,$out0,$twk1 # last block prep for stealing
- le?vperm $out0,$out0,$out0,$leperm
- stvx_u $out0,$x00,$out # store output
- addi $out,$out,0x10
- bne Lxts_enc6x_steal
- b Lxts_enc6x_done
-
-.align 4
-Lxts_enc6x_zero:
- cmpwi $taillen,0
- beq Lxts_enc6x_done
-
- add $inp,$inp,$taillen
- subi $inp,$inp,16
- lvx_u $in0,0,$inp
- lvsr $inpperm,0,$taillen # $in5 is no more
- le?vperm $in0,$in0,$in0,$leperm
- vperm $in0,$in0,$in0,$inpperm
- vxor $tmp,$tmp,$twk0
-Lxts_enc6x_steal:
- vxor $in0,$in0,$twk0
- vxor $out0,$out0,$out0
- vspltisb $out1,-1
- vperm $out0,$out0,$out1,$inpperm
- vsel $out0,$in0,$tmp,$out0 # $tmp is last block, remember?
-
- subi r30,$out,17
- subi $out,$out,16
- mtctr $taillen
-Loop_xts_enc6x_steal:
- lbzu r0,1(r30)
- stb r0,16(r30)
- bdnz Loop_xts_enc6x_steal
-
- li $taillen,0
- mtctr $rounds
- b Loop_xts_enc1x # one more time...
-
-.align 4
-Lxts_enc6x_done:
- ${UCMP}i $ivp,0
- beq Lxts_enc6x_ret
-
- vxor $tweak,$twk0,$rndkey0
- le?vperm $tweak,$tweak,$tweak,$leperm
- stvx_u $tweak,0,$ivp
-
-Lxts_enc6x_ret:
- mtlr r11
- li r10,`$FRAME+15`
- li r11,`$FRAME+31`
- stvx $seven,r10,$sp # wipe copies of round keys
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
- stvx $seven,r10,$sp
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
- stvx $seven,r10,$sp
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
- stvx $seven,r10,$sp
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
-
- mtspr 256,$vrsave
- lvx v20,r10,$sp # ABI says so
- addi r10,r10,32
- lvx v21,r11,$sp
- addi r11,r11,32
- lvx v22,r10,$sp
- addi r10,r10,32
- lvx v23,r11,$sp
- addi r11,r11,32
- lvx v24,r10,$sp
- addi r10,r10,32
- lvx v25,r11,$sp
- addi r11,r11,32
- lvx v26,r10,$sp
- addi r10,r10,32
- lvx v27,r11,$sp
- addi r11,r11,32
- lvx v28,r10,$sp
- addi r10,r10,32
- lvx v29,r11,$sp
- addi r11,r11,32
- lvx v30,r10,$sp
- lvx v31,r11,$sp
- $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
- blr
- .long 0
- .byte 0,12,0x04,1,0x80,6,6,0
- .long 0
-
-.align 5
-_aesp8_xts_enc5x:
- vcipher $out0,$out0,v24
- vcipher $out1,$out1,v24
- vcipher $out2,$out2,v24
- vcipher $out3,$out3,v24
- vcipher $out4,$out4,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vcipher $out0,$out0,v25
- vcipher $out1,$out1,v25
- vcipher $out2,$out2,v25
- vcipher $out3,$out3,v25
- vcipher $out4,$out4,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz _aesp8_xts_enc5x
-
- add $inp,$inp,$taillen
- cmpwi $taillen,0
- vcipher $out0,$out0,v24
- vcipher $out1,$out1,v24
- vcipher $out2,$out2,v24
- vcipher $out3,$out3,v24
- vcipher $out4,$out4,v24
-
- subi $inp,$inp,16
- vcipher $out0,$out0,v25
- vcipher $out1,$out1,v25
- vcipher $out2,$out2,v25
- vcipher $out3,$out3,v25
- vcipher $out4,$out4,v25
- vxor $twk0,$twk0,v31
-
- vcipher $out0,$out0,v26
- lvsr $inpperm,r0,$taillen # $in5 is no more
- vcipher $out1,$out1,v26
- vcipher $out2,$out2,v26
- vcipher $out3,$out3,v26
- vcipher $out4,$out4,v26
- vxor $in1,$twk1,v31
-
- vcipher $out0,$out0,v27
- lvx_u $in0,0,$inp
- vcipher $out1,$out1,v27
- vcipher $out2,$out2,v27
- vcipher $out3,$out3,v27
- vcipher $out4,$out4,v27
- vxor $in2,$twk2,v31
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vcipher $out0,$out0,v28
- vcipher $out1,$out1,v28
- vcipher $out2,$out2,v28
- vcipher $out3,$out3,v28
- vcipher $out4,$out4,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
- vxor $in3,$twk3,v31
-
- vcipher $out0,$out0,v29
- le?vperm $in0,$in0,$in0,$leperm
- vcipher $out1,$out1,v29
- vcipher $out2,$out2,v29
- vcipher $out3,$out3,v29
- vcipher $out4,$out4,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vxor $in4,$twk4,v31
-
- vcipher $out0,$out0,v30
- vperm $in0,$in0,$in0,$inpperm
- vcipher $out1,$out1,v30
- vcipher $out2,$out2,v30
- vcipher $out3,$out3,v30
- vcipher $out4,$out4,v30
-
- vcipherlast $out0,$out0,$twk0
- vcipherlast $out1,$out1,$in1
- vcipherlast $out2,$out2,$in2
- vcipherlast $out3,$out3,$in3
- vcipherlast $out4,$out4,$in4
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,0,0
-
-.align 5
-_aesp8_xts_decrypt6x:
- $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
- mflr r11
- li r7,`$FRAME+8*16+15`
- li r3,`$FRAME+8*16+31`
- $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp)
- stvx v20,r7,$sp # ABI says so
- addi r7,r7,32
- stvx v21,r3,$sp
- addi r3,r3,32
- stvx v22,r7,$sp
- addi r7,r7,32
- stvx v23,r3,$sp
- addi r3,r3,32
- stvx v24,r7,$sp
- addi r7,r7,32
- stvx v25,r3,$sp
- addi r3,r3,32
- stvx v26,r7,$sp
- addi r7,r7,32
- stvx v27,r3,$sp
- addi r3,r3,32
- stvx v28,r7,$sp
- addi r7,r7,32
- stvx v29,r3,$sp
- addi r3,r3,32
- stvx v30,r7,$sp
- stvx v31,r3,$sp
- li r0,-1
- stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
- li $x10,0x10
- $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- li $x20,0x20
- $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- li $x30,0x30
- $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- li $x40,0x40
- $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- li $x50,0x50
- $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- li $x60,0x60
- $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- li $x70,0x70
- mtspr 256,r0
-
- subi $rounds,$rounds,3 # -4 in total
-
- lvx $rndkey0,$x00,$key1 # load key schedule
- lvx v30,$x10,$key1
- addi $key1,$key1,0x20
- lvx v31,$x00,$key1
- ?vperm $rndkey0,$rndkey0,v30,$keyperm
- addi $key_,$sp,$FRAME+15
- mtctr $rounds
-
-Load_xts_dec_key:
- ?vperm v24,v30,v31,$keyperm
- lvx v30,$x10,$key1
- addi $key1,$key1,0x20
- stvx v24,$x00,$key_ # off-load round[1]
- ?vperm v25,v31,v30,$keyperm
- lvx v31,$x00,$key1
- stvx v25,$x10,$key_ # off-load round[2]
- addi $key_,$key_,0x20
- bdnz Load_xts_dec_key
-
- lvx v26,$x10,$key1
- ?vperm v24,v30,v31,$keyperm
- lvx v27,$x20,$key1
- stvx v24,$x00,$key_ # off-load round[3]
- ?vperm v25,v31,v26,$keyperm
- lvx v28,$x30,$key1
- stvx v25,$x10,$key_ # off-load round[4]
- addi $key_,$sp,$FRAME+15 # rewind $key_
- ?vperm v26,v26,v27,$keyperm
- lvx v29,$x40,$key1
- ?vperm v27,v27,v28,$keyperm
- lvx v30,$x50,$key1
- ?vperm v28,v28,v29,$keyperm
- lvx v31,$x60,$key1
- ?vperm v29,v29,v30,$keyperm
- lvx $twk5,$x70,$key1 # borrow $twk5
- ?vperm v30,v30,v31,$keyperm
- lvx v24,$x00,$key_ # pre-load round[1]
- ?vperm v31,v31,$twk5,$keyperm
- lvx v25,$x10,$key_ # pre-load round[2]
-
- vperm $in0,$inout,$inptail,$inpperm
- subi $inp,$inp,31 # undo "caller"
- vxor $twk0,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vand $tmp,$tmp,$eighty7
- vxor $out0,$in0,$twk0
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in1,$x10,$inp
- vxor $twk1,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in1,$in1,$in1,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out1,$in1,$twk1
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in2,$x20,$inp
- andi. $taillen,$len,15
- vxor $twk2,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in2,$in2,$in2,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out2,$in2,$twk2
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in3,$x30,$inp
- sub $len,$len,$taillen
- vxor $twk3,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in3,$in3,$in3,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out3,$in3,$twk3
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in4,$x40,$inp
- subi $len,$len,0x60
- vxor $twk4,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in4,$in4,$in4,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out4,$in4,$twk4
- vxor $tweak,$tweak,$tmp
-
- lvx_u $in5,$x50,$inp
- addi $inp,$inp,0x60
- vxor $twk5,$tweak,$rndkey0
- vsrab $tmp,$tweak,$seven # next tweak value
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- le?vperm $in5,$in5,$in5,$leperm
- vand $tmp,$tmp,$eighty7
- vxor $out5,$in5,$twk5
- vxor $tweak,$tweak,$tmp
-
- vxor v31,v31,$rndkey0
- mtctr $rounds
- b Loop_xts_dec6x
-
-.align 5
-Loop_xts_dec6x:
- vncipher $out0,$out0,v24
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
- vncipher $out5,$out5,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vncipher $out0,$out0,v25
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- vncipher $out5,$out5,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_xts_dec6x
-
- subic $len,$len,96 # $len-=96
- vxor $in0,$twk0,v31 # xor with last round key
- vncipher $out0,$out0,v24
- vncipher $out1,$out1,v24
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk0,$tweak,$rndkey0
- vaddubm $tweak,$tweak,$tweak
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vsldoi $tmp,$tmp,$tmp,15
- vncipher $out4,$out4,v24
- vncipher $out5,$out5,v24
-
- subfe. r0,r0,r0 # borrow?-1:0
- vand $tmp,$tmp,$eighty7
- vncipher $out0,$out0,v25
- vncipher $out1,$out1,v25
- vxor $tweak,$tweak,$tmp
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vxor $in1,$twk1,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk1,$tweak,$rndkey0
- vncipher $out4,$out4,v25
- vncipher $out5,$out5,v25
-
- and r0,r0,$len
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vncipher $out0,$out0,v26
- vncipher $out1,$out1,v26
- vand $tmp,$tmp,$eighty7
- vncipher $out2,$out2,v26
- vncipher $out3,$out3,v26
- vxor $tweak,$tweak,$tmp
- vncipher $out4,$out4,v26
- vncipher $out5,$out5,v26
-
- add $inp,$inp,r0 # $inp is adjusted in such
- # way that at exit from the
- # loop inX-in5 are loaded
- # with last "words"
- vxor $in2,$twk2,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk2,$tweak,$rndkey0
- vaddubm $tweak,$tweak,$tweak
- vncipher $out0,$out0,v27
- vncipher $out1,$out1,v27
- vsldoi $tmp,$tmp,$tmp,15
- vncipher $out2,$out2,v27
- vncipher $out3,$out3,v27
- vand $tmp,$tmp,$eighty7
- vncipher $out4,$out4,v27
- vncipher $out5,$out5,v27
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vxor $tweak,$tweak,$tmp
- vncipher $out0,$out0,v28
- vncipher $out1,$out1,v28
- vxor $in3,$twk3,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk3,$tweak,$rndkey0
- vncipher $out2,$out2,v28
- vncipher $out3,$out3,v28
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vncipher $out4,$out4,v28
- vncipher $out5,$out5,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
- vand $tmp,$tmp,$eighty7
-
- vncipher $out0,$out0,v29
- vncipher $out1,$out1,v29
- vxor $tweak,$tweak,$tmp
- vncipher $out2,$out2,v29
- vncipher $out3,$out3,v29
- vxor $in4,$twk4,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk4,$tweak,$rndkey0
- vncipher $out4,$out4,v29
- vncipher $out5,$out5,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
-
- vncipher $out0,$out0,v30
- vncipher $out1,$out1,v30
- vand $tmp,$tmp,$eighty7
- vncipher $out2,$out2,v30
- vncipher $out3,$out3,v30
- vxor $tweak,$tweak,$tmp
- vncipher $out4,$out4,v30
- vncipher $out5,$out5,v30
- vxor $in5,$twk5,v31
- vsrab $tmp,$tweak,$seven # next tweak value
- vxor $twk5,$tweak,$rndkey0
-
- vncipherlast $out0,$out0,$in0
- lvx_u $in0,$x00,$inp # load next input block
- vaddubm $tweak,$tweak,$tweak
- vsldoi $tmp,$tmp,$tmp,15
- vncipherlast $out1,$out1,$in1
- lvx_u $in1,$x10,$inp
- vncipherlast $out2,$out2,$in2
- le?vperm $in0,$in0,$in0,$leperm
- lvx_u $in2,$x20,$inp
- vand $tmp,$tmp,$eighty7
- vncipherlast $out3,$out3,$in3
- le?vperm $in1,$in1,$in1,$leperm
- lvx_u $in3,$x30,$inp
- vncipherlast $out4,$out4,$in4
- le?vperm $in2,$in2,$in2,$leperm
- lvx_u $in4,$x40,$inp
- vxor $tweak,$tweak,$tmp
- vncipherlast $out5,$out5,$in5
- le?vperm $in3,$in3,$in3,$leperm
- lvx_u $in5,$x50,$inp
- addi $inp,$inp,0x60
- le?vperm $in4,$in4,$in4,$leperm
- le?vperm $in5,$in5,$in5,$leperm
-
- le?vperm $out0,$out0,$out0,$leperm
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $out0,$in0,$twk0
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- vxor $out1,$in1,$twk1
- le?vperm $out3,$out3,$out3,$leperm
- stvx_u $out2,$x20,$out
- vxor $out2,$in2,$twk2
- le?vperm $out4,$out4,$out4,$leperm
- stvx_u $out3,$x30,$out
- vxor $out3,$in3,$twk3
- le?vperm $out5,$out5,$out5,$leperm
- stvx_u $out4,$x40,$out
- vxor $out4,$in4,$twk4
- stvx_u $out5,$x50,$out
- vxor $out5,$in5,$twk5
- addi $out,$out,0x60
-
- mtctr $rounds
- beq Loop_xts_dec6x # did $len-=96 borrow?
-
- addic. $len,$len,0x60
- beq Lxts_dec6x_zero
- cmpwi $len,0x20
- blt Lxts_dec6x_one
- nop
- beq Lxts_dec6x_two
- cmpwi $len,0x40
- blt Lxts_dec6x_three
- nop
- beq Lxts_dec6x_four
-
-Lxts_dec6x_five:
- vxor $out0,$in1,$twk0
- vxor $out1,$in2,$twk1
- vxor $out2,$in3,$twk2
- vxor $out3,$in4,$twk3
- vxor $out4,$in5,$twk4
-
- bl _aesp8_xts_dec5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk5 # unused tweak
- vxor $twk1,$tweak,$rndkey0
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $out0,$in0,$twk1
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$leperm
- stvx_u $out2,$x20,$out
- le?vperm $out4,$out4,$out4,$leperm
- stvx_u $out3,$x30,$out
- stvx_u $out4,$x40,$out
- addi $out,$out,0x50
- bne Lxts_dec6x_steal
- b Lxts_dec6x_done
-
-.align 4
-Lxts_dec6x_four:
- vxor $out0,$in2,$twk0
- vxor $out1,$in3,$twk1
- vxor $out2,$in4,$twk2
- vxor $out3,$in5,$twk3
- vxor $out4,$out4,$out4
-
- bl _aesp8_xts_dec5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk4 # unused tweak
- vmr $twk1,$twk5
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $out0,$in0,$twk5
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- le?vperm $out3,$out3,$out3,$leperm
- stvx_u $out2,$x20,$out
- stvx_u $out3,$x30,$out
- addi $out,$out,0x40
- bne Lxts_dec6x_steal
- b Lxts_dec6x_done
-
-.align 4
-Lxts_dec6x_three:
- vxor $out0,$in3,$twk0
- vxor $out1,$in4,$twk1
- vxor $out2,$in5,$twk2
- vxor $out3,$out3,$out3
- vxor $out4,$out4,$out4
-
- bl _aesp8_xts_dec5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk3 # unused tweak
- vmr $twk1,$twk4
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $out0,$in0,$twk4
- le?vperm $out2,$out2,$out2,$leperm
- stvx_u $out1,$x10,$out
- stvx_u $out2,$x20,$out
- addi $out,$out,0x30
- bne Lxts_dec6x_steal
- b Lxts_dec6x_done
-
-.align 4
-Lxts_dec6x_two:
- vxor $out0,$in4,$twk0
- vxor $out1,$in5,$twk1
- vxor $out2,$out2,$out2
- vxor $out3,$out3,$out3
- vxor $out4,$out4,$out4
-
- bl _aesp8_xts_dec5x
-
- le?vperm $out0,$out0,$out0,$leperm
- vmr $twk0,$twk2 # unused tweak
- vmr $twk1,$twk3
- le?vperm $out1,$out1,$out1,$leperm
- stvx_u $out0,$x00,$out # store output
- vxor $out0,$in0,$twk3
- stvx_u $out1,$x10,$out
- addi $out,$out,0x20
- bne Lxts_dec6x_steal
- b Lxts_dec6x_done
-
-.align 4
-Lxts_dec6x_one:
- vxor $out0,$in5,$twk0
- nop
-Loop_xts_dec1x:
- vncipher $out0,$out0,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vncipher $out0,$out0,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Loop_xts_dec1x
-
- subi r0,$taillen,1
- vncipher $out0,$out0,v24
-
- andi. r0,r0,16
- cmpwi $taillen,0
- vncipher $out0,$out0,v25
-
- sub $inp,$inp,r0
- vncipher $out0,$out0,v26
-
- lvx_u $in0,0,$inp
- vncipher $out0,$out0,v27
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vncipher $out0,$out0,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
-
- vncipher $out0,$out0,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vxor $twk0,$twk0,v31
-
- le?vperm $in0,$in0,$in0,$leperm
- vncipher $out0,$out0,v30
-
- mtctr $rounds
- vncipherlast $out0,$out0,$twk0
-
- vmr $twk0,$twk1 # unused tweak
- vmr $twk1,$twk2
- le?vperm $out0,$out0,$out0,$leperm
- stvx_u $out0,$x00,$out # store output
- addi $out,$out,0x10
- vxor $out0,$in0,$twk2
- bne Lxts_dec6x_steal
- b Lxts_dec6x_done
-
-.align 4
-Lxts_dec6x_zero:
- cmpwi $taillen,0
- beq Lxts_dec6x_done
-
- lvx_u $in0,0,$inp
- le?vperm $in0,$in0,$in0,$leperm
- vxor $out0,$in0,$twk1
-Lxts_dec6x_steal:
- vncipher $out0,$out0,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vncipher $out0,$out0,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz Lxts_dec6x_steal
-
- add $inp,$inp,$taillen
- vncipher $out0,$out0,v24
-
- cmpwi $taillen,0
- vncipher $out0,$out0,v25
-
- lvx_u $in0,0,$inp
- vncipher $out0,$out0,v26
-
- lvsr $inpperm,0,$taillen # $in5 is no more
- vncipher $out0,$out0,v27
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vncipher $out0,$out0,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
-
- vncipher $out0,$out0,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vxor $twk1,$twk1,v31
-
- le?vperm $in0,$in0,$in0,$leperm
- vncipher $out0,$out0,v30
-
- vperm $in0,$in0,$in0,$inpperm
- vncipherlast $tmp,$out0,$twk1
-
- le?vperm $out0,$tmp,$tmp,$leperm
- le?stvx_u $out0,0,$out
- be?stvx_u $tmp,0,$out
-
- vxor $out0,$out0,$out0
- vspltisb $out1,-1
- vperm $out0,$out0,$out1,$inpperm
- vsel $out0,$in0,$tmp,$out0
- vxor $out0,$out0,$twk0
-
- subi r30,$out,1
- mtctr $taillen
-Loop_xts_dec6x_steal:
- lbzu r0,1(r30)
- stb r0,16(r30)
- bdnz Loop_xts_dec6x_steal
-
- li $taillen,0
- mtctr $rounds
- b Loop_xts_dec1x # one more time...
-
-.align 4
-Lxts_dec6x_done:
- ${UCMP}i $ivp,0
- beq Lxts_dec6x_ret
-
- vxor $tweak,$twk0,$rndkey0
- le?vperm $tweak,$tweak,$tweak,$leperm
- stvx_u $tweak,0,$ivp
-
-Lxts_dec6x_ret:
- mtlr r11
- li r10,`$FRAME+15`
- li r11,`$FRAME+31`
- stvx $seven,r10,$sp # wipe copies of round keys
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
- stvx $seven,r10,$sp
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
- stvx $seven,r10,$sp
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
- stvx $seven,r10,$sp
- addi r10,r10,32
- stvx $seven,r11,$sp
- addi r11,r11,32
-
- mtspr 256,$vrsave
- lvx v20,r10,$sp # ABI says so
- addi r10,r10,32
- lvx v21,r11,$sp
- addi r11,r11,32
- lvx v22,r10,$sp
- addi r10,r10,32
- lvx v23,r11,$sp
- addi r11,r11,32
- lvx v24,r10,$sp
- addi r10,r10,32
- lvx v25,r11,$sp
- addi r11,r11,32
- lvx v26,r10,$sp
- addi r10,r10,32
- lvx v27,r11,$sp
- addi r11,r11,32
- lvx v28,r10,$sp
- addi r10,r10,32
- lvx v29,r11,$sp
- addi r11,r11,32
- lvx v30,r10,$sp
- lvx v31,r11,$sp
- $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
- $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
- $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
- $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
- $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
- $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
- addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
- blr
- .long 0
- .byte 0,12,0x04,1,0x80,6,6,0
- .long 0
-
-.align 5
-_aesp8_xts_dec5x:
- vncipher $out0,$out0,v24
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
- lvx v24,$x20,$key_ # round[3]
- addi $key_,$key_,0x20
-
- vncipher $out0,$out0,v25
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- lvx v25,$x10,$key_ # round[4]
- bdnz _aesp8_xts_dec5x
-
- subi r0,$taillen,1
- vncipher $out0,$out0,v24
- vncipher $out1,$out1,v24
- vncipher $out2,$out2,v24
- vncipher $out3,$out3,v24
- vncipher $out4,$out4,v24
-
- andi. r0,r0,16
- cmpwi $taillen,0
- vncipher $out0,$out0,v25
- vncipher $out1,$out1,v25
- vncipher $out2,$out2,v25
- vncipher $out3,$out3,v25
- vncipher $out4,$out4,v25
- vxor $twk0,$twk0,v31
-
- sub $inp,$inp,r0
- vncipher $out0,$out0,v26
- vncipher $out1,$out1,v26
- vncipher $out2,$out2,v26
- vncipher $out3,$out3,v26
- vncipher $out4,$out4,v26
- vxor $in1,$twk1,v31
-
- vncipher $out0,$out0,v27
- lvx_u $in0,0,$inp
- vncipher $out1,$out1,v27
- vncipher $out2,$out2,v27
- vncipher $out3,$out3,v27
- vncipher $out4,$out4,v27
- vxor $in2,$twk2,v31
-
- addi $key_,$sp,$FRAME+15 # rewind $key_
- vncipher $out0,$out0,v28
- vncipher $out1,$out1,v28
- vncipher $out2,$out2,v28
- vncipher $out3,$out3,v28
- vncipher $out4,$out4,v28
- lvx v24,$x00,$key_ # re-pre-load round[1]
- vxor $in3,$twk3,v31
-
- vncipher $out0,$out0,v29
- le?vperm $in0,$in0,$in0,$leperm
- vncipher $out1,$out1,v29
- vncipher $out2,$out2,v29
- vncipher $out3,$out3,v29
- vncipher $out4,$out4,v29
- lvx v25,$x10,$key_ # re-pre-load round[2]
- vxor $in4,$twk4,v31
-
- vncipher $out0,$out0,v30
- vncipher $out1,$out1,v30
- vncipher $out2,$out2,v30
- vncipher $out3,$out3,v30
- vncipher $out4,$out4,v30
-
- vncipherlast $out0,$out0,$twk0
- vncipherlast $out1,$out1,$in1
- vncipherlast $out2,$out2,$in2
- vncipherlast $out3,$out3,$in3
- vncipherlast $out4,$out4,$in4
- mtctr $rounds
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,0,0
-___
-}} }}}
-
-my $consts=1;
-foreach(split("\n",$code)) {
- s/\`([^\`]*)\`/eval($1)/geo;
-
- # constants table endian-specific conversion
- if ($consts && m/\.(long|byte)\s+(.+)\s+(\?[a-z]*)$/o) {
- my $conv=$3;
- my @bytes=();
-
- # convert to endian-agnostic format
- if ($1 eq "long") {
- foreach (split(/,\s*/,$2)) {
- my $l = /^0/?oct:int;
- push @bytes,($l>>24)&0xff,($l>>16)&0xff,($l>>8)&0xff,$l&0xff;
- }
- } else {
- @bytes = map(/^0/?oct:int,split(/,\s*/,$2));
- }
-
- # little-endian conversion
- if ($flavour =~ /le$/o) {
- SWITCH: for($conv) {
- /\?inv/ && do { @bytes=map($_^0xf,@bytes); last; };
- /\?rev/ && do { @bytes=reverse(@bytes); last; };
- }
- }
-
- #emit
- print ".byte\t",join(',',map (sprintf("0x%02x",$_),@bytes)),"\n";
- next;
- }
- $consts=0 if (m/Lconsts:/o); # end of table
-
- # instructions prefixed with '?' are endian-specific and need
- # to be adjusted accordingly...
- if ($flavour =~ /le$/o) { # little-endian
- s/le\?//o or
- s/be\?/#be#/o or
- s/\?lvsr/lvsl/o or
- s/\?lvsl/lvsr/o or
- s/\?(vperm\s+v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+)/$1$3$2$4/o or
- s/\?(vsldoi\s+v[0-9]+,\s*)(v[0-9]+,)\s*(v[0-9]+,\s*)([0-9]+)/$1$3$2 16-$4/o or
- s/\?(vspltw\s+v[0-9]+,\s*)(v[0-9]+,)\s*([0-9])/$1$2 3-$3/o;
- } else { # big-endian
- s/le\?/#le#/o or
- s/be\?//o or
- s/\?([a-z]+)/$1/o;
- }
-
- print $_,"\n";
-}
-
-close STDOUT;
diff --git a/drivers/crypto/vmx/ghash.c b/drivers/crypto/vmx/ghash.c
deleted file mode 100644
index 27a94a119009..000000000000
--- a/drivers/crypto/vmx/ghash.c
+++ /dev/null
@@ -1,227 +0,0 @@
-/**
- * GHASH routines supporting VMX instructions on the Power 8
- *
- * Copyright (C) 2015 International Business Machines Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; version 2 only.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- *
- * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
- */
-
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/crypto.h>
-#include <linux/delay.h>
-#include <linux/hardirq.h>
-#include <asm/switch_to.h>
-#include <crypto/aes.h>
-#include <crypto/ghash.h>
-#include <crypto/scatterwalk.h>
-#include <crypto/internal/hash.h>
-#include <crypto/b128ops.h>
-
-#define IN_INTERRUPT in_interrupt()
-
-void gcm_init_p8(u128 htable[16], const u64 Xi[2]);
-void gcm_gmult_p8(u64 Xi[2], const u128 htable[16]);
-void gcm_ghash_p8(u64 Xi[2], const u128 htable[16],
- const u8 *in, size_t len);
-
-struct p8_ghash_ctx {
- u128 htable[16];
- struct crypto_shash *fallback;
-};
-
-struct p8_ghash_desc_ctx {
- u64 shash[2];
- u8 buffer[GHASH_DIGEST_SIZE];
- int bytes;
- struct shash_desc fallback_desc;
-};
-
-static int p8_ghash_init_tfm(struct crypto_tfm *tfm)
-{
- const char *alg = "ghash-generic";
- struct crypto_shash *fallback;
- struct crypto_shash *shash_tfm = __crypto_shash_cast(tfm);
- struct p8_ghash_ctx *ctx = crypto_tfm_ctx(tfm);
-
- fallback = crypto_alloc_shash(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
- if (IS_ERR(fallback)) {
- printk(KERN_ERR
- "Failed to allocate transformation for '%s': %ld\n",
- alg, PTR_ERR(fallback));
- return PTR_ERR(fallback);
- }
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name(crypto_shash_tfm(fallback)));
-
- crypto_shash_set_flags(fallback,
- crypto_shash_get_flags((struct crypto_shash
- *) tfm));
-
- /* Check if the descsize defined in the algorithm is still enough. */
- if (shash_tfm->descsize < sizeof(struct p8_ghash_desc_ctx)
- + crypto_shash_descsize(fallback)) {
- printk(KERN_ERR
- "Desc size of the fallback implementation (%s) does not match the expected value: %lu vs %u\n",
- alg,
- shash_tfm->descsize - sizeof(struct p8_ghash_desc_ctx),
- crypto_shash_descsize(fallback));
- return -EINVAL;
- }
- ctx->fallback = fallback;
-
- return 0;
-}
-
-static void p8_ghash_exit_tfm(struct crypto_tfm *tfm)
-{
- struct p8_ghash_ctx *ctx = crypto_tfm_ctx(tfm);
-
- if (ctx->fallback) {
- crypto_free_shash(ctx->fallback);
- ctx->fallback = NULL;
- }
-}
-
-static int p8_ghash_init(struct shash_desc *desc)
-{
- struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
- struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
-
- dctx->bytes = 0;
- memset(dctx->shash, 0, GHASH_DIGEST_SIZE);
- dctx->fallback_desc.tfm = ctx->fallback;
- dctx->fallback_desc.flags = desc->flags;
- return crypto_shash_init(&dctx->fallback_desc);
-}
-
-static int p8_ghash_setkey(struct crypto_shash *tfm, const u8 *key,
- unsigned int keylen)
-{
- struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(tfm));
-
- if (keylen != GHASH_BLOCK_SIZE)
- return -EINVAL;
-
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- gcm_init_p8(ctx->htable, (const u64 *) key);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- return crypto_shash_setkey(ctx->fallback, key, keylen);
-}
-
-static int p8_ghash_update(struct shash_desc *desc,
- const u8 *src, unsigned int srclen)
-{
- unsigned int len;
- struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
- struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
-
- if (IN_INTERRUPT) {
- return crypto_shash_update(&dctx->fallback_desc, src,
- srclen);
- } else {
- if (dctx->bytes) {
- if (dctx->bytes + srclen < GHASH_DIGEST_SIZE) {
- memcpy(dctx->buffer + dctx->bytes, src,
- srclen);
- dctx->bytes += srclen;
- return 0;
- }
- memcpy(dctx->buffer + dctx->bytes, src,
- GHASH_DIGEST_SIZE - dctx->bytes);
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- gcm_ghash_p8(dctx->shash, ctx->htable,
- dctx->buffer, GHASH_DIGEST_SIZE);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- src += GHASH_DIGEST_SIZE - dctx->bytes;
- srclen -= GHASH_DIGEST_SIZE - dctx->bytes;
- dctx->bytes = 0;
- }
- len = srclen & ~(GHASH_DIGEST_SIZE - 1);
- if (len) {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- gcm_ghash_p8(dctx->shash, ctx->htable, src, len);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- src += len;
- srclen -= len;
- }
- if (srclen) {
- memcpy(dctx->buffer, src, srclen);
- dctx->bytes = srclen;
- }
- return 0;
- }
-}
-
-static int p8_ghash_final(struct shash_desc *desc, u8 *out)
-{
- int i;
- struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
- struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
-
- if (IN_INTERRUPT) {
- return crypto_shash_final(&dctx->fallback_desc, out);
- } else {
- if (dctx->bytes) {
- for (i = dctx->bytes; i < GHASH_DIGEST_SIZE; i++)
- dctx->buffer[i] = 0;
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
- gcm_ghash_p8(dctx->shash, ctx->htable,
- dctx->buffer, GHASH_DIGEST_SIZE);
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
- dctx->bytes = 0;
- }
- memcpy(out, dctx->shash, GHASH_DIGEST_SIZE);
- return 0;
- }
-}
-
-struct shash_alg p8_ghash_alg = {
- .digestsize = GHASH_DIGEST_SIZE,
- .init = p8_ghash_init,
- .update = p8_ghash_update,
- .final = p8_ghash_final,
- .setkey = p8_ghash_setkey,
- .descsize = sizeof(struct p8_ghash_desc_ctx)
- + sizeof(struct ghash_desc_ctx),
- .base = {
- .cra_name = "ghash",
- .cra_driver_name = "p8_ghash",
- .cra_priority = 1000,
- .cra_flags = CRYPTO_ALG_TYPE_SHASH | CRYPTO_ALG_NEED_FALLBACK,
- .cra_blocksize = GHASH_BLOCK_SIZE,
- .cra_ctxsize = sizeof(struct p8_ghash_ctx),
- .cra_module = THIS_MODULE,
- .cra_init = p8_ghash_init_tfm,
- .cra_exit = p8_ghash_exit_tfm,
- },
-};
diff --git a/drivers/crypto/vmx/ghashp8-ppc.pl b/drivers/crypto/vmx/ghashp8-ppc.pl
deleted file mode 100644
index d8429cb71f02..000000000000
--- a/drivers/crypto/vmx/ghashp8-ppc.pl
+++ /dev/null
@@ -1,234 +0,0 @@
-#!/usr/bin/env perl
-#
-# ====================================================================
-# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
-# project. The module is, however, dual licensed under OpenSSL and
-# CRYPTOGAMS licenses depending on where you obtain it. For further
-# details see http://www.openssl.org/~appro/cryptogams/.
-# ====================================================================
-#
-# GHASH for for PowerISA v2.07.
-#
-# July 2014
-#
-# Accurate performance measurements are problematic, because it's
-# always virtualized setup with possibly throttled processor.
-# Relative comparison is therefore more informative. This initial
-# version is ~2.1x slower than hardware-assisted AES-128-CTR, ~12x
-# faster than "4-bit" integer-only compiler-generated 64-bit code.
-# "Initial version" means that there is room for futher improvement.
-
-$flavour=shift;
-$output =shift;
-
-if ($flavour =~ /64/) {
- $SIZE_T=8;
- $LRSAVE=2*$SIZE_T;
- $STU="stdu";
- $POP="ld";
- $PUSH="std";
-} elsif ($flavour =~ /32/) {
- $SIZE_T=4;
- $LRSAVE=$SIZE_T;
- $STU="stwu";
- $POP="lwz";
- $PUSH="stw";
-} else { die "nonsense $flavour"; }
-
-$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
-( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or
-( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or
-die "can't locate ppc-xlate.pl";
-
-open STDOUT,"| $^X $xlate $flavour $output" || die "can't call $xlate: $!";
-
-my ($Xip,$Htbl,$inp,$len)=map("r$_",(3..6)); # argument block
-
-my ($Xl,$Xm,$Xh,$IN)=map("v$_",(0..3));
-my ($zero,$t0,$t1,$t2,$xC2,$H,$Hh,$Hl,$lemask)=map("v$_",(4..12));
-my $vrsave="r12";
-
-$code=<<___;
-.machine "any"
-
-.text
-
-.globl .gcm_init_p8
- lis r0,0xfff0
- li r8,0x10
- mfspr $vrsave,256
- li r9,0x20
- mtspr 256,r0
- li r10,0x30
- lvx_u $H,0,r4 # load H
- le?xor r7,r7,r7
- le?addi r7,r7,0x8 # need a vperm start with 08
- le?lvsr 5,0,r7
- le?vspltisb 6,0x0f
- le?vxor 5,5,6 # set a b-endian mask
- le?vperm $H,$H,$H,5
-
- vspltisb $xC2,-16 # 0xf0
- vspltisb $t0,1 # one
- vaddubm $xC2,$xC2,$xC2 # 0xe0
- vxor $zero,$zero,$zero
- vor $xC2,$xC2,$t0 # 0xe1
- vsldoi $xC2,$xC2,$zero,15 # 0xe1...
- vsldoi $t1,$zero,$t0,1 # ...1
- vaddubm $xC2,$xC2,$xC2 # 0xc2...
- vspltisb $t2,7
- vor $xC2,$xC2,$t1 # 0xc2....01
- vspltb $t1,$H,0 # most significant byte
- vsl $H,$H,$t0 # H<<=1
- vsrab $t1,$t1,$t2 # broadcast carry bit
- vand $t1,$t1,$xC2
- vxor $H,$H,$t1 # twisted H
-
- vsldoi $H,$H,$H,8 # twist even more ...
- vsldoi $xC2,$zero,$xC2,8 # 0xc2.0
- vsldoi $Hl,$zero,$H,8 # ... and split
- vsldoi $Hh,$H,$zero,8
-
- stvx_u $xC2,0,r3 # save pre-computed table
- stvx_u $Hl,r8,r3
- stvx_u $H, r9,r3
- stvx_u $Hh,r10,r3
-
- mtspr 256,$vrsave
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,2,0
- .long 0
-.size .gcm_init_p8,.-.gcm_init_p8
-
-.globl .gcm_gmult_p8
- lis r0,0xfff8
- li r8,0x10
- mfspr $vrsave,256
- li r9,0x20
- mtspr 256,r0
- li r10,0x30
- lvx_u $IN,0,$Xip # load Xi
-
- lvx_u $Hl,r8,$Htbl # load pre-computed table
- le?lvsl $lemask,r0,r0
- lvx_u $H, r9,$Htbl
- le?vspltisb $t0,0x07
- lvx_u $Hh,r10,$Htbl
- le?vxor $lemask,$lemask,$t0
- lvx_u $xC2,0,$Htbl
- le?vperm $IN,$IN,$IN,$lemask
- vxor $zero,$zero,$zero
-
- vpmsumd $Xl,$IN,$Hl # H.lo·Xi.lo
- vpmsumd $Xm,$IN,$H # H.hi·Xi.lo+H.lo·Xi.hi
- vpmsumd $Xh,$IN,$Hh # H.hi·Xi.hi
-
- vpmsumd $t2,$Xl,$xC2 # 1st phase
-
- vsldoi $t0,$Xm,$zero,8
- vsldoi $t1,$zero,$Xm,8
- vxor $Xl,$Xl,$t0
- vxor $Xh,$Xh,$t1
-
- vsldoi $Xl,$Xl,$Xl,8
- vxor $Xl,$Xl,$t2
-
- vsldoi $t1,$Xl,$Xl,8 # 2nd phase
- vpmsumd $Xl,$Xl,$xC2
- vxor $t1,$t1,$Xh
- vxor $Xl,$Xl,$t1
-
- le?vperm $Xl,$Xl,$Xl,$lemask
- stvx_u $Xl,0,$Xip # write out Xi
-
- mtspr 256,$vrsave
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,2,0
- .long 0
-.size .gcm_gmult_p8,.-.gcm_gmult_p8
-
-.globl .gcm_ghash_p8
- lis r0,0xfff8
- li r8,0x10
- mfspr $vrsave,256
- li r9,0x20
- mtspr 256,r0
- li r10,0x30
- lvx_u $Xl,0,$Xip # load Xi
-
- lvx_u $Hl,r8,$Htbl # load pre-computed table
- le?lvsl $lemask,r0,r0
- lvx_u $H, r9,$Htbl
- le?vspltisb $t0,0x07
- lvx_u $Hh,r10,$Htbl
- le?vxor $lemask,$lemask,$t0
- lvx_u $xC2,0,$Htbl
- le?vperm $Xl,$Xl,$Xl,$lemask
- vxor $zero,$zero,$zero
-
- lvx_u $IN,0,$inp
- addi $inp,$inp,16
- subi $len,$len,16
- le?vperm $IN,$IN,$IN,$lemask
- vxor $IN,$IN,$Xl
- b Loop
-
-.align 5
-Loop:
- subic $len,$len,16
- vpmsumd $Xl,$IN,$Hl # H.lo·Xi.lo
- subfe. r0,r0,r0 # borrow?-1:0
- vpmsumd $Xm,$IN,$H # H.hi·Xi.lo+H.lo·Xi.hi
- and r0,r0,$len
- vpmsumd $Xh,$IN,$Hh # H.hi·Xi.hi
- add $inp,$inp,r0
-
- vpmsumd $t2,$Xl,$xC2 # 1st phase
-
- vsldoi $t0,$Xm,$zero,8
- vsldoi $t1,$zero,$Xm,8
- vxor $Xl,$Xl,$t0
- vxor $Xh,$Xh,$t1
-
- vsldoi $Xl,$Xl,$Xl,8
- vxor $Xl,$Xl,$t2
- lvx_u $IN,0,$inp
- addi $inp,$inp,16
-
- vsldoi $t1,$Xl,$Xl,8 # 2nd phase
- vpmsumd $Xl,$Xl,$xC2
- le?vperm $IN,$IN,$IN,$lemask
- vxor $t1,$t1,$Xh
- vxor $IN,$IN,$t1
- vxor $IN,$IN,$Xl
- beq Loop # did $len-=16 borrow?
-
- vxor $Xl,$Xl,$t1
- le?vperm $Xl,$Xl,$Xl,$lemask
- stvx_u $Xl,0,$Xip # write out Xi
-
- mtspr 256,$vrsave
- blr
- .long 0
- .byte 0,12,0x14,0,0,0,4,0
- .long 0
-.size .gcm_ghash_p8,.-.gcm_ghash_p8
-
-.asciz "GHASH for PowerISA 2.07, CRYPTOGAMS by <appro\@openssl.org>"
-.align 2
-___
-
-foreach (split("\n",$code)) {
- if ($flavour =~ /le$/o) { # little-endian
- s/le\?//o or
- s/be\?/#be#/o;
- } else {
- s/le\?/#le#/o or
- s/be\?//o;
- }
- print $_,"\n";
-}
-
-close STDOUT; # enforce flush
diff --git a/drivers/crypto/vmx/ppc-xlate.pl b/drivers/crypto/vmx/ppc-xlate.pl
deleted file mode 100644
index b18e67d0e065..000000000000
--- a/drivers/crypto/vmx/ppc-xlate.pl
+++ /dev/null
@@ -1,228 +0,0 @@
-#!/usr/bin/env perl
-
-# PowerPC assembler distiller by <appro>.
-
-my $flavour = shift;
-my $output = shift;
-open STDOUT,">$output" || die "can't open $output: $!";
-
-my %GLOBALS;
-my $dotinlocallabels=($flavour=~/linux/)?1:0;
-
-################################################################
-# directives which need special treatment on different platforms
-################################################################
-my $globl = sub {
- my $junk = shift;
- my $name = shift;
- my $global = \$GLOBALS{$name};
- my $ret;
-
- $name =~ s|^[\.\_]||;
-
- SWITCH: for ($flavour) {
- /aix/ && do { $name = ".$name";
- last;
- };
- /osx/ && do { $name = "_$name";
- last;
- };
- /linux/
- && do { $ret = "_GLOBAL($name)";
- last;
- };
- }
-
- $ret = ".globl $name\nalign 5\n$name:" if (!$ret);
- $$global = $name;
- $ret;
-};
-my $text = sub {
- my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text";
- $ret = ".abiversion 2\n".$ret if ($flavour =~ /linux.*64le/);
- $ret;
-};
-my $machine = sub {
- my $junk = shift;
- my $arch = shift;
- if ($flavour =~ /osx/)
- { $arch =~ s/\"//g;
- $arch = ($flavour=~/64/) ? "ppc970-64" : "ppc970" if ($arch eq "any");
- }
- ".machine $arch";
-};
-my $size = sub {
- if ($flavour =~ /linux/)
- { shift;
- my $name = shift; $name =~ s|^[\.\_]||;
- my $ret = ".size $name,.-".($flavour=~/64$/?".":"").$name;
- $ret .= "\n.size .$name,.-.$name" if ($flavour=~/64$/);
- $ret;
- }
- else
- { ""; }
-};
-my $asciz = sub {
- shift;
- my $line = join(",",@_);
- if ($line =~ /^"(.*)"$/)
- { ".byte " . join(",",unpack("C*",$1),0) . "\n.align 2"; }
- else
- { ""; }
-};
-my $quad = sub {
- shift;
- my @ret;
- my ($hi,$lo);
- for (@_) {
- if (/^0x([0-9a-f]*?)([0-9a-f]{1,8})$/io)
- { $hi=$1?"0x$1":"0"; $lo="0x$2"; }
- elsif (/^([0-9]+)$/o)
- { $hi=$1>>32; $lo=$1&0xffffffff; } # error-prone with 32-bit perl
- else
- { $hi=undef; $lo=$_; }
-
- if (defined($hi))
- { push(@ret,$flavour=~/le$/o?".long\t$lo,$hi":".long\t$hi,$lo"); }
- else
- { push(@ret,".quad $lo"); }
- }
- join("\n",@ret);
-};
-
-################################################################
-# simplified mnemonics not handled by at least one assembler
-################################################################
-my $cmplw = sub {
- my $f = shift;
- my $cr = 0; $cr = shift if ($#_>1);
- # Some out-of-date 32-bit GNU assembler just can't handle cmplw...
- ($flavour =~ /linux.*32/) ?
- " .long ".sprintf "0x%x",31<<26|$cr<<23|$_[0]<<16|$_[1]<<11|64 :
- " cmplw ".join(',',$cr,@_);
-};
-my $bdnz = sub {
- my $f = shift;
- my $bo = $f=~/[\+\-]/ ? 16+9 : 16; # optional "to be taken" hint
- " bc $bo,0,".shift;
-} if ($flavour!~/linux/);
-my $bltlr = sub {
- my $f = shift;
- my $bo = $f=~/\-/ ? 12+2 : 12; # optional "not to be taken" hint
- ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
- " .long ".sprintf "0x%x",19<<26|$bo<<21|16<<1 :
- " bclr $bo,0";
-};
-my $bnelr = sub {
- my $f = shift;
- my $bo = $f=~/\-/ ? 4+2 : 4; # optional "not to be taken" hint
- ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
- " .long ".sprintf "0x%x",19<<26|$bo<<21|2<<16|16<<1 :
- " bclr $bo,2";
-};
-my $beqlr = sub {
- my $f = shift;
- my $bo = $f=~/-/ ? 12+2 : 12; # optional "not to be taken" hint
- ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
- " .long ".sprintf "0x%X",19<<26|$bo<<21|2<<16|16<<1 :
- " bclr $bo,2";
-};
-# GNU assembler can't handle extrdi rA,rS,16,48, or when sum of last two
-# arguments is 64, with "operand out of range" error.
-my $extrdi = sub {
- my ($f,$ra,$rs,$n,$b) = @_;
- $b = ($b+$n)&63; $n = 64-$n;
- " rldicl $ra,$rs,$b,$n";
-};
-my $vmr = sub {
- my ($f,$vx,$vy) = @_;
- " vor $vx,$vy,$vy";
-};
-
-# Some ABIs specify vrsave, special-purpose register #256, as reserved
-# for system use.
-my $no_vrsave = ($flavour =~ /linux-ppc64le/);
-my $mtspr = sub {
- my ($f,$idx,$ra) = @_;
- if ($idx == 256 && $no_vrsave) {
- " or $ra,$ra,$ra";
- } else {
- " mtspr $idx,$ra";
- }
-};
-my $mfspr = sub {
- my ($f,$rd,$idx) = @_;
- if ($idx == 256 && $no_vrsave) {
- " li $rd,-1";
- } else {
- " mfspr $rd,$idx";
- }
-};
-
-# PowerISA 2.06 stuff
-sub vsxmem_op {
- my ($f, $vrt, $ra, $rb, $op) = @_;
- " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|($rb<<11)|($op*2+1);
-}
-# made-up unaligned memory reference AltiVec/VMX instructions
-my $lvx_u = sub { vsxmem_op(@_, 844); }; # lxvd2x
-my $stvx_u = sub { vsxmem_op(@_, 972); }; # stxvd2x
-my $lvdx_u = sub { vsxmem_op(@_, 588); }; # lxsdx
-my $stvdx_u = sub { vsxmem_op(@_, 716); }; # stxsdx
-my $lvx_4w = sub { vsxmem_op(@_, 780); }; # lxvw4x
-my $stvx_4w = sub { vsxmem_op(@_, 908); }; # stxvw4x
-
-# PowerISA 2.07 stuff
-sub vcrypto_op {
- my ($f, $vrt, $vra, $vrb, $op) = @_;
- " .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|$op;
-}
-my $vcipher = sub { vcrypto_op(@_, 1288); };
-my $vcipherlast = sub { vcrypto_op(@_, 1289); };
-my $vncipher = sub { vcrypto_op(@_, 1352); };
-my $vncipherlast= sub { vcrypto_op(@_, 1353); };
-my $vsbox = sub { vcrypto_op(@_, 0, 1480); };
-my $vshasigmad = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1730); };
-my $vshasigmaw = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1666); };
-my $vpmsumb = sub { vcrypto_op(@_, 1032); };
-my $vpmsumd = sub { vcrypto_op(@_, 1224); };
-my $vpmsubh = sub { vcrypto_op(@_, 1096); };
-my $vpmsumw = sub { vcrypto_op(@_, 1160); };
-my $vaddudm = sub { vcrypto_op(@_, 192); };
-my $vadduqm = sub { vcrypto_op(@_, 256); };
-
-my $mtsle = sub {
- my ($f, $arg) = @_;
- " .long ".sprintf "0x%X",(31<<26)|($arg<<21)|(147*2);
-};
-
-print "#include <asm/ppc_asm.h>\n" if $flavour =~ /linux/;
-
-while($line=<>) {
-
- $line =~ s|[#!;].*$||; # get rid of asm-style comments...
- $line =~ s|/\*.*\*/||; # ... and C-style comments...
- $line =~ s|^\s+||; # ... and skip white spaces in beginning...
- $line =~ s|\s+$||; # ... and at the end
-
- {
- $line =~ s|\b\.L(\w+)|L$1|g; # common denominator for Locallabel
- $line =~ s|\bL(\w+)|\.L$1|g if ($dotinlocallabels);
- }
-
- {
- $line =~ s|^\s*(\.?)(\w+)([\.\+\-]?)\s*||;
- my $c = $1; $c = "\t" if ($c eq "");
- my $mnemonic = $2;
- my $f = $3;
- my $opcode = eval("\$$mnemonic");
- $line =~ s/\b(c?[rf]|v|vs)([0-9]+)\b/$2/g if ($c ne "." and $flavour !~ /osx/);
- if (ref($opcode) eq 'CODE') { $line = &$opcode($f,split(',',$line)); }
- elsif ($mnemonic) { $line = $c.$mnemonic.$f."\t".$line; }
- }
-
- print $line if ($line);
- print "\n";
-}
-
-close STDOUT;
diff --git a/drivers/crypto/vmx/vmx.c b/drivers/crypto/vmx/vmx.c
deleted file mode 100644
index 31a98dc6f849..000000000000
--- a/drivers/crypto/vmx/vmx.c
+++ /dev/null
@@ -1,88 +0,0 @@
-/**
- * Routines supporting VMX instructions on the Power 8
- *
- * Copyright (C) 2015 International Business Machines Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; version 2 only.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- *
- * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
- */
-
-#include <linux/module.h>
-#include <linux/moduleparam.h>
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/cpufeature.h>
-#include <linux/crypto.h>
-#include <asm/cputable.h>
-#include <crypto/internal/hash.h>
-
-extern struct shash_alg p8_ghash_alg;
-extern struct crypto_alg p8_aes_alg;
-extern struct crypto_alg p8_aes_cbc_alg;
-extern struct crypto_alg p8_aes_ctr_alg;
-extern struct crypto_alg p8_aes_xts_alg;
-static struct crypto_alg *algs[] = {
- &p8_aes_alg,
- &p8_aes_cbc_alg,
- &p8_aes_ctr_alg,
- &p8_aes_xts_alg,
- NULL,
-};
-
-int __init p8_init(void)
-{
- int ret = 0;
- struct crypto_alg **alg_it;
-
- for (alg_it = algs; *alg_it; alg_it++) {
- ret = crypto_register_alg(*alg_it);
- printk(KERN_INFO "crypto_register_alg '%s' = %d\n",
- (*alg_it)->cra_name, ret);
- if (ret) {
- for (alg_it--; alg_it >= algs; alg_it--)
- crypto_unregister_alg(*alg_it);
- break;
- }
- }
- if (ret)
- return ret;
-
- ret = crypto_register_shash(&p8_ghash_alg);
- if (ret) {
- for (alg_it = algs; *alg_it; alg_it++)
- crypto_unregister_alg(*alg_it);
- }
- return ret;
-}
-
-void __exit p8_exit(void)
-{
- struct crypto_alg **alg_it;
-
- for (alg_it = algs; *alg_it; alg_it++) {
- printk(KERN_INFO "Removing '%s'\n", (*alg_it)->cra_name);
- crypto_unregister_alg(*alg_it);
- }
- crypto_unregister_shash(&p8_ghash_alg);
-}
-
-module_cpu_feature_match(PPC_MODULE_FEATURE_VEC_CRYPTO, p8_init);
-module_exit(p8_exit);
-
-MODULE_AUTHOR("Marcelo Cerri<mhcerri@br.ibm.com>");
-MODULE_DESCRIPTION("IBM VMX cryptographic acceleration instructions "
- "support on Power 8");
-MODULE_LICENSE("GPL");
-MODULE_VERSION("1.0.0");
--
2.10.2
^ permalink raw reply related
* Re: [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Greg Kroah-Hartman @ 2017-03-29 14:51 UTC (permalink / raw)
To: Michal Suchanek, Leonidas S. Barbosa, Paulo Flabiano Smorigo
Cc: Herbert Xu, David S. Miller, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman, Geert Uytterhoeven,
Mauro Carvalho Chehab, linux-kernel, linux-crypto, linuxppc-dev
In-Reply-To: <20170329125639.14288-1-msuchanek@suse.de>
On Wed, Mar 29, 2017 at 02:56:39PM +0200, Michal Suchanek wrote:
> While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
> subroutines for XTS") which adds the OpenSSL license header to
> drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
> qestion. The whole license reads:
>
> # Licensed under the OpenSSL license (the "License"). You may not use
> # this file except in compliance with the License. You can obtain a
> # copy
> # in the file LICENSE in the source distribution or at
> # https://www.openssl.org/source/license.html
>
> #
> # ====================================================================
> # Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
> # project. The module is, however, dual licensed under OpenSSL and
> # CRYPTOGAMS licenses depending on where you obtain it. For further
> # details see http://www.openssl.org/~appro/cryptogams/.
> # ====================================================================
>
> After seeking legal advice it is still not clear that this driver can be
> legally used in Linux. In particular the "depending on where you obtain
> it" part does not make it clear when you can apply the GPL and when the
> OpenSSL license.
>
> I tried contacting the author of the code for clarification but did not
> hear back. In absence of clear licensing the only solution I see is
> removing this code.
>
> Signed-off-by: Michal Suchanek <msuchanek@suse.de>
> ---
> MAINTAINERS | 12 -
> drivers/crypto/Kconfig | 8 -
> drivers/crypto/Makefile | 1 -
> drivers/crypto/vmx/.gitignore | 2 -
> drivers/crypto/vmx/Kconfig | 9 -
> drivers/crypto/vmx/Makefile | 21 -
> drivers/crypto/vmx/aes.c | 150 --
> drivers/crypto/vmx/aes_cbc.c | 202 --
> drivers/crypto/vmx/aes_ctr.c | 191 --
> drivers/crypto/vmx/aes_xts.c | 190 --
> drivers/crypto/vmx/aesp8-ppc.h | 25 -
> drivers/crypto/vmx/aesp8-ppc.pl | 3789 -------------------------------------
> drivers/crypto/vmx/ghash.c | 227 ---
> drivers/crypto/vmx/ghashp8-ppc.pl | 234 ---
> drivers/crypto/vmx/ppc-xlate.pl | 228 ---
> drivers/crypto/vmx/vmx.c | 88 -
> 16 files changed, 5377 deletions(-)
> delete mode 100644 drivers/crypto/vmx/.gitignore
> delete mode 100644 drivers/crypto/vmx/Kconfig
> delete mode 100644 drivers/crypto/vmx/Makefile
> delete mode 100644 drivers/crypto/vmx/aes.c
> delete mode 100644 drivers/crypto/vmx/aes_cbc.c
> delete mode 100644 drivers/crypto/vmx/aes_ctr.c
> delete mode 100644 drivers/crypto/vmx/aes_xts.c
> delete mode 100644 drivers/crypto/vmx/aesp8-ppc.h
> delete mode 100644 drivers/crypto/vmx/aesp8-ppc.pl
> delete mode 100644 drivers/crypto/vmx/ghash.c
> delete mode 100644 drivers/crypto/vmx/ghashp8-ppc.pl
> delete mode 100644 drivers/crypto/vmx/ppc-xlate.pl
> delete mode 100644 drivers/crypto/vmx/vmx.c
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 1b0a87ffffab..fd4cbf046ab4 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6190,18 +6190,6 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
> S: Maintained
> F: arch/ia64/
>
> -IBM Power VMX Cryptographic instructions
> -M: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
> -M: Paulo Flabiano Smorigo <pfsmorigo@linux.vnet.ibm.com>
Any reason why you didn't cc: these maintainers on your patch?
thanks,
greg k-h
^ permalink raw reply
* Re: [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Michal Suchánek @ 2017-03-29 15:13 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Leonidas S. Barbosa, Paulo Flabiano Smorigo, Herbert Xu,
David S. Miller, Benjamin Herrenschmidt, Paul Mackerras,
Michael Ellerman, Geert Uytterhoeven, Mauro Carvalho Chehab,
linux-kernel, linux-crypto, linuxppc-dev
In-Reply-To: <20170329145135.GA28057@kroah.com>
On Wed, 29 Mar 2017 16:51:35 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> On Wed, Mar 29, 2017 at 02:56:39PM +0200, Michal Suchanek wrote:
> > While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
> > subroutines for XTS") which adds the OpenSSL license header to
> > drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
> > qestion. The whole license reads:
> >
> > # Licensed under the OpenSSL license (the "License"). You may not
> > use # this file except in compliance with the License. You can
> > obtain a # copy
> > # in the file LICENSE in the source distribution or at
> > # https://www.openssl.org/source/license.html
> >
> > #
> > #
> > ====================================================================
> > # Written by Andy Polyakov <appro@openssl.org> for the OpenSSL #
> > project. The module is, however, dual licensed under OpenSSL and #
> > CRYPTOGAMS licenses depending on where you obtain it. For further #
> > details see http://www.openssl.org/~appro/cryptogams/. #
> > ====================================================================
> >
> > After seeking legal advice it is still not clear that this driver
> > can be legally used in Linux. In particular the "depending on where
> > you obtain it" part does not make it clear when you can apply the
> > GPL and when the OpenSSL license.
> >
> > I tried contacting the author of the code for clarification but did
> > not hear back. In absence of clear licensing the only solution I
> > see is removing this code.
> >
> > Signed-off-by: Michal Suchanek <msuchanek@suse.de>
> > ---
> > MAINTAINERS | 12 -
> > drivers/crypto/Kconfig | 8 -
> > drivers/crypto/Makefile | 1 -
> > drivers/crypto/vmx/.gitignore | 2 -
> > drivers/crypto/vmx/Kconfig | 9 -
> > drivers/crypto/vmx/Makefile | 21 -
> > drivers/crypto/vmx/aes.c | 150 --
> > drivers/crypto/vmx/aes_cbc.c | 202 --
> > drivers/crypto/vmx/aes_ctr.c | 191 --
> > drivers/crypto/vmx/aes_xts.c | 190 --
> > drivers/crypto/vmx/aesp8-ppc.h | 25 -
> > drivers/crypto/vmx/aesp8-ppc.pl | 3789
> > -------------------------------------
> > drivers/crypto/vmx/ghash.c | 227 ---
> > drivers/crypto/vmx/ghashp8-ppc.pl | 234 ---
> > drivers/crypto/vmx/ppc-xlate.pl | 228 ---
> > drivers/crypto/vmx/vmx.c | 88 - 16 files changed, 5377
> > deletions(-) delete mode 100644 drivers/crypto/vmx/.gitignore
> > delete mode 100644 drivers/crypto/vmx/Kconfig
> > delete mode 100644 drivers/crypto/vmx/Makefile
> > delete mode 100644 drivers/crypto/vmx/aes.c
> > delete mode 100644 drivers/crypto/vmx/aes_cbc.c
> > delete mode 100644 drivers/crypto/vmx/aes_ctr.c
> > delete mode 100644 drivers/crypto/vmx/aes_xts.c
> > delete mode 100644 drivers/crypto/vmx/aesp8-ppc.h
> > delete mode 100644 drivers/crypto/vmx/aesp8-ppc.pl
> > delete mode 100644 drivers/crypto/vmx/ghash.c
> > delete mode 100644 drivers/crypto/vmx/ghashp8-ppc.pl
> > delete mode 100644 drivers/crypto/vmx/ppc-xlate.pl
> > delete mode 100644 drivers/crypto/vmx/vmx.c
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 1b0a87ffffab..fd4cbf046ab4 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6190,18 +6190,6 @@ T: git
> > git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
> > S: Maintained F: arch/ia64/
> >
> > -IBM Power VMX Cryptographic instructions
> > -M: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
> > -M: Paulo Flabiano Smorigo <pfsmorigo@linux.vnet.ibm.com>
>
> Any reason why you didn't cc: these maintainers on your patch?
>
I used get_maintainers.pl with a filter that turns it into a valid
e-mail list and did not particularly thoroughly check the output.
Removing the maintainers from MAINTAINERS in the patch is probably what
causes the omission.
Thanks
Michal
^ permalink raw reply
* Re: [RFC PATCH v2 18/32] kvm: svm: Use the hardware provided GPA instead of page walk
From: Borislav Petkov @ 2017-03-29 15:14 UTC (permalink / raw)
To: Brijesh Singh
Cc: linux-efi, kvm, rkrcmar, matt, linux-pci, linus.walleij,
gary.hook, linux-mm, hpa, cl, tglx, aarcange, sfr, mchehab,
simon.guinot, bhe, xemul, joro, x86, peterz, piotr.luc, mingo,
msalter, ross.zwisler, labbott, dyoung, thomas.lendacky, jroedel,
keescook, arnd, toshi.kani, mathieu.desnoyers, luto, pbonzini,
bhelgaas, dan.j.williams, andriy.shevchenko, akpm, herbert,
tony.luck, pau
In-Reply-To: <148846776540.2349.3123530065053870721.stgit@brijesh-build-machine>
On Thu, Mar 02, 2017 at 10:16:05AM -0500, Brijesh Singh wrote:
> From: Tom Lendacky <thomas.lendacky@amd.com>
>
> When a guest causes a NPF which requires emulation, KVM sometimes walks
> the guest page tables to translate the GVA to a GPA. This is unnecessary
> most of the time on AMD hardware since the hardware provides the GPA in
> EXITINFO2.
>
> The only exception cases involve string operations involving rep or
> operations that use two memory locations. With rep, the GPA will only be
> the value of the initial NPF and with dual memory locations we won't know
> which memory address was translated into EXITINFO2.
>
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> Reviewed-by: Borislav Petkov <bp@suse.de>
I think I already asked you to remove Revewed-by tags when you have to
change an already reviewed patch in non-trivial manner. Why does this
one still have my Reviewed-by tag?
--
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
^ permalink raw reply
* Re: [RFC PATCH v2 16/32] x86: kvm: Provide support to create Guest and HV shared per-CPU variables
From: Paolo Bonzini @ 2017-03-29 15:21 UTC (permalink / raw)
To: Borislav Petkov, Brijesh Singh
Cc: simon.guinot, linux-efi, kvm, rkrcmar, matt, linux-pci,
linus.walleij, gary.hook, linux-mm, paul.gortmaker, hpa, cl,
dan.j.williams, aarcange, sfr, andriy.shevchenko, herbert, bhe,
xemul, joro, x86, peterz, piotr.luc, mingo, msalter, ross.zwisler,
dyoung, thomas.lendacky, jroedel, keescook, arnd, toshi.kani,
mathieu.desnoyers, luto, devel, bhelgaas
In-Reply-To: <20170328183931.rqorduu5fnp5r3y2@pd.tnic>
On 28/03/2017 20:39, Borislav Petkov wrote:
>> 2) Since the encryption attributes works on PAGE_SIZE hence add some extra
>> padding to 'struct kvm-steal-time' to make it PAGE_SIZE and then at runtime
>> clear the encryption attribute of the full PAGE. The downside of this was
>> now we need to modify structure which may break the compatibility.
> From SEV-ES whitepaper:
>
> "To facilitate this communication, the SEV-ES architecture defines
> a Guest Hypervisor Communication Block (GHCB). The GHCB resides in
> page of shared memory so it is accessible to both the guest VM and the
> hypervisor."
>
> So this is kinda begging to be implemented with a shared page between
> guest and host. And then put steal-time, ... etc in there too. Provided
> there's enough room in the single page for the GHCB *and* our stuff.
The GHCB would have to be allocated much earlier, possibly even by
firmware depending on how things will be designed. I think it's
premature to consider SEV-ES requirements.
Paolo
^ permalink raw reply
* Re: [RFC PATCH v2 16/32] x86: kvm: Provide support to create Guest and HV shared per-CPU variables
From: Borislav Petkov @ 2017-03-29 15:32 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Brijesh Singh, simon.guinot, linux-efi, kvm, rkrcmar, matt,
linux-pci, linus.walleij, gary.hook, linux-mm, paul.gortmaker,
hpa, cl, dan.j.williams, aarcange, sfr, andriy.shevchenko,
herbert, bhe, xemul, joro, x86, peterz, piotr.luc, mingo, msalter,
ross.zwisler, dyoung, thomas.lendacky, jroedel, keescook, arnd,
toshi.kani, mathieu.desnoyers, luto, devel, bhelgaas, tglx,
mchehab
In-Reply-To: <9a8723fc-300d-eb76-deb1-cbc8492e9d49@redhat.com>
On Wed, Mar 29, 2017 at 05:21:13PM +0200, Paolo Bonzini wrote:
> The GHCB would have to be allocated much earlier, possibly even by
> firmware depending on how things will be designed.
How about a statically allocated page like we do with the early
pagetable pages in head_64.S?
> I think it's premature to consider SEV-ES requirements.
My only concern is not to have to redo a lot when SEV-ES gets enabled.
So it would be prudent to design with SEV-ES in the back of our minds.
--
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC PATCH v2 18/32] kvm: svm: Use the hardware provided GPA instead of page walk
From: Brijesh Singh @ 2017-03-29 17:08 UTC (permalink / raw)
To: Borislav Petkov
Cc: brijesh.singh, simon.guinot, linux-efi, kvm, rkrcmar, matt,
linux-pci, linus.walleij, gary.hook, linux-mm, paul.gortmaker,
hpa, cl, dan.j.williams, aarcange, sfr, andriy.shevchenko,
herbert, bhe, xemul, joro, x86, peterz, piotr.luc, mingo, msalter,
ross.zwisler, dyoung, thomas.lendacky, jroedel, keescook, arnd,
toshi.kani
In-Reply-To: <20170329151413.l2on26mdyyskwqlu@pd.tnic>
Hi Boris,
On 03/29/2017 10:14 AM, Borislav Petkov wrote:
> On Thu, Mar 02, 2017 at 10:16:05AM -0500, Brijesh Singh wrote:
>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>
>> When a guest causes a NPF which requires emulation, KVM sometimes walks
>> the guest page tables to translate the GVA to a GPA. This is unnecessary
>> most of the time on AMD hardware since the hardware provides the GPA in
>> EXITINFO2.
>>
>> The only exception cases involve string operations involving rep or
>> operations that use two memory locations. With rep, the GPA will only be
>> the value of the initial NPF and with dual memory locations we won't know
>> which memory address was translated into EXITINFO2.
>>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> Reviewed-by: Borislav Petkov <bp@suse.de>
>
> I think I already asked you to remove Revewed-by tags when you have to
> change an already reviewed patch in non-trivial manner. Why does this
> one still have my Reviewed-by tag?
>
Actually this patch is included in RFCv2 series for the completeness.
The patch is already been reviewed and accepted in kvm upstream tree but it
was not present in the tip branch hence I cherry-picked into RFC so that we do
not break the build. SEV runtime behavior needs this patch. I have tried to
highlight it in cover letter. It was my bad that I missed fixing the Reviewed-by
tag during cherry picking. Sorry about that and will be extra careful next time around. Thanks
~ Brijesh
^ permalink raw reply
* Re: [RFC TLS Offload Support 00/15] cover letter
From: David Miller @ 2017-03-29 17:41 UTC (permalink / raw)
To: aviadye
Cc: ilyal, borisp, davejwatson, netdev, matanb, liranl, haggaie, tom,
herbert, nmav, fridolin.pokorny, ilant, kliteyn, linux-crypto,
saeedm, aviadye
In-Reply-To: <1490707592-1430-1-git-send-email-aviadye@mellanox.com>
From: Aviad Yehezkel <aviadye@mellanox.com>
Date: Tue, 28 Mar 2017 16:26:17 +0300
> TLS Tx crypto offload is a new feature of network devices. It
> enables the kernel TLS socket to skip encryption and authentication
> operations on the transmit side of the data path, delegating those
> to the NIC. In turn, the NIC encrypts packets that belong to an
> offloaded TLS socket on the fly. The NIC does not modify any packet
> headers. It expects to receive fully framed TCP packets with TLS
> records as payload. The NIC replaces plaintext with ciphertext and
> fills the authentication tag. The NIC does not hold any state beyond
> the context needed to encrypt the next expected packet,
> i.e. expected TCP sequence number and crypto state.
It seems like, since you do the TLS framing in TCP and the card is
expecting to fill in certain aspects, there is a requirement that the
packet contents aren't mangled between the TLS framing code and when
the SKB hits the card.
Is this right?
For example, what happens if netfilter splits a TLS Tx offloaded frame
into two TCP segments?
^ permalink raw reply
* Re: [RFC TLS Offload Support 00/15] cover letter
From: Hannes Frederic Sowa @ 2017-03-29 18:03 UTC (permalink / raw)
To: David Miller, aviadye
Cc: ilyal, borisp, davejwatson, netdev, matanb, liranl, haggaie, tom,
herbert, nmav, fridolin.pokorny, ilant, kliteyn, linux-crypto,
saeedm, aviadye
In-Reply-To: <20170329.104122.139835608303809432.davem@davemloft.net>
Hello,
On 29.03.2017 19:41, David Miller wrote:
> From: Aviad Yehezkel <aviadye@mellanox.com>
> Date: Tue, 28 Mar 2017 16:26:17 +0300
>
>> TLS Tx crypto offload is a new feature of network devices. It
>> enables the kernel TLS socket to skip encryption and authentication
>> operations on the transmit side of the data path, delegating those
>> to the NIC. In turn, the NIC encrypts packets that belong to an
>> offloaded TLS socket on the fly. The NIC does not modify any packet
>> headers. It expects to receive fully framed TCP packets with TLS
>> records as payload. The NIC replaces plaintext with ciphertext and
>> fills the authentication tag. The NIC does not hold any state beyond
>> the context needed to encrypt the next expected packet,
>> i.e. expected TCP sequence number and crypto state.
>
> It seems like, since you do the TLS framing in TCP and the card is
> expecting to fill in certain aspects, there is a requirement that the
> packet contents aren't mangled between the TLS framing code and when
> the SKB hits the card.
>
> Is this right?
>
> For example, what happens if netfilter splits a TLS Tx offloaded frame
> into two TCP segments?
Furthermore, it doesn't seem to work with bonding or any other virtual
interface, which could move the skb's to be processed on another NIC, as
the context is put onto the NIC. Even a redirect can not be processed
anymore (seems like those patches try to stick the connection to an
interface anyway).
Wouldn't it be possible to keep the state in software and push down a
security context per skb, which get applied during sending? If not
possible via hw, slowpath can encrypt packet in sw.
Also sticking connections to outgoing interfaces might work for TX, but
you can't force the interface where packets come in.
Bye,
Hannes
^ permalink raw reply
* [PATCH 1/6] virtio: wrap find_vqs
From: Michael S. Tsirkin @ 2017-03-29 20:48 UTC (permalink / raw)
To: linux-kernel
Cc: Dmitry Tarnyagin, kvm, David Airlie, linux-remoteproc, dri-devel,
Bjorn Andersson, James E.J. Bottomley, Herbert Xu, linux-scsi,
John Fastabend, Arnd Bergmann, Amit Shah, Stefan Hajnoczi,
virtualization, Martin K. Petersen, Greg Kroah-Hartman,
linux-crypto, netdev, David S. Miller
In-Reply-To: <1490820507-8005-1-git-send-email-mst@redhat.com>
We are going to add more parameters to find_vqs, let's wrap the call so
we don't need to tweak all drivers every time.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
drivers/block/virtio_blk.c | 3 +--
drivers/char/virtio_console.c | 6 +++---
drivers/crypto/virtio/virtio_crypto_core.c | 3 +--
drivers/gpu/drm/virtio/virtgpu_kms.c | 3 +--
drivers/net/caif/caif_virtio.c | 3 +--
drivers/net/virtio_net.c | 3 +--
drivers/rpmsg/virtio_rpmsg_bus.c | 2 +-
drivers/scsi/virtio_scsi.c | 3 +--
drivers/virtio/virtio_balloon.c | 3 +--
drivers/virtio/virtio_input.c | 3 +--
include/linux/virtio_config.h | 9 +++++++++
net/vmw_vsock/virtio_transport.c | 6 +++---
12 files changed, 24 insertions(+), 23 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 1d4c9f8..c08c30c 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -455,8 +455,7 @@ static int init_vq(struct virtio_blk *vblk)
}
/* Discover virtqueues and write information to configuration. */
- err = vdev->config->find_vqs(vdev, num_vqs, vqs, callbacks, names,
- &desc);
+ err = virtio_find_vqs(vdev, num_vqs, vqs, callbacks, names, &desc);
if (err)
goto out;
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index e9b7e0b..5da4c8e 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1945,9 +1945,9 @@ static int init_vqs(struct ports_device *portdev)
}
}
/* Find the queues. */
- err = portdev->vdev->config->find_vqs(portdev->vdev, nr_queues, vqs,
- io_callbacks,
- (const char **)io_names, NULL);
+ err = virtio_find_vqs(portdev->vdev, nr_queues, vqs,
+ io_callbacks,
+ (const char **)io_names, NULL);
if (err)
goto free;
diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
index 21472e4..a111cd72 100644
--- a/drivers/crypto/virtio/virtio_crypto_core.c
+++ b/drivers/crypto/virtio/virtio_crypto_core.c
@@ -119,8 +119,7 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi)
names[i] = vi->data_vq[i].name;
}
- ret = vi->vdev->config->find_vqs(vi->vdev, total_vqs, vqs, callbacks,
- names, NULL);
+ ret = virtio_find_vqs(vi->vdev, total_vqs, vqs, callbacks, names, NULL);
if (ret)
goto err_find;
diff --git a/drivers/gpu/drm/virtio/virtgpu_kms.c b/drivers/gpu/drm/virtio/virtgpu_kms.c
index 4918668..1e1c90b 100644
--- a/drivers/gpu/drm/virtio/virtgpu_kms.c
+++ b/drivers/gpu/drm/virtio/virtgpu_kms.c
@@ -175,8 +175,7 @@ int virtio_gpu_driver_load(struct drm_device *dev, unsigned long flags)
DRM_INFO("virgl 3d acceleration not supported by guest\n");
#endif
- ret = vgdev->vdev->config->find_vqs(vgdev->vdev, 2, vqs,
- callbacks, names, NULL);
+ ret = virtio_find_vqs(vgdev->vdev, 2, vqs, callbacks, names, NULL);
if (ret) {
DRM_ERROR("failed to find virt queues\n");
goto err_vqs;
diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
index bc0eb47..6122768 100644
--- a/drivers/net/caif/caif_virtio.c
+++ b/drivers/net/caif/caif_virtio.c
@@ -679,8 +679,7 @@ static int cfv_probe(struct virtio_device *vdev)
goto err;
/* Get the TX virtio ring. This is a "guest side vring". */
- err = vdev->config->find_vqs(vdev, 1, &cfv->vq_tx, &vq_cbs, &names,
- NULL);
+ err = virtio_find_vqs(vdev, 1, &cfv->vq_tx, &vq_cbs, &names, NULL);
if (err)
goto err;
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index ea9890d..6802169 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -2079,8 +2079,7 @@ static int virtnet_find_vqs(struct virtnet_info *vi)
names[txq2vq(i)] = vi->sq[i].name;
}
- ret = vi->vdev->config->find_vqs(vi->vdev, total_vqs, vqs, callbacks,
- names, NULL);
+ ret = virtio_find_vqs(vi->vdev, total_vqs, vqs, callbacks, names, NULL);
if (ret)
goto err_find;
diff --git a/drivers/rpmsg/virtio_rpmsg_bus.c b/drivers/rpmsg/virtio_rpmsg_bus.c
index 5e66e08..f7cade0 100644
--- a/drivers/rpmsg/virtio_rpmsg_bus.c
+++ b/drivers/rpmsg/virtio_rpmsg_bus.c
@@ -869,7 +869,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
init_waitqueue_head(&vrp->sendq);
/* We expect two virtqueues, rx and tx (and in this order) */
- err = vdev->config->find_vqs(vdev, 2, vqs, vq_cbs, names, NULL);
+ err = virtio_find_vqs(vdev, 2, vqs, vq_cbs, names, NULL);
if (err)
goto free_vrp;
diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
index 939c47d..e9222dc 100644
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -870,8 +870,7 @@ static int virtscsi_init(struct virtio_device *vdev,
}
/* Discover virtqueues and write information to configuration. */
- err = vdev->config->find_vqs(vdev, num_vqs, vqs, callbacks, names,
- &desc);
+ err = virtio_find_vqs(vdev, num_vqs, vqs, callbacks, names, &desc);
if (err)
goto out;
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 34adf9b..408c174 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -418,8 +418,7 @@ static int init_vqs(struct virtio_balloon *vb)
* optionally stat.
*/
nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
- err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names,
- NULL);
+ err = virtio_find_vqs(vb->vdev, nvqs, vqs, callbacks, names, NULL);
if (err)
return err;
diff --git a/drivers/virtio/virtio_input.c b/drivers/virtio/virtio_input.c
index 79f1293..3a0468f 100644
--- a/drivers/virtio/virtio_input.c
+++ b/drivers/virtio/virtio_input.c
@@ -173,8 +173,7 @@ static int virtinput_init_vqs(struct virtio_input *vi)
static const char * const names[] = { "events", "status" };
int err;
- err = vi->vdev->config->find_vqs(vi->vdev, 2, vqs, cbs, names,
- NULL);
+ err = virtio_find_vqs(vi->vdev, 2, vqs, cbs, names, NULL);
if (err)
return err;
vi->evt = vqs[0];
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 8355bab..47f3d80 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -179,6 +179,15 @@ struct virtqueue *virtio_find_single_vq(struct virtio_device *vdev,
return vq;
}
+static inline
+int virtio_find_vqs(struct virtio_device *vdev, unsigned nvqs,
+ struct virtqueue *vqs[], vq_callback_t *callbacks[],
+ const char * const names[],
+ struct irq_affinity *desc)
+{
+ return vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names, desc);
+}
+
/**
* virtio_device_ready - enable vq use in probe function
* @vdev: the device
diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
index 68675a1..97e26e2 100644
--- a/net/vmw_vsock/virtio_transport.c
+++ b/net/vmw_vsock/virtio_transport.c
@@ -573,9 +573,9 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
vsock->vdev = vdev;
- ret = vsock->vdev->config->find_vqs(vsock->vdev, VSOCK_VQ_MAX,
- vsock->vqs, callbacks, names,
- NULL);
+ ret = virtio_find_vqs(vsock->vdev, VSOCK_VQ_MAX,
+ vsock->vqs, callbacks, names,
+ NULL);
if (ret < 0)
goto out;
--
MST
^ permalink raw reply related
* Re: [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Tyrel Datwyler @ 2017-03-29 23:08 UTC (permalink / raw)
To: Michal Suchánek, Greg Kroah-Hartman
Cc: Leonidas S. Barbosa, Herbert Xu, Geert Uytterhoeven, linux-kernel,
Paul Mackerras, linux-crypto, Paulo Flabiano Smorigo,
Mauro Carvalho Chehab, linuxppc-dev, David S. Miller
In-Reply-To: <20170329171327.38d4fdd6@kitsune.suse.cz>
On 03/29/2017 08:13 AM, Michal Suchánek wrote:
> On Wed, 29 Mar 2017 16:51:35 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
>
>> On Wed, Mar 29, 2017 at 02:56:39PM +0200, Michal Suchanek wrote:
>>> While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
>>> subroutines for XTS") which adds the OpenSSL license header to
>>> drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
>>> qestion. The whole license reads:
>>>
>>> # Licensed under the OpenSSL license (the "License"). You may not
>>> use # this file except in compliance with the License. You can
>>> obtain a # copy
>>> # in the file LICENSE in the source distribution or at
>>> # https://www.openssl.org/source/license.html
>>>
>>> #
>>> #
>>> ====================================================================
>>> # Written by Andy Polyakov <appro@openssl.org> for the OpenSSL #
>>> project. The module is, however, dual licensed under OpenSSL and #
>>> CRYPTOGAMS licenses depending on where you obtain it. For further #
>>> details see http://www.openssl.org/~appro/cryptogams/. #
>>> ====================================================================
>>>
>>> After seeking legal advice it is still not clear that this driver
>>> can be legally used in Linux. In particular the "depending on where
>>> you obtain it" part does not make it clear when you can apply the
>>> GPL and when the OpenSSL license.
>>>
>>> I tried contacting the author of the code for clarification but did
>>> not hear back. In absence of clear licensing the only solution I
>>> see is removing this code.
A quick 'git grep OpenSSL' of the Linux tree returns several other
crypto files under the ARM architecture that are similarly licensed. Namely:
arch/arm/crypto/sha1-armv4-large.S
arch/arm/crypto/sha256-armv4.pl
arch/arm/crypto/sha256-core.S_shipped
arch/arm/crypto/sha512-armv4.pl
arch/arm/crypto/sha512-core.S_shipped
arch/arm64/crypto/sha256-core.S_shipped
arch/arm64/crypto/sha512-armv8.pl
arch/arm64/crypto/sha512-core.S_shipped
On closer inspection of some of those files have the addendum that
"Permission to use under GPL terms is granted", but not all of them.
-Tyrel
^ permalink raw reply
* Re: [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Tyrel Datwyler @ 2017-03-29 23:29 UTC (permalink / raw)
To: Michal Suchanek, Herbert Xu, David S. Miller,
Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Greg Kroah-Hartman, Geert Uytterhoeven, Mauro Carvalho Chehab,
linux-kernel, linux-crypto, linuxppc-dev, appro
In-Reply-To: <20170329125639.14288-1-msuchanek@suse.de>
Adding Andy to the discussion as he may be able to shine light on
licensing issue of the crypto perl scripts in question.
-Tyrel
On 03/29/2017 05:56 AM, Michal Suchanek wrote:
> While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
> subroutines for XTS") which adds the OpenSSL license header to
> drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
> qestion. The whole license reads:
>
> # Licensed under the OpenSSL license (the "License"). You may not use
> # this file except in compliance with the License. You can obtain a
> # copy
> # in the file LICENSE in the source distribution or at
> # https://www.openssl.org/source/license.html
>
> #
> # ====================================================================
> # Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
> # project. The module is, however, dual licensed under OpenSSL and
> # CRYPTOGAMS licenses depending on where you obtain it. For further
> # details see http://www.openssl.org/~appro/cryptogams/.
> # ====================================================================
>
> After seeking legal advice it is still not clear that this driver can be
> legally used in Linux. In particular the "depending on where you obtain
> it" part does not make it clear when you can apply the GPL and when the
> OpenSSL license.
>
> I tried contacting the author of the code for clarification but did not
> hear back. In absence of clear licensing the only solution I see is
> removing this code.
>
> Signed-off-by: Michal Suchanek <msuchanek@suse.de>
> ---
> MAINTAINERS | 12 -
> drivers/crypto/Kconfig | 8 -
> drivers/crypto/Makefile | 1 -
> drivers/crypto/vmx/.gitignore | 2 -
> drivers/crypto/vmx/Kconfig | 9 -
> drivers/crypto/vmx/Makefile | 21 -
> drivers/crypto/vmx/aes.c | 150 --
> drivers/crypto/vmx/aes_cbc.c | 202 --
> drivers/crypto/vmx/aes_ctr.c | 191 --
> drivers/crypto/vmx/aes_xts.c | 190 --
> drivers/crypto/vmx/aesp8-ppc.h | 25 -
> drivers/crypto/vmx/aesp8-ppc.pl | 3789 -------------------------------------
> drivers/crypto/vmx/ghash.c | 227 ---
> drivers/crypto/vmx/ghashp8-ppc.pl | 234 ---
> drivers/crypto/vmx/ppc-xlate.pl | 228 ---
> drivers/crypto/vmx/vmx.c | 88 -
> 16 files changed, 5377 deletions(-)
> delete mode 100644 drivers/crypto/vmx/.gitignore
> delete mode 100644 drivers/crypto/vmx/Kconfig
> delete mode 100644 drivers/crypto/vmx/Makefile
> delete mode 100644 drivers/crypto/vmx/aes.c
> delete mode 100644 drivers/crypto/vmx/aes_cbc.c
> delete mode 100644 drivers/crypto/vmx/aes_ctr.c
> delete mode 100644 drivers/crypto/vmx/aes_xts.c
> delete mode 100644 drivers/crypto/vmx/aesp8-ppc.h
> delete mode 100644 drivers/crypto/vmx/aesp8-ppc.pl
> delete mode 100644 drivers/crypto/vmx/ghash.c
> delete mode 100644 drivers/crypto/vmx/ghashp8-ppc.pl
> delete mode 100644 drivers/crypto/vmx/ppc-xlate.pl
> delete mode 100644 drivers/crypto/vmx/vmx.c
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 1b0a87ffffab..fd4cbf046ab4 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6190,18 +6190,6 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
> S: Maintained
> F: arch/ia64/
>
> -IBM Power VMX Cryptographic instructions
> -M: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
> -M: Paulo Flabiano Smorigo <pfsmorigo@linux.vnet.ibm.com>
> -L: linux-crypto@vger.kernel.org
> -S: Supported
> -F: drivers/crypto/vmx/Makefile
> -F: drivers/crypto/vmx/Kconfig
> -F: drivers/crypto/vmx/vmx.c
> -F: drivers/crypto/vmx/aes*
> -F: drivers/crypto/vmx/ghash*
> -F: drivers/crypto/vmx/ppc-xlate.pl
> -
> IBM Power in-Nest Crypto Acceleration
> M: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
> M: Paulo Flabiano Smorigo <pfsmorigo@linux.vnet.ibm.com>
> diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
> index 473d31288ad8..9fcd3af1f2f1 100644
> --- a/drivers/crypto/Kconfig
> +++ b/drivers/crypto/Kconfig
> @@ -530,14 +530,6 @@ config CRYPTO_DEV_QCE
> hardware. To compile this driver as a module, choose M here. The
> module will be called qcrypto.
>
> -config CRYPTO_DEV_VMX
> - bool "Support for VMX cryptographic acceleration instructions"
> - depends on PPC64 && VSX
> - help
> - Support for VMX cryptographic acceleration instructions.
> -
> -source "drivers/crypto/vmx/Kconfig"
> -
> config CRYPTO_DEV_IMGTEC_HASH
> tristate "Imagination Technologies hardware hash accelerator"
> depends on MIPS || COMPILE_TEST
> diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
> index 739609471169..486e57e10e7a 100644
> --- a/drivers/crypto/Makefile
> +++ b/drivers/crypto/Makefile
> @@ -34,5 +34,4 @@ obj-$(CONFIG_CRYPTO_DEV_SUN4I_SS) += sunxi-ss/
> obj-$(CONFIG_CRYPTO_DEV_TALITOS) += talitos.o
> obj-$(CONFIG_CRYPTO_DEV_UX500) += ux500/
> obj-$(CONFIG_CRYPTO_DEV_VIRTIO) += virtio/
> -obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
> obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/
> diff --git a/drivers/crypto/vmx/.gitignore b/drivers/crypto/vmx/.gitignore
> deleted file mode 100644
> index af4a7ce4738d..000000000000
> --- a/drivers/crypto/vmx/.gitignore
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -aesp8-ppc.S
> -ghashp8-ppc.S
> diff --git a/drivers/crypto/vmx/Kconfig b/drivers/crypto/vmx/Kconfig
> deleted file mode 100644
> index c3d524ea6998..000000000000
> --- a/drivers/crypto/vmx/Kconfig
> +++ /dev/null
> @@ -1,9 +0,0 @@
> -config CRYPTO_DEV_VMX_ENCRYPT
> - tristate "Encryption acceleration support on P8 CPU"
> - depends on CRYPTO_DEV_VMX
> - select CRYPTO_GHASH
> - default m
> - help
> - Support for VMX cryptographic acceleration instructions on Power8 CPU.
> - This module supports acceleration for AES and GHASH in hardware. If you
> - choose 'M' here, this module will be called vmx-crypto.
> diff --git a/drivers/crypto/vmx/Makefile b/drivers/crypto/vmx/Makefile
> deleted file mode 100644
> index 55f7c392582f..000000000000
> --- a/drivers/crypto/vmx/Makefile
> +++ /dev/null
> @@ -1,21 +0,0 @@
> -obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
> -vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
> -
> -ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
> -TARGET := linux-ppc64le
> -else
> -TARGET := linux-ppc64
> -endif
> -
> -quiet_cmd_perl = PERL $@
> - cmd_perl = $(PERL) $(<) $(TARGET) > $(@)
> -
> -targets += aesp8-ppc.S ghashp8-ppc.S
> -
> -$(obj)/aesp8-ppc.S: $(src)/aesp8-ppc.pl FORCE
> - $(call if_changed,perl)
> -
> -$(obj)/ghashp8-ppc.S: $(src)/ghashp8-ppc.pl FORCE
> - $(call if_changed,perl)
> -
> -clean-files := aesp8-ppc.S ghashp8-ppc.S
> diff --git a/drivers/crypto/vmx/aes.c b/drivers/crypto/vmx/aes.c
> deleted file mode 100644
> index 022c7ab7351a..000000000000
> --- a/drivers/crypto/vmx/aes.c
> +++ /dev/null
> @@ -1,150 +0,0 @@
> -/**
> - * AES routines supporting VMX instructions on the Power 8
> - *
> - * Copyright (C) 2015 International Business Machines Inc.
> - *
> - * This program is free software; you can redistribute it and/or modify
> - * it under the terms of the GNU General Public License as published by
> - * the Free Software Foundation; version 2 only.
> - *
> - * This program is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - * GNU General Public License for more details.
> - *
> - * You should have received a copy of the GNU General Public License
> - * along with this program; if not, write to the Free Software
> - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> - *
> - * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
> - */
> -
> -#include <linux/types.h>
> -#include <linux/err.h>
> -#include <linux/crypto.h>
> -#include <linux/delay.h>
> -#include <linux/hardirq.h>
> -#include <asm/switch_to.h>
> -#include <crypto/aes.h>
> -
> -#include "aesp8-ppc.h"
> -
> -struct p8_aes_ctx {
> - struct crypto_cipher *fallback;
> - struct aes_key enc_key;
> - struct aes_key dec_key;
> -};
> -
> -static int p8_aes_init(struct crypto_tfm *tfm)
> -{
> - const char *alg;
> - struct crypto_cipher *fallback;
> - struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (!(alg = crypto_tfm_alg_name(tfm))) {
> - printk(KERN_ERR "Failed to get algorithm name.\n");
> - return -ENOENT;
> - }
> -
> - fallback = crypto_alloc_cipher(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
> - if (IS_ERR(fallback)) {
> - printk(KERN_ERR
> - "Failed to allocate transformation for '%s': %ld\n",
> - alg, PTR_ERR(fallback));
> - return PTR_ERR(fallback);
> - }
> - printk(KERN_INFO "Using '%s' as fallback implementation.\n",
> - crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
> -
> - crypto_cipher_set_flags(fallback,
> - crypto_cipher_get_flags((struct
> - crypto_cipher *)
> - tfm));
> - ctx->fallback = fallback;
> -
> - return 0;
> -}
> -
> -static void p8_aes_exit(struct crypto_tfm *tfm)
> -{
> - struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (ctx->fallback) {
> - crypto_free_cipher(ctx->fallback);
> - ctx->fallback = NULL;
> - }
> -}
> -
> -static int p8_aes_setkey(struct crypto_tfm *tfm, const u8 *key,
> - unsigned int keylen)
> -{
> - int ret;
> - struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
> - ret += aes_p8_set_decrypt_key(key, keylen * 8, &ctx->dec_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> -
> - ret += crypto_cipher_setkey(ctx->fallback, key, keylen);
> - return ret;
> -}
> -
> -static void p8_aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
> -{
> - struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (in_interrupt()) {
> - crypto_cipher_encrypt_one(ctx->fallback, dst, src);
> - } else {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - aes_p8_encrypt(src, dst, &ctx->enc_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - }
> -}
> -
> -static void p8_aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
> -{
> - struct p8_aes_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (in_interrupt()) {
> - crypto_cipher_decrypt_one(ctx->fallback, dst, src);
> - } else {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - aes_p8_decrypt(src, dst, &ctx->dec_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - }
> -}
> -
> -struct crypto_alg p8_aes_alg = {
> - .cra_name = "aes",
> - .cra_driver_name = "p8_aes",
> - .cra_module = THIS_MODULE,
> - .cra_priority = 1000,
> - .cra_type = NULL,
> - .cra_flags = CRYPTO_ALG_TYPE_CIPHER | CRYPTO_ALG_NEED_FALLBACK,
> - .cra_alignmask = 0,
> - .cra_blocksize = AES_BLOCK_SIZE,
> - .cra_ctxsize = sizeof(struct p8_aes_ctx),
> - .cra_init = p8_aes_init,
> - .cra_exit = p8_aes_exit,
> - .cra_cipher = {
> - .cia_min_keysize = AES_MIN_KEY_SIZE,
> - .cia_max_keysize = AES_MAX_KEY_SIZE,
> - .cia_setkey = p8_aes_setkey,
> - .cia_encrypt = p8_aes_encrypt,
> - .cia_decrypt = p8_aes_decrypt,
> - },
> -};
> diff --git a/drivers/crypto/vmx/aes_cbc.c b/drivers/crypto/vmx/aes_cbc.c
> deleted file mode 100644
> index 72a26eb4e954..000000000000
> --- a/drivers/crypto/vmx/aes_cbc.c
> +++ /dev/null
> @@ -1,202 +0,0 @@
> -/**
> - * AES CBC routines supporting VMX instructions on the Power 8
> - *
> - * Copyright (C) 2015 International Business Machines Inc.
> - *
> - * This program is free software; you can redistribute it and/or modify
> - * it under the terms of the GNU General Public License as published by
> - * the Free Software Foundation; version 2 only.
> - *
> - * This program is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - * GNU General Public License for more details.
> - *
> - * You should have received a copy of the GNU General Public License
> - * along with this program; if not, write to the Free Software
> - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> - *
> - * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
> - */
> -
> -#include <linux/types.h>
> -#include <linux/err.h>
> -#include <linux/crypto.h>
> -#include <linux/delay.h>
> -#include <linux/hardirq.h>
> -#include <asm/switch_to.h>
> -#include <crypto/aes.h>
> -#include <crypto/scatterwalk.h>
> -#include <crypto/skcipher.h>
> -
> -#include "aesp8-ppc.h"
> -
> -struct p8_aes_cbc_ctx {
> - struct crypto_skcipher *fallback;
> - struct aes_key enc_key;
> - struct aes_key dec_key;
> -};
> -
> -static int p8_aes_cbc_init(struct crypto_tfm *tfm)
> -{
> - const char *alg;
> - struct crypto_skcipher *fallback;
> - struct p8_aes_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (!(alg = crypto_tfm_alg_name(tfm))) {
> - printk(KERN_ERR "Failed to get algorithm name.\n");
> - return -ENOENT;
> - }
> -
> - fallback = crypto_alloc_skcipher(alg, 0,
> - CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
> -
> - if (IS_ERR(fallback)) {
> - printk(KERN_ERR
> - "Failed to allocate transformation for '%s': %ld\n",
> - alg, PTR_ERR(fallback));
> - return PTR_ERR(fallback);
> - }
> - printk(KERN_INFO "Using '%s' as fallback implementation.\n",
> - crypto_skcipher_driver_name(fallback));
> -
> -
> - crypto_skcipher_set_flags(
> - fallback,
> - crypto_skcipher_get_flags((struct crypto_skcipher *)tfm));
> - ctx->fallback = fallback;
> -
> - return 0;
> -}
> -
> -static void p8_aes_cbc_exit(struct crypto_tfm *tfm)
> -{
> - struct p8_aes_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (ctx->fallback) {
> - crypto_free_skcipher(ctx->fallback);
> - ctx->fallback = NULL;
> - }
> -}
> -
> -static int p8_aes_cbc_setkey(struct crypto_tfm *tfm, const u8 *key,
> - unsigned int keylen)
> -{
> - int ret;
> - struct p8_aes_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
> - ret += aes_p8_set_decrypt_key(key, keylen * 8, &ctx->dec_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> -
> - ret += crypto_skcipher_setkey(ctx->fallback, key, keylen);
> - return ret;
> -}
> -
> -static int p8_aes_cbc_encrypt(struct blkcipher_desc *desc,
> - struct scatterlist *dst,
> - struct scatterlist *src, unsigned int nbytes)
> -{
> - int ret;
> - struct blkcipher_walk walk;
> - struct p8_aes_cbc_ctx *ctx =
> - crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
> -
> - if (in_interrupt()) {
> - SKCIPHER_REQUEST_ON_STACK(req, ctx->fallback);
> - skcipher_request_set_tfm(req, ctx->fallback);
> - skcipher_request_set_callback(req, desc->flags, NULL, NULL);
> - skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
> - ret = crypto_skcipher_encrypt(req);
> - skcipher_request_zero(req);
> - } else {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> -
> - blkcipher_walk_init(&walk, dst, src, nbytes);
> - ret = blkcipher_walk_virt(desc, &walk);
> - while ((nbytes = walk.nbytes)) {
> - aes_p8_cbc_encrypt(walk.src.virt.addr,
> - walk.dst.virt.addr,
> - nbytes & AES_BLOCK_MASK,
> - &ctx->enc_key, walk.iv, 1);
> - nbytes &= AES_BLOCK_SIZE - 1;
> - ret = blkcipher_walk_done(desc, &walk, nbytes);
> - }
> -
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - }
> -
> - return ret;
> -}
> -
> -static int p8_aes_cbc_decrypt(struct blkcipher_desc *desc,
> - struct scatterlist *dst,
> - struct scatterlist *src, unsigned int nbytes)
> -{
> - int ret;
> - struct blkcipher_walk walk;
> - struct p8_aes_cbc_ctx *ctx =
> - crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
> -
> - if (in_interrupt()) {
> - SKCIPHER_REQUEST_ON_STACK(req, ctx->fallback);
> - skcipher_request_set_tfm(req, ctx->fallback);
> - skcipher_request_set_callback(req, desc->flags, NULL, NULL);
> - skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
> - ret = crypto_skcipher_decrypt(req);
> - skcipher_request_zero(req);
> - } else {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> -
> - blkcipher_walk_init(&walk, dst, src, nbytes);
> - ret = blkcipher_walk_virt(desc, &walk);
> - while ((nbytes = walk.nbytes)) {
> - aes_p8_cbc_encrypt(walk.src.virt.addr,
> - walk.dst.virt.addr,
> - nbytes & AES_BLOCK_MASK,
> - &ctx->dec_key, walk.iv, 0);
> - nbytes &= AES_BLOCK_SIZE - 1;
> - ret = blkcipher_walk_done(desc, &walk, nbytes);
> - }
> -
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - }
> -
> - return ret;
> -}
> -
> -
> -struct crypto_alg p8_aes_cbc_alg = {
> - .cra_name = "cbc(aes)",
> - .cra_driver_name = "p8_aes_cbc",
> - .cra_module = THIS_MODULE,
> - .cra_priority = 2000,
> - .cra_type = &crypto_blkcipher_type,
> - .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER | CRYPTO_ALG_NEED_FALLBACK,
> - .cra_alignmask = 0,
> - .cra_blocksize = AES_BLOCK_SIZE,
> - .cra_ctxsize = sizeof(struct p8_aes_cbc_ctx),
> - .cra_init = p8_aes_cbc_init,
> - .cra_exit = p8_aes_cbc_exit,
> - .cra_blkcipher = {
> - .ivsize = AES_BLOCK_SIZE,
> - .min_keysize = AES_MIN_KEY_SIZE,
> - .max_keysize = AES_MAX_KEY_SIZE,
> - .setkey = p8_aes_cbc_setkey,
> - .encrypt = p8_aes_cbc_encrypt,
> - .decrypt = p8_aes_cbc_decrypt,
> - },
> -};
> diff --git a/drivers/crypto/vmx/aes_ctr.c b/drivers/crypto/vmx/aes_ctr.c
> deleted file mode 100644
> index 7cf6d31c1123..000000000000
> --- a/drivers/crypto/vmx/aes_ctr.c
> +++ /dev/null
> @@ -1,191 +0,0 @@
> -/**
> - * AES CTR routines supporting VMX instructions on the Power 8
> - *
> - * Copyright (C) 2015 International Business Machines Inc.
> - *
> - * This program is free software; you can redistribute it and/or modify
> - * it under the terms of the GNU General Public License as published by
> - * the Free Software Foundation; version 2 only.
> - *
> - * This program is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - * GNU General Public License for more details.
> - *
> - * You should have received a copy of the GNU General Public License
> - * along with this program; if not, write to the Free Software
> - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> - *
> - * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
> - */
> -
> -#include <linux/types.h>
> -#include <linux/err.h>
> -#include <linux/crypto.h>
> -#include <linux/delay.h>
> -#include <linux/hardirq.h>
> -#include <asm/switch_to.h>
> -#include <crypto/aes.h>
> -#include <crypto/scatterwalk.h>
> -#include "aesp8-ppc.h"
> -
> -struct p8_aes_ctr_ctx {
> - struct crypto_blkcipher *fallback;
> - struct aes_key enc_key;
> -};
> -
> -static int p8_aes_ctr_init(struct crypto_tfm *tfm)
> -{
> - const char *alg;
> - struct crypto_blkcipher *fallback;
> - struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (!(alg = crypto_tfm_alg_name(tfm))) {
> - printk(KERN_ERR "Failed to get algorithm name.\n");
> - return -ENOENT;
> - }
> -
> - fallback =
> - crypto_alloc_blkcipher(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
> - if (IS_ERR(fallback)) {
> - printk(KERN_ERR
> - "Failed to allocate transformation for '%s': %ld\n",
> - alg, PTR_ERR(fallback));
> - return PTR_ERR(fallback);
> - }
> - printk(KERN_INFO "Using '%s' as fallback implementation.\n",
> - crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
> -
> - crypto_blkcipher_set_flags(
> - fallback,
> - crypto_blkcipher_get_flags((struct crypto_blkcipher *)tfm));
> - ctx->fallback = fallback;
> -
> - return 0;
> -}
> -
> -static void p8_aes_ctr_exit(struct crypto_tfm *tfm)
> -{
> - struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (ctx->fallback) {
> - crypto_free_blkcipher(ctx->fallback);
> - ctx->fallback = NULL;
> - }
> -}
> -
> -static int p8_aes_ctr_setkey(struct crypto_tfm *tfm, const u8 *key,
> - unsigned int keylen)
> -{
> - int ret;
> - struct p8_aes_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - ret = aes_p8_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> -
> - ret += crypto_blkcipher_setkey(ctx->fallback, key, keylen);
> - return ret;
> -}
> -
> -static void p8_aes_ctr_final(struct p8_aes_ctr_ctx *ctx,
> - struct blkcipher_walk *walk)
> -{
> - u8 *ctrblk = walk->iv;
> - u8 keystream[AES_BLOCK_SIZE];
> - u8 *src = walk->src.virt.addr;
> - u8 *dst = walk->dst.virt.addr;
> - unsigned int nbytes = walk->nbytes;
> -
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - aes_p8_encrypt(ctrblk, keystream, &ctx->enc_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> -
> - crypto_xor(keystream, src, nbytes);
> - memcpy(dst, keystream, nbytes);
> - crypto_inc(ctrblk, AES_BLOCK_SIZE);
> -}
> -
> -static int p8_aes_ctr_crypt(struct blkcipher_desc *desc,
> - struct scatterlist *dst,
> - struct scatterlist *src, unsigned int nbytes)
> -{
> - int ret;
> - u64 inc;
> - struct blkcipher_walk walk;
> - struct p8_aes_ctr_ctx *ctx =
> - crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
> - struct blkcipher_desc fallback_desc = {
> - .tfm = ctx->fallback,
> - .info = desc->info,
> - .flags = desc->flags
> - };
> -
> - if (in_interrupt()) {
> - ret = crypto_blkcipher_encrypt(&fallback_desc, dst, src,
> - nbytes);
> - } else {
> - blkcipher_walk_init(&walk, dst, src, nbytes);
> - ret = blkcipher_walk_virt_block(desc, &walk, AES_BLOCK_SIZE);
> - while ((nbytes = walk.nbytes) >= AES_BLOCK_SIZE) {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - aes_p8_ctr32_encrypt_blocks(walk.src.virt.addr,
> - walk.dst.virt.addr,
> - (nbytes &
> - AES_BLOCK_MASK) /
> - AES_BLOCK_SIZE,
> - &ctx->enc_key,
> - walk.iv);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> -
> - /* We need to update IV mostly for last bytes/round */
> - inc = (nbytes & AES_BLOCK_MASK) / AES_BLOCK_SIZE;
> - if (inc > 0)
> - while (inc--)
> - crypto_inc(walk.iv, AES_BLOCK_SIZE);
> -
> - nbytes &= AES_BLOCK_SIZE - 1;
> - ret = blkcipher_walk_done(desc, &walk, nbytes);
> - }
> - if (walk.nbytes) {
> - p8_aes_ctr_final(ctx, &walk);
> - ret = blkcipher_walk_done(desc, &walk, 0);
> - }
> - }
> -
> - return ret;
> -}
> -
> -struct crypto_alg p8_aes_ctr_alg = {
> - .cra_name = "ctr(aes)",
> - .cra_driver_name = "p8_aes_ctr",
> - .cra_module = THIS_MODULE,
> - .cra_priority = 2000,
> - .cra_type = &crypto_blkcipher_type,
> - .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER | CRYPTO_ALG_NEED_FALLBACK,
> - .cra_alignmask = 0,
> - .cra_blocksize = 1,
> - .cra_ctxsize = sizeof(struct p8_aes_ctr_ctx),
> - .cra_init = p8_aes_ctr_init,
> - .cra_exit = p8_aes_ctr_exit,
> - .cra_blkcipher = {
> - .ivsize = AES_BLOCK_SIZE,
> - .min_keysize = AES_MIN_KEY_SIZE,
> - .max_keysize = AES_MAX_KEY_SIZE,
> - .setkey = p8_aes_ctr_setkey,
> - .encrypt = p8_aes_ctr_crypt,
> - .decrypt = p8_aes_ctr_crypt,
> - },
> -};
> diff --git a/drivers/crypto/vmx/aes_xts.c b/drivers/crypto/vmx/aes_xts.c
> deleted file mode 100644
> index 6adc9290557a..000000000000
> --- a/drivers/crypto/vmx/aes_xts.c
> +++ /dev/null
> @@ -1,190 +0,0 @@
> -/**
> - * AES XTS routines supporting VMX In-core instructions on Power 8
> - *
> - * Copyright (C) 2015 International Business Machines Inc.
> - *
> - * This program is free software; you can redistribute it and/or modify
> - * it under the terms of the GNU General Public License as published by
> - * the Free Software Foundations; version 2 only.
> - *
> - * This program is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY of FITNESS FOR A PARTICUPAR PURPOSE. See the
> - * GNU General Public License for more details.
> - *
> - * You should have received a copy of the GNU General Public License
> - * along with this program; if not, write to the Free Software
> - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> - *
> - * Author: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
> - */
> -
> -#include <linux/types.h>
> -#include <linux/err.h>
> -#include <linux/crypto.h>
> -#include <linux/delay.h>
> -#include <linux/hardirq.h>
> -#include <asm/switch_to.h>
> -#include <crypto/aes.h>
> -#include <crypto/scatterwalk.h>
> -#include <crypto/xts.h>
> -#include <crypto/skcipher.h>
> -
> -#include "aesp8-ppc.h"
> -
> -struct p8_aes_xts_ctx {
> - struct crypto_skcipher *fallback;
> - struct aes_key enc_key;
> - struct aes_key dec_key;
> - struct aes_key tweak_key;
> -};
> -
> -static int p8_aes_xts_init(struct crypto_tfm *tfm)
> -{
> - const char *alg;
> - struct crypto_skcipher *fallback;
> - struct p8_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (!(alg = crypto_tfm_alg_name(tfm))) {
> - printk(KERN_ERR "Failed to get algorithm name.\n");
> - return -ENOENT;
> - }
> -
> - fallback = crypto_alloc_skcipher(alg, 0,
> - CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK);
> - if (IS_ERR(fallback)) {
> - printk(KERN_ERR
> - "Failed to allocate transformation for '%s': %ld\n",
> - alg, PTR_ERR(fallback));
> - return PTR_ERR(fallback);
> - }
> - printk(KERN_INFO "Using '%s' as fallback implementation.\n",
> - crypto_skcipher_driver_name(fallback));
> -
> - crypto_skcipher_set_flags(
> - fallback,
> - crypto_skcipher_get_flags((struct crypto_skcipher *)tfm));
> - ctx->fallback = fallback;
> -
> - return 0;
> -}
> -
> -static void p8_aes_xts_exit(struct crypto_tfm *tfm)
> -{
> - struct p8_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (ctx->fallback) {
> - crypto_free_skcipher(ctx->fallback);
> - ctx->fallback = NULL;
> - }
> -}
> -
> -static int p8_aes_xts_setkey(struct crypto_tfm *tfm, const u8 *key,
> - unsigned int keylen)
> -{
> - int ret;
> - struct p8_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - ret = xts_check_key(tfm, key, keylen);
> - if (ret)
> - return ret;
> -
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - ret = aes_p8_set_encrypt_key(key + keylen/2, (keylen/2) * 8, &ctx->tweak_key);
> - ret += aes_p8_set_encrypt_key(key, (keylen/2) * 8, &ctx->enc_key);
> - ret += aes_p8_set_decrypt_key(key, (keylen/2) * 8, &ctx->dec_key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> -
> - ret += crypto_skcipher_setkey(ctx->fallback, key, keylen);
> - return ret;
> -}
> -
> -static int p8_aes_xts_crypt(struct blkcipher_desc *desc,
> - struct scatterlist *dst,
> - struct scatterlist *src,
> - unsigned int nbytes, int enc)
> -{
> - int ret;
> - u8 tweak[AES_BLOCK_SIZE];
> - u8 *iv;
> - struct blkcipher_walk walk;
> - struct p8_aes_xts_ctx *ctx =
> - crypto_tfm_ctx(crypto_blkcipher_tfm(desc->tfm));
> -
> - if (in_interrupt()) {
> - SKCIPHER_REQUEST_ON_STACK(req, ctx->fallback);
> - skcipher_request_set_tfm(req, ctx->fallback);
> - skcipher_request_set_callback(req, desc->flags, NULL, NULL);
> - skcipher_request_set_crypt(req, src, dst, nbytes, desc->info);
> - ret = enc? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req);
> - skcipher_request_zero(req);
> - } else {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> -
> - blkcipher_walk_init(&walk, dst, src, nbytes);
> -
> - ret = blkcipher_walk_virt(desc, &walk);
> - iv = walk.iv;
> - memset(tweak, 0, AES_BLOCK_SIZE);
> - aes_p8_encrypt(iv, tweak, &ctx->tweak_key);
> -
> - while ((nbytes = walk.nbytes)) {
> - if (enc)
> - aes_p8_xts_encrypt(walk.src.virt.addr, walk.dst.virt.addr,
> - nbytes & AES_BLOCK_MASK, &ctx->enc_key, NULL, tweak);
> - else
> - aes_p8_xts_decrypt(walk.src.virt.addr, walk.dst.virt.addr,
> - nbytes & AES_BLOCK_MASK, &ctx->dec_key, NULL, tweak);
> -
> - nbytes &= AES_BLOCK_SIZE - 1;
> - ret = blkcipher_walk_done(desc, &walk, nbytes);
> - }
> -
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - }
> - return ret;
> -}
> -
> -static int p8_aes_xts_encrypt(struct blkcipher_desc *desc,
> - struct scatterlist *dst,
> - struct scatterlist *src, unsigned int nbytes)
> -{
> - return p8_aes_xts_crypt(desc, dst, src, nbytes, 1);
> -}
> -
> -static int p8_aes_xts_decrypt(struct blkcipher_desc *desc,
> - struct scatterlist *dst,
> - struct scatterlist *src, unsigned int nbytes)
> -{
> - return p8_aes_xts_crypt(desc, dst, src, nbytes, 0);
> -}
> -
> -struct crypto_alg p8_aes_xts_alg = {
> - .cra_name = "xts(aes)",
> - .cra_driver_name = "p8_aes_xts",
> - .cra_module = THIS_MODULE,
> - .cra_priority = 2000,
> - .cra_type = &crypto_blkcipher_type,
> - .cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER | CRYPTO_ALG_NEED_FALLBACK,
> - .cra_alignmask = 0,
> - .cra_blocksize = AES_BLOCK_SIZE,
> - .cra_ctxsize = sizeof(struct p8_aes_xts_ctx),
> - .cra_init = p8_aes_xts_init,
> - .cra_exit = p8_aes_xts_exit,
> - .cra_blkcipher = {
> - .ivsize = AES_BLOCK_SIZE,
> - .min_keysize = 2 * AES_MIN_KEY_SIZE,
> - .max_keysize = 2 * AES_MAX_KEY_SIZE,
> - .setkey = p8_aes_xts_setkey,
> - .encrypt = p8_aes_xts_encrypt,
> - .decrypt = p8_aes_xts_decrypt,
> - }
> -};
> diff --git a/drivers/crypto/vmx/aesp8-ppc.h b/drivers/crypto/vmx/aesp8-ppc.h
> deleted file mode 100644
> index 01972e16a6c0..000000000000
> --- a/drivers/crypto/vmx/aesp8-ppc.h
> +++ /dev/null
> @@ -1,25 +0,0 @@
> -#include <linux/types.h>
> -#include <crypto/aes.h>
> -
> -#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE-1))
> -
> -struct aes_key {
> - u8 key[AES_MAX_KEYLENGTH];
> - int rounds;
> -};
> -
> -int aes_p8_set_encrypt_key(const u8 *userKey, const int bits,
> - struct aes_key *key);
> -int aes_p8_set_decrypt_key(const u8 *userKey, const int bits,
> - struct aes_key *key);
> -void aes_p8_encrypt(const u8 *in, u8 *out, const struct aes_key *key);
> -void aes_p8_decrypt(const u8 *in, u8 *out, const struct aes_key *key);
> -void aes_p8_cbc_encrypt(const u8 *in, u8 *out, size_t len,
> - const struct aes_key *key, u8 *iv, const int enc);
> -void aes_p8_ctr32_encrypt_blocks(const u8 *in, u8 *out,
> - size_t len, const struct aes_key *key,
> - const u8 *iv);
> -void aes_p8_xts_encrypt(const u8 *in, u8 *out, size_t len,
> - const struct aes_key *key1, const struct aes_key *key2, u8 *iv);
> -void aes_p8_xts_decrypt(const u8 *in, u8 *out, size_t len,
> - const struct aes_key *key1, const struct aes_key *key2, u8 *iv);
> diff --git a/drivers/crypto/vmx/aesp8-ppc.pl b/drivers/crypto/vmx/aesp8-ppc.pl
> deleted file mode 100644
> index 0b4a293b8a1e..000000000000
> --- a/drivers/crypto/vmx/aesp8-ppc.pl
> +++ /dev/null
> @@ -1,3789 +0,0 @@
> -#! /usr/bin/env perl
> -# Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved.
> -#
> -# Licensed under the OpenSSL license (the "License"). You may not use
> -# this file except in compliance with the License. You can obtain a copy
> -# in the file LICENSE in the source distribution or at
> -# https://www.openssl.org/source/license.html
> -
> -#
> -# ====================================================================
> -# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
> -# project. The module is, however, dual licensed under OpenSSL and
> -# CRYPTOGAMS licenses depending on where you obtain it. For further
> -# details see http://www.openssl.org/~appro/cryptogams/.
> -# ====================================================================
> -#
> -# This module implements support for AES instructions as per PowerISA
> -# specification version 2.07, first implemented by POWER8 processor.
> -# The module is endian-agnostic in sense that it supports both big-
> -# and little-endian cases. Data alignment in parallelizable modes is
> -# handled with VSX loads and stores, which implies MSR.VSX flag being
> -# set. It should also be noted that ISA specification doesn't prohibit
> -# alignment exceptions for these instructions on page boundaries.
> -# Initially alignment was handled in pure AltiVec/VMX way [when data
> -# is aligned programmatically, which in turn guarantees exception-
> -# free execution], but it turned to hamper performance when vcipher
> -# instructions are interleaved. It's reckoned that eventual
> -# misalignment penalties at page boundaries are in average lower
> -# than additional overhead in pure AltiVec approach.
> -#
> -# May 2016
> -#
> -# Add XTS subroutine, 9x on little- and 12x improvement on big-endian
> -# systems were measured.
> -#
> -######################################################################
> -# Current large-block performance in cycles per byte processed with
> -# 128-bit key (less is better).
> -#
> -# CBC en-/decrypt CTR XTS
> -# POWER8[le] 3.96/0.72 0.74 1.1
> -# POWER8[be] 3.75/0.65 0.66 1.0
> -
> -$flavour = shift;
> -
> -if ($flavour =~ /64/) {
> - $SIZE_T =8;
> - $LRSAVE =2*$SIZE_T;
> - $STU ="stdu";
> - $POP ="ld";
> - $PUSH ="std";
> - $UCMP ="cmpld";
> - $SHL ="sldi";
> -} elsif ($flavour =~ /32/) {
> - $SIZE_T =4;
> - $LRSAVE =$SIZE_T;
> - $STU ="stwu";
> - $POP ="lwz";
> - $PUSH ="stw";
> - $UCMP ="cmplw";
> - $SHL ="slwi";
> -} else { die "nonsense $flavour"; }
> -
> -$LITTLE_ENDIAN = ($flavour=~/le$/) ? $SIZE_T : 0;
> -
> -$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
> -( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or
> -( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or
> -die "can't locate ppc-xlate.pl";
> -
> -open STDOUT,"| $^X $xlate $flavour ".shift || die "can't call $xlate: $!";
> -
> -$FRAME=8*$SIZE_T;
> -$prefix="aes_p8";
> -
> -$sp="r1";
> -$vrsave="r12";
> -
> -#########################################################################
> -{{{ # Key setup procedures #
> -my ($inp,$bits,$out,$ptr,$cnt,$rounds)=map("r$_",(3..8));
> -my ($zero,$in0,$in1,$key,$rcon,$mask,$tmp)=map("v$_",(0..6));
> -my ($stage,$outperm,$outmask,$outhead,$outtail)=map("v$_",(7..11));
> -
> -$code.=<<___;
> -.machine "any"
> -
> -.text
> -
> -.align 7
> -rcon:
> -.long 0x01000000, 0x01000000, 0x01000000, 0x01000000 ?rev
> -.long 0x1b000000, 0x1b000000, 0x1b000000, 0x1b000000 ?rev
> -.long 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c ?rev
> -.long 0,0,0,0 ?asis
> -Lconsts:
> - mflr r0
> - bcl 20,31,\$+4
> - mflr $ptr #vvvvv "distance between . and rcon
> - addi $ptr,$ptr,-0x48
> - mtlr r0
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,0,0
> -.asciz "AES for PowerISA 2.07, CRYPTOGAMS by <appro\@openssl.org>"
> -
> -.globl .${prefix}_set_encrypt_key
> -Lset_encrypt_key:
> - mflr r11
> - $PUSH r11,$LRSAVE($sp)
> -
> - li $ptr,-1
> - ${UCMP}i $inp,0
> - beq- Lenc_key_abort # if ($inp==0) return -1;
> - ${UCMP}i $out,0
> - beq- Lenc_key_abort # if ($out==0) return -1;
> - li $ptr,-2
> - cmpwi $bits,128
> - blt- Lenc_key_abort
> - cmpwi $bits,256
> - bgt- Lenc_key_abort
> - andi. r0,$bits,0x3f
> - bne- Lenc_key_abort
> -
> - lis r0,0xfff0
> - mfspr $vrsave,256
> - mtspr 256,r0
> -
> - bl Lconsts
> - mtlr r11
> -
> - neg r9,$inp
> - lvx $in0,0,$inp
> - addi $inp,$inp,15 # 15 is not typo
> - lvsr $key,0,r9 # borrow $key
> - li r8,0x20
> - cmpwi $bits,192
> - lvx $in1,0,$inp
> - le?vspltisb $mask,0x0f # borrow $mask
> - lvx $rcon,0,$ptr
> - le?vxor $key,$key,$mask # adjust for byte swap
> - lvx $mask,r8,$ptr
> - addi $ptr,$ptr,0x10
> - vperm $in0,$in0,$in1,$key # align [and byte swap in LE]
> - li $cnt,8
> - vxor $zero,$zero,$zero
> - mtctr $cnt
> -
> - ?lvsr $outperm,0,$out
> - vspltisb $outmask,-1
> - lvx $outhead,0,$out
> - ?vperm $outmask,$zero,$outmask,$outperm
> -
> - blt Loop128
> - addi $inp,$inp,8
> - beq L192
> - addi $inp,$inp,8
> - b L256
> -
> -.align 4
> -Loop128:
> - vperm $key,$in0,$in0,$mask # rotate-n-splat
> - vsldoi $tmp,$zero,$in0,12 # >>32
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - vcipherlast $key,$key,$rcon
> - stvx $stage,0,$out
> - addi $out,$out,16
> -
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vadduwm $rcon,$rcon,$rcon
> - vxor $in0,$in0,$key
> - bdnz Loop128
> -
> - lvx $rcon,0,$ptr # last two round keys
> -
> - vperm $key,$in0,$in0,$mask # rotate-n-splat
> - vsldoi $tmp,$zero,$in0,12 # >>32
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - vcipherlast $key,$key,$rcon
> - stvx $stage,0,$out
> - addi $out,$out,16
> -
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vadduwm $rcon,$rcon,$rcon
> - vxor $in0,$in0,$key
> -
> - vperm $key,$in0,$in0,$mask # rotate-n-splat
> - vsldoi $tmp,$zero,$in0,12 # >>32
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - vcipherlast $key,$key,$rcon
> - stvx $stage,0,$out
> - addi $out,$out,16
> -
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vxor $in0,$in0,$key
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - stvx $stage,0,$out
> -
> - addi $inp,$out,15 # 15 is not typo
> - addi $out,$out,0x50
> -
> - li $rounds,10
> - b Ldone
> -
> -.align 4
> -L192:
> - lvx $tmp,0,$inp
> - li $cnt,4
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - stvx $stage,0,$out
> - addi $out,$out,16
> - vperm $in1,$in1,$tmp,$key # align [and byte swap in LE]
> - vspltisb $key,8 # borrow $key
> - mtctr $cnt
> - vsububm $mask,$mask,$key # adjust the mask
> -
> -Loop192:
> - vperm $key,$in1,$in1,$mask # roate-n-splat
> - vsldoi $tmp,$zero,$in0,12 # >>32
> - vcipherlast $key,$key,$rcon
> -
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> -
> - vsldoi $stage,$zero,$in1,8
> - vspltw $tmp,$in0,3
> - vxor $tmp,$tmp,$in1
> - vsldoi $in1,$zero,$in1,12 # >>32
> - vadduwm $rcon,$rcon,$rcon
> - vxor $in1,$in1,$tmp
> - vxor $in0,$in0,$key
> - vxor $in1,$in1,$key
> - vsldoi $stage,$stage,$in0,8
> -
> - vperm $key,$in1,$in1,$mask # rotate-n-splat
> - vsldoi $tmp,$zero,$in0,12 # >>32
> - vperm $outtail,$stage,$stage,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - vcipherlast $key,$key,$rcon
> - stvx $stage,0,$out
> - addi $out,$out,16
> -
> - vsldoi $stage,$in0,$in1,8
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vperm $outtail,$stage,$stage,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - stvx $stage,0,$out
> - addi $out,$out,16
> -
> - vspltw $tmp,$in0,3
> - vxor $tmp,$tmp,$in1
> - vsldoi $in1,$zero,$in1,12 # >>32
> - vadduwm $rcon,$rcon,$rcon
> - vxor $in1,$in1,$tmp
> - vxor $in0,$in0,$key
> - vxor $in1,$in1,$key
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - stvx $stage,0,$out
> - addi $inp,$out,15 # 15 is not typo
> - addi $out,$out,16
> - bdnz Loop192
> -
> - li $rounds,12
> - addi $out,$out,0x20
> - b Ldone
> -
> -.align 4
> -L256:
> - lvx $tmp,0,$inp
> - li $cnt,7
> - li $rounds,14
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - stvx $stage,0,$out
> - addi $out,$out,16
> - vperm $in1,$in1,$tmp,$key # align [and byte swap in LE]
> - mtctr $cnt
> -
> -Loop256:
> - vperm $key,$in1,$in1,$mask # rotate-n-splat
> - vsldoi $tmp,$zero,$in0,12 # >>32
> - vperm $outtail,$in1,$in1,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - vcipherlast $key,$key,$rcon
> - stvx $stage,0,$out
> - addi $out,$out,16
> -
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in0,$in0,$tmp
> - vadduwm $rcon,$rcon,$rcon
> - vxor $in0,$in0,$key
> - vperm $outtail,$in0,$in0,$outperm # rotate
> - vsel $stage,$outhead,$outtail,$outmask
> - vmr $outhead,$outtail
> - stvx $stage,0,$out
> - addi $inp,$out,15 # 15 is not typo
> - addi $out,$out,16
> - bdz Ldone
> -
> - vspltw $key,$in0,3 # just splat
> - vsldoi $tmp,$zero,$in1,12 # >>32
> - vsbox $key,$key
> -
> - vxor $in1,$in1,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in1,$in1,$tmp
> - vsldoi $tmp,$zero,$tmp,12 # >>32
> - vxor $in1,$in1,$tmp
> -
> - vxor $in1,$in1,$key
> - b Loop256
> -
> -.align 4
> -Ldone:
> - lvx $in1,0,$inp # redundant in aligned case
> - vsel $in1,$outhead,$in1,$outmask
> - stvx $in1,0,$inp
> - li $ptr,0
> - mtspr 256,$vrsave
> - stw $rounds,0($out)
> -
> -Lenc_key_abort:
> - mr r3,$ptr
> - blr
> - .long 0
> - .byte 0,12,0x14,1,0,0,3,0
> - .long 0
> -.size .${prefix}_set_encrypt_key,.-.${prefix}_set_encrypt_key
> -
> -.globl .${prefix}_set_decrypt_key
> - $STU $sp,-$FRAME($sp)
> - mflr r10
> - $PUSH r10,$FRAME+$LRSAVE($sp)
> - bl Lset_encrypt_key
> - mtlr r10
> -
> - cmpwi r3,0
> - bne- Ldec_key_abort
> -
> - slwi $cnt,$rounds,4
> - subi $inp,$out,240 # first round key
> - srwi $rounds,$rounds,1
> - add $out,$inp,$cnt # last round key
> - mtctr $rounds
> -
> -Ldeckey:
> - lwz r0, 0($inp)
> - lwz r6, 4($inp)
> - lwz r7, 8($inp)
> - lwz r8, 12($inp)
> - addi $inp,$inp,16
> - lwz r9, 0($out)
> - lwz r10,4($out)
> - lwz r11,8($out)
> - lwz r12,12($out)
> - stw r0, 0($out)
> - stw r6, 4($out)
> - stw r7, 8($out)
> - stw r8, 12($out)
> - subi $out,$out,16
> - stw r9, -16($inp)
> - stw r10,-12($inp)
> - stw r11,-8($inp)
> - stw r12,-4($inp)
> - bdnz Ldeckey
> -
> - xor r3,r3,r3 # return value
> -Ldec_key_abort:
> - addi $sp,$sp,$FRAME
> - blr
> - .long 0
> - .byte 0,12,4,1,0x80,0,3,0
> - .long 0
> -.size .${prefix}_set_decrypt_key,.-.${prefix}_set_decrypt_key
> -___
> -}}}
> -#########################################################################
> -{{{ # Single block en- and decrypt procedures #
> -sub gen_block () {
> -my $dir = shift;
> -my $n = $dir eq "de" ? "n" : "";
> -my ($inp,$out,$key,$rounds,$idx)=map("r$_",(3..7));
> -
> -$code.=<<___;
> -.globl .${prefix}_${dir}crypt
> - lwz $rounds,240($key)
> - lis r0,0xfc00
> - mfspr $vrsave,256
> - li $idx,15 # 15 is not typo
> - mtspr 256,r0
> -
> - lvx v0,0,$inp
> - neg r11,$out
> - lvx v1,$idx,$inp
> - lvsl v2,0,$inp # inpperm
> - le?vspltisb v4,0x0f
> - ?lvsl v3,0,r11 # outperm
> - le?vxor v2,v2,v4
> - li $idx,16
> - vperm v0,v0,v1,v2 # align [and byte swap in LE]
> - lvx v1,0,$key
> - ?lvsl v5,0,$key # keyperm
> - srwi $rounds,$rounds,1
> - lvx v2,$idx,$key
> - addi $idx,$idx,16
> - subi $rounds,$rounds,1
> - ?vperm v1,v1,v2,v5 # align round key
> -
> - vxor v0,v0,v1
> - lvx v1,$idx,$key
> - addi $idx,$idx,16
> - mtctr $rounds
> -
> -Loop_${dir}c:
> - ?vperm v2,v2,v1,v5
> - v${n}cipher v0,v0,v2
> - lvx v2,$idx,$key
> - addi $idx,$idx,16
> - ?vperm v1,v1,v2,v5
> - v${n}cipher v0,v0,v1
> - lvx v1,$idx,$key
> - addi $idx,$idx,16
> - bdnz Loop_${dir}c
> -
> - ?vperm v2,v2,v1,v5
> - v${n}cipher v0,v0,v2
> - lvx v2,$idx,$key
> - ?vperm v1,v1,v2,v5
> - v${n}cipherlast v0,v0,v1
> -
> - vspltisb v2,-1
> - vxor v1,v1,v1
> - li $idx,15 # 15 is not typo
> - ?vperm v2,v1,v2,v3 # outmask
> - le?vxor v3,v3,v4
> - lvx v1,0,$out # outhead
> - vperm v0,v0,v0,v3 # rotate [and byte swap in LE]
> - vsel v1,v1,v0,v2
> - lvx v4,$idx,$out
> - stvx v1,0,$out
> - vsel v0,v0,v4,v2
> - stvx v0,$idx,$out
> -
> - mtspr 256,$vrsave
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,3,0
> - .long 0
> -.size .${prefix}_${dir}crypt,.-.${prefix}_${dir}crypt
> -___
> -}
> -&gen_block("en");
> -&gen_block("de");
> -}}}
> -#########################################################################
> -{{{ # CBC en- and decrypt procedures #
> -my ($inp,$out,$len,$key,$ivp,$enc,$rounds,$idx)=map("r$_",(3..10));
> -my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3));
> -my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm)=
> - map("v$_",(4..10));
> -$code.=<<___;
> -.globl .${prefix}_cbc_encrypt
> - ${UCMP}i $len,16
> - bltlr-
> -
> - cmpwi $enc,0 # test direction
> - lis r0,0xffe0
> - mfspr $vrsave,256
> - mtspr 256,r0
> -
> - li $idx,15
> - vxor $rndkey0,$rndkey0,$rndkey0
> - le?vspltisb $tmp,0x0f
> -
> - lvx $ivec,0,$ivp # load [unaligned] iv
> - lvsl $inpperm,0,$ivp
> - lvx $inptail,$idx,$ivp
> - le?vxor $inpperm,$inpperm,$tmp
> - vperm $ivec,$ivec,$inptail,$inpperm
> -
> - neg r11,$inp
> - ?lvsl $keyperm,0,$key # prepare for unaligned key
> - lwz $rounds,240($key)
> -
> - lvsr $inpperm,0,r11 # prepare for unaligned load
> - lvx $inptail,0,$inp
> - addi $inp,$inp,15 # 15 is not typo
> - le?vxor $inpperm,$inpperm,$tmp
> -
> - ?lvsr $outperm,0,$out # prepare for unaligned store
> - vspltisb $outmask,-1
> - lvx $outhead,0,$out
> - ?vperm $outmask,$rndkey0,$outmask,$outperm
> - le?vxor $outperm,$outperm,$tmp
> -
> - srwi $rounds,$rounds,1
> - li $idx,16
> - subi $rounds,$rounds,1
> - beq Lcbc_dec
> -
> -Lcbc_enc:
> - vmr $inout,$inptail
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> - mtctr $rounds
> - subi $len,$len,16 # len-=16
> -
> - lvx $rndkey0,0,$key
> - vperm $inout,$inout,$inptail,$inpperm
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> - vxor $inout,$inout,$ivec
> -
> -Loop_cbc_enc:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipher $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> - bdnz Loop_cbc_enc
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key
> - li $idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipherlast $ivec,$inout,$rndkey0
> - ${UCMP}i $len,16
> -
> - vperm $tmp,$ivec,$ivec,$outperm
> - vsel $inout,$outhead,$tmp,$outmask
> - vmr $outhead,$tmp
> - stvx $inout,0,$out
> - addi $out,$out,16
> - bge Lcbc_enc
> -
> - b Lcbc_done
> -
> -.align 4
> -Lcbc_dec:
> - ${UCMP}i $len,128
> - bge _aesp8_cbc_decrypt8x
> - vmr $tmp,$inptail
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> - mtctr $rounds
> - subi $len,$len,16 # len-=16
> -
> - lvx $rndkey0,0,$key
> - vperm $tmp,$tmp,$inptail,$inpperm
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$tmp,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> -
> -Loop_cbc_dec:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vncipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vncipher $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> - bdnz Loop_cbc_dec
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vncipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key
> - li $idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vncipherlast $inout,$inout,$rndkey0
> - ${UCMP}i $len,16
> -
> - vxor $inout,$inout,$ivec
> - vmr $ivec,$tmp
> - vperm $tmp,$inout,$inout,$outperm
> - vsel $inout,$outhead,$tmp,$outmask
> - vmr $outhead,$tmp
> - stvx $inout,0,$out
> - addi $out,$out,16
> - bge Lcbc_dec
> -
> -Lcbc_done:
> - addi $out,$out,-1
> - lvx $inout,0,$out # redundant in aligned case
> - vsel $inout,$outhead,$inout,$outmask
> - stvx $inout,0,$out
> -
> - neg $enc,$ivp # write [unaligned] iv
> - li $idx,15 # 15 is not typo
> - vxor $rndkey0,$rndkey0,$rndkey0
> - vspltisb $outmask,-1
> - le?vspltisb $tmp,0x0f
> - ?lvsl $outperm,0,$enc
> - ?vperm $outmask,$rndkey0,$outmask,$outperm
> - le?vxor $outperm,$outperm,$tmp
> - lvx $outhead,0,$ivp
> - vperm $ivec,$ivec,$ivec,$outperm
> - vsel $inout,$outhead,$ivec,$outmask
> - lvx $inptail,$idx,$ivp
> - stvx $inout,0,$ivp
> - vsel $inout,$ivec,$inptail,$outmask
> - stvx $inout,$idx,$ivp
> -
> - mtspr 256,$vrsave
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,6,0
> - .long 0
> -___
> -#########################################################################
> -{{ # Optimized CBC decrypt procedure #
> -my $key_="r11";
> -my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31));
> -my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10..13));
> -my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(14..21));
> -my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys
> - # v26-v31 last 6 round keys
> -my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment
> -
> -$code.=<<___;
> -.align 5
> -_aesp8_cbc_decrypt8x:
> - $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
> - li r10,`$FRAME+8*16+15`
> - li r11,`$FRAME+8*16+31`
> - stvx v20,r10,$sp # ABI says so
> - addi r10,r10,32
> - stvx v21,r11,$sp
> - addi r11,r11,32
> - stvx v22,r10,$sp
> - addi r10,r10,32
> - stvx v23,r11,$sp
> - addi r11,r11,32
> - stvx v24,r10,$sp
> - addi r10,r10,32
> - stvx v25,r11,$sp
> - addi r11,r11,32
> - stvx v26,r10,$sp
> - addi r10,r10,32
> - stvx v27,r11,$sp
> - addi r11,r11,32
> - stvx v28,r10,$sp
> - addi r10,r10,32
> - stvx v29,r11,$sp
> - addi r11,r11,32
> - stvx v30,r10,$sp
> - stvx v31,r11,$sp
> - li r0,-1
> - stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
> - li $x10,0x10
> - $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - li $x20,0x20
> - $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - li $x30,0x30
> - $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - li $x40,0x40
> - $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - li $x50,0x50
> - $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - li $x60,0x60
> - $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - li $x70,0x70
> - mtspr 256,r0
> -
> - subi $rounds,$rounds,3 # -4 in total
> - subi $len,$len,128 # bias
> -
> - lvx $rndkey0,$x00,$key # load key schedule
> - lvx v30,$x10,$key
> - addi $key,$key,0x20
> - lvx v31,$x00,$key
> - ?vperm $rndkey0,$rndkey0,v30,$keyperm
> - addi $key_,$sp,$FRAME+15
> - mtctr $rounds
> -
> -Load_cbc_dec_key:
> - ?vperm v24,v30,v31,$keyperm
> - lvx v30,$x10,$key
> - addi $key,$key,0x20
> - stvx v24,$x00,$key_ # off-load round[1]
> - ?vperm v25,v31,v30,$keyperm
> - lvx v31,$x00,$key
> - stvx v25,$x10,$key_ # off-load round[2]
> - addi $key_,$key_,0x20
> - bdnz Load_cbc_dec_key
> -
> - lvx v26,$x10,$key
> - ?vperm v24,v30,v31,$keyperm
> - lvx v27,$x20,$key
> - stvx v24,$x00,$key_ # off-load round[3]
> - ?vperm v25,v31,v26,$keyperm
> - lvx v28,$x30,$key
> - stvx v25,$x10,$key_ # off-load round[4]
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - ?vperm v26,v26,v27,$keyperm
> - lvx v29,$x40,$key
> - ?vperm v27,v27,v28,$keyperm
> - lvx v30,$x50,$key
> - ?vperm v28,v28,v29,$keyperm
> - lvx v31,$x60,$key
> - ?vperm v29,v29,v30,$keyperm
> - lvx $out0,$x70,$key # borrow $out0
> - ?vperm v30,v30,v31,$keyperm
> - lvx v24,$x00,$key_ # pre-load round[1]
> - ?vperm v31,v31,$out0,$keyperm
> - lvx v25,$x10,$key_ # pre-load round[2]
> -
> - #lvx $inptail,0,$inp # "caller" already did this
> - #addi $inp,$inp,15 # 15 is not typo
> - subi $inp,$inp,15 # undo "caller"
> -
> - le?li $idx,8
> - lvx_u $in0,$x00,$inp # load first 8 "words"
> - le?lvsl $inpperm,0,$idx
> - le?vspltisb $tmp,0x0f
> - lvx_u $in1,$x10,$inp
> - le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u
> - lvx_u $in2,$x20,$inp
> - le?vperm $in0,$in0,$in0,$inpperm
> - lvx_u $in3,$x30,$inp
> - le?vperm $in1,$in1,$in1,$inpperm
> - lvx_u $in4,$x40,$inp
> - le?vperm $in2,$in2,$in2,$inpperm
> - vxor $out0,$in0,$rndkey0
> - lvx_u $in5,$x50,$inp
> - le?vperm $in3,$in3,$in3,$inpperm
> - vxor $out1,$in1,$rndkey0
> - lvx_u $in6,$x60,$inp
> - le?vperm $in4,$in4,$in4,$inpperm
> - vxor $out2,$in2,$rndkey0
> - lvx_u $in7,$x70,$inp
> - addi $inp,$inp,0x80
> - le?vperm $in5,$in5,$in5,$inpperm
> - vxor $out3,$in3,$rndkey0
> - le?vperm $in6,$in6,$in6,$inpperm
> - vxor $out4,$in4,$rndkey0
> - le?vperm $in7,$in7,$in7,$inpperm
> - vxor $out5,$in5,$rndkey0
> - vxor $out6,$in6,$rndkey0
> - vxor $out7,$in7,$rndkey0
> -
> - mtctr $rounds
> - b Loop_cbc_dec8x
> -.align 5
> -Loop_cbc_dec8x:
> - vncipher $out0,$out0,v24
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> - vncipher $out5,$out5,v24
> - vncipher $out6,$out6,v24
> - vncipher $out7,$out7,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vncipher $out0,$out0,v25
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - vncipher $out5,$out5,v25
> - vncipher $out6,$out6,v25
> - vncipher $out7,$out7,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_cbc_dec8x
> -
> - subic $len,$len,128 # $len-=128
> - vncipher $out0,$out0,v24
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> - vncipher $out5,$out5,v24
> - vncipher $out6,$out6,v24
> - vncipher $out7,$out7,v24
> -
> - subfe. r0,r0,r0 # borrow?-1:0
> - vncipher $out0,$out0,v25
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - vncipher $out5,$out5,v25
> - vncipher $out6,$out6,v25
> - vncipher $out7,$out7,v25
> -
> - and r0,r0,$len
> - vncipher $out0,$out0,v26
> - vncipher $out1,$out1,v26
> - vncipher $out2,$out2,v26
> - vncipher $out3,$out3,v26
> - vncipher $out4,$out4,v26
> - vncipher $out5,$out5,v26
> - vncipher $out6,$out6,v26
> - vncipher $out7,$out7,v26
> -
> - add $inp,$inp,r0 # $inp is adjusted in such
> - # way that at exit from the
> - # loop inX-in7 are loaded
> - # with last "words"
> - vncipher $out0,$out0,v27
> - vncipher $out1,$out1,v27
> - vncipher $out2,$out2,v27
> - vncipher $out3,$out3,v27
> - vncipher $out4,$out4,v27
> - vncipher $out5,$out5,v27
> - vncipher $out6,$out6,v27
> - vncipher $out7,$out7,v27
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vncipher $out0,$out0,v28
> - vncipher $out1,$out1,v28
> - vncipher $out2,$out2,v28
> - vncipher $out3,$out3,v28
> - vncipher $out4,$out4,v28
> - vncipher $out5,$out5,v28
> - vncipher $out6,$out6,v28
> - vncipher $out7,$out7,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> -
> - vncipher $out0,$out0,v29
> - vncipher $out1,$out1,v29
> - vncipher $out2,$out2,v29
> - vncipher $out3,$out3,v29
> - vncipher $out4,$out4,v29
> - vncipher $out5,$out5,v29
> - vncipher $out6,$out6,v29
> - vncipher $out7,$out7,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> -
> - vncipher $out0,$out0,v30
> - vxor $ivec,$ivec,v31 # xor with last round key
> - vncipher $out1,$out1,v30
> - vxor $in0,$in0,v31
> - vncipher $out2,$out2,v30
> - vxor $in1,$in1,v31
> - vncipher $out3,$out3,v30
> - vxor $in2,$in2,v31
> - vncipher $out4,$out4,v30
> - vxor $in3,$in3,v31
> - vncipher $out5,$out5,v30
> - vxor $in4,$in4,v31
> - vncipher $out6,$out6,v30
> - vxor $in5,$in5,v31
> - vncipher $out7,$out7,v30
> - vxor $in6,$in6,v31
> -
> - vncipherlast $out0,$out0,$ivec
> - vncipherlast $out1,$out1,$in0
> - lvx_u $in0,$x00,$inp # load next input block
> - vncipherlast $out2,$out2,$in1
> - lvx_u $in1,$x10,$inp
> - vncipherlast $out3,$out3,$in2
> - le?vperm $in0,$in0,$in0,$inpperm
> - lvx_u $in2,$x20,$inp
> - vncipherlast $out4,$out4,$in3
> - le?vperm $in1,$in1,$in1,$inpperm
> - lvx_u $in3,$x30,$inp
> - vncipherlast $out5,$out5,$in4
> - le?vperm $in2,$in2,$in2,$inpperm
> - lvx_u $in4,$x40,$inp
> - vncipherlast $out6,$out6,$in5
> - le?vperm $in3,$in3,$in3,$inpperm
> - lvx_u $in5,$x50,$inp
> - vncipherlast $out7,$out7,$in6
> - le?vperm $in4,$in4,$in4,$inpperm
> - lvx_u $in6,$x60,$inp
> - vmr $ivec,$in7
> - le?vperm $in5,$in5,$in5,$inpperm
> - lvx_u $in7,$x70,$inp
> - addi $inp,$inp,0x80
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $in6,$in6,$in6,$inpperm
> - vxor $out0,$in0,$rndkey0
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - le?vperm $in7,$in7,$in7,$inpperm
> - vxor $out1,$in1,$rndkey0
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x20,$out
> - vxor $out2,$in2,$rndkey0
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x30,$out
> - vxor $out3,$in3,$rndkey0
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x40,$out
> - vxor $out4,$in4,$rndkey0
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x50,$out
> - vxor $out5,$in5,$rndkey0
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x60,$out
> - vxor $out6,$in6,$rndkey0
> - stvx_u $out7,$x70,$out
> - addi $out,$out,0x80
> - vxor $out7,$in7,$rndkey0
> -
> - mtctr $rounds
> - beq Loop_cbc_dec8x # did $len-=128 borrow?
> -
> - addic. $len,$len,128
> - beq Lcbc_dec8x_done
> - nop
> - nop
> -
> -Loop_cbc_dec8x_tail: # up to 7 "words" tail...
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> - vncipher $out5,$out5,v24
> - vncipher $out6,$out6,v24
> - vncipher $out7,$out7,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - vncipher $out5,$out5,v25
> - vncipher $out6,$out6,v25
> - vncipher $out7,$out7,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_cbc_dec8x_tail
> -
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> - vncipher $out5,$out5,v24
> - vncipher $out6,$out6,v24
> - vncipher $out7,$out7,v24
> -
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - vncipher $out5,$out5,v25
> - vncipher $out6,$out6,v25
> - vncipher $out7,$out7,v25
> -
> - vncipher $out1,$out1,v26
> - vncipher $out2,$out2,v26
> - vncipher $out3,$out3,v26
> - vncipher $out4,$out4,v26
> - vncipher $out5,$out5,v26
> - vncipher $out6,$out6,v26
> - vncipher $out7,$out7,v26
> -
> - vncipher $out1,$out1,v27
> - vncipher $out2,$out2,v27
> - vncipher $out3,$out3,v27
> - vncipher $out4,$out4,v27
> - vncipher $out5,$out5,v27
> - vncipher $out6,$out6,v27
> - vncipher $out7,$out7,v27
> -
> - vncipher $out1,$out1,v28
> - vncipher $out2,$out2,v28
> - vncipher $out3,$out3,v28
> - vncipher $out4,$out4,v28
> - vncipher $out5,$out5,v28
> - vncipher $out6,$out6,v28
> - vncipher $out7,$out7,v28
> -
> - vncipher $out1,$out1,v29
> - vncipher $out2,$out2,v29
> - vncipher $out3,$out3,v29
> - vncipher $out4,$out4,v29
> - vncipher $out5,$out5,v29
> - vncipher $out6,$out6,v29
> - vncipher $out7,$out7,v29
> -
> - vncipher $out1,$out1,v30
> - vxor $ivec,$ivec,v31 # last round key
> - vncipher $out2,$out2,v30
> - vxor $in1,$in1,v31
> - vncipher $out3,$out3,v30
> - vxor $in2,$in2,v31
> - vncipher $out4,$out4,v30
> - vxor $in3,$in3,v31
> - vncipher $out5,$out5,v30
> - vxor $in4,$in4,v31
> - vncipher $out6,$out6,v30
> - vxor $in5,$in5,v31
> - vncipher $out7,$out7,v30
> - vxor $in6,$in6,v31
> -
> - cmplwi $len,32 # switch($len)
> - blt Lcbc_dec8x_one
> - nop
> - beq Lcbc_dec8x_two
> - cmplwi $len,64
> - blt Lcbc_dec8x_three
> - nop
> - beq Lcbc_dec8x_four
> - cmplwi $len,96
> - blt Lcbc_dec8x_five
> - nop
> - beq Lcbc_dec8x_six
> -
> -Lcbc_dec8x_seven:
> - vncipherlast $out1,$out1,$ivec
> - vncipherlast $out2,$out2,$in1
> - vncipherlast $out3,$out3,$in2
> - vncipherlast $out4,$out4,$in3
> - vncipherlast $out5,$out5,$in4
> - vncipherlast $out6,$out6,$in5
> - vncipherlast $out7,$out7,$in6
> - vmr $ivec,$in7
> -
> - le?vperm $out1,$out1,$out1,$inpperm
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x00,$out
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x10,$out
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x20,$out
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x30,$out
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x40,$out
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x50,$out
> - stvx_u $out7,$x60,$out
> - addi $out,$out,0x70
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lcbc_dec8x_six:
> - vncipherlast $out2,$out2,$ivec
> - vncipherlast $out3,$out3,$in2
> - vncipherlast $out4,$out4,$in3
> - vncipherlast $out5,$out5,$in4
> - vncipherlast $out6,$out6,$in5
> - vncipherlast $out7,$out7,$in6
> - vmr $ivec,$in7
> -
> - le?vperm $out2,$out2,$out2,$inpperm
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x00,$out
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x10,$out
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x20,$out
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x30,$out
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x40,$out
> - stvx_u $out7,$x50,$out
> - addi $out,$out,0x60
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lcbc_dec8x_five:
> - vncipherlast $out3,$out3,$ivec
> - vncipherlast $out4,$out4,$in3
> - vncipherlast $out5,$out5,$in4
> - vncipherlast $out6,$out6,$in5
> - vncipherlast $out7,$out7,$in6
> - vmr $ivec,$in7
> -
> - le?vperm $out3,$out3,$out3,$inpperm
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x00,$out
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x10,$out
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x20,$out
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x30,$out
> - stvx_u $out7,$x40,$out
> - addi $out,$out,0x50
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lcbc_dec8x_four:
> - vncipherlast $out4,$out4,$ivec
> - vncipherlast $out5,$out5,$in4
> - vncipherlast $out6,$out6,$in5
> - vncipherlast $out7,$out7,$in6
> - vmr $ivec,$in7
> -
> - le?vperm $out4,$out4,$out4,$inpperm
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x00,$out
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x10,$out
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x20,$out
> - stvx_u $out7,$x30,$out
> - addi $out,$out,0x40
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lcbc_dec8x_three:
> - vncipherlast $out5,$out5,$ivec
> - vncipherlast $out6,$out6,$in5
> - vncipherlast $out7,$out7,$in6
> - vmr $ivec,$in7
> -
> - le?vperm $out5,$out5,$out5,$inpperm
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x00,$out
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x10,$out
> - stvx_u $out7,$x20,$out
> - addi $out,$out,0x30
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lcbc_dec8x_two:
> - vncipherlast $out6,$out6,$ivec
> - vncipherlast $out7,$out7,$in6
> - vmr $ivec,$in7
> -
> - le?vperm $out6,$out6,$out6,$inpperm
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x00,$out
> - stvx_u $out7,$x10,$out
> - addi $out,$out,0x20
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lcbc_dec8x_one:
> - vncipherlast $out7,$out7,$ivec
> - vmr $ivec,$in7
> -
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out7,0,$out
> - addi $out,$out,0x10
> -
> -Lcbc_dec8x_done:
> - le?vperm $ivec,$ivec,$ivec,$inpperm
> - stvx_u $ivec,0,$ivp # write [unaligned] iv
> -
> - li r10,`$FRAME+15`
> - li r11,`$FRAME+31`
> - stvx $inpperm,r10,$sp # wipe copies of round keys
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> - stvx $inpperm,r10,$sp
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> - stvx $inpperm,r10,$sp
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> - stvx $inpperm,r10,$sp
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> -
> - mtspr 256,$vrsave
> - lvx v20,r10,$sp # ABI says so
> - addi r10,r10,32
> - lvx v21,r11,$sp
> - addi r11,r11,32
> - lvx v22,r10,$sp
> - addi r10,r10,32
> - lvx v23,r11,$sp
> - addi r11,r11,32
> - lvx v24,r10,$sp
> - addi r10,r10,32
> - lvx v25,r11,$sp
> - addi r11,r11,32
> - lvx v26,r10,$sp
> - addi r10,r10,32
> - lvx v27,r11,$sp
> - addi r11,r11,32
> - lvx v28,r10,$sp
> - addi r10,r10,32
> - lvx v29,r11,$sp
> - addi r11,r11,32
> - lvx v30,r10,$sp
> - lvx v31,r11,$sp
> - $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0x80,6,6,0
> - .long 0
> -.size .${prefix}_cbc_encrypt,.-.${prefix}_cbc_encrypt
> -___
> -}} }}}
> -
> -#########################################################################
> -{{{ # CTR procedure[s] #
> -my ($inp,$out,$len,$key,$ivp,$x10,$rounds,$idx)=map("r$_",(3..10));
> -my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3));
> -my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm,$one)=
> - map("v$_",(4..11));
> -my $dat=$tmp;
> -
> -$code.=<<___;
> -.globl .${prefix}_ctr32_encrypt_blocks
> - ${UCMP}i $len,1
> - bltlr-
> -
> - lis r0,0xfff0
> - mfspr $vrsave,256
> - mtspr 256,r0
> -
> - li $idx,15
> - vxor $rndkey0,$rndkey0,$rndkey0
> - le?vspltisb $tmp,0x0f
> -
> - lvx $ivec,0,$ivp # load [unaligned] iv
> - lvsl $inpperm,0,$ivp
> - lvx $inptail,$idx,$ivp
> - vspltisb $one,1
> - le?vxor $inpperm,$inpperm,$tmp
> - vperm $ivec,$ivec,$inptail,$inpperm
> - vsldoi $one,$rndkey0,$one,1
> -
> - neg r11,$inp
> - ?lvsl $keyperm,0,$key # prepare for unaligned key
> - lwz $rounds,240($key)
> -
> - lvsr $inpperm,0,r11 # prepare for unaligned load
> - lvx $inptail,0,$inp
> - addi $inp,$inp,15 # 15 is not typo
> - le?vxor $inpperm,$inpperm,$tmp
> -
> - srwi $rounds,$rounds,1
> - li $idx,16
> - subi $rounds,$rounds,1
> -
> - ${UCMP}i $len,8
> - bge _aesp8_ctr32_encrypt8x
> -
> - ?lvsr $outperm,0,$out # prepare for unaligned store
> - vspltisb $outmask,-1
> - lvx $outhead,0,$out
> - ?vperm $outmask,$rndkey0,$outmask,$outperm
> - le?vxor $outperm,$outperm,$tmp
> -
> - lvx $rndkey0,0,$key
> - mtctr $rounds
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$ivec,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> - b Loop_ctr32_enc
> -
> -.align 5
> -Loop_ctr32_enc:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipher $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> - bdnz Loop_ctr32_enc
> -
> - vadduwm $ivec,$ivec,$one
> - vmr $dat,$inptail
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> - subic. $len,$len,1 # blocks--
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key
> - vperm $dat,$dat,$inptail,$inpperm
> - li $idx,16
> - ?vperm $rndkey1,$rndkey0,$rndkey1,$keyperm
> - lvx $rndkey0,0,$key
> - vxor $dat,$dat,$rndkey1 # last round key
> - vcipherlast $inout,$inout,$dat
> -
> - lvx $rndkey1,$idx,$key
> - addi $idx,$idx,16
> - vperm $inout,$inout,$inout,$outperm
> - vsel $dat,$outhead,$inout,$outmask
> - mtctr $rounds
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vmr $outhead,$inout
> - vxor $inout,$ivec,$rndkey0
> - lvx $rndkey0,$idx,$key
> - addi $idx,$idx,16
> - stvx $dat,0,$out
> - addi $out,$out,16
> - bne Loop_ctr32_enc
> -
> - addi $out,$out,-1
> - lvx $inout,0,$out # redundant in aligned case
> - vsel $inout,$outhead,$inout,$outmask
> - stvx $inout,0,$out
> -
> - mtspr 256,$vrsave
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,6,0
> - .long 0
> -___
> -#########################################################################
> -{{ # Optimized CTR procedure #
> -my $key_="r11";
> -my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31));
> -my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10,12..14));
> -my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(15..22));
> -my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys
> - # v26-v31 last 6 round keys
> -my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment
> -my ($two,$three,$four)=($outhead,$outperm,$outmask);
> -
> -$code.=<<___;
> -.align 5
> -_aesp8_ctr32_encrypt8x:
> - $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
> - li r10,`$FRAME+8*16+15`
> - li r11,`$FRAME+8*16+31`
> - stvx v20,r10,$sp # ABI says so
> - addi r10,r10,32
> - stvx v21,r11,$sp
> - addi r11,r11,32
> - stvx v22,r10,$sp
> - addi r10,r10,32
> - stvx v23,r11,$sp
> - addi r11,r11,32
> - stvx v24,r10,$sp
> - addi r10,r10,32
> - stvx v25,r11,$sp
> - addi r11,r11,32
> - stvx v26,r10,$sp
> - addi r10,r10,32
> - stvx v27,r11,$sp
> - addi r11,r11,32
> - stvx v28,r10,$sp
> - addi r10,r10,32
> - stvx v29,r11,$sp
> - addi r11,r11,32
> - stvx v30,r10,$sp
> - stvx v31,r11,$sp
> - li r0,-1
> - stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
> - li $x10,0x10
> - $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - li $x20,0x20
> - $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - li $x30,0x30
> - $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - li $x40,0x40
> - $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - li $x50,0x50
> - $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - li $x60,0x60
> - $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - li $x70,0x70
> - mtspr 256,r0
> -
> - subi $rounds,$rounds,3 # -4 in total
> -
> - lvx $rndkey0,$x00,$key # load key schedule
> - lvx v30,$x10,$key
> - addi $key,$key,0x20
> - lvx v31,$x00,$key
> - ?vperm $rndkey0,$rndkey0,v30,$keyperm
> - addi $key_,$sp,$FRAME+15
> - mtctr $rounds
> -
> -Load_ctr32_enc_key:
> - ?vperm v24,v30,v31,$keyperm
> - lvx v30,$x10,$key
> - addi $key,$key,0x20
> - stvx v24,$x00,$key_ # off-load round[1]
> - ?vperm v25,v31,v30,$keyperm
> - lvx v31,$x00,$key
> - stvx v25,$x10,$key_ # off-load round[2]
> - addi $key_,$key_,0x20
> - bdnz Load_ctr32_enc_key
> -
> - lvx v26,$x10,$key
> - ?vperm v24,v30,v31,$keyperm
> - lvx v27,$x20,$key
> - stvx v24,$x00,$key_ # off-load round[3]
> - ?vperm v25,v31,v26,$keyperm
> - lvx v28,$x30,$key
> - stvx v25,$x10,$key_ # off-load round[4]
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - ?vperm v26,v26,v27,$keyperm
> - lvx v29,$x40,$key
> - ?vperm v27,v27,v28,$keyperm
> - lvx v30,$x50,$key
> - ?vperm v28,v28,v29,$keyperm
> - lvx v31,$x60,$key
> - ?vperm v29,v29,v30,$keyperm
> - lvx $out0,$x70,$key # borrow $out0
> - ?vperm v30,v30,v31,$keyperm
> - lvx v24,$x00,$key_ # pre-load round[1]
> - ?vperm v31,v31,$out0,$keyperm
> - lvx v25,$x10,$key_ # pre-load round[2]
> -
> - vadduqm $two,$one,$one
> - subi $inp,$inp,15 # undo "caller"
> - $SHL $len,$len,4
> -
> - vadduqm $out1,$ivec,$one # counter values ...
> - vadduqm $out2,$ivec,$two
> - vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0]
> - le?li $idx,8
> - vadduqm $out3,$out1,$two
> - vxor $out1,$out1,$rndkey0
> - le?lvsl $inpperm,0,$idx
> - vadduqm $out4,$out2,$two
> - vxor $out2,$out2,$rndkey0
> - le?vspltisb $tmp,0x0f
> - vadduqm $out5,$out3,$two
> - vxor $out3,$out3,$rndkey0
> - le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u
> - vadduqm $out6,$out4,$two
> - vxor $out4,$out4,$rndkey0
> - vadduqm $out7,$out5,$two
> - vxor $out5,$out5,$rndkey0
> - vadduqm $ivec,$out6,$two # next counter value
> - vxor $out6,$out6,$rndkey0
> - vxor $out7,$out7,$rndkey0
> -
> - mtctr $rounds
> - b Loop_ctr32_enc8x
> -.align 5
> -Loop_ctr32_enc8x:
> - vcipher $out0,$out0,v24
> - vcipher $out1,$out1,v24
> - vcipher $out2,$out2,v24
> - vcipher $out3,$out3,v24
> - vcipher $out4,$out4,v24
> - vcipher $out5,$out5,v24
> - vcipher $out6,$out6,v24
> - vcipher $out7,$out7,v24
> -Loop_ctr32_enc8x_middle:
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vcipher $out0,$out0,v25
> - vcipher $out1,$out1,v25
> - vcipher $out2,$out2,v25
> - vcipher $out3,$out3,v25
> - vcipher $out4,$out4,v25
> - vcipher $out5,$out5,v25
> - vcipher $out6,$out6,v25
> - vcipher $out7,$out7,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_ctr32_enc8x
> -
> - subic r11,$len,256 # $len-256, borrow $key_
> - vcipher $out0,$out0,v24
> - vcipher $out1,$out1,v24
> - vcipher $out2,$out2,v24
> - vcipher $out3,$out3,v24
> - vcipher $out4,$out4,v24
> - vcipher $out5,$out5,v24
> - vcipher $out6,$out6,v24
> - vcipher $out7,$out7,v24
> -
> - subfe r0,r0,r0 # borrow?-1:0
> - vcipher $out0,$out0,v25
> - vcipher $out1,$out1,v25
> - vcipher $out2,$out2,v25
> - vcipher $out3,$out3,v25
> - vcipher $out4,$out4,v25
> - vcipher $out5,$out5,v25
> - vcipher $out6,$out6,v25
> - vcipher $out7,$out7,v25
> -
> - and r0,r0,r11
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vcipher $out0,$out0,v26
> - vcipher $out1,$out1,v26
> - vcipher $out2,$out2,v26
> - vcipher $out3,$out3,v26
> - vcipher $out4,$out4,v26
> - vcipher $out5,$out5,v26
> - vcipher $out6,$out6,v26
> - vcipher $out7,$out7,v26
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> -
> - subic $len,$len,129 # $len-=129
> - vcipher $out0,$out0,v27
> - addi $len,$len,1 # $len-=128 really
> - vcipher $out1,$out1,v27
> - vcipher $out2,$out2,v27
> - vcipher $out3,$out3,v27
> - vcipher $out4,$out4,v27
> - vcipher $out5,$out5,v27
> - vcipher $out6,$out6,v27
> - vcipher $out7,$out7,v27
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> -
> - vcipher $out0,$out0,v28
> - lvx_u $in0,$x00,$inp # load input
> - vcipher $out1,$out1,v28
> - lvx_u $in1,$x10,$inp
> - vcipher $out2,$out2,v28
> - lvx_u $in2,$x20,$inp
> - vcipher $out3,$out3,v28
> - lvx_u $in3,$x30,$inp
> - vcipher $out4,$out4,v28
> - lvx_u $in4,$x40,$inp
> - vcipher $out5,$out5,v28
> - lvx_u $in5,$x50,$inp
> - vcipher $out6,$out6,v28
> - lvx_u $in6,$x60,$inp
> - vcipher $out7,$out7,v28
> - lvx_u $in7,$x70,$inp
> - addi $inp,$inp,0x80
> -
> - vcipher $out0,$out0,v29
> - le?vperm $in0,$in0,$in0,$inpperm
> - vcipher $out1,$out1,v29
> - le?vperm $in1,$in1,$in1,$inpperm
> - vcipher $out2,$out2,v29
> - le?vperm $in2,$in2,$in2,$inpperm
> - vcipher $out3,$out3,v29
> - le?vperm $in3,$in3,$in3,$inpperm
> - vcipher $out4,$out4,v29
> - le?vperm $in4,$in4,$in4,$inpperm
> - vcipher $out5,$out5,v29
> - le?vperm $in5,$in5,$in5,$inpperm
> - vcipher $out6,$out6,v29
> - le?vperm $in6,$in6,$in6,$inpperm
> - vcipher $out7,$out7,v29
> - le?vperm $in7,$in7,$in7,$inpperm
> -
> - add $inp,$inp,r0 # $inp is adjusted in such
> - # way that at exit from the
> - # loop inX-in7 are loaded
> - # with last "words"
> - subfe. r0,r0,r0 # borrow?-1:0
> - vcipher $out0,$out0,v30
> - vxor $in0,$in0,v31 # xor with last round key
> - vcipher $out1,$out1,v30
> - vxor $in1,$in1,v31
> - vcipher $out2,$out2,v30
> - vxor $in2,$in2,v31
> - vcipher $out3,$out3,v30
> - vxor $in3,$in3,v31
> - vcipher $out4,$out4,v30
> - vxor $in4,$in4,v31
> - vcipher $out5,$out5,v30
> - vxor $in5,$in5,v31
> - vcipher $out6,$out6,v30
> - vxor $in6,$in6,v31
> - vcipher $out7,$out7,v30
> - vxor $in7,$in7,v31
> -
> - bne Lctr32_enc8x_break # did $len-129 borrow?
> -
> - vcipherlast $in0,$out0,$in0
> - vcipherlast $in1,$out1,$in1
> - vadduqm $out1,$ivec,$one # counter values ...
> - vcipherlast $in2,$out2,$in2
> - vadduqm $out2,$ivec,$two
> - vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0]
> - vcipherlast $in3,$out3,$in3
> - vadduqm $out3,$out1,$two
> - vxor $out1,$out1,$rndkey0
> - vcipherlast $in4,$out4,$in4
> - vadduqm $out4,$out2,$two
> - vxor $out2,$out2,$rndkey0
> - vcipherlast $in5,$out5,$in5
> - vadduqm $out5,$out3,$two
> - vxor $out3,$out3,$rndkey0
> - vcipherlast $in6,$out6,$in6
> - vadduqm $out6,$out4,$two
> - vxor $out4,$out4,$rndkey0
> - vcipherlast $in7,$out7,$in7
> - vadduqm $out7,$out5,$two
> - vxor $out5,$out5,$rndkey0
> - le?vperm $in0,$in0,$in0,$inpperm
> - vadduqm $ivec,$out6,$two # next counter value
> - vxor $out6,$out6,$rndkey0
> - le?vperm $in1,$in1,$in1,$inpperm
> - vxor $out7,$out7,$rndkey0
> - mtctr $rounds
> -
> - vcipher $out0,$out0,v24
> - stvx_u $in0,$x00,$out
> - le?vperm $in2,$in2,$in2,$inpperm
> - vcipher $out1,$out1,v24
> - stvx_u $in1,$x10,$out
> - le?vperm $in3,$in3,$in3,$inpperm
> - vcipher $out2,$out2,v24
> - stvx_u $in2,$x20,$out
> - le?vperm $in4,$in4,$in4,$inpperm
> - vcipher $out3,$out3,v24
> - stvx_u $in3,$x30,$out
> - le?vperm $in5,$in5,$in5,$inpperm
> - vcipher $out4,$out4,v24
> - stvx_u $in4,$x40,$out
> - le?vperm $in6,$in6,$in6,$inpperm
> - vcipher $out5,$out5,v24
> - stvx_u $in5,$x50,$out
> - le?vperm $in7,$in7,$in7,$inpperm
> - vcipher $out6,$out6,v24
> - stvx_u $in6,$x60,$out
> - vcipher $out7,$out7,v24
> - stvx_u $in7,$x70,$out
> - addi $out,$out,0x80
> -
> - b Loop_ctr32_enc8x_middle
> -
> -.align 5
> -Lctr32_enc8x_break:
> - cmpwi $len,-0x60
> - blt Lctr32_enc8x_one
> - nop
> - beq Lctr32_enc8x_two
> - cmpwi $len,-0x40
> - blt Lctr32_enc8x_three
> - nop
> - beq Lctr32_enc8x_four
> - cmpwi $len,-0x20
> - blt Lctr32_enc8x_five
> - nop
> - beq Lctr32_enc8x_six
> - cmpwi $len,0x00
> - blt Lctr32_enc8x_seven
> -
> -Lctr32_enc8x_eight:
> - vcipherlast $out0,$out0,$in0
> - vcipherlast $out1,$out1,$in1
> - vcipherlast $out2,$out2,$in2
> - vcipherlast $out3,$out3,$in3
> - vcipherlast $out4,$out4,$in4
> - vcipherlast $out5,$out5,$in5
> - vcipherlast $out6,$out6,$in6
> - vcipherlast $out7,$out7,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x20,$out
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x30,$out
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x40,$out
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x50,$out
> - le?vperm $out7,$out7,$out7,$inpperm
> - stvx_u $out6,$x60,$out
> - stvx_u $out7,$x70,$out
> - addi $out,$out,0x80
> - b Lctr32_enc8x_done
> -
> -.align 5
> -Lctr32_enc8x_seven:
> - vcipherlast $out0,$out0,$in1
> - vcipherlast $out1,$out1,$in2
> - vcipherlast $out2,$out2,$in3
> - vcipherlast $out3,$out3,$in4
> - vcipherlast $out4,$out4,$in5
> - vcipherlast $out5,$out5,$in6
> - vcipherlast $out6,$out6,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x20,$out
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x30,$out
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x40,$out
> - le?vperm $out6,$out6,$out6,$inpperm
> - stvx_u $out5,$x50,$out
> - stvx_u $out6,$x60,$out
> - addi $out,$out,0x70
> - b Lctr32_enc8x_done
> -
> -.align 5
> -Lctr32_enc8x_six:
> - vcipherlast $out0,$out0,$in2
> - vcipherlast $out1,$out1,$in3
> - vcipherlast $out2,$out2,$in4
> - vcipherlast $out3,$out3,$in5
> - vcipherlast $out4,$out4,$in6
> - vcipherlast $out5,$out5,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x20,$out
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x30,$out
> - le?vperm $out5,$out5,$out5,$inpperm
> - stvx_u $out4,$x40,$out
> - stvx_u $out5,$x50,$out
> - addi $out,$out,0x60
> - b Lctr32_enc8x_done
> -
> -.align 5
> -Lctr32_enc8x_five:
> - vcipherlast $out0,$out0,$in3
> - vcipherlast $out1,$out1,$in4
> - vcipherlast $out2,$out2,$in5
> - vcipherlast $out3,$out3,$in6
> - vcipherlast $out4,$out4,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x20,$out
> - le?vperm $out4,$out4,$out4,$inpperm
> - stvx_u $out3,$x30,$out
> - stvx_u $out4,$x40,$out
> - addi $out,$out,0x50
> - b Lctr32_enc8x_done
> -
> -.align 5
> -Lctr32_enc8x_four:
> - vcipherlast $out0,$out0,$in4
> - vcipherlast $out1,$out1,$in5
> - vcipherlast $out2,$out2,$in6
> - vcipherlast $out3,$out3,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$inpperm
> - stvx_u $out2,$x20,$out
> - stvx_u $out3,$x30,$out
> - addi $out,$out,0x40
> - b Lctr32_enc8x_done
> -
> -.align 5
> -Lctr32_enc8x_three:
> - vcipherlast $out0,$out0,$in5
> - vcipherlast $out1,$out1,$in6
> - vcipherlast $out2,$out2,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - le?vperm $out2,$out2,$out2,$inpperm
> - stvx_u $out1,$x10,$out
> - stvx_u $out2,$x20,$out
> - addi $out,$out,0x30
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lctr32_enc8x_two:
> - vcipherlast $out0,$out0,$in6
> - vcipherlast $out1,$out1,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - le?vperm $out1,$out1,$out1,$inpperm
> - stvx_u $out0,$x00,$out
> - stvx_u $out1,$x10,$out
> - addi $out,$out,0x20
> - b Lcbc_dec8x_done
> -
> -.align 5
> -Lctr32_enc8x_one:
> - vcipherlast $out0,$out0,$in7
> -
> - le?vperm $out0,$out0,$out0,$inpperm
> - stvx_u $out0,0,$out
> - addi $out,$out,0x10
> -
> -Lctr32_enc8x_done:
> - li r10,`$FRAME+15`
> - li r11,`$FRAME+31`
> - stvx $inpperm,r10,$sp # wipe copies of round keys
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> - stvx $inpperm,r10,$sp
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> - stvx $inpperm,r10,$sp
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> - stvx $inpperm,r10,$sp
> - addi r10,r10,32
> - stvx $inpperm,r11,$sp
> - addi r11,r11,32
> -
> - mtspr 256,$vrsave
> - lvx v20,r10,$sp # ABI says so
> - addi r10,r10,32
> - lvx v21,r11,$sp
> - addi r11,r11,32
> - lvx v22,r10,$sp
> - addi r10,r10,32
> - lvx v23,r11,$sp
> - addi r11,r11,32
> - lvx v24,r10,$sp
> - addi r10,r10,32
> - lvx v25,r11,$sp
> - addi r11,r11,32
> - lvx v26,r10,$sp
> - addi r10,r10,32
> - lvx v27,r11,$sp
> - addi r11,r11,32
> - lvx v28,r10,$sp
> - addi r10,r10,32
> - lvx v29,r11,$sp
> - addi r11,r11,32
> - lvx v30,r10,$sp
> - lvx v31,r11,$sp
> - $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0x80,6,6,0
> - .long 0
> -.size .${prefix}_ctr32_encrypt_blocks,.-.${prefix}_ctr32_encrypt_blocks
> -___
> -}} }}}
> -
> -#########################################################################
> -{{{ # XTS procedures #
> -# int aes_p8_xts_[en|de]crypt(const char *inp, char *out, size_t len, #
> -# const AES_KEY *key1, const AES_KEY *key2, #
> -# [const] unsigned char iv[16]); #
> -# If $key2 is NULL, then a "tweak chaining" mode is engaged, in which #
> -# input tweak value is assumed to be encrypted already, and last tweak #
> -# value, one suitable for consecutive call on same chunk of data, is #
> -# written back to original buffer. In addition, in "tweak chaining" #
> -# mode only complete input blocks are processed. #
> -
> -my ($inp,$out,$len,$key1,$key2,$ivp,$rounds,$idx) = map("r$_",(3..10));
> -my ($rndkey0,$rndkey1,$inout) = map("v$_",(0..2));
> -my ($output,$inptail,$inpperm,$leperm,$keyperm) = map("v$_",(3..7));
> -my ($tweak,$seven,$eighty7,$tmp,$tweak1) = map("v$_",(8..12));
> -my $taillen = $key2;
> -
> - ($inp,$idx) = ($idx,$inp); # reassign
> -
> -$code.=<<___;
> -.globl .${prefix}_xts_encrypt
> - mr $inp,r3 # reassign
> - li r3,-1
> - ${UCMP}i $len,16
> - bltlr-
> -
> - lis r0,0xfff0
> - mfspr r12,256 # save vrsave
> - li r11,0
> - mtspr 256,r0
> -
> - vspltisb $seven,0x07 # 0x070707..07
> - le?lvsl $leperm,r11,r11
> - le?vspltisb $tmp,0x0f
> - le?vxor $leperm,$leperm,$seven
> -
> - li $idx,15
> - lvx $tweak,0,$ivp # load [unaligned] iv
> - lvsl $inpperm,0,$ivp
> - lvx $inptail,$idx,$ivp
> - le?vxor $inpperm,$inpperm,$tmp
> - vperm $tweak,$tweak,$inptail,$inpperm
> -
> - neg r11,$inp
> - lvsr $inpperm,0,r11 # prepare for unaligned load
> - lvx $inout,0,$inp
> - addi $inp,$inp,15 # 15 is not typo
> - le?vxor $inpperm,$inpperm,$tmp
> -
> - ${UCMP}i $key2,0 # key2==NULL?
> - beq Lxts_enc_no_key2
> -
> - ?lvsl $keyperm,0,$key2 # prepare for unaligned key
> - lwz $rounds,240($key2)
> - srwi $rounds,$rounds,1
> - subi $rounds,$rounds,1
> - li $idx,16
> -
> - lvx $rndkey0,0,$key2
> - lvx $rndkey1,$idx,$key2
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $tweak,$tweak,$rndkey0
> - lvx $rndkey0,$idx,$key2
> - addi $idx,$idx,16
> - mtctr $rounds
> -
> -Ltweak_xts_enc:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $tweak,$tweak,$rndkey1
> - lvx $rndkey1,$idx,$key2
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipher $tweak,$tweak,$rndkey0
> - lvx $rndkey0,$idx,$key2
> - addi $idx,$idx,16
> - bdnz Ltweak_xts_enc
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $tweak,$tweak,$rndkey1
> - lvx $rndkey1,$idx,$key2
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipherlast $tweak,$tweak,$rndkey0
> -
> - li $ivp,0 # don't chain the tweak
> - b Lxts_enc
> -
> -Lxts_enc_no_key2:
> - li $idx,-16
> - and $len,$len,$idx # in "tweak chaining"
> - # mode only complete
> - # blocks are processed
> -Lxts_enc:
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> -
> - ?lvsl $keyperm,0,$key1 # prepare for unaligned key
> - lwz $rounds,240($key1)
> - srwi $rounds,$rounds,1
> - subi $rounds,$rounds,1
> - li $idx,16
> -
> - vslb $eighty7,$seven,$seven # 0x808080..80
> - vor $eighty7,$eighty7,$seven # 0x878787..87
> - vspltisb $tmp,1 # 0x010101..01
> - vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01
> -
> - ${UCMP}i $len,96
> - bge _aesp8_xts_encrypt6x
> -
> - andi. $taillen,$len,15
> - subic r0,$len,32
> - subi $taillen,$taillen,16
> - subfe r0,r0,r0
> - and r0,r0,$taillen
> - add $inp,$inp,r0
> -
> - lvx $rndkey0,0,$key1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> - vperm $inout,$inout,$inptail,$inpperm
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$inout,$tweak
> - vxor $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> - mtctr $rounds
> - b Loop_xts_enc
> -
> -.align 5
> -Loop_xts_enc:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipher $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> - bdnz Loop_xts_enc
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key1
> - li $idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $rndkey0,$rndkey0,$tweak
> - vcipherlast $output,$inout,$rndkey0
> -
> - le?vperm $tmp,$output,$output,$leperm
> - be?nop
> - le?stvx_u $tmp,0,$out
> - be?stvx_u $output,0,$out
> - addi $out,$out,16
> -
> - subic. $len,$len,16
> - beq Lxts_enc_done
> -
> - vmr $inout,$inptail
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> - lvx $rndkey0,0,$key1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> -
> - subic r0,$len,32
> - subfe r0,r0,r0
> - and r0,r0,$taillen
> - add $inp,$inp,r0
> -
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $tweak,$tweak,$tmp
> -
> - vperm $inout,$inout,$inptail,$inpperm
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$inout,$tweak
> - vxor $output,$output,$rndkey0 # just in case $len<16
> - vxor $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> -
> - mtctr $rounds
> - ${UCMP}i $len,16
> - bge Loop_xts_enc
> -
> - vxor $output,$output,$tweak
> - lvsr $inpperm,0,$len # $inpperm is no longer needed
> - vxor $inptail,$inptail,$inptail # $inptail is no longer needed
> - vspltisb $tmp,-1
> - vperm $inptail,$inptail,$tmp,$inpperm
> - vsel $inout,$inout,$output,$inptail
> -
> - subi r11,$out,17
> - subi $out,$out,16
> - mtctr $len
> - li $len,16
> -Loop_xts_enc_steal:
> - lbzu r0,1(r11)
> - stb r0,16(r11)
> - bdnz Loop_xts_enc_steal
> -
> - mtctr $rounds
> - b Loop_xts_enc # one more time...
> -
> -Lxts_enc_done:
> - ${UCMP}i $ivp,0
> - beq Lxts_enc_ret
> -
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $tweak,$tweak,$tmp
> -
> - le?vperm $tweak,$tweak,$tweak,$leperm
> - stvx_u $tweak,0,$ivp
> -
> -Lxts_enc_ret:
> - mtspr 256,r12 # restore vrsave
> - li r3,0
> - blr
> - .long 0
> - .byte 0,12,0x04,0,0x80,6,6,0
> - .long 0
> -.size .${prefix}_xts_encrypt,.-.${prefix}_xts_encrypt
> -
> -.globl .${prefix}_xts_decrypt
> - mr $inp,r3 # reassign
> - li r3,-1
> - ${UCMP}i $len,16
> - bltlr-
> -
> - lis r0,0xfff8
> - mfspr r12,256 # save vrsave
> - li r11,0
> - mtspr 256,r0
> -
> - andi. r0,$len,15
> - neg r0,r0
> - andi. r0,r0,16
> - sub $len,$len,r0
> -
> - vspltisb $seven,0x07 # 0x070707..07
> - le?lvsl $leperm,r11,r11
> - le?vspltisb $tmp,0x0f
> - le?vxor $leperm,$leperm,$seven
> -
> - li $idx,15
> - lvx $tweak,0,$ivp # load [unaligned] iv
> - lvsl $inpperm,0,$ivp
> - lvx $inptail,$idx,$ivp
> - le?vxor $inpperm,$inpperm,$tmp
> - vperm $tweak,$tweak,$inptail,$inpperm
> -
> - neg r11,$inp
> - lvsr $inpperm,0,r11 # prepare for unaligned load
> - lvx $inout,0,$inp
> - addi $inp,$inp,15 # 15 is not typo
> - le?vxor $inpperm,$inpperm,$tmp
> -
> - ${UCMP}i $key2,0 # key2==NULL?
> - beq Lxts_dec_no_key2
> -
> - ?lvsl $keyperm,0,$key2 # prepare for unaligned key
> - lwz $rounds,240($key2)
> - srwi $rounds,$rounds,1
> - subi $rounds,$rounds,1
> - li $idx,16
> -
> - lvx $rndkey0,0,$key2
> - lvx $rndkey1,$idx,$key2
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $tweak,$tweak,$rndkey0
> - lvx $rndkey0,$idx,$key2
> - addi $idx,$idx,16
> - mtctr $rounds
> -
> -Ltweak_xts_dec:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $tweak,$tweak,$rndkey1
> - lvx $rndkey1,$idx,$key2
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipher $tweak,$tweak,$rndkey0
> - lvx $rndkey0,$idx,$key2
> - addi $idx,$idx,16
> - bdnz Ltweak_xts_dec
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vcipher $tweak,$tweak,$rndkey1
> - lvx $rndkey1,$idx,$key2
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vcipherlast $tweak,$tweak,$rndkey0
> -
> - li $ivp,0 # don't chain the tweak
> - b Lxts_dec
> -
> -Lxts_dec_no_key2:
> - neg $idx,$len
> - andi. $idx,$idx,15
> - add $len,$len,$idx # in "tweak chaining"
> - # mode only complete
> - # blocks are processed
> -Lxts_dec:
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> -
> - ?lvsl $keyperm,0,$key1 # prepare for unaligned key
> - lwz $rounds,240($key1)
> - srwi $rounds,$rounds,1
> - subi $rounds,$rounds,1
> - li $idx,16
> -
> - vslb $eighty7,$seven,$seven # 0x808080..80
> - vor $eighty7,$eighty7,$seven # 0x878787..87
> - vspltisb $tmp,1 # 0x010101..01
> - vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01
> -
> - ${UCMP}i $len,96
> - bge _aesp8_xts_decrypt6x
> -
> - lvx $rndkey0,0,$key1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> - vperm $inout,$inout,$inptail,$inpperm
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$inout,$tweak
> - vxor $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> - mtctr $rounds
> -
> - ${UCMP}i $len,16
> - blt Ltail_xts_dec
> - be?b Loop_xts_dec
> -
> -.align 5
> -Loop_xts_dec:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vncipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vncipher $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> - bdnz Loop_xts_dec
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vncipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key1
> - li $idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $rndkey0,$rndkey0,$tweak
> - vncipherlast $output,$inout,$rndkey0
> -
> - le?vperm $tmp,$output,$output,$leperm
> - be?nop
> - le?stvx_u $tmp,0,$out
> - be?stvx_u $output,0,$out
> - addi $out,$out,16
> -
> - subic. $len,$len,16
> - beq Lxts_dec_done
> -
> - vmr $inout,$inptail
> - lvx $inptail,0,$inp
> - addi $inp,$inp,16
> - lvx $rndkey0,0,$key1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> -
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $tweak,$tweak,$tmp
> -
> - vperm $inout,$inout,$inptail,$inpperm
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $inout,$inout,$tweak
> - vxor $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> -
> - mtctr $rounds
> - ${UCMP}i $len,16
> - bge Loop_xts_dec
> -
> -Ltail_xts_dec:
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak1,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $tweak1,$tweak1,$tmp
> -
> - subi $inp,$inp,16
> - add $inp,$inp,$len
> -
> - vxor $inout,$inout,$tweak # :-(
> - vxor $inout,$inout,$tweak1 # :-)
> -
> -Loop_xts_dec_short:
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vncipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vncipher $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> - bdnz Loop_xts_dec_short
> -
> - ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm
> - vncipher $inout,$inout,$rndkey1
> - lvx $rndkey1,$idx,$key1
> - li $idx,16
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> - vxor $rndkey0,$rndkey0,$tweak1
> - vncipherlast $output,$inout,$rndkey0
> -
> - le?vperm $tmp,$output,$output,$leperm
> - be?nop
> - le?stvx_u $tmp,0,$out
> - be?stvx_u $output,0,$out
> -
> - vmr $inout,$inptail
> - lvx $inptail,0,$inp
> - #addi $inp,$inp,16
> - lvx $rndkey0,0,$key1
> - lvx $rndkey1,$idx,$key1
> - addi $idx,$idx,16
> - vperm $inout,$inout,$inptail,$inpperm
> - ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm
> -
> - lvsr $inpperm,0,$len # $inpperm is no longer needed
> - vxor $inptail,$inptail,$inptail # $inptail is no longer needed
> - vspltisb $tmp,-1
> - vperm $inptail,$inptail,$tmp,$inpperm
> - vsel $inout,$inout,$output,$inptail
> -
> - vxor $rndkey0,$rndkey0,$tweak
> - vxor $inout,$inout,$rndkey0
> - lvx $rndkey0,$idx,$key1
> - addi $idx,$idx,16
> -
> - subi r11,$out,1
> - mtctr $len
> - li $len,16
> -Loop_xts_dec_steal:
> - lbzu r0,1(r11)
> - stb r0,16(r11)
> - bdnz Loop_xts_dec_steal
> -
> - mtctr $rounds
> - b Loop_xts_dec # one more time...
> -
> -Lxts_dec_done:
> - ${UCMP}i $ivp,0
> - beq Lxts_dec_ret
> -
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $tweak,$tweak,$tmp
> -
> - le?vperm $tweak,$tweak,$tweak,$leperm
> - stvx_u $tweak,0,$ivp
> -
> -Lxts_dec_ret:
> - mtspr 256,r12 # restore vrsave
> - li r3,0
> - blr
> - .long 0
> - .byte 0,12,0x04,0,0x80,6,6,0
> - .long 0
> -.size .${prefix}_xts_decrypt,.-.${prefix}_xts_decrypt
> -___
> -#########################################################################
> -{{ # Optimized XTS procedures #
> -my $key_=$key2;
> -my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,3,26..31));
> - $x00=0 if ($flavour =~ /osx/);
> -my ($in0, $in1, $in2, $in3, $in4, $in5 )=map("v$_",(0..5));
> -my ($out0, $out1, $out2, $out3, $out4, $out5)=map("v$_",(7,12..16));
> -my ($twk0, $twk1, $twk2, $twk3, $twk4, $twk5)=map("v$_",(17..22));
> -my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys
> - # v26-v31 last 6 round keys
> -my ($keyperm)=($out0); # aliases with "caller", redundant assignment
> -my $taillen=$x70;
> -
> -$code.=<<___;
> -.align 5
> -_aesp8_xts_encrypt6x:
> - $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
> - mflr r11
> - li r7,`$FRAME+8*16+15`
> - li r3,`$FRAME+8*16+31`
> - $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp)
> - stvx v20,r7,$sp # ABI says so
> - addi r7,r7,32
> - stvx v21,r3,$sp
> - addi r3,r3,32
> - stvx v22,r7,$sp
> - addi r7,r7,32
> - stvx v23,r3,$sp
> - addi r3,r3,32
> - stvx v24,r7,$sp
> - addi r7,r7,32
> - stvx v25,r3,$sp
> - addi r3,r3,32
> - stvx v26,r7,$sp
> - addi r7,r7,32
> - stvx v27,r3,$sp
> - addi r3,r3,32
> - stvx v28,r7,$sp
> - addi r7,r7,32
> - stvx v29,r3,$sp
> - addi r3,r3,32
> - stvx v30,r7,$sp
> - stvx v31,r3,$sp
> - li r0,-1
> - stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
> - li $x10,0x10
> - $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - li $x20,0x20
> - $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - li $x30,0x30
> - $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - li $x40,0x40
> - $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - li $x50,0x50
> - $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - li $x60,0x60
> - $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - li $x70,0x70
> - mtspr 256,r0
> -
> - subi $rounds,$rounds,3 # -4 in total
> -
> - lvx $rndkey0,$x00,$key1 # load key schedule
> - lvx v30,$x10,$key1
> - addi $key1,$key1,0x20
> - lvx v31,$x00,$key1
> - ?vperm $rndkey0,$rndkey0,v30,$keyperm
> - addi $key_,$sp,$FRAME+15
> - mtctr $rounds
> -
> -Load_xts_enc_key:
> - ?vperm v24,v30,v31,$keyperm
> - lvx v30,$x10,$key1
> - addi $key1,$key1,0x20
> - stvx v24,$x00,$key_ # off-load round[1]
> - ?vperm v25,v31,v30,$keyperm
> - lvx v31,$x00,$key1
> - stvx v25,$x10,$key_ # off-load round[2]
> - addi $key_,$key_,0x20
> - bdnz Load_xts_enc_key
> -
> - lvx v26,$x10,$key1
> - ?vperm v24,v30,v31,$keyperm
> - lvx v27,$x20,$key1
> - stvx v24,$x00,$key_ # off-load round[3]
> - ?vperm v25,v31,v26,$keyperm
> - lvx v28,$x30,$key1
> - stvx v25,$x10,$key_ # off-load round[4]
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - ?vperm v26,v26,v27,$keyperm
> - lvx v29,$x40,$key1
> - ?vperm v27,v27,v28,$keyperm
> - lvx v30,$x50,$key1
> - ?vperm v28,v28,v29,$keyperm
> - lvx v31,$x60,$key1
> - ?vperm v29,v29,v30,$keyperm
> - lvx $twk5,$x70,$key1 # borrow $twk5
> - ?vperm v30,v30,v31,$keyperm
> - lvx v24,$x00,$key_ # pre-load round[1]
> - ?vperm v31,v31,$twk5,$keyperm
> - lvx v25,$x10,$key_ # pre-load round[2]
> -
> - vperm $in0,$inout,$inptail,$inpperm
> - subi $inp,$inp,31 # undo "caller"
> - vxor $twk0,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $out0,$in0,$twk0
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in1,$x10,$inp
> - vxor $twk1,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in1,$in1,$in1,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out1,$in1,$twk1
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in2,$x20,$inp
> - andi. $taillen,$len,15
> - vxor $twk2,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in2,$in2,$in2,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out2,$in2,$twk2
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in3,$x30,$inp
> - sub $len,$len,$taillen
> - vxor $twk3,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in3,$in3,$in3,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out3,$in3,$twk3
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in4,$x40,$inp
> - subi $len,$len,0x60
> - vxor $twk4,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in4,$in4,$in4,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out4,$in4,$twk4
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in5,$x50,$inp
> - addi $inp,$inp,0x60
> - vxor $twk5,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in5,$in5,$in5,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out5,$in5,$twk5
> - vxor $tweak,$tweak,$tmp
> -
> - vxor v31,v31,$rndkey0
> - mtctr $rounds
> - b Loop_xts_enc6x
> -
> -.align 5
> -Loop_xts_enc6x:
> - vcipher $out0,$out0,v24
> - vcipher $out1,$out1,v24
> - vcipher $out2,$out2,v24
> - vcipher $out3,$out3,v24
> - vcipher $out4,$out4,v24
> - vcipher $out5,$out5,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vcipher $out0,$out0,v25
> - vcipher $out1,$out1,v25
> - vcipher $out2,$out2,v25
> - vcipher $out3,$out3,v25
> - vcipher $out4,$out4,v25
> - vcipher $out5,$out5,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_xts_enc6x
> -
> - subic $len,$len,96 # $len-=96
> - vxor $in0,$twk0,v31 # xor with last round key
> - vcipher $out0,$out0,v24
> - vcipher $out1,$out1,v24
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk0,$tweak,$rndkey0
> - vaddubm $tweak,$tweak,$tweak
> - vcipher $out2,$out2,v24
> - vcipher $out3,$out3,v24
> - vsldoi $tmp,$tmp,$tmp,15
> - vcipher $out4,$out4,v24
> - vcipher $out5,$out5,v24
> -
> - subfe. r0,r0,r0 # borrow?-1:0
> - vand $tmp,$tmp,$eighty7
> - vcipher $out0,$out0,v25
> - vcipher $out1,$out1,v25
> - vxor $tweak,$tweak,$tmp
> - vcipher $out2,$out2,v25
> - vcipher $out3,$out3,v25
> - vxor $in1,$twk1,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk1,$tweak,$rndkey0
> - vcipher $out4,$out4,v25
> - vcipher $out5,$out5,v25
> -
> - and r0,r0,$len
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vcipher $out0,$out0,v26
> - vcipher $out1,$out1,v26
> - vand $tmp,$tmp,$eighty7
> - vcipher $out2,$out2,v26
> - vcipher $out3,$out3,v26
> - vxor $tweak,$tweak,$tmp
> - vcipher $out4,$out4,v26
> - vcipher $out5,$out5,v26
> -
> - add $inp,$inp,r0 # $inp is adjusted in such
> - # way that at exit from the
> - # loop inX-in5 are loaded
> - # with last "words"
> - vxor $in2,$twk2,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk2,$tweak,$rndkey0
> - vaddubm $tweak,$tweak,$tweak
> - vcipher $out0,$out0,v27
> - vcipher $out1,$out1,v27
> - vsldoi $tmp,$tmp,$tmp,15
> - vcipher $out2,$out2,v27
> - vcipher $out3,$out3,v27
> - vand $tmp,$tmp,$eighty7
> - vcipher $out4,$out4,v27
> - vcipher $out5,$out5,v27
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vxor $tweak,$tweak,$tmp
> - vcipher $out0,$out0,v28
> - vcipher $out1,$out1,v28
> - vxor $in3,$twk3,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk3,$tweak,$rndkey0
> - vcipher $out2,$out2,v28
> - vcipher $out3,$out3,v28
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vcipher $out4,$out4,v28
> - vcipher $out5,$out5,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> - vand $tmp,$tmp,$eighty7
> -
> - vcipher $out0,$out0,v29
> - vcipher $out1,$out1,v29
> - vxor $tweak,$tweak,$tmp
> - vcipher $out2,$out2,v29
> - vcipher $out3,$out3,v29
> - vxor $in4,$twk4,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk4,$tweak,$rndkey0
> - vcipher $out4,$out4,v29
> - vcipher $out5,$out5,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> -
> - vcipher $out0,$out0,v30
> - vcipher $out1,$out1,v30
> - vand $tmp,$tmp,$eighty7
> - vcipher $out2,$out2,v30
> - vcipher $out3,$out3,v30
> - vxor $tweak,$tweak,$tmp
> - vcipher $out4,$out4,v30
> - vcipher $out5,$out5,v30
> - vxor $in5,$twk5,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk5,$tweak,$rndkey0
> -
> - vcipherlast $out0,$out0,$in0
> - lvx_u $in0,$x00,$inp # load next input block
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vcipherlast $out1,$out1,$in1
> - lvx_u $in1,$x10,$inp
> - vcipherlast $out2,$out2,$in2
> - le?vperm $in0,$in0,$in0,$leperm
> - lvx_u $in2,$x20,$inp
> - vand $tmp,$tmp,$eighty7
> - vcipherlast $out3,$out3,$in3
> - le?vperm $in1,$in1,$in1,$leperm
> - lvx_u $in3,$x30,$inp
> - vcipherlast $out4,$out4,$in4
> - le?vperm $in2,$in2,$in2,$leperm
> - lvx_u $in4,$x40,$inp
> - vxor $tweak,$tweak,$tmp
> - vcipherlast $tmp,$out5,$in5 # last block might be needed
> - # in stealing mode
> - le?vperm $in3,$in3,$in3,$leperm
> - lvx_u $in5,$x50,$inp
> - addi $inp,$inp,0x60
> - le?vperm $in4,$in4,$in4,$leperm
> - le?vperm $in5,$in5,$in5,$leperm
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $out0,$in0,$twk0
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - vxor $out1,$in1,$twk1
> - le?vperm $out3,$out3,$out3,$leperm
> - stvx_u $out2,$x20,$out
> - vxor $out2,$in2,$twk2
> - le?vperm $out4,$out4,$out4,$leperm
> - stvx_u $out3,$x30,$out
> - vxor $out3,$in3,$twk3
> - le?vperm $out5,$tmp,$tmp,$leperm
> - stvx_u $out4,$x40,$out
> - vxor $out4,$in4,$twk4
> - le?stvx_u $out5,$x50,$out
> - be?stvx_u $tmp, $x50,$out
> - vxor $out5,$in5,$twk5
> - addi $out,$out,0x60
> -
> - mtctr $rounds
> - beq Loop_xts_enc6x # did $len-=96 borrow?
> -
> - addic. $len,$len,0x60
> - beq Lxts_enc6x_zero
> - cmpwi $len,0x20
> - blt Lxts_enc6x_one
> - nop
> - beq Lxts_enc6x_two
> - cmpwi $len,0x40
> - blt Lxts_enc6x_three
> - nop
> - beq Lxts_enc6x_four
> -
> -Lxts_enc6x_five:
> - vxor $out0,$in1,$twk0
> - vxor $out1,$in2,$twk1
> - vxor $out2,$in3,$twk2
> - vxor $out3,$in4,$twk3
> - vxor $out4,$in5,$twk4
> -
> - bl _aesp8_xts_enc5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk5 # unused tweak
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$leperm
> - stvx_u $out2,$x20,$out
> - vxor $tmp,$out4,$twk5 # last block prep for stealing
> - le?vperm $out4,$out4,$out4,$leperm
> - stvx_u $out3,$x30,$out
> - stvx_u $out4,$x40,$out
> - addi $out,$out,0x50
> - bne Lxts_enc6x_steal
> - b Lxts_enc6x_done
> -
> -.align 4
> -Lxts_enc6x_four:
> - vxor $out0,$in2,$twk0
> - vxor $out1,$in3,$twk1
> - vxor $out2,$in4,$twk2
> - vxor $out3,$in5,$twk3
> - vxor $out4,$out4,$out4
> -
> - bl _aesp8_xts_enc5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk4 # unused tweak
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - vxor $tmp,$out3,$twk4 # last block prep for stealing
> - le?vperm $out3,$out3,$out3,$leperm
> - stvx_u $out2,$x20,$out
> - stvx_u $out3,$x30,$out
> - addi $out,$out,0x40
> - bne Lxts_enc6x_steal
> - b Lxts_enc6x_done
> -
> -.align 4
> -Lxts_enc6x_three:
> - vxor $out0,$in3,$twk0
> - vxor $out1,$in4,$twk1
> - vxor $out2,$in5,$twk2
> - vxor $out3,$out3,$out3
> - vxor $out4,$out4,$out4
> -
> - bl _aesp8_xts_enc5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk3 # unused tweak
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $tmp,$out2,$twk3 # last block prep for stealing
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - stvx_u $out2,$x20,$out
> - addi $out,$out,0x30
> - bne Lxts_enc6x_steal
> - b Lxts_enc6x_done
> -
> -.align 4
> -Lxts_enc6x_two:
> - vxor $out0,$in4,$twk0
> - vxor $out1,$in5,$twk1
> - vxor $out2,$out2,$out2
> - vxor $out3,$out3,$out3
> - vxor $out4,$out4,$out4
> -
> - bl _aesp8_xts_enc5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk2 # unused tweak
> - vxor $tmp,$out1,$twk2 # last block prep for stealing
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - stvx_u $out1,$x10,$out
> - addi $out,$out,0x20
> - bne Lxts_enc6x_steal
> - b Lxts_enc6x_done
> -
> -.align 4
> -Lxts_enc6x_one:
> - vxor $out0,$in5,$twk0
> - nop
> -Loop_xts_enc1x:
> - vcipher $out0,$out0,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vcipher $out0,$out0,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_xts_enc1x
> -
> - add $inp,$inp,$taillen
> - cmpwi $taillen,0
> - vcipher $out0,$out0,v24
> -
> - subi $inp,$inp,16
> - vcipher $out0,$out0,v25
> -
> - lvsr $inpperm,0,$taillen
> - vcipher $out0,$out0,v26
> -
> - lvx_u $in0,0,$inp
> - vcipher $out0,$out0,v27
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vcipher $out0,$out0,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> -
> - vcipher $out0,$out0,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vxor $twk0,$twk0,v31
> -
> - le?vperm $in0,$in0,$in0,$leperm
> - vcipher $out0,$out0,v30
> -
> - vperm $in0,$in0,$in0,$inpperm
> - vcipherlast $out0,$out0,$twk0
> -
> - vmr $twk0,$twk1 # unused tweak
> - vxor $tmp,$out0,$twk1 # last block prep for stealing
> - le?vperm $out0,$out0,$out0,$leperm
> - stvx_u $out0,$x00,$out # store output
> - addi $out,$out,0x10
> - bne Lxts_enc6x_steal
> - b Lxts_enc6x_done
> -
> -.align 4
> -Lxts_enc6x_zero:
> - cmpwi $taillen,0
> - beq Lxts_enc6x_done
> -
> - add $inp,$inp,$taillen
> - subi $inp,$inp,16
> - lvx_u $in0,0,$inp
> - lvsr $inpperm,0,$taillen # $in5 is no more
> - le?vperm $in0,$in0,$in0,$leperm
> - vperm $in0,$in0,$in0,$inpperm
> - vxor $tmp,$tmp,$twk0
> -Lxts_enc6x_steal:
> - vxor $in0,$in0,$twk0
> - vxor $out0,$out0,$out0
> - vspltisb $out1,-1
> - vperm $out0,$out0,$out1,$inpperm
> - vsel $out0,$in0,$tmp,$out0 # $tmp is last block, remember?
> -
> - subi r30,$out,17
> - subi $out,$out,16
> - mtctr $taillen
> -Loop_xts_enc6x_steal:
> - lbzu r0,1(r30)
> - stb r0,16(r30)
> - bdnz Loop_xts_enc6x_steal
> -
> - li $taillen,0
> - mtctr $rounds
> - b Loop_xts_enc1x # one more time...
> -
> -.align 4
> -Lxts_enc6x_done:
> - ${UCMP}i $ivp,0
> - beq Lxts_enc6x_ret
> -
> - vxor $tweak,$twk0,$rndkey0
> - le?vperm $tweak,$tweak,$tweak,$leperm
> - stvx_u $tweak,0,$ivp
> -
> -Lxts_enc6x_ret:
> - mtlr r11
> - li r10,`$FRAME+15`
> - li r11,`$FRAME+31`
> - stvx $seven,r10,$sp # wipe copies of round keys
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> - stvx $seven,r10,$sp
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> - stvx $seven,r10,$sp
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> - stvx $seven,r10,$sp
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> -
> - mtspr 256,$vrsave
> - lvx v20,r10,$sp # ABI says so
> - addi r10,r10,32
> - lvx v21,r11,$sp
> - addi r11,r11,32
> - lvx v22,r10,$sp
> - addi r10,r10,32
> - lvx v23,r11,$sp
> - addi r11,r11,32
> - lvx v24,r10,$sp
> - addi r10,r10,32
> - lvx v25,r11,$sp
> - addi r11,r11,32
> - lvx v26,r10,$sp
> - addi r10,r10,32
> - lvx v27,r11,$sp
> - addi r11,r11,32
> - lvx v28,r10,$sp
> - addi r10,r10,32
> - lvx v29,r11,$sp
> - addi r11,r11,32
> - lvx v30,r10,$sp
> - lvx v31,r11,$sp
> - $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
> - blr
> - .long 0
> - .byte 0,12,0x04,1,0x80,6,6,0
> - .long 0
> -
> -.align 5
> -_aesp8_xts_enc5x:
> - vcipher $out0,$out0,v24
> - vcipher $out1,$out1,v24
> - vcipher $out2,$out2,v24
> - vcipher $out3,$out3,v24
> - vcipher $out4,$out4,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vcipher $out0,$out0,v25
> - vcipher $out1,$out1,v25
> - vcipher $out2,$out2,v25
> - vcipher $out3,$out3,v25
> - vcipher $out4,$out4,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz _aesp8_xts_enc5x
> -
> - add $inp,$inp,$taillen
> - cmpwi $taillen,0
> - vcipher $out0,$out0,v24
> - vcipher $out1,$out1,v24
> - vcipher $out2,$out2,v24
> - vcipher $out3,$out3,v24
> - vcipher $out4,$out4,v24
> -
> - subi $inp,$inp,16
> - vcipher $out0,$out0,v25
> - vcipher $out1,$out1,v25
> - vcipher $out2,$out2,v25
> - vcipher $out3,$out3,v25
> - vcipher $out4,$out4,v25
> - vxor $twk0,$twk0,v31
> -
> - vcipher $out0,$out0,v26
> - lvsr $inpperm,r0,$taillen # $in5 is no more
> - vcipher $out1,$out1,v26
> - vcipher $out2,$out2,v26
> - vcipher $out3,$out3,v26
> - vcipher $out4,$out4,v26
> - vxor $in1,$twk1,v31
> -
> - vcipher $out0,$out0,v27
> - lvx_u $in0,0,$inp
> - vcipher $out1,$out1,v27
> - vcipher $out2,$out2,v27
> - vcipher $out3,$out3,v27
> - vcipher $out4,$out4,v27
> - vxor $in2,$twk2,v31
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vcipher $out0,$out0,v28
> - vcipher $out1,$out1,v28
> - vcipher $out2,$out2,v28
> - vcipher $out3,$out3,v28
> - vcipher $out4,$out4,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> - vxor $in3,$twk3,v31
> -
> - vcipher $out0,$out0,v29
> - le?vperm $in0,$in0,$in0,$leperm
> - vcipher $out1,$out1,v29
> - vcipher $out2,$out2,v29
> - vcipher $out3,$out3,v29
> - vcipher $out4,$out4,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vxor $in4,$twk4,v31
> -
> - vcipher $out0,$out0,v30
> - vperm $in0,$in0,$in0,$inpperm
> - vcipher $out1,$out1,v30
> - vcipher $out2,$out2,v30
> - vcipher $out3,$out3,v30
> - vcipher $out4,$out4,v30
> -
> - vcipherlast $out0,$out0,$twk0
> - vcipherlast $out1,$out1,$in1
> - vcipherlast $out2,$out2,$in2
> - vcipherlast $out3,$out3,$in3
> - vcipherlast $out4,$out4,$in4
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,0,0
> -
> -.align 5
> -_aesp8_xts_decrypt6x:
> - $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)
> - mflr r11
> - li r7,`$FRAME+8*16+15`
> - li r3,`$FRAME+8*16+31`
> - $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp)
> - stvx v20,r7,$sp # ABI says so
> - addi r7,r7,32
> - stvx v21,r3,$sp
> - addi r3,r3,32
> - stvx v22,r7,$sp
> - addi r7,r7,32
> - stvx v23,r3,$sp
> - addi r3,r3,32
> - stvx v24,r7,$sp
> - addi r7,r7,32
> - stvx v25,r3,$sp
> - addi r3,r3,32
> - stvx v26,r7,$sp
> - addi r7,r7,32
> - stvx v27,r3,$sp
> - addi r3,r3,32
> - stvx v28,r7,$sp
> - addi r7,r7,32
> - stvx v29,r3,$sp
> - addi r3,r3,32
> - stvx v30,r7,$sp
> - stvx v31,r3,$sp
> - li r0,-1
> - stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave
> - li $x10,0x10
> - $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - li $x20,0x20
> - $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - li $x30,0x30
> - $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - li $x40,0x40
> - $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - li $x50,0x50
> - $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - li $x60,0x60
> - $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - li $x70,0x70
> - mtspr 256,r0
> -
> - subi $rounds,$rounds,3 # -4 in total
> -
> - lvx $rndkey0,$x00,$key1 # load key schedule
> - lvx v30,$x10,$key1
> - addi $key1,$key1,0x20
> - lvx v31,$x00,$key1
> - ?vperm $rndkey0,$rndkey0,v30,$keyperm
> - addi $key_,$sp,$FRAME+15
> - mtctr $rounds
> -
> -Load_xts_dec_key:
> - ?vperm v24,v30,v31,$keyperm
> - lvx v30,$x10,$key1
> - addi $key1,$key1,0x20
> - stvx v24,$x00,$key_ # off-load round[1]
> - ?vperm v25,v31,v30,$keyperm
> - lvx v31,$x00,$key1
> - stvx v25,$x10,$key_ # off-load round[2]
> - addi $key_,$key_,0x20
> - bdnz Load_xts_dec_key
> -
> - lvx v26,$x10,$key1
> - ?vperm v24,v30,v31,$keyperm
> - lvx v27,$x20,$key1
> - stvx v24,$x00,$key_ # off-load round[3]
> - ?vperm v25,v31,v26,$keyperm
> - lvx v28,$x30,$key1
> - stvx v25,$x10,$key_ # off-load round[4]
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - ?vperm v26,v26,v27,$keyperm
> - lvx v29,$x40,$key1
> - ?vperm v27,v27,v28,$keyperm
> - lvx v30,$x50,$key1
> - ?vperm v28,v28,v29,$keyperm
> - lvx v31,$x60,$key1
> - ?vperm v29,v29,v30,$keyperm
> - lvx $twk5,$x70,$key1 # borrow $twk5
> - ?vperm v30,v30,v31,$keyperm
> - lvx v24,$x00,$key_ # pre-load round[1]
> - ?vperm v31,v31,$twk5,$keyperm
> - lvx v25,$x10,$key_ # pre-load round[2]
> -
> - vperm $in0,$inout,$inptail,$inpperm
> - subi $inp,$inp,31 # undo "caller"
> - vxor $twk0,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vand $tmp,$tmp,$eighty7
> - vxor $out0,$in0,$twk0
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in1,$x10,$inp
> - vxor $twk1,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in1,$in1,$in1,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out1,$in1,$twk1
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in2,$x20,$inp
> - andi. $taillen,$len,15
> - vxor $twk2,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in2,$in2,$in2,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out2,$in2,$twk2
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in3,$x30,$inp
> - sub $len,$len,$taillen
> - vxor $twk3,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in3,$in3,$in3,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out3,$in3,$twk3
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in4,$x40,$inp
> - subi $len,$len,0x60
> - vxor $twk4,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in4,$in4,$in4,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out4,$in4,$twk4
> - vxor $tweak,$tweak,$tmp
> -
> - lvx_u $in5,$x50,$inp
> - addi $inp,$inp,0x60
> - vxor $twk5,$tweak,$rndkey0
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - le?vperm $in5,$in5,$in5,$leperm
> - vand $tmp,$tmp,$eighty7
> - vxor $out5,$in5,$twk5
> - vxor $tweak,$tweak,$tmp
> -
> - vxor v31,v31,$rndkey0
> - mtctr $rounds
> - b Loop_xts_dec6x
> -
> -.align 5
> -Loop_xts_dec6x:
> - vncipher $out0,$out0,v24
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> - vncipher $out5,$out5,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vncipher $out0,$out0,v25
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - vncipher $out5,$out5,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_xts_dec6x
> -
> - subic $len,$len,96 # $len-=96
> - vxor $in0,$twk0,v31 # xor with last round key
> - vncipher $out0,$out0,v24
> - vncipher $out1,$out1,v24
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk0,$tweak,$rndkey0
> - vaddubm $tweak,$tweak,$tweak
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vsldoi $tmp,$tmp,$tmp,15
> - vncipher $out4,$out4,v24
> - vncipher $out5,$out5,v24
> -
> - subfe. r0,r0,r0 # borrow?-1:0
> - vand $tmp,$tmp,$eighty7
> - vncipher $out0,$out0,v25
> - vncipher $out1,$out1,v25
> - vxor $tweak,$tweak,$tmp
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vxor $in1,$twk1,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk1,$tweak,$rndkey0
> - vncipher $out4,$out4,v25
> - vncipher $out5,$out5,v25
> -
> - and r0,r0,$len
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vncipher $out0,$out0,v26
> - vncipher $out1,$out1,v26
> - vand $tmp,$tmp,$eighty7
> - vncipher $out2,$out2,v26
> - vncipher $out3,$out3,v26
> - vxor $tweak,$tweak,$tmp
> - vncipher $out4,$out4,v26
> - vncipher $out5,$out5,v26
> -
> - add $inp,$inp,r0 # $inp is adjusted in such
> - # way that at exit from the
> - # loop inX-in5 are loaded
> - # with last "words"
> - vxor $in2,$twk2,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk2,$tweak,$rndkey0
> - vaddubm $tweak,$tweak,$tweak
> - vncipher $out0,$out0,v27
> - vncipher $out1,$out1,v27
> - vsldoi $tmp,$tmp,$tmp,15
> - vncipher $out2,$out2,v27
> - vncipher $out3,$out3,v27
> - vand $tmp,$tmp,$eighty7
> - vncipher $out4,$out4,v27
> - vncipher $out5,$out5,v27
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vxor $tweak,$tweak,$tmp
> - vncipher $out0,$out0,v28
> - vncipher $out1,$out1,v28
> - vxor $in3,$twk3,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk3,$tweak,$rndkey0
> - vncipher $out2,$out2,v28
> - vncipher $out3,$out3,v28
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vncipher $out4,$out4,v28
> - vncipher $out5,$out5,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> - vand $tmp,$tmp,$eighty7
> -
> - vncipher $out0,$out0,v29
> - vncipher $out1,$out1,v29
> - vxor $tweak,$tweak,$tmp
> - vncipher $out2,$out2,v29
> - vncipher $out3,$out3,v29
> - vxor $in4,$twk4,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk4,$tweak,$rndkey0
> - vncipher $out4,$out4,v29
> - vncipher $out5,$out5,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> -
> - vncipher $out0,$out0,v30
> - vncipher $out1,$out1,v30
> - vand $tmp,$tmp,$eighty7
> - vncipher $out2,$out2,v30
> - vncipher $out3,$out3,v30
> - vxor $tweak,$tweak,$tmp
> - vncipher $out4,$out4,v30
> - vncipher $out5,$out5,v30
> - vxor $in5,$twk5,v31
> - vsrab $tmp,$tweak,$seven # next tweak value
> - vxor $twk5,$tweak,$rndkey0
> -
> - vncipherlast $out0,$out0,$in0
> - lvx_u $in0,$x00,$inp # load next input block
> - vaddubm $tweak,$tweak,$tweak
> - vsldoi $tmp,$tmp,$tmp,15
> - vncipherlast $out1,$out1,$in1
> - lvx_u $in1,$x10,$inp
> - vncipherlast $out2,$out2,$in2
> - le?vperm $in0,$in0,$in0,$leperm
> - lvx_u $in2,$x20,$inp
> - vand $tmp,$tmp,$eighty7
> - vncipherlast $out3,$out3,$in3
> - le?vperm $in1,$in1,$in1,$leperm
> - lvx_u $in3,$x30,$inp
> - vncipherlast $out4,$out4,$in4
> - le?vperm $in2,$in2,$in2,$leperm
> - lvx_u $in4,$x40,$inp
> - vxor $tweak,$tweak,$tmp
> - vncipherlast $out5,$out5,$in5
> - le?vperm $in3,$in3,$in3,$leperm
> - lvx_u $in5,$x50,$inp
> - addi $inp,$inp,0x60
> - le?vperm $in4,$in4,$in4,$leperm
> - le?vperm $in5,$in5,$in5,$leperm
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $out0,$in0,$twk0
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - vxor $out1,$in1,$twk1
> - le?vperm $out3,$out3,$out3,$leperm
> - stvx_u $out2,$x20,$out
> - vxor $out2,$in2,$twk2
> - le?vperm $out4,$out4,$out4,$leperm
> - stvx_u $out3,$x30,$out
> - vxor $out3,$in3,$twk3
> - le?vperm $out5,$out5,$out5,$leperm
> - stvx_u $out4,$x40,$out
> - vxor $out4,$in4,$twk4
> - stvx_u $out5,$x50,$out
> - vxor $out5,$in5,$twk5
> - addi $out,$out,0x60
> -
> - mtctr $rounds
> - beq Loop_xts_dec6x # did $len-=96 borrow?
> -
> - addic. $len,$len,0x60
> - beq Lxts_dec6x_zero
> - cmpwi $len,0x20
> - blt Lxts_dec6x_one
> - nop
> - beq Lxts_dec6x_two
> - cmpwi $len,0x40
> - blt Lxts_dec6x_three
> - nop
> - beq Lxts_dec6x_four
> -
> -Lxts_dec6x_five:
> - vxor $out0,$in1,$twk0
> - vxor $out1,$in2,$twk1
> - vxor $out2,$in3,$twk2
> - vxor $out3,$in4,$twk3
> - vxor $out4,$in5,$twk4
> -
> - bl _aesp8_xts_dec5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk5 # unused tweak
> - vxor $twk1,$tweak,$rndkey0
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $out0,$in0,$twk1
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$leperm
> - stvx_u $out2,$x20,$out
> - le?vperm $out4,$out4,$out4,$leperm
> - stvx_u $out3,$x30,$out
> - stvx_u $out4,$x40,$out
> - addi $out,$out,0x50
> - bne Lxts_dec6x_steal
> - b Lxts_dec6x_done
> -
> -.align 4
> -Lxts_dec6x_four:
> - vxor $out0,$in2,$twk0
> - vxor $out1,$in3,$twk1
> - vxor $out2,$in4,$twk2
> - vxor $out3,$in5,$twk3
> - vxor $out4,$out4,$out4
> -
> - bl _aesp8_xts_dec5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk4 # unused tweak
> - vmr $twk1,$twk5
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $out0,$in0,$twk5
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - le?vperm $out3,$out3,$out3,$leperm
> - stvx_u $out2,$x20,$out
> - stvx_u $out3,$x30,$out
> - addi $out,$out,0x40
> - bne Lxts_dec6x_steal
> - b Lxts_dec6x_done
> -
> -.align 4
> -Lxts_dec6x_three:
> - vxor $out0,$in3,$twk0
> - vxor $out1,$in4,$twk1
> - vxor $out2,$in5,$twk2
> - vxor $out3,$out3,$out3
> - vxor $out4,$out4,$out4
> -
> - bl _aesp8_xts_dec5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk3 # unused tweak
> - vmr $twk1,$twk4
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $out0,$in0,$twk4
> - le?vperm $out2,$out2,$out2,$leperm
> - stvx_u $out1,$x10,$out
> - stvx_u $out2,$x20,$out
> - addi $out,$out,0x30
> - bne Lxts_dec6x_steal
> - b Lxts_dec6x_done
> -
> -.align 4
> -Lxts_dec6x_two:
> - vxor $out0,$in4,$twk0
> - vxor $out1,$in5,$twk1
> - vxor $out2,$out2,$out2
> - vxor $out3,$out3,$out3
> - vxor $out4,$out4,$out4
> -
> - bl _aesp8_xts_dec5x
> -
> - le?vperm $out0,$out0,$out0,$leperm
> - vmr $twk0,$twk2 # unused tweak
> - vmr $twk1,$twk3
> - le?vperm $out1,$out1,$out1,$leperm
> - stvx_u $out0,$x00,$out # store output
> - vxor $out0,$in0,$twk3
> - stvx_u $out1,$x10,$out
> - addi $out,$out,0x20
> - bne Lxts_dec6x_steal
> - b Lxts_dec6x_done
> -
> -.align 4
> -Lxts_dec6x_one:
> - vxor $out0,$in5,$twk0
> - nop
> -Loop_xts_dec1x:
> - vncipher $out0,$out0,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vncipher $out0,$out0,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Loop_xts_dec1x
> -
> - subi r0,$taillen,1
> - vncipher $out0,$out0,v24
> -
> - andi. r0,r0,16
> - cmpwi $taillen,0
> - vncipher $out0,$out0,v25
> -
> - sub $inp,$inp,r0
> - vncipher $out0,$out0,v26
> -
> - lvx_u $in0,0,$inp
> - vncipher $out0,$out0,v27
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vncipher $out0,$out0,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> -
> - vncipher $out0,$out0,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vxor $twk0,$twk0,v31
> -
> - le?vperm $in0,$in0,$in0,$leperm
> - vncipher $out0,$out0,v30
> -
> - mtctr $rounds
> - vncipherlast $out0,$out0,$twk0
> -
> - vmr $twk0,$twk1 # unused tweak
> - vmr $twk1,$twk2
> - le?vperm $out0,$out0,$out0,$leperm
> - stvx_u $out0,$x00,$out # store output
> - addi $out,$out,0x10
> - vxor $out0,$in0,$twk2
> - bne Lxts_dec6x_steal
> - b Lxts_dec6x_done
> -
> -.align 4
> -Lxts_dec6x_zero:
> - cmpwi $taillen,0
> - beq Lxts_dec6x_done
> -
> - lvx_u $in0,0,$inp
> - le?vperm $in0,$in0,$in0,$leperm
> - vxor $out0,$in0,$twk1
> -Lxts_dec6x_steal:
> - vncipher $out0,$out0,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vncipher $out0,$out0,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz Lxts_dec6x_steal
> -
> - add $inp,$inp,$taillen
> - vncipher $out0,$out0,v24
> -
> - cmpwi $taillen,0
> - vncipher $out0,$out0,v25
> -
> - lvx_u $in0,0,$inp
> - vncipher $out0,$out0,v26
> -
> - lvsr $inpperm,0,$taillen # $in5 is no more
> - vncipher $out0,$out0,v27
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vncipher $out0,$out0,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> -
> - vncipher $out0,$out0,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vxor $twk1,$twk1,v31
> -
> - le?vperm $in0,$in0,$in0,$leperm
> - vncipher $out0,$out0,v30
> -
> - vperm $in0,$in0,$in0,$inpperm
> - vncipherlast $tmp,$out0,$twk1
> -
> - le?vperm $out0,$tmp,$tmp,$leperm
> - le?stvx_u $out0,0,$out
> - be?stvx_u $tmp,0,$out
> -
> - vxor $out0,$out0,$out0
> - vspltisb $out1,-1
> - vperm $out0,$out0,$out1,$inpperm
> - vsel $out0,$in0,$tmp,$out0
> - vxor $out0,$out0,$twk0
> -
> - subi r30,$out,1
> - mtctr $taillen
> -Loop_xts_dec6x_steal:
> - lbzu r0,1(r30)
> - stb r0,16(r30)
> - bdnz Loop_xts_dec6x_steal
> -
> - li $taillen,0
> - mtctr $rounds
> - b Loop_xts_dec1x # one more time...
> -
> -.align 4
> -Lxts_dec6x_done:
> - ${UCMP}i $ivp,0
> - beq Lxts_dec6x_ret
> -
> - vxor $tweak,$twk0,$rndkey0
> - le?vperm $tweak,$tweak,$tweak,$leperm
> - stvx_u $tweak,0,$ivp
> -
> -Lxts_dec6x_ret:
> - mtlr r11
> - li r10,`$FRAME+15`
> - li r11,`$FRAME+31`
> - stvx $seven,r10,$sp # wipe copies of round keys
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> - stvx $seven,r10,$sp
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> - stvx $seven,r10,$sp
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> - stvx $seven,r10,$sp
> - addi r10,r10,32
> - stvx $seven,r11,$sp
> - addi r11,r11,32
> -
> - mtspr 256,$vrsave
> - lvx v20,r10,$sp # ABI says so
> - addi r10,r10,32
> - lvx v21,r11,$sp
> - addi r11,r11,32
> - lvx v22,r10,$sp
> - addi r10,r10,32
> - lvx v23,r11,$sp
> - addi r11,r11,32
> - lvx v24,r10,$sp
> - addi r10,r10,32
> - lvx v25,r11,$sp
> - addi r11,r11,32
> - lvx v26,r10,$sp
> - addi r10,r10,32
> - lvx v27,r11,$sp
> - addi r11,r11,32
> - lvx v28,r10,$sp
> - addi r10,r10,32
> - lvx v29,r11,$sp
> - addi r11,r11,32
> - lvx v30,r10,$sp
> - lvx v31,r11,$sp
> - $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp)
> - $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp)
> - $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp)
> - $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp)
> - $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp)
> - $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp)
> - addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T`
> - blr
> - .long 0
> - .byte 0,12,0x04,1,0x80,6,6,0
> - .long 0
> -
> -.align 5
> -_aesp8_xts_dec5x:
> - vncipher $out0,$out0,v24
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> - lvx v24,$x20,$key_ # round[3]
> - addi $key_,$key_,0x20
> -
> - vncipher $out0,$out0,v25
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - lvx v25,$x10,$key_ # round[4]
> - bdnz _aesp8_xts_dec5x
> -
> - subi r0,$taillen,1
> - vncipher $out0,$out0,v24
> - vncipher $out1,$out1,v24
> - vncipher $out2,$out2,v24
> - vncipher $out3,$out3,v24
> - vncipher $out4,$out4,v24
> -
> - andi. r0,r0,16
> - cmpwi $taillen,0
> - vncipher $out0,$out0,v25
> - vncipher $out1,$out1,v25
> - vncipher $out2,$out2,v25
> - vncipher $out3,$out3,v25
> - vncipher $out4,$out4,v25
> - vxor $twk0,$twk0,v31
> -
> - sub $inp,$inp,r0
> - vncipher $out0,$out0,v26
> - vncipher $out1,$out1,v26
> - vncipher $out2,$out2,v26
> - vncipher $out3,$out3,v26
> - vncipher $out4,$out4,v26
> - vxor $in1,$twk1,v31
> -
> - vncipher $out0,$out0,v27
> - lvx_u $in0,0,$inp
> - vncipher $out1,$out1,v27
> - vncipher $out2,$out2,v27
> - vncipher $out3,$out3,v27
> - vncipher $out4,$out4,v27
> - vxor $in2,$twk2,v31
> -
> - addi $key_,$sp,$FRAME+15 # rewind $key_
> - vncipher $out0,$out0,v28
> - vncipher $out1,$out1,v28
> - vncipher $out2,$out2,v28
> - vncipher $out3,$out3,v28
> - vncipher $out4,$out4,v28
> - lvx v24,$x00,$key_ # re-pre-load round[1]
> - vxor $in3,$twk3,v31
> -
> - vncipher $out0,$out0,v29
> - le?vperm $in0,$in0,$in0,$leperm
> - vncipher $out1,$out1,v29
> - vncipher $out2,$out2,v29
> - vncipher $out3,$out3,v29
> - vncipher $out4,$out4,v29
> - lvx v25,$x10,$key_ # re-pre-load round[2]
> - vxor $in4,$twk4,v31
> -
> - vncipher $out0,$out0,v30
> - vncipher $out1,$out1,v30
> - vncipher $out2,$out2,v30
> - vncipher $out3,$out3,v30
> - vncipher $out4,$out4,v30
> -
> - vncipherlast $out0,$out0,$twk0
> - vncipherlast $out1,$out1,$in1
> - vncipherlast $out2,$out2,$in2
> - vncipherlast $out3,$out3,$in3
> - vncipherlast $out4,$out4,$in4
> - mtctr $rounds
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,0,0
> -___
> -}} }}}
> -
> -my $consts=1;
> -foreach(split("\n",$code)) {
> - s/\`([^\`]*)\`/eval($1)/geo;
> -
> - # constants table endian-specific conversion
> - if ($consts && m/\.(long|byte)\s+(.+)\s+(\?[a-z]*)$/o) {
> - my $conv=$3;
> - my @bytes=();
> -
> - # convert to endian-agnostic format
> - if ($1 eq "long") {
> - foreach (split(/,\s*/,$2)) {
> - my $l = /^0/?oct:int;
> - push @bytes,($l>>24)&0xff,($l>>16)&0xff,($l>>8)&0xff,$l&0xff;
> - }
> - } else {
> - @bytes = map(/^0/?oct:int,split(/,\s*/,$2));
> - }
> -
> - # little-endian conversion
> - if ($flavour =~ /le$/o) {
> - SWITCH: for($conv) {
> - /\?inv/ && do { @bytes=map($_^0xf,@bytes); last; };
> - /\?rev/ && do { @bytes=reverse(@bytes); last; };
> - }
> - }
> -
> - #emit
> - print ".byte\t",join(',',map (sprintf("0x%02x",$_),@bytes)),"\n";
> - next;
> - }
> - $consts=0 if (m/Lconsts:/o); # end of table
> -
> - # instructions prefixed with '?' are endian-specific and need
> - # to be adjusted accordingly...
> - if ($flavour =~ /le$/o) { # little-endian
> - s/le\?//o or
> - s/be\?/#be#/o or
> - s/\?lvsr/lvsl/o or
> - s/\?lvsl/lvsr/o or
> - s/\?(vperm\s+v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+)/$1$3$2$4/o or
> - s/\?(vsldoi\s+v[0-9]+,\s*)(v[0-9]+,)\s*(v[0-9]+,\s*)([0-9]+)/$1$3$2 16-$4/o or
> - s/\?(vspltw\s+v[0-9]+,\s*)(v[0-9]+,)\s*([0-9])/$1$2 3-$3/o;
> - } else { # big-endian
> - s/le\?/#le#/o or
> - s/be\?//o or
> - s/\?([a-z]+)/$1/o;
> - }
> -
> - print $_,"\n";
> -}
> -
> -close STDOUT;
> diff --git a/drivers/crypto/vmx/ghash.c b/drivers/crypto/vmx/ghash.c
> deleted file mode 100644
> index 27a94a119009..000000000000
> --- a/drivers/crypto/vmx/ghash.c
> +++ /dev/null
> @@ -1,227 +0,0 @@
> -/**
> - * GHASH routines supporting VMX instructions on the Power 8
> - *
> - * Copyright (C) 2015 International Business Machines Inc.
> - *
> - * This program is free software; you can redistribute it and/or modify
> - * it under the terms of the GNU General Public License as published by
> - * the Free Software Foundation; version 2 only.
> - *
> - * This program is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - * GNU General Public License for more details.
> - *
> - * You should have received a copy of the GNU General Public License
> - * along with this program; if not, write to the Free Software
> - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> - *
> - * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
> - */
> -
> -#include <linux/types.h>
> -#include <linux/err.h>
> -#include <linux/crypto.h>
> -#include <linux/delay.h>
> -#include <linux/hardirq.h>
> -#include <asm/switch_to.h>
> -#include <crypto/aes.h>
> -#include <crypto/ghash.h>
> -#include <crypto/scatterwalk.h>
> -#include <crypto/internal/hash.h>
> -#include <crypto/b128ops.h>
> -
> -#define IN_INTERRUPT in_interrupt()
> -
> -void gcm_init_p8(u128 htable[16], const u64 Xi[2]);
> -void gcm_gmult_p8(u64 Xi[2], const u128 htable[16]);
> -void gcm_ghash_p8(u64 Xi[2], const u128 htable[16],
> - const u8 *in, size_t len);
> -
> -struct p8_ghash_ctx {
> - u128 htable[16];
> - struct crypto_shash *fallback;
> -};
> -
> -struct p8_ghash_desc_ctx {
> - u64 shash[2];
> - u8 buffer[GHASH_DIGEST_SIZE];
> - int bytes;
> - struct shash_desc fallback_desc;
> -};
> -
> -static int p8_ghash_init_tfm(struct crypto_tfm *tfm)
> -{
> - const char *alg = "ghash-generic";
> - struct crypto_shash *fallback;
> - struct crypto_shash *shash_tfm = __crypto_shash_cast(tfm);
> - struct p8_ghash_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - fallback = crypto_alloc_shash(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
> - if (IS_ERR(fallback)) {
> - printk(KERN_ERR
> - "Failed to allocate transformation for '%s': %ld\n",
> - alg, PTR_ERR(fallback));
> - return PTR_ERR(fallback);
> - }
> - printk(KERN_INFO "Using '%s' as fallback implementation.\n",
> - crypto_tfm_alg_driver_name(crypto_shash_tfm(fallback)));
> -
> - crypto_shash_set_flags(fallback,
> - crypto_shash_get_flags((struct crypto_shash
> - *) tfm));
> -
> - /* Check if the descsize defined in the algorithm is still enough. */
> - if (shash_tfm->descsize < sizeof(struct p8_ghash_desc_ctx)
> - + crypto_shash_descsize(fallback)) {
> - printk(KERN_ERR
> - "Desc size of the fallback implementation (%s) does not match the expected value: %lu vs %u\n",
> - alg,
> - shash_tfm->descsize - sizeof(struct p8_ghash_desc_ctx),
> - crypto_shash_descsize(fallback));
> - return -EINVAL;
> - }
> - ctx->fallback = fallback;
> -
> - return 0;
> -}
> -
> -static void p8_ghash_exit_tfm(struct crypto_tfm *tfm)
> -{
> - struct p8_ghash_ctx *ctx = crypto_tfm_ctx(tfm);
> -
> - if (ctx->fallback) {
> - crypto_free_shash(ctx->fallback);
> - ctx->fallback = NULL;
> - }
> -}
> -
> -static int p8_ghash_init(struct shash_desc *desc)
> -{
> - struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
> - struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
> -
> - dctx->bytes = 0;
> - memset(dctx->shash, 0, GHASH_DIGEST_SIZE);
> - dctx->fallback_desc.tfm = ctx->fallback;
> - dctx->fallback_desc.flags = desc->flags;
> - return crypto_shash_init(&dctx->fallback_desc);
> -}
> -
> -static int p8_ghash_setkey(struct crypto_shash *tfm, const u8 *key,
> - unsigned int keylen)
> -{
> - struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(tfm));
> -
> - if (keylen != GHASH_BLOCK_SIZE)
> - return -EINVAL;
> -
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - gcm_init_p8(ctx->htable, (const u64 *) key);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - return crypto_shash_setkey(ctx->fallback, key, keylen);
> -}
> -
> -static int p8_ghash_update(struct shash_desc *desc,
> - const u8 *src, unsigned int srclen)
> -{
> - unsigned int len;
> - struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
> - struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
> -
> - if (IN_INTERRUPT) {
> - return crypto_shash_update(&dctx->fallback_desc, src,
> - srclen);
> - } else {
> - if (dctx->bytes) {
> - if (dctx->bytes + srclen < GHASH_DIGEST_SIZE) {
> - memcpy(dctx->buffer + dctx->bytes, src,
> - srclen);
> - dctx->bytes += srclen;
> - return 0;
> - }
> - memcpy(dctx->buffer + dctx->bytes, src,
> - GHASH_DIGEST_SIZE - dctx->bytes);
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - gcm_ghash_p8(dctx->shash, ctx->htable,
> - dctx->buffer, GHASH_DIGEST_SIZE);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - src += GHASH_DIGEST_SIZE - dctx->bytes;
> - srclen -= GHASH_DIGEST_SIZE - dctx->bytes;
> - dctx->bytes = 0;
> - }
> - len = srclen & ~(GHASH_DIGEST_SIZE - 1);
> - if (len) {
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - gcm_ghash_p8(dctx->shash, ctx->htable, src, len);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - src += len;
> - srclen -= len;
> - }
> - if (srclen) {
> - memcpy(dctx->buffer, src, srclen);
> - dctx->bytes = srclen;
> - }
> - return 0;
> - }
> -}
> -
> -static int p8_ghash_final(struct shash_desc *desc, u8 *out)
> -{
> - int i;
> - struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
> - struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
> -
> - if (IN_INTERRUPT) {
> - return crypto_shash_final(&dctx->fallback_desc, out);
> - } else {
> - if (dctx->bytes) {
> - for (i = dctx->bytes; i < GHASH_DIGEST_SIZE; i++)
> - dctx->buffer[i] = 0;
> - preempt_disable();
> - pagefault_disable();
> - enable_kernel_vsx();
> - gcm_ghash_p8(dctx->shash, ctx->htable,
> - dctx->buffer, GHASH_DIGEST_SIZE);
> - disable_kernel_vsx();
> - pagefault_enable();
> - preempt_enable();
> - dctx->bytes = 0;
> - }
> - memcpy(out, dctx->shash, GHASH_DIGEST_SIZE);
> - return 0;
> - }
> -}
> -
> -struct shash_alg p8_ghash_alg = {
> - .digestsize = GHASH_DIGEST_SIZE,
> - .init = p8_ghash_init,
> - .update = p8_ghash_update,
> - .final = p8_ghash_final,
> - .setkey = p8_ghash_setkey,
> - .descsize = sizeof(struct p8_ghash_desc_ctx)
> - + sizeof(struct ghash_desc_ctx),
> - .base = {
> - .cra_name = "ghash",
> - .cra_driver_name = "p8_ghash",
> - .cra_priority = 1000,
> - .cra_flags = CRYPTO_ALG_TYPE_SHASH | CRYPTO_ALG_NEED_FALLBACK,
> - .cra_blocksize = GHASH_BLOCK_SIZE,
> - .cra_ctxsize = sizeof(struct p8_ghash_ctx),
> - .cra_module = THIS_MODULE,
> - .cra_init = p8_ghash_init_tfm,
> - .cra_exit = p8_ghash_exit_tfm,
> - },
> -};
> diff --git a/drivers/crypto/vmx/ghashp8-ppc.pl b/drivers/crypto/vmx/ghashp8-ppc.pl
> deleted file mode 100644
> index d8429cb71f02..000000000000
> --- a/drivers/crypto/vmx/ghashp8-ppc.pl
> +++ /dev/null
> @@ -1,234 +0,0 @@
> -#!/usr/bin/env perl
> -#
> -# ====================================================================
> -# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
> -# project. The module is, however, dual licensed under OpenSSL and
> -# CRYPTOGAMS licenses depending on where you obtain it. For further
> -# details see http://www.openssl.org/~appro/cryptogams/.
> -# ====================================================================
> -#
> -# GHASH for for PowerISA v2.07.
> -#
> -# July 2014
> -#
> -# Accurate performance measurements are problematic, because it's
> -# always virtualized setup with possibly throttled processor.
> -# Relative comparison is therefore more informative. This initial
> -# version is ~2.1x slower than hardware-assisted AES-128-CTR, ~12x
> -# faster than "4-bit" integer-only compiler-generated 64-bit code.
> -# "Initial version" means that there is room for futher improvement.
> -
> -$flavour=shift;
> -$output =shift;
> -
> -if ($flavour =~ /64/) {
> - $SIZE_T=8;
> - $LRSAVE=2*$SIZE_T;
> - $STU="stdu";
> - $POP="ld";
> - $PUSH="std";
> -} elsif ($flavour =~ /32/) {
> - $SIZE_T=4;
> - $LRSAVE=$SIZE_T;
> - $STU="stwu";
> - $POP="lwz";
> - $PUSH="stw";
> -} else { die "nonsense $flavour"; }
> -
> -$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
> -( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or
> -( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or
> -die "can't locate ppc-xlate.pl";
> -
> -open STDOUT,"| $^X $xlate $flavour $output" || die "can't call $xlate: $!";
> -
> -my ($Xip,$Htbl,$inp,$len)=map("r$_",(3..6)); # argument block
> -
> -my ($Xl,$Xm,$Xh,$IN)=map("v$_",(0..3));
> -my ($zero,$t0,$t1,$t2,$xC2,$H,$Hh,$Hl,$lemask)=map("v$_",(4..12));
> -my $vrsave="r12";
> -
> -$code=<<___;
> -.machine "any"
> -
> -.text
> -
> -.globl .gcm_init_p8
> - lis r0,0xfff0
> - li r8,0x10
> - mfspr $vrsave,256
> - li r9,0x20
> - mtspr 256,r0
> - li r10,0x30
> - lvx_u $H,0,r4 # load H
> - le?xor r7,r7,r7
> - le?addi r7,r7,0x8 # need a vperm start with 08
> - le?lvsr 5,0,r7
> - le?vspltisb 6,0x0f
> - le?vxor 5,5,6 # set a b-endian mask
> - le?vperm $H,$H,$H,5
> -
> - vspltisb $xC2,-16 # 0xf0
> - vspltisb $t0,1 # one
> - vaddubm $xC2,$xC2,$xC2 # 0xe0
> - vxor $zero,$zero,$zero
> - vor $xC2,$xC2,$t0 # 0xe1
> - vsldoi $xC2,$xC2,$zero,15 # 0xe1...
> - vsldoi $t1,$zero,$t0,1 # ...1
> - vaddubm $xC2,$xC2,$xC2 # 0xc2...
> - vspltisb $t2,7
> - vor $xC2,$xC2,$t1 # 0xc2....01
> - vspltb $t1,$H,0 # most significant byte
> - vsl $H,$H,$t0 # H<<=1
> - vsrab $t1,$t1,$t2 # broadcast carry bit
> - vand $t1,$t1,$xC2
> - vxor $H,$H,$t1 # twisted H
> -
> - vsldoi $H,$H,$H,8 # twist even more ...
> - vsldoi $xC2,$zero,$xC2,8 # 0xc2.0
> - vsldoi $Hl,$zero,$H,8 # ... and split
> - vsldoi $Hh,$H,$zero,8
> -
> - stvx_u $xC2,0,r3 # save pre-computed table
> - stvx_u $Hl,r8,r3
> - stvx_u $H, r9,r3
> - stvx_u $Hh,r10,r3
> -
> - mtspr 256,$vrsave
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,2,0
> - .long 0
> -.size .gcm_init_p8,.-.gcm_init_p8
> -
> -.globl .gcm_gmult_p8
> - lis r0,0xfff8
> - li r8,0x10
> - mfspr $vrsave,256
> - li r9,0x20
> - mtspr 256,r0
> - li r10,0x30
> - lvx_u $IN,0,$Xip # load Xi
> -
> - lvx_u $Hl,r8,$Htbl # load pre-computed table
> - le?lvsl $lemask,r0,r0
> - lvx_u $H, r9,$Htbl
> - le?vspltisb $t0,0x07
> - lvx_u $Hh,r10,$Htbl
> - le?vxor $lemask,$lemask,$t0
> - lvx_u $xC2,0,$Htbl
> - le?vperm $IN,$IN,$IN,$lemask
> - vxor $zero,$zero,$zero
> -
> - vpmsumd $Xl,$IN,$Hl # H.lo�Xi.lo
> - vpmsumd $Xm,$IN,$H # H.hi�Xi.lo+H.lo�Xi.hi
> - vpmsumd $Xh,$IN,$Hh # H.hi�Xi.hi
> -
> - vpmsumd $t2,$Xl,$xC2 # 1st phase
> -
> - vsldoi $t0,$Xm,$zero,8
> - vsldoi $t1,$zero,$Xm,8
> - vxor $Xl,$Xl,$t0
> - vxor $Xh,$Xh,$t1
> -
> - vsldoi $Xl,$Xl,$Xl,8
> - vxor $Xl,$Xl,$t2
> -
> - vsldoi $t1,$Xl,$Xl,8 # 2nd phase
> - vpmsumd $Xl,$Xl,$xC2
> - vxor $t1,$t1,$Xh
> - vxor $Xl,$Xl,$t1
> -
> - le?vperm $Xl,$Xl,$Xl,$lemask
> - stvx_u $Xl,0,$Xip # write out Xi
> -
> - mtspr 256,$vrsave
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,2,0
> - .long 0
> -.size .gcm_gmult_p8,.-.gcm_gmult_p8
> -
> -.globl .gcm_ghash_p8
> - lis r0,0xfff8
> - li r8,0x10
> - mfspr $vrsave,256
> - li r9,0x20
> - mtspr 256,r0
> - li r10,0x30
> - lvx_u $Xl,0,$Xip # load Xi
> -
> - lvx_u $Hl,r8,$Htbl # load pre-computed table
> - le?lvsl $lemask,r0,r0
> - lvx_u $H, r9,$Htbl
> - le?vspltisb $t0,0x07
> - lvx_u $Hh,r10,$Htbl
> - le?vxor $lemask,$lemask,$t0
> - lvx_u $xC2,0,$Htbl
> - le?vperm $Xl,$Xl,$Xl,$lemask
> - vxor $zero,$zero,$zero
> -
> - lvx_u $IN,0,$inp
> - addi $inp,$inp,16
> - subi $len,$len,16
> - le?vperm $IN,$IN,$IN,$lemask
> - vxor $IN,$IN,$Xl
> - b Loop
> -
> -.align 5
> -Loop:
> - subic $len,$len,16
> - vpmsumd $Xl,$IN,$Hl # H.lo�Xi.lo
> - subfe. r0,r0,r0 # borrow?-1:0
> - vpmsumd $Xm,$IN,$H # H.hi�Xi.lo+H.lo�Xi.hi
> - and r0,r0,$len
> - vpmsumd $Xh,$IN,$Hh # H.hi�Xi.hi
> - add $inp,$inp,r0
> -
> - vpmsumd $t2,$Xl,$xC2 # 1st phase
> -
> - vsldoi $t0,$Xm,$zero,8
> - vsldoi $t1,$zero,$Xm,8
> - vxor $Xl,$Xl,$t0
> - vxor $Xh,$Xh,$t1
> -
> - vsldoi $Xl,$Xl,$Xl,8
> - vxor $Xl,$Xl,$t2
> - lvx_u $IN,0,$inp
> - addi $inp,$inp,16
> -
> - vsldoi $t1,$Xl,$Xl,8 # 2nd phase
> - vpmsumd $Xl,$Xl,$xC2
> - le?vperm $IN,$IN,$IN,$lemask
> - vxor $t1,$t1,$Xh
> - vxor $IN,$IN,$t1
> - vxor $IN,$IN,$Xl
> - beq Loop # did $len-=16 borrow?
> -
> - vxor $Xl,$Xl,$t1
> - le?vperm $Xl,$Xl,$Xl,$lemask
> - stvx_u $Xl,0,$Xip # write out Xi
> -
> - mtspr 256,$vrsave
> - blr
> - .long 0
> - .byte 0,12,0x14,0,0,0,4,0
> - .long 0
> -.size .gcm_ghash_p8,.-.gcm_ghash_p8
> -
> -.asciz "GHASH for PowerISA 2.07, CRYPTOGAMS by <appro\@openssl.org>"
> -.align 2
> -___
> -
> -foreach (split("\n",$code)) {
> - if ($flavour =~ /le$/o) { # little-endian
> - s/le\?//o or
> - s/be\?/#be#/o;
> - } else {
> - s/le\?/#le#/o or
> - s/be\?//o;
> - }
> - print $_,"\n";
> -}
> -
> -close STDOUT; # enforce flush
> diff --git a/drivers/crypto/vmx/ppc-xlate.pl b/drivers/crypto/vmx/ppc-xlate.pl
> deleted file mode 100644
> index b18e67d0e065..000000000000
> --- a/drivers/crypto/vmx/ppc-xlate.pl
> +++ /dev/null
> @@ -1,228 +0,0 @@
> -#!/usr/bin/env perl
> -
> -# PowerPC assembler distiller by <appro>.
> -
> -my $flavour = shift;
> -my $output = shift;
> -open STDOUT,">$output" || die "can't open $output: $!";
> -
> -my %GLOBALS;
> -my $dotinlocallabels=($flavour=~/linux/)?1:0;
> -
> -################################################################
> -# directives which need special treatment on different platforms
> -################################################################
> -my $globl = sub {
> - my $junk = shift;
> - my $name = shift;
> - my $global = \$GLOBALS{$name};
> - my $ret;
> -
> - $name =~ s|^[\.\_]||;
> -
> - SWITCH: for ($flavour) {
> - /aix/ && do { $name = ".$name";
> - last;
> - };
> - /osx/ && do { $name = "_$name";
> - last;
> - };
> - /linux/
> - && do { $ret = "_GLOBAL($name)";
> - last;
> - };
> - }
> -
> - $ret = ".globl $name\nalign 5\n$name:" if (!$ret);
> - $$global = $name;
> - $ret;
> -};
> -my $text = sub {
> - my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text";
> - $ret = ".abiversion 2\n".$ret if ($flavour =~ /linux.*64le/);
> - $ret;
> -};
> -my $machine = sub {
> - my $junk = shift;
> - my $arch = shift;
> - if ($flavour =~ /osx/)
> - { $arch =~ s/\"//g;
> - $arch = ($flavour=~/64/) ? "ppc970-64" : "ppc970" if ($arch eq "any");
> - }
> - ".machine $arch";
> -};
> -my $size = sub {
> - if ($flavour =~ /linux/)
> - { shift;
> - my $name = shift; $name =~ s|^[\.\_]||;
> - my $ret = ".size $name,.-".($flavour=~/64$/?".":"").$name;
> - $ret .= "\n.size .$name,.-.$name" if ($flavour=~/64$/);
> - $ret;
> - }
> - else
> - { ""; }
> -};
> -my $asciz = sub {
> - shift;
> - my $line = join(",",@_);
> - if ($line =~ /^"(.*)"$/)
> - { ".byte " . join(",",unpack("C*",$1),0) . "\n.align 2"; }
> - else
> - { ""; }
> -};
> -my $quad = sub {
> - shift;
> - my @ret;
> - my ($hi,$lo);
> - for (@_) {
> - if (/^0x([0-9a-f]*?)([0-9a-f]{1,8})$/io)
> - { $hi=$1?"0x$1":"0"; $lo="0x$2"; }
> - elsif (/^([0-9]+)$/o)
> - { $hi=$1>>32; $lo=$1&0xffffffff; } # error-prone with 32-bit perl
> - else
> - { $hi=undef; $lo=$_; }
> -
> - if (defined($hi))
> - { push(@ret,$flavour=~/le$/o?".long\t$lo,$hi":".long\t$hi,$lo"); }
> - else
> - { push(@ret,".quad $lo"); }
> - }
> - join("\n",@ret);
> -};
> -
> -################################################################
> -# simplified mnemonics not handled by at least one assembler
> -################################################################
> -my $cmplw = sub {
> - my $f = shift;
> - my $cr = 0; $cr = shift if ($#_>1);
> - # Some out-of-date 32-bit GNU assembler just can't handle cmplw...
> - ($flavour =~ /linux.*32/) ?
> - " .long ".sprintf "0x%x",31<<26|$cr<<23|$_[0]<<16|$_[1]<<11|64 :
> - " cmplw ".join(',',$cr,@_);
> -};
> -my $bdnz = sub {
> - my $f = shift;
> - my $bo = $f=~/[\+\-]/ ? 16+9 : 16; # optional "to be taken" hint
> - " bc $bo,0,".shift;
> -} if ($flavour!~/linux/);
> -my $bltlr = sub {
> - my $f = shift;
> - my $bo = $f=~/\-/ ? 12+2 : 12; # optional "not to be taken" hint
> - ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
> - " .long ".sprintf "0x%x",19<<26|$bo<<21|16<<1 :
> - " bclr $bo,0";
> -};
> -my $bnelr = sub {
> - my $f = shift;
> - my $bo = $f=~/\-/ ? 4+2 : 4; # optional "not to be taken" hint
> - ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
> - " .long ".sprintf "0x%x",19<<26|$bo<<21|2<<16|16<<1 :
> - " bclr $bo,2";
> -};
> -my $beqlr = sub {
> - my $f = shift;
> - my $bo = $f=~/-/ ? 12+2 : 12; # optional "not to be taken" hint
> - ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
> - " .long ".sprintf "0x%X",19<<26|$bo<<21|2<<16|16<<1 :
> - " bclr $bo,2";
> -};
> -# GNU assembler can't handle extrdi rA,rS,16,48, or when sum of last two
> -# arguments is 64, with "operand out of range" error.
> -my $extrdi = sub {
> - my ($f,$ra,$rs,$n,$b) = @_;
> - $b = ($b+$n)&63; $n = 64-$n;
> - " rldicl $ra,$rs,$b,$n";
> -};
> -my $vmr = sub {
> - my ($f,$vx,$vy) = @_;
> - " vor $vx,$vy,$vy";
> -};
> -
> -# Some ABIs specify vrsave, special-purpose register #256, as reserved
> -# for system use.
> -my $no_vrsave = ($flavour =~ /linux-ppc64le/);
> -my $mtspr = sub {
> - my ($f,$idx,$ra) = @_;
> - if ($idx == 256 && $no_vrsave) {
> - " or $ra,$ra,$ra";
> - } else {
> - " mtspr $idx,$ra";
> - }
> -};
> -my $mfspr = sub {
> - my ($f,$rd,$idx) = @_;
> - if ($idx == 256 && $no_vrsave) {
> - " li $rd,-1";
> - } else {
> - " mfspr $rd,$idx";
> - }
> -};
> -
> -# PowerISA 2.06 stuff
> -sub vsxmem_op {
> - my ($f, $vrt, $ra, $rb, $op) = @_;
> - " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|($rb<<11)|($op*2+1);
> -}
> -# made-up unaligned memory reference AltiVec/VMX instructions
> -my $lvx_u = sub { vsxmem_op(@_, 844); }; # lxvd2x
> -my $stvx_u = sub { vsxmem_op(@_, 972); }; # stxvd2x
> -my $lvdx_u = sub { vsxmem_op(@_, 588); }; # lxsdx
> -my $stvdx_u = sub { vsxmem_op(@_, 716); }; # stxsdx
> -my $lvx_4w = sub { vsxmem_op(@_, 780); }; # lxvw4x
> -my $stvx_4w = sub { vsxmem_op(@_, 908); }; # stxvw4x
> -
> -# PowerISA 2.07 stuff
> -sub vcrypto_op {
> - my ($f, $vrt, $vra, $vrb, $op) = @_;
> - " .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|$op;
> -}
> -my $vcipher = sub { vcrypto_op(@_, 1288); };
> -my $vcipherlast = sub { vcrypto_op(@_, 1289); };
> -my $vncipher = sub { vcrypto_op(@_, 1352); };
> -my $vncipherlast= sub { vcrypto_op(@_, 1353); };
> -my $vsbox = sub { vcrypto_op(@_, 0, 1480); };
> -my $vshasigmad = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1730); };
> -my $vshasigmaw = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1666); };
> -my $vpmsumb = sub { vcrypto_op(@_, 1032); };
> -my $vpmsumd = sub { vcrypto_op(@_, 1224); };
> -my $vpmsubh = sub { vcrypto_op(@_, 1096); };
> -my $vpmsumw = sub { vcrypto_op(@_, 1160); };
> -my $vaddudm = sub { vcrypto_op(@_, 192); };
> -my $vadduqm = sub { vcrypto_op(@_, 256); };
> -
> -my $mtsle = sub {
> - my ($f, $arg) = @_;
> - " .long ".sprintf "0x%X",(31<<26)|($arg<<21)|(147*2);
> -};
> -
> -print "#include <asm/ppc_asm.h>\n" if $flavour =~ /linux/;
> -
> -while($line=<>) {
> -
> - $line =~ s|[#!;].*$||; # get rid of asm-style comments...
> - $line =~ s|/\*.*\*/||; # ... and C-style comments...
> - $line =~ s|^\s+||; # ... and skip white spaces in beginning...
> - $line =~ s|\s+$||; # ... and at the end
> -
> - {
> - $line =~ s|\b\.L(\w+)|L$1|g; # common denominator for Locallabel
> - $line =~ s|\bL(\w+)|\.L$1|g if ($dotinlocallabels);
> - }
> -
> - {
> - $line =~ s|^\s*(\.?)(\w+)([\.\+\-]?)\s*||;
> - my $c = $1; $c = "\t" if ($c eq "");
> - my $mnemonic = $2;
> - my $f = $3;
> - my $opcode = eval("\$$mnemonic");
> - $line =~ s/\b(c?[rf]|v|vs)([0-9]+)\b/$2/g if ($c ne "." and $flavour !~ /osx/);
> - if (ref($opcode) eq 'CODE') { $line = &$opcode($f,split(',',$line)); }
> - elsif ($mnemonic) { $line = $c.$mnemonic.$f."\t".$line; }
> - }
> -
> - print $line if ($line);
> - print "\n";
> -}
> -
> -close STDOUT;
> diff --git a/drivers/crypto/vmx/vmx.c b/drivers/crypto/vmx/vmx.c
> deleted file mode 100644
> index 31a98dc6f849..000000000000
> --- a/drivers/crypto/vmx/vmx.c
> +++ /dev/null
> @@ -1,88 +0,0 @@
> -/**
> - * Routines supporting VMX instructions on the Power 8
> - *
> - * Copyright (C) 2015 International Business Machines Inc.
> - *
> - * This program is free software; you can redistribute it and/or modify
> - * it under the terms of the GNU General Public License as published by
> - * the Free Software Foundation; version 2 only.
> - *
> - * This program is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - * GNU General Public License for more details.
> - *
> - * You should have received a copy of the GNU General Public License
> - * along with this program; if not, write to the Free Software
> - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> - *
> - * Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
> - */
> -
> -#include <linux/module.h>
> -#include <linux/moduleparam.h>
> -#include <linux/types.h>
> -#include <linux/err.h>
> -#include <linux/cpufeature.h>
> -#include <linux/crypto.h>
> -#include <asm/cputable.h>
> -#include <crypto/internal/hash.h>
> -
> -extern struct shash_alg p8_ghash_alg;
> -extern struct crypto_alg p8_aes_alg;
> -extern struct crypto_alg p8_aes_cbc_alg;
> -extern struct crypto_alg p8_aes_ctr_alg;
> -extern struct crypto_alg p8_aes_xts_alg;
> -static struct crypto_alg *algs[] = {
> - &p8_aes_alg,
> - &p8_aes_cbc_alg,
> - &p8_aes_ctr_alg,
> - &p8_aes_xts_alg,
> - NULL,
> -};
> -
> -int __init p8_init(void)
> -{
> - int ret = 0;
> - struct crypto_alg **alg_it;
> -
> - for (alg_it = algs; *alg_it; alg_it++) {
> - ret = crypto_register_alg(*alg_it);
> - printk(KERN_INFO "crypto_register_alg '%s' = %d\n",
> - (*alg_it)->cra_name, ret);
> - if (ret) {
> - for (alg_it--; alg_it >= algs; alg_it--)
> - crypto_unregister_alg(*alg_it);
> - break;
> - }
> - }
> - if (ret)
> - return ret;
> -
> - ret = crypto_register_shash(&p8_ghash_alg);
> - if (ret) {
> - for (alg_it = algs; *alg_it; alg_it++)
> - crypto_unregister_alg(*alg_it);
> - }
> - return ret;
> -}
> -
> -void __exit p8_exit(void)
> -{
> - struct crypto_alg **alg_it;
> -
> - for (alg_it = algs; *alg_it; alg_it++) {
> - printk(KERN_INFO "Removing '%s'\n", (*alg_it)->cra_name);
> - crypto_unregister_alg(*alg_it);
> - }
> - crypto_unregister_shash(&p8_ghash_alg);
> -}
> -
> -module_cpu_feature_match(PPC_MODULE_FEATURE_VEC_CRYPTO, p8_init);
> -module_exit(p8_exit);
> -
> -MODULE_AUTHOR("Marcelo Cerri<mhcerri@br.ibm.com>");
> -MODULE_DESCRIPTION("IBM VMX cryptographic acceleration instructions "
> - "support on Power 8");
> -MODULE_LICENSE("GPL");
> -MODULE_VERSION("1.0.0");
>
^ permalink raw reply
* Re: [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Michael Ellerman @ 2017-03-30 0:17 UTC (permalink / raw)
To: Michal Suchanek, Herbert Xu, David S. Miller,
Benjamin Herrenschmidt, Paul Mackerras, Greg Kroah-Hartman,
Geert Uytterhoeven, Mauro Carvalho Chehab, linux-kernel,
linux-crypto, linuxppc-dev, paulmck
Cc: Michal Suchanek
In-Reply-To: <20170329125639.14288-1-msuchanek@suse.de>
Michal Suchanek <msuchanek@suse.de> writes:
> While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
> subroutines for XTS") which adds the OpenSSL license header to
> drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
> qestion. The whole license reads:
>
> # Licensed under the OpenSSL license (the "License"). You may not use
> # this file except in compliance with the License. You can obtain a
> # copy
> # in the file LICENSE in the source distribution or at
> # https://www.openssl.org/source/license.html
>
> #
> # ====================================================================
> # Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
> # project. The module is, however, dual licensed under OpenSSL and
> # CRYPTOGAMS licenses depending on where you obtain it. For further
> # details see http://www.openssl.org/~appro/cryptogams/.
> # ====================================================================
>
> After seeking legal advice it is still not clear that this driver can be
> legally used in Linux. In particular the "depending on where you obtain
> it" part does not make it clear when you can apply the GPL and when the
> OpenSSL license.
It seems pretty clear to me that the intention is that the CRYPTOGAM
license applies.
If you visit it's URL it includes:
ALTERNATIVELY, provided that this notice is retained in full, this
product may be distributed under the terms of the GNU General Public
License (GPL), in which case the provisions of the GPL apply INSTEAD OF
those given above.
I agree that the text in the file is not sufficiently clear about what
license applies, but I'm unconvinced that there is any code here that is
actually being distributed incorrectly.
Getting the text in the header changed to be clearer seems like the
obvious solution.
> I tried contacting the author of the code for clarification but did not
> hear back. In absence of clear licensing the only solution I see is
> removing this code.
Did you try contacting anyone else? Like perhaps the powerpc or crypto
maintainers, or anyone else who's worked on the driver?
Sending a patch to delete all the code clearly works to get people's
attention, I'll give you that.
^ permalink raw reply
* Re: [PATCH] crypto: vmx: Remove dubiously licensed crypto code
From: Tyrel Datwyler @ 2017-03-30 0:49 UTC (permalink / raw)
To: Michael Ellerman, Michal Suchanek, Herbert Xu, David S. Miller,
Benjamin Herrenschmidt, Paul Mackerras, Greg Kroah-Hartman,
Geert Uytterhoeven, Mauro Carvalho Chehab, linux-kernel,
linux-crypto, linuxppc-dev, paulmck@linux.vnet.ibm.com, appro
In-Reply-To: <87r31flisi.fsf@concordia.ellerman.id.au>
On 03/29/2017 05:17 PM, Michael Ellerman wrote:
> Michal Suchanek <msuchanek@suse.de> writes:
>
>> While reviewing commit 11c6e16ee13a ("crypto: vmx - Adding asm
>> subroutines for XTS") which adds the OpenSSL license header to
>> drivers/crypto/vmx/aesp8-ppc.pl licensing of this driver came into
>> qestion. The whole license reads:
>>
>> # Licensed under the OpenSSL license (the "License"). You may not use
>> # this file except in compliance with the License. You can obtain a
>> # copy
>> # in the file LICENSE in the source distribution or at
>> # https://www.openssl.org/source/license.html
>>
>> #
>> # ====================================================================
>> # Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
>> # project. The module is, however, dual licensed under OpenSSL and
>> # CRYPTOGAMS licenses depending on where you obtain it. For further
>> # details see http://www.openssl.org/~appro/cryptogams/.
>> # ====================================================================
>>
>> After seeking legal advice it is still not clear that this driver can be
>> legally used in Linux. In particular the "depending on where you obtain
>> it" part does not make it clear when you can apply the GPL and when the
>> OpenSSL license.
>
> It seems pretty clear to me that the intention is that the CRYPTOGAM
> license applies.
>
> If you visit it's URL it includes:
>
> ALTERNATIVELY, provided that this notice is retained in full, this
> product may be distributed under the terms of the GNU General Public
> License (GPL), in which case the provisions of the GPL apply INSTEAD OF
> those given above.
>
>
> I agree that the text in the file is not sufficiently clear about what
> license applies, but I'm unconvinced that there is any code here that is
> actually being distributed incorrectly.
The original commit message also outlines that the authors collaborated
directly with Andy.
commit 5c380d623ed30b71a2441fb4f2e053a4e1a50794
Author: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
Date: Fri Feb 6 14:59:35 2015 -0200
crypto: vmx - Add support for VMS instructions by ASM
OpenSSL implements optimized ASM algorithms which support
VMX instructions on Power 8 CPU.
These scripts generate an endian-agnostic ASM implementation
in order to support both big and little-endian.
- aesp8-ppc.pl: implements suport for AES instructions
implemented by POWER8 processor.
- ghashp8-ppc.pl: implements support for GHASH for Power8.
- ppc-xlate.pl: ppc assembler distiller.
These code has been adopted from OpenSSL project in collaboration
with the original author (Andy Polyakov <appro@openssl.org>).
Signed-off-by: Leonidas S. Barbosa <leosilva@linux.vnet.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
-Tyrel
>
> Getting the text in the header changed to be clearer seems like the
> obvious solution.
>
>> I tried contacting the author of the code for clarification but did not
>> hear back. In absence of clear licensing the only solution I see is
>> removing this code.
>
> Did you try contacting anyone else? Like perhaps the powerpc or crypto
> maintainers, or anyone else who's worked on the driver?
>
> Sending a patch to delete all the code clearly works to get people's
> attention, I'll give you that.
>
^ permalink raw reply
* RE: [PATCH 1/6] virtio: wrap find_vqs
From: Gonglei (Arei) @ 2017-03-30 0:50 UTC (permalink / raw)
To: Michael S. Tsirkin, linux-kernel@vger.kernel.org
Cc: John Fastabend, Jason Wang, Arnd Bergmann, Greg Kroah-Hartman,
Amit Shah, Herbert Xu, David S. Miller, David Airlie,
Gerd Hoffmann, Dmitry Tarnyagin, Ohad Ben-Cohen, Bjorn Andersson,
James E.J. Bottomley, Martin K. Petersen, Stefan Hajnoczi,
virtualization@lists.linux-foundation.org,
linux-crypto@vger.kernel.org
In-Reply-To: <1490820507-8005-2-git-send-email-mst@redhat.com>
> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Thursday, March 30, 2017 4:49 AM
> Subject: [PATCH 1/6] virtio: wrap find_vqs
>
> We are going to add more parameters to find_vqs, let's wrap the call so
> we don't need to tweak all drivers every time.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
> drivers/block/virtio_blk.c | 3 +--
> drivers/char/virtio_console.c | 6 +++---
> drivers/crypto/virtio/virtio_crypto_core.c | 3 +--
> drivers/gpu/drm/virtio/virtgpu_kms.c | 3 +--
> drivers/net/caif/caif_virtio.c | 3 +--
> drivers/net/virtio_net.c | 3 +--
> drivers/rpmsg/virtio_rpmsg_bus.c | 2 +-
> drivers/scsi/virtio_scsi.c | 3 +--
> drivers/virtio/virtio_balloon.c | 3 +--
> drivers/virtio/virtio_input.c | 3 +--
> include/linux/virtio_config.h | 9 +++++++++
> net/vmw_vsock/virtio_transport.c | 6 +++---
> 12 files changed, 24 insertions(+), 23 deletions(-)
>
Acked-by: Gonglei <arei.gonglei@huawei.com>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 1d4c9f8..c08c30c 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -455,8 +455,7 @@ static int init_vq(struct virtio_blk *vblk)
> }
>
> /* Discover virtqueues and write information to configuration. */
> - err = vdev->config->find_vqs(vdev, num_vqs, vqs, callbacks, names,
> - &desc);
> + err = virtio_find_vqs(vdev, num_vqs, vqs, callbacks, names, &desc);
> if (err)
> goto out;
>
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index e9b7e0b..5da4c8e 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -1945,9 +1945,9 @@ static int init_vqs(struct ports_device *portdev)
> }
> }
> /* Find the queues. */
> - err = portdev->vdev->config->find_vqs(portdev->vdev, nr_queues, vqs,
> - io_callbacks,
> - (const char **)io_names, NULL);
> + err = virtio_find_vqs(portdev->vdev, nr_queues, vqs,
> + io_callbacks,
> + (const char **)io_names, NULL);
> if (err)
> goto free;
>
> diff --git a/drivers/crypto/virtio/virtio_crypto_core.c
> b/drivers/crypto/virtio/virtio_crypto_core.c
> index 21472e4..a111cd72 100644
> --- a/drivers/crypto/virtio/virtio_crypto_core.c
> +++ b/drivers/crypto/virtio/virtio_crypto_core.c
> @@ -119,8 +119,7 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi)
> names[i] = vi->data_vq[i].name;
> }
>
> - ret = vi->vdev->config->find_vqs(vi->vdev, total_vqs, vqs, callbacks,
> - names, NULL);
> + ret = virtio_find_vqs(vi->vdev, total_vqs, vqs, callbacks, names, NULL);
> if (ret)
> goto err_find;
>
> diff --git a/drivers/gpu/drm/virtio/virtgpu_kms.c
> b/drivers/gpu/drm/virtio/virtgpu_kms.c
> index 4918668..1e1c90b 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_kms.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_kms.c
> @@ -175,8 +175,7 @@ int virtio_gpu_driver_load(struct drm_device *dev,
> unsigned long flags)
> DRM_INFO("virgl 3d acceleration not supported by guest\n");
> #endif
>
> - ret = vgdev->vdev->config->find_vqs(vgdev->vdev, 2, vqs,
> - callbacks, names, NULL);
> + ret = virtio_find_vqs(vgdev->vdev, 2, vqs, callbacks, names, NULL);
> if (ret) {
> DRM_ERROR("failed to find virt queues\n");
> goto err_vqs;
> diff --git a/drivers/net/caif/caif_virtio.c b/drivers/net/caif/caif_virtio.c
> index bc0eb47..6122768 100644
> --- a/drivers/net/caif/caif_virtio.c
> +++ b/drivers/net/caif/caif_virtio.c
> @@ -679,8 +679,7 @@ static int cfv_probe(struct virtio_device *vdev)
> goto err;
>
> /* Get the TX virtio ring. This is a "guest side vring". */
> - err = vdev->config->find_vqs(vdev, 1, &cfv->vq_tx, &vq_cbs, &names,
> - NULL);
> + err = virtio_find_vqs(vdev, 1, &cfv->vq_tx, &vq_cbs, &names, NULL);
> if (err)
> goto err;
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index ea9890d..6802169 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -2079,8 +2079,7 @@ static int virtnet_find_vqs(struct virtnet_info *vi)
> names[txq2vq(i)] = vi->sq[i].name;
> }
>
> - ret = vi->vdev->config->find_vqs(vi->vdev, total_vqs, vqs, callbacks,
> - names, NULL);
> + ret = virtio_find_vqs(vi->vdev, total_vqs, vqs, callbacks, names, NULL);
> if (ret)
> goto err_find;
>
> diff --git a/drivers/rpmsg/virtio_rpmsg_bus.c
> b/drivers/rpmsg/virtio_rpmsg_bus.c
> index 5e66e08..f7cade0 100644
> --- a/drivers/rpmsg/virtio_rpmsg_bus.c
> +++ b/drivers/rpmsg/virtio_rpmsg_bus.c
> @@ -869,7 +869,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
> init_waitqueue_head(&vrp->sendq);
>
> /* We expect two virtqueues, rx and tx (and in this order) */
> - err = vdev->config->find_vqs(vdev, 2, vqs, vq_cbs, names, NULL);
> + err = virtio_find_vqs(vdev, 2, vqs, vq_cbs, names, NULL);
> if (err)
> goto free_vrp;
>
> diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
> index 939c47d..e9222dc 100644
> --- a/drivers/scsi/virtio_scsi.c
> +++ b/drivers/scsi/virtio_scsi.c
> @@ -870,8 +870,7 @@ static int virtscsi_init(struct virtio_device *vdev,
> }
>
> /* Discover virtqueues and write information to configuration. */
> - err = vdev->config->find_vqs(vdev, num_vqs, vqs, callbacks, names,
> - &desc);
> + err = virtio_find_vqs(vdev, num_vqs, vqs, callbacks, names, &desc);
> if (err)
> goto out;
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 34adf9b..408c174 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -418,8 +418,7 @@ static int init_vqs(struct virtio_balloon *vb)
> * optionally stat.
> */
> nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 :
> 2;
> - err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names,
> - NULL);
> + err = virtio_find_vqs(vb->vdev, nvqs, vqs, callbacks, names, NULL);
> if (err)
> return err;
>
> diff --git a/drivers/virtio/virtio_input.c b/drivers/virtio/virtio_input.c
> index 79f1293..3a0468f 100644
> --- a/drivers/virtio/virtio_input.c
> +++ b/drivers/virtio/virtio_input.c
> @@ -173,8 +173,7 @@ static int virtinput_init_vqs(struct virtio_input *vi)
> static const char * const names[] = { "events", "status" };
> int err;
>
> - err = vi->vdev->config->find_vqs(vi->vdev, 2, vqs, cbs, names,
> - NULL);
> + err = virtio_find_vqs(vi->vdev, 2, vqs, cbs, names, NULL);
> if (err)
> return err;
> vi->evt = vqs[0];
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index 8355bab..47f3d80 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -179,6 +179,15 @@ struct virtqueue *virtio_find_single_vq(struct
> virtio_device *vdev,
> return vq;
> }
>
> +static inline
> +int virtio_find_vqs(struct virtio_device *vdev, unsigned nvqs,
> + struct virtqueue *vqs[], vq_callback_t *callbacks[],
> + const char * const names[],
> + struct irq_affinity *desc)
> +{
> + return vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names, desc);
> +}
> +
> /**
> * virtio_device_ready - enable vq use in probe function
> * @vdev: the device
> diff --git a/net/vmw_vsock/virtio_transport.c
> b/net/vmw_vsock/virtio_transport.c
> index 68675a1..97e26e2 100644
> --- a/net/vmw_vsock/virtio_transport.c
> +++ b/net/vmw_vsock/virtio_transport.c
> @@ -573,9 +573,9 @@ static int virtio_vsock_probe(struct virtio_device
> *vdev)
>
> vsock->vdev = vdev;
>
> - ret = vsock->vdev->config->find_vqs(vsock->vdev, VSOCK_VQ_MAX,
> - vsock->vqs, callbacks, names,
> - NULL);
> + ret = virtio_find_vqs(vsock->vdev, VSOCK_VQ_MAX,
> + vsock->vqs, callbacks, names,
> + NULL);
> if (ret < 0)
> goto out;
>
> --
> MST
^ permalink raw reply
* Re: [PATCH v6 0/4] Broadcom SBA RAID support
From: Vinod Koul @ 2017-03-30 4:49 UTC (permalink / raw)
To: Jassi Brar
Cc: Anup Patel, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Dan Williams, Ray Jui, Scott Branden, Jon Mason,
Rob Rice, BCM Kernel Feedback, dmaengine, Device Tree,
Linux ARM Kernel, Linux Kernel, linux-crypto, linux-raid
In-Reply-To: <CAALAos-P6CDdWTJxg8YQfT1V+iCxUcbYHiQ35EdZzz9PnN3cuA@mail.gmail.com>
On Wed, Mar 29, 2017 at 11:35:43AM +0530, Anup Patel wrote:
> On Tue, Mar 21, 2017 at 2:48 PM, Vinod Koul <vinod.koul@intel.com> wrote:
> > On Tue, Mar 21, 2017 at 02:17:21PM +0530, Anup Patel wrote:
> >> On Tue, Mar 21, 2017 at 2:00 PM, Vinod Koul <vinod.koul@intel.com> wrote:
> >> > On Mon, Mar 06, 2017 at 03:13:24PM +0530, Anup Patel wrote:
> >> >> The Broadcom SBA RAID is a stream-based device which provides
> >> >> RAID5/6 offload.
> >> >>
> >> >> It requires a SoC specific ring manager (such as Broadcom FlexRM
> >> >> ring manager) to provide ring-based programming interface. Due to
> >> >> this, the Broadcom SBA RAID driver (mailbox client) implements
> >> >> DMA device having one DMA channel using a set of mailbox channels
> >> >> provided by Broadcom SoC specific ring manager driver (mailbox
> >> >> controller).
> >> >>
> >> >> The Broadcom SBA RAID hardware requires PQ disk position instead
> >> >> of PQ disk coefficient. To address this, we have added raid_gflog
> >> >> table which will help driver to convert PQ disk coefficient to PQ
> >> >> disk position.
> >> >>
> >> >> This patchset is based on Linux-4.11-rc1 and depends on patchset
> >> >> "[PATCH v5 0/2] Broadcom FlexRM ring manager support"
> >> >
> >> > Okay I applied and was about to push when I noticed this :(
> >> >
> >> > So what is the status of this..?
> >>
> >> PATCH2 is Acked but PATCH1 is under-review. Currently, its
> >> v6 of that patchset.
> >>
> >> The only dependency on that patchset is the changes in
> >> brcm-message.h which are required by this BCM-SBA-RAID
> >> driver.
> >>
> >> @Jassi,
> >> Can you please have a look at PATCH v6?
> >
> > And I would need an immutable branch/tag once merged. I am going to keep
> > this series pending till then.
>
> The Broadcom FlexRM patchset is pickedup by Jassi and
> can be found in mailbox-for-next branch of
> git://git.linaro.org/landing-teams/working/fujitsu/integration
>
> Both patchset (Broadcom FlexRM patchset and this one) are
> also available in sba-raid-v7 branch of
> https://github.com/Broadcom/arm64-linux.git
Jassi,
Can you provide an immutable branch/tag please for this, latter is
preferred.
Btw didn't find your tree in MAINTAINERS..
>
> Regards,
> Anup
--
~Vinod
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox