* [PATCH 4/6] async_tx: Fix DMA_PREP_FENCE usage in do_async_gen_syndrome()
From: Anup Patel @ 2017-02-02 4:47 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Jassi Brar
Cc: Dan Williams, Ray Jui, Scott Branden, Jon Mason, Rob Rice,
bcm-kernel-feedback-list, dmaengine, devicetree, linux-arm-kernel,
linux-kernel, linux-crypto, linux-raid, Anup Patel
In-Reply-To: <1486010836-25228-1-git-send-email-anup.patel@broadcom.com>
The DMA_PREP_FENCE is to be used when preparing Tx descriptor if output
of Tx descriptor is to be used by next/dependent Tx descriptor.
The DMA_PREP_FENSE will not be set correctly in do_async_gen_syndrome()
when calling dma->device_prep_dma_pq() under following conditions:
1. ASYNC_TX_FENCE not set in submit->flags
2. DMA_PREP_FENCE not set in dma_flags
3. src_cnt (= (disks - 2)) is greater than dma_maxpq(dma, dma_flags)
This patch fixes DMA_PREP_FENCE usage in do_async_gen_syndrome() taking
inspiration from do_async_xor() implementation.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
---
crypto/async_tx/async_pq.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/crypto/async_tx/async_pq.c b/crypto/async_tx/async_pq.c
index 16c6526..947cf35 100644
--- a/crypto/async_tx/async_pq.c
+++ b/crypto/async_tx/async_pq.c
@@ -62,9 +62,6 @@ do_async_gen_syndrome(struct dma_chan *chan,
dma_addr_t dma_dest[2];
int src_off = 0;
- if (submit->flags & ASYNC_TX_FENCE)
- dma_flags |= DMA_PREP_FENCE;
-
while (src_cnt > 0) {
submit->flags = flags_orig;
pq_src_cnt = min(src_cnt, dma_maxpq(dma, dma_flags));
@@ -83,6 +80,8 @@ do_async_gen_syndrome(struct dma_chan *chan,
if (cb_fn_orig)
dma_flags |= DMA_PREP_INTERRUPT;
}
+ if (submit->flags & ASYNC_TX_FENCE)
+ dma_flags |= DMA_PREP_FENCE;
/* Drivers force forward progress in case they can not provide
* a descriptor
--
2.7.4
^ permalink raw reply related
* [PATCH 6/6] dt-bindings: Add DT bindings document for Broadcom SBA RAID driver
From: Anup Patel @ 2017-02-02 4:47 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Jassi Brar
Cc: Dan Williams, Ray Jui, Scott Branden, Jon Mason, Rob Rice,
bcm-kernel-feedback-list-dY08KVG/lbpWk0Htik3J/w,
dmaengine-u79uwXL29TY76Z2rM5mHXA,
devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-crypto-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA, Anup Patel
In-Reply-To: <1486010836-25228-1-git-send-email-anup.patel-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
This patch adds the DT bindings document for newly added Broadcom
SBA RAID driver.
Signed-off-by: Anup Patel <anup.patel-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
Reviewed-by: Ray Jui <ray.jui-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
Reviewed-by: Scott Branden <scott.branden-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
---
.../devicetree/bindings/dma/brcm,iproc-sba.txt | 29 ++++++++++++++++++++++
1 file changed, 29 insertions(+)
create mode 100644 Documentation/devicetree/bindings/dma/brcm,iproc-sba.txt
diff --git a/Documentation/devicetree/bindings/dma/brcm,iproc-sba.txt b/Documentation/devicetree/bindings/dma/brcm,iproc-sba.txt
new file mode 100644
index 0000000..092913a
--- /dev/null
+++ b/Documentation/devicetree/bindings/dma/brcm,iproc-sba.txt
@@ -0,0 +1,29 @@
+* Broadcom SBA RAID engine
+
+Required properties:
+- compatible: Should be one of the following
+ "brcm,iproc-sba"
+ "brcm,iproc-sba-v2"
+ The "brcm,iproc-sba" has support for only 6 PQ coefficients
+ The "brcm,iproc-sba-v2" has support for only 30 PQ coefficients
+- mboxes: List of phandle and mailbox channel specifiers
+
+Example:
+
+raid_mbox: mbox@67400000 {
+ ...
+ #mbox-cells = <3>;
+ ...
+};
+
+raid0 {
+ compatible = "brcm,iproc-sba-v2";
+ mboxes = <&raid_mbox 0 0x1 0xffff>,
+ <&raid_mbox 1 0x1 0xffff>,
+ <&raid_mbox 2 0x1 0xffff>,
+ <&raid_mbox 3 0x1 0xffff>,
+ <&raid_mbox 4 0x1 0xffff>,
+ <&raid_mbox 5 0x1 0xffff>,
+ <&raid_mbox 6 0x1 0xffff>,
+ <&raid_mbox 7 0x1 0xffff>;
+};
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 2/6] lib/raid6: Add log-of-2 table for RAID6 HW requiring disk position
From: Anup Patel @ 2017-02-02 4:47 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Jassi Brar
Cc: Dan Williams, Ray Jui, Scott Branden, Jon Mason, Rob Rice,
bcm-kernel-feedback-list, dmaengine, devicetree, linux-arm-kernel,
linux-kernel, linux-crypto, linux-raid, Anup Patel
In-Reply-To: <1486010836-25228-1-git-send-email-anup.patel@broadcom.com>
The raid6_gfexp table represents {2}^n values for 0 <= n < 256. The
Linux async_tx framework pass values from raid6_gfexp as coefficients
for each source to prep_dma_pq() callback of DMA channel with PQ
capability. This creates problem for RAID6 offload engines (such as
Broadcom SBA) which take disk position (i.e. log of {2}) instead of
multiplicative cofficients from raid6_gfexp table.
This patch adds raid6_gflog table having log-of-2 value for any given
x such that 0 <= x < 256. For any given disk coefficient x, the
corresponding disk position is given by raid6_gflog[x]. The RAID6
offload engine driver can use this newly added raid6_gflog table to
get disk position from multiplicative coefficient.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
---
include/linux/raid/pq.h | 1 +
lib/raid6/mktables.c | 20 ++++++++++++++++++++
2 files changed, 21 insertions(+)
diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h
index 4d57bba..30f9453 100644
--- a/include/linux/raid/pq.h
+++ b/include/linux/raid/pq.h
@@ -142,6 +142,7 @@ int raid6_select_algo(void);
extern const u8 raid6_gfmul[256][256] __attribute__((aligned(256)));
extern const u8 raid6_vgfmul[256][32] __attribute__((aligned(256)));
extern const u8 raid6_gfexp[256] __attribute__((aligned(256)));
+extern const u8 raid6_gflog[256] __attribute__((aligned(256)));
extern const u8 raid6_gfinv[256] __attribute__((aligned(256)));
extern const u8 raid6_gfexi[256] __attribute__((aligned(256)));
diff --git a/lib/raid6/mktables.c b/lib/raid6/mktables.c
index 39787db..e824d08 100644
--- a/lib/raid6/mktables.c
+++ b/lib/raid6/mktables.c
@@ -125,6 +125,26 @@ int main(int argc, char *argv[])
printf("EXPORT_SYMBOL(raid6_gfexp);\n");
printf("#endif\n");
+ /* Compute log-of-2 table */
+ printf("\nconst u8 __attribute__((aligned(256)))\n"
+ "raid6_gflog[256] =\n" "{\n");
+ for (i = 0; i < 256; i += 8) {
+ printf("\t");
+ for (j = 0; j < 8; j++) {
+ v = 255;
+ for (k = 0; k < 256; k++)
+ if (exptbl[k] == (i + j)) {
+ v = k;
+ break;
+ }
+ printf("0x%02x,%c", v, (j == 7) ? '\n' : ' ');
+ }
+ }
+ printf("};\n");
+ printf("#ifdef __KERNEL__\n");
+ printf("EXPORT_SYMBOL(raid6_gflog);\n");
+ printf("#endif\n");
+
/* Compute inverse table x^-1 == x^254 */
printf("\nconst u8 __attribute__((aligned(256)))\n"
"raid6_gfinv[256] =\n" "{\n");
--
2.7.4
^ permalink raw reply related
* [PATCH 3/6] async_tx: Handle DMA devices having support for fewer PQ coefficients
From: Anup Patel @ 2017-02-02 4:47 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Jassi Brar
Cc: Dan Williams, Ray Jui, Scott Branden, Jon Mason, Rob Rice,
bcm-kernel-feedback-list, dmaengine, devicetree, linux-arm-kernel,
linux-kernel, linux-crypto, linux-raid, Anup Patel
In-Reply-To: <1486010836-25228-1-git-send-email-anup.patel@broadcom.com>
The DMAENGINE framework assumes that if PQ offload is supported by a
DMA device then all 256 PQ coefficients are supported. This assumption
does not hold anymore because we now have BCM-SBA-RAID offload engine
which supports PQ offload with limited number of PQ coefficients.
This patch extends async_tx APIs to handle DMA devices with support
for fewer PQ coefficients.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
---
crypto/async_tx/async_pq.c | 3 +++
crypto/async_tx/async_raid6_recov.c | 12 ++++++++++--
include/linux/dmaengine.h | 19 +++++++++++++++++++
include/linux/raid/pq.h | 3 +++
4 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/crypto/async_tx/async_pq.c b/crypto/async_tx/async_pq.c
index f83de99..16c6526 100644
--- a/crypto/async_tx/async_pq.c
+++ b/crypto/async_tx/async_pq.c
@@ -187,6 +187,9 @@ async_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
BUG_ON(disks > 255 || !(P(blocks, disks) || Q(blocks, disks)));
+ if (device && dma_maxpqcoef(device) < src_cnt)
+ device = NULL;
+
if (device)
unmap = dmaengine_get_unmap_data(device->dev, disks, GFP_NOWAIT);
diff --git a/crypto/async_tx/async_raid6_recov.c b/crypto/async_tx/async_raid6_recov.c
index 8fab627..2916f95 100644
--- a/crypto/async_tx/async_raid6_recov.c
+++ b/crypto/async_tx/async_raid6_recov.c
@@ -352,6 +352,7 @@ async_raid6_2data_recov(int disks, size_t bytes, int faila, int failb,
{
void *scribble = submit->scribble;
int non_zero_srcs, i;
+ struct dma_chan *chan = async_dma_find_channel(DMA_PQ);
BUG_ON(faila == failb);
if (failb < faila)
@@ -359,12 +360,15 @@ async_raid6_2data_recov(int disks, size_t bytes, int faila, int failb,
pr_debug("%s: disks: %d len: %zu\n", __func__, disks, bytes);
+ if (chan && dma_maxpqcoef(chan->device) < RAID6_PQ_MAX_COEF)
+ chan = NULL;
+
/* if a dma resource is not available or a scribble buffer is not
* available punt to the synchronous path. In the 'dma not
* available' case be sure to use the scribble buffer to
* preserve the content of 'blocks' as the caller intended.
*/
- if (!async_dma_find_channel(DMA_PQ) || !scribble) {
+ if (!chan || !scribble) {
void **ptrs = scribble ? scribble : (void **) blocks;
async_tx_quiesce(&submit->depend_tx);
@@ -432,15 +436,19 @@ async_raid6_datap_recov(int disks, size_t bytes, int faila,
void *scribble = submit->scribble;
int good_srcs, good, i;
struct page *srcs[2];
+ struct dma_chan *chan = async_dma_find_channel(DMA_PQ);
pr_debug("%s: disks: %d len: %zu\n", __func__, disks, bytes);
+ if (chan && dma_maxpqcoef(chan->device) < RAID6_PQ_MAX_COEF)
+ chan = NULL;
+
/* if a dma resource is not available or a scribble buffer is not
* available punt to the synchronous path. In the 'dma not
* available' case be sure to use the scribble buffer to
* preserve the content of 'blocks' as the caller intended.
*/
- if (!async_dma_find_channel(DMA_PQ) || !scribble) {
+ if (!chan || !scribble) {
void **ptrs = scribble ? scribble : (void **) blocks;
async_tx_quiesce(&submit->depend_tx);
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index feee6ec..d938a8b 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -24,6 +24,7 @@
#include <linux/scatterlist.h>
#include <linux/bitmap.h>
#include <linux/types.h>
+#include <linux/raid/pq.h>
#include <asm/page.h>
/**
@@ -668,6 +669,7 @@ struct dma_filter {
* @cap_mask: one or more dma_capability flags
* @max_xor: maximum number of xor sources, 0 if no capability
* @max_pq: maximum number of PQ sources and PQ-continue capability
+ * @max_pqcoef: maximum number of PQ coefficients, 0 if all supported
* @copy_align: alignment shift for memcpy operations
* @xor_align: alignment shift for xor operations
* @pq_align: alignment shift for pq operations
@@ -727,11 +729,13 @@ struct dma_device {
dma_cap_mask_t cap_mask;
unsigned short max_xor;
unsigned short max_pq;
+ unsigned short max_pqcoef;
enum dmaengine_alignment copy_align;
enum dmaengine_alignment xor_align;
enum dmaengine_alignment pq_align;
enum dmaengine_alignment fill_align;
#define DMA_HAS_PQ_CONTINUE (1 << 15)
+ #define DMA_HAS_FEWER_PQ_COEF (1 << 15)
int dev_id;
struct device *dev;
@@ -1122,6 +1126,21 @@ static inline int dma_maxpq(struct dma_device *dma, enum dma_ctrl_flags flags)
BUG();
}
+static inline void dma_set_maxpqcoef(struct dma_device *dma,
+ unsigned short max_pqcoef)
+{
+ if (max_pqcoef < RAID6_PQ_MAX_COEF) {
+ dma->max_pqcoef = max_pqcoef;
+ dma->max_pqcoef |= DMA_HAS_FEWER_PQ_COEF;
+ }
+}
+
+static inline unsigned short dma_maxpqcoef(struct dma_device *dma)
+{
+ return (dma->max_pqcoef & DMA_HAS_FEWER_PQ_COEF) ?
+ (dma->max_pqcoef & ~DMA_HAS_FEWER_PQ_COEF) : RAID6_PQ_MAX_COEF;
+}
+
static inline size_t dmaengine_get_icg(bool inc, bool sgl, size_t icg,
size_t dir_icg)
{
diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h
index 30f9453..f3a04bb 100644
--- a/include/linux/raid/pq.h
+++ b/include/linux/raid/pq.h
@@ -15,6 +15,9 @@
#ifdef __KERNEL__
+/* Max number of PQ coefficients */
+#define RAID6_PQ_MAX_COEF 256
+
/* Set to 1 to use kernel-wide empty_zero_page */
#define RAID6_USE_EMPTY_ZERO_PAGE 0
#include <linux/blkdev.h>
--
2.7.4
^ permalink raw reply related
* [PATCH 5/6] dmaengine: Add Broadcom SBA RAID driver
From: Anup Patel @ 2017-02-02 4:47 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Jassi Brar
Cc: Dan Williams, Ray Jui, Scott Branden, Jon Mason, Rob Rice,
bcm-kernel-feedback-list, dmaengine, devicetree, linux-arm-kernel,
linux-kernel, linux-crypto, linux-raid, Anup Patel
In-Reply-To: <1486010836-25228-1-git-send-email-anup.patel@broadcom.com>
The Broadcom stream buffer accelerator (SBA) provides offloading
capabilities for RAID operations. This SBA offload engine is
accessible via Broadcom SoC specific ring manager.
This patch adds Broadcom SBA RAID driver which provides one
DMA device with RAID capabilities using one or more Broadcom
SoC specific ring manager channels. The SBA RAID driver in its
current shape implements memcpy, xor, and pq operations.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
---
drivers/dma/Kconfig | 13 +
drivers/dma/Makefile | 1 +
drivers/dma/bcm-sba-raid.c | 1309 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 1323 insertions(+)
create mode 100644 drivers/dma/bcm-sba-raid.c
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index 263495d..58d0463 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -99,6 +99,19 @@ config AXI_DMAC
controller is often used in Analog Device's reference designs for FPGA
platforms.
+config BCM_SBA_RAID
+ tristate "Broadcom SBA RAID engine support"
+ depends on (ARM64 && MAILBOX && RAID6_PQ) || COMPILE_TEST
+ select DMA_ENGINE
+ select DMA_ENGINE_RAID
+ select ASYNC_TX_ENABLE_CHANNEL_SWITCH
+ default ARCH_BCM_IPROC
+ help
+ Enable support for Broadcom SBA RAID Engine. The SBA RAID
+ engine is available on most of the Broadcom iProc SoCs. It
+ has the capability to offload memcpy, xor and pq computation
+ for raid5/6.
+
config COH901318
bool "ST-Ericsson COH901318 DMA support"
select DMA_ENGINE
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index a4fa336..ba96bdd 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -17,6 +17,7 @@ obj-$(CONFIG_AMCC_PPC440SPE_ADMA) += ppc4xx/
obj-$(CONFIG_AT_HDMAC) += at_hdmac.o
obj-$(CONFIG_AT_XDMAC) += at_xdmac.o
obj-$(CONFIG_AXI_DMAC) += dma-axi-dmac.o
+obj-$(CONFIG_BCM_SBA_RAID) += bcm-sba-raid.o
obj-$(CONFIG_COH901318) += coh901318.o coh901318_lli.o
obj-$(CONFIG_DMA_BCM2835) += bcm2835-dma.o
obj-$(CONFIG_DMA_JZ4740) += dma-jz4740.o
diff --git a/drivers/dma/bcm-sba-raid.c b/drivers/dma/bcm-sba-raid.c
new file mode 100644
index 0000000..bf39f3f
--- /dev/null
+++ b/drivers/dma/bcm-sba-raid.c
@@ -0,0 +1,1309 @@
+/*
+ * Copyright (C) 2017 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/*
+ * Broadcom SBA RAID Driver
+ *
+ * The Broadcom stream buffer accelerator (SBA) provides offloading
+ * capabilities for RAID operations. The SBA offload engine is accessible
+ * via Broadcom SoC specific ring manager. Two or more offload engines
+ * can share same Broadcom SoC specific ring manager due to this Broadcom
+ * SoC specific ring manager driver is implemented as a mailbox controller
+ * driver and offload engine drivers are implemented as mallbox clients.
+ *
+ * Typically, Broadcom SoC specific ring manager will implement larger
+ * number of hardware rings over one or more SBA hardware devices. By
+ * design, the internal buffer size of SBA hardware device is limited
+ * but all offload operations supported by SBA can be broken down into
+ * multiple small size requests and executed parallely on multiple SBA
+ * hardware devices for achieving high through-put.
+ *
+ * The Broadcom SBA RAID driver does not require any register programming
+ * except submitting request to SBA hardware device via mailbox channels.
+ * This driver implements a DMA device with one DMA channel using a set
+ * of mailbox channels provided by Broadcom SoC specific ring manager
+ * driver. To exploit parallelism (as described above), all DMA request
+ * coming to SBA RAID DMA channel are broken down to smaller requests
+ * and submitted to multiple mailbox channels in round-robin fashion.
+ * For having more SBA DMA channels, we can create more SBA device nodes
+ * in Broadcom SoC specific DTS based on number of hardware rings supported
+ * by Broadcom SoC ring manager.
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox/brcm-message.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/slab.h>
+#include <linux/raid/pq.h>
+
+#include "dmaengine.h"
+
+/* SBA command helper macros */
+#define SBA_DEC(_d, _s, _m) (((_d) >> (_s)) & (_m))
+#define SBA_ENC(_d, _v, _s, _m) \
+ do { \
+ (_d) &= ~((u64)(_m) << (_s)); \
+ (_d) |= (((u64)(_v) & (_m)) << (_s)); \
+ } while (0)
+
+/* SBA command related defines */
+#define SBA_TYPE_SHIFT 48
+#define SBA_TYPE_MASK 0x3
+#define SBA_TYPE_A 0x0
+#define SBA_TYPE_B 0x2
+#define SBA_TYPE_C 0x3
+#define SBA_USER_DEF_SHIFT 32
+#define SBA_USER_DEF_MASK 0xffff
+#define SBA_R_MDATA_SHIFT 24
+#define SBA_R_MDATA_MASK 0xff
+#define SBA_C_MDATA_MS_SHIFT 18
+#define SBA_C_MDATA_MS_MASK 0x3
+#define SBA_INT_SHIFT 17
+#define SBA_INT_MASK 0x1
+#define SBA_RESP_SHIFT 16
+#define SBA_RESP_MASK 0x1
+#define SBA_C_MDATA_SHIFT 8
+#define SBA_C_MDATA_MASK 0xff
+#define SBA_CMD_SHIFT 0
+#define SBA_CMD_MASK 0xf
+#define SBA_CMD_ZERO_ALL_BUFFERS 0x8
+#define SBA_CMD_LOAD_BUFFER 0x9
+#define SBA_CMD_XOR 0xa
+#define SBA_CMD_GALOIS_XOR 0xb
+#define SBA_CMD_ZERO_BUFFER 0x4
+#define SBA_CMD_WRITE_BUFFER 0xc
+
+/* SBA C_MDATA helper macros */
+#define SBA_C_MDATA_LOAD_VAL(__bnum0) ((__bnum0) & 0x3)
+#define SBA_C_MDATA_WRITE_VAL(__bnum0) ((__bnum0) & 0x3)
+#define SBA_C_MDATA_XOR_VAL(__bnum1, __bnum0) \
+ ({ u32 __v = ((__bnum0) & 0x3); \
+ __v |= ((__bnum1) & 0x3) << 2; \
+ __v; \
+ })
+#define SBA_C_MDATA_PQ_VAL(__dnum, __bnum1, __bnum0) \
+ ({ u32 __v = ((__bnum0) & 0x3); \
+ __v |= ((__bnum1) & 0x3) << 2; \
+ __v |= ((__dnum) & 0x1f) << 5; \
+ __v; \
+ })
+#define SBA_C_MDATA_LS(__c_mdata_val) ((__c_mdata_val) & 0xff)
+#define SBA_C_MDATA_MS(__c_mdata_val) (((__c_mdata_val) >> 8) & 0x3)
+
+/* Driver helper macros */
+#define to_sba_request(tx) \
+ container_of(tx, struct sba_request, tx)
+#define to_sba_device(dchan) \
+ container_of(dchan, struct sba_device, dma_chan)
+
+enum sba_request_state {
+ SBA_REQUEST_STATE_FREE = 1,
+ SBA_REQUEST_STATE_ALLOCED = 2,
+ SBA_REQUEST_STATE_PENDING = 3,
+ SBA_REQUEST_STATE_ACTIVE = 4,
+ SBA_REQUEST_STATE_COMPLETED = 5,
+ SBA_REQUEST_STATE_ABORTED = 6,
+};
+
+struct sba_request {
+ struct list_head node;
+ struct sba_device *sba;
+ enum sba_request_state state;
+ bool fence;
+ void *resp;
+ dma_addr_t resp_dma;
+ struct brcm_sba_command *cmds;
+ struct brcm_message *msgs;
+ struct brcm_message bmsg;
+ atomic_t msgs_pending_count;
+ struct dma_async_tx_descriptor tx;
+};
+
+enum sba_version {
+ SBA_VER_1 = 0,
+ SBA_VER_2
+};
+
+struct sba_device {
+ /* Underlying device */
+ struct device *dev;
+ /* DT configuration parameters */
+ enum sba_version ver;
+ u32 max_req;
+ u32 req_size;
+ /* Derived configuration parameters */
+ u32 hw_buf_size;
+ u32 hw_resp_size;
+ u32 max_pq_coefs;
+ u32 max_pq_srcs;
+ u32 max_msg_per_req;
+ u32 max_cmd_per_msg;
+ u32 max_cmd_per_req;
+ u32 max_xor_srcs;
+ u32 max_resp_pool_size;
+ u32 max_cmds_pool_size;
+ /* Maibox client and Mailbox channels */
+ struct mbox_client client;
+ int mchans_count;
+ atomic_t mchans_current;
+ struct mbox_chan **mchans;
+ struct device *mbox_dev;
+ /* DMA device and DMA channel */
+ struct dma_device dma_dev;
+ struct dma_chan dma_chan;
+ /* DMA channel resources */
+ void *resp_base;
+ dma_addr_t resp_dma_base;
+ void *cmds_base;
+ dma_addr_t cmds_dma_base;
+ spinlock_t reqs_lock;
+ struct sba_request *reqs;
+ bool reqs_fence;
+ struct list_head reqs_alloc_list;
+ struct list_head reqs_pending_list;
+ struct list_head reqs_active_list;
+ struct list_head reqs_completed_list;
+ struct list_head reqs_aborted_list;
+ struct list_head reqs_free_list;
+ int reqs_free_count;
+};
+
+/* ====== Channel resource management routines ===== */
+
+static struct sba_request *sba_alloc_request(struct sba_device *sba)
+{
+ unsigned long flags;
+ struct sba_request *req = NULL;
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+
+ if (!list_empty(&sba->reqs_free_list)) {
+ req = list_first_entry(&sba->reqs_free_list,
+ struct sba_request,
+ node);
+
+ req->state = SBA_REQUEST_STATE_ALLOCED;
+ req->fence = false;
+ atomic_set(&req->msgs_pending_count, 0);
+ list_move_tail(&req->node, &sba->reqs_alloc_list);
+ sba->reqs_free_count--;
+
+ dma_async_tx_descriptor_init(&req->tx, &sba->dma_chan);
+ }
+
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+
+ return req;
+}
+
+/* Note: Must be called with sba->reqs_lock held */
+static void _sba_pending_request(struct sba_device *sba,
+ struct sba_request *req)
+{
+ req->state = SBA_REQUEST_STATE_PENDING;
+ list_move_tail(&req->node, &sba->reqs_pending_list);
+ if (list_empty(&sba->reqs_active_list))
+ sba->reqs_fence = false;
+}
+
+/* Note: Must be called with sba->reqs_lock held */
+static bool _sba_active_request(struct sba_device *sba,
+ struct sba_request *req)
+{
+ if (list_empty(&sba->reqs_active_list))
+ sba->reqs_fence = false;
+ if (sba->reqs_fence)
+ return false;
+ req->state = SBA_REQUEST_STATE_ACTIVE;
+ list_move_tail(&req->node, &sba->reqs_active_list);
+ if (req->fence)
+ sba->reqs_fence = true;
+ return true;
+}
+
+/* Note: Must be called with sba->reqs_lock held */
+static void _sba_abort_request(struct sba_device *sba,
+ struct sba_request *req)
+{
+ req->state = SBA_REQUEST_STATE_ABORTED;
+ list_move_tail(&req->node, &sba->reqs_aborted_list);
+ if (list_empty(&sba->reqs_active_list))
+ sba->reqs_fence = false;
+}
+
+/* Note: Must be called with sba->reqs_lock held */
+static void _sba_free_request(struct sba_device *sba,
+ struct sba_request *req)
+{
+ req->state = SBA_REQUEST_STATE_FREE;
+ list_move_tail(&req->node, &sba->reqs_free_list);
+ if (list_empty(&sba->reqs_active_list))
+ sba->reqs_fence = false;
+ sba->reqs_free_count++;
+}
+
+static void sba_complete_request(struct sba_request *req)
+{
+ unsigned long flags;
+ struct sba_device *sba = req->sba;
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+ req->state = SBA_REQUEST_STATE_COMPLETED;
+ list_move_tail(&req->node, &sba->reqs_completed_list);
+ if (list_empty(&sba->reqs_active_list))
+ sba->reqs_fence = false;
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+}
+
+static void sba_free_request(struct sba_request *req)
+{
+ unsigned long flags;
+ struct sba_device *sba = req->sba;
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+ _sba_free_request(sba, req);
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+}
+
+static int sba_free_request_count(struct sba_device *sba)
+{
+ int ret;
+ unsigned long flags;
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+ ret = sba->reqs_free_count;
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+
+ return ret;
+}
+
+static void sba_cleanup_inflight_requests(struct sba_device *sba)
+{
+ unsigned long flags;
+ struct sba_request *req, *req1;
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+
+ /* Freeup all alloced request */
+ list_for_each_entry_safe(req, req1, &sba->reqs_alloc_list, node) {
+ _sba_free_request(sba, req);
+ }
+
+ /* Freeup all pending request */
+ list_for_each_entry_safe(req, req1, &sba->reqs_pending_list, node) {
+ if (req->bmsg.batch.msgs_queued < req->bmsg.batch.msgs_count)
+ /* Set partially-queued request as aborted */
+ _sba_abort_request(sba, req);
+ else
+ /* Freeup rest of the pending request */
+ _sba_free_request(sba, req);
+ }
+
+ /* Freeup all completed request */
+ list_for_each_entry_safe(req, req1, &sba->reqs_completed_list, node) {
+ _sba_free_request(sba, req);
+ }
+
+ /* Set all active requests as aborted */
+ list_for_each_entry_safe(req, req1, &sba->reqs_active_list, node) {
+ _sba_abort_request(sba, req);
+ }
+
+ /*
+ * Note: We expect that aborted request will be eventually
+ * freed by sba_receive_message()
+ */
+
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+}
+
+/* ====== DMAENGINE callbacks ===== */
+
+static int sba_alloc_chan_resources(struct dma_chan *dchan)
+{
+ /*
+ * We only have one channel so we have pre-alloced
+ * channel resources. Over here we just return number
+ * of free request.
+ */
+ return sba_free_request_count(to_sba_device(dchan));
+}
+
+static void sba_free_chan_resources(struct dma_chan *dchan)
+{
+ /*
+ * Channel resources are pre-alloced so we just free-up
+ * whatever we can so that we can re-use pre-alloced
+ * channel resources next time.
+ */
+ sba_cleanup_inflight_requests(to_sba_device(dchan));
+}
+
+static int sba_send_mbox_request(struct sba_device *sba,
+ struct sba_request *req)
+{
+ int mchans_idx, ret = 0;
+
+ /* Select mailbox channel in round-robin fashion */
+ mchans_idx = atomic_inc_return(&sba->mchans_current);
+ mchans_idx = mchans_idx % sba->mchans_count;
+
+ /* Send batch message for the request */
+ req->bmsg.batch.msgs_queued = 0;
+ ret = mbox_send_message(sba->mchans[mchans_idx], &req->bmsg);
+ if (ret < 0) {
+ dev_info(sba->dev, "channel %d message %d (total %d)",
+ mchans_idx, req->bmsg.batch.msgs_queued,
+ req->bmsg.batch.msgs_count);
+ dev_err(sba->dev, "send message failed with error %d", ret);
+ return ret;
+ }
+ ret = req->bmsg.error;
+ if (ret < 0) {
+ dev_info(sba->dev,
+ "mbox channel %d message %d (total %d)",
+ mchans_idx, req->bmsg.batch.msgs_queued,
+ req->bmsg.batch.msgs_count);
+ dev_err(sba->dev, "message error %d", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void sba_issue_pending(struct dma_chan *dchan)
+{
+ int ret;
+ unsigned long flags;
+ struct sba_request *req, *req1;
+ struct sba_device *sba = to_sba_device(dchan);
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+
+ /* Process all pending request */
+ list_for_each_entry_safe(req, req1, &sba->reqs_pending_list, node) {
+ /* Try to make request active */
+ if (!_sba_active_request(sba, req))
+ break;
+
+ /* Send request to mailbox channel */
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+ ret = sba_send_mbox_request(sba, req);
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+
+ /* If something went wrong then keep request pending */
+ if (ret < 0) {
+ _sba_pending_request(sba, req);
+ break;
+ }
+ }
+
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+}
+
+static dma_cookie_t sba_tx_submit(struct dma_async_tx_descriptor *tx)
+{
+ unsigned long flags;
+ dma_cookie_t cookie;
+ struct sba_request *req;
+ struct sba_device *sba;
+
+ if (unlikely(!tx))
+ return -EINVAL;
+
+ sba = to_sba_device(tx->chan);
+ req = to_sba_request(tx);
+
+ /* Assign cookie and mark request pending */
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+ cookie = dma_cookie_assign(tx);
+ _sba_pending_request(sba, req);
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+
+ /* Try to submit pending request */
+ sba_issue_pending(&sba->dma_chan);
+
+ return cookie;
+}
+
+static enum dma_status sba_tx_status(struct dma_chan *dchan,
+ dma_cookie_t cookie,
+ struct dma_tx_state *txstate)
+{
+ int mchan_idx;
+ enum dma_status ret;
+ struct sba_device *sba = to_sba_device(dchan);
+
+ ret = dma_cookie_status(dchan, cookie, txstate);
+ if (ret == DMA_COMPLETE)
+ return ret;
+
+ for (mchan_idx = 0; mchan_idx < sba->mchans_count; mchan_idx++)
+ mbox_client_peek_data(sba->mchans[mchan_idx]);
+
+ return dma_cookie_status(dchan, cookie, txstate);
+}
+
+static unsigned int sba_fillup_memcpy_msg(struct sba_request *req,
+ struct brcm_sba_command *cmds,
+ struct brcm_message *msg,
+ dma_addr_t msg_offset, size_t msg_len,
+ dma_addr_t dst, dma_addr_t src)
+{
+ u64 cmd;
+ u32 c_mdata;
+ struct brcm_sba_command *cmdsp = cmds;
+
+ /* Type-B command to load data into buf0 */
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_B, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ c_mdata = SBA_C_MDATA_LOAD_VAL(0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_LOAD_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_B;
+ cmdsp->data = src + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+
+ /* Type-A command to write buf0 */
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_A, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ SBA_ENC(cmd, 0x1, SBA_RESP_SHIFT, SBA_RESP_MASK);
+ c_mdata = SBA_C_MDATA_WRITE_VAL(0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_WRITE_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_A;
+ if (req->sba->hw_resp_size) {
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_RESP;
+ cmdsp->resp = req->resp_dma;
+ cmdsp->resp_len = req->sba->hw_resp_size;
+ }
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_OUTPUT;
+ cmdsp->data = dst + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+
+ /* Fillup brcm_message */
+ msg->type = BRCM_MESSAGE_SBA;
+ msg->sba.cmds = cmds;
+ msg->sba.cmds_count = cmdsp - cmds;
+ msg->ctx = req;
+ msg->error = 0;
+
+ return cmdsp - cmds;
+}
+
+static struct dma_async_tx_descriptor *
+sba_prep_dma_memcpy(struct dma_chan *dchan, dma_addr_t dst, dma_addr_t src,
+ size_t len, unsigned long flags)
+{
+ size_t msg_len;
+ dma_addr_t msg_offset = 0;
+ unsigned int msgs_count = 0, cmds_count, cmds_idx = 0;
+ struct sba_device *sba = to_sba_device(dchan);
+ struct sba_request *req = NULL;
+
+ /* Sanity checks */
+ if (unlikely(len > sba->req_size))
+ return NULL;
+
+ /* Alloc new request */
+ req = sba_alloc_request(sba);
+ if (!req)
+ return NULL;
+ req->fence = (flags & DMA_PREP_FENCE) ? true : false;
+
+ /* Fillup request messages */
+ while (len) {
+ msg_len = (len < sba->hw_buf_size) ? len : sba->hw_buf_size;
+ cmds_count = sba_fillup_memcpy_msg(req,
+ &req->cmds[cmds_idx],
+ &req->msgs[msgs_count],
+ msg_offset, msg_len, dst, src);
+ msgs_count++;
+ cmds_idx += cmds_count;
+ msg_offset += msg_len;
+ len -= msg_len;
+ }
+ req->bmsg.type = BRCM_MESSAGE_BATCH;
+ req->bmsg.batch.msgs = &req->msgs[0];
+ req->bmsg.batch.msgs_queued = 0;
+ req->bmsg.batch.msgs_count = msgs_count;
+ req->bmsg.ctx = req;
+ req->bmsg.error = 0;
+ atomic_set(&req->msgs_pending_count, msgs_count);
+
+ /* Init async_tx descriptor */
+ req->tx.flags = flags;
+ req->tx.cookie = -EBUSY;
+
+ return &req->tx;
+}
+
+static unsigned int sba_fillup_xor_msg(struct sba_request *req,
+ struct brcm_sba_command *cmds,
+ struct brcm_message *msg,
+ dma_addr_t msg_offset, size_t msg_len,
+ dma_addr_t dst, dma_addr_t *src, u32 src_cnt)
+{
+ u64 cmd;
+ u32 c_mdata;
+ unsigned int i;
+ struct brcm_sba_command *cmdsp = cmds;
+
+ /* Type-B command to load data into buf0 */
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_B, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ c_mdata = SBA_C_MDATA_LOAD_VAL(0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_LOAD_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_B;
+ cmdsp->data = src[0] + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+
+ /* Type-B commands to xor data with buf0 and put it back in buf0 */
+ for (i = 1; i < src_cnt; i++) {
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_B, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ c_mdata = SBA_C_MDATA_XOR_VAL(0, 0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_XOR, SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_B;
+ cmdsp->data = src[i] + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+ }
+
+ /* Type-A command to write buf0 */
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_A, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ SBA_ENC(cmd, 0x1, SBA_RESP_SHIFT, SBA_RESP_MASK);
+ c_mdata = SBA_C_MDATA_WRITE_VAL(0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_WRITE_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_A;
+ if (req->sba->hw_resp_size) {
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_RESP;
+ cmdsp->resp = req->resp_dma;
+ cmdsp->resp_len = req->sba->hw_resp_size;
+ }
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_OUTPUT;
+ cmdsp->data = dst + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+
+ /* Fillup brcm_message */
+ msg->type = BRCM_MESSAGE_SBA;
+ msg->sba.cmds = cmds;
+ msg->sba.cmds_count = cmdsp - cmds;
+ msg->ctx = req;
+ msg->error = 0;
+
+ return cmdsp - cmds;
+}
+
+static struct dma_async_tx_descriptor *
+sba_prep_dma_xor(struct dma_chan *dchan, dma_addr_t dst, dma_addr_t *src,
+ u32 src_cnt, size_t len, unsigned long flags)
+{
+ size_t msg_len;
+ dma_addr_t msg_offset = 0;
+ unsigned int msgs_count = 0, cmds_count, cmds_idx = 0;
+ struct sba_device *sba = to_sba_device(dchan);
+ struct sba_request *req = NULL;
+
+ /* Sanity checks */
+ if (unlikely(len > sba->req_size))
+ return NULL;
+ if (unlikely(src_cnt > sba->max_xor_srcs))
+ return NULL;
+
+ /* Alloc new request */
+ req = sba_alloc_request(sba);
+ if (!req)
+ return NULL;
+ req->fence = (flags & DMA_PREP_FENCE) ? true : false;
+
+ /* Fillup request messages */
+ while (len) {
+ msg_len = (len < sba->hw_buf_size) ? len : sba->hw_buf_size;
+ cmds_count = sba_fillup_xor_msg(req,
+ &req->cmds[cmds_idx],
+ &req->msgs[msgs_count],
+ msg_offset, msg_len,
+ dst, src, src_cnt);
+ msgs_count++;
+ cmds_idx += cmds_count;
+ msg_offset += msg_len;
+ len -= msg_len;
+ }
+ req->bmsg.type = BRCM_MESSAGE_BATCH;
+ req->bmsg.batch.msgs = &req->msgs[0];
+ req->bmsg.batch.msgs_queued = 0;
+ req->bmsg.batch.msgs_count = msgs_count;
+ req->bmsg.ctx = req;
+ req->bmsg.error = 0;
+ atomic_set(&req->msgs_pending_count, msgs_count);
+
+ /* Init async_tx descriptor */
+ req->tx.flags = flags;
+ req->tx.cookie = -EBUSY;
+
+ return &req->tx;
+}
+
+static unsigned int sba_fillup_pq_msg(struct sba_request *req,
+ bool pq_continue,
+ struct brcm_sba_command *cmds,
+ struct brcm_message *msg,
+ dma_addr_t msg_offset, size_t msg_len,
+ dma_addr_t *dst_p, dma_addr_t *dst_q,
+ const u8 *scf, dma_addr_t *src, u32 src_cnt)
+{
+ u64 cmd;
+ u32 c_mdata;
+ unsigned int i;
+ struct brcm_sba_command *cmdsp = cmds;
+
+ if (pq_continue) {
+ /* Type-B command to load old P into buf0 */
+ if (dst_p) {
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_B,
+ SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ c_mdata = SBA_C_MDATA_LOAD_VAL(0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_LOAD_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_B;
+ cmdsp->data = *dst_p + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+ }
+
+ /* Type-B command to load old Q into buf1 */
+ if (dst_q) {
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_B,
+ SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ c_mdata = SBA_C_MDATA_LOAD_VAL(1);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_LOAD_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_B;
+ cmdsp->data = *dst_q + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+ }
+ } else {
+ /* Type-A command to load data into buf0 */
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_A, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ SBA_ENC(cmd, SBA_CMD_ZERO_ALL_BUFFERS,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_A;
+ cmdsp++;
+ }
+
+ /* Type-B commands for generate P onto buf0 and Q onto buf1 */
+ for (i = 0; i < src_cnt; i++) {
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_B, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ c_mdata = SBA_C_MDATA_PQ_VAL(raid6_gflog[scf[i]], 1, 0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_C_MDATA_MS(c_mdata),
+ SBA_C_MDATA_MS_SHIFT, SBA_C_MDATA_MS_MASK);
+ SBA_ENC(cmd, SBA_CMD_GALOIS_XOR,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_B;
+ cmdsp->data = src[i] + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+ }
+
+ /* Type-A command to write buf0 */
+ if (dst_p) {
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_A, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ SBA_ENC(cmd, 0x1, SBA_RESP_SHIFT, SBA_RESP_MASK);
+ c_mdata = SBA_C_MDATA_WRITE_VAL(0);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_WRITE_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_A;
+ if (req->sba->hw_resp_size) {
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_RESP;
+ cmdsp->resp = req->resp_dma;
+ cmdsp->resp_len = req->sba->hw_resp_size;
+ }
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_OUTPUT;
+ cmdsp->data = *dst_p + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+ }
+
+ /* Type-A command to write buf1 */
+ if (dst_q) {
+ cmd = 0;
+ SBA_ENC(cmd, SBA_TYPE_A, SBA_TYPE_SHIFT, SBA_TYPE_MASK);
+ SBA_ENC(cmd, msg_len,
+ SBA_USER_DEF_SHIFT, SBA_USER_DEF_MASK);
+ SBA_ENC(cmd, 0x1, SBA_RESP_SHIFT, SBA_RESP_MASK);
+ c_mdata = SBA_C_MDATA_WRITE_VAL(1);
+ SBA_ENC(cmd, SBA_C_MDATA_LS(c_mdata),
+ SBA_C_MDATA_SHIFT, SBA_C_MDATA_MASK);
+ SBA_ENC(cmd, SBA_CMD_WRITE_BUFFER,
+ SBA_CMD_SHIFT, SBA_CMD_MASK);
+ cmdsp->cmd = cmd;
+ *cmdsp->cmd_dma = cpu_to_le64(cmd);
+ cmdsp->flags = BRCM_SBA_CMD_TYPE_A;
+ if (req->sba->hw_resp_size) {
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_RESP;
+ cmdsp->resp = req->resp_dma;
+ cmdsp->resp_len = req->sba->hw_resp_size;
+ }
+ cmdsp->flags |= BRCM_SBA_CMD_HAS_OUTPUT;
+ cmdsp->data = *dst_q + msg_offset;
+ cmdsp->data_len = msg_len;
+ cmdsp++;
+ }
+
+ /* Fillup brcm_message */
+ msg->type = BRCM_MESSAGE_SBA;
+ msg->sba.cmds = cmds;
+ msg->sba.cmds_count = cmdsp - cmds;
+ msg->ctx = req;
+ msg->error = 0;
+
+ return cmdsp - cmds;
+}
+
+static struct dma_async_tx_descriptor *
+sba_prep_dma_pq(struct dma_chan *dchan, dma_addr_t *dst, dma_addr_t *src,
+ u32 src_cnt, const u8 *scf, size_t len, unsigned long flags)
+{
+ u32 i;
+ size_t dst_count, msg_len;
+ unsigned int msgs_count = 0, cmds_count, cmds_idx = 0;
+ dma_addr_t *dst_p = NULL, *dst_q = NULL;
+ dma_addr_t msg_offset = 0;
+ struct sba_device *sba = to_sba_device(dchan);
+ struct sba_request *req = NULL;
+
+ /* Sanity checks */
+ if (unlikely(len > sba->req_size))
+ return NULL;
+ if (unlikely(src_cnt > sba->max_pq_srcs))
+ return NULL;
+ for (i = 0; i < src_cnt; i++)
+ if (sba->max_pq_coefs <= raid6_gflog[scf[i]])
+ return NULL;
+
+ /* Figure-out P and Q destination addresses */
+ dst_count = 0;
+ if (!(flags & DMA_PREP_PQ_DISABLE_P))
+ dst_p = &dst[dst_count++];
+ if (!(flags & DMA_PREP_PQ_DISABLE_Q))
+ dst_q = &dst[dst_count++];
+
+ /* Alloc new request */
+ req = sba_alloc_request(sba);
+ if (!req)
+ return NULL;
+ req->fence = (flags & DMA_PREP_FENCE) ? true : false;
+
+ /* Fillup request messages */
+ while (len) {
+ msg_len = (len < sba->hw_buf_size) ? len : sba->hw_buf_size;
+ cmds_count = sba_fillup_pq_msg(req, dmaf_continue(flags),
+ &req->cmds[cmds_idx],
+ &req->msgs[msgs_count],
+ msg_offset, msg_len,
+ dst_p, dst_q, scf, src, src_cnt);
+ msgs_count++;
+ cmds_idx += cmds_count;
+ msg_offset += msg_len;
+ len -= msg_len;
+ }
+ req->bmsg.type = BRCM_MESSAGE_BATCH;
+ req->bmsg.batch.msgs = &req->msgs[0];
+ req->bmsg.batch.msgs_queued = 0;
+ req->bmsg.batch.msgs_count = msgs_count;
+ req->bmsg.ctx = req;
+ req->bmsg.error = 0;
+ atomic_set(&req->msgs_pending_count, msgs_count);
+
+ /* Init async_tx descriptor */
+ req->tx.flags = flags;
+ req->tx.cookie = -EBUSY;
+
+ return &req->tx;
+}
+
+/* ====== Mailbox callbacks ===== */
+
+static void sba_dma_tx_actions(struct sba_request *req)
+{
+ struct dma_async_tx_descriptor *tx = &req->tx;
+
+ WARN_ON(tx->cookie < 0);
+
+ if (tx->cookie > 0) {
+ dma_cookie_complete(tx);
+
+ /* call the callback (must not sleep or submit new
+ * operations to this channel)
+ */
+ if (tx->callback)
+ tx->callback(tx->callback_param);
+
+ dma_descriptor_unmap(tx);
+ }
+
+ /* run dependent operations */
+ dma_run_dependencies(tx);
+}
+
+static void sba_dma_clean(struct sba_request *req)
+{
+ /* If waiting for 'ack' then move to completed list */
+ if (!async_tx_test_ack(&req->tx))
+ sba_complete_request(req);
+ else
+ sba_free_request(req);
+}
+
+static void sba_receive_message(struct mbox_client *cl, void *msg)
+{
+ unsigned long flags;
+ struct brcm_message *m = msg;
+ struct sba_request *req = m->ctx, *req1;
+ struct sba_device *sba = req->sba;
+
+ /* error count if message has error */
+ if (m->error < 0) {
+ dev_err(sba->dev, "%s got message with error %d",
+ dma_chan_name(&sba->dma_chan), m->error);
+ }
+
+ /* Wait for all messages to be completed */
+ if (atomic_dec_return(&req->msgs_pending_count))
+ return;
+
+ /* Update request */
+ if (req->state == SBA_REQUEST_STATE_ACTIVE) {
+ sba_dma_tx_actions(req);
+ sba_dma_clean(req);
+ } else {
+ sba_free_request(req);
+ }
+
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+
+ /* Re-check all completed request waiting for 'ack' */
+ list_for_each_entry_safe(req, req1, &sba->reqs_completed_list, node) {
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+ sba_dma_tx_actions(req);
+ sba_dma_clean(req);
+ spin_lock_irqsave(&sba->reqs_lock, flags);
+ }
+
+ spin_unlock_irqrestore(&sba->reqs_lock, flags);
+
+ /* Try to submit pending request */
+ sba_issue_pending(&sba->dma_chan);
+}
+
+/* ====== Platform driver routines ===== */
+
+static int sba_prealloc_channel_resources(struct sba_device *sba)
+{
+ int i, j, p, ret = 0;
+ struct sba_request *req = NULL;
+
+ sba->resp_base = dma_alloc_coherent(sba->dma_dev.dev,
+ sba->max_resp_pool_size,
+ &sba->resp_dma_base, GFP_KERNEL);
+ if (!sba->resp_base)
+ return -ENOMEM;
+
+ sba->cmds_base = dma_alloc_coherent(sba->dma_dev.dev,
+ sba->max_cmds_pool_size,
+ &sba->cmds_dma_base, GFP_KERNEL);
+ if (!sba->cmds_base) {
+ ret = -ENOMEM;
+ goto fail_free_resp_pool;
+ }
+
+ spin_lock_init(&sba->reqs_lock);
+ sba->reqs_fence = false;
+ INIT_LIST_HEAD(&sba->reqs_alloc_list);
+ INIT_LIST_HEAD(&sba->reqs_pending_list);
+ INIT_LIST_HEAD(&sba->reqs_active_list);
+ INIT_LIST_HEAD(&sba->reqs_completed_list);
+ INIT_LIST_HEAD(&sba->reqs_aborted_list);
+ INIT_LIST_HEAD(&sba->reqs_free_list);
+
+ sba->reqs = devm_kcalloc(sba->dev, sba->max_req,
+ sizeof(*req), GFP_KERNEL);
+ if (!sba->reqs) {
+ ret = -ENOMEM;
+ goto fail_free_cmds_pool;
+ }
+
+ for (i = 0, p = 0; i < sba->max_req; i++) {
+ req = &sba->reqs[i];
+ INIT_LIST_HEAD(&req->node);
+ req->sba = sba;
+ req->state = SBA_REQUEST_STATE_FREE;
+ req->fence = false;
+ req->resp = sba->resp_base + p;
+ req->resp_dma = sba->resp_dma_base + p;
+ p += sba->hw_resp_size;
+ req->cmds = devm_kcalloc(sba->dev, sba->max_cmd_per_req,
+ sizeof(*req->cmds), GFP_KERNEL);
+ if (!req->cmds) {
+ ret = -ENOMEM;
+ goto fail_free_cmds_pool;
+ }
+ for (j = 0; j < sba->max_cmd_per_req; j++) {
+ req->cmds[j].cmd = 0;
+ req->cmds[j].cmd_dma = sba->cmds_base +
+ (i * sba->max_cmd_per_req + j) * sizeof(u64);
+ req->cmds[j].cmd_dma_addr = sba->cmds_dma_base +
+ (i * sba->max_cmd_per_req + j) * sizeof(u64);
+ req->cmds[j].flags = 0;
+ }
+ req->msgs = devm_kcalloc(sba->dev, sba->max_msg_per_req,
+ sizeof(*req->msgs), GFP_KERNEL);
+ if (!req->msgs) {
+ ret = -ENOMEM;
+ goto fail_free_cmds_pool;
+ }
+ memset(&req->bmsg, 0, sizeof(req->bmsg));
+ atomic_set(&req->msgs_pending_count, 0);
+ dma_async_tx_descriptor_init(&req->tx, &sba->dma_chan);
+ req->tx.tx_submit = sba_tx_submit;
+ req->tx.phys = req->resp_dma;
+ list_add_tail(&req->node, &sba->reqs_free_list);
+ }
+
+ sba->reqs_free_count = sba->max_req;
+
+ return 0;
+
+fail_free_cmds_pool:
+ dma_free_coherent(sba->dma_dev.dev,
+ sba->max_cmds_pool_size,
+ sba->cmds_base, sba->cmds_dma_base);
+fail_free_resp_pool:
+ dma_free_coherent(sba->dma_dev.dev,
+ sba->max_resp_pool_size,
+ sba->resp_base, sba->resp_dma_base);
+ return ret;
+}
+
+static void sba_freeup_channel_resources(struct sba_device *sba)
+{
+ dmaengine_terminate_all(&sba->dma_chan);
+ dma_free_coherent(sba->dma_dev.dev, sba->max_cmds_pool_size,
+ sba->cmds_base, sba->cmds_dma_base);
+ dma_free_coherent(sba->dma_dev.dev, sba->max_resp_pool_size,
+ sba->resp_base, sba->resp_dma_base);
+ sba->resp_base = NULL;
+ sba->resp_dma_base = 0;
+}
+
+static int sba_async_register(struct sba_device *sba)
+{
+ int ret;
+ struct dma_device *dma_dev = &sba->dma_dev;
+
+ /* Initialize DMA channel cookie */
+ sba->dma_chan.device = dma_dev;
+ dma_cookie_init(&sba->dma_chan);
+
+ /* Initialize DMA device capability mask */
+ dma_cap_zero(dma_dev->cap_mask);
+ dma_cap_set(DMA_MEMCPY, dma_dev->cap_mask);
+ dma_cap_set(DMA_XOR, dma_dev->cap_mask);
+ dma_cap_set(DMA_PQ, dma_dev->cap_mask);
+
+ /*
+ * Set mailbox channel device as the base device of
+ * our dma_device because the actual memory accesses
+ * will be done by mailbox controller
+ */
+ dma_dev->dev = sba->mbox_dev;
+
+ /* Set base prep routines */
+ dma_dev->device_alloc_chan_resources = sba_alloc_chan_resources;
+ dma_dev->device_free_chan_resources = sba_free_chan_resources;
+ dma_dev->device_issue_pending = sba_issue_pending;
+ dma_dev->device_tx_status = sba_tx_status;
+
+ /* Set memcpy routines and capability */
+ if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask))
+ dma_dev->device_prep_dma_memcpy = sba_prep_dma_memcpy;
+
+ /* Set xor routines and capability */
+ if (dma_has_cap(DMA_XOR, dma_dev->cap_mask)) {
+ dma_dev->device_prep_dma_xor = sba_prep_dma_xor;
+ dma_dev->max_xor = sba->max_xor_srcs;
+ }
+
+ /* Set pq routines and capability */
+ if (dma_has_cap(DMA_PQ, dma_dev->cap_mask)) {
+ dma_dev->device_prep_dma_pq = sba_prep_dma_pq;
+ dma_set_maxpq(dma_dev, sba->max_pq_srcs, 0);
+ dma_set_maxpqcoef(dma_dev, sba->max_pq_coefs);
+ }
+
+ /* Initialize DMA device channel list */
+ INIT_LIST_HEAD(&dma_dev->channels);
+ list_add_tail(&sba->dma_chan.device_node, &dma_dev->channels);
+
+ /* Register with Linux async DMA framework*/
+ ret = dma_async_device_register(dma_dev);
+ if (ret) {
+ dev_err(sba->dev, "async device register error %d", ret);
+ return ret;
+ }
+
+ dev_info(sba->dev, "%s capabilities: %s%s%s\n",
+ dma_chan_name(&sba->dma_chan),
+ dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask) ? "memcpy " : "",
+ dma_has_cap(DMA_XOR, dma_dev->cap_mask) ? "xor " : "",
+ dma_has_cap(DMA_PQ, dma_dev->cap_mask) ? "pq " : "");
+
+ return 0;
+}
+
+static int sba_probe(struct platform_device *pdev)
+{
+ int i, ret = 0, mchans_count;
+ struct sba_device *sba;
+
+ /* Allocate main SBA struct */
+ sba = devm_kzalloc(&pdev->dev, sizeof(*sba), GFP_KERNEL);
+ if (!sba)
+ return -ENOMEM;
+
+ sba->dev = &pdev->dev;
+ platform_set_drvdata(pdev, sba);
+
+ /* Determine SBA version from DT compatible string */
+ if (of_device_is_compatible(sba->dev->of_node, "brcm,iproc-sba"))
+ sba->ver = SBA_VER_1;
+ else if (of_device_is_compatible(sba->dev->of_node,
+ "brcm,iproc-sba-v2"))
+ sba->ver = SBA_VER_2;
+ else
+ return -ENODEV;
+
+ /* Derived Configuration parameters */
+ switch (sba->ver) {
+ case SBA_VER_1:
+ sba->max_req = 128;
+ sba->req_size = PAGE_SIZE;
+ sba->hw_buf_size = 4096;
+ sba->hw_resp_size = 8;
+ sba->max_pq_coefs = 6;
+ sba->max_pq_srcs = 6;
+ break;
+ case SBA_VER_2:
+ sba->max_req = 128;
+ sba->req_size = PAGE_SIZE;
+ sba->hw_buf_size = 4096;
+ sba->hw_resp_size = 8;
+ sba->max_pq_coefs = 30;
+ /*
+ * We can support max_pq_srcs == max_pq_coefs because
+ * we are limited by number of SBA commands that we can
+ * fit in one message for underlying ring manager HW.
+ */
+ sba->max_pq_srcs = 12;
+ break;
+ default:
+ return -EINVAL;
+ }
+ sba->max_msg_per_req = sba->req_size / sba->hw_buf_size;
+ if ((sba->max_msg_per_req * sba->hw_buf_size) < sba->req_size)
+ sba->max_msg_per_req++;
+ sba->max_cmd_per_msg = sba->max_pq_srcs + 3;
+ sba->max_cmd_per_req = sba->max_msg_per_req * sba->max_cmd_per_msg;
+ sba->max_xor_srcs = sba->max_cmd_per_msg - 1;
+ sba->max_resp_pool_size = sba->max_req * sba->hw_resp_size;
+ sba->max_cmds_pool_size = sba->max_req *
+ sba->max_cmd_per_req * sizeof(u64);
+
+ /* Setup mailbox client */
+ sba->client.dev = &pdev->dev;
+ sba->client.rx_callback = sba_receive_message;
+ sba->client.tx_block = false;
+ sba->client.knows_txdone = false;
+ sba->client.tx_tout = 0;
+
+ /* Number of channels equals number of mailbox channels */
+ ret = of_count_phandle_with_args(pdev->dev.of_node,
+ "mboxes", "#mbox-cells");
+ if (ret <= 0)
+ return -ENODEV;
+ mchans_count = ret;
+ sba->mchans_count = 0;
+ atomic_set(&sba->mchans_current, 0);
+
+ /* Allocate mailbox channel array */
+ sba->mchans = devm_kcalloc(&pdev->dev, sba->mchans_count,
+ sizeof(*sba->mchans), GFP_KERNEL);
+ if (!sba->mchans)
+ return -ENOMEM;
+
+ /* Request mailbox channels */
+ for (i = 0; i < mchans_count; i++) {
+ sba->mchans[i] = mbox_request_channel(&sba->client, i);
+ if (IS_ERR(sba->mchans[i])) {
+ ret = PTR_ERR(sba->mchans[i]);
+ goto fail_free_mchans;
+ }
+ sba->mchans_count++;
+ }
+
+ /* Find-out underlying mailbox device */
+ sba->mbox_dev = mbox_channel_device(sba->mchans[0]);
+ if (IS_ERR(sba->mbox_dev)) {
+ ret = PTR_ERR(sba->mbox_dev);
+ goto fail_free_mchans;
+ }
+
+ /* All mailbox channels should be of same ring manager device */
+ for (i = 1; i < mchans_count; i++) {
+ if (mbox_channel_device(sba->mchans[i]) != sba->mbox_dev) {
+ ret = -EINVAL;
+ goto fail_free_mchans;
+ }
+ }
+
+ /* Register DMA device with linux async framework */
+ ret = sba_async_register(sba);
+ if (ret)
+ goto fail_free_mchans;
+
+ /* Prealloc channel resource */
+ ret = sba_prealloc_channel_resources(sba);
+ if (ret)
+ goto fail_async_dev_unreg;
+
+ /* Print device info */
+ dev_info(sba->dev, "%s using SBAv%d and %d mailbox channels",
+ dma_chan_name(&sba->dma_chan), sba->ver+1,
+ sba->mchans_count);
+
+ return 0;
+
+fail_async_dev_unreg:
+ dma_async_device_unregister(&sba->dma_dev);
+fail_free_mchans:
+ for (i = 0; i < sba->mchans_count; i++)
+ mbox_free_channel(sba->mchans[i]);
+ return ret;
+}
+
+static int sba_remove(struct platform_device *pdev)
+{
+ int i;
+ struct sba_device *sba = platform_get_drvdata(pdev);
+
+ sba_freeup_channel_resources(sba);
+
+ dma_async_device_unregister(&sba->dma_dev);
+
+ for (i = 0; i < sba->mchans_count; i++)
+ mbox_free_channel(sba->mchans[i]);
+
+ return 0;
+}
+
+static const struct of_device_id sba_of_match[] = {
+ { .compatible = "brcm,iproc-sba", },
+ { .compatible = "brcm,iproc-sba-v2", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, sba_of_match);
+
+static struct platform_driver sba_driver = {
+ .probe = sba_probe,
+ .remove = sba_remove,
+ .driver = {
+ .name = "bcm-sba-raid",
+ .of_match_table = sba_of_match,
+ },
+};
+module_platform_driver(sba_driver);
+
+MODULE_DESCRIPTION("Broadcom SBA RAID driver");
+MODULE_AUTHOR("Anup Patel <anup.patel@broadcom.com>");
+MODULE_LICENSE("GPL v2");
--
2.7.4
^ permalink raw reply related
* Re: 4.10 aesni-intel no longer having lrw/ablk_helper dependencies?
From: Herbert Xu @ 2017-02-02 5:14 UTC (permalink / raw)
To: Arkadiusz Miśkiewicz; +Cc: Eric Biggers, linux-crypto
In-Reply-To: <201702011708.03940.arekm@maven.pl>
On Wed, Feb 01, 2017 at 05:08:03PM +0100, Arkadiusz Miśkiewicz wrote:
>
> q: Will later loading of pcbc (so intel-aseni loaded from initrd, no pcbc
> available, rootfs gets mounted; pcbc is loaded) enable its "functionality" for
> intel-aesni just like it would be available at intel-aesni load time ?
No it won't but it's no big deal as pcbc(aes) isn't actually used
anywhere so it's really just an example of using fpu. AFAIK pcbc
is only ever used with fcrypt.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH -stable] crypto: ccm - deal with CTR ciphers that honour iv_out
From: Herbert Xu @ 2017-02-02 5:13 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto@vger.kernel.org
In-Reply-To: <CAKv+Gu9Em15JVQOo9civMMWOtwBR0R_a+K0P1Y-JOZJU5zPtnw@mail.gmail.com>
On Wed, Feb 01, 2017 at 08:08:09PM +0000, Ard Biesheuvel wrote:
>
> Could you please forward this patch to Linus as well? I noticed that the patch
Sure, I will do that.
> crypto: arm64/aes-blk - honour iv_out requirement in CBC and CTR modes
>
> is now in mainline, which means CCM is now broken on arm64, given that
> the iv_out requirement for CTR apparently isn't honored by *any*
> implementation, and CCM wrongly assumes that req->iv retains its value
> across the call into the CTR skcipher
Hmm, I wonder why we don't see this breakage with the generic
CTR as it seems to do exactly the same thing.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH 3/6] async_tx: Handle DMA devices having support for fewer PQ coefficients
From: Dan Williams @ 2017-02-02 6:01 UTC (permalink / raw)
To: Anup Patel
Cc: Vinod Koul, Rob Herring, Mark Rutland, Herbert Xu,
David S . Miller, Jassi Brar, Ray Jui, Scott Branden, Jon Mason,
Rob Rice, bcm-kernel-feedback-list, dmaengine@vger.kernel.org,
devicetree, linux-arm-kernel@lists.infradead.org,
linux-kernel@vger.kernel.org, linux-crypto, linux-raid
In-Reply-To: <1486010836-25228-4-git-send-email-anup.patel@broadcom.com>
On Wed, Feb 1, 2017 at 8:47 PM, Anup Patel <anup.patel@broadcom.com> wrote:
> The DMAENGINE framework assumes that if PQ offload is supported by a
> DMA device then all 256 PQ coefficients are supported. This assumption
> does not hold anymore because we now have BCM-SBA-RAID offload engine
> which supports PQ offload with limited number of PQ coefficients.
>
> This patch extends async_tx APIs to handle DMA devices with support
> for fewer PQ coefficients.
>
> Signed-off-by: Anup Patel <anup.patel@broadcom.com>
> Reviewed-by: Scott Branden <scott.branden@broadcom.com>
> ---
> crypto/async_tx/async_pq.c | 3 +++
> crypto/async_tx/async_raid6_recov.c | 12 ++++++++++--
> include/linux/dmaengine.h | 19 +++++++++++++++++++
> include/linux/raid/pq.h | 3 +++
> 4 files changed, 35 insertions(+), 2 deletions(-)
So, I hate the way async_tx does these checks on each operation, and
it's ok for me to say that because it's my fault. Really it's md that
should be validating engine offload capabilities once at the beginning
of time. I'd rather we move in that direction than continue to pile
onto a bad design.
^ permalink raw reply
* Re: [RFC PATCH] crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic
From: Eric Biggers @ 2017-02-02 6:47 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, herbert
In-Reply-To: <1485785489-5116-1-git-send-email-ard.biesheuvel@linaro.org>
On Mon, Jan 30, 2017 at 02:11:29PM +0000, Ard Biesheuvel wrote:
> Instead of unconditionally forcing 4 byte alignment for all generic
> chaining modes that rely on crypto_xor() or crypto_inc() (which may
> result in unnecessary copying of data when the underlying hardware
> can perform unaligned accesses efficiently), make those functions
> deal with unaligned input explicitly, but only if the Kconfig symbol
> HAVE_EFFICIENT_UNALIGNED_ACCESS is set. This will allow us to drop
> the alignmasks from the CBC, CMAC, CTR, CTS, PCBC and SEQIV drivers.
>
> For crypto_inc(), this simply involves making the 4-byte stride
> conditional on HAVE_EFFICIENT_UNALIGNED_ACCESS being set, given that
> it typically operates on 16 byte buffers.
>
> For crypto_xor(), an algorithm is implemented that simply runs through
> the input using the largest strides possible if unaligned accesses are
> allowed. If they are not, an optimal sequence of memory accesses is
> emitted that takes the relative alignment of the input buffers into
> account, e.g., if the relative misalignment of dst and src is 4 bytes,
> the entire xor operation will be completed using 4 byte loads and stores
> (modulo unaligned bits at the start and end). Note that all expressions
> involving startalign and misalign are simply eliminated by the compiler
> if HAVE_EFFICIENT_UNALIGNED_ACCESS is defined.
>
Hi Ard,
This is a good idea, and I think it was error-prone to be requiring 4-byte
alignment always, and also inefficient on many architectures.
The new crypto_inc() looks fine, but the new crypto_xor() is quite complicated.
I'm wondering whether it has to be that way, especially since it seems to most
commonly be used on very small input buffers, e.g. 8 or 16-byte blocks. There
are a couple trivial ways it could be simplified, e.g. using 'dst' and 'src'
directly instead of 'a' and 'b' (which also seems to improve code generation by
getting rid of the '+= len & ~mask' parts), or using sizeof(long) directly
instead of 'size' and 'mask'.
But also when I tried testing the proposed crypto_xor() on MIPS, it didn't work
correctly on a misaligned buffer. With startalign=1, it did one iteration of
the following loop and then exited with startalign=0 and entered the "unsigned
long at a time" loop, which is incorrect since at that point the buffers were
not yet fully aligned:
> do {
> if (len < sizeof(u8))
> break;
>
> if (len >= size && !(startalign & 1) && !(misalign & 1))
> break;
>
> *dst++ ^= *src++;
> len -= sizeof(u8);
> startalign &= ~sizeof(u8);
> } while (misalign & 1);
I think it would need to do instead:
startalign += sizeof(u8);
startalign %= sizeof(unsigned long);
But I am wondering whether you considered something simpler, using the
get_unaligned/put_unaligned helpers, maybe even using a switch statement for the
last (sizeof(long) - 1) bytes so it can be compiled as a jump table. Something
like this:
#define xor_unaligned(dst, src) \
put_unaligned(get_unaligned(dst) ^ get_unaligned(src), (dst))
void crypto_xor(u8 *dst, const u8 *src, unsigned int len)
{
while (len >= sizeof(unsigned long)) {
xor_unaligned((unsigned long *)dst, (unsigned long *)src);
dst += sizeof(unsigned long);
src += sizeof(unsigned long);
len -= sizeof(unsigned long);
}
switch (len) {
#ifdef CONFIG_64BIT
case 7:
dst[6] ^= src[6];
/* fall through */
case 6:
xor_unaligned((u16 *)&dst[4], (u16 *)&src[4]);
goto len_4;
case 5:
dst[4] ^= src[4];
/* fall through */
case 4:
len_4:
xor_unaligned((u32 *)dst, (u32 *)src);
break;
#endif
case 3:
dst[2] ^= src[2];
/* fall through */
case 2:
xor_unaligned((u16 *)dst, (u16 *)src);
break;
case 1:
dst[0] ^= src[0];
break;
}
}
That would seem like a better choice for small buffers, which seems to be the
more common case. It should generate slightly faster code on architectures with
fast unaligned access like x86_64, while still being sufficient on architectures
without --- perhaps even faster, since it wouldn't have as many branches.
Eric
^ permalink raw reply
* Re: [RFC PATCH v2 4/4] crypto: aes - add generic time invariant AES for CTR/CCM/GCM
From: Eric Biggers @ 2017-02-02 7:38 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto, linux-arm-kernel, herbert
In-Reply-To: <1485646413-17491-5-git-send-email-ard.biesheuvel@linaro.org>
Hi Ard,
On Sat, Jan 28, 2017 at 11:33:33PM +0000, Ard Biesheuvel wrote:
>
> Note that this only implements AES encryption, which is all we need
> for CTR and CBC-MAC. AES decryption can easily be implemented in a
> similar way, but is significantly more costly.
Is the expectation of decryption being more costly the only reason why "aes-ti"
couldn't be a "cipher" algorithm, allowing it to automatically be used by the
existing templates for CTR, CBC-MAC, CBC, ECB, XTS, CMAC, etc.? It doesn't seem
to do anything expensive on a per-block basis like loading SSE registers, so it
seems it would fit better as a "cipher" algorithm if at all possible. Then
there would be no need to implement all these modes yet again.
Also, what would be the feasibility of simply replacing aes-generic with the
time-invariant implementation, rather than offering two implementations and
requiring users to choose one, usually without the needed expertise?
> +
> +/*
> + * Emit the sbox as __weak with external linkage to prevent the compiler
> + * from doing constant folding on sbox references involving fixed indexes.
> + */
> +__weak const u8 __cacheline_aligned __aesti_sbox[] = {
Did you consider marking it 'volatile' instead?
> +static int aesti_set_key(struct aes_ti_ctx *ctx, const u8 *in_key,
> + unsigned int key_len)
> +{
> + struct crypto_aes_ctx rk;
> + int err;
> +
> + err = crypto_aes_expand_key(&rk, in_key, key_len);
> + if (err)
> + return err;
crypto_aes_expand_key() assumes that the key is u32-aligned; I don't think
that's guaranteed here.
Eric
^ permalink raw reply
* Re: [RFC PATCH v2 4/4] crypto: aes - add generic time invariant AES for CTR/CCM/GCM
From: Ard Biesheuvel @ 2017-02-02 7:48 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, Herbert Xu
In-Reply-To: <20170202073824.GC582@zzz>
On 2 February 2017 at 07:38, Eric Biggers <ebiggers3@gmail.com> wrote:
> Hi Ard,
>
> On Sat, Jan 28, 2017 at 11:33:33PM +0000, Ard Biesheuvel wrote:
>>
>> Note that this only implements AES encryption, which is all we need
>> for CTR and CBC-MAC. AES decryption can easily be implemented in a
>> similar way, but is significantly more costly.
>
> Is the expectation of decryption being more costly the only reason why "aes-ti"
> couldn't be a "cipher" algorithm, allowing it to automatically be used by the
> existing templates for CTR, CBC-MAC, CBC, ECB, XTS, CMAC, etc.?
Yes.
> It doesn't seem
> to do anything expensive on a per-block basis like loading SSE registers, so it
> seems it would fit better as a "cipher" algorithm if at all possible. Then
> there would be no need to implement all these modes yet again.
>
True.
> Also, what would be the feasibility of simply replacing aes-generic with the
> time-invariant implementation, rather than offering two implementations and
> requiring users to choose one, usually without the needed expertise?
>
Well, it is a policy decision whether you want the best performance or
reduced correlation between timing and the input, so there is no way
to make everybody happy by replacing one with the other. But I can
certainly implement is as a cipher, and we can take the discussion
from there.
>> +
>> +/*
>> + * Emit the sbox as __weak with external linkage to prevent the compiler
>> + * from doing constant folding on sbox references involving fixed indexes.
>> + */
>> +__weak const u8 __cacheline_aligned __aesti_sbox[] = {
>
> Did you consider marking it 'volatile' instead?
>
I did not, and I expect the result to be the same. I can replace it if
it matters.
>> +static int aesti_set_key(struct aes_ti_ctx *ctx, const u8 *in_key,
>> + unsigned int key_len)
>> +{
>> + struct crypto_aes_ctx rk;
>> + int err;
>> +
>> + err = crypto_aes_expand_key(&rk, in_key, key_len);
>> + if (err)
>> + return err;
>
> crypto_aes_expand_key() assumes that the key is u32-aligned; I don't think
> that's guaranteed here.
>
No, I don't think so. AFAICT it expects the round keys to be u32
aligned, which is guaranteed due to the fact that struct
crypto_aes_ctx encapsulates an array of u32
Regards,
Ard.
^ permalink raw reply
* Re: [RFC PATCH] crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic
From: Ard Biesheuvel @ 2017-02-02 7:52 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto@vger.kernel.org, Herbert Xu
In-Reply-To: <20170202064716.GB582@zzz>
On 2 February 2017 at 06:47, Eric Biggers <ebiggers3@gmail.com> wrote:
> On Mon, Jan 30, 2017 at 02:11:29PM +0000, Ard Biesheuvel wrote:
>> Instead of unconditionally forcing 4 byte alignment for all generic
>> chaining modes that rely on crypto_xor() or crypto_inc() (which may
>> result in unnecessary copying of data when the underlying hardware
>> can perform unaligned accesses efficiently), make those functions
>> deal with unaligned input explicitly, but only if the Kconfig symbol
>> HAVE_EFFICIENT_UNALIGNED_ACCESS is set. This will allow us to drop
>> the alignmasks from the CBC, CMAC, CTR, CTS, PCBC and SEQIV drivers.
>>
>> For crypto_inc(), this simply involves making the 4-byte stride
>> conditional on HAVE_EFFICIENT_UNALIGNED_ACCESS being set, given that
>> it typically operates on 16 byte buffers.
>>
>> For crypto_xor(), an algorithm is implemented that simply runs through
>> the input using the largest strides possible if unaligned accesses are
>> allowed. If they are not, an optimal sequence of memory accesses is
>> emitted that takes the relative alignment of the input buffers into
>> account, e.g., if the relative misalignment of dst and src is 4 bytes,
>> the entire xor operation will be completed using 4 byte loads and stores
>> (modulo unaligned bits at the start and end). Note that all expressions
>> involving startalign and misalign are simply eliminated by the compiler
>> if HAVE_EFFICIENT_UNALIGNED_ACCESS is defined.
>>
>
> Hi Ard,
>
> This is a good idea, and I think it was error-prone to be requiring 4-byte
> alignment always, and also inefficient on many architectures.
>
> The new crypto_inc() looks fine, but the new crypto_xor() is quite complicated.
> I'm wondering whether it has to be that way, especially since it seems to most
> commonly be used on very small input buffers, e.g. 8 or 16-byte blocks. There
> are a couple trivial ways it could be simplified, e.g. using 'dst' and 'src'
> directly instead of 'a' and 'b' (which also seems to improve code generation by
> getting rid of the '+= len & ~mask' parts), or using sizeof(long) directly
> instead of 'size' and 'mask'.
>
> But also when I tried testing the proposed crypto_xor() on MIPS, it didn't work
> correctly on a misaligned buffer. With startalign=1, it did one iteration of
> the following loop and then exited with startalign=0 and entered the "unsigned
> long at a time" loop, which is incorrect since at that point the buffers were
> not yet fully aligned:
>
Right. I knew it was convoluted but I thought that was justified by
its correctness :-)
>> do {
>> if (len < sizeof(u8))
>> break;
>>
>> if (len >= size && !(startalign & 1) && !(misalign & 1))
>> break;
>>
>> *dst++ ^= *src++;
>> len -= sizeof(u8);
>> startalign &= ~sizeof(u8);
>> } while (misalign & 1);
>
> I think it would need to do instead:
>
> startalign += sizeof(u8);
> startalign %= sizeof(unsigned long);
>
> But I am wondering whether you considered something simpler, using the
> get_unaligned/put_unaligned helpers, maybe even using a switch statement for the
> last (sizeof(long) - 1) bytes so it can be compiled as a jump table. Something
> like this:
>
> #define xor_unaligned(dst, src) \
> put_unaligned(get_unaligned(dst) ^ get_unaligned(src), (dst))
>
> void crypto_xor(u8 *dst, const u8 *src, unsigned int len)
> {
> while (len >= sizeof(unsigned long)) {
> xor_unaligned((unsigned long *)dst, (unsigned long *)src);
> dst += sizeof(unsigned long);
> src += sizeof(unsigned long);
> len -= sizeof(unsigned long);
> }
>
> switch (len) {
> #ifdef CONFIG_64BIT
> case 7:
> dst[6] ^= src[6];
> /* fall through */
> case 6:
> xor_unaligned((u16 *)&dst[4], (u16 *)&src[4]);
> goto len_4;
> case 5:
> dst[4] ^= src[4];
> /* fall through */
> case 4:
> len_4:
> xor_unaligned((u32 *)dst, (u32 *)src);
> break;
> #endif
> case 3:
> dst[2] ^= src[2];
> /* fall through */
> case 2:
> xor_unaligned((u16 *)dst, (u16 *)src);
> break;
> case 1:
> dst[0] ^= src[0];
> break;
> }
> }
>
> That would seem like a better choice for small buffers, which seems to be the
> more common case. It should generate slightly faster code on architectures with
> fast unaligned access like x86_64, while still being sufficient on architectures
> without --- perhaps even faster, since it wouldn't have as many branches.
>
Well, what I tried to deal with explicitly is misaligned dst and src
by, e.g., 4 bytes. In my implementation, the idea was that it would
run through the entire input using 32-bit loads and stores, and the
standard unaligned accessors always take the hit of bytewise accesses
on architectures that don't have hardware support.
But I have an idea how I could simplify this, stay tuned please
^ permalink raw reply
* Re: [RFC PATCH v2 4/4] crypto: aes - add generic time invariant AES for CTR/CCM/GCM
From: Ard Biesheuvel @ 2017-02-02 7:55 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, Herbert Xu
In-Reply-To: <CAKv+Gu-0WK7AkgcTsFNUUCeyaafQiWziSuYzY6VEWAPZ41E1ag@mail.gmail.com>
On 2 February 2017 at 07:48, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 2 February 2017 at 07:38, Eric Biggers <ebiggers3@gmail.com> wrote:
>> Hi Ard,
>>
>> On Sat, Jan 28, 2017 at 11:33:33PM +0000, Ard Biesheuvel wrote:
>>>
>>> Note that this only implements AES encryption, which is all we need
>>> for CTR and CBC-MAC. AES decryption can easily be implemented in a
>>> similar way, but is significantly more costly.
>>
>> Is the expectation of decryption being more costly the only reason why "aes-ti"
>> couldn't be a "cipher" algorithm, allowing it to automatically be used by the
>> existing templates for CTR, CBC-MAC, CBC, ECB, XTS, CMAC, etc.?
>
> Yes.
>
>> It doesn't seem
>> to do anything expensive on a per-block basis like loading SSE registers, so it
>> seems it would fit better as a "cipher" algorithm if at all possible. Then
>> there would be no need to implement all these modes yet again.
>>
>
> True.
>
>> Also, what would be the feasibility of simply replacing aes-generic with the
>> time-invariant implementation, rather than offering two implementations and
>> requiring users to choose one, usually without the needed expertise?
>>
>
> Well, it is a policy decision whether you want the best performance or
> reduced correlation between timing and the input, so there is no way
> to make everybody happy by replacing one with the other. But I can
> certainly implement is as a cipher, and we can take the discussion
> from there.
>
>>> +
>>> +/*
>>> + * Emit the sbox as __weak with external linkage to prevent the compiler
>>> + * from doing constant folding on sbox references involving fixed indexes.
>>> + */
>>> +__weak const u8 __cacheline_aligned __aesti_sbox[] = {
>>
>> Did you consider marking it 'volatile' instead?
>>
>
> I did not, and I expect the result to be the same. I can replace it if
> it matters.
>
>>> +static int aesti_set_key(struct aes_ti_ctx *ctx, const u8 *in_key,
>>> + unsigned int key_len)
>>> +{
>>> + struct crypto_aes_ctx rk;
>>> + int err;
>>> +
>>> + err = crypto_aes_expand_key(&rk, in_key, key_len);
>>> + if (err)
>>> + return err;
>>
>> crypto_aes_expand_key() assumes that the key is u32-aligned; I don't think
>> that's guaranteed here.
>>
>
> No, I don't think so. AFAICT it expects the round keys to be u32
> aligned, which is guaranteed due to the fact that struct
> crypto_aes_ctx encapsulates an array of u32
>
I stand corrected: I misread le32_to_cpu() for get_unaligned_le32()
^ permalink raw reply
* Re: [PATCH -stable] crypto: ccm - deal with CTR ciphers that honour iv_out
From: Ard Biesheuvel @ 2017-02-02 8:01 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto@vger.kernel.org
In-Reply-To: <20170202051300.GA4349@gondor.apana.org.au>
On 2 February 2017 at 05:13, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Wed, Feb 01, 2017 at 08:08:09PM +0000, Ard Biesheuvel wrote:
>>
>> Could you please forward this patch to Linus as well? I noticed that the patch
>
> Sure, I will do that.
>
>> crypto: arm64/aes-blk - honour iv_out requirement in CBC and CTR modes
>>
>> is now in mainline, which means CCM is now broken on arm64, given that
>> the iv_out requirement for CTR apparently isn't honored by *any*
>> implementation, and CCM wrongly assumes that req->iv retains its value
>> across the call into the CTR skcipher
>
> Hmm, I wonder why we don't see this breakage with the generic
> CTR as it seems to do exactly the same thing.
>
You are right: due to its construction, the CCM mode does not care
about the incremented counter because it clears the counter part of
the IV before encrypting the MAC. So this is caused by an optimization
in my code rather than the CCM code being incorrect.
^ permalink raw reply
* Re: [PATCH -stable] crypto: ccm - deal with CTR ciphers that honour iv_out
From: Herbert Xu @ 2017-02-02 9:53 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: linux-crypto@vger.kernel.org
In-Reply-To: <CAKv+Gu-fUAofB+_Td=bq=m2KsMoc7V3N+NffW5GTuo4fdCwRTw@mail.gmail.com>
On Thu, Feb 02, 2017 at 08:01:47AM +0000, Ard Biesheuvel wrote:
>
> You are right: due to its construction, the CCM mode does not care
> about the incremented counter because it clears the counter part of
> the IV before encrypting the MAC. So this is caused by an optimization
> in my code rather than the CCM code being incorrect.
OK so you will send me an update for the ARM64 code, right?
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: crypto: NULL deref in sha512_mb_mgr_get_comp_job_avx2
From: Dmitry Vyukov @ 2017-02-02 10:58 UTC (permalink / raw)
To: tim.c.chen
Cc: Herbert Xu, David Miller, linux-crypto, LKML, megha.dey,
fenghua.yu, syzkaller
In-Reply-To: <20170201184502.GA11649@linux.intel.com>
On Wed, Feb 1, 2017 at 7:45 PM, Tim Chen <tim.c.chen@linux.intel.com> wrote:
> On Tue, Jan 31, 2017 at 02:16:31PM +0100, Dmitry Vyukov wrote:
>> Hello,
>>
>> I am getting the following reports with low frequency while running
>> syzkaller fuzzer. Unfortunately they are not reproducible and happen
>> in a background thread, so it is difficult to extract any context on
>> my side. I see only few such crashes per week, so most likely it is
>> some hard to trigger data race. The following reports are from mmotm
>> tree, commits 00e20cfc2bf04a0cbe1f5405f61c8426f43eee84 and
>> fff7e71eac7788904753136f09bcad7471f7799e. Any ideas as to how this can
>> happen?
>>
>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
>> IP: [<ffffffff813fc09e>] sha512_mb_mgr_get_comp_job_avx2+0x6e/0xee
>> arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S:251
>> PGD 1d2395067 [ 220.874864] PUD 1d2860067
>> Oops: 0002 [#1] SMP KASAN
>> Dumping ftrace buffer:
>> (ftrace buffer empty)
>> Modules linked in:
>> CPU: 0 PID: 516 Comm: kworker/0:1 Not tainted 4.9.0 #4
>> Hardware name: Google Google Compute Engine/Google Compute Engine,
>> BIOS Google 01/01/2011
>> Workqueue: crypto mcryptd_queue_worker
>> task: ffff8801d9f346c0 task.stack: ffff8801d9f08000
>> RIP: 0010:[<ffffffff813fc09e>] [<ffffffff813fc09e>]
>> sha512_mb_mgr_get_comp_job_avx2+0x6e/0xee
>> arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S:251
>> RSP: 0018:ffff8801d9f0eef8 EFLAGS: 00010202
>> RAX: 0000000000000000 RBX: ffff8801d7db1190 RCX: 0000000000000006
>> RDX: 0000000000000001 RSI: ffff8801d9f34ee8 RDI: ffff8801d7db1040
>> RBP: ffff8801d9f0f258 R08: 0000000100000000 R09: 0000000000000001
>> R10: 0000000000000002 R11: 0000000000000003 R12: ffff8801d9f0f230
>> R13: ffff8801c8bbc4e0 R14: ffff8801c8bbc530 R15: ffff8801d9f0ef70
>> FS: 0000000000000000(0000) GS:ffff8801dc000000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000000000060 CR3: 00000001cc15a000 CR4: 00000000001406f0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Stack:
>> ffff8801d7db1040 ffffffff813fa207 dffffc0000000000 ffffe8ffffc0f238
>> 0000000000000002 1ffff1003b3e1dea ffffe8ffffc0f218 ffff8801d9f0f190
>> 0000000000000282 ffffe8ffffc0f140 ffffe8ffffc0f220 0000000041b58ab3
>> Call Trace:
>> [<ffffffff813fb407>] sha512_mb_update+0x2f7/0x4e0
>> arch/x86/crypto/sha512-mb/sha512_mb.c:588
>> [<ffffffff8219d4ad>] crypto_ahash_update include/crypto/hash.h:512 [inline]
>> [<ffffffff8219d4ad>] ahash_mcryptd_update crypto/mcryptd.c:627 [inline]
>> [<ffffffff8219d4ad>] mcryptd_hash_update+0xcd/0x1c0 crypto/mcryptd.c:373
>> [<ffffffff8219c99f>] mcryptd_queue_worker+0xff/0x6a0 crypto/mcryptd.c:181
>> [<ffffffff81492960>] process_one_work+0xbd0/0x1c10 kernel/workqueue.c:2096
>> [<ffffffff81493bc3>] worker_thread+0x223/0x1990 kernel/workqueue.c:2230
>> [<ffffffff814abb33>] kthread+0x323/0x3e0 kernel/kthread.c:209
>> [<ffffffff8436fbaa>] ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:433
>> Code: 49 0f 42 d3 48 f7 c2 f0 ff ff ff 0f 85 9a 00 00 00 48 83 e2 0f
>> 48 6b da 08 48 8d 9c 1f 48 01 00 00 48 8b 03 48 c7 03 00 00 00 00 <c7>
>> 40 60 02 00 00 00 48 8b 9f 40 01 00 00 48 c1 e3 08 48 09 d3
>> RIP [<ffffffff813fc09e>] sha512_mb_mgr_get_comp_job_avx2+0x6e/0xee
>> arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S:251
>> RSP <ffff8801d9f0eef8>
>> CR2: 0000000000000060
>> ---[ end trace 139fd4cda5dfe2c4 ]---
>>
>
> Dmitry,
>
> One theory that Mehga and I have is that perhaps the flusher
> and regular computaion updates are stepping on each other.
> Can you try this patch and see if it helps?
No, for this one I can't. Sorry.
It happens with very low frequency and only one fuzzer that tests
mmotm tree. If/when this is committed, I can keep an eye on these
reports and notify if I still see them.
If you have a hypothesis as to how it happens, perhaps you could write
a test that provokes the crash and maybe add some sleeps to kernel
code or alter timeouts to increase probability.
> --->8---
>
> From: Tim Chen <tim.c.chen@linux.intel.com>
> Subject: [PATCH] crypto/sha512-mb: Protect sha512 mb ctx mgr access
> To: Herbert Xu <herbert@gondor.apana.org.au>, Dmitry Vyukov <dvyukov@google.com>
> Cc: Tim Chen <tim.c.chen@linux.intel.com>, David Miller <davem@davemloft.net>, linux-crypto@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>, megha.dey@linux.intel.com, fenghua.yu@intel.com
>
> The flusher and regular multi-buffer computation via mcryptd may race with another.
> Add here a lock and turn off interrupt to to access multi-buffer
> computation state cstate->mgr before a round of computation. This should
> prevent the flusher code jumping in.
>
> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> ---
> arch/x86/crypto/sha512-mb/sha512_mb.c | 64 +++++++++++++++++++++++------------
> 1 file changed, 42 insertions(+), 22 deletions(-)
>
> diff --git a/arch/x86/crypto/sha512-mb/sha512_mb.c b/arch/x86/crypto/sha512-mb/sha512_mb.c
> index d210174..f3c1c21 100644
> --- a/arch/x86/crypto/sha512-mb/sha512_mb.c
> +++ b/arch/x86/crypto/sha512-mb/sha512_mb.c
> @@ -221,7 +221,7 @@ static struct sha512_hash_ctx *sha512_ctx_mgr_resubmit
> }
>
> static struct sha512_hash_ctx
> - *sha512_ctx_mgr_get_comp_ctx(struct sha512_ctx_mgr *mgr)
> + *sha512_ctx_mgr_get_comp_ctx(struct mcryptd_alg_cstate *cstate)
> {
> /*
> * If get_comp_job returns NULL, there are no jobs complete.
> @@ -233,11 +233,17 @@ static struct sha512_hash_ctx
> * Otherwise, all jobs currently being managed by the hash_ctx_mgr
> * still need processing.
> */
> + struct sha512_ctx_mgr *mgr;
> struct sha512_hash_ctx *ctx;
> + unsigned long flags;
>
> + mgr = cstate->mgr;
> + spin_lock_irqsave(&cstate->work_lock, flags);
> ctx = (struct sha512_hash_ctx *)
> sha512_job_mgr_get_comp_job(&mgr->mgr);
> - return sha512_ctx_mgr_resubmit(mgr, ctx);
> + ctx = sha512_ctx_mgr_resubmit(mgr, ctx);
> + spin_unlock_irqrestore(&cstate->work_lock, flags);
> + return ctx;
> }
>
> static void sha512_ctx_mgr_init(struct sha512_ctx_mgr *mgr)
> @@ -246,12 +252,17 @@ static void sha512_ctx_mgr_init(struct sha512_ctx_mgr *mgr)
> }
>
> static struct sha512_hash_ctx
> - *sha512_ctx_mgr_submit(struct sha512_ctx_mgr *mgr,
> + *sha512_ctx_mgr_submit(struct mcryptd_alg_cstate *cstate,
> struct sha512_hash_ctx *ctx,
> const void *buffer,
> uint32_t len,
> int flags)
> {
> + struct sha512_ctx_mgr *mgr;
> + unsigned long irqflags;
> +
> + mgr = cstate->mgr;
> + spin_lock_irqsave(&cstate->work_lock, irqflags);
> if (flags & (~HASH_ENTIRE)) {
> /*
> * User should not pass anything other than FIRST, UPDATE, or
> @@ -351,20 +362,26 @@ static struct sha512_hash_ctx
> }
> }
>
> - return sha512_ctx_mgr_resubmit(mgr, ctx);
> + ctx = sha512_ctx_mgr_resubmit(mgr, ctx);
> + spin_unlock_irqrestore(&cstate->work_lock, irqflags);
> + return ctx;
> }
>
> -static struct sha512_hash_ctx *sha512_ctx_mgr_flush(struct sha512_ctx_mgr *mgr)
> +static struct sha512_hash_ctx *sha512_ctx_mgr_flush(struct mcryptd_alg_cstate *cstate)
> {
> + struct sha512_ctx_mgr *mgr;
> struct sha512_hash_ctx *ctx;
> + unsigned long flags;
>
> + mgr = cstate->mgr;
> + spin_lock_irqsave(&cstate->work_lock, flags);
> while (1) {
> ctx = (struct sha512_hash_ctx *)
> sha512_job_mgr_flush(&mgr->mgr);
>
> /* If flush returned 0, there are no more jobs in flight. */
> if (!ctx)
> - return NULL;
> + break;
>
> /*
> * If flush returned a job, resubmit the job to finish
> @@ -378,8 +395,10 @@ static struct sha512_hash_ctx *sha512_ctx_mgr_flush(struct sha512_ctx_mgr *mgr)
> * the sha512_ctx_mgr still need processing. Loop.
> */
> if (ctx)
> - return ctx;
> + break;
> }
> + spin_unlock_irqrestore(&cstate->work_lock, flags);
> + return ctx;
> }
>
> static int sha512_mb_init(struct ahash_request *areq)
> @@ -439,11 +458,11 @@ static int sha_finish_walk(struct mcryptd_hash_request_ctx **ret_rctx,
> sha_ctx = (struct sha512_hash_ctx *)
> ahash_request_ctx(&rctx->areq);
> kernel_fpu_begin();
> - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx,
> + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx,
> rctx->walk.data, nbytes, flag);
> if (!sha_ctx) {
> if (flush)
> - sha_ctx = sha512_ctx_mgr_flush(cstate->mgr);
> + sha_ctx = sha512_ctx_mgr_flush(cstate);
> }
> kernel_fpu_end();
> if (sha_ctx)
> @@ -471,11 +490,12 @@ static int sha_complete_job(struct mcryptd_hash_request_ctx *rctx,
> struct sha512_hash_ctx *sha_ctx;
> struct mcryptd_hash_request_ctx *req_ctx;
> int ret;
> + unsigned long flags;
>
> /* remove from work list */
> - spin_lock(&cstate->work_lock);
> + spin_lock_irqsave(&cstate->work_lock, flags);
> list_del(&rctx->waiter);
> - spin_unlock(&cstate->work_lock);
> + spin_unlock_irqrestore(&cstate->work_lock, flags);
>
> if (irqs_disabled())
> rctx->complete(&req->base, err);
> @@ -486,14 +506,14 @@ static int sha_complete_job(struct mcryptd_hash_request_ctx *rctx,
> }
>
> /* check to see if there are other jobs that are done */
> - sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate->mgr);
> + sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate);
> while (sha_ctx) {
> req_ctx = cast_hash_to_mcryptd_ctx(sha_ctx);
> ret = sha_finish_walk(&req_ctx, cstate, false);
> if (req_ctx) {
> - spin_lock(&cstate->work_lock);
> + spin_lock_irqsave(&cstate->work_lock, flags);
> list_del(&req_ctx->waiter);
> - spin_unlock(&cstate->work_lock);
> + spin_unlock_irqrestore(&cstate->work_lock, flags);
>
> req = cast_mcryptd_ctx_to_req(req_ctx);
> if (irqs_disabled())
> @@ -504,7 +524,7 @@ static int sha_complete_job(struct mcryptd_hash_request_ctx *rctx,
> local_bh_enable();
> }
> }
> - sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate->mgr);
> + sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate);
> }
>
> return 0;
> @@ -515,6 +535,7 @@ static void sha512_mb_add_list(struct mcryptd_hash_request_ctx *rctx,
> {
> unsigned long next_flush;
> unsigned long delay = usecs_to_jiffies(FLUSH_INTERVAL);
> + unsigned long flags;
>
> /* initialize tag */
> rctx->tag.arrival = jiffies; /* tag the arrival time */
> @@ -522,9 +543,9 @@ static void sha512_mb_add_list(struct mcryptd_hash_request_ctx *rctx,
> next_flush = rctx->tag.arrival + delay;
> rctx->tag.expire = next_flush;
>
> - spin_lock(&cstate->work_lock);
> + spin_lock_irqsave(&cstate->work_lock, flags);
> list_add_tail(&rctx->waiter, &cstate->work_list);
> - spin_unlock(&cstate->work_lock);
> + spin_unlock_irqrestore(&cstate->work_lock, flags);
>
> mcryptd_arm_flusher(cstate, delay);
> }
> @@ -565,7 +586,7 @@ static int sha512_mb_update(struct ahash_request *areq)
> sha_ctx = (struct sha512_hash_ctx *) ahash_request_ctx(areq);
> sha512_mb_add_list(rctx, cstate);
> kernel_fpu_begin();
> - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data,
> + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx, rctx->walk.data,
> nbytes, HASH_UPDATE);
> kernel_fpu_end();
>
> @@ -628,7 +649,7 @@ static int sha512_mb_finup(struct ahash_request *areq)
> sha512_mb_add_list(rctx, cstate);
>
> kernel_fpu_begin();
> - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data,
> + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx, rctx->walk.data,
> nbytes, flag);
> kernel_fpu_end();
>
> @@ -677,8 +698,7 @@ static int sha512_mb_final(struct ahash_request *areq)
> /* flag HASH_FINAL and 0 data size */
> sha512_mb_add_list(rctx, cstate);
> kernel_fpu_begin();
> - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx, &data, 0,
> - HASH_LAST);
> + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx, &data, 0, HASH_LAST);
> kernel_fpu_end();
>
> /* check if anything is returned */
> @@ -940,7 +960,7 @@ static unsigned long sha512_mb_flusher(struct mcryptd_alg_cstate *cstate)
> break;
> kernel_fpu_begin();
> sha_ctx = (struct sha512_hash_ctx *)
> - sha512_ctx_mgr_flush(cstate->mgr);
> + sha512_ctx_mgr_flush(cstate);
> kernel_fpu_end();
> if (!sha_ctx) {
> pr_err("sha512_mb error: nothing got flushed for"
> --
> 2.5.5
>
^ permalink raw reply
* [PATCH 1/2] crypto: arm64/aes - don't use IV buffer to return final keystream block
From: Ard Biesheuvel @ 2017-02-02 11:38 UTC (permalink / raw)
To: linux-crypto, herbert; +Cc: linux-arm-kernel, Ard Biesheuvel
The arm64 bit sliced AES core code uses the IV buffer to pass the final
keystream block back to the glue code if the input is not a multiple of
the block size, so that the asm code does not have to deal with anything
except 16 byte blocks. This is done under the assumption that the outgoing
IV is meaningless anyway in this case, given that chaining is no longer
possible under these circumstances.
However, as it turns out, the CCM driver does expect the IV to retain
a value that is equal to the original IV except for the counter value,
and even interprets byte zero as a length indicator, which may result
in memory corruption if the IV is overwritten with something else.
So use a separate buffer to return the final keystream block.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
Note that this patch includes the fix
crypto: arm64/aes-neon-bs - honour iv_out requirement in CTR mode
which I sent out earlier.
arch/arm64/crypto/aes-neonbs-core.S | 37 ++++++++++++--------
arch/arm64/crypto/aes-neonbs-glue.c | 9 ++---
2 files changed, 28 insertions(+), 18 deletions(-)
diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S
index 8d0cdaa2768d..ca0472500433 100644
--- a/arch/arm64/crypto/aes-neonbs-core.S
+++ b/arch/arm64/crypto/aes-neonbs-core.S
@@ -853,13 +853,15 @@ ENDPROC(aesbs_xts_decrypt)
/*
* aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
- * int rounds, int blocks, u8 iv[], bool final)
+ * int rounds, int blocks, u8 iv[], u8 final[])
*/
ENTRY(aesbs_ctr_encrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
- add x4, x4, x6 // do one extra block if final
+ cmp x6, #0
+ cset x10, ne
+ add x4, x4, x10 // do one extra block if final
ldp x7, x8, [x5]
ld1 {v0.16b}, [x5]
@@ -874,19 +876,26 @@ CPU_LE( rev x8, x8 )
csel x4, x4, xzr, pl
csel x9, x9, xzr, le
+ tbnz x9, #1, 0f
next_ctr v1
+ tbnz x9, #2, 0f
next_ctr v2
+ tbnz x9, #3, 0f
next_ctr v3
+ tbnz x9, #4, 0f
next_ctr v4
+ tbnz x9, #5, 0f
next_ctr v5
+ tbnz x9, #6, 0f
next_ctr v6
+ tbnz x9, #7, 0f
next_ctr v7
0: mov bskey, x2
mov rounds, x3
bl aesbs_encrypt8
- lsr x9, x9, x6 // disregard the extra block
+ lsr x9, x9, x10 // disregard the extra block
tbnz x9, #0, 0f
ld1 {v8.16b}, [x1], #16
@@ -928,36 +937,36 @@ CPU_LE( rev x8, x8 )
eor v5.16b, v5.16b, v15.16b
st1 {v5.16b}, [x0], #16
- next_ctr v0
+8: next_ctr v0
cbnz x4, 99b
0: st1 {v0.16b}, [x5]
-8: ldp x29, x30, [sp], #16
+ ldp x29, x30, [sp], #16
ret
/*
- * If we are handling the tail of the input (x6 == 1), return the
- * final keystream block back to the caller via the IV buffer.
+ * If we are handling the tail of the input (x6 != NULL), return the
+ * final keystream block back to the caller.
*/
1: cbz x6, 8b
- st1 {v1.16b}, [x5]
+ st1 {v1.16b}, [x6]
b 8b
2: cbz x6, 8b
- st1 {v4.16b}, [x5]
+ st1 {v4.16b}, [x6]
b 8b
3: cbz x6, 8b
- st1 {v6.16b}, [x5]
+ st1 {v6.16b}, [x6]
b 8b
4: cbz x6, 8b
- st1 {v3.16b}, [x5]
+ st1 {v3.16b}, [x6]
b 8b
5: cbz x6, 8b
- st1 {v7.16b}, [x5]
+ st1 {v7.16b}, [x6]
b 8b
6: cbz x6, 8b
- st1 {v2.16b}, [x5]
+ st1 {v2.16b}, [x6]
b 8b
7: cbz x6, 8b
- st1 {v5.16b}, [x5]
+ st1 {v5.16b}, [x6]
b 8b
ENDPROC(aesbs_ctr_encrypt)
diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c
index 863e436ecf89..db2501d93550 100644
--- a/arch/arm64/crypto/aes-neonbs-glue.c
+++ b/arch/arm64/crypto/aes-neonbs-glue.c
@@ -34,7 +34,7 @@ asmlinkage void aesbs_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
- int rounds, int blocks, u8 iv[], bool final);
+ int rounds, int blocks, u8 iv[], u8 final[]);
asmlinkage void aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
@@ -201,6 +201,7 @@ static int ctr_encrypt(struct skcipher_request *req)
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct aesbs_ctx *ctx = crypto_skcipher_ctx(tfm);
struct skcipher_walk walk;
+ u8 buf[AES_BLOCK_SIZE];
int err;
err = skcipher_walk_virt(&walk, req, true);
@@ -208,12 +209,12 @@ static int ctr_encrypt(struct skcipher_request *req)
kernel_neon_begin();
while (walk.nbytes > 0) {
unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE;
- bool final = (walk.total % AES_BLOCK_SIZE) != 0;
+ u8 *final = (walk.total % AES_BLOCK_SIZE) ? buf : NULL;
if (walk.nbytes < walk.total) {
blocks = round_down(blocks,
walk.stride / AES_BLOCK_SIZE);
- final = false;
+ final = NULL;
}
aesbs_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
@@ -225,7 +226,7 @@ static int ctr_encrypt(struct skcipher_request *req)
if (dst != src)
memcpy(dst, src, walk.total % AES_BLOCK_SIZE);
- crypto_xor(dst, walk.iv, walk.total % AES_BLOCK_SIZE);
+ crypto_xor(dst, final, walk.total % AES_BLOCK_SIZE);
err = skcipher_walk_done(&walk, 0);
break;
--
2.7.4
^ permalink raw reply related
* [PATCH 2/2] crypto: arm/aes - don't use IV buffer to return final keystream block
From: Ard Biesheuvel @ 2017-02-02 11:38 UTC (permalink / raw)
To: linux-crypto, herbert; +Cc: linux-arm-kernel, Ard Biesheuvel
In-Reply-To: <1486035536-895-1-git-send-email-ard.biesheuvel@linaro.org>
The ARM bit sliced AES core code uses the IV buffer to pass the final
keystream block back to the glue code if the input is not a multiple of
the block size, so that the asm code does not have to deal with anything
except 16 byte blocks. This is done under the assumption that the outgoing
IV is meaningless anyway in this case, given that chaining is no longer
possible under these circumstances.
However, as it turns out, the CCM driver does expect the IV to retain
a value that is equal to the original IV except for the counter value,
and even interprets byte zero as a length indicator, which may result
in memory corruption if the IV is overwritten with something else.
So use a separate buffer to return the final keystream block.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm/crypto/aes-neonbs-core.S | 16 +++++++++-------
arch/arm/crypto/aes-neonbs-glue.c | 9 +++++----
2 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/arch/arm/crypto/aes-neonbs-core.S b/arch/arm/crypto/aes-neonbs-core.S
index c9477044fbba..2764edc56467 100644
--- a/arch/arm/crypto/aes-neonbs-core.S
+++ b/arch/arm/crypto/aes-neonbs-core.S
@@ -779,14 +779,15 @@ ENDPROC(aesbs_cbc_decrypt)
/*
* aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
- * int rounds, int blocks, u8 ctr[], bool final)
+ * int rounds, int blocks, u8 ctr[], u8 final[])
*/
ENTRY(aesbs_ctr_encrypt)
mov ip, sp
push {r4-r10, lr}
ldm ip, {r5-r7} // load args 4-6
- add r5, r5, r7 // one extra block if final == 1
+ teq r7, #0
+ addne r5, r5, #1 // one extra block if final != 0
vld1.8 {q0}, [r6] // load counter
vrev32.8 q1, q0
@@ -865,19 +866,20 @@ ENTRY(aesbs_ctr_encrypt)
veor q2, q2, q14
vst1.8 {q2}, [r0]!
teq r4, #0 // skip last block if 'final'
- W(bne) 4f
+ W(bne) 5f
3: veor q5, q5, q15
vst1.8 {q5}, [r0]!
- next_ctr q0
+4: next_ctr q0
subs r5, r5, #8
bgt 99b
- vmov q5, q0
-
-4: vst1.8 {q5}, [r6]
+ vst1.8 {q0}, [r6]
pop {r4-r10, pc}
+
+5: vst1.8 {q5}, [r4]
+ b 4b
ENDPROC(aesbs_ctr_encrypt)
.macro next_tweak, out, in, const, tmp
diff --git a/arch/arm/crypto/aes-neonbs-glue.c b/arch/arm/crypto/aes-neonbs-glue.c
index e262f99a44d3..2920b96dbd36 100644
--- a/arch/arm/crypto/aes-neonbs-glue.c
+++ b/arch/arm/crypto/aes-neonbs-glue.c
@@ -35,7 +35,7 @@ asmlinkage void aesbs_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
- int rounds, int blocks, u8 ctr[], bool final);
+ int rounds, int blocks, u8 ctr[], u8 final[]);
asmlinkage void aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
@@ -186,6 +186,7 @@ static int ctr_encrypt(struct skcipher_request *req)
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct aesbs_ctx *ctx = crypto_skcipher_ctx(tfm);
struct skcipher_walk walk;
+ u8 buf[AES_BLOCK_SIZE];
int err;
err = skcipher_walk_virt(&walk, req, true);
@@ -193,12 +194,12 @@ static int ctr_encrypt(struct skcipher_request *req)
kernel_neon_begin();
while (walk.nbytes > 0) {
unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE;
- bool final = (walk.total % AES_BLOCK_SIZE) != 0;
+ u8 *final = (walk.total % AES_BLOCK_SIZE) ? buf : NULL;
if (walk.nbytes < walk.total) {
blocks = round_down(blocks,
walk.stride / AES_BLOCK_SIZE);
- final = false;
+ final = NULL;
}
aesbs_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
@@ -210,7 +211,7 @@ static int ctr_encrypt(struct skcipher_request *req)
if (dst != src)
memcpy(dst, src, walk.total % AES_BLOCK_SIZE);
- crypto_xor(dst, walk.iv, walk.total % AES_BLOCK_SIZE);
+ crypto_xor(dst, final, walk.total % AES_BLOCK_SIZE);
err = skcipher_walk_done(&walk, 0);
break;
--
2.7.4
^ permalink raw reply related
* Re: [PATCH -stable] crypto: ccm - deal with CTR ciphers that honour iv_out
From: Ard Biesheuvel @ 2017-02-02 11:39 UTC (permalink / raw)
To: Herbert Xu; +Cc: linux-crypto@vger.kernel.org
In-Reply-To: <20170202095352.GA4818@gondor.apana.org.au>
On 2 February 2017 at 09:53, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Thu, Feb 02, 2017 at 08:01:47AM +0000, Ard Biesheuvel wrote:
>>
>> You are right: due to its construction, the CCM mode does not care
>> about the incremented counter because it clears the counter part of
>> the IV before encrypting the MAC. So this is caused by an optimization
>> in my code rather than the CCM code being incorrect.
>
> OK so you will send me an update for the ARM64 code, right?
>
Yes, on their way
Thanks,
Ard.
^ permalink raw reply
* Re: [PATCH 0/6] Add support for ECDSA algorithm
From: Herbert Xu @ 2017-02-02 13:57 UTC (permalink / raw)
To: Nitin Kumbhar; +Cc: davem, linux-crypto
In-Reply-To: <113d3ef8-ef34-0651-71e7-be09ef55727e@nvidia.com>
On Thu, Jan 26, 2017 at 11:30:04AM +0530, Nitin Kumbhar wrote:
>
> This ECDSA implementation is analogous to the RSA kernel implementation for
> signature generation / verification. It extends ECC family of algorithms
> like ECDH to support signature verification using akcipher. This will be
> used in a way similar to RSA.
Yes but RSA had an in-kernel user in the form of module signature
verification. We don't add algorithms to the kernel without
actual users. So this patch-set needs to come with an actual
in-kernel user of ECDSA.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v3 2/3] crypto: brcm: Add Broadcom SPU driver
From: Herbert Xu @ 2017-02-02 14:05 UTC (permalink / raw)
To: Rob Rice
Cc: David S. Miller, Rob Herring, Mark Rutland,
linux-crypto-u79uwXL29TY76Z2rM5mHXA,
devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
bcm-kernel-feedback-list-dY08KVG/lbpWk0Htik3J/w, Catalin Marinas,
Will Deacon, linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
Steve Lin
In-Reply-To: <1485362689-11294-3-git-send-email-rob.rice-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
On Wed, Jan 25, 2017 at 11:44:48AM -0500, Rob Rice wrote:
>
> +static int ahash_export(struct ahash_request *req, void *out)
> +{
> + const struct iproc_reqctx_s *rctx = ahash_request_ctx(req);
> +
> + memcpy(out, rctx, offsetof(struct iproc_reqctx_s, msg_buf));
> + return 0;
> +}
The reqctx data structure seems to contain a lot of info unrelated
to the hash state. Can't we get away with just copying the hash
state (incr_hash) itself?
Cheers,
--
Email: Herbert Xu <herbert-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 2/3] crypto: brcm: Add Broadcom SPU driver
From: Rob Rice @ 2017-02-02 15:11 UTC (permalink / raw)
To: Herbert Xu
Cc: David S. Miller, Rob Herring, Mark Rutland,
linux-crypto-u79uwXL29TY76Z2rM5mHXA,
devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
bcm-kernel-feedback-list-dY08KVG/lbpWk0Htik3J/w, Catalin Marinas,
Will Deacon, linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
Steve Lin
In-Reply-To: <20170202140510.GB5289-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org>
Herbert,
> On Feb 2, 2017, at 9:05 AM, Herbert Xu <herbert-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org> wrote:
>
> On Wed, Jan 25, 2017 at 11:44:48AM -0500, Rob Rice wrote:
>>
>> +static int ahash_export(struct ahash_request *req, void *out)
>> +{
>> + const struct iproc_reqctx_s *rctx = ahash_request_ctx(req);
>> +
>> + memcpy(out, rctx, offsetof(struct iproc_reqctx_s, msg_buf));
>> + return 0;
>> +}
>
> The reqctx data structure seems to contain a lot of info unrelated
> to the hash state. Can't we get away with just copying the hash
> state (incr_hash) itself?
Yes, I see your point. I’ll whittle the export state down to just what’s needed for the hash.
>
> Cheers,
> --
> Email: Herbert Xu <herbert-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH v2] crypto: algapi - make crypto_xor() and crypto_inc() alignment agnostic
From: Ard Biesheuvel @ 2017-02-02 15:56 UTC (permalink / raw)
To: linux-crypto, herbert, ebiggers3; +Cc: Ard Biesheuvel
Instead of unconditionally forcing 4 byte alignment for all generic
chaining modes that rely on crypto_xor() or crypto_inc() (which may
result in unnecessary copying of data when the underlying hardware
can perform unaligned accesses efficiently), make those functions
deal with unaligned input explicitly, but only if the Kconfig symbol
HAVE_EFFICIENT_UNALIGNED_ACCESS is set. This will allow us to drop
the alignmasks from the CBC, CMAC, CTR, CTS, PCBC and SEQIV drivers.
For crypto_inc(), this simply involves making the 4-byte stride
conditional on HAVE_EFFICIENT_UNALIGNED_ACCESS being set, given that
it typically operates on 16 byte buffers.
For crypto_xor(), an algorithm is implemented that simply runs through
the input using the largest strides possible if unaligned accesses are
allowed. If they are not, an optimal sequence of memory accesses is
emitted that takes the relative alignment of the input buffers into
account, e.g., if the relative misalignment of dst and src is 4 bytes,
the entire xor operation will be completed using 4 byte loads and stores
(modulo unaligned bits at the start and end). Note that all expressions
involving misalign are simply eliminated by the compiler when
HAVE_EFFICIENT_UNALIGNED_ACCESS is defined.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
I have greatly simplified the code, but it should still emit an optimal
sequence of loads and stores depending on the misalignment.
crypto/algapi.c | 65 +++++++++++++++-----
crypto/cbc.c | 3 -
crypto/cmac.c | 3 +-
crypto/ctr.c | 2 +-
crypto/cts.c | 3 -
crypto/pcbc.c | 3 -
crypto/seqiv.c | 2 -
7 files changed, 50 insertions(+), 31 deletions(-)
diff --git a/crypto/algapi.c b/crypto/algapi.c
index df939b54b09f..e05ed1c5f5d1 100644
--- a/crypto/algapi.c
+++ b/crypto/algapi.c
@@ -961,32 +961,63 @@ void crypto_inc(u8 *a, unsigned int size)
__be32 *b = (__be32 *)(a + size);
u32 c;
- for (; size >= 4; size -= 4) {
- c = be32_to_cpu(*--b) + 1;
- *b = cpu_to_be32(c);
- if (c)
- return;
- }
+ if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+ !((unsigned long)b & (__alignof__(*b) - 1)))
+ for (; size >= 4; size -= 4) {
+ c = be32_to_cpu(*--b) + 1;
+ *b = cpu_to_be32(c);
+ if (c)
+ return;
+ }
crypto_inc_byte(a, size);
}
EXPORT_SYMBOL_GPL(crypto_inc);
-static inline void crypto_xor_byte(u8 *a, const u8 *b, unsigned int size)
+void crypto_xor(u8 *dst, const u8 *src, unsigned int len)
{
- for (; size; size--)
- *a++ ^= *b++;
-}
+ const int size = sizeof(unsigned long);
+ int delta = ((unsigned long)dst ^ (unsigned long)src) & (size - 1);
+ int misalign = 0;
+
+ if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && delta) {
+ misalign = 1 << __ffs(delta);
+
+ /*
+ * If we care about alignment, process as many bytes as
+ * needed to advance dst and src to values whose alignments
+ * equal their relative misalignment. This will allow us to
+ * process the remainder of the input using optimal strides.
+ */
+ while (((unsigned long)dst & (misalign - 1)) && len > 0) {
+ *dst++ ^= *src++;
+ len--;
+ }
+ }
-void crypto_xor(u8 *dst, const u8 *src, unsigned int size)
-{
- u32 *a = (u32 *)dst;
- u32 *b = (u32 *)src;
+ while (len >= size && !misalign) {
+ *(unsigned long *)dst ^= *(unsigned long *)src;
+ dst += size;
+ src += size;
+ len -= size;
+ }
- for (; size >= 4; size -= 4)
- *a++ ^= *b++;
+ while (IS_ENABLED(CONFIG_64BIT) && len >= 4 && !(misalign & 3)) {
+ *(u32 *)dst ^= *(u32 *)src;
+ dst += 4;
+ src += 4;
+ len -= 4;
+ }
+
+ while (len >= 2 && !(misalign & 1)) {
+ *(u16 *)dst ^= *(u16 *)src;
+ dst += 2;
+ src += 2;
+ len -= 2;
+ }
- crypto_xor_byte((u8 *)a, (u8 *)b, size);
+ while (len--)
+ *dst++ ^= *src++;
}
EXPORT_SYMBOL_GPL(crypto_xor);
diff --git a/crypto/cbc.c b/crypto/cbc.c
index 68f751a41a84..bc160a3186dc 100644
--- a/crypto/cbc.c
+++ b/crypto/cbc.c
@@ -145,9 +145,6 @@ static int crypto_cbc_create(struct crypto_template *tmpl, struct rtattr **tb)
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
- /* We access the data as u32s when xoring. */
- inst->alg.base.cra_alignmask |= __alignof__(u32) - 1;
-
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
diff --git a/crypto/cmac.c b/crypto/cmac.c
index 04080dca8f0c..16301f52858c 100644
--- a/crypto/cmac.c
+++ b/crypto/cmac.c
@@ -260,8 +260,7 @@ static int cmac_create(struct crypto_template *tmpl, struct rtattr **tb)
if (err)
goto out_free_inst;
- /* We access the data as u32s when xoring. */
- alignmask = alg->cra_alignmask | (__alignof__(u32) - 1);
+ alignmask = alg->cra_alignmask;
inst->alg.base.cra_alignmask = alignmask;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
diff --git a/crypto/ctr.c b/crypto/ctr.c
index a9a7a44f2783..a4f4a8983169 100644
--- a/crypto/ctr.c
+++ b/crypto/ctr.c
@@ -209,7 +209,7 @@ static struct crypto_instance *crypto_ctr_alloc(struct rtattr **tb)
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER;
inst->alg.cra_priority = alg->cra_priority;
inst->alg.cra_blocksize = 1;
- inst->alg.cra_alignmask = alg->cra_alignmask | (__alignof__(u32) - 1);
+ inst->alg.cra_alignmask = alg->cra_alignmask;
inst->alg.cra_type = &crypto_blkcipher_type;
inst->alg.cra_blkcipher.ivsize = alg->cra_blocksize;
diff --git a/crypto/cts.c b/crypto/cts.c
index a1335d6c35fb..243f591dc409 100644
--- a/crypto/cts.c
+++ b/crypto/cts.c
@@ -374,9 +374,6 @@ static int crypto_cts_create(struct crypto_template *tmpl, struct rtattr **tb)
inst->alg.base.cra_blocksize = alg->base.cra_blocksize;
inst->alg.base.cra_alignmask = alg->base.cra_alignmask;
- /* We access the data as u32s when xoring. */
- inst->alg.base.cra_alignmask |= __alignof__(u32) - 1;
-
inst->alg.ivsize = alg->base.cra_blocksize;
inst->alg.chunksize = crypto_skcipher_alg_chunksize(alg);
inst->alg.min_keysize = crypto_skcipher_alg_min_keysize(alg);
diff --git a/crypto/pcbc.c b/crypto/pcbc.c
index 11d248673ad4..29dd2b4a3b85 100644
--- a/crypto/pcbc.c
+++ b/crypto/pcbc.c
@@ -260,9 +260,6 @@ static int crypto_pcbc_create(struct crypto_template *tmpl, struct rtattr **tb)
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
- /* We access the data as u32s when xoring. */
- inst->alg.base.cra_alignmask |= __alignof__(u32) - 1;
-
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
diff --git a/crypto/seqiv.c b/crypto/seqiv.c
index c7049231861f..570b7d1aa0ca 100644
--- a/crypto/seqiv.c
+++ b/crypto/seqiv.c
@@ -153,8 +153,6 @@ static int seqiv_aead_create(struct crypto_template *tmpl, struct rtattr **tb)
if (IS_ERR(inst))
return PTR_ERR(inst);
- inst->alg.base.cra_alignmask |= __alignof__(u32) - 1;
-
spawn = aead_instance_ctx(inst);
alg = crypto_spawn_aead_alg(spawn);
--
2.7.4
^ permalink raw reply related
* [PATCH] crypto: generic/aes - drop alignment requirement
From: Ard Biesheuvel @ 2017-02-02 15:58 UTC (permalink / raw)
To: linux-crypto, herbert, ebiggers3; +Cc: Ard Biesheuvel
The generic AES code exposes a 32-bit align mask, which forces all
users of the code to use temporary buffers or take other measures to
ensure the alignment requirement is adhered to, even on architectures
that don't care about alignment for software algorithms such as this
one.
So drop the align mask, and fix the code to use get_unaligned_le32()
where appropriate, which will resolve to whatever is optimal for the
architecture.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
crypto/aes_generic.c | 64 ++++++++++----------
1 file changed, 32 insertions(+), 32 deletions(-)
diff --git a/crypto/aes_generic.c b/crypto/aes_generic.c
index 3dd101144a58..ca554d57d01e 100644
--- a/crypto/aes_generic.c
+++ b/crypto/aes_generic.c
@@ -54,6 +54,7 @@
#include <linux/errno.h>
#include <linux/crypto.h>
#include <asm/byteorder.h>
+#include <asm/unaligned.h>
static inline u8 byte(const u32 x, const unsigned n)
{
@@ -1216,7 +1217,6 @@ EXPORT_SYMBOL_GPL(crypto_il_tab);
int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
unsigned int key_len)
{
- const __le32 *key = (const __le32 *)in_key;
u32 i, t, u, v, w, j;
if (key_len != AES_KEYSIZE_128 && key_len != AES_KEYSIZE_192 &&
@@ -1225,10 +1225,15 @@ int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
ctx->key_length = key_len;
- ctx->key_dec[key_len + 24] = ctx->key_enc[0] = le32_to_cpu(key[0]);
- ctx->key_dec[key_len + 25] = ctx->key_enc[1] = le32_to_cpu(key[1]);
- ctx->key_dec[key_len + 26] = ctx->key_enc[2] = le32_to_cpu(key[2]);
- ctx->key_dec[key_len + 27] = ctx->key_enc[3] = le32_to_cpu(key[3]);
+ ctx->key_enc[0] = get_unaligned_le32(in_key);
+ ctx->key_enc[1] = get_unaligned_le32(in_key + 4);
+ ctx->key_enc[2] = get_unaligned_le32(in_key + 8);
+ ctx->key_enc[3] = get_unaligned_le32(in_key + 12);
+
+ ctx->key_dec[key_len + 24] = ctx->key_enc[0];
+ ctx->key_dec[key_len + 25] = ctx->key_enc[1];
+ ctx->key_dec[key_len + 26] = ctx->key_enc[2];
+ ctx->key_dec[key_len + 27] = ctx->key_enc[3];
switch (key_len) {
case AES_KEYSIZE_128:
@@ -1238,17 +1243,17 @@ int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
break;
case AES_KEYSIZE_192:
- ctx->key_enc[4] = le32_to_cpu(key[4]);
- t = ctx->key_enc[5] = le32_to_cpu(key[5]);
+ ctx->key_enc[4] = get_unaligned_le32(in_key + 16);
+ t = ctx->key_enc[5] = get_unaligned_le32(in_key + 20);
for (i = 0; i < 8; ++i)
loop6(i);
break;
case AES_KEYSIZE_256:
- ctx->key_enc[4] = le32_to_cpu(key[4]);
- ctx->key_enc[5] = le32_to_cpu(key[5]);
- ctx->key_enc[6] = le32_to_cpu(key[6]);
- t = ctx->key_enc[7] = le32_to_cpu(key[7]);
+ ctx->key_enc[4] = get_unaligned_le32(in_key + 16);
+ ctx->key_enc[5] = get_unaligned_le32(in_key + 20);
+ ctx->key_enc[6] = get_unaligned_le32(in_key + 24);
+ t = ctx->key_enc[7] = get_unaligned_le32(in_key + 28);
for (i = 0; i < 6; ++i)
loop8(i);
loop8tophalf(i);
@@ -1329,16 +1334,14 @@ EXPORT_SYMBOL_GPL(crypto_aes_set_key);
static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
{
const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
- const __le32 *src = (const __le32 *)in;
- __le32 *dst = (__le32 *)out;
u32 b0[4], b1[4];
const u32 *kp = ctx->key_enc + 4;
const int key_len = ctx->key_length;
- b0[0] = le32_to_cpu(src[0]) ^ ctx->key_enc[0];
- b0[1] = le32_to_cpu(src[1]) ^ ctx->key_enc[1];
- b0[2] = le32_to_cpu(src[2]) ^ ctx->key_enc[2];
- b0[3] = le32_to_cpu(src[3]) ^ ctx->key_enc[3];
+ b0[0] = ctx->key_enc[0] ^ get_unaligned_le32(in);
+ b0[1] = ctx->key_enc[1] ^ get_unaligned_le32(in + 4);
+ b0[2] = ctx->key_enc[2] ^ get_unaligned_le32(in + 8);
+ b0[3] = ctx->key_enc[3] ^ get_unaligned_le32(in + 12);
if (key_len > 24) {
f_nround(b1, b0, kp);
@@ -1361,10 +1364,10 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
f_nround(b1, b0, kp);
f_lround(b0, b1, kp);
- dst[0] = cpu_to_le32(b0[0]);
- dst[1] = cpu_to_le32(b0[1]);
- dst[2] = cpu_to_le32(b0[2]);
- dst[3] = cpu_to_le32(b0[3]);
+ put_unaligned_le32(b0[0], out);
+ put_unaligned_le32(b0[1], out + 4);
+ put_unaligned_le32(b0[2], out + 8);
+ put_unaligned_le32(b0[3], out + 12);
}
/* decrypt a block of text */
@@ -1401,16 +1404,14 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
{
const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
- const __le32 *src = (const __le32 *)in;
- __le32 *dst = (__le32 *)out;
u32 b0[4], b1[4];
const int key_len = ctx->key_length;
const u32 *kp = ctx->key_dec + 4;
- b0[0] = le32_to_cpu(src[0]) ^ ctx->key_dec[0];
- b0[1] = le32_to_cpu(src[1]) ^ ctx->key_dec[1];
- b0[2] = le32_to_cpu(src[2]) ^ ctx->key_dec[2];
- b0[3] = le32_to_cpu(src[3]) ^ ctx->key_dec[3];
+ b0[0] = ctx->key_dec[0] ^ get_unaligned_le32(in);
+ b0[1] = ctx->key_dec[1] ^ get_unaligned_le32(in + 4);
+ b0[2] = ctx->key_dec[2] ^ get_unaligned_le32(in + 8);
+ b0[3] = ctx->key_dec[3] ^ get_unaligned_le32(in + 12);
if (key_len > 24) {
i_nround(b1, b0, kp);
@@ -1433,10 +1434,10 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
i_nround(b1, b0, kp);
i_lround(b0, b1, kp);
- dst[0] = cpu_to_le32(b0[0]);
- dst[1] = cpu_to_le32(b0[1]);
- dst[2] = cpu_to_le32(b0[2]);
- dst[3] = cpu_to_le32(b0[3]);
+ put_unaligned_le32(b0[0], out);
+ put_unaligned_le32(b0[1], out + 4);
+ put_unaligned_le32(b0[2], out + 8);
+ put_unaligned_le32(b0[3], out + 12);
}
static struct crypto_alg aes_alg = {
@@ -1446,7 +1447,6 @@ static struct crypto_alg aes_alg = {
.cra_flags = CRYPTO_ALG_TYPE_CIPHER,
.cra_blocksize = AES_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct crypto_aes_ctx),
- .cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_u = {
.cipher = {
--
2.7.4
^ permalink raw reply related
* Re: crypto: NULL deref in sha512_mb_mgr_get_comp_job_avx2
From: Tim Chen @ 2017-02-02 16:00 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Herbert Xu, David Miller, linux-crypto, LKML, megha.dey,
fenghua.yu, syzkaller
In-Reply-To: <CACT4Y+bwD4ktMKVEkcmvJ_EuW834jAZbJfANxfyDKpmGXFxr4Q@mail.gmail.com>
On Thu, 2017-02-02 at 11:58 +0100, Dmitry Vyukov wrote:
> On Wed, Feb 1, 2017 at 7:45 PM, Tim Chen <tim.c.chen@linux.intel.com> wrote:
> >
> > On Tue, Jan 31, 2017 at 02:16:31PM +0100, Dmitry Vyukov wrote:
> > >
> > > Hello,
> > >
> > > I am getting the following reports with low frequency while running
> > > syzkaller fuzzer. Unfortunately they are not reproducible and happen
> > > in a background thread, so it is difficult to extract any context on
> > > my side. I see only few such crashes per week, so most likely it is
> > > some hard to trigger data race. The following reports are from mmotm
> > > tree, commits 00e20cfc2bf04a0cbe1f5405f61c8426f43eee84 and
> > > fff7e71eac7788904753136f09bcad7471f7799e. Any ideas as to how this can
> > > happen?
> > >
> > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
> > > IP: [<ffffffff813fc09e>] sha512_mb_mgr_get_comp_job_avx2+0x6e/0xee
> > > arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S:251
> > > PGD 1d2395067 [ 220.874864] PUD 1d2860067
> > > Oops: 0002 [#1] SMP KASAN
> > > Dumping ftrace buffer:
> > > (ftrace buffer empty)
> > > Modules linked in:
> > > CPU: 0 PID: 516 Comm: kworker/0:1 Not tainted 4.9.0 #4
> > > Hardware name: Google Google Compute Engine/Google Compute Engine,
> > > BIOS Google 01/01/2011
> > > Workqueue: crypto mcryptd_queue_worker
> > > task: ffff8801d9f346c0 task.stack: ffff8801d9f08000
> > > RIP: 0010:[<ffffffff813fc09e>] [<ffffffff813fc09e>]
> > > sha512_mb_mgr_get_comp_job_avx2+0x6e/0xee
> > > arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S:251
> > > RSP: 0018:ffff8801d9f0eef8 EFLAGS: 00010202
> > > RAX: 0000000000000000 RBX: ffff8801d7db1190 RCX: 0000000000000006
> > > RDX: 0000000000000001 RSI: ffff8801d9f34ee8 RDI: ffff8801d7db1040
> > > RBP: ffff8801d9f0f258 R08: 0000000100000000 R09: 0000000000000001
> > > R10: 0000000000000002 R11: 0000000000000003 R12: ffff8801d9f0f230
> > > R13: ffff8801c8bbc4e0 R14: ffff8801c8bbc530 R15: ffff8801d9f0ef70
> > > FS: 0000000000000000(0000) GS:ffff8801dc000000(0000) knlGS:0000000000000000
> > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > CR2: 0000000000000060 CR3: 00000001cc15a000 CR4: 00000000001406f0
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > Stack:
> > > ffff8801d7db1040 ffffffff813fa207 dffffc0000000000 ffffe8ffffc0f238
> > > 0000000000000002 1ffff1003b3e1dea ffffe8ffffc0f218 ffff8801d9f0f190
> > > 0000000000000282 ffffe8ffffc0f140 ffffe8ffffc0f220 0000000041b58ab3
> > > Call Trace:
> > > [<ffffffff813fb407>] sha512_mb_update+0x2f7/0x4e0
> > > arch/x86/crypto/sha512-mb/sha512_mb.c:588
> > > [<ffffffff8219d4ad>] crypto_ahash_update include/crypto/hash.h:512 [inline]
> > > [<ffffffff8219d4ad>] ahash_mcryptd_update crypto/mcryptd.c:627 [inline]
> > > [<ffffffff8219d4ad>] mcryptd_hash_update+0xcd/0x1c0 crypto/mcryptd.c:373
> > > [<ffffffff8219c99f>] mcryptd_queue_worker+0xff/0x6a0 crypto/mcryptd.c:181
> > > [<ffffffff81492960>] process_one_work+0xbd0/0x1c10 kernel/workqueue.c:2096
> > > [<ffffffff81493bc3>] worker_thread+0x223/0x1990 kernel/workqueue.c:2230
> > > [<ffffffff814abb33>] kthread+0x323/0x3e0 kernel/kthread.c:209
> > > [<ffffffff8436fbaa>] ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:433
> > > Code: 49 0f 42 d3 48 f7 c2 f0 ff ff ff 0f 85 9a 00 00 00 48 83 e2 0f
> > > 48 6b da 08 48 8d 9c 1f 48 01 00 00 48 8b 03 48 c7 03 00 00 00 00 <c7>
> > > 40 60 02 00 00 00 48 8b 9f 40 01 00 00 48 c1 e3 08 48 09 d3
> > > RIP [<ffffffff813fc09e>] sha512_mb_mgr_get_comp_job_avx2+0x6e/0xee
> > > arch/x86/crypto/sha512-mb/sha512_mb_mgr_flush_avx2.S:251
> > > RSP <ffff8801d9f0eef8>
> > > CR2: 0000000000000060
> > > ---[ end trace 139fd4cda5dfe2c4 ]---
> > >
> > Dmitry,
> >
> > One theory that Mehga and I have is that perhaps the flusher
> > and regular computaion updates are stepping on each other.
> > Can you try this patch and see if it helps?
>
> No, for this one I can't. Sorry.
> It happens with very low frequency and only one fuzzer that tests
> mmotm tree. If/when this is committed, I can keep an eye on these
> reports and notify if I still see them.
> If you have a hypothesis as to how it happens, perhaps you could write
> a test that provokes the crash and maybe add some sleeps to kernel
> code or alter timeouts to increase probability.
>
If this patch is merged, it will most likely go into Herbert's crypto-dev
tree and not Andrew's mm tree for testing. We will try to do the best
on our side for testing to replicate the crash scenario.
Will it be possible to have one of your fuzzer run the crypto-dev tree
once the patch got merged there?
Thanks.
Tim
>
>
> >
> > --->8---
> >
> > From: Tim Chen <tim.c.chen@linux.intel.com>
> > Subject: [PATCH] crypto/sha512-mb: Protect sha512 mb ctx mgr access
> > To: Herbert Xu <herbert@gondor.apana.org.au>, Dmitry Vyukov <dvyukov@google.com>
> > Cc: Tim Chen <tim.c.chen@linux.intel.com>, David Miller <davem@davemloft.net>, linux-crypto@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>, megha.dey@linux.intel.com, fenghua.yu@intel.com
> >
> > The flusher and regular multi-buffer computation via mcryptd may race with another.
> > Add here a lock and turn off interrupt to to access multi-buffer
> > computation state cstate->mgr before a round of computation. This should
> > prevent the flusher code jumping in.
> >
> > Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> > ---
> > arch/x86/crypto/sha512-mb/sha512_mb.c | 64 +++++++++++++++++++++++------------
> > 1 file changed, 42 insertions(+), 22 deletions(-)
> >
> > diff --git a/arch/x86/crypto/sha512-mb/sha512_mb.c b/arch/x86/crypto/sha512-mb/sha512_mb.c
> > index d210174..f3c1c21 100644
> > --- a/arch/x86/crypto/sha512-mb/sha512_mb.c
> > +++ b/arch/x86/crypto/sha512-mb/sha512_mb.c
> > @@ -221,7 +221,7 @@ static struct sha512_hash_ctx *sha512_ctx_mgr_resubmit
> > }
> >
> > static struct sha512_hash_ctx
> > - *sha512_ctx_mgr_get_comp_ctx(struct sha512_ctx_mgr *mgr)
> > + *sha512_ctx_mgr_get_comp_ctx(struct mcryptd_alg_cstate *cstate)
> > {
> > /*
> > * If get_comp_job returns NULL, there are no jobs complete.
> > @@ -233,11 +233,17 @@ static struct sha512_hash_ctx
> > * Otherwise, all jobs currently being managed by the hash_ctx_mgr
> > * still need processing.
> > */
> > + struct sha512_ctx_mgr *mgr;
> > struct sha512_hash_ctx *ctx;
> > + unsigned long flags;
> >
> > + mgr = cstate->mgr;
> > + spin_lock_irqsave(&cstate->work_lock, flags);
> > ctx = (struct sha512_hash_ctx *)
> > sha512_job_mgr_get_comp_job(&mgr->mgr);
> > - return sha512_ctx_mgr_resubmit(mgr, ctx);
> > + ctx = sha512_ctx_mgr_resubmit(mgr, ctx);
> > + spin_unlock_irqrestore(&cstate->work_lock, flags);
> > + return ctx;
> > }
> >
> > static void sha512_ctx_mgr_init(struct sha512_ctx_mgr *mgr)
> > @@ -246,12 +252,17 @@ static void sha512_ctx_mgr_init(struct sha512_ctx_mgr *mgr)
> > }
> >
> > static struct sha512_hash_ctx
> > - *sha512_ctx_mgr_submit(struct sha512_ctx_mgr *mgr,
> > + *sha512_ctx_mgr_submit(struct mcryptd_alg_cstate *cstate,
> > struct sha512_hash_ctx *ctx,
> > const void *buffer,
> > uint32_t len,
> > int flags)
> > {
> > + struct sha512_ctx_mgr *mgr;
> > + unsigned long irqflags;
> > +
> > + mgr = cstate->mgr;
> > + spin_lock_irqsave(&cstate->work_lock, irqflags);
> > if (flags & (~HASH_ENTIRE)) {
> > /*
> > * User should not pass anything other than FIRST, UPDATE, or
> > @@ -351,20 +362,26 @@ static struct sha512_hash_ctx
> > }
> > }
> >
> > - return sha512_ctx_mgr_resubmit(mgr, ctx);
> > + ctx = sha512_ctx_mgr_resubmit(mgr, ctx);
> > + spin_unlock_irqrestore(&cstate->work_lock, irqflags);
> > + return ctx;
> > }
> >
> > -static struct sha512_hash_ctx *sha512_ctx_mgr_flush(struct sha512_ctx_mgr *mgr)
> > +static struct sha512_hash_ctx *sha512_ctx_mgr_flush(struct mcryptd_alg_cstate *cstate)
> > {
> > + struct sha512_ctx_mgr *mgr;
> > struct sha512_hash_ctx *ctx;
> > + unsigned long flags;
> >
> > + mgr = cstate->mgr;
> > + spin_lock_irqsave(&cstate->work_lock, flags);
> > while (1) {
> > ctx = (struct sha512_hash_ctx *)
> > sha512_job_mgr_flush(&mgr->mgr);
> >
> > /* If flush returned 0, there are no more jobs in flight. */
> > if (!ctx)
> > - return NULL;
> > + break;
> >
> > /*
> > * If flush returned a job, resubmit the job to finish
> > @@ -378,8 +395,10 @@ static struct sha512_hash_ctx *sha512_ctx_mgr_flush(struct sha512_ctx_mgr *mgr)
> > * the sha512_ctx_mgr still need processing. Loop.
> > */
> > if (ctx)
> > - return ctx;
> > + break;
> > }
> > + spin_unlock_irqrestore(&cstate->work_lock, flags);
> > + return ctx;
> > }
> >
> > static int sha512_mb_init(struct ahash_request *areq)
> > @@ -439,11 +458,11 @@ static int sha_finish_walk(struct mcryptd_hash_request_ctx **ret_rctx,
> > sha_ctx = (struct sha512_hash_ctx *)
> > ahash_request_ctx(&rctx->areq);
> > kernel_fpu_begin();
> > - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx,
> > + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx,
> > rctx->walk.data, nbytes, flag);
> > if (!sha_ctx) {
> > if (flush)
> > - sha_ctx = sha512_ctx_mgr_flush(cstate->mgr);
> > + sha_ctx = sha512_ctx_mgr_flush(cstate);
> > }
> > kernel_fpu_end();
> > if (sha_ctx)
> > @@ -471,11 +490,12 @@ static int sha_complete_job(struct mcryptd_hash_request_ctx *rctx,
> > struct sha512_hash_ctx *sha_ctx;
> > struct mcryptd_hash_request_ctx *req_ctx;
> > int ret;
> > + unsigned long flags;
> >
> > /* remove from work list */
> > - spin_lock(&cstate->work_lock);
> > + spin_lock_irqsave(&cstate->work_lock, flags);
> > list_del(&rctx->waiter);
> > - spin_unlock(&cstate->work_lock);
> > + spin_unlock_irqrestore(&cstate->work_lock, flags);
> >
> > if (irqs_disabled())
> > rctx->complete(&req->base, err);
> > @@ -486,14 +506,14 @@ static int sha_complete_job(struct mcryptd_hash_request_ctx *rctx,
> > }
> >
> > /* check to see if there are other jobs that are done */
> > - sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate->mgr);
> > + sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate);
> > while (sha_ctx) {
> > req_ctx = cast_hash_to_mcryptd_ctx(sha_ctx);
> > ret = sha_finish_walk(&req_ctx, cstate, false);
> > if (req_ctx) {
> > - spin_lock(&cstate->work_lock);
> > + spin_lock_irqsave(&cstate->work_lock, flags);
> > list_del(&req_ctx->waiter);
> > - spin_unlock(&cstate->work_lock);
> > + spin_unlock_irqrestore(&cstate->work_lock, flags);
> >
> > req = cast_mcryptd_ctx_to_req(req_ctx);
> > if (irqs_disabled())
> > @@ -504,7 +524,7 @@ static int sha_complete_job(struct mcryptd_hash_request_ctx *rctx,
> > local_bh_enable();
> > }
> > }
> > - sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate->mgr);
> > + sha_ctx = sha512_ctx_mgr_get_comp_ctx(cstate);
> > }
> >
> > return 0;
> > @@ -515,6 +535,7 @@ static void sha512_mb_add_list(struct mcryptd_hash_request_ctx *rctx,
> > {
> > unsigned long next_flush;
> > unsigned long delay = usecs_to_jiffies(FLUSH_INTERVAL);
> > + unsigned long flags;
> >
> > /* initialize tag */
> > rctx->tag.arrival = jiffies; /* tag the arrival time */
> > @@ -522,9 +543,9 @@ static void sha512_mb_add_list(struct mcryptd_hash_request_ctx *rctx,
> > next_flush = rctx->tag.arrival + delay;
> > rctx->tag.expire = next_flush;
> >
> > - spin_lock(&cstate->work_lock);
> > + spin_lock_irqsave(&cstate->work_lock, flags);
> > list_add_tail(&rctx->waiter, &cstate->work_list);
> > - spin_unlock(&cstate->work_lock);
> > + spin_unlock_irqrestore(&cstate->work_lock, flags);
> >
> > mcryptd_arm_flusher(cstate, delay);
> > }
> > @@ -565,7 +586,7 @@ static int sha512_mb_update(struct ahash_request *areq)
> > sha_ctx = (struct sha512_hash_ctx *) ahash_request_ctx(areq);
> > sha512_mb_add_list(rctx, cstate);
> > kernel_fpu_begin();
> > - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data,
> > + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx, rctx->walk.data,
> > nbytes, HASH_UPDATE);
> > kernel_fpu_end();
> >
> > @@ -628,7 +649,7 @@ static int sha512_mb_finup(struct ahash_request *areq)
> > sha512_mb_add_list(rctx, cstate);
> >
> > kernel_fpu_begin();
> > - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx, rctx->walk.data,
> > + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx, rctx->walk.data,
> > nbytes, flag);
> > kernel_fpu_end();
> >
> > @@ -677,8 +698,7 @@ static int sha512_mb_final(struct ahash_request *areq)
> > /* flag HASH_FINAL and 0 data size */
> > sha512_mb_add_list(rctx, cstate);
> > kernel_fpu_begin();
> > - sha_ctx = sha512_ctx_mgr_submit(cstate->mgr, sha_ctx, &data, 0,
> > - HASH_LAST);
> > + sha_ctx = sha512_ctx_mgr_submit(cstate, sha_ctx, &data, 0, HASH_LAST);
> > kernel_fpu_end();
> >
> > /* check if anything is returned */
> > @@ -940,7 +960,7 @@ static unsigned long sha512_mb_flusher(struct mcryptd_alg_cstate *cstate)
> > break;
> > kernel_fpu_begin();
> > sha_ctx = (struct sha512_hash_ctx *)
> > - sha512_ctx_mgr_flush(cstate->mgr);
> > + sha512_ctx_mgr_flush(cstate);
> > kernel_fpu_end();
> > if (!sha_ctx) {
> > pr_err("sha512_mb error: nothing got flushed for"
> > --
> > 2.5.5
> >
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox