* Re: XTS Crypto Not Found In /proc/crypto Even After Compiled for 4.10.1.
From: Nathan Royce @ 2017-03-03 14:08 UTC (permalink / raw)
To: Herbert Xu
Cc: davem, linux-crypto, linux-kernel, Krzysztof Kozlowski,
Marek Szyprowski
In-Reply-To: <20170303120432.GA26540@gondor.apana.org.au>
Yup, when I disabled the s5p driver, xts DID show in the /proc/crypto list.
Heh, I was about to ask if it was something I should push towards
another maintainer for s5p stuff, but found you listed in that as
well.
If I am incorrect in that assumption, do let me know whom else I
should make aware of this issue.
Also let me know if you would like the rest of the kernel panic. Maybe
you already have enough to go on and don't need it.
Thanks for all that clarity.
On Fri, Mar 3, 2017 at 6:04 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Fri, Mar 03, 2017 at 04:36:18AM -0600, Nathan Royce wrote:
>> I do have ECB selected as well:
>> DM_CRYPT=y
>> CRYPTO_ECB=y
>> CRYPTO_XTS=y
>>
>> name : ecb(aes)
>> driver : ecb-aes-s5p
>> module : kernel
>> priority : 100
>> refcnt : 1
>> selftest : passed
>> internal : no
>> type : ablkcipher
>> async : yes
>> blocksize : 16
>> min keysize : 16
>> max keysize : 32
>> ivsize : 0
>> geniv : <default>
>> //still no "xts" can be found in the list
>
> Weird. So you can't find any instances of xts in /proc/crypto
> at all? Even if the self-test fails it should still register an
> entry there...
>
> In any case, I think disabling the s5p driver should work at
> least.
>
> Cheers,
> --
> Email: Herbert Xu <herbert@gondor.apana.org.au>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [RFC 0/7] crypto: caam - add Queue Interface (QI) support
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Alexandru Porosanu
The patchset adds support for CAAM Queue Interface (QI), the additional
interface (besides job ring) available for submitting jobs to the engine
on platforms having DPAA (Datapath Acceleration Architecture).
Patches 1-4 are QMan dependencies.
I would prefer to take them through the crypto tree,
but I am open to suggestions.
Patch 5 adds a missing double inclusion guard in desc_constr.h.
Patch 6 adds the caam/qi job submission backend.
Patch 7 adds algorithms (ablkcipher and authenc) that run on top
of caam/qi. For now, their priority is set lower than caam/jr.
Thanks,
Horia
Horia Geantă (7):
soc/qman: export volatile dequeue related structs
soc/qman: add dedicated channel ID for CAAM
soc/qman: export non-programmable FQD fields query
soc/qman: add helper functions needed by caam/qi driver
crypto: caam - avoid double inclusion in desc_constr.h
crypto: caam - add Queue Interface (QI) backend support
crypto: caam/qi - add ablkcipher and authenc algorithms
drivers/crypto/caam/Kconfig | 20 +-
drivers/crypto/caam/Makefile | 5 +
drivers/crypto/caam/caamalg.c | 9 +-
drivers/crypto/caam/caamalg_desc.c | 77 +-
drivers/crypto/caam/caamalg_desc.h | 15 +-
drivers/crypto/caam/caamalg_qi.c | 2387 +++++++++++++++++++++++++++++++
drivers/crypto/caam/ctrl.c | 58 +-
drivers/crypto/caam/desc_constr.h | 5 +
drivers/crypto/caam/intern.h | 24 +
drivers/crypto/caam/qi.c | 805 +++++++++++
drivers/crypto/caam/qi.h | 201 +++
drivers/crypto/caam/sg_sw_qm.h | 107 ++
drivers/soc/fsl/qbman/qman.c | 20 +-
drivers/soc/fsl/qbman/qman_ccsr.c | 6 +-
drivers/soc/fsl/qbman/qman_priv.h | 97 --
drivers/soc/fsl/qbman/qman_test_stash.c | 5 +-
include/soc/fsl/qman.h | 154 ++
17 files changed, 3839 insertions(+), 156 deletions(-)
create mode 100644 drivers/crypto/caam/caamalg_qi.c
create mode 100644 drivers/crypto/caam/qi.c
create mode 100644 drivers/crypto/caam/qi.h
create mode 100644 drivers/crypto/caam/sg_sw_qm.h
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply
* [RFC 1/7] soc/qman: export volatile dequeue related structs
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
Since qman_volatile_dequeue() is already exported, move the related
structures into the public header too.
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/soc/fsl/qbman/qman_priv.h | 36 ------------------------------------
include/soc/fsl/qman.h | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 36 insertions(+), 36 deletions(-)
diff --git a/drivers/soc/fsl/qbman/qman_priv.h b/drivers/soc/fsl/qbman/qman_priv.h
index 53685b59718e..64781eff6974 100644
--- a/drivers/soc/fsl/qbman/qman_priv.h
+++ b/drivers/soc/fsl/qbman/qman_priv.h
@@ -271,42 +271,6 @@ const struct qm_portal_config *qman_destroy_affine_portal(void);
*/
int qman_query_fq(struct qman_fq *fq, struct qm_fqd *fqd);
-/*
- * For qman_volatile_dequeue(); Choose one PRECEDENCE. EXACT is optional. Use
- * NUMFRAMES(n) (6-bit) or NUMFRAMES_TILLEMPTY to fill in the frame-count. Use
- * FQID(n) to fill in the frame queue ID.
- */
-#define QM_VDQCR_PRECEDENCE_VDQCR 0x0
-#define QM_VDQCR_PRECEDENCE_SDQCR 0x80000000
-#define QM_VDQCR_EXACT 0x40000000
-#define QM_VDQCR_NUMFRAMES_MASK 0x3f000000
-#define QM_VDQCR_NUMFRAMES_SET(n) (((n) & 0x3f) << 24)
-#define QM_VDQCR_NUMFRAMES_GET(n) (((n) >> 24) & 0x3f)
-#define QM_VDQCR_NUMFRAMES_TILLEMPTY QM_VDQCR_NUMFRAMES_SET(0)
-
-#define QMAN_VOLATILE_FLAG_WAIT 0x00000001 /* wait if VDQCR is in use */
-#define QMAN_VOLATILE_FLAG_WAIT_INT 0x00000002 /* if wait, interruptible? */
-#define QMAN_VOLATILE_FLAG_FINISH 0x00000004 /* wait till VDQCR completes */
-
-/*
- * qman_volatile_dequeue - Issue a volatile dequeue command
- * @fq: the frame queue object to dequeue from
- * @flags: a bit-mask of QMAN_VOLATILE_FLAG_*** options
- * @vdqcr: bit mask of QM_VDQCR_*** options, as per qm_dqrr_vdqcr_set()
- *
- * Attempts to lock access to the portal's VDQCR volatile dequeue functionality.
- * The function will block and sleep if QMAN_VOLATILE_FLAG_WAIT is specified and
- * the VDQCR is already in use, otherwise returns non-zero for failure. If
- * QMAN_VOLATILE_FLAG_FINISH is specified, the function will only return once
- * the VDQCR command has finished executing (ie. once the callback for the last
- * DQRR entry resulting from the VDQCR command has been called). If not using
- * the FINISH flag, completion can be determined either by detecting the
- * presence of the QM_DQRR_STAT_UNSCHEDULED and QM_DQRR_STAT_DQCR_EXPIRED bits
- * in the "stat" parameter passed to the FQ's dequeue callback, or by waiting
- * for the QMAN_FQ_STATE_VDQCR bit to disappear.
- */
-int qman_volatile_dequeue(struct qman_fq *fq, u32 flags, u32 vdqcr);
-
int qman_alloc_fq_table(u32 num_fqids);
/* QMan s/w corenet portal, low-level i/face */
diff --git a/include/soc/fsl/qman.h b/include/soc/fsl/qman.h
index 3d4df74a96de..4de1ffcc8982 100644
--- a/include/soc/fsl/qman.h
+++ b/include/soc/fsl/qman.h
@@ -791,6 +791,23 @@ struct qman_cgr {
#define QMAN_INITFQ_FLAG_SCHED 0x00000001 /* schedule rather than park */
#define QMAN_INITFQ_FLAG_LOCAL 0x00000004 /* set dest portal */
+/*
+ * For qman_volatile_dequeue(); Choose one PRECEDENCE. EXACT is optional. Use
+ * NUMFRAMES(n) (6-bit) or NUMFRAMES_TILLEMPTY to fill in the frame-count. Use
+ * FQID(n) to fill in the frame queue ID.
+ */
+#define QM_VDQCR_PRECEDENCE_VDQCR 0x0
+#define QM_VDQCR_PRECEDENCE_SDQCR 0x80000000
+#define QM_VDQCR_EXACT 0x40000000
+#define QM_VDQCR_NUMFRAMES_MASK 0x3f000000
+#define QM_VDQCR_NUMFRAMES_SET(n) (((n) & 0x3f) << 24)
+#define QM_VDQCR_NUMFRAMES_GET(n) (((n) >> 24) & 0x3f)
+#define QM_VDQCR_NUMFRAMES_TILLEMPTY QM_VDQCR_NUMFRAMES_SET(0)
+
+#define QMAN_VOLATILE_FLAG_WAIT 0x00000001 /* wait if VDQCR is in use */
+#define QMAN_VOLATILE_FLAG_WAIT_INT 0x00000002 /* if wait, interruptible? */
+#define QMAN_VOLATILE_FLAG_FINISH 0x00000004 /* wait till VDQCR completes */
+
/* Portal Management */
/**
* qman_p_irqsource_add - add processing sources to be interrupt-driven
@@ -963,6 +980,25 @@ int qman_retire_fq(struct qman_fq *fq, u32 *flags);
*/
int qman_oos_fq(struct qman_fq *fq);
+/*
+ * qman_volatile_dequeue - Issue a volatile dequeue command
+ * @fq: the frame queue object to dequeue from
+ * @flags: a bit-mask of QMAN_VOLATILE_FLAG_*** options
+ * @vdqcr: bit mask of QM_VDQCR_*** options, as per qm_dqrr_vdqcr_set()
+ *
+ * Attempts to lock access to the portal's VDQCR volatile dequeue functionality.
+ * The function will block and sleep if QMAN_VOLATILE_FLAG_WAIT is specified and
+ * the VDQCR is already in use, otherwise returns non-zero for failure. If
+ * QMAN_VOLATILE_FLAG_FINISH is specified, the function will only return once
+ * the VDQCR command has finished executing (ie. once the callback for the last
+ * DQRR entry resulting from the VDQCR command has been called). If not using
+ * the FINISH flag, completion can be determined either by detecting the
+ * presence of the QM_DQRR_STAT_UNSCHEDULED and QM_DQRR_STAT_DQCR_EXPIRED bits
+ * in the "stat" parameter passed to the FQ's dequeue callback, or by waiting
+ * for the QMAN_FQ_STATE_VDQCR bit to disappear.
+ */
+int qman_volatile_dequeue(struct qman_fq *fq, u32 flags, u32 vdqcr);
+
/**
* qman_enqueue - Enqueue a frame to a frame queue
* @fq: the frame queue object to enqueue to
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [RFC 2/7] soc/qman: add dedicated channel ID for CAAM
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
Add and export the ID of the channel serviced by the
CAAM (Cryptographic Acceleration and Assurance Module) DCP.
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/soc/fsl/qbman/qman_ccsr.c | 6 +++++-
include/soc/fsl/qman.h | 3 +++
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/soc/fsl/qbman/qman_ccsr.c b/drivers/soc/fsl/qbman/qman_ccsr.c
index f4e6e70de259..90bc40c48675 100644
--- a/drivers/soc/fsl/qbman/qman_ccsr.c
+++ b/drivers/soc/fsl/qbman/qman_ccsr.c
@@ -34,6 +34,8 @@ u16 qman_ip_rev;
EXPORT_SYMBOL(qman_ip_rev);
u16 qm_channel_pool1 = QMAN_CHANNEL_POOL1;
EXPORT_SYMBOL(qm_channel_pool1);
+u16 qm_channel_caam = QMAN_CHANNEL_CAAM;
+EXPORT_SYMBOL(qm_channel_caam);
/* Register offsets */
#define REG_QCSP_LIO_CFG(n) (0x0000 + ((n) * 0x10))
@@ -720,8 +722,10 @@ static int fsl_qman_probe(struct platform_device *pdev)
return -ENODEV;
}
- if ((qman_ip_rev & 0xff00) >= QMAN_REV30)
+ if ((qman_ip_rev & 0xff00) >= QMAN_REV30) {
qm_channel_pool1 = QMAN_CHANNEL_POOL1_REV3;
+ qm_channel_caam = QMAN_CHANNEL_CAAM_REV3;
+ }
ret = zero_priv_mem(dev, node, fqd_a, fqd_sz);
WARN_ON(ret);
diff --git a/include/soc/fsl/qman.h b/include/soc/fsl/qman.h
index 4de1ffcc8982..10b549783ec5 100644
--- a/include/soc/fsl/qman.h
+++ b/include/soc/fsl/qman.h
@@ -36,8 +36,11 @@
/* Hardware constants */
#define QM_CHANNEL_SWPORTAL0 0
#define QMAN_CHANNEL_POOL1 0x21
+#define QMAN_CHANNEL_CAAM 0x80
#define QMAN_CHANNEL_POOL1_REV3 0x401
+#define QMAN_CHANNEL_CAAM_REV3 0x840
extern u16 qm_channel_pool1;
+extern u16 qm_channel_caam;
/* Portal processing (interrupt) sources */
#define QM_PIRQ_CSCI 0x00100000 /* Congestion State Change */
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [RFC 3/7] soc/qman: export non-programmable FQD fields query
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
Export qman_query_fq_np() function and related structures.
This will be needed in the caam/qi driver, where "queue empty"
condition will be decided based on the frm_cnt.
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/soc/fsl/qbman/qman.c | 4 +--
drivers/soc/fsl/qbman/qman_priv.h | 61 -----------------------------------
include/soc/fsl/qman.h | 68 +++++++++++++++++++++++++++++++++++++++
3 files changed, 70 insertions(+), 63 deletions(-)
diff --git a/drivers/soc/fsl/qbman/qman.c b/drivers/soc/fsl/qbman/qman.c
index 6f509f68085e..3d891db57ee6 100644
--- a/drivers/soc/fsl/qbman/qman.c
+++ b/drivers/soc/fsl/qbman/qman.c
@@ -2019,8 +2019,7 @@ int qman_query_fq(struct qman_fq *fq, struct qm_fqd *fqd)
return ret;
}
-static int qman_query_fq_np(struct qman_fq *fq,
- struct qm_mcr_queryfq_np *np)
+int qman_query_fq_np(struct qman_fq *fq, struct qm_mcr_queryfq_np *np)
{
union qm_mc_command *mcc;
union qm_mc_result *mcr;
@@ -2046,6 +2045,7 @@ static int qman_query_fq_np(struct qman_fq *fq,
put_affine_portal();
return ret;
}
+EXPORT_SYMBOL(qman_query_fq_np);
static int qman_query_cgr(struct qman_cgr *cgr,
struct qm_mcr_querycgr *cgrd)
diff --git a/drivers/soc/fsl/qbman/qman_priv.h b/drivers/soc/fsl/qbman/qman_priv.h
index 64781eff6974..22725bdc6f15 100644
--- a/drivers/soc/fsl/qbman/qman_priv.h
+++ b/drivers/soc/fsl/qbman/qman_priv.h
@@ -89,67 +89,6 @@ static inline u64 qm_mcr_querycgr_a_get64(const struct qm_mcr_querycgr *q)
return ((u64)q->a_bcnt_hi << 32) | be32_to_cpu(q->a_bcnt_lo);
}
-/* "Query FQ Non-Programmable Fields" */
-
-struct qm_mcr_queryfq_np {
- u8 verb;
- u8 result;
- u8 __reserved1;
- u8 state; /* QM_MCR_NP_STATE_*** */
- u32 fqd_link; /* 24-bit, _res2[24-31] */
- u16 odp_seq; /* 14-bit, _res3[14-15] */
- u16 orp_nesn; /* 14-bit, _res4[14-15] */
- u16 orp_ea_hseq; /* 15-bit, _res5[15] */
- u16 orp_ea_tseq; /* 15-bit, _res6[15] */
- u32 orp_ea_hptr; /* 24-bit, _res7[24-31] */
- u32 orp_ea_tptr; /* 24-bit, _res8[24-31] */
- u32 pfdr_hptr; /* 24-bit, _res9[24-31] */
- u32 pfdr_tptr; /* 24-bit, _res10[24-31] */
- u8 __reserved2[5];
- u8 is; /* 1-bit, _res12[1-7] */
- u16 ics_surp;
- u32 byte_cnt;
- u32 frm_cnt; /* 24-bit, _res13[24-31] */
- u32 __reserved3;
- u16 ra1_sfdr; /* QM_MCR_NP_RA1_*** */
- u16 ra2_sfdr; /* QM_MCR_NP_RA2_*** */
- u16 __reserved4;
- u16 od1_sfdr; /* QM_MCR_NP_OD1_*** */
- u16 od2_sfdr; /* QM_MCR_NP_OD2_*** */
- u16 od3_sfdr; /* QM_MCR_NP_OD3_*** */
-} __packed;
-
-#define QM_MCR_NP_STATE_FE 0x10
-#define QM_MCR_NP_STATE_R 0x08
-#define QM_MCR_NP_STATE_MASK 0x07 /* Reads FQD::STATE; */
-#define QM_MCR_NP_STATE_OOS 0x00
-#define QM_MCR_NP_STATE_RETIRED 0x01
-#define QM_MCR_NP_STATE_TEN_SCHED 0x02
-#define QM_MCR_NP_STATE_TRU_SCHED 0x03
-#define QM_MCR_NP_STATE_PARKED 0x04
-#define QM_MCR_NP_STATE_ACTIVE 0x05
-#define QM_MCR_NP_PTR_MASK 0x07ff /* for RA[12] & OD[123] */
-#define QM_MCR_NP_RA1_NRA(v) (((v) >> 14) & 0x3) /* FQD::NRA */
-#define QM_MCR_NP_RA2_IT(v) (((v) >> 14) & 0x1) /* FQD::IT */
-#define QM_MCR_NP_OD1_NOD(v) (((v) >> 14) & 0x3) /* FQD::NOD */
-#define QM_MCR_NP_OD3_NPC(v) (((v) >> 14) & 0x3) /* FQD::NPC */
-
-enum qm_mcr_queryfq_np_masks {
- qm_mcr_fqd_link_mask = BIT(24)-1,
- qm_mcr_odp_seq_mask = BIT(14)-1,
- qm_mcr_orp_nesn_mask = BIT(14)-1,
- qm_mcr_orp_ea_hseq_mask = BIT(15)-1,
- qm_mcr_orp_ea_tseq_mask = BIT(15)-1,
- qm_mcr_orp_ea_hptr_mask = BIT(24)-1,
- qm_mcr_orp_ea_tptr_mask = BIT(24)-1,
- qm_mcr_pfdr_hptr_mask = BIT(24)-1,
- qm_mcr_pfdr_tptr_mask = BIT(24)-1,
- qm_mcr_is_mask = BIT(1)-1,
- qm_mcr_frm_cnt_mask = BIT(24)-1,
-};
-#define qm_mcr_np_get(np, field) \
- ((np)->field & (qm_mcr_##field##_mask))
-
/* Congestion Groups */
/*
diff --git a/include/soc/fsl/qman.h b/include/soc/fsl/qman.h
index 10b549783ec5..0252c32f7421 100644
--- a/include/soc/fsl/qman.h
+++ b/include/soc/fsl/qman.h
@@ -811,6 +811,67 @@ struct qman_cgr {
#define QMAN_VOLATILE_FLAG_WAIT_INT 0x00000002 /* if wait, interruptible? */
#define QMAN_VOLATILE_FLAG_FINISH 0x00000004 /* wait till VDQCR completes */
+/* "Query FQ Non-Programmable Fields" */
+struct qm_mcr_queryfq_np {
+ u8 verb;
+ u8 result;
+ u8 __reserved1;
+ u8 state; /* QM_MCR_NP_STATE_*** */
+ u32 fqd_link; /* 24-bit, _res2[24-31] */
+ u16 odp_seq; /* 14-bit, _res3[14-15] */
+ u16 orp_nesn; /* 14-bit, _res4[14-15] */
+ u16 orp_ea_hseq; /* 15-bit, _res5[15] */
+ u16 orp_ea_tseq; /* 15-bit, _res6[15] */
+ u32 orp_ea_hptr; /* 24-bit, _res7[24-31] */
+ u32 orp_ea_tptr; /* 24-bit, _res8[24-31] */
+ u32 pfdr_hptr; /* 24-bit, _res9[24-31] */
+ u32 pfdr_tptr; /* 24-bit, _res10[24-31] */
+ u8 __reserved2[5];
+ u8 is; /* 1-bit, _res12[1-7] */
+ u16 ics_surp;
+ u32 byte_cnt;
+ u32 frm_cnt; /* 24-bit, _res13[24-31] */
+ u32 __reserved3;
+ u16 ra1_sfdr; /* QM_MCR_NP_RA1_*** */
+ u16 ra2_sfdr; /* QM_MCR_NP_RA2_*** */
+ u16 __reserved4;
+ u16 od1_sfdr; /* QM_MCR_NP_OD1_*** */
+ u16 od2_sfdr; /* QM_MCR_NP_OD2_*** */
+ u16 od3_sfdr; /* QM_MCR_NP_OD3_*** */
+} __packed;
+
+#define QM_MCR_NP_STATE_FE 0x10
+#define QM_MCR_NP_STATE_R 0x08
+#define QM_MCR_NP_STATE_MASK 0x07 /* Reads FQD::STATE; */
+#define QM_MCR_NP_STATE_OOS 0x00
+#define QM_MCR_NP_STATE_RETIRED 0x01
+#define QM_MCR_NP_STATE_TEN_SCHED 0x02
+#define QM_MCR_NP_STATE_TRU_SCHED 0x03
+#define QM_MCR_NP_STATE_PARKED 0x04
+#define QM_MCR_NP_STATE_ACTIVE 0x05
+#define QM_MCR_NP_PTR_MASK 0x07ff /* for RA[12] & OD[123] */
+#define QM_MCR_NP_RA1_NRA(v) (((v) >> 14) & 0x3) /* FQD::NRA */
+#define QM_MCR_NP_RA2_IT(v) (((v) >> 14) & 0x1) /* FQD::IT */
+#define QM_MCR_NP_OD1_NOD(v) (((v) >> 14) & 0x3) /* FQD::NOD */
+#define QM_MCR_NP_OD3_NPC(v) (((v) >> 14) & 0x3) /* FQD::NPC */
+
+enum qm_mcr_queryfq_np_masks {
+ qm_mcr_fqd_link_mask = BIT(24) - 1,
+ qm_mcr_odp_seq_mask = BIT(14) - 1,
+ qm_mcr_orp_nesn_mask = BIT(14) - 1,
+ qm_mcr_orp_ea_hseq_mask = BIT(15) - 1,
+ qm_mcr_orp_ea_tseq_mask = BIT(15) - 1,
+ qm_mcr_orp_ea_hptr_mask = BIT(24) - 1,
+ qm_mcr_orp_ea_tptr_mask = BIT(24) - 1,
+ qm_mcr_pfdr_hptr_mask = BIT(24) - 1,
+ qm_mcr_pfdr_tptr_mask = BIT(24) - 1,
+ qm_mcr_is_mask = BIT(1) - 1,
+ qm_mcr_frm_cnt_mask = BIT(24) - 1,
+};
+
+#define qm_mcr_np_get(np, field) \
+ ((np)->field & (qm_mcr_##field##_mask))
+
/* Portal Management */
/**
* qman_p_irqsource_add - add processing sources to be interrupt-driven
@@ -1033,6 +1094,13 @@ int qman_alloc_fqid_range(u32 *result, u32 count);
*/
int qman_release_fqid(u32 fqid);
+/**
+ * qman_query_fq_np - Queries non-programmable FQD fields
+ * @fq: the frame queue object to be queried
+ * @np: storage for the queried FQD fields
+ */
+int qman_query_fq_np(struct qman_fq *fq, struct qm_mcr_queryfq_np *np);
+
/* Pool-channel management */
/**
* qman_alloc_pool_range - Allocate a contiguous range of pool-channel IDs
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [RFC 4/7] soc/qman: add helper functions needed by caam/qi driver
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
Add helper functions, macros, #defines for accessing / enabling
qman functionality from caam/qi driver, such that this driver
is not aware of the need for data conversion to big endian.
qman is updated to use these helpers internally.
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/soc/fsl/qbman/qman.c | 16 +++++------
drivers/soc/fsl/qbman/qman_test_stash.c | 5 ++--
include/soc/fsl/qman.h | 47 +++++++++++++++++++++++++++++++++
3 files changed, 56 insertions(+), 12 deletions(-)
diff --git a/drivers/soc/fsl/qbman/qman.c b/drivers/soc/fsl/qbman/qman.c
index 3d891db57ee6..7668ff53cd90 100644
--- a/drivers/soc/fsl/qbman/qman.c
+++ b/drivers/soc/fsl/qbman/qman.c
@@ -1764,16 +1764,15 @@ int qman_init_fq(struct qman_fq *fq, u32 flags, struct qm_mcc_initfq *opts)
if (fq_isclear(fq, QMAN_FQ_FLAG_TO_DCPORTAL)) {
dma_addr_t phys_fq;
- mcc->initfq.we_mask |= cpu_to_be16(QM_INITFQ_WE_CONTEXTB);
- mcc->initfq.fqd.context_b = cpu_to_be32(fq_to_tag(fq));
+ qm_initfq_setbits(&mcc->initfq, QM_INITFQ_WE_CONTEXTB);
+ qm_fqd_set_contextb(&mcc->initfq.fqd, fq_to_tag(fq));
/*
* and the physical address - NB, if the user wasn't trying to
* set CONTEXTA, clear the stashing settings.
*/
if (!(be16_to_cpu(mcc->initfq.we_mask) &
QM_INITFQ_WE_CONTEXTA)) {
- mcc->initfq.we_mask |=
- cpu_to_be16(QM_INITFQ_WE_CONTEXTA);
+ qm_initfq_setbits(&mcc->initfq, QM_INITFQ_WE_CONTEXTA);
memset(&mcc->initfq.fqd.context_a, 0,
sizeof(mcc->initfq.fqd.context_a));
} else {
@@ -1795,8 +1794,7 @@ int qman_init_fq(struct qman_fq *fq, u32 flags, struct qm_mcc_initfq *opts)
if (!(be16_to_cpu(mcc->initfq.we_mask) &
QM_INITFQ_WE_DESTWQ)) {
- mcc->initfq.we_mask |=
- cpu_to_be16(QM_INITFQ_WE_DESTWQ);
+ qm_initfq_setbits(&mcc->initfq, QM_INITFQ_WE_DESTWQ);
wq = 4;
}
qm_fqd_set_destwq(&mcc->initfq.fqd, p->config->channel, wq);
@@ -1816,7 +1814,7 @@ int qman_init_fq(struct qman_fq *fq, u32 flags, struct qm_mcc_initfq *opts)
}
if (opts) {
if (be16_to_cpu(opts->we_mask) & QM_INITFQ_WE_FQCTRL) {
- if (be16_to_cpu(opts->fqd.fq_ctrl) & QM_FQCTRL_CGE)
+ if (qm_fqd_isset_fqctrl(&opts->fqd, QM_FQCTRL_CGE))
fq_set(fq, QMAN_FQ_STATE_CGR_EN);
else
fq_clear(fq, QMAN_FQ_STATE_CGR_EN);
@@ -2321,7 +2319,7 @@ int qman_create_cgr(struct qman_cgr *cgr, u32 flags,
qm_cgr_cscn_targ_set(&local_opts.cgr, PORTAL_IDX(p),
be32_to_cpu(cgr_state.cgr.cscn_targ));
- local_opts.we_mask |= cpu_to_be16(QM_CGR_WE_CSCN_TARG);
+ qm_initcgr_setbits(&local_opts, QM_CGR_WE_CSCN_TARG);
/* send init if flags indicate so */
if (flags & QMAN_CGR_FLAG_USE_INIT)
@@ -2840,7 +2838,7 @@ static int cgr_cleanup(u32 cgrid)
err = qman_query_fq(&fq, &fqd);
if (WARN_ON(err))
return err;
- if (be16_to_cpu(fqd.fq_ctrl) & QM_FQCTRL_CGE &&
+ if (qm_fqd_isset_fqctrl(&fqd, QM_FQCTRL_CGE) &&
fqd.cgid == cgrid) {
pr_err("CRGID 0x%x is being used by FQID 0x%x, CGR will be leaked\n",
cgrid, fq.fqid);
diff --git a/drivers/soc/fsl/qbman/qman_test_stash.c b/drivers/soc/fsl/qbman/qman_test_stash.c
index e87b65403b67..d2bf453092d7 100644
--- a/drivers/soc/fsl/qbman/qman_test_stash.c
+++ b/drivers/soc/fsl/qbman/qman_test_stash.c
@@ -406,9 +406,8 @@ static int init_handler(void *h)
goto failed;
}
memset(&opts, 0, sizeof(opts));
- opts.we_mask = cpu_to_be16(QM_INITFQ_WE_FQCTRL |
- QM_INITFQ_WE_CONTEXTA);
- opts.fqd.fq_ctrl = cpu_to_be16(QM_FQCTRL_CTXASTASHING);
+ qm_initfq_setbits(&opts, QM_INITFQ_WE_FQCTRL | QM_INITFQ_WE_CONTEXTA);
+ qm_fqd_set_fqctrl(&opts.fqd, QM_FQCTRL_CTXASTASHING);
qm_fqd_set_stashing(&opts.fqd, 0, STASH_DATA_CL, STASH_CTX_CL);
err = qman_init_fq(&handler->rx, QMAN_INITFQ_FLAG_SCHED |
QMAN_INITFQ_FLAG_LOCAL, &opts);
diff --git a/include/soc/fsl/qman.h b/include/soc/fsl/qman.h
index 0252c32f7421..fc133c658385 100644
--- a/include/soc/fsl/qman.h
+++ b/include/soc/fsl/qman.h
@@ -168,6 +168,12 @@ static inline void qm_fd_set_param(struct qm_fd *fd, enum qm_fd_format fmt,
#define qm_fd_set_contig_big(fd, len) \
qm_fd_set_param(fd, qm_fd_contig_big, 0, len)
#define qm_fd_set_sg_big(fd, len) qm_fd_set_param(fd, qm_fd_sg_big, 0, len)
+#define qm_fd_set_compound(fd, len) qm_fd_set_param(fd, qm_fd_compound, 0, len)
+
+static inline int qm_fd_get_status(const struct qm_fd *fd)
+{
+ return be32_to_cpu(fd->status);
+}
static inline void qm_fd_clear_fd(struct qm_fd *fd)
{
@@ -233,16 +239,31 @@ static inline void qm_sg_entry_set_len(struct qm_sg_entry *sg, int len)
sg->cfg = cpu_to_be32(len & QM_SG_LEN_MASK);
}
+static inline void qm_sg_entry_set_e(struct qm_sg_entry *sg, int len)
+{
+ sg->cfg = cpu_to_be32(QM_SG_EXT | (len & QM_SG_LEN_MASK));
+}
+
static inline void qm_sg_entry_set_f(struct qm_sg_entry *sg, int len)
{
sg->cfg = cpu_to_be32(QM_SG_FIN | (len & QM_SG_LEN_MASK));
}
+static inline void qm_sg_entry_set_ef(struct qm_sg_entry *sg, int len)
+{
+ sg->cfg = cpu_to_be32(QM_SG_EXT | QM_SG_FIN | (len & QM_SG_LEN_MASK));
+}
+
static inline int qm_sg_entry_get_off(const struct qm_sg_entry *sg)
{
return be32_to_cpu(sg->offset) & QM_SG_OFF_MASK;
}
+static inline void qm_sg_entry_set_off(struct qm_sg_entry *sg, int off)
+{
+ sg->offset = cpu_to_be16(off & QM_SG_OFF_MASK);
+}
+
/* "Frame Dequeue Response" */
struct qm_dqrr_entry {
u8 verb;
@@ -494,6 +515,21 @@ static inline int qm_fqd_get_wq(const struct qm_fqd *fqd)
return be16_to_cpu(fqd->dest_wq) & QM_FQD_WQ_MASK;
}
+static inline bool qm_fqd_isset_fqctrl(const struct qm_fqd *fqd, u16 mask)
+{
+ return be16_to_cpu(fqd->fq_ctrl) & mask;
+}
+
+static inline void qm_fqd_set_fqctrl(struct qm_fqd *fqd, int val)
+{
+ fqd->fq_ctrl = cpu_to_be16(val);
+}
+
+static inline void qm_fqd_set_contextb(struct qm_fqd *fqd, int val)
+{
+ fqd->context_b = cpu_to_be32(val);
+}
+
/* See "Frame Queue Descriptor (FQD)" */
/* Frame Queue Descriptor (FQD) field 'fq_ctrl' uses these constants */
#define QM_FQCTRL_MASK 0x07ff /* 'fq_ctrl' flags; */
@@ -616,6 +652,16 @@ struct qm_mcc_initcgr {
u8 __reserved3[32];
} __packed;
+static inline void qm_initfq_setbits(struct qm_mcc_initfq *p, u16 we_mask)
+{
+ p->we_mask |= cpu_to_be16(we_mask);
+}
+
+static inline void qm_initcgr_setbits(struct qm_mcc_initcgr *p, u16 we_mask)
+{
+ p->we_mask |= cpu_to_be16(we_mask);
+}
+
/* INITFQ-specific flags */
#define QM_INITFQ_WE_MASK 0x01ff /* 'Write Enable' flags; */
#define QM_INITFQ_WE_OAC 0x0100
@@ -642,6 +688,7 @@ struct qm_mcc_initcgr {
#define QM_CGR_WE_MODE 0x0001
#define QMAN_CGR_FLAG_USE_INIT 0x00000001
+#define QMAN_CGR_MODE_FRAME 0x00000001
/* Portal and Frame Queues */
/* Represents a managed portal */
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [RFC 5/7] crypto: caam - avoid double inclusion in desc_constr.h
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/crypto/caam/desc_constr.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/drivers/crypto/caam/desc_constr.h b/drivers/crypto/caam/desc_constr.h
index b9c8d98ef826..d8e83ca104e0 100644
--- a/drivers/crypto/caam/desc_constr.h
+++ b/drivers/crypto/caam/desc_constr.h
@@ -4,6 +4,9 @@
* Copyright 2008-2012 Freescale Semiconductor, Inc.
*/
+#ifndef DESC_CONSTR_H
+#define DESC_CONSTR_H
+
#include "desc.h"
#include "regs.h"
@@ -491,3 +494,5 @@ static inline int desc_inline_query(unsigned int sd_base_len,
return (rem_bytes >= 0) ? 0 : -1;
}
+
+#endif /* DESC_CONSTR_H */
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [RFC 6/7] crypto: caam - add Queue Interface (QI) backend support
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
CAAM engine supports two interfaces for crypto job submission:
-job ring interface - already existing caam/jr driver
-Queue Interface (QI) - caam/qi driver added in current patch
QI is present in CAAM engines found on DPAA platforms.
QI gets its I/O (frame descriptors) from QMan (Queue Manager) queues.
This patch adds a platform device for accessing CAAM's queue interface.
The requests are submitted to CAAM using one frame queue per
cryptographic context. Each crypto context has one shared descriptor.
This shared descriptor is attached to frame queue associated with
corresponding driver context using context_a.
The driver hides the mechanics of FQ creation, initialisation from its
applications. Each cryptographic context needs to be associated with
driver context which houses the FQ to be used to transport the job to
CAAM. The driver provides API for:
(a) Context creation
(b) Job submission
(c) Context deletion
(d) Congestion indication - whether path to/from CAAM is congested
The driver supports affining its context to a particular CPU.
This means that any responses from CAAM for the context in question
would arrive at the given CPU. This helps in implementing one CPU
per packet round trip in IPsec application.
The driver processes CAAM responses under NAPI contexts.
NAPI contexts are instantiated only on cores with affined portals since
only cores having their own portal can receive responses from DQRR.
The responses from CAAM for all cryptographic contexts ride on a fixed
set of FQs. We use one response FQ per portal owning core. The response
FQ is configured in each core's and thus portal's dedicated channel.
This gives the flexibility to direct CAAM's responses for a crypto
context on a given core.
Signed-off-by: Vakul Garg <vakul.garg@nxp.com>
Signed-off-by: Alex Porosanu <alexandru.porosanu@nxp.com>
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/crypto/caam/Makefile | 4 +
drivers/crypto/caam/ctrl.c | 58 ++--
drivers/crypto/caam/intern.h | 24 ++
drivers/crypto/caam/qi.c | 805 +++++++++++++++++++++++++++++++++++++++++++
drivers/crypto/caam/qi.h | 201 +++++++++++
5 files changed, 1064 insertions(+), 28 deletions(-)
create mode 100644 drivers/crypto/caam/qi.c
create mode 100644 drivers/crypto/caam/qi.h
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index 6554742f357e..2e60e45c2bf1 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -16,3 +16,7 @@ obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API) += caam_pkc.o
caam-objs := ctrl.o
caam_jr-objs := jr.o key_gen.o error.o
caam_pkc-y := caampkc.o pkc_desc.o
+ifneq ($(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI),)
+ ccflags-y += -DCONFIG_CAAM_QI
+ caam-objs += qi.o
+endif
diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index 579f8263c479..9f91de315c30 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -18,6 +18,10 @@
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
+#ifdef CONFIG_CAAM_QI
+#include "qi.h"
+#endif
+
/*
* i.MX targets tend to have clock control subsystems that can
* enable/disable clocking to our device.
@@ -311,6 +315,11 @@ static int caam_remove(struct platform_device *pdev)
for (ring = 0; ring < ctrlpriv->total_jobrs; ring++)
of_device_unregister(ctrlpriv->jrpdev[ring]);
+#ifdef CONFIG_CAAM_QI
+ if (ctrlpriv->qidev)
+ caam_qi_shutdown(ctrlpriv->qidev);
+#endif
+
/* De-initialize RNG state handles initialized by this driver. */
if (ctrlpriv->rng4_sh_init)
deinstantiate_rng(ctrldev, ctrlpriv->rng4_sh_init);
@@ -401,23 +410,6 @@ int caam_get_era(void)
}
EXPORT_SYMBOL(caam_get_era);
-#ifdef CONFIG_DEBUG_FS
-static int caam_debugfs_u64_get(void *data, u64 *val)
-{
- *val = caam64_to_cpu(*(u64 *)data);
- return 0;
-}
-
-static int caam_debugfs_u32_get(void *data, u64 *val)
-{
- *val = caam32_to_cpu(*(u32 *)data);
- return 0;
-}
-
-DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u32_ro, caam_debugfs_u32_get, NULL, "%llu\n");
-DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u64_ro, caam_debugfs_u64_get, NULL, "%llu\n");
-#endif
-
/* Probe routine for CAAM top (controller) level */
static int caam_probe(struct platform_device *pdev)
{
@@ -615,6 +607,17 @@ static int caam_probe(struct platform_device *pdev)
goto iounmap_ctrl;
}
+#ifdef CONFIG_DEBUG_FS
+ /*
+ * FIXME: needs better naming distinction, as some amalgamation of
+ * "caam" and nprop->full_name. The OF name isn't distinctive,
+ * but does separate instances
+ */
+ perfmon = (struct caam_perfmon __force *)&ctrl->perfmon;
+
+ ctrlpriv->dfs_root = debugfs_create_dir(dev_name(dev), NULL);
+ ctrlpriv->ctl = debugfs_create_dir("ctl", ctrlpriv->dfs_root);
+#endif
ring = 0;
ridx = 0;
ctrlpriv->total_jobrs = 0;
@@ -650,6 +653,13 @@ static int caam_probe(struct platform_device *pdev)
);
/* This is all that's required to physically enable QI */
wr_reg32(&ctrlpriv->qi->qi_control_lo, QICTL_DQEN);
+
+ /* If QMAN driver is present, init CAAM-QI backend */
+#ifdef CONFIG_CAAM_QI
+ ret = caam_qi_init(pdev);
+ if (ret)
+ dev_err(dev, "caam qi i/f init failed: %d\n", ret);
+#endif
}
/* If no QI and no rings specified, quit and go home */
@@ -737,17 +747,6 @@ static int caam_probe(struct platform_device *pdev)
ctrlpriv->total_jobrs, ctrlpriv->qi_present);
#ifdef CONFIG_DEBUG_FS
- /*
- * FIXME: needs better naming distinction, as some amalgamation of
- * "caam" and nprop->full_name. The OF name isn't distinctive,
- * but does separate instances
- */
- perfmon = (struct caam_perfmon __force *)&ctrl->perfmon;
-
- ctrlpriv->dfs_root = debugfs_create_dir(dev_name(dev), NULL);
- ctrlpriv->ctl = debugfs_create_dir("ctl", ctrlpriv->dfs_root);
-
- /* Controller-level - performance monitor counters */
ctrlpriv->ctl_rq_dequeued =
debugfs_create_file("rq_dequeued",
@@ -830,6 +829,9 @@ static int caam_probe(struct platform_device *pdev)
return 0;
caam_remove:
+#ifdef CONFIG_DEBUG_FS
+ debugfs_remove_recursive(ctrlpriv->dfs_root);
+#endif
caam_remove(pdev);
return ret;
diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h
index e2bcacc1a921..c334df638ff6 100644
--- a/drivers/crypto/caam/intern.h
+++ b/drivers/crypto/caam/intern.h
@@ -67,6 +67,9 @@ struct caam_drv_private {
struct device *dev;
struct platform_device **jrpdev; /* Alloc'ed array per sub-device */
+#ifdef CONFIG_CAAM_QI
+ struct device *qidev;
+#endif
struct platform_device *pdev;
/* Physical-presence section */
@@ -110,9 +113,30 @@ struct caam_drv_private {
struct debugfs_blob_wrapper ctl_kek_wrap, ctl_tkek_wrap, ctl_tdsk_wrap;
struct dentry *ctl_kek, *ctl_tkek, *ctl_tdsk;
+#ifdef CONFIG_CAAM_QI
+ struct dentry *qi_congested;
+#endif
#endif
};
void caam_jr_algapi_init(struct device *dev);
void caam_jr_algapi_remove(struct device *dev);
+
+#ifdef CONFIG_DEBUG_FS
+static int caam_debugfs_u64_get(void *data, u64 *val)
+{
+ *val = caam64_to_cpu(*(u64 *)data);
+ return 0;
+}
+
+static int caam_debugfs_u32_get(void *data, u64 *val)
+{
+ *val = caam32_to_cpu(*(u32 *)data);
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u32_ro, caam_debugfs_u32_get, NULL, "%llu\n");
+DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u64_ro, caam_debugfs_u64_get, NULL, "%llu\n");
+#endif
+
#endif /* INTERN_H */
diff --git a/drivers/crypto/caam/qi.c b/drivers/crypto/caam/qi.c
new file mode 100644
index 000000000000..6a7f32fc7557
--- /dev/null
+++ b/drivers/crypto/caam/qi.c
@@ -0,0 +1,805 @@
+/*
+ * CAAM/SEC 4.x QI transport/backend driver
+ * Queue Interface backend functionality
+ *
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ */
+
+#include <linux/cpumask.h>
+#include <linux/kthread.h>
+#include <soc/fsl/qman.h>
+
+#include "regs.h"
+#include "qi.h"
+#include "desc.h"
+#include "intern.h"
+#include "desc_constr.h"
+
+#define PREHDR_RSLS_SHIFT 31
+
+/*
+ * Use a reasonable backlog of frames (per CPU) as congestion threshold,
+ * so that resources used by the in-flight buffers do not become a memory hog.
+ */
+#define MAX_RSP_FQ_BACKLOG_PER_CPU 256
+
+/* Length of a single buffer in the QI driver memory cache */
+#define CAAM_QI_MEMCACHE_SIZE 512
+
+#define CAAM_QI_ENQUEUE_RETRIES 10000
+
+#define CAAM_NAPI_WEIGHT 63
+
+/*
+ * caam_napi - struct holding CAAM NAPI-related params
+ * @irqtask: IRQ task for QI backend
+ * @p: QMan portal
+ */
+struct caam_napi {
+ struct napi_struct irqtask;
+ struct qman_portal *p;
+};
+
+/*
+ * caam_qi_pcpu_priv - percpu private data structure to main list of pending
+ * responses expected on each cpu.
+ * @caam_napi: CAAM NAPI params
+ * @net_dev: netdev used by NAPI
+ * @rsp_fq: response FQ from CAAM
+ */
+struct caam_qi_pcpu_priv {
+ struct caam_napi caam_napi;
+ struct net_device net_dev;
+ struct qman_fq *rsp_fq;
+} ____cacheline_aligned;
+
+static DEFINE_PER_CPU(struct caam_qi_pcpu_priv, pcpu_qipriv);
+
+/*
+ * caam_qi_priv - CAAM QI backend private params
+ * @cgr: QMan congestion group
+ * @qi_pdev: platform device for QI backend
+ */
+struct caam_qi_priv {
+ struct qman_cgr cgr;
+ struct platform_device *qi_pdev;
+};
+
+static struct caam_qi_priv qipriv ____cacheline_aligned;
+
+/*
+ * This is written by only one core - the one that initialized the CGR - and
+ * read by multiple cores (all the others).
+ */
+bool caam_congested __read_mostly;
+EXPORT_SYMBOL(caam_congested);
+
+#ifdef CONFIG_DEBUG_FS
+/*
+ * This is a counter for the number of times the congestion group (where all
+ * the request and response queueus are) reached congestion. Incremented
+ * each time the congestion callback is called with congested == true.
+ */
+static u64 times_congested;
+#endif
+
+/*
+ * CPU from where the module initialised. This is required because QMan driver
+ * requires CGRs to be removed from same CPU from where they were originally
+ * allocated.
+ */
+static int mod_init_cpu;
+
+/*
+ * This is a a cache of buffers, from which the users of CAAM QI driver
+ * can allocate short (CAAM_QI_MEMCACHE_SIZE) buffers. It's faster than
+ * doing malloc on the hotpath.
+ * NOTE: A more elegant solution would be to have some headroom in the frames
+ * being processed. This could be added by the dpaa-ethernet driver.
+ * This would pose a problem for userspace application processing which
+ * cannot know of this limitation. So for now, this will work.
+ * NOTE: The memcache is SMP-safe. No need to handle spinlocks in-here
+ */
+static struct kmem_cache *qi_cache;
+
+int caam_qi_enqueue(struct device *qidev, struct caam_drv_req *req)
+{
+ struct qm_fd fd;
+ dma_addr_t addr;
+ int ret;
+ int num_retries = 0;
+
+ qm_fd_clear_fd(&fd);
+ qm_fd_set_compound(&fd, qm_sg_entry_get_len(&req->fd_sgt[1]));
+
+ addr = dma_map_single(qidev, req->fd_sgt, sizeof(req->fd_sgt),
+ DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(qidev, addr)) {
+ dev_err(qidev, "DMA mapping error for QI enqueue request\n");
+ return -EIO;
+ }
+ qm_fd_addr_set64(&fd, addr);
+
+ do {
+ ret = qman_enqueue(req->drv_ctx->req_fq, &fd);
+ if (likely(!ret))
+ return 0;
+
+ if (ret != -EBUSY)
+ break;
+ num_retries++;
+ } while (num_retries < CAAM_QI_ENQUEUE_RETRIES);
+
+ dev_err(qidev, "qman_enqueue failed: %d\n", ret);
+
+ return ret;
+}
+EXPORT_SYMBOL(caam_qi_enqueue);
+
+static void caam_fq_ern_cb(struct qman_portal *qm, struct qman_fq *fq,
+ const union qm_mr_entry *msg)
+{
+ const struct qm_fd *fd;
+ struct caam_drv_req *drv_req;
+ struct device *qidev = &(raw_cpu_ptr(&pcpu_qipriv)->net_dev.dev);
+
+ fd = &msg->ern.fd;
+
+ if (qm_fd_get_format(fd) != qm_fd_compound) {
+ dev_err(qidev, "Non-compound FD from CAAM\n");
+ return;
+ }
+
+ drv_req = (struct caam_drv_req *)phys_to_virt(qm_fd_addr_get64(fd));
+ if (!drv_req) {
+ dev_err(qidev,
+ "Can't find original request for CAAM response\n");
+ return;
+ }
+
+ dma_unmap_single(drv_req->drv_ctx->qidev, qm_fd_addr(fd),
+ sizeof(drv_req->fd_sgt), DMA_BIDIRECTIONAL);
+
+ drv_req->cbk(drv_req, -EIO);
+}
+
+static struct qman_fq *create_caam_req_fq(struct device *qidev,
+ struct qman_fq *rsp_fq,
+ dma_addr_t hwdesc,
+ int fq_sched_flag)
+{
+ int ret;
+ struct qman_fq *req_fq;
+ struct qm_mcc_initfq opts;
+
+ req_fq = kzalloc(sizeof(*req_fq), GFP_ATOMIC);
+ if (!req_fq)
+ return ERR_PTR(-ENOMEM);
+
+ req_fq->cb.ern = caam_fq_ern_cb;
+ req_fq->cb.fqs = NULL;
+
+ ret = qman_create_fq(0, QMAN_FQ_FLAG_DYNAMIC_FQID |
+ QMAN_FQ_FLAG_TO_DCPORTAL, req_fq);
+ if (ret) {
+ dev_err(qidev, "Failed to create session req FQ\n");
+ goto create_req_fq_fail;
+ }
+
+ memset(&opts, 0, sizeof(opts));
+ qm_initfq_setbits(&opts, QM_INITFQ_WE_FQCTRL | QM_INITFQ_WE_DESTWQ |
+ QM_INITFQ_WE_CONTEXTB | QM_INITFQ_WE_CONTEXTA |
+ QM_INITFQ_WE_CGID);
+ qm_fqd_set_fqctrl(&opts.fqd, QM_FQCTRL_CPCSTASH | QM_FQCTRL_CGE);
+ qm_fqd_set_destwq(&opts.fqd, qm_channel_caam, 2);
+ qm_fqd_set_contextb(&opts.fqd, qman_fq_fqid(rsp_fq));
+ qm_fqd_context_a_set64(&opts.fqd, hwdesc);
+ opts.fqd.cgid = qipriv.cgr.cgrid;
+
+ ret = qman_init_fq(req_fq, fq_sched_flag, &opts);
+ if (ret) {
+ dev_err(qidev, "Failed to init session req FQ\n");
+ goto init_req_fq_fail;
+ }
+
+ dev_info(qidev, "Allocated request FQ %u for CPU %u\n", req_fq->fqid,
+ smp_processor_id());
+ return req_fq;
+
+init_req_fq_fail:
+ qman_destroy_fq(req_fq);
+create_req_fq_fail:
+ kfree(req_fq);
+ return ERR_PTR(ret);
+}
+
+static int empty_retired_fq(struct device *qidev, struct qman_fq *fq)
+{
+ int ret;
+
+ ret = qman_volatile_dequeue(fq, QMAN_VOLATILE_FLAG_WAIT_INT |
+ QMAN_VOLATILE_FLAG_FINISH,
+ QM_VDQCR_PRECEDENCE_VDQCR |
+ QM_VDQCR_NUMFRAMES_TILLEMPTY);
+ if (ret) {
+ dev_err(qidev, "Volatile dequeue fail for FQ: %u\n", fq->fqid);
+ return ret;
+ }
+
+ do {
+ struct qman_portal *p;
+
+ p = qman_get_affine_portal(smp_processor_id());
+ qman_p_poll_dqrr(p, 16);
+ } while (fq->flags & QMAN_FQ_STATE_NE);
+
+ return 0;
+}
+
+static int kill_fq(struct device *qidev, struct qman_fq *fq)
+{
+ u32 flags;
+ int ret;
+
+ ret = qman_retire_fq(fq, &flags);
+ if (ret < 0) {
+ dev_err(qidev, "qman_retire_fq failed: %d\n", ret);
+ return ret;
+ }
+
+ if (!ret)
+ goto empty_fq;
+
+ /* Async FQ retirement condition */
+ if (ret == 1) {
+ /* Retry till FQ gets in retired state */
+ do {
+ msleep(20);
+ } while (fq->state != qman_fq_state_retired);
+
+ WARN_ON(fq->flags & QMAN_FQ_STATE_BLOCKOOS);
+ WARN_ON(fq->flags & QMAN_FQ_STATE_ORL);
+ }
+
+empty_fq:
+ if (fq->flags & QMAN_FQ_STATE_NE) {
+ ret = empty_retired_fq(qidev, fq);
+ if (ret) {
+ dev_err(qidev, "empty_retired_fq fail for FQ: %u\n",
+ fq->fqid);
+ return ret;
+ }
+ }
+
+ ret = qman_oos_fq(fq);
+ if (ret)
+ dev_err(qidev, "OOS of FQID: %u failed\n", fq->fqid);
+
+ qman_destroy_fq(fq);
+
+ return ret;
+}
+
+static int empty_caam_fq(struct qman_fq *fq)
+{
+ int ret;
+ struct qm_mcr_queryfq_np np;
+
+ /* Wait till the older CAAM FQ get empty */
+ do {
+ ret = qman_query_fq_np(fq, &np);
+ if (ret)
+ return ret;
+
+ if (!qm_mcr_np_get(&np, frm_cnt))
+ break;
+
+ msleep(20);
+ } while (1);
+
+ /*
+ * Give extra time for pending jobs from this FQ in holding tanks
+ * to get processed
+ */
+ msleep(20);
+ return 0;
+}
+
+int caam_drv_ctx_update(struct caam_drv_ctx *drv_ctx, u32 *sh_desc)
+{
+ int ret;
+ u32 num_words;
+ struct qman_fq *new_fq, *old_fq;
+ struct device *qidev = drv_ctx->qidev;
+
+ num_words = desc_len(sh_desc);
+ if (num_words > MAX_SDLEN) {
+ dev_err(qidev, "Invalid descriptor len: %d words\n", num_words);
+ return -EINVAL;
+ }
+
+ /* Note down older req FQ */
+ old_fq = drv_ctx->req_fq;
+
+ /* Create a new req FQ in parked state */
+ new_fq = create_caam_req_fq(drv_ctx->qidev, drv_ctx->rsp_fq,
+ drv_ctx->context_a, 0);
+ if (unlikely(IS_ERR_OR_NULL(new_fq))) {
+ dev_err(qidev, "FQ allocation for shdesc update failed\n");
+ return PTR_ERR(new_fq);
+ }
+
+ /* Hook up new FQ to context so that new requests keep queuing */
+ drv_ctx->req_fq = new_fq;
+
+ /* Empty and remove the older FQ */
+ ret = empty_caam_fq(old_fq);
+ if (ret) {
+ dev_err(qidev, "Old CAAM FQ empty failed: %d\n", ret);
+
+ /* We can revert to older FQ */
+ drv_ctx->req_fq = old_fq;
+
+ if (kill_fq(qidev, new_fq))
+ dev_warn(qidev, "New CAAM FQ: %u kill failed\n",
+ new_fq->fqid);
+
+ return ret;
+ }
+
+ /*
+ * Re-initialise pre-header. Set RSLS and SDLEN.
+ * Update the shared descriptor for driver context.
+ */
+ drv_ctx->prehdr[0] = cpu_to_caam32((1 << PREHDR_RSLS_SHIFT) |
+ num_words);
+ memcpy(drv_ctx->sh_desc, sh_desc, desc_bytes(sh_desc));
+ dma_sync_single_for_device(qidev, drv_ctx->context_a,
+ sizeof(drv_ctx->sh_desc) +
+ sizeof(drv_ctx->prehdr),
+ DMA_BIDIRECTIONAL);
+
+ /* Put the new FQ in scheduled state */
+ ret = qman_schedule_fq(new_fq);
+ if (ret) {
+ dev_err(qidev, "Fail to sched new CAAM FQ, ecode = %d\n", ret);
+
+ /*
+ * We can kill new FQ and revert to old FQ.
+ * Since the desc is already modified, it is success case
+ */
+
+ drv_ctx->req_fq = old_fq;
+
+ if (kill_fq(qidev, new_fq))
+ dev_warn(qidev, "New CAAM FQ: %u kill failed\n",
+ new_fq->fqid);
+ } else if (kill_fq(qidev, old_fq)) {
+ dev_warn(qidev, "Old CAAM FQ: %u kill failed\n", old_fq->fqid);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(caam_drv_ctx_update);
+
+struct caam_drv_ctx *caam_drv_ctx_init(struct device *qidev,
+ int *cpu,
+ u32 *sh_desc)
+{
+ size_t size;
+ u32 num_words;
+ dma_addr_t hwdesc;
+ struct caam_drv_ctx *drv_ctx;
+ const cpumask_t *cpus = qman_affine_cpus();
+ static DEFINE_PER_CPU(int, last_cpu);
+
+ num_words = desc_len(sh_desc);
+ if (num_words > MAX_SDLEN) {
+ dev_err(qidev, "Invalid descriptor len: %d words\n",
+ num_words);
+ return ERR_PTR(-EINVAL);
+ }
+
+ drv_ctx = kzalloc(sizeof(*drv_ctx), GFP_ATOMIC);
+ if (!drv_ctx)
+ return ERR_PTR(-ENOMEM);
+
+ /*
+ * Initialise pre-header - set RSLS and SDLEN - and shared descriptor
+ * and dma-map them.
+ */
+ drv_ctx->prehdr[0] = cpu_to_caam32((1 << PREHDR_RSLS_SHIFT) |
+ num_words);
+ memcpy(drv_ctx->sh_desc, sh_desc, desc_bytes(sh_desc));
+ size = sizeof(drv_ctx->prehdr) + sizeof(drv_ctx->sh_desc);
+ hwdesc = dma_map_single(qidev, drv_ctx->prehdr, size,
+ DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(qidev, hwdesc)) {
+ dev_err(qidev, "DMA map error for preheader + shdesc\n");
+ kfree(drv_ctx);
+ return ERR_PTR(-ENOMEM);
+ }
+ drv_ctx->context_a = hwdesc;
+
+ /* If given CPU does not own the portal, choose another one that does */
+ if (!cpumask_test_cpu(*cpu, cpus)) {
+ int *pcpu = &get_cpu_var(last_cpu);
+
+ *pcpu = cpumask_next(*pcpu, cpus);
+ if (*pcpu >= nr_cpu_ids)
+ *pcpu = cpumask_first(cpus);
+ *cpu = *pcpu;
+
+ put_cpu_var(last_cpu);
+ }
+ drv_ctx->cpu = *cpu;
+
+ /* Find response FQ hooked with this CPU */
+ drv_ctx->rsp_fq = per_cpu(pcpu_qipriv.rsp_fq, drv_ctx->cpu);
+
+ /* Attach request FQ */
+ drv_ctx->req_fq = create_caam_req_fq(qidev, drv_ctx->rsp_fq, hwdesc,
+ QMAN_INITFQ_FLAG_SCHED);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx->req_fq))) {
+ dev_err(qidev, "create_caam_req_fq failed\n");
+ dma_unmap_single(qidev, hwdesc, size, DMA_BIDIRECTIONAL);
+ kfree(drv_ctx);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ drv_ctx->qidev = qidev;
+ return drv_ctx;
+}
+EXPORT_SYMBOL(caam_drv_ctx_init);
+
+void *qi_cache_alloc(gfp_t flags)
+{
+ return kmem_cache_alloc(qi_cache, flags);
+}
+EXPORT_SYMBOL(qi_cache_alloc);
+
+void qi_cache_free(void *obj)
+{
+ kmem_cache_free(qi_cache, obj);
+}
+EXPORT_SYMBOL(qi_cache_free);
+
+static int caam_qi_poll(struct napi_struct *napi, int budget)
+{
+ struct caam_napi *np = container_of(napi, struct caam_napi, irqtask);
+
+ int cleaned = qman_p_poll_dqrr(np->p, budget);
+
+ if (cleaned < budget) {
+ napi_complete(napi);
+ qman_p_irqsource_add(np->p, QM_PIRQ_DQRI);
+ }
+
+ return cleaned;
+}
+
+void caam_drv_ctx_rel(struct caam_drv_ctx *drv_ctx)
+{
+ if (IS_ERR_OR_NULL(drv_ctx))
+ return;
+
+ /* Remove request FQ */
+ if (kill_fq(drv_ctx->qidev, drv_ctx->req_fq))
+ dev_err(drv_ctx->qidev, "Crypto session req FQ kill failed\n");
+
+ dma_unmap_single(drv_ctx->qidev, drv_ctx->context_a,
+ sizeof(drv_ctx->sh_desc) + sizeof(drv_ctx->prehdr),
+ DMA_BIDIRECTIONAL);
+ kfree(drv_ctx);
+}
+EXPORT_SYMBOL(caam_drv_ctx_rel);
+
+int caam_qi_shutdown(struct device *qidev)
+{
+ int i, ret;
+ struct caam_qi_priv *priv = dev_get_drvdata(qidev);
+ const cpumask_t *cpus = qman_affine_cpus();
+ struct cpumask old_cpumask = *tsk_cpus_allowed(current);
+
+ for_each_cpu(i, cpus) {
+ struct napi_struct *irqtask;
+
+ irqtask = &per_cpu_ptr(&pcpu_qipriv.caam_napi, i)->irqtask;
+ napi_disable(irqtask);
+ netif_napi_del(irqtask);
+
+ if (kill_fq(qidev, per_cpu(pcpu_qipriv.rsp_fq, i)))
+ dev_err(qidev, "Rsp FQ kill failed, cpu: %d\n", i);
+ kfree(per_cpu(pcpu_qipriv.rsp_fq, i));
+ }
+
+ /*
+ * QMan driver requires CGRs to be deleted from same CPU from where they
+ * were instantiated. Hence we get the module removal execute from the
+ * same CPU from where it was originally inserted.
+ */
+ set_cpus_allowed_ptr(current, get_cpu_mask(mod_init_cpu));
+
+ ret = qman_delete_cgr(&priv->cgr);
+ if (ret)
+ dev_err(qidev, "Deletion of CGR failed: %d\n", ret);
+ else
+ qman_release_cgrid(priv->cgr.cgrid);
+
+ kmem_cache_destroy(qi_cache);
+
+ /* Now that we're done with the CGRs, restore the cpus allowed mask */
+ set_cpus_allowed_ptr(current, &old_cpumask);
+
+ platform_device_unregister(priv->qi_pdev);
+ return ret;
+}
+
+static void cgr_cb(struct qman_portal *qm, struct qman_cgr *cgr, int congested)
+{
+ caam_congested = congested;
+
+ if (congested) {
+#ifdef CONFIG_DEBUG_FS
+ times_congested++;
+#endif
+ pr_debug_ratelimited("CAAM entered congestion\n");
+
+ } else {
+ pr_debug_ratelimited("CAAM exited congestion\n");
+ }
+}
+
+static int caam_qi_napi_schedule(struct qman_portal *p, struct caam_napi *np)
+{
+ /*
+ * In case of threaded ISR, for RT kernels in_irq() does not return
+ * appropriate value, so use in_serving_softirq to distinguish between
+ * softirq and irq contexts.
+ */
+ if (unlikely(in_irq() || !in_serving_softirq())) {
+ /* Disable QMan IRQ source and invoke NAPI */
+ qman_p_irqsource_remove(p, QM_PIRQ_DQRI);
+ np->p = p;
+ napi_schedule(&np->irqtask);
+ return 1;
+ }
+ return 0;
+}
+
+static enum qman_cb_dqrr_result caam_rsp_fq_dqrr_cb(struct qman_portal *p,
+ struct qman_fq *rsp_fq,
+ const struct qm_dqrr_entry *dqrr)
+{
+ struct caam_napi *caam_napi = raw_cpu_ptr(&pcpu_qipriv.caam_napi);
+ struct caam_drv_req *drv_req;
+ const struct qm_fd *fd;
+ struct device *qidev = &(raw_cpu_ptr(&pcpu_qipriv)->net_dev.dev);
+ u32 status;
+
+ if (caam_qi_napi_schedule(p, caam_napi))
+ return qman_cb_dqrr_stop;
+
+ fd = &dqrr->fd;
+ status = qm_fd_get_status(fd);
+ if (unlikely(status))
+ dev_err(qidev, "Error: %#x in CAAM response FD\n", status);
+
+ if (unlikely(qm_fd_get_format(fd) != qm_fd_compound)) {
+ dev_err(qidev, "Non-compound FD from CAAM\n");
+ return qman_cb_dqrr_consume;
+ }
+
+ drv_req = (struct caam_drv_req *)phys_to_virt(qm_fd_addr_get64(fd));
+ if (unlikely(!drv_req)) {
+ dev_err(qidev,
+ "Can't find original request for caam response\n");
+ return qman_cb_dqrr_consume;
+ }
+
+ dma_unmap_single(drv_req->drv_ctx->qidev, qm_fd_addr(fd),
+ sizeof(drv_req->fd_sgt), DMA_BIDIRECTIONAL);
+
+ drv_req->cbk(drv_req, status);
+ return qman_cb_dqrr_consume;
+}
+
+static int alloc_rsp_fq_cpu(struct device *qidev, unsigned int cpu)
+{
+ struct qm_mcc_initfq opts;
+ struct qman_fq *fq;
+ int ret;
+
+ fq = kzalloc(sizeof(*fq), GFP_KERNEL | GFP_DMA);
+ if (!fq)
+ return -ENOMEM;
+
+ fq->cb.dqrr = caam_rsp_fq_dqrr_cb;
+
+ ret = qman_create_fq(0, QMAN_FQ_FLAG_NO_ENQUEUE |
+ QMAN_FQ_FLAG_DYNAMIC_FQID, fq);
+ if (ret) {
+ dev_err(qidev, "Rsp FQ create failed\n");
+ kfree(fq);
+ return -ENODEV;
+ }
+
+ memset(&opts, 0, sizeof(opts));
+ qm_initfq_setbits(&opts, QM_INITFQ_WE_FQCTRL | QM_INITFQ_WE_DESTWQ |
+ QM_INITFQ_WE_CONTEXTB | QM_INITFQ_WE_CONTEXTA |
+ QM_INITFQ_WE_CGID);
+ qm_fqd_set_fqctrl(&opts.fqd, QM_FQCTRL_CTXASTASHING |
+ QM_FQCTRL_CPCSTASH | QM_FQCTRL_CGE);
+ qm_fqd_set_destwq(&opts.fqd, qman_affine_channel(cpu), 3);
+ opts.fqd.cgid = qipriv.cgr.cgrid;
+ opts.fqd.context_a.stashing.exclusive = QM_STASHING_EXCL_CTX |
+ QM_STASHING_EXCL_DATA;
+ qm_fqd_set_stashing(&opts.fqd, 0, 1, 1);
+
+ ret = qman_init_fq(fq, QMAN_INITFQ_FLAG_SCHED, &opts);
+ if (ret) {
+ dev_err(qidev, "Rsp FQ init failed\n");
+ kfree(fq);
+ return -ENODEV;
+ }
+
+ per_cpu(pcpu_qipriv.rsp_fq, cpu) = fq;
+
+ dev_info(qidev, "Allocated response FQ %u for CPU %u", fq->fqid, cpu);
+ return 0;
+}
+
+static int init_cgr(struct device *qidev)
+{
+ int ret;
+ struct qm_mcc_initcgr opts;
+ const u64 cpus = *(u64 *)qman_affine_cpus();
+ const int num_cpus = hweight64(cpus);
+ const u64 val = num_cpus * MAX_RSP_FQ_BACKLOG_PER_CPU;
+
+ ret = qman_alloc_cgrid(&qipriv.cgr.cgrid);
+ if (ret) {
+ dev_err(qidev, "CGR alloc failed for rsp FQs: %d\n", ret);
+ return ret;
+ }
+
+ qipriv.cgr.cb = cgr_cb;
+ memset(&opts, 0, sizeof(opts));
+ qm_initcgr_setbits(&opts, QM_CGR_WE_CSCN_EN | QM_CGR_WE_CS_THRES |
+ QM_CGR_WE_MODE);
+ opts.cgr.cscn_en = QM_CGR_EN;
+ opts.cgr.mode = QMAN_CGR_MODE_FRAME;
+ qm_cgr_cs_thres_set64(&opts.cgr.cs_thres, val, 1);
+
+ ret = qman_create_cgr(&qipriv.cgr, QMAN_CGR_FLAG_USE_INIT, &opts);
+ if (ret) {
+ dev_err(qidev, "Error %d creating CAAM CGRID: %u\n", ret,
+ qipriv.cgr.cgrid);
+ return ret;
+ }
+
+ dev_info(qidev, "Congestion threshold set to %llu\n", val);
+ return 0;
+}
+
+static int alloc_rsp_fqs(struct device *qidev)
+{
+ int ret, i;
+ const cpumask_t *cpus = qman_affine_cpus();
+
+ /*Now create response FQs*/
+ for_each_cpu(i, cpus) {
+ ret = alloc_rsp_fq_cpu(qidev, i);
+ if (ret) {
+ dev_err(qidev, "CAAM rsp FQ alloc failed, cpu: %u", i);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static void free_rsp_fqs(void)
+{
+ int i;
+ const cpumask_t *cpus = qman_affine_cpus();
+
+ for_each_cpu(i, cpus)
+ kfree(per_cpu(pcpu_qipriv.rsp_fq, i));
+}
+
+int caam_qi_init(struct platform_device *caam_pdev)
+{
+ int err, i;
+ struct platform_device *qi_pdev;
+ struct device *ctrldev = &caam_pdev->dev, *qidev;
+ struct caam_drv_private *ctrlpriv;
+ const cpumask_t *cpus = qman_affine_cpus();
+ struct cpumask old_cpumask = *tsk_cpus_allowed(current);
+ static struct platform_device_info qi_pdev_info = {
+ .name = "caam_qi",
+ .id = PLATFORM_DEVID_NONE
+ };
+
+ /*
+ * QMAN requires CGRs to be removed from same CPU+portal from where it
+ * was originally allocated. Hence we need to note down the
+ * initialisation CPU and use the same CPU for module exit.
+ * We select the first CPU to from the list of portal owning CPUs.
+ * Then we pin module init to this CPU.
+ */
+ mod_init_cpu = cpumask_first(cpus);
+ set_cpus_allowed_ptr(current, get_cpu_mask(mod_init_cpu));
+
+ qi_pdev_info.parent = ctrldev;
+ qi_pdev_info.dma_mask = dma_get_mask(ctrldev);
+ qi_pdev = platform_device_register_full(&qi_pdev_info);
+ if (IS_ERR(qi_pdev))
+ return PTR_ERR(qi_pdev);
+
+ ctrlpriv = dev_get_drvdata(ctrldev);
+ qidev = &qi_pdev->dev;
+
+ qipriv.qi_pdev = qi_pdev;
+ dev_set_drvdata(qidev, &qipriv);
+
+ /* Initialize the congestion detection */
+ err = init_cgr(qidev);
+ if (err) {
+ dev_err(qidev, "CGR initialization failed: %d\n", err);
+ platform_device_unregister(qi_pdev);
+ return err;
+ }
+
+ /* Initialise response FQs */
+ err = alloc_rsp_fqs(qidev);
+ if (err) {
+ dev_err(qidev, "Can't allocate CAAM response FQs: %d\n", err);
+ free_rsp_fqs();
+ platform_device_unregister(qi_pdev);
+ return err;
+ }
+
+ /*
+ * Enable the NAPI contexts on each of the core which has an affine
+ * portal.
+ */
+ for_each_cpu(i, cpus) {
+ struct caam_qi_pcpu_priv *priv = per_cpu_ptr(&pcpu_qipriv, i);
+ struct caam_napi *caam_napi = &priv->caam_napi;
+ struct napi_struct *irqtask = &caam_napi->irqtask;
+ struct net_device *net_dev = &priv->net_dev;
+
+ net_dev->dev = *qidev;
+ INIT_LIST_HEAD(&net_dev->napi_list);
+
+ netif_napi_add(net_dev, irqtask, caam_qi_poll,
+ CAAM_NAPI_WEIGHT);
+
+ napi_enable(irqtask);
+ }
+
+ /* Hook up QI device to parent controlling caam device */
+ ctrlpriv->qidev = qidev;
+
+ qi_cache = kmem_cache_create("caamqicache", CAAM_QI_MEMCACHE_SIZE, 0,
+ SLAB_CACHE_DMA, NULL);
+ if (!qi_cache) {
+ dev_err(qidev, "Can't allocate CAAM cache\n");
+ free_rsp_fqs();
+ platform_device_unregister(qi_pdev);
+ return err;
+ }
+
+ /* Done with the CGRs; restore the cpus allowed mask */
+ set_cpus_allowed_ptr(current, &old_cpumask);
+#ifdef CONFIG_DEBUG_FS
+ ctrlpriv->qi_congested = debugfs_create_file("qi_congested", 0444,
+ ctrlpriv->ctl,
+ ×_congested,
+ &caam_fops_u64_ro);
+#endif
+ dev_info(qidev, "Linux CAAM Queue I/F driver initialised\n");
+ return 0;
+}
diff --git a/drivers/crypto/caam/qi.h b/drivers/crypto/caam/qi.h
new file mode 100644
index 000000000000..33b0433f5f22
--- /dev/null
+++ b/drivers/crypto/caam/qi.h
@@ -0,0 +1,201 @@
+/*
+ * Public definitions for the CAAM/QI (Queue Interface) backend.
+ *
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ */
+
+#ifndef __QI_H__
+#define __QI_H__
+
+#include <soc/fsl/qman.h>
+#include "compat.h"
+#include "desc.h"
+#include "desc_constr.h"
+
+/*
+ * CAAM hardware constructs a job descriptor which points to a shared descriptor
+ * (as pointed by context_a of to-CAAM FQ).
+ * When the job descriptor is executed by DECO, the whole job descriptor
+ * together with shared descriptor gets loaded in DECO buffer, which is
+ * 64 words (each 32-bit) long.
+ *
+ * The job descriptor constructed by CAAM hardware has the following layout:
+ *
+ * HEADER (1 word)
+ * Shdesc ptr (1 or 2 words)
+ * SEQ_OUT_PTR (1 word)
+ * Out ptr (1 or 2 words)
+ * Out length (1 word)
+ * SEQ_IN_PTR (1 word)
+ * In ptr (1 or 2 words)
+ * In length (1 word)
+ *
+ * The shdesc ptr is used to fetch shared descriptor contents into DECO buffer.
+ *
+ * Apart from shdesc contents, the total number of words that get loaded in DECO
+ * buffer are '8' or '11'. The remaining words in DECO buffer can be used for
+ * storing shared descriptor.
+ */
+#define MAX_SDLEN ((CAAM_DESC_BYTES_MAX - DESC_JOB_IO_LEN) / CAAM_CMD_SZ)
+
+extern bool caam_congested __read_mostly;
+
+/*
+ * This is the request structure the driver application should fill while
+ * submitting a job to driver.
+ */
+struct caam_drv_req;
+
+/*
+ * caam_qi_cbk - application's callback function invoked by the driver when the
+ * request has been successfully processed.
+ * @drv_req: original request that was submitted
+ * @status: completion status of request (0 - success, non-zero - error code)
+ */
+typedef void (*caam_qi_cbk)(struct caam_drv_req *drv_req, u32 status);
+
+enum optype {
+ ENCRYPT,
+ DECRYPT,
+ GIVENCRYPT,
+ NUM_OP
+};
+
+/**
+ * caam_drv_ctx - CAAM/QI backend driver context
+ *
+ * The jobs are processed by the driver against a driver context.
+ * With every cryptographic context, a driver context is attached.
+ * The driver context contains data for private use by driver.
+ * For the applications, this is an opaque structure.
+ *
+ * @prehdr: preheader placed before shrd desc
+ * @sh_desc: shared descriptor
+ * @context_a: shared descriptor dma address
+ * @req_fq: to-CAAM request frame queue
+ * @rsp_fq: from-CAAM response frame queue
+ * @cpu: cpu on which to receive CAAM response
+ * @op_type: operation type
+ * @qidev: device pointer for CAAM/QI backend
+ */
+struct caam_drv_ctx {
+ u32 prehdr[2];
+ u32 sh_desc[MAX_SDLEN];
+ dma_addr_t context_a;
+ struct qman_fq *req_fq;
+ struct qman_fq *rsp_fq;
+ int cpu;
+ enum optype op_type;
+ struct device *qidev;
+} ____cacheline_aligned;
+
+/**
+ * caam_drv_req - The request structure the driver application should fill while
+ * submitting a job to driver.
+ * @fd_sgt: QMan S/G pointing to output (fd_sgt[0]) and input (fd_sgt[1])
+ * buffers.
+ * @cbk: callback function to invoke when job is completed
+ * @app_ctx: arbitrary context attached with request by the application
+ *
+ * The fields mentioned below should not be used by application.
+ * These are for private use by driver.
+ *
+ * @hdr__: linked list header to maintain list of outstanding requests to CAAM
+ * @hwaddr: DMA address for the S/G table.
+ */
+struct caam_drv_req {
+ struct qm_sg_entry fd_sgt[2];
+ struct caam_drv_ctx *drv_ctx;
+ caam_qi_cbk cbk;
+ void *app_ctx;
+} ____cacheline_aligned;
+
+/**
+ * caam_drv_ctx_init - Initialise a CAAM/QI driver context
+ *
+ * A CAAM/QI driver context must be attached with each cryptographic context.
+ * This function allocates memory for CAAM/QI context and returns a handle to
+ * the application. This handle must be submitted along with each enqueue
+ * request to the driver by the application.
+ *
+ * @cpu: CPU where the application prefers to the driver to receive CAAM
+ * responses. The request completion callback would be issued from this
+ * CPU.
+ * @sh_desc: shared descriptor pointer to be attached with CAAM/QI driver
+ * context.
+ *
+ * Returns a driver context on success or negative error code on failure.
+ */
+struct caam_drv_ctx *caam_drv_ctx_init(struct device *qidev, int *cpu,
+ u32 *sh_desc);
+
+/**
+ * caam_qi_enqueue - Submit a request to QI backend driver.
+ *
+ * The request structure must be properly filled as described above.
+ *
+ * @qidev: device pointer for QI backend
+ * @req: CAAM QI request structure
+ *
+ * Returns 0 on success or negative error code on failure.
+ */
+int caam_qi_enqueue(struct device *qidev, struct caam_drv_req *req);
+
+/**
+ * caam_drv_ctx_busy - Check if there are too many jobs pending with CAAM
+ * or too many CAAM responses are pending to be processed.
+ * @drv_ctx: driver context for which job is to be submitted
+ *
+ * Returns caam congestion status 'true/false'
+ */
+bool caam_drv_ctx_busy(struct caam_drv_ctx *drv_ctx);
+
+/**
+ * caam_drv_ctx_update - Update QI driver context
+ *
+ * Invoked when shared descriptor is required to be change in driver context.
+ *
+ * @drv_ctx: driver context to be updated
+ * @sh_desc: new shared descriptor pointer to be updated in QI driver context
+ *
+ * Returns 0 on success or negative error code on failure.
+ */
+int caam_drv_ctx_update(struct caam_drv_ctx *drv_ctx, u32 *sh_desc);
+
+/**
+ * caam_drv_ctx_rel - Release a QI driver context
+ * @drv_ctx: context to be released
+ */
+void caam_drv_ctx_rel(struct caam_drv_ctx *drv_ctx);
+
+int caam_qi_init(struct platform_device *pdev);
+int caam_qi_shutdown(struct device *dev);
+
+/**
+ * qi_cache_alloc - Allocate buffers from CAAM-QI cache
+ *
+ * Invoked when a user of the CAAM-QI (i.e. caamalg-qi) needs data which has
+ * to be allocated on the hotpath. Instead of using malloc, one can use the
+ * services of the CAAM QI memory cache (backed by kmem_cache). The buffers
+ * will have a size of 256B, which is sufficient for hosting 16 SG entries.
+ *
+ * @flags: flags that would be used for the equivalent malloc(..) call
+ *
+ * Returns a pointer to a retrieved buffer on success or NULL on failure.
+ */
+void *qi_cache_alloc(gfp_t flags);
+
+/**
+ * qi_cache_free - Frees buffers allocated from CAAM-QI cache
+ *
+ * Invoked when a user of the CAAM-QI (i.e. caamalg-qi) no longer needs
+ * the buffer previously allocated by a qi_cache_alloc call.
+ * No checking is being done, the call is a passthrough call to
+ * kmem_cache_free(...)
+ *
+ * @obj: object previously allocated using qi_cache_alloc()
+ */
+void qi_cache_free(void *obj);
+
+#endif /* __QI_H__ */
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* [RFC 7/7] crypto: caam/qi - add ablkcipher and authenc algorithms
From: Horia Geantă @ 2017-03-03 14:52 UTC (permalink / raw)
To: Herbert Xu, Scott Wood, Roy Pledge
Cc: linux-arm-kernel, Claudiu Manoil, Cristian Stoica, Dan Douglass,
linux-crypto, Vakul Garg, Vakul Garg, Alexandru Porosanu
In-Reply-To: <1488552733-20806-1-git-send-email-horia.geanta@nxp.com>
Add support to submit ablkcipher and authenc algorithms
via the QI backend:
-ablkcipher:
cbc({aes,des,des3_ede})
ctr(aes), rfc3686(ctr(aes))
xts(aes)
-authenc:
authenc(hmac(md5),cbc({aes,des,des3_ede}))
authenc(hmac(sha*),cbc({aes,des,des3_ede}))
caam/qi being a new driver, let's wait some time to settle down without
interfering with existing caam/jr driver.
Accordingly, for now all caam/qi algorithms (caamalg_qi module) are
marked to be of lower priority than caam/jr ones (caamalg module).
Signed-off-by: Vakul Garg <vakul.garg@nxp.com>
Signed-off-by: Alex Porosanu <alexandru.porosanu@nxp.com>
Signed-off-by: Horia Geantă <horia.geanta@nxp.com>
---
drivers/crypto/caam/Kconfig | 20 +-
drivers/crypto/caam/Makefile | 1 +
drivers/crypto/caam/caamalg.c | 9 +-
drivers/crypto/caam/caamalg_desc.c | 77 +-
drivers/crypto/caam/caamalg_desc.h | 15 +-
drivers/crypto/caam/caamalg_qi.c | 2387 ++++++++++++++++++++++++++++++++++++
drivers/crypto/caam/sg_sw_qm.h | 107 ++
7 files changed, 2600 insertions(+), 16 deletions(-)
create mode 100644 drivers/crypto/caam/caamalg_qi.c
create mode 100644 drivers/crypto/caam/sg_sw_qm.h
diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index bc0d3569f8d9..e36aeacd7635 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -87,6 +87,23 @@ config CRYPTO_DEV_FSL_CAAM_CRYPTO_API
To compile this as a module, choose M here: the module
will be called caamalg.
+config CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI
+ tristate "Queue Interface as Crypto API backend"
+ depends on CRYPTO_DEV_FSL_CAAM_JR && FSL_DPAA && NET
+ default y
+ select CRYPTO_AUTHENC
+ select CRYPTO_BLKCIPHER
+ help
+ Selecting this will use CAAM Queue Interface (QI) for sending
+ & receiving crypto jobs to/from CAAM. This gives better performance
+ than job ring interface when the number of cores are more than the
+ number of job rings assigned to the kernel. The number of portals
+ assigned to the kernel should also be more than the number of
+ job rings.
+
+ To compile this as a module, choose M here: the module
+ will be called caamalg_qi.
+
config CRYPTO_DEV_FSL_CAAM_AHASH_API
tristate "Register hash algorithm implementations with Crypto API"
depends on CRYPTO_DEV_FSL_CAAM_JR
@@ -136,4 +153,5 @@ config CRYPTO_DEV_FSL_CAAM_DEBUG
information in the CAAM driver.
config CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC
- def_tristate CRYPTO_DEV_FSL_CAAM_CRYPTO_API
+ def_tristate (CRYPTO_DEV_FSL_CAAM_CRYPTO_API || \
+ CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI)
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index 2e60e45c2bf1..9e2e98856b9b 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -8,6 +8,7 @@ endif
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_JR) += caam_jr.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o
+obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI) += caamalg_qi.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC) += caamalg_desc.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API) += caamrng.o
diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index 9bc80eb06934..398807d1b77e 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -266,8 +266,9 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
/* aead_encrypt shared descriptor */
desc = ctx->sh_desc_enc;
- cnstr_shdsc_aead_encap(desc, &ctx->cdata, &ctx->adata, ctx->authsize,
- is_rfc3686, nonce, ctx1_iv_off);
+ cnstr_shdsc_aead_encap(desc, &ctx->cdata, &ctx->adata, ivsize,
+ ctx->authsize, is_rfc3686, nonce, ctx1_iv_off,
+ false);
dma_sync_single_for_device(jrdev, ctx->sh_desc_enc_dma,
desc_bytes(desc), DMA_TO_DEVICE);
@@ -299,7 +300,7 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
desc = ctx->sh_desc_dec;
cnstr_shdsc_aead_decap(desc, &ctx->cdata, &ctx->adata, ivsize,
ctx->authsize, alg->caam.geniv, is_rfc3686,
- nonce, ctx1_iv_off);
+ nonce, ctx1_iv_off, false);
dma_sync_single_for_device(jrdev, ctx->sh_desc_dec_dma,
desc_bytes(desc), DMA_TO_DEVICE);
@@ -333,7 +334,7 @@ static int aead_set_sh_desc(struct crypto_aead *aead)
desc = ctx->sh_desc_enc;
cnstr_shdsc_aead_givencap(desc, &ctx->cdata, &ctx->adata, ivsize,
ctx->authsize, is_rfc3686, nonce,
- ctx1_iv_off);
+ ctx1_iv_off, false);
dma_sync_single_for_device(jrdev, ctx->sh_desc_enc_dma,
desc_bytes(desc), DMA_TO_DEVICE);
diff --git a/drivers/crypto/caam/caamalg_desc.c b/drivers/crypto/caam/caamalg_desc.c
index f3f48c10b9d6..6f9c7ec0e339 100644
--- a/drivers/crypto/caam/caamalg_desc.c
+++ b/drivers/crypto/caam/caamalg_desc.c
@@ -265,17 +265,19 @@ static void init_sh_desc_key_aead(u32 * const desc,
* split key is to be used, the size of the split key itself is
* specified. Valid algorithm values - one of OP_ALG_ALGSEL_{MD5, SHA1,
* SHA224, SHA256, SHA384, SHA512} ANDed with OP_ALG_AAI_HMAC_PRECOMP.
+ * @ivsize: initialization vector size
* @icvsize: integrity check value (ICV) size (truncated or full)
* @is_rfc3686: true when ctr(aes) is wrapped by rfc3686 template
* @nonce: pointer to rfc3686 nonce
* @ctx1_iv_off: IV offset in CONTEXT1 register
+ * @is_qi: true when called from caam/qi
*
* Note: Requires an MDHA split key.
*/
void cnstr_shdsc_aead_encap(u32 * const desc, struct alginfo *cdata,
- struct alginfo *adata, unsigned int icvsize,
- const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off)
+ struct alginfo *adata, unsigned int ivsize,
+ unsigned int icvsize, const bool is_rfc3686,
+ u32 *nonce, const u32 ctx1_iv_off, const bool is_qi)
{
/* Note: Context registers are saved. */
init_sh_desc_key_aead(desc, cdata, adata, is_rfc3686, nonce);
@@ -284,6 +286,25 @@ void cnstr_shdsc_aead_encap(u32 * const desc, struct alginfo *cdata,
append_operation(desc, adata->algtype | OP_ALG_AS_INITFINAL |
OP_ALG_ENCRYPT);
+ if (is_qi) {
+ u32 *wait_load_cmd;
+
+ /* REG3 = assoclen */
+ append_seq_load(desc, 4, LDST_CLASS_DECO |
+ LDST_SRCDST_WORD_DECO_MATH3 |
+ (4 << LDST_OFFSET_SHIFT));
+
+ wait_load_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_CALM | JUMP_COND_NCP |
+ JUMP_COND_NOP | JUMP_COND_NIP |
+ JUMP_COND_NIFP);
+ set_jump_tgt_here(desc, wait_load_cmd);
+
+ append_seq_load(desc, ivsize, LDST_CLASS_1_CCB |
+ LDST_SRCDST_BYTE_CONTEXT |
+ (ctx1_iv_off << LDST_OFFSET_SHIFT));
+ }
+
/* Read and write assoclen bytes */
append_math_add(desc, VARSEQINLEN, ZERO, REG3, CAAM_CMD_SZ);
append_math_add(desc, VARSEQOUTLEN, ZERO, REG3, CAAM_CMD_SZ);
@@ -338,6 +359,7 @@ EXPORT_SYMBOL(cnstr_shdsc_aead_encap);
* @is_rfc3686: true when ctr(aes) is wrapped by rfc3686 template
* @nonce: pointer to rfc3686 nonce
* @ctx1_iv_off: IV offset in CONTEXT1 register
+ * @is_qi: true when called from caam/qi
*
* Note: Requires an MDHA split key.
*/
@@ -345,7 +367,7 @@ void cnstr_shdsc_aead_decap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool geniv,
const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off)
+ const u32 ctx1_iv_off, const bool is_qi)
{
/* Note: Context registers are saved. */
init_sh_desc_key_aead(desc, cdata, adata, is_rfc3686, nonce);
@@ -354,6 +376,26 @@ void cnstr_shdsc_aead_decap(u32 * const desc, struct alginfo *cdata,
append_operation(desc, adata->algtype | OP_ALG_AS_INITFINAL |
OP_ALG_DECRYPT | OP_ALG_ICV_ON);
+ if (is_qi) {
+ u32 *wait_load_cmd;
+
+ /* REG3 = assoclen */
+ append_seq_load(desc, 4, LDST_CLASS_DECO |
+ LDST_SRCDST_WORD_DECO_MATH3 |
+ (4 << LDST_OFFSET_SHIFT));
+
+ wait_load_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_CALM | JUMP_COND_NCP |
+ JUMP_COND_NOP | JUMP_COND_NIP |
+ JUMP_COND_NIFP);
+ set_jump_tgt_here(desc, wait_load_cmd);
+
+ if (!geniv)
+ append_seq_load(desc, ivsize, LDST_CLASS_1_CCB |
+ LDST_SRCDST_BYTE_CONTEXT |
+ (ctx1_iv_off << LDST_OFFSET_SHIFT));
+ }
+
/* Read and write assoclen bytes */
append_math_add(desc, VARSEQINLEN, ZERO, REG3, CAAM_CMD_SZ);
if (geniv)
@@ -423,21 +465,44 @@ EXPORT_SYMBOL(cnstr_shdsc_aead_decap);
* @is_rfc3686: true when ctr(aes) is wrapped by rfc3686 template
* @nonce: pointer to rfc3686 nonce
* @ctx1_iv_off: IV offset in CONTEXT1 register
+ * @is_qi: true when called from caam/qi
*
* Note: Requires an MDHA split key.
*/
void cnstr_shdsc_aead_givencap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool is_rfc3686,
- u32 *nonce, const u32 ctx1_iv_off)
+ u32 *nonce, const u32 ctx1_iv_off,
+ const bool is_qi)
{
u32 geniv, moveiv;
/* Note: Context registers are saved. */
init_sh_desc_key_aead(desc, cdata, adata, is_rfc3686, nonce);
- if (is_rfc3686)
+ if (is_qi) {
+ u32 *wait_load_cmd;
+
+ /* REG3 = assoclen */
+ append_seq_load(desc, 4, LDST_CLASS_DECO |
+ LDST_SRCDST_WORD_DECO_MATH3 |
+ (4 << LDST_OFFSET_SHIFT));
+
+ wait_load_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_CALM | JUMP_COND_NCP |
+ JUMP_COND_NOP | JUMP_COND_NIP |
+ JUMP_COND_NIFP);
+ set_jump_tgt_here(desc, wait_load_cmd);
+ }
+
+ if (is_rfc3686) {
+ if (is_qi)
+ append_seq_load(desc, ivsize, LDST_CLASS_1_CCB |
+ LDST_SRCDST_BYTE_CONTEXT |
+ (ctx1_iv_off << LDST_OFFSET_SHIFT));
+
goto copy_iv;
+ }
/* Generate IV */
geniv = NFIFOENTRY_STYPE_PAD | NFIFOENTRY_DEST_DECO |
diff --git a/drivers/crypto/caam/caamalg_desc.h b/drivers/crypto/caam/caamalg_desc.h
index 95551737333a..8731e4a7ff05 100644
--- a/drivers/crypto/caam/caamalg_desc.h
+++ b/drivers/crypto/caam/caamalg_desc.h
@@ -12,6 +12,9 @@
#define DESC_AEAD_ENC_LEN (DESC_AEAD_BASE + 11 * CAAM_CMD_SZ)
#define DESC_AEAD_DEC_LEN (DESC_AEAD_BASE + 15 * CAAM_CMD_SZ)
#define DESC_AEAD_GIVENC_LEN (DESC_AEAD_ENC_LEN + 7 * CAAM_CMD_SZ)
+#define DESC_QI_AEAD_ENC_LEN (DESC_AEAD_ENC_LEN + 3 * CAAM_CMD_SZ)
+#define DESC_QI_AEAD_DEC_LEN (DESC_AEAD_DEC_LEN + 3 * CAAM_CMD_SZ)
+#define DESC_QI_AEAD_GIVENC_LEN (DESC_AEAD_GIVENC_LEN + 3 * CAAM_CMD_SZ)
/* Note: Nonce is counted in cdata.keylen */
#define DESC_AEAD_CTR_RFC3686_LEN (4 * CAAM_CMD_SZ)
@@ -45,20 +48,22 @@ void cnstr_shdsc_aead_null_decap(u32 * const desc, struct alginfo *adata,
unsigned int icvsize);
void cnstr_shdsc_aead_encap(u32 * const desc, struct alginfo *cdata,
- struct alginfo *adata, unsigned int icvsize,
- const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off);
+ struct alginfo *adata, unsigned int ivsize,
+ unsigned int icvsize, const bool is_rfc3686,
+ u32 *nonce, const u32 ctx1_iv_off,
+ const bool is_qi);
void cnstr_shdsc_aead_decap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool geniv,
const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off);
+ const u32 ctx1_iv_off, const bool is_qi);
void cnstr_shdsc_aead_givencap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool is_rfc3686,
- u32 *nonce, const u32 ctx1_iv_off);
+ u32 *nonce, const u32 ctx1_iv_off,
+ const bool is_qi);
void cnstr_shdsc_gcm_encap(u32 * const desc, struct alginfo *cdata,
unsigned int icvsize);
diff --git a/drivers/crypto/caam/caamalg_qi.c b/drivers/crypto/caam/caamalg_qi.c
new file mode 100644
index 000000000000..ea0e5b8b9171
--- /dev/null
+++ b/drivers/crypto/caam/caamalg_qi.c
@@ -0,0 +1,2387 @@
+/*
+ * Freescale FSL CAAM support for crypto API over QI backend.
+ * Based on caamalg.c
+ *
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ */
+
+#include "compat.h"
+
+#include "regs.h"
+#include "intern.h"
+#include "desc_constr.h"
+#include "error.h"
+#include "sg_sw_sec4.h"
+#include "sg_sw_qm.h"
+#include "key_gen.h"
+#include "qi.h"
+#include "jr.h"
+#include "caamalg_desc.h"
+
+/*
+ * crypto alg
+ */
+#define CAAM_CRA_PRIORITY 2000
+/* max key is sum of AES_MAX_KEY_SIZE, max split key size */
+#define CAAM_MAX_KEY_SIZE (AES_MAX_KEY_SIZE + \
+ SHA512_DIGEST_SIZE * 2)
+
+#define DESC_MAX_USED_BYTES (DESC_QI_AEAD_GIVENC_LEN + \
+ CAAM_MAX_KEY_SIZE)
+#define DESC_MAX_USED_LEN (DESC_MAX_USED_BYTES / CAAM_CMD_SZ)
+
+struct caam_alg_entry {
+ int class1_alg_type;
+ int class2_alg_type;
+ bool rfc3686;
+ bool geniv;
+};
+
+struct caam_aead_alg {
+ struct aead_alg aead;
+ struct caam_alg_entry caam;
+ bool registered;
+};
+
+/*
+ * per-session context
+ */
+struct caam_ctx {
+ struct device *jrdev;
+ u32 sh_desc_enc[DESC_MAX_USED_LEN];
+ u32 sh_desc_dec[DESC_MAX_USED_LEN];
+ u32 sh_desc_givenc[DESC_MAX_USED_LEN];
+ u8 key[CAAM_MAX_KEY_SIZE];
+ dma_addr_t key_dma;
+ struct alginfo adata;
+ struct alginfo cdata;
+ unsigned int authsize;
+ struct device *qidev;
+ spinlock_t lock; /* Protects multiple init of driver context */
+ struct caam_drv_ctx *drv_ctx[NUM_OP];
+};
+
+static int aead_set_sh_desc(struct crypto_aead *aead)
+{
+ struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
+ typeof(*alg), aead);
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ unsigned int ivsize = crypto_aead_ivsize(aead);
+ u32 ctx1_iv_off = 0;
+ u32 *nonce = NULL;
+ unsigned int data_len[2];
+ u32 inl_mask;
+ const bool ctr_mode = ((ctx->cdata.algtype & OP_ALG_AAI_MASK) ==
+ OP_ALG_AAI_CTR_MOD128);
+ const bool is_rfc3686 = alg->caam.rfc3686;
+
+ if (!ctx->cdata.keylen || !ctx->authsize)
+ return 0;
+
+ /*
+ * AES-CTR needs to load IV in CONTEXT1 reg
+ * at an offset of 128bits (16bytes)
+ * CONTEXT1[255:128] = IV
+ */
+ if (ctr_mode)
+ ctx1_iv_off = 16;
+
+ /*
+ * RFC3686 specific:
+ * CONTEXT1[255:128] = {NONCE, IV, COUNTER}
+ */
+ if (is_rfc3686) {
+ ctx1_iv_off = 16 + CTR_RFC3686_NONCE_SIZE;
+ nonce = (u32 *)((void *)ctx->key + ctx->adata.keylen_pad +
+ ctx->cdata.keylen - CTR_RFC3686_NONCE_SIZE);
+ }
+
+ data_len[0] = ctx->adata.keylen_pad;
+ data_len[1] = ctx->cdata.keylen;
+
+ if (alg->caam.geniv)
+ goto skip_enc;
+
+ /* aead_encrypt shared descriptor */
+ if (desc_inline_query(DESC_QI_AEAD_ENC_LEN +
+ (is_rfc3686 ? DESC_AEAD_CTR_RFC3686_LEN : 0),
+ DESC_JOB_IO_LEN, data_len, &inl_mask,
+ ARRAY_SIZE(data_len)) < 0)
+ return -EINVAL;
+
+ if (inl_mask & 1)
+ ctx->adata.key_virt = ctx->key;
+ else
+ ctx->adata.key_dma = ctx->key_dma;
+
+ if (inl_mask & 2)
+ ctx->cdata.key_virt = ctx->key + ctx->adata.keylen_pad;
+ else
+ ctx->cdata.key_dma = ctx->key_dma + ctx->adata.keylen_pad;
+
+ ctx->adata.key_inline = !!(inl_mask & 1);
+ ctx->cdata.key_inline = !!(inl_mask & 2);
+
+ cnstr_shdsc_aead_encap(ctx->sh_desc_enc, &ctx->cdata, &ctx->adata,
+ ivsize, ctx->authsize, is_rfc3686, nonce,
+ ctx1_iv_off, true);
+
+skip_enc:
+ /* aead_decrypt shared descriptor */
+ if (desc_inline_query(DESC_QI_AEAD_DEC_LEN +
+ (is_rfc3686 ? DESC_AEAD_CTR_RFC3686_LEN : 0),
+ DESC_JOB_IO_LEN, data_len, &inl_mask,
+ ARRAY_SIZE(data_len)) < 0)
+ return -EINVAL;
+
+ if (inl_mask & 1)
+ ctx->adata.key_virt = ctx->key;
+ else
+ ctx->adata.key_dma = ctx->key_dma;
+
+ if (inl_mask & 2)
+ ctx->cdata.key_virt = ctx->key + ctx->adata.keylen_pad;
+ else
+ ctx->cdata.key_dma = ctx->key_dma + ctx->adata.keylen_pad;
+
+ ctx->adata.key_inline = !!(inl_mask & 1);
+ ctx->cdata.key_inline = !!(inl_mask & 2);
+
+ cnstr_shdsc_aead_decap(ctx->sh_desc_dec, &ctx->cdata, &ctx->adata,
+ ivsize, ctx->authsize, alg->caam.geniv,
+ is_rfc3686, nonce, ctx1_iv_off, true);
+
+ if (!alg->caam.geniv)
+ goto skip_givenc;
+
+ /* aead_givencrypt shared descriptor */
+ if (desc_inline_query(DESC_QI_AEAD_GIVENC_LEN +
+ (is_rfc3686 ? DESC_AEAD_CTR_RFC3686_LEN : 0),
+ DESC_JOB_IO_LEN, data_len, &inl_mask,
+ ARRAY_SIZE(data_len)) < 0)
+ return -EINVAL;
+
+ if (inl_mask & 1)
+ ctx->adata.key_virt = ctx->key;
+ else
+ ctx->adata.key_dma = ctx->key_dma;
+
+ if (inl_mask & 2)
+ ctx->cdata.key_virt = ctx->key + ctx->adata.keylen_pad;
+ else
+ ctx->cdata.key_dma = ctx->key_dma + ctx->adata.keylen_pad;
+
+ ctx->adata.key_inline = !!(inl_mask & 1);
+ ctx->cdata.key_inline = !!(inl_mask & 2);
+
+ cnstr_shdsc_aead_givencap(ctx->sh_desc_enc, &ctx->cdata, &ctx->adata,
+ ivsize, ctx->authsize, is_rfc3686, nonce,
+ ctx1_iv_off, true);
+
+skip_givenc:
+ return 0;
+}
+
+static int aead_setauthsize(struct crypto_aead *authenc, unsigned int authsize)
+{
+ struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+
+ ctx->authsize = authsize;
+ aead_set_sh_desc(authenc);
+
+ return 0;
+}
+
+static int aead_setkey(struct crypto_aead *aead, const u8 *key,
+ unsigned int keylen)
+{
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ struct device *jrdev = ctx->jrdev;
+ struct crypto_authenc_keys keys;
+ int ret = 0;
+
+ if (crypto_authenc_extractkeys(&keys, key, keylen) != 0)
+ goto badkey;
+
+#ifdef DEBUG
+ dev_err(jrdev, "keylen %d enckeylen %d authkeylen %d\n",
+ keys.authkeylen + keys.enckeylen, keys.enckeylen,
+ keys.authkeylen);
+ print_hex_dump(KERN_ERR, "key in @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
+#endif
+
+ ret = gen_split_key(jrdev, ctx->key, &ctx->adata, keys.authkey,
+ keys.authkeylen, CAAM_MAX_KEY_SIZE -
+ keys.enckeylen);
+ if (ret)
+ goto badkey;
+
+ /* postpend encryption key to auth split key */
+ memcpy(ctx->key + ctx->adata.keylen_pad, keys.enckey, keys.enckeylen);
+ dma_sync_single_for_device(jrdev, ctx->key_dma, ctx->adata.keylen_pad +
+ keys.enckeylen, DMA_TO_DEVICE);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx.key@" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, ctx->key,
+ ctx->adata.keylen_pad + keys.enckeylen, 1);
+#endif
+
+ ctx->cdata.keylen = keys.enckeylen;
+
+ ret = aead_set_sh_desc(aead);
+ if (ret)
+ goto badkey;
+
+ /* Now update the driver contexts with the new shared descriptor */
+ if (ctx->drv_ctx[ENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[ENCRYPT],
+ ctx->sh_desc_enc);
+ if (ret) {
+ dev_err(jrdev, "driver enc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[DECRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[DECRYPT],
+ ctx->sh_desc_dec);
+ if (ret) {
+ dev_err(jrdev, "driver dec context update failed\n");
+ goto badkey;
+ }
+ }
+
+ return ret;
+badkey:
+ crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+}
+
+static int ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
+ const u8 *key, unsigned int keylen)
+{
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct crypto_tfm *tfm = crypto_ablkcipher_tfm(ablkcipher);
+ const char *alg_name = crypto_tfm_alg_name(tfm);
+ struct device *jrdev = ctx->jrdev;
+ unsigned int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ u32 ctx1_iv_off = 0;
+ const bool ctr_mode = ((ctx->cdata.algtype & OP_ALG_AAI_MASK) ==
+ OP_ALG_AAI_CTR_MOD128);
+ const bool is_rfc3686 = (ctr_mode && strstr(alg_name, "rfc3686"));
+ int ret = 0;
+
+ memcpy(ctx->key, key, keylen);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "key in @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
+#endif
+ /*
+ * AES-CTR needs to load IV in CONTEXT1 reg
+ * at an offset of 128bits (16bytes)
+ * CONTEXT1[255:128] = IV
+ */
+ if (ctr_mode)
+ ctx1_iv_off = 16;
+
+ /*
+ * RFC3686 specific:
+ * | CONTEXT1[255:128] = {NONCE, IV, COUNTER}
+ * | *key = {KEY, NONCE}
+ */
+ if (is_rfc3686) {
+ ctx1_iv_off = 16 + CTR_RFC3686_NONCE_SIZE;
+ keylen -= CTR_RFC3686_NONCE_SIZE;
+ }
+
+ dma_sync_single_for_device(jrdev, ctx->key_dma, keylen, DMA_TO_DEVICE);
+ ctx->cdata.keylen = keylen;
+ ctx->cdata.key_virt = ctx->key;
+ ctx->cdata.key_inline = true;
+
+ /* ablkcipher encrypt, decrypt, givencrypt shared descriptors */
+ cnstr_shdsc_ablkcipher_encap(ctx->sh_desc_enc, &ctx->cdata, ivsize,
+ is_rfc3686, ctx1_iv_off);
+ cnstr_shdsc_ablkcipher_decap(ctx->sh_desc_dec, &ctx->cdata, ivsize,
+ is_rfc3686, ctx1_iv_off);
+ cnstr_shdsc_ablkcipher_givencap(ctx->sh_desc_givenc, &ctx->cdata,
+ ivsize, is_rfc3686, ctx1_iv_off);
+
+ /* Now update the driver contexts with the new shared descriptor */
+ if (ctx->drv_ctx[ENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[ENCRYPT],
+ ctx->sh_desc_enc);
+ if (ret) {
+ dev_err(jrdev, "driver enc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[DECRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[DECRYPT],
+ ctx->sh_desc_dec);
+ if (ret) {
+ dev_err(jrdev, "driver dec context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[GIVENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[GIVENCRYPT],
+ ctx->sh_desc_givenc);
+ if (ret) {
+ dev_err(jrdev, "driver givenc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ return ret;
+badkey:
+ crypto_ablkcipher_set_flags(ablkcipher, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+}
+
+static int xts_ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
+ const u8 *key, unsigned int keylen)
+{
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *jrdev = ctx->jrdev;
+ int ret = 0;
+
+ if (keylen != 2 * AES_MIN_KEY_SIZE && keylen != 2 * AES_MAX_KEY_SIZE) {
+ crypto_ablkcipher_set_flags(ablkcipher,
+ CRYPTO_TFM_RES_BAD_KEY_LEN);
+ dev_err(jrdev, "key size mismatch\n");
+ return -EINVAL;
+ }
+
+ memcpy(ctx->key, key, keylen);
+ dma_sync_single_for_device(jrdev, ctx->key_dma, keylen, DMA_TO_DEVICE);
+ ctx->cdata.keylen = keylen;
+ ctx->cdata.key_virt = ctx->key;
+ ctx->cdata.key_inline = true;
+
+ /* xts ablkcipher encrypt, decrypt shared descriptors */
+ cnstr_shdsc_xts_ablkcipher_encap(ctx->sh_desc_enc, &ctx->cdata);
+ cnstr_shdsc_xts_ablkcipher_decap(ctx->sh_desc_dec, &ctx->cdata);
+
+ /* Now update the driver contexts with the new shared descriptor */
+ if (ctx->drv_ctx[ENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[ENCRYPT],
+ ctx->sh_desc_enc);
+ if (ret) {
+ dev_err(jrdev, "driver enc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[DECRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[DECRYPT],
+ ctx->sh_desc_dec);
+ if (ret) {
+ dev_err(jrdev, "driver dec context update failed\n");
+ goto badkey;
+ }
+ }
+
+ return ret;
+badkey:
+ crypto_ablkcipher_set_flags(ablkcipher, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return 0;
+}
+
+/*
+ * aead_edesc - s/w-extended aead descriptor
+ * @src_nents: number of segments in input scatterlist
+ * @dst_nents: number of segments in output scatterlist
+ * @iv_dma: dma address of iv for checking continuity and link table
+ * @qm_sg_bytes: length of dma mapped h/w link table
+ * @qm_sg_dma: bus physical mapped address of h/w link table
+ * @assoclen_dma: bus physical mapped address of req->assoclen
+ * @drv_req: driver-specific request structure
+ * @sgt: the h/w link table
+ */
+struct aead_edesc {
+ int src_nents;
+ int dst_nents;
+ dma_addr_t iv_dma;
+ int qm_sg_bytes;
+ dma_addr_t qm_sg_dma;
+ dma_addr_t assoclen_dma;
+ struct caam_drv_req drv_req;
+ struct qm_sg_entry sgt[0];
+};
+
+/*
+ * ablkcipher_edesc - s/w-extended ablkcipher descriptor
+ * @src_nents: number of segments in input scatterlist
+ * @dst_nents: number of segments in output scatterlist
+ * @iv_dma: dma address of iv for checking continuity and link table
+ * @qm_sg_bytes: length of dma mapped h/w link table
+ * @qm_sg_dma: bus physical mapped address of h/w link table
+ * @drv_req: driver-specific request structure
+ * @sgt: the h/w link table
+ */
+struct ablkcipher_edesc {
+ int src_nents;
+ int dst_nents;
+ dma_addr_t iv_dma;
+ int qm_sg_bytes;
+ dma_addr_t qm_sg_dma;
+ struct caam_drv_req drv_req;
+ struct qm_sg_entry sgt[0];
+};
+
+static struct caam_drv_ctx *get_drv_ctx(struct caam_ctx *ctx,
+ enum optype type)
+{
+ /*
+ * This function is called on the fast path with values of 'type'
+ * known at compile time. Invalid arguments are not expected and
+ * thus no checks are made.
+ */
+ struct caam_drv_ctx *drv_ctx = ctx->drv_ctx[type];
+ u32 *desc;
+
+ if (unlikely(!drv_ctx)) {
+ spin_lock(&ctx->lock);
+
+ /* Read again to check if some other core init drv_ctx */
+ drv_ctx = ctx->drv_ctx[type];
+ if (!drv_ctx) {
+ int cpu;
+
+ if (type == ENCRYPT)
+ desc = ctx->sh_desc_enc;
+ else if (type == DECRYPT)
+ desc = ctx->sh_desc_dec;
+ else /* (type == GIVENCRYPT) */
+ desc = ctx->sh_desc_givenc;
+
+ cpu = smp_processor_id();
+ drv_ctx = caam_drv_ctx_init(ctx->qidev, &cpu, desc);
+ if (likely(!IS_ERR_OR_NULL(drv_ctx)))
+ drv_ctx->op_type = type;
+
+ ctx->drv_ctx[type] = drv_ctx;
+ }
+
+ spin_unlock(&ctx->lock);
+ }
+
+ return drv_ctx;
+}
+
+static void caam_unmap(struct device *dev, struct scatterlist *src,
+ struct scatterlist *dst, int src_nents,
+ int dst_nents, dma_addr_t iv_dma, int ivsize,
+ enum optype op_type, dma_addr_t qm_sg_dma,
+ int qm_sg_bytes)
+{
+ if (dst != src) {
+ if (src_nents)
+ dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
+ dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
+ } else {
+ dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
+ }
+
+ if (iv_dma)
+ dma_unmap_single(dev, iv_dma, ivsize,
+ op_type == GIVENCRYPT ? DMA_FROM_DEVICE :
+ DMA_TO_DEVICE);
+ if (qm_sg_bytes)
+ dma_unmap_single(dev, qm_sg_dma, qm_sg_bytes, DMA_TO_DEVICE);
+}
+
+static void aead_unmap(struct device *dev,
+ struct aead_edesc *edesc,
+ struct aead_request *req)
+{
+ struct crypto_aead *aead = crypto_aead_reqtfm(req);
+ int ivsize = crypto_aead_ivsize(aead);
+
+ caam_unmap(dev, req->src, req->dst, edesc->src_nents, edesc->dst_nents,
+ edesc->iv_dma, ivsize, edesc->drv_req.drv_ctx->op_type,
+ edesc->qm_sg_dma, edesc->qm_sg_bytes);
+ dma_unmap_single(dev, edesc->assoclen_dma, 4, DMA_TO_DEVICE);
+}
+
+static void ablkcipher_unmap(struct device *dev,
+ struct ablkcipher_edesc *edesc,
+ struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+
+ caam_unmap(dev, req->src, req->dst, edesc->src_nents, edesc->dst_nents,
+ edesc->iv_dma, ivsize, edesc->drv_req.drv_ctx->op_type,
+ edesc->qm_sg_dma, edesc->qm_sg_bytes);
+}
+
+static void aead_done(struct caam_drv_req *drv_req, u32 status)
+{
+ struct device *qidev;
+ struct aead_edesc *edesc;
+ struct aead_request *aead_req = drv_req->app_ctx;
+ struct crypto_aead *aead = crypto_aead_reqtfm(aead_req);
+ struct caam_ctx *caam_ctx = crypto_aead_ctx(aead);
+ int ecode = 0;
+
+ qidev = caam_ctx->qidev;
+
+ if (unlikely(status)) {
+ caam_jr_strstatus(qidev, status);
+ ecode = -EIO;
+ }
+
+ edesc = container_of(drv_req, typeof(*edesc), drv_req);
+ aead_unmap(qidev, edesc, aead_req);
+
+ aead_request_complete(aead_req, ecode);
+ qi_cache_free(edesc);
+}
+
+/*
+ * allocate and map the aead extended descriptor
+ */
+static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
+ bool encrypt)
+{
+ struct crypto_aead *aead = crypto_aead_reqtfm(req);
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
+ typeof(*alg), aead);
+ struct device *qidev = ctx->qidev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
+ struct aead_edesc *edesc;
+ dma_addr_t qm_sg_dma, iv_dma = 0;
+ int ivsize = 0;
+ unsigned int authsize = ctx->authsize;
+ int qm_sg_index = 0, qm_sg_ents = 0, qm_sg_bytes;
+ int in_len, out_len;
+ struct qm_sg_entry *sg_table, *fd_sgt;
+ struct caam_drv_ctx *drv_ctx;
+ enum optype op_type = encrypt ? ENCRYPT : DECRYPT;
+
+ drv_ctx = get_drv_ctx(ctx, op_type);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
+ return (struct aead_edesc *)drv_ctx;
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = qi_cache_alloc(GFP_DMA | flags);
+ if (unlikely(!edesc)) {
+ dev_err(qidev, "could not allocate extended descriptor\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ if (likely(req->src == req->dst)) {
+ src_nents = sg_nents_for_len(req->src, req->assoclen +
+ req->cryptlen +
+ (encrypt ? authsize : 0));
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->assoclen + req->cryptlen +
+ (encrypt ? authsize : 0));
+ qi_cache_free(edesc);
+ return ERR_PTR(src_nents);
+ }
+
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ src_nents = sg_nents_for_len(req->src, req->assoclen +
+ req->cryptlen);
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->assoclen + req->cryptlen);
+ qi_cache_free(edesc);
+ return ERR_PTR(src_nents);
+ }
+
+ dst_nents = sg_nents_for_len(req->dst, req->assoclen +
+ req->cryptlen +
+ (encrypt ? authsize :
+ (-authsize)));
+ if (unlikely(dst_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in dst S/G\n",
+ req->assoclen + req->cryptlen +
+ (encrypt ? authsize : (-authsize)));
+ qi_cache_free(edesc);
+ return ERR_PTR(dst_nents);
+ }
+
+ if (src_nents) {
+ mapped_src_nents = dma_map_sg(qidev, req->src,
+ src_nents, DMA_TO_DEVICE);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ mapped_src_nents = 0;
+ }
+
+ mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
+ DMA_FROM_DEVICE);
+ if (unlikely(!mapped_dst_nents)) {
+ dev_err(qidev, "unable to map destination\n");
+ dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ }
+
+ if ((alg->caam.rfc3686 && encrypt) || !alg->caam.geniv) {
+ ivsize = crypto_aead_ivsize(aead);
+ iv_dma = dma_map_single(qidev, req->iv, ivsize, DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, iv_dma)) {
+ dev_err(qidev, "unable to map IV\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents,
+ dst_nents, 0, 0, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ }
+
+ /*
+ * Create S/G table: req->assoclen, [IV,] req->src [, req->dst].
+ * Input is not contiguous.
+ */
+ qm_sg_ents = 1 + !!ivsize + mapped_src_nents +
+ (mapped_dst_nents > 1 ? mapped_dst_nents : 0);
+ sg_table = &edesc->sgt[0];
+ qm_sg_bytes = qm_sg_ents * sizeof(*sg_table);
+
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+ edesc->iv_dma = iv_dma;
+ edesc->drv_req.app_ctx = req;
+ edesc->drv_req.cbk = aead_done;
+ edesc->drv_req.drv_ctx = drv_ctx;
+
+ edesc->assoclen_dma = dma_map_single(qidev, &req->assoclen, 4,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, edesc->assoclen_dma)) {
+ dev_err(qidev, "unable to map assoclen\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ dma_to_qm_sg_one(sg_table, edesc->assoclen_dma, 4, 0);
+ qm_sg_index++;
+ if (ivsize) {
+ dma_to_qm_sg_one(sg_table + qm_sg_index, iv_dma, ivsize, 0);
+ qm_sg_index++;
+ }
+ sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table + qm_sg_index, 0);
+ qm_sg_index += mapped_src_nents;
+
+ if (mapped_dst_nents > 1)
+ sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table +
+ qm_sg_index, 0);
+
+ qm_sg_dma = dma_map_single(qidev, sg_table, qm_sg_bytes, DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, qm_sg_dma)) {
+ dev_err(qidev, "unable to map S/G table\n");
+ dma_unmap_single(qidev, edesc->assoclen_dma, 4, DMA_TO_DEVICE);
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ edesc->qm_sg_dma = qm_sg_dma;
+ edesc->qm_sg_bytes = qm_sg_bytes;
+
+ out_len = req->assoclen + req->cryptlen +
+ (encrypt ? ctx->authsize : (-ctx->authsize));
+ in_len = 4 + ivsize + req->assoclen + req->cryptlen;
+
+ fd_sgt = &edesc->drv_req.fd_sgt[0];
+ dma_to_qm_sg_one_last_ext(&fd_sgt[1], qm_sg_dma, in_len, 0);
+
+ if (req->dst == req->src) {
+ if (mapped_src_nents == 1)
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->src),
+ out_len, 0);
+ else
+ dma_to_qm_sg_one_ext(&fd_sgt[0], qm_sg_dma +
+ (1 + !!ivsize) * sizeof(*sg_table),
+ out_len, 0);
+ } else if (mapped_dst_nents == 1) {
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst), out_len,
+ 0);
+ } else {
+ dma_to_qm_sg_one_ext(&fd_sgt[0], qm_sg_dma + sizeof(*sg_table) *
+ qm_sg_index, out_len, 0);
+ }
+
+ return edesc;
+}
+
+static inline int aead_crypt(struct aead_request *req, bool encrypt)
+{
+ struct aead_edesc *edesc;
+ struct crypto_aead *aead = crypto_aead_reqtfm(req);
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ int ret;
+
+ if (unlikely(caam_congested))
+ return -EAGAIN;
+
+ /* allocate extended descriptor */
+ edesc = aead_edesc_alloc(req, encrypt);
+ if (IS_ERR_OR_NULL(edesc))
+ return PTR_ERR(edesc);
+
+ /* Create and submit job descriptor */
+ ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ aead_unmap(ctx->qidev, edesc, req);
+ qi_cache_free(edesc);
+ }
+
+ return ret;
+}
+
+static int aead_encrypt(struct aead_request *req)
+{
+ return aead_crypt(req, true);
+}
+
+static int aead_decrypt(struct aead_request *req)
+{
+ return aead_crypt(req, false);
+}
+
+static void ablkcipher_done(struct caam_drv_req *drv_req, u32 status)
+{
+ struct ablkcipher_edesc *edesc;
+ struct ablkcipher_request *req = drv_req->app_ctx;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *caam_ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *qidev = caam_ctx->qidev;
+#ifdef DEBUG
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+
+ dev_err(qidev, "%s %d: status 0x%x\n", __func__, __LINE__, status);
+#endif
+
+ edesc = container_of(drv_req, typeof(*edesc), drv_req);
+
+ if (status)
+ caam_jr_strstatus(qidev, status);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "dstiv @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->info,
+ edesc->src_nents > 1 ? 100 : ivsize, 1);
+ dbg_dump_sg(KERN_ERR, "dst @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->dst,
+ edesc->dst_nents > 1 ? 100 : req->nbytes, 1);
+#endif
+
+ ablkcipher_unmap(qidev, edesc, req);
+ qi_cache_free(edesc);
+
+ ablkcipher_request_complete(req, status);
+}
+
+static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
+ *req, bool encrypt)
+{
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *qidev = ctx->qidev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ?
+ GFP_KERNEL : GFP_ATOMIC;
+ int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
+ struct ablkcipher_edesc *edesc;
+ dma_addr_t iv_dma;
+ bool in_contig;
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ int dst_sg_idx, qm_sg_ents;
+ struct qm_sg_entry *sg_table, *fd_sgt;
+ struct caam_drv_ctx *drv_ctx;
+ enum optype op_type = encrypt ? ENCRYPT : DECRYPT;
+
+ drv_ctx = get_drv_ctx(ctx, op_type);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
+ return (struct ablkcipher_edesc *)drv_ctx;
+
+ src_nents = sg_nents_for_len(req->src, req->nbytes);
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->nbytes);
+ return ERR_PTR(src_nents);
+ }
+
+ if (unlikely(req->src != req->dst)) {
+ dst_nents = sg_nents_for_len(req->dst, req->nbytes);
+ if (unlikely(dst_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in dst S/G\n",
+ req->nbytes);
+ return ERR_PTR(dst_nents);
+ }
+
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_TO_DEVICE);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
+ DMA_FROM_DEVICE);
+ if (unlikely(!mapped_dst_nents)) {
+ dev_err(qidev, "unable to map destination\n");
+ dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+ }
+
+ iv_dma = dma_map_single(qidev, req->info, ivsize, DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, iv_dma)) {
+ dev_err(qidev, "unable to map IV\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0,
+ 0, 0, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ if (mapped_src_nents == 1 &&
+ iv_dma + ivsize == sg_dma_address(req->src)) {
+ in_contig = true;
+ qm_sg_ents = 0;
+ } else {
+ in_contig = false;
+ qm_sg_ents = 1 + mapped_src_nents;
+ }
+ dst_sg_idx = qm_sg_ents;
+
+ /* allocate space for base edesc and link tables */
+ edesc = qi_cache_alloc(GFP_DMA | flags);
+ if (unlikely(!edesc)) {
+ dev_err(qidev, "could not allocate extended descriptor\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+ edesc->iv_dma = iv_dma;
+ qm_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0;
+ sg_table = &edesc->sgt[0];
+ edesc->qm_sg_bytes = qm_sg_ents * sizeof(*sg_table);
+ edesc->drv_req.app_ctx = req;
+ edesc->drv_req.cbk = ablkcipher_done;
+ edesc->drv_req.drv_ctx = drv_ctx;
+
+ if (!in_contig) {
+ dma_to_qm_sg_one(sg_table, iv_dma, ivsize, 0);
+ sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table + 1, 0);
+ }
+
+ if (mapped_dst_nents > 1)
+ sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table +
+ dst_sg_idx, 0);
+
+ edesc->qm_sg_dma = dma_map_single(qidev, sg_table, edesc->qm_sg_bytes,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, edesc->qm_sg_dma)) {
+ dev_err(qidev, "unable to map S/G table\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ fd_sgt = &edesc->drv_req.fd_sgt[0];
+
+ if (!in_contig)
+ dma_to_qm_sg_one_last_ext(&fd_sgt[1], edesc->qm_sg_dma,
+ ivsize + req->nbytes, 0);
+ else
+ dma_to_qm_sg_one_last(&fd_sgt[1], iv_dma, ivsize + req->nbytes,
+ 0);
+
+ if (req->src == req->dst) {
+ if (!in_contig)
+ dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma +
+ sizeof(*sg_table), req->nbytes, 0);
+ else
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->src),
+ req->nbytes, 0);
+ } else if (mapped_dst_nents > 1) {
+ dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx *
+ sizeof(*sg_table), req->nbytes, 0);
+ } else {
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst),
+ req->nbytes, 0);
+ }
+
+ return edesc;
+}
+
+static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc(
+ struct skcipher_givcrypt_request *creq)
+{
+ struct ablkcipher_request *req = &creq->creq;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *qidev = ctx->qidev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ?
+ GFP_KERNEL : GFP_ATOMIC;
+ int src_nents, mapped_src_nents, dst_nents, mapped_dst_nents;
+ struct ablkcipher_edesc *edesc;
+ dma_addr_t iv_dma;
+ bool out_contig;
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ struct qm_sg_entry *sg_table, *fd_sgt;
+ int dst_sg_idx, qm_sg_ents;
+ struct caam_drv_ctx *drv_ctx;
+
+ drv_ctx = get_drv_ctx(ctx, GIVENCRYPT);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
+ return (struct ablkcipher_edesc *)drv_ctx;
+
+ src_nents = sg_nents_for_len(req->src, req->nbytes);
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->nbytes);
+ return ERR_PTR(src_nents);
+ }
+
+ if (unlikely(req->src != req->dst)) {
+ dst_nents = sg_nents_for_len(req->dst, req->nbytes);
+ if (unlikely(dst_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in dst S/G\n",
+ req->nbytes);
+ return ERR_PTR(dst_nents);
+ }
+
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_TO_DEVICE);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
+ DMA_FROM_DEVICE);
+ if (unlikely(!mapped_dst_nents)) {
+ dev_err(qidev, "unable to map destination\n");
+ dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ dst_nents = src_nents;
+ mapped_dst_nents = src_nents;
+ }
+
+ iv_dma = dma_map_single(qidev, creq->giv, ivsize, DMA_FROM_DEVICE);
+ if (dma_mapping_error(qidev, iv_dma)) {
+ dev_err(qidev, "unable to map IV\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0,
+ 0, 0, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ qm_sg_ents = mapped_src_nents > 1 ? mapped_src_nents : 0;
+ dst_sg_idx = qm_sg_ents;
+ if (mapped_dst_nents == 1 &&
+ iv_dma + ivsize == sg_dma_address(req->dst)) {
+ out_contig = true;
+ } else {
+ out_contig = false;
+ qm_sg_ents += 1 + mapped_dst_nents;
+ }
+
+ /* allocate space for base edesc and link tables */
+ edesc = qi_cache_alloc(GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(qidev, "could not allocate extended descriptor\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, GIVENCRYPT, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+ edesc->iv_dma = iv_dma;
+ sg_table = &edesc->sgt[0];
+ edesc->qm_sg_bytes = qm_sg_ents * sizeof(*sg_table);
+ edesc->drv_req.app_ctx = req;
+ edesc->drv_req.cbk = ablkcipher_done;
+ edesc->drv_req.drv_ctx = drv_ctx;
+
+ if (mapped_src_nents > 1)
+ sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table, 0);
+
+ if (!out_contig) {
+ dma_to_qm_sg_one(sg_table + dst_sg_idx, iv_dma, ivsize, 0);
+ sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table +
+ dst_sg_idx + 1, 0);
+ }
+
+ edesc->qm_sg_dma = dma_map_single(qidev, sg_table, edesc->qm_sg_bytes,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, edesc->qm_sg_dma)) {
+ dev_err(qidev, "unable to map S/G table\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, GIVENCRYPT, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ fd_sgt = &edesc->drv_req.fd_sgt[0];
+
+ if (mapped_src_nents > 1)
+ dma_to_qm_sg_one_ext(&fd_sgt[1], edesc->qm_sg_dma, req->nbytes,
+ 0);
+ else
+ dma_to_qm_sg_one(&fd_sgt[1], sg_dma_address(req->src),
+ req->nbytes, 0);
+
+ if (!out_contig)
+ dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx *
+ sizeof(*sg_table), ivsize + req->nbytes,
+ 0);
+ else
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst),
+ ivsize + req->nbytes, 0);
+
+ return edesc;
+}
+
+static inline int ablkcipher_crypt(struct ablkcipher_request *req, bool encrypt)
+{
+ struct ablkcipher_edesc *edesc;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ int ret;
+
+ if (unlikely(caam_congested))
+ return -EAGAIN;
+
+ /* allocate extended descriptor */
+ edesc = ablkcipher_edesc_alloc(req, encrypt);
+ if (IS_ERR(edesc))
+ return PTR_ERR(edesc);
+
+ ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ablkcipher_unmap(ctx->qidev, edesc, req);
+ qi_cache_free(edesc);
+ }
+
+ return ret;
+}
+
+static int ablkcipher_encrypt(struct ablkcipher_request *req)
+{
+ return ablkcipher_crypt(req, true);
+}
+
+static int ablkcipher_decrypt(struct ablkcipher_request *req)
+{
+ return ablkcipher_crypt(req, false);
+}
+
+static int ablkcipher_givencrypt(struct skcipher_givcrypt_request *creq)
+{
+ struct ablkcipher_request *req = &creq->creq;
+ struct ablkcipher_edesc *edesc;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ int ret;
+
+ if (unlikely(caam_congested))
+ return -EAGAIN;
+
+ /* allocate extended descriptor */
+ edesc = ablkcipher_giv_edesc_alloc(creq);
+ if (IS_ERR(edesc))
+ return PTR_ERR(edesc);
+
+ ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ablkcipher_unmap(ctx->qidev, edesc, req);
+ qi_cache_free(edesc);
+ }
+
+ return ret;
+}
+
+#define template_ablkcipher template_u.ablkcipher
+struct caam_alg_template {
+ char name[CRYPTO_MAX_ALG_NAME];
+ char driver_name[CRYPTO_MAX_ALG_NAME];
+ unsigned int blocksize;
+ u32 type;
+ union {
+ struct ablkcipher_alg ablkcipher;
+ } template_u;
+ u32 class1_alg_type;
+ u32 class2_alg_type;
+};
+
+static struct caam_alg_template driver_algs[] = {
+ /* ablkcipher descriptor */
+ {
+ .name = "cbc(aes)",
+ .driver_name = "cbc-aes-caam-qi",
+ .blocksize = AES_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ },
+ {
+ .name = "cbc(des3_ede)",
+ .driver_name = "cbc-3des-caam-qi",
+ .blocksize = DES3_EDE_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = DES3_EDE_KEY_SIZE,
+ .max_keysize = DES3_EDE_KEY_SIZE,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ },
+ {
+ .name = "cbc(des)",
+ .driver_name = "cbc-des-caam-qi",
+ .blocksize = DES_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = DES_KEY_SIZE,
+ .max_keysize = DES_KEY_SIZE,
+ .ivsize = DES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ },
+ {
+ .name = "ctr(aes)",
+ .driver_name = "ctr-aes-caam-qi",
+ .blocksize = 1,
+ .type = CRYPTO_ALG_TYPE_ABLKCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .geniv = "chainiv",
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CTR_MOD128,
+ },
+ {
+ .name = "rfc3686(ctr(aes))",
+ .driver_name = "rfc3686-ctr-aes-caam-qi",
+ .blocksize = 1,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = AES_MIN_KEY_SIZE +
+ CTR_RFC3686_NONCE_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE +
+ CTR_RFC3686_NONCE_SIZE,
+ .ivsize = CTR_RFC3686_IV_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CTR_MOD128,
+ },
+ {
+ .name = "xts(aes)",
+ .driver_name = "xts-aes-caam-qi",
+ .blocksize = AES_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_ABLKCIPHER,
+ .template_ablkcipher = {
+ .setkey = xts_ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .geniv = "eseqiv",
+ .min_keysize = 2 * AES_MIN_KEY_SIZE,
+ .max_keysize = 2 * AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XTS,
+ },
+};
+
+static struct caam_aead_alg driver_aeads[] = {
+ /* single-pass ipsec_esp descriptor */
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(md5),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-md5-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(md5),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-hmac-md5-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha1),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha1-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha1),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha1-cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha224),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha224-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha224),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha224-cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha256),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha256-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha256),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha256-cbc-aes-"
+ "caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha384),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha384-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha384),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha384-cbc-aes-"
+ "caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha512),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha512-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha512),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha512-cbc-aes-"
+ "caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(md5),cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-md5-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(md5),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-hmac-md5-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha1),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha1-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha1),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha1-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha224),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha224-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha224),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha224-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha256),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha256-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha256),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha256-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha384),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha384-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha384),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha384-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha512),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha512-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha512),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha512-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(md5),cbc(des))",
+ .cra_driver_name = "authenc-hmac-md5-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(md5),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-hmac-md5-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha1),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha1-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha1),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha1-cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha224),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha224-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha224),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha224-cbc-des-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha256),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha256-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha256),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha256-cbc-desi-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha384),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha384-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha384),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha384-cbc-des-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha512),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha512-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha512),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha512-cbc-des-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+};
+
+struct caam_crypto_alg {
+ struct list_head entry;
+ struct crypto_alg crypto_alg;
+ struct caam_alg_entry caam;
+};
+
+static int caam_init_common(struct caam_ctx *ctx, struct caam_alg_entry *caam)
+{
+ struct caam_drv_private *priv;
+
+ /*
+ * distribute tfms across job rings to ensure in-order
+ * crypto request processing per tfm
+ */
+ ctx->jrdev = caam_jr_alloc();
+ if (IS_ERR(ctx->jrdev)) {
+ pr_err("Job Ring Device allocation for transform failed\n");
+ return PTR_ERR(ctx->jrdev);
+ }
+
+ ctx->key_dma = dma_map_single(ctx->jrdev, ctx->key, sizeof(ctx->key),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(ctx->jrdev, ctx->key_dma)) {
+ dev_err(ctx->jrdev, "unable to map key\n");
+ caam_jr_free(ctx->jrdev);
+ return -ENOMEM;
+ }
+
+ /* copy descriptor header template value */
+ ctx->cdata.algtype = OP_TYPE_CLASS1_ALG | caam->class1_alg_type;
+ ctx->adata.algtype = OP_TYPE_CLASS2_ALG | caam->class2_alg_type;
+
+ priv = dev_get_drvdata(ctx->jrdev->parent);
+ ctx->qidev = priv->qidev;
+
+ spin_lock_init(&ctx->lock);
+ ctx->drv_ctx[ENCRYPT] = NULL;
+ ctx->drv_ctx[DECRYPT] = NULL;
+ ctx->drv_ctx[GIVENCRYPT] = NULL;
+
+ return 0;
+}
+
+static int caam_cra_init(struct crypto_tfm *tfm)
+{
+ struct crypto_alg *alg = tfm->__crt_alg;
+ struct caam_crypto_alg *caam_alg = container_of(alg, typeof(*caam_alg),
+ crypto_alg);
+ struct caam_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return caam_init_common(ctx, &caam_alg->caam);
+}
+
+static int caam_aead_init(struct crypto_aead *tfm)
+{
+ struct aead_alg *alg = crypto_aead_alg(tfm);
+ struct caam_aead_alg *caam_alg = container_of(alg, typeof(*caam_alg),
+ aead);
+ struct caam_ctx *ctx = crypto_aead_ctx(tfm);
+
+ return caam_init_common(ctx, &caam_alg->caam);
+}
+
+static void caam_exit_common(struct caam_ctx *ctx)
+{
+ caam_drv_ctx_rel(ctx->drv_ctx[ENCRYPT]);
+ caam_drv_ctx_rel(ctx->drv_ctx[DECRYPT]);
+ caam_drv_ctx_rel(ctx->drv_ctx[GIVENCRYPT]);
+
+ dma_unmap_single(ctx->jrdev, ctx->key_dma, sizeof(ctx->key),
+ DMA_TO_DEVICE);
+
+ caam_jr_free(ctx->jrdev);
+}
+
+static void caam_cra_exit(struct crypto_tfm *tfm)
+{
+ caam_exit_common(crypto_tfm_ctx(tfm));
+}
+
+static void caam_aead_exit(struct crypto_aead *tfm)
+{
+ caam_exit_common(crypto_aead_ctx(tfm));
+}
+
+static struct list_head alg_list;
+static void __exit caam_qi_algapi_exit(void)
+{
+ struct caam_crypto_alg *t_alg, *n;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(driver_aeads); i++) {
+ struct caam_aead_alg *t_alg = driver_aeads + i;
+
+ if (t_alg->registered)
+ crypto_unregister_aead(&t_alg->aead);
+ }
+
+ if (!alg_list.next)
+ return;
+
+ list_for_each_entry_safe(t_alg, n, &alg_list, entry) {
+ crypto_unregister_alg(&t_alg->crypto_alg);
+ list_del(&t_alg->entry);
+ kfree(t_alg);
+ }
+}
+
+static struct caam_crypto_alg *caam_alg_alloc(struct caam_alg_template
+ *template)
+{
+ struct caam_crypto_alg *t_alg;
+ struct crypto_alg *alg;
+
+ t_alg = kzalloc(sizeof(*t_alg), GFP_KERNEL);
+ if (!t_alg)
+ return ERR_PTR(-ENOMEM);
+
+ alg = &t_alg->crypto_alg;
+
+ snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s", template->name);
+ snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->driver_name);
+ alg->cra_module = THIS_MODULE;
+ alg->cra_init = caam_cra_init;
+ alg->cra_exit = caam_cra_exit;
+ alg->cra_priority = CAAM_CRA_PRIORITY;
+ alg->cra_blocksize = template->blocksize;
+ alg->cra_alignmask = 0;
+ alg->cra_ctxsize = sizeof(struct caam_ctx);
+ alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY |
+ template->type;
+ switch (template->type) {
+ case CRYPTO_ALG_TYPE_GIVCIPHER:
+ alg->cra_type = &crypto_givcipher_type;
+ alg->cra_ablkcipher = template->template_ablkcipher;
+ break;
+ case CRYPTO_ALG_TYPE_ABLKCIPHER:
+ alg->cra_type = &crypto_ablkcipher_type;
+ alg->cra_ablkcipher = template->template_ablkcipher;
+ break;
+ }
+
+ t_alg->caam.class1_alg_type = template->class1_alg_type;
+ t_alg->caam.class2_alg_type = template->class2_alg_type;
+
+ return t_alg;
+}
+
+static void caam_aead_alg_init(struct caam_aead_alg *t_alg)
+{
+ struct aead_alg *alg = &t_alg->aead;
+
+ alg->base.cra_module = THIS_MODULE;
+ alg->base.cra_priority = CAAM_CRA_PRIORITY;
+ alg->base.cra_ctxsize = sizeof(struct caam_ctx);
+ alg->base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY;
+
+ alg->init = caam_aead_init;
+ alg->exit = caam_aead_exit;
+}
+
+static int __init caam_qi_algapi_init(void)
+{
+ struct device_node *dev_node;
+ struct platform_device *pdev;
+ struct device *ctrldev;
+ struct caam_drv_private *priv;
+ int i = 0, err = 0;
+ u32 cha_vid, cha_inst, des_inst, aes_inst, md_inst;
+ unsigned int md_limit = SHA512_DIGEST_SIZE;
+ bool registered = false;
+
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
+ if (!dev_node) {
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec4.0");
+ if (!dev_node)
+ return -ENODEV;
+ }
+
+ pdev = of_find_device_by_node(dev_node);
+ of_node_put(dev_node);
+ if (!pdev)
+ return -ENODEV;
+
+ ctrldev = &pdev->dev;
+ priv = dev_get_drvdata(ctrldev);
+
+ /*
+ * If priv is NULL, it's probably because the caam driver wasn't
+ * properly initialized (e.g. RNG4 init failed). Thus, bail out here.
+ */
+ if (!priv || !priv->qi_present)
+ return -ENODEV;
+
+ INIT_LIST_HEAD(&alg_list);
+
+ /*
+ * Register crypto algorithms the device supports.
+ * First, detect presence and attributes of DES, AES, and MD blocks.
+ */
+ cha_vid = rd_reg32(&priv->ctrl->perfmon.cha_id_ls);
+ cha_inst = rd_reg32(&priv->ctrl->perfmon.cha_num_ls);
+ des_inst = (cha_inst & CHA_ID_LS_DES_MASK) >> CHA_ID_LS_DES_SHIFT;
+ aes_inst = (cha_inst & CHA_ID_LS_AES_MASK) >> CHA_ID_LS_AES_SHIFT;
+ md_inst = (cha_inst & CHA_ID_LS_MD_MASK) >> CHA_ID_LS_MD_SHIFT;
+
+ /* If MD is present, limit digest size based on LP256 */
+ if (md_inst && ((cha_vid & CHA_ID_LS_MD_MASK) == CHA_ID_LS_MD_LP256))
+ md_limit = SHA256_DIGEST_SIZE;
+
+ for (i = 0; i < ARRAY_SIZE(driver_algs); i++) {
+ struct caam_crypto_alg *t_alg;
+ struct caam_alg_template *alg = driver_algs + i;
+ u32 alg_sel = alg->class1_alg_type & OP_ALG_ALGSEL_MASK;
+
+ /* Skip DES algorithms if not supported by device */
+ if (!des_inst &&
+ ((alg_sel == OP_ALG_ALGSEL_3DES) ||
+ (alg_sel == OP_ALG_ALGSEL_DES)))
+ continue;
+
+ /* Skip AES algorithms if not supported by device */
+ if (!aes_inst && (alg_sel == OP_ALG_ALGSEL_AES))
+ continue;
+
+ t_alg = caam_alg_alloc(alg);
+ if (IS_ERR(t_alg)) {
+ err = PTR_ERR(t_alg);
+ dev_warn(priv->qidev, "%s alg allocation failed\n",
+ alg->driver_name);
+ continue;
+ }
+
+ err = crypto_register_alg(&t_alg->crypto_alg);
+ if (err) {
+ dev_warn(priv->qidev, "%s alg registration failed\n",
+ t_alg->crypto_alg.cra_driver_name);
+ kfree(t_alg);
+ continue;
+ }
+
+ list_add_tail(&t_alg->entry, &alg_list);
+ registered = true;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(driver_aeads); i++) {
+ struct caam_aead_alg *t_alg = driver_aeads + i;
+ u32 c1_alg_sel = t_alg->caam.class1_alg_type &
+ OP_ALG_ALGSEL_MASK;
+ u32 c2_alg_sel = t_alg->caam.class2_alg_type &
+ OP_ALG_ALGSEL_MASK;
+ u32 alg_aai = t_alg->caam.class1_alg_type & OP_ALG_AAI_MASK;
+
+ /* Skip DES algorithms if not supported by device */
+ if (!des_inst &&
+ ((c1_alg_sel == OP_ALG_ALGSEL_3DES) ||
+ (c1_alg_sel == OP_ALG_ALGSEL_DES)))
+ continue;
+
+ /* Skip AES algorithms if not supported by device */
+ if (!aes_inst && (c1_alg_sel == OP_ALG_ALGSEL_AES))
+ continue;
+
+ /*
+ * Check support for AES algorithms not available
+ * on LP devices.
+ */
+ if (((cha_vid & CHA_ID_LS_AES_MASK) == CHA_ID_LS_AES_LP) &&
+ (alg_aai == OP_ALG_AAI_GCM))
+ continue;
+
+ /*
+ * Skip algorithms requiring message digests
+ * if MD or MD size is not supported by device.
+ */
+ if (c2_alg_sel &&
+ (!md_inst || (t_alg->aead.maxauthsize > md_limit)))
+ continue;
+
+ caam_aead_alg_init(t_alg);
+
+ err = crypto_register_aead(&t_alg->aead);
+ if (err) {
+ pr_warn("%s alg registration failed\n",
+ t_alg->aead.base.cra_driver_name);
+ continue;
+ }
+
+ t_alg->registered = true;
+ registered = true;
+ }
+
+ if (registered)
+ dev_info(priv->qidev, "algorithms registered in /proc/crypto\n");
+
+ return err;
+}
+
+module_init(caam_qi_algapi_init);
+module_exit(caam_qi_algapi_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Support for crypto API using CAAM-QI backend");
+MODULE_AUTHOR("Freescale Semiconductor");
diff --git a/drivers/crypto/caam/sg_sw_qm.h b/drivers/crypto/caam/sg_sw_qm.h
new file mode 100644
index 000000000000..65b0f3c5670c
--- /dev/null
+++ b/drivers/crypto/caam/sg_sw_qm.h
@@ -0,0 +1,107 @@
+/*
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * * Neither the name of Freescale Semiconductor nor the
+ * names of its contributors may be used to endorse or promote products
+ * derived from this software without specific prior written permission.
+ *
+ *
+ * ALTERNATIVELY, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") as published by the Free Software
+ * Foundation, either version 2 of that License or (at your option) any
+ * later version.
+ *
+ * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY
+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+ * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __SG_SW_QM_H
+#define __SG_SW_QM_H
+
+#include <soc/fsl/qman.h>
+#include "regs.h"
+
+static inline void __dma_to_qm_sg(struct qm_sg_entry *qm_sg_ptr, dma_addr_t dma,
+ u16 offset)
+{
+ qm_sg_entry_set64(qm_sg_ptr, dma);
+ qm_sg_ptr->__reserved2 = 0;
+ qm_sg_ptr->bpid = 0;
+ qm_sg_entry_set_off(qm_sg_ptr, offset);
+}
+
+static inline void dma_to_qm_sg_one(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len, u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_entry_set_len(qm_sg_ptr, len);
+}
+
+static inline void dma_to_qm_sg_one_last(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len, u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_entry_set_f(qm_sg_ptr, len);
+}
+
+static inline void dma_to_qm_sg_one_ext(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len, u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_entry_set_e(qm_sg_ptr, len);
+}
+
+static inline void dma_to_qm_sg_one_last_ext(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len,
+ u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_entry_set_ef(qm_sg_ptr, len);
+}
+
+/*
+ * convert scatterlist to h/w link table format
+ * but does not have final bit; instead, returns last entry
+ */
+static inline struct qm_sg_entry *
+sg_to_qm_sg(struct scatterlist *sg, int sg_count,
+ struct qm_sg_entry *qm_sg_ptr, u16 offset)
+{
+ while (sg_count && sg) {
+ dma_to_qm_sg_one(qm_sg_ptr, sg_dma_address(sg),
+ sg_dma_len(sg), offset);
+ qm_sg_ptr++;
+ sg = sg_next(sg);
+ sg_count--;
+ }
+ return qm_sg_ptr - 1;
+}
+
+/*
+ * convert scatterlist to h/w link table format
+ * scatterlist must have been previously dma mapped
+ */
+static inline void sg_to_qm_sg_last(struct scatterlist *sg, int sg_count,
+ struct qm_sg_entry *qm_sg_ptr, u16 offset)
+{
+ qm_sg_ptr = sg_to_qm_sg(sg, sg_count, qm_sg_ptr, offset);
+ qm_sg_entry_set_f(qm_sg_ptr, qm_sg_entry_get_len(qm_sg_ptr));
+}
+
+#endif /* __SG_SW_QM_H */
--
2.4.4
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related
* Re: [RFC PATCH v2 01/32] x86: Add the Secure Encrypted Virtualization CPU feature
From: Borislav Petkov @ 2017-03-03 16:59 UTC (permalink / raw)
To: Brijesh Singh
Cc: simon.guinot, linux-efi, kvm, rkrcmar, matt, linux-pci,
linus.walleij, gary.hook, linux-mm, paul.gortmaker, hpa, cl,
dan.j.williams, aarcange, sfr, andriy.shevchenko, herbert, bhe,
xemul, joro, x86, peterz, piotr.luc, mingo, msalter, ross.zwisler,
dyoung, thomas.lendacky, jroedel, keescook, arnd, toshi.kani,
mathieu.desnoyers, luto, devel, bhelgaas
In-Reply-To: <148846752953.2349.17505492128445909591.stgit@brijesh-build-machine>
On Thu, Mar 02, 2017 at 10:12:09AM -0500, Brijesh Singh wrote:
> From: Tom Lendacky <thomas.lendacky@amd.com>
>
> Update the CPU features to include identifying and reporting on the
> Secure Encrypted Virtualization (SEV) feature. SME is identified by
> CPUID 0x8000001f, but requires BIOS support to enable it (set bit 23 of
> MSR_K8_SYSCFG and set bit 0 of MSR_K7_HWCR). Only show the SEV feature
> as available if reported by CPUID and enabled by BIOS.
>
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
> arch/x86/include/asm/cpufeatures.h | 1 +
> arch/x86/include/asm/msr-index.h | 2 ++
> arch/x86/kernel/cpu/amd.c | 22 ++++++++++++++++++----
> arch/x86/kernel/cpu/scattered.c | 1 +
> 4 files changed, 22 insertions(+), 4 deletions(-)
So this patchset is not really ontop of Tom's patchset because this
patch doesn't apply. The reason is, Tom did the SME bit this way:
https://lkml.kernel.org/r/20170216154236.19244.7580.stgit@tlendack-t1.amdoffice.net
but it should've been in scattered.c.
> diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
> index cabda87..c3f58d9 100644
> --- a/arch/x86/kernel/cpu/scattered.c
> +++ b/arch/x86/kernel/cpu/scattered.c
> @@ -31,6 +31,7 @@ static const struct cpuid_bit cpuid_bits[] = {
> { X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 },
> { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
> { X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 },
> + { X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 },
> { 0, 0, 0, 0, 0 }
... and here it is in scattered.c, as it should be. So you've used an
older version of the patch, it seems.
Please sync with Tom to see whether he's reworked the v4 version of that
patch already. If yes, then you could send only the SME and SEV adding
patches as a reply to this message so that I can continue reviewing in
the meantime.
Thanks.
--
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
^ permalink raw reply
* Re: [RFC PATCH v2 00/32] x86: Secure Encrypted Virtualization (AMD)
From: Bjorn Helgaas @ 2017-03-03 20:33 UTC (permalink / raw)
To: Brijesh Singh
Cc: simon.guinot, linux-efi, kvm, rkrcmar, matt, linux-pci,
linus.walleij, gary.hook, linux-mm, paul.gortmaker, hpa, cl,
dan.j.williams, aarcange, sfr, andriy.shevchenko, herbert, bhe,
xemul, joro, x86, peterz, piotr.luc, mingo, msalter, ross.zwisler,
bp, dyoung, thomas.lendacky, jroedel, keescook, arnd, toshi.kani,
mathieu.desnoyers, luto, devel, bhelgaas, tglx, mchehab,
iamjoonsoo.kim
In-Reply-To: <148846752022.2349.13667498174822419498.stgit@brijesh-build-machine>
On Thu, Mar 02, 2017 at 10:12:01AM -0500, Brijesh Singh wrote:
> This RFC series provides support for AMD's new Secure Encrypted Virtualization
> (SEV) feature. This RFC is build upon Secure Memory Encryption (SME) RFCv4 [1].
What kernel version is this series based on?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC PATCH v2 06/32] x86/pci: Use memremap when walking setup data
From: Bjorn Helgaas @ 2017-03-03 20:42 UTC (permalink / raw)
To: Brijesh Singh
Cc: linux-efi, labbott, kvm, rkrcmar, matt, linux-pci, linus.walleij,
gary.hook, linux-mm, hpa, cl, tglx, aarcange, sfr, mchehab,
simon.guinot, bhe, xemul, joro, x86, peterz, piotr.luc, mingo,
msalter, ross.zwisler, bp, dyoung, thomas.lendacky, jroedel,
keescook, arnd, toshi.kani, mathieu.desnoyers, luto, pbonzini,
bhelgaas, dan.j.williams, andriy.shevchenko, akpm, herbert,
tony.luck
In-Reply-To: <148846759008.2349.8274808454274279039.stgit@brijesh-build-machine>
On Thu, Mar 02, 2017 at 10:13:10AM -0500, Brijesh Singh wrote:
> From: Tom Lendacky <thomas.lendacky@amd.com>
>
> The use of ioremap will force the setup data to be mapped decrypted even
> though setup data is encrypted. Switch to using memremap which will be
> able to perform the proper mapping.
How should callers decide whether to use ioremap() or memremap()?
memremap() existed before SME and SEV, and this code is used even if
SME and SEV aren't supported, so the rationale for this change should
not need the decryption argument.
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
> arch/x86/pci/common.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index a4fdfa7..0b06670 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -691,7 +691,7 @@ int pcibios_add_device(struct pci_dev *dev)
>
> pa_data = boot_params.hdr.setup_data;
> while (pa_data) {
> - data = ioremap(pa_data, sizeof(*rom));
> + data = memremap(pa_data, sizeof(*rom), MEMREMAP_WB);
I can't quite connect the dots here. ioremap() on x86 would do
ioremap_nocache(). memremap(MEMREMAP_WB) would do arch_memremap_wb(),
which is ioremap_cache(). Is making a cacheable mapping the important
difference?
> if (!data)
> return -ENOMEM;
>
> @@ -710,7 +710,7 @@ int pcibios_add_device(struct pci_dev *dev)
> }
> }
> pa_data = data->next;
> - iounmap(data);
> + memunmap(data);
> }
> set_dma_domain_ops(dev);
> set_dev_domain_options(dev);
>
^ permalink raw reply
* Re: [RFC PATCH v2 00/32] x86: Secure Encrypted Virtualization (AMD)
From: Borislav Petkov @ 2017-03-03 20:51 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Brijesh Singh, simon.guinot, linux-efi, kvm, rkrcmar, matt,
linux-pci, linus.walleij, gary.hook, linux-mm, paul.gortmaker,
hpa, cl, dan.j.williams, aarcange, sfr, andriy.shevchenko,
herbert, bhe, xemul, joro, x86, peterz, piotr.luc, mingo, msalter,
ross.zwisler, dyoung, thomas.lendacky, jroedel, keescook, arnd,
toshi.kani, mathieu.desnoyers, luto
In-Reply-To: <20170303203323.GA5305@bhelgaas-glaptop.roam.corp.google.com>
On Fri, Mar 03, 2017 at 02:33:23PM -0600, Bjorn Helgaas wrote:
> On Thu, Mar 02, 2017 at 10:12:01AM -0500, Brijesh Singh wrote:
> > This RFC series provides support for AMD's new Secure Encrypted Virtualization
> > (SEV) feature. This RFC is build upon Secure Memory Encryption (SME) RFCv4 [1].
>
> What kernel version is this series based on?
Yeah, see that mail in [1]:
https://lkml.kernel.org/r/20170216154158.19244.66630.stgit@tlendack-t1.amdoffice.net
"This patch series is based off of the master branch of tip.
Commit a27cb9e1b2b4 ("Merge branch 'WIP.sched/core'")"
$ git describe a27cb9e1b2b4
v4.10-rc7-681-ga27cb9e1b2b4
So you need the SME pile first and then that SVE pile. But the first
patch needs refreshing as it is using a different base than the SME
pile. :-)
Tom, Brijesh, perhaps you guys could push a full tree somewhere - github
or so - for people to pull, in addition to the patchset on lkml.
Thanks.
--
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
^ permalink raw reply
* Re: [RFC PATCH v2 01/32] x86: Add the Secure Encrypted Virtualization CPU feature
From: Brijesh Singh @ 2017-03-03 21:01 UTC (permalink / raw)
To: Borislav Petkov
Cc: brijesh.singh, simon.guinot, linux-efi, kvm, rkrcmar, matt,
linux-pci, linus.walleij, gary.hook, linux-mm, paul.gortmaker,
hpa, cl, dan.j.williams, aarcange, sfr, andriy.shevchenko,
herbert, bhe, xemul, joro, x86, peterz, piotr.luc, mingo, msalter,
ross.zwisler, dyoung, thomas.lendacky, jroedel, keescook, arnd,
toshi.kani, mathieu.desnoyers, luto, devel, <bhel
In-Reply-To: <20170303165915.3233fx7wo74vsslx@pd.tnic>
Hi Boris,
On 03/03/2017 10:59 AM, Borislav Petkov wrote:
> On Thu, Mar 02, 2017 at 10:12:09AM -0500, Brijesh Singh wrote:
>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>
>> Update the CPU features to include identifying and reporting on the
>> Secure Encrypted Virtualization (SEV) feature. SME is identified by
>> CPUID 0x8000001f, but requires BIOS support to enable it (set bit 23 of
>> MSR_K8_SYSCFG and set bit 0 of MSR_K7_HWCR). Only show the SEV feature
>> as available if reported by CPUID and enabled by BIOS.
>>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>> arch/x86/include/asm/cpufeatures.h | 1 +
>> arch/x86/include/asm/msr-index.h | 2 ++
>> arch/x86/kernel/cpu/amd.c | 22 ++++++++++++++++++----
>> arch/x86/kernel/cpu/scattered.c | 1 +
>> 4 files changed, 22 insertions(+), 4 deletions(-)
>
> So this patchset is not really ontop of Tom's patchset because this
> patch doesn't apply. The reason is, Tom did the SME bit this way:
>
> https://lkml.kernel.org/r/20170216154236.19244.7580.stgit@tlendack-t1.amdoffice.net
>
> but it should've been in scattered.c.
>
>> diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
>> index cabda87..c3f58d9 100644
>> --- a/arch/x86/kernel/cpu/scattered.c
>> +++ b/arch/x86/kernel/cpu/scattered.c
>> @@ -31,6 +31,7 @@ static const struct cpuid_bit cpuid_bits[] = {
>> { X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 },
>> { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
>> { X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 },
>> + { X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 },
>> { 0, 0, 0, 0, 0 }
>
> ... and here it is in scattered.c, as it should be. So you've used an
> older version of the patch, it seems.
>
> Please sync with Tom to see whether he's reworked the v4 version of that
> patch already. If yes, then you could send only the SME and SEV adding
> patches as a reply to this message so that I can continue reviewing in
> the meantime.
>
Just realized my error, I actually end up using Tom's recent updates to
v4 instead of original v4. Here is the diff. If you have Tom's v4
applied then apply this diff before applying SEV v2 version. Sorry about
that.
Optionally, you also pull the complete tree from github [1].
[1] https://github.com/codomania/tip/tree/sev-rfc-v2
diff --git a/Documentation/admin-guide/kernel-parameters.txt
b/Documentation/admin-guide/kernel-parameters.txt
index 91c40fa..b91e2495 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2153,8 +2153,8 @@
mem_encrypt=on: Activate SME
mem_encrypt=off: Do not activate SME
- Refer to the SME documentation for details on when
- memory encryption can be activated.
+ Refer to Documentation/x86/amd-memory-encryption.txt
+ for details on when memory encryption can be activated.
mem_sleep_default= [SUSPEND] Default system suspend mode:
s2idle - Suspend-To-Idle
diff --git a/Documentation/x86/amd-memory-encryption.txt
b/Documentation/x86/amd-memory-encryption.txt
index 0938e89..0b72ff2 100644
--- a/Documentation/x86/amd-memory-encryption.txt
+++ b/Documentation/x86/amd-memory-encryption.txt
@@ -7,9 +7,9 @@ DRAM. SME can therefore be used to protect the contents
of DRAM from physical
attacks on the system.
A page is encrypted when a page table entry has the encryption bit set
(see
-below how to determine the position of the bit). The encryption bit can be
-specified in the cr3 register, allowing the PGD table to be encrypted. Each
-successive level of page tables can also be encrypted.
+below on how to determine its position). The encryption bit can be
specified
+in the cr3 register, allowing the PGD table to be encrypted. Each
successive
+level of page tables can also be encrypted.
Support for SME can be determined through the CPUID instruction. The CPUID
function 0x8000001f reports information related to SME:
@@ -17,13 +17,14 @@ function 0x8000001f reports information related to SME:
0x8000001f[eax]:
Bit[0] indicates support for SME
0x8000001f[ebx]:
- Bit[5:0] pagetable bit number used to activate memory
- encryption
- Bit[11:6] reduction in physical address space, in bits, when
- memory encryption is enabled (this only affects system
- physical addresses, not guest physical addresses)
-
-If support for SME is present, MSR 0xc00100010 (SYS_CFG) can be used to
+ Bits[5:0] pagetable bit number used to activate memory
+ encryption
+ Bits[11:6] reduction in physical address space, in bits, when
+ memory encryption is enabled (this only affects
+ system physical addresses, not guest physical
+ addresses)
+
+If support for SME is present, MSR 0xc00100010 (MSR_K8_SYSCFG) can be
used to
determine if SME is enabled and/or to enable memory encryption:
0xc0010010:
@@ -41,7 +42,7 @@ The state of SME in the Linux kernel can be documented
as follows:
The CPU supports SME (determined through CPUID instruction).
- Enabled:
- Supported and bit 23 of the SYS_CFG MSR is set.
+ Supported and bit 23 of MSR_K8_SYSCFG is set.
- Active:
Supported, Enabled and the Linux kernel is actively applying
@@ -51,7 +52,9 @@ The state of SME in the Linux kernel can be documented
as follows:
SME can also be enabled and activated in the BIOS. If SME is enabled and
activated in the BIOS, then all memory accesses will be encrypted and
it will
not be necessary to activate the Linux memory encryption support. If
the BIOS
-merely enables SME (sets bit 23 of the SYS_CFG MSR), then Linux can
activate
-memory encryption. However, if BIOS does not enable SME, then Linux
will not
-attempt to activate memory encryption, even if configured to do so by
default
-or the mem_encrypt=on command line parameter is specified.
+merely enables SME (sets bit 23 of the MSR_K8_SYSCFG), then Linux can
activate
+memory encryption by default
(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=y) or
+by supplying mem_encrypt=on on the kernel command line. However, if
BIOS does
+not enable SME, then Linux will not be able to activate memory
encryption, even
+if configured to do so by default or the mem_encrypt=on command line
parameter
+is specified.
diff --git a/arch/x86/include/asm/cpufeature.h
b/arch/x86/include/asm/cpufeature.h
index ea2de6a..d59c15c 100644
--- a/arch/x86/include/asm/cpufeature.h
+++ b/arch/x86/include/asm/cpufeature.h
@@ -28,7 +28,6 @@ enum cpuid_leafs
CPUID_8000_000A_EDX,
CPUID_7_ECX,
CPUID_8000_0007_EBX,
- CPUID_8000_001F_EAX,
};
#ifdef CONFIG_X86_FEATURE_NAMES
@@ -79,9 +78,8 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 15, feature_bit) || \
CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 16, feature_bit) || \
CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 17, feature_bit) || \
- CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 18, feature_bit) || \
REQUIRED_MASK_CHECK || \
- BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+ BUILD_BUG_ON_ZERO(NCAPINTS != 18))
#define DISABLED_MASK_BIT_SET(feature_bit) \
( CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 0, feature_bit) || \
@@ -102,9 +100,8 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 15, feature_bit) || \
CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 16, feature_bit) || \
CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 17, feature_bit) || \
- CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 18, feature_bit) || \
DISABLED_MASK_CHECK || \
- BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+ BUILD_BUG_ON_ZERO(NCAPINTS != 18))
#define cpu_has(c, bit) \
(__builtin_constant_p(bit) && REQUIRED_MASK_BIT_SET(bit) ? 1 : \
diff --git a/arch/x86/include/asm/cpufeatures.h
b/arch/x86/include/asm/cpufeatures.h
index 331fb81..b1a4468 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -12,7 +12,7 @@
/*
* Defines x86 CPU feature bits
*/
-#define NCAPINTS 19 /* N 32-bit words worth of info */
+#define NCAPINTS 18 /* N 32-bit words worth of info */
#define NBUGINTS 1 /* N 32-bit bug flags */
/*
@@ -187,6 +187,7 @@
* Reuse free bits when adding new feature flags!
*/
+#define X86_FEATURE_SME ( 7*32+ 0) /* AMD Secure Memory Encryption */
#define X86_FEATURE_CPB ( 7*32+ 2) /* AMD Core Performance Boost */
#define X86_FEATURE_EPB ( 7*32+ 3) /* IA32_ENERGY_PERF_BIAS support */
#define X86_FEATURE_CAT_L3 ( 7*32+ 4) /* Cache Allocation Technology L3 */
@@ -296,9 +297,6 @@
#define X86_FEATURE_SUCCOR (17*32+1) /* Uncorrectable error
containment and recovery */
#define X86_FEATURE_SMCA (17*32+3) /* Scalable MCA */
-/* AMD-defined CPU features, CPUID level 0x8000001f (eax), word 18 */
-#define X86_FEATURE_SME (18*32+0) /* Secure Memory Encryption */
-
/*
* BUG word(s)
*/
diff --git a/arch/x86/include/asm/disabled-features.h
b/arch/x86/include/asm/disabled-features.h
index 8b45e08..85599ad 100644
--- a/arch/x86/include/asm/disabled-features.h
+++ b/arch/x86/include/asm/disabled-features.h
@@ -57,7 +57,6 @@
#define DISABLED_MASK15 0
#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE)
#define DISABLED_MASK17 0
-#define DISABLED_MASK18 0
-#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
#endif /* _ASM_X86_DISABLED_FEATURES_H */
diff --git a/arch/x86/include/asm/required-features.h
b/arch/x86/include/asm/required-features.h
index 6847d85..fac9a5c 100644
--- a/arch/x86/include/asm/required-features.h
+++ b/arch/x86/include/asm/required-features.h
@@ -100,7 +100,6 @@
#define REQUIRED_MASK15 0
#define REQUIRED_MASK16 0
#define REQUIRED_MASK17 0
-#define REQUIRED_MASK18 0
-#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
#endif /* _ASM_X86_REQUIRED_FEATURES_H */
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index 35a5d5d..6bddda3 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -615,6 +615,29 @@ static void early_init_amd(struct cpuinfo_x86 *c)
*/
if (cpu_has_amd_erratum(c, amd_erratum_400))
set_cpu_bug(c, X86_BUG_AMD_E400);
+
+ /*
+ * BIOS support is required for SME. If BIOS has enabld SME then
+ * adjust x86_phys_bits by the SME physical address space reduction
+ * value. If BIOS has not enabled SME then don't advertise the
+ * feature (set in scattered.c).
+ */
+ if (c->extended_cpuid_level >= 0x8000001f) {
+ if (cpu_has(c, X86_FEATURE_SME)) {
+ u64 msr;
+
+ /* Check if SME is enabled */
+ rdmsrl(MSR_K8_SYSCFG, msr);
+ if (msr & MSR_K8_SYSCFG_MEM_ENCRYPT) {
+ unsigned int ebx;
+
+ ebx = cpuid_ebx(0x8000001f);
+ c->x86_phys_bits -= (ebx >> 6) & 0x3f;
+ } else {
+ clear_cpu_cap(c, X86_FEATURE_SME);
+ }
+ }
+ }
}
static void init_amd_k8(struct cpuinfo_x86 *c)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 358208d7..c188ae5 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -763,29 +763,6 @@ void get_cpu_cap(struct cpuinfo_x86 *c)
if (c->extended_cpuid_level >= 0x8000000a)
c->x86_capability[CPUID_8000_000A_EDX] = cpuid_edx(0x8000000a);
- if (c->extended_cpuid_level >= 0x8000001f) {
- cpuid(0x8000001f, &eax, &ebx, &ecx, &edx);
-
- /* SME feature support */
- if ((c->x86_vendor == X86_VENDOR_AMD) && (eax & 0x01)) {
- u64 msr;
-
- /*
- * For SME, BIOS support is required. If BIOS has
- * enabled SME adjust x86_phys_bits by the SME
- * physical address space reduction value. If BIOS
- * has not enabled SME don't advertise the feature.
- */
- rdmsrl(MSR_K8_SYSCFG, msr);
- if (msr & MSR_K8_SYSCFG_MEM_ENCRYPT)
- c->x86_phys_bits -= (ebx >> 6) & 0x3f;
- else
- eax &= ~0x01;
- }
-
- c->x86_capability[CPUID_8000_001F_EAX] = eax;
- }
-
init_scattered_cpuid_features(c);
/*
diff --git a/arch/x86/kernel/cpu/scattered.c
b/arch/x86/kernel/cpu/scattered.c
index d979406..cabda87 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -30,6 +30,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_HW_PSTATE, CPUID_EDX, 7, 0x80000007, 0 },
{ X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 },
{ X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
+ { X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 },
{ 0, 0, 0, 0, 0 }
};
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [RFC PATCH v2 00/32] x86: Secure Encrypted Virtualization (AMD)
From: Brijesh Singh @ 2017-03-03 21:15 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: brijesh.singh, simon.guinot, linux-efi, kvm, rkrcmar, matt,
linux-pci, linus.walleij, gary.hook, linux-mm, paul.gortmaker,
hpa, cl, dan.j.williams, aarcange, sfr, andriy.shevchenko,
herbert, bhe, xemul, joro, x86, peterz, piotr.luc, mingo, msalter,
ross.zwisler, bp, dyoung, thomas.lendacky, jroedel, keescook,
arnd, toshi.kani, mathieu.desnoyers, luto, devel
In-Reply-To: <20170303203323.GA5305@bhelgaas-glaptop.roam.corp.google.com>
Hi Bjorn,
On 03/03/2017 02:33 PM, Bjorn Helgaas wrote:
> On Thu, Mar 02, 2017 at 10:12:01AM -0500, Brijesh Singh wrote:
>> This RFC series provides support for AMD's new Secure Encrypted Virtualization
>> (SEV) feature. This RFC is build upon Secure Memory Encryption (SME) RFCv4 [1].
>
> What kernel version is this series based on?
>
This patch series is based off of the master branch of tip.
Commit a27cb9e1b2b4 ("Merge branch 'WIP.sched/core'")
Tom's RFC v4 patches (http://marc.info/?l=linux-mm&m=148725973013686&w=2)
Accidentally, I ended up rebasing SEV RFCv2 patches from updated SME v4
instead of original SME v4. So you may need to apply patch [1]
[1] http://marc.info/?l=linux-mm&m=148857523132253&w=2
Optionally, I have posted the full git tree here [2]
[2] https://github.com/codomania/tip/branches
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC PATCH v2 06/32] x86/pci: Use memremap when walking setup data
From: Tom Lendacky @ 2017-03-03 21:15 UTC (permalink / raw)
To: Bjorn Helgaas, Brijesh Singh
Cc: simon.guinot, linux-efi, kvm, rkrcmar, matt, linux-pci,
linus.walleij, gary.hook, linux-mm, paul.gortmaker, hpa, cl,
dan.j.williams, aarcange, sfr, andriy.shevchenko, herbert, bhe,
xemul, joro, x86, peterz, piotr.luc, mingo, msalter, ross.zwisler,
bp, dyoung, jroedel, keescook, arnd, toshi.kani,
mathieu.desnoyers, luto, devel, bhelgaas, tglx
In-Reply-To: <20170303204209.GA31767@bhelgaas-glaptop.roam.corp.google.com>
On 3/3/2017 2:42 PM, Bjorn Helgaas wrote:
> On Thu, Mar 02, 2017 at 10:13:10AM -0500, Brijesh Singh wrote:
>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>
>> The use of ioremap will force the setup data to be mapped decrypted even
>> though setup data is encrypted. Switch to using memremap which will be
>> able to perform the proper mapping.
>
> How should callers decide whether to use ioremap() or memremap()?
>
> memremap() existed before SME and SEV, and this code is used even if
> SME and SEV aren't supported, so the rationale for this change should
> not need the decryption argument.
When SME or SEV is active an ioremap() will remove the encryption bit
from the pagetable entry when it is mapped. This allows MMIO, which
doesn't support SME/SEV, to be performed successfully. So my take is
that ioremap() should be used for MMIO and memremap() for pages in RAM.
>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>> arch/x86/pci/common.c | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>> index a4fdfa7..0b06670 100644
>> --- a/arch/x86/pci/common.c
>> +++ b/arch/x86/pci/common.c
>> @@ -691,7 +691,7 @@ int pcibios_add_device(struct pci_dev *dev)
>>
>> pa_data = boot_params.hdr.setup_data;
>> while (pa_data) {
>> - data = ioremap(pa_data, sizeof(*rom));
>> + data = memremap(pa_data, sizeof(*rom), MEMREMAP_WB);
>
> I can't quite connect the dots here. ioremap() on x86 would do
> ioremap_nocache(). memremap(MEMREMAP_WB) would do arch_memremap_wb(),
> which is ioremap_cache(). Is making a cacheable mapping the important
> difference?
The memremap(MEMREMAP_WB) will actually check to see if it can perform
a __va(pa_data) in try_ram_remap() and then fallback to the
arch_memremap_wb(). So it's actually the __va() vs the ioremap_cache()
that is the difference.
Thanks,
Tom
>
>> if (!data)
>> return -ENOMEM;
>>
>> @@ -710,7 +710,7 @@ int pcibios_add_device(struct pci_dev *dev)
>> }
>> }
>> pa_data = data->next;
>> - iounmap(data);
>> + memunmap(data);
>> }
>> set_dma_domain_ops(dev);
>> set_dev_domain_options(dev);
>>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Crypto Fixes for 4.11
From: Herbert Xu @ 2017-03-04 7:41 UTC (permalink / raw)
To: Linus Torvalds, David S. Miller, Linux Kernel Mailing List,
Linux Crypto Mailing List
In-Reply-To: <20170111115646.GA8943@gondor.apana.org.au>
Hi Linus:
This push fixes the following issues:
- vmalloc stack regression in CCM.
- Build problem in CRC32 on ARM.
- Memory leak in cavium.
- Missing Kconfig dependencies in atmel and mediatek.
- XTS Regression on some platforms (s390 and ppc).
- Memory overrun in CCM test vector.
Please pull from
git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6.git linus
Ard Biesheuvel (3):
crypto: ccm - move cbcmac input off the stack
crypto: arm/crc32 - fix build error with outdated binutils
crypto: arm/crc32 - add build time test for CRC instruction support
Colin Ian King (1):
crypto: cavium - fix leak on curr if curr->head fails to be allocated
Geert Uytterhoeven (2):
crypto: atmel - CRYPTO_DEV_ATMEL_TDES and CRYPTO_DEV_ATMEL_SHA should depend on HAS_DMA
crypto: atmel - CRYPTO_DEV_MEDIATEK should depend on HAS_DMA
George Cherian (1):
crypto: cavium - Fix couple of static checker errors
Herbert Xu (2):
crypto: api - Add crypto_requires_off helper
crypto: xts - Propagate NEED_FALLBACK bit
Laura Abbott (1):
crypto: testmgr - Pad aes_ccm_enc_tv_template vector
Paulo Flabiano Smorigo (2):
crypto: vmx - Use skcipher for cbc fallback
crypto: vmx - Use skcipher for xts fallback
arch/arm/crypto/Makefile | 12 ++++++-
arch/arm/crypto/crc32-ce-core.S | 2 +-
crypto/ccm.c | 5 +--
crypto/testmgr.h | 2 +-
crypto/xts.c | 14 ++++----
drivers/crypto/Kconfig | 3 ++
drivers/crypto/cavium/cpt/cptvf_main.c | 5 ++-
drivers/crypto/cavium/cpt/cptvf_reqmanager.c | 4 +--
drivers/crypto/vmx/aes_cbc.c | 47 +++++++++++++-------------
drivers/crypto/vmx/aes_xts.c | 32 +++++++++---------
include/crypto/algapi.h | 7 +++-
11 files changed, 79 insertions(+), 54 deletions(-)
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [RFC PATCH v2 01/32] x86: Add the Secure Encrypted Virtualization CPU feature
From: Borislav Petkov @ 2017-03-04 10:11 UTC (permalink / raw)
To: Brijesh Singh
Cc: linux-efi, kvm, rkrcmar, matt, linux-pci, linus.walleij,
gary.hook, linux-mm, hpa, cl, tglx, aarcange, sfr, mchehab,
simon.guinot, bhe, xemul, joro, x86, peterz, piotr.luc, mingo,
msalter, ross.zwisler, labbott, dyoung, thomas.lendacky, jroedel,
keescook, arnd, toshi.kani, mathieu.desnoyers, luto, pbonzini,
bhelgaas, dan.j.williams, andriy.shevchenko, akpm, herbert,
tony.luck, pau
In-Reply-To: <404fafd8-bbd6-b8c7-1abb-787ac083ea41@amd.com>
On Fri, Mar 03, 2017 at 03:01:23PM -0600, Brijesh Singh wrote:
> +merely enables SME (sets bit 23 of the MSR_K8_SYSCFG), then Linux can
> activate
> +memory encryption by default (CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=y)
> or
> +by supplying mem_encrypt=on on the kernel command line. However, if BIOS
> does
> +not enable SME, then Linux will not be able to activate memory encryption,
> even
> +if configured to do so by default or the mem_encrypt=on command line
> parameter
> +is specified.
This looks like a wraparound...
$ test-apply.sh /tmp/brijesh.singh.delta
checking file Documentation/admin-guide/kernel-parameters.txt
Hunk #1 succeeded at 2144 (offset -9 lines).
checking file Documentation/x86/amd-memory-encryption.txt
patch: **** malformed patch at line 23: DRAM from physical
Yap.
Looks like exchange or your mail client decided to do some patch editing
on its own.
Please send it to yourself first and try applying.
--
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
^ permalink raw reply
* crypto: deadlock between crypto_alg_sem/rtnl_mutex/genl_mutex
From: Dmitry Vyukov @ 2017-03-05 15:08 UTC (permalink / raw)
To: Herbert Xu, David Miller, linux-crypto, LKML, Eric Dumazet,
Cong Wang, netdev
Cc: syzkaller
Hello,
I am getting the following deadlock reports while running syzkaller
fuzzer on net-next/8d70eeb84ab277377c017af6a21d0a337025dede:
======================================================
[ INFO: possible circular locking dependency detected ]
4.10.0+ #5 Not tainted
-------------------------------------------------------
syz-executor6/6143 is trying to acquire lock:
(nlk->cb_mutex){+.+.+.}, at: [<ffffffff837df634>]
__netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
but task is already holding lock:
(crypto_alg_sem){+++++.}, at: [<ffffffff821cd1f6>]
crypto_user_rcv_msg+0x136/0x4f0 crypto/crypto_user.c:507
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #4 (crypto_alg_sem){+++++.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
down_read+0x9b/0x150 kernel/locking/rwsem.c:23
crypto_alg_lookup+0x23/0x50 crypto/api.c:199
crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
crypto_larval_lookup crypto/api.c:211 [inline]
crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
crypto_alloc_base+0x50/0x1e0 crypto/api.c:416
crypto_alloc_cipher include/linux/crypto.h:1407 [inline]
tcp_fastopen_reset_cipher+0xc2/0x2e0 net/ipv4/tcp_fastopen.c:48
tcp_fastopen_init_key_once+0x114/0x120 net/ipv4/tcp_fastopen.c:29
do_tcp_setsockopt.isra.36+0x140a/0x20a0 net/ipv4/tcp.c:2684
tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #3 (sk_lock-AF_INET){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
lock_sock include/net/sock.h:1460 [inline]
rds_tcp_listen_stop+0x57/0x140 net/rds/tcp_listen.c:284
rds_tcp_kill_sock net/rds/tcp.c:529 [inline]
rds_tcp_dev_event+0x383/0xc50 net/rds/tcp.c:568
notifier_call_chain+0x1b5/0x2b0 kernel/notifier.c:93
__raw_notifier_call_chain kernel/notifier.c:394 [inline]
raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1646
call_netdevice_notifiers net/core/dev.c:1662 [inline]
netdev_run_todo+0x3b2/0xa30 net/core/dev.c:7530
rtnl_unlock+0xe/0x10 net/core/rtnetlink.c:104
default_device_exit_batch+0x504/0x620 net/core/dev.c:8334
ops_exit_list.isra.6+0x100/0x150 net/core/net_namespace.c:144
cleanup_net+0x551/0xa90 net/core/net_namespace.c:463
process_one_work+0xbd0/0x1c10 kernel/workqueue.c:2096
worker_thread+0x223/0x1990 kernel/workqueue.c:2230
kthread+0x326/0x3f0 kernel/kthread.c:229
ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
-> #2 (rtnl_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
tipc_nl_bearer_dump+0x3ef/0x720 net/tipc/bearer.c:774
genl_lock_dumpit+0x68/0x90 net/netlink/genetlink.c:479
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #1 (genl_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
genl_lock net/netlink/genetlink.c:32 [inline]
genl_lock_dumpit+0x41/0x90 net/netlink/genetlink.c:478
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #0 (nlk->cb_mutex){+.+.+.}:
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
__netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
netlink_dump_start include/linux/netlink.h:165 [inline]
crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
__sys_sendmsg+0x138/0x300 net/socket.c:2019
SYSC_sendmsg net/socket.c:2030 [inline]
SyS_sendmsg+0x2d/0x50 net/socket.c:2026
entry_SYSCALL_64_fastpath+0x1f/0xc2
other info that might help us debug this:
Chain exists of:
nlk->cb_mutex --> sk_lock-AF_INET --> crypto_alg_sem
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(crypto_alg_sem);
lock(sk_lock-AF_INET);
lock(crypto_alg_sem);
lock(nlk->cb_mutex);
*** DEADLOCK ***
2 locks held by syz-executor6/6143:
#0: (crypto_cfg_mutex){+.+...}, at: [<ffffffff821cad9b>]
crypto_netlink_rcv+0x1b/0x40 crypto/crypto_user.c:537
#1: (crypto_alg_sem){+++++.}, at: [<ffffffff821cd1f6>]
crypto_user_rcv_msg+0x136/0x4f0 crypto/crypto_user.c:507
stack backtrace:
CPU: 0 PID: 6143 Comm: syz-executor6 Not tainted 4.10.0+ #5
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
__netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
netlink_dump_start include/linux/netlink.h:165 [inline]
crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
__sys_sendmsg+0x138/0x300 net/socket.c:2019
SYSC_sendmsg net/socket.c:2030 [inline]
SyS_sendmsg+0x2d/0x50 net/socket.c:2026
entry_SYSCALL_64_fastpath+0x1f/0xc2
^ permalink raw reply
* [PATCH] crypto: s5p-sss - Fix completing crypto request in IRQ handler
From: Krzysztof Kozlowski @ 2017-03-05 17:14 UTC (permalink / raw)
To: Herbert Xu, David S . Miller, linux-crypto, linux-kernel
Cc: Nathan Royce, Krzysztof Kozlowski
In a regular interrupt handler driver was finishing the crypt/decrypt
request by calling complete on crypto request. This is disallowed since
converting to skcipher in commit b286d8b1a690 ("crypto: skcipher - Add
skcipher walk interface") and causes a warning:
WARNING: CPU: 0 PID: 0 at crypto/skcipher.c:430 skcipher_walk_first+0x13c/0x14c
The interrupt is marked shared but in fact there are no other users
sharing it. Thus the simplest solution seems to be to just use a
threaded interrupt handler, after converting it to oneshot.
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
---
Tested on Odroid U3 and XU3.
Not sure which commit introduced it so I am also not sure if this should
be cc-stable.
---
drivers/crypto/s5p-sss.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/s5p-sss.c b/drivers/crypto/s5p-sss.c
index dce1af0ce85c..a668286d62cb 100644
--- a/drivers/crypto/s5p-sss.c
+++ b/drivers/crypto/s5p-sss.c
@@ -805,8 +805,9 @@ static int s5p_aes_probe(struct platform_device *pdev)
dev_warn(dev, "feed control interrupt is not available.\n");
goto err_irq;
}
- err = devm_request_irq(dev, pdata->irq_fc, s5p_aes_interrupt,
- IRQF_SHARED, pdev->name, pdev);
+ err = devm_request_threaded_irq(dev, pdata->irq_fc, NULL,
+ s5p_aes_interrupt, IRQF_ONESHOT,
+ pdev->name, pdev);
if (err < 0) {
dev_warn(dev, "feed control interrupt is not available.\n");
goto err_irq;
--
2.9.3
^ permalink raw reply related
* Re: XTS Crypto Not Found In /proc/crypto Even After Compiled for 4.10.1.
From: Krzysztof Kozlowski @ 2017-03-05 17:16 UTC (permalink / raw)
To: Herbert Xu
Cc: Nathan Royce, davem, linux-crypto, linux-kernel, Marek Szyprowski
In-Reply-To: <20170303040210.GA24737@gondor.apana.org.au>
On Fri, Mar 03, 2017 at 12:02:10PM +0800, Herbert Xu wrote:
> On Thu, Mar 02, 2017 at 05:35:30PM -0600, Nathan Royce wrote:
> > ARM ODroid XU4
> >
> > $ cat /proc/config.gz | gunzip | grep XTS
> > CONFIG_CRYPTO_XTS=y
> >
> > $ grep xts /proc/crypto
> > //4.9.13
> > name : xts(aes)
> > driver : xts(aes-generic)
> > //4.10.1
> > <blank>
> > //cbc can be found though
> >
> > CRYPTTAB:
> > cryptswap1 UUID=<sanitized> /dev/urandom
> > swap,offset=2048,cipher=aes-xts-plain64:sha512,size=512,nofail
> >
> > FSTAB:
> > /dev/mapper/cryptswap1 none swap sw 0 0
> >
> > Boot Log:
> > [ 10.535985] ------------[ cut here ]------------
> > [ 10.539252] WARNING: CPU: 0 PID: 0 at crypto/skcipher.c:430
> > skcipher_walk_first+0x13c/0x14c
> > [ 10.547542] Modules linked in: xor xor_neon aes_arm zlib_deflate
> > dm_crypt raid6_pq nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
> > ip_tables x_tables
> > [ 10.561716] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.10.1-dirty #1
> > [ 10.568049] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
> > [ 10.574171] [<c010eb54>] (unwind_backtrace) from [<c010b408>]
> > (show_stack+0x10/0x14)
> > [ 10.581893] [<c010b408>] (show_stack) from [<c036772c>]
> > (dump_stack+0x84/0x98)
> > [ 10.589073] [<c036772c>] (dump_stack) from [<c011bb20>]
> > (__warn+0xe8/0x100)
> > [ 10.595975] [<c011bb20>] (__warn) from [<c011bc30>]
> > (warn_slowpath_null+0x20/0x28)
> > [ 10.603546] [<c011bc30>] (warn_slowpath_null) from [<c0329a48>]
> > (skcipher_walk_first+0x13c/0x14c)
> > [ 10.612390] [<c0329a48>] (skcipher_walk_first) from [<c0329b30>]
> > (skcipher_walk_virt+0x1c/0x38)
> > [ 10.621056] [<c0329b30>] (skcipher_walk_virt) from [<c0330ed0>]
> > (post_crypt+0x38/0x1c4)
> > [ 10.629022] [<c0330ed0>] (post_crypt) from [<c0331470>]
> > (decrypt_done+0x4c/0x54)
> > [ 10.636389] [<c0331470>] (decrypt_done) from [<c05a03f0>]
> > (s5p_aes_complete+0x70/0xfc)
> > [ 10.644274] [<c05a03f0>] (s5p_aes_complete) from [<c05a05b0>]
> > (s5p_aes_interrupt+0x134/0x1a0)
> > [ 10.652771] [<c05a05b0>] (s5p_aes_interrupt) from [<c016dc3c>]
> > (__handle_irq_event_percpu+0x9c/0x124)
>
> This looks like a bug in the s5p driver. It's calling the completion
> function straight from the IRQ handler, which is triggering the
> sanity check in skcipher_walk_first.
>
> The s5p driver needs to schedule a tasklet to call the completion
> function.
Tasklet... or threaded IRQ handler maybe? I sent a fix.
BTW, I subscribe the crypto list but I could not find the original email
there.
Best regards,
Krzysztof
^ permalink raw reply
* Re: crypto: deadlock between crypto_alg_sem/rtnl_mutex/genl_mutex
From: Dmitry Vyukov @ 2017-03-05 17:36 UTC (permalink / raw)
To: Herbert Xu, David Miller, linux-crypto, LKML, Eric Dumazet,
Cong Wang, netdev
Cc: syzkaller
In-Reply-To: <CACT4Y+bLbNbf0T376JHvemuY+za0+qgrfOKWzvJg5vAkJTCRFg@mail.gmail.com>
On Sun, Mar 5, 2017 at 4:08 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> Hello,
>
> I am getting the following deadlock reports while running syzkaller
> fuzzer on net-next/8d70eeb84ab277377c017af6a21d0a337025dede:
>
> ======================================================
> [ INFO: possible circular locking dependency detected ]
> 4.10.0+ #5 Not tainted
> -------------------------------------------------------
> syz-executor6/6143 is trying to acquire lock:
> (nlk->cb_mutex){+.+.+.}, at: [<ffffffff837df634>]
> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
>
> but task is already holding lock:
> (crypto_alg_sem){+++++.}, at: [<ffffffff821cd1f6>]
> crypto_user_rcv_msg+0x136/0x4f0 crypto/crypto_user.c:507
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #4 (crypto_alg_sem){+++++.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> down_read+0x9b/0x150 kernel/locking/rwsem.c:23
> crypto_alg_lookup+0x23/0x50 crypto/api.c:199
> crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
> crypto_larval_lookup crypto/api.c:211 [inline]
> crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
> crypto_alloc_base+0x50/0x1e0 crypto/api.c:416
> crypto_alloc_cipher include/linux/crypto.h:1407 [inline]
> tcp_fastopen_reset_cipher+0xc2/0x2e0 net/ipv4/tcp_fastopen.c:48
> tcp_fastopen_init_key_once+0x114/0x120 net/ipv4/tcp_fastopen.c:29
> do_tcp_setsockopt.isra.36+0x140a/0x20a0 net/ipv4/tcp.c:2684
> tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
> sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
> SYSC_setsockopt net/socket.c:1786 [inline]
> SyS_setsockopt+0x25c/0x390 net/socket.c:1765
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #3 (sk_lock-AF_INET){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
> lock_sock include/net/sock.h:1460 [inline]
> rds_tcp_listen_stop+0x57/0x140 net/rds/tcp_listen.c:284
> rds_tcp_kill_sock net/rds/tcp.c:529 [inline]
> rds_tcp_dev_event+0x383/0xc50 net/rds/tcp.c:568
> notifier_call_chain+0x1b5/0x2b0 kernel/notifier.c:93
> __raw_notifier_call_chain kernel/notifier.c:394 [inline]
> raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
> call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1646
> call_netdevice_notifiers net/core/dev.c:1662 [inline]
> netdev_run_todo+0x3b2/0xa30 net/core/dev.c:7530
> rtnl_unlock+0xe/0x10 net/core/rtnetlink.c:104
> default_device_exit_batch+0x504/0x620 net/core/dev.c:8334
> ops_exit_list.isra.6+0x100/0x150 net/core/net_namespace.c:144
> cleanup_net+0x551/0xa90 net/core/net_namespace.c:463
> process_one_work+0xbd0/0x1c10 kernel/workqueue.c:2096
> worker_thread+0x223/0x1990 kernel/workqueue.c:2230
> kthread+0x326/0x3f0 kernel/kthread.c:229
> ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
>
> -> #2 (rtnl_mutex){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
> tipc_nl_bearer_dump+0x3ef/0x720 net/tipc/bearer.c:774
> genl_lock_dumpit+0x68/0x90 net/netlink/genetlink.c:479
> netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
> __netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
> genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
> genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> sock_write_iter+0x326/0x600 net/socket.c:846
> call_write_iter include/linux/fs.h:1733 [inline]
> new_sync_write fs/read_write.c:497 [inline]
> __vfs_write+0x483/0x740 fs/read_write.c:510
> vfs_write+0x187/0x530 fs/read_write.c:558
> SYSC_write fs/read_write.c:605 [inline]
> SyS_write+0xfb/0x230 fs/read_write.c:597
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #1 (genl_mutex){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> genl_lock net/netlink/genetlink.c:32 [inline]
> genl_lock_dumpit+0x41/0x90 net/netlink/genetlink.c:478
> netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
> __netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
> genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
> genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> sock_write_iter+0x326/0x600 net/socket.c:846
> call_write_iter include/linux/fs.h:1733 [inline]
> new_sync_write fs/read_write.c:497 [inline]
> __vfs_write+0x483/0x740 fs/read_write.c:510
> vfs_write+0x187/0x530 fs/read_write.c:558
> SYSC_write fs/read_write.c:605 [inline]
> SyS_write+0xfb/0x230 fs/read_write.c:597
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #0 (nlk->cb_mutex){+.+.+.}:
> check_prev_add kernel/locking/lockdep.c:1830 [inline]
> check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
> netlink_dump_start include/linux/netlink.h:165 [inline]
> crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> ___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
> __sys_sendmsg+0x138/0x300 net/socket.c:2019
> SYSC_sendmsg net/socket.c:2030 [inline]
> SyS_sendmsg+0x2d/0x50 net/socket.c:2026
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> other info that might help us debug this:
>
> Chain exists of:
> nlk->cb_mutex --> sk_lock-AF_INET --> crypto_alg_sem
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(crypto_alg_sem);
> lock(sk_lock-AF_INET);
> lock(crypto_alg_sem);
> lock(nlk->cb_mutex);
>
> *** DEADLOCK ***
>
> 2 locks held by syz-executor6/6143:
> #0: (crypto_cfg_mutex){+.+...}, at: [<ffffffff821cad9b>]
> crypto_netlink_rcv+0x1b/0x40 crypto/crypto_user.c:537
> #1: (crypto_alg_sem){+++++.}, at: [<ffffffff821cd1f6>]
> crypto_user_rcv_msg+0x136/0x4f0 crypto/crypto_user.c:507
>
> stack backtrace:
> CPU: 0 PID: 6143 Comm: syz-executor6 Not tainted 4.10.0+ #5
> Hardware name: Google Google Compute Engine/Google Compute Engine,
> BIOS Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:16 [inline]
> dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
> print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
> check_prev_add kernel/locking/lockdep.c:1830 [inline]
> check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
> netlink_dump_start include/linux/netlink.h:165 [inline]
> crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> ___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
> __sys_sendmsg+0x138/0x300 net/socket.c:2019
> SYSC_sendmsg net/socket.c:2030 [inline]
> SyS_sendmsg+0x2d/0x50 net/socket.c:2026
> entry_SYSCALL_64_fastpath+0x1f/0xc2
Another one involving tcp_md5sig_mutex:
======================================================
[ INFO: possible circular locking dependency detected ]
SELinux: unrecognized netlink message: protocol=9 nlmsg_type=2
sclass=netlink_audit_socket pig=4033 comm=syz-executor4
4.10.0+ #5 Not tainted
-------------------------------------------------------
syz-executor8/4018 is trying to acquire lock:
(crypto_alg_sem){++++++}, at: [<ffffffff82193473>]
crypto_alg_lookup+0x23/0x50 crypto/api.c:199
but task is already holding lock:
(tcp_md5sig_mutex){+.+...}, at: [<ffffffff838e4efa>]
tcp_alloc_md5sig_pool+0x4a/0x470 net/ipv4/tcp.c:3196
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #5 (tcp_md5sig_mutex){+.+...}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
tcp_alloc_md5sig_pool+0x4a/0x470 net/ipv4/tcp.c:3196
tcp_md5_do_add+0x1d8/0x5d0 net/ipv4/tcp_ipv4.c:969
tcp_v4_parse_md5_keys+0x1c7/0x2b0 net/ipv4/tcp_ipv4.c:1037
do_tcp_setsockopt.isra.36+0x657/0x20a0 net/ipv4/tcp.c:2668
tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
-> #4 (sk_lock-AF_INET){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
lock_sock include/net/sock.h:1460 [inline]
do_ip_setsockopt.isra.12+0x301/0x3760 net/ipv4/ip_sockglue.c:653
ip_setsockopt+0x3a/0xb0 net/ipv4/ip_sockglue.c:1265
tcp_setsockopt+0x82/0xd0 net/ipv4/tcp.c:2731
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #3 (rtnl_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
nl80211_prepare_wdev_dump.isra.37+0x2c/0x5d0 net/wireless/nl80211.c:548
nl80211_dump_station+0x178/0xd80 net/wireless/nl80211.c:4455
genl_lock_dumpit+0x68/0x90 net/netlink/genetlink.c:479
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #2 (genl_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
genl_lock net/netlink/genetlink.c:32 [inline]
genl_lock_dumpit+0x41/0x90 net/netlink/genetlink.c:478
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #1 (nlk->cb_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
__netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
netlink_dump_start include/linux/netlink.h:165 [inline]
crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
__sys_sendmsg+0x138/0x300 net/socket.c:2019
SYSC_sendmsg net/socket.c:2030 [inline]
SyS_sendmsg+0x2d/0x50 net/socket.c:2026
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #0 (crypto_alg_sem){++++++}:
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
down_read+0x9b/0x150 kernel/locking/rwsem.c:23
crypto_alg_lookup+0x23/0x50 crypto/api.c:199
crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
crypto_larval_lookup crypto/api.c:211 [inline]
crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
crypto_find_alg crypto/api.c:500 [inline]
crypto_alloc_tfm+0x101/0x2e0 crypto/api.c:533
crypto_alloc_ahash+0x2c/0x40 crypto/ahash.c:525
__tcp_alloc_md5sig_pool net/ipv4/tcp.c:3158 [inline]
tcp_alloc_md5sig_pool+0x85/0x470 net/ipv4/tcp.c:3199
tcp_md5_do_add+0x1d8/0x5d0 net/ipv4/tcp_ipv4.c:969
tcp_v4_parse_md5_keys+0x1c7/0x2b0 net/ipv4/tcp_ipv4.c:1037
do_tcp_setsockopt.isra.36+0x657/0x20a0 net/ipv4/tcp.c:2668
tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
other info that might help us debug this:
Chain exists of:
crypto_alg_sem --> sk_lock-AF_INET --> tcp_md5sig_mutex
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(tcp_md5sig_mutex);
lock(sk_lock-AF_INET);
lock(tcp_md5sig_mutex);
lock(crypto_alg_sem);
*** DEADLOCK ***
2 locks held by syz-executor8/4018:
#0: (sk_lock-AF_INET){+.+.+.}, at: [<ffffffff838e7a26>] lock_sock
include/net/sock.h:1460 [inline]
#0: (sk_lock-AF_INET){+.+.+.}, at: [<ffffffff838e7a26>]
do_tcp_setsockopt.isra.36+0x216/0x20a0 net/ipv4/tcp.c:2466
#1: (tcp_md5sig_mutex){+.+...}, at: [<ffffffff838e4efa>]
tcp_alloc_md5sig_pool+0x4a/0x470 net/ipv4/tcp.c:3196
stack backtrace:
CPU: 0 PID: 4018 Comm: syz-executor8 Not tainted 4.10.0+ #5
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
down_read+0x9b/0x150 kernel/locking/rwsem.c:23
crypto_alg_lookup+0x23/0x50 crypto/api.c:199
crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
crypto_larval_lookup crypto/api.c:211 [inline]
crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
crypto_find_alg crypto/api.c:500 [inline]
crypto_alloc_tfm+0x101/0x2e0 crypto/api.c:533
crypto_alloc_ahash+0x2c/0x40 crypto/ahash.c:525
__tcp_alloc_md5sig_pool net/ipv4/tcp.c:3158 [inline]
tcp_alloc_md5sig_pool+0x85/0x470 net/ipv4/tcp.c:3199
tcp_md5_do_add+0x1d8/0x5d0 net/ipv4/tcp_ipv4.c:969
tcp_v4_parse_md5_keys+0x1c7/0x2b0 net/ipv4/tcp_ipv4.c:1037
do_tcp_setsockopt.isra.36+0x657/0x20a0 net/ipv4/tcp.c:2668
tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
^ permalink raw reply
* Re: [PATCH] crypto: powerpc - Fix initialisation of crc32c context
From: Anton Blanchard @ 2017-03-06 2:48 UTC (permalink / raw)
To: Daniel Axtens; +Cc: linuxppc-dev, linux-crypto
In-Reply-To: <20170303065655.7317-1-dja@axtens.net>
Hi Daniel,
> Turning on crypto self-tests on a POWER8 shows:
>
> alg: hash: Test 1 failed for crc32c-vpmsum
> 00000000: ff ff ff ff
>
> Comparing the code with the Intel CRC32c implementation on which
> ours is based shows that we are doing an init with 0, not ~0
> as CRC32c requires.
>
> This probably wasn't caught because btrfs does its own weird
> open-coded initialisation.
>
> Initialise our internal context to ~0 on init.
>
> This makes the self-tests pass, and btrfs continues to work.
Thanks! Not sure how I screwed that up.
Acked-by: Anton Blanchard <anton@samba.org>
> Fixes: 6dd7a82cc54e ("crypto: powerpc - Add POWER8 optimised crc32c")
> Cc: Anton Blanchard <anton@samba.org>
> Cc: stable@vger.kernel.org
> Signed-off-by: Daniel Axtens <dja@axtens.net>
> ---
> arch/powerpc/crypto/crc32c-vpmsum_glue.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/crypto/crc32c-vpmsum_glue.c
> b/arch/powerpc/crypto/crc32c-vpmsum_glue.c index
> 9fa046d56eba..411994551afc 100644 ---
> a/arch/powerpc/crypto/crc32c-vpmsum_glue.c +++
> b/arch/powerpc/crypto/crc32c-vpmsum_glue.c @@ -52,7 +52,7 @@ static
> int crc32c_vpmsum_cra_init(struct crypto_tfm *tfm) {
> u32 *key = crypto_tfm_ctx(tfm);
>
> - *key = 0;
> + *key = ~0;
>
> return 0;
> }
^ permalink raw reply
* Re: [PATCH] crypto: powerpc - Fix initialisation of crc32c context
From: Michael Ellerman @ 2017-03-06 4:43 UTC (permalink / raw)
To: Daniel Axtens, linuxppc-dev, linux-crypto; +Cc: anton, Daniel Axtens
In-Reply-To: <20170303065655.7317-1-dja@axtens.net>
Daniel Axtens <dja@axtens.net> writes:
> Turning on crypto self-tests on a POWER8 shows:
>
> alg: hash: Test 1 failed for crc32c-vpmsum
> 00000000: ff ff ff ff
>
> Comparing the code with the Intel CRC32c implementation on which
> ours is based shows that we are doing an init with 0, not ~0
> as CRC32c requires.
>
> This probably wasn't caught because btrfs does its own weird
> open-coded initialisation.
>
> Initialise our internal context to ~0 on init.
>
> This makes the self-tests pass, and btrfs continues to work.
>
> Fixes: 6dd7a82cc54e ("crypto: powerpc - Add POWER8 optimised crc32c")
> Cc: Anton Blanchard <anton@samba.org>
> Cc: stable@vger.kernel.org
> Signed-off-by: Daniel Axtens <dja@axtens.net>
> ---
> arch/powerpc/crypto/crc32c-vpmsum_glue.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
This driver was originally merged via the crypto tree, so I'll assume
Herbert will pick up the fix. If he hasn't in a few days I'll take it.
cheers
^ permalink raw reply
* Re: crypto: deadlock between crypto_alg_sem/rtnl_mutex/genl_mutex
From: Dmitry Vyukov @ 2017-03-06 9:36 UTC (permalink / raw)
To: Herbert Xu, David Miller, linux-crypto, LKML, Eric Dumazet,
Cong Wang, netdev
Cc: syzkaller
In-Reply-To: <CACT4Y+bJ1-d3DhCvzEN0PeDd7+P2LNgjF2NHwALXBubOKaOEsw@mail.gmail.com>
On Sun, Mar 5, 2017 at 6:36 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Sun, Mar 5, 2017 at 4:08 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> Hello,
>>
>> I am getting the following deadlock reports while running syzkaller
>> fuzzer on net-next/8d70eeb84ab277377c017af6a21d0a337025dede:
>>
>> ======================================================
>> [ INFO: possible circular locking dependency detected ]
>> 4.10.0+ #5 Not tainted
>> -------------------------------------------------------
>> syz-executor6/6143 is trying to acquire lock:
>> (nlk->cb_mutex){+.+.+.}, at: [<ffffffff837df634>]
>> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
>>
>> but task is already holding lock:
>> (crypto_alg_sem){+++++.}, at: [<ffffffff821cd1f6>]
>> crypto_user_rcv_msg+0x136/0x4f0 crypto/crypto_user.c:507
>>
>> which lock already depends on the new lock.
>>
>>
>> the existing dependency chain (in reverse order) is:
>>
>> -> #4 (crypto_alg_sem){+++++.}:
>> validate_chain kernel/locking/lockdep.c:2267 [inline]
>> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
>> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
>> down_read+0x9b/0x150 kernel/locking/rwsem.c:23
>> crypto_alg_lookup+0x23/0x50 crypto/api.c:199
>> crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
>> crypto_larval_lookup crypto/api.c:211 [inline]
>> crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
>> crypto_alloc_base+0x50/0x1e0 crypto/api.c:416
>> crypto_alloc_cipher include/linux/crypto.h:1407 [inline]
>> tcp_fastopen_reset_cipher+0xc2/0x2e0 net/ipv4/tcp_fastopen.c:48
>> tcp_fastopen_init_key_once+0x114/0x120 net/ipv4/tcp_fastopen.c:29
>> do_tcp_setsockopt.isra.36+0x140a/0x20a0 net/ipv4/tcp.c:2684
>> tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
>> sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
>> SYSC_setsockopt net/socket.c:1786 [inline]
>> SyS_setsockopt+0x25c/0x390 net/socket.c:1765
>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>
>> -> #3 (sk_lock-AF_INET){+.+.+.}:
>> validate_chain kernel/locking/lockdep.c:2267 [inline]
>> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
>> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
>> lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
>> lock_sock include/net/sock.h:1460 [inline]
>> rds_tcp_listen_stop+0x57/0x140 net/rds/tcp_listen.c:284
>> rds_tcp_kill_sock net/rds/tcp.c:529 [inline]
>> rds_tcp_dev_event+0x383/0xc50 net/rds/tcp.c:568
>> notifier_call_chain+0x1b5/0x2b0 kernel/notifier.c:93
>> __raw_notifier_call_chain kernel/notifier.c:394 [inline]
>> raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
>> call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1646
>> call_netdevice_notifiers net/core/dev.c:1662 [inline]
>> netdev_run_todo+0x3b2/0xa30 net/core/dev.c:7530
>> rtnl_unlock+0xe/0x10 net/core/rtnetlink.c:104
>> default_device_exit_batch+0x504/0x620 net/core/dev.c:8334
>> ops_exit_list.isra.6+0x100/0x150 net/core/net_namespace.c:144
>> cleanup_net+0x551/0xa90 net/core/net_namespace.c:463
>> process_one_work+0xbd0/0x1c10 kernel/workqueue.c:2096
>> worker_thread+0x223/0x1990 kernel/workqueue.c:2230
>> kthread+0x326/0x3f0 kernel/kthread.c:229
>> ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
>>
>> -> #2 (rtnl_mutex){+.+.+.}:
>> validate_chain kernel/locking/lockdep.c:2267 [inline]
>> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
>> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
>> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
>> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>> rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
>> tipc_nl_bearer_dump+0x3ef/0x720 net/tipc/bearer.c:774
>> genl_lock_dumpit+0x68/0x90 net/netlink/genetlink.c:479
>> netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
>> __netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
>> genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
>> genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
>> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
>> genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
>> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
>> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
>> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
>> sock_sendmsg_nosec net/socket.c:633 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:643
>> sock_write_iter+0x326/0x600 net/socket.c:846
>> call_write_iter include/linux/fs.h:1733 [inline]
>> new_sync_write fs/read_write.c:497 [inline]
>> __vfs_write+0x483/0x740 fs/read_write.c:510
>> vfs_write+0x187/0x530 fs/read_write.c:558
>> SYSC_write fs/read_write.c:605 [inline]
>> SyS_write+0xfb/0x230 fs/read_write.c:597
>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>
>> -> #1 (genl_mutex){+.+.+.}:
>> validate_chain kernel/locking/lockdep.c:2267 [inline]
>> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
>> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
>> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
>> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>> genl_lock net/netlink/genetlink.c:32 [inline]
>> genl_lock_dumpit+0x41/0x90 net/netlink/genetlink.c:478
>> netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
>> __netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
>> genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
>> genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
>> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
>> genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
>> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
>> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
>> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
>> sock_sendmsg_nosec net/socket.c:633 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:643
>> sock_write_iter+0x326/0x600 net/socket.c:846
>> call_write_iter include/linux/fs.h:1733 [inline]
>> new_sync_write fs/read_write.c:497 [inline]
>> __vfs_write+0x483/0x740 fs/read_write.c:510
>> vfs_write+0x187/0x530 fs/read_write.c:558
>> SYSC_write fs/read_write.c:605 [inline]
>> SyS_write+0xfb/0x230 fs/read_write.c:597
>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>
>> -> #0 (nlk->cb_mutex){+.+.+.}:
>> check_prev_add kernel/locking/lockdep.c:1830 [inline]
>> check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
>> validate_chain kernel/locking/lockdep.c:2267 [inline]
>> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
>> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
>> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
>> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
>> netlink_dump_start include/linux/netlink.h:165 [inline]
>> crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
>> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
>> crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
>> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
>> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
>> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
>> sock_sendmsg_nosec net/socket.c:633 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:643
>> ___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
>> __sys_sendmsg+0x138/0x300 net/socket.c:2019
>> SYSC_sendmsg net/socket.c:2030 [inline]
>> SyS_sendmsg+0x2d/0x50 net/socket.c:2026
>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>
>> other info that might help us debug this:
>>
>> Chain exists of:
>> nlk->cb_mutex --> sk_lock-AF_INET --> crypto_alg_sem
>>
>> Possible unsafe locking scenario:
>>
>> CPU0 CPU1
>> ---- ----
>> lock(crypto_alg_sem);
>> lock(sk_lock-AF_INET);
>> lock(crypto_alg_sem);
>> lock(nlk->cb_mutex);
>>
>> *** DEADLOCK ***
>>
>> 2 locks held by syz-executor6/6143:
>> #0: (crypto_cfg_mutex){+.+...}, at: [<ffffffff821cad9b>]
>> crypto_netlink_rcv+0x1b/0x40 crypto/crypto_user.c:537
>> #1: (crypto_alg_sem){+++++.}, at: [<ffffffff821cd1f6>]
>> crypto_user_rcv_msg+0x136/0x4f0 crypto/crypto_user.c:507
>>
>> stack backtrace:
>> CPU: 0 PID: 6143 Comm: syz-executor6 Not tainted 4.10.0+ #5
>> Hardware name: Google Google Compute Engine/Google Compute Engine,
>> BIOS Google 01/01/2011
>> Call Trace:
>> __dump_stack lib/dump_stack.c:16 [inline]
>> dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
>> print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
>> check_prev_add kernel/locking/lockdep.c:1830 [inline]
>> check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
>> validate_chain kernel/locking/lockdep.c:2267 [inline]
>> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
>> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
>> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
>> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
>> netlink_dump_start include/linux/netlink.h:165 [inline]
>> crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
>> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
>> crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
>> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
>> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
>> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
>> sock_sendmsg_nosec net/socket.c:633 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:643
>> ___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
>> __sys_sendmsg+0x138/0x300 net/socket.c:2019
>> SYSC_sendmsg net/socket.c:2030 [inline]
>> SyS_sendmsg+0x2d/0x50 net/socket.c:2026
>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
>
>
> Another one involving tcp_md5sig_mutex:
>
>
> ======================================================
> [ INFO: possible circular locking dependency detected ]
> SELinux: unrecognized netlink message: protocol=9 nlmsg_type=2
> sclass=netlink_audit_socket pig=4033 comm=syz-executor4
> 4.10.0+ #5 Not tainted
> -------------------------------------------------------
> syz-executor8/4018 is trying to acquire lock:
> (crypto_alg_sem){++++++}, at: [<ffffffff82193473>]
> crypto_alg_lookup+0x23/0x50 crypto/api.c:199
>
> but task is already holding lock:
> (tcp_md5sig_mutex){+.+...}, at: [<ffffffff838e4efa>]
> tcp_alloc_md5sig_pool+0x4a/0x470 net/ipv4/tcp.c:3196
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #5 (tcp_md5sig_mutex){+.+...}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> tcp_alloc_md5sig_pool+0x4a/0x470 net/ipv4/tcp.c:3196
> tcp_md5_do_add+0x1d8/0x5d0 net/ipv4/tcp_ipv4.c:969
> tcp_v4_parse_md5_keys+0x1c7/0x2b0 net/ipv4/tcp_ipv4.c:1037
> do_tcp_setsockopt.isra.36+0x657/0x20a0 net/ipv4/tcp.c:2668
> tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
> sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
> SYSC_setsockopt net/socket.c:1786 [inline]
> SyS_setsockopt+0x25c/0x390 net/socket.c:1765
> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
> return_from_SYSCALL_64+0x0/0x7a
>
> -> #4 (sk_lock-AF_INET){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
> lock_sock include/net/sock.h:1460 [inline]
> do_ip_setsockopt.isra.12+0x301/0x3760 net/ipv4/ip_sockglue.c:653
> ip_setsockopt+0x3a/0xb0 net/ipv4/ip_sockglue.c:1265
> tcp_setsockopt+0x82/0xd0 net/ipv4/tcp.c:2731
> sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
> SYSC_setsockopt net/socket.c:1786 [inline]
> SyS_setsockopt+0x25c/0x390 net/socket.c:1765
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #3 (rtnl_mutex){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
> nl80211_prepare_wdev_dump.isra.37+0x2c/0x5d0 net/wireless/nl80211.c:548
> nl80211_dump_station+0x178/0xd80 net/wireless/nl80211.c:4455
> genl_lock_dumpit+0x68/0x90 net/netlink/genetlink.c:479
> netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
> __netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
> genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
> genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> sock_write_iter+0x326/0x600 net/socket.c:846
> call_write_iter include/linux/fs.h:1733 [inline]
> new_sync_write fs/read_write.c:497 [inline]
> __vfs_write+0x483/0x740 fs/read_write.c:510
> vfs_write+0x187/0x530 fs/read_write.c:558
> SYSC_write fs/read_write.c:605 [inline]
> SyS_write+0xfb/0x230 fs/read_write.c:597
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #2 (genl_mutex){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> genl_lock net/netlink/genetlink.c:32 [inline]
> genl_lock_dumpit+0x41/0x90 net/netlink/genetlink.c:478
> netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
> __netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
> genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
> genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> sock_write_iter+0x326/0x600 net/socket.c:846
> call_write_iter include/linux/fs.h:1733 [inline]
> new_sync_write fs/read_write.c:497 [inline]
> __vfs_write+0x483/0x740 fs/read_write.c:510
> vfs_write+0x187/0x530 fs/read_write.c:558
> SYSC_write fs/read_write.c:605 [inline]
> SyS_write+0xfb/0x230 fs/read_write.c:597
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #1 (nlk->cb_mutex){+.+.+.}:
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> __netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
> netlink_dump_start include/linux/netlink.h:165 [inline]
> crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
> netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
> crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
> netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
> netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
> netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
> sock_sendmsg_nosec net/socket.c:633 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:643
> ___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
> __sys_sendmsg+0x138/0x300 net/socket.c:2019
> SYSC_sendmsg net/socket.c:2030 [inline]
> SyS_sendmsg+0x2d/0x50 net/socket.c:2026
> entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> -> #0 (crypto_alg_sem){++++++}:
> check_prev_add kernel/locking/lockdep.c:1830 [inline]
> check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> down_read+0x9b/0x150 kernel/locking/rwsem.c:23
> crypto_alg_lookup+0x23/0x50 crypto/api.c:199
> crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
> crypto_larval_lookup crypto/api.c:211 [inline]
> crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
> crypto_find_alg crypto/api.c:500 [inline]
> crypto_alloc_tfm+0x101/0x2e0 crypto/api.c:533
> crypto_alloc_ahash+0x2c/0x40 crypto/ahash.c:525
> __tcp_alloc_md5sig_pool net/ipv4/tcp.c:3158 [inline]
> tcp_alloc_md5sig_pool+0x85/0x470 net/ipv4/tcp.c:3199
> tcp_md5_do_add+0x1d8/0x5d0 net/ipv4/tcp_ipv4.c:969
> tcp_v4_parse_md5_keys+0x1c7/0x2b0 net/ipv4/tcp_ipv4.c:1037
> do_tcp_setsockopt.isra.36+0x657/0x20a0 net/ipv4/tcp.c:2668
> tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
> sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
> SYSC_setsockopt net/socket.c:1786 [inline]
> SyS_setsockopt+0x25c/0x390 net/socket.c:1765
> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
> return_from_SYSCALL_64+0x0/0x7a
>
> other info that might help us debug this:
>
> Chain exists of:
> crypto_alg_sem --> sk_lock-AF_INET --> tcp_md5sig_mutex
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(tcp_md5sig_mutex);
> lock(sk_lock-AF_INET);
> lock(tcp_md5sig_mutex);
> lock(crypto_alg_sem);
>
> *** DEADLOCK ***
>
> 2 locks held by syz-executor8/4018:
> #0: (sk_lock-AF_INET){+.+.+.}, at: [<ffffffff838e7a26>] lock_sock
> include/net/sock.h:1460 [inline]
> #0: (sk_lock-AF_INET){+.+.+.}, at: [<ffffffff838e7a26>]
> do_tcp_setsockopt.isra.36+0x216/0x20a0 net/ipv4/tcp.c:2466
> #1: (tcp_md5sig_mutex){+.+...}, at: [<ffffffff838e4efa>]
> tcp_alloc_md5sig_pool+0x4a/0x470 net/ipv4/tcp.c:3196
>
> stack backtrace:
> CPU: 0 PID: 4018 Comm: syz-executor8 Not tainted 4.10.0+ #5
> Hardware name: Google Google Compute Engine/Google Compute Engine,
> BIOS Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:16 [inline]
> dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
> print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
> check_prev_add kernel/locking/lockdep.c:1830 [inline]
> check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
> validate_chain kernel/locking/lockdep.c:2267 [inline]
> __lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
> lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
> down_read+0x9b/0x150 kernel/locking/rwsem.c:23
> crypto_alg_lookup+0x23/0x50 crypto/api.c:199
> crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
> crypto_larval_lookup crypto/api.c:211 [inline]
> crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
> crypto_find_alg crypto/api.c:500 [inline]
> crypto_alloc_tfm+0x101/0x2e0 crypto/api.c:533
> crypto_alloc_ahash+0x2c/0x40 crypto/ahash.c:525
> __tcp_alloc_md5sig_pool net/ipv4/tcp.c:3158 [inline]
> tcp_alloc_md5sig_pool+0x85/0x470 net/ipv4/tcp.c:3199
> tcp_md5_do_add+0x1d8/0x5d0 net/ipv4/tcp_ipv4.c:969
> tcp_v4_parse_md5_keys+0x1c7/0x2b0 net/ipv4/tcp_ipv4.c:1037
> do_tcp_setsockopt.isra.36+0x657/0x20a0 net/ipv4/tcp.c:2668
> tcp_setsockopt+0xb0/0xd0 net/ipv4/tcp.c:2733
> sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
> SYSC_setsockopt net/socket.c:1786 [inline]
> SyS_setsockopt+0x25c/0x390 net/socket.c:1765
> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
Another one:
======================================================
[ INFO: possible circular locking dependency detected ]
4.10.0+ #6 Not tainted
-------------------------------------------------------
syz-executor8/3613 is trying to acquire lock:
(sk_lock-AF_INET6){+.+.+.}, at: [<ffffffff83b72eb9>] lock_sock
include/net/sock.h:1460 [inline]
(sk_lock-AF_INET6){+.+.+.}, at: [<ffffffff83b72eb9>]
do_ipv6_setsockopt.isra.11+0x229/0x36e0 net/ipv6/ipv6_sockglue.c:167
but task is already holding lock:
(rtnl_mutex){+.+.+.}, at: [<ffffffff8370a197>] rtnl_lock+0x17/0x20
net/core/rtnetlink.c:70
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #4 (rtnl_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
tipc_nl_node_dump_monitor+0x260/0x510 net/tipc/node.c:2128
genl_lock_dumpit+0x68/0x90 net/netlink/genetlink.c:479
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #3 (genl_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
genl_lock net/netlink/genetlink.c:32 [inline]
genl_lock_dumpit+0x41/0x90 net/netlink/genetlink.c:478
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
genl_family_rcv_msg+0xd9d/0x1040 net/netlink/genetlink.c:546
genl_rcv_msg+0xa6/0x140 net/netlink/genetlink.c:620
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
genl_rcv+0x28/0x40 net/netlink/genetlink.c:631
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #2 (nlk->cb_mutex){+.+.+.}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x172/0x1730 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
__netlink_dump_start+0xf4/0x760 net/netlink/af_netlink.c:2187
netlink_dump_start include/linux/netlink.h:165 [inline]
crypto_user_rcv_msg+0x2ad/0x4f0 crypto/crypto_user.c:517
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
crypto_netlink_rcv+0x2a/0x40 crypto/crypto_user.c:538
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x8fa/0x9f0 net/socket.c:1985
__sys_sendmsg+0x138/0x300 net/socket.c:2019
SYSC_sendmsg net/socket.c:2030 [inline]
SyS_sendmsg+0x2d/0x50 net/socket.c:2026
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #1 (crypto_alg_sem){++++++}:
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
down_read+0x9b/0x150 kernel/locking/rwsem.c:23
crypto_alg_lookup+0x23/0x50 crypto/api.c:199
crypto_larval_lookup.part.10+0x9a/0x3b0 crypto/api.c:217
crypto_larval_lookup crypto/api.c:211 [inline]
crypto_alg_mod_lookup+0x77/0x1b0 crypto/api.c:270
crypto_find_alg crypto/api.c:500 [inline]
crypto_alloc_tfm+0x101/0x2e0 crypto/api.c:533
crypto_alloc_shash+0x2c/0x40 crypto/shash.c:433
sctp_listen_start net/sctp/socket.c:6969 [inline]
sctp_inet_listen+0x5b7/0x7e0 net/sctp/socket.c:7054
SYSC_listen net/socket.c:1440 [inline]
SyS_listen+0x2c9/0x390 net/socket.c:1426
entry_SYSCALL_64_fastpath+0x1f/0xc2
-> #0 (sk_lock-AF_INET6){+.+.+.}:
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
lock_sock include/net/sock.h:1460 [inline]
do_ipv6_setsockopt.isra.11+0x229/0x36e0 net/ipv6/ipv6_sockglue.c:167
ipv6_setsockopt+0x9b/0x140 net/ipv6/ipv6_sockglue.c:919
tcp_setsockopt+0x82/0xd0 net/ipv4/tcp.c:2731
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
entry_SYSCALL_64_fastpath+0x1f/0xc2
other info that might help us debug this:
Chain exists of:
sk_lock-AF_INET6 --> genl_mutex --> rtnl_mutex
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(rtnl_mutex);
lock(genl_mutex);
lock(rtnl_mutex);
lock(sk_lock-AF_INET6);
*** DEADLOCK ***
1 lock held by syz-executor8/3613:
#0: (rtnl_mutex){+.+.+.}, at: [<ffffffff8370a197>]
rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70
stack backtrace:
CPU: 1 PID: 3613 Comm: syz-executor8 Not tainted 4.10.0+ #6
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
print_circular_bug+0x307/0x3b0 kernel/locking/lockdep.c:1204
check_prev_add kernel/locking/lockdep.c:1830 [inline]
check_prevs_add+0xa8f/0x19f0 kernel/locking/lockdep.c:1940
validate_chain kernel/locking/lockdep.c:2267 [inline]
__lock_acquire+0x2149/0x3430 kernel/locking/lockdep.c:3340
lock_acquire+0x2a1/0x630 kernel/locking/lockdep.c:3755
lock_sock_nested+0xcb/0x120 net/core/sock.c:2536
lock_sock include/net/sock.h:1460 [inline]
do_ipv6_setsockopt.isra.11+0x229/0x36e0 net/ipv6/ipv6_sockglue.c:167
ipv6_setsockopt+0x9b/0x140 net/ipv6/ipv6_sockglue.c:919
tcp_setsockopt+0x82/0xd0 net/ipv4/tcp.c:2731
sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2731
SYSC_setsockopt net/socket.c:1786 [inline]
SyS_setsockopt+0x25c/0x390 net/socket.c:1765
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox