* [RFC PATCH 0/3] Cavium ThunderX ZIP driver
From: Jan Glauber @ 2016-12-12 15:04 UTC (permalink / raw)
To: Herbert Xu
Cc: linux-crypto, linux-kernel, David S . Miller, Mahipal Challa,
Vishnu Nair, Jan Glauber
Hi Herbert,
this series adds support for hardware accelerated compression & decompression
as found on ThunderX (arm64) SOCs. I've been reviewing this driver internally
for some time and would like to get feedback on the RFC to see if this goes
into the right direction and to see if there are any concerns.
We've discussed switching to the new acomp algorithm but for the time being
decided against acomp because our test cases are not yet supported with it.
To test the ZIP driver we've used ZSWAP and IPComp.
Performance numbers from ZSWAP look promising.
The "average time" for compressing a 4KB page:
Compression Software : 128 usec
Compression HW deflate : 16 usec
Compression HW LZS : 10 usec
Decompression Software : 20 usec
Decompression HW deflate: 7 usec
Decompression HW LZS : 5 usec
Patches are on top of 4.9.
Feedback welcome!
Jan
---------------------
Mahipal Challa (3):
crypto: zip - Add ThunderX ZIP driver core
crypto: zip - Wire-up Compression / decompression HW offload
crypto: zip - Add Compression/decompression statistics
drivers/crypto/Kconfig | 7 +
drivers/crypto/Makefile | 1 +
drivers/crypto/cavium/Makefile | 4 +
drivers/crypto/cavium/zip/Makefile | 11 +
drivers/crypto/cavium/zip/common.h | 258 ++++++
drivers/crypto/cavium/zip/zip_crypto.c | 243 ++++++
drivers/crypto/cavium/zip/zip_crypto.h | 67 ++
drivers/crypto/cavium/zip/zip_deflate.c | 200 +++++
drivers/crypto/cavium/zip/zip_deflate.h | 62 ++
drivers/crypto/cavium/zip/zip_device.c | 209 +++++
drivers/crypto/cavium/zip/zip_device.h | 138 ++++
drivers/crypto/cavium/zip/zip_inflate.c | 223 ++++++
drivers/crypto/cavium/zip/zip_inflate.h | 62 ++
drivers/crypto/cavium/zip/zip_main.c | 698 ++++++++++++++++
drivers/crypto/cavium/zip/zip_main.h | 141 ++++
drivers/crypto/cavium/zip/zip_mem.c | 120 +++
drivers/crypto/cavium/zip/zip_mem.h | 78 ++
drivers/crypto/cavium/zip/zip_regs.h | 1326 +++++++++++++++++++++++++++++++
18 files changed, 3848 insertions(+)
create mode 100644 drivers/crypto/cavium/Makefile
create mode 100644 drivers/crypto/cavium/zip/Makefile
create mode 100644 drivers/crypto/cavium/zip/common.h
create mode 100644 drivers/crypto/cavium/zip/zip_crypto.c
create mode 100644 drivers/crypto/cavium/zip/zip_crypto.h
create mode 100644 drivers/crypto/cavium/zip/zip_deflate.c
create mode 100644 drivers/crypto/cavium/zip/zip_deflate.h
create mode 100644 drivers/crypto/cavium/zip/zip_device.c
create mode 100644 drivers/crypto/cavium/zip/zip_device.h
create mode 100644 drivers/crypto/cavium/zip/zip_inflate.c
create mode 100644 drivers/crypto/cavium/zip/zip_inflate.h
create mode 100644 drivers/crypto/cavium/zip/zip_main.c
create mode 100644 drivers/crypto/cavium/zip/zip_main.h
create mode 100644 drivers/crypto/cavium/zip/zip_mem.c
create mode 100644 drivers/crypto/cavium/zip/zip_mem.h
create mode 100644 drivers/crypto/cavium/zip/zip_regs.h
--
2.9.0.rc0.21.g7777322
^ permalink raw reply
* [RFC PATCH 3/3] crypto: zip - Add Compression/decompression statistics
From: Jan Glauber @ 2016-12-12 15:04 UTC (permalink / raw)
To: Herbert Xu
Cc: linux-crypto, linux-kernel, David S . Miller, Mahipal Challa,
Vishnu Nair, Jan Glauber
In-Reply-To: <20161212150439.18627-1-jglauber@cavium.com>
From: Mahipal Challa <Mahipal.Challa@cavium.com>
Add statistics for compression/decompression hardware offload
under debugfs.
Signed-off-by: Mahipal Challa <Mahipal.Challa@cavium.com>
Signed-off-by: Vishnu Nair <Vishnu.Nair@cavium.com>
Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
drivers/crypto/cavium/zip/zip_deflate.c | 10 ++
drivers/crypto/cavium/zip/zip_inflate.c | 12 ++
drivers/crypto/cavium/zip/zip_main.c | 227 ++++++++++++++++++++++++++++++++
drivers/crypto/cavium/zip/zip_main.h | 15 +++
4 files changed, 264 insertions(+)
diff --git a/drivers/crypto/cavium/zip/zip_deflate.c b/drivers/crypto/cavium/zip/zip_deflate.c
index 913cc25..11052d8 100644
--- a/drivers/crypto/cavium/zip/zip_deflate.c
+++ b/drivers/crypto/cavium/zip/zip_deflate.c
@@ -122,12 +122,19 @@ int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s,
/* Prepares zip command based on the input parameters */
prepare_zip_command(zip_ops, s, zip_cmd);
+ atomic64_add(zip_ops->input_len, &zip_dev->stats.comp_in_bytes);
/* Loads zip command into command queues and rings door bell */
queue = zip_load_instr(zip_cmd, zip_dev);
+ /* Stats update for compression requests submitted */
+ atomic64_inc(&zip_dev->stats.comp_req_submit);
+
while (!result_ptr->s.compcode)
continue;
+ /* Stats update for compression requests completed */
+ atomic64_inc(&zip_dev->stats.comp_req_complete);
+
zip_ops->compcode = result_ptr->s.compcode;
switch (zip_ops->compcode) {
case ZIP_NOTDONE:
@@ -175,6 +182,9 @@ int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s,
zip_err("Unknown Format:%d\n", zip_ops->format);
}
+ atomic64_add(result_ptr->s.totalbyteswritten,
+ &zip_dev->stats.comp_out_bytes);
+
/* Update output_len */
if (zip_ops->output_len < result_ptr->s.totalbyteswritten) {
/* Dynamic stop && strm->output_len < zipconstants[onfsize] */
diff --git a/drivers/crypto/cavium/zip/zip_inflate.c b/drivers/crypto/cavium/zip/zip_inflate.c
index 849c4c85..44503d8 100644
--- a/drivers/crypto/cavium/zip/zip_inflate.c
+++ b/drivers/crypto/cavium/zip/zip_inflate.c
@@ -135,12 +135,20 @@ int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
/* Prepare inflate zip command */
prepare_inflate_zcmd(zip_ops, s, zip_cmd);
+ atomic64_add(zip_ops->input_len, &zip_dev->stats.decomp_in_bytes);
+
/* Load inflate command to zip queue and ring the doorbell */
queue = zip_load_instr(zip_cmd, zip_dev);
+ /* Decompression requests submitted stats update */
+ atomic64_inc(&zip_dev->stats.decomp_req_submit);
+
while (!result_ptr->s.compcode)
continue;
+ /* Decompression requests completed stats update */
+ atomic64_inc(&zip_dev->stats.decomp_req_complete);
+
zip_ops->compcode = result_ptr->s.compcode;
switch (zip_ops->compcode) {
case ZIP_NOTDONE:
@@ -157,6 +165,7 @@ int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
default:
zip_dbg("Instruction failed. Code = %d\n", zip_ops->compcode);
+ atomic64_inc(&zip_dev->stats.decomp_bad_reqs);
zip_update_cmd_bufs(zip_dev, queue);
return ZIP_ERROR;
}
@@ -169,6 +178,9 @@ int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
zip_ops->csum = result_ptr->s.adler32;
+ atomic64_add(result_ptr->s.totalbyteswritten,
+ &zip_dev->stats.decomp_out_bytes);
+
if (zip_ops->output_len < result_ptr->s.totalbyteswritten) {
zip_err("output_len (%d) < total bytes written (%d)\n",
zip_ops->output_len, result_ptr->s.totalbyteswritten);
diff --git a/drivers/crypto/cavium/zip/zip_main.c b/drivers/crypto/cavium/zip/zip_main.c
index ae3395f..56631bf 100644
--- a/drivers/crypto/cavium/zip/zip_main.c
+++ b/drivers/crypto/cavium/zip/zip_main.c
@@ -427,6 +427,228 @@ static void zip_unregister_compression_device(void)
crypto_unregister_alg(&zip_comp_lzs);
}
+/*
+ * debugfs functions
+ */
+#ifdef CONFIG_DEBUG_FS
+#include <linux/debugfs.h>
+
+/* Displays ZIP device statistics */
+static int zip_show_stats(struct seq_file *s, void *unused)
+{
+ u64 val = 0ull;
+ u64 avg_chunk = 0ull, avg_cr = 0ull;
+ u32 q = 0;
+
+ int index = 0;
+ struct zip_device *zip;
+ struct zip_stats *st;
+
+ for (index = 0; index < MAX_ZIP_DEVICES; index++) {
+ if (zip_dev[index]) {
+ zip = zip_dev[index];
+ st = &zip->stats;
+
+ /* Get all the pending requests */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ val = zip_reg_read((zip->reg_base +
+ ZIP_DBG_COREX_STA(q)));
+ val = (val >> 32);
+ val = val & 0xffffff;
+ atomic64_add(val, &st->pending_req);
+ }
+
+ avg_chunk = (atomic64_read(&st->comp_in_bytes) /
+ atomic64_read(&st->comp_req_complete));
+ avg_cr = (atomic64_read(&st->comp_in_bytes) /
+ atomic64_read(&st->comp_out_bytes));
+ seq_printf(s, " ZIP Device %d Stats\n"
+ "-----------------------------------\n"
+ "Comp Req Submitted : \t%ld\n"
+ "Comp Req Completed : \t%ld\n"
+ "Compress In Bytes : \t%ld\n"
+ "Compressed Out Bytes : \t%ld\n"
+ "Average Chunk size : \t%llu\n"
+ "Average Compression ratio : \t%llu\n"
+ "Decomp Req Submitted : \t%ld\n"
+ "Decomp Req Completed : \t%ld\n"
+ "Decompress In Bytes : \t%ld\n"
+ "Decompressed Out Bytes : \t%ld\n"
+ "Decompress Bad requests : \t%ld\n"
+ "Pending Req : \t%ld\n"
+ "---------------------------------\n",
+ index,
+ atomic64_read(&st->comp_req_submit),
+ atomic64_read(&st->comp_req_complete),
+ atomic64_read(&st->comp_in_bytes),
+ atomic64_read(&st->comp_out_bytes),
+ avg_chunk,
+ avg_cr,
+ atomic64_read(&st->decomp_req_submit),
+ atomic64_read(&st->decomp_req_complete),
+ atomic64_read(&st->decomp_in_bytes),
+ atomic64_read(&st->decomp_out_bytes),
+ atomic64_read(&st->decomp_bad_reqs),
+ atomic64_read(&st->pending_req));
+
+ /* Reset pending requests count */
+ atomic64_set(&st->pending_req, 0);
+ }
+ }
+ return 0;
+}
+
+/* Clears stats data */
+static int zip_clear_stats(struct seq_file *s, void *unused)
+{
+ int index = 0;
+
+ for (index = 0; index < MAX_ZIP_DEVICES; index++) {
+ if (zip_dev[index]) {
+ memset(&zip_dev[index]->stats, 0,
+ sizeof(struct zip_state));
+ seq_printf(s, "Cleared stats for zip %d\n", index);
+ }
+ }
+
+ return 0;
+}
+
+static struct zip_registers zipregs[64] = {
+ {"ZIP_CMD_CTL ", 0x0000ull},
+ {"ZIP_THROTTLE ", 0x0010ull},
+ {"ZIP_CONSTANTS ", 0x00A0ull},
+ {"ZIP_QUE0_MAP ", 0x1400ull},
+ {"ZIP_QUE1_MAP ", 0x1408ull},
+ {"ZIP_QUE_ENA ", 0x0500ull},
+ {"ZIP_QUE_PRI ", 0x0508ull},
+ {"ZIP_QUE0_DONE ", 0x2000ull},
+ {"ZIP_QUE1_DONE ", 0x2008ull},
+ {"ZIP_QUE0_DOORBELL ", 0x4000ull},
+ {"ZIP_QUE1_DOORBELL ", 0x4008ull},
+ {"ZIP_QUE0_SBUF_ADDR ", 0x1000ull},
+ {"ZIP_QUE1_SBUF_ADDR ", 0x1008ull},
+ {"ZIP_QUE0_SBUF_CTL ", 0x1200ull},
+ {"ZIP_QUE1_SBUF_CTL ", 0x1208ull},
+ { NULL, 0}
+};
+
+/* Prints registers' contents */
+static int zip_print_regs(struct seq_file *s, void *unused)
+{
+ u64 val = 0;
+ int i = 0, index = 0;
+
+ for (index = 0; index < MAX_ZIP_DEVICES; index++) {
+ if (zip_dev[index]) {
+ seq_printf(s, "--------------------------------\n"
+ " ZIP Device %d Registers\n"
+ "--------------------------------\n",
+ index);
+
+ i = 0;
+
+ while (zipregs[i].reg_name) {
+ val = zip_reg_read((zip_dev[index]->reg_base +
+ zipregs[i].reg_offset));
+ seq_printf(s, "%s: 0x%016llx\n",
+ zipregs[i].reg_name, val);
+ i++;
+ }
+ }
+ }
+ return 0;
+}
+
+static int zip_stats_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, zip_show_stats, NULL);
+}
+
+static const struct file_operations zip_stats_fops = {
+ .owner = THIS_MODULE,
+ .open = zip_stats_open,
+ .read = seq_read,
+};
+
+static int zip_clear_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, zip_clear_stats, NULL);
+}
+
+static const struct file_operations zip_clear_fops = {
+ .owner = THIS_MODULE,
+ .open = zip_clear_open,
+ .read = seq_read,
+};
+
+static int zip_regs_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, zip_print_regs, NULL);
+}
+
+static const struct file_operations zip_regs_fops = {
+ .owner = THIS_MODULE,
+ .open = zip_regs_open,
+ .read = seq_read,
+};
+
+/* Root directory for thunderx_zip debugfs entry */
+static struct dentry *zip_debugfs_root;
+
+static int __init zip_debugfs_init(void)
+{
+ struct dentry *zip_stats, *zip_clear, *zip_regs;
+
+ if (!debugfs_initialized())
+ return -ENODEV;
+
+ zip_debugfs_root = debugfs_create_dir("thunderx_zip", NULL);
+ if (!zip_debugfs_root)
+ return -ENOMEM;
+
+ /* Creating files for entries inside thunderx_zip directory */
+ zip_stats = debugfs_create_file("zip_stats", S_IRUGO,
+ zip_debugfs_root,
+ NULL, &zip_stats_fops);
+ if (!zip_stats)
+ goto failed_to_create;
+
+ zip_clear = debugfs_create_file("zip_clear", S_IRUGO,
+ zip_debugfs_root,
+ NULL, &zip_clear_fops);
+ if (!zip_clear)
+ goto failed_to_create;
+
+ zip_regs = debugfs_create_file("zip_regs", S_IRUGO,
+ zip_debugfs_root,
+ NULL, &zip_regs_fops);
+ if (!zip_regs)
+ goto failed_to_create;
+
+ return 0;
+
+failed_to_create:
+ debugfs_remove_recursive(zip_debugfs_root);
+ return -ENOENT;
+}
+
+static void __exit zip_debugfs_exit(void)
+{
+ debugfs_remove_recursive(zip_debugfs_root);
+}
+
+#else
+static int __init zip_debugfs_init(void)
+{
+ return 0;
+}
+
+static void __exit zip_debugfs_exit(void) { }
+
+#endif
+/* debugfs - end */
+
static int __init zip_init_module(void)
{
int ret;
@@ -448,11 +670,16 @@ static int __init zip_init_module(void)
return 1;
}
+ if (zip_debugfs_init())
+ zip_msg("debugfs initialization failed\n");
+
return ret;
}
static void __exit zip_cleanup_module(void)
{
+ zip_debugfs_exit();
+
/* Unregister this driver for pci zip devices */
pci_unregister_driver(&zip_driver);
diff --git a/drivers/crypto/cavium/zip/zip_main.h b/drivers/crypto/cavium/zip/zip_main.h
index 73b9e6d..cd7963e 100644
--- a/drivers/crypto/cavium/zip/zip_main.h
+++ b/drivers/crypto/cavium/zip/zip_main.h
@@ -87,6 +87,20 @@ struct zip_registers {
u64 reg_offset;
};
+/* ZIP Compression - Decompression stats */
+struct zip_stats {
+ atomic64_t comp_req_submit;
+ atomic64_t comp_req_complete;
+ atomic64_t decomp_req_submit;
+ atomic64_t decomp_req_complete;
+ atomic64_t pending_req;
+ atomic64_t comp_in_bytes;
+ atomic64_t comp_out_bytes;
+ atomic64_t decomp_in_bytes;
+ atomic64_t decomp_out_bytes;
+ atomic64_t decomp_bad_reqs;
+};
+
/* ZIP Instruction Queue */
struct zip_iq {
u64 *sw_head;
@@ -112,6 +126,7 @@ struct zip_device {
u64 ctxsize;
struct zip_iq iq[ZIP_MAX_NUM_QUEUES];
+ struct zip_stats stats;
};
/* Prototypes */
--
2.9.0.rc0.21.g7777322
^ permalink raw reply related
* [RFC PATCH 2/3] crypto: zip - Wire-up Compression / decompression HW offload
From: Jan Glauber @ 2016-12-12 15:04 UTC (permalink / raw)
To: Herbert Xu
Cc: linux-crypto, linux-kernel, David S . Miller, Mahipal Challa,
Vishnu Nair, Jan Glauber
In-Reply-To: <20161212150439.18627-1-jglauber@cavium.com>
From: Mahipal Challa <Mahipal.Challa@cavium.com>
This contains changes for adding compression/decompression h/w offload
functionality for both DEFLATE and LZS.
Signed-off-by: Mahipal Challa <Mahipal.Challa@cavium.com>
Signed-off-by: Vishnu Nair <Vishnu.Nair@cavium.com>
Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
drivers/crypto/cavium/zip/Makefile | 5 +-
drivers/crypto/cavium/zip/zip_crypto.c | 243 ++++++++++++++++++++++++++++++++
drivers/crypto/cavium/zip/zip_crypto.h | 6 +
drivers/crypto/cavium/zip/zip_deflate.c | 190 +++++++++++++++++++++++++
drivers/crypto/cavium/zip/zip_deflate.h | 62 ++++++++
drivers/crypto/cavium/zip/zip_device.c | 1 +
drivers/crypto/cavium/zip/zip_inflate.c | 211 +++++++++++++++++++++++++++
drivers/crypto/cavium/zip/zip_inflate.h | 62 ++++++++
drivers/crypto/cavium/zip/zip_main.c | 29 ----
9 files changed, 779 insertions(+), 30 deletions(-)
create mode 100644 drivers/crypto/cavium/zip/zip_crypto.c
create mode 100644 drivers/crypto/cavium/zip/zip_deflate.c
create mode 100644 drivers/crypto/cavium/zip/zip_deflate.h
create mode 100644 drivers/crypto/cavium/zip/zip_inflate.c
create mode 100644 drivers/crypto/cavium/zip/zip_inflate.h
diff --git a/drivers/crypto/cavium/zip/Makefile b/drivers/crypto/cavium/zip/Makefile
index 2c07508..b2f3baaf 100644
--- a/drivers/crypto/cavium/zip/Makefile
+++ b/drivers/crypto/cavium/zip/Makefile
@@ -5,4 +5,7 @@
obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += thunderx_zip.o
thunderx_zip-y := zip_main.o \
zip_device.o \
- zip_mem.o
+ zip_crypto.o \
+ zip_mem.o \
+ zip_deflate.o \
+ zip_inflate.o
diff --git a/drivers/crypto/cavium/zip/zip_crypto.c b/drivers/crypto/cavium/zip/zip_crypto.c
new file mode 100644
index 0000000..888e18b
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_crypto.c
@@ -0,0 +1,243 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include "zip_crypto.h"
+
+static void zip_static_init_zip_ops(struct zip_operation *zip_ops,
+ int lzs_flag)
+{
+ zip_ops->flush = ZIP_FLUSH_FINISH;
+
+ /* equivalent to level 6 of opensource zlib */
+ zip_ops->speed = 1;
+
+ if (!lzs_flag) {
+ zip_ops->ccode = 0; /* Auto Huffman */
+ zip_ops->lzs_flag = 0;
+ zip_ops->format = ZLIB_FORMAT;
+ } else {
+ zip_ops->ccode = 3; /* LZS Encoding */
+ zip_ops->lzs_flag = 1;
+ zip_ops->format = LZS_FORMAT;
+ }
+ zip_ops->begin_file = 1;
+ zip_ops->history_len = 0;
+ zip_ops->end_file = 1;
+ zip_ops->compcode = 0;
+ zip_ops->csum = 1; /* Adler checksum desired */
+}
+
+/* Legacy Compress framework start */
+
+int zip_alloc_zip_ctx(struct crypto_tfm *tfm)
+{
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+ struct zip_operation *comp_ctx = &zip_ctx->zip_comp;
+ struct zip_operation *decomp_ctx = &zip_ctx->zip_decomp;
+
+ zip_static_init_zip_ops(comp_ctx, 0);
+ zip_static_init_zip_ops(decomp_ctx, 0);
+
+ comp_ctx->input = zip_data_buf_alloc(MAX_INPUT_BUFFER_SIZE);
+ if (!comp_ctx->input)
+ return -ENOMEM;
+
+ comp_ctx->output = zip_data_buf_alloc(MAX_OUTPUT_BUFFER_SIZE);
+ if (!comp_ctx->output)
+ goto err_comp_input;
+
+ decomp_ctx->input = zip_data_buf_alloc(MAX_INPUT_BUFFER_SIZE);
+ if (!decomp_ctx->input)
+ goto err_comp_output;
+
+ decomp_ctx->output = zip_data_buf_alloc(MAX_OUTPUT_BUFFER_SIZE);
+ if (!decomp_ctx->output)
+ goto err_decomp_input;
+
+ return 0;
+
+err_decomp_input:
+ zip_data_buf_free(decomp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+
+err_comp_output:
+ zip_data_buf_free(comp_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+
+err_comp_input:
+ zip_data_buf_free(comp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+
+ return -ENOMEM;
+}
+
+int zip_alloc_lzs_ctx(struct crypto_tfm *tfm)
+{
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+ struct zip_operation *comp_ctx = &zip_ctx->zip_comp;
+ struct zip_operation *decomp_ctx = &zip_ctx->zip_decomp;
+
+ zip_static_init_zip_ops(comp_ctx, 1);
+ zip_static_init_zip_ops(decomp_ctx, 1);
+
+ comp_ctx->input = zip_data_buf_alloc(MAX_INPUT_BUFFER_SIZE);
+ if (!comp_ctx->input)
+ return -ENOMEM;
+
+ comp_ctx->output = zip_data_buf_alloc(MAX_OUTPUT_BUFFER_SIZE);
+ if (!comp_ctx->output)
+ goto err_comp_input;
+
+ decomp_ctx->input = zip_data_buf_alloc(MAX_INPUT_BUFFER_SIZE);
+ if (!decomp_ctx->input)
+ goto err_comp_output;
+
+ decomp_ctx->output = zip_data_buf_alloc(MAX_OUTPUT_BUFFER_SIZE);
+ if (!decomp_ctx->output)
+ goto err_decomp_input;
+
+ return 0;
+
+err_decomp_input:
+ zip_data_buf_free(decomp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+
+err_comp_output:
+ zip_data_buf_free(comp_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+
+err_comp_input:
+ zip_data_buf_free(comp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+
+ return -ENOMEM;
+}
+
+void zip_free_zip_ctx(struct crypto_tfm *tfm)
+{
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+ struct zip_operation *comp_ctx = &zip_ctx->zip_comp;
+ struct zip_operation *dec_ctx = &zip_ctx->zip_decomp;
+
+ zip_data_buf_free(comp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+ zip_data_buf_free(comp_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+
+ zip_data_buf_free(dec_ctx->input, MAX_INPUT_BUFFER_SIZE);
+ zip_data_buf_free(dec_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+}
+
+int zip_deflate_comp(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen)
+{
+ struct zip_kernel_ctx *zip_ctx = NULL;
+ struct zip_operation *zip_ops = NULL;
+ struct zip_state zip_state;
+ struct zip_device *zip = NULL;
+ int ret;
+
+ if (!tfm || !src || !dst || !dlen)
+ return -ENOMEM;
+
+ zip = zip_get_device(zip_get_node_id());
+ if (!zip)
+ return -ENODEV;
+
+ memset(&zip_state, 0, sizeof(struct zip_state));
+
+ zip_ctx = crypto_tfm_ctx(tfm);
+ zip_ops = &zip_ctx->zip_comp;
+
+ zip_ops->input_len = slen;
+ zip_ops->output_len = *dlen;
+
+ memcpy(zip_ops->input, src, slen);
+
+ ret = zip_deflate(zip_ops, &zip_state, zip);
+
+ if (!ret) {
+ *dlen = zip_ops->output_len;
+ memcpy(dst, zip_ops->output, *dlen);
+ }
+
+ return ret;
+}
+
+int zip_inflate_comp(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen)
+{
+ struct zip_kernel_ctx *zip_ctx = NULL;
+ struct zip_operation *zip_ops = NULL;
+ struct zip_state zip_state;
+ struct zip_device *zip = NULL;
+ int ret;
+
+ if (!tfm || !src || !dst || !dlen)
+ return -ENOMEM;
+
+ zip = zip_get_device(zip_get_node_id());
+ if (!zip)
+ return -ENODEV;
+
+ memset(&zip_state, 0, sizeof(struct zip_state));
+
+ zip_ctx = crypto_tfm_ctx(tfm);
+ zip_ops = &zip_ctx->zip_decomp;
+
+ memcpy(zip_ops->input, src, slen);
+
+ /* Work around for a bug in zlib which needs an extra bytes sometimes */
+ if (zip_ops->ccode != 3) /* Not LZS Encoding */
+ zip_ops->input[slen++] = 0;
+
+ zip_ops->input_len = slen;
+ zip_ops->output_len = *dlen;
+
+ ret = zip_inflate(zip_ops, &zip_state, zip);
+
+ if (!ret) {
+ *dlen = zip_ops->output_len;
+ memcpy(dst, zip_ops->output, *dlen);
+ }
+
+ return ret;
+}
+
+/* Legacy compress framework end */
diff --git a/drivers/crypto/cavium/zip/zip_crypto.h b/drivers/crypto/cavium/zip/zip_crypto.h
index 1215049..26792e9 100644
--- a/drivers/crypto/cavium/zip/zip_crypto.h
+++ b/drivers/crypto/cavium/zip/zip_crypto.h
@@ -48,6 +48,8 @@
#include <linux/crypto.h>
#include "common.h"
+#include "zip_deflate.h"
+#include "zip_inflate.h"
struct zip_kernel_ctx {
struct zip_operation zip_comp;
@@ -57,5 +59,9 @@ struct zip_kernel_ctx {
int zip_alloc_zip_ctx(struct crypto_tfm *tfm);
int zip_alloc_lzs_ctx(struct crypto_tfm *tfm);
void zip_free_zip_ctx(struct crypto_tfm *tfm);
+int zip_deflate_comp(struct crypto_tfm *tfm, const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen);
+int zip_inflate_comp(struct crypto_tfm *tfm, const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen);
#endif
diff --git a/drivers/crypto/cavium/zip/zip_deflate.c b/drivers/crypto/cavium/zip/zip_deflate.c
new file mode 100644
index 0000000..913cc25
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_deflate.c
@@ -0,0 +1,190 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include <linux/delay.h>
+#include <linux/sched.h>
+
+#include "common.h"
+#include "zip_deflate.h"
+
+/* Prepares the deflate zip command */
+static int prepare_zip_command(struct zip_operation *zip_ops,
+ struct zip_state *s, union zip_inst_s *zip_cmd)
+{
+ union zip_zres_s *result_ptr = &s->result;
+
+ memset(zip_cmd, 0, sizeof(s->zip_cmd));
+ memset(result_ptr, 0, sizeof(s->result));
+
+ /* IWORD #0 */
+ /* History gather */
+ zip_cmd->s.hg = 0;
+ /* compression enable = 1 for deflate */
+ zip_cmd->s.ce = 1;
+ /* sf (sync flush) */
+ zip_cmd->s.sf = 1;
+ /* ef (end of file) */
+ if (zip_ops->flush == ZIP_FLUSH_FINISH) {
+ zip_cmd->s.ef = 1;
+ zip_cmd->s.sf = 0;
+ }
+
+ zip_cmd->s.cc = zip_ops->ccode;
+ /* ss (compression speed/storage) */
+ zip_cmd->s.ss = zip_ops->speed;
+
+ /* IWORD #1 */
+ /* adler checksum */
+ zip_cmd->s.adlercrc32 = zip_ops->csum;
+ zip_cmd->s.historylength = zip_ops->history_len;
+ zip_cmd->s.dg = 0;
+
+ /* IWORD # 6 and 7 - compression input/history pointer */
+ zip_cmd->s.inp_ptr_addr.s.addr = __pa(zip_ops->input);
+ zip_cmd->s.inp_ptr_ctl.s.length = (zip_ops->input_len +
+ zip_ops->history_len);
+ zip_cmd->s.ds = 0;
+
+ /* IWORD # 8 and 9 - Output pointer */
+ zip_cmd->s.out_ptr_addr.s.addr = __pa(zip_ops->output);
+ zip_cmd->s.out_ptr_ctl.s.length = zip_ops->output_len;
+ /* maximum number of output-stream bytes that can be written */
+ zip_cmd->s.totaloutputlength = zip_ops->output_len;
+
+ /* IWORD # 10 and 11 - Result pointer */
+ zip_cmd->s.res_ptr_addr.s.addr = __pa(result_ptr);
+ /* Clearing completion code */
+ result_ptr->s.compcode = 0;
+
+ return 0;
+}
+
+/**
+ * zip_deflate - API to offload deflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to zip device structure
+ *
+ * This function prepares the zip deflate command and submits it to the zip
+ * engine for processing.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev)
+{
+ union zip_inst_s *zip_cmd = &s->zip_cmd;
+ union zip_zres_s *result_ptr = &s->result;
+ u32 queue;
+
+ /* Prepares zip command based on the input parameters */
+ prepare_zip_command(zip_ops, s, zip_cmd);
+
+ /* Loads zip command into command queues and rings door bell */
+ queue = zip_load_instr(zip_cmd, zip_dev);
+
+ while (!result_ptr->s.compcode)
+ continue;
+
+ zip_ops->compcode = result_ptr->s.compcode;
+ switch (zip_ops->compcode) {
+ case ZIP_NOTDONE:
+ zip_dbg("Zip instruction not yet completed");
+ return ZIP_ERROR;
+
+ case ZIP_SUCCESS:
+ zip_dbg("Zip instruction completed successfully");
+ zip_update_cmd_bufs(zip_dev, queue);
+ break;
+
+ case ZIP_DTRUNC:
+ zip_dbg("Output Truncate error");
+ /* Returning ZIP_ERROR to avoid copy to user */
+ return ZIP_ERROR;
+
+ default:
+ zip_err("Zip instruction failed. Code:%d", zip_ops->compcode);
+ return ZIP_ERROR;
+ }
+
+ /* Update the CRC depending on the format */
+ switch (zip_ops->format) {
+ case RAW_FORMAT:
+ zip_dbg("RAW Format: %d ", zip_ops->format);
+ /* Get checksum from engine, need to feed it again */
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case ZLIB_FORMAT:
+ zip_dbg("ZLIB Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case GZIP_FORMAT:
+ zip_dbg("GZIP Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.crc32;
+ break;
+
+ case LZS_FORMAT:
+ zip_dbg("LZS Format: %d ", zip_ops->format);
+ break;
+
+ default:
+ zip_err("Unknown Format:%d\n", zip_ops->format);
+ }
+
+ /* Update output_len */
+ if (zip_ops->output_len < result_ptr->s.totalbyteswritten) {
+ /* Dynamic stop && strm->output_len < zipconstants[onfsize] */
+ zip_err("output_len (%d) < total bytes written(%d)\n",
+ zip_ops->output_len, result_ptr->s.totalbyteswritten);
+ zip_ops->output_len = 0;
+
+ } else {
+ zip_ops->output_len = result_ptr->s.totalbyteswritten;
+ }
+
+ return 0;
+}
diff --git a/drivers/crypto/cavium/zip/zip_deflate.h b/drivers/crypto/cavium/zip/zip_deflate.h
new file mode 100644
index 0000000..bdb5207
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_deflate.h
@@ -0,0 +1,62 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_DEFLATE_H__
+#define __ZIP_DEFLATE_H__
+
+/**
+ * zip_deflate - API to offload deflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to the structure representing zip device
+ *
+ * This function prepares the zip deflate command and submits it to the zip
+ * engine by ringing the doorbell.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev);
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_device.c b/drivers/crypto/cavium/zip/zip_device.c
index ed21c5a..a72cdcf0 100644
--- a/drivers/crypto/cavium/zip/zip_device.c
+++ b/drivers/crypto/cavium/zip/zip_device.c
@@ -44,6 +44,7 @@
***********************license end**************************************/
#include "common.h"
+#include "zip_deflate.h"
/**
* zip_cmd_queue_consumed - Calculates the space consumed in the command queue.
diff --git a/drivers/crypto/cavium/zip/zip_inflate.c b/drivers/crypto/cavium/zip/zip_inflate.c
new file mode 100644
index 0000000..849c4c85
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_inflate.c
@@ -0,0 +1,211 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include <linux/delay.h>
+#include <linux/sched.h>
+
+#include "common.h"
+#include "zip_inflate.h"
+
+static int prepare_inflate_zcmd(struct zip_operation *zip_ops,
+ struct zip_state *s, union zip_inst_s *zip_cmd)
+{
+ union zip_zres_s *result_ptr = &s->result;
+
+ memset(zip_cmd, 0, sizeof(s->zip_cmd));
+ memset(result_ptr, 0, sizeof(s->result));
+
+ /* IWORD#0 */
+
+ /* Decompression History Gather list - no gather list */
+ zip_cmd->s.hg = 0;
+ /* For decompression, CE must be 0x0. */
+ zip_cmd->s.ce = 0;
+ /* For decompression, SS must be 0x0. */
+ zip_cmd->s.ss = 0;
+ /* For decompression, SF should always be set. */
+ zip_cmd->s.sf = 1;
+
+ /* Begin File */
+ if (zip_ops->begin_file == 0)
+ zip_cmd->s.bf = 0;
+ else
+ zip_cmd->s.bf = 1;
+
+ zip_cmd->s.ef = 1;
+ /* 0: for Deflate decompression, 3: for LZS decompression */
+ zip_cmd->s.cc = zip_ops->ccode;
+
+ /* IWORD #1*/
+
+ /* adler checksum */
+ zip_cmd->s.adlercrc32 = zip_ops->csum;
+
+ /*
+ * HISTORYLENGTH must be 0x0 for any ZIP decompress operation.
+ * History data is added to a decompression operation via IWORD3.
+ */
+ zip_cmd->s.historylength = 0;
+ zip_cmd->s.ds = 0;
+
+ /* IWORD # 8 and 9 - Output pointer */
+ zip_cmd->s.out_ptr_addr.s.addr = __pa(zip_ops->output);
+ zip_cmd->s.out_ptr_ctl.s.length = zip_ops->output_len;
+
+ /* Maximum number of output-stream bytes that can be written */
+ zip_cmd->s.totaloutputlength = zip_ops->output_len;
+
+ zip_dbg("Data Direct Input case ");
+
+ /* IWORD # 6 and 7 - input pointer */
+ zip_cmd->s.dg = 0;
+ zip_cmd->s.inp_ptr_addr.s.addr = __pa((u8 *)zip_ops->input);
+ zip_cmd->s.inp_ptr_ctl.s.length = zip_ops->input_len;
+
+ /* IWORD # 10 and 11 - Result pointer */
+ zip_cmd->s.res_ptr_addr.s.addr = __pa(result_ptr);
+
+ /* Clearing completion code */
+ result_ptr->s.compcode = 0;
+
+ /* Returning 0 for time being.*/
+ return 0;
+}
+
+/**
+ * zip_inflate - API to offload inflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to zip device structure
+ *
+ * This function prepares the zip inflate command and submits it to the zip
+ * engine for processing.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev)
+{
+ union zip_inst_s *zip_cmd = &s->zip_cmd;
+ union zip_zres_s *result_ptr = &s->result;
+ u32 queue;
+
+ /* Prepare inflate zip command */
+ prepare_inflate_zcmd(zip_ops, s, zip_cmd);
+
+ /* Load inflate command to zip queue and ring the doorbell */
+ queue = zip_load_instr(zip_cmd, zip_dev);
+
+ while (!result_ptr->s.compcode)
+ continue;
+
+ zip_ops->compcode = result_ptr->s.compcode;
+ switch (zip_ops->compcode) {
+ case ZIP_NOTDONE:
+ zip_dbg("Zip Instruction not yet completed\n");
+ return ZIP_ERROR;
+
+ case ZIP_SUCCESS:
+ zip_dbg("Zip Instruction completed successfully\n");
+ break;
+
+ case ZIP_DYNAMIC_STOP:
+ zip_dbg(" Dynamic stop Initiated\n");
+ break;
+
+ default:
+ zip_dbg("Instruction failed. Code = %d\n", zip_ops->compcode);
+ zip_update_cmd_bufs(zip_dev, queue);
+ return ZIP_ERROR;
+ }
+
+ zip_update_cmd_bufs(zip_dev, queue);
+
+ if ((zip_ops->ccode == 3) && (zip_ops->flush == 4) &&
+ (zip_ops->compcode != ZIP_DYNAMIC_STOP))
+ result_ptr->s.ef = 1;
+
+ zip_ops->csum = result_ptr->s.adler32;
+
+ if (zip_ops->output_len < result_ptr->s.totalbyteswritten) {
+ zip_err("output_len (%d) < total bytes written (%d)\n",
+ zip_ops->output_len, result_ptr->s.totalbyteswritten);
+ zip_ops->output_len = 0;
+ } else {
+ zip_ops->output_len = result_ptr->s.totalbyteswritten;
+ }
+
+ zip_ops->bytes_read = result_ptr->s.totalbytesread;
+ zip_ops->bits_processed = result_ptr->s.totalbitsprocessed;
+ zip_ops->end_file = result_ptr->s.ef;
+ if (zip_ops->end_file) {
+ switch (zip_ops->format) {
+ case RAW_FORMAT:
+ zip_dbg("RAW Format: %d ", zip_ops->format);
+ /* Get checksum from engine */
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case ZLIB_FORMAT:
+ zip_dbg("ZLIB Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case GZIP_FORMAT:
+ zip_dbg("GZIP Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.crc32;
+ break;
+
+ case LZS_FORMAT:
+ zip_dbg("LZS Format: %d ", zip_ops->format);
+ break;
+
+ default:
+ zip_err("Format error:%d\n", zip_ops->format);
+ }
+ }
+
+ return 0;
+}
diff --git a/drivers/crypto/cavium/zip/zip_inflate.h b/drivers/crypto/cavium/zip/zip_inflate.h
new file mode 100644
index 0000000..4cee4c9
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_inflate.h
@@ -0,0 +1,62 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_INFLATE_H__
+#define __ZIP_INFLATE_H__
+
+/**
+ * zip_inflate - API to offload inflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to the structure representing zip device
+ *
+ * This function prepares the zip inflate command and submits it to the zip
+ * engine for processing.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev);
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_main.c b/drivers/crypto/cavium/zip/zip_main.c
index 052c42d..ae3395f 100644
--- a/drivers/crypto/cavium/zip/zip_main.c
+++ b/drivers/crypto/cavium/zip/zip_main.c
@@ -364,35 +364,6 @@ static void zip_remove(struct pci_dev *pdev)
zip_dbg_exit();
}
-/* Dummy Functions */
-int zip_alloc_lzs_ctx(struct crypto_tfm *tfm)
-{
- return 0;
-}
-
-int zip_alloc_zip_ctx(struct crypto_tfm *tfm)
-{
- return 0;
-}
-
-void zip_free_zip_ctx(struct crypto_tfm *tfm)
-{
-}
-
-int zip_deflate_comp(struct crypto_tfm *tfm,
- const u8 *src, unsigned int slen,
- u8 *dst, unsigned int *dlen)
-{
- return 0;
-}
-
-int zip_inflate_comp(struct crypto_tfm *tfm,
- const u8 *src, unsigned int slen,
- u8 *dst, unsigned int *dlen)
-{
- return 0;
-}
-
/* PCI Sub-System Interface */
static struct pci_driver zip_driver = {
.name = DRV_NAME,
--
2.9.0.rc0.21.g7777322
^ permalink raw reply related
* [RFC PATCH 1/3] crypto: zip - Add ThunderX ZIP driver core
From: Jan Glauber @ 2016-12-12 15:04 UTC (permalink / raw)
To: Herbert Xu
Cc: linux-crypto, linux-kernel, David S . Miller, Mahipal Challa,
Vishnu Nair, Jan Glauber
In-Reply-To: <20161212150439.18627-1-jglauber@cavium.com>
From: Mahipal Challa <Mahipal.Challa@cavium.com>
Add a driver for the ZIP engine found on Cavium ThunderX SOCs.
The ZIP engine supports hardware accelerated compression and
decompression. It includes 2 independent ZIP cores and supports:
- DEFLATE compression and decompression (RFC 1951)
- LZS compression and decompression (RFC 2395 and ANSI X3.241-1994)
- ADLER32 and CRC32 checksums for ZLIB (RFC 1950) and GZIP (RFC 1952)
The ZIP engine is presented as a PCI device. It supports DMA and
scatter-gather.
Signed-off-by: Mahipal Challa <Mahipal.Challa@cavium.com>
Signed-off-by: Vishnu Nair <Vishnu.Nair@cavium.com>
Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
drivers/crypto/Kconfig | 7 +
drivers/crypto/Makefile | 1 +
drivers/crypto/cavium/Makefile | 4 +
drivers/crypto/cavium/zip/Makefile | 8 +
drivers/crypto/cavium/zip/common.h | 258 +++++++
drivers/crypto/cavium/zip/zip_crypto.h | 61 ++
drivers/crypto/cavium/zip/zip_device.c | 208 +++++
drivers/crypto/cavium/zip/zip_device.h | 138 ++++
drivers/crypto/cavium/zip/zip_main.c | 500 ++++++++++++
drivers/crypto/cavium/zip/zip_main.h | 126 +++
drivers/crypto/cavium/zip/zip_mem.c | 120 +++
drivers/crypto/cavium/zip/zip_mem.h | 78 ++
drivers/crypto/cavium/zip/zip_regs.h | 1326 ++++++++++++++++++++++++++++++++
13 files changed, 2835 insertions(+)
create mode 100644 drivers/crypto/cavium/Makefile
create mode 100644 drivers/crypto/cavium/zip/Makefile
create mode 100644 drivers/crypto/cavium/zip/common.h
create mode 100644 drivers/crypto/cavium/zip/zip_crypto.h
create mode 100644 drivers/crypto/cavium/zip/zip_device.c
create mode 100644 drivers/crypto/cavium/zip/zip_device.h
create mode 100644 drivers/crypto/cavium/zip/zip_main.c
create mode 100644 drivers/crypto/cavium/zip/zip_main.h
create mode 100644 drivers/crypto/cavium/zip/zip_mem.c
create mode 100644 drivers/crypto/cavium/zip/zip_mem.h
create mode 100644 drivers/crypto/cavium/zip/zip_regs.h
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 4d2b81f..da48d93 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -485,6 +485,13 @@ config CRYPTO_DEV_MXS_DCP
source "drivers/crypto/qat/Kconfig"
+config CRYPTO_DEV_CAVIUM_ZIP
+ tristate "Cavium ZIP driver"
+ depends on PCI && 64BIT && (ARM64 || COMPILE_TEST)
+ ---help---
+ Select this option if you want to enable compression/decompression
+ acceleration on Cavium's ARM based SoCs
+
config CRYPTO_DEV_QCE
tristate "Qualcomm crypto engine accelerator"
depends on (ARCH_QCOM || COMPILE_TEST) && HAS_DMA && HAS_IOMEM
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index ad7250f..3d152d4 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -27,6 +27,7 @@ obj-$(CONFIG_CRYPTO_DEV_MXC_SCC) += mxc-scc.o
obj-$(CONFIG_CRYPTO_DEV_TALITOS) += talitos.o
obj-$(CONFIG_CRYPTO_DEV_UX500) += ux500/
obj-$(CONFIG_CRYPTO_DEV_QAT) += qat/
+obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += cavium/
obj-$(CONFIG_CRYPTO_DEV_QCE) += qce/
obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
obj-$(CONFIG_CRYPTO_DEV_SUN4I_SS) += sunxi-ss/
diff --git a/drivers/crypto/cavium/Makefile b/drivers/crypto/cavium/Makefile
new file mode 100644
index 0000000..641268b
--- /dev/null
+++ b/drivers/crypto/cavium/Makefile
@@ -0,0 +1,4 @@
+#
+# Makefile for Cavium crypto device drivers
+#
+obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += zip/
diff --git a/drivers/crypto/cavium/zip/Makefile b/drivers/crypto/cavium/zip/Makefile
new file mode 100644
index 0000000..2c07508
--- /dev/null
+++ b/drivers/crypto/cavium/zip/Makefile
@@ -0,0 +1,8 @@
+#
+# Makefile for Cavium's ZIP Driver.
+#
+
+obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += thunderx_zip.o
+thunderx_zip-y := zip_main.o \
+ zip_device.o \
+ zip_mem.o
diff --git a/drivers/crypto/cavium/zip/common.h b/drivers/crypto/cavium/zip/common.h
new file mode 100644
index 0000000..f0694f4
--- /dev/null
+++ b/drivers/crypto/cavium/zip/common.h
@@ -0,0 +1,258 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __COMMON_H__
+#define __COMMON_H__
+
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/version.h>
+
+/* Device specific zlib function definitions */
+#include "zip_device.h"
+
+/* ZIP device definitions */
+#include "zip_main.h"
+
+/* ZIP memory allocation/deallocation related definitions */
+#include "zip_mem.h"
+
+/* Device specific structure definitions */
+#include "zip_regs.h"
+
+#define ZIP_ERROR -1
+
+#define ZIP_FLUSH_FINISH 4
+
+#define RAW_FORMAT 0 /* for rawpipe */
+#define ZLIB_FORMAT 1 /* for zpipe */
+#define GZIP_FORMAT 2 /* for gzpipe */
+#define LZS_FORMAT 3 /* for lzspipe */
+
+/* Max number of ZIP devices supported */
+#define MAX_ZIP_DEVICES 2
+
+/* Configures the number of zip queues to be used */
+#define ZIP_NUM_QUEUES 2
+
+#define DYNAMIC_STOP_EXCESS 1024
+
+/* Maximum buffer sizes in direct mode */
+#define MAX_INPUT_BUFFER_SIZE ((64 * 1024) - 1)
+#define MAX_OUTPUT_BUFFER_SIZE ((64 * 1024) - 1)
+
+/* ZIP invocation result completion status codes */
+#define ZIP_NOTDONE 0x0
+
+/* Successful completion. */
+#define ZIP_SUCCESS 0x1
+
+/* Output truncated */
+#define ZIP_DTRUNC 0x2
+
+/* Dynamic Stop */
+#define ZIP_DYNAMIC_STOP 0x3
+
+/* Uncompress ran out of input data when IWORD0[EF] was set */
+#define ZIP_ITRUNC 0x4
+
+/* Uncompress found the reserved block type 3 */
+#define ZIP_RBLOCK 0x5
+
+/* Uncompress found LEN != ZIP_NLEN in an uncompressed block in the input */
+#define ZIP_NLEN 0x6
+
+/* Uncompress found a bad code in the main Huffman codes. */
+#define ZIP_BADCODE 0x7
+
+/* Uncompress found a bad code in the 19 Huffman codes encoding lengths. */
+#define ZIP_BADCODE2 0x8
+
+/* Compress found a zero-length input. */
+#define ZIP_ZERO_LEN 0x9
+
+/* The compress or decompress encountered an internal parity error. */
+#define ZIP_PARITY 0xA
+
+/*
+ * Uncompress found a string identifier that precedes the uncompressed data and
+ * decompression history.
+ */
+#define ZIP_FATAL 0xB
+
+/**
+ * struct zip_operation - common data structure for comp and decomp operations
+ * @input: Next input byte is read from here
+ * @output: Next output byte written here
+ * @ctx_addr: Inflate context buffer address
+ * @history: Pointer to the history buffer
+ * @input_len: Number of bytes available at next_in
+ * @input_total_len: Total number of input bytes read
+ * @output_len: Remaining free space at next_out
+ * @output_total_len: Total number of bytes output so far
+ * @csum: Checksum value of the uncompressed data
+ * @flush: Flush flag
+ * @format: Format (depends on stream's wrap)
+ * @speed: Speed depends on stream's level
+ * @ccode: Compression code ( stream's strategy)
+ * @lzs_flag: Flag for LZS support
+ * @begin_file: Beginning of file indication for inflate
+ * @history_len: Size of the history data
+ * @end_file: Ending of the file indication for inflate
+ * @compcode: Completion status of the ZIP invocation
+ * @bytes_read: Input bytes read in current instruction
+ * @bits_processed: Total bits processed for entire file
+ * @sizeofptr: To distinguish between ILP32 and LP64
+ * @sizeofzops: Optional just for padding
+ *
+ * This structure is used to maintain the required meta data for the
+ * comp and decomp operations.
+ */
+struct zip_operation {
+ u8 *input;
+ u8 *output;
+ u64 ctx_addr;
+ u64 history;
+
+ u32 input_len;
+ u32 input_total_len;
+
+ u32 output_len;
+ u32 output_total_len;
+
+ u32 csum;
+ u32 flush;
+
+ u32 format;
+ u32 speed;
+ u32 ccode;
+ u32 lzs_flag;
+
+ u32 begin_file;
+ u32 history_len;
+
+ u32 end_file;
+ u32 compcode;
+ u32 bytes_read;
+ u32 bits_processed;
+
+ u32 sizeofptr;
+ u32 sizeofzops;
+};
+
+/* error messages */
+#define zip_err(fmt, args...) pr_err("ZIP ERR:%s():%d: " \
+ fmt "\n", __func__, __LINE__, ## args)
+
+#ifdef MSG_ENABLE
+/* Enable all messages */
+#define zip_msg(fmt, args...) pr_info("ZIP_MSG:" fmt "\n", ## args)
+#else
+#define zip_msg(fmt, args...)
+#endif
+
+#if defined(ZIP_DEBUG_ENABLE) && defined(MSG_ENABLE)
+
+#ifdef DEBUG_LEVEL
+
+#define FILE_NAME (strrchr(__FILE__, '/') ? strrchr(__FILE__, '/') + 1 : \
+ strrchr(__FILE__, '\\') ? strrchr(__FILE__, '\\') + 1 : __FILE__)
+
+#if DEBUG_LEVEL >= 4
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG: %s: %s() : %d: " \
+ fmt "\n", FILE_NAME, __func__, __LINE__, ## args)
+
+#define zip_dbg_enter(fmt, args...) pr_info("ZIP_DBG: %s() in %s" \
+ fmt "\n", __func__, FILE_NAME, ## args)
+
+#define zip_dbg_exit(fmt, args...) pr_info("ZIP_DBG:Exit %s() in %s" \
+ fmt "\n", __func__, FILE_NAME, ## args)
+
+#elif DEBUG_LEVEL >= 3
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG: %s: %s() : %d: " \
+ fmt "\n", FILE_NAME, __func__, __LINE__, ## args)
+
+#elif DEBUG_LEVEL >= 2
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG: %s() : %d: " \
+ fmt "\n", __func__, __LINE__, ## args)
+
+#else
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG:" fmt "\n", ## args)
+
+#endif /* DEBUG LEVEL >= */
+
+#if DEBUG_LEVEL <= 3
+
+#define zip_dbg_enter(fmt, args...)
+#define zip_dbg_exit(fmt, args...)
+
+#endif /* DEBUG_LEVEL <= 3 */
+#else
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG:" fmt "\n", ## args)
+
+#define zip_dbg_enter(fmt, args...)
+#define zip_dbg_exit(fmt, args...)
+
+#endif /* DEBUG_LEVEL */
+#else
+
+#define zip_dbg(fmt, args...)
+#define zip_dbg_enter(fmt, args...)
+#define zip_dbg_exit(fmt, args...)
+
+#endif /* ZIP_DEBUG_ENABLE */
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_crypto.h b/drivers/crypto/cavium/zip/zip_crypto.h
new file mode 100644
index 0000000..1215049
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_crypto.h
@@ -0,0 +1,61 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_CRYPTO_H__
+#define __ZIP_CRYPTO_H__
+
+#include <linux/crypto.h>
+#include "common.h"
+
+struct zip_kernel_ctx {
+ struct zip_operation zip_comp;
+ struct zip_operation zip_decomp;
+};
+
+int zip_alloc_zip_ctx(struct crypto_tfm *tfm);
+int zip_alloc_lzs_ctx(struct crypto_tfm *tfm);
+void zip_free_zip_ctx(struct crypto_tfm *tfm);
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_device.c b/drivers/crypto/cavium/zip/zip_device.c
new file mode 100644
index 0000000..ed21c5a
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_device.c
@@ -0,0 +1,208 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include "common.h"
+
+/**
+ * zip_cmd_queue_consumed - Calculates the space consumed in the command queue.
+ *
+ * @zip_dev: Pointer to zip device structure
+ * @queue: Queue number
+ *
+ * Return: Bytes consumed in the command queue buffer.
+ */
+static inline u32 zip_cmd_queue_consumed(struct zip_device *zip_dev, int queue)
+{
+ return ((zip_dev->iq[queue].sw_head - zip_dev->iq[queue].sw_tail) *
+ sizeof(u64 *));
+}
+
+/**
+ * zip_load_instr - Submits the instruction into the ZIP command queue
+ * @instr: Pointer to the instruction to be submitted
+ * @zip_dev: Pointer to ZIP device structure to which the instruction is to
+ * be submitted
+ *
+ * This function copies the ZIP instruction to the command queue and rings the
+ * doorbell to notify the engine of the instruction submission. The command
+ * queue is maintained in a circular fashion. When there is space for exactly
+ * one instruction in the queue, next chunk pointer of the queue is made to
+ * point to the head of the queue, thus maintaining a circular queue.
+ *
+ * Return: Queue number to which the instruction was submitted
+ */
+u32 zip_load_instr(union zip_inst_s *instr,
+ struct zip_device *zip_dev)
+{
+ union zip_quex_doorbell dbell;
+ u32 queue = 0;
+ u32 consumed = 0;
+ u64 *ncb_ptr = NULL;
+ union zip_nptr_s ncp;
+
+ /*
+ * Distribute the instructions between the enabled queues based on
+ * the CPU id.
+ */
+ if (raw_smp_processor_id() % 2 == 0)
+ queue = 0;
+ else
+ queue = 1;
+
+ zip_dbg("CPU Core: %d Queue number:%d", raw_smp_processor_id(), queue);
+
+ /* Take cmd buffer lock */
+ spin_lock(&zip_dev->iq[queue].lock);
+
+ /*
+ * Command Queue implementation
+ * 1. If there is place for new instructions, push the cmd at sw_head.
+ * 2. If there is place for exactly one instruction, push the new cmd
+ * at the sw_head. Make sw_head point to the sw_tail to make it
+ * circular. Write sw_head's physical address to the "Next-Chunk
+ * Buffer Ptr" to make it cmd_hw_tail.
+ * 3. Ring the door bell.
+ */
+ zip_dbg("sw_head : %lx", zip_dev->iq[queue].sw_head);
+ zip_dbg("sw_tail : %lx", zip_dev->iq[queue].sw_tail);
+
+ consumed = zip_cmd_queue_consumed(zip_dev, queue);
+ /* Check if there is space to push just one cmd */
+ if ((consumed + 128) == (ZIP_CMD_QBUF_SIZE - 8)) {
+ zip_dbg("Cmd queue space available for single command");
+ /* Space for one cmd, pust it and make it circular queue */
+ memcpy((u8 *)zip_dev->iq[queue].sw_head, (u8 *)instr,
+ sizeof(union zip_inst_s));
+ zip_dev->iq[queue].sw_head += 16; /* 16 64_bit words = 128B */
+
+ /* Now, point the "Next-Chunk Buffer Ptr" to sw_head */
+ ncb_ptr = zip_dev->iq[queue].sw_head;
+
+ zip_dbg("ncb addr :0x%lx sw_head addr :0x%lx",
+ ncb_ptr, zip_dev->iq[queue].sw_head - 16);
+
+ /* Using Circular command queue */
+ zip_dev->iq[queue].sw_head = zip_dev->iq[queue].sw_tail;
+ /* Mark this buffer for free */
+ zip_dev->iq[queue].free_flag = 1;
+
+ /* Write new chunk buffer address at "Next-Chunk Buffer Ptr" */
+ ncp.u_reg64 = 0ull;
+ ncp.s.addr = __pa(zip_dev->iq[queue].sw_head);
+ *ncb_ptr = ncp.u_reg64;
+ zip_dbg("*ncb_ptr :0x%lx sw_head[phys] :0x%lx",
+ *ncb_ptr, __pa(zip_dev->iq[queue].sw_head));
+
+ zip_dev->iq[queue].pend_cnt++;
+
+ } else {
+ zip_dbg("Enough space is available for commands");
+ /* Push this cmd to cmd queue buffer */
+ memcpy((u8 *)zip_dev->iq[queue].sw_head, (u8 *)instr,
+ sizeof(union zip_inst_s));
+ zip_dev->iq[queue].sw_head += 16; /* 16 64_bit words = 128B */
+
+ zip_dev->iq[queue].pend_cnt++;
+ }
+ zip_dbg("sw_head :0x%lx sw_tail :0x%lx hw_tail :0x%lx",
+ zip_dev->iq[queue].sw_head, zip_dev->iq[queue].sw_tail,
+ zip_dev->iq[queue].hw_tail);
+
+ zip_dbg(" Pushed the new cmd : pend_cnt : %d",
+ zip_dev->iq[queue].pend_cnt);
+
+ /* Ring the doorbell */
+ dbell.u_reg64 = 0ull;
+ dbell.s.dbell_cnt = 1;
+ zip_reg_write(dbell.u_reg64,
+ (zip_dev->reg_base + ZIP_QUEX_DOORBELL(queue)));
+
+ /* Unlock cmd buffer lock */
+ spin_unlock(&zip_dev->iq[queue].lock);
+
+ /* Poll for the IQ cmd completion code */
+ zip_dbg_exit();
+
+ return queue;
+}
+
+/**
+ * zip_update_cmd_bufs - Updates the queue statistics after posting the
+ * instruction
+ * @zip_dev: Pointer to zip device structure
+ * @queue: Queue number
+ */
+void zip_update_cmd_bufs(struct zip_device *zip_dev, u32 queue)
+{
+ zip_dbg_enter();
+
+ /* Take cmd buffer lock */
+ spin_lock(&zip_dev->iq[queue].lock);
+
+ /* Check if the previous buffer can be freed */
+ if (zip_dev->iq[queue].free_flag == 1) {
+ zip_dbg("Free flag. Free cmd buffer, adjust sw head and tail");
+ /* Reset the free flag */
+ zip_dev->iq[queue].free_flag = 0;
+
+ /* Point the hw_tail to start of the new chunk buffer */
+ zip_dev->iq[queue].hw_tail = zip_dev->iq[queue].sw_head;
+ } else {
+ zip_dbg("Free flag not set. increment hw tail");
+ zip_dev->iq[queue].hw_tail += 16; /* 16 64_bit words = 128B */
+ }
+
+ zip_dev->iq[queue].done_cnt++;
+ zip_dev->iq[queue].pend_cnt--;
+
+ zip_dbg("sw_head :0x%lx sw_tail :0x%lx hw_tail :0x%lx",
+ zip_dev->iq[queue].sw_head, zip_dev->iq[queue].sw_tail,
+ zip_dev->iq[queue].hw_tail);
+ zip_dbg(" Got CC : pend_cnt : %d\n", zip_dev->iq[queue].pend_cnt);
+
+ spin_unlock(&zip_dev->iq[queue].lock);
+
+ zip_dbg_exit();
+}
diff --git a/drivers/crypto/cavium/zip/zip_device.h b/drivers/crypto/cavium/zip/zip_device.h
new file mode 100644
index 0000000..7f864e0
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_device.h
@@ -0,0 +1,138 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_DEVICE_H__
+#define __ZIP_DEVICE_H__
+
+#include <linux/types.h>
+#include "zip_main.h"
+
+struct sg_info {
+ /*
+ * Pointer to the input data when scatter_gather == 0 and
+ * pointer to the input gather list buffer when scatter_gather == 1
+ */
+ union zip_zptr_s *gather;
+
+ /*
+ * Pointer to the output data when scatter_gather == 0 and
+ * pointer to the output scatter list buffer when scatter_gather == 1
+ */
+ union zip_zptr_s *scatter;
+
+ /*
+ * Holds size of the output buffer pointed by scatter list
+ * when scatter_gather == 1
+ */
+ u64 scatter_buf_size;
+
+ /* for gather data */
+ u64 gather_enable;
+
+ /* for scatter data */
+ u64 scatter_enable;
+
+ /* Number of gather list pointers for gather data */
+ u32 gbuf_cnt;
+
+ /* Number of scatter list pointers for scatter data */
+ u32 sbuf_cnt;
+
+ /* Buffers allocation state */
+ u8 alloc_state;
+};
+
+/**
+ * struct zip_state - Structure representing the required information related
+ * to a command
+ * @zip_cmd: Pointer to zip instruction structure
+ * @result: Pointer to zip result structure
+ * @ctx: Context pointer for inflate
+ * @history: Decompression history pointer
+ * @sginfo: Scatter-gather info structure
+ */
+struct zip_state {
+ union zip_inst_s zip_cmd;
+ union zip_zres_s result;
+ union zip_zptr_s *ctx;
+ union zip_zptr_s *history;
+ struct sg_info sginfo;
+};
+
+static inline u64 zip_depth(void)
+{
+ struct zip_device *zip_dev = zip_get_device(zip_get_node_id());
+
+ if (!zip_dev)
+ return -ENODEV;
+
+ return zip_dev->depth;
+}
+
+static inline u64 zip_onfsize(void)
+{
+ struct zip_device *zip_dev = zip_get_device(zip_get_node_id());
+
+ if (!zip_dev)
+ return -ENODEV;
+
+ return zip_dev->onfsize;
+}
+
+static inline u64 zip_ctxsize(void)
+{
+ struct zip_device *zip_dev = zip_get_device(zip_get_node_id());
+
+ if (!zip_dev)
+ return -ENODEV;
+
+ return zip_dev->ctxsize;
+}
+
+#define ZIP_CONTEXT_SIZE 2048
+#define ZIP_INFLATE_HISTORY_SIZE 32768
+#define ZIP_DEFLATE_HISTORY_SIZE 32768
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_main.c b/drivers/crypto/cavium/zip/zip_main.c
new file mode 100644
index 0000000..052c42d
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_main.c
@@ -0,0 +1,500 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include "common.h"
+#include "zip_crypto.h"
+
+#define DRV_NAME "ThunderX-ZIP"
+
+static struct zip_device *zip_dev[MAX_ZIP_DEVICES];
+
+static const struct pci_device_id zip_id_table[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDERX_ZIP) },
+ { 0, }
+};
+
+void zip_reg_write(u64 val, u64 __iomem *addr)
+{
+ writeq(val, addr);
+}
+
+u64 zip_reg_read(u64 __iomem *addr)
+{
+ return readq(addr);
+}
+
+/*
+ * Allocates new ZIP device structure
+ * Returns zip_device pointer or NULL if cannot allocate memory for zip_device
+ */
+static struct zip_device *zip_alloc_device(struct pci_dev *pdev)
+{
+ struct zip_device *zip = NULL;
+ int idx = 0;
+
+ for (idx = 0; idx < MAX_ZIP_DEVICES; idx++) {
+ if (!zip_dev[idx])
+ break;
+ }
+
+ zip = kzalloc(sizeof(*zip), GFP_KERNEL);
+
+ if (!zip)
+ return NULL;
+
+ zip_dev[idx] = zip;
+ zip->index = idx;
+ return zip;
+}
+
+/**
+ * zip_get_device - Get ZIP device based on node id of cpu
+ *
+ * @node: Node id of the current cpu
+ * Return: Pointer to Zip device structure
+ */
+struct zip_device *zip_get_device(int node)
+{
+ if ((node < MAX_ZIP_DEVICES) && (node >= 0))
+ return zip_dev[node];
+
+ zip_err("ZIP device not found for node id %d\n", node);
+ return NULL;
+}
+
+/**
+ * zip_get_node_id - Get the node id of the current cpu
+ *
+ * Return: Node id of the current cpu
+ */
+int zip_get_node_id(void)
+{
+ return cpu_to_node(raw_smp_processor_id());
+}
+
+/**
+ * zip-get_zipeng_count - Returns No. of ZIP Cores present in CN88XX
+ *
+ * Return: Number of zip engines in the system
+ */
+int zip_get_zipeng_count(void)
+{
+ return ZIP_NUMENG_CN88XX;
+}
+
+/* Initializes the ZIP h/w sub-system */
+static int zip_init_hw(struct zip_device *zip)
+{
+ union zip_cmd_ctl cmd_ctl;
+ union zip_constants constants;
+ union zip_que_ena que_ena;
+ union zip_quex_map que_map;
+ union zip_que_pri que_pri;
+
+ union zip_quex_sbuf_addr que_sbuf_addr;
+ union zip_quex_sbuf_ctl que_sbuf_ctl;
+
+ int q = 0;
+
+ zip_dbg_enter();
+
+ /* ZIP Engine Init / Enable */
+
+ /* Enable the ZIP Engine(Core) Clock */
+ cmd_ctl.u_reg64 = zip_reg_read(zip->reg_base + ZIP_CMD_CTL);
+ cmd_ctl.s.forceclk = 1;
+ zip_reg_write(cmd_ctl.u_reg64 & 0xFF, (zip->reg_base + ZIP_CMD_CTL));
+
+ zip_msg("ZIP_CMD_CTL : 0x%016llx",
+ zip_reg_read(zip->reg_base + ZIP_CMD_CTL));
+
+ constants.u_reg64 = zip_reg_read(zip->reg_base + ZIP_CONSTANTS);
+ zip->depth = constants.s.depth;
+ zip->onfsize = constants.s.onfsize;
+ zip->ctxsize = constants.s.ctxsize;
+
+ zip_msg("depth: 0x%016llx , onfsize : 0x%016llx , ctxsize : 0x%016llx",
+ zip->depth, zip->onfsize, zip->ctxsize);
+
+ /*
+ * Program ZIP_QUE(0..7)_SBUF_ADDR and ZIP_QUE(0..7)_SBUF_CTL to
+ * have the correct buffer pointer and size configured for each
+ * instruction queue.
+ */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ que_sbuf_ctl.u_reg64 = 0ull;
+ que_sbuf_ctl.s.size = (ZIP_CMD_QBUF_SIZE / sizeof(u64));
+ que_sbuf_ctl.s.inst_be = 0;
+ que_sbuf_ctl.s.stream_id = 0;
+ zip_reg_write(que_sbuf_ctl.u_reg64,
+ (zip->reg_base + ZIP_QUEX_SBUF_CTL(q)));
+
+ zip_msg("QUEX_SBUF_CTL[%d]: 0x%016llx", q,
+ zip_reg_read(zip->reg_base + ZIP_QUEX_SBUF_CTL(q)));
+ }
+
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ memset(&zip->iq[q], 0x0, sizeof(struct zip_iq));
+
+ spin_lock_init(&zip->iq[q].lock);
+
+ if (zip_cmd_qbuf_alloc(zip, q)) {
+ while (q != 0) {
+ q--;
+ zip_cmd_qbuf_free(zip, q);
+ }
+ return -ENOMEM;
+ }
+
+ /* Initialize tail ptr to head */
+ zip->iq[q].sw_tail = zip->iq[q].sw_head;
+ zip->iq[q].hw_tail = zip->iq[q].sw_head;
+
+ /* Write the physical addr to register */
+ que_sbuf_addr.u_reg64 = 0ull;
+ que_sbuf_addr.s.ptr = (__pa(zip->iq[q].sw_head) >>
+ ZIP_128B_ALIGN);
+
+ zip_msg("QUE[%d]_PTR(PHYS): 0x%016llx", q,
+ (u64)que_sbuf_addr.s.ptr);
+
+ zip_reg_write(que_sbuf_addr.u_reg64,
+ (zip->reg_base + ZIP_QUEX_SBUF_ADDR(q)));
+
+ zip_msg("QUEX_SBUF_ADDR[%d]: 0x%016llx", q,
+ zip_reg_read(zip->reg_base + ZIP_QUEX_SBUF_ADDR(q)));
+
+ zip_dbg("sw_head :0x%lx sw_tail :0x%lx hw_tail :0x%lx",
+ zip->iq[q].sw_head, zip->iq[q].sw_tail,
+ zip->iq[q].hw_tail);
+ zip_dbg("sw_head phy addr : 0x%lx", que_sbuf_addr.s.ptr);
+ }
+
+ /*
+ * Queue-to-ZIP core mapping
+ * If a queue is not mapped to a particular core, it is equivalent to
+ * the ZIP core being disabled.
+ */
+ que_ena.u_reg64 = 0x0ull;
+ /* Enabling queues based on ZIP_NUM_QUEUES */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++)
+ que_ena.s.ena |= (0x1 << q);
+ zip_reg_write(que_ena.u_reg64, (zip->reg_base + ZIP_QUE_ENA));
+
+ zip_msg("QUE_ENA : 0x%016llx",
+ zip_reg_read(zip->reg_base + ZIP_QUE_ENA));
+
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ que_map.u_reg64 = 0ull;
+ /* Mapping each queue to two ZIP cores */
+ que_map.s.zce = 0x3;
+ zip_reg_write(que_map.u_reg64,
+ (zip->reg_base + ZIP_QUEX_MAP(q)));
+
+ zip_msg("QUE_MAP(%d) : 0x%016llx", q,
+ zip_reg_read(zip->reg_base + ZIP_QUEX_MAP(q)));
+ }
+
+ que_pri.u_reg64 = 0ull;
+ for (q = 0; q < ZIP_NUM_QUEUES; q++)
+ que_pri.s.pri |= (0x1 << q); /* Higher Priority RR */
+ zip_reg_write(que_pri.u_reg64, (zip->reg_base + ZIP_QUE_PRI));
+
+ zip_msg("QUE_PRI %016llx", zip_reg_read(zip->reg_base + ZIP_QUE_PRI));
+
+ zip_dbg_exit();
+ return 0;
+}
+
+static int zip_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+ struct device *dev = &pdev->dev;
+ struct zip_device *zip = NULL;
+ int err;
+
+ zip_dbg_enter();
+
+ zip = zip_alloc_device(pdev);
+
+ if (!zip)
+ return -ENOMEM;
+
+ pr_info("Found ZIP device %d %x:%x on Node %d\n", zip->index,
+ pdev->vendor, pdev->device, dev_to_node(dev));
+
+ zip->pdev = pdev;
+
+ pci_set_drvdata(pdev, zip);
+
+ err = pci_enable_device(pdev);
+ if (err) {
+ zip_err("Failed to enable PCI device");
+ goto err_free_device;
+ }
+
+ err = pci_request_regions(pdev, DRV_NAME);
+ if (err) {
+ zip_err("PCI request regions failed 0x%x", err);
+ goto err_disable_device;
+ }
+
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(48));
+ if (err) {
+ dev_err(dev, "Unable to get usable DMA configuration\n");
+ goto err_release_regions;
+ }
+
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(48));
+ if (err) {
+ dev_err(dev, "Unable to get 48-bit DMA for allocations\n");
+ goto err_release_regions;
+ }
+
+ /* MAP configuration registers */
+ zip->reg_base = pci_ioremap_bar(pdev, PCI_CFG_ZIP_PF_BAR0);
+ if (!zip->reg_base) {
+ zip_err("ZIP: Cannot map BAR0 CSR memory space, aborting");
+ err = -ENOMEM;
+ goto err_release_regions;
+ }
+
+ /* Initialize ZIP Hardware */
+ err = zip_init_hw(zip);
+ if (err)
+ goto err_release_regions;
+
+ return 0;
+
+err_release_regions:
+ if (zip->reg_base)
+ iounmap(zip->reg_base);
+ pci_release_regions(pdev);
+
+err_disable_device:
+ pci_disable_device(pdev);
+
+err_free_device:
+ pci_set_drvdata(pdev, NULL);
+
+ /* remove zip_dev from zip_device list, free the zip_device memory */
+ zip_dev[zip->index] = NULL;
+ kfree(zip);
+
+ zip_dbg_exit();
+ return err;
+}
+
+static void zip_remove(struct pci_dev *pdev)
+{
+ struct zip_device *zip = pci_get_drvdata(pdev);
+ int q = 0;
+
+ zip_dbg_enter();
+
+ if (!zip)
+ return;
+
+ if (zip->reg_base) {
+ union zip_cmd_ctl cmd_ctl;
+
+ cmd_ctl.u_reg64 = 0x0ull;
+ cmd_ctl.s.reset = 1; /* Forces ZIP cores to do reset */
+ zip_reg_write(cmd_ctl.u_reg64, (zip->reg_base + ZIP_CMD_CTL));
+ iounmap(zip->reg_base);
+ }
+
+ pci_release_regions(pdev);
+ pci_disable_device(pdev);
+
+ /*
+ * Free Command Queue buffers. This free should be called for all
+ * the enabled Queues.
+ */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++)
+ zip_cmd_qbuf_free(zip, q);
+
+ pci_set_drvdata(pdev, NULL);
+
+ /* remove zip device from zip device list */
+ zip_dev[zip->index] = NULL;
+ kfree(zip);
+
+ zip_dbg_exit();
+}
+
+/* Dummy Functions */
+int zip_alloc_lzs_ctx(struct crypto_tfm *tfm)
+{
+ return 0;
+}
+
+int zip_alloc_zip_ctx(struct crypto_tfm *tfm)
+{
+ return 0;
+}
+
+void zip_free_zip_ctx(struct crypto_tfm *tfm)
+{
+}
+
+int zip_deflate_comp(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen)
+{
+ return 0;
+}
+
+int zip_inflate_comp(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen)
+{
+ return 0;
+}
+
+/* PCI Sub-System Interface */
+static struct pci_driver zip_driver = {
+ .name = DRV_NAME,
+ .id_table = zip_id_table,
+ .probe = zip_probe,
+ .remove = zip_remove,
+};
+
+/* Kernel Crypto Subsystem Interface */
+
+static struct crypto_alg zip_comp_deflate = {
+ .cra_name = "deflate",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zip_kernel_ctx),
+ .cra_priority = 300,
+ .cra_module = THIS_MODULE,
+ .cra_init = zip_alloc_zip_ctx,
+ .cra_exit = zip_free_zip_ctx,
+ .cra_u = { .compress = {
+ .coa_compress = zip_deflate_comp,
+ .coa_decompress = zip_inflate_comp
+ } }
+};
+
+static struct crypto_alg zip_comp_lzs = {
+ .cra_name = "lzs",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zip_kernel_ctx),
+ .cra_priority = 300,
+ .cra_module = THIS_MODULE,
+ .cra_init = zip_alloc_lzs_ctx,
+ .cra_exit = zip_free_zip_ctx,
+ .cra_u = { .compress = {
+ .coa_compress = zip_deflate_comp,
+ .coa_decompress = zip_inflate_comp
+ } }
+};
+
+static int zip_register_compression_device(void)
+{
+ int ret;
+
+ ret = crypto_register_alg(&zip_comp_deflate);
+ if (ret < 0) {
+ zip_err("Deflate algorithm registration failed\n");
+ return ret;
+ }
+
+ ret = crypto_register_alg(&zip_comp_lzs);
+ if (ret < 0) {
+ zip_err("LZS algorithm registration failed\n");
+ crypto_unregister_alg(&zip_comp_deflate);
+ }
+
+ return ret;
+}
+
+static void zip_unregister_compression_device(void)
+{
+ crypto_unregister_alg(&zip_comp_deflate);
+ crypto_unregister_alg(&zip_comp_lzs);
+}
+
+static int __init zip_init_module(void)
+{
+ int ret;
+
+ memset(&zip_dev, 0, sizeof(zip_dev));
+
+ zip_msg("%s\n", DRV_NAME);
+
+ ret = pci_register_driver(&zip_driver);
+ if (ret < 0) {
+ zip_err("ZIP: pci_register_driver() returned %d\n", ret);
+ return ret;
+ }
+
+ /* Register with the Kernel Crypto Interface */
+ ret = zip_register_compression_device();
+ if (ret < 0) {
+ zip_err("ZIP: Kernel Crypto Registration failed\n");
+ return 1;
+ }
+
+ return ret;
+}
+
+static void __exit zip_cleanup_module(void)
+{
+ /* Unregister this driver for pci zip devices */
+ pci_unregister_driver(&zip_driver);
+
+ /* Unregister from the kernel crypto interface */
+ zip_unregister_compression_device();
+
+ pr_info("ThunderX-ZIP driver is removed successfully\n");
+}
+
+module_init(zip_init_module);
+module_exit(zip_cleanup_module);
+
+MODULE_AUTHOR("Cavium Inc");
+MODULE_DESCRIPTION("Cavium Inc ThunderX ZIP Driver");
+MODULE_LICENSE("GPL v2");
+MODULE_DEVICE_TABLE(pci, zip_id_table);
diff --git a/drivers/crypto/cavium/zip/zip_main.h b/drivers/crypto/cavium/zip/zip_main.h
new file mode 100644
index 0000000..73b9e6d
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_main.h
@@ -0,0 +1,126 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_MAIN_H__
+#define __ZIP_MAIN_H__
+
+#include "zip_device.h"
+#include "zip_regs.h"
+
+/* PCI device IDs */
+#define PCI_DEVICE_ID_THUNDERX_ZIP 0xA01A
+
+/* ZIP device BARs */
+#define PCI_CFG_ZIP_PF_BAR0 0 /* Base addr for normal regs */
+#define PCI_CFG_ZIP_PF_BAR4 4 /* Base addr for MSI-X regs */
+
+/* Maximum available zip queues */
+#define ZIP_MAX_NUM_QUEUES 8
+#define ZIP_MAXQ_PER_ZIPENG 4
+#define ZIP_NUMENG_CN88XX 2
+
+#define ZIP_128B_ALIGN 7
+
+/* Buffer size and alignment */
+#define ZIP_CMD_QBUF_SIZE (8064 + 8)
+#define ZIP_CMD_QBUF_ALIGN 128
+#define ZIP_DATA_BUF_ALIGN 8
+
+/*
+ * There will be max of 2^20 zip cmds in the zip instruction queue.
+ * So no of zip Chunk buffers = ((2^20) / ((2*1024)/64))
+ */
+#define ZIP_MAX_CMD (1024 * 1024)
+#define ZIP_CMD_PER_BUF (ZIP_CMD_QBUF_SIZE / 64)
+#define ZIP_CMD_QBUF_MAX_CNT (1 * 1024)
+
+/* Data buffer size 64K for time being */
+#define ZIP_DATA_BUF_SIZE (64 * 1024)
+
+/* Number of data buffers */
+#define ZIP_DATA_BUF_CNT (32 * 1024)
+
+struct zip_registers {
+ char *reg_name;
+ u64 reg_offset;
+};
+
+/* ZIP Instruction Queue */
+struct zip_iq {
+ u64 *sw_head;
+ u64 *sw_tail;
+ u64 *hw_tail;
+ u64 done_cnt;
+ u64 pend_cnt;
+ u64 free_flag;
+
+ /* ZIP IQ lock */
+ spinlock_t lock;
+};
+
+/* ZIP Device */
+struct zip_device {
+ u32 index;
+ void __iomem *reg_base;
+ struct pci_dev *pdev;
+
+ /* Different ZIP Constants */
+ u64 depth;
+ u64 onfsize;
+ u64 ctxsize;
+
+ struct zip_iq iq[ZIP_MAX_NUM_QUEUES];
+};
+
+/* Prototypes */
+struct zip_device *zip_get_device(int node_id);
+int zip_get_node_id(void);
+int zip_get_zipeng_count(void);
+void zip_reg_write(u64 val, u64 __iomem *addr);
+u64 zip_reg_read(u64 __iomem *addr);
+void zip_update_cmd_bufs(struct zip_device *zip_dev, u32 queue);
+u32 zip_load_instr(union zip_inst_s *instr, struct zip_device *zip_dev);
+
+#endif /* ZIP_MAIN_H */
diff --git a/drivers/crypto/cavium/zip/zip_mem.c b/drivers/crypto/cavium/zip/zip_mem.c
new file mode 100644
index 0000000..cf1800f5
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_mem.c
@@ -0,0 +1,120 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include <linux/types.h>
+#include <linux/vmalloc.h>
+
+#include "common.h"
+
+/**
+ * zip_cmd_qbuf_alloc - Allocates a cmd buffer for ZIP Instruction Queue
+ * @zip: Pointer to zip device structure
+ * @q: Queue number to allocate bufffer to
+ * Return: 0 if successful, -ENOMEM otherwise
+ */
+int zip_cmd_qbuf_alloc(struct zip_device *zip, int q)
+{
+ zip_dbg_enter();
+
+ zip->iq[q].sw_head = (u64 *)__get_free_pages((GFP_KERNEL | GFP_DMA),
+ get_order(ZIP_CMD_QBUF_SIZE));
+
+ if (!zip->iq[q].sw_head)
+ return -ENOMEM;
+
+ memset(zip->iq[q].sw_head, 0, ZIP_CMD_QBUF_SIZE);
+
+ zip_dbg("cmd_qbuf_alloc[%d] Success : %p\n", q, zip->iq[q].sw_head);
+ zip_dbg_exit();
+ return 0;
+}
+
+/**
+ * zip_cmd_qbuf_free - Frees the cmd Queue buffer
+ * @zip: Pointer to zip device structure
+ * @q: Queue number to free buffer of
+ */
+void zip_cmd_qbuf_free(struct zip_device *zip, int q)
+{
+ zip_dbg("Freeing cmd_qbuf 0x%lx\n", zip->iq[q].sw_tail);
+
+ free_pages((u64)zip->iq[q].sw_tail, get_order(ZIP_CMD_QBUF_SIZE));
+}
+
+/**
+ * zip_data_buf_alloc - Allocates memory for a data bufffer
+ * @size: Size of the buffer to allocate
+ * Returns: Pointer to the buffer allocated
+ */
+u8 *zip_data_buf_alloc(u64 size)
+{
+ u8 *ptr;
+
+ zip_dbg_enter();
+
+ ptr = (u8 *)__get_free_pages((GFP_ATOMIC | GFP_DMA),
+ get_order(size));
+
+ if (!ptr)
+ return NULL;
+
+ memset(ptr, 0, size);
+
+ zip_dbg("Data buffer allocation success\n");
+ zip_dbg_exit();
+ return ptr;
+}
+
+/**
+ * zip_data_buf_free - Frees the memory of a data buffer
+ * @ptr: Pointer to the buffer
+ * @size: Buffer size
+ */
+void zip_data_buf_free(u8 *ptr, u64 size)
+{
+ zip_dbg("Freeing data buffer 0x%lx\n", ptr);
+
+ free_pages((u64)ptr, get_order(size));
+}
diff --git a/drivers/crypto/cavium/zip/zip_mem.h b/drivers/crypto/cavium/zip/zip_mem.h
new file mode 100644
index 0000000..23591d8
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_mem.h
@@ -0,0 +1,78 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_MEM_H__
+#define __ZIP_MEM_H__
+
+/**
+ * zip_cmd_qbuf_free - Frees the cmd Queue buffer
+ * @zip: Pointer to zip device structure
+ * @q: Queue nmber to free buffer of
+ */
+void zip_cmd_qbuf_free(struct zip_device *zip, int q);
+
+/**
+ * zip_cmd_qbuf_alloc - Allocates a Chunk/cmd buffer for ZIP Inst(cmd) Queue
+ * @zip: Pointer to zip device structure
+ * @q: Queue number to allocate bufffer to
+ * Return: 0 if successful, 1 otherwise
+ */
+int zip_cmd_qbuf_alloc(struct zip_device *zip, int q);
+
+/**
+ * zip_data_buf_alloc - Allocates memory for a data bufffer
+ * @size: Size of the buffer to allocate
+ * Returns: Pointer to the buffer allocated
+ */
+u8 *zip_data_buf_alloc(u64 size);
+
+/**
+ * zip_data_buf_free - Frees the memory of a data buffer
+ * @ptr: Pointer to the buffer
+ * @size: Buffer size
+ */
+void zip_data_buf_free(u8 *ptr, u64 size);
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_regs.h b/drivers/crypto/cavium/zip/zip_regs.h
new file mode 100644
index 0000000..ec913ad
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_regs.h
@@ -0,0 +1,1326 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2016 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_REGS_H__
+#define __ZIP_REGS_H__
+
+/*
+ * Configuration and status register (CSR) address and type definitions for
+ * Cavium ZIP.
+ */
+
+#include <linux/kern_levels.h>
+
+/**
+ * enum zip_comp_e - ZIP Completion Enumeration, enumerates the values of
+ * ZIP_ZRES_S[COMPCODE].
+ */
+enum zip_comp_e {
+ ZIP_COMP_E_BADCODE = 0x7,
+ ZIP_COMP_E_BADCODE2 = 0x8,
+ ZIP_COMP_E_DTRUNC = 0x2,
+ ZIP_COMP_E_FATAL = 0xb,
+ ZIP_COMP_E_ITRUNC = 0x4,
+ ZIP_COMP_E_NLEN = 0x6,
+ ZIP_COMP_E_NOTDONE = 0x0,
+ ZIP_COMP_E_PARITY = 0xa,
+ ZIP_COMP_E_RBLOCK = 0x5,
+ ZIP_COMP_E_STOP = 0x3,
+ ZIP_COMP_E_SUCCESS = 0x1,
+ ZIP_COMP_E_ZERO_LEN = 0x9,
+ ZIP_COMP_E_ENUM_LAST = 0xc,
+};
+
+/**
+ * enum zip_int_vec_e - ZIP MSI-X Vector Enumeration, enumerates the MSI-X
+ * interrupt vectors.
+ */
+enum zip_int_vec_e {
+ ZIP_INT_VEC_E_ECCE = 0x10,
+ ZIP_INT_VEC_E_FIFE = 0x11,
+ ZIP_INT_VEC_E_QUE0_DONE = 0x0,
+ ZIP_INT_VEC_E_QUE0_ERR = 0x8,
+ ZIP_INT_VEC_E_QUE1_DONE = 0x1,
+ ZIP_INT_VEC_E_QUE1_ERR = 0x9,
+ ZIP_INT_VEC_E_QUE2_DONE = 0x2,
+ ZIP_INT_VEC_E_QUE2_ERR = 0xa,
+ ZIP_INT_VEC_E_QUE3_DONE = 0x3,
+ ZIP_INT_VEC_E_QUE3_ERR = 0xb,
+ ZIP_INT_VEC_E_QUE4_DONE = 0x4,
+ ZIP_INT_VEC_E_QUE4_ERR = 0xc,
+ ZIP_INT_VEC_E_QUE5_DONE = 0x5,
+ ZIP_INT_VEC_E_QUE5_ERR = 0xd,
+ ZIP_INT_VEC_E_QUE6_DONE = 0x6,
+ ZIP_INT_VEC_E_QUE6_ERR = 0xe,
+ ZIP_INT_VEC_E_QUE7_DONE = 0x7,
+ ZIP_INT_VEC_E_QUE7_ERR = 0xf,
+ ZIP_INT_VEC_E_ENUM_LAST = 0x12,
+};
+
+/**
+ * union zip_zptr_addr_s - ZIP Generic Pointer Structure for ADDR.
+ *
+ * It is the generic format of pointers in ZIP_INST_S.
+ */
+union zip_zptr_addr_s {
+ u64 u_reg64;
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 addr : 49;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+
+};
+
+/**
+ * union zip_zptr_ctl_s - ZIP Generic Pointer Structure for CTL.
+ *
+ * It is the generic format of pointers in ZIP_INST_S.
+ */
+union zip_zptr_ctl_s {
+ u64 u_reg64;
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_112_127 : 16;
+ u64 length : 16;
+ u64 reserved_67_95 : 29;
+ u64 fw : 1;
+ u64 nc : 1;
+ u64 data_be : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 data_be : 1;
+ u64 nc : 1;
+ u64 fw : 1;
+ u64 reserved_67_95 : 29;
+ u64 length : 16;
+ u64 reserved_112_127 : 16;
+#endif
+ } s;
+};
+
+/**
+ * union zip_inst_s - ZIP Instruction Structure.
+ * Each ZIP instruction has 16 words (they are called IWORD0 to IWORD15 within
+ * the structure).
+ */
+union zip_inst_s {
+ u64 u_reg64[16];
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 doneint : 1;
+ u64 reserved_56_62 : 7;
+ u64 totaloutputlength : 24;
+ u64 reserved_27_31 : 5;
+ u64 exn : 3;
+ u64 reserved_23_23 : 1;
+ u64 exbits : 7;
+ u64 reserved_12_15 : 4;
+ u64 sf : 1;
+ u64 ss : 2;
+ u64 cc : 2;
+ u64 ef : 1;
+ u64 bf : 1;
+ u64 ce : 1;
+ u64 reserved_3_3 : 1;
+ u64 ds : 1;
+ u64 dg : 1;
+ u64 hg : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 hg : 1;
+ u64 dg : 1;
+ u64 ds : 1;
+ u64 reserved_3_3 : 1;
+ u64 ce : 1;
+ u64 bf : 1;
+ u64 ef : 1;
+ u64 cc : 2;
+ u64 ss : 2;
+ u64 sf : 1;
+ u64 reserved_12_15 : 4;
+ u64 exbits : 7;
+ u64 reserved_23_23 : 1;
+ u64 exn : 3;
+ u64 reserved_27_31 : 5;
+ u64 totaloutputlength : 24;
+ u64 reserved_56_62 : 7;
+ u64 doneint : 1;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 historylength : 16;
+ u64 reserved_96_111 : 16;
+ u64 adlercrc32 : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 adlercrc32 : 32;
+ u64 reserved_96_111 : 16;
+ u64 historylength : 16;
+#endif
+ union zip_zptr_addr_s ctx_ptr_addr;
+ union zip_zptr_ctl_s ctx_ptr_ctl;
+ union zip_zptr_addr_s his_ptr_addr;
+ union zip_zptr_ctl_s his_ptr_ctl;
+ union zip_zptr_addr_s inp_ptr_addr;
+ union zip_zptr_ctl_s inp_ptr_ctl;
+ union zip_zptr_addr_s out_ptr_addr;
+ union zip_zptr_ctl_s out_ptr_ctl;
+ union zip_zptr_addr_s res_ptr_addr;
+ union zip_zptr_ctl_s res_ptr_ctl;
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_817_831 : 15;
+ u64 wq_ptr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 wq_ptr : 49;
+ u64 reserved_817_831 : 15;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_882_895 : 14;
+ u64 tt : 2;
+ u64 reserved_874_879 : 6;
+ u64 grp : 10;
+ u64 tag : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 tag : 32;
+ u64 grp : 10;
+ u64 reserved_874_879 : 6;
+ u64 tt : 2;
+ u64 reserved_882_895 : 14;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_896_959 : 64;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reserved_896_959 : 64;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_960_1023 : 64;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reserved_960_1023 : 64;
+#endif
+ } s;
+};
+
+/**
+ * union zip_nptr_s - ZIP Instruction Next-Chunk-Buffer Pointer (NPTR)
+ * Structure
+ *
+ * ZIP_NPTR structure is used to chain all the zip instruction buffers
+ * together. ZIP instruction buffers are managed (allocated and released) by
+ * the software.
+ */
+union zip_nptr_s {
+ u64 u_reg64;
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 addr : 49;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+};
+
+/**
+ * union zip_zptr_s - ZIP Generic Pointer Structure.
+ *
+ * It is the generic format of pointers in ZIP_INST_S.
+ */
+union zip_zptr_s {
+ u64 u_reg64[2];
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 addr : 49;
+ u64 reserved_49_63 : 15;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_112_127 : 16;
+ u64 length : 16;
+ u64 reserved_67_95 : 29;
+ u64 fw : 1;
+ u64 nc : 1;
+ u64 data_be : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 data_be : 1;
+ u64 nc : 1;
+ u64 fw : 1;
+ u64 reserved_67_95 : 29;
+ u64 length : 16;
+ u64 reserved_112_127 : 16;
+#endif
+ } s;
+};
+
+/**
+ * union zip_zres_s - ZIP Result Structure
+ *
+ * The ZIP coprocessor writes the result structure after it completes the
+ * invocation. The result structure is exactly 24 bytes, and each invocation of
+ * the ZIP coprocessor produces exactly one result structure.
+ */
+union zip_zres_s {
+ u64 u_reg64[3];
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 crc32 : 32;
+ u64 adler32 : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 adler32 : 32;
+ u64 crc32 : 32;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 totalbyteswritten : 32;
+ u64 totalbytesread : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 totalbytesread : 32;
+ u64 totalbyteswritten : 32;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 totalbitsprocessed : 32;
+ u64 doneint : 1;
+ u64 reserved_155_158 : 4;
+ u64 exn : 3;
+ u64 reserved_151_151 : 1;
+ u64 exbits : 7;
+ u64 reserved_137_143 : 7;
+ u64 ef : 1;
+
+ volatile u64 compcode : 8;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+
+ volatile u64 compcode : 8;
+ u64 ef : 1;
+ u64 reserved_137_143 : 7;
+ u64 exbits : 7;
+ u64 reserved_151_151 : 1;
+ u64 exn : 3;
+ u64 reserved_155_158 : 4;
+ u64 doneint : 1;
+ u64 totalbitsprocessed : 32;
+#endif
+ } s;
+};
+
+/**
+ * union zip_cmd_ctl - Structure representing the register that controls
+ * clock and reset.
+ */
+union zip_cmd_ctl {
+ u64 u_reg64;
+ struct zip_cmd_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_2_63 : 62;
+ u64 forceclk : 1;
+ u64 reset : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reset : 1;
+ u64 forceclk : 1;
+ u64 reserved_2_63 : 62;
+#endif
+ } s;
+};
+
+#define ZIP_CMD_CTL 0x0ull
+
+/**
+ * union zip_constants - Data structure representing the register that contains
+ * all of the current implementation-related parameters of the zip core in this
+ * chip.
+ */
+union zip_constants {
+ u64 u_reg64;
+ struct zip_constants_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 nexec : 8;
+ u64 reserved_49_55 : 7;
+ u64 syncflush_capable : 1;
+ u64 depth : 16;
+ u64 onfsize : 12;
+ u64 ctxsize : 12;
+ u64 reserved_1_7 : 7;
+ u64 disabled : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 disabled : 1;
+ u64 reserved_1_7 : 7;
+ u64 ctxsize : 12;
+ u64 onfsize : 12;
+ u64 depth : 16;
+ u64 syncflush_capable : 1;
+ u64 reserved_49_55 : 7;
+ u64 nexec : 8;
+#endif
+ } s;
+};
+
+#define ZIP_CONSTANTS 0x00A0ull
+
+/**
+ * union zip_corex_bist_status - Represents registers which have the BIST
+ * status of memories in zip cores.
+ *
+ * Each bit is the BIST result of an individual memory
+ * (per bit, 0 = pass and 1 = fail).
+ */
+union zip_corex_bist_status {
+ u64 u_reg64;
+ struct zip_corex_bist_status_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_53_63 : 11;
+ u64 bstatus : 53;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 bstatus : 53;
+ u64 reserved_53_63 : 11;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_COREX_BIST_STATUS(u64 param1)
+{
+ if (((param1 <= 1)))
+ return 0x0520ull + (param1 & 1) * 0x8ull;
+ pr_err("ZIP_COREX_BIST_STATUS: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_ctl_bist_status - Represents register that has the BIST status of
+ * memories in ZIP_CTL (instruction buffer, G/S pointer FIFO, input data
+ * buffer, output data buffers).
+ *
+ * Each bit is the BIST result of an individual memory
+ * (per bit, 0 = pass and 1 = fail).
+ */
+union zip_ctl_bist_status {
+ u64 u_reg64;
+ struct zip_ctl_bist_status_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_9_63 : 55;
+ u64 bstatus : 9;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 bstatus : 9;
+ u64 reserved_9_63 : 55;
+#endif
+ } s;
+};
+
+#define ZIP_CTL_BIST_STATUS 0x0510ull
+
+/**
+ * union zip_ctl_cfg - Represents the register that controls the behavior of
+ * the ZIP DMA engines.
+ *
+ * It is recommended to keep default values for normal operation. Changing the
+ * values of the fields may be useful for diagnostics.
+ */
+union zip_ctl_cfg {
+ u64 u_reg64;
+ struct zip_ctl_cfg_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_52_63 : 12;
+ u64 ildf : 4;
+ u64 reserved_36_47 : 12;
+ u64 drtf : 4;
+ u64 reserved_27_31 : 5;
+ u64 stcf : 3;
+ u64 reserved_19_23 : 5;
+ u64 ldf : 3;
+ u64 reserved_2_15 : 14;
+ u64 busy : 1;
+ u64 reserved_0_0 : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reserved_0_0 : 1;
+ u64 busy : 1;
+ u64 reserved_2_15 : 14;
+ u64 ldf : 3;
+ u64 reserved_19_23 : 5;
+ u64 stcf : 3;
+ u64 reserved_27_31 : 5;
+ u64 drtf : 4;
+ u64 reserved_36_47 : 12;
+ u64 ildf : 4;
+ u64 reserved_52_63 : 12;
+#endif
+ } s;
+};
+
+#define ZIP_CTL_CFG 0x0560ull
+
+/**
+ * union zip_dbg_corex_inst - Represents the registers that reflect the status
+ * of the current instruction that the ZIP core is executing or has executed.
+ *
+ * These registers are only for debug use.
+ */
+union zip_dbg_corex_inst {
+ u64 u_reg64;
+ struct zip_dbg_corex_inst_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 busy : 1;
+ u64 reserved_35_62 : 28;
+ u64 qid : 3;
+ u64 iid : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 iid : 32;
+ u64 qid : 3;
+ u64 reserved_35_62 : 28;
+ u64 busy : 1;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_DBG_COREX_INST(u64 param1)
+{
+ if (((param1 <= 1)))
+ return 0x0640ull + (param1 & 1) * 0x8ull;
+ pr_err("ZIP_DBG_COREX_INST: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_dbg_corex_sta - Represents registers that reflect the status of
+ * the zip cores.
+ *
+ * They are for debug use only.
+ */
+union zip_dbg_corex_sta {
+ u64 u_reg64;
+ struct zip_dbg_corex_sta_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 busy : 1;
+ u64 reserved_37_62 : 26;
+ u64 ist : 5;
+ u64 nie : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 nie : 32;
+ u64 ist : 5;
+ u64 reserved_37_62 : 26;
+ u64 busy : 1;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_DBG_COREX_STA(u64 param1)
+{
+ if (((param1 <= 1)))
+ return 0x0680ull + (param1 & 1) * 0x8ull;
+ pr_err("ZIP_DBG_COREX_STA: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_dbg_quex_sta - Represets registers that reflect status of the zip
+ * instruction queues.
+ *
+ * They are for debug use only.
+ */
+union zip_dbg_quex_sta {
+ u64 u_reg64;
+ struct zip_dbg_quex_sta_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 busy : 1;
+ u64 reserved_56_62 : 7;
+ u64 rqwc : 24;
+ u64 nii : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 nii : 32;
+ u64 rqwc : 24;
+ u64 reserved_56_62 : 7;
+ u64 busy : 1;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_DBG_QUEX_STA(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1800ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_DBG_QUEX_STA: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_ecc_ctl - Represents the register that enables ECC for each
+ * individual internal memory that requires ECC.
+ *
+ * For debug purpose, it can also flip one or two bits in the ECC data.
+ */
+union zip_ecc_ctl {
+ u64 u_reg64;
+ struct zip_ecc_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_19_63 : 45;
+ u64 vmem_cdis : 1;
+ u64 vmem_fs : 2;
+ u64 reserved_15_15 : 1;
+ u64 idf1_cdis : 1;
+ u64 idf1_fs : 2;
+ u64 reserved_11_11 : 1;
+ u64 idf0_cdis : 1;
+ u64 idf0_fs : 2;
+ u64 reserved_7_7 : 1;
+ u64 gspf_cdis : 1;
+ u64 gspf_fs : 2;
+ u64 reserved_3_3 : 1;
+ u64 iqf_cdis : 1;
+ u64 iqf_fs : 2;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 iqf_fs : 2;
+ u64 iqf_cdis : 1;
+ u64 reserved_3_3 : 1;
+ u64 gspf_fs : 2;
+ u64 gspf_cdis : 1;
+ u64 reserved_7_7 : 1;
+ u64 idf0_fs : 2;
+ u64 idf0_cdis : 1;
+ u64 reserved_11_11 : 1;
+ u64 idf1_fs : 2;
+ u64 idf1_cdis : 1;
+ u64 reserved_15_15 : 1;
+ u64 vmem_fs : 2;
+ u64 vmem_cdis : 1;
+ u64 reserved_19_63 : 45;
+#endif
+ } s;
+};
+
+#define ZIP_ECC_CTL 0x0568ull
+
+/* NCB - zip_ecce_ena_w1c */
+union zip_ecce_ena_w1c {
+ u64 u_reg64;
+ struct zip_ecce_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_ENA_W1C 0x0598ull
+
+/* NCB - zip_ecce_ena_w1s */
+union zip_ecce_ena_w1s {
+ u64 u_reg64;
+ struct zip_ecce_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_ENA_W1S 0x0590ull
+
+/**
+ * union zip_ecce_int - Represents the register that contains the status of the
+ * ECC interrupt sources.
+ */
+union zip_ecce_int {
+ u64 u_reg64;
+ struct zip_ecce_int_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_INT 0x0580ull
+
+/* NCB - zip_ecce_int_w1s */
+union zip_ecce_int_w1s {
+ u64 u_reg64;
+ struct zip_ecce_int_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_INT_W1S 0x0588ull
+
+/* NCB - zip_fife_ena_w1c */
+union zip_fife_ena_w1c {
+ u64 u_reg64;
+ struct zip_fife_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_ENA_W1C 0x0090ull
+
+/* NCB - zip_fife_ena_w1s */
+union zip_fife_ena_w1s {
+ u64 u_reg64;
+ struct zip_fife_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_ENA_W1S 0x0088ull
+
+/* NCB - zip_fife_int */
+union zip_fife_int {
+ u64 u_reg64;
+ struct zip_fife_int_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_INT 0x0078ull
+
+/* NCB - zip_fife_int_w1s */
+union zip_fife_int_w1s {
+ u64 u_reg64;
+ struct zip_fife_int_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_INT_W1S 0x0080ull
+
+/**
+ * union zip_msix_pbax - Represents the register that is the MSI-X PBA table
+ *
+ * The bit number is indexed by the ZIP_INT_VEC_E enumeration.
+ */
+union zip_msix_pbax {
+ u64 u_reg64;
+ struct zip_msix_pbax_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 pend : 64;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 pend : 64;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_MSIX_PBAX(u64 param1)
+{
+ if (((param1 == 0)))
+ return 0x0000838000FF0000ull;
+ pr_err("ZIP_MSIX_PBAX: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_msix_vecx_addr - Represents the register that is the MSI-X vector
+ * table, indexed by the ZIP_INT_VEC_E enumeration.
+ */
+union zip_msix_vecx_addr {
+ u64 u_reg64;
+ struct zip_msix_vecx_addr_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 47;
+ u64 reserved_1_1 : 1;
+ u64 secvec : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 secvec : 1;
+ u64 reserved_1_1 : 1;
+ u64 addr : 47;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_MSIX_VECX_ADDR(u64 param1)
+{
+ if (((param1 <= 17)))
+ return 0x0000838000F00000ull + (param1 & 31) * 0x10ull;
+ pr_err("ZIP_MSIX_VECX_ADDR: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_msix_vecx_ctl - Represents the register that is the MSI-X vector
+ * table, indexed by the ZIP_INT_VEC_E enumeration.
+ */
+union zip_msix_vecx_ctl {
+ u64 u_reg64;
+ struct zip_msix_vecx_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_33_63 : 31;
+ u64 mask : 1;
+ u64 reserved_20_31 : 12;
+ u64 data : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 data : 20;
+ u64 reserved_20_31 : 12;
+ u64 mask : 1;
+ u64 reserved_33_63 : 31;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_MSIX_VECX_CTL(u64 param1)
+{
+ if (((param1 <= 17)))
+ return 0x0000838000F00008ull + (param1 & 31) * 0x10ull;
+ pr_err("ZIP_MSIX_VECX_CTL: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done - Represents the registers that contain the per-queue
+ * instruction done count.
+ */
+union zip_quex_done {
+ u64 u_reg64;
+ struct zip_quex_done_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_20_63 : 44;
+ u64 done : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done : 20;
+ u64 reserved_20_63 : 44;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_ack - Represents the registers on write to which will
+ * decrement the per-queue instructiona done count.
+ */
+union zip_quex_done_ack {
+ u64 u_reg64;
+ struct zip_quex_done_ack_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_20_63 : 44;
+ u64 done_ack : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done_ack : 20;
+ u64 reserved_20_63 : 44;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_ACK(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2200ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_ACK: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_ena_w1c - Represents the register which when written
+ * 1 to will disable the DONEINT interrupt for the queue.
+ */
+union zip_quex_done_ena_w1c {
+ u64 u_reg64;
+ struct zip_quex_done_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_1_63 : 63;
+ u64 done_ena : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done_ena : 1;
+ u64 reserved_1_63 : 63;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_ENA_W1C(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2600ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_ENA_W1C: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_ena_w1s - Represents the register that when written 1 to
+ * will enable the DONEINT interrupt for the queue.
+ */
+union zip_quex_done_ena_w1s {
+ u64 u_reg64;
+ struct zip_quex_done_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_1_63 : 63;
+ u64 done_ena : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done_ena : 1;
+ u64 reserved_1_63 : 63;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_ENA_W1S(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2400ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_ENA_W1S: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_wait - Represents the register that specifies the per
+ * queue interrupt coalescing settings.
+ */
+union zip_quex_done_wait {
+ u64 u_reg64;
+ struct zip_quex_done_wait_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_48_63 : 16;
+ u64 time_wait : 16;
+ u64 reserved_20_31 : 12;
+ u64 num_wait : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 num_wait : 20;
+ u64 reserved_20_31 : 12;
+ u64 time_wait : 16;
+ u64 reserved_48_63 : 16;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_WAIT(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2800ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_WAIT: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_doorbell - Represents doorbell registers for the ZIP
+ * instruction queues.
+ */
+union zip_quex_doorbell {
+ u64 u_reg64;
+ struct zip_quex_doorbell_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_20_63 : 44;
+ u64 dbell_cnt : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dbell_cnt : 20;
+ u64 reserved_20_63 : 44;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DOORBELL(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x4000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DOORBELL: %llu\n", param1);
+ return 0;
+}
+
+union zip_quex_err_ena_w1c {
+ u64 u_reg64;
+ struct zip_quex_err_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_ENA_W1C(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3600ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_ENA_W1C: %llu\n", param1);
+ return 0;
+}
+
+union zip_quex_err_ena_w1s {
+ u64 u_reg64;
+ struct zip_quex_err_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_ENA_W1S(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3400ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_ENA_W1S: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_err_int - Represents registers that contain the per-queue
+ * error interrupts.
+ */
+union zip_quex_err_int {
+ u64 u_reg64;
+ struct zip_quex_err_int_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_INT(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_INT: %llu\n", param1);
+ return 0;
+}
+
+/* NCB - zip_que#_err_int_w1s */
+union zip_quex_err_int_w1s {
+ u64 u_reg64;
+ struct zip_quex_err_int_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_INT_W1S(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3200ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_INT_W1S: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_gcfg - Represents the registers that reflect status of the
+ * zip instruction queues,debug use only.
+ */
+union zip_quex_gcfg {
+ u64 u_reg64;
+ struct zip_quex_gcfg_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_4_63 : 60;
+ u64 iqb_ldwb : 1;
+ u64 cbw_sty : 1;
+ u64 l2ld_cmd : 2;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 l2ld_cmd : 2;
+ u64 cbw_sty : 1;
+ u64 iqb_ldwb : 1;
+ u64 reserved_4_63 : 60;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_GCFG(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1A00ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_GCFG: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_map - Represents the registers that control how each
+ * instruction queue maps to zip cores.
+ */
+union zip_quex_map {
+ u64 u_reg64;
+ struct zip_quex_map_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_2_63 : 62;
+ u64 zce : 2;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 zce : 2;
+ u64 reserved_2_63 : 62;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_MAP(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1400ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_MAP: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_sbuf_addr - Represents the registers that set the buffer
+ * parameters for the instruction queues.
+ *
+ * When quiescent (i.e. outstanding doorbell count is 0), it is safe to rewrite
+ * this register to effectively reset the command buffer state machine.
+ * These registers must be programmed after SW programs the corresponding
+ * ZIP_QUE(0..7)_SBUF_CTL.
+ */
+union zip_quex_sbuf_addr {
+ u64 u_reg64;
+ struct zip_quex_sbuf_addr_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 ptr : 42;
+ u64 off : 7;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 off : 7;
+ u64 ptr : 42;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_SBUF_ADDR(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_SBUF_ADDR: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_sbuf_ctl - Represents the registers that set the buffer
+ * parameters for the instruction queues.
+ *
+ * When quiescent (i.e. outstanding doorbell count is 0), it is safe to rewrite
+ * this register to effectively reset the command buffer state machine.
+ * These registers must be programmed before SW programs the corresponding
+ * ZIP_QUE(0..7)_SBUF_ADDR.
+ */
+union zip_quex_sbuf_ctl {
+ u64 u_reg64;
+ struct zip_quex_sbuf_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_45_63 : 19;
+ u64 size : 13;
+ u64 inst_be : 1;
+ u64 reserved_24_30 : 7;
+ u64 stream_id : 8;
+ u64 reserved_12_15 : 4;
+ u64 aura : 12;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 aura : 12;
+ u64 reserved_12_15 : 4;
+ u64 stream_id : 8;
+ u64 reserved_24_30 : 7;
+ u64 inst_be : 1;
+ u64 size : 13;
+ u64 reserved_45_63 : 19;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_SBUF_CTL(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1200ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_SBUF_CTL: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_que_ena - Represents queue enable register
+ *
+ * If a queue is disabled, ZIP_CTL stops fetching instructions from the queue.
+ */
+union zip_que_ena {
+ u64 u_reg64;
+ struct zip_que_ena_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_8_63 : 56;
+ u64 ena : 8;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 ena : 8;
+ u64 reserved_8_63 : 56;
+#endif
+ } s;
+};
+
+#define ZIP_QUE_ENA 0x0500ull
+
+/**
+ * union zip_que_pri - Represents the register that defines the priority
+ * between instruction queues.
+ */
+union zip_que_pri {
+ u64 u_reg64;
+ struct zip_que_pri_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_8_63 : 56;
+ u64 pri : 8;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 pri : 8;
+ u64 reserved_8_63 : 56;
+#endif
+ } s;
+};
+
+#define ZIP_QUE_PRI 0x0508ull
+
+/**
+ * union zip_throttle - Represents the register that controls the maximum
+ * number of in-flight X2I data fetch transactions.
+ *
+ * Writing 0 to this register causes the ZIP module to temporarily suspend NCB
+ * accesses; it is not recommended for normal operation, but may be useful for
+ * diagnostics.
+ */
+union zip_throttle {
+ u64 u_reg64;
+ struct zip_throttle_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_6_63 : 58;
+ u64 ld_infl : 6;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 ld_infl : 6;
+ u64 reserved_6_63 : 58;
+#endif
+ } s;
+};
+
+#define ZIP_THROTTLE 0x0010ull
+
+#endif /* _CSRS_ZIP__ */
--
2.9.0.rc0.21.g7777322
^ permalink raw reply related
* Re: [PATCH v6 2/2] crypto: add virtio-crypto driver
From: Herbert Xu @ 2016-12-12 10:54 UTC (permalink / raw)
To: Gonglei (Arei)
Cc: virtio-dev@lists.oasis-open.org, Huangweidong (C),
Claudio Fontana, mst@redhat.com, qemu-devel@nongnu.org,
Hanweidong (Randy), Luonengjun, linux-kernel@vger.kernel.org,
Wanzongshun (Vincent), virtualization@lists.linux-foundation.org,
Xuquan (Quan Xu), linux-crypto@vger.kernel.org,
stefanha@redhat.com, Zhoujian (jay, Euler), longpeng,
davem@davemloft.net, Wubin (H), "arei.gonglei@hotmail.co
In-Reply-To: <33183CC9F5247A488A2544077AF19020DA15A07C@DGGEMA505-MBX.china.huawei.com>
On Mon, Dec 12, 2016 at 06:25:12AM +0000, Gonglei (Arei) wrote:
> Hi, Michael & Herbert
>
> Because the virtio-crypto device emulation had been in QEMU 2.8,
> would you please merge the virtio-crypto driver for 4.10 if no other
> comments? If so, Miachel pls ack and/or review the patch, then
> Herbert will take it (I asked him last week). Thank you!
>
> Ps: Note on 4.10 merge window timing from Linus
> https://lkml.org/lkml/2016/12/7/506
>
> Dec 23rd is the deadline for 4.10 merge window.
Sorry but it's too late for 4.10. It needed to have been in my
tree before the merge window opened to make it for this cycle.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* RE: [PATCH v6 2/2] crypto: add virtio-crypto driver
From: Gonglei (Arei) @ 2016-12-12 6:25 UTC (permalink / raw)
To: Gonglei (Arei), linux-kernel@vger.kernel.org,
qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org,
virtualization@lists.linux-foundation.org,
linux-crypto@vger.kernel.org
Cc: Luonengjun, mst@redhat.com, stefanha@redhat.com, Huangweidong (C),
Wubin (H), xin.zeng@intel.com, Claudio Fontana,
herbert@gondor.apana.org.au, pasic@linux.vnet.ibm.com,
davem@davemloft.net, Zhoujian (jay, Euler), Hanweidong (Randy),
arei.gonglei@hotmail.com, cornelia.huck@de.ibm.com,
Xuquan (Quan Xu), longpeng, Wanzongshun (Vincent)
In-Reply-To: <1481171829-116496-3-git-send-email-arei.gonglei@huawei.com>
Hi, Michael & Herbert
Because the virtio-crypto device emulation had been in QEMU 2.8,
would you please merge the virtio-crypto driver for 4.10 if no other
comments? If so, Miachel pls ack and/or review the patch, then
Herbert will take it (I asked him last week). Thank you!
Ps: Note on 4.10 merge window timing from Linus
https://lkml.org/lkml/2016/12/7/506
Dec 23rd is the deadline for 4.10 merge window.
Regards,
-Gonglei
> -----Original Message-----
> From: Gonglei (Arei)
> Sent: Thursday, December 08, 2016 12:37 PM
> Subject: [PATCH v6 2/2] crypto: add virtio-crypto driver
>
> This patch introduces virtio-crypto driver for Linux Kernel.
>
> The virtio crypto device is a virtual cryptography device
> as well as a kind of virtual hardware accelerator for
> virtual machines. The encryption anddecryption requests
> are placed in the data queue and are ultimately handled by
> thebackend crypto accelerators. The second queue is the
> control queue used to create or destroy sessions for
> symmetric algorithms and will control some advanced features
> in the future. The virtio crypto device provides the following
> cryptoservices: CIPHER, MAC, HASH, and AEAD.
>
> For more information about virtio-crypto device, please see:
> http://qemu-project.org/Features/VirtioCrypto
>
> CC: Michael S. Tsirkin <mst@redhat.com>
> CC: Cornelia Huck <cornelia.huck@de.ibm.com>
> CC: Stefan Hajnoczi <stefanha@redhat.com>
> CC: Herbert Xu <herbert@gondor.apana.org.au>
> CC: Halil Pasic <pasic@linux.vnet.ibm.com>
> CC: David S. Miller <davem@davemloft.net>
> CC: Zeng Xin <xin.zeng@intel.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> MAINTAINERS | 9 +
> drivers/crypto/Kconfig | 2 +
> drivers/crypto/Makefile | 1 +
> drivers/crypto/virtio/Kconfig | 10 +
> drivers/crypto/virtio/Makefile | 5 +
> drivers/crypto/virtio/virtio_crypto_algs.c | 541
> +++++++++++++++++++++++++++
> drivers/crypto/virtio/virtio_crypto_common.h | 122 ++++++
> drivers/crypto/virtio/virtio_crypto_core.c | 464
> +++++++++++++++++++++++
> drivers/crypto/virtio/virtio_crypto_mgr.c | 264 +++++++++++++
> include/uapi/linux/Kbuild | 1 +
> include/uapi/linux/virtio_crypto.h | 450
> ++++++++++++++++++++++
> include/uapi/linux/virtio_ids.h | 1 +
> 12 files changed, 1870 insertions(+)
> create mode 100644 drivers/crypto/virtio/Kconfig
> create mode 100644 drivers/crypto/virtio/Makefile
> create mode 100644 drivers/crypto/virtio/virtio_crypto_algs.c
> create mode 100644 drivers/crypto/virtio/virtio_crypto_common.h
> create mode 100644 drivers/crypto/virtio/virtio_crypto_core.c
> create mode 100644 drivers/crypto/virtio/virtio_crypto_mgr.c
> create mode 100644 include/uapi/linux/virtio_crypto.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index ad9b965..cccaaf0 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -12810,6 +12810,7 @@ F: drivers/net/virtio_net.c
> F: drivers/block/virtio_blk.c
> F: include/linux/virtio_*.h
> F: include/uapi/linux/virtio_*.h
> +F: drivers/crypto/virtio/
>
> VIRTIO DRIVERS FOR S390
> M: Christian Borntraeger <borntraeger@de.ibm.com>
> @@ -12846,6 +12847,14 @@ S: Maintained
> F: drivers/virtio/virtio_input.c
> F: include/uapi/linux/virtio_input.h
>
> +VIRTIO CRYPTO DRIVER
> +M: Gonglei <arei.gonglei@huawei.com>
> +L: virtualization@lists.linux-foundation.org
> +L: linux-crypto@vger.kernel.org
> +S: Maintained
> +F: drivers/crypto/virtio/
> +F: include/uapi/linux/virtio_crypto.h
> +
> VIA RHINE NETWORK DRIVER
> S: Orphan
> F: drivers/net/ethernet/via/via-rhine.c
> diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
> index 4d2b81f..7956478 100644
> --- a/drivers/crypto/Kconfig
> +++ b/drivers/crypto/Kconfig
> @@ -555,4 +555,6 @@ config CRYPTO_DEV_ROCKCHIP
>
> source "drivers/crypto/chelsio/Kconfig"
>
> +source "drivers/crypto/virtio/Kconfig"
> +
> endif # CRYPTO_HW
> diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
> index ad7250f..bc53cb8 100644
> --- a/drivers/crypto/Makefile
> +++ b/drivers/crypto/Makefile
> @@ -32,3 +32,4 @@ obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
> obj-$(CONFIG_CRYPTO_DEV_SUN4I_SS) += sunxi-ss/
> obj-$(CONFIG_CRYPTO_DEV_ROCKCHIP) += rockchip/
> obj-$(CONFIG_CRYPTO_DEV_CHELSIO) += chelsio/
> +obj-$(CONFIG_CRYPTO_DEV_VIRTIO) += virtio/
> diff --git a/drivers/crypto/virtio/Kconfig b/drivers/crypto/virtio/Kconfig
> new file mode 100644
> index 0000000..d80f733
> --- /dev/null
> +++ b/drivers/crypto/virtio/Kconfig
> @@ -0,0 +1,10 @@
> +config CRYPTO_DEV_VIRTIO
> + tristate "VirtIO crypto driver"
> + depends on VIRTIO
> + select CRYPTO_AEAD
> + select CRYPTO_AUTHENC
> + select CRYPTO_BLKCIPHER
> + default m
> + help
> + This driver provides support for virtio crypto device. If you
> + choose 'M' here, this module will be called virtio_crypto.
> diff --git a/drivers/crypto/virtio/Makefile b/drivers/crypto/virtio/Makefile
> new file mode 100644
> index 0000000..dd342c9
> --- /dev/null
> +++ b/drivers/crypto/virtio/Makefile
> @@ -0,0 +1,5 @@
> +obj-$(CONFIG_CRYPTO_DEV_VIRTIO) += virtio_crypto.o
> +virtio_crypto-objs := \
> + virtio_crypto_algs.o \
> + virtio_crypto_mgr.o \
> + virtio_crypto_core.o
> diff --git a/drivers/crypto/virtio/virtio_crypto_algs.c
> b/drivers/crypto/virtio/virtio_crypto_algs.c
> new file mode 100644
> index 0000000..7130dc9
> --- /dev/null
> +++ b/drivers/crypto/virtio/virtio_crypto_algs.c
> @@ -0,0 +1,541 @@
> + /* Algorithms supported by virtio crypto device
> + *
> + * Authors: Gonglei <arei.gonglei@huawei.com>
> + *
> + * Copyright 2016 HUAWEI TECHNOLOGIES CO., LTD.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/scatterlist.h>
> +#include <crypto/algapi.h>
> +#include <linux/err.h>
> +#include <crypto/scatterwalk.h>
> +#include <linux/atomic.h>
> +
> +#include <uapi/linux/virtio_crypto.h>
> +#include "virtio_crypto_common.h"
> +
> +/*
> + * The algs_lock protects the below global virtio_crypto_active_devs
> + * and crypto algorithms registion.
> + */
> +static DEFINE_MUTEX(algs_lock);
> +static unsigned int virtio_crypto_active_devs;
> +
> +static u64 virtio_crypto_alg_sg_nents_length(struct scatterlist *sg)
> +{
> + u64 total = 0;
> +
> + for (total = 0; sg; sg = sg_next(sg))
> + total += sg->length;
> +
> + return total;
> +}
> +
> +static int
> +virtio_crypto_alg_validate_key(int key_len, uint32_t *alg)
> +{
> + switch (key_len) {
> + case AES_KEYSIZE_128:
> + case AES_KEYSIZE_192:
> + case AES_KEYSIZE_256:
> + *alg = VIRTIO_CRYPTO_CIPHER_AES_CBC;
> + break;
> + default:
> + pr_err("virtio_crypto: Unsupported key length: %d\n",
> + key_len);
> + return -EINVAL;
> + }
> + return 0;
> +}
> +
> +static int virtio_crypto_alg_ablkcipher_init_session(
> + struct virtio_crypto_ablkcipher_ctx *ctx,
> + uint32_t alg, const uint8_t *key,
> + unsigned int keylen,
> + int encrypt)
> +{
> + struct scatterlist outhdr, key_sg, inhdr, *sgs[3];
> + unsigned int tmp;
> + struct virtio_crypto *vcrypto = ctx->vcrypto;
> + int op = encrypt ? VIRTIO_CRYPTO_OP_ENCRYPT :
> VIRTIO_CRYPTO_OP_DECRYPT;
> + int err;
> + unsigned int num_out = 0, num_in = 0;
> +
> + /*
> + * Avoid to do DMA from the stack, switch to using
> + * dynamically-allocated for the key
> + */
> + uint8_t *cipher_key = kmalloc(keylen, GFP_ATOMIC);
> +
> + if (!cipher_key)
> + return -ENOMEM;
> +
> + memcpy(cipher_key, key, keylen);
> +
> + spin_lock(&vcrypto->ctrl_lock);
> + /* Pad ctrl header */
> + vcrypto->ctrl.header.opcode =
> + cpu_to_le32(VIRTIO_CRYPTO_CIPHER_CREATE_SESSION);
> + vcrypto->ctrl.header.algo = cpu_to_le32(alg);
> + /* Set the default dataqueue id to 0 */
> + vcrypto->ctrl.header.queue_id = 0;
> +
> + vcrypto->input.status = cpu_to_le32(VIRTIO_CRYPTO_ERR);
> + /* Pad cipher's parameters */
> + vcrypto->ctrl.u.sym_create_session.op_type =
> + cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER);
> + vcrypto->ctrl.u.sym_create_session.u.cipher.para.algo =
> + vcrypto->ctrl.header.algo;
> + vcrypto->ctrl.u.sym_create_session.u.cipher.para.keylen =
> + cpu_to_le32(keylen);
> + vcrypto->ctrl.u.sym_create_session.u.cipher.para.op =
> + cpu_to_le32(op);
> +
> + sg_init_one(&outhdr, &vcrypto->ctrl, sizeof(vcrypto->ctrl));
> + sgs[num_out++] = &outhdr;
> +
> + /* Set key */
> + sg_init_one(&key_sg, cipher_key, keylen);
> + sgs[num_out++] = &key_sg;
> +
> + /* Return status and session id back */
> + sg_init_one(&inhdr, &vcrypto->input, sizeof(vcrypto->input));
> + sgs[num_out + num_in++] = &inhdr;
> +
> + err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out,
> + num_in, vcrypto, GFP_ATOMIC);
> + if (err < 0) {
> + spin_unlock(&vcrypto->ctrl_lock);
> + kzfree(cipher_key);
> + return err;
> + }
> + virtqueue_kick(vcrypto->ctrl_vq);
> +
> + /*
> + * Trapping into the hypervisor, so the request should be
> + * handled immediately.
> + */
> + while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) &&
> + !virtqueue_is_broken(vcrypto->ctrl_vq))
> + cpu_relax();
> +
> + if (le32_to_cpu(vcrypto->input.status) != VIRTIO_CRYPTO_OK) {
> + spin_unlock(&vcrypto->ctrl_lock);
> + pr_err("virtio_crypto: Create session failed status: %u\n",
> + le32_to_cpu(vcrypto->input.status));
> + kzfree(cipher_key);
> + return -EINVAL;
> + }
> +
> + if (encrypt)
> + ctx->enc_sess_info.session_id =
> + le64_to_cpu(vcrypto->input.session_id);
> + else
> + ctx->dec_sess_info.session_id =
> + le64_to_cpu(vcrypto->input.session_id);
> +
> + spin_unlock(&vcrypto->ctrl_lock);
> +
> + kzfree(cipher_key);
> + return 0;
> +}
> +
> +static int virtio_crypto_alg_ablkcipher_close_session(
> + struct virtio_crypto_ablkcipher_ctx *ctx,
> + int encrypt)
> +{
> + struct scatterlist outhdr, status_sg, *sgs[2];
> + unsigned int tmp;
> + struct virtio_crypto_destroy_session_req *destroy_session;
> + struct virtio_crypto *vcrypto = ctx->vcrypto;
> + int err;
> + unsigned int num_out = 0, num_in = 0;
> +
> + spin_lock(&vcrypto->ctrl_lock);
> + vcrypto->ctrl_status.status = VIRTIO_CRYPTO_ERR;
> + /* Pad ctrl header */
> + vcrypto->ctrl.header.opcode =
> + cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION);
> + /* Set the default virtqueue id to 0 */
> + vcrypto->ctrl.header.queue_id = 0;
> +
> + destroy_session = &vcrypto->ctrl.u.destroy_session;
> +
> + if (encrypt)
> + destroy_session->session_id =
> + cpu_to_le64(ctx->enc_sess_info.session_id);
> + else
> + destroy_session->session_id =
> + cpu_to_le64(ctx->dec_sess_info.session_id);
> +
> + sg_init_one(&outhdr, &vcrypto->ctrl, sizeof(vcrypto->ctrl));
> + sgs[num_out++] = &outhdr;
> +
> + /* Return status and session id back */
> + sg_init_one(&status_sg, &vcrypto->ctrl_status.status,
> + sizeof(vcrypto->ctrl_status.status));
> + sgs[num_out + num_in++] = &status_sg;
> +
> + err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out,
> + num_in, vcrypto, GFP_ATOMIC);
> + if (err < 0) {
> + spin_unlock(&vcrypto->ctrl_lock);
> + return err;
> + }
> + virtqueue_kick(vcrypto->ctrl_vq);
> +
> + while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) &&
> + !virtqueue_is_broken(vcrypto->ctrl_vq))
> + cpu_relax();
> +
> + if (vcrypto->ctrl_status.status != VIRTIO_CRYPTO_OK) {
> + spin_unlock(&vcrypto->ctrl_lock);
> + pr_err("virtio_crypto: Close session failed status: %u, session_id:
> 0x%llx\n",
> + vcrypto->ctrl_status.status,
> + destroy_session->session_id);
> +
> + return -EINVAL;
> + }
> + spin_unlock(&vcrypto->ctrl_lock);
> +
> + return 0;
> +}
> +
> +static int virtio_crypto_alg_ablkcipher_init_sessions(
> + struct virtio_crypto_ablkcipher_ctx *ctx,
> + const uint8_t *key, unsigned int keylen)
> +{
> + uint32_t alg;
> + int ret;
> + struct virtio_crypto *vcrypto = ctx->vcrypto;
> +
> + if (keylen > vcrypto->max_cipher_key_len) {
> + pr_err("virtio_crypto: the key is too long\n");
> + goto bad_key;
> + }
> +
> + if (virtio_crypto_alg_validate_key(keylen, &alg))
> + goto bad_key;
> +
> + /* Create encryption session */
> + ret = virtio_crypto_alg_ablkcipher_init_session(ctx,
> + alg, key, keylen, 1);
> + if (ret)
> + return ret;
> + /* Create decryption session */
> + ret = virtio_crypto_alg_ablkcipher_init_session(ctx,
> + alg, key, keylen, 0);
> + if (ret) {
> + virtio_crypto_alg_ablkcipher_close_session(ctx, 1);
> + return ret;
> + }
> + return 0;
> +
> +bad_key:
> + crypto_tfm_set_flags(ctx->tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
> + return -EINVAL;
> +}
> +
> +/* Note: kernel crypto API realization */
> +static int virtio_crypto_ablkcipher_setkey(struct crypto_ablkcipher *tfm,
> + const uint8_t *key,
> + unsigned int keylen)
> +{
> + struct virtio_crypto_ablkcipher_ctx *ctx = crypto_ablkcipher_ctx(tfm);
> + int ret;
> +
> + if (!ctx->vcrypto) {
> + /* New key */
> + int node = virtio_crypto_get_current_node();
> + struct virtio_crypto *vcrypto =
> + virtcrypto_get_dev_node(node);
> + if (!vcrypto) {
> + pr_err("virtio_crypto: Could not find a virtio device in the
> system");
> + return -ENODEV;
> + }
> +
> + ctx->vcrypto = vcrypto;
> + } else {
> + /* Rekeying, we should close the created sessions previously */
> + virtio_crypto_alg_ablkcipher_close_session(ctx, 1);
> + virtio_crypto_alg_ablkcipher_close_session(ctx, 0);
> + }
> +
> + ret = virtio_crypto_alg_ablkcipher_init_sessions(ctx, key, keylen);
> + if (ret) {
> + virtcrypto_dev_put(ctx->vcrypto);
> + ctx->vcrypto = NULL;
> +
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int
> +__virtio_crypto_ablkcipher_do_req(struct virtio_crypto_request *vc_req,
> + struct ablkcipher_request *req,
> + struct data_queue *data_vq,
> + __u8 op)
> +{
> + struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
> + unsigned int ivsize = crypto_ablkcipher_ivsize(tfm);
> + struct virtio_crypto_ablkcipher_ctx *ctx = vc_req->ablkcipher_ctx;
> + struct virtio_crypto *vcrypto = ctx->vcrypto;
> + struct virtio_crypto_op_data_req *req_data;
> + int src_nents, dst_nents;
> + int err;
> + unsigned long flags;
> + struct scatterlist outhdr, iv_sg, status_sg, **sgs;
> + int i;
> + u64 dst_len;
> + unsigned int num_out = 0, num_in = 0;
> + int sg_total;
> + uint8_t *iv;
> +
> + src_nents = sg_nents_for_len(req->src, req->nbytes);
> + dst_nents = sg_nents(req->dst);
> +
> + pr_debug("virtio_crypto: Number of sgs (src_nents: %d,
> dst_nents: %d)\n",
> + src_nents, dst_nents);
> +
> + /* Why 3? outhdr + iv + inhdr */
> + sg_total = src_nents + dst_nents + 3;
> + sgs = kzalloc_node(sg_total * sizeof(*sgs), GFP_ATOMIC,
> + dev_to_node(&vcrypto->vdev->dev));
> + if (!sgs)
> + return -ENOMEM;
> +
> + req_data = kzalloc_node(sizeof(*req_data), GFP_ATOMIC,
> + dev_to_node(&vcrypto->vdev->dev));
> + if (!req_data) {
> + kfree(sgs);
> + return -ENOMEM;
> + }
> +
> + vc_req->req_data = req_data;
> + vc_req->type = VIRTIO_CRYPTO_SYM_OP_CIPHER;
> + /* Head of operation */
> + if (op) {
> + req_data->header.session_id =
> + cpu_to_le64(ctx->enc_sess_info.session_id);
> + req_data->header.opcode =
> + cpu_to_le32(VIRTIO_CRYPTO_CIPHER_ENCRYPT);
> + } else {
> + req_data->header.session_id =
> + cpu_to_le64(ctx->dec_sess_info.session_id);
> + req_data->header.opcode =
> + cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DECRYPT);
> + }
> + req_data->u.sym_req.op_type =
> cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER);
> + req_data->u.sym_req.u.cipher.para.iv_len = cpu_to_le32(ivsize);
> + req_data->u.sym_req.u.cipher.para.src_data_len =
> + cpu_to_le32(req->nbytes);
> +
> + dst_len = virtio_crypto_alg_sg_nents_length(req->dst);
> + if (unlikely(dst_len > U32_MAX)) {
> + pr_err("virtio_crypto: The dst_len is beyond U32_MAX\n");
> + err = -EINVAL;
> + goto free;
> + }
> +
> + pr_debug("virtio_crypto: src_len: %u, dst_len: %llu\n",
> + req->nbytes, dst_len);
> +
> + if (unlikely(req->nbytes + dst_len + ivsize +
> + sizeof(vc_req->status) > vcrypto->max_size)) {
> + pr_err("virtio_crypto: The length is too big\n");
> + err = -EINVAL;
> + goto free;
> + }
> +
> + req_data->u.sym_req.u.cipher.para.dst_data_len =
> + cpu_to_le32((uint32_t)dst_len);
> +
> + /* Outhdr */
> + sg_init_one(&outhdr, req_data, sizeof(*req_data));
> + sgs[num_out++] = &outhdr;
> +
> + /* IV */
> +
> + /*
> + * Avoid to do DMA from the stack, switch to using
> + * dynamically-allocated for the IV
> + */
> + iv = kzalloc_node(ivsize, GFP_ATOMIC,
> + dev_to_node(&vcrypto->vdev->dev));
> + if (!iv) {
> + err = -ENOMEM;
> + goto free;
> + }
> + memcpy(iv, req->info, ivsize);
> + sg_init_one(&iv_sg, iv, ivsize);
> + sgs[num_out++] = &iv_sg;
> + vc_req->iv = iv;
> +
> + /* Source data */
> + for (i = 0; i < src_nents; i++)
> + sgs[num_out++] = &req->src[i];
> +
> + /* Destination data */
> + for (i = 0; i < dst_nents; i++)
> + sgs[num_out + num_in++] = &req->dst[i];
> +
> + /* Status */
> + sg_init_one(&status_sg, &vc_req->status, sizeof(vc_req->status));
> + sgs[num_out + num_in++] = &status_sg;
> +
> + vc_req->sgs = sgs;
> +
> + spin_lock_irqsave(&vcrypto->lock, flags);
> + err = virtqueue_add_sgs(data_vq->vq, sgs, num_out,
> + num_in, vc_req, GFP_ATOMIC);
> + spin_unlock_irqrestore(&vcrypto->lock, flags);
> + if (unlikely(err < 0))
> + goto free_iv;
> +
> + return 0;
> +
> +free_iv:
> + kzfree(iv);
> +free:
> + kzfree(req_data);
> + kfree(sgs);
> + return err;
> +}
> +
> +static int virtio_crypto_ablkcipher_encrypt(struct ablkcipher_request *req)
> +{
> + struct crypto_ablkcipher *atfm = crypto_ablkcipher_reqtfm(req);
> + struct virtio_crypto_ablkcipher_ctx *ctx = crypto_ablkcipher_ctx(atfm);
> + struct virtio_crypto_request *vc_req = ablkcipher_request_ctx(req);
> + struct virtio_crypto *vcrypto = ctx->vcrypto;
> + int ret;
> + /* Use the first data virtqueue as default */
> + struct data_queue *data_vq = &vcrypto->data_vq[0];
> +
> + vc_req->ablkcipher_ctx = ctx;
> + vc_req->ablkcipher_req = req;
> + ret = __virtio_crypto_ablkcipher_do_req(vc_req, req, data_vq, 1);
> + if (ret < 0) {
> + pr_err("virtio_crypto: Encryption failed!\n");
> + return ret;
> + }
> + virtqueue_kick(data_vq->vq);
> +
> + return -EINPROGRESS;
> +}
> +
> +static int virtio_crypto_ablkcipher_decrypt(struct ablkcipher_request *req)
> +{
> + struct crypto_ablkcipher *atfm = crypto_ablkcipher_reqtfm(req);
> + struct virtio_crypto_ablkcipher_ctx *ctx = crypto_ablkcipher_ctx(atfm);
> + struct virtio_crypto_request *vc_req = ablkcipher_request_ctx(req);
> + struct virtio_crypto *vcrypto = ctx->vcrypto;
> + int ret;
> + /* Use the first data virtqueue as default */
> + struct data_queue *data_vq = &vcrypto->data_vq[0];
> +
> + vc_req->ablkcipher_ctx = ctx;
> + vc_req->ablkcipher_req = req;
> +
> + ret = __virtio_crypto_ablkcipher_do_req(vc_req, req, data_vq, 0);
> + if (ret < 0) {
> + pr_err("virtio_crypto: Decryption failed!\n");
> + return ret;
> + }
> + virtqueue_kick(data_vq->vq);
> +
> + return -EINPROGRESS;
> +}
> +
> +static int virtio_crypto_ablkcipher_init(struct crypto_tfm *tfm)
> +{
> + struct virtio_crypto_ablkcipher_ctx *ctx = crypto_tfm_ctx(tfm);
> +
> + tfm->crt_ablkcipher.reqsize = sizeof(struct virtio_crypto_request);
> + ctx->tfm = tfm;
> +
> + return 0;
> +}
> +
> +static void virtio_crypto_ablkcipher_exit(struct crypto_tfm *tfm)
> +{
> + struct virtio_crypto_ablkcipher_ctx *ctx = crypto_tfm_ctx(tfm);
> +
> + if (!ctx->vcrypto)
> + return;
> +
> + virtio_crypto_alg_ablkcipher_close_session(ctx, 1);
> + virtio_crypto_alg_ablkcipher_close_session(ctx, 0);
> + virtcrypto_dev_put(ctx->vcrypto);
> + ctx->vcrypto = NULL;
> +}
> +
> +static struct crypto_alg virtio_crypto_algs[] = { {
> + .cra_name = "cbc(aes)",
> + .cra_driver_name = "virtio_crypto_aes_cbc",
> + .cra_priority = 501,
> + .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | CRYPTO_ALG_ASYNC,
> + .cra_blocksize = AES_BLOCK_SIZE,
> + .cra_ctxsize = sizeof(struct virtio_crypto_ablkcipher_ctx),
> + .cra_alignmask = 0,
> + .cra_module = THIS_MODULE,
> + .cra_type = &crypto_ablkcipher_type,
> + .cra_init = virtio_crypto_ablkcipher_init,
> + .cra_exit = virtio_crypto_ablkcipher_exit,
> + .cra_u = {
> + .ablkcipher = {
> + .setkey = virtio_crypto_ablkcipher_setkey,
> + .decrypt = virtio_crypto_ablkcipher_decrypt,
> + .encrypt = virtio_crypto_ablkcipher_encrypt,
> + .min_keysize = AES_MIN_KEY_SIZE,
> + .max_keysize = AES_MAX_KEY_SIZE,
> + .ivsize = AES_BLOCK_SIZE,
> + },
> + },
> +} };
> +
> +int virtio_crypto_algs_register(void)
> +{
> + int ret = 0;
> +
> + mutex_lock(&algs_lock);
> + if (++virtio_crypto_active_devs != 1)
> + goto unlock;
> +
> + ret = crypto_register_algs(virtio_crypto_algs,
> + ARRAY_SIZE(virtio_crypto_algs));
> + if (ret)
> + virtio_crypto_active_devs--;
> +
> +unlock:
> + mutex_unlock(&algs_lock);
> + return ret;
> +}
> +
> +void virtio_crypto_algs_unregister(void)
> +{
> + mutex_lock(&algs_lock);
> + if (--virtio_crypto_active_devs != 0)
> + goto unlock;
> +
> + crypto_unregister_algs(virtio_crypto_algs,
> + ARRAY_SIZE(virtio_crypto_algs));
> +
> +unlock:
> + mutex_unlock(&algs_lock);
> +}
> diff --git a/drivers/crypto/virtio/virtio_crypto_common.h
> b/drivers/crypto/virtio/virtio_crypto_common.h
> new file mode 100644
> index 0000000..975404b
> --- /dev/null
> +++ b/drivers/crypto/virtio/virtio_crypto_common.h
> @@ -0,0 +1,122 @@
> +/* Common header for Virtio crypto device.
> + *
> + * Copyright 2016 HUAWEI TECHNOLOGIES CO., LTD.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef _VIRTIO_CRYPTO_COMMON_H
> +#define _VIRTIO_CRYPTO_COMMON_H
> +
> +#include <linux/virtio.h>
> +#include <linux/crypto.h>
> +#include <linux/spinlock.h>
> +#include <crypto/aead.h>
> +#include <crypto/aes.h>
> +#include <crypto/authenc.h>
> +
> +
> +/* Internal representation of a data virtqueue */
> +struct data_queue {
> + /* Virtqueue associated with this send _queue */
> + struct virtqueue *vq;
> +
> + /* Name of the tx queue: dataq.$index */
> + char name[32];
> +};
> +
> +struct virtio_crypto {
> + struct virtio_device *vdev;
> + struct virtqueue *ctrl_vq;
> + struct data_queue *data_vq;
> +
> + /* To protect the vq operations for the dataq */
> + spinlock_t lock;
> +
> + /* To protect the vq operations for the controlq */
> + spinlock_t ctrl_lock;
> +
> + /* Maximum of data queues supported by the device */
> + u32 max_data_queues;
> +
> + /* Number of queue currently used by the driver */
> + u32 curr_queue;
> +
> + /* Maximum length of cipher key */
> + u32 max_cipher_key_len;
> + /* Maximum length of authenticated key */
> + u32 max_auth_key_len;
> + /* Maximum size of per request */
> + u64 max_size;
> +
> + /* Control VQ buffers: protected by the ctrl_lock */
> + struct virtio_crypto_op_ctrl_req ctrl;
> + struct virtio_crypto_session_input input;
> + struct virtio_crypto_inhdr ctrl_status;
> +
> + unsigned long status;
> + atomic_t ref_count;
> + struct list_head list;
> + struct module *owner;
> + uint8_t dev_id;
> +
> + /* Does the affinity hint is set for virtqueues? */
> + bool affinity_hint_set;
> +};
> +
> +struct virtio_crypto_sym_session_info {
> + /* Backend session id, which come from the host side */
> + __u64 session_id;
> +};
> +
> +struct virtio_crypto_ablkcipher_ctx {
> + struct virtio_crypto *vcrypto;
> + struct crypto_tfm *tfm;
> +
> + struct virtio_crypto_sym_session_info enc_sess_info;
> + struct virtio_crypto_sym_session_info dec_sess_info;
> +};
> +
> +struct virtio_crypto_request {
> + /* Cipher or aead */
> + uint32_t type;
> + uint8_t status;
> + struct virtio_crypto_ablkcipher_ctx *ablkcipher_ctx;
> + struct ablkcipher_request *ablkcipher_req;
> + struct virtio_crypto_op_data_req *req_data;
> + struct scatterlist **sgs;
> + uint8_t *iv;
> +};
> +
> +int virtcrypto_devmgr_add_dev(struct virtio_crypto *vcrypto_dev);
> +struct list_head *virtcrypto_devmgr_get_head(void);
> +void virtcrypto_devmgr_rm_dev(struct virtio_crypto *vcrypto_dev);
> +struct virtio_crypto *virtcrypto_devmgr_get_first(void);
> +int virtcrypto_dev_in_use(struct virtio_crypto *vcrypto_dev);
> +int virtcrypto_dev_get(struct virtio_crypto *vcrypto_dev);
> +void virtcrypto_dev_put(struct virtio_crypto *vcrypto_dev);
> +int virtcrypto_dev_started(struct virtio_crypto *vcrypto_dev);
> +struct virtio_crypto *virtcrypto_get_dev_node(int node);
> +int virtcrypto_dev_start(struct virtio_crypto *vcrypto);
> +void virtcrypto_dev_stop(struct virtio_crypto *vcrypto);
> +
> +static inline int virtio_crypto_get_current_node(void)
> +{
> + return topology_physical_package_id(smp_processor_id());
> +}
> +
> +int virtio_crypto_algs_register(void);
> +void virtio_crypto_algs_unregister(void);
> +
> +#endif /* _VIRTIO_CRYPTO_COMMON_H */
> diff --git a/drivers/crypto/virtio/virtio_crypto_core.c
> b/drivers/crypto/virtio/virtio_crypto_core.c
> new file mode 100644
> index 0000000..286d829
> --- /dev/null
> +++ b/drivers/crypto/virtio/virtio_crypto_core.c
> @@ -0,0 +1,464 @@
> + /* Driver for Virtio crypto device.
> + *
> + * Copyright 2016 HUAWEI TECHNOLOGIES CO., LTD.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/err.h>
> +#include <linux/module.h>
> +#include <linux/virtio_config.h>
> +#include <linux/cpu.h>
> +
> +#include <uapi/linux/virtio_crypto.h>
> +#include "virtio_crypto_common.h"
> +
> +
> +static void virtcrypto_dataq_callback(struct virtqueue *vq)
> +{
> + struct virtio_crypto *vcrypto = vq->vdev->priv;
> + struct virtio_crypto_request *vc_req;
> + unsigned long flags;
> + unsigned int len;
> + struct ablkcipher_request *ablk_req;
> + int error;
> +
> + spin_lock_irqsave(&vcrypto->lock, flags);
> + do {
> + virtqueue_disable_cb(vq);
> + while ((vc_req = virtqueue_get_buf(vq, &len)) != NULL) {
> + if (vc_req->type == VIRTIO_CRYPTO_SYM_OP_CIPHER) {
> + switch (vc_req->status) {
> + case VIRTIO_CRYPTO_OK:
> + error = 0;
> + break;
> + case VIRTIO_CRYPTO_INVSESS:
> + case VIRTIO_CRYPTO_ERR:
> + error = -EINVAL;
> + break;
> + case VIRTIO_CRYPTO_BADMSG:
> + error = -EBADMSG;
> + break;
> + default:
> + error = -EIO;
> + break;
> + }
> + ablk_req = vc_req->ablkcipher_req;
> + /* Finish the encrypt or decrypt process */
> + ablk_req->base.complete(&ablk_req->base, error);
> + }
> +
> + kzfree(vc_req->iv);
> + kzfree(vc_req->req_data);
> + kfree(vc_req->sgs);
> + }
> + } while (!virtqueue_enable_cb(vq));
> + spin_unlock_irqrestore(&vcrypto->lock, flags);
> +}
> +
> +static int virtcrypto_find_vqs(struct virtio_crypto *vi)
> +{
> + vq_callback_t **callbacks;
> + struct virtqueue **vqs;
> + int ret = -ENOMEM;
> + int i, total_vqs;
> + const char **names;
> +
> + /*
> + * We expect 1 data virtqueue, followed by
> + * possible N-1 data queues used in multiqueue mode,
> + * followed by control vq.
> + */
> + total_vqs = vi->max_data_queues + 1;
> +
> + /* Allocate space for find_vqs parameters */
> + vqs = kcalloc(total_vqs, sizeof(*vqs), GFP_KERNEL);
> + if (!vqs)
> + goto err_vq;
> + callbacks = kcalloc(total_vqs, sizeof(*callbacks), GFP_KERNEL);
> + if (!callbacks)
> + goto err_callback;
> + names = kcalloc(total_vqs, sizeof(*names), GFP_KERNEL);
> + if (!names)
> + goto err_names;
> +
> + /* Parameters for control virtqueue */
> + callbacks[total_vqs - 1] = NULL;
> + names[total_vqs - 1] = "controlq";
> +
> + /* Allocate/initialize parameters for data virtqueues */
> + for (i = 0; i < vi->max_data_queues; i++) {
> + callbacks[i] = virtcrypto_dataq_callback;
> + snprintf(vi->data_vq[i].name, sizeof(vi->data_vq[i].name),
> + "dataq.%d", i);
> + names[i] = vi->data_vq[i].name;
> + }
> +
> + ret = vi->vdev->config->find_vqs(vi->vdev, total_vqs, vqs, callbacks,
> + names);
> + if (ret)
> + goto err_find;
> +
> + vi->ctrl_vq = vqs[total_vqs - 1];
> +
> + for (i = 0; i < vi->max_data_queues; i++)
> + vi->data_vq[i].vq = vqs[i];
> +
> + kfree(names);
> + kfree(callbacks);
> + kfree(vqs);
> +
> + return 0;
> +
> +err_find:
> + kfree(names);
> +err_names:
> + kfree(callbacks);
> +err_callback:
> + kfree(vqs);
> +err_vq:
> + return ret;
> +}
> +
> +static int virtcrypto_alloc_queues(struct virtio_crypto *vi)
> +{
> + vi->data_vq = kcalloc(vi->max_data_queues, sizeof(*vi->data_vq),
> + GFP_KERNEL);
> + if (!vi->data_vq)
> + return -ENOMEM;
> +
> + return 0;
> +}
> +
> +static void virtcrypto_clean_affinity(struct virtio_crypto *vi, long hcpu)
> +{
> + int i;
> +
> + if (vi->affinity_hint_set) {
> + for (i = 0; i < vi->max_data_queues; i++)
> + virtqueue_set_affinity(vi->data_vq[i].vq, -1);
> +
> + vi->affinity_hint_set = false;
> + }
> +}
> +
> +static void virtcrypto_set_affinity(struct virtio_crypto *vcrypto)
> +{
> + int i = 0;
> + int cpu;
> +
> + /*
> + * In single queue mode, we don't set the cpu affinity.
> + */
> + if (vcrypto->curr_queue == 1 || vcrypto->max_data_queues == 1) {
> + virtcrypto_clean_affinity(vcrypto, -1);
> + return;
> + }
> +
> + /*
> + * In multiqueue mode, we let the queue to be private to one cpu
> + * by setting the affinity hint to eliminate the contention.
> + *
> + * TODO: adds cpu hotplug support by register cpu notifier.
> + *
> + */
> + for_each_online_cpu(cpu) {
> + virtqueue_set_affinity(vcrypto->data_vq[i].vq, cpu);
> + if (++i >= vcrypto->max_data_queues)
> + break;
> + }
> +
> + vcrypto->affinity_hint_set = true;
> +}
> +
> +static void virtcrypto_free_queues(struct virtio_crypto *vi)
> +{
> + kfree(vi->data_vq);
> +}
> +
> +static int virtcrypto_init_vqs(struct virtio_crypto *vi)
> +{
> + int ret;
> +
> + /* Allocate send & receive queues */
> + ret = virtcrypto_alloc_queues(vi);
> + if (ret)
> + goto err;
> +
> + ret = virtcrypto_find_vqs(vi);
> + if (ret)
> + goto err_free;
> +
> + get_online_cpus();
> + virtcrypto_set_affinity(vi);
> + put_online_cpus();
> +
> + return 0;
> +
> +err_free:
> + virtcrypto_free_queues(vi);
> +err:
> + return ret;
> +}
> +
> +static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
> +{
> + u32 status;
> + int err;
> + unsigned long flags;
> +
> + virtio_cread(vcrypto->vdev,
> + struct virtio_crypto_config, status, &status);
> +
> + /*
> + * Unknown status bits would be a host error and the driver
> + * should consider the device to be broken.
> + */
> + if (status & (~VIRTIO_CRYPTO_S_HW_READY)) {
> + dev_warn(&vcrypto->vdev->dev,
> + "Unknown status bits: 0x%x\n", status);
> +
> + spin_lock_irqsave(&vcrypto->lock, flags);
> + virtio_break_device(vcrypto->vdev);
> + spin_unlock_irqrestore(&vcrypto->lock, flags);
> + return -EPERM;
> + }
> +
> + if (vcrypto->status == status)
> + return 0;
> +
> + vcrypto->status = status;
> +
> + if (vcrypto->status & VIRTIO_CRYPTO_S_HW_READY) {
> + err = virtcrypto_dev_start(vcrypto);
> + if (err) {
> + dev_err(&vcrypto->vdev->dev,
> + "Failed to start virtio crypto device.\n");
> +
> + return -EPERM;
> + }
> + dev_info(&vcrypto->vdev->dev, "Accelerator is ready\n");
> + } else {
> + virtcrypto_dev_stop(vcrypto);
> + dev_info(&vcrypto->vdev->dev, "Accelerator is not ready\n");
> + }
> +
> + return 0;
> +}
> +
> +static void virtcrypto_del_vqs(struct virtio_crypto *vcrypto)
> +{
> + struct virtio_device *vdev = vcrypto->vdev;
> +
> + virtcrypto_clean_affinity(vcrypto, -1);
> +
> + vdev->config->del_vqs(vdev);
> +
> + virtcrypto_free_queues(vcrypto);
> +}
> +
> +static int virtcrypto_probe(struct virtio_device *vdev)
> +{
> + int err = -EFAULT;
> + struct virtio_crypto *vcrypto;
> + u32 max_data_queues = 0, max_cipher_key_len = 0;
> + u32 max_auth_key_len = 0;
> + u64 max_size = 0;
> +
> + if (!virtio_has_feature(vdev, VIRTIO_F_VERSION_1))
> + return -ENODEV;
> +
> + if (!vdev->config->get) {
> + dev_err(&vdev->dev, "%s failure: config access disabled\n",
> + __func__);
> + return -EINVAL;
> + }
> +
> + if (num_possible_nodes() > 1 && dev_to_node(&vdev->dev) < 0) {
> + /*
> + * If the accelerator is connected to a node with no memory
> + * there is no point in using the accelerator since the remote
> + * memory transaction will be very slow.
> + */
> + dev_err(&vdev->dev, "Invalid NUMA configuration.\n");
> + return -EINVAL;
> + }
> +
> + vcrypto = kzalloc_node(sizeof(*vcrypto), GFP_KERNEL,
> + dev_to_node(&vdev->dev));
> + if (!vcrypto)
> + return -ENOMEM;
> +
> + virtio_cread(vdev, struct virtio_crypto_config,
> + max_dataqueues, &max_data_queues);
> + if (max_data_queues < 1)
> + max_data_queues = 1;
> +
> + virtio_cread(vdev, struct virtio_crypto_config,
> + max_cipher_key_len, &max_cipher_key_len);
> + virtio_cread(vdev, struct virtio_crypto_config,
> + max_auth_key_len, &max_auth_key_len);
> + virtio_cread(vdev, struct virtio_crypto_config,
> + max_size, &max_size);
> +
> + /* Add virtio crypto device to global table */
> + err = virtcrypto_devmgr_add_dev(vcrypto);
> + if (err) {
> + dev_err(&vdev->dev, "Failed to add new virtio crypto device.\n");
> + goto free;
> + }
> + vcrypto->owner = THIS_MODULE;
> + vcrypto = vdev->priv = vcrypto;
> + vcrypto->vdev = vdev;
> + spin_lock_init(&vcrypto->lock);
> + spin_lock_init(&vcrypto->ctrl_lock);
> +
> + /* Use single data queue as default */
> + vcrypto->curr_queue = 1;
> + vcrypto->max_data_queues = max_data_queues;
> + vcrypto->max_cipher_key_len = max_cipher_key_len;
> + vcrypto->max_auth_key_len = max_auth_key_len;
> + vcrypto->max_size = max_size;
> +
> + dev_info(&vdev->dev,
> + "max_queues: %u, max_cipher_key_len: %u, max_auth_key_len: %u,
> max_size 0x%llx\n",
> + vcrypto->max_data_queues,
> + vcrypto->max_cipher_key_len,
> + vcrypto->max_auth_key_len,
> + vcrypto->max_size);
> +
> + err = virtcrypto_init_vqs(vcrypto);
> + if (err) {
> + dev_err(&vdev->dev, "Failed to initialize vqs.\n");
> + goto free_dev;
> + }
> + virtio_device_ready(vdev);
> +
> + err = virtcrypto_update_status(vcrypto);
> + if (err)
> + goto free_vqs;
> +
> + return 0;
> +
> +free_vqs:
> + vcrypto->vdev->config->reset(vdev);
> + virtcrypto_del_vqs(vcrypto);
> +free_dev:
> + virtcrypto_devmgr_rm_dev(vcrypto);
> +free:
> + kfree(vcrypto);
> + return err;
> +}
> +
> +static void virtcrypto_free_unused_reqs(struct virtio_crypto *vcrypto)
> +{
> + struct virtio_crypto_request *vc_req;
> + int i;
> + struct virtqueue *vq;
> +
> + for (i = 0; i < vcrypto->max_data_queues; i++) {
> + vq = vcrypto->data_vq[i].vq;
> + while ((vc_req = virtqueue_detach_unused_buf(vq)) != NULL) {
> + kfree(vc_req->req_data);
> + kfree(vc_req->sgs);
> + }
> + }
> +}
> +
> +static void virtcrypto_remove(struct virtio_device *vdev)
> +{
> + struct virtio_crypto *vcrypto = vdev->priv;
> +
> + dev_info(&vdev->dev, "Start virtcrypto_remove.\n");
> +
> + if (virtcrypto_dev_started(vcrypto))
> + virtcrypto_dev_stop(vcrypto);
> + vdev->config->reset(vdev);
> + virtcrypto_free_unused_reqs(vcrypto);
> + virtcrypto_del_vqs(vcrypto);
> + virtcrypto_devmgr_rm_dev(vcrypto);
> + kfree(vcrypto);
> +}
> +
> +static void virtcrypto_config_changed(struct virtio_device *vdev)
> +{
> + struct virtio_crypto *vcrypto = vdev->priv;
> +
> + virtcrypto_update_status(vcrypto);
> +}
> +
> +#ifdef CONFIG_PM_SLEEP
> +static int virtcrypto_freeze(struct virtio_device *vdev)
> +{
> + struct virtio_crypto *vcrypto = vdev->priv;
> +
> + vdev->config->reset(vdev);
> + virtcrypto_free_unused_reqs(vcrypto);
> + if (virtcrypto_dev_started(vcrypto))
> + virtcrypto_dev_stop(vcrypto);
> +
> + virtcrypto_del_vqs(vcrypto);
> + return 0;
> +}
> +
> +static int virtcrypto_restore(struct virtio_device *vdev)
> +{
> + struct virtio_crypto *vcrypto = vdev->priv;
> + int err;
> +
> + err = virtcrypto_init_vqs(vcrypto);
> + if (err)
> + return err;
> +
> + virtio_device_ready(vdev);
> + err = virtcrypto_dev_start(vcrypto);
> + if (err) {
> + dev_err(&vdev->dev, "Failed to start virtio crypto device.\n");
> + return -EFAULT;
> + }
> +
> + return 0;
> +}
> +#endif
> +
> +static unsigned int features[] = {
> + /* none */
> +};
> +
> +static struct virtio_device_id id_table[] = {
> + { VIRTIO_ID_CRYPTO, VIRTIO_DEV_ANY_ID },
> + { 0 },
> +};
> +
> +static struct virtio_driver virtio_crypto_driver = {
> + .driver.name = KBUILD_MODNAME,
> + .driver.owner = THIS_MODULE,
> + .feature_table = features,
> + .feature_table_size = ARRAY_SIZE(features),
> + .id_table = id_table,
> + .probe = virtcrypto_probe,
> + .remove = virtcrypto_remove,
> + .config_changed = virtcrypto_config_changed,
> +#ifdef CONFIG_PM_SLEEP
> + .freeze = virtcrypto_freeze,
> + .restore = virtcrypto_restore,
> +#endif
> +};
> +
> +module_virtio_driver(virtio_crypto_driver);
> +
> +MODULE_DEVICE_TABLE(virtio, id_table);
> +MODULE_DESCRIPTION("virtio crypto device driver");
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Gonglei <arei.gonglei@huawei.com>");
> diff --git a/drivers/crypto/virtio/virtio_crypto_mgr.c
> b/drivers/crypto/virtio/virtio_crypto_mgr.c
> new file mode 100644
> index 0000000..a69ff71
> --- /dev/null
> +++ b/drivers/crypto/virtio/virtio_crypto_mgr.c
> @@ -0,0 +1,264 @@
> + /* Management for virtio crypto devices (refer to adf_dev_mgr.c)
> + *
> + * Copyright 2016 HUAWEI TECHNOLOGIES CO., LTD.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/mutex.h>
> +#include <linux/list.h>
> +#include <linux/module.h>
> +
> +#include <uapi/linux/virtio_crypto.h>
> +#include "virtio_crypto_common.h"
> +
> +static LIST_HEAD(virtio_crypto_table);
> +static uint32_t num_devices;
> +
> +/* The table_lock protects the above global list and num_devices */
> +static DEFINE_MUTEX(table_lock);
> +
> +#define VIRTIO_CRYPTO_MAX_DEVICES 32
> +
> +
> +/*
> + * virtcrypto_devmgr_add_dev() - Add vcrypto_dev to the acceleration
> + * framework.
> + * @vcrypto_dev: Pointer to virtio crypto device.
> + *
> + * Function adds virtio crypto device to the global list.
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: 0 on success, error code othewise.
> + */
> +int virtcrypto_devmgr_add_dev(struct virtio_crypto *vcrypto_dev)
> +{
> + struct list_head *itr;
> +
> + mutex_lock(&table_lock);
> + if (num_devices == VIRTIO_CRYPTO_MAX_DEVICES) {
> + pr_info("virtio_crypto: only support up to %d devices\n",
> + VIRTIO_CRYPTO_MAX_DEVICES);
> + mutex_unlock(&table_lock);
> + return -EFAULT;
> + }
> +
> + list_for_each(itr, &virtio_crypto_table) {
> + struct virtio_crypto *ptr =
> + list_entry(itr, struct virtio_crypto, list);
> +
> + if (ptr == vcrypto_dev) {
> + mutex_unlock(&table_lock);
> + return -EEXIST;
> + }
> + }
> + atomic_set(&vcrypto_dev->ref_count, 0);
> + list_add_tail(&vcrypto_dev->list, &virtio_crypto_table);
> + vcrypto_dev->dev_id = num_devices++;
> + mutex_unlock(&table_lock);
> + return 0;
> +}
> +
> +struct list_head *virtcrypto_devmgr_get_head(void)
> +{
> + return &virtio_crypto_table;
> +}
> +
> +/*
> + * virtcrypto_devmgr_rm_dev() - Remove vcrypto_dev from the acceleration
> + * framework.
> + * @vcrypto_dev: Pointer to virtio crypto device.
> + *
> + * Function removes virtio crypto device from the acceleration framework.
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: void
> + */
> +void virtcrypto_devmgr_rm_dev(struct virtio_crypto *vcrypto_dev)
> +{
> + mutex_lock(&table_lock);
> + list_del(&vcrypto_dev->list);
> + num_devices--;
> + mutex_unlock(&table_lock);
> +}
> +
> +/*
> + * virtcrypto_devmgr_get_first()
> + *
> + * Function returns the first virtio crypto device from the acceleration
> + * framework.
> + *
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: pointer to vcrypto_dev or NULL if not found.
> + */
> +struct virtio_crypto *virtcrypto_devmgr_get_first(void)
> +{
> + struct virtio_crypto *dev = NULL;
> +
> + mutex_lock(&table_lock);
> + if (!list_empty(&virtio_crypto_table))
> + dev = list_first_entry(&virtio_crypto_table,
> + struct virtio_crypto,
> + list);
> + mutex_unlock(&table_lock);
> + return dev;
> +}
> +
> +/*
> + * virtcrypto_dev_in_use() - Check whether vcrypto_dev is currently in use
> + * @vcrypto_dev: Pointer to virtio crypto device.
> + *
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: 1 when device is in use, 0 otherwise.
> + */
> +int virtcrypto_dev_in_use(struct virtio_crypto *vcrypto_dev)
> +{
> + return atomic_read(&vcrypto_dev->ref_count) != 0;
> +}
> +
> +/*
> + * virtcrypto_dev_get() - Increment vcrypto_dev reference count
> + * @vcrypto_dev: Pointer to virtio crypto device.
> + *
> + * Increment the vcrypto_dev refcount and if this is the first time
> + * incrementing it during this period the vcrypto_dev is in use,
> + * increment the module refcount too.
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: 0 when successful, EFAULT when fail to bump module refcount
> + */
> +int virtcrypto_dev_get(struct virtio_crypto *vcrypto_dev)
> +{
> + if (atomic_add_return(1, &vcrypto_dev->ref_count) == 1)
> + if (!try_module_get(vcrypto_dev->owner))
> + return -EFAULT;
> + return 0;
> +}
> +
> +/*
> + * virtcrypto_dev_put() - Decrement vcrypto_dev reference count
> + * @vcrypto_dev: Pointer to virtio crypto device.
> + *
> + * Decrement the vcrypto_dev refcount and if this is the last time
> + * decrementing it during this period the vcrypto_dev is in use,
> + * decrement the module refcount too.
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: void
> + */
> +void virtcrypto_dev_put(struct virtio_crypto *vcrypto_dev)
> +{
> + if (atomic_sub_return(1, &vcrypto_dev->ref_count) == 0)
> + module_put(vcrypto_dev->owner);
> +}
> +
> +/*
> + * virtcrypto_dev_started() - Check whether device has started
> + * @vcrypto_dev: Pointer to virtio crypto device.
> + *
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: 1 when the device has started, 0 otherwise
> + */
> +int virtcrypto_dev_started(struct virtio_crypto *vcrypto_dev)
> +{
> + return (vcrypto_dev->status & VIRTIO_CRYPTO_S_HW_READY);
> +}
> +
> +/*
> + * virtcrypto_get_dev_node() - Get vcrypto_dev on the node.
> + * @node: Node id the driver works.
> + *
> + * Function returns the virtio crypto device used fewest on the node.
> + *
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: pointer to vcrypto_dev or NULL if not found.
> + */
> +struct virtio_crypto *virtcrypto_get_dev_node(int node)
> +{
> + struct virtio_crypto *vcrypto_dev = NULL, *tmp_dev;
> + unsigned long best = ~0;
> + unsigned long ctr;
> +
> + mutex_lock(&table_lock);
> + list_for_each_entry(tmp_dev, virtcrypto_devmgr_get_head(), list) {
> +
> + if ((node == dev_to_node(&tmp_dev->vdev->dev) ||
> + dev_to_node(&tmp_dev->vdev->dev) < 0) &&
> + virtcrypto_dev_started(tmp_dev)) {
> + ctr = atomic_read(&tmp_dev->ref_count);
> + if (best > ctr) {
> + vcrypto_dev = tmp_dev;
> + best = ctr;
> + }
> + }
> + }
> +
> + if (!vcrypto_dev) {
> + pr_info("virtio_crypto: Could not find a device on node %d\n",
> + node);
> + /* Get any started device */
> + list_for_each_entry(tmp_dev,
> + virtcrypto_devmgr_get_head(), list) {
> + if (virtcrypto_dev_started(tmp_dev)) {
> + vcrypto_dev = tmp_dev;
> + break;
> + }
> + }
> + }
> + mutex_unlock(&table_lock);
> + if (!vcrypto_dev)
> + return NULL;
> +
> + virtcrypto_dev_get(vcrypto_dev);
> + return vcrypto_dev;
> +}
> +
> +/*
> + * virtcrypto_dev_start() - Start virtio crypto device
> + * @vcrypto: Pointer to virtio crypto device.
> + *
> + * Function notifies all the registered services that the virtio crypto device
> + * is ready to be used.
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: 0 on success, EFAULT when fail to register algorithms
> + */
> +int virtcrypto_dev_start(struct virtio_crypto *vcrypto)
> +{
> + if (virtio_crypto_algs_register()) {
> + pr_err("virtio_crypto: Failed to register crypto algs\n");
> + return -EFAULT;
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * virtcrypto_dev_stop() - Stop virtio crypto device
> + * @vcrypto: Pointer to virtio crypto device.
> + *
> + * Function notifies all the registered services that the virtio crypto device
> + * is ready to be used.
> + * To be used by virtio crypto device specific drivers.
> + *
> + * Return: void
> + */
> +void virtcrypto_dev_stop(struct virtio_crypto *vcrypto)
> +{
> + virtio_crypto_algs_unregister();
> +}
> diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild
> index cd2be1c..4bdb84c 100644
> --- a/include/uapi/linux/Kbuild
> +++ b/include/uapi/linux/Kbuild
> @@ -460,6 +460,7 @@ header-y += virtio_rng.h
> header-y += virtio_scsi.h
> header-y += virtio_types.h
> header-y += virtio_vsock.h
> +header-y += virtio_crypto.h
> header-y += vm_sockets.h
> header-y += vt.h
> header-y += vtpm_proxy.h
> diff --git a/include/uapi/linux/virtio_crypto.h
> b/include/uapi/linux/virtio_crypto.h
> new file mode 100644
> index 0000000..50cdc8a
> --- /dev/null
> +++ b/include/uapi/linux/virtio_crypto.h
> @@ -0,0 +1,450 @@
> +#ifndef _VIRTIO_CRYPTO_H
> +#define _VIRTIO_CRYPTO_H
> +/* This header is BSD licensed so anyone can use the definitions to implement
> + * compatible drivers/servers.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the distribution.
> + * 3. Neither the name of IBM nor the names of its contributors
> + * may be used to endorse or promote products derived from this
> software
> + * without specific prior written permission.
> + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
> CONTRIBUTORS
> + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
> NOT
> + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
> FITNESS
> + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL IBM
> OR
> + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
> NOT
> + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
> OF
> + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
> AND
> + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
> + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> OUT
> + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +#include <linux/types.h>
> +#include <linux/virtio_types.h>
> +#include <linux/virtio_ids.h>
> +#include <linux/virtio_config.h>
> +
> +
> +#define VIRTIO_CRYPTO_SERVICE_CIPHER 0
> +#define VIRTIO_CRYPTO_SERVICE_HASH 1
> +#define VIRTIO_CRYPTO_SERVICE_MAC 2
> +#define VIRTIO_CRYPTO_SERVICE_AEAD 3
> +
> +#define VIRTIO_CRYPTO_OPCODE(service, op) (((service) << 8) | (op))
> +
> +struct virtio_crypto_ctrl_header {
> +#define VIRTIO_CRYPTO_CIPHER_CREATE_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_CIPHER, 0x02)
> +#define VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_CIPHER, 0x03)
> +#define VIRTIO_CRYPTO_HASH_CREATE_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_HASH, 0x02)
> +#define VIRTIO_CRYPTO_HASH_DESTROY_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_HASH, 0x03)
> +#define VIRTIO_CRYPTO_MAC_CREATE_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_MAC, 0x02)
> +#define VIRTIO_CRYPTO_MAC_DESTROY_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_MAC, 0x03)
> +#define VIRTIO_CRYPTO_AEAD_CREATE_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AEAD, 0x02)
> +#define VIRTIO_CRYPTO_AEAD_DESTROY_SESSION \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AEAD, 0x03)
> + __le32 opcode;
> + __le32 algo;
> + __le32 flag;
> + /* data virtqueue id */
> + __le32 queue_id;
> +};
> +
> +struct virtio_crypto_cipher_session_para {
> +#define VIRTIO_CRYPTO_NO_CIPHER 0
> +#define VIRTIO_CRYPTO_CIPHER_ARC4 1
> +#define VIRTIO_CRYPTO_CIPHER_AES_ECB 2
> +#define VIRTIO_CRYPTO_CIPHER_AES_CBC 3
> +#define VIRTIO_CRYPTO_CIPHER_AES_CTR 4
> +#define VIRTIO_CRYPTO_CIPHER_DES_ECB 5
> +#define VIRTIO_CRYPTO_CIPHER_DES_CBC 6
> +#define VIRTIO_CRYPTO_CIPHER_3DES_ECB 7
> +#define VIRTIO_CRYPTO_CIPHER_3DES_CBC 8
> +#define VIRTIO_CRYPTO_CIPHER_3DES_CTR 9
> +#define VIRTIO_CRYPTO_CIPHER_KASUMI_F8 10
> +#define VIRTIO_CRYPTO_CIPHER_SNOW3G_UEA2 11
> +#define VIRTIO_CRYPTO_CIPHER_AES_F8 12
> +#define VIRTIO_CRYPTO_CIPHER_AES_XTS 13
> +#define VIRTIO_CRYPTO_CIPHER_ZUC_EEA3 14
> + __le32 algo;
> + /* length of key */
> + __le32 keylen;
> +
> +#define VIRTIO_CRYPTO_OP_ENCRYPT 1
> +#define VIRTIO_CRYPTO_OP_DECRYPT 2
> + /* encrypt or decrypt */
> + __le32 op;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_session_input {
> + /* Device-writable part */
> + __le64 session_id;
> + __le32 status;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_cipher_session_req {
> + struct virtio_crypto_cipher_session_para para;
> + __u8 padding[32];
> +};
> +
> +struct virtio_crypto_hash_session_para {
> +#define VIRTIO_CRYPTO_NO_HASH 0
> +#define VIRTIO_CRYPTO_HASH_MD5 1
> +#define VIRTIO_CRYPTO_HASH_SHA1 2
> +#define VIRTIO_CRYPTO_HASH_SHA_224 3
> +#define VIRTIO_CRYPTO_HASH_SHA_256 4
> +#define VIRTIO_CRYPTO_HASH_SHA_384 5
> +#define VIRTIO_CRYPTO_HASH_SHA_512 6
> +#define VIRTIO_CRYPTO_HASH_SHA3_224 7
> +#define VIRTIO_CRYPTO_HASH_SHA3_256 8
> +#define VIRTIO_CRYPTO_HASH_SHA3_384 9
> +#define VIRTIO_CRYPTO_HASH_SHA3_512 10
> +#define VIRTIO_CRYPTO_HASH_SHA3_SHAKE128 11
> +#define VIRTIO_CRYPTO_HASH_SHA3_SHAKE256 12
> + __le32 algo;
> + /* hash result length */
> + __le32 hash_result_len;
> + __u8 padding[8];
> +};
> +
> +struct virtio_crypto_hash_create_session_req {
> + struct virtio_crypto_hash_session_para para;
> + __u8 padding[40];
> +};
> +
> +struct virtio_crypto_mac_session_para {
> +#define VIRTIO_CRYPTO_NO_MAC 0
> +#define VIRTIO_CRYPTO_MAC_HMAC_MD5 1
> +#define VIRTIO_CRYPTO_MAC_HMAC_SHA1 2
> +#define VIRTIO_CRYPTO_MAC_HMAC_SHA_224 3
> +#define VIRTIO_CRYPTO_MAC_HMAC_SHA_256 4
> +#define VIRTIO_CRYPTO_MAC_HMAC_SHA_384 5
> +#define VIRTIO_CRYPTO_MAC_HMAC_SHA_512 6
> +#define VIRTIO_CRYPTO_MAC_CMAC_3DES 25
> +#define VIRTIO_CRYPTO_MAC_CMAC_AES 26
> +#define VIRTIO_CRYPTO_MAC_KASUMI_F9 27
> +#define VIRTIO_CRYPTO_MAC_SNOW3G_UIA2 28
> +#define VIRTIO_CRYPTO_MAC_GMAC_AES 41
> +#define VIRTIO_CRYPTO_MAC_GMAC_TWOFISH 42
> +#define VIRTIO_CRYPTO_MAC_CBCMAC_AES 49
> +#define VIRTIO_CRYPTO_MAC_CBCMAC_KASUMI_F9 50
> +#define VIRTIO_CRYPTO_MAC_XCBC_AES 53
> + __le32 algo;
> + /* hash result length */
> + __le32 hash_result_len;
> + /* length of authenticated key */
> + __le32 auth_key_len;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_mac_create_session_req {
> + struct virtio_crypto_mac_session_para para;
> + __u8 padding[40];
> +};
> +
> +struct virtio_crypto_aead_session_para {
> +#define VIRTIO_CRYPTO_NO_AEAD 0
> +#define VIRTIO_CRYPTO_AEAD_GCM 1
> +#define VIRTIO_CRYPTO_AEAD_CCM 2
> +#define VIRTIO_CRYPTO_AEAD_CHACHA20_POLY1305 3
> + __le32 algo;
> + /* length of key */
> + __le32 key_len;
> + /* hash result length */
> + __le32 hash_result_len;
> + /* length of the additional authenticated data (AAD) in bytes */
> + __le32 aad_len;
> + /* encrypt or decrypt, See above VIRTIO_CRYPTO_OP_* */
> + __le32 op;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_aead_create_session_req {
> + struct virtio_crypto_aead_session_para para;
> + __u8 padding[32];
> +};
> +
> +struct virtio_crypto_alg_chain_session_para {
> +#define VIRTIO_CRYPTO_SYM_ALG_CHAIN_ORDER_HASH_THEN_CIPHER 1
> +#define VIRTIO_CRYPTO_SYM_ALG_CHAIN_ORDER_CIPHER_THEN_HASH 2
> + __le32 alg_chain_order;
> +/* Plain hash */
> +#define VIRTIO_CRYPTO_SYM_HASH_MODE_PLAIN 1
> +/* Authenticated hash (mac) */
> +#define VIRTIO_CRYPTO_SYM_HASH_MODE_AUTH 2
> +/* Nested hash */
> +#define VIRTIO_CRYPTO_SYM_HASH_MODE_NESTED 3
> + __le32 hash_mode;
> + struct virtio_crypto_cipher_session_para cipher_param;
> + union {
> + struct virtio_crypto_hash_session_para hash_param;
> + struct virtio_crypto_mac_session_para mac_param;
> + __u8 padding[16];
> + } u;
> + /* length of the additional authenticated data (AAD) in bytes */
> + __le32 aad_len;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_alg_chain_session_req {
> + struct virtio_crypto_alg_chain_session_para para;
> +};
> +
> +struct virtio_crypto_sym_create_session_req {
> + union {
> + struct virtio_crypto_cipher_session_req cipher;
> + struct virtio_crypto_alg_chain_session_req chain;
> + __u8 padding[48];
> + } u;
> +
> + /* Device-readable part */
> +
> +/* No operation */
> +#define VIRTIO_CRYPTO_SYM_OP_NONE 0
> +/* Cipher only operation on the data */
> +#define VIRTIO_CRYPTO_SYM_OP_CIPHER 1
> +/*
> + * Chain any cipher with any hash or mac operation. The order
> + * depends on the value of alg_chain_order param
> + */
> +#define VIRTIO_CRYPTO_SYM_OP_ALGORITHM_CHAINING 2
> + __le32 op_type;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_destroy_session_req {
> + /* Device-readable part */
> + __le64 session_id;
> + __u8 padding[48];
> +};
> +
> +/* The request of the control virtqueue's packet */
> +struct virtio_crypto_op_ctrl_req {
> + struct virtio_crypto_ctrl_header header;
> +
> + union {
> + struct virtio_crypto_sym_create_session_req
> + sym_create_session;
> + struct virtio_crypto_hash_create_session_req
> + hash_create_session;
> + struct virtio_crypto_mac_create_session_req
> + mac_create_session;
> + struct virtio_crypto_aead_create_session_req
> + aead_create_session;
> + struct virtio_crypto_destroy_session_req
> + destroy_session;
> + __u8 padding[56];
> + } u;
> +};
> +
> +struct virtio_crypto_op_header {
> +#define VIRTIO_CRYPTO_CIPHER_ENCRYPT \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_CIPHER, 0x00)
> +#define VIRTIO_CRYPTO_CIPHER_DECRYPT \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_CIPHER, 0x01)
> +#define VIRTIO_CRYPTO_HASH \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_HASH, 0x00)
> +#define VIRTIO_CRYPTO_MAC \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_MAC, 0x00)
> +#define VIRTIO_CRYPTO_AEAD_ENCRYPT \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AEAD, 0x00)
> +#define VIRTIO_CRYPTO_AEAD_DECRYPT \
> + VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AEAD, 0x01)
> + __le32 opcode;
> + /* algo should be service-specific algorithms */
> + __le32 algo;
> + /* session_id should be service-specific algorithms */
> + __le64 session_id;
> + /* control flag to control the request */
> + __le32 flag;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_cipher_para {
> + /*
> + * Byte Length of valid IV/Counter
> + *
> + * For block ciphers in CBC or F8 mode, or for Kasumi in F8 mode, or for
> + * SNOW3G in UEA2 mode, this is the length of the IV (which
> + * must be the same as the block length of the cipher).
> + * For block ciphers in CTR mode, this is the length of the counter
> + * (which must be the same as the block length of the cipher).
> + * For AES-XTS, this is the 128bit tweak, i, from IEEE Std 1619-2007.
> + *
> + * The IV/Counter will be updated after every partial cryptographic
> + * operation.
> + */
> + __le32 iv_len;
> + /* length of source data */
> + __le32 src_data_len;
> + /* length of dst data */
> + __le32 dst_data_len;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_hash_para {
> + /* length of source data */
> + __le32 src_data_len;
> + /* hash result length */
> + __le32 hash_result_len;
> +};
> +
> +struct virtio_crypto_mac_para {
> + struct virtio_crypto_hash_para hash;
> +};
> +
> +struct virtio_crypto_aead_para {
> + /*
> + * Byte Length of valid IV data pointed to by the below iv_addr
> + * parameter.
> + *
> + * For GCM mode, this is either 12 (for 96-bit IVs) or 16, in which
> + * case iv_addr points to J0.
> + * For CCM mode, this is the length of the nonce, which can be in the
> + * range 7 to 13 inclusive.
> + */
> + __le32 iv_len;
> + /* length of additional auth data */
> + __le32 aad_len;
> + /* length of source data */
> + __le32 src_data_len;
> + /* length of dst data */
> + __le32 dst_data_len;
> +};
> +
> +struct virtio_crypto_cipher_data_req {
> + /* Device-readable part */
> + struct virtio_crypto_cipher_para para;
> + __u8 padding[24];
> +};
> +
> +struct virtio_crypto_hash_data_req {
> + /* Device-readable part */
> + struct virtio_crypto_hash_para para;
> + __u8 padding[40];
> +};
> +
> +struct virtio_crypto_mac_data_req {
> + /* Device-readable part */
> + struct virtio_crypto_mac_para para;
> + __u8 padding[40];
> +};
> +
> +struct virtio_crypto_alg_chain_data_para {
> + __le32 iv_len;
> + /* Length of source data */
> + __le32 src_data_len;
> + /* Length of destination data */
> + __le32 dst_data_len;
> + /* Starting point for cipher processing in source data */
> + __le32 cipher_start_src_offset;
> + /* Length of the source data that the cipher will be computed on */
> + __le32 len_to_cipher;
> + /* Starting point for hash processing in source data */
> + __le32 hash_start_src_offset;
> + /* Length of the source data that the hash will be computed on */
> + __le32 len_to_hash;
> + /* Length of the additional auth data */
> + __le32 aad_len;
> + /* Length of the hash result */
> + __le32 hash_result_len;
> + __le32 reserved;
> +};
> +
> +struct virtio_crypto_alg_chain_data_req {
> + /* Device-readable part */
> + struct virtio_crypto_alg_chain_data_para para;
> +};
> +
> +struct virtio_crypto_sym_data_req {
> + union {
> + struct virtio_crypto_cipher_data_req cipher;
> + struct virtio_crypto_alg_chain_data_req chain;
> + __u8 padding[40];
> + } u;
> +
> + /* See above VIRTIO_CRYPTO_SYM_OP_* */
> + __le32 op_type;
> + __le32 padding;
> +};
> +
> +struct virtio_crypto_aead_data_req {
> + /* Device-readable part */
> + struct virtio_crypto_aead_para para;
> + __u8 padding[32];
> +};
> +
> +/* The request of the data virtqueue's packet */
> +struct virtio_crypto_op_data_req {
> + struct virtio_crypto_op_header header;
> +
> + union {
> + struct virtio_crypto_sym_data_req sym_req;
> + struct virtio_crypto_hash_data_req hash_req;
> + struct virtio_crypto_mac_data_req mac_req;
> + struct virtio_crypto_aead_data_req aead_req;
> + __u8 padding[48];
> + } u;
> +};
> +
> +#define VIRTIO_CRYPTO_OK 0
> +#define VIRTIO_CRYPTO_ERR 1
> +#define VIRTIO_CRYPTO_BADMSG 2
> +#define VIRTIO_CRYPTO_NOTSUPP 3
> +#define VIRTIO_CRYPTO_INVSESS 4 /* Invalid session id */
> +
> +/* The accelerator hardware is ready */
> +#define VIRTIO_CRYPTO_S_HW_READY (1 << 0)
> +
> +struct virtio_crypto_config {
> + /* See VIRTIO_CRYPTO_OP_* above */
> + __u32 status;
> +
> + /*
> + * Maximum number of data queue
> + */
> + __u32 max_dataqueues;
> +
> + /*
> + * Specifies the services mask which the device support,
> + * see VIRTIO_CRYPTO_SERVICE_* above
> + */
> + __u32 crypto_services;
> +
> + /* Detailed algorithms mask */
> + __u32 cipher_algo_l;
> + __u32 cipher_algo_h;
> + __u32 hash_algo;
> + __u32 mac_algo_l;
> + __u32 mac_algo_h;
> + __u32 aead_algo;
> + /* Maximum length of cipher key */
> + __u32 max_cipher_key_len;
> + /* Maximum length of authenticated key */
> + __u32 max_auth_key_len;
> + __u32 reserve;
> + /* Maximum size of each crypto request's content */
> + __u64 max_size;
> +};
> +
> +struct virtio_crypto_inhdr {
> + /* See VIRTIO_CRYPTO_* above */
> + __u8 status;
> +};
> +#endif
> diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
> index 3228d58..6d5c3b2 100644
> --- a/include/uapi/linux/virtio_ids.h
> +++ b/include/uapi/linux/virtio_ids.h
> @@ -42,5 +42,6 @@
> #define VIRTIO_ID_GPU 16 /* virtio GPU */
> #define VIRTIO_ID_INPUT 18 /* virtio input */
> #define VIRTIO_ID_VSOCK 19 /* virtio vsock transport */
> +#define VIRTIO_ID_CRYPTO 20 /* virtio crypto */
>
> #endif /* _LINUX_VIRTIO_IDS_H */
> --
> 1.8.3.1
>
^ permalink raw reply
* Re: [PATCH v2] siphash: add cryptographically secure hashtable function
From: Jason A. Donenfeld @ 2016-12-12 5:48 UTC (permalink / raw)
To: Linus Torvalds
Cc: kernel-hardening@lists.openwall.com, LKML,
Linux Crypto Mailing List, George Spelvin, Scott Bauer,
Andi Kleen, Andy Lutomirski, Greg KH, Jean-Philippe Aumasson,
Daniel J . Bernstein
In-Reply-To: <CA+55aFyfijNTvi0AN1kC4oWZqdGyoRD4WUVAf+kjFytVOE3kNw@mail.gmail.com>
Hey Linus,
On Mon, Dec 12, 2016 at 5:01 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> The above is extremely inefficient. Considering that most kernel data
> would be expected to be smallish, that matters (ie the usual benchmark
> would not be about hashing megabytes of data, but instead millions of
> hashes of small data).
>
> I think this could be rewritten (at least for 64-bit architectures) as
>
> #ifdef CONFIG_DCACHE_WORD_ACCESS
>
> if (left)
> b |= le64_to_cpu(load_unaligned_zeropad(data) &
> bytemask_from_count(left));
>
> #else
>
> .. do the duff's device thing with the switch() ..
>
> #endif
>
> which should give you basically perfect code generation (ie a single
> 64-bit load and a byte mask).
I modified the test to hash data of size 0 through 7 repeatedly
100000000 times, and benchmarked that a few times on a Skylake laptop.
The `load_unaligned_zeropad & bytemask_from_count` version was
consistently 7% slower.
I then modified it again to simply hash a 4 byte constant repeatedly
1000000000 times. The `load_unaligned_zeropad & bytemask_from_count`
version was around 6% faster. I tried again with a 7 byte constant and
got more or less a similar result.
Then I tried with a 1 byte constant, and found that the
`load_unaligned_zeropad & bytemask_from_count` version was slower.
So, it would seem that between the `if (left)` and the `switch
(left)`, there's the same number of branches. But for small values of
`left`, the duff's device just has simpler arithmetic, whereas for
large values of `left`, the `load_unaligned_zeropad` prevails. If
micro-optimization is really appealing, one could imagine a hybrid of
the two:
switch (left) {
case 7:
case 6:
case 5:
case 4:
b |= le64_to_cpu(load_unaligned_zeropad(data) &
bytemask_from_count(left));
break;
case 3: b |= ((u64)data[2]) << 16;
case 2: b |= ((u64)data[1]) << 8;
case 1: b |= ((u64)data[0]); break;
case 0: break;
}
But I'm not sure this complication is worth it, and it might be more
likely that the left-over size is 4 bytes most of the time, so we
should just use your trick on platforms that support it.
Jason
^ permalink raw reply
* Re: [PATCH v2] siphash: add cryptographically secure hashtable function
From: Eric Biggers @ 2016-12-12 5:42 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: kernel-hardening, LKML, linux-crypto, Linus Torvalds,
George Spelvin, Scott Bauer, ak, Andy Lutomirski, Greg KH,
Jean-Philippe Aumasson, Daniel J . Bernstein
In-Reply-To: <20161212034817.1773-1-Jason@zx2c4.com>
On Mon, Dec 12, 2016 at 04:48:17AM +0100, Jason A. Donenfeld wrote:
>
> diff --git a/lib/Makefile b/lib/Makefile
> index 50144a3aeebd..71d398b04a74 100644
> --- a/lib/Makefile
> +++ b/lib/Makefile
> @@ -22,7 +22,8 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
> sha1.o chacha20.o md5.o irq_regs.o argv_split.o \
> flex_proportions.o ratelimit.o show_mem.o \
> is_single_threaded.o plist.o decompress.o kobject_uevent.o \
> - earlycpio.o seq_buf.o nmi_backtrace.o nodemask.o win_minmax.o
> + earlycpio.o seq_buf.o siphash.o \
> + nmi_backtrace.o nodemask.o win_minmax.o
>
> lib-$(CONFIG_MMU) += ioremap.o
> lib-$(CONFIG_SMP) += cpumask.o
> @@ -44,7 +45,7 @@ obj-$(CONFIG_TEST_HEXDUMP) += test_hexdump.o
> obj-y += kstrtox.o
> obj-$(CONFIG_TEST_BPF) += test_bpf.o
> obj-$(CONFIG_TEST_FIRMWARE) += test_firmware.o
> -obj-$(CONFIG_TEST_HASH) += test_hash.o
> +obj-$(CONFIG_TEST_HASH) += test_hash.o test_siphash.o
Maybe add to the help text for CONFIG_TEST_HASH that it now tests siphash too?
> +static inline u64 le64_to_cpuvp(const void *p)
> +{
> + return le64_to_cpup(p);
> +}
This assumes the key and message buffers are aligned to __alignof__(u64).
Unless that's going to be a clearly documented requirement for callers, you
should use get_unaligned_le64() instead. And you can pass a 'u8 *' directly to
get_unaligned_le64(), no need for a helper function.
> + b = (v0 ^ v1) ^ (v2 ^ v3);
> + return (__force u64)cpu_to_le64(b);
> +}
It makes sense for this to return a u64, but that means the cpu_to_le64() is
wrong, since u64 indicates CPU endianness. It should just return 'b'.
> +++ b/lib/test_siphash.c
> @@ -0,0 +1,116 @@
> +/* Test cases for siphash.c
> + *
> + * Copyright (C) 2015-2016 Jason A. Donenfeld <Jason@zx2c4.com>
> + *
> + * This file is provided under a dual BSD/GPLv2 license.
> + *
> + * SipHash: a fast short-input PRF
> + * https://131002.net/siphash/
> + */
> +
> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
> +
> +#include <linux/siphash.h>
> +#include <linux/kernel.h>
> +#include <linux/string.h>
> +#include <linux/errno.h>
> +#include <linux/module.h>
> +
> +static const u8 test_vectors[64][8] = {
> + { 0x31, 0x0e, 0x0e, 0xdd, 0x47, 0xdb, 0x6f, 0x72 },
Can you mention in a comment where the test vectors came from?
> + if (memcmp(&out, test_vectors[i], 8)) {
> + pr_info("self-test %u: FAIL\n", i + 1);
> + ret = -EINVAL;
> + }
If you make the output really be CPU-endian like I'm suggesting then this will
need to be something like:
if (out != get_unaligned_le64(test_vectors[i])) {
Or else make the test vectors be an array of u64.
- Eric
^ permalink raw reply
* Re: [PATCH v2] siphash: add cryptographically secure hashtable function
From: Linus Torvalds @ 2016-12-12 4:01 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: kernel-hardening@lists.openwall.com, LKML,
Linux Crypto Mailing List, George Spelvin, Scott Bauer,
Andi Kleen, Andy Lutomirski, Greg KH, Jean-Philippe Aumasson,
Daniel J . Bernstein
In-Reply-To: <20161212034817.1773-1-Jason@zx2c4.com>
On Sun, Dec 11, 2016 at 7:48 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> + switch (left) {
> + case 7: b |= ((u64)data[6]) << 48;
> + case 6: b |= ((u64)data[5]) << 40;
> + case 5: b |= ((u64)data[4]) << 32;
> + case 4: b |= ((u64)data[3]) << 24;
> + case 3: b |= ((u64)data[2]) << 16;
> + case 2: b |= ((u64)data[1]) << 8;
> + case 1: b |= ((u64)data[0]); break;
> + case 0: break;
> + }
The above is extremely inefficient. Considering that most kernel data
would be expected to be smallish, that matters (ie the usual benchmark
would not be about hashing megabytes of data, but instead millions of
hashes of small data).
I think this could be rewritten (at least for 64-bit architectures) as
#ifdef CONFIG_DCACHE_WORD_ACCESS
if (left)
b |= le64_to_cpu(load_unaligned_zeropad(data) &
bytemask_from_count(left));
#else
.. do the duff's device thing with the switch() ..
#endif
which should give you basically perfect code generation (ie a single
64-bit load and a byte mask).
Totally untested, just looking at the code and trying to make sense of it.
... and obviously, it requires an actual high-performance use-case to
make any difference.
Linus
^ permalink raw reply
* [PATCH v2] siphash: add cryptographically secure hashtable function
From: Jason A. Donenfeld @ 2016-12-12 3:48 UTC (permalink / raw)
To: kernel-hardening, LKML, linux-crypto, Linus Torvalds,
George Spelvin, Scott Bauer, ak, Andy Lutomirski, Greg KH
Cc: Jason A. Donenfeld, Jean-Philippe Aumasson, Daniel J . Bernstein
In-Reply-To: <20161211204345.GA1558@kroah.com>
SipHash is a 64-bit keyed hash function that is actually a
cryptographically secure PRF, like HMAC. Except SipHash is super fast,
and is meant to be used as a hashtable keyed lookup function.
SipHash isn't just some new trendy hash function. It's been around for a
while, and there really isn't anything that comes remotely close to
being useful in the way SipHash is. With that said, why do we need this?
There are a variety of attacks known as "hashtable poisoning" in which an
attacker forms some data such that the hash of that data will be the
same, and then preceeds to fill up all entries of a hashbucket. This is
a realistic and well-known denial-of-service vector.
Linux developers already seem to be aware that this is an issue, and
various places that use hash tables in, say, a network context, use a
non-cryptographically secure function (usually jhash) and then try to
twiddle with the key on a time basis (or in many cases just do nothing
and hope that nobody notices). While this is an admirable attempt at
solving the problem, it doesn't actually fix it. SipHash fixes it.
(It fixes it in such a sound way that you could even build a stream
cipher out of SipHash that would resist the modern cryptanalysis.)
There are a modicum of places in the kernel that are vulnerable to
hashtable poisoning attacks, either via userspace vectors or network
vectors, and there's not a reliable mechanism inside the kernel at the
moment to fix it. The first step toward fixing these issues is actually
getting a secure primitive into the kernel for developers to use. Then
we can, bit by bit, port things over to it as deemed appropriate.
Dozens of languages are already using this internally for their hash
tables. Some of the BSDs already use this in their kernels. SipHash is
a widely known high-speed solution to a widely known problem, and it's
time we catch-up.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
Cc: Daniel J. Bernstein <djb@cr.yp.to>
---
include/linux/siphash.h | 20 +++++++++
lib/Makefile | 5 ++-
lib/siphash.c | 72 ++++++++++++++++++++++++++++++
lib/test_siphash.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 211 insertions(+), 2 deletions(-)
create mode 100644 include/linux/siphash.h
create mode 100644 lib/siphash.c
create mode 100644 lib/test_siphash.c
diff --git a/include/linux/siphash.h b/include/linux/siphash.h
new file mode 100644
index 000000000000..6623b3090645
--- /dev/null
+++ b/include/linux/siphash.h
@@ -0,0 +1,20 @@
+/* Copyright (C) 2016 Jason A. Donenfeld <Jason@zx2c4.com>
+ *
+ * This file is provided under a dual BSD/GPLv2 license.
+ *
+ * SipHash: a fast short-input PRF
+ * https://131002.net/siphash/
+ */
+
+#ifndef _LINUX_SIPHASH_H
+#define _LINUX_SIPHASH_H
+
+#include <linux/types.h>
+
+enum siphash_lengths {
+ SIPHASH24_KEY_LEN = 16
+};
+
+u64 siphash24(const u8 *data, size_t len, const u8 key[SIPHASH24_KEY_LEN]);
+
+#endif /* _LINUX_SIPHASH_H */
diff --git a/lib/Makefile b/lib/Makefile
index 50144a3aeebd..71d398b04a74 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -22,7 +22,8 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
sha1.o chacha20.o md5.o irq_regs.o argv_split.o \
flex_proportions.o ratelimit.o show_mem.o \
is_single_threaded.o plist.o decompress.o kobject_uevent.o \
- earlycpio.o seq_buf.o nmi_backtrace.o nodemask.o win_minmax.o
+ earlycpio.o seq_buf.o siphash.o \
+ nmi_backtrace.o nodemask.o win_minmax.o
lib-$(CONFIG_MMU) += ioremap.o
lib-$(CONFIG_SMP) += cpumask.o
@@ -44,7 +45,7 @@ obj-$(CONFIG_TEST_HEXDUMP) += test_hexdump.o
obj-y += kstrtox.o
obj-$(CONFIG_TEST_BPF) += test_bpf.o
obj-$(CONFIG_TEST_FIRMWARE) += test_firmware.o
-obj-$(CONFIG_TEST_HASH) += test_hash.o
+obj-$(CONFIG_TEST_HASH) += test_hash.o test_siphash.o
obj-$(CONFIG_TEST_KASAN) += test_kasan.o
obj-$(CONFIG_TEST_KSTRTOX) += test-kstrtox.o
obj-$(CONFIG_TEST_LKM) += test_module.o
diff --git a/lib/siphash.c b/lib/siphash.c
new file mode 100644
index 000000000000..e78dc36d19b9
--- /dev/null
+++ b/lib/siphash.c
@@ -0,0 +1,72 @@
+/* Copyright (C) 2015-2016 Jason A. Donenfeld <Jason@zx2c4.com>
+ * Copyright (C) 2012-2014 Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
+ * Copyright (C) 2012-2014 Daniel J. Bernstein <djb@cr.yp.to>
+ *
+ * This file is provided under a dual BSD/GPLv2 license.
+ *
+ * SipHash: a fast short-input PRF
+ * https://131002.net/siphash/
+ */
+
+#include <linux/siphash.h>
+#include <linux/kernel.h>
+
+static inline u64 le64_to_cpuvp(const void *p)
+{
+ return le64_to_cpup(p);
+}
+
+#define SIPROUND \
+ do { \
+ v0 += v1; v1 = rol64(v1, 13); v1 ^= v0; v0 = rol64(v0, 32); \
+ v2 += v3; v3 = rol64(v3, 16); v3 ^= v2; \
+ v0 += v3; v3 = rol64(v3, 21); v3 ^= v0; \
+ v2 += v1; v1 = rol64(v1, 17); v1 ^= v2; v2 = rol64(v2, 32); \
+ } while(0)
+
+u64 siphash24(const u8 *data, size_t len, const u8 key[SIPHASH24_KEY_LEN])
+{
+ u64 v0 = 0x736f6d6570736575ULL;
+ u64 v1 = 0x646f72616e646f6dULL;
+ u64 v2 = 0x6c7967656e657261ULL;
+ u64 v3 = 0x7465646279746573ULL;
+ u64 b = ((u64)len) << 56;
+ u64 k0 = le64_to_cpuvp(key);
+ u64 k1 = le64_to_cpuvp(key + sizeof(u64));
+ u64 m;
+ const u8 *end = data + len - (len % sizeof(u64));
+ const u8 left = len & (sizeof(u64) - 1);
+ v3 ^= k1;
+ v2 ^= k0;
+ v1 ^= k1;
+ v0 ^= k0;
+ for (; data != end; data += sizeof(u64)) {
+ m = le64_to_cpuvp(data);
+ v3 ^= m;
+ SIPROUND;
+ SIPROUND;
+ v0 ^= m;
+ }
+ switch (left) {
+ case 7: b |= ((u64)data[6]) << 48;
+ case 6: b |= ((u64)data[5]) << 40;
+ case 5: b |= ((u64)data[4]) << 32;
+ case 4: b |= ((u64)data[3]) << 24;
+ case 3: b |= ((u64)data[2]) << 16;
+ case 2: b |= ((u64)data[1]) << 8;
+ case 1: b |= ((u64)data[0]); break;
+ case 0: break;
+ }
+ v3 ^= b;
+ SIPROUND;
+ SIPROUND;
+ v0 ^= b;
+ v2 ^= 0xff;
+ SIPROUND;
+ SIPROUND;
+ SIPROUND;
+ SIPROUND;
+ b = (v0 ^ v1) ^ (v2 ^ v3);
+ return (__force u64)cpu_to_le64(b);
+}
+EXPORT_SYMBOL(siphash24);
diff --git a/lib/test_siphash.c b/lib/test_siphash.c
new file mode 100644
index 000000000000..45b5435540e9
--- /dev/null
+++ b/lib/test_siphash.c
@@ -0,0 +1,116 @@
+/* Test cases for siphash.c
+ *
+ * Copyright (C) 2015-2016 Jason A. Donenfeld <Jason@zx2c4.com>
+ *
+ * This file is provided under a dual BSD/GPLv2 license.
+ *
+ * SipHash: a fast short-input PRF
+ * https://131002.net/siphash/
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/siphash.h>
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <linux/errno.h>
+#include <linux/module.h>
+
+static const u8 test_vectors[64][8] = {
+ { 0x31, 0x0e, 0x0e, 0xdd, 0x47, 0xdb, 0x6f, 0x72 },
+ { 0xfd, 0x67, 0xdc, 0x93, 0xc5, 0x39, 0xf8, 0x74 },
+ { 0x5a, 0x4f, 0xa9, 0xd9, 0x09, 0x80, 0x6c, 0x0d },
+ { 0x2d, 0x7e, 0xfb, 0xd7, 0x96, 0x66, 0x67, 0x85 },
+ { 0xb7, 0x87, 0x71, 0x27, 0xe0, 0x94, 0x27, 0xcf },
+ { 0x8d, 0xa6, 0x99, 0xcd, 0x64, 0x55, 0x76, 0x18 },
+ { 0xce, 0xe3, 0xfe, 0x58, 0x6e, 0x46, 0xc9, 0xcb },
+ { 0x37, 0xd1, 0x01, 0x8b, 0xf5, 0x00, 0x02, 0xab },
+ { 0x62, 0x24, 0x93, 0x9a, 0x79, 0xf5, 0xf5, 0x93 },
+ { 0xb0, 0xe4, 0xa9, 0x0b, 0xdf, 0x82, 0x00, 0x9e },
+ { 0xf3, 0xb9, 0xdd, 0x94, 0xc5, 0xbb, 0x5d, 0x7a },
+ { 0xa7, 0xad, 0x6b, 0x22, 0x46, 0x2f, 0xb3, 0xf4 },
+ { 0xfb, 0xe5, 0x0e, 0x86, 0xbc, 0x8f, 0x1e, 0x75 },
+ { 0x90, 0x3d, 0x84, 0xc0, 0x27, 0x56, 0xea, 0x14 },
+ { 0xee, 0xf2, 0x7a, 0x8e, 0x90, 0xca, 0x23, 0xf7 },
+ { 0xe5, 0x45, 0xbe, 0x49, 0x61, 0xca, 0x29, 0xa1 },
+ { 0xdb, 0x9b, 0xc2, 0x57, 0x7f, 0xcc, 0x2a, 0x3f },
+ { 0x94, 0x47, 0xbe, 0x2c, 0xf5, 0xe9, 0x9a, 0x69 },
+ { 0x9c, 0xd3, 0x8d, 0x96, 0xf0, 0xb3, 0xc1, 0x4b },
+ { 0xbd, 0x61, 0x79, 0xa7, 0x1d, 0xc9, 0x6d, 0xbb },
+ { 0x98, 0xee, 0xa2, 0x1a, 0xf2, 0x5c, 0xd6, 0xbe },
+ { 0xc7, 0x67, 0x3b, 0x2e, 0xb0, 0xcb, 0xf2, 0xd0 },
+ { 0x88, 0x3e, 0xa3, 0xe3, 0x95, 0x67, 0x53, 0x93 },
+ { 0xc8, 0xce, 0x5c, 0xcd, 0x8c, 0x03, 0x0c, 0xa8 },
+ { 0x94, 0xaf, 0x49, 0xf6, 0xc6, 0x50, 0xad, 0xb8 },
+ { 0xea, 0xb8, 0x85, 0x8a, 0xde, 0x92, 0xe1, 0xbc },
+ { 0xf3, 0x15, 0xbb, 0x5b, 0xb8, 0x35, 0xd8, 0x17 },
+ { 0xad, 0xcf, 0x6b, 0x07, 0x63, 0x61, 0x2e, 0x2f },
+ { 0xa5, 0xc9, 0x1d, 0xa7, 0xac, 0xaa, 0x4d, 0xde },
+ { 0x71, 0x65, 0x95, 0x87, 0x66, 0x50, 0xa2, 0xa6 },
+ { 0x28, 0xef, 0x49, 0x5c, 0x53, 0xa3, 0x87, 0xad },
+ { 0x42, 0xc3, 0x41, 0xd8, 0xfa, 0x92, 0xd8, 0x32 },
+ { 0xce, 0x7c, 0xf2, 0x72, 0x2f, 0x51, 0x27, 0x71 },
+ { 0xe3, 0x78, 0x59, 0xf9, 0x46, 0x23, 0xf3, 0xa7 },
+ { 0x38, 0x12, 0x05, 0xbb, 0x1a, 0xb0, 0xe0, 0x12 },
+ { 0xae, 0x97, 0xa1, 0x0f, 0xd4, 0x34, 0xe0, 0x15 },
+ { 0xb4, 0xa3, 0x15, 0x08, 0xbe, 0xff, 0x4d, 0x31 },
+ { 0x81, 0x39, 0x62, 0x29, 0xf0, 0x90, 0x79, 0x02 },
+ { 0x4d, 0x0c, 0xf4, 0x9e, 0xe5, 0xd4, 0xdc, 0xca },
+ { 0x5c, 0x73, 0x33, 0x6a, 0x76, 0xd8, 0xbf, 0x9a },
+ { 0xd0, 0xa7, 0x04, 0x53, 0x6b, 0xa9, 0x3e, 0x0e },
+ { 0x92, 0x59, 0x58, 0xfc, 0xd6, 0x42, 0x0c, 0xad },
+ { 0xa9, 0x15, 0xc2, 0x9b, 0xc8, 0x06, 0x73, 0x18 },
+ { 0x95, 0x2b, 0x79, 0xf3, 0xbc, 0x0a, 0xa6, 0xd4 },
+ { 0xf2, 0x1d, 0xf2, 0xe4, 0x1d, 0x45, 0x35, 0xf9 },
+ { 0x87, 0x57, 0x75, 0x19, 0x04, 0x8f, 0x53, 0xa9 },
+ { 0x10, 0xa5, 0x6c, 0xf5, 0xdf, 0xcd, 0x9a, 0xdb },
+ { 0xeb, 0x75, 0x09, 0x5c, 0xcd, 0x98, 0x6c, 0xd0 },
+ { 0x51, 0xa9, 0xcb, 0x9e, 0xcb, 0xa3, 0x12, 0xe6 },
+ { 0x96, 0xaf, 0xad, 0xfc, 0x2c, 0xe6, 0x66, 0xc7 },
+ { 0x72, 0xfe, 0x52, 0x97, 0x5a, 0x43, 0x64, 0xee },
+ { 0x5a, 0x16, 0x45, 0xb2, 0x76, 0xd5, 0x92, 0xa1 },
+ { 0xb2, 0x74, 0xcb, 0x8e, 0xbf, 0x87, 0x87, 0x0a },
+ { 0x6f, 0x9b, 0xb4, 0x20, 0x3d, 0xe7, 0xb3, 0x81 },
+ { 0xea, 0xec, 0xb2, 0xa3, 0x0b, 0x22, 0xa8, 0x7f },
+ { 0x99, 0x24, 0xa4, 0x3c, 0xc1, 0x31, 0x57, 0x24 },
+ { 0xbd, 0x83, 0x8d, 0x3a, 0xaf, 0xbf, 0x8d, 0xb7 },
+ { 0x0b, 0x1a, 0x2a, 0x32, 0x65, 0xd5, 0x1a, 0xea },
+ { 0x13, 0x50, 0x79, 0xa3, 0x23, 0x1c, 0xe6, 0x60 },
+ { 0x93, 0x2b, 0x28, 0x46, 0xe4, 0xd7, 0x06, 0x66 },
+ { 0xe1, 0x91, 0x5f, 0x5c, 0xb1, 0xec, 0xa4, 0x6c },
+ { 0xf3, 0x25, 0x96, 0x5c, 0xa1, 0x6d, 0x62, 0x9f },
+ { 0x57, 0x5f, 0xf2, 0x8e, 0x60, 0x38, 0x1b, 0xe5 },
+ { 0x72, 0x45, 0x06, 0xeb, 0x4c, 0x32, 0x8a, 0x95 }
+};
+
+static int __init siphash_test_init(void)
+{
+ u8 in[64], k[16], i;
+ u64 out;
+ int ret = 0;
+
+ for (i = 0; i < 16; ++i)
+ k[i] = i;
+
+ for (i = 0; i < 64; ++i) {
+ in[i] = i;
+ out = siphash24(in, i, k);
+ if (memcmp(&out, test_vectors[i], 8)) {
+ pr_info("self-test %u: FAIL\n", i + 1);
+ ret = -EINVAL;
+ }
+ }
+ if (!ret)
+ pr_info("self-tests: pass\n");
+ return ret;
+}
+
+static void __exit siphash_test_exit(void)
+{
+}
+
+module_init(siphash_test_init);
+module_exit(siphash_test_exit);
+
+MODULE_AUTHOR("Jason A. Donenfeld <Jason@zx2c4.com>");
+MODULE_LICENSE("Dual BSD/GPL");
--
2.11.0
^ permalink raw reply related
* kernel crash when using sha1 as csums-alg for drbd
From: Zhang Zhuoyu @ 2016-12-12 3:31 UTC (permalink / raw)
To: mouli; +Cc: marex, hpa, herbert, 'lixiubo', linux-crypto
Hello, Chandramouli
Sorry for last email.
These days we experienced 5 times kernel crash issue when using sha1 as
csums-alg for drbd on our CentOS7.2 3.10.0-327.el7.x86_64:
Kernel log as below:
[19839335.792807] BUG: unable to handle kernel paging request at
ffff88007bd4f000
[19839335.793145] IP: [<ffffffff8106a908>] _begin+0x28/0x187
[19839335.793326] PGD 1f32067 PUD 607ffff067 PMD 1f35067 PTE 0
[19839335.793510] Oops: 0000 [#1] SMP
[19839335.793683] Modules linked in: dm_service_time iscsi_tcp libiscsi_tcp
libiscsi scsi_transport_iscsi nf_conntrack_netlink nf_conntrack_ipv6
nf_defrag_ipv6 xt_mac xt_set xt_physdev xt_CT ip_set_hash_net ip_set
nfnetlink vhost_net vhost macvtap macvlan veth iptable_raw iptable_filter
iptable_nat nf_nat_ipv4 iptable_mangle ip_tables dm_multipath ip6table_raw
vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch xt_multiport
ipmi_devintf xt_comment ext4 mbcache jbd2 xt_CHECKSUM ipt_MASQUERADE
nf_nat_masquerade_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
nf_conntrack ipt_REJECT tun bridge ebtable_filter ebtables ip6table_filter
ip6_tables drbd(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash
dm_log iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl
kvm_intel kvm
[19839335.795640] crc32_pclmul dm_mod ghash_clmulni_intel aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd pcspkr ses ipmi_ssif enclosure sg
sb_edac edac_core lpc_ich mei_me i2c_i801 mfd_core mei ioatdma shpchp wmi
ipmi_si ipmi_msghandler acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl
lockd grace sunrpc xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common
crc32c_intel drm_kms_helper ttm ixgbe drm igb mdio ptp mpt3sas pps_core
i2c_algo_bit raid_class dca i2c_core scsi_transport_sas [last unloaded:
ip_tables][19839335.797216] CPU: 1 PID: 2912 Comm: drbd_w_drbd1 Tainted: G
OE ------------ 3.10.0-327.el7.x86_64 #1
[19839335.797550] Hardware name: Inspur NF5280M4/YZMB-00326-101, BIOS 4.0.18
11/09/2015
[19839335.797877] task: ffff885f749b9700 ti: ffff882f62fc4000 task.ti:
ffff882f62fc4000
[19839335.798203] RIP: 0010:[<ffffffff8106a908>] [<ffffffff8106a908>]
_begin+0x28/0x187
[19839335.798532] RSP: 0018:ffff882f62fc75f8 EFLAGS: 00010202
[19839335.798702] RAX: 000000002fced277 RBX: 00000000e9cee1cc RCX:
00000000a73b8733
[19839335.799030] RDX: 00000000b573ac7c RSI: 00000000bb6b5097 RDI:
00000000da4f4b14
[19839335.799356] RBP: 0000000058444804 R08: ffffffff81656100 R09:
ffff882f33147998
[19839335.799680] R10: ffff88007bd4ef80 R11: ffff88007bd4f040 R12:
00000000e770e674
[19839335.800010] R13: ffff88007bd4efc0 R14: ffff882f62fc75f8 R15:
ffff882f62fc7898
[19839335.800336] FS: 0000000000000000(0000) GS:ffff882fbf840000(0000)
knlGS:0000000000000000
[19839335.800664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[19839335.800835] CR2: ffff88007bd4f000 CR3: 000000000194a000 CR4:
00000000001427e0
[19839335.801160] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[19839335.801486] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[19839335.801812] Stack:
[19839335.801974] 5a8279995a827999 5a8279995a827999 5a8279995a827999
5a8279995a827999
[19839335.802317] 5a8279995a827999 5a8279995a827999 5a8279995a827999
5a8279995a827999
[19839335.802663] 5a8279995a827999 5a8279995a827999 5a8279995a827999
5a8279995a827999
[19839335.803005] Call Trace:
[19839335.803180] [<ffffffff81569a41>] ? ip_local_out_sk+0x31/0x40
[19839335.803355] [<ffffffff8106a31d>] ?
sha1_apply_transform_avx2+0x1d/0x30
[19839335.803530] [<ffffffff8106a063>] ? __sha1_ssse3_update+0x53/0xd0
[19839335.803704] [<ffffffff8106a388>] ? sha1_ssse3_update+0x58/0xf0
[19839335.803881] [<ffffffff812b1878>] ? crypto_shash_update+0x38/0x100
[19839335.804056] [<ffffffff812b1d6e>] ? shash_compat_update+0x4e/0x80
[19839335.804242] [<ffffffffa05245ab>] ? drbd_csum_bio+0x9b/0xe0 [drbd]
[19839335.804427] [<ffffffffa0546701>] ? drbd_send_dblock+0x3b1/0x480
[drbd]
[19839335.804608] [<ffffffffa0522a80>] ? dequeue_work_batch+0x20/0x90
[drbd]
[19839335.804788] [<ffffffffa0522d37>] ? wait_for_work+0x67/0x370 [drbd]
[19839335.804969] [<ffffffffa052726f>] ? w_send_dblock+0xaf/0x1d0 [drbd]
[19839335.805168] [<ffffffffa052867b>] ? drbd_worker+0xfb/0x390 [drbd]
[19839335.805349] [<ffffffffa0542430>] ?
drbd_destroy_connection+0x160/0x160 [drbd]
[19839335.805684] [<ffffffffa054244d>] ? drbd_thread_setup+0x1d/0x110
[drbd]
[19839335.805864] [<ffffffffa0542430>] ?
drbd_destroy_connection+0x160/0x160 [drbd]
[19839335.806195] [<ffffffff810a5aef>] ? kthread+0xcf/0xe0
[19839335.806367] [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[19839335.806545] [<ffffffff81645858>] ? ret_from_fork+0x58/0x90
[19839335.806717] [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[19839335.806889] Code: 00 00 00 89 f3 c4 e3 7b f0 f6 02 c4 e2 60 f2 e8 21
fb 31 eb 41 03 17 c4 e2 70 f2 ef 8d 14 1a c4 63 7b f0 e1 1b c4 e3 7b f0 d9
02 <c4> c1 7a 6f 82 80 00 00 00 21 f1 31 e9 42 8d 14 22 41 03 47 04
[19839335.807640] RIP [<ffffffff8106a908>] _begin+0x28/0x187
[19839335.807814] RSP <ffff882f62fc75f8>
[19839335.807979] CR2: ffff88007bd4f000
We debug it by using crash:
crash> bt
PID: 2912 TASK: ffff885f749b9700 CPU: 1 COMMAND: "drbd_w_drbd1"
#0 [ffff882f62fc72c0] machine_kexec at ffffffff81051beb
#1 [ffff882f62fc7320] crash_kexec at ffffffff810f2542
#2 [ffff882f62fc73f0] oops_end at ffffffff8163e1a8
#3 [ffff882f62fc7418] no_context at ffffffff8162e2b8
#4 [ffff882f62fc7468] __bad_area_nosemaphore at ffffffff8162e34e
#5 [ffff882f62fc74b0] bad_area_nosemaphore at ffffffff8162e4b8
#6 [ffff882f62fc74c0] __do_page_fault at ffffffff81640fce
#7 [ffff882f62fc7518] do_page_fault at ffffffff81641113
#8 [ffff882f62fc7540] page_fault at ffffffff8163d408
[exception RIP: _begin+40]
RIP: ffffffff8106a908 RSP: ffff882f62fc75f8 RFLAGS: 00010202
RAX: 000000002fced277 RBX: 00000000e9cee1cc RCX: 00000000a73b8733
RDX: 00000000b573ac7c RSI: 00000000bb6b5097 RDI: 00000000da4f4b14
RBP: 0000000058444804 R8: ffffffff81656100 R9: ffff882f33147998
R10: ffff88007bd4ef80 R11: ffff88007bd4f040 R12: 00000000e770e674
R13: ffff88007bd4efc0 R14: ffff882f62fc75f8 R15: ffff882f62fc7898
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff882f62fc7878] ip_local_out_sk at ffffffff81569a41
#10 [ffff882f62fc7ba8] sha1_apply_transform_avx2 at ffffffff8106a31d
#11 [ffff882f62fc7bb8] __sha1_ssse3_update at ffffffff8106a063
#12 [ffff882f62fc7bf8] sha1_ssse3_update at ffffffff8106a388
#13 [ffff882f62fc7c28] crypto_shash_update at ffffffff812b1878
#14 [ffff882f62fc7c78] shash_compat_update at ffffffff812b1d6e
#15 [ffff882f62fc7cc8] drbd_csum_bio at ffffffffa05245ab [drbd]
#16 [ffff882f62fc7d28] drbd_send_dblock at ffffffffa0546701 [drbd]
#17 [ffff882f62fc7de0] w_send_dblock at ffffffffa052726f [drbd]
#18 [ffff882f62fc7e28] drbd_worker at ffffffffa052867b [drbd]
#19 [ffff882f62fc7e98] drbd_thread_setup at ffffffffa054244d [drbd]
#20 [ffff882f62fc7ec8] kthread at ffffffff810a5aef
#21 [ffff882f62fc7f50] ret_from_fork at ffffffff81645858
crash> dis -l ffffffff8106a908
/usr/src/debug/kernel-3.10.0-327.el7/linux-3.10.0-327.el7.x86_64/arch/x86/cr
ypto/sha1_avx2_x86_64_asm.S: 677
0xffffffff8106a908 <_begin+40>: vmovdqu 0x80(%r10),%xmm0
crash> dis -l _begin
/usr/src/debug/kernel-3.10.0-327.el7/linux-3.10.0-327.el7.x86_64/arch/x86/cr
ypto/sha1_avx2_x86_64_asm.S: 677
0xffffffff8106a8e0 <_begin>: mov %esi,%ebx
0xffffffff8106a8e2 <_begin+2>: rorx $0x2,%esi,%esi
0xffffffff8106a8e8 <_begin+8>: andn %eax,%ebx,%ebp
0xffffffff8106a8ed <_begin+13>: and %edi,%ebx
0xffffffff8106a8ef <_begin+15>: xor %ebp,%ebx
0xffffffff8106a8f1 <_begin+17>: add (%r15),%edx
0xffffffff8106a8f4 <_begin+20>: andn %edi,%ecx,%ebp
0xffffffff8106a8f9 <_begin+25>: lea (%rdx,%rbx,1),%edx
0xffffffff8106a8fc <_begin+28>: rorx $0x1b,%ecx,%r12d
0xffffffff8106a902 <_begin+34>: rorx $0x2,%ecx,%ebx
0xffffffff8106a908 <_begin+40>: vmovdqu 0x80(%r10),%xmm0
<--------------- crash here
0xffffffff8106a911 <_begin+49>: and %esi,%ecx
0xffffffff8106a913 <_begin+51>: xor %ebp,%ecx
0xffffffff8106a915 <_begin+53>: lea (%rdx,%r12,1),%edx
0xffffffff8106a919 <_begin+57>: add 0x4(%r15),%eax
0xffffffff8106a91d <_begin+61>: andn %esi,%edx,%ebp
0xffffffff8106a922 <_begin+66>: lea (%rax,%rcx,1),%eax
0xffffffff8106a925 <_begin+69>: rorx $0x1b,%edx,%r12d
0xffffffff8106a92b <_begin+75>: rorx $0x2,%edx,%ecx
0xffffffff8106a931 <_begin+81>: vinsertf128 $0x1,0x80(%r13),%ymm0,%ymm0
0xffffffff8106a93b <_begin+91>: and %ebx,%edx
0xffffffff8106a93d <_begin+93>: xor %ebp,%edx
0xffffffff8106a93f <_begin+95>: lea (%rax,%r12,1),%eax
0xffffffff8106a943 <_begin+99>: add 0x8(%r15),%edi
It crashed at arch/x86/crypto/sha1_avx2_x86_64_asm.S, and according to the
stack trace, I deduced some useful information:
crash> struct -x sha1_state 0xffff882f33147990
struct sha1_state {
count = 0x4e000,
state = {0xa73b8733, 0xedad425e, 0xda4f4b14, 0x2fced277, 0x90a160ae},
buffer =
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000"
}
crash> rd ffff882f62fc7c78 24
ffff882f62fc7c78: ffffffff812b1d6e ffff88007bd4e000 n.+........{....
ffff882f62fc7c88: 0000000000000000 ffffea0001ef5380 .........S......
ffff882f62fc7c98: 0000000000000000 ffff882f62fc7ce0 .........|.b/...
ffff882f62fc7ca8: ffffffff00000000 00000000f6275b17 .........['.....
ffff882f62fc7cb8: 000000000000004e ffff882f62fc7d20 N....... }.b/...
ffff882f62fc7cc8: ffffffffa05245ab ffff885f66044120 .ER..... A.f_...
ffff882f62fc7cd8: ffff882f00000000 ffffea0001ef5382 ..../....S......
ffff882f62fc7ce8: 0000100000000000 0000000000000000 ................
ffff882f62fc7cf8: 0000000000000000 00000000f6275b17 .........['.....
ffff882f62fc7d08: ffff882f73c0a000 ffff880111b94540 ...s/...@E......
ffff882f62fc7d18: ffff882f6aff0010 ffff882f62fc7dd8 ...j/....}.b/...
ffff882f62fc7d28: ffffffffa0546701 0000000000000000 .gT.............
crash>
crash> struct hash_desc ffff882f62fc7cd0
struct hash_desc {
tfm = 0xffff885f66044120,
flags = 0
}
crash> struct scatterlist ffff882f62fc7ce0
struct scatterlist {
page_link = 18446719884486202242,
offset = 0,
length = 4096,
dma_address = 0,
dma_length = 0
}
crash> rd ffff882f62fc7c28 22
ffff882f62fc7c28: ffffffff812b1878 ffff882f33147980 x.+......y.3/...
ffff882f62fc7c38: ffff882f6aff0028 ffff882ae84cd500 (..j/.....L.*...
ffff882f62fc7c48: ffff882f33147980 ffff882f6aff0028 .y.3/...(..j/...
ffff882f62fc7c58: ffff882ae84cd500 ffff882f70846800 ..L.*....h.p/...
ffff882f62fc7c68: ffff885f738a12a0 ffff882f62fc7cc0 ...s_....|.b/...
ffff882f62fc7c78: ffffffff812b1d6e ffff88007bd4e000 n.+........{....
ffff882f62fc7c88: 0000000000000000 ffffea0001ef5380 .........S......
ffff882f62fc7c98: 0000000000000000 ffff882f62fc7ce0 .........|.b/...
ffff882f62fc7ca8: ffffffff00000000 00000000f6275b17 .........['.....
ffff882f62fc7cb8: 000000000000004e ffff882f62fc7d20 N....... }.b/...
ffff882f62fc7cc8: ffffffffa05245ab ffff885f66044120 .ER..... A.f_...
crash>
crash>
crash>
crash> struct crypto_hash_walk ffff882f62fc7c80
struct crypto_hash_walk {
data = 0xffff88007bd4e000 struct: page excluded: kernel virtual address:
ffff88007bd4e000 type: "gdb_readmem_callback"
struct: page excluded: kernel virtual address: ffff88007bd4e000 type:
"gdb_readmem_callback"
<Address 0xffff88007bd4e000 out of bounds>,
offset = 0,
alignmask = 0,
pg = 0xffffea0001ef5380,
entrylen = 0,
total = 0,
sg = 0xffff882f62fc7ce0,
flags = 0
}
According to the above information, after call shash_compat_update and, we
got one page sized 4k after kmap, which started at virtual address
0xffff88007bd4e000.
So, the value pass to void sha1_transform_avx2(int *hash, const char* data,
size_t num_blocks ); data = 0xffff88007bd4e000, rounds = 64, which means we
have 64 blocks(4k) to handle.
But the BUFFER_END we calculated out in sha1_avx2_x86_64_asm.S is rounds <<6
+ data + 64 = 64 <<6 + 0xffff88007bd4e000 + 64 = 0xffff88007bd4f040 which
exceed one page.
I think maybe it is the reason why we got the "BUG: unable to handle kernel
paging request at ffff88007bd4f000".
I am not so familiar with the sha1 algorithm, so I email you for your kindly
help, can you give me some suggestion on this issue?
Sincerely
Zhuoyu
^ permalink raw reply
* Re: Remaining crypto API regressions with CONFIG_VMAP_STACK
From: Eric Biggers @ 2016-12-11 23:31 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-crypto, linux-kernel@vger.kernel.org, linux-mm@kvack.org,
kernel-hardening@lists.openwall.com, Herbert Xu,
Andrew Lutomirski, Stephan Mueller
In-Reply-To: <CALCETrVBGPijiacbY-trdbgRPYC8grNrGA7TVu0xvxUaqud08w@mail.gmail.com>
On Sun, Dec 11, 2016 at 11:13:55AM -0800, Andy Lutomirski wrote:
> On Fri, Dec 9, 2016 at 3:08 PM, Eric Biggers <ebiggers3@gmail.com> wrote:
> > In the 4.9 kernel, virtually-mapped stacks will be supported and enabled by
> > default on x86_64. This has been exposing a number of problems in which
> > on-stack buffers are being passed into the crypto API, which to support crypto
> > accelerators operates on 'struct page' rather than on virtual memory.
> >
>
> > fs/cifs/smbencrypt.c:96
>
> This should use crypto_cipher_encrypt_one(), I think.
>
> --Andy
Yes, I believe that's correct. It encrypts 8 bytes with ecb(des) which is
equivalent to simply encrypting one block with DES. Maybe try the following
(untested):
static int
smbhash(unsigned char *out, const unsigned char *in, unsigned char *key)
{
unsigned char key2[8];
struct crypto_cipher *cipher;
str_to_key(key, key2);
cipher = crypto_alloc_cipher("des", 0, 0);
if (IS_ERR(cipher)) {
cifs_dbg(VFS, "could not allocate des cipher\n");
return PTR_ERR(cipher);
}
crypto_cipher_setkey(cipher, key2, 8);
crypto_cipher_encrypt_one(cipher, out, in);
crypto_free_cipher(cipher);
return 0;
}
- Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [kernel-hardening] [PATCH] siphash: add cryptographically secure hashtable function
From: Greg KH @ 2016-12-11 20:43 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: kernel-hardening, LKML, linux-crypto, Linus Torvalds,
Jean-Philippe Aumasson, Daniel J . Bernstein, Herbert Xu,
George Spelvin, Scott Bauer, ak, Andy Lutomirski
In-Reply-To: <CAHmME9qFjeiSFkXfyQQgpRyQCj7dDV9r+UM7S8W=QBsQX+=McQ@mail.gmail.com>
On Sun, Dec 11, 2016 at 04:30:31PM +0100, Jason A. Donenfeld wrote:
> Hi Greg,
>
> Thanks for the review. Responses to your suggestions are inline below:
>
> On Sat, Dec 10, 2016 at 1:37 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > Please use u64 and u8 instead of the userspace uint64_t and uint8_t
> > types for kernel code. Yes, the ship has probably sailed for trying to
> > strictly enforce it, but it's a good idea to do where ever possible.
>
> I didn't know this was a rule. Since I had seen a hodgepodge
> throughout the kernel I just sort of assumed it was a free for all.
> I've fixed this up for v2, and I've also gone through all of my other
> [not yet submitted] code and made this change.
>
> > Any specific license for this code? It's good to at the least say what
> > it is. Yes, we know it will default to GPLv2 only as part of the whole
> > kernel tree, but it's good to be explicit for when someone wants to copy
> > this code for their own projects...
>
> Public domain, actually. I'll add notice of this to the header.
Hm, there really is no such license as "Public domain" that works in all
countries, sorry. You will note it's not one of the "valid module
license list" we have in module.h because of that.
So, I don't know where you got the code from, but perhaps "Dual BSD/GPL"
is the correct one for you?
Note, I'm not a lawyer, so this isn't legal advice about the license of
code, but I do spend way too much time with lawyers dealing with license
issues...
> >> +#define U8TO64(p) le64_to_cpu(*(__le64 *)(p))
> >
> > Why the crazy casting behind a macro?
>
> le64_to_cpup doesn't take the right type. But I agree the macro is not
> the nicest way to do this. Instead, I'll copy what
> crypto/chacha20_generic.c does and define locally le64_to_cpuvp which
> takes a void pointer:
>
> static inline u64 le64_to_cpuvp(const void *p)
> {
> return le64_to_cpup(p);
> }
Ah much better.
> >> +__attribute__((optimize("unroll-loops")))
> >
> > Care to document why this attribute is needed? Older versions of gcc
> > doesn't know how to handle it properly? Faster with newer versions?
> > Black magic? :)
>
> It tells gcc to unroll the loops. Comparing the assembly, it looks
> like on x86_64, gcc does twice as many rounds per loop iteration when
> asked to unroll the loops. This allows the code to remain neat and
> straightforward, while instructing gcc to do the gnarly part.
>
> Is this too much rice? Have I been Gentoo-ing for too long? Or are
> limited uses of unroll-loops acceptable?
Given that this would be the first use of it in the kernel, it might be
too much rice :)
Unless you can show real numbers that it actually matters, I wouldn't do
it, it's not worth the hassle. That's the same rule for when you use
likely()/unlikely(), if you can not measure it, don't use it as you
almost always get it wrong, the compiler is smarter, and keeps getting
better over time.
> > s/uint64_t/u64/g please.
>
> Done.
>
> > EXPORT_SYMBOL_GPL()? I have to ask, sorry :)
>
> Since it's public domain, EXPORT_SYMBOL() is fine.
>
> If you have some reason for preferring GPL2 over public domain, I'm
> happy to make the change. Maybe you want to preclude the new&shiny
> from proprietary modules? That's fine with me, if you desire it being
> that way.
Nope, I just have to ask :)
If it's dual licensed, a normal EXPORT_SYMBOL() is just fine, I have no
objection to that at all.
thanks,
greg k-h
^ permalink raw reply
* [ANNOUNCE] libkcapi v0.13.0 released
From: Stephan Müller @ 2016-12-11 20:31 UTC (permalink / raw)
To: linux-crypto
Hi,
The Linux kernel exports a network interface of type AF_ALG to allow user
space to utilize the kernel crypto API. libkcapi uses this network interface
and exports an easy to use API so that a developer does not need to consider
the low-level network interface handling.
The library does not implement any low level cipher algorithms. All consumer
requests are sent to the kernel for processing. Results from the kernel crypto
API are returned to the consumer via the library API.
The kernel interface and therefore this library can be used by unprivileged
processes.
The library code archive also provides a drop-in replacement for the command
line tools of sha*sum, fipscheck/fipshmac and sha512hmac.
The source code and the documentation is available at [1].
[1] http://www.chronox.de/libkcapi.html
Changes v0.13.0:
* change kcapi_aead_encrypt_aio, kcapi_aead_decrypt_aio,
kcapi_cipher_encrypt_aio and kcapi_cipher_decrypt_aio to require the
user to provide IOVECs for input and output buffers separately
* addition of kcapi_aead_inbuflen_enc, kcapi_aead_inbuflen_dec,
kcapi_aead_outbuflen_enc, kcapi_aead_outbuflen_dec,
kcapi_aead_getdata_input,
kcapi_aead_getdata_output to allow apps to be programmed without specific
code handling for old and new AEAD AF_ALG interface (AAD and tag handling).
See the documentation section "AEAD Memory Structure" for an explanation
on how to use the API in a way to make the calling application agnostic
of the kernel interface differences.
* significant addition to library to handle old / new AEAD AF_ALG interface
without the caller being aware of that
* change AEAD tests such to use the new API calls to make code independent
of AEAD interface changes
* split up of the library implementation into individual files to allow
a more clear code management and to allow even to selectively disable
code to make the library smaller
* various small fixes suggested by Zbigniew Jędrzejewski-Szmek
* fix memleak in kcapi_*_destroy suggested by Zbigniew Jędrzejewski-Szmek
* use hard-links for the kcapi-hasher apps
* add bi-arch tests
* add check that AIO interface is only initialized if the kernel supports
AIO (library requires kernel 4.1.0 or larger for skcipher AIO and
4.7.0 or larger for AEAD AIO support)
* add transparent fallback in case the caller requests AIO operation but
the AIO interface was not or could not be initialized -- the AIO API can be
used on systems without AIO support as the library transparently falls back
to the non-AIO operation (however, the library complains at the beginning
about the use of the AIO API on unsupported systems).
Ciao
Stephan
^ permalink raw reply
* Re: Remaining crypto API regressions with CONFIG_VMAP_STACK
From: Andy Lutomirski @ 2016-12-11 19:13 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel@vger.kernel.org, linux-mm@kvack.org,
kernel-hardening@lists.openwall.com, Herbert Xu,
Andrew Lutomirski, Stephan Mueller
In-Reply-To: <20161209230851.GB64048@google.com>
On Fri, Dec 9, 2016 at 3:08 PM, Eric Biggers <ebiggers3@gmail.com> wrote:
> In the 4.9 kernel, virtually-mapped stacks will be supported and enabled by
> default on x86_64. This has been exposing a number of problems in which
> on-stack buffers are being passed into the crypto API, which to support crypto
> accelerators operates on 'struct page' rather than on virtual memory.
>
> fs/cifs/smbencrypt.c:96
This should use crypto_cipher_encrypt_one(), I think.
--Andy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] siphash: add cryptographically secure hashtable function
From: Jason A. Donenfeld @ 2016-12-11 15:30 UTC (permalink / raw)
To: Greg KH
Cc: kernel-hardening, LKML, linux-crypto, Linus Torvalds,
Jean-Philippe Aumasson, Daniel J . Bernstein, Herbert Xu,
George Spelvin, Scott Bauer, ak, Andy Lutomirski
In-Reply-To: <20161210123725.GC21421@kroah.com>
Hi Greg,
Thanks for the review. Responses to your suggestions are inline below:
On Sat, Dec 10, 2016 at 1:37 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> Please use u64 and u8 instead of the userspace uint64_t and uint8_t
> types for kernel code. Yes, the ship has probably sailed for trying to
> strictly enforce it, but it's a good idea to do where ever possible.
I didn't know this was a rule. Since I had seen a hodgepodge
throughout the kernel I just sort of assumed it was a free for all.
I've fixed this up for v2, and I've also gone through all of my other
[not yet submitted] code and made this change.
> Any specific license for this code? It's good to at the least say what
> it is. Yes, we know it will default to GPLv2 only as part of the whole
> kernel tree, but it's good to be explicit for when someone wants to copy
> this code for their own projects...
Public domain, actually. I'll add notice of this to the header.
> Don't we have this in kernel.h somewhere? Ah, yeah, it's rol64() in
> bitops.h, no need to define it again please.
Thanks!
>
>> +#define U8TO64(p) le64_to_cpu(*(__le64 *)(p))
>
> Why the crazy casting behind a macro?
le64_to_cpup doesn't take the right type. But I agree the macro is not
the nicest way to do this. Instead, I'll copy what
crypto/chacha20_generic.c does and define locally le64_to_cpuvp which
takes a void pointer:
static inline u64 le64_to_cpuvp(const void *p)
{
return le64_to_cpup(p);
}
>> +__attribute__((optimize("unroll-loops")))
>
> Care to document why this attribute is needed? Older versions of gcc
> doesn't know how to handle it properly? Faster with newer versions?
> Black magic? :)
It tells gcc to unroll the loops. Comparing the assembly, it looks
like on x86_64, gcc does twice as many rounds per loop iteration when
asked to unroll the loops. This allows the code to remain neat and
straightforward, while instructing gcc to do the gnarly part.
Is this too much rice? Have I been Gentoo-ing for too long? Or are
limited uses of unroll-loops acceptable?
> s/uint64_t/u64/g please.
Done.
> EXPORT_SYMBOL_GPL()? I have to ask, sorry :)
Since it's public domain, EXPORT_SYMBOL() is fine.
If you have some reason for preferring GPL2 over public domain, I'm
happy to make the change. Maybe you want to preclude the new&shiny
from proprietary modules? That's fine with me, if you desire it being
that way.
>
>
> pr_info()?
Ack.
>
>> + ret = -1;
>
> Pick a real error number?
I started to do this, but couldn't make up my mind, and then resigned
to -1. I'll redouble my efforts to pick something decent.
> pr_info()?
Ack.
> Don't we have a "do crypto/library/whatever selftests at boot" config
> option that this little test could be put under? It would be great to
> not have to manually add DEBUG to the build to verify this works on a
> specific arch.
There is crypto/testmgr.c, but it's designed for testing things that
use the actual crypto API. Clearly for a hashtable function, nobody
would accept the overhead of the enormous crypto API, so siphash has
to remain an ordinary fast function call. Also, it lives in lib/, not
in crypto/. For that reason, I thought that Herbert might object if I
clutter up his testmgr with non crypto API functions. I'll CC him on
this email to double check.
> This looks really nice, but we don't usually add stuff into lib/ unless
> there is an actual user of the code :)
>
> Have you tried converting any of the existing hash users to use this
> instead? If you did that, and it shows a solution for the known
> problems with our existing hashes (as you point out above), I doubt
> there would be any objection for this patch at all.
Here's where the controversy begins! As we've seen from this thread,
there are two hurdles:
1. Convincing people that the cryptographic properties of siphash are
important, and jhash does not have these. I think JP Aumasson's email
described things pretty clearly, and this isn't really up for debate
anymore.
2. Convincing people that for a particular use case, siphash _is_
sufficiently fast, and that any potential (tiny) slowdown, compared to
insecure function like jhash, is either a) not worth having a
known-vulnerability or b) not even measurably relavent for the actual
real life workload.
I suspect that the debates about (2.a) and (2.b) will need to be duked
out one-by-one for a bit of time. I thought that since this will be
more of an evolutionary change, it'd be best to at least get the
primitives into lib/ so they can actually be used.
For example, I found some patches from George Spelvin (CC'd) trying to
get this added a few years back, for reasons related to extfs code. I
found a discussion between Scott Bauer (CC'd) and Andy&Andi (CC'd)
about adding siphash for the purpose of SROP mitigation, but not doing
so because there wasn't the primitive in lib/.
Seeing that siphash is both a solution to current existing problems,
future security mechanisms, and current things people clearly seem to
want, I thought it might be worthwhile to add this straight-up.
But if you really really want me to submit this alongside a patch
series of places that could be changed, I guess I could take the time
to pick out the most uncontroversial places -- some network stack /
netfilter places, some userspace API hashtable DoS places, etc -- but
I fear that's going to drag so many different consumers into the fold
that in the end nothing will get merged. So I think this might be a
good case for an exception with /lib, as a means of making forward
progress in general. Feel free to disagree, though; you know best.
Regards,
Jason
^ permalink raw reply
* Re: [PATCH 3/3] crypto: brcm: Add Broadcom SPU driver DT entry.
From: kbuild test robot @ 2016-12-11 0:14 UTC (permalink / raw)
To: Rob Rice
Cc: kbuild-all-JC7UmRfGjtg, Herbert Xu, David S. Miller, Rob Herring,
Mark Rutland, linux-crypto-u79uwXL29TY76Z2rM5mHXA,
devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Ray Jui, Scott Branden,
Jon Mason, bcm-kernel-feedback-list-dY08KVG/lbpWk0Htik3J/w,
Catalin Marinas, Will Deacon,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Steve Lin,
Rob Rice
In-Reply-To: <1480536453-24781-4-git-send-email-rob.rice-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]
Hi Rob,
[auto build test ERROR on cryptodev/master]
[also build test ERROR on v4.9-rc8]
[cannot apply to next-20161209]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Rob-Rice/crypto-brcm-DT-documentation-for-Broadcom-SPU-driver/20161202-010038
base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master
config: arm64-defconfig (attached as .config)
compiler: aarch64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm64
All errors (new ones prefixed by >>):
>> ERROR: Input tree has errors, aborting (use -f to force output)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 31957 bytes --]
^ permalink raw reply
* Re: [PATCH] siphash: add cryptographically secure hashtable function
From: Jean-Philippe Aumasson @ 2016-12-10 18:13 UTC (permalink / raw)
To: Vegard Nossum, Jason A. Donenfeld
Cc: LKML, kernel-hardening, linux-crypto, Rusty Russell,
Linus Torvalds, Daniel J . Bernstein, linux
In-Reply-To: <CAOMGZ=HMTZhBOh0jTBT4cyMuK5s-D51FFUtWUWyMV7VX0U2L0w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 5804 bytes --]
SipHash co-designer here.
SipHash is secure when it takes a secret key/seed as parameter, meaning
that its output values are unpredictable. Concretely, when SipHash produces
64-bit output values then you've a chance 1/2^64 to guess the hash value of
a given message, provided that the key/seed is kept secret. That's the
standard security definition of a pseudorandom function (PRF), which is
typically instantiated with a MAC such as HMAC-somehash.
With djb we demonstrated that this security notion is sufficient to protect
from hash-flooding attacks wherein an attacker creates many different input
values that hash to a same value and therefore may DoS the underlying data
structure.
I admit that the naming is confusing: "SipHash" is not a hash function,
strictly speaking. In crypto we only call hash function algorithms that are
unkeyed. PRFs/MACs are sometimes called keyed hash functions though.
On Sat, Dec 10, 2016 at 3:17 PM Vegard Nossum <vegard.nossum@gmail.com>
wrote:
> On 9 December 2016 at 19:36, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > SipHash is a 64-bit keyed hash function that is actually a
> > cryptographically secure PRF, like HMAC. Except SipHash is super fast,
> > and is meant to be used as a hashtable keyed lookup function.
> >
> > SipHash isn't just some new trendy hash function. It's been around for a
> > while, and there really isn't anything that comes remotely close to
> > being useful in the way SipHash is. With that said, why do we need this?
> >
> > There are a variety of attacks known as "hashtable poisoning" in which an
> > attacker forms some data such that the hash of that data will be the
> > same, and then preceeds to fill up all entries of a hashbucket. This is
> > a realistic and well-known denial-of-service vector.
> >
> > Linux developers already seem to be aware that this is an issue, and
> > various places that use hash tables in, say, a network context, use a
> > non-cryptographically secure function (usually jhash) and then try to
> > twiddle with the key on a time basis (or in many cases just do nothing
> > and hope that nobody notices). While this is an admirable attempt at
> > solving the problem, it doesn't actually fix it. SipHash fixes it.
>
> Could you give some more concrete details/examples? Here's the IPv4
> hash table from include/net/inet_sock.h / net/ipv4/inet_hashtables.c:
>
> static inline unsigned int __inet_ehashfn(const __be32 laddr,
> const __u16 lport,
> const __be32 faddr,
> const __be16 fport,
> u32 initval)
> {
> return jhash_3words((__force __u32) laddr,
> (__force __u32) faddr,
> ((__u32) lport) << 16 | (__force __u32)fport,
> initval);
> }
>
> static u32 inet_ehashfn(const struct net *net, const __be32 laddr,
> const __u16 lport, const __be32 faddr,
> const __be16 fport)
> {
> static u32 inet_ehash_secret __read_mostly;
>
> net_get_random_once(&inet_ehash_secret, sizeof(inet_ehash_secret));
>
> return __inet_ehashfn(laddr, lport, faddr, fport,
> inet_ehash_secret + net_hash_mix(net));
> }
>
> There's a 32-bit secret random salt (inet_ehash_secret) which means
> that in practice, inet_ehashfn() will select 1 out of 2^32 different
> hash functions at random each time you boot the kernel; without
> knowing which one it selected, how can a local or remote attacker can
> force IPv4 connections/whatever to go into a single hash bucket?
>
> It is not possible to obtain the secret salt directly (except by
> reading from kernel memory, in which case you've lost already), nor is
> it possible to obtain the result of inet_ehashfn() other than (maybe)
> by a timing attack where you somehow need to detect that two
> connections went into the same hash bucket and work backwards from
> that to figure out how to land more connections into into the same
> bucket -- but if they can do that, you've also already lost.
>
> The same pattern is used for IPv6 hashtables and the dentry cache.
>
> I suppose that using a hash function proven to be cryptographically
> secure gives a hard guarantee (under some assumptions) that the
> salt/key will give enough diversity between the (in the example above)
> 2^32 different hash functions that you cannot improve your chances of
> guessing that two values will map to the same bucket regardless of the
> salt/key. However, I am a bit doubtful that using a cryptographically
> secure hash function will make much of a difference as long as the
> attacker doesn't actually have any way to get the output/result of the
> hash function (and given that the hash function isn't completely
> trivial, of course).
>
> I am happy to be proven wrong, but you make it sound very easy to
> exploit the current situation, so I would just like to ask whether you
> have a concrete way to do that?
>
>
> Vegard
>
> > There are a modicum of places in the kernel that are vulnerable to
> > hashtable poisoning attacks, either via userspace vectors or network
> > vectors, and there's not a reliable mechanism inside the kernel at the
> > moment to fix it. The first step toward fixing these issues is actually
> > getting a secure primitive into the kernel for developers to use. Then
> > we can, bit by bit, port things over to it as deemed appropriate.
> >
> > Dozens of languages are already using this internally for their hash
> > tables. Some of the BSDs already use this in their kernels. SipHash is
> > a widely known high-speed solution to a widely known problem, and it's
> > time we catch-up.
>
[-- Attachment #2: Type: text/html, Size: 8549 bytes --]
^ permalink raw reply
* Re: [kernel-hardening] Re: Remaining crypto API regressions with CONFIG_VMAP_STACK
From: Andy Lutomirski @ 2016-12-10 17:48 UTC (permalink / raw)
To: Jason A. Donenfeld, Al Viro
Cc: kernel-hardening@lists.openwall.com, Eric Biggers, linux-crypto,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Andrew Lutomirski, Stephan Mueller
In-Reply-To: <CAHmME9pzT=bxuEVVGDOJkm2PaEAVjbo=8na7URy=g-1sKvv0yw@mail.gmail.com>
cc: Viro because I'm talking about iov_iter.
On Sat, Dec 10, 2016 at 6:45 AM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> Hi Herbert,
>
> On Sat, Dec 10, 2016 at 6:37 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
>> As for AEAD we never had a sync interface to begin with and I
>> don't think I'm going to add one.
>
> That's too bad to hear. I hope you'll reconsider. Modern cryptographic
> design is heading more and more in the direction of using AEADs for
> interesting things, and having a sync interface would be a lot easier
> for implementing these protocols. In the same way many protocols need
> a hash of some data, now protocols often want some particular data
> encrypted with an AEAD using a particular key and nonce and AD. One
> protocol that comes to mind is Noise [1].
>
I think that sync vs async has gotten conflated with
vectored-vs-nonvectored and the results are unfortunate.
There are a lot of users in the tree that are trying to do crypto on
very small pieces of data and want to have that data consist of the
concatenation of two small buffers and/or want to use primitives that
don't have "sync" interfaces. These users are stuck using async
interfaces even though using async implementations makes no sense for
them.
I'd love to see the API restructured a bit to decouple all of these
considerations. One approach might be to teach iov_iter about
scatterlists. Then, for each primitive, there could be two entry
points:
1. A simplified and lower-overhead entry. You pass it an iov_iter
(and, depending on what the operation is, an output iov_iter), it does
the crypto synchronously, and returns. Operating in-place might be
permitted for some primitives.
2. A full-featured async entry. You pass it iov_iters and it requires
that the iov_iters be backed by scatterlists in order to operate
asynchronously.
I see no reason that the decisions to use virtual vs physical
addressing or to do vectored vs non-vectored IO should be tied up with
asynchronicity.
--Andy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] siphash: add cryptographically secure hashtable function
From: George Spelvin @ 2016-12-10 15:35 UTC (permalink / raw)
To: Jason, vegard.nossum
Cc: djb, jeanphilippe.aumasson, kernel-hardening, linux-crypto,
linux-kernel, linux, rusty, torvalds
In-Reply-To: <CAOMGZ=HMTZhBOh0jTBT4cyMuK5s-D51FFUtWUWyMV7VX0U2L0w@mail.gmail.com>
> There's a 32-bit secret random salt (inet_ehash_secret) which means
> that in practice, inet_ehashfn() will select 1 out of 2^32 different
> hash functions at random each time you boot the kernel; without
> knowing which one it selected, how can a local or remote attacker can
> force IPv4 connections/whatever to go into a single hash bucket?
By figuring out the salt. The thing is, the timing of hash table lookups
*is externally visible*. If I create connections to the target, then
see which ones make responses on previous connections slightly slower,
I gain information about the salt.
I dont't know *where* in the hash table the collissions occur, but I
know *which* inputs collide, and that's enough for me to learn something.
(I need more connections than the size of the hash table, but even
with just one IP source I can use 64K ports on my end times however
many the target has open on its end.)
With enough information (google "unicity distance") I can recover the
entire salt. It's not like I care about the cryptographic strength of
the hash; simply trying all 4 billion possible seeds is pretty fast on
a 4 GHz processor.
Once that happens, I can choose a target connection whose timing I can't
observe directly and pack its hash chain without being obvious about it.
> I am happy to be proven wrong, but you make it sound very easy to
> exploit the current situation, so I would just like to ask whether you
> have a concrete way to do that?
I don't think anyone's implemented an attack on this particular hash
table yet, and the reason it hasn't been urgent is that it's just a mild
DoS attack it makes the computer noticeably slower withough disabling
it completely.
But the general style of attack is well known and has been repeatedly
demonstrated. Its practicality is not in question. The only question is
whether it's *more* practical that simpler techniques that don't depend
on any such algorithmic subtlety like brute-force flooding.
But if the history of Internet security has taught us one thing, it's
that naively hoping something won't be a problem is doomed.
The main issue is performance. IPv6 addresses are big, and although
SipHash is fast by the standard of cryptographic hashes, it's far slower
than jhash or any other non-cryptographic hash.
^ permalink raw reply
* Re: [kernel-hardening] Re: Remaining crypto API regressions with CONFIG_VMAP_STACK
From: Jason A. Donenfeld @ 2016-12-10 14:45 UTC (permalink / raw)
To: kernel-hardening
Cc: Andy Lutomirski, Eric Biggers, linux-crypto,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Andrew Lutomirski, Stephan Mueller
In-Reply-To: <20161210053711.GB27951@gondor.apana.org.au>
Hi Herbert,
On Sat, Dec 10, 2016 at 6:37 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> As for AEAD we never had a sync interface to begin with and I
> don't think I'm going to add one.
That's too bad to hear. I hope you'll reconsider. Modern cryptographic
design is heading more and more in the direction of using AEADs for
interesting things, and having a sync interface would be a lot easier
for implementing these protocols. In the same way many protocols need
a hash of some data, now protocols often want some particular data
encrypted with an AEAD using a particular key and nonce and AD. One
protocol that comes to mind is Noise [1].
I know that in my own [currently external to the tree] kernel code, I
just forego the use of the crypto API all together, and one of the
primary reasons for that is lack of a sync interface for AEADs. When I
eventually send this upstream, presumably everyone will want me to use
the crypto API, and having a sync AEAD interface would be personally
helpful for that. I guess I could always write the sync interface
myself, but I imagine you'd prefer having the design control etc.
Jason
[1] http://noiseprotocol.org/
^ permalink raw reply
* Re: [PATCH] siphash: add cryptographically secure hashtable function
From: Vegard Nossum @ 2016-12-10 14:17 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: LKML, kernel-hardening, linux-crypto, Rusty Russell,
Linus Torvalds, Jean-Philippe Aumasson, Daniel J . Bernstein,
linux
In-Reply-To: <20161209183659.25727-1-Jason@zx2c4.com>
On 9 December 2016 at 19:36, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> SipHash is a 64-bit keyed hash function that is actually a
> cryptographically secure PRF, like HMAC. Except SipHash is super fast,
> and is meant to be used as a hashtable keyed lookup function.
>
> SipHash isn't just some new trendy hash function. It's been around for a
> while, and there really isn't anything that comes remotely close to
> being useful in the way SipHash is. With that said, why do we need this?
>
> There are a variety of attacks known as "hashtable poisoning" in which an
> attacker forms some data such that the hash of that data will be the
> same, and then preceeds to fill up all entries of a hashbucket. This is
> a realistic and well-known denial-of-service vector.
>
> Linux developers already seem to be aware that this is an issue, and
> various places that use hash tables in, say, a network context, use a
> non-cryptographically secure function (usually jhash) and then try to
> twiddle with the key on a time basis (or in many cases just do nothing
> and hope that nobody notices). While this is an admirable attempt at
> solving the problem, it doesn't actually fix it. SipHash fixes it.
Could you give some more concrete details/examples? Here's the IPv4
hash table from include/net/inet_sock.h / net/ipv4/inet_hashtables.c:
static inline unsigned int __inet_ehashfn(const __be32 laddr,
const __u16 lport,
const __be32 faddr,
const __be16 fport,
u32 initval)
{
return jhash_3words((__force __u32) laddr,
(__force __u32) faddr,
((__u32) lport) << 16 | (__force __u32)fport,
initval);
}
static u32 inet_ehashfn(const struct net *net, const __be32 laddr,
const __u16 lport, const __be32 faddr,
const __be16 fport)
{
static u32 inet_ehash_secret __read_mostly;
net_get_random_once(&inet_ehash_secret, sizeof(inet_ehash_secret));
return __inet_ehashfn(laddr, lport, faddr, fport,
inet_ehash_secret + net_hash_mix(net));
}
There's a 32-bit secret random salt (inet_ehash_secret) which means
that in practice, inet_ehashfn() will select 1 out of 2^32 different
hash functions at random each time you boot the kernel; without
knowing which one it selected, how can a local or remote attacker can
force IPv4 connections/whatever to go into a single hash bucket?
It is not possible to obtain the secret salt directly (except by
reading from kernel memory, in which case you've lost already), nor is
it possible to obtain the result of inet_ehashfn() other than (maybe)
by a timing attack where you somehow need to detect that two
connections went into the same hash bucket and work backwards from
that to figure out how to land more connections into into the same
bucket -- but if they can do that, you've also already lost.
The same pattern is used for IPv6 hashtables and the dentry cache.
I suppose that using a hash function proven to be cryptographically
secure gives a hard guarantee (under some assumptions) that the
salt/key will give enough diversity between the (in the example above)
2^32 different hash functions that you cannot improve your chances of
guessing that two values will map to the same bucket regardless of the
salt/key. However, I am a bit doubtful that using a cryptographically
secure hash function will make much of a difference as long as the
attacker doesn't actually have any way to get the output/result of the
hash function (and given that the hash function isn't completely
trivial, of course).
I am happy to be proven wrong, but you make it sound very easy to
exploit the current situation, so I would just like to ask whether you
have a concrete way to do that?
Vegard
> There are a modicum of places in the kernel that are vulnerable to
> hashtable poisoning attacks, either via userspace vectors or network
> vectors, and there's not a reliable mechanism inside the kernel at the
> moment to fix it. The first step toward fixing these issues is actually
> getting a secure primitive into the kernel for developers to use. Then
> we can, bit by bit, port things over to it as deemed appropriate.
>
> Dozens of languages are already using this internally for their hash
> tables. Some of the BSDs already use this in their kernels. SipHash is
> a widely known high-speed solution to a widely known problem, and it's
> time we catch-up.
^ permalink raw reply
* Re: [kernel-hardening] [PATCH] siphash: add cryptographically secure hashtable function
From: Greg KH @ 2016-12-10 12:37 UTC (permalink / raw)
To: kernel-hardening
Cc: linux-kernel, linux-crypto, rusty, torvalds, Jason A. Donenfeld,
Jean-Philippe Aumasson, Daniel J . Bernstein
In-Reply-To: <20161209183659.25727-1-Jason@zx2c4.com>
On Fri, Dec 09, 2016 at 07:36:59PM +0100, Jason A. Donenfeld wrote:
> SipHash is a 64-bit keyed hash function that is actually a
> cryptographically secure PRF, like HMAC. Except SipHash is super fast,
> and is meant to be used as a hashtable keyed lookup function.
>
> SipHash isn't just some new trendy hash function. It's been around for a
> while, and there really isn't anything that comes remotely close to
> being useful in the way SipHash is. With that said, why do we need this?
>
> There are a variety of attacks known as "hashtable poisoning" in which an
> attacker forms some data such that the hash of that data will be the
> same, and then preceeds to fill up all entries of a hashbucket. This is
> a realistic and well-known denial-of-service vector.
>
> Linux developers already seem to be aware that this is an issue, and
> various places that use hash tables in, say, a network context, use a
> non-cryptographically secure function (usually jhash) and then try to
> twiddle with the key on a time basis (or in many cases just do nothing
> and hope that nobody notices). While this is an admirable attempt at
> solving the problem, it doesn't actually fix it. SipHash fixes it.
>
> (It fixes it in such a sound way that you could even build a stream
> cipher out of SipHash that would resist the modern cryptanalysis.)
>
> There are a modicum of places in the kernel that are vulnerable to
> hashtable poisoning attacks, either via userspace vectors or network
> vectors, and there's not a reliable mechanism inside the kernel at the
> moment to fix it. The first step toward fixing these issues is actually
> getting a secure primitive into the kernel for developers to use. Then
> we can, bit by bit, port things over to it as deemed appropriate.
>
> Dozens of languages are already using this internally for their hash
> tables. Some of the BSDs already use this in their kernels. SipHash is
> a widely known high-speed solution to a widely known problem, and it's
> time we catch-up.
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> Cc: Daniel J. Bernstein <djb@cr.yp.to>
> ---
> include/linux/siphash.h | 18 ++++++
> lib/Makefile | 3 +-
> lib/siphash.c | 163 ++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 183 insertions(+), 1 deletion(-)
> create mode 100644 include/linux/siphash.h
> create mode 100644 lib/siphash.c
This looks really nice, but we don't usually add stuff into lib/ unless
there is an actual user of the code :)
Have you tried converting any of the existing hash users to use this
instead? If you did that, and it shows a solution for the known
problems with our existing hashes (as you point out above), I doubt
there would be any objection for this patch at all.
Minor coding style nits below:
> @@ -0,0 +1,18 @@
> +/* Copyright (C) 2016 Jason A. Donenfeld <Jason@zx2c4.com>
> + *
> + * SipHash: a fast short-input PRF
> + * https://131002.net/siphash/
> + */
> +
> +#ifndef _LINUX_SIPHASH_H
> +#define _LINUX_SIPHASH_H
> +
> +#include <linux/types.h>
> +
> +enum siphash24_lengths {
> + SIPHASH24_KEY_LEN = 16
> +};
> +
> +uint64_t siphash24(const uint8_t *data, size_t len, const uint8_t key[SIPHASH24_KEY_LEN]);
Please use u64 and u8 instead of the userspace uint64_t and uint8_t
types for kernel code. Yes, the ship has probably sailed for trying to
strictly enforce it, but it's a good idea to do where ever possible.
> +
> +#endif /* _LINUX_SIPHASH_H */
> diff --git a/lib/Makefile b/lib/Makefile
> index 50144a3aeebd..d224337b0d01 100644
> --- a/lib/Makefile
> +++ b/lib/Makefile
> @@ -22,7 +22,8 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
> sha1.o chacha20.o md5.o irq_regs.o argv_split.o \
> flex_proportions.o ratelimit.o show_mem.o \
> is_single_threaded.o plist.o decompress.o kobject_uevent.o \
> - earlycpio.o seq_buf.o nmi_backtrace.o nodemask.o win_minmax.o
> + earlycpio.o seq_buf.o siphash.o \
> + nmi_backtrace.o nodemask.o win_minmax.o
>
> lib-$(CONFIG_MMU) += ioremap.o
> lib-$(CONFIG_SMP) += cpumask.o
> diff --git a/lib/siphash.c b/lib/siphash.c
> new file mode 100644
> index 000000000000..022d86f04b9b
> --- /dev/null
> +++ b/lib/siphash.c
> @@ -0,0 +1,163 @@
> +/* Copyright (C) 2015-2016 Jason A. Donenfeld <Jason@zx2c4.com>
> + * Copyright (C) 2012-2014 Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> + * Copyright (C) 2012-2014 Daniel J. Bernstein <djb@cr.yp.to>
> + *
> + * SipHash: a fast short-input PRF
> + * https://131002.net/siphash/
> + */
Any specific license for this code? It's good to at the least say what
it is. Yes, we know it will default to GPLv2 only as part of the whole
kernel tree, but it's good to be explicit for when someone wants to copy
this code for their own projects...
> +
> +#include <linux/siphash.h>
> +#include <linux/kernel.h>
> +#include <linux/string.h>
> +
> +#define ROTL(x,b) (uint64_t)(((x) << (b)) | ((x) >> (64 - (b))))
Don't we have this in kernel.h somewhere? Ah, yeah, it's rol64() in
bitops.h, no need to define it again please.
> +#define U8TO64(p) le64_to_cpu(*(__le64 *)(p))
Why the crazy casting behind a macro?
> +
> +#define SIPROUND \
> + do { \
> + v0 += v1; v1 = ROTL(v1, 13); v1 ^= v0; v0 = ROTL(v0, 32); \
> + v2 += v3; v3 = ROTL(v3, 16); v3 ^= v2; \
> + v0 += v3; v3 = ROTL(v3, 21); v3 ^= v0; \
> + v2 += v1; v1 = ROTL(v1, 17); v1 ^= v2; v2 = ROTL(v2, 32); \
> + } while(0)
> +
> +__attribute__((optimize("unroll-loops")))
Care to document why this attribute is needed? Older versions of gcc
doesn't know how to handle it properly? Faster with newer versions?
Black magic? :)
> +uint64_t siphash24(const uint8_t *data, size_t len, const uint8_t key[SIPHASH24_KEY_LEN])
> +{
> + uint64_t v0 = 0x736f6d6570736575ULL;
s/uint64_t/u64/g please.
> + uint64_t v1 = 0x646f72616e646f6dULL;
> + uint64_t v2 = 0x6c7967656e657261ULL;
> + uint64_t v3 = 0x7465646279746573ULL;
> + uint64_t b;
> + uint64_t k0 = U8TO64(key);
> + uint64_t k1 = U8TO64(key + sizeof(uint64_t));
> + uint64_t m;
> + const uint8_t *end = data + len - (len % sizeof(uint64_t));
> + const uint8_t left = len & (sizeof(uint64_t) - 1);
> + b = ((uint64_t)len) << 56;
> + v3 ^= k1;
> + v2 ^= k0;
> + v1 ^= k1;
> + v0 ^= k0;
> + for (; data != end; data += sizeof(uint64_t)) {
> + m = U8TO64(data);
> + v3 ^= m;
> + SIPROUND;
> + SIPROUND;
> + v0 ^= m;
> + }
> + switch (left) {
> + case 7: b |= ((uint64_t)data[6]) << 48;
> + case 6: b |= ((uint64_t)data[5]) << 40;
> + case 5: b |= ((uint64_t)data[4]) << 32;
> + case 4: b |= ((uint64_t)data[3]) << 24;
> + case 3: b |= ((uint64_t)data[2]) << 16;
> + case 2: b |= ((uint64_t)data[1]) << 8;
> + case 1: b |= ((uint64_t)data[0]); break;
> + case 0: break;
> + }
> + v3 ^= b;
> + SIPROUND;
> + SIPROUND;
> + v0 ^= b;
> + v2 ^= 0xff;
> + SIPROUND;
> + SIPROUND;
> + SIPROUND;
> + SIPROUND;
> + b = (v0 ^ v1) ^ (v2 ^ v3);
> + return (__force uint64_t)cpu_to_le64(b);
> +}
> +EXPORT_SYMBOL(siphash24);
EXPORT_SYMBOL_GPL()? I have to ask, sorry :)
> +
> +#ifdef DEBUG
> +static const uint8_t test_vectors[64][8] = {
> + { 0x31, 0x0e, 0x0e, 0xdd, 0x47, 0xdb, 0x6f, 0x72 },
> + { 0xfd, 0x67, 0xdc, 0x93, 0xc5, 0x39, 0xf8, 0x74 },
> + { 0x5a, 0x4f, 0xa9, 0xd9, 0x09, 0x80, 0x6c, 0x0d },
> + { 0x2d, 0x7e, 0xfb, 0xd7, 0x96, 0x66, 0x67, 0x85 },
> + { 0xb7, 0x87, 0x71, 0x27, 0xe0, 0x94, 0x27, 0xcf },
> + { 0x8d, 0xa6, 0x99, 0xcd, 0x64, 0x55, 0x76, 0x18 },
> + { 0xce, 0xe3, 0xfe, 0x58, 0x6e, 0x46, 0xc9, 0xcb },
> + { 0x37, 0xd1, 0x01, 0x8b, 0xf5, 0x00, 0x02, 0xab },
> + { 0x62, 0x24, 0x93, 0x9a, 0x79, 0xf5, 0xf5, 0x93 },
> + { 0xb0, 0xe4, 0xa9, 0x0b, 0xdf, 0x82, 0x00, 0x9e },
> + { 0xf3, 0xb9, 0xdd, 0x94, 0xc5, 0xbb, 0x5d, 0x7a },
> + { 0xa7, 0xad, 0x6b, 0x22, 0x46, 0x2f, 0xb3, 0xf4 },
> + { 0xfb, 0xe5, 0x0e, 0x86, 0xbc, 0x8f, 0x1e, 0x75 },
> + { 0x90, 0x3d, 0x84, 0xc0, 0x27, 0x56, 0xea, 0x14 },
> + { 0xee, 0xf2, 0x7a, 0x8e, 0x90, 0xca, 0x23, 0xf7 },
> + { 0xe5, 0x45, 0xbe, 0x49, 0x61, 0xca, 0x29, 0xa1 },
> + { 0xdb, 0x9b, 0xc2, 0x57, 0x7f, 0xcc, 0x2a, 0x3f },
> + { 0x94, 0x47, 0xbe, 0x2c, 0xf5, 0xe9, 0x9a, 0x69 },
> + { 0x9c, 0xd3, 0x8d, 0x96, 0xf0, 0xb3, 0xc1, 0x4b },
> + { 0xbd, 0x61, 0x79, 0xa7, 0x1d, 0xc9, 0x6d, 0xbb },
> + { 0x98, 0xee, 0xa2, 0x1a, 0xf2, 0x5c, 0xd6, 0xbe },
> + { 0xc7, 0x67, 0x3b, 0x2e, 0xb0, 0xcb, 0xf2, 0xd0 },
> + { 0x88, 0x3e, 0xa3, 0xe3, 0x95, 0x67, 0x53, 0x93 },
> + { 0xc8, 0xce, 0x5c, 0xcd, 0x8c, 0x03, 0x0c, 0xa8 },
> + { 0x94, 0xaf, 0x49, 0xf6, 0xc6, 0x50, 0xad, 0xb8 },
> + { 0xea, 0xb8, 0x85, 0x8a, 0xde, 0x92, 0xe1, 0xbc },
> + { 0xf3, 0x15, 0xbb, 0x5b, 0xb8, 0x35, 0xd8, 0x17 },
> + { 0xad, 0xcf, 0x6b, 0x07, 0x63, 0x61, 0x2e, 0x2f },
> + { 0xa5, 0xc9, 0x1d, 0xa7, 0xac, 0xaa, 0x4d, 0xde },
> + { 0x71, 0x65, 0x95, 0x87, 0x66, 0x50, 0xa2, 0xa6 },
> + { 0x28, 0xef, 0x49, 0x5c, 0x53, 0xa3, 0x87, 0xad },
> + { 0x42, 0xc3, 0x41, 0xd8, 0xfa, 0x92, 0xd8, 0x32 },
> + { 0xce, 0x7c, 0xf2, 0x72, 0x2f, 0x51, 0x27, 0x71 },
> + { 0xe3, 0x78, 0x59, 0xf9, 0x46, 0x23, 0xf3, 0xa7 },
> + { 0x38, 0x12, 0x05, 0xbb, 0x1a, 0xb0, 0xe0, 0x12 },
> + { 0xae, 0x97, 0xa1, 0x0f, 0xd4, 0x34, 0xe0, 0x15 },
> + { 0xb4, 0xa3, 0x15, 0x08, 0xbe, 0xff, 0x4d, 0x31 },
> + { 0x81, 0x39, 0x62, 0x29, 0xf0, 0x90, 0x79, 0x02 },
> + { 0x4d, 0x0c, 0xf4, 0x9e, 0xe5, 0xd4, 0xdc, 0xca },
> + { 0x5c, 0x73, 0x33, 0x6a, 0x76, 0xd8, 0xbf, 0x9a },
> + { 0xd0, 0xa7, 0x04, 0x53, 0x6b, 0xa9, 0x3e, 0x0e },
> + { 0x92, 0x59, 0x58, 0xfc, 0xd6, 0x42, 0x0c, 0xad },
> + { 0xa9, 0x15, 0xc2, 0x9b, 0xc8, 0x06, 0x73, 0x18 },
> + { 0x95, 0x2b, 0x79, 0xf3, 0xbc, 0x0a, 0xa6, 0xd4 },
> + { 0xf2, 0x1d, 0xf2, 0xe4, 0x1d, 0x45, 0x35, 0xf9 },
> + { 0x87, 0x57, 0x75, 0x19, 0x04, 0x8f, 0x53, 0xa9 },
> + { 0x10, 0xa5, 0x6c, 0xf5, 0xdf, 0xcd, 0x9a, 0xdb },
> + { 0xeb, 0x75, 0x09, 0x5c, 0xcd, 0x98, 0x6c, 0xd0 },
> + { 0x51, 0xa9, 0xcb, 0x9e, 0xcb, 0xa3, 0x12, 0xe6 },
> + { 0x96, 0xaf, 0xad, 0xfc, 0x2c, 0xe6, 0x66, 0xc7 },
> + { 0x72, 0xfe, 0x52, 0x97, 0x5a, 0x43, 0x64, 0xee },
> + { 0x5a, 0x16, 0x45, 0xb2, 0x76, 0xd5, 0x92, 0xa1 },
> + { 0xb2, 0x74, 0xcb, 0x8e, 0xbf, 0x87, 0x87, 0x0a },
> + { 0x6f, 0x9b, 0xb4, 0x20, 0x3d, 0xe7, 0xb3, 0x81 },
> + { 0xea, 0xec, 0xb2, 0xa3, 0x0b, 0x22, 0xa8, 0x7f },
> + { 0x99, 0x24, 0xa4, 0x3c, 0xc1, 0x31, 0x57, 0x24 },
> + { 0xbd, 0x83, 0x8d, 0x3a, 0xaf, 0xbf, 0x8d, 0xb7 },
> + { 0x0b, 0x1a, 0x2a, 0x32, 0x65, 0xd5, 0x1a, 0xea },
> + { 0x13, 0x50, 0x79, 0xa3, 0x23, 0x1c, 0xe6, 0x60 },
> + { 0x93, 0x2b, 0x28, 0x46, 0xe4, 0xd7, 0x06, 0x66 },
> + { 0xe1, 0x91, 0x5f, 0x5c, 0xb1, 0xec, 0xa4, 0x6c },
> + { 0xf3, 0x25, 0x96, 0x5c, 0xa1, 0x6d, 0x62, 0x9f },
> + { 0x57, 0x5f, 0xf2, 0x8e, 0x60, 0x38, 0x1b, 0xe5 },
> + { 0x72, 0x45, 0x06, 0xeb, 0x4c, 0x32, 0x8a, 0x95 }
> +};
> +
> +static int siphash24_selftest(void)
> +{
> + uint8_t in[64], k[16], i;
> + uint64_t out;
> + int ret = 0;
> +
> + for (i = 0; i < 16; ++i)
> + k[i] = i;
> +
> + for (i = 0; i < 64; ++i) {
> + in[i] = i;
> + out = siphash24(in, i, k);
> + if (memcmp(&out, test_vectors[i], 8)) {
> + printk(KERN_INFO "siphash24: self-test %u: FAIL\n", i + 1);
pr_info()?
> + ret = -1;
Pick a real error number?
> + }
> + }
> + if (!ret)
> + printk(KERN_INFO "siphash24: self-tests: pass\n");
pr_info()?
> + return ret;
> +}
> +__initcall(siphash24_selftest);
Don't we have a "do crypto/library/whatever selftests at boot" config
option that this little test could be put under? It would be great to
not have to manually add DEBUG to the build to verify this works on a
specific arch.
thanks,
greg k-h
^ permalink raw reply
* RE: [PATCH v6 1/2] sparc: fix a building error reported by kbuild
From: Gonglei (Arei) @ 2016-12-10 8:40 UTC (permalink / raw)
To: Sam Ravnborg
Cc: linux-kernel@vger.kernel.org, qemu-devel@nongnu.org,
virtio-dev@lists.oasis-open.org,
virtualization@lists.linux-foundation.org,
linux-crypto@vger.kernel.org, Luonengjun, mst@redhat.com,
stefanha@redhat.com, Huangweidong (C), Wubin (H),
xin.zeng@intel.com, Claudio Fontana, herbert@gondor.apana.org.au,
pasic@linux.vnet.ibm.com, davem@davemloft.net
In-Reply-To: <20161209215851.GA7717@ravnborg.org>
Regards,
-Gonglei
> -----Original Message-----
> From: linux-crypto-owner@vger.kernel.org
> [mailto:linux-crypto-owner@vger.kernel.org] On Behalf Of Sam Ravnborg
> Sent: Saturday, December 10, 2016 5:59 AM
> To: Gonglei (Arei)
> Cc: linux-kernel@vger.kernel.org; qemu-devel@nongnu.org;
> virtio-dev@lists.oasis-open.org; virtualization@lists.linux-foundation.org;
> linux-crypto@vger.kernel.org; Luonengjun; mst@redhat.com;
> stefanha@redhat.com; Huangweidong (C); Wubin (H); xin.zeng@intel.com;
> Claudio Fontana; herbert@gondor.apana.org.au; pasic@linux.vnet.ibm.com;
> davem@davemloft.net; Zhoujian (jay, Euler); Hanweidong (Randy);
> arei.gonglei@hotmail.com; cornelia.huck@de.ibm.com; Xuquan (Quan Xu);
> longpeng; Wanzongshun (Vincent); sparclinux@vger.kernel.org
> Subject: Re: [PATCH v6 1/2] sparc: fix a building error reported by kbuild
>
> Hi Gonglei.
>
> On Thu, Dec 08, 2016 at 12:37:08PM +0800, Gonglei wrote:
> > >> arch/sparc/include/asm/topology_64.h:44:44:
> > error: implicit declaration of function 'cpu_data'
> > [-Werror=implicit-function-declaration]
> >
> > #define topology_physical_package_id(cpu) (cpu_data(cpu).proc_id)
> > ^
> > Let's include cpudata.h in topology_64.h.
> >
> > Cc: Sam Ravnborg <sam@ravnborg.org>
> > Cc: David S. Miller <davem@davemloft.net>
> > Cc: sparclinux@vger.kernel.org
> > Suggested-by: Sam Ravnborg <sam@ravnborg.org>
> > Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> Acked-by: Sam Ravnborg <sam@ravnborg.org>
>
Thanks.
> > ---
> > arch/sparc/include/asm/topology_64.h | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/arch/sparc/include/asm/topology_64.h
> b/arch/sparc/include/asm/topology_64.h
> > index 7b4898a..2255430 100644
> > --- a/arch/sparc/include/asm/topology_64.h
> > +++ b/arch/sparc/include/asm/topology_64.h
> > @@ -4,6 +4,7 @@
> > #ifdef CONFIG_NUMA
> >
> > #include <asm/mmzone.h>
> > +#include <asm/cpudata.h>
>
> Nitpick - if you are going to resend this patch,
It depends on the maintainer's thought. :)
> then please order the two includes in alphabetic order.
>
> For two includes this looks like bikeshedding, but when we add
> more having them in a defined arder prevents merge conflicts.
> And makes it readable too.
>
> We also sometimes order the includes with the longest lines topmost,
> and lines with the ame length are ordered alphabetically.
> But this is not seen so often.
>
Regards,
-Gonglei
^ permalink raw reply
* Re: Remaining crypto API regressions with CONFIG_VMAP_STACK
From: Eric Biggers @ 2016-12-10 8:39 UTC (permalink / raw)
To: Herbert Xu
Cc: kernel-hardening, luto, linux-crypto, linux-kernel, linux-mm,
luto, smueller
In-Reply-To: <20161210081643.GA384@gondor.apana.org.au>
On Sat, Dec 10, 2016 at 04:16:43PM +0800, Herbert Xu wrote:
> Why did you drop me from the CC list when you were replying to
> my email?
>
Sorry --- this thread is Cc'ed to the kernel-hardening mailing list (which was
somewhat recently revived), and I replied to the email that reached me from
there. It looks like it currently behaves a little differently from the vger
mailing lists, in that it replaces "Reply-To" with the address of the mailing
list itself rather than the sender. So that's how you got dropped. It also
seems to add a prefix to the subject...
I
> >> Are you sure? Any instance of *_ON_STACK must only be used with
> >> sync algorithms and most drivers under drivers/crypto declare
> >> themselves as async.
> >
> > Why exactly is that? Obviously, it wouldn't work if you returned from the stack
> > frame before the request completed, but does anything stop someone from using an
> > *_ON_STACK() request and then waiting for the request to complete before
> > returning from the stack frame?
>
> The *_ON_STACK variants (except SHASH of course) were simply hacks
> to help legacy crypto API users to cope with the new async interface.
> In general we should avoid using the sync interface when possible.
>
> It's a bad idea for the obvious reason that most of our async
> algorithms want to DMA and that doesn't work very well when you're
> using memory from the stack.
Sure, I just feel that the idea of "is this algorithm asynchronous?" is being
conflated with the idea of "does this algorithm operate on physical memory?".
Also, if *_ON_STACK are really not allowed with asynchronous algorithms can
there at least be a comment or a WARN_ON() to express this?
Thanks,
Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox