[PATCH 00/11] iSER target initial support for T10-DIF offload

[PATCH 00/11] iSER target initial support for T10-DIF offload

[Sagi Grimberg]

Hey Nic, MKP, SCSI and RDMA folks,

This patchset adds basic support for T10-DIF protection information offload
in iSER target on top of Nic's recent work and RDMA signature verbs API.

This code was tested with my own implementation of the target core T10-PI support
which was designed mainly to activate the transport DIF offload. In order to
actually get Linux SCSI target to work with iSER T10-DIF offload a couple of
patches needs to be added to Nic's work which is ongoing.

Apart from doing the actual iser implementation for  T10-DIF offload, this
series would help to see the full picture by:

* Showing how the T10-DIF offload verbs are used
* Showing how fabric transport offload plugs into the target core

The T10-DIF signature offload verbs and mlx5 driver implementation patches are available
from the for-next branch of git://beany.openfabrics.org/~ogerlitz/linux-2.6.git
as the below commits:

2b4316b IB/mlx5: Publish support in signature feature
ef3130d IB/mlx5: Collect signature error completion
c1b37b1 IB/mlx5: Support IB_WR_REG_SIG_MR
f5d8496 IB/mlx5: Keep mlx5 MRs in a radix tree under device
72a72ee IB/mlx5: remove MTT access mode from umr flags helper function
ccb0a907 IB/mlx5: Break wqe handling to begin & finish routines
cda0569 IB/mlx5: Initialize mlx5_ib_qp signature related
33b4079 IB/mlx5, mlx5_core: Support for create_mr and destroy_mr
8b343e6 IB/core: Introduce Signature Verbs API
c1b0358 IB/core: Introduce protected memory regions

Sagi Grimberg (11):
  Target/core: Fixes for isert compilation
  IB/isert: seperate connection protection domains and dma MRs
  IB/isert: Avoid frwr notation, user fastreg
  IB/isert: Move fastreg descriptor creation to a function
  Target/iscsi: Add T10-PI indication for iscsi_portal_group
  IB/isert: Initialize T10-PI resources
  IB/isert: pass scatterlist instead of cmd to fast_reg_mr routine
  IB/isert: pass mr and frpl to isert_fast_reg_mr routine
  IB/isert: Accept RDMA_WRITE completions
  IB/isert: Support T10-PI protected transactions
  Target/configfs: Expose iSCSI network portal group T10-PI support

 drivers/infiniband/ulp/isert/ib_isert.c      |  708 +++++++++++++++++++------
 drivers/infiniband/ulp/isert/ib_isert.h      |   29 +-
 drivers/target/iscsi/iscsi_target_configfs.c |    6 +
 drivers/target/iscsi/iscsi_target_core.h     |    5 +-
 drivers/target/iscsi/iscsi_target_tpg.c      |   21 +
 drivers/target/iscsi/iscsi_target_tpg.h      |    1 +
 include/target/target_core_base.h            |   22 +-
 7 files changed, 603 insertions(+), 189 deletions(-)

[PATCH 01/11] Target/core: Fixes for isert compilation

[Sagi Grimberg]

replace prot_interleaved with prot_handover in se_cmd.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 include/target/target_core_base.h |   22 ++++++++++++++--------
 1 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 13daea5..2ae304d 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -439,14 +439,20 @@ struct se_tmr_req {
 	struct list_head	tmr_list;
 };
 
+#define TARGET_DIF_SIZE 8
 enum target_prot_op {
-	TARGET_PROT_NORMAL,
-	TARGET_PROT_READ_INSERT,
-	TARGET_PROT_WRITE_INSERT,
-	TARGET_PROT_READ_STRIP,
-	TARGET_PROT_WRITE_STRIP,
-	TARGET_PROT_READ_PASS,
-	TARGET_PROT_WRITE_PASS,
+	TARGET_PROT_NORMAL = 0,
+	TARGET_PROT_DIN_INSERT,
+	TARGET_PROT_DOUT_INSERT,
+	TARGET_PROT_DIN_STRIP,
+	TARGET_PROT_DOUT_STRIP,
+	TARGET_PROT_DIN_PASS,
+	TARGET_PROT_DOUT_PASS
+};
+
+enum target_prot_ho {
+	PROT_SEPERATED,
+	PROT_INTERLEAVED,
 };
 
 enum target_prot_type {
@@ -573,7 +579,7 @@ struct se_cmd {
 	u32			prot_length;
 	struct scatterlist	*t_prot_sg;
 	unsigned int		t_prot_nents;
-	bool			prot_interleaved;
+	enum target_prot_ho     prot_handover;
 	enum target_pi_error	pi_err;
 	u32			block_num;
 };
-- 
1.7.1

[PATCH 02/11] IB/isert: seperate connection protection domains and dma MRs

[Sagi Grimberg]

It is more correct to seperate connections protection domains
and dma_mr handles. protection information support requires to
do so.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   46 ++++++++++++++++---------------
 drivers/infiniband/ulp/isert/ib_isert.h |    2 -
 2 files changed, 24 insertions(+), 24 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 6be57c3..3dd2427 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -248,13 +248,6 @@ isert_create_device_ib_res(struct isert_device *device)
 	}
 	cq_desc = device->cq_desc;
 
-	device->dev_pd = ib_alloc_pd(ib_dev);
-	if (IS_ERR(device->dev_pd)) {
-		ret = PTR_ERR(device->dev_pd);
-		pr_err("ib_alloc_pd failed for dev_pd: %d\n", ret);
-		goto out_cq_desc;
-	}
-
 	for (i = 0; i < device->cqs_used; i++) {
 		cq_desc[i].device = device;
 		cq_desc[i].cq_index = i;
@@ -282,13 +275,6 @@ isert_create_device_ib_res(struct isert_device *device)
 			goto out_cq;
 	}
 
-	device->dev_mr = ib_get_dma_mr(device->dev_pd, IB_ACCESS_LOCAL_WRITE);
-	if (IS_ERR(device->dev_mr)) {
-		ret = PTR_ERR(device->dev_mr);
-		pr_err("ib_get_dma_mr failed for dev_mr: %d\n", ret);
-		goto out_cq;
-	}
-
 	return 0;
 
 out_cq:
@@ -304,9 +290,6 @@ out_cq:
 			ib_destroy_cq(device->dev_tx_cq[j]);
 		}
 	}
-	ib_dealloc_pd(device->dev_pd);
-
-out_cq_desc:
 	kfree(device->cq_desc);
 
 	return ret;
@@ -329,8 +312,6 @@ isert_free_device_ib_res(struct isert_device *device)
 		device->dev_tx_cq[i] = NULL;
 	}
 
-	ib_dereg_mr(device->dev_mr);
-	ib_dealloc_pd(device->dev_pd);
 	kfree(device->cq_desc);
 }
 
@@ -437,7 +418,7 @@ isert_conn_create_frwr_pool(struct isert_conn *isert_conn)
 			goto err;
 		}
 
-		fr_desc->data_mr = ib_alloc_fast_reg_mr(device->dev_pd,
+		fr_desc->data_mr = ib_alloc_fast_reg_mr(isert_conn->conn_pd,
 					ISCSI_ISER_SG_TABLESIZE);
 		if (IS_ERR(fr_desc->data_mr)) {
 			pr_err("Failed to allocate frmr err=%ld\n",
@@ -546,8 +527,22 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 	}
 
 	isert_conn->conn_device = device;
-	isert_conn->conn_pd = device->dev_pd;
-	isert_conn->conn_mr = device->dev_mr;
+	isert_conn->conn_pd = ib_alloc_pd(isert_conn->conn_device->ib_device);
+	if (IS_ERR(isert_conn->conn_pd)) {
+		ret = PTR_ERR(isert_conn->conn_pd);
+		pr_err("ib_alloc_pd failed for conn %p: ret=%d\n",
+		       isert_conn, ret);
+		goto out_pd;
+	}
+
+	isert_conn->conn_mr = ib_get_dma_mr(isert_conn->conn_pd,
+					   IB_ACCESS_LOCAL_WRITE);
+	if (IS_ERR(isert_conn->conn_mr)) {
+		ret = PTR_ERR(isert_conn->conn_mr);
+		pr_err("ib_get_dma_mr failed for conn %p: ret=%d\n",
+		       isert_conn, ret);
+		goto out_mr;
+	}
 
 	if (device->use_frwr) {
 		ret = isert_conn_create_frwr_pool(isert_conn);
@@ -573,6 +568,10 @@ out_conn_dev:
 	if (device->use_frwr)
 		isert_conn_free_frwr_pool(isert_conn);
 out_frwr:
+	ib_dereg_mr(isert_conn->conn_mr);
+out_mr:
+	ib_dealloc_pd(isert_conn->conn_pd);
+out_pd:
 	isert_device_try_release(device);
 out_rsp_dma_map:
 	ib_dma_unmap_single(ib_dev, isert_conn->login_rsp_dma,
@@ -611,6 +610,9 @@ isert_connect_release(struct isert_conn *isert_conn)
 	isert_free_rx_descriptors(isert_conn);
 	rdma_destroy_id(isert_conn->conn_cm_id);
 
+	ib_dereg_mr(isert_conn->conn_mr);
+	ib_dealloc_pd(isert_conn->conn_pd);
+
 	if (isert_conn->login_buf) {
 		ib_dma_unmap_single(ib_dev, isert_conn->login_rsp_dma,
 				    ISER_RX_LOGIN_SIZE, DMA_TO_DEVICE);
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 691f90f..dec74d4 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -144,8 +144,6 @@ struct isert_device {
 	int			refcount;
 	int			cq_active_qps[ISERT_MAX_CQ];
 	struct ib_device	*ib_device;
-	struct ib_pd		*dev_pd;
-	struct ib_mr		*dev_mr;
 	struct ib_cq		*dev_rx_cq[ISERT_MAX_CQ];
 	struct ib_cq		*dev_tx_cq[ISERT_MAX_CQ];
 	struct isert_cq_desc	*cq_desc;
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 03/11] IB/isert: Avoid frwr notation, user fastreg

[Sagi Grimberg]

Use fast registration lingo. fast registration will
also incorporate signature/DIF registration.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   84 ++++++++++++++++---------------
 drivers/infiniband/ulp/isert/ib_isert.h |    8 ++--
 2 files changed, 47 insertions(+), 45 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 3dd2427..295d2be 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -47,10 +47,10 @@ static int
 isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	       struct isert_rdma_wr *wr);
 static void
-isert_unreg_rdma_frwr(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn);
+isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn);
 static int
-isert_reg_rdma_frwr(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
-		    struct isert_rdma_wr *wr);
+isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
+	       struct isert_rdma_wr *wr);
 
 static void
 isert_qp_event_callback(struct ib_event *e, void *context)
@@ -225,11 +225,11 @@ isert_create_device_ib_res(struct isert_device *device)
 
 	/* asign function handlers */
 	if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) {
-		device->use_frwr = 1;
-		device->reg_rdma_mem = isert_reg_rdma_frwr;
-		device->unreg_rdma_mem = isert_unreg_rdma_frwr;
+		device->use_fastreg = 1;
+		device->reg_rdma_mem = isert_reg_rdma;
+		device->unreg_rdma_mem = isert_unreg_rdma;
 	} else {
-		device->use_frwr = 0;
+		device->use_fastreg = 0;
 		device->reg_rdma_mem = isert_map_rdma;
 		device->unreg_rdma_mem = isert_unmap_cmd;
 	}
@@ -237,9 +237,10 @@ isert_create_device_ib_res(struct isert_device *device)
 	device->cqs_used = min_t(int, num_online_cpus(),
 				 device->ib_device->num_comp_vectors);
 	device->cqs_used = min(ISERT_MAX_CQ, device->cqs_used);
-	pr_debug("Using %d CQs, device %s supports %d vectors support FRWR %d\n",
+	pr_debug("Using %d CQs, device %s supports %d vectors support "
+		 "Fast registration %d\n",
 		 device->cqs_used, device->ib_device->name,
-		 device->ib_device->num_comp_vectors, device->use_frwr);
+		 device->ib_device->num_comp_vectors, device->use_fastreg);
 	device->cq_desc = kzalloc(sizeof(struct isert_cq_desc) *
 				device->cqs_used, GFP_KERNEL);
 	if (!device->cq_desc) {
@@ -367,18 +368,18 @@ isert_device_find_by_ib_dev(struct rdma_cm_id *cma_id)
 }
 
 static void
-isert_conn_free_frwr_pool(struct isert_conn *isert_conn)
+isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
 {
 	struct fast_reg_descriptor *fr_desc, *tmp;
 	int i = 0;
 
-	if (list_empty(&isert_conn->conn_frwr_pool))
+	if (list_empty(&isert_conn->conn_fr_pool))
 		return;
 
-	pr_debug("Freeing conn %p frwr pool", isert_conn);
+	pr_debug("Freeing conn %p fastreg pool", isert_conn);
 
 	list_for_each_entry_safe(fr_desc, tmp,
-				 &isert_conn->conn_frwr_pool, list) {
+				 &isert_conn->conn_fr_pool, list) {
 		list_del(&fr_desc->list);
 		ib_free_fast_reg_page_list(fr_desc->data_frpl);
 		ib_dereg_mr(fr_desc->data_mr);
@@ -386,20 +387,20 @@ isert_conn_free_frwr_pool(struct isert_conn *isert_conn)
 		++i;
 	}
 
-	if (i < isert_conn->conn_frwr_pool_size)
+	if (i < isert_conn->conn_fr_pool_size)
 		pr_warn("Pool still has %d regions registered\n",
-			isert_conn->conn_frwr_pool_size - i);
+			isert_conn->conn_fr_pool_size - i);
 }
 
 static int
-isert_conn_create_frwr_pool(struct isert_conn *isert_conn)
+isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
 {
 	struct fast_reg_descriptor *fr_desc;
 	struct isert_device *device = isert_conn->conn_device;
 	int i, ret;
 
-	INIT_LIST_HEAD(&isert_conn->conn_frwr_pool);
-	isert_conn->conn_frwr_pool_size = 0;
+	INIT_LIST_HEAD(&isert_conn->conn_fr_pool);
+	isert_conn->conn_fr_pool_size = 0;
 	for (i = 0; i < ISCSI_DEF_XMIT_CMDS_MAX; i++) {
 		fr_desc = kzalloc(sizeof(*fr_desc), GFP_KERNEL);
 		if (!fr_desc) {
@@ -431,17 +432,17 @@ isert_conn_create_frwr_pool(struct isert_conn *isert_conn)
 			 fr_desc, fr_desc->data_frpl->page_list);
 
 		fr_desc->valid = true;
-		list_add_tail(&fr_desc->list, &isert_conn->conn_frwr_pool);
-		isert_conn->conn_frwr_pool_size++;
+		list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
+		isert_conn->conn_fr_pool_size++;
 	}
 
-	pr_debug("Creating conn %p frwr pool size=%d",
-		 isert_conn, isert_conn->conn_frwr_pool_size);
+	pr_debug("Creating conn %p fastreg pool size=%d",
+		 isert_conn, isert_conn->conn_fr_pool_size);
 
 	return 0;
 
 err:
-	isert_conn_free_frwr_pool(isert_conn);
+	isert_conn_free_fastreg_pool(isert_conn);
 	return ret;
 }
 
@@ -544,11 +545,12 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 		goto out_mr;
 	}
 
-	if (device->use_frwr) {
-		ret = isert_conn_create_frwr_pool(isert_conn);
+	if (device->use_fastreg) {
+		ret = isert_conn_create_fastreg_pool(isert_conn);
 		if (ret) {
-			pr_err("Conn: %p failed to create frwr_pool\n", isert_conn);
-			goto out_frwr;
+			pr_err("Conn: %p failed to create fastreg pool\n",
+			       isert_conn);
+			goto out_fastreg;
 		}
 	}
 
@@ -565,9 +567,9 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 	return 0;
 
 out_conn_dev:
-	if (device->use_frwr)
-		isert_conn_free_frwr_pool(isert_conn);
-out_frwr:
+	if (device->use_fastreg)
+		isert_conn_free_fastreg_pool(isert_conn);
+out_fastreg:
 	ib_dereg_mr(isert_conn->conn_mr);
 out_mr:
 	ib_dealloc_pd(isert_conn->conn_pd);
@@ -595,8 +597,8 @@ isert_connect_release(struct isert_conn *isert_conn)
 
 	pr_debug("Entering isert_connect_release(): >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>\n");
 
-	if (device && device->use_frwr)
-		isert_conn_free_frwr_pool(isert_conn);
+	if (device && device->use_fastreg)
+		isert_conn_free_fastreg_pool(isert_conn);
 
 	if (isert_conn->conn_qp) {
 		cq_index = ((struct isert_cq_desc *)
@@ -1394,25 +1396,25 @@ isert_unmap_cmd(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 }
 
 static void
-isert_unreg_rdma_frwr(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
+isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 {
 	struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 	LIST_HEAD(unmap_list);
 
-	pr_debug("unreg_frwr_cmd: %p\n", isert_cmd);
+	pr_debug("unreg_fastreg_cmd: %p\n", isert_cmd);
 
 	if (wr->fr_desc) {
-		pr_debug("unreg_frwr_cmd: %p free fr_desc %p\n",
+		pr_debug("unreg_fastreg_cmd: %p free fr_desc %p\n",
 			 isert_cmd, wr->fr_desc);
 		spin_lock_bh(&isert_conn->conn_lock);
-		list_add_tail(&wr->fr_desc->list, &isert_conn->conn_frwr_pool);
+		list_add_tail(&wr->fr_desc->list, &isert_conn->conn_fr_pool);
 		spin_unlock_bh(&isert_conn->conn_lock);
 		wr->fr_desc = NULL;
 	}
 
 	if (wr->sge) {
-		pr_debug("unreg_frwr_cmd: %p unmap_sg op\n", isert_cmd);
+		pr_debug("unreg_fastreg_cmd: %p unmap_sg op\n", isert_cmd);
 		ib_dma_unmap_sg(ib_dev, wr->sge, wr->num_sge,
 				(wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
 				DMA_TO_DEVICE : DMA_FROM_DEVICE);
@@ -2224,8 +2226,8 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 }
 
 static int
-isert_reg_rdma_frwr(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
-		    struct isert_rdma_wr *wr)
+isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
+	       struct isert_rdma_wr *wr)
 {
 	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
@@ -2303,7 +2305,7 @@ isert_reg_rdma_frwr(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		wr->fr_desc = NULL;
 	} else {
 		spin_lock_irqsave(&isert_conn->conn_lock, flags);
-		fr_desc = list_first_entry(&isert_conn->conn_frwr_pool,
+		fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
 					   struct fast_reg_descriptor, list);
 		list_del(&fr_desc->list);
 		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
@@ -2312,7 +2314,7 @@ isert_reg_rdma_frwr(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		ret = isert_fast_reg_mr(fr_desc, isert_cmd, isert_conn,
 				  ib_sge, offset, data_len);
 		if (ret) {
-			list_add_tail(&fr_desc->list, &isert_conn->conn_frwr_pool);
+			list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
 			goto unmap_sg;
 		}
 	}
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index dec74d4..708a069 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -119,9 +119,9 @@ struct isert_conn {
 	wait_queue_head_t	conn_wait;
 	wait_queue_head_t	conn_wait_comp_err;
 	struct kref		conn_kref;
-	struct list_head	conn_frwr_pool;
-	int			conn_frwr_pool_size;
-	/* lock to protect frwr_pool */
+	struct list_head	conn_fr_pool;
+	int			conn_fr_pool_size;
+	/* lock to protect fastreg pool */
 	spinlock_t		conn_lock;
 #define ISERT_COMP_BATCH_COUNT	8
 	int			conn_comp_batch;
@@ -139,7 +139,7 @@ struct isert_cq_desc {
 };
 
 struct isert_device {
-	int			use_frwr;
+	int			use_fastreg;
 	int			cqs_used;
 	int			refcount;
 	int			cq_active_qps[ISERT_MAX_CQ];
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 04/11] IB/isert: Move fastreg descriptor creation to a function

[Sagi Grimberg]

This routine may be called both by fast registration
descriptors for data and for integrity buffers.

This patch does not change any functionality.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   52 +++++++++++++++++++------------
 1 files changed, 32 insertions(+), 20 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 295d2be..9ef9193 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -393,6 +393,33 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
 }
 
 static int
+isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
+		     struct fast_reg_descriptor *fr_desc)
+{
+	fr_desc->data_frpl = ib_alloc_fast_reg_page_list(ib_device,
+							 ISCSI_ISER_SG_TABLESIZE);
+	if (IS_ERR(fr_desc->data_frpl)) {
+		pr_err("Failed to allocate data frpl err=%ld\n",
+		       PTR_ERR(fr_desc->data_frpl));
+		return PTR_ERR(fr_desc->data_frpl);
+	}
+
+	fr_desc->data_mr = ib_alloc_fast_reg_mr(pd, ISCSI_ISER_SG_TABLESIZE);
+	if (IS_ERR(fr_desc->data_mr)) {
+		pr_err("Failed to allocate data frmr err=%ld\n",
+		       PTR_ERR(fr_desc->data_mr));
+		ib_free_fast_reg_page_list(fr_desc->data_frpl);
+		return PTR_ERR(fr_desc->data_mr);
+	}
+	pr_debug("Create fr_desc %p page_list %p\n",
+		 fr_desc, fr_desc->data_frpl->page_list);
+
+	fr_desc->valid = true;
+
+	return 0;
+}
+
+static int
 isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
 {
 	struct fast_reg_descriptor *fr_desc;
@@ -409,29 +436,14 @@ isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
 			goto err;
 		}
 
-		fr_desc->data_frpl =
-			ib_alloc_fast_reg_page_list(device->ib_device,
-						    ISCSI_ISER_SG_TABLESIZE);
-		if (IS_ERR(fr_desc->data_frpl)) {
-			pr_err("Failed to allocate fr_pg_list err=%ld\n",
-			       PTR_ERR(fr_desc->data_frpl));
-			ret = PTR_ERR(fr_desc->data_frpl);
-			goto err;
-		}
-
-		fr_desc->data_mr = ib_alloc_fast_reg_mr(isert_conn->conn_pd,
-					ISCSI_ISER_SG_TABLESIZE);
-		if (IS_ERR(fr_desc->data_mr)) {
-			pr_err("Failed to allocate frmr err=%ld\n",
-			       PTR_ERR(fr_desc->data_mr));
-			ret = PTR_ERR(fr_desc->data_mr);
-			ib_free_fast_reg_page_list(fr_desc->data_frpl);
+		ret = isert_create_fr_desc(device->ib_device,
+					   isert_conn->conn_pd, fr_desc);
+		if (ret) {
+			pr_err("Failed to create fastreg descriptor err=%d\n",
+			       ret);
 			goto err;
 		}
-		pr_debug("Create fr_desc %p page_list %p\n",
-			 fr_desc, fr_desc->data_frpl->page_list);
 
-		fr_desc->valid = true;
 		list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
 		isert_conn->conn_fr_pool_size++;
 	}
-- 
1.7.1

[PATCH 05/11] Target/iscsi: Add T10-PI indication for iscsi_portal_group

[Sagi Grimberg]

In case an iscsi portal group will be defined as t10_pi enabled,
all connections on top of it will support protected transactions.

T10-PI support may require extra reource allocation and maintenance by
the transport layer, so we don't want to apply them on non-t10_pi network
portals. This is a hook for the iscsi target layer to signal the transport
at connection establishment that this connection will carry protected
transactions.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/target/iscsi/iscsi_target_core.h |    5 ++++-
 drivers/target/iscsi/iscsi_target_tpg.c  |    2 ++
 2 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_core.h b/drivers/target/iscsi/iscsi_target_core.h
index 48f7b3b..886d74d 100644
--- a/drivers/target/iscsi/iscsi_target_core.h
+++ b/drivers/target/iscsi/iscsi_target_core.h
@@ -58,7 +58,8 @@
 #define TA_DEMO_MODE_DISCOVERY		1
 #define TA_DEFAULT_ERL			0
 #define TA_CACHE_CORE_NPS		0
-
+/* T10 protection information disabled by default */
+#define TA_DEFAULT_T10_PI		0
 
 #define ISCSI_IOV_DATA_BUFFER		5
 
@@ -765,6 +766,7 @@ struct iscsi_tpg_attrib {
 	u32			prod_mode_write_protect;
 	u32			demo_mode_discovery;
 	u32			default_erl;
+	u8			t10_pi;
 	struct iscsi_portal_group *tpg;
 };
 
@@ -787,6 +789,7 @@ struct iscsi_np {
 	void			*np_context;
 	struct iscsit_transport *np_transport;
 	struct list_head	np_list;
+	struct iscsi_tpg_np	*tpg_np;
 } ____cacheline_aligned;
 
 struct iscsi_tpg_np {
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 3976183..80ae14c 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -225,6 +225,7 @@ static void iscsit_set_default_tpg_attribs(struct iscsi_portal_group *tpg)
 	a->prod_mode_write_protect = TA_PROD_MODE_WRITE_PROTECT;
 	a->demo_mode_discovery = TA_DEMO_MODE_DISCOVERY;
 	a->default_erl = TA_DEFAULT_ERL;
+	a->t10_pi = TA_DEFAULT_T10_PI;
 }
 
 int iscsit_tpg_add_portal_group(struct iscsi_tiqn *tiqn, struct iscsi_portal_group *tpg)
@@ -500,6 +501,7 @@ struct iscsi_tpg_np *iscsit_tpg_add_network_portal(
 	init_completion(&tpg_np->tpg_np_comp);
 	kref_init(&tpg_np->tpg_np_kref);
 	tpg_np->tpg_np		= np;
+	np->tpg_np		= tpg_np;
 	tpg_np->tpg		= tpg;
 
 	spin_lock(&tpg->tpg_np_lock);
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 06/11] IB/isert: Initialize T10-PI resources

[Sagi Grimberg]

Upon connection establishment check if network portal is T10-PI
enabled and allocate T10-PI resources if necessary, allocate
signature enabled memory regions and mark connection queue-pair
as signature enabled.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c |  104 +++++++++++++++++++++++++++----
 drivers/infiniband/ulp/isert/ib_isert.h |   19 +++++-
 2 files changed, 106 insertions(+), 17 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 9ef9193..98f23f4 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -87,7 +87,8 @@ isert_query_device(struct ib_device *ib_dev, struct ib_device_attr *devattr)
 }
 
 static int
-isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id)
+isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id,
+		    u8 protection)
 {
 	struct isert_device *device = isert_conn->conn_device;
 	struct ib_qp_init_attr attr;
@@ -119,6 +120,8 @@ isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id)
 	attr.cap.max_recv_sge = 1;
 	attr.sq_sig_type = IB_SIGNAL_REQ_WR;
 	attr.qp_type = IB_QPT_RC;
+	if (protection)
+		attr.create_flags |= IB_QP_CREATE_SIGNATURE_EN;
 
 	pr_debug("isert_conn_setup_qp cma_id->device: %p\n",
 		 cma_id->device);
@@ -234,13 +237,18 @@ isert_create_device_ib_res(struct isert_device *device)
 		device->unreg_rdma_mem = isert_unmap_cmd;
 	}
 
+	/* Check signature cap */
+	device->pi_capable = dev_attr->device_cap_flags &
+			     IB_DEVICE_SIGNATURE_HANDOVER ? true : false;
+
 	device->cqs_used = min_t(int, num_online_cpus(),
 				 device->ib_device->num_comp_vectors);
 	device->cqs_used = min(ISERT_MAX_CQ, device->cqs_used);
 	pr_debug("Using %d CQs, device %s supports %d vectors support "
-		 "Fast registration %d\n",
+		 "Fast registration %d pi_capable %d\n",
 		 device->cqs_used, device->ib_device->name,
-		 device->ib_device->num_comp_vectors, device->use_fastreg);
+		 device->ib_device->num_comp_vectors, device->use_fastreg,
+		 device->pi_capable);
 	device->cq_desc = kzalloc(sizeof(struct isert_cq_desc) *
 				device->cqs_used, GFP_KERNEL);
 	if (!device->cq_desc) {
@@ -383,6 +391,12 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
 		list_del(&fr_desc->list);
 		ib_free_fast_reg_page_list(fr_desc->data_frpl);
 		ib_dereg_mr(fr_desc->data_mr);
+		if (fr_desc->pi_ctx) {
+			ib_free_fast_reg_page_list(fr_desc->pi_ctx->prot_frpl);
+			ib_dereg_mr(fr_desc->pi_ctx->prot_mr);
+			ib_destroy_mr(fr_desc->pi_ctx->sig_mr);
+			kfree(fr_desc->pi_ctx);
+		}
 		kfree(fr_desc);
 		++i;
 	}
@@ -394,8 +408,10 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
 
 static int
 isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
-		     struct fast_reg_descriptor *fr_desc)
+		     struct fast_reg_descriptor *fr_desc, u8 protection)
 {
+	int ret;
+
 	fr_desc->data_frpl = ib_alloc_fast_reg_page_list(ib_device,
 							 ISCSI_ISER_SG_TABLESIZE);
 	if (IS_ERR(fr_desc->data_frpl)) {
@@ -408,19 +424,73 @@ isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
 	if (IS_ERR(fr_desc->data_mr)) {
 		pr_err("Failed to allocate data frmr err=%ld\n",
 		       PTR_ERR(fr_desc->data_mr));
-		ib_free_fast_reg_page_list(fr_desc->data_frpl);
-		return PTR_ERR(fr_desc->data_mr);
+		ret = PTR_ERR(fr_desc->data_mr);
+		goto err_data_frpl;
 	}
 	pr_debug("Create fr_desc %p page_list %p\n",
 		 fr_desc, fr_desc->data_frpl->page_list);
+	fr_desc->data_key_valid = true;
 
-	fr_desc->valid = true;
+	if (protection) {
+		struct ib_mr_init_attr mr_init_attr = {0};
+		struct pi_context *pi_ctx;
+
+		fr_desc->pi_ctx = kzalloc(sizeof(*fr_desc->pi_ctx), GFP_KERNEL);
+		if (!fr_desc->pi_ctx) {
+			pr_err("Failed to allocate pi context\n");
+			ret = -ENOMEM;
+			goto err_data_mr;
+		}
+		pi_ctx = fr_desc->pi_ctx;
+
+		pi_ctx->prot_frpl = ib_alloc_fast_reg_page_list(ib_device,
+						    ISCSI_ISER_SG_TABLESIZE);
+		if (IS_ERR(pi_ctx->prot_frpl)) {
+			pr_err("Failed to allocate prot frpl err=%ld\n",
+			       PTR_ERR(pi_ctx->prot_frpl));
+			ret = PTR_ERR(pi_ctx->prot_frpl);
+			goto err_pi_ctx;
+		}
+
+		pi_ctx->prot_mr = ib_alloc_fast_reg_mr(pd, ISCSI_ISER_SG_TABLESIZE);
+		if (IS_ERR(pi_ctx->prot_mr)) {
+			pr_err("Failed to allocate prot frmr err=%ld\n",
+			       PTR_ERR(pi_ctx->prot_mr));
+			ret = PTR_ERR(pi_ctx->prot_mr);
+			goto err_prot_frpl;
+		}
+		pi_ctx->prot_key_valid = true;
+
+		mr_init_attr.max_reg_descriptors = 2;
+		mr_init_attr.flags |= IB_MR_SIGNATURE_EN;
+		pi_ctx->sig_mr = ib_create_mr(pd, &mr_init_attr);
+		if (IS_ERR(pi_ctx->sig_mr)) {
+			pr_err("Failed to allocate signature enabled mr err=%ld\n",
+			       PTR_ERR(pi_ctx->sig_mr));
+			ret = PTR_ERR(pi_ctx->sig_mr);
+			goto err_prot_mr;
+		}
+		pi_ctx->sig_key_valid = true;
+	}
+	fr_desc->protected = false;
 
 	return 0;
+err_prot_mr:
+	ib_dereg_mr(fr_desc->pi_ctx->prot_mr);
+err_prot_frpl:
+	ib_free_fast_reg_page_list(fr_desc->pi_ctx->prot_frpl);
+err_pi_ctx:
+	kfree(fr_desc->pi_ctx);
+err_data_mr:
+	ib_dereg_mr(fr_desc->data_mr);
+err_data_frpl:
+	ib_free_fast_reg_page_list(fr_desc->data_frpl);
+
+	return ret;
 }
 
 static int
-isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
+isert_conn_create_fastreg_pool(struct isert_conn *isert_conn, u8 pi_support)
 {
 	struct fast_reg_descriptor *fr_desc;
 	struct isert_device *device = isert_conn->conn_device;
@@ -437,7 +507,8 @@ isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
 		}
 
 		ret = isert_create_fr_desc(device->ib_device,
-					   isert_conn->conn_pd, fr_desc);
+					   isert_conn->conn_pd, fr_desc,
+					   pi_support);
 		if (ret) {
 			pr_err("Failed to create fastreg descriptor err=%d\n",
 			       ret);
@@ -467,6 +538,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 	struct isert_device *device;
 	struct ib_device *ib_dev = cma_id->device;
 	int ret = 0;
+	u8 pi_support = np->tpg_np->tpg->tpg_attrib.t10_pi;
 
 	pr_debug("Entering isert_connect_request cma_id: %p, context: %p\n",
 		 cma_id, cma_id->context);
@@ -557,8 +629,14 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 		goto out_mr;
 	}
 
+	if (pi_support && !device->pi_capable) {
+		pr_err("Protection information requested but not supported\n");
+		ret = -EINVAL;
+		goto out_mr;
+	}
+
 	if (device->use_fastreg) {
-		ret = isert_conn_create_fastreg_pool(isert_conn);
+		ret = isert_conn_create_fastreg_pool(isert_conn, pi_support);
 		if (ret) {
 			pr_err("Conn: %p failed to create fastreg pool\n",
 			       isert_conn);
@@ -566,7 +644,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 		}
 	}
 
-	ret = isert_conn_setup_qp(isert_conn, cma_id);
+	ret = isert_conn_setup_qp(isert_conn, cma_id, pi_support);
 	if (ret)
 		goto out_conn_dev;
 
@@ -2193,7 +2271,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 	pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
 					     &fr_desc->data_frpl->page_list[0]);
 
-	if (!fr_desc->valid) {
+	if (!fr_desc->data_key_valid) {
 		memset(&inv_wr, 0, sizeof(inv_wr));
 		inv_wr.opcode = IB_WR_LOCAL_INV;
 		inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
@@ -2225,7 +2303,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 		pr_err("fast registration failed, ret:%d\n", ret);
 		return ret;
 	}
-	fr_desc->valid = false;
+	fr_desc->data_key_valid = false;
 
 	ib_sge->lkey = fr_desc->data_mr->lkey;
 	ib_sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 708a069..fab8b50 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -48,11 +48,21 @@ struct iser_tx_desc {
 	struct ib_send_wr send_wr;
 } __packed;
 
+struct pi_context {
+	struct ib_mr		       *prot_mr;
+	bool				prot_key_valid;
+	struct ib_fast_reg_page_list   *prot_frpl;
+	struct ib_mr		       *sig_mr;
+	bool				sig_key_valid;
+};
+
 struct fast_reg_descriptor {
-	struct list_head	list;
-	struct ib_mr		*data_mr;
-	struct ib_fast_reg_page_list	*data_frpl;
-	bool			valid;
+	struct list_head		list;
+	struct ib_mr		       *data_mr;
+	bool				data_key_valid;
+	struct ib_fast_reg_page_list   *data_frpl;
+	bool				protected;
+	struct pi_context	       *pi_ctx;
 };
 
 struct isert_rdma_wr {
@@ -140,6 +150,7 @@ struct isert_cq_desc {
 
 struct isert_device {
 	int			use_fastreg;
+	bool			pi_capable;
 	int			cqs_used;
 	int			refcount;
 	int			cq_active_qps[ISERT_MAX_CQ];
-- 
1.7.1

[PATCH 07/11] IB/isert: pass scatterlist instead of cmd to fast_reg_mr routine

[Sagi Grimberg]

This routine may help for protection registration as well.
This patch does not change any functionality.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   28 ++++++++++++----------------
 1 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 98f23f4..3495e73 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -2247,26 +2247,22 @@ isert_map_fr_pagelist(struct ib_device *ib_dev,
 
 static int
 isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
-		  struct isert_cmd *isert_cmd, struct isert_conn *isert_conn,
-		  struct ib_sge *ib_sge, u32 offset, unsigned int data_len)
+		  struct isert_conn *isert_conn, struct scatterlist *sg_start,
+		  struct ib_sge *ib_sge, u32 sg_nents, u32 offset,
+		  unsigned int data_len)
 {
-	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
-	struct scatterlist *sg_start;
-	u32 sg_off, page_off;
 	struct ib_send_wr fr_wr, inv_wr;
 	struct ib_send_wr *bad_wr, *wr = NULL;
+	int ret, pagelist_len;
+	u32 page_off;
 	u8 key;
-	int ret, sg_nents, pagelist_len;
 
-	sg_off = offset / PAGE_SIZE;
-	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
-	sg_nents = min_t(unsigned int, cmd->se_cmd.t_data_nents - sg_off,
-			 ISCSI_ISER_SG_TABLESIZE);
+	sg_nents = min_t(unsigned int, sg_nents, ISCSI_ISER_SG_TABLESIZE);
 	page_off = offset % PAGE_SIZE;
 
-	pr_debug("Cmd: %p use fr_desc %p sg_nents %d sg_off %d offset %u\n",
-		 isert_cmd, fr_desc, sg_nents, sg_off, offset);
+	pr_debug("Use fr_desc %p sg_nents %d offset %u\n",
+		 fr_desc, sg_nents, offset);
 
 	pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
 					     &fr_desc->data_frpl->page_list[0]);
@@ -2335,9 +2331,9 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
 		data_left = se_cmd->data_length;
 	} else {
-		sg_off = cmd->write_data_done / PAGE_SIZE;
-		data_left = se_cmd->data_length - cmd->write_data_done;
 		offset = cmd->write_data_done;
+		sg_off = offset / PAGE_SIZE;
+		data_left = se_cmd->data_length - cmd->write_data_done;
 		isert_cmd->tx_desc.isert_cmd = isert_cmd;
 	}
 
@@ -2401,8 +2397,8 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
 		wr->fr_desc = fr_desc;
 
-		ret = isert_fast_reg_mr(fr_desc, isert_cmd, isert_conn,
-				  ib_sge, offset, data_len);
+		ret = isert_fast_reg_mr(fr_desc, isert_conn, sg_start,
+					ib_sge, sg_nents, offset, data_len);
 		if (ret) {
 			list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
 			goto unmap_sg;
-- 
1.7.1

[PATCH 08/11] IB/isert: pass mr and frpl to isert_fast_reg_mr routine

[Sagi Grimberg]

This commit generalizes isert_fast_reg_mr to receive mr
and frpl instead of fr_desc to do registration. In T10-PI
we also register protection memory region so we want to
use this routine.

This commit does not change any functionality.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   62 +++++++++++++++----------------
 1 files changed, 30 insertions(+), 32 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 3495e73..98aab21 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -2246,10 +2246,10 @@ isert_map_fr_pagelist(struct ib_device *ib_dev,
 }
 
 static int
-isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
-		  struct isert_conn *isert_conn, struct scatterlist *sg_start,
-		  struct ib_sge *ib_sge, u32 sg_nents, u32 offset,
-		  unsigned int data_len)
+isert_fast_reg_mr(struct isert_conn *isert_conn, struct ib_mr *mr,
+	          struct ib_fast_reg_page_list *frpl, bool *key_valid,
+	          struct scatterlist *sg_start, u32 sg_nents, u32 offset,
+		  unsigned int data_len, struct ib_sge *ib_sge)
 {
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 	struct ib_send_wr fr_wr, inv_wr;
@@ -2260,33 +2260,31 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 
 	sg_nents = min_t(unsigned int, sg_nents, ISCSI_ISER_SG_TABLESIZE);
 	page_off = offset % PAGE_SIZE;
-
-	pr_debug("Use fr_desc %p sg_nents %d offset %u\n",
-		 fr_desc, sg_nents, offset);
+	pr_debug("Use mr %p frpl %p sg_nents %d offset %u\n",
+		 mr, frpl, sg_nents, offset);
 
 	pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
-					     &fr_desc->data_frpl->page_list[0]);
+					     &frpl->page_list[0]);
 
-	if (!fr_desc->data_key_valid) {
+	if (!*key_valid) {
 		memset(&inv_wr, 0, sizeof(inv_wr));
 		inv_wr.opcode = IB_WR_LOCAL_INV;
-		inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
+		inv_wr.ex.invalidate_rkey = mr->rkey;
 		wr = &inv_wr;
 		/* Bump the key */
-		key = (u8)(fr_desc->data_mr->rkey & 0x000000FF);
-		ib_update_fast_reg_key(fr_desc->data_mr, ++key);
+		key = (u8)(mr->rkey & 0x000000FF);
+		ib_update_fast_reg_key(mr, ++key);
 	}
 
 	/* Prepare FASTREG WR */
 	memset(&fr_wr, 0, sizeof(fr_wr));
 	fr_wr.opcode = IB_WR_FAST_REG_MR;
-	fr_wr.wr.fast_reg.iova_start =
-		fr_desc->data_frpl->page_list[0] + page_off;
-	fr_wr.wr.fast_reg.page_list = fr_desc->data_frpl;
+	fr_wr.wr.fast_reg.iova_start = frpl->page_list[0] + page_off;
+	fr_wr.wr.fast_reg.page_list = frpl;
 	fr_wr.wr.fast_reg.page_list_len = pagelist_len;
 	fr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
 	fr_wr.wr.fast_reg.length = data_len;
-	fr_wr.wr.fast_reg.rkey = fr_desc->data_mr->rkey;
+	fr_wr.wr.fast_reg.rkey = mr->rkey;
 	fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
 
 	if (!wr)
@@ -2299,14 +2297,14 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 		pr_err("fast registration failed, ret:%d\n", ret);
 		return ret;
 	}
-	fr_desc->data_key_valid = false;
 
-	ib_sge->lkey = fr_desc->data_mr->lkey;
-	ib_sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
+	*key_valid = false;
+	ib_sge->lkey = mr->lkey;
+	ib_sge->addr = frpl->page_list[0] + page_off;
 	ib_sge->length = data_len;
 
-	pr_debug("RDMA ib_sge: addr: 0x%16llx  length: %u lkey: %08x\n",
-		 ib_sge->addr, ib_sge->length, ib_sge->lkey);
+	pr_debug("fastreg ib_sge: addr: 0x%16llx  length: %u lkey: %08x\n",
+		 ib_sge->addr + page_off, ib_sge->length, ib_sge->lkey);
 
 	return ret;
 }
@@ -2320,7 +2318,7 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	struct isert_conn *isert_conn = (struct isert_conn *)conn->context;
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 	struct ib_send_wr *send_wr;
-	struct ib_sge *ib_sge;
+	struct ib_sge data_sge;
 	struct scatterlist *sg_start;
 	struct fast_reg_descriptor *fr_desc;
 	u32 sg_off = 0, sg_nents;
@@ -2352,10 +2350,7 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
 		 isert_cmd, count, sg_start, sg_nents, data_left);
 
-	memset(&wr->s_ib_sge, 0, sizeof(*ib_sge));
-	ib_sge = &wr->s_ib_sge;
-	wr->ib_sge = ib_sge;
-
+	wr->ib_sge = &wr->s_ib_sge;
 	wr->send_wr_num = 1;
 	memset(&wr->s_send_wr, 0, sizeof(*send_wr));
 	wr->send_wr = &wr->s_send_wr;
@@ -2364,7 +2359,7 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	rdma_write_max = ISCSI_ISER_SG_TABLESIZE * PAGE_SIZE;
 
 	send_wr = &isert_cmd->rdma_wr.s_send_wr;
-	send_wr->sg_list = ib_sge;
+	send_wr->sg_list = &wr->s_ib_sge;
 	send_wr->num_sge = 1;
 	send_wr->wr_id = (unsigned long)&isert_cmd->tx_desc;
 	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
@@ -2385,9 +2380,9 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 
 	/* if there is a single dma entry, dma mr is sufficient */
 	if (count == 1) {
-		ib_sge->addr = ib_sg_dma_address(ib_dev, &sg_start[0]);
-		ib_sge->length = ib_sg_dma_len(ib_dev, &sg_start[0]);
-		ib_sge->lkey = isert_conn->conn_mr->lkey;
+		wr->s_ib_sge.addr = ib_sg_dma_address(ib_dev, &sg_start[0]);
+		wr->s_ib_sge.length = ib_sg_dma_len(ib_dev, &sg_start[0]);
+		wr->s_ib_sge.lkey = isert_conn->conn_mr->lkey;
 		wr->fr_desc = NULL;
 	} else {
 		spin_lock_irqsave(&isert_conn->conn_lock, flags);
@@ -2397,12 +2392,15 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
 		wr->fr_desc = fr_desc;
 
-		ret = isert_fast_reg_mr(fr_desc, isert_conn, sg_start,
-					ib_sge, sg_nents, offset, data_len);
+		ret = isert_fast_reg_mr(isert_conn, fr_desc->data_mr,
+					fr_desc->data_frpl,
+					&fr_desc->data_key_valid, sg_start,
+					sg_nents, offset, data_len, &data_sge);
 		if (ret) {
 			list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
 			goto unmap_sg;
 		}
+		memcpy(&wr->s_ib_sge, &data_sge, sizeof(data_sge));
 	}
 
 	return 0;
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 09/11] IB/isert: Accept RDMA_WRITE completions

[Sagi Grimberg]

In case of protected transactions, we will need to check the
protection status of the transaction before sending SCSI response.
So be ready for RDMA_WRITE completions. currently we don't ask
for these completions, but for T10-PI we will.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   20 +++++++++++++++++---
 1 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 98aab21..9aa933e 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -51,6 +51,8 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn);
 static int
 isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	       struct isert_rdma_wr *wr);
+static int
+isert_put_response(struct iscsi_conn *conn, struct iscsi_cmd *cmd);
 
 static void
 isert_qp_event_callback(struct ib_event *e, void *context)
@@ -1602,6 +1604,18 @@ isert_completion_put(struct iser_tx_desc *tx_desc, struct isert_cmd *isert_cmd,
 }
 
 static void
+isert_completion_rdma_write(struct iser_tx_desc *tx_desc,
+			    struct isert_cmd *isert_cmd)
+{
+	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+	struct isert_conn *isert_conn = isert_cmd->conn;
+	struct isert_device *device = isert_conn->conn_device;
+
+	device->unreg_rdma_mem(isert_cmd, isert_conn);
+	isert_put_response(isert_conn->conn, cmd);
+}
+
+static void
 isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
 			   struct isert_cmd *isert_cmd)
 {
@@ -1721,9 +1735,9 @@ __isert_send_completion(struct iser_tx_desc *tx_desc,
 					  isert_conn, ib_dev);
 		break;
 	case ISER_IB_RDMA_WRITE:
-		pr_err("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
-		dump_stack();
-		break;
+		pr_debug("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
+		atomic_dec(&isert_conn->post_send_buf_count);
+		isert_completion_rdma_write(tx_desc, isert_cmd);
 	case ISER_IB_RDMA_READ:
 		pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n");
 
-- 
1.7.1

[PATCH 10/11] IB/isert: Support T10-PI protected transactions

[Sagi Grimberg]

In case the Target core passed transport T10 protection
operation:

1. Register data buffer (data memory region)
2. Register protection buffer if exsists (prot memory region)
3. Register signature region (signature memory region)
   - use work request IB_WR_REG_SIG_MR
4. Execute RDMA
5. Upon RDMA completion check the signature status
   - if succeeded send good SCSI response
   - if failed send SCSI bad response with appropriate sense buffer

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c |  376 ++++++++++++++++++++++++++-----
 1 files changed, 321 insertions(+), 55 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 9aa933e..8a888f0 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1499,6 +1499,7 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 	if (wr->fr_desc) {
 		pr_debug("unreg_fastreg_cmd: %p free fr_desc %p\n",
 			 isert_cmd, wr->fr_desc);
+		wr->fr_desc->protected = false;
 		spin_lock_bh(&isert_conn->conn_lock);
 		list_add_tail(&wr->fr_desc->list, &isert_conn->conn_fr_pool);
 		spin_unlock_bh(&isert_conn->conn_lock);
@@ -1604,13 +1605,65 @@ isert_completion_put(struct iser_tx_desc *tx_desc, struct isert_cmd *isert_cmd,
 }
 
 static void
+isert_pi_err_sense_buffer(u8 *buf, u8 key, u8 asc, u8 ascq)
+{
+	buf[0] = 0x70;
+	buf[SPC_SENSE_KEY_OFFSET] = key;
+	buf[SPC_ASC_KEY_OFFSET] = asc;
+	buf[SPC_ASCQ_KEY_OFFSET] = ascq;
+}
+
+static void
 isert_completion_rdma_write(struct iser_tx_desc *tx_desc,
 			    struct isert_cmd *isert_cmd)
 {
+	struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
 	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_conn *isert_conn = isert_cmd->conn;
 	struct isert_device *device = isert_conn->conn_device;
+	struct ib_mr_status mr_status;
+	int ret;
 
+	if (wr->fr_desc && wr->fr_desc->protected) {
+		ret = ib_check_mr_status(wr->fr_desc->pi_ctx->sig_mr,
+					 IB_MR_CHECK_SIG_STATUS, &mr_status);
+		if (ret) {
+			pr_err("ib_check_mr_status failed, ret %d\n", ret);
+			goto fail_mr_status;
+		}
+		if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
+			u32 block_size = se_cmd->se_dev->dev_attrib.block_size;
+
+			pr_err("PI error found type %d at offset %llx "
+			       "expected %x vs actual %x\n",
+			       mr_status.sig_err.err_type,
+			       mr_status.sig_err.sig_err_offset,
+			       mr_status.sig_err.expected,
+			       mr_status.sig_err.actual);
+			switch (mr_status.sig_err.err_type) {
+			case IB_SIG_BAD_GUARD:
+				se_cmd->pi_err = TARGET_GUARD_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_REFTAG:
+				se_cmd->pi_err = TARGET_REFTAG_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_APPTAG:
+				se_cmd->pi_err = TARGET_APPTAG_CHECK_FAILED;
+				break;
+			}
+			se_cmd->block_num =
+				mr_status.sig_err.sig_err_offset / block_size;
+			isert_pi_err_sense_buffer(se_cmd->sense_buffer,
+						  ILLEGAL_REQUEST, 0x10,
+						  (u8)se_cmd->pi_err);
+			se_cmd->scsi_status = SAM_STAT_CHECK_CONDITION;
+			se_cmd->scsi_sense_length = TRANSPORT_SENSE_BUFFER;
+			se_cmd->se_cmd_flags |= SCF_EMULATED_TASK_SENSE;
+		}
+	}
+
+fail_mr_status:
 	device->unreg_rdma_mem(isert_cmd, isert_conn);
 	isert_put_response(isert_conn->conn, cmd);
 }
@@ -1624,7 +1677,43 @@ isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
 	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_conn *isert_conn = isert_cmd->conn;
 	struct isert_device *device = isert_conn->conn_device;
+	struct ib_mr_status mr_status;
+	int ret;
 
+	if (wr->fr_desc && wr->fr_desc->protected) {
+		ret = ib_check_mr_status(wr->fr_desc->pi_ctx->sig_mr,
+					 IB_MR_CHECK_SIG_STATUS, &mr_status);
+		if (ret) {
+			pr_err("ib_check_mr_status failed, ret %d\n", ret);
+			goto fail_mr_status;
+		}
+		if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
+			u32 block_size = se_cmd->se_dev->dev_attrib.block_size;
+
+			pr_err("PI error found key %x type %d at offset %llx "
+			       "expected %x vs actual %x\n",
+			       mr_status.sig_err.key,
+			       mr_status.sig_err.err_type,
+			       mr_status.sig_err.sig_err_offset,
+			       mr_status.sig_err.expected,
+			       mr_status.sig_err.actual);
+			switch (mr_status.sig_err.err_type) {
+			case IB_SIG_BAD_GUARD:
+				se_cmd->pi_err = TARGET_GUARD_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_REFTAG:
+				se_cmd->pi_err = TARGET_REFTAG_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_APPTAG:
+				se_cmd->pi_err = TARGET_APPTAG_CHECK_FAILED;
+				break;
+			}
+			se_cmd->block_num =
+				mr_status.sig_err.sig_err_offset / block_size;
+		}
+	}
+
+fail_mr_status:
 	iscsit_stop_dataout_timer(cmd);
 	device->unreg_rdma_mem(isert_cmd, isert_conn);
 	cmd->write_data_done = wr->cur_rdma_length;
@@ -1738,6 +1827,7 @@ __isert_send_completion(struct iser_tx_desc *tx_desc,
 		pr_debug("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
 		atomic_dec(&isert_conn->post_send_buf_count);
 		isert_completion_rdma_write(tx_desc, isert_cmd);
+		break;
 	case ISER_IB_RDMA_READ:
 		pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n");
 
@@ -2324,6 +2414,128 @@ isert_fast_reg_mr(struct isert_conn *isert_conn, struct ib_mr *mr,
 }
 
 static int
+isert_reg_sig_mr(struct isert_conn *isert_conn, struct se_cmd *se_cmd,
+		 struct pi_context *pi_ctx, struct ib_sge *data_sge,
+		 struct ib_sge *prot_sge, struct ib_sge *sig_sge)
+{
+	struct ib_send_wr sig_wr, inv_wr;
+	struct ib_send_wr *bad_wr, *wr = NULL;
+	struct ib_sig_attrs sig_attrs = {0};
+	int ret;
+	u32 key;
+
+	if (!pi_ctx->sig_key_valid) {
+		memset(&inv_wr, 0, sizeof(inv_wr));
+		inv_wr.opcode = IB_WR_LOCAL_INV;
+		inv_wr.ex.invalidate_rkey = pi_ctx->sig_mr->rkey;
+		wr = &inv_wr;
+		/* Bump the key */
+		key = (u8)(pi_ctx->sig_mr->rkey & 0x000000FF);
+		ib_update_fast_reg_key(pi_ctx->sig_mr, ++key);
+	}
+
+	memset(&sig_wr, 0, sizeof(sig_wr));
+	sig_wr.opcode = IB_WR_REG_SIG_MR;
+	sig_wr.sg_list = data_sge;
+	sig_wr.num_sge = 1;
+	sig_wr.wr.sig_handover.access_flags = IB_ACCESS_LOCAL_WRITE;
+	sig_wr.wr.sig_handover.sig_attrs = &sig_attrs;
+	sig_wr.wr.sig_handover.sig_mr = pi_ctx->sig_mr;
+	if (se_cmd->t_prot_sg)
+		sig_wr.wr.sig_handover.prot = prot_sge;
+	else if (se_cmd->prot_handover == PROT_INTERLEAVED)
+		sig_wr.wr.sig_handover.prot = sig_wr.sg_list;
+
+	/* Set signature attributes */
+	sig_attrs.mem.sig_type = IB_SIG_TYPE_T10_DIF;
+	sig_attrs.wire.sig_type = IB_SIG_TYPE_T10_DIF;
+	sig_attrs.mem.sig.dif.pi_interval =
+	     			se_cmd->se_dev->dev_attrib.block_size;
+    	sig_attrs.wire.sig.dif.pi_interval =
+				se_cmd->se_dev->dev_attrib.block_size;
+
+	switch (se_cmd->prot_op) {
+	case TARGET_PROT_DIN_INSERT:
+	case TARGET_PROT_DOUT_STRIP:
+		sig_attrs.mem.sig.dif.type = IB_T10DIF_NONE;
+		sig_attrs.wire.sig.dif.type = se_cmd->prot_type;
+		sig_attrs.wire.sig.dif.bg_type = IB_T10DIF_CRC;
+		sig_attrs.wire.sig.dif.bg = se_cmd->bg_seed;
+		sig_attrs.wire.sig.dif.app_tag = se_cmd->apptag_seed;
+		sig_attrs.wire.sig.dif.ref_tag = se_cmd->reftag_seed;
+		break;
+	case TARGET_PROT_DOUT_INSERT:
+	case TARGET_PROT_DIN_STRIP:
+		sig_attrs.mem.sig.dif.type = se_cmd->prot_type;
+		sig_attrs.mem.sig.dif.bg_type = se_cmd->bg_type;
+		sig_attrs.mem.sig.dif.bg = se_cmd->bg_seed;
+		sig_attrs.mem.sig.dif.app_tag = se_cmd->apptag_seed;
+		sig_attrs.mem.sig.dif.ref_tag = se_cmd->reftag_seed;
+		sig_attrs.wire.sig.dif.type = IB_T10DIF_NONE;
+		break;
+	case TARGET_PROT_DIN_PASS:
+	case TARGET_PROT_DOUT_PASS:
+		sig_attrs.mem.sig.dif.type = se_cmd->prot_type;
+		sig_attrs.mem.sig.dif.bg_type = se_cmd->bg_type;
+		sig_attrs.mem.sig.dif.bg = se_cmd->bg_seed;
+		sig_attrs.mem.sig.dif.app_tag = se_cmd->apptag_seed;
+		sig_attrs.mem.sig.dif.ref_tag = se_cmd->reftag_seed;
+		sig_attrs.wire.sig.dif.type = se_cmd->prot_type;
+		sig_attrs.wire.sig.dif.bg_type = IB_T10DIF_CRC;
+		sig_attrs.wire.sig.dif.bg = se_cmd->bg_seed;
+		sig_attrs.wire.sig.dif.app_tag = se_cmd->apptag_seed;
+		sig_attrs.wire.sig.dif.ref_tag = se_cmd->reftag_seed;
+		break;
+	default:
+		pr_err("Unsupported PI operation %d\n", se_cmd->prot_op);
+		ret = -EINVAL;
+		goto err;
+	}
+
+	switch (se_cmd->prot_type) {
+	case TARGET_DIF_TYPE0_PROT:
+		sig_attrs.check_mask = 0x0;
+		break;
+	case TARGET_DIF_TYPE1_PROT:
+		sig_attrs.check_mask = 0xcf;
+		break;
+	case TARGET_DIF_TYPE2_PROT:
+		sig_attrs.check_mask = 0xcf;
+		break;
+	case TARGET_DIF_TYPE3_PROT:
+		sig_attrs.check_mask = 0xc0;
+		break;
+	default:
+		pr_err("Unsupported protection type %d\n", se_cmd->prot_type);
+		ret = -EINVAL;
+		goto err;
+	}
+
+	if (!wr)
+		wr = &sig_wr;
+	else
+		wr->next = &sig_wr;
+
+	ret = ib_post_send(isert_conn->conn_qp, wr, &bad_wr);
+	if (ret) {
+		pr_err("fast registration failed, ret:%d\n", ret);
+		goto err;
+	}
+	pi_ctx->sig_key_valid = false;
+
+	sig_sge->lkey = pi_ctx->sig_mr->lkey;
+	sig_sge->addr = 0;
+	sig_sge->length = se_cmd->data_length;
+
+	pr_debug("sig_sge: addr: 0x%16llx  length: %u lkey: %08x\n",
+		 sig_sge->addr, sig_sge->length,
+		 sig_sge->lkey);
+
+err:
+	return ret;
+}
+
+static int
 isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	       struct isert_rdma_wr *wr)
 {
@@ -2332,37 +2544,109 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	struct isert_conn *isert_conn = (struct isert_conn *)conn->context;
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 	struct ib_send_wr *send_wr;
-	struct ib_sge data_sge;
+	struct ib_sge data_sge, prot_sge = {0}, sig_sge;
 	struct scatterlist *sg_start;
 	struct fast_reg_descriptor *fr_desc;
+	enum dma_data_direction dir;
 	u32 sg_off = 0, sg_nents;
 	u32 offset = 0, data_len, data_left, rdma_write_max;
-	int ret = 0, count;
+	int ret = 0, count, pcount;
 	unsigned long flags;
 
 	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
+		dir = DMA_TO_DEVICE;
 		data_left = se_cmd->data_length;
 	} else {
+		dir = DMA_FROM_DEVICE;
 		offset = cmd->write_data_done;
 		sg_off = offset / PAGE_SIZE;
 		data_left = se_cmd->data_length - cmd->write_data_done;
-		isert_cmd->tx_desc.isert_cmd = isert_cmd;
 	}
 
+	isert_cmd->tx_desc.isert_cmd = isert_cmd;
 	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
 	sg_nents = se_cmd->t_data_nents - sg_off;
 
-	count = ib_dma_map_sg(ib_dev, sg_start, sg_nents,
-			      (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-			      DMA_TO_DEVICE : DMA_FROM_DEVICE);
+	wr->sge = sg_start;
+	wr->num_sge = sg_nents;
+	rdma_write_max = ISCSI_ISER_SG_TABLESIZE * PAGE_SIZE;
+	data_len = min(data_left, rdma_write_max);
+	wr->cur_rdma_length = data_len;
+
+	/* dma map data sg */
+	count = ib_dma_map_sg(ib_dev, sg_start, sg_nents, dir);
 	if (unlikely(!count)) {
 		pr_err("Cmd: %p unrable to map SGs\n", isert_cmd);
 		return -EINVAL;
 	}
 	wr->sge = sg_start;
 	wr->num_sge = sg_nents;
-	pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
-		 isert_cmd, count, sg_start, sg_nents, data_left);
+	pr_debug("Mapped cmd %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
+		 isert_cmd, count, sg_start, sg_nents, data_len);
+
+	if (count == 1 && se_cmd->prot_type == TARGET_PROT_NORMAL) {
+		wr->s_ib_sge.addr = ib_sg_dma_address(ib_dev, &sg_start[0]);
+		wr->s_ib_sge.length = ib_sg_dma_len(ib_dev, &sg_start[0]);
+		wr->s_ib_sge.lkey = isert_conn->conn_mr->lkey;
+	} else {
+		/* pop fastreg descriptor */
+		spin_lock_irqsave(&isert_conn->conn_lock, flags);
+		fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
+					   struct fast_reg_descriptor, list);
+		list_del(&fr_desc->list);
+		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
+		wr->fr_desc = fr_desc;
+        
+		/* fast register data memory */
+		if (se_cmd->prot_handover == PROT_INTERLEAVED)
+			data_len += data_len / 512 * 8;
+		ret = isert_fast_reg_mr(isert_conn, fr_desc->data_mr,
+					fr_desc->data_frpl,
+				        &fr_desc->data_key_valid, sg_start,
+					sg_nents, offset, data_len, &data_sge);
+		if (ret)
+			goto unmap_sg;
+
+		if (se_cmd->prot_type != TARGET_PROT_NORMAL) {
+			/* Protection information */
+			struct pi_context *pi_ctx = fr_desc->pi_ctx;
+        
+			if (se_cmd->t_prot_sg) {
+				/* dma map prot sg */
+				pcount = ib_dma_map_sg(ib_dev, se_cmd->t_prot_sg,
+						       se_cmd->t_prot_nents, dir);
+				if (unlikely(!pcount)) {
+					pr_err("unable to map scatterlist %p\n",
+					       se_cmd->t_prot_sg);
+					ret = -EINVAL;
+					goto unmap_prot_sg;
+				}
+				pr_debug("Mapped cmd %p prot count: %u prot_sg: %p "
+					 "prot_sg_nents: %u prot_len: %d\n",
+					 isert_cmd, pcount, se_cmd->t_prot_sg,
+					 se_cmd->t_prot_nents, se_cmd->prot_length);
+
+				/* fast register data memory */
+				ret = isert_fast_reg_mr(isert_conn, pi_ctx->prot_mr,
+						        pi_ctx->prot_frpl, &pi_ctx->prot_key_valid,
+						        se_cmd->t_prot_sg, se_cmd->t_prot_nents,
+						        0, se_cmd->prot_length, &prot_sge);
+				if (ret)
+					goto unmap_prot_sg;
+			}
+        
+			ret = isert_reg_sig_mr(isert_conn, se_cmd, pi_ctx,
+					       &data_sge, &prot_sge, &sig_sge);
+			if (ret)
+				goto unmap_prot_sg;
+        
+			fr_desc->protected = true;
+			/* record sig_mr ib_sge to RDMA */
+			memcpy(&wr->s_ib_sge, &sig_sge, sizeof(sig_sge));
+		} else
+			/* record data ib_sge to RDMA */
+			memcpy(&wr->s_ib_sge, &data_sge, sizeof(data_sge));
+	}
 
 	wr->ib_sge = &wr->s_ib_sge;
 	wr->send_wr_num = 1;
@@ -2370,18 +2654,19 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	wr->send_wr = &wr->s_send_wr;
 
 	wr->isert_cmd = isert_cmd;
-	rdma_write_max = ISCSI_ISER_SG_TABLESIZE * PAGE_SIZE;
 
 	send_wr = &isert_cmd->rdma_wr.s_send_wr;
 	send_wr->sg_list = &wr->s_ib_sge;
 	send_wr->num_sge = 1;
+	pr_debug("RDMA addr %llx length %x lkey %04x\n",
+	       send_wr->sg_list->addr, send_wr->sg_list->length, send_wr->sg_list->lkey);
 	send_wr->wr_id = (unsigned long)&isert_cmd->tx_desc;
 	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
 		send_wr->opcode = IB_WR_RDMA_WRITE;
 		send_wr->wr.rdma.remote_addr = isert_cmd->read_va;
 		send_wr->wr.rdma.rkey = isert_cmd->read_stag;
-		send_wr->send_flags = 0;
-		send_wr->next = &isert_cmd->tx_desc.send_wr;
+		send_wr->send_flags = se_cmd->prot_type == TARGET_PROT_NORMAL ?
+				      0 : IB_SEND_SIGNALED;
 	} else {
 		send_wr->opcode = IB_WR_RDMA_READ;
 		send_wr->wr.rdma.remote_addr = isert_cmd->write_va;
@@ -2389,40 +2674,14 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		send_wr->send_flags = IB_SEND_SIGNALED;
 	}
 
-	data_len = min(data_left, rdma_write_max);
-	wr->cur_rdma_length = data_len;
-
-	/* if there is a single dma entry, dma mr is sufficient */
-	if (count == 1) {
-		wr->s_ib_sge.addr = ib_sg_dma_address(ib_dev, &sg_start[0]);
-		wr->s_ib_sge.length = ib_sg_dma_len(ib_dev, &sg_start[0]);
-		wr->s_ib_sge.lkey = isert_conn->conn_mr->lkey;
-		wr->fr_desc = NULL;
-	} else {
-		spin_lock_irqsave(&isert_conn->conn_lock, flags);
-		fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
-					   struct fast_reg_descriptor, list);
-		list_del(&fr_desc->list);
-		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
-		wr->fr_desc = fr_desc;
-
-		ret = isert_fast_reg_mr(isert_conn, fr_desc->data_mr,
-					fr_desc->data_frpl,
-					&fr_desc->data_key_valid, sg_start,
-					sg_nents, offset, data_len, &data_sge);
-		if (ret) {
-			list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
-			goto unmap_sg;
-		}
-		memcpy(&wr->s_ib_sge, &data_sge, sizeof(data_sge));
-	}
-
 	return 0;
-
+unmap_prot_sg:
+	if (se_cmd->t_prot_sg)
+		ib_dma_unmap_sg(ib_dev, se_cmd->t_prot_sg,
+				se_cmd->t_prot_nents, dir);
 unmap_sg:
-	ib_dma_unmap_sg(ib_dev, sg_start, sg_nents,
-			(wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-			DMA_TO_DEVICE : DMA_FROM_DEVICE);
+	list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
+	ib_dma_unmap_sg(ib_dev, sg_start, sg_nents, dir);
 	return ret;
 }
 
@@ -2446,26 +2705,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
 		return rc;
 	}
 
-	/*
-	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
-	 */
-	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
-	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
-			     &isert_cmd->tx_desc.iscsi_header);
-	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
-	isert_init_send_wr(isert_conn, isert_cmd,
-			   &isert_cmd->tx_desc.send_wr, true);
+	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
+		/*
+		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
+		 */
+		isert_create_send_desc(isert_conn, isert_cmd,
+				       &isert_cmd->tx_desc);
+		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
+				     &isert_cmd->tx_desc.iscsi_header);
+		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
+		isert_init_send_wr(isert_conn, isert_cmd,
+				   &isert_cmd->tx_desc.send_wr, true);
+		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
+	}
 
 	atomic_inc(&isert_conn->post_send_buf_count);
-
 	rc = ib_post_send(isert_conn->conn_qp, wr->send_wr, &wr_failed);
 	if (rc) {
 		pr_warn("ib_post_send() failed for IB_WR_RDMA_WRITE\n");
 		atomic_dec(&isert_conn->post_send_buf_count);
 	}
-	pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data READ\n",
-		 isert_cmd);
 
+	if (se_cmd->prot_type == TARGET_PROT_NORMAL)
+		pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data "
+			 "READ\n", isert_cmd);
+	else
+		pr_debug("Cmd: %p posted RDMA_WRITE for iSER Data READ\n",
+			 isert_cmd);
 	return 1;
 }
 
-- 
1.7.1

[PATCH 11/11] Target/configfs: Expose iSCSI network portal group T10-PI support

[Sagi Grimberg]

User may enable T10-PI support per network portal group. any connection
established on top of it, will be required to serve protected transactions.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/iscsi/iscsi_target_configfs.c |    6 ++++++
 drivers/target/iscsi/iscsi_target_tpg.c      |   19 +++++++++++++++++++
 drivers/target/iscsi/iscsi_target_tpg.h      |    1 +
 3 files changed, 26 insertions(+), 0 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index e3318ed..8f3f585 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -1051,6 +1051,11 @@ TPG_ATTR(demo_mode_discovery, S_IRUGO | S_IWUSR);
  */
 DEF_TPG_ATTRIB(default_erl);
 TPG_ATTR(default_erl, S_IRUGO | S_IWUSR);
+/*
+ * Define iscsi_tpg_attrib_s_t10_pi
+ */
+DEF_TPG_ATTRIB(t10_pi);
+TPG_ATTR(t10_pi, S_IRUGO | S_IWUSR);
 
 static struct configfs_attribute *lio_target_tpg_attrib_attrs[] = {
 	&iscsi_tpg_attrib_authentication.attr,
@@ -1063,6 +1068,7 @@ static struct configfs_attribute *lio_target_tpg_attrib_attrs[] = {
 	&iscsi_tpg_attrib_prod_mode_write_protect.attr,
 	&iscsi_tpg_attrib_demo_mode_discovery.attr,
 	&iscsi_tpg_attrib_default_erl.attr,
+	&iscsi_tpg_attrib_t10_pi.attr,
 	NULL,
 };
 
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 80ae14c..d95a5f2 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -860,3 +860,22 @@ int iscsit_ta_default_erl(
 
 	return 0;
 }
+
+int iscsit_ta_t10_pi(
+	struct iscsi_portal_group *tpg,
+	u32 flag)
+{
+	struct iscsi_tpg_attrib *a = &tpg->tpg_attrib;
+
+	if ((flag != 0) && (flag != 1)) {
+		pr_err("Illegal value %d\n", flag);
+		return -EINVAL;
+	}
+
+	a->t10_pi = flag;
+	pr_debug("iSCSI_TPG[%hu] - T10 Protection information bit:"
+		" %s\n", tpg->tpgt, (a->t10_pi) ?
+		"ON" : "OFF");
+
+	return 0;
+}
diff --git a/drivers/target/iscsi/iscsi_target_tpg.h b/drivers/target/iscsi/iscsi_target_tpg.h
index 213c0fc..0a182f2 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.h
+++ b/drivers/target/iscsi/iscsi_target_tpg.h
@@ -39,5 +39,6 @@ extern int iscsit_ta_demo_mode_write_protect(struct iscsi_portal_group *, u32);
 extern int iscsit_ta_prod_mode_write_protect(struct iscsi_portal_group *, u32);
 extern int iscsit_ta_demo_mode_discovery(struct iscsi_portal_group *, u32);
 extern int iscsit_ta_default_erl(struct iscsi_portal_group *, u32);
+extern int iscsit_ta_t10_pi(struct iscsi_portal_group *, u32);
 
 #endif /* ISCSI_TARGET_TPG_H */
-- 
1.7.1

Re: [PATCH 06/11] IB/isert: Initialize T10-PI resources

[Or Gerlitz]

On Thu, Jan 9, 2014 at 6:40 PM, Sagi Grimberg <sagig@mellanox.com> wrote:
> @@ -557,8 +629,14 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
>                 goto out_mr;
>         }
>
> +       if (pi_support && !device->pi_capable) {
> +               pr_err("Protection information requested but not supported\n");
> +               ret = -EINVAL;
> +               goto out_mr;
> +       }
> +
>         if (device->use_fastreg) {
> -               ret = isert_conn_create_fastreg_pool(isert_conn);
> +               ret = isert_conn_create_fastreg_pool(isert_conn, pi_support);

just a nit, the pi_support bit can be looked up from the isert_conn
struct, isn't it?

>                 if (ret) {
>                         pr_err("Conn: %p failed to create fastreg pool\n",
>                                isert_conn);
> @@ -566,7 +644,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
>                 }
>         }
>
> -       ret = isert_conn_setup_qp(isert_conn, cma_id);
> +       ret = isert_conn_setup_qp(isert_conn, cma_id, pi_support);
>         if (ret)
>                 goto out_conn_dev;
>
> @@ -2193,7 +2271,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
>         pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
>                                              &fr_desc->data_frpl->page_list[0]);
>
> -       if (!fr_desc->valid) {
> +       if (!fr_desc->data_key_valid) {
>                 memset(&inv_wr, 0, sizeof(inv_wr));
>                 inv_wr.opcode = IB_WR_LOCAL_INV;
>                 inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
> @@ -2225,7 +2303,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
>                 pr_err("fast registration failed, ret:%d\n", ret);
>                 return ret;
>         }
> -       fr_desc->valid = false;
> +       fr_desc->data_key_valid = false;
>
>         ib_sge->lkey = fr_desc->data_mr->lkey;
>         ib_sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
> diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
> index 708a069..fab8b50 100644
> --- a/drivers/infiniband/ulp/isert/ib_isert.h
> +++ b/drivers/infiniband/ulp/isert/ib_isert.h
> @@ -48,11 +48,21 @@ struct iser_tx_desc {
>         struct ib_send_wr send_wr;
>  } __packed;
>
> +struct pi_context {
> +       struct ib_mr                   *prot_mr;
> +       bool                            prot_key_valid;
> +       struct ib_fast_reg_page_list   *prot_frpl;
> +       struct ib_mr                   *sig_mr;
> +       bool                            sig_key_valid;
> +};
> +
>  struct fast_reg_descriptor {
> -       struct list_head        list;
> -       struct ib_mr            *data_mr;
> -       struct ib_fast_reg_page_list    *data_frpl;
> -       bool                    valid;
> +       struct list_head                list;
> +       struct ib_mr                   *data_mr;
> +       bool                            data_key_valid;
> +       struct ib_fast_reg_page_list   *data_frpl;
> +       bool                            protected;

no need for many bools in one structure... each one needs a bit,
correct? so embed them in one variable

> +       struct pi_context              *pi_ctx;
>  };



>
>  struct isert_rdma_wr {
> @@ -140,6 +150,7 @@ struct isert_cq_desc {
>
>  struct isert_device {
>         int                     use_fastreg;
> +       bool                    pi_capable;

this one (and its such) is/are derived from the ib device
capabilities, so I would suggest to keep a copy of the caps instead of
derived bools

>         int                     cqs_used;
>         int                     refcount;
>         int                     cq_active_qps[ISERT_MAX_CQ];

Re: [PATCH 09/11] IB/isert: Accept RDMA_WRITE completions

[Or Gerlitz]

On Thu, Jan 9, 2014 at 6:40 PM, Sagi Grimberg <sagig@mellanox.com> wrote:
> In case of protected transactions, we will need to check the
> protection status of the transaction before sending SCSI response.
> So be ready for RDMA_WRITE completions. currently we don't ask
> for these completions, but for T10-PI we will.
> @@ -1721,9 +1735,9 @@ __isert_send_completion(struct iser_tx_desc *tx_desc,
>                                           isert_conn, ib_dev);
>                 break;
>         case ISER_IB_RDMA_WRITE:
> -               pr_err("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
> -               dump_stack();
> -               break;
> +               pr_debug("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
> +               atomic_dec(&isert_conn->post_send_buf_count);
> +               isert_completion_rdma_write(tx_desc, isert_cmd);

are we doing fall through here? why?
>         case ISER_IB_RDMA_READ:
>                 pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n");


>
> --
> 1.7.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/11] IB/isert: Accept RDMA_WRITE completions

[Sagi Grimberg]

On 1/11/2014 11:14 PM, Or Gerlitz wrote:
> On Thu, Jan 9, 2014 at 6:40 PM, Sagi Grimberg <sagig@mellanox.com> wrote:
>> In case of protected transactions, we will need to check the
>> protection status of the transaction before sending SCSI response.
>> So be ready for RDMA_WRITE completions. currently we don't ask
>> for these completions, but for T10-PI we will.
>> @@ -1721,9 +1735,9 @@ __isert_send_completion(struct iser_tx_desc *tx_desc,
>>                                            isert_conn, ib_dev);
>>                  break;
>>          case ISER_IB_RDMA_WRITE:
>> -               pr_err("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
>> -               dump_stack();
>> -               break;
>> +               pr_debug("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
>> +               atomic_dec(&isert_conn->post_send_buf_count);
>> +               isert_completion_rdma_write(tx_desc, isert_cmd);
> are we doing fall through here? why?

Ohhhh, somehow i missed it in squash... Will fix, Thanks!

>>          case ISER_IB_RDMA_READ:
>>                  pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n");
>

Re: [PATCH 06/11] IB/isert: Initialize T10-PI resources

[Sagi Grimberg]

On 1/11/2014 11:09 PM, Or Gerlitz wrote:
> On Thu, Jan 9, 2014 at 6:40 PM, Sagi Grimberg <sagig@mellanox.com> wrote:
>> @@ -557,8 +629,14 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
>>                  goto out_mr;
>>          }
>>
>> +       if (pi_support && !device->pi_capable) {
>> +               pr_err("Protection information requested but not supported\n");
>> +               ret = -EINVAL;
>> +               goto out_mr;
>> +       }
>> +
>>          if (device->use_fastreg) {
>> -               ret = isert_conn_create_fastreg_pool(isert_conn);
>> +               ret = isert_conn_create_fastreg_pool(isert_conn, pi_support);
> just a nit, the pi_support bit can be looked up from the isert_conn
> struct, isn't it?
>
>>                  if (ret) {
>>                          pr_err("Conn: %p failed to create fastreg pool\n",
>>                                 isert_conn);
>> @@ -566,7 +644,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
>>                  }
>>          }
>>
>> -       ret = isert_conn_setup_qp(isert_conn, cma_id);
>> +       ret = isert_conn_setup_qp(isert_conn, cma_id, pi_support);
>>          if (ret)
>>                  goto out_conn_dev;
>>
>> @@ -2193,7 +2271,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
>>          pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
>>                                               &fr_desc->data_frpl->page_list[0]);
>>
>> -       if (!fr_desc->valid) {
>> +       if (!fr_desc->data_key_valid) {
>>                  memset(&inv_wr, 0, sizeof(inv_wr));
>>                  inv_wr.opcode = IB_WR_LOCAL_INV;
>>                  inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
>> @@ -2225,7 +2303,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
>>                  pr_err("fast registration failed, ret:%d\n", ret);
>>                  return ret;
>>          }
>> -       fr_desc->valid = false;
>> +       fr_desc->data_key_valid = false;
>>
>>          ib_sge->lkey = fr_desc->data_mr->lkey;
>>          ib_sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
>> diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
>> index 708a069..fab8b50 100644
>> --- a/drivers/infiniband/ulp/isert/ib_isert.h
>> +++ b/drivers/infiniband/ulp/isert/ib_isert.h
>> @@ -48,11 +48,21 @@ struct iser_tx_desc {
>>          struct ib_send_wr send_wr;
>>   } __packed;
>>
>> +struct pi_context {
>> +       struct ib_mr                   *prot_mr;
>> +       bool                            prot_key_valid;
>> +       struct ib_fast_reg_page_list   *prot_frpl;
>> +       struct ib_mr                   *sig_mr;
>> +       bool                            sig_key_valid;
>> +};
>> +
>>   struct fast_reg_descriptor {
>> -       struct list_head        list;
>> -       struct ib_mr            *data_mr;
>> -       struct ib_fast_reg_page_list    *data_frpl;
>> -       bool                    valid;
>> +       struct list_head                list;
>> +       struct ib_mr                   *data_mr;
>> +       bool                            data_key_valid;
>> +       struct ib_fast_reg_page_list   *data_frpl;
>> +       bool                            protected;
> no need for many bools in one structure... each one needs a bit,
> correct? so embed them in one variable

I figured it will be more explicit this way.
protected boolean indicates if we should check the data-integrity 
status, and the other 3 indicates if the relevant MR is valid (no need 
to execute local invalidation).
Do you think I should compact it somehow? usually xxx_valid booleans 
will align together although not always.

>
>> +       struct pi_context              *pi_ctx;
>>   };
>
>
>>   struct isert_rdma_wr {
>> @@ -140,6 +150,7 @@ struct isert_cq_desc {
>>
>>   struct isert_device {
>>          int                     use_fastreg;
>> +       bool                    pi_capable;
> this one (and its such) is/are derived from the ib device
> capabilities, so I would suggest to keep a copy of the caps instead of
> derived bools

Yes, I'll keep the device capabilities instead.

>
>>          int                     cqs_used;
>>          int                     refcount;
>>          int                     cq_active_qps[ISERT_MAX_CQ];
> --
> To unsubscribe from this list: send the line "unsubscribe target-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 06/11] IB/isert: Initialize T10-PI resources

[Or Gerlitz]

On 12/01/2014 14:41, Sagi Grimberg wrote:
>>> --- a/drivers/infiniband/ulp/isert/ib_isert.h
>>> +++ b/drivers/infiniband/ulp/isert/ib_isert.h
>>> @@ -48,11 +48,21 @@ struct iser_tx_desc {
>>>          struct ib_send_wr send_wr;
>>>   } __packed;
>>>
>>> +struct pi_context {
>>> +       struct ib_mr                   *prot_mr;
>>> +       bool prot_key_valid;
>>> +       struct ib_fast_reg_page_list   *prot_frpl;
>>> +       struct ib_mr                   *sig_mr;
>>> +       bool sig_key_valid;
>>> +};
>>> +
>>>   struct fast_reg_descriptor {
>>> -       struct list_head        list;
>>> -       struct ib_mr            *data_mr;
>>> -       struct ib_fast_reg_page_list    *data_frpl;
>>> -       bool                    valid;
>>> +       struct list_head                list;
>>> +       struct ib_mr                   *data_mr;
>>> +       bool data_key_valid;
>>> +       struct ib_fast_reg_page_list   *data_frpl;
>>> +       bool                            protected;
>> no need for many bools in one structure... each one needs a bit,
>> correct? so embed them in one variable
>
> I figured it will be more explicit this way. protected boolean 
> indicates if we should check the data-integrity status, and the other 
> 3 indicates if the relevant MR is valid (no need to execute local 
> invalidation). Do you think I should compact it somehow? usually 
> xxx_valid booleans will align together although not always. 

I didn't note there are so many booleans there... sure, my personal 
preference is compaction, e.g have one field names flags and multiple 
bit locations marking the different flags e.g

FAST_REG_DESC_VALID
FAST_REG_DESC_DATA_KEY_VALID
FAST_REG_DESC_PROTECTED
etc

Re: [PATCH 02/11] IB/isert: seperate connection protection domains and dma MRs

[Nicholas A. Bellinger]

Hi Sagi,

On Thu, 2014-01-09 at 18:40 +0200, Sagi Grimberg wrote:
> It is more correct to seperate connections protection domains
> and dma_mr handles. protection information support requires to
> do so.
> 
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
>  drivers/infiniband/ulp/isert/ib_isert.c |   46 ++++++++++++++++---------------
>  drivers/infiniband/ulp/isert/ib_isert.h |    2 -
>  2 files changed, 24 insertions(+), 24 deletions(-)
> 

Applied to for-next, given this patch does not depend on the proposed
VERBS protection changes.

--nab 

Re: [PATCH 03/11] IB/isert: Avoid frwr notation, user fastreg

[Nicholas A. Bellinger]

On Thu, 2014-01-09 at 18:40 +0200, Sagi Grimberg wrote:
> Use fast registration lingo. fast registration will
> also incorporate signature/DIF registration.
> 
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
>  drivers/infiniband/ulp/isert/ib_isert.c |   84 ++++++++++++++++---------------
>  drivers/infiniband/ulp/isert/ib_isert.h |    8 ++--
>  2 files changed, 47 insertions(+), 45 deletions(-)

Applied to for-next.

Thanks Sagi!

--nab

Re: [PATCH 04/11] IB/isert: Move fastreg descriptor creation to a function

[Nicholas A. Bellinger]

On Thu, 2014-01-09 at 18:40 +0200, Sagi Grimberg wrote:
> This routine may be called both by fast registration
> descriptors for data and for integrity buffers.
> 
> This patch does not change any functionality.
> 
> Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  drivers/infiniband/ulp/isert/ib_isert.c |   52 +++++++++++++++++++------------
>  1 files changed, 32 insertions(+), 20 deletions(-)

Also applied to for-next.

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/11] IB/isert: pass scatterlist instead of cmd to fast_reg_mr routine

[Nicholas A. Bellinger]

On Thu, 2014-01-09 at 18:40 +0200, Sagi Grimberg wrote:
> This routine may help for protection registration as well.
> This patch does not change any functionality.
> 
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
>  drivers/infiniband/ulp/isert/ib_isert.c |   28 ++++++++++++----------------
>  1 files changed, 12 insertions(+), 16 deletions(-)

Also applied to for-next.

--nab