[PATCH v2 00/12] T10-DIF Initial support for iSER target

[PATCH v2 00/12] T10-DIF Initial support for iSER target

[Sagi Grimberg]

Hey Nic,

I addressed your comments in the this set. I'll try to send the
initiator code for review in the next couple of days.

This patchset introduces target side T10-PI offload support over
RDMA. Currently the implementation is for iSER transport but can
be easily extended to SRP (or FCoE in the future).

Should mention that this patchset depends on RDMA signature
verbs making it for inclusion which will hopefully happen in
near future.

This code was tested against iSER legacy initiator, and also
initiator that uses T10-PI offload as well. I'll clean up the
initiator code in the following days and submit that as well.

This code works under medium loads using backstores:
- FileIO with DIF emulation.
- RD with DIF emulation.
- iBLOCK (scsi_debug with DIF support).

Chnages from v1:
- Rebased rdma_dif (3.14-rc2)
- Target core:
  - Pass zero_flag=true to alloc_sgl for protection buffers
  - Removed Unneeded inline function rwprotect
  - Pass is_write bool to sbc_set_prot_op_checks
  - Send failure response for absence of protection buffers
    when needed.
- iSER:
  - Removed unneeded assignments in isert_set_sig_attrs

Chnages from v0:
- Rebased from for-next
- Target core:
  - Don't minor fixes for check_prot - to support transports
    that doesn't use submit_map_sgls.
  - file  format - use escape values.
  - Removed redundant prot_handover.
  - Added protection checks and operation set.
- iSER:
  - Added preperation routines for mapping/unmapping buffers
    to ease the amount of code in isert_reg_rdma.
  - Fixed print of DIF error (sector instead of offset).
  - Fix RDMA length for protection on wire domain.
  - Refactored reg_sig_mr to use helper routines.

Sagi Grimberg (12):
  Target/transport: Allocate protection sg if needed
  Target/sbc: Set protection operation and relevant checks
  Target/file: place escape values for protection information format
  Target/sbc: don't return from sbc_check for non prot_sg
  Target/core: Remove prot_handover use for now
  Target/iscsi: Add T10-PI indication for iscsi_portal_group
  Target/configfs: Expose iSCSI network portal group T10-PI support
  Target/sbc: add debug print
  IB/isert: Introduce isert_map/unmap_data_buf
  IB/isert: Initialize T10-PI resources
  IB/isert: Accept RDMA_WRITE completions
  IB/isert: Support T10-PI protected transactions

 drivers/infiniband/ulp/isert/ib_isert.c      |  668 ++++++++++++++++++++------
 drivers/infiniband/ulp/isert/ib_isert.h      |   38 ++-
 drivers/target/iscsi/iscsi_target_configfs.c |    6 +
 drivers/target/iscsi/iscsi_target_core.h     |    5 +-
 drivers/target/iscsi/iscsi_target_tpg.c      |   21 +
 drivers/target/iscsi/iscsi_target_tpg.h      |    1 +
 drivers/target/target_core_file.c            |   40 +--
 drivers/target/target_core_sbc.c             |   91 +++-
 drivers/target/target_core_transport.c       |   33 ++-
 include/target/target_core_base.h            |   13 +-
 10 files changed, 683 insertions(+), 233 deletions(-)

[PATCH v2 01/12] Target/transport: Allocate protection sg if needed

[Sagi Grimberg]

In case protection information is involved, allocate
protection SG-list for transport.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/target_core_transport.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 24b4f65..0f9e1ea 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -2036,6 +2036,10 @@ static inline void transport_free_pages(struct se_cmd *cmd)
 	transport_free_sgl(cmd->t_bidi_data_sg, cmd->t_bidi_data_nents);
 	cmd->t_bidi_data_sg = NULL;
 	cmd->t_bidi_data_nents = 0;
+
+	transport_free_sgl(cmd->t_prot_sg, cmd->t_prot_nents);
+	cmd->t_prot_sg = NULL;
+	cmd->t_prot_nents = 0;
 }
 
 /**
@@ -2199,6 +2203,14 @@ transport_generic_new_cmd(struct se_cmd *cmd)
 				return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 		}
 
+		if (cmd->prot_type != TARGET_PROT_NORMAL) {
+			ret = target_alloc_sgl(&cmd->t_prot_sg,
+					       &cmd->t_prot_nents,
+					       cmd->prot_length, true);
+			if (ret < 0)
+				return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
+		}
+
 		ret = target_alloc_sgl(&cmd->t_data_sg, &cmd->t_data_nents,
 				       cmd->data_length, zero_flag);
 		if (ret < 0)
-- 
1.7.1

[PATCH v2 02/12] Target/sbc: Set protection operation and relevant checks

[Sagi Grimberg]

SBC-3 mandates the protection checks that must be
performed in the rdprotect/wrprotect field. Use them.
According to backstore device pi_attributes and
cdb rdprotect/wrprotect field.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/target_core_sbc.c  |   86 +++++++++++++++++++++++++++++++------
 include/target/target_core_base.h |    7 +++
 2 files changed, 79 insertions(+), 14 deletions(-)

diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index a448944..6939947 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -569,30 +569,84 @@ sbc_compare_and_write(struct se_cmd *cmd)
 	return TCM_NO_SENSE;
 }
 
+static int
+sbc_set_prot_op_checks(u8 protect, enum target_prot_type prot_type,
+		       bool is_write, struct se_cmd *cmd)
+{
+	if (is_write) {
+		cmd->prot_op = protect ? TARGET_PROT_DOUT_PASS :
+					 TARGET_PROT_DOUT_INSERT;
+		switch (protect) {
+		case 0x0:
+		case 0x3:
+			cmd->prot_checks = 0;
+		case 0x1:
+		case 0x5:
+			cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+			if (prot_type == TARGET_DIF_TYPE1_PROT)
+				cmd->prot_checks |= TARGET_DIF_CHECK_REFTAG;
+			break;
+		case 0x2:
+			if (prot_type == TARGET_DIF_TYPE1_PROT)
+				cmd->prot_checks = TARGET_DIF_CHECK_REFTAG;
+			break;
+		case 0x4:
+			cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+			break;
+		default:
+			pr_err("Unsupported protect field %d\n", protect);
+			return -EINVAL;
+		}
+	} else {
+		cmd->prot_op = protect ? TARGET_PROT_DIN_PASS :
+					 TARGET_PROT_DIN_STRIP;
+		switch (protect) {
+		case 0x0:
+		case 0x1:
+		case 0x5:
+			cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+			if (prot_type == TARGET_DIF_TYPE1_PROT)
+				cmd->prot_checks |= TARGET_DIF_CHECK_REFTAG;
+			break;
+		case 0x2:
+			if (prot_type == TARGET_DIF_TYPE1_PROT)
+				cmd->prot_checks = TARGET_DIF_CHECK_REFTAG;
+			break;
+		case 0x3:
+			cmd->prot_checks = 0;
+			break;
+		case 0x4:
+			cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+			break;
+		default:
+			pr_err("Unsupported protect field %d\n", protect);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
 static bool
 sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
-	       u32 sectors)
+	       u32 sectors, bool is_write)
 {
+	u8 protect = cdb[1] >> 5;
+
 	if (!cmd->t_prot_sg || !cmd->t_prot_nents)
 		return true;
 
 	switch (dev->dev_attrib.pi_prot_type) {
 	case TARGET_DIF_TYPE3_PROT:
-		if (!(cdb[1] & 0xe0))
-			return true;
-
 		cmd->reftag_seed = 0xffffffff;
 		break;
 	case TARGET_DIF_TYPE2_PROT:
-		if (cdb[1] & 0xe0)
+		if (protect)
 			return false;
 
 		cmd->reftag_seed = cmd->t_task_lba;
 		break;
 	case TARGET_DIF_TYPE1_PROT:
-		if (!(cdb[1] & 0xe0))
-			return true;
-
 		cmd->reftag_seed = cmd->t_task_lba;
 		break;
 	case TARGET_DIF_TYPE0_PROT:
@@ -600,6 +654,10 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
 		return true;
 	}
 
+	if (sbc_set_prot_op_checks(protect, dev->dev_attrib.pi_prot_type,
+				   is_write, cmd))
+		return false;
+
 	cmd->prot_type = dev->dev_attrib.pi_prot_type;
 	cmd->prot_length = dev->prot_length * sectors;
 	cmd->prot_handover = PROT_SEPERATED;
@@ -628,7 +686,7 @@ sbc_parse_cdb(struct se_cmd *cmd, struct sbc_ops *ops)
 		sectors = transport_get_sectors_10(cdb);
 		cmd->t_task_lba = transport_lba_32(cdb);
 
-		if (!sbc_check_prot(dev, cmd, cdb, sectors))
+		if (!sbc_check_prot(dev, cmd, cdb, sectors, false))
 			return TCM_UNSUPPORTED_SCSI_OPCODE;
 
 		cmd->se_cmd_flags |= SCF_SCSI_DATA_CDB;
@@ -639,7 +697,7 @@ sbc_parse_cdb(struct se_cmd *cmd, struct sbc_ops *ops)
 		sectors = transport_get_sectors_12(cdb);
 		cmd->t_task_lba = transport_lba_32(cdb);
 
-		if (!sbc_check_prot(dev, cmd, cdb, sectors))
+		if (!sbc_check_prot(dev, cmd, cdb, sectors, false))
 			return TCM_UNSUPPORTED_SCSI_OPCODE;
 
 		cmd->se_cmd_flags |= SCF_SCSI_DATA_CDB;
@@ -650,7 +708,7 @@ sbc_parse_cdb(struct se_cmd *cmd, struct sbc_ops *ops)
 		sectors = transport_get_sectors_16(cdb);
 		cmd->t_task_lba = transport_lba_64(cdb);
 
-		if (!sbc_check_prot(dev, cmd, cdb, sectors))
+		if (!sbc_check_prot(dev, cmd, cdb, sectors, false))
 			return TCM_UNSUPPORTED_SCSI_OPCODE;
 
 		cmd->se_cmd_flags |= SCF_SCSI_DATA_CDB;
@@ -669,7 +727,7 @@ sbc_parse_cdb(struct se_cmd *cmd, struct sbc_ops *ops)
 		sectors = transport_get_sectors_10(cdb);
 		cmd->t_task_lba = transport_lba_32(cdb);
 
-		if (!sbc_check_prot(dev, cmd, cdb, sectors))
+		if (!sbc_check_prot(dev, cmd, cdb, sectors, true))
 			return TCM_UNSUPPORTED_SCSI_OPCODE;
 
 		if (cdb[1] & 0x8)
@@ -682,7 +740,7 @@ sbc_parse_cdb(struct se_cmd *cmd, struct sbc_ops *ops)
 		sectors = transport_get_sectors_12(cdb);
 		cmd->t_task_lba = transport_lba_32(cdb);
 
-		if (!sbc_check_prot(dev, cmd, cdb, sectors))
+		if (!sbc_check_prot(dev, cmd, cdb, sectors, true))
 			return TCM_UNSUPPORTED_SCSI_OPCODE;
 
 		if (cdb[1] & 0x8)
@@ -695,7 +753,7 @@ sbc_parse_cdb(struct se_cmd *cmd, struct sbc_ops *ops)
 		sectors = transport_get_sectors_16(cdb);
 		cmd->t_task_lba = transport_lba_64(cdb);
 
-		if (!sbc_check_prot(dev, cmd, cdb, sectors))
+		if (!sbc_check_prot(dev, cmd, cdb, sectors, true))
 			return TCM_UNSUPPORTED_SCSI_OPCODE;
 
 		if (cdb[1] & 0x8)
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 1772fad..5ae9249 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -463,6 +463,12 @@ enum target_prot_type {
 	TARGET_DIF_TYPE3_PROT,
 };
 
+enum target_core_dif_check {
+	TARGET_DIF_CHECK_GUARD  = 0x1 << 0,
+	TARGET_DIF_CHECK_APPTAG = 0x1 << 1,
+	TARGET_DIF_CHECK_REFTAG = 0x1 << 2,
+};
+
 struct se_dif_v1_tuple {
 	__be16			guard_tag;
 	__be16			app_tag;
@@ -556,6 +562,7 @@ struct se_cmd {
 	/* DIF related members */
 	enum target_prot_op	prot_op;
 	enum target_prot_type	prot_type;
+	u8			prot_checks;
 	u32			prot_length;
 	u32			reftag_seed;
 	struct scatterlist	*t_prot_sg;
-- 
1.7.1

[PATCH v2 03/12] Target/file: place escape values for protection information format

[Sagi Grimberg]

No need to actually compute protection information when formatting

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/target_core_file.c |   40 +------------------------------------
 1 files changed, 1 insertions(+), 39 deletions(-)

diff --git a/drivers/target/target_core_file.c b/drivers/target/target_core_file.c
index cf991a9..7d6cdda 100644
--- a/drivers/target/target_core_file.c
+++ b/drivers/target/target_core_file.c
@@ -854,25 +854,6 @@ static int fd_init_prot(struct se_device *dev)
 	return 0;
 }
 
-static void fd_init_format_buf(struct se_device *dev, unsigned char *buf,
-			       u32 unit_size, u32 *ref_tag, u16 app_tag,
-			       bool inc_reftag)
-{
-	unsigned char *p = buf;
-	int i;
-
-	for (i = 0; i < unit_size; i += dev->prot_length) {
-		*((u16 *)&p[0]) = 0xffff;
-		*((__be16 *)&p[2]) = cpu_to_be16(app_tag);
-		*((__be32 *)&p[4]) = cpu_to_be32(*ref_tag);
-
-		if (inc_reftag)
-			(*ref_tag)++;
-
-		p += dev->prot_length;
-	}
-}
-
 static int fd_format_prot(struct se_device *dev)
 {
 	struct fd_dev *fd_dev = FD_DEV(dev);
@@ -880,10 +861,8 @@ static int fd_format_prot(struct se_device *dev)
 	sector_t prot_length, prot;
 	unsigned char *buf;
 	loff_t pos = 0;
-	u32 ref_tag = 0;
 	int unit_size = FDBD_FORMAT_UNIT_SIZE * dev->dev_attrib.block_size;
 	int rc, ret = 0, size, len;
-	bool inc_reftag = false;
 
 	if (!dev->dev_attrib.pi_prot_type) {
 		pr_err("Unable to format_prot while pi_prot_type == 0\n");
@@ -894,37 +873,20 @@ static int fd_format_prot(struct se_device *dev)
 		return -ENODEV;
 	}
 
-	switch (dev->dev_attrib.pi_prot_type) {
-	case TARGET_DIF_TYPE3_PROT:
-		ref_tag = 0xffffffff;
-		break;
-	case TARGET_DIF_TYPE2_PROT:
-	case TARGET_DIF_TYPE1_PROT:
-		inc_reftag = true;
-		break;
-	default:
-		break;
-	}
-
 	buf = vzalloc(unit_size);
 	if (!buf) {
 		pr_err("Unable to allocate FILEIO prot buf\n");
 		return -ENOMEM;
 	}
-
 	prot_length = (dev->transport->get_blocks(dev) + 1) * dev->prot_length;
 	size = prot_length;
 
 	pr_debug("Using FILEIO prot_length: %llu\n",
 		 (unsigned long long)prot_length);
 
+	memset(buf, 0xff, unit_size);
 	for (prot = 0; prot < prot_length; prot += unit_size) {
-
-		fd_init_format_buf(dev, buf, unit_size, &ref_tag, 0xffff,
-				   inc_reftag);
-
 		len = min(unit_size, size);
-
 		rc = kernel_write(prot_fd, buf, len, pos);
 		if (rc != len) {
 			pr_err("vfs_write to prot file failed: %d\n", rc);
-- 
1.7.1

[PATCH v2 04/12] Target/sbc: don't return from sbc_check for non prot_sg

[Sagi Grimberg]

For transports which use generic new command these
buffers have yet to be allocated. Instead check afterwards
if command required prot buffers but none are provided, In
this case send CHECK_CONDITION response.

Also this way, target may support protection information
against legacy initiators (writes are inserted and reads
are stripped).

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/target_core_sbc.c       |    3 ---
 drivers/target/target_core_transport.c |   21 ++++++++++++++++++---
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index 6939947..312f522 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -633,9 +633,6 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
 {
 	u8 protect = cdb[1] >> 5;
 
-	if (!cmd->t_prot_sg || !cmd->t_prot_nents)
-		return true;
-
 	switch (dev->dev_attrib.pi_prot_type) {
 	case TARGET_DIF_TYPE3_PROT:
 		cmd->reftag_seed = 0xffffffff;
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 0f9e1ea..a45d628 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -1365,6 +1365,13 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess
 		target_put_sess_cmd(se_sess, se_cmd);
 		return 0;
 	}
+
+	rc = target_setup_cmd_from_cdb(se_cmd, cdb);
+	if (rc != 0) {
+		transport_generic_request_failure(se_cmd, rc);
+		return 0;
+	}
+
 	/*
 	 * Save pointers for SGLs containing protection information,
 	 * if present.
@@ -1374,11 +1381,19 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess
 		se_cmd->t_prot_nents = sgl_prot_count;
 	}
 
-	rc = target_setup_cmd_from_cdb(se_cmd, cdb);
-	if (rc != 0) {
-		transport_generic_request_failure(se_cmd, rc);
+	/*
+	 * Fail if protection operation requiers protection
+	 * information buffers but None are provided!
+	 */
+	if ((!se_cmd->t_prot_sg || !se_cmd->t_prot_nents) &&
+	    (se_cmd->prot_op != TARGET_PROT_NORMAL)) {
+		pr_err("ERROR: protection information was requested but "
+		       "protection buffers weren't provided.\n");
+		transport_generic_request_failure(se_cmd,
+						  TCM_INVALID_CDB_FIELD);
 		return 0;
 	}
+
 	/*
 	 * When a non zero sgl_count has been passed perform SGL passthrough
 	 * mapping for pre-allocated fabric memory instead of having target
-- 
1.7.1

[PATCH v2 05/12] Target/core: Remove prot_handover use for now

[Sagi Grimberg]

This is not going to be supported soon - so drop it.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/target_core_sbc.c  |    1 -
 include/target/target_core_base.h |    6 ------
 2 files changed, 0 insertions(+), 7 deletions(-)

diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index 312f522..33f3163 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -657,7 +657,6 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
 
 	cmd->prot_type = dev->dev_attrib.pi_prot_type;
 	cmd->prot_length = dev->prot_length * sectors;
-	cmd->prot_handover = PROT_SEPERATED;
 
 	return true;
 }
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 5ae9249..344fd05 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -451,11 +451,6 @@ enum target_prot_op {
 	TARGET_PROT_DOUT_PASS,
 };
 
-enum target_prot_ho {
-	PROT_SEPERATED,
-	PROT_INTERLEAVED,
-};
-
 enum target_prot_type {
 	TARGET_DIF_TYPE0_PROT,
 	TARGET_DIF_TYPE1_PROT,
@@ -567,7 +562,6 @@ struct se_cmd {
 	u32			reftag_seed;
 	struct scatterlist	*t_prot_sg;
 	unsigned int		t_prot_nents;
-	enum target_prot_ho	prot_handover;
 	sense_reason_t		pi_err;
 	sector_t		bad_sector;
 };
-- 
1.7.1

[PATCH v2 06/12] Target/iscsi: Add T10-PI indication for iscsi_portal_group

[Sagi Grimberg]

In case an iscsi portal group will be defined as t10_pi enabled,
all connections on top of it will support protected transactions.

T10-PI support may require extra reource allocation and maintenance by
the transport layer, so we don't want to apply them on non-t10_pi network
portals. This is a hook for the iscsi target layer to signal the transport
at connection establishment that this connection will carry protected
transactions.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/iscsi/iscsi_target_core.h |    5 ++++-
 drivers/target/iscsi/iscsi_target_tpg.c  |    2 ++
 2 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_core.h b/drivers/target/iscsi/iscsi_target_core.h
index 48f7b3b..886d74d 100644
--- a/drivers/target/iscsi/iscsi_target_core.h
+++ b/drivers/target/iscsi/iscsi_target_core.h
@@ -58,7 +58,8 @@
 #define TA_DEMO_MODE_DISCOVERY		1
 #define TA_DEFAULT_ERL			0
 #define TA_CACHE_CORE_NPS		0
-
+/* T10 protection information disabled by default */
+#define TA_DEFAULT_T10_PI		0
 
 #define ISCSI_IOV_DATA_BUFFER		5
 
@@ -765,6 +766,7 @@ struct iscsi_tpg_attrib {
 	u32			prod_mode_write_protect;
 	u32			demo_mode_discovery;
 	u32			default_erl;
+	u8			t10_pi;
 	struct iscsi_portal_group *tpg;
 };
 
@@ -787,6 +789,7 @@ struct iscsi_np {
 	void			*np_context;
 	struct iscsit_transport *np_transport;
 	struct list_head	np_list;
+	struct iscsi_tpg_np	*tpg_np;
 } ____cacheline_aligned;
 
 struct iscsi_tpg_np {
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 3976183..80ae14c 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -225,6 +225,7 @@ static void iscsit_set_default_tpg_attribs(struct iscsi_portal_group *tpg)
 	a->prod_mode_write_protect = TA_PROD_MODE_WRITE_PROTECT;
 	a->demo_mode_discovery = TA_DEMO_MODE_DISCOVERY;
 	a->default_erl = TA_DEFAULT_ERL;
+	a->t10_pi = TA_DEFAULT_T10_PI;
 }
 
 int iscsit_tpg_add_portal_group(struct iscsi_tiqn *tiqn, struct iscsi_portal_group *tpg)
@@ -500,6 +501,7 @@ struct iscsi_tpg_np *iscsit_tpg_add_network_portal(
 	init_completion(&tpg_np->tpg_np_comp);
 	kref_init(&tpg_np->tpg_np_kref);
 	tpg_np->tpg_np		= np;
+	np->tpg_np		= tpg_np;
 	tpg_np->tpg		= tpg;
 
 	spin_lock(&tpg->tpg_np_lock);
-- 
1.7.1

[PATCH v2 07/12] Target/configfs: Expose iSCSI network portal group T10-PI support

[Sagi Grimberg]

User may enable T10-PI support per network portal group. any connection
established on top of it, will be required to serve protected transactions.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
 drivers/target/iscsi/iscsi_target_configfs.c |    6 ++++++
 drivers/target/iscsi/iscsi_target_tpg.c      |   19 +++++++++++++++++++
 drivers/target/iscsi/iscsi_target_tpg.h      |    1 +
 3 files changed, 26 insertions(+), 0 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index 1c0088f..fe35dcb 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -1052,6 +1052,11 @@ TPG_ATTR(demo_mode_discovery, S_IRUGO | S_IWUSR);
  */
 DEF_TPG_ATTRIB(default_erl);
 TPG_ATTR(default_erl, S_IRUGO | S_IWUSR);
+/*
+ * Define iscsi_tpg_attrib_s_t10_pi
+ */
+DEF_TPG_ATTRIB(t10_pi);
+TPG_ATTR(t10_pi, S_IRUGO | S_IWUSR);
 
 static struct configfs_attribute *lio_target_tpg_attrib_attrs[] = {
 	&iscsi_tpg_attrib_authentication.attr,
@@ -1064,6 +1069,7 @@ static struct configfs_attribute *lio_target_tpg_attrib_attrs[] = {
 	&iscsi_tpg_attrib_prod_mode_write_protect.attr,
 	&iscsi_tpg_attrib_demo_mode_discovery.attr,
 	&iscsi_tpg_attrib_default_erl.attr,
+	&iscsi_tpg_attrib_t10_pi.attr,
 	NULL,
 };
 
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 80ae14c..d95a5f2 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -860,3 +860,22 @@ int iscsit_ta_default_erl(
 
 	return 0;
 }
+
+int iscsit_ta_t10_pi(
+	struct iscsi_portal_group *tpg,
+	u32 flag)
+{
+	struct iscsi_tpg_attrib *a = &tpg->tpg_attrib;
+
+	if ((flag != 0) && (flag != 1)) {
+		pr_err("Illegal value %d\n", flag);
+		return -EINVAL;
+	}
+
+	a->t10_pi = flag;
+	pr_debug("iSCSI_TPG[%hu] - T10 Protection information bit:"
+		" %s\n", tpg->tpgt, (a->t10_pi) ?
+		"ON" : "OFF");
+
+	return 0;
+}
diff --git a/drivers/target/iscsi/iscsi_target_tpg.h b/drivers/target/iscsi/iscsi_target_tpg.h
index 213c0fc..0a182f2 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.h
+++ b/drivers/target/iscsi/iscsi_target_tpg.h
@@ -39,5 +39,6 @@ extern int iscsit_ta_demo_mode_write_protect(struct iscsi_portal_group *, u32);
 extern int iscsit_ta_prod_mode_write_protect(struct iscsi_portal_group *, u32);
 extern int iscsit_ta_demo_mode_discovery(struct iscsi_portal_group *, u32);
 extern int iscsit_ta_default_erl(struct iscsi_portal_group *, u32);
+extern int iscsit_ta_t10_pi(struct iscsi_portal_group *, u32);
 
 #endif /* ISCSI_TARGET_TPG_H */
-- 
1.7.1

[PATCH v2 08/12] Target/sbc: add debug print

[Sagi Grimberg]

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/target/target_core_sbc.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index 33f3163..59fdc9f 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -657,6 +657,9 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
 
 	cmd->prot_type = dev->dev_attrib.pi_prot_type;
 	cmd->prot_length = dev->prot_length * sectors;
+	pr_debug("%s: prot_type=%d, prot_length=%d prot_op=%d prot_checks=%d\n",
+		 __func__, cmd->prot_type, cmd->prot_length,
+		 cmd->prot_op, cmd->prot_checks);
 
 	return true;
 }
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 09/12] IB/isert: Introduce isert_map/unmap_data_buf

[Sagi Grimberg]

export map/unmap data buffer to a routine that may
be used in various places in the code and keep the
mapping data in a designated descriptor. Also, let
isert_fast_reg_mr to decide weather to use global
MR or do fast registration.

This commit does not change any functionality.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |  238 +++++++++++++++----------------
 drivers/infiniband/ulp/isert/ib_isert.h |   14 ++-
 2 files changed, 127 insertions(+), 125 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index d18d08a..cbf0248 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1392,19 +1392,60 @@ isert_rx_completion(struct iser_rx_desc *desc, struct isert_conn *isert_conn,
 	}
 }
 
+static int
+isert_map_data_buf(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
+		   struct scatterlist *sg, u32 nents, u32 length, u32 offset,
+		   enum iser_ib_op_code op, struct isert_data_buf *data)
+{
+	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+
+	data->dma_dir = op == ISER_IB_RDMA_WRITE ?
+			      DMA_TO_DEVICE : DMA_FROM_DEVICE;
+
+	data->len = length - offset;
+	data->offset = offset;
+	data->sg_off = data->offset / PAGE_SIZE;
+
+	data->sg = &sg[data->sg_off];
+	data->nents = min_t(unsigned int, nents - data->sg_off,
+					  ISCSI_ISER_SG_TABLESIZE);
+	data->len = min_t(unsigned int, data->len, ISCSI_ISER_SG_TABLESIZE *
+					PAGE_SIZE);
+
+	data->dma_nents = ib_dma_map_sg(ib_dev, data->sg, data->nents,
+					data->dma_dir);
+	if (unlikely(!data->dma_nents)) {
+		pr_err("Cmd: unable to dma map SGs %p\n", sg);
+		return -EINVAL;
+	}
+
+	pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
+		 isert_cmd, data->dma_nents, data->sg, data->nents, data->len);
+
+	return 0;
+}
+
+static void
+isert_unmap_data_buf(struct isert_conn *isert_conn, struct isert_data_buf *data)
+{
+	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+
+	ib_dma_unmap_sg(ib_dev, data->sg, data->nents, data->dma_dir);
+	memset(data, 0, sizeof(*data));
+}
+
+
+
 static void
 isert_unmap_cmd(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 {
 	struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
-	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 
 	pr_debug("isert_unmap_cmd: %p\n", isert_cmd);
-	if (wr->sge) {
+
+	if (wr->data.sg) {
 		pr_debug("isert_unmap_cmd: %p unmap_sg op\n", isert_cmd);
-		ib_dma_unmap_sg(ib_dev, wr->sge, wr->num_sge,
-				(wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-				DMA_TO_DEVICE : DMA_FROM_DEVICE);
-		wr->sge = NULL;
+		isert_unmap_data_buf(isert_conn, &wr->data);
 	}
 
 	if (wr->send_wr) {
@@ -1424,7 +1465,6 @@ static void
 isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 {
 	struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
-	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 	LIST_HEAD(unmap_list);
 
 	pr_debug("unreg_fastreg_cmd: %p\n", isert_cmd);
@@ -1438,12 +1478,9 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 		wr->fr_desc = NULL;
 	}
 
-	if (wr->sge) {
+	if (wr->data.sg) {
 		pr_debug("unreg_fastreg_cmd: %p unmap_sg op\n", isert_cmd);
-		ib_dma_unmap_sg(ib_dev, wr->sge, wr->num_sge,
-				(wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-				DMA_TO_DEVICE : DMA_FROM_DEVICE);
-		wr->sge = NULL;
+		isert_unmap_data_buf(isert_conn, &wr->data);
 	}
 
 	wr->ib_sge = NULL;
@@ -1548,7 +1585,7 @@ isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
 
 	iscsit_stop_dataout_timer(cmd);
 	device->unreg_rdma_mem(isert_cmd, isert_conn);
-	cmd->write_data_done = wr->cur_rdma_length;
+	cmd->write_data_done = wr->data.len;
 
 	pr_debug("Cmd: %p RDMA_READ comp calling execute_cmd\n", isert_cmd);
 	spin_lock_bh(&cmd->istate_lock);
@@ -2046,54 +2083,39 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
 	struct isert_conn *isert_conn = (struct isert_conn *)conn->context;
-	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+	struct isert_data_buf *data = &wr->data;
 	struct ib_send_wr *send_wr;
 	struct ib_sge *ib_sge;
-	struct scatterlist *sg_start;
-	u32 sg_off = 0, sg_nents;
-	u32 offset = 0, data_len, data_left, rdma_write_max, va_offset = 0;
-	int ret = 0, count, i, ib_sge_cnt;
+	u32 offset, data_len, data_left, rdma_write_max, va_offset = 0;
+	int ret = 0, i, ib_sge_cnt;
 
-	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
-		data_left = se_cmd->data_length;
-	} else {
-		sg_off = cmd->write_data_done / PAGE_SIZE;
-		data_left = se_cmd->data_length - cmd->write_data_done;
-		offset = cmd->write_data_done;
-		isert_cmd->tx_desc.isert_cmd = isert_cmd;
-	}
+	isert_cmd->tx_desc.isert_cmd = isert_cmd;
 
-	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
-	sg_nents = se_cmd->t_data_nents - sg_off;
+	offset = wr->iser_ib_op == ISER_IB_RDMA_READ ? cmd->write_data_done : 0;
+	ret = isert_map_data_buf(isert_conn, isert_cmd, se_cmd->t_data_sg,
+				 se_cmd->t_data_nents, se_cmd->data_length,
+				 offset, wr->iser_ib_op, &wr->data);
+	if (ret)
+		return ret;
 
-	count = ib_dma_map_sg(ib_dev, sg_start, sg_nents,
-			      (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-			      DMA_TO_DEVICE : DMA_FROM_DEVICE);
-	if (unlikely(!count)) {
-		pr_err("Cmd: %p unrable to map SGs\n", isert_cmd);
-		return -EINVAL;
-	}
-	wr->sge = sg_start;
-	wr->num_sge = sg_nents;
-	wr->cur_rdma_length = data_left;
-	pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
-		 isert_cmd, count, sg_start, sg_nents, data_left);
+	data_left = data->len;
+	offset = data->offset;
 
-	ib_sge = kzalloc(sizeof(struct ib_sge) * sg_nents, GFP_KERNEL);
+	ib_sge = kzalloc(sizeof(struct ib_sge) * data->nents, GFP_KERNEL);
 	if (!ib_sge) {
 		pr_warn("Unable to allocate ib_sge\n");
 		ret = -ENOMEM;
-		goto unmap_sg;
+		goto unmap_cmd;
 	}
 	wr->ib_sge = ib_sge;
 
-	wr->send_wr_num = DIV_ROUND_UP(sg_nents, isert_conn->max_sge);
+	wr->send_wr_num = DIV_ROUND_UP(data->nents, isert_conn->max_sge);
 	wr->send_wr = kzalloc(sizeof(struct ib_send_wr) * wr->send_wr_num,
 				GFP_KERNEL);
 	if (!wr->send_wr) {
 		pr_debug("Unable to allocate wr->send_wr\n");
 		ret = -ENOMEM;
-		goto unmap_sg;
+		goto unmap_cmd;
 	}
 
 	wr->isert_cmd = isert_cmd;
@@ -2132,10 +2154,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 
 	return 0;
-unmap_sg:
-	ib_dma_unmap_sg(ib_dev, sg_start, sg_nents,
-			(wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-			DMA_TO_DEVICE : DMA_FROM_DEVICE);
+unmap_cmd:
+	isert_unmap_data_buf(isert_conn, data);
+
 	return ret;
 }
 
@@ -2179,10 +2200,10 @@ isert_map_fr_pagelist(struct ib_device *ib_dev,
 }
 
 static int
-isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
-		  struct isert_conn *isert_conn, struct scatterlist *sg_start,
-		  struct ib_sge *ib_sge, u32 sg_nents, u32 offset,
-		  unsigned int data_len)
+isert_fast_reg_mr(struct isert_conn *isert_conn,
+		  struct fast_reg_descriptor *fr_desc,
+		  struct isert_data_buf *mem,
+		  struct ib_sge *sge)
 {
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
 	struct ib_send_wr fr_wr, inv_wr;
@@ -2191,13 +2212,19 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 	u32 page_off;
 	u8 key;
 
-	sg_nents = min_t(unsigned int, sg_nents, ISCSI_ISER_SG_TABLESIZE);
-	page_off = offset % PAGE_SIZE;
+	if (mem->dma_nents == 1) {
+		sge->lkey = isert_conn->conn_mr->lkey;
+		sge->addr = ib_sg_dma_address(ib_dev, &mem->sg[0]);
+		sge->length = ib_sg_dma_len(ib_dev, &mem->sg[0]);
+		return 0;
+	}
+
+	page_off = mem->offset % PAGE_SIZE;
 
 	pr_debug("Use fr_desc %p sg_nents %d offset %u\n",
-		 fr_desc, sg_nents, offset);
+		 fr_desc, mem->nents, mem->offset);
 
-	pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
+	pagelist_len = isert_map_fr_pagelist(ib_dev, mem->sg, mem->nents,
 					     &fr_desc->data_frpl->page_list[0]);
 
 	if (!fr_desc->valid) {
@@ -2218,7 +2245,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 	fr_wr.wr.fast_reg.page_list = fr_desc->data_frpl;
 	fr_wr.wr.fast_reg.page_list_len = pagelist_len;
 	fr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
-	fr_wr.wr.fast_reg.length = data_len;
+	fr_wr.wr.fast_reg.length = mem->len;
 	fr_wr.wr.fast_reg.rkey = fr_desc->data_mr->rkey;
 	fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
 
@@ -2234,12 +2261,12 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
 	}
 	fr_desc->valid = false;
 
-	ib_sge->lkey = fr_desc->data_mr->lkey;
-	ib_sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
-	ib_sge->length = data_len;
+	sge->lkey = fr_desc->data_mr->lkey;
+	sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
+	sge->length = mem->len;
 
 	pr_debug("RDMA ib_sge: addr: 0x%16llx  length: %u lkey: %08x\n",
-		 ib_sge->addr, ib_sge->length, ib_sge->lkey);
+		 sge->addr, sge->length, sge->lkey);
 
 	return ret;
 }
@@ -2250,54 +2277,43 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 {
 	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
-	struct isert_conn *isert_conn = (struct isert_conn *)conn->context;
-	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+	struct isert_conn *isert_conn = conn->context;
 	struct ib_send_wr *send_wr;
-	struct ib_sge *ib_sge;
-	struct scatterlist *sg_start;
-	struct fast_reg_descriptor *fr_desc;
-	u32 sg_off = 0, sg_nents;
-	u32 offset = 0, data_len, data_left, rdma_write_max;
-	int ret = 0, count;
+	struct fast_reg_descriptor *fr_desc = NULL;
+	u32 offset;
+	int ret = 0;
 	unsigned long flags;
 
-	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
-		data_left = se_cmd->data_length;
-	} else {
-		offset = cmd->write_data_done;
-		sg_off = offset / PAGE_SIZE;
-		data_left = se_cmd->data_length - cmd->write_data_done;
-		isert_cmd->tx_desc.isert_cmd = isert_cmd;
-	}
+	isert_cmd->tx_desc.isert_cmd = isert_cmd;
 
-	sg_start = &cmd->se_cmd.t_data_sg[sg_off];
-	sg_nents = se_cmd->t_data_nents - sg_off;
+	offset = wr->iser_ib_op == ISER_IB_RDMA_READ ? cmd->write_data_done : 0;
+	ret = isert_map_data_buf(isert_conn, isert_cmd, se_cmd->t_data_sg,
+				 se_cmd->t_data_nents, se_cmd->data_length,
+				 offset, wr->iser_ib_op, &wr->data);
+	if (ret)
+		return ret;
 
-	count = ib_dma_map_sg(ib_dev, sg_start, sg_nents,
-			      (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-			      DMA_TO_DEVICE : DMA_FROM_DEVICE);
-	if (unlikely(!count)) {
-		pr_err("Cmd: %p unrable to map SGs\n", isert_cmd);
-		return -EINVAL;
+	if (wr->data.dma_nents != 1) {
+		spin_lock_irqsave(&isert_conn->conn_lock, flags);
+		fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
+					   struct fast_reg_descriptor, list);
+		list_del(&fr_desc->list);
+		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
+		wr->fr_desc = fr_desc;
 	}
-	wr->sge = sg_start;
-	wr->num_sge = sg_nents;
-	pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
-		 isert_cmd, count, sg_start, sg_nents, data_left);
 
-	memset(&wr->s_ib_sge, 0, sizeof(*ib_sge));
-	ib_sge = &wr->s_ib_sge;
-	wr->ib_sge = ib_sge;
+	ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->data, &wr->s_ib_sge);
+	if (ret)
+		goto unmap_cmd;
 
+	wr->ib_sge = &wr->s_ib_sge;
 	wr->send_wr_num = 1;
 	memset(&wr->s_send_wr, 0, sizeof(*send_wr));
 	wr->send_wr = &wr->s_send_wr;
-
 	wr->isert_cmd = isert_cmd;
-	rdma_write_max = ISCSI_ISER_SG_TABLESIZE * PAGE_SIZE;
 
 	send_wr = &isert_cmd->rdma_wr.s_send_wr;
-	send_wr->sg_list = ib_sge;
+	send_wr->sg_list = &wr->s_ib_sge;
 	send_wr->num_sge = 1;
 	send_wr->wr_id = (unsigned long)&isert_cmd->tx_desc;
 	if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
@@ -2313,37 +2329,15 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		send_wr->send_flags = IB_SEND_SIGNALED;
 	}
 
-	data_len = min(data_left, rdma_write_max);
-	wr->cur_rdma_length = data_len;
-
-	/* if there is a single dma entry, dma mr is sufficient */
-	if (count == 1) {
-		ib_sge->addr = ib_sg_dma_address(ib_dev, &sg_start[0]);
-		ib_sge->length = ib_sg_dma_len(ib_dev, &sg_start[0]);
-		ib_sge->lkey = isert_conn->conn_mr->lkey;
-		wr->fr_desc = NULL;
-	} else {
+	return 0;
+unmap_cmd:
+	if (fr_desc) {
 		spin_lock_irqsave(&isert_conn->conn_lock, flags);
-		fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
-					   struct fast_reg_descriptor, list);
-		list_del(&fr_desc->list);
+		list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
 		spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
-		wr->fr_desc = fr_desc;
-
-		ret = isert_fast_reg_mr(fr_desc, isert_conn, sg_start,
-					ib_sge, sg_nents, offset, data_len);
-		if (ret) {
-			list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
-			goto unmap_sg;
-		}
 	}
+	isert_unmap_data_buf(isert_conn, &wr->data);
 
-	return 0;
-
-unmap_sg:
-	ib_dma_unmap_sg(ib_dev, sg_start, sg_nents,
-			(wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
-			DMA_TO_DEVICE : DMA_FROM_DEVICE);
 	return ret;
 }
 
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 708a069..18cb2e4 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -55,18 +55,26 @@ struct fast_reg_descriptor {
 	bool			valid;
 };
 
+struct isert_data_buf {
+	struct scatterlist     *sg;
+	int			nents;
+	u32			sg_off;
+	u32			len; /* cur_rdma_length */
+	u32			offset;
+	unsigned int		dma_nents;
+	enum dma_data_direction dma_dir;
+};
+
 struct isert_rdma_wr {
 	struct list_head	wr_list;
 	struct isert_cmd	*isert_cmd;
 	enum iser_ib_op_code	iser_ib_op;
 	struct ib_sge		*ib_sge;
 	struct ib_sge		s_ib_sge;
-	int			num_sge;
-	struct scatterlist	*sge;
 	int			send_wr_num;
 	struct ib_send_wr	*send_wr;
 	struct ib_send_wr	s_send_wr;
-	u32			cur_rdma_length;
+	struct isert_data_buf	data;
 	struct fast_reg_descriptor *fr_desc;
 };
 
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 10/12] IB/isert: Initialize T10-PI resources

[Sagi Grimberg]

Introduce pi_context to hold relevant RDMA protection resources.
We eliminate data_key_valid boolean and replace it with indicators
container to indicate:
- Is the descriptor protected (registered via signature MR)
- Is the data_mr key valid (can spare LOCAL_INV WR)
- Is the prot_mr key valid (can spare LOCAL_INV WR)
- Is the sig_mr key valid (can spare LOCAL_INV WR)

Upon connection establishment check if network portal is T10-PI
enabled and allocate T10-PI resources if necessary, allocate
signature enabled memory regions and mark connection queue-pair
as signature enabled.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |  104 +++++++++++++++++++++++++++----
 drivers/infiniband/ulp/isert/ib_isert.h |   23 ++++++-
 2 files changed, 110 insertions(+), 17 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index cbf0248..84c98f5 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -87,7 +87,8 @@ isert_query_device(struct ib_device *ib_dev, struct ib_device_attr *devattr)
 }
 
 static int
-isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id)
+isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id,
+		    u8 protection)
 {
 	struct isert_device *device = isert_conn->conn_device;
 	struct ib_qp_init_attr attr;
@@ -119,6 +120,8 @@ isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id)
 	attr.cap.max_recv_sge = 1;
 	attr.sq_sig_type = IB_SIGNAL_REQ_WR;
 	attr.qp_type = IB_QPT_RC;
+	if (protection)
+		attr.create_flags |= IB_QP_CREATE_SIGNATURE_EN;
 
 	pr_debug("isert_conn_setup_qp cma_id->device: %p\n",
 		 cma_id->device);
@@ -236,13 +239,18 @@ isert_create_device_ib_res(struct isert_device *device)
 		device->unreg_rdma_mem = isert_unmap_cmd;
 	}
 
+	/* Check signature cap */
+	device->pi_capable = dev_attr->device_cap_flags &
+			     IB_DEVICE_SIGNATURE_HANDOVER ? true : false;
+
 	device->cqs_used = min_t(int, num_online_cpus(),
 				 device->ib_device->num_comp_vectors);
 	device->cqs_used = min(ISERT_MAX_CQ, device->cqs_used);
 	pr_debug("Using %d CQs, device %s supports %d vectors support "
-		 "Fast registration %d\n",
+		 "Fast registration %d pi_capable %d\n",
 		 device->cqs_used, device->ib_device->name,
-		 device->ib_device->num_comp_vectors, device->use_fastreg);
+		 device->ib_device->num_comp_vectors, device->use_fastreg,
+		 device->pi_capable);
 	device->cq_desc = kzalloc(sizeof(struct isert_cq_desc) *
 				device->cqs_used, GFP_KERNEL);
 	if (!device->cq_desc) {
@@ -395,6 +403,12 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
 		list_del(&fr_desc->list);
 		ib_free_fast_reg_page_list(fr_desc->data_frpl);
 		ib_dereg_mr(fr_desc->data_mr);
+		if (fr_desc->pi_ctx) {
+			ib_free_fast_reg_page_list(fr_desc->pi_ctx->prot_frpl);
+			ib_dereg_mr(fr_desc->pi_ctx->prot_mr);
+			ib_destroy_mr(fr_desc->pi_ctx->sig_mr);
+			kfree(fr_desc->pi_ctx);
+		}
 		kfree(fr_desc);
 		++i;
 	}
@@ -406,8 +420,10 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
 
 static int
 isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
-		     struct fast_reg_descriptor *fr_desc)
+		     struct fast_reg_descriptor *fr_desc, u8 protection)
 {
+	int ret;
+
 	fr_desc->data_frpl = ib_alloc_fast_reg_page_list(ib_device,
 							 ISCSI_ISER_SG_TABLESIZE);
 	if (IS_ERR(fr_desc->data_frpl)) {
@@ -420,19 +436,73 @@ isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
 	if (IS_ERR(fr_desc->data_mr)) {
 		pr_err("Failed to allocate data frmr err=%ld\n",
 		       PTR_ERR(fr_desc->data_mr));
-		ib_free_fast_reg_page_list(fr_desc->data_frpl);
-		return PTR_ERR(fr_desc->data_mr);
+		ret = PTR_ERR(fr_desc->data_mr);
+		goto err_data_frpl;
 	}
 	pr_debug("Create fr_desc %p page_list %p\n",
 		 fr_desc, fr_desc->data_frpl->page_list);
+	fr_desc->ind |= ISERT_DATA_KEY_VALID;
 
-	fr_desc->valid = true;
+	if (protection) {
+		struct ib_mr_init_attr mr_init_attr = {0};
+		struct pi_context *pi_ctx;
+
+		fr_desc->pi_ctx = kzalloc(sizeof(*fr_desc->pi_ctx), GFP_KERNEL);
+		if (!fr_desc->pi_ctx) {
+			pr_err("Failed to allocate pi context\n");
+			ret = -ENOMEM;
+			goto err_data_mr;
+		}
+		pi_ctx = fr_desc->pi_ctx;
+
+		pi_ctx->prot_frpl = ib_alloc_fast_reg_page_list(ib_device,
+						    ISCSI_ISER_SG_TABLESIZE);
+		if (IS_ERR(pi_ctx->prot_frpl)) {
+			pr_err("Failed to allocate prot frpl err=%ld\n",
+			       PTR_ERR(pi_ctx->prot_frpl));
+			ret = PTR_ERR(pi_ctx->prot_frpl);
+			goto err_pi_ctx;
+		}
+
+		pi_ctx->prot_mr = ib_alloc_fast_reg_mr(pd, ISCSI_ISER_SG_TABLESIZE);
+		if (IS_ERR(pi_ctx->prot_mr)) {
+			pr_err("Failed to allocate prot frmr err=%ld\n",
+			       PTR_ERR(pi_ctx->prot_mr));
+			ret = PTR_ERR(pi_ctx->prot_mr);
+			goto err_prot_frpl;
+		}
+		fr_desc->ind |= ISERT_PROT_KEY_VALID;
+
+		mr_init_attr.max_reg_descriptors = 2;
+		mr_init_attr.flags |= IB_MR_SIGNATURE_EN;
+		pi_ctx->sig_mr = ib_create_mr(pd, &mr_init_attr);
+		if (IS_ERR(pi_ctx->sig_mr)) {
+			pr_err("Failed to allocate signature enabled mr err=%ld\n",
+			       PTR_ERR(pi_ctx->sig_mr));
+			ret = PTR_ERR(pi_ctx->sig_mr);
+			goto err_prot_mr;
+		}
+		fr_desc->ind |= ISERT_SIG_KEY_VALID;
+	}
+	fr_desc->ind &= ~ISERT_PROTECTED;
 
 	return 0;
+err_prot_mr:
+	ib_dereg_mr(fr_desc->pi_ctx->prot_mr);
+err_prot_frpl:
+	ib_free_fast_reg_page_list(fr_desc->pi_ctx->prot_frpl);
+err_pi_ctx:
+	kfree(fr_desc->pi_ctx);
+err_data_mr:
+	ib_dereg_mr(fr_desc->data_mr);
+err_data_frpl:
+	ib_free_fast_reg_page_list(fr_desc->data_frpl);
+
+	return ret;
 }
 
 static int
-isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
+isert_conn_create_fastreg_pool(struct isert_conn *isert_conn, u8 pi_support)
 {
 	struct fast_reg_descriptor *fr_desc;
 	struct isert_device *device = isert_conn->conn_device;
@@ -449,7 +519,8 @@ isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
 		}
 
 		ret = isert_create_fr_desc(device->ib_device,
-					   isert_conn->conn_pd, fr_desc);
+					   isert_conn->conn_pd, fr_desc,
+					   pi_support);
 		if (ret) {
 			pr_err("Failed to create fastreg descriptor err=%d\n",
 			       ret);
@@ -480,6 +551,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 	struct isert_device *device;
 	struct ib_device *ib_dev = cma_id->device;
 	int ret = 0;
+	u8 pi_support = np->tpg_np->tpg->tpg_attrib.t10_pi;
 
 	pr_debug("Entering isert_connect_request cma_id: %p, context: %p\n",
 		 cma_id, cma_id->context);
@@ -570,8 +642,14 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 		goto out_mr;
 	}
 
+	if (pi_support && !device->pi_capable) {
+		pr_err("Protection information requested but not supported\n");
+		ret = -EINVAL;
+		goto out_mr;
+	}
+
 	if (device->use_fastreg) {
-		ret = isert_conn_create_fastreg_pool(isert_conn);
+		ret = isert_conn_create_fastreg_pool(isert_conn, pi_support);
 		if (ret) {
 			pr_err("Conn: %p failed to create fastreg pool\n",
 			       isert_conn);
@@ -579,7 +657,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
 		}
 	}
 
-	ret = isert_conn_setup_qp(isert_conn, cma_id);
+	ret = isert_conn_setup_qp(isert_conn, cma_id, pi_support);
 	if (ret)
 		goto out_conn_dev;
 
@@ -2227,7 +2305,7 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 	pagelist_len = isert_map_fr_pagelist(ib_dev, mem->sg, mem->nents,
 					     &fr_desc->data_frpl->page_list[0]);
 
-	if (!fr_desc->valid) {
+	if (!(fr_desc->ind & ISERT_DATA_KEY_VALID)) {
 		memset(&inv_wr, 0, sizeof(inv_wr));
 		inv_wr.opcode = IB_WR_LOCAL_INV;
 		inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
@@ -2259,7 +2337,7 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 		pr_err("fast registration failed, ret:%d\n", ret);
 		return ret;
 	}
-	fr_desc->valid = false;
+	fr_desc->ind &= ~ISERT_DATA_KEY_VALID;
 
 	sge->lkey = fr_desc->data_mr->lkey;
 	sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 18cb2e4..d984172 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -48,11 +48,25 @@ struct iser_tx_desc {
 	struct ib_send_wr send_wr;
 } __packed;
 
+enum isert_indicator {
+	ISERT_PROTECTED		= 1 << 0,
+	ISERT_DATA_KEY_VALID	= 1 << 1,
+	ISERT_PROT_KEY_VALID	= 1 << 2,
+	ISERT_SIG_KEY_VALID	= 1 << 3,
+};
+
+struct pi_context {
+	struct ib_mr		       *prot_mr;
+	struct ib_fast_reg_page_list   *prot_frpl;
+	struct ib_mr		       *sig_mr;
+};
+
 struct fast_reg_descriptor {
-	struct list_head	list;
-	struct ib_mr		*data_mr;
-	struct ib_fast_reg_page_list	*data_frpl;
-	bool			valid;
+	struct list_head		list;
+	struct ib_mr		       *data_mr;
+	struct ib_fast_reg_page_list   *data_frpl;
+	u8				ind;
+	struct pi_context	       *pi_ctx;
 };
 
 struct isert_data_buf {
@@ -148,6 +162,7 @@ struct isert_cq_desc {
 
 struct isert_device {
 	int			use_fastreg;
+	bool			pi_capable;
 	int			cqs_used;
 	int			refcount;
 	int			cq_active_qps[ISERT_MAX_CQ];
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 11/12] IB/isert: Accept RDMA_WRITE completions

[Sagi Grimberg]

In case of protected transactions, we will need to check the
protection status of the transaction before sending SCSI response.
So be ready for RDMA_WRITE completions. currently we don't ask
for these completions, but for T10-PI we will.

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |   19 +++++++++++++++++--
 1 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 84c98f5..3871ab2 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -51,6 +51,8 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn);
 static int
 isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	       struct isert_rdma_wr *wr);
+static int
+isert_put_response(struct iscsi_conn *conn, struct iscsi_cmd *cmd);
 
 static void
 isert_qp_event_callback(struct ib_event *e, void *context)
@@ -1652,6 +1654,18 @@ isert_completion_put(struct iser_tx_desc *tx_desc, struct isert_cmd *isert_cmd,
 }
 
 static void
+isert_completion_rdma_write(struct iser_tx_desc *tx_desc,
+			    struct isert_cmd *isert_cmd)
+{
+	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+	struct isert_conn *isert_conn = isert_cmd->conn;
+	struct isert_device *device = isert_conn->conn_device;
+
+	device->unreg_rdma_mem(isert_cmd, isert_conn);
+	isert_put_response(isert_conn->conn, cmd);
+}
+
+static void
 isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
 			   struct isert_cmd *isert_cmd)
 {
@@ -1771,8 +1785,9 @@ __isert_send_completion(struct iser_tx_desc *tx_desc,
 					  isert_conn, ib_dev);
 		break;
 	case ISER_IB_RDMA_WRITE:
-		pr_err("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
-		dump_stack();
+		pr_debug("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
+		atomic_dec(&isert_conn->post_send_buf_count);
+		isert_completion_rdma_write(tx_desc, isert_cmd);
 		break;
 	case ISER_IB_RDMA_READ:
 		pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n");
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 12/12] IB/isert: Support T10-PI protected transactions

[Sagi Grimberg]

In case the Target core passed transport T10 protection
operation:

1. Register data buffer (data memory region)
2. Register protection buffer if exsists (prot memory region)
3. Register signature region (signature memory region)
   - use work request IB_WR_REG_SIG_MR
4. Execute RDMA
5. Upon RDMA completion check the signature status
   - if succeeded send good SCSI response
   - if failed send SCSI bad response with appropriate sense buffer

Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/isert/ib_isert.c |  321 ++++++++++++++++++++++++++++---
 drivers/infiniband/ulp/isert/ib_isert.h |    1 +
 2 files changed, 292 insertions(+), 30 deletions(-)

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 3871ab2..04bdd79 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1552,6 +1552,10 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
 	if (wr->fr_desc) {
 		pr_debug("unreg_fastreg_cmd: %p free fr_desc %p\n",
 			 isert_cmd, wr->fr_desc);
+		if (wr->fr_desc->ind & ISERT_PROTECTED) {
+			isert_unmap_data_buf(isert_conn, &wr->prot);
+			wr->fr_desc->ind &= ~ISERT_PROTECTED;
+		}
 		spin_lock_bh(&isert_conn->conn_lock);
 		list_add_tail(&wr->fr_desc->list, &isert_conn->conn_fr_pool);
 		spin_unlock_bh(&isert_conn->conn_lock);
@@ -1657,12 +1661,55 @@ static void
 isert_completion_rdma_write(struct iser_tx_desc *tx_desc,
 			    struct isert_cmd *isert_cmd)
 {
+	struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
 	struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_conn *isert_conn = isert_cmd->conn;
 	struct isert_device *device = isert_conn->conn_device;
+	struct ib_mr_status mr_status;
+	int ret = 0;
 
+	if (wr->fr_desc && wr->fr_desc->ind & ISERT_PROTECTED) {
+		ret = ib_check_mr_status(wr->fr_desc->pi_ctx->sig_mr,
+					 IB_MR_CHECK_SIG_STATUS, &mr_status);
+		if (ret) {
+			pr_err("ib_check_mr_status failed, ret %d\n", ret);
+			goto fail_mr_status;
+		}
+		if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
+			u32 block_size = se_cmd->se_dev->dev_attrib.block_size;
+
+			switch (mr_status.sig_err.err_type) {
+			case IB_SIG_BAD_GUARD:
+				se_cmd->pi_err = TCM_LOGICAL_BLOCK_GUARD_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_REFTAG:
+				se_cmd->pi_err = TCM_LOGICAL_BLOCK_REF_TAG_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_APPTAG:
+				se_cmd->pi_err = TCM_LOGICAL_BLOCK_APP_TAG_CHECK_FAILED;
+				break;
+			}
+			se_cmd->bad_sector =
+				mr_status.sig_err.sig_err_offset / block_size;
+
+			pr_err("isert: PI error found type %d at sector 0x%llx "
+			       "expected 0x%x vs actual 0x%x\n",
+			       mr_status.sig_err.err_type,
+			       (unsigned long long)se_cmd->bad_sector,
+			       mr_status.sig_err.expected,
+			       mr_status.sig_err.actual);
+			ret = 1;
+		}
+	}
+
+fail_mr_status:
 	device->unreg_rdma_mem(isert_cmd, isert_conn);
-	isert_put_response(isert_conn->conn, cmd);
+	if (ret)
+		transport_send_check_condition_and_sense(se_cmd,
+							 se_cmd->pi_err, 0);
+	else
+		isert_put_response(isert_conn->conn, cmd);
 }
 
 static void
@@ -1674,7 +1721,43 @@ isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
 	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_conn *isert_conn = isert_cmd->conn;
 	struct isert_device *device = isert_conn->conn_device;
+	struct ib_mr_status mr_status;
+	int ret;
 
+	if (wr->fr_desc && wr->fr_desc->ind & ISERT_PROTECTED) {
+		ret = ib_check_mr_status(wr->fr_desc->pi_ctx->sig_mr,
+					 IB_MR_CHECK_SIG_STATUS, &mr_status);
+		if (ret) {
+			pr_err("ib_check_mr_status failed, ret %d\n", ret);
+			goto fail_mr_status;
+		}
+		if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
+			u32 block_size = se_cmd->se_dev->dev_attrib.block_size;
+
+			switch (mr_status.sig_err.err_type) {
+			case IB_SIG_BAD_GUARD:
+				se_cmd->pi_err = TCM_LOGICAL_BLOCK_GUARD_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_REFTAG:
+				se_cmd->pi_err = TCM_LOGICAL_BLOCK_REF_TAG_CHECK_FAILED;
+				break;
+			case IB_SIG_BAD_APPTAG:
+				se_cmd->pi_err = TCM_LOGICAL_BLOCK_APP_TAG_CHECK_FAILED;
+				break;
+			}
+			se_cmd->bad_sector =
+				mr_status.sig_err.sig_err_offset / block_size;
+
+			pr_err("isert: PI error found type %d at sector 0x%llx "
+			       "expected 0x%x vs actual 0x%x\n",
+			       mr_status.sig_err.err_type,
+			       (unsigned long long)se_cmd->bad_sector,
+			       mr_status.sig_err.expected,
+			       mr_status.sig_err.actual);
+		}
+	}
+
+fail_mr_status:
 	iscsit_stop_dataout_timer(cmd);
 	device->unreg_rdma_mem(isert_cmd, isert_conn);
 	cmd->write_data_done = wr->data.len;
@@ -2296,9 +2379,12 @@ static int
 isert_fast_reg_mr(struct isert_conn *isert_conn,
 		  struct fast_reg_descriptor *fr_desc,
 		  struct isert_data_buf *mem,
+		  enum isert_indicator ind,
 		  struct ib_sge *sge)
 {
 	struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+	struct ib_mr *mr;
+	struct ib_fast_reg_page_list *frpl;
 	struct ib_send_wr fr_wr, inv_wr;
 	struct ib_send_wr *bad_wr, *wr = NULL;
 	int ret, pagelist_len;
@@ -2309,37 +2395,50 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 		sge->lkey = isert_conn->conn_mr->lkey;
 		sge->addr = ib_sg_dma_address(ib_dev, &mem->sg[0]);
 		sge->length = ib_sg_dma_len(ib_dev, &mem->sg[0]);
+		pr_debug("%s:%d sge: addr: 0x%llx  length: %u lkey: %x\n",
+			 __func__, __LINE__, sge->addr, sge->length,
+			 sge->lkey);
 		return 0;
 	}
 
+	if (ind == ISERT_DATA_KEY_VALID) {
+		/* Registering data buffer */
+		mr = fr_desc->data_mr;
+		frpl = fr_desc->data_frpl;
+	} else {
+		/* Registering protection buffer */
+		mr = fr_desc->pi_ctx->prot_mr;
+		frpl = fr_desc->pi_ctx->prot_frpl;
+
+	}
+
 	page_off = mem->offset % PAGE_SIZE;
 
 	pr_debug("Use fr_desc %p sg_nents %d offset %u\n",
 		 fr_desc, mem->nents, mem->offset);
 
 	pagelist_len = isert_map_fr_pagelist(ib_dev, mem->sg, mem->nents,
-					     &fr_desc->data_frpl->page_list[0]);
+					     &frpl->page_list[0]);
 
 	if (!(fr_desc->ind & ISERT_DATA_KEY_VALID)) {
 		memset(&inv_wr, 0, sizeof(inv_wr));
 		inv_wr.opcode = IB_WR_LOCAL_INV;
-		inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
+		inv_wr.ex.invalidate_rkey = mr->rkey;
 		wr = &inv_wr;
 		/* Bump the key */
-		key = (u8)(fr_desc->data_mr->rkey & 0x000000FF);
-		ib_update_fast_reg_key(fr_desc->data_mr, ++key);
+		key = (u8)(mr->rkey & 0x000000FF);
+		ib_update_fast_reg_key(mr, ++key);
 	}
 
 	/* Prepare FASTREG WR */
 	memset(&fr_wr, 0, sizeof(fr_wr));
 	fr_wr.opcode = IB_WR_FAST_REG_MR;
-	fr_wr.wr.fast_reg.iova_start =
-		fr_desc->data_frpl->page_list[0] + page_off;
-	fr_wr.wr.fast_reg.page_list = fr_desc->data_frpl;
+	fr_wr.wr.fast_reg.iova_start = frpl->page_list[0] + page_off;
+	fr_wr.wr.fast_reg.page_list = frpl;
 	fr_wr.wr.fast_reg.page_list_len = pagelist_len;
 	fr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
 	fr_wr.wr.fast_reg.length = mem->len;
-	fr_wr.wr.fast_reg.rkey = fr_desc->data_mr->rkey;
+	fr_wr.wr.fast_reg.rkey = mr->rkey;
 	fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
 
 	if (!wr)
@@ -2352,25 +2451,147 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
 		pr_err("fast registration failed, ret:%d\n", ret);
 		return ret;
 	}
-	fr_desc->ind &= ~ISERT_DATA_KEY_VALID;
+	fr_desc->ind &= ~ind;
 
-	sge->lkey = fr_desc->data_mr->lkey;
-	sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
+	sge->lkey = mr->lkey;
+	sge->addr = frpl->page_list[0] + page_off;
 	sge->length = mem->len;
 
-	pr_debug("RDMA ib_sge: addr: 0x%16llx  length: %u lkey: %08x\n",
-		 sge->addr, sge->length, sge->lkey);
+	pr_debug("%s:%d sge: addr: 0x%llx  length: %u lkey: %x\n",
+		 __func__, __LINE__, sge->addr, sge->length,
+		 sge->lkey);
 
 	return ret;
 }
 
 static int
+isert_set_sig_attrs(struct se_cmd *se_cmd, struct ib_sig_attrs *sig_attrs)
+{
+	sig_attrs->mem.sig_type = IB_SIG_TYPE_T10_DIF;
+	sig_attrs->wire.sig_type = IB_SIG_TYPE_T10_DIF;
+	sig_attrs->mem.sig.dif.pi_interval =
+				se_cmd->se_dev->dev_attrib.block_size;
+	sig_attrs->wire.sig.dif.pi_interval =
+				se_cmd->se_dev->dev_attrib.block_size;
+
+	switch (se_cmd->prot_op) {
+	case TARGET_PROT_DIN_INSERT:
+	case TARGET_PROT_DOUT_STRIP:
+		sig_attrs->mem.sig.dif.type = IB_T10DIF_NONE;
+		sig_attrs->wire.sig.dif.type = se_cmd->prot_type;
+		sig_attrs->wire.sig.dif.bg_type = IB_T10DIF_CRC;
+		sig_attrs->wire.sig.dif.ref_tag = se_cmd->reftag_seed;
+		break;
+	case TARGET_PROT_DOUT_INSERT:
+	case TARGET_PROT_DIN_STRIP:
+		sig_attrs->mem.sig.dif.type = se_cmd->prot_type;
+		sig_attrs->mem.sig.dif.bg_type = IB_T10DIF_CRC;
+		sig_attrs->mem.sig.dif.ref_tag = se_cmd->reftag_seed;
+		sig_attrs->wire.sig.dif.type = IB_T10DIF_NONE;
+		break;
+	case TARGET_PROT_DIN_PASS:
+	case TARGET_PROT_DOUT_PASS:
+		sig_attrs->mem.sig.dif.type = se_cmd->prot_type;
+		sig_attrs->mem.sig.dif.bg_type = IB_T10DIF_CRC;
+		sig_attrs->mem.sig.dif.ref_tag = se_cmd->reftag_seed;
+		sig_attrs->wire.sig.dif.type = se_cmd->prot_type;
+		sig_attrs->wire.sig.dif.bg_type = IB_T10DIF_CRC;
+		sig_attrs->wire.sig.dif.ref_tag = se_cmd->reftag_seed;
+		break;
+	default:
+		pr_err("Unsupported PI operation %d\n", se_cmd->prot_op);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static inline u8
+isert_set_prot_checks(u8 prot_checks)
+{
+	return (prot_checks & TARGET_DIF_CHECK_GUARD  ? 0xc0 : 0) |
+	       (prot_checks & TARGET_DIF_CHECK_REFTAG ? 0x30 : 0) |
+	       (prot_checks & TARGET_DIF_CHECK_REFTAG ? 0x0f : 0);
+}
+
+static int
+isert_reg_sig_mr(struct isert_conn *isert_conn, struct se_cmd *se_cmd,
+		 struct fast_reg_descriptor *fr_desc,
+		 struct ib_sge *data_sge, struct ib_sge *prot_sge,
+		 struct ib_sge *sig_sge)
+{
+	struct ib_send_wr sig_wr, inv_wr;
+	struct ib_send_wr *bad_wr, *wr = NULL;
+	struct pi_context *pi_ctx = fr_desc->pi_ctx;
+	struct ib_sig_attrs sig_attrs;
+	int ret;
+	u32 key;
+
+	memset(&sig_attrs, 0, sizeof(sig_attrs))
+	ret = isert_set_sig_attrs(se_cmd, &sig_attrs);
+	if (ret)
+		goto err;
+
+	sig_attrs.check_mask = isert_set_prot_checks(se_cmd->prot_checks);
+
+	if (!(fr_desc->ind & ISERT_SIG_KEY_VALID)) {
+		memset(&inv_wr, 0, sizeof(inv_wr));
+		inv_wr.opcode = IB_WR_LOCAL_INV;
+		inv_wr.ex.invalidate_rkey = pi_ctx->sig_mr->rkey;
+		wr = &inv_wr;
+		/* Bump the key */
+		key = (u8)(pi_ctx->sig_mr->rkey & 0x000000FF);
+		ib_update_fast_reg_key(pi_ctx->sig_mr, ++key);
+	}
+
+	memset(&sig_wr, 0, sizeof(sig_wr));
+	sig_wr.opcode = IB_WR_REG_SIG_MR;
+	sig_wr.sg_list = data_sge;
+	sig_wr.num_sge = 1;
+	sig_wr.wr.sig_handover.access_flags = IB_ACCESS_LOCAL_WRITE;
+	sig_wr.wr.sig_handover.sig_attrs = &sig_attrs;
+	sig_wr.wr.sig_handover.sig_mr = pi_ctx->sig_mr;
+	if (se_cmd->t_prot_sg)
+		sig_wr.wr.sig_handover.prot = prot_sge;
+
+	if (!wr)
+		wr = &sig_wr;
+	else
+		wr->next = &sig_wr;
+
+	ret = ib_post_send(isert_conn->conn_qp, wr, &bad_wr);
+	if (ret) {
+		pr_err("fast registration failed, ret:%d\n", ret);
+		goto err;
+	}
+	fr_desc->ind &= ~ISERT_SIG_KEY_VALID;
+
+	sig_sge->lkey = pi_ctx->sig_mr->lkey;
+	sig_sge->addr = 0;
+	sig_sge->length = se_cmd->data_length;
+	if (se_cmd->prot_op != TARGET_PROT_DIN_STRIP &&
+	    se_cmd->prot_op != TARGET_PROT_DOUT_INSERT)
+		/*
+		 * We have protection guards on the wire
+		 * so we need to set a larget transfer
+		 */
+		sig_sge->length += se_cmd->prot_length;
+
+	pr_debug("sig_sge: addr: 0x%llx  length: %u lkey: %x\n",
+		 sig_sge->addr, sig_sge->length,
+		 sig_sge->lkey);
+err:
+	return ret;
+}
+
+static int
 isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	       struct isert_rdma_wr *wr)
 {
 	struct se_cmd *se_cmd = &cmd->se_cmd;
 	struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
 	struct isert_conn *isert_conn = conn->context;
+	struct ib_sge data_sge;
 	struct ib_send_wr *send_wr;
 	struct fast_reg_descriptor *fr_desc = NULL;
 	u32 offset;
@@ -2386,7 +2607,8 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	if (ret)
 		return ret;
 
-	if (wr->data.dma_nents != 1) {
+	if (wr->data.dma_nents != 1 ||
+	    se_cmd->prot_type != TARGET_PROT_NORMAL) {
 		spin_lock_irqsave(&isert_conn->conn_lock, flags);
 		fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
 					   struct fast_reg_descriptor, list);
@@ -2395,10 +2617,39 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		wr->fr_desc = fr_desc;
 	}
 
-	ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->data, &wr->s_ib_sge);
+	ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->data,
+				ISERT_DATA_KEY_VALID, &data_sge);
 	if (ret)
 		goto unmap_cmd;
 
+	if (se_cmd->prot_type != TARGET_PROT_NORMAL) {
+		struct ib_sge prot_sge, sig_sge;
+
+		if (se_cmd->t_prot_sg) {
+			ret = isert_map_data_buf(isert_conn, isert_cmd,
+						 se_cmd->t_prot_sg,
+						 se_cmd->t_prot_nents,
+						 se_cmd->prot_length,
+						 0, wr->iser_ib_op, &wr->prot);
+			if (ret)
+				goto unmap_cmd;
+
+			ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->prot,
+						ISERT_PROT_KEY_VALID, &prot_sge);
+			if (ret)
+				goto unmap_prot_cmd;
+		}
+
+		ret = isert_reg_sig_mr(isert_conn, se_cmd, fr_desc,
+				       &data_sge, &prot_sge, &sig_sge);
+		if (ret)
+			goto unmap_prot_cmd;
+
+		fr_desc->ind |= ISERT_PROTECTED;
+		memcpy(&wr->s_ib_sge, &sig_sge, sizeof(sig_sge));
+	} else
+		memcpy(&wr->s_ib_sge, &data_sge, sizeof(data_sge));
+
 	wr->ib_sge = &wr->s_ib_sge;
 	wr->send_wr_num = 1;
 	memset(&wr->s_send_wr, 0, sizeof(*send_wr));
@@ -2413,8 +2664,8 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 		send_wr->opcode = IB_WR_RDMA_WRITE;
 		send_wr->wr.rdma.remote_addr = isert_cmd->read_va;
 		send_wr->wr.rdma.rkey = isert_cmd->read_stag;
-		send_wr->send_flags = 0;
-		send_wr->next = &isert_cmd->tx_desc.send_wr;
+		send_wr->send_flags = se_cmd->prot_type == TARGET_PROT_NORMAL ?
+				      0 : IB_SEND_SIGNALED;
 	} else {
 		send_wr->opcode = IB_WR_RDMA_READ;
 		send_wr->wr.rdma.remote_addr = isert_cmd->write_va;
@@ -2423,6 +2674,9 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
 	}
 
 	return 0;
+unmap_prot_cmd:
+	if (se_cmd->t_prot_sg)
+		isert_unmap_data_buf(isert_conn, &wr->prot);
 unmap_cmd:
 	if (fr_desc) {
 		spin_lock_irqsave(&isert_conn->conn_lock, flags);
@@ -2454,26 +2708,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
 		return rc;
 	}
 
-	/*
-	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
-	 */
-	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
-	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
-			     &isert_cmd->tx_desc.iscsi_header);
-	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
-	isert_init_send_wr(isert_conn, isert_cmd,
-			   &isert_cmd->tx_desc.send_wr, true);
+	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
+		/*
+		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
+		 */
+		isert_create_send_desc(isert_conn, isert_cmd,
+				       &isert_cmd->tx_desc);
+		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
+				     &isert_cmd->tx_desc.iscsi_header);
+		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
+		isert_init_send_wr(isert_conn, isert_cmd,
+				   &isert_cmd->tx_desc.send_wr, true);
+		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
+	}
 
 	atomic_inc(&isert_conn->post_send_buf_count);
-
 	rc = ib_post_send(isert_conn->conn_qp, wr->send_wr, &wr_failed);
 	if (rc) {
 		pr_warn("ib_post_send() failed for IB_WR_RDMA_WRITE\n");
 		atomic_dec(&isert_conn->post_send_buf_count);
 	}
-	pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data READ\n",
-		 isert_cmd);
 
+	if (se_cmd->prot_type == TARGET_PROT_NORMAL)
+		pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data "
+			 "READ\n", isert_cmd);
+	else
+		pr_debug("Cmd: %p posted RDMA_WRITE for iSER Data READ\n",
+			 isert_cmd);
 	return 1;
 }
 
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index d984172..0e2253e 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -89,6 +89,7 @@ struct isert_rdma_wr {
 	struct ib_send_wr	*send_wr;
 	struct ib_send_wr	s_send_wr;
 	struct isert_data_buf	data;
+	struct isert_data_buf	prot;
 	struct fast_reg_descriptor *fr_desc;
 };
 
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 12/12] IB/isert: Support T10-PI protected transactions

[Nicholas A. Bellinger]

Hey Sagi,

On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
> In case the Target core passed transport T10 protection
> operation:
> 
> 1. Register data buffer (data memory region)
> 2. Register protection buffer if exsists (prot memory region)
> 3. Register signature region (signature memory region)
>    - use work request IB_WR_REG_SIG_MR
> 4. Execute RDMA
> 5. Upon RDMA completion check the signature status
>    - if succeeded send good SCSI response
>    - if failed send SCSI bad response with appropriate sense buffer
> 
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
>  drivers/infiniband/ulp/isert/ib_isert.c |  321 ++++++++++++++++++++++++++++---
>  drivers/infiniband/ulp/isert/ib_isert.h |    1 +
>  2 files changed, 292 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
> index 3871ab2..04bdd79 100644
> --- a/drivers/infiniband/ulp/isert/ib_isert.c
> +++ b/drivers/infiniband/ulp/isert/ib_isert.c

<SNIP>

> @@ -2454,26 +2708,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
>  		return rc;
>  	}
>  
> -	/*
> -	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
> -	 */
> -	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
> -	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
> -			     &isert_cmd->tx_desc.iscsi_header);
> -	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
> -	isert_init_send_wr(isert_conn, isert_cmd,
> -			   &isert_cmd->tx_desc.send_wr, true);
> +	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
> +		/*
> +		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
> +		 */
> +		isert_create_send_desc(isert_conn, isert_cmd,
> +				       &isert_cmd->tx_desc);
> +		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
> +				     &isert_cmd->tx_desc.iscsi_header);
> +		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
> +		isert_init_send_wr(isert_conn, isert_cmd,
> +				   &isert_cmd->tx_desc.send_wr, true);
> +		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
> +	}
>  

So I'm fixing up some v3.14-rc6 context changes before applying this
series to for-next, and noticed something above..

Namely, that iscsi_build_rsp_pdu() has been changed to pass 'false' as
it's second argument vs. the original case passing 'true'.  This boolean
controls the incrementing of the StatSN in the traditional response
header, which AFAICT should still be 'true' for this case.

Is there a reason why this was changed..?

--nab

Re: [PATCH v2 12/12] IB/isert: Support T10-PI protected transactions

[Sagi Grimberg]

On 3/13/2014 8:15 PM, Nicholas A. Bellinger wrote:
> Hey Sagi,
>
> On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
>> In case the Target core passed transport T10 protection
>> operation:
>>
>> 1. Register data buffer (data memory region)
>> 2. Register protection buffer if exsists (prot memory region)
>> 3. Register signature region (signature memory region)
>>     - use work request IB_WR_REG_SIG_MR
>> 4. Execute RDMA
>> 5. Upon RDMA completion check the signature status
>>     - if succeeded send good SCSI response
>>     - if failed send SCSI bad response with appropriate sense buffer
>>
>> Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> ---
>>   drivers/infiniband/ulp/isert/ib_isert.c |  321 ++++++++++++++++++++++++++++---
>>   drivers/infiniband/ulp/isert/ib_isert.h |    1 +
>>   2 files changed, 292 insertions(+), 30 deletions(-)
>>
>> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
>> index 3871ab2..04bdd79 100644
>> --- a/drivers/infiniband/ulp/isert/ib_isert.c
>> +++ b/drivers/infiniband/ulp/isert/ib_isert.c
> <SNIP>
>
>> @@ -2454,26 +2708,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
>>   		return rc;
>>   	}
>>   
>> -	/*
>> -	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
>> -	 */
>> -	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
>> -	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
>> -			     &isert_cmd->tx_desc.iscsi_header);
>> -	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
>> -	isert_init_send_wr(isert_conn, isert_cmd,
>> -			   &isert_cmd->tx_desc.send_wr, true);
>> +	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
>> +		/*
>> +		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
>> +		 */
>> +		isert_create_send_desc(isert_conn, isert_cmd,
>> +				       &isert_cmd->tx_desc);
>> +		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
>> +				     &isert_cmd->tx_desc.iscsi_header);
>> +		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
>> +		isert_init_send_wr(isert_conn, isert_cmd,
>> +				   &isert_cmd->tx_desc.send_wr, true);
>> +		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
>> +	}
>>   
> So I'm fixing up some v3.14-rc6 context changes before applying this
> series to for-next, and noticed something above..
>
> Namely, that iscsi_build_rsp_pdu() has been changed to pass 'false' as
> it's second argument vs. the original case passing 'true'.  This boolean
> controls the incrementing of the StatSN in the traditional response
> header, which AFAICT should still be 'true' for this case.
>
> Is there a reason why this was changed..?

Well, it wrong... This probably was left around when I rebased to 3.14 
(earlier versions incremented StatSN explicitly).
I wonder how that works though...

Want me to fix and resend?

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 12/12] IB/isert: Support T10-PI protected transactions

[Nicholas A. Bellinger]

On Thu, 2014-03-13 at 20:57 +0200, Sagi Grimberg wrote:
> On 3/13/2014 8:15 PM, Nicholas A. Bellinger wrote:
> > Hey Sagi,
> >
> > On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
> >> In case the Target core passed transport T10 protection
> >> operation:
> >>
> >> 1. Register data buffer (data memory region)
> >> 2. Register protection buffer if exsists (prot memory region)
> >> 3. Register signature region (signature memory region)
> >>     - use work request IB_WR_REG_SIG_MR
> >> 4. Execute RDMA
> >> 5. Upon RDMA completion check the signature status
> >>     - if succeeded send good SCSI response
> >>     - if failed send SCSI bad response with appropriate sense buffer
> >>
> >> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> >> ---
> >>   drivers/infiniband/ulp/isert/ib_isert.c |  321 ++++++++++++++++++++++++++++---
> >>   drivers/infiniband/ulp/isert/ib_isert.h |    1 +
> >>   2 files changed, 292 insertions(+), 30 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
> >> index 3871ab2..04bdd79 100644
> >> --- a/drivers/infiniband/ulp/isert/ib_isert.c
> >> +++ b/drivers/infiniband/ulp/isert/ib_isert.c
> > <SNIP>
> >
> >> @@ -2454,26 +2708,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
> >>   		return rc;
> >>   	}
> >>   
> >> -	/*
> >> -	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
> >> -	 */
> >> -	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
> >> -	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
> >> -			     &isert_cmd->tx_desc.iscsi_header);
> >> -	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
> >> -	isert_init_send_wr(isert_conn, isert_cmd,
> >> -			   &isert_cmd->tx_desc.send_wr, true);
> >> +	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
> >> +		/*
> >> +		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
> >> +		 */
> >> +		isert_create_send_desc(isert_conn, isert_cmd,
> >> +				       &isert_cmd->tx_desc);
> >> +		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
> >> +				     &isert_cmd->tx_desc.iscsi_header);
> >> +		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
> >> +		isert_init_send_wr(isert_conn, isert_cmd,
> >> +				   &isert_cmd->tx_desc.send_wr, true);
> >> +		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
> >> +	}
> >>   
> > So I'm fixing up some v3.14-rc6 context changes before applying this
> > series to for-next, and noticed something above..
> >
> > Namely, that iscsi_build_rsp_pdu() has been changed to pass 'false' as
> > it's second argument vs. the original case passing 'true'.  This boolean
> > controls the incrementing of the StatSN in the traditional response
> > header, which AFAICT should still be 'true' for this case.
> >
> > Is there a reason why this was changed..?
> 
> Well, it wrong... This probably was left around when I rebased to 3.14 
> (earlier versions incremented StatSN explicitly).
> I wonder how that works though...
> 
> Want me to fix and resend?
> 

Nah, fixing this in for-next now.

--nab

Re: [PATCH v2 12/12] IB/isert: Support T10-PI protected transactions

[Sagi Grimberg]

On 3/13/2014 8:59 PM, Nicholas A. Bellinger wrote:
> On Thu, 2014-03-13 at 20:57 +0200, Sagi Grimberg wrote:
>> On 3/13/2014 8:15 PM, Nicholas A. Bellinger wrote:
>>> Hey Sagi,
>>>
>>> On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
>>>> In case the Target core passed transport T10 protection
>>>> operation:
>>>>
>>>> 1. Register data buffer (data memory region)
>>>> 2. Register protection buffer if exsists (prot memory region)
>>>> 3. Register signature region (signature memory region)
>>>>      - use work request IB_WR_REG_SIG_MR
>>>> 4. Execute RDMA
>>>> 5. Upon RDMA completion check the signature status
>>>>      - if succeeded send good SCSI response
>>>>      - if failed send SCSI bad response with appropriate sense buffer
>>>>
>>>> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
>>>> ---
>>>>    drivers/infiniband/ulp/isert/ib_isert.c |  321 ++++++++++++++++++++++++++++---
>>>>    drivers/infiniband/ulp/isert/ib_isert.h |    1 +
>>>>    2 files changed, 292 insertions(+), 30 deletions(-)
>>>>
>>>> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
>>>> index 3871ab2..04bdd79 100644
>>>> --- a/drivers/infiniband/ulp/isert/ib_isert.c
>>>> +++ b/drivers/infiniband/ulp/isert/ib_isert.c
>>> <SNIP>
>>>
>>>> @@ -2454,26 +2708,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
>>>>    		return rc;
>>>>    	}
>>>>    
>>>> -	/*
>>>> -	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
>>>> -	 */
>>>> -	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
>>>> -	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
>>>> -			     &isert_cmd->tx_desc.iscsi_header);
>>>> -	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
>>>> -	isert_init_send_wr(isert_conn, isert_cmd,
>>>> -			   &isert_cmd->tx_desc.send_wr, true);
>>>> +	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
>>>> +		/*
>>>> +		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
>>>> +		 */
>>>> +		isert_create_send_desc(isert_conn, isert_cmd,
>>>> +				       &isert_cmd->tx_desc);
>>>> +		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
>>>> +				     &isert_cmd->tx_desc.iscsi_header);
>>>> +		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
>>>> +		isert_init_send_wr(isert_conn, isert_cmd,
>>>> +				   &isert_cmd->tx_desc.send_wr, true);
>>>> +		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
>>>> +	}
>>>>    
>>> So I'm fixing up some v3.14-rc6 context changes before applying this
>>> series to for-next, and noticed something above..
>>>
>>> Namely, that iscsi_build_rsp_pdu() has been changed to pass 'false' as
>>> it's second argument vs. the original case passing 'true'.  This boolean
>>> controls the incrementing of the StatSN in the traditional response
>>> header, which AFAICT should still be 'true' for this case.
>>>
>>> Is there a reason why this was changed..?
>> Well, it wrong... This probably was left around when I rebased to 3.14
>> (earlier versions incremented StatSN explicitly).
>> I wonder how that works though...
>>
>> Want me to fix and resend?
>>
> Nah, fixing this in for-next now.
>
> --nab
>

Thanks,

I just spotted something strange, I think it was already fixed sometime:

-	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {

+	if (se_cmd->prot_op == TARGET_PROT_NORMAL) {

This is strange that I find a mismatch in our trees.
I'll check it out on Sunday.

Sagi.

Re: [PATCH v2 12/12] IB/isert: Support T10-PI protected transactions

[Nicholas A. Bellinger]

On Thu, 2014-03-13 at 21:14 +0200, Sagi Grimberg wrote:
> On 3/13/2014 8:59 PM, Nicholas A. Bellinger wrote:
> > On Thu, 2014-03-13 at 20:57 +0200, Sagi Grimberg wrote:
> >> On 3/13/2014 8:15 PM, Nicholas A. Bellinger wrote:
> >>> Hey Sagi,
> >>>
> >>> On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
> >>>> In case the Target core passed transport T10 protection
> >>>> operation:
> >>>>
> >>>> 1. Register data buffer (data memory region)
> >>>> 2. Register protection buffer if exsists (prot memory region)
> >>>> 3. Register signature region (signature memory region)
> >>>>      - use work request IB_WR_REG_SIG_MR
> >>>> 4. Execute RDMA
> >>>> 5. Upon RDMA completion check the signature status
> >>>>      - if succeeded send good SCSI response
> >>>>      - if failed send SCSI bad response with appropriate sense buffer
> >>>>
> >>>> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> >>>> ---
> >>>>    drivers/infiniband/ulp/isert/ib_isert.c |  321 ++++++++++++++++++++++++++++---
> >>>>    drivers/infiniband/ulp/isert/ib_isert.h |    1 +
> >>>>    2 files changed, 292 insertions(+), 30 deletions(-)
> >>>>
> >>>> diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
> >>>> index 3871ab2..04bdd79 100644
> >>>> --- a/drivers/infiniband/ulp/isert/ib_isert.c
> >>>> +++ b/drivers/infiniband/ulp/isert/ib_isert.c
> >>> <SNIP>
> >>>
> >>>> @@ -2454,26 +2708,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
> >>>>    		return rc;
> >>>>    	}
> >>>>    
> >>>> -	/*
> >>>> -	 * Build isert_conn->tx_desc for iSCSI response PDU and attach
> >>>> -	 */
> >>>> -	isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
> >>>> -	iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
> >>>> -			     &isert_cmd->tx_desc.iscsi_header);
> >>>> -	isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
> >>>> -	isert_init_send_wr(isert_conn, isert_cmd,
> >>>> -			   &isert_cmd->tx_desc.send_wr, true);
> >>>> +	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
> >>>> +		/*
> >>>> +		 * Build isert_conn->tx_desc for iSCSI response PDU and attach
> >>>> +		 */
> >>>> +		isert_create_send_desc(isert_conn, isert_cmd,
> >>>> +				       &isert_cmd->tx_desc);
> >>>> +		iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
> >>>> +				     &isert_cmd->tx_desc.iscsi_header);
> >>>> +		isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
> >>>> +		isert_init_send_wr(isert_conn, isert_cmd,
> >>>> +				   &isert_cmd->tx_desc.send_wr, true);
> >>>> +		isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
> >>>> +	}
> >>>>    
> >>> So I'm fixing up some v3.14-rc6 context changes before applying this
> >>> series to for-next, and noticed something above..
> >>>
> >>> Namely, that iscsi_build_rsp_pdu() has been changed to pass 'false' as
> >>> it's second argument vs. the original case passing 'true'.  This boolean
> >>> controls the incrementing of the StatSN in the traditional response
> >>> header, which AFAICT should still be 'true' for this case.
> >>>
> >>> Is there a reason why this was changed..?
> >> Well, it wrong... This probably was left around when I rebased to 3.14
> >> (earlier versions incremented StatSN explicitly).
> >> I wonder how that works though...
> >>
> >> Want me to fix and resend?
> >>
> > Nah, fixing this in for-next now.
> >
> > --nab
> >
> 
> Thanks,
> 
> I just spotted something strange, I think it was already fixed sometime:
> 
> -	if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
> 
> +	if (se_cmd->prot_op == TARGET_PROT_NORMAL) {
> 
> This is strange that I find a mismatch in our trees.
> I'll check it out on Sunday.
> 

Mmmm, yeah.  I thought that was fixed a while ago, but is still present
in the -v2 patches from 02192014..

In any event, those are now merged in for-next, and should be included
in tomorrow's next build.

--nab

Re: [PATCH v2 00/12] T10-DIF Initial support for iSER target

[Nicholas A. Bellinger]

On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
> Hey Nic,
> 
> I addressed your comments in the this set. I'll try to send the
> initiator code for review in the next couple of days.
> 
> This patchset introduces target side T10-PI offload support over
> RDMA. Currently the implementation is for iSER transport but can
> be easily extended to SRP (or FCoE in the future).
> 
> Should mention that this patchset depends on RDMA signature
> verbs making it for inclusion which will hopefully happen in
> near future.
> 
> This code was tested against iSER legacy initiator, and also
> initiator that uses T10-PI offload as well. I'll clean up the
> initiator code in the following days and submit that as well.
> 
> This code works under medium loads using backstores:
> - FileIO with DIF emulation.
> - RD with DIF emulation.
> - iBLOCK (scsi_debug with DIF support).
> 
> Chnages from v1:
> - Rebased rdma_dif (3.14-rc2)
> - Target core:
>   - Pass zero_flag=true to alloc_sgl for protection buffers
>   - Removed Unneeded inline function rwprotect
>   - Pass is_write bool to sbc_set_prot_op_checks
>   - Send failure response for absence of protection buffers
>     when needed.
> - iSER:
>   - Removed unneeded assignments in isert_set_sig_attrs
> 
> Chnages from v0:
> - Rebased from for-next
> - Target core:
>   - Don't minor fixes for check_prot - to support transports
>     that doesn't use submit_map_sgls.
>   - file  format - use escape values.
>   - Removed redundant prot_handover.
>   - Added protection checks and operation set.
> - iSER:
>   - Added preperation routines for mapping/unmapping buffers
>     to ease the amount of code in isert_reg_rdma.
>   - Fixed print of DIF error (sector instead of offset).
>   - Fix RDMA length for protection on wire domain.
>   - Refactored reg_sig_mr to use helper routines.
> 

Hey Sagi,

FYI, this series did not compile:

drivers/infiniband/ulp/isert/ib_isert.c: In function ‘isert_reg_sig_mr’:
drivers/infiniband/ulp/isert/ib_isert.c:2531:2: error: expected ‘;’ before ‘ret’
drivers/infiniband/ulp/isert/ib_isert.c: In function ‘isert_reg_rdma’:
drivers/infiniband/ulp/isert/ib_isert.c:2611:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
drivers/infiniband/ulp/isert/ib_isert.c:2625:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
drivers/infiniband/ulp/isert/ib_isert.c:2667:43: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
drivers/infiniband/ulp/isert/ib_isert.c: In function ‘isert_put_datain’:
drivers/infiniband/ulp/isert/ib_isert.c:2711:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
drivers/infiniband/ulp/isert/ib_isert.c:2732:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
drivers/infiniband/ulp/isert/ib_isert.c: At top level:
drivers/infiniband/ulp/isert/ib_isert.c:2468:1: warning: ‘isert_set_sig_attrs’ defined but not used [-Wunused-function]
drivers/target/target_core_transport.c: In function ‘transport_generic_new_cmd’:
drivers/target/target_core_transport.c:2221:22: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]

Squashing the following patch(es) into your original series, and applied
to target-pending/rdma-dif.

Thank you,

--nab

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.
index 04bdd79..b7c8cd7 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -2527,7 +2527,7 @@ isert_reg_sig_mr(struct isert_conn *isert_conn, struct se_cmd *se_cmd,
        int ret;
        u32 key;
 
-       memset(&sig_attrs, 0, sizeof(sig_attrs))
+       memset(&sig_attrs, 0, sizeof(sig_attrs));
        ret = isert_set_sig_attrs(se_cmd, &sig_attrs);
        if (ret)
                goto err;
@@ -2608,7 +2608,7 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
                return ret;
 
        if (wr->data.dma_nents != 1 ||
-           se_cmd->prot_type != TARGET_PROT_NORMAL) {
+           se_cmd->prot_op != TARGET_PROT_NORMAL) {
                spin_lock_irqsave(&isert_conn->conn_lock, flags);
                fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
                                           struct fast_reg_descriptor, list);
@@ -2622,7 +2622,7 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
        if (ret)
                goto unmap_cmd;
 
-       if (se_cmd->prot_type != TARGET_PROT_NORMAL) {
+       if (se_cmd->prot_op != TARGET_PROT_NORMAL) {
                struct ib_sge prot_sge, sig_sge;
 
                if (se_cmd->t_prot_sg) {
@@ -2664,7 +2664,7 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
                send_wr->opcode = IB_WR_RDMA_WRITE;
                send_wr->wr.rdma.remote_addr = isert_cmd->read_va;
                send_wr->wr.rdma.rkey = isert_cmd->read_stag;
-               send_wr->send_flags = se_cmd->prot_type == TARGET_PROT_NORMAL ?
+               send_wr->send_flags = se_cmd->prot_op == TARGET_PROT_NORMAL ?
                                      0 : IB_SEND_SIGNALED;
        } else {
                send_wr->opcode = IB_WR_RDMA_READ;
@@ -2708,7 +2708,7 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
                return rc;
        }
 
-       if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
+       if (se_cmd->prot_op == TARGET_PROT_NORMAL) {
                /*
                 * Build isert_conn->tx_desc for iSCSI response PDU and attach
                 */
@@ -2729,7 +2729,7 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
                atomic_dec(&isert_conn->post_send_buf_count);
        }
 
-       if (se_cmd->prot_type == TARGET_PROT_NORMAL)
+       if (se_cmd->prot_op == TARGET_PROT_NORMAL)
                pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data "
                         "READ\n", isert_cmd);
        else
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index a45d628..954e7ed 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -2218,7 +2218,7 @@ transport_generic_new_cmd(struct se_cmd *cmd)
                                return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
                }
 
-               if (cmd->prot_type != TARGET_PROT_NORMAL) {
+               if (cmd->prot_op != TARGET_PROT_NORMAL) {
                        ret = target_alloc_sgl(&cmd->t_prot_sg,
                                               &cmd->t_prot_nents,
                                               cmd->prot_length, true);

Re: [PATCH v2 00/12] T10-DIF Initial support for iSER target

[Sagi Grimberg]

On 2/20/2014 1:14 AM, Nicholas A. Bellinger wrote:
> On Wed, 2014-02-19 at 17:50 +0200, Sagi Grimberg wrote:
>> Hey Nic,
>>
>> I addressed your comments in the this set. I'll try to send the
>> initiator code for review in the next couple of days.
>>
>> This patchset introduces target side T10-PI offload support over
>> RDMA. Currently the implementation is for iSER transport but can
>> be easily extended to SRP (or FCoE in the future).
>>
>> Should mention that this patchset depends on RDMA signature
>> verbs making it for inclusion which will hopefully happen in
>> near future.
>>
>> This code was tested against iSER legacy initiator, and also
>> initiator that uses T10-PI offload as well. I'll clean up the
>> initiator code in the following days and submit that as well.
>>
>> This code works under medium loads using backstores:
>> - FileIO with DIF emulation.
>> - RD with DIF emulation.
>> - iBLOCK (scsi_debug with DIF support).
>>
>> Chnages from v1:
>> - Rebased rdma_dif (3.14-rc2)
>> - Target core:
>>    - Pass zero_flag=true to alloc_sgl for protection buffers
>>    - Removed Unneeded inline function rwprotect
>>    - Pass is_write bool to sbc_set_prot_op_checks
>>    - Send failure response for absence of protection buffers
>>      when needed.
>> - iSER:
>>    - Removed unneeded assignments in isert_set_sig_attrs
>>
>> Chnages from v0:
>> - Rebased from for-next
>> - Target core:
>>    - Don't minor fixes for check_prot - to support transports
>>      that doesn't use submit_map_sgls.
>>    - file  format - use escape values.
>>    - Removed redundant prot_handover.
>>    - Added protection checks and operation set.
>> - iSER:
>>    - Added preperation routines for mapping/unmapping buffers
>>      to ease the amount of code in isert_reg_rdma.
>>    - Fixed print of DIF error (sector instead of offset).
>>    - Fix RDMA length for protection on wire domain.
>>    - Refactored reg_sig_mr to use helper routines.
>>
> Hey Sagi,
>
> FYI, this series did not compile:
>
> drivers/infiniband/ulp/isert/ib_isert.c: In function ‘isert_reg_sig_mr’:
> drivers/infiniband/ulp/isert/ib_isert.c:2531:2: error: expected ‘;’ before ‘ret’
> drivers/infiniband/ulp/isert/ib_isert.c: In function ‘isert_reg_rdma’:
> drivers/infiniband/ulp/isert/ib_isert.c:2611:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
> drivers/infiniband/ulp/isert/ib_isert.c:2625:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
> drivers/infiniband/ulp/isert/ib_isert.c:2667:43: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
> drivers/infiniband/ulp/isert/ib_isert.c: In function ‘isert_put_datain’:
> drivers/infiniband/ulp/isert/ib_isert.c:2711:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
> drivers/infiniband/ulp/isert/ib_isert.c:2732:24: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
> drivers/infiniband/ulp/isert/ib_isert.c: At top level:
> drivers/infiniband/ulp/isert/ib_isert.c:2468:1: warning: ‘isert_set_sig_attrs’ defined but not used [-Wunused-function]
> drivers/target/target_core_transport.c: In function ‘transport_generic_new_cmd’:
> drivers/target/target_core_transport.c:2221:22: warning: comparison between ‘enum target_prot_type’ and ‘enum target_prot_op’ [-Wenum-compare]
>
> Squashing the following patch(es) into your original series, and applied
> to target-pending/rdma-dif.

Ohhh, last second cosmetics...

Thanks for fixing it, won't happen again...

> Thank you,

Thank you...

> --nab
>