[PATCH 0/5] scsi: target: Allow userspace to config cmd submission

[PATCH 0/5] scsi: target: Allow userspace to config cmd submission

[Mike Christie]

The following patches were made over Linus's tree but apply over Martin's
branches. They allow userspace to configure how fabric drivers submit cmds
to backend drivers.

Right now loop and vhost use a worker thread, and the other drivers submit
from the contexts they receive/process the cmd from. For multiple LUN
cases where the target can queue more cmds than the backend can handle
then defering to a worker thread is safest because the backend driver can
block when doing things like waiting for a free request/tag. For cases
where the backend devices can queue everything the target sends, then
there is no need to defer to a workqueue and you can see a perf boost of
up to 26% for small IO workloads. For a nvme device and vhost-scsi I can
see with 4K IOs:

fio jobs        1       2       4       8       10
--------------------------------------------------
workqueue
submit           94K     190K    394K    770K    890K

direct
submit          128K    252K    488K    950K    -

[PATCH 1/5] scsi: target: Make write_pending_must_be_called a bit field

[Mike Christie]

The next patches add more on/off type of settings to the
target_core_fabric_ops struct so this makes write_pending_must_be_called
a bit field instead of a bool to better organize the settings.

Signed-off-by: Mike Christie <michael.christie@oracle.com>
---
 drivers/target/iscsi/iscsi_target_configfs.c | 2 +-
 include/target/target_core_fabric.h          | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index 5d0f51822414..620a04e23be1 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -1591,5 +1591,5 @@ const struct target_core_fabric_ops iscsi_ops = {
 	.tfc_tpg_nacl_auth_attrs	= lio_target_nacl_auth_attrs,
 	.tfc_tpg_nacl_param_attrs	= lio_target_nacl_param_attrs,
 
-	.write_pending_must_be_called	= true,
+	.write_pending_must_be_called	= 1,
 };
diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
index b188b1e90e1e..2a6c4c935666 100644
--- a/include/target/target_core_fabric.h
+++ b/include/target/target_core_fabric.h
@@ -113,11 +113,11 @@ struct target_core_fabric_ops {
 	struct configfs_attribute **tfc_tpg_nacl_param_attrs;
 
 	/*
-	 * Set this member variable to true if the SCSI transport protocol
+	 * Set this member variable if the SCSI transport protocol
 	 * (e.g. iSCSI) requires that the Data-Out buffer is transferred in
 	 * its entirety before a command is aborted.
 	 */
-	bool write_pending_must_be_called;
+	unsigned int write_pending_must_be_called:1;
 };
 
 int target_register_template(const struct target_core_fabric_ops *fo);
-- 
2.34.1

[PATCH 2/5] scsi: target: Have drivers report if they support direct submissions

[Mike Christie]

In some cases, like with multiple LUN targets, it can be better to defer
the cmd submission to a helper thread, because if the backend driver
blocks on something like request/tag allocation it can block the entire
submission path for all LUNs. In other cases like single LUN targets
with fast storage that can support all the commands that the target can
queue, then it's best to submit the cmd to the backend from the target's
cmd recv context.

The next patch will allow the user to config what they prefer, but
drivers like loop can't directly submit because they can be called from a
context that can't sleep. And, drivers like vhost-scsi can support direct
submission, but need to keep their default behavior of deferring
execution to avoid perf regressions for the multiple LUN case.

This patch has the drivers tell LIO core if they support direct
submissions and their current default, so in the next patch we can
prevent users from misconfiguring the system and initialize devices
correctly.

Signed-off-by: Mike Christie <michael.christie@oracle.com>
---
 drivers/infiniband/ulp/srpt/ib_srpt.c    |  3 +++
 drivers/scsi/elx/efct/efct_lio.c         |  5 +++++
 drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c |  3 +++
 drivers/scsi/qla2xxx/tcm_qla2xxx.c       |  6 ++++++
 drivers/target/loopback/tcm_loop.c       |  1 +
 drivers/target/sbp/sbp_target.c          |  3 +++
 drivers/target/tcm_fc/tfc_conf.c         |  3 +++
 drivers/usb/gadget/function/f_tcm.c      |  3 +++
 drivers/vhost/scsi.c                     |  3 +++
 drivers/xen/xen-scsiback.c               |  3 +++
 include/target/target_core_fabric.h      | 10 ++++++++++
 11 files changed, 43 insertions(+)

diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index c12005eab14c..a7e21d0cca0f 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -3867,6 +3867,9 @@ static const struct target_core_fabric_ops srpt_template = {
 	.tfc_discovery_attrs		= srpt_da_attrs,
 	.tfc_wwn_attrs			= srpt_wwn_attrs,
 	.tfc_tpg_attrib_attrs		= srpt_tpg_attrib_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 /**
diff --git a/drivers/scsi/elx/efct/efct_lio.c b/drivers/scsi/elx/efct/efct_lio.c
index a982b9cf9870..b7df6fadae9c 100644
--- a/drivers/scsi/elx/efct/efct_lio.c
+++ b/drivers/scsi/elx/efct/efct_lio.c
@@ -1611,6 +1611,8 @@ static const struct target_core_fabric_ops efct_lio_ops = {
 	.sess_get_initiator_sid		= NULL,
 	.tfc_tpg_base_attrs		= efct_lio_tpg_attrs,
 	.tfc_tpg_attrib_attrs           = efct_lio_tpg_attrib_attrs,
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static const struct target_core_fabric_ops efct_lio_npiv_ops = {
@@ -1646,6 +1648,9 @@ static const struct target_core_fabric_ops efct_lio_npiv_ops = {
 	.sess_get_initiator_sid		= NULL,
 	.tfc_tpg_base_attrs		= efct_lio_npiv_tpg_attrs,
 	.tfc_tpg_attrib_attrs		= efct_lio_npiv_tpg_attrib_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 int efct_scsi_tgt_driver_init(void)
diff --git a/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c b/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c
index 385f812b8793..d345cf14196e 100644
--- a/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c
+++ b/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c
@@ -3975,6 +3975,9 @@ static const struct target_core_fabric_ops ibmvscsis_ops = {
 	.fabric_drop_tpg		= ibmvscsis_drop_tpg,
 
 	.tfc_wwn_attrs			= ibmvscsis_wwn_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static void ibmvscsis_dev_release(struct device *dev) {};
diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
index 3b5ba4b47b3b..a005eb4e347a 100644
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
@@ -1822,6 +1822,9 @@ static const struct target_core_fabric_ops tcm_qla2xxx_ops = {
 	.tfc_wwn_attrs			= tcm_qla2xxx_wwn_attrs,
 	.tfc_tpg_base_attrs		= tcm_qla2xxx_tpg_attrs,
 	.tfc_tpg_attrib_attrs		= tcm_qla2xxx_tpg_attrib_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static const struct target_core_fabric_ops tcm_qla2xxx_npiv_ops = {
@@ -1859,6 +1862,9 @@ static const struct target_core_fabric_ops tcm_qla2xxx_npiv_ops = {
 	.fabric_init_nodeacl		= tcm_qla2xxx_init_nodeacl,
 
 	.tfc_wwn_attrs			= tcm_qla2xxx_wwn_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static int tcm_qla2xxx_register_configfs(void)
diff --git a/drivers/target/loopback/tcm_loop.c b/drivers/target/loopback/tcm_loop.c
index 4ec99a55ac30..75804303b175 100644
--- a/drivers/target/loopback/tcm_loop.c
+++ b/drivers/target/loopback/tcm_loop.c
@@ -1102,6 +1102,7 @@ static const struct target_core_fabric_ops loop_ops = {
 	.tfc_wwn_attrs			= tcm_loop_wwn_attrs,
 	.tfc_tpg_base_attrs		= tcm_loop_tpg_attrs,
 	.tfc_tpg_attrib_attrs		= tcm_loop_tpg_attrib_attrs,
+	.direct_submit_supp		= 0,
 };
 
 static int __init tcm_loop_fabric_init(void)
diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
index 2a761bc09193..1cac139af3e3 100644
--- a/drivers/target/sbp/sbp_target.c
+++ b/drivers/target/sbp/sbp_target.c
@@ -2278,6 +2278,9 @@ static const struct target_core_fabric_ops sbp_ops = {
 	.tfc_wwn_attrs			= sbp_wwn_attrs,
 	.tfc_tpg_base_attrs		= sbp_tpg_base_attrs,
 	.tfc_tpg_attrib_attrs		= sbp_tpg_attrib_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static int __init sbp_init(void)
diff --git a/drivers/target/tcm_fc/tfc_conf.c b/drivers/target/tcm_fc/tfc_conf.c
index 6ac3fc1a7d39..e66fd7131da5 100644
--- a/drivers/target/tcm_fc/tfc_conf.c
+++ b/drivers/target/tcm_fc/tfc_conf.c
@@ -432,6 +432,9 @@ static const struct target_core_fabric_ops ft_fabric_ops = {
 
 	.tfc_wwn_attrs			= ft_wwn_attrs,
 	.tfc_tpg_nacl_base_attrs	= ft_nacl_base_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static struct notifier_block ft_notifier = {
diff --git a/drivers/usb/gadget/function/f_tcm.c b/drivers/usb/gadget/function/f_tcm.c
index 79ed2e6e576a..eb618ff72165 100644
--- a/drivers/usb/gadget/function/f_tcm.c
+++ b/drivers/usb/gadget/function/f_tcm.c
@@ -1687,6 +1687,9 @@ static const struct target_core_fabric_ops usbg_ops = {
 
 	.tfc_wwn_attrs			= usbg_wwn_attrs,
 	.tfc_tpg_base_attrs		= usbg_base_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 /* Start gadget.c code */
diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c
index c83f7f043470..ed6cc84c0b85 100644
--- a/drivers/vhost/scsi.c
+++ b/drivers/vhost/scsi.c
@@ -2461,6 +2461,9 @@ static const struct target_core_fabric_ops vhost_scsi_ops = {
 	.tfc_wwn_attrs			= vhost_scsi_wwn_attrs,
 	.tfc_tpg_base_attrs		= vhost_scsi_tpg_attrs,
 	.tfc_tpg_attrib_attrs		= vhost_scsi_tpg_attrib_attrs,
+
+	.default_direct_submit		= 0,
+	.direct_submit_supp		= 1,
 };
 
 static int __init vhost_scsi_init(void)
diff --git a/drivers/xen/xen-scsiback.c b/drivers/xen/xen-scsiback.c
index 8b77e4c06e43..fc41f936b84b 100644
--- a/drivers/xen/xen-scsiback.c
+++ b/drivers/xen/xen-scsiback.c
@@ -1832,6 +1832,9 @@ static const struct target_core_fabric_ops scsiback_ops = {
 	.tfc_wwn_attrs			= scsiback_wwn_attrs,
 	.tfc_tpg_base_attrs		= scsiback_tpg_attrs,
 	.tfc_tpg_param_attrs		= scsiback_param_attrs,
+
+	.default_direct_submit		= 1,
+	.direct_submit_supp		= 1,
 };
 
 static const struct xenbus_device_id scsiback_ids[] = {
diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
index 2a6c4c935666..5b2367c0839e 100644
--- a/include/target/target_core_fabric.h
+++ b/include/target/target_core_fabric.h
@@ -118,6 +118,16 @@ struct target_core_fabric_ops {
 	 * its entirety before a command is aborted.
 	 */
 	unsigned int write_pending_must_be_called:1;
+	/*
+	 * Set this if the driver supports submitting commands to the backend
+	 * from target_submit/target_submit_cmd.
+	 */
+	unsigned int direct_submit_supp:1;
+	/*
+	 * Set this if the driver wants to default to directly submitting
+	 * commands.
+	 */
+	unsigned int default_direct_submit:1;
 };
 
 int target_register_template(const struct target_core_fabric_ops *fo);
-- 
2.34.1

[PATCH 3/5] scsi: target: Allow userspace to request direct submissions

[Mike Christie]

This allows userspace to request the fabric drivers do direct submissions
if they support it.

When using a nvme drive and vhost-scsi we see around a 20% improvement
in 4K IOs:

fio jobs        1       2       4       8       10
--------------------------------------------------
defer           94K     190K    394K    770K    890K
direct          128K    252K    488K    950K    -

Signed-off-by: Mike Christie <michael.christie@oracle.com>
---
 drivers/target/loopback/tcm_loop.c     |  2 +-
 drivers/target/target_core_configfs.c  |  5 ++++
 drivers/target/target_core_device.c    |  1 +
 drivers/target/target_core_transport.c | 33 +++++++++++++++++++++++---
 drivers/vhost/scsi.c                   |  2 +-
 include/target/target_core_base.h      |  5 ++++
 6 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/drivers/target/loopback/tcm_loop.c b/drivers/target/loopback/tcm_loop.c
index 75804303b175..fbbf80ae9766 100644
--- a/drivers/target/loopback/tcm_loop.c
+++ b/drivers/target/loopback/tcm_loop.c
@@ -154,7 +154,7 @@ static void tcm_loop_target_queue_cmd(struct tcm_loop_cmd *tl_cmd)
 			       GFP_ATOMIC))
 		return;
 
-	target_queue_submission(se_cmd);
+	target_submit(se_cmd);
 	return;
 
 out_done:
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 936e5ff1b209..e4ddd8882f83 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -577,6 +577,7 @@ DEF_CONFIGFS_ATTRIB_SHOW(unmap_granularity_alignment);
 DEF_CONFIGFS_ATTRIB_SHOW(unmap_zeroes_data);
 DEF_CONFIGFS_ATTRIB_SHOW(max_write_same_len);
 DEF_CONFIGFS_ATTRIB_SHOW(emulate_rsoc);
+DEF_CONFIGFS_ATTRIB_SHOW(direct_submit);
 
 #define DEF_CONFIGFS_ATTRIB_STORE_U32(_name)				\
 static ssize_t _name##_store(struct config_item *item, const char *page,\
@@ -598,6 +599,7 @@ DEF_CONFIGFS_ATTRIB_STORE_U32(max_unmap_block_desc_count);
 DEF_CONFIGFS_ATTRIB_STORE_U32(unmap_granularity);
 DEF_CONFIGFS_ATTRIB_STORE_U32(unmap_granularity_alignment);
 DEF_CONFIGFS_ATTRIB_STORE_U32(max_write_same_len);
+DEF_CONFIGFS_ATTRIB_STORE_U32(direct_submit);
 
 #define DEF_CONFIGFS_ATTRIB_STORE_BOOL(_name)				\
 static ssize_t _name##_store(struct config_item *item, const char *page,	\
@@ -1266,6 +1268,7 @@ CONFIGFS_ATTR(, unmap_zeroes_data);
 CONFIGFS_ATTR(, max_write_same_len);
 CONFIGFS_ATTR(, alua_support);
 CONFIGFS_ATTR(, pgr_support);
+CONFIGFS_ATTR(, direct_submit);
 
 /*
  * dev_attrib attributes for devices using the target core SBC/SPC
@@ -1308,6 +1311,7 @@ struct configfs_attribute *sbc_attrib_attrs[] = {
 	&attr_alua_support,
 	&attr_pgr_support,
 	&attr_emulate_rsoc,
+	&attr_direct_submit,
 	NULL,
 };
 EXPORT_SYMBOL(sbc_attrib_attrs);
@@ -1325,6 +1329,7 @@ struct configfs_attribute *passthrough_attrib_attrs[] = {
 	&attr_emulate_pr,
 	&attr_alua_support,
 	&attr_pgr_support,
+	&attr_direct_submit,
 	NULL,
 };
 EXPORT_SYMBOL(passthrough_attrib_attrs);
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index b7ac60f4a219..b4c3e17ebfc9 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -779,6 +779,7 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	dev->dev_attrib.unmap_zeroes_data =
 				DA_UNMAP_ZEROES_DATA_DEFAULT;
 	dev->dev_attrib.max_write_same_len = DA_MAX_WRITE_SAME_LEN;
+	dev->dev_attrib.direct_submit = DA_FABRIC_DEFAULT_SUBMIT;
 
 	xcopy_lun = &dev->xcopy_lun;
 	rcu_assign_pointer(xcopy_lun->lun_se_dev, dev);
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 687adc9e086c..5349a2dd8187 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -1781,13 +1781,13 @@ int target_submit_prep(struct se_cmd *se_cmd, unsigned char *cdb,
 EXPORT_SYMBOL_GPL(target_submit_prep);
 
 /**
- * target_submit - perform final initialization and submit cmd to LIO core
+ * __target_submit - perform final initialization and submit cmd to LIO core
  * @se_cmd: command descriptor to submit
  *
  * target_submit_prep must have been called on the cmd, and this must be
  * called from process context.
  */
-void target_submit(struct se_cmd *se_cmd)
+static void __target_submit(struct se_cmd *se_cmd)
 {
 	struct scatterlist *sgl = se_cmd->t_data_sg;
 	unsigned char *buf = NULL;
@@ -1825,6 +1825,33 @@ void target_submit(struct se_cmd *se_cmd)
 
 	transport_handle_cdb_direct(se_cmd);
 }
+
+/**
+ * target_submit - submit cmd to LIO core or queue it's submission
+ * @se_cmd: command descriptor to submit
+ */
+void target_submit(struct se_cmd *se_cmd)
+{
+	struct se_dev_attrib *da = &se_cmd->se_dev->dev_attrib;
+	u32 direct_submit;
+
+	if (da->direct_submit == DA_FABRIC_DEFAULT_SUBMIT) {
+		if (se_cmd->se_sess->se_tpg->se_tpg_tfo->default_direct_submit)
+			direct_submit = DA_DIRECT_SUBMIT;
+		else
+			direct_submit = DA_QUEUE_SUBMIT;
+	} else if (da->direct_submit == DA_DIRECT_SUBMIT &&
+		   se_cmd->se_sess->se_tpg->se_tpg_tfo->direct_submit_supp) {
+		direct_submit = DA_DIRECT_SUBMIT;
+	} else {
+		direct_submit = DA_QUEUE_SUBMIT;
+	}
+
+	if (direct_submit == DA_DIRECT_SUBMIT)
+		__target_submit(se_cmd);
+	else
+		target_queue_submission(se_cmd);
+}
 EXPORT_SYMBOL_GPL(target_submit);
 
 /**
@@ -1922,7 +1949,7 @@ void target_queued_submit_work(struct work_struct *work)
 			se_plug = target_plug_device(se_dev);
 		}
 
-		target_submit(se_cmd);
+		__target_submit(se_cmd);
 	}
 
 	if (se_plug)
diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c
index ed6cc84c0b85..e2603eb1b638 100644
--- a/drivers/vhost/scsi.c
+++ b/drivers/vhost/scsi.c
@@ -772,7 +772,7 @@ static void vhost_scsi_target_queue_cmd(struct vhost_scsi_cmd *cmd)
 			       cmd->tvc_prot_sgl_count, GFP_KERNEL))
 		return;
 
-	target_queue_submission(se_cmd);
+	target_submit(se_cmd);
 }
 
 static void
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 159567359bbb..4f37324ada5d 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -103,6 +103,10 @@
 #define DA_IS_NONROT				0
 /* Queue Algorithm Modifier default for restricted reordering in control mode page */
 #define DA_EMULATE_REST_REORD			0
+/* Command submission settings */
+#define DA_FABRIC_DEFAULT_SUBMIT		0
+#define DA_DIRECT_SUBMIT			1
+#define DA_QUEUE_SUBMIT				2
 
 #define SE_INQUIRY_BUF				1024
 #define SE_MODE_PAGE_BUF			512
@@ -717,6 +721,7 @@ struct se_dev_attrib {
 	u32		unmap_granularity;
 	u32		unmap_granularity_alignment;
 	u32		max_write_same_len;
+	u32		direct_submit;
 	struct se_device *da_dev;
 	struct config_group da_group;
 };
-- 
2.34.1

[PATCH 4/5] scsi: target: Unexport target_queue_submission

[Mike Christie]

target_queue_submission is not called by drivers anymore so unexport it.
This moves target_submit to before target_queue_submission so we can
easily call it. It does not change any behavior.

Signed-off-by: Mike Christie <michael.christie@oracle.com>
---
 drivers/target/target_core_transport.c | 59 +++++++++++++-------------
 include/target/target_core_fabric.h    |  1 -
 2 files changed, 29 insertions(+), 31 deletions(-)

diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 5349a2dd8187..701ca3593599 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -1826,34 +1826,6 @@ static void __target_submit(struct se_cmd *se_cmd)
 	transport_handle_cdb_direct(se_cmd);
 }
 
-/**
- * target_submit - submit cmd to LIO core or queue it's submission
- * @se_cmd: command descriptor to submit
- */
-void target_submit(struct se_cmd *se_cmd)
-{
-	struct se_dev_attrib *da = &se_cmd->se_dev->dev_attrib;
-	u32 direct_submit;
-
-	if (da->direct_submit == DA_FABRIC_DEFAULT_SUBMIT) {
-		if (se_cmd->se_sess->se_tpg->se_tpg_tfo->default_direct_submit)
-			direct_submit = DA_DIRECT_SUBMIT;
-		else
-			direct_submit = DA_QUEUE_SUBMIT;
-	} else if (da->direct_submit == DA_DIRECT_SUBMIT &&
-		   se_cmd->se_sess->se_tpg->se_tpg_tfo->direct_submit_supp) {
-		direct_submit = DA_DIRECT_SUBMIT;
-	} else {
-		direct_submit = DA_QUEUE_SUBMIT;
-	}
-
-	if (direct_submit == DA_DIRECT_SUBMIT)
-		__target_submit(se_cmd);
-	else
-		target_queue_submission(se_cmd);
-}
-EXPORT_SYMBOL_GPL(target_submit);
-
 /**
  * target_submit_cmd - lookup unpacked lun and submit uninitialized se_cmd
  *
@@ -1960,7 +1932,7 @@ void target_queued_submit_work(struct work_struct *work)
  * target_queue_submission - queue the cmd to run on the LIO workqueue
  * @se_cmd: command descriptor to submit
  */
-void target_queue_submission(struct se_cmd *se_cmd)
+static void target_queue_submission(struct se_cmd *se_cmd)
 {
 	struct se_device *se_dev = se_cmd->se_dev;
 	int cpu = se_cmd->cpuid;
@@ -1970,7 +1942,34 @@ void target_queue_submission(struct se_cmd *se_cmd)
 	llist_add(&se_cmd->se_cmd_list, &sq->cmd_list);
 	queue_work_on(cpu, target_submission_wq, &sq->work);
 }
-EXPORT_SYMBOL_GPL(target_queue_submission);
+
+/**
+ * target_submit - submit cmd to LIO core or queue it's submission
+ * @se_cmd: command descriptor to submit
+ */
+void target_submit(struct se_cmd *se_cmd)
+{
+	struct se_dev_attrib *da = &se_cmd->se_dev->dev_attrib;
+	u32 direct_submit;
+
+	if (da->direct_submit == DA_FABRIC_DEFAULT_SUBMIT) {
+		if (se_cmd->se_sess->se_tpg->se_tpg_tfo->default_direct_submit)
+			direct_submit = DA_DIRECT_SUBMIT;
+		else
+			direct_submit = DA_QUEUE_SUBMIT;
+	} else if (da->direct_submit == DA_DIRECT_SUBMIT &&
+		   se_cmd->se_sess->se_tpg->se_tpg_tfo->direct_submit_supp) {
+		direct_submit = DA_DIRECT_SUBMIT;
+	} else {
+		direct_submit = DA_QUEUE_SUBMIT;
+	}
+
+	if (direct_submit == DA_DIRECT_SUBMIT)
+		__target_submit(se_cmd);
+	else
+		target_queue_submission(se_cmd);
+}
+EXPORT_SYMBOL_GPL(target_submit);
 
 static void target_complete_tmr_failure(struct work_struct *work)
 {
diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
index 5b2367c0839e..77f69b75e851 100644
--- a/include/target/target_core_fabric.h
+++ b/include/target/target_core_fabric.h
@@ -183,7 +183,6 @@ sense_reason_t target_cmd_init_cdb(struct se_cmd *se_cmd, unsigned char *cdb,
 sense_reason_t target_cmd_parse_cdb(struct se_cmd *);
 void	target_submit_cmd(struct se_cmd *, struct se_session *, unsigned char *,
 		unsigned char *, u64, u32, int, int, int);
-void	target_queue_submission(struct se_cmd *se_cmd);
 
 int	target_submit_tmr(struct se_cmd *se_cmd, struct se_session *se_sess,
 		unsigned char *sense, u64 unpacked_lun,
-- 
2.34.1

[PATCH 5/5] scsi: target: Export fabric driver direct submit settings

[Mike Christie]

This exports the fabric driver's direct submit settings, so users know
what the driver supports. It will be helpful when they are exporting
a device through different targets and one doesn't support direct
submission.

Signed-off-by: Mike Christie <michael.christie@oracle.com>
---
 drivers/target/target_core_fabric_configfs.c | 22 ++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index b7c637644cd4..6fd6a9135dca 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -1065,8 +1065,30 @@ target_fabric_wwn_cmd_completion_affinity_store(struct config_item *item,
 }
 CONFIGFS_ATTR(target_fabric_wwn_, cmd_completion_affinity);
 
+static ssize_t
+target_fabric_wwn_default_to_direct_submit_show(struct config_item *item,
+						char *page)
+{
+	struct se_wwn *wwn = container_of(to_config_group(item), struct se_wwn,
+					  param_group);
+	return sprintf(page, "%u\n", wwn->wwn_tf->tf_ops->default_direct_submit);
+}
+CONFIGFS_ATTR_RO(target_fabric_wwn_, default_to_direct_submit);
+
+static ssize_t
+target_fabric_wwn_direct_submit_supported_show(struct config_item *item,
+					       char *page)
+{
+	struct se_wwn *wwn = container_of(to_config_group(item), struct se_wwn,
+					  param_group);
+	return sprintf(page, "%u\n", wwn->wwn_tf->tf_ops->direct_submit_supp);
+}
+CONFIGFS_ATTR_RO(target_fabric_wwn_, direct_submit_supported);
+
 static struct configfs_attribute *target_fabric_wwn_param_attrs[] = {
 	&target_fabric_wwn_attr_cmd_completion_affinity,
+	&target_fabric_wwn_attr_default_to_direct_submit,
+	&target_fabric_wwn_attr_direct_submit_supported,
 	NULL,
 };
 
-- 
2.34.1

Re: [PATCH 5/5] scsi: target: Export fabric driver direct submit settings

[Konstantin Shelekhin]

On Mon, Jul 10, 2023 at 04:44:42PM -0500, Mike Christie wrote:
> +static ssize_t
> +target_fabric_wwn_default_to_direct_submit_show(struct config_item *item,
> +                                               char *page)
> +{
> +       struct se_wwn *wwn = container_of(to_config_group(item), struct se_wwn,
> +                                         param_group);
> +       return sprintf(page, "%u\n", wwn->wwn_tf->tf_ops->default_direct_submit);

I belive we should do the right thing and use sysfs_emit() here. We
already have a bunch of issues with bad sprintf() usage and I think it
would be wise to promote safer interfaces.

> +static ssize_t
> +target_fabric_wwn_direct_submit_supported_show(struct config_item *item,
> +                                              char *page)
> +{
> +       struct se_wwn *wwn = container_of(to_config_group(item), struct se_wwn,
> +                                         param_group);
> +       return sprintf(page, "%u\n", wwn->wwn_tf->tf_ops->direct_submit_supp);

Same.

Re: [PATCH 5/5] scsi: target: Export fabric driver direct submit settings

[Mike Christie]

On 7/13/23 8:06 AM, Konstantin Shelekhin wrote:
> On Mon, Jul 10, 2023 at 04:44:42PM -0500, Mike Christie wrote:
>> +static ssize_t
>> +target_fabric_wwn_default_to_direct_submit_show(struct config_item *item,
>> +                                               char *page)
>> +{
>> +       struct se_wwn *wwn = container_of(to_config_group(item), struct se_wwn,
>> +                                         param_group);
>> +       return sprintf(page, "%u\n", wwn->wwn_tf->tf_ops->default_direct_submit);
> 
> I belive we should do the right thing and use sysfs_emit() here. We
> already have a bunch of issues with bad sprintf() usage and I think it
> would be wise to promote safer interfaces.

You're right. Forgot about that function. Will fix. Thanks.

> 
>> +static ssize_t
>> +target_fabric_wwn_direct_submit_supported_show(struct config_item *item,
>> +                                              char *page)
>> +{
>> +       struct se_wwn *wwn = container_of(to_config_group(item), struct se_wwn,
>> +                                         param_group);
>> +       return sprintf(page, "%u\n", wwn->wwn_tf->tf_ops->direct_submit_supp);
> 
> Same.