[PATCH v4 00/29] Optimize the hot path in the UFS driver

[PATCH v4 00/29] Optimize the hot path in the UFS driver

[Bart Van Assche]

Hi Martin,

This patch series optimizes the hot path of the UFS driver by making
struct scsi_cmnd and struct ufshcd_lrb adjacent. Making these two data
structures adjacent is realized as follows:

@@ -9040,6 +9046,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
     .name            = UFSHCD,
     .proc_name        = UFSHCD,
     .map_queues        = ufshcd_map_queues,
+    .cmd_size        = sizeof(struct ufshcd_lrb),
     .init_cmd_priv        = ufshcd_init_cmd_priv,
     .queuecommand        = ufshcd_queuecommand,
     .mq_poll        = ufshcd_poll,

The following changes had to be made prior to making these two data
structures adjacent:
* Add support for driver-internal and reserved commands in the SCSI core.
* Instead of making the reserved command slot (hba->reserved_slot)
  invisible to the SCSI core, let the SCSI core allocate a reserved command.
* Remove all UFS data structure members that are no longer needed
  because struct scsi_cmnd and struct ufshcd_lrb are now adjacent
* Call ufshcd_init_lrb() from inside ufshcd_queuecommand() instead of
  calling this function before I/O starts. This is necessary because
  ufshcd_memory_alloc() allocates fewer instances than the block layer
  allocates requests. See also the following code in the block layer
  core:

    if (blk_mq_init_request(set, hctx->fq->flush_rq, hctx_idx,
                hctx->numa_node))

  Although the UFS driver could be modified such that ufshcd_init_lrb()
  is called from ufshcd_init_cmd_priv(), realizing this would require
  moving the memory allocations that happen from inside
  ufshcd_memory_alloc() into ufshcd_init_cmd_priv(). That would make
  this patch series even larger. Although ufshcd_init_lrb() is called for each
  command, the benefits of reduced indirection and better cache efficiency
  outweigh the small overhead of per-command lrb initialization.
* ufshcd_add_scsi_host() happens now before any device management
  commands are submitted. This change is necessary because this patch
  makes device management command allocation happen when the SCSI host
  is allocated.
* Allocate as many command slots as the host controller supports. Decrease
  host->cmds_per_lun if necessary once it is clear whether or not the UFS
  device supports less command slots than the host controller.

Changes compared to v3:
 - Fixed a spelling error in patch 1 and left out a superfluous if-statement.
 - Left out scsi_host_template.alloc_pseudo_sdev and allocate a pseudo SCSI
   device if either nr_reserved_cmds > 0 or .queue_reserved_commands has been
   set.
 - Left out the 'pseudo_sdev' local variable from scsi_forget_host().
 - Removed a backwards jump from scsi_get_pseudo_dev().
 - Included a bug fix for synchronous scanning.
 - Skip scsi_track_queue_full() and scsi_handle_queue_ramp_up() for pseudo SCSI
   devices.
 - Extended the scsi_execute_rq() functionality.
 - Use scsi_execute_rq() for submitting reserved commands instead of
   blk_execute_rq().
 - Dropped the patch that introduces scsi_get_internal_cmd() and
   scsi_put_internal_cmd().

Changes compared to v2:
 - Removed scsi_host_update_can_queue() and also the UFS driver refactoring
   patches that were introduced to support this call.
 - Added .queue_reserved_command(). Added ufshcd_queue_reserved_command().
 - Removed a BUG_ON() statement from ufshcd_get_dev_mgmt_cmd().
 - Modified and renamed ufshcd_mcq_decide_queue_depth().

Changes compared to v1:
 - Left out the kernel patches related to support for const SCSI command
   arguments.
 - Added SCSI core patches for allocating a pseudo SCSI device and reserved
   command support.
 - Added several kernel patches to switch the UFS driver from a hardcoded
   reserved slot to calling scsi_get_internal_cmd().
 - Enable .alloc_pseudo_sdev in the scsi_debug driver.

Bart Van Assche (26):
  scsi: core: Move two statements
  scsi: core: Make the budget map optional
  scsi: core: Extend the scsi_execute_cmd() functionality
  scsi_debug: Allocate a pseudo SCSI device
  ufs: core: Move an assignment in ufshcd_mcq_process_cqe()
  ufs: core: Change the type of one ufshcd_add_cmd_upiu_trace() argument
  ufs: core: Only call ufshcd_add_command_trace() for SCSI commands
  ufs: core: Change the type of one ufshcd_add_command_trace() argument
  ufs: core: Change the type of one ufshcd_send_command() argument
  ufs: core: Only call ufshcd_should_inform_monitor() for SCSI commands
  ufs: core: Change the monitor function argument types
  ufs: core: Rework ufshcd_mcq_compl_pending_transfer()
  ufs: core: Rework ufshcd_eh_device_reset_handler()
  ufs: core: Rework the SCSI host queue depth calculation code
  ufs: core: Allocate the SCSI host earlier
  ufs: core: Call ufshcd_init_lrb() later
  ufs: core: Use hba->reserved_slot
  ufs: core: Make the reserved slot a reserved request
  ufs: core: Do not clear driver-private command data
  ufs: core: Optimize the hot path
  ufs: core: Pass a SCSI pointer instead of an LRB pointer
  ufs: core: Remove the ufshcd_lrb task_tag member
  ufs: core: Make blk_mq_tagset_busy_iter() skip reserved requests
  ufs: core: Move code out of ufshcd_wait_for_dev_cmd()
  ufs: core: Rework the ufshcd_issue_dev_cmd() callers
  ufs: core: Switch to scsi_execute_cmd()

Hannes Reinecke (2):
  scsi: core: Support allocating reserved commands
  scsi: core: Support allocating a pseudo SCSI device

John Garry (1):
  scsi: core: Introduce .queue_reserved_command()

 drivers/scsi/hosts.c             |   15 +
 drivers/scsi/scsi.c              |   11 +-
 drivers/scsi/scsi_debug.c        |   14 +
 drivers/scsi/scsi_error.c        |    3 +
 drivers/scsi/scsi_lib.c          |   99 ++-
 drivers/scsi/scsi_priv.h         |    2 +
 drivers/scsi/scsi_scan.c         |   76 ++-
 drivers/scsi/scsi_sysfs.c        |    5 +-
 drivers/ufs/core/ufs-mcq.c       |   51 +-
 drivers/ufs/core/ufshcd-crypto.h |   18 +-
 drivers/ufs/core/ufshcd-priv.h   |   20 +-
 drivers/ufs/core/ufshcd.c        | 1045 +++++++++++++++++-------------
 include/scsi/scsi_cmnd.h         |    2 +
 include/scsi/scsi_device.h       |   53 +-
 include/scsi/scsi_host.h         |   33 +-
 include/ufs/ufshcd.h             |   12 -
 16 files changed, 902 insertions(+), 557 deletions(-)

[PATCH v4 01/29] scsi: core: Support allocating reserved commands

[Bart Van Assche]

From: Hannes Reinecke <hare@suse.de>

Quite some drivers are using management commands internally. These
commands typically use the same tag pool as regular SCSI commands. Tags
for these management commands are set aside before allocating the
block-mq tag bitmap for regular SCSI commands. The block layer already
supports this via the reserved tag mechanism. Add a new field
'nr_reserved_cmds' to the SCSI host template to instruct the block layer
to set aside a tag space for these management commands by using reserved
tags. Exclude reserved commands from .can_queue because .can_queue is
visible in sysfs.

Signed-off-by: Hannes Reinecke <hare@suse.de>
[ bvanassche: modified patch title and patch description. Left out the
  following statements: "if (sht->nr_reserved_cmds)" and also
  "if (sdev->host->nr_reserved_cmds) flags |= BLK_MQ_REQ_RESERVED;". Moved
  nr_reserved_cmds declarations and statements close to the
  corresponding can_queue declarations and statements. See also
  https://lore.kernel.org/linux-scsi/20210503150333.130310-11-hare@suse.de/ ]
Reviewed-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/hosts.c     |  1 +
 drivers/scsi/scsi_lib.c  |  3 ++-
 include/scsi/scsi_host.h | 21 ++++++++++++++++++++-
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
index cc5d05dc395c..9bb7f0114763 100644
--- a/drivers/scsi/hosts.c
+++ b/drivers/scsi/hosts.c
@@ -436,6 +436,7 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int priv
 	shost->hostt = sht;
 	shost->this_id = sht->this_id;
 	shost->can_queue = sht->can_queue;
+	shost->nr_reserved_cmds = sht->nr_reserved_cmds;
 	shost->sg_tablesize = sht->sg_tablesize;
 	shost->sg_prot_tablesize = sht->sg_prot_tablesize;
 	shost->cmd_per_lun = sht->cmd_per_lun;
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 0c65ecfedfbd..9c67e04265ce 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -2083,7 +2083,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
 		tag_set->ops = &scsi_mq_ops_no_commit;
 	tag_set->nr_hw_queues = shost->nr_hw_queues ? : 1;
 	tag_set->nr_maps = shost->nr_maps ? : 1;
-	tag_set->queue_depth = shost->can_queue;
+	tag_set->queue_depth = shost->can_queue + shost->nr_reserved_cmds;
+	tag_set->reserved_tags = shost->nr_reserved_cmds;
 	tag_set->cmd_size = cmd_size;
 	tag_set->numa_node = dev_to_node(shost->dma_dev);
 	if (shost->hostt->tag_alloc_policy_rr)
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index c53812b9026f..91eb3f52b3d0 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -375,10 +375,19 @@ struct scsi_host_template {
 	/*
 	 * This determines if we will use a non-interrupt driven
 	 * or an interrupt driven scheme.  It is set to the maximum number
-	 * of simultaneous commands a single hw queue in HBA will accept.
+	 * of simultaneous commands a single hw queue in HBA will accept
+	 * excluding internal commands.
 	 */
 	int can_queue;
 
+	/*
+	 * This determines how many commands the HBA will set aside
+	 * for internal commands. This number will be added to
+	 * @can_queue to calculate the maximum number of simultaneous
+	 * commands sent to the host.
+	 */
+	int nr_reserved_cmds;
+
 	/*
 	 * In many instances, especially where disconnect / reconnect are
 	 * supported, our host also has an ID on the SCSI bus.  If this is
@@ -611,7 +620,17 @@ struct Scsi_Host {
 	unsigned short max_cmd_len;
 
 	int this_id;
+
+	/*
+	 * Number of commands this host can handle at the same time.
+	 * This excludes reserved commands as specified by nr_reserved_cmds.
+	 */
 	int can_queue;
+	/*
+	 * Number of reserved commands to allocate, if any.
+	 */
+	unsigned int nr_reserved_cmds;
+
 	short cmd_per_lun;
 	short unsigned int sg_tablesize;
 	short unsigned int sg_prot_tablesize;

[PATCH v4 02/29] scsi: core: Move two statements

[Bart Van Assche]

Move two statements that will be needed for pseudo SCSI devices in front
of code that won't be needed for pseudo SCSI devices. No functionality
has been changed.

CC: John Garry <john.g.garry@oracle.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/scsi_scan.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index 3c6e089e80c3..de039efef290 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -347,6 +347,8 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
 	kref_get(&sdev->host->tagset_refcnt);
 	sdev->request_queue = q;
 
+	scsi_sysfs_device_initialize(sdev);
+
 	depth = sdev->host->cmd_per_lun ?: 1;
 
 	/*
@@ -363,8 +365,6 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
 
 	scsi_change_queue_depth(sdev, depth);
 
-	scsi_sysfs_device_initialize(sdev);
-
 	if (shost->hostt->sdev_init) {
 		ret = shost->hostt->sdev_init(sdev);
 		if (ret) {
@@ -1068,6 +1068,8 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
 
 	transport_configure_device(&sdev->sdev_gendev);
 
+	sdev->sdev_bflags = *bflags;
+
 	/*
 	 * No need to freeze the queue as it isn't reachable to anyone else yet.
 	 */
@@ -1113,7 +1115,6 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
 
 	sdev->max_queue_depth = sdev->queue_depth;
 	WARN_ON_ONCE(sdev->max_queue_depth > sdev->budget_map.depth);
-	sdev->sdev_bflags = *bflags;
 
 	/*
 	 * Ok, the device is now all set up, we can

[PATCH v4 03/29] scsi: core: Make the budget map optional

[Bart Van Assche]

Prepare for not allocating a budget map for pseudo SCSI devices by
checking whether a budget map has been allocated before using a budget
map.

Cc: Hannes Reinecke <hare@suse.de>
Cc: John Garry <john.g.garry@oracle.com>
Cc: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/scsi.c     |  2 ++
 drivers/scsi/scsi_lib.c | 14 ++++++++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 9a0f467264b3..ff6b0973d3b4 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -216,6 +216,8 @@ int scsi_device_max_queue_depth(struct scsi_device *sdev)
  */
 int scsi_change_queue_depth(struct scsi_device *sdev, int depth)
 {
+	WARN_ON_ONCE(!sdev->budget_map.map);
+
 	depth = min_t(int, depth, scsi_device_max_queue_depth(sdev));
 
 	if (depth > 0) {
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 9c67e04265ce..91a0c7f843c1 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)
 	if (starget->can_queue > 0)
 		atomic_dec(&starget->target_busy);
 
-	sbitmap_put(&sdev->budget_map, cmd->budget_token);
+	if (sdev->budget_map.map)
+		sbitmap_put(&sdev->budget_map, cmd->budget_token);
 	cmd->budget_token = -1;
 }
 
@@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q,
 {
 	int token;
 
+	/*
+	 * Do not allocate a budget token for reserved SCSI commands. Budget
+	 * tokens are used to enforce the cmd_per_lun limit. That limit does not
+	 * apply to reserved commands.
+	 */
+	if (!sdev->budget_map.map)
+		return INT_MAX;
+
 	token = sbitmap_get(&sdev->budget_map);
 	if (token < 0)
 		return -1;
@@ -1749,7 +1758,8 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token)
 {
 	struct scsi_device *sdev = q->queuedata;
 
-	sbitmap_put(&sdev->budget_map, budget_token);
+	if (sdev->budget_map.map)
+		sbitmap_put(&sdev->budget_map, budget_token);
 }
 
 /*

[PATCH v4 04/29] scsi: core: Support allocating a pseudo SCSI device

[Bart Van Assche]

From: Hannes Reinecke <hare@suse.de>

Allocate a pseudo SCSI device if 'nr_reserved_cmds' has been set. Pseudo
SCSI devices have the SCSI ID <max_id>:U64_MAX so they won't clash with
any devices the LLD might create. Pseudo SCSI devices are excluded from
scanning and will not show up in sysfs. Additionally, pseudo SCSI
devices are skipped by shost_for_each_device(). This prevents that the
SCSI error handler tries to submit a reset to a non-existent logical unit.

Do not allocate a budget map for pseudo SCSI devices since the
cmd_per_lun limit does not apply to pseudo SCSI devices.

Do not perform queue depth ramp up / ramp down for pseudo SCSI devices.

Pseudo SCSI devices will be used to send internal commands to a storage
device.

Cc: John Garry <john.g.garry@oracle.com>
Signed-off-by: Hannes Reinecke <hare@suse.de>
[ bvanassche: edited patch description / renamed host_sdev into
  pseudo_sdev / unexported scsi_get_host_dev() / modified error path in
  scsi_get_pseudo_dev() / skip pseudo devices in __scsi_iterate_devices()
  and also when calling sdev_init(), sdev_configure() and sdev_destroy().
  See also
  https://lore.kernel.org/linux-scsi/20211125151048.103910-2-hare@suse.de/ ]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/hosts.c       |  8 +++++
 drivers/scsi/scsi.c        |  9 +++--
 drivers/scsi/scsi_error.c  |  3 ++
 drivers/scsi/scsi_priv.h   |  2 ++
 drivers/scsi/scsi_scan.c   | 69 +++++++++++++++++++++++++++++++++++++-
 drivers/scsi/scsi_sysfs.c  |  5 ++-
 include/scsi/scsi_device.h | 16 +++++++++
 include/scsi/scsi_host.h   |  6 ++++
 8 files changed, 114 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
index 9bb7f0114763..986586bf67dc 100644
--- a/drivers/scsi/hosts.c
+++ b/drivers/scsi/hosts.c
@@ -307,6 +307,14 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 	if (error)
 		goto out_del_dev;
 
+	if (sht->nr_reserved_cmds) {
+		shost->pseudo_sdev = scsi_get_pseudo_dev(shost);
+		if (!shost->pseudo_sdev) {
+			error = -ENOMEM;
+			goto out_del_dev;
+		}
+	}
+
 	scsi_proc_host_add(shost);
 	scsi_autopm_put_host(shost);
 	return error;
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index ff6b0973d3b4..2d2a52c3ef49 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -257,6 +257,8 @@ EXPORT_SYMBOL(scsi_change_queue_depth);
  */
 int scsi_track_queue_full(struct scsi_device *sdev, int depth)
 {
+	if (scsi_device_is_pseudo_dev(sdev))
+		return 0;
 
 	/*
 	 * Don't let QUEUE_FULLs on the same
@@ -828,8 +830,11 @@ struct scsi_device *__scsi_iterate_devices(struct Scsi_Host *shost,
 	spin_lock_irqsave(shost->host_lock, flags);
 	while (list->next != &shost->__devices) {
 		next = list_entry(list->next, struct scsi_device, siblings);
-		/* skip devices that we can't get a reference to */
-		if (!scsi_device_get(next))
+		/*
+		 * Skip pseudo devices and also devices for which
+		 * scsi_device_get() fails.
+		 */
+		if (!scsi_device_is_pseudo_dev(next) && !scsi_device_get(next))
 			break;
 		next = NULL;
 		list = list->next;
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 746ff6a1f309..540d82974529 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -749,6 +749,9 @@ static void scsi_handle_queue_ramp_up(struct scsi_device *sdev)
 	const struct scsi_host_template *sht = sdev->host->hostt;
 	struct scsi_device *tmp_sdev;
 
+	if (scsi_device_is_pseudo_dev(sdev))
+		return;
+
 	if (!sht->track_queue_depth ||
 	    sdev->queue_depth >= sdev->max_queue_depth)
 		return;
diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h
index 5b2b19f5e8ec..da3bc87ac5a6 100644
--- a/drivers/scsi/scsi_priv.h
+++ b/drivers/scsi/scsi_priv.h
@@ -135,6 +135,8 @@ extern int scsi_complete_async_scans(void);
 extern int scsi_scan_host_selected(struct Scsi_Host *, unsigned int,
 				   unsigned int, u64, enum scsi_scan_mode);
 extern void scsi_forget_host(struct Scsi_Host *);
+struct scsi_device *scsi_get_pseudo_dev(struct Scsi_Host *);
+bool scsi_device_is_pseudo_dev(struct scsi_device *sdev);
 
 /* scsi_sysctl.c */
 #ifdef CONFIG_SYSCTL
diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index de039efef290..a3523f964bc1 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -225,6 +225,8 @@ static int scsi_realloc_sdev_budget_map(struct scsi_device *sdev,
 	int ret;
 	struct sbitmap sb_backup;
 
+	WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev));
+
 	depth = min_t(unsigned int, depth, scsi_device_max_queue_depth(sdev));
 
 	/*
@@ -349,6 +351,9 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
 
 	scsi_sysfs_device_initialize(sdev);
 
+	if (scsi_device_is_pseudo_dev(sdev))
+		return sdev;
+
 	depth = sdev->host->cmd_per_lun ?: 1;
 
 	/*
@@ -1070,6 +1075,9 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
 
 	sdev->sdev_bflags = *bflags;
 
+	if (scsi_device_is_pseudo_dev(sdev))
+		return SCSI_SCAN_LUN_PRESENT;
+
 	/*
 	 * No need to freeze the queue as it isn't reachable to anyone else yet.
 	 */
@@ -1213,6 +1221,12 @@ static int scsi_probe_and_add_lun(struct scsi_target *starget,
 	if (!sdev)
 		goto out;
 
+	if (scsi_device_is_pseudo_dev(sdev)) {
+		if (bflagsp)
+			*bflagsp = BLIST_NOLUN;
+		return SCSI_SCAN_LUN_PRESENT;
+	}
+
 	result = kmalloc(result_len, GFP_KERNEL);
 	if (!result)
 		goto out_free_sdev;
@@ -2084,12 +2098,65 @@ void scsi_forget_host(struct Scsi_Host *shost)
  restart:
 	spin_lock_irqsave(shost->host_lock, flags);
 	list_for_each_entry(sdev, &shost->__devices, siblings) {
-		if (sdev->sdev_state == SDEV_DEL)
+		if (scsi_device_is_pseudo_dev(sdev) ||
+		    sdev->sdev_state == SDEV_DEL)
 			continue;
 		spin_unlock_irqrestore(shost->host_lock, flags);
 		__scsi_remove_device(sdev);
 		goto restart;
 	}
 	spin_unlock_irqrestore(shost->host_lock, flags);
+
+	/*
+	 * Remove the pseudo device last since it may be needed during removal
+	 * of other SCSI devices.
+	 */
+	if (shost->pseudo_sdev)
+		__scsi_remove_device(shost->pseudo_sdev);
 }
 
+/**
+ * scsi_get_pseudo_dev() - Attach a pseudo SCSI device to a SCSI host
+ * @shost: Host that needs a pseudo SCSI device
+ *
+ * Lock status: None assumed.
+ *
+ * Returns:     The scsi_device or NULL
+ *
+ * Notes:
+ *	Attach a single scsi_device to the Scsi_Host. The primary aim for this
+ *	device is to serve as a container from which SCSI commands can be
+ *	allocated. Each SCSI command will carry a command tag allocated by the
+ *	block layer. These SCSI commands can be used by the LLDD to send
+ *	internal or passthrough commands without having to manage tag allocation
+ *	inside the LLDD.
+ */
+struct scsi_device *scsi_get_pseudo_dev(struct Scsi_Host *shost)
+{
+	struct scsi_device *sdev = NULL;
+	struct scsi_target *starget;
+
+	guard(mutex)(&shost->scan_mutex);
+
+	if (!scsi_host_scan_allowed(shost))
+		goto out;
+
+	starget = scsi_alloc_target(&shost->shost_gendev, 0, shost->max_id);
+	if (!starget)
+		goto out;
+
+	sdev = scsi_alloc_sdev(starget, U64_MAX, NULL);
+	if (!sdev) {
+		scsi_target_reap(starget);
+		goto put_target;
+	}
+
+	sdev->borken = 0;
+
+put_target:
+	/* See also the get_device(dev) call in scsi_alloc_target(). */
+	put_device(&starget->dev);
+
+out:
+	return sdev;
+}
diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
index 169af7d47ce7..22f76a1ca23b 100644
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -1406,6 +1406,9 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev)
 	int error;
 	struct scsi_target *starget = sdev->sdev_target;
 
+	if (WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev)))
+		return -EINVAL;
+
 	error = scsi_target_add(starget);
 	if (error)
 		return error;
@@ -1513,7 +1516,7 @@ void __scsi_remove_device(struct scsi_device *sdev)
 	kref_put(&sdev->host->tagset_refcnt, scsi_mq_free_tags);
 	cancel_work_sync(&sdev->requeue_work);
 
-	if (sdev->host->hostt->sdev_destroy)
+	if (!scsi_device_is_pseudo_dev(sdev) && sdev->host->hostt->sdev_destroy)
 		sdev->host->hostt->sdev_destroy(sdev);
 	transport_destroy_device(dev);
 
diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
index 6d6500148c4b..3846f5dfc51c 100644
--- a/include/scsi/scsi_device.h
+++ b/include/scsi/scsi_device.h
@@ -589,6 +589,22 @@ static inline unsigned int sdev_id(struct scsi_device *sdev)
 #define scmd_id(scmd) sdev_id((scmd)->device)
 #define scmd_channel(scmd) sdev_channel((scmd)->device)
 
+/**
+ * scsi_device_is_pseudo_dev() - Whether a device is a pseudo SCSI device.
+ * @sdev: SCSI device to examine
+ *
+ * A pseudo SCSI device can be used to allocate SCSI commands but does not show
+ * up in sysfs. Additionally, the logical unit information in *@sdev is made up.
+ *
+ * This function tests the LUN number instead of comparing @sdev with
+ * @sdev->host->pseudo_sdev because this function may be called before
+ * @sdev->host->pseudo_sdev has been initialized.
+ */
+static inline bool scsi_device_is_pseudo_dev(struct scsi_device *sdev)
+{
+	return sdev->lun == U64_MAX;
+}
+
 /*
  * checks for positions of the SCSI state machine
  */
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 91eb3f52b3d0..3bfb53cf5dfc 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -721,6 +721,12 @@ struct Scsi_Host {
 	/* ldm bits */
 	struct device		shost_gendev, shost_dev;
 
+	/*
+	 * A SCSI device structure used for sending internal commands to the
+	 * HBA. There is no corresponding logical unit inside the SCSI device.
+	 */
+	struct scsi_device *pseudo_sdev;
+
 	/*
 	 * Points to the transport data (if any) which is allocated
 	 * separately

[PATCH v4 05/29] scsi: core: Introduce .queue_reserved_command()

[Bart Van Assche]

From: John Garry <john.garry@huawei.com>

Reserved commands will be used by SCSI LLDs for submitting internal
commands. Since the SCSI host, target and device limits do not apply to
the reserved command use cases, bypass the SCSI host limit checks for
reserved commands. Introduce the .queue_reserved_command() callback for
reserved commands. Additionally, do not activate the SCSI error handler
if a reserved command fails such that reserved commands can be submitted
from inside the SCSI error handler.

Signed-off-by: John Garry <john.garry@huawei.com>
[ bvanassche: modified patch title and patch description. Renamed
  .reserved_queuecommand() into .queue_reserved_command(). Changed
  the second argument of __blk_mq_end_request() from 0 into error
  code in the completion path if cmd->result != 0. Rewrote the
  scsi_queue_rq() changes. See also
  https://lore.kernel.org/linux-scsi/1666693096-180008-5-git-send-email-john.garry@huawei.com/ ]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/hosts.c     |  8 +++++-
 drivers/scsi/scsi_lib.c  | 54 ++++++++++++++++++++++++++++------------
 include/scsi/scsi_host.h |  6 +++++
 3 files changed, 51 insertions(+), 17 deletions(-)

diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
index 986586bf67dc..3a62c51379ef 100644
--- a/drivers/scsi/hosts.c
+++ b/drivers/scsi/hosts.c
@@ -231,6 +231,12 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 		goto fail;
 	}
 
+	if (shost->nr_reserved_cmds && !sht->queue_reserved_command) {
+		shost_printk(KERN_ERR, shost,
+			     "nr_reserved_cmds set but no method to queue\n");
+		goto fail;
+	}
+
 	/* Use min_t(int, ...) in case shost->can_queue exceeds SHRT_MAX */
 	shost->cmd_per_lun = min_t(int, shost->cmd_per_lun,
 				   shost->can_queue);
@@ -307,7 +313,7 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 	if (error)
 		goto out_del_dev;
 
-	if (sht->nr_reserved_cmds) {
+	if (sht->nr_reserved_cmds || sht->queue_reserved_command) {
 		shost->pseudo_sdev = scsi_get_pseudo_dev(shost);
 		if (!shost->pseudo_sdev) {
 			error = -ENOMEM;
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 91a0c7f843c1..5e636e015352 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1539,6 +1539,14 @@ static void scsi_complete(struct request *rq)
 	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
 	enum scsi_disposition disposition;
 
+	if (blk_mq_is_reserved_rq(rq)) {
+		/* Only pass-through requests are supported in this code path. */
+		WARN_ON_ONCE(!blk_rq_is_passthrough(scsi_cmd_to_rq(cmd)));
+		scsi_mq_uninit_cmd(cmd);
+		__blk_mq_end_request(rq, scsi_result_to_blk_status(cmd->result));
+		return;
+	}
+
 	INIT_LIST_HEAD(&cmd->eh_entry);
 
 	atomic_inc(&cmd->device->iodone_cnt);
@@ -1828,25 +1836,31 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 	WARN_ON_ONCE(cmd->budget_token < 0);
 
 	/*
-	 * If the device is not in running state we will reject some or all
-	 * commands.
+	 * Bypass the SCSI device, SCSI target and SCSI host checks for
+	 * reserved commands.
 	 */
-	if (unlikely(sdev->sdev_state != SDEV_RUNNING)) {
-		ret = scsi_device_state_check(sdev, req);
-		if (ret != BLK_STS_OK)
-			goto out_put_budget;
-	}
+	if (!blk_mq_is_reserved_rq(req)) {
+		/*
+		 * If the device is not in running state we will reject some or
+		 * all commands.
+		 */
+		if (unlikely(sdev->sdev_state != SDEV_RUNNING)) {
+			ret = scsi_device_state_check(sdev, req);
+			if (ret != BLK_STS_OK)
+				goto out_put_budget;
+		}
 
-	ret = BLK_STS_RESOURCE;
-	if (!scsi_target_queue_ready(shost, sdev))
-		goto out_put_budget;
-	if (unlikely(scsi_host_in_recovery(shost))) {
-		if (cmd->flags & SCMD_FAIL_IF_RECOVERING)
-			ret = BLK_STS_OFFLINE;
-		goto out_dec_target_busy;
+		ret = BLK_STS_RESOURCE;
+		if (!scsi_target_queue_ready(shost, sdev))
+			goto out_put_budget;
+		if (unlikely(scsi_host_in_recovery(shost))) {
+			if (cmd->flags & SCMD_FAIL_IF_RECOVERING)
+				ret = BLK_STS_OFFLINE;
+			goto out_dec_target_busy;
+		}
+		if (!scsi_host_queue_ready(q, shost, sdev, cmd))
+			goto out_dec_target_busy;
 	}
-	if (!scsi_host_queue_ready(q, shost, sdev, cmd))
-		goto out_dec_target_busy;
 
 	/*
 	 * Only clear the driver-private command data if the LLD does not supply
@@ -1875,6 +1889,14 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 	cmd->submitter = SUBMITTED_BY_BLOCK_LAYER;
 
 	blk_mq_start_request(req);
+	if (blk_mq_is_reserved_rq(req)) {
+		reason = shost->hostt->queue_reserved_command(shost, cmd);
+		if (reason) {
+			ret = BLK_STS_RESOURCE;
+			goto out_put_budget;
+		}
+		return BLK_STS_OK;
+	}
 	reason = scsi_dispatch_cmd(cmd);
 	if (reason) {
 		scsi_set_blocked(cmd, reason);
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 3bfb53cf5dfc..9a0b07bfd559 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -86,6 +86,12 @@ struct scsi_host_template {
 	 */
 	int (* queuecommand)(struct Scsi_Host *, struct scsi_cmnd *);
 
+	/*
+	 * Queue a reserved command (BLK_MQ_REQ_RESERVED). The .queuecommand()
+	 * documentation also applies to the .queue_reserved_command() callback.
+	 */
+	int (*queue_reserved_command)(struct Scsi_Host *, struct scsi_cmnd *);
+
 	/*
 	 * The commit_rqs function is used to trigger a hardware
 	 * doorbell after some requests have been queued with

[PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[Bart Van Assche]

Make the @cmd argument optional. Add .init_cmd() and .copy_result()
callbacks in struct scsi_exec_args. Support allocating from a specific
hardware queue. This patch prepares for submitting reserved commands
with scsi_execute_cmd().

Cc: John Garry <john.g.garry@oracle.com>
Cc: Hannes Reinecke <hare@suse.de>
Cc: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/scsi_lib.c    | 28 +++++++++++++++++++++++++---
 include/scsi/scsi_cmnd.h   |  2 ++
 include/scsi/scsi_device.h | 37 +++++++++++++++++++++++++++++--------
 3 files changed, 56 insertions(+), 11 deletions(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 5e636e015352..022cd454d658 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -308,7 +308,10 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
 		return -EINVAL;
 
 retry:
-	req = scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
+	req = args->specify_hctx ?
+		scsi_alloc_request_hctx(sdev->request_queue, opf,
+					args->req_flags, args->hctx_idx) :
+		scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
 	if (IS_ERR(req))
 		return PTR_ERR(req);
 
@@ -318,8 +321,12 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
 			goto out;
 	}
 	scmd = blk_mq_rq_to_pdu(req);
-	scmd->cmd_len = COMMAND_SIZE(cmd[0]);
-	memcpy(scmd->cmnd, cmd, scmd->cmd_len);
+	if (cmd) {
+		scmd->cmd_len = COMMAND_SIZE(cmd[0]);
+		memcpy(scmd->cmnd, cmd, scmd->cmd_len);
+	}
+	if (args->init_cmd)
+		args->init_cmd(scmd, args);
 	scmd->allowed = ml_retries;
 	scmd->flags |= args->scmd_flags;
 	req->timeout = timeout;
@@ -353,6 +360,9 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
 				     args->sshdr);
 
 	ret = scmd->result;
+	if (ret == 0 && args->copy_result)
+		args->copy_result(scmd, args);
+
  out:
 	blk_mq_free_request(req);
 
@@ -1247,6 +1257,18 @@ struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf,
 }
 EXPORT_SYMBOL_GPL(scsi_alloc_request);
 
+struct request *scsi_alloc_request_hctx(struct request_queue *q, blk_opf_t opf,
+			blk_mq_req_flags_t flags, unsigned int hctx_idx)
+{
+	struct request *rq;
+
+	rq = blk_mq_alloc_request_hctx(q, opf, flags, hctx_idx);
+	if (!IS_ERR(rq))
+		scsi_initialize_rq(rq);
+	return rq;
+}
+EXPORT_SYMBOL_GPL(scsi_alloc_request_hctx);
+
 /*
  * Only called when the request isn't completed by SCSI, and not freed by
  * SCSI
diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h
index 8ecfb94049db..a4cb836809df 100644
--- a/include/scsi/scsi_cmnd.h
+++ b/include/scsi/scsi_cmnd.h
@@ -396,5 +396,7 @@ extern void scsi_build_sense(struct scsi_cmnd *scmd, int desc,
 
 struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf,
 				   blk_mq_req_flags_t flags);
+struct request *scsi_alloc_request_hctx(struct request_queue *q, blk_opf_t opf,
+			blk_mq_req_flags_t flags, unsigned int hctx_idx);
 
 #endif /* _SCSI_SCSI_CMND_H */
diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
index 3846f5dfc51c..2b2dc08962a2 100644
--- a/include/scsi/scsi_device.h
+++ b/include/scsi/scsi_device.h
@@ -541,15 +541,36 @@ struct scsi_failures {
 	struct scsi_failure *failure_definitions;
 };
 
-/* Optional arguments to scsi_execute_cmd */
+/**
+ * struct scsi_exec_args - Optional arguments to scsi_execute_cmd()
+ * @init_cmd: called before the command is executed.
+ * @copy_result: called after the command has been executed.
+ * @sense: sense buffer.
+ * @sense_len: sense buffer len
+ * @sshdr: decoded sense header
+ * @req_flags: BLK_MQ_REQ flags
+ * @scmd_flags: SCMD flags.
+ * @resid: residual length.
+ * @failures: which failures to retry.
+ * @specify_hctx: call scsi_alloc_request_hctx() if %true or
+	scsi_alloc_request() otherwise.
+ * @hctx_idx: Passed as fourth argument for scsi_alloc_request_hctx() if
+	@specify_hctx is %true.
+ */
 struct scsi_exec_args {
-	unsigned char *sense;		/* sense buffer */
-	unsigned int sense_len;		/* sense buffer len */
-	struct scsi_sense_hdr *sshdr;	/* decoded sense header */
-	blk_mq_req_flags_t req_flags;	/* BLK_MQ_REQ flags */
-	int scmd_flags;			/* SCMD flags */
-	int *resid;			/* residual length */
-	struct scsi_failures *failures;	/* failures to retry */
+	int (*init_cmd)(struct scsi_cmnd *cmd,
+			const struct scsi_exec_args *args);
+	void (*copy_result)(struct scsi_cmnd *cmd,
+			    const struct scsi_exec_args *args);
+	unsigned char *sense;
+	unsigned int sense_len;
+	struct scsi_sense_hdr *sshdr;
+	blk_mq_req_flags_t req_flags;
+	int scmd_flags;
+	int *resid;
+	struct scsi_failures *failures;
+	bool specify_hctx;
+	int hctx_idx;
 };
 
 int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,

[PATCH v4 07/29] scsi_debug: Allocate a pseudo SCSI device

[Bart Van Assche]

Make sure that the code for allocating a pseudo SCSI device gets triggered
while running blktests.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/scsi_debug.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
index 2a8638937d23..3f7884025d19 100644
--- a/drivers/scsi/scsi_debug.c
+++ b/drivers/scsi/scsi_debug.c
@@ -9197,6 +9197,19 @@ static int sdebug_fail_cmd(struct scsi_cmnd *cmnd, int *retval,
 	return ret;
 }
 
+/*
+ * The only purpose of this function is to make the SCSI core allocate a
+ * pseudo SCSI device.
+ */
+static int scsi_debug_queue_reserved_command(struct Scsi_Host *shost,
+					     struct scsi_cmnd *scp)
+{
+	WARN_ON_ONCE(true);
+	scp->result = DID_ERROR << 16;
+	scsi_done(scp);
+	return 0;
+}
+
 static int scsi_debug_queuecommand(struct Scsi_Host *shost,
 				   struct scsi_cmnd *scp)
 {
@@ -9416,6 +9429,7 @@ static const struct scsi_host_template sdebug_driver_template = {
 	.sdev_destroy =		scsi_debug_sdev_destroy,
 	.ioctl =		scsi_debug_ioctl,
 	.queuecommand =		scsi_debug_queuecommand,
+	.queue_reserved_command = scsi_debug_queue_reserved_command,
 	.change_queue_depth =	sdebug_change_qdepth,
 	.map_queues =		sdebug_map_queues,
 	.mq_poll =		sdebug_blk_mq_poll,

[PATCH v4 08/29] ufs: core: Move an assignment in ufshcd_mcq_process_cqe()

[Bart Van Assche]

Prepare for introducing a WARN_ON_ONCE() call in ufshcd_mcq_get_tag() if
the tag lookup fails.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufs-mcq.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c
index 1e50675772fe..4eb4f1af7800 100644
--- a/drivers/ufs/core/ufs-mcq.c
+++ b/drivers/ufs/core/ufs-mcq.c
@@ -303,9 +303,10 @@ static void ufshcd_mcq_process_cqe(struct ufs_hba *hba,
 				   struct ufs_hw_queue *hwq)
 {
 	struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq);
-	int tag = ufshcd_mcq_get_tag(hba, cqe);
 
 	if (cqe->command_desc_base_addr) {
+		int tag = ufshcd_mcq_get_tag(hba, cqe);
+
 		ufshcd_compl_one_cqe(hba, tag, cqe);
 		/* After processed the cqe, mark it empty (invalid) entry */
 		cqe->command_desc_base_addr = 0;

[PATCH v4 09/29] ufs: core: Change the type of one ufshcd_add_cmd_upiu_trace() argument

[Bart Van Assche]

Change the 'tag' argument into an LRB pointer. This patch prepares for the
removal of the hba->lrb[] array.

Reviewed-by: Avri Altman <avri.altman@sandisk.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index e2157128e3bf..9eb11bb33e85 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -405,10 +405,11 @@ static void ufshcd_configure_wb(struct ufs_hba *hba)
 		ufshcd_wb_toggle_buf_flush(hba, true);
 }
 
-static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba, unsigned int tag,
+static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba,
+				      struct ufshcd_lrb *lrb,
 				      enum ufs_trace_str_t str_t)
 {
-	struct utp_upiu_req *rq = hba->lrb[tag].ucd_req_ptr;
+	struct utp_upiu_req *rq = lrb->ucd_req_ptr;
 	struct utp_upiu_header *header;
 
 	if (!trace_ufshcd_upiu_enabled())
@@ -417,7 +418,7 @@ static void ufshcd_add_cmd_upiu_trace(struct ufs_hba *hba, unsigned int tag,
 	if (str_t == UFS_CMD_SEND)
 		header = &rq->header;
 	else
-		header = &hba->lrb[tag].ucd_rsp_ptr->header;
+		header = &lrb->ucd_rsp_ptr->header;
 
 	trace_ufshcd_upiu(hba, str_t, header, &rq->sc.cdb,
 			  UFS_TSF_CDB);
@@ -491,7 +492,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 		return;
 
 	/* trace UPIU also */
-	ufshcd_add_cmd_upiu_trace(hba, tag, str_t);
+	ufshcd_add_cmd_upiu_trace(hba, lrbp, str_t);
 	if (!trace_ufshcd_command_enabled())
 		return;
 

[PATCH v4 10/29] ufs: core: Only call ufshcd_add_command_trace() for SCSI commands

[Bart Van Assche]

Instead of checking inside ufshcd_add_command_trace() whether 'cmd' points
at a SCSI command, let the caller perform that check. This patch prepares
for removing the lrbp->cmd pointer.

Reviewed-by: Avri Altman <avri.altman@sandisk.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 9eb11bb33e85..2b14d1510d92 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -488,9 +488,6 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 	struct request *rq = scsi_cmd_to_rq(cmd);
 	int transfer_len = -1;
 
-	if (!cmd)
-		return;
-
 	/* trace UPIU also */
 	ufshcd_add_cmd_upiu_trace(hba, lrbp, str_t);
 	if (!trace_ufshcd_command_enabled())
@@ -2365,9 +2362,10 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag,
 		lrbp->compl_time_stamp = ktime_set(0, 0);
 		lrbp->compl_time_stamp_local_clock = 0;
 	}
-	ufshcd_add_command_trace(hba, task_tag, UFS_CMD_SEND);
-	if (lrbp->cmd)
+	if (lrbp->cmd) {
+		ufshcd_add_command_trace(hba, task_tag, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
+	}
 	if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
 		ufshcd_start_monitor(hba, lrbp);
 

[PATCH v4 11/29] ufs: core: Change the type of one ufshcd_add_command_trace() argument

[Bart Van Assche]

Change the 'tag' argument into a SCSI command pointer. This patch prepares
for the removal of the hba->lrb[] array.

Reviewed-by: Avri Altman <avri.altman@sandisk.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 2b14d1510d92..a87983dde5cd 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -475,7 +475,7 @@ static void ufshcd_add_uic_command_trace(struct ufs_hba *hba,
 				 ufshcd_readl(hba, REG_UIC_COMMAND_ARG_3));
 }
 
-static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
+static void ufshcd_add_command_trace(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 				     enum ufs_trace_str_t str_t)
 {
 	u64 lba = 0;
@@ -483,9 +483,9 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 	u32 doorbell = 0;
 	u32 intr;
 	u32 hwq_id = 0;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
-	struct scsi_cmnd *cmd = lrbp->cmd;
 	struct request *rq = scsi_cmd_to_rq(cmd);
+	unsigned int tag = rq->tag;
+	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
 	int transfer_len = -1;
 
 	/* trace UPIU also */
@@ -503,7 +503,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, unsigned int tag,
 		       be32_to_cpu(lrbp->ucd_req_ptr->sc.exp_data_transfer_len);
 		lba = scsi_get_lba(cmd);
 		if (opcode == WRITE_10)
-			group_id = lrbp->cmd->cmnd[6];
+			group_id = cmd->cmnd[6];
 	} else if (opcode == UNMAP) {
 		/*
 		 * The number of Bytes to be unmapped beginning with the lba.
@@ -2363,7 +2363,7 @@ void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag,
 		lrbp->compl_time_stamp_local_clock = 0;
 	}
 	if (lrbp->cmd) {
-		ufshcd_add_command_trace(hba, task_tag, UFS_CMD_SEND);
+		ufshcd_add_command_trace(hba, lrbp->cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
 	}
 	if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
@@ -5630,7 +5630,7 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 	if (cmd) {
 		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
 			ufshcd_update_monitor(hba, lrbp);
-		ufshcd_add_command_trace(hba, task_tag, UFS_CMD_COMP);
+		ufshcd_add_command_trace(hba, cmd, UFS_CMD_COMP);
 		cmd->result = ufshcd_transfer_rsp_status(hba, lrbp, cqe);
 		ufshcd_release_scsi_cmd(hba, lrbp);
 		/* Do not touch lrbp after scsi done */

[PATCH v4 12/29] ufs: core: Change the type of one ufshcd_send_command() argument

[Bart Van Assche]

Change the 'task_tag' argument into an LRB pointer. This patch prepares
for the removal of the hba->lrb[] array.

Reviewed-by: Avri Altman <avri.altman@sandisk.com>
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index a87983dde5cd..eb2c6fe5e7ea 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2346,14 +2346,13 @@ static void ufshcd_update_monitor(struct ufs_hba *hba, const struct ufshcd_lrb *
 /**
  * ufshcd_send_command - Send SCSI or device management commands
  * @hba: per adapter instance
- * @task_tag: Task tag of the command
+ * @lrbp: Local reference block of SCSI command
  * @hwq: pointer to hardware queue instance
  */
-static inline
-void ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag,
-			 struct ufs_hw_queue *hwq)
+static inline void ufshcd_send_command(struct ufs_hba *hba,
+				       struct ufshcd_lrb *lrbp,
+				       struct ufs_hw_queue *hwq)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
 	unsigned long flags;
 
 	if (hba->monitor.enabled) {
@@ -3062,7 +3061,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	if (hba->mcq_enabled)
 		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
 
-	ufshcd_send_command(hba, tag, hwq);
+	ufshcd_send_command(hba, lrbp, hwq);
 
 out:
 	if (ufs_trigger_eh(hba)) {
@@ -3307,7 +3306,7 @@ static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 	int err;
 
 	ufshcd_add_query_upiu_trace(hba, UFS_QUERY_SEND, lrbp->ucd_req_ptr);
-	ufshcd_send_command(hba, tag, hba->dev_cmd_queue);
+	ufshcd_send_command(hba, lrbp, hba->dev_cmd_queue);
 	err = ufshcd_wait_for_dev_cmd(hba, lrbp, timeout);
 
 	ufshcd_add_query_upiu_trace(hba, err ? UFS_QUERY_ERR : UFS_QUERY_COMP,

[PATCH v4 13/29] ufs: core: Only call ufshcd_should_inform_monitor() for SCSI commands

[Bart Van Assche]

ufshcd_should_inform_monitor() only returns 'true' for SCSI commands.
Instead of checking inside ufshcd_should_inform_monitor() whether its
second argument represents a SCSI command, only call this function for
SCSI commands. This patch prepares for removing the lrbp->cmd member.

Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index eb2c6fe5e7ea..bd57baa15e8d 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2289,12 +2289,13 @@ static inline int ufshcd_monitor_opcode2dir(u8 opcode)
 		return -EINVAL;
 }
 
+/* Must only be called for SCSI commands. */
 static inline bool ufshcd_should_inform_monitor(struct ufs_hba *hba,
 						struct ufshcd_lrb *lrbp)
 {
 	const struct ufs_hba_monitor *m = &hba->monitor;
 
-	return (m->enabled && lrbp && lrbp->cmd &&
+	return (m->enabled &&
 		(!m->chunk_size || m->chunk_size == lrbp->cmd->sdb.length) &&
 		ktime_before(hba->monitor.enabled_ts, lrbp->issue_time_stamp));
 }
@@ -2364,9 +2365,9 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	if (lrbp->cmd) {
 		ufshcd_add_command_trace(hba, lrbp->cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
+		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
+			ufshcd_start_monitor(hba, lrbp);
 	}
-	if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
-		ufshcd_start_monitor(hba, lrbp);
 
 	if (hba->mcq_enabled) {
 		int utrd_size = sizeof(struct utp_transfer_req_desc);

[PATCH v4 14/29] ufs: core: Change the monitor function argument types

[Bart Van Assche]

Pass a SCSI command pointer instead of a struct ufshcd_lrb pointer. This
patch prepares for combining the SCSI command and ufshcd_lrb data
structures into a single data structure.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 44 +++++++++++++++++++--------------------
 1 file changed, 22 insertions(+), 22 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index bd57baa15e8d..23c2e94bfd89 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2291,19 +2291,20 @@ static inline int ufshcd_monitor_opcode2dir(u8 opcode)
 
 /* Must only be called for SCSI commands. */
 static inline bool ufshcd_should_inform_monitor(struct ufs_hba *hba,
-						struct ufshcd_lrb *lrbp)
+						struct scsi_cmnd *cmd)
 {
 	const struct ufs_hba_monitor *m = &hba->monitor;
+	struct request *rq = scsi_cmd_to_rq(cmd);
+	struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag];
 
-	return (m->enabled &&
-		(!m->chunk_size || m->chunk_size == lrbp->cmd->sdb.length) &&
-		ktime_before(hba->monitor.enabled_ts, lrbp->issue_time_stamp));
+	return m->enabled &&
+	       (!m->chunk_size || m->chunk_size == cmd->sdb.length) &&
+	       ktime_before(hba->monitor.enabled_ts, lrbp->issue_time_stamp);
 }
 
-static void ufshcd_start_monitor(struct ufs_hba *hba,
-				 const struct ufshcd_lrb *lrbp)
+static void ufshcd_start_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	int dir = ufshcd_monitor_opcode2dir(*lrbp->cmd->cmnd);
+	int dir = ufshcd_monitor_opcode2dir(cmd->cmnd[0]);
 	unsigned long flags;
 
 	spin_lock_irqsave(hba->host->host_lock, flags);
@@ -2312,14 +2313,15 @@ static void ufshcd_start_monitor(struct ufs_hba *hba,
 	spin_unlock_irqrestore(hba->host->host_lock, flags);
 }
 
-static void ufshcd_update_monitor(struct ufs_hba *hba, const struct ufshcd_lrb *lrbp)
+static void ufshcd_update_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	int dir = ufshcd_monitor_opcode2dir(*lrbp->cmd->cmnd);
+	struct request *req = scsi_cmd_to_rq(cmd);
+	struct ufshcd_lrb *lrbp = &hba->lrb[req->tag];
+	int dir = ufshcd_monitor_opcode2dir(cmd->cmnd[0]);
 	unsigned long flags;
 
 	spin_lock_irqsave(hba->host->host_lock, flags);
 	if (dir >= 0 && hba->monitor.nr_queued[dir] > 0) {
-		const struct request *req = scsi_cmd_to_rq(lrbp->cmd);
 		struct ufs_hba_monitor *m = &hba->monitor;
 		ktime_t now, inc, lat;
 
@@ -2354,6 +2356,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 				       struct ufshcd_lrb *lrbp,
 				       struct ufs_hw_queue *hwq)
 {
+	struct scsi_cmnd *cmd = lrbp->cmd;
 	unsigned long flags;
 
 	if (hba->monitor.enabled) {
@@ -2362,11 +2365,11 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 		lrbp->compl_time_stamp = ktime_set(0, 0);
 		lrbp->compl_time_stamp_local_clock = 0;
 	}
-	if (lrbp->cmd) {
-		ufshcd_add_command_trace(hba, lrbp->cmd, UFS_CMD_SEND);
+	if (cmd) {
+		ufshcd_add_command_trace(hba, cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
-		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
-			ufshcd_start_monitor(hba, lrbp);
+		if (unlikely(ufshcd_should_inform_monitor(hba, cmd)))
+			ufshcd_start_monitor(hba, cmd);
 	}
 
 	if (hba->mcq_enabled) {
@@ -2382,8 +2385,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	} else {
 		spin_lock_irqsave(&hba->outstanding_lock, flags);
 		if (hba->vops && hba->vops->setup_xfer_req)
-			hba->vops->setup_xfer_req(hba, lrbp->task_tag,
-						  !!lrbp->cmd);
+			hba->vops->setup_xfer_req(hba, lrbp->task_tag, !!cmd);
 		__set_bit(lrbp->task_tag, &hba->outstanding_reqs);
 		ufshcd_writel(hba, 1 << lrbp->task_tag,
 			      REG_UTP_TRANSFER_REQ_DOOR_BELL);
@@ -5617,19 +5619,17 @@ void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
 void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 			  struct cq_entry *cqe)
 {
-	struct ufshcd_lrb *lrbp;
-	struct scsi_cmnd *cmd;
+	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
+	struct scsi_cmnd *cmd = lrbp->cmd;
 	enum utp_ocs ocs;
 
-	lrbp = &hba->lrb[task_tag];
 	if (hba->monitor.enabled) {
 		lrbp->compl_time_stamp = ktime_get();
 		lrbp->compl_time_stamp_local_clock = local_clock();
 	}
-	cmd = lrbp->cmd;
 	if (cmd) {
-		if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
-			ufshcd_update_monitor(hba, lrbp);
+		if (unlikely(ufshcd_should_inform_monitor(hba, cmd)))
+			ufshcd_update_monitor(hba, cmd);
 		ufshcd_add_command_trace(hba, cmd, UFS_CMD_COMP);
 		cmd->result = ufshcd_transfer_rsp_status(hba, lrbp, cqe);
 		ufshcd_release_scsi_cmd(hba, lrbp);

[PATCH v4 15/29] ufs: core: Rework ufshcd_mcq_compl_pending_transfer()

[Bart Van Assche]

Replace a tag loop with blk_mq_tagset_busy_iter(). This patch prepares
for removing the hba->lrb[] array.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 80 ++++++++++++++++++++++-----------------
 1 file changed, 46 insertions(+), 34 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 23c2e94bfd89..876c4de86b26 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -5714,6 +5714,48 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)
 	return completed_reqs != 0;
 }
 
+static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
+{
+	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
+	struct scsi_device *sdev = rq->q->queuedata;
+	struct Scsi_Host *shost = sdev->host;
+	struct ufs_hba *hba = shost_priv(shost);
+	struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag];
+	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
+
+	if (!hwq)
+		return true;
+
+	ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
+
+	/*
+	 * For those cmds of which the cqes are not present in the cq, complete
+	 * them explicitly.
+	 */
+	scoped_guard(spinlock_irqsave, &hwq->cq_lock) {
+		if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
+			set_host_byte(cmd, DID_REQUEUE);
+			ufshcd_release_scsi_cmd(hba, lrbp);
+			scsi_done(cmd);
+		}
+	}
+
+	return true;
+}
+
+static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)
+{
+	struct scsi_device *sdev = rq->q->queuedata;
+	struct Scsi_Host *shost = sdev->host;
+	struct ufs_hba *hba = shost_priv(shost);
+	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
+
+	if (hwq)
+		ufshcd_mcq_poll_cqe_lock(hba, hwq);
+
+	return true;
+}
+
 /**
  * ufshcd_mcq_compl_pending_transfer - MCQ mode function. It is
  * invoked from the error handler context or ufshcd_host_reset_and_restore()
@@ -5728,40 +5770,10 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)
 static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba,
 					      bool force_compl)
 {
-	struct ufs_hw_queue *hwq;
-	struct ufshcd_lrb *lrbp;
-	struct scsi_cmnd *cmd;
-	unsigned long flags;
-	int tag;
-
-	for (tag = 0; tag < hba->nutrs; tag++) {
-		lrbp = &hba->lrb[tag];
-		cmd = lrbp->cmd;
-		if (!ufshcd_cmd_inflight(cmd) ||
-		    test_bit(SCMD_STATE_COMPLETE, &cmd->state))
-			continue;
-
-		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
-		if (!hwq)
-			continue;
-
-		if (force_compl) {
-			ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
-			/*
-			 * For those cmds of which the cqes are not present
-			 * in the cq, complete them explicitly.
-			 */
-			spin_lock_irqsave(&hwq->cq_lock, flags);
-			if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
-				set_host_byte(cmd, DID_REQUEUE);
-				ufshcd_release_scsi_cmd(hba, lrbp);
-				scsi_done(cmd);
-			}
-			spin_unlock_irqrestore(&hwq->cq_lock, flags);
-		} else {
-			ufshcd_mcq_poll_cqe_lock(hba, hwq);
-		}
-	}
+	blk_mq_tagset_busy_iter(&hba->host->tag_set,
+				force_compl ? ufshcd_mcq_force_compl_one :
+					      ufshcd_mcq_compl_one,
+				NULL);
 }
 
 /**

[PATCH v4 16/29] ufs: core: Rework ufshcd_eh_device_reset_handler()

[Bart Van Assche]

Merge the MCQ mode and legacy mode loops into a single loop. This patch
prepares for optimizing the hot path by removing the direct hba->lrb[]
accesses from ufshcd_eh_device_reset_handler().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 84 ++++++++++++++++++---------------------
 1 file changed, 38 insertions(+), 46 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 876c4de86b26..bdc6476694f3 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -7535,6 +7535,36 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 	return err ? : result;
 }
 
+static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)
+{
+	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(req);
+	struct scsi_device *sdev = cmd->device;
+	struct Scsi_Host *shost = sdev->host;
+	struct ufs_hba *hba = shost_priv(shost);
+	const u64 lun = *(u64 *)priv;
+	const u32 tag = req->tag;
+
+	if (sdev->lun != lun)
+		return true;
+
+	if (ufshcd_clear_cmd(hba, tag) < 0) {
+		dev_err(hba->dev, "%s: failed to clear request %d\n", __func__,
+			tag);
+		return true;
+	}
+
+	if (hba->mcq_enabled) {
+		struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, req);
+
+		if (hwq)
+			ufshcd_mcq_poll_cqe_lock(hba, hwq);
+		return true;
+	}
+
+	ufshcd_compl_one_cqe(hba, tag, NULL);
+	return true;
+}
+
 /**
  * ufshcd_eh_device_reset_handler() - Reset a single logical unit.
  * @cmd: SCSI command pointer
@@ -7543,12 +7573,8 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
  */
 static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd)
 {
-	unsigned long flags, pending_reqs = 0, not_cleared = 0;
 	struct Scsi_Host *host;
 	struct ufs_hba *hba;
-	struct ufs_hw_queue *hwq;
-	struct ufshcd_lrb *lrbp;
-	u32 pos, not_cleared_mask = 0;
 	int err;
 	u8 resp = 0xF, lun;
 
@@ -7557,50 +7583,16 @@ static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd)
 
 	lun = ufshcd_scsi_to_upiu_lun(cmd->device->lun);
 	err = ufshcd_issue_tm_cmd(hba, lun, 0, UFS_LOGICAL_RESET, &resp);
-	if (err || resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) {
-		if (!err)
-			err = resp;
-		goto out;
-	}
-
-	if (hba->mcq_enabled) {
-		for (pos = 0; pos < hba->nutrs; pos++) {
-			lrbp = &hba->lrb[pos];
-			if (ufshcd_cmd_inflight(lrbp->cmd) &&
-			    lrbp->lun == lun) {
-				ufshcd_clear_cmd(hba, pos);
-				hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(lrbp->cmd));
-				ufshcd_mcq_poll_cqe_lock(hba, hwq);
-			}
-		}
-		err = 0;
-		goto out;
-	}
-
-	/* clear the commands that were pending for corresponding LUN */
-	spin_lock_irqsave(&hba->outstanding_lock, flags);
-	for_each_set_bit(pos, &hba->outstanding_reqs, hba->nutrs)
-		if (hba->lrb[pos].lun == lun)
-			__set_bit(pos, &pending_reqs);
-	hba->outstanding_reqs &= ~pending_reqs;
-	spin_unlock_irqrestore(&hba->outstanding_lock, flags);
-
-	for_each_set_bit(pos, &pending_reqs, hba->nutrs) {
-		if (ufshcd_clear_cmd(hba, pos) < 0) {
-			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			not_cleared = 1U << pos &
-				ufshcd_readl(hba, REG_UTP_TRANSFER_REQ_DOOR_BELL);
-			hba->outstanding_reqs |= not_cleared;
-			not_cleared_mask |= not_cleared;
-			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
-
-			dev_err(hba->dev, "%s: failed to clear request %d\n",
-				__func__, pos);
-		}
+	if (err) {
+	} else if (resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) {
+		err = resp;
+	} else {
+		/* clear the commands that were pending for corresponding LUN */
+		blk_mq_tagset_busy_iter(&hba->host->tag_set,
+					ufshcd_clear_lu_cmds,
+					&cmd->device->lun);
 	}
-	__ufshcd_transfer_req_compl(hba, pending_reqs & ~not_cleared_mask);
 
-out:
 	hba->req_abort_count = 0;
 	ufshcd_update_evt_hist(hba, UFS_EVT_DEV_RESET, (u32)err);
 	if (!err) {

[PATCH v4 17/29] ufs: core: Rework the SCSI host queue depth calculation code

[Bart Van Assche]

Prepare for allocating the SCSI host earlier by making the SCSI host
queue depth independent of the queue depth supported by the UFS device.
This patch may increase the queue depth of the UFS SCSI host.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufs-mcq.c     | 20 +++++---------------
 drivers/ufs/core/ufshcd-priv.h |  2 +-
 drivers/ufs/core/ufshcd.c      | 17 +++++++++++++++--
 3 files changed, 21 insertions(+), 18 deletions(-)

diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c
index 4eb4f1af7800..b99e482ad8da 100644
--- a/drivers/ufs/core/ufs-mcq.c
+++ b/drivers/ufs/core/ufs-mcq.c
@@ -130,17 +130,15 @@ unsigned int ufshcd_mcq_queue_cfg_addr(struct ufs_hba *hba)
 EXPORT_SYMBOL_GPL(ufshcd_mcq_queue_cfg_addr);
 
 /**
- * ufshcd_mcq_decide_queue_depth - decide the queue depth
+ * ufshcd_get_hba_mac - Maximum number of commands supported by the host
+ *	controller.
  * @hba: per adapter instance
  *
- * Return: queue-depth on success, non-zero on error
+ * Return: queue depth on success; negative upon error.
  *
- * MAC - Max. Active Command of the Host Controller (HC)
- * HC wouldn't send more than this commands to the device.
- * Calculates and adjusts the queue depth based on the depth
- * supported by the HC and ufs device.
+ * MAC = Maximum number of Active Commands supported by the Host Controller.
  */
-int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba)
+int ufshcd_get_hba_mac(struct ufs_hba *hba)
 {
 	int mac;
 
@@ -160,14 +158,6 @@ int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba)
 	if (mac < 0)
 		goto err;
 
-	WARN_ON_ONCE(!hba->dev_info.bqueuedepth);
-	/*
-	 * max. value of bqueuedepth = 256, mac is host dependent.
-	 * It is mandatory for UFS device to define bQueueDepth if
-	 * shared queuing architecture is enabled.
-	 */
-	return min_t(int, mac, hba->dev_info.bqueuedepth);
-
 err:
 	dev_err(hba->dev, "Failed to get mac, err=%d\n", mac);
 	return mac;
diff --git a/drivers/ufs/core/ufshcd-priv.h b/drivers/ufs/core/ufshcd-priv.h
index d0a2c963a27d..ec1bb818bc73 100644
--- a/drivers/ufs/core/ufshcd-priv.h
+++ b/drivers/ufs/core/ufshcd-priv.h
@@ -65,7 +65,7 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 			  struct cq_entry *cqe);
 int ufshcd_mcq_init(struct ufs_hba *hba);
 void ufshcd_mcq_disable(struct ufs_hba *hba);
-int ufshcd_mcq_decide_queue_depth(struct ufs_hba *hba);
+int ufshcd_get_hba_mac(struct ufs_hba *hba);
 int ufshcd_mcq_memory_alloc(struct ufs_hba *hba);
 struct ufs_hw_queue *ufshcd_mcq_req_to_hwq(struct ufs_hba *hba,
 					   struct request *req);
diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index bdc6476694f3..f54599490c86 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -8435,10 +8435,11 @@ static void ufs_init_rtc(struct ufs_hba *hba, u8 *desc_buf)
 
 static int ufs_get_device_desc(struct ufs_hba *hba)
 {
+	struct ufs_dev_info *dev_info = &hba->dev_info;
+	struct Scsi_Host *shost = hba->host;
 	int err;
 	u8 model_index;
 	u8 *desc_buf;
-	struct ufs_dev_info *dev_info = &hba->dev_info;
 
 	desc_buf = kzalloc(QUERY_DESC_MAX_SIZE, GFP_KERNEL);
 	if (!desc_buf) {
@@ -8466,6 +8467,18 @@ static int ufs_get_device_desc(struct ufs_hba *hba)
 				      desc_buf[DEVICE_DESC_PARAM_SPEC_VER + 1];
 	dev_info->bqueuedepth = desc_buf[DEVICE_DESC_PARAM_Q_DPTH];
 
+	/*
+	 * According to the UFS standard, the UFS device queue depth
+	 * (bQueueDepth) must be in the range 1..255 if the shared queueing
+	 * architecture is supported. bQueueDepth is zero if the shared queueing
+	 * architecture is not supported.
+	 */
+	if (dev_info->bqueuedepth)
+		shost->cmd_per_lun = min(hba->nutrs, dev_info->bqueuedepth) -
+				     UFSHCD_NUM_RESERVED;
+	else
+		shost->cmd_per_lun = shost->can_queue;
+
 	dev_info->rtt_cap = desc_buf[DEVICE_DESC_PARAM_RTT_CAP];
 
 	dev_info->hid_sup = get_unaligned_be32(desc_buf +
@@ -8871,7 +8884,7 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba)
 	int ret;
 	int old_nutrs = hba->nutrs;
 
-	ret = ufshcd_mcq_decide_queue_depth(hba);
+	ret = ufshcd_get_hba_mac(hba);
 	if (ret < 0)
 		return ret;
 

[PATCH v4 18/29] ufs: core: Allocate the SCSI host earlier

[Bart Van Assche]

Call ufshcd_add_scsi_host() before any UPIU commands are sent to the UFS
device. This patch prepares for letting ufshcd_add_scsi_host() allocate
memory for both SCSI and UPIU commands.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index f54599490c86..161d54a532fe 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -10577,6 +10577,9 @@ static int ufshcd_add_scsi_host(struct ufs_hba *hba)
 {
 	int err;
 
+	WARN_ON_ONCE(!hba->host->can_queue);
+	WARN_ON_ONCE(!hba->host->cmd_per_lun);
+
 	if (is_mcq_supported(hba)) {
 		ufshcd_mcq_enable(hba);
 		err = ufshcd_alloc_mcq(hba);
@@ -10727,7 +10730,11 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 	ufshcd_host_memory_configure(hba);
 
 	host->can_queue = hba->nutrs - UFSHCD_NUM_RESERVED;
-	host->cmd_per_lun = hba->nutrs - UFSHCD_NUM_RESERVED;
+	/*
+	 * Set the queue depth for WLUNs. ufs_get_device_desc() will increase
+	 * host->cmd_per_lun to a larger value.
+	 */
+	host->cmd_per_lun = 1;
 	host->max_id = UFSHCD_MAX_ID;
 	host->max_lun = UFS_MAX_LUNS;
 	host->max_channel = UFSHCD_MAX_CHANNEL;
@@ -10818,6 +10825,10 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 			    FIELD_PREP(UFSHCI_AHIBERN8_SCALE_MASK, 3);
 	}
 
+	err = ufshcd_add_scsi_host(hba);
+	if (err)
+		goto out_disable;
+
 	/* Hold auto suspend until async scan completes */
 	pm_runtime_get_sync(dev);
 
@@ -10868,10 +10879,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 	if (err)
 		goto out_disable;
 
-	err = ufshcd_add_scsi_host(hba);
-	if (err)
-		goto out_disable;
-
 	async_schedule(ufshcd_async_scan, hba);
 	ufs_sysfs_add_nodes(hba->dev);
 

[PATCH v4 19/29] ufs: core: Call ufshcd_init_lrb() later

[Bart Van Assche]

Call ufshcd_init_lrb() from inside ufshcd_setup_dev_cmd() instead of
ufshcd_host_memory_configure(). This patch prepares for calling
ufshcd_host_memory_configure() before the information is available
that is required to call ufshcd_setup_dev_cmd().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 53 ++++++++++++++++++++-------------------
 1 file changed, 27 insertions(+), 26 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 161d54a532fe..c43fd0a224f4 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2900,8 +2900,32 @@ static void ufshcd_comp_scsi_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	ufshcd_prepare_utp_scsi_cmd_upiu(lrbp, upiu_flags);
 }
 
-static void __ufshcd_setup_cmd(struct ufshcd_lrb *lrbp, struct scsi_cmnd *cmd, u8 lun, int tag)
+static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
+{
+	struct utp_transfer_cmd_desc *cmd_descp =
+		(void *)hba->ucdl_base_addr + i * ufshcd_get_ucd_size(hba);
+	struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;
+	dma_addr_t cmd_desc_element_addr =
+		hba->ucdl_dma_addr + i * ufshcd_get_ucd_size(hba);
+	u16 response_offset = le16_to_cpu(utrdlp[i].response_upiu_offset);
+	u16 prdt_offset = le16_to_cpu(utrdlp[i].prd_table_offset);
+
+	lrb->utr_descriptor_ptr = utrdlp + i;
+	lrb->utrd_dma_addr =
+		hba->utrdl_dma_addr + i * sizeof(struct utp_transfer_req_desc);
+	lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp->command_upiu;
+	lrb->ucd_req_dma_addr = cmd_desc_element_addr;
+	lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu;
+	lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset;
+	lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table;
+	lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset;
+}
+
+static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
+			       struct scsi_cmnd *cmd, u8 lun, int tag)
 {
+	ufshcd_init_lrb(hba, lrbp, tag);
+
 	memset(lrbp->ucd_req_ptr, 0, sizeof(*lrbp->ucd_req_ptr));
 
 	lrbp->cmd = cmd;
@@ -2913,7 +2937,7 @@ static void __ufshcd_setup_cmd(struct ufshcd_lrb *lrbp, struct scsi_cmnd *cmd, u
 static void ufshcd_setup_scsi_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 				  struct scsi_cmnd *cmd, u8 lun, int tag)
 {
-	__ufshcd_setup_cmd(lrbp, cmd, lun, tag);
+	__ufshcd_setup_cmd(hba, lrbp, cmd, lun, tag);
 	lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed(hba);
 	lrbp->req_abort_skip = false;
 
@@ -2968,27 +2992,6 @@ static void ufshcd_map_queues(struct Scsi_Host *shost)
 	}
 }
 
-static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
-{
-	struct utp_transfer_cmd_desc *cmd_descp = (void *)hba->ucdl_base_addr +
-		i * ufshcd_get_ucd_size(hba);
-	struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;
-	dma_addr_t cmd_desc_element_addr = hba->ucdl_dma_addr +
-		i * ufshcd_get_ucd_size(hba);
-	u16 response_offset = le16_to_cpu(utrdlp[i].response_upiu_offset);
-	u16 prdt_offset = le16_to_cpu(utrdlp[i].prd_table_offset);
-
-	lrb->utr_descriptor_ptr = utrdlp + i;
-	lrb->utrd_dma_addr = hba->utrdl_dma_addr +
-		i * sizeof(struct utp_transfer_req_desc);
-	lrb->ucd_req_ptr = (struct utp_upiu_req *)cmd_descp->command_upiu;
-	lrb->ucd_req_dma_addr = cmd_desc_element_addr;
-	lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp->response_upiu;
-	lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset;
-	lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp->prd_table;
-	lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset;
-}
-
 /**
  * ufshcd_queuecommand - main entry point for SCSI requests
  * @host: SCSI host pointer
@@ -3081,7 +3084,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 			     enum dev_cmd_type cmd_type, u8 lun, int tag)
 {
-	__ufshcd_setup_cmd(lrbp, NULL, lun, tag);
+	__ufshcd_setup_cmd(hba, lrbp, NULL, lun, tag);
 	lrbp->intr_cmd = true; /* No interrupt aggregation */
 	hba->dev_cmd.type = cmd_type;
 }
@@ -4039,8 +4042,6 @@ static void ufshcd_host_memory_configure(struct ufs_hba *hba)
 			utrdlp[i].response_upiu_length =
 				cpu_to_le16(ALIGNED_UPIU_SIZE >> 2);
 		}
-
-		ufshcd_init_lrb(hba, &hba->lrb[i], i);
 	}
 }
 

[PATCH v4 20/29] ufs: core: Use hba->reserved_slot

[Bart Van Assche]

Use hba->reserved_slot instead of open-coding it. This patch prepares
for changing the value of hba->reserved_slot.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufs-mcq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c
index b99e482ad8da..9b6674d6f21c 100644
--- a/drivers/ufs/core/ufs-mcq.c
+++ b/drivers/ufs/core/ufs-mcq.c
@@ -536,7 +536,7 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 	if (hba->quirks & UFSHCD_QUIRK_MCQ_BROKEN_RTC)
 		return -ETIMEDOUT;
 
-	if (task_tag != hba->nutrs - UFSHCD_NUM_RESERVED) {
+	if (task_tag != hba->reserved_slot) {
 		if (!cmd)
 			return -EINVAL;
 		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));

[PATCH v4 21/29] ufs: core: Make the reserved slot a reserved request

[Bart Van Assche]

Instead of letting the SCSI core allocate hba->nutrs - 1 commands, let
the SCSI core allocate hba->nutrs commands, set the number of reserved
tags to 1 and use the reserved tag for device management commands. This
patch changes the 'reserved slot' from hba->nutrs - 1 into 0 because
the block layer reserves the smallest tags for reserved commands.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index c43fd0a224f4..ce766295d4ca 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2472,7 +2472,7 @@ static inline int ufshcd_hba_capabilities(struct ufs_hba *hba)
 	hba->nutrs = (hba->capabilities & MASK_TRANSFER_REQUESTS_SLOTS_SDB) + 1;
 	hba->nutmrs =
 	((hba->capabilities & MASK_TASK_MANAGEMENT_REQUEST_SLOTS) >> 16) + 1;
-	hba->reserved_slot = hba->nutrs - 1;
+	hba->reserved_slot = 0;
 
 	hba->nortt = FIELD_GET(MASK_NUMBER_OUTSTANDING_RTT, hba->capabilities) + 1;
 
@@ -8911,7 +8911,8 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba)
 		goto err;
 
 	hba->host->can_queue = hba->nutrs - UFSHCD_NUM_RESERVED;
-	hba->reserved_slot = hba->nutrs - UFSHCD_NUM_RESERVED;
+	hba->host->nr_reserved_cmds = UFSHCD_NUM_RESERVED;
+	hba->reserved_slot = 0;
 
 	return 0;
 err:
@@ -10736,6 +10737,7 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 	 * host->cmd_per_lun to a larger value.
 	 */
 	host->cmd_per_lun = 1;
+	host->nr_reserved_cmds = UFSHCD_NUM_RESERVED;
 	host->max_id = UFSHCD_MAX_ID;
 	host->max_lun = UFS_MAX_LUNS;
 	host->max_channel = UFSHCD_MAX_CHANNEL;

[PATCH v4 22/29] ufs: core: Do not clear driver-private command data

[Bart Van Assche]

Tell the SCSI core to skip the memset() call that clears driver-private
data because __ufshcd_setup_cmd() performs all necessary initialization.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index ce766295d4ca..f25671ae1b30 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2992,6 +2992,15 @@ static void ufshcd_map_queues(struct Scsi_Host *shost)
 	}
 }
 
+/*
+ * The only purpose of this function is to make the SCSI core skip the memset()
+ * call for the private command data.
+ */
+static int ufshcd_init_cmd_priv(struct Scsi_Host *host, struct scsi_cmnd *cmd)
+{
+	return 0;
+}
+
 /**
  * ufshcd_queuecommand - main entry point for SCSI requests
  * @host: SCSI host pointer
@@ -9155,6 +9164,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
 	.name			= UFSHCD,
 	.proc_name		= UFSHCD,
 	.map_queues		= ufshcd_map_queues,
+	.init_cmd_priv		= ufshcd_init_cmd_priv,
 	.queuecommand		= ufshcd_queuecommand,
 	.mq_poll		= ufshcd_poll,
 	.sdev_init		= ufshcd_sdev_init,

[PATCH v4 23/29] ufs: core: Optimize the hot path

[Bart Van Assche]

Set .cmd_size in the SCSI host template such that the SCSI core makes
struct scsi_cmnd and struct ufshcd_lrb adjacent. Convert the cmd->lrbp
and lrbp->cmd memory loads into pointer offset calculations. Remove the
data structure members that became superfluous, namely ufshcd_lrb.cmd
and ufs_hba.lrb.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufs-mcq.c       |   9 +-
 drivers/ufs/core/ufshcd-crypto.h |  18 ++-
 drivers/ufs/core/ufshcd-priv.h   |  41 +++++-
 drivers/ufs/core/ufshcd.c        | 232 ++++++++++++++++---------------
 include/ufs/ufshcd.h             |   5 -
 5 files changed, 177 insertions(+), 128 deletions(-)

diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c
index 9b6674d6f21c..c6c6cca400de 100644
--- a/drivers/ufs/core/ufs-mcq.c
+++ b/drivers/ufs/core/ufs-mcq.c
@@ -526,8 +526,8 @@ static int ufshcd_mcq_sq_start(struct ufs_hba *hba, struct ufs_hw_queue *hwq)
  */
 int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
-	struct scsi_cmnd *cmd = lrbp->cmd;
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct ufs_hw_queue *hwq;
 	void __iomem *reg, *opr_sqd_base;
 	u32 nexus, id, val;
@@ -612,7 +612,8 @@ static void ufshcd_mcq_nullify_sqe(struct utp_transfer_req_desc *utrd)
 static bool ufshcd_mcq_sqe_search(struct ufs_hba *hba,
 				  struct ufs_hw_queue *hwq, int task_tag)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_transfer_req_desc *utrd;
 	__le64  cmd_desc_base_addr;
 	bool ret = false;
@@ -663,7 +664,7 @@ int ufshcd_mcq_abort(struct scsi_cmnd *cmd)
 	struct Scsi_Host *host = cmd->device->host;
 	struct ufs_hba *hba = shost_priv(host);
 	int tag = scsi_cmd_to_rq(cmd)->tag;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct ufs_hw_queue *hwq;
 	int err;
 
diff --git a/drivers/ufs/core/ufshcd-crypto.h b/drivers/ufs/core/ufshcd-crypto.h
index 89bb97c14c15..c148a5194378 100644
--- a/drivers/ufs/core/ufshcd-crypto.h
+++ b/drivers/ufs/core/ufshcd-crypto.h
@@ -38,10 +38,10 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp,
 }
 
 static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba,
-					  struct ufshcd_lrb *lrbp)
+					  struct scsi_cmnd *cmd)
 {
-	struct scsi_cmnd *cmd = lrbp->cmd;
 	const struct bio_crypt_ctx *crypt_ctx = scsi_cmd_to_rq(cmd)->crypt_ctx;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 
 	if (crypt_ctx && hba->vops && hba->vops->fill_crypto_prdt)
 		return hba->vops->fill_crypto_prdt(hba, crypt_ctx,
@@ -51,17 +51,19 @@ static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba,
 }
 
 static inline void ufshcd_crypto_clear_prdt(struct ufs_hba *hba,
-					    struct ufshcd_lrb *lrbp)
+					    struct scsi_cmnd *cmd)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
 	if (!(hba->quirks & UFSHCD_QUIRK_KEYS_IN_PRDT))
 		return;
 
-	if (!(scsi_cmd_to_rq(lrbp->cmd)->crypt_ctx))
+	if (!(scsi_cmd_to_rq(cmd)->crypt_ctx))
 		return;
 
 	/* Zeroize the PRDT because it can contain cryptographic keys. */
 	memzero_explicit(lrbp->ucd_prdt_ptr,
-			 ufshcd_sg_entry_size(hba) * scsi_sg_count(lrbp->cmd));
+			 ufshcd_sg_entry_size(hba) * scsi_sg_count(cmd));
 }
 
 bool ufshcd_crypto_enable(struct ufs_hba *hba);
@@ -82,13 +84,15 @@ ufshcd_prepare_req_desc_hdr_crypto(struct ufshcd_lrb *lrbp,
 				   struct request_desc_header *h) { }
 
 static inline int ufshcd_crypto_fill_prdt(struct ufs_hba *hba,
-					  struct ufshcd_lrb *lrbp)
+					  struct scsi_cmnd *cmd)
 {
 	return 0;
 }
 
 static inline void ufshcd_crypto_clear_prdt(struct ufs_hba *hba,
-					    struct ufshcd_lrb *lrbp) { }
+					    struct scsi_cmnd *cmd)
+{
+}
 
 static inline bool ufshcd_crypto_enable(struct ufs_hba *hba)
 {
diff --git a/drivers/ufs/core/ufshcd-priv.h b/drivers/ufs/core/ufshcd-priv.h
index ec1bb818bc73..3222c4d3ceb4 100644
--- a/drivers/ufs/core/ufshcd-priv.h
+++ b/drivers/ufs/core/ufshcd-priv.h
@@ -75,8 +75,7 @@ bool ufshcd_cmd_inflight(struct scsi_cmnd *cmd);
 int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag);
 int ufshcd_mcq_abort(struct scsi_cmnd *cmd);
 int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag);
-void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
-			     struct ufshcd_lrb *lrbp);
+void ufshcd_release_scsi_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd);
 
 #define SD_ASCII_STD true
 #define SD_RAW false
@@ -361,6 +360,44 @@ static inline bool ufs_is_valid_unit_desc_lun(struct ufs_dev_info *dev_info, u8
 	return lun == UFS_UPIU_RPMB_WLUN || (lun < dev_info->max_lu_supported);
 }
 
+/*
+ * Convert a block layer tag into a SCSI command pointer. This function is
+ * called once per I/O completion path and is also called from error paths.
+ */
+static inline struct scsi_cmnd *ufshcd_tag_to_cmd(struct ufs_hba *hba, u32 tag)
+{
+	struct blk_mq_tags *tags = hba->host->tag_set.shared_tags;
+	struct request *rq;
+
+	/*
+	 * Handle reserved tags differently because the UFS driver does not
+	 * call blk_mq_alloc_request() for allocating reserved requests.
+	 * Allocating reserved tags with blk_mq_alloc_request() would require
+	 * the following:
+	 * - Allocate an additional request queue from &hba->host->tag_set for
+	 *   allocating reserved requests from.
+	 * - For that request queue, allocate a SCSI device.
+	 * - Calling blk_mq_alloc_request(hba->dev_mgmt_queue, REQ_OP_DRV_OUT,
+	 *   BLK_MQ_REQ_RESERVED) for allocating a reserved request and
+	 *   blk_mq_free_request() for freeing reserved requests.
+	 * - Set the .device pointer for these reserved requests.
+	 * - Submit reserved requests with blk_execute_rq().
+	 * - Modify ufshcd_queuecommand() such that it handles reserved requests
+	 *   in another way than SCSI requests.
+	 * - Modify ufshcd_compl_one_cqe() such that it calls scsi_done() for
+	 *   device management commands.
+	 * - Modify all callback functions called by blk_mq_tagset_busy_iter()
+	 *   calls in the UFS driver and skip device management commands.
+	 */
+	rq = tag < UFSHCD_NUM_RESERVED ? tags->static_rqs[tag] :
+					 blk_mq_tag_to_rq(tags, tag);
+
+	if (WARN_ON_ONCE(!rq))
+		return NULL;
+
+	return blk_mq_rq_to_pdu(rq);
+}
+
 static inline void ufshcd_inc_sq_tail(struct ufs_hw_queue *q)
 	__must_hold(&q->sq_lock)
 {
diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index f25671ae1b30..3990b161ef78 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -28,6 +28,7 @@
 #include <scsi/scsi_dbg.h>
 #include <scsi/scsi_driver.h>
 #include <scsi/scsi_eh.h>
+#include <scsi/scsi_tcq.h>
 #include "ufshcd-priv.h"
 #include <ufs/ufs_quirks.h>
 #include <ufs/unipro.h>
@@ -485,7 +486,7 @@ static void ufshcd_add_command_trace(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 	u32 hwq_id = 0;
 	struct request *rq = scsi_cmd_to_rq(cmd);
 	unsigned int tag = rq->tag;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int transfer_len = -1;
 
 	/* trace UPIU also */
@@ -596,14 +597,13 @@ static void ufshcd_print_evt_hist(struct ufs_hba *hba)
 	ufshcd_vops_dbg_register_dump(hba);
 }
 
-static
-void ufshcd_print_tr(struct ufs_hba *hba, int tag, bool pr_prdt)
+static void ufshcd_print_tr(struct ufs_hba *hba, struct scsi_cmnd *cmd,
+			    bool pr_prdt)
 {
-	const struct ufshcd_lrb *lrbp;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = lrbp->task_tag;
 	int prdt_length;
 
-	lrbp = &hba->lrb[tag];
-
 	if (hba->monitor.enabled) {
 		dev_err(hba->dev, "UPIU[%d] - issue time %lld us\n", tag,
 			div_u64(lrbp->issue_time_stamp_local_clock, 1000));
@@ -646,7 +646,7 @@ static bool ufshcd_print_tr_iter(struct request *req, void *priv)
 	struct Scsi_Host *shost = sdev->host;
 	struct ufs_hba *hba = shost_priv(shost);
 
-	ufshcd_print_tr(hba, req->tag, *(bool *)priv);
+	ufshcd_print_tr(hba, blk_mq_rq_to_pdu(req), *(bool *)priv);
 
 	return true;
 }
@@ -2294,8 +2294,7 @@ static inline bool ufshcd_should_inform_monitor(struct ufs_hba *hba,
 						struct scsi_cmnd *cmd)
 {
 	const struct ufs_hba_monitor *m = &hba->monitor;
-	struct request *rq = scsi_cmd_to_rq(cmd);
-	struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag];
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 
 	return m->enabled &&
 	       (!m->chunk_size || m->chunk_size == cmd->sdb.length) &&
@@ -2316,7 +2315,7 @@ static void ufshcd_start_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 static void ufshcd_update_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
 	struct request *req = scsi_cmd_to_rq(cmd);
-	struct ufshcd_lrb *lrbp = &hba->lrb[req->tag];
+	const struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int dir = ufshcd_monitor_opcode2dir(cmd->cmnd[0]);
 	unsigned long flags;
 
@@ -2346,17 +2345,26 @@ static void ufshcd_update_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 	spin_unlock_irqrestore(hba->host->host_lock, flags);
 }
 
+/*
+ * Returns %true if @cmd represents a SCSI command that is in flight and %false
+ * if it represents a device management command.
+ */
+static bool ufshcd_is_scsi_cmd(struct scsi_cmnd *cmd)
+{
+	return blk_mq_request_started(scsi_cmd_to_rq(cmd));
+}
+
 /**
  * ufshcd_send_command - Send SCSI or device management commands
  * @hba: per adapter instance
- * @lrbp: Local reference block of SCSI command
+ * @cmd: SCSI command or device management command pointer
  * @hwq: pointer to hardware queue instance
  */
 static inline void ufshcd_send_command(struct ufs_hba *hba,
-				       struct ufshcd_lrb *lrbp,
+				       struct scsi_cmnd *cmd,
 				       struct ufs_hw_queue *hwq)
 {
-	struct scsi_cmnd *cmd = lrbp->cmd;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	unsigned long flags;
 
 	if (hba->monitor.enabled) {
@@ -2365,7 +2373,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 		lrbp->compl_time_stamp = ktime_set(0, 0);
 		lrbp->compl_time_stamp_local_clock = 0;
 	}
-	if (cmd) {
+	if (ufshcd_is_scsi_cmd(cmd)) {
 		ufshcd_add_command_trace(hba, cmd, UFS_CMD_SEND);
 		ufshcd_clk_scaling_start_busy(hba);
 		if (unlikely(ufshcd_should_inform_monitor(hba, cmd)))
@@ -2385,7 +2393,8 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	} else {
 		spin_lock_irqsave(&hba->outstanding_lock, flags);
 		if (hba->vops && hba->vops->setup_xfer_req)
-			hba->vops->setup_xfer_req(hba, lrbp->task_tag, !!cmd);
+			hba->vops->setup_xfer_req(hba, lrbp->task_tag,
+						  ufshcd_is_scsi_cmd(cmd));
 		__set_bit(lrbp->task_tag, &hba->outstanding_reqs);
 		ufshcd_writel(hba, 1 << lrbp->task_tag,
 			      REG_UTP_TRANSFER_REQ_DOOR_BELL);
@@ -2395,11 +2404,12 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 
 /**
  * ufshcd_copy_sense_data - Copy sense data in case of check condition
- * @lrbp: pointer to local reference block
+ * @cmd: SCSI command
  */
-static inline void ufshcd_copy_sense_data(struct ufshcd_lrb *lrbp)
+static inline void ufshcd_copy_sense_data(struct scsi_cmnd *cmd)
 {
-	u8 *const sense_buffer = lrbp->cmd->sense_buffer;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	u8 *const sense_buffer = cmd->sense_buffer;
 	u16 resp_len;
 	int len;
 
@@ -2704,13 +2714,13 @@ static void ufshcd_sgl_to_prdt(struct ufs_hba *hba, struct ufshcd_lrb *lrbp, int
 /**
  * ufshcd_map_sg - Map scatter-gather list to prdt
  * @hba: per adapter instance
- * @lrbp: pointer to local reference block
+ * @cmd: SCSI command
  *
  * Return: 0 in case of success, non-zero value in case of failure.
  */
-static int ufshcd_map_sg(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+static int ufshcd_map_sg(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	struct scsi_cmnd *cmd = lrbp->cmd;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int sg_segments = scsi_dma_map(cmd);
 
 	if (sg_segments < 0)
@@ -2718,7 +2728,7 @@ static int ufshcd_map_sg(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 
 	ufshcd_sgl_to_prdt(hba, lrbp, sg_segments, scsi_sglist(cmd));
 
-	return ufshcd_crypto_fill_prdt(hba, lrbp);
+	return ufshcd_crypto_fill_prdt(hba, cmd);
 }
 
 /**
@@ -2777,13 +2787,13 @@ ufshcd_prepare_req_desc_hdr(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 /**
  * ufshcd_prepare_utp_scsi_cmd_upiu() - fills the utp_transfer_req_desc,
  * for scsi commands
- * @lrbp: local reference block pointer
+ * @cmd: SCSI command
  * @upiu_flags: flags
  */
-static
-void ufshcd_prepare_utp_scsi_cmd_upiu(struct ufshcd_lrb *lrbp, u8 upiu_flags)
+static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,
+					     u8 upiu_flags)
 {
-	struct scsi_cmnd *cmd = lrbp->cmd;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
 	unsigned short cdb_len;
 
@@ -2886,22 +2896,25 @@ static int ufshcd_compose_devman_upiu(struct ufs_hba *hba,
  * ufshcd_comp_scsi_upiu - UFS Protocol Information Unit(UPIU)
  *			   for SCSI Purposes
  * @hba: per adapter instance
- * @lrbp: pointer to local reference block
+ * @cmd: SCSI command
  */
-static void ufshcd_comp_scsi_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+static void ufshcd_comp_scsi_upiu(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	struct request *rq = scsi_cmd_to_rq(lrbp->cmd);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	struct request *rq = scsi_cmd_to_rq(cmd);
 	unsigned int ioprio_class = IOPRIO_PRIO_CLASS(req_get_ioprio(rq));
 	u8 upiu_flags;
 
-	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags, lrbp->cmd->sc_data_direction, 0);
+	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags,
+				    cmd->sc_data_direction, 0);
 	if (ioprio_class == IOPRIO_CLASS_RT)
 		upiu_flags |= UPIU_CMD_FLAGS_CP;
-	ufshcd_prepare_utp_scsi_cmd_upiu(lrbp, upiu_flags);
+	ufshcd_prepare_utp_scsi_cmd_upiu(cmd, upiu_flags);
 }
 
-static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
+static void ufshcd_init_lrb(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
+	const int i = scsi_cmd_to_rq(cmd)->tag;
 	struct utp_transfer_cmd_desc *cmd_descp =
 		(void *)hba->ucdl_base_addr + i * ufshcd_get_ucd_size(hba);
 	struct utp_transfer_req_desc *utrdlp = hba->utrdl_base_addr;
@@ -2909,6 +2922,7 @@ static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
 		hba->ucdl_dma_addr + i * ufshcd_get_ucd_size(hba);
 	u16 response_offset = le16_to_cpu(utrdlp[i].response_upiu_offset);
 	u16 prdt_offset = le16_to_cpu(utrdlp[i].prd_table_offset);
+	struct ufshcd_lrb *lrb = scsi_cmd_priv(cmd);
 
 	lrb->utr_descriptor_ptr = utrdlp + i;
 	lrb->utrd_dma_addr =
@@ -2921,27 +2935,30 @@ static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int i)
 	lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset;
 }
 
-static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
-			       struct scsi_cmnd *cmd, u8 lun, int tag)
+static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
+			       u8 lun, int tag)
 {
-	ufshcd_init_lrb(hba, lrbp, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
+	ufshcd_init_lrb(hba, cmd);
 
 	memset(lrbp->ucd_req_ptr, 0, sizeof(*lrbp->ucd_req_ptr));
 
-	lrbp->cmd = cmd;
 	lrbp->task_tag = tag;
 	lrbp->lun = lun;
 	ufshcd_prepare_lrbp_crypto(cmd ? scsi_cmd_to_rq(cmd) : NULL, lrbp);
 }
 
-static void ufshcd_setup_scsi_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
-				  struct scsi_cmnd *cmd, u8 lun, int tag)
+static void ufshcd_setup_scsi_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
+				  u8 lun, int tag)
 {
-	__ufshcd_setup_cmd(hba, lrbp, cmd, lun, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
+	__ufshcd_setup_cmd(hba, cmd, lun, tag);
 	lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed(hba);
 	lrbp->req_abort_skip = false;
 
-	ufshcd_comp_scsi_upiu(hba, lrbp);
+	ufshcd_comp_scsi_upiu(hba, cmd);
 }
 
 /**
@@ -3012,7 +3029,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 {
 	struct ufs_hba *hba = shost_priv(host);
 	int tag = scsi_cmd_to_rq(cmd)->tag;
-	struct ufshcd_lrb *lrbp;
 	int err = 0;
 	struct ufs_hw_queue *hwq = NULL;
 
@@ -3063,11 +3079,10 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 
 	ufshcd_hold(hba);
 
-	lrbp = &hba->lrb[tag];
-
-	ufshcd_setup_scsi_cmd(hba, lrbp, cmd, ufshcd_scsi_to_upiu_lun(cmd->device->lun), tag);
+	ufshcd_setup_scsi_cmd(hba, cmd,
+			      ufshcd_scsi_to_upiu_lun(cmd->device->lun), tag);
 
-	err = ufshcd_map_sg(hba, lrbp);
+	err = ufshcd_map_sg(hba, cmd);
 	if (err) {
 		ufshcd_release(hba);
 		goto out;
@@ -3076,7 +3091,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	if (hba->mcq_enabled)
 		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
 
-	ufshcd_send_command(hba, lrbp, hwq);
+	ufshcd_send_command(hba, cmd, hwq);
 
 out:
 	if (ufs_trigger_eh(hba)) {
@@ -3090,10 +3105,12 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	return err;
 }
 
-static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
-			     enum dev_cmd_type cmd_type, u8 lun, int tag)
+static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
+				 enum dev_cmd_type cmd_type, u8 lun, int tag)
 {
-	__ufshcd_setup_cmd(hba, lrbp, NULL, lun, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
+	__ufshcd_setup_cmd(hba, cmd, lun, tag);
 	lrbp->intr_cmd = true; /* No interrupt aggregation */
 	hba->dev_cmd.type = cmd_type;
 }
@@ -3101,10 +3118,12 @@ static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 /*
  * Return: 0 upon success; < 0 upon failure.
  */
-static int ufshcd_compose_dev_cmd(struct ufs_hba *hba,
-		struct ufshcd_lrb *lrbp, enum dev_cmd_type cmd_type, int tag)
+static int ufshcd_compose_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
+				  enum dev_cmd_type cmd_type, int tag)
 {
-	ufshcd_setup_dev_cmd(hba, lrbp, cmd_type, 0, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
+	ufshcd_setup_dev_cmd(hba, cmd, cmd_type, 0, tag);
 
 	return ufshcd_compose_devman_upiu(hba, lrbp);
 }
@@ -3315,13 +3334,14 @@ static void ufshcd_dev_man_unlock(struct ufs_hba *hba)
 /*
  * Return: 0 upon success; < 0 upon failure.
  */
-static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
-			  const u32 tag, int timeout)
+static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
+				const u32 tag, int timeout)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int err;
 
 	ufshcd_add_query_upiu_trace(hba, UFS_QUERY_SEND, lrbp->ucd_req_ptr);
-	ufshcd_send_command(hba, lrbp, hba->dev_cmd_queue);
+	ufshcd_send_command(hba, cmd, hba->dev_cmd_queue);
 	err = ufshcd_wait_for_dev_cmd(hba, lrbp, timeout);
 
 	ufshcd_add_query_upiu_trace(hba, err ? UFS_QUERY_ERR : UFS_QUERY_COMP,
@@ -3345,17 +3365,17 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,
 		enum dev_cmd_type cmd_type, int timeout)
 {
 	const u32 tag = hba->reserved_slot;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
 	int err;
 
 	/* Protects use of hba->reserved_slot. */
 	lockdep_assert_held(&hba->dev_cmd.lock);
 
-	err = ufshcd_compose_dev_cmd(hba, lrbp, cmd_type, tag);
+	err = ufshcd_compose_dev_cmd(hba, cmd, cmd_type, tag);
 	if (unlikely(err))
 		return err;
 
-	return ufshcd_issue_dev_cmd(hba, lrbp, tag, timeout);
+	return ufshcd_issue_dev_cmd(hba, cmd, tag, timeout);
 }
 
 /**
@@ -3982,14 +4002,6 @@ static int ufshcd_memory_alloc(struct ufs_hba *hba)
 	}
 
 skip_utmrdl:
-	/* Allocate memory for local reference block */
-	hba->lrb = devm_kcalloc(hba->dev,
-				hba->nutrs, sizeof(struct ufshcd_lrb),
-				GFP_KERNEL);
-	if (!hba->lrb) {
-		dev_err(hba->dev, "LRB Memory allocation failed\n");
-		goto out;
-	}
 	return 0;
 out:
 	return -ENOMEM;
@@ -5400,19 +5412,18 @@ static void ufshcd_sdev_destroy(struct scsi_device *sdev)
 
 /**
  * ufshcd_scsi_cmd_status - Update SCSI command result based on SCSI status
- * @lrbp: pointer to local reference block of completed command
+ * @cmd: SCSI command
  * @scsi_status: SCSI command status
  *
  * Return: value base on SCSI command status.
  */
-static inline int
-ufshcd_scsi_cmd_status(struct ufshcd_lrb *lrbp, int scsi_status)
+static inline int ufshcd_scsi_cmd_status(struct scsi_cmnd *cmd, int scsi_status)
 {
 	int result = 0;
 
 	switch (scsi_status) {
 	case SAM_STAT_CHECK_CONDITION:
-		ufshcd_copy_sense_data(lrbp);
+		ufshcd_copy_sense_data(cmd);
 		fallthrough;
 	case SAM_STAT_GOOD:
 		result |= DID_OK << 16 | scsi_status;
@@ -5420,7 +5431,7 @@ ufshcd_scsi_cmd_status(struct ufshcd_lrb *lrbp, int scsi_status)
 	case SAM_STAT_TASK_SET_FULL:
 	case SAM_STAT_BUSY:
 	case SAM_STAT_TASK_ABORTED:
-		ufshcd_copy_sense_data(lrbp);
+		ufshcd_copy_sense_data(cmd);
 		result |= scsi_status;
 		break;
 	default:
@@ -5434,15 +5445,16 @@ ufshcd_scsi_cmd_status(struct ufshcd_lrb *lrbp, int scsi_status)
 /**
  * ufshcd_transfer_rsp_status - Get overall status of the response
  * @hba: per adapter instance
- * @lrbp: pointer to local reference block of completed command
+ * @cmd: SCSI command
  * @cqe: pointer to the completion queue entry
  *
  * Return: result of the command to notify SCSI midlayer.
  */
-static inline int
-ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
-			   struct cq_entry *cqe)
+static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
+					     struct scsi_cmnd *cmd,
+					     struct cq_entry *cqe)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int result = 0;
 	int scsi_status;
 	enum utp_ocs ocs;
@@ -5456,7 +5468,7 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 	 * not set either flag.
 	 */
 	if (resid && !(upiu_flags & UPIU_RSP_FLAG_OVERFLOW))
-		scsi_set_resid(lrbp->cmd, resid);
+		scsi_set_resid(cmd, resid);
 
 	/* overall command status of utrd */
 	ocs = ufshcd_get_tr_ocs(lrbp, cqe);
@@ -5477,7 +5489,7 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 			 * to notify the SCSI midlayer of the command status
 			 */
 			scsi_status = lrbp->ucd_rsp_ptr->header.status;
-			result = ufshcd_scsi_cmd_status(lrbp, scsi_status);
+			result = ufshcd_scsi_cmd_status(cmd, scsi_status);
 
 			/*
 			 * Currently we are only supporting BKOPs exception
@@ -5540,7 +5552,7 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 
 	if ((host_byte(result) != DID_OK) &&
 	    (host_byte(result) != DID_REQUEUE) && !hba->silence_err_logs)
-		ufshcd_print_tr(hba, lrbp->task_tag, true);
+		ufshcd_print_tr(hba, cmd, true);
 	return result;
 }
 
@@ -5609,13 +5621,10 @@ static irqreturn_t ufshcd_uic_cmd_compl(struct ufs_hba *hba, u32 intr_status)
 }
 
 /* Release the resources allocated for processing a SCSI command. */
-void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
-			     struct ufshcd_lrb *lrbp)
+void ufshcd_release_scsi_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd)
 {
-	struct scsi_cmnd *cmd = lrbp->cmd;
-
 	scsi_dma_unmap(cmd);
-	ufshcd_crypto_clear_prdt(hba, lrbp);
+	ufshcd_crypto_clear_prdt(hba, cmd);
 	ufshcd_release(hba);
 	ufshcd_clk_scaling_update_busy(hba);
 }
@@ -5629,20 +5638,20 @@ void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
 void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 			  struct cq_entry *cqe)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[task_tag];
-	struct scsi_cmnd *cmd = lrbp->cmd;
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	enum utp_ocs ocs;
 
 	if (hba->monitor.enabled) {
 		lrbp->compl_time_stamp = ktime_get();
 		lrbp->compl_time_stamp_local_clock = local_clock();
 	}
-	if (cmd) {
+	if (ufshcd_is_scsi_cmd(cmd)) {
 		if (unlikely(ufshcd_should_inform_monitor(hba, cmd)))
 			ufshcd_update_monitor(hba, cmd);
 		ufshcd_add_command_trace(hba, cmd, UFS_CMD_COMP);
-		cmd->result = ufshcd_transfer_rsp_status(hba, lrbp, cqe);
-		ufshcd_release_scsi_cmd(hba, lrbp);
+		cmd->result = ufshcd_transfer_rsp_status(hba, cmd, cqe);
+		ufshcd_release_scsi_cmd(hba, cmd);
 		/* Do not touch lrbp after scsi done */
 		scsi_done(cmd);
 	} else {
@@ -5679,7 +5688,7 @@ static void ufshcd_clear_polled(struct ufs_hba *hba,
 	int tag;
 
 	for_each_set_bit(tag, completed_reqs, hba->nutrs) {
-		struct scsi_cmnd *cmd = hba->lrb[tag].cmd;
+		struct scsi_cmnd *cmd = scsi_host_find_tag(hba->host, tag);
 
 		if (!cmd)
 			continue;
@@ -5730,7 +5739,6 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
 	struct scsi_device *sdev = rq->q->queuedata;
 	struct Scsi_Host *shost = sdev->host;
 	struct ufs_hba *hba = shost_priv(shost);
-	struct ufshcd_lrb *lrbp = &hba->lrb[rq->tag];
 	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
 
 	if (!hwq)
@@ -5745,7 +5753,7 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
 	scoped_guard(spinlock_irqsave, &hwq->cq_lock) {
 		if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
 			set_host_byte(cmd, DID_REQUEUE);
-			ufshcd_release_scsi_cmd(hba, lrbp);
+			ufshcd_release_scsi_cmd(hba, cmd);
 			scsi_done(cmd);
 		}
 	}
@@ -6624,7 +6632,7 @@ static bool ufshcd_abort_one(struct request *rq, void *priv)
 
 	*ret = ufshcd_try_to_abort_task(hba, tag);
 	dev_err(hba->dev, "Aborting tag %d / CDB %#02x %s\n", tag,
-		hba->lrb[tag].cmd ? hba->lrb[tag].cmd->cmnd[0] : -1,
+		ufshcd_is_scsi_cmd(cmd) ? cmd->cmnd[0] : -1,
 		*ret ? "failed" : "succeeded");
 
 	return *ret == 0;
@@ -7341,14 +7349,15 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,
 					enum query_opcode desc_op)
 {
 	const u32 tag = hba->reserved_slot;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int err = 0;
 	u8 upiu_flags;
 
 	/* Protects use of hba->reserved_slot. */
 	lockdep_assert_held(&hba->dev_cmd.lock);
 
-	ufshcd_setup_dev_cmd(hba, lrbp, cmd_type, 0, tag);
+	ufshcd_setup_dev_cmd(hba, cmd, cmd_type, 0, tag);
 
 	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags, DMA_NONE, 0);
 
@@ -7373,7 +7382,7 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,
 	 * bound to fail since dev_cmd.query and dev_cmd.type were left empty.
 	 * read the response directly ignoring all errors.
 	 */
-	ufshcd_issue_dev_cmd(hba, lrbp, tag, dev_cmd_timeout);
+	ufshcd_issue_dev_cmd(hba, cmd, tag, dev_cmd_timeout);
 
 	/* just copy the upiu response as it is */
 	memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));
@@ -7487,7 +7496,8 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 			 enum dma_data_direction dir)
 {
 	const u32 tag = hba->reserved_slot;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int err = 0;
 	int result;
 	u8 upiu_flags;
@@ -7498,7 +7508,8 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 	/* Protects use of hba->reserved_slot. */
 	ufshcd_dev_man_lock(hba);
 
-	ufshcd_setup_dev_cmd(hba, lrbp, DEV_CMD_TYPE_RPMB, UFS_UPIU_RPMB_WLUN, tag);
+	ufshcd_setup_dev_cmd(hba, cmd, DEV_CMD_TYPE_RPMB, UFS_UPIU_RPMB_WLUN,
+			     tag);
 
 	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags, DMA_NONE, ehs);
 
@@ -7515,7 +7526,7 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 
 	memset(lrbp->ucd_rsp_ptr, 0, sizeof(struct utp_upiu_rsp));
 
-	err = ufshcd_issue_dev_cmd(hba, lrbp, tag, ADVANCED_RPMB_REQ_TIMEOUT);
+	err = ufshcd_issue_dev_cmd(hba, cmd, tag, ADVANCED_RPMB_REQ_TIMEOUT);
 
 	if (!err) {
 		/* Just copy the upiu response as it is */
@@ -7616,11 +7627,12 @@ static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd)
 
 static void ufshcd_set_req_abort_skip(struct ufs_hba *hba, unsigned long bitmap)
 {
-	struct ufshcd_lrb *lrbp;
 	int tag;
 
 	for_each_set_bit(tag, &bitmap, hba->nutrs) {
-		lrbp = &hba->lrb[tag];
+		struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+		struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
 		lrbp->req_abort_skip = true;
 	}
 }
@@ -7628,7 +7640,7 @@ static void ufshcd_set_req_abort_skip(struct ufs_hba *hba, unsigned long bitmap)
 /**
  * ufshcd_try_to_abort_task - abort a specific task
  * @hba: Pointer to adapter instance
- * @tag: Task tag/index to be aborted
+ * @tag: Tag of the task to be aborted
  *
  * Abort the pending command in device by sending UFS_ABORT_TASK task management
  * command, and in host controller by clearing the door-bell register. There can
@@ -7640,7 +7652,8 @@ static void ufshcd_set_req_abort_skip(struct ufs_hba *hba, unsigned long bitmap)
  */
 int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)
 {
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int err;
 	int poll_cnt;
 	u8 resp = 0xF;
@@ -7662,7 +7675,7 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)
 				hba->dev,
 				"%s: cmd with tag %d not pending in the device.\n",
 				__func__, tag);
-			if (!ufshcd_cmd_inflight(lrbp->cmd)) {
+			if (!ufshcd_cmd_inflight(cmd)) {
 				dev_info(hba->dev,
 					 "%s: cmd with tag=%d completed.\n",
 					 __func__, tag);
@@ -7710,7 +7723,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 	struct Scsi_Host *host = cmd->device->host;
 	struct ufs_hba *hba = shost_priv(host);
 	int tag = scsi_cmd_to_rq(cmd)->tag;
-	struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	unsigned long flags;
 	int err = FAILED;
 	bool outstanding;
@@ -7745,9 +7758,9 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 		ufshcd_print_evt_hist(hba);
 		ufshcd_print_host_state(hba);
 		ufshcd_print_pwr_info(hba);
-		ufshcd_print_tr(hba, tag, true);
+		ufshcd_print_tr(hba, cmd, true);
 	} else {
-		ufshcd_print_tr(hba, tag, false);
+		ufshcd_print_tr(hba, cmd, false);
 	}
 	hba->req_abort_count++;
 
@@ -7791,7 +7804,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 		goto release;
 	}
 
-	err = ufshcd_try_to_abort_task(hba, tag);
+	err = ufshcd_try_to_abort_task(hba, lrbp->task_tag);
 	if (err) {
 		dev_err(hba->dev, "%s: failed with err %d\n", __func__, err);
 		ufshcd_set_req_abort_skip(hba, hba->outstanding_reqs);
@@ -7808,7 +7821,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 	spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 
 	if (outstanding)
-		ufshcd_release_scsi_cmd(hba, lrbp);
+		ufshcd_release_scsi_cmd(hba, cmd);
 
 	err = SUCCESS;
 
@@ -8885,8 +8898,6 @@ static void ufshcd_release_sdb_queue(struct ufs_hba *hba, int nutrs)
 	utrdl_size = sizeof(struct utp_transfer_req_desc) * nutrs;
 	dmam_free_coherent(hba->dev, utrdl_size, hba->utrdl_base_addr,
 			   hba->utrdl_dma_addr);
-
-	devm_kfree(hba->dev, hba->lrb);
 }
 
 static int ufshcd_alloc_mcq(struct ufs_hba *hba)
@@ -9164,6 +9175,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
 	.name			= UFSHCD,
 	.proc_name		= UFSHCD,
 	.map_queues		= ufshcd_map_queues,
+	.cmd_size		= sizeof(struct ufshcd_lrb),
 	.init_cmd_priv		= ufshcd_init_cmd_priv,
 	.queuecommand		= ufshcd_queuecommand,
 	.mq_poll		= ufshcd_poll,
diff --git a/include/ufs/ufshcd.h b/include/ufs/ufshcd.h
index ea0021f067c9..89dcac34c957 100644
--- a/include/ufs/ufshcd.h
+++ b/include/ufs/ufshcd.h
@@ -161,7 +161,6 @@ struct ufs_pm_lvl_states {
  * @ucd_prdt_dma_addr: PRDT dma address for debug
  * @ucd_rsp_dma_addr: UPIU response dma address for debug
  * @ucd_req_dma_addr: UPIU request dma address for debug
- * @cmd: pointer to SCSI command
  * @scsi_status: SCSI status of the command
  * @command_type: SCSI, UFS, Query.
  * @task_tag: Task tag of the command
@@ -186,7 +185,6 @@ struct ufshcd_lrb {
 	dma_addr_t ucd_rsp_dma_addr;
 	dma_addr_t ucd_prdt_dma_addr;
 
-	struct scsi_cmnd *cmd;
 	int scsi_status;
 
 	int command_type;
@@ -833,7 +831,6 @@ enum ufshcd_mcq_opr {
  * @spm_lvl: desired UFS power management level during system PM.
  * @pm_op_in_progress: whether or not a PM operation is in progress.
  * @ahit: value of Auto-Hibernate Idle Timer register.
- * @lrb: local reference block
  * @outstanding_tasks: Bits representing outstanding task requests
  * @outstanding_lock: Protects @outstanding_reqs.
  * @outstanding_reqs: Bits representing outstanding transfer requests
@@ -975,8 +972,6 @@ struct ufs_hba {
 	/* Auto-Hibernate Idle Timer register value */
 	u32 ahit;
 
-	struct ufshcd_lrb *lrb;
-
 	unsigned long outstanding_tasks;
 	spinlock_t outstanding_lock;
 	unsigned long outstanding_reqs;

[PATCH v4 24/29] ufs: core: Pass a SCSI pointer instead of an LRB pointer

[Bart Van Assche]

Prepare for removing the ufshcd_lrb task_tag member.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 3990b161ef78..e59e80d8e720 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2818,12 +2818,13 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,
 /**
  * ufshcd_prepare_utp_query_req_upiu() - fill the utp_transfer_req_desc for query request
  * @hba: UFS hba
- * @lrbp: local reference block pointer
+ * @cmd: SCSI command pointer
  * @upiu_flags: flags
  */
 static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
-				struct ufshcd_lrb *lrbp, u8 upiu_flags)
+				struct scsi_cmnd *cmd, u8 upiu_flags)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
 	struct ufs_query *query = &hba->dev_cmd.query;
 	u16 len = be16_to_cpu(query->request.upiu_req.length);
@@ -2852,8 +2853,9 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
 		memcpy(ucd_req_ptr + 1, query->descriptor, len);
 }
 
-static inline void ufshcd_prepare_utp_nop_upiu(struct ufshcd_lrb *lrbp)
+static inline void ufshcd_prepare_utp_nop_upiu(struct scsi_cmnd *cmd)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
 
 	memset(ucd_req_ptr, 0, sizeof(struct utp_upiu_req));
@@ -2868,22 +2870,23 @@ static inline void ufshcd_prepare_utp_nop_upiu(struct ufshcd_lrb *lrbp)
  * ufshcd_compose_devman_upiu - UFS Protocol Information Unit(UPIU)
  *			     for Device Management Purposes
  * @hba: per adapter instance
- * @lrbp: pointer to local reference block
+ * @cmd: SCSI command pointer
  *
  * Return: 0 upon success; < 0 upon failure.
  */
 static int ufshcd_compose_devman_upiu(struct ufs_hba *hba,
-				      struct ufshcd_lrb *lrbp)
+				      struct scsi_cmnd *cmd)
 {
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	u8 upiu_flags;
 	int ret = 0;
 
 	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags, DMA_NONE, 0);
 
 	if (hba->dev_cmd.type == DEV_CMD_TYPE_QUERY)
-		ufshcd_prepare_utp_query_req_upiu(hba, lrbp, upiu_flags);
+		ufshcd_prepare_utp_query_req_upiu(hba, cmd, upiu_flags);
 	else if (hba->dev_cmd.type == DEV_CMD_TYPE_NOP)
-		ufshcd_prepare_utp_nop_upiu(lrbp);
+		ufshcd_prepare_utp_nop_upiu(cmd);
 	else
 		ret = -EINVAL;
 
@@ -3121,11 +3124,9 @@ static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 static int ufshcd_compose_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 				  enum dev_cmd_type cmd_type, int tag)
 {
-	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-
 	ufshcd_setup_dev_cmd(hba, cmd, cmd_type, 0, tag);
 
-	return ufshcd_compose_devman_upiu(hba, lrbp);
+	return ufshcd_compose_devman_upiu(hba, cmd);
 }
 
 /*

[PATCH v4 25/29] ufs: core: Remove the ufshcd_lrb task_tag member

[Bart Van Assche]

Remove the ufshcd_lrb task_tag member and use scsi_cmd_to_rq(cmd)->tag
instead. Use rq->tag instead of lrbp->task_tag.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 62 +++++++++++++++++++--------------------
 include/ufs/ufshcd.h      |  1 -
 2 files changed, 30 insertions(+), 33 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index e59e80d8e720..0961432e9ee0 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -601,7 +601,7 @@ static void ufshcd_print_tr(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 			    bool pr_prdt)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-	const int tag = lrbp->task_tag;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	int prdt_length;
 
 	if (hba->monitor.enabled) {
@@ -2365,6 +2365,7 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 				       struct ufs_hw_queue *hwq)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	unsigned long flags;
 
 	if (hba->monitor.enabled) {
@@ -2393,11 +2394,10 @@ static inline void ufshcd_send_command(struct ufs_hba *hba,
 	} else {
 		spin_lock_irqsave(&hba->outstanding_lock, flags);
 		if (hba->vops && hba->vops->setup_xfer_req)
-			hba->vops->setup_xfer_req(hba, lrbp->task_tag,
+			hba->vops->setup_xfer_req(hba, tag,
 						  ufshcd_is_scsi_cmd(cmd));
-		__set_bit(lrbp->task_tag, &hba->outstanding_reqs);
-		ufshcd_writel(hba, 1 << lrbp->task_tag,
-			      REG_UTP_TRANSFER_REQ_DOOR_BELL);
+		__set_bit(tag, &hba->outstanding_reqs);
+		ufshcd_writel(hba, 1 << tag, REG_UTP_TRANSFER_REQ_DOOR_BELL);
 		spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 	}
 }
@@ -2794,6 +2794,7 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,
 					     u8 upiu_flags)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
 	unsigned short cdb_len;
 
@@ -2801,11 +2802,11 @@ static void ufshcd_prepare_utp_scsi_cmd_upiu(struct scsi_cmnd *cmd,
 		.transaction_code = UPIU_TRANSACTION_COMMAND,
 		.flags = upiu_flags,
 		.lun = lrbp->lun,
-		.task_tag = lrbp->task_tag,
+		.task_tag = tag,
 		.command_set_type = UPIU_COMMAND_SET_TYPE_SCSI,
 	};
 
-	WARN_ON_ONCE(ucd_req_ptr->header.task_tag != lrbp->task_tag);
+	WARN_ON_ONCE(ucd_req_ptr->header.task_tag != tag);
 
 	ucd_req_ptr->sc.exp_data_transfer_len = cpu_to_be32(cmd->sdb.length);
 
@@ -2826,6 +2827,7 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	struct ufs_query *query = &hba->dev_cmd.query;
 	u16 len = be16_to_cpu(query->request.upiu_req.length);
 
@@ -2834,7 +2836,7 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba,
 		.transaction_code = UPIU_TRANSACTION_QUERY_REQ,
 		.flags = upiu_flags,
 		.lun = lrbp->lun,
-		.task_tag = lrbp->task_tag,
+		.task_tag = tag,
 		.query_function = query->request.query_func,
 		/* Data segment length only need for WRITE_DESC */
 		.data_segment_length =
@@ -2857,12 +2859,13 @@ static inline void ufshcd_prepare_utp_nop_upiu(struct scsi_cmnd *cmd)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 
 	memset(ucd_req_ptr, 0, sizeof(struct utp_upiu_req));
 
 	ucd_req_ptr->header = (struct utp_upiu_header){
 		.transaction_code = UPIU_TRANSACTION_NOP_OUT,
-		.task_tag = lrbp->task_tag,
+		.task_tag = tag,
 	};
 }
 
@@ -2947,7 +2950,6 @@ static void __ufshcd_setup_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 
 	memset(lrbp->ucd_req_ptr, 0, sizeof(*lrbp->ucd_req_ptr));
 
-	lrbp->task_tag = tag;
 	lrbp->lun = lun;
 	ufshcd_prepare_lrbp_crypto(cmd ? scsi_cmd_to_rq(cmd) : NULL, lrbp);
 }
@@ -3243,6 +3245,8 @@ ufshcd_dev_cmd_completion(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 		struct ufshcd_lrb *lrbp, int max_timeout)
 {
+	struct scsi_cmnd *cmd = (struct scsi_cmnd *)lrbp - 1;
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	unsigned long time_left = msecs_to_jiffies(max_timeout);
 	unsigned long flags;
 	bool pending;
@@ -3259,18 +3263,18 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 	} else {
 		err = -ETIMEDOUT;
 		dev_dbg(hba->dev, "%s: dev_cmd request timedout, tag %d\n",
-			__func__, lrbp->task_tag);
+			__func__, tag);
 
 		/* MCQ mode */
 		if (hba->mcq_enabled) {
 			/* successfully cleared the command, retry if needed */
-			if (ufshcd_clear_cmd(hba, lrbp->task_tag) == 0)
+			if (ufshcd_clear_cmd(hba, tag) == 0)
 				err = -EAGAIN;
 			return err;
 		}
 
 		/* SDB mode */
-		if (ufshcd_clear_cmd(hba, lrbp->task_tag) == 0) {
+		if (ufshcd_clear_cmd(hba, tag) == 0) {
 			/* successfully cleared the command, retry if needed */
 			err = -EAGAIN;
 			/*
@@ -3279,11 +3283,9 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 			 * variable.
 			 */
 			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			pending = test_bit(lrbp->task_tag,
-					   &hba->outstanding_reqs);
+			pending = test_bit(tag, &hba->outstanding_reqs);
 			if (pending)
-				__clear_bit(lrbp->task_tag,
-					    &hba->outstanding_reqs);
+				__clear_bit(tag, &hba->outstanding_reqs);
 			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 
 			if (!pending) {
@@ -3296,11 +3298,10 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 			}
 		} else {
 			dev_err(hba->dev, "%s: failed to clear tag %d\n",
-				__func__, lrbp->task_tag);
+				__func__, tag);
 
 			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			pending = test_bit(lrbp->task_tag,
-					   &hba->outstanding_reqs);
+			pending = test_bit(tag, &hba->outstanding_reqs);
 			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
 
 			if (!pending) {
@@ -5456,6 +5457,7 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
 					     struct cq_entry *cqe)
 {
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	const int tag = scsi_cmd_to_rq(cmd)->tag;
 	int result = 0;
 	int scsi_status;
 	enum utp_ocs ocs;
@@ -5527,10 +5529,8 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
 	case OCS_ABORTED:
 	case OCS_INVALID_COMMAND_STATUS:
 		result |= DID_REQUEUE << 16;
-		dev_warn(hba->dev,
-				"OCS %s from controller for tag %d\n",
-				(ocs == OCS_ABORTED ? "aborted" : "invalid"),
-				lrbp->task_tag);
+		dev_warn(hba->dev, "OCS %s from controller for tag %d\n",
+			 ocs == OCS_ABORTED ? "aborted" : "invalid", tag);
 		break;
 	case OCS_INVALID_CMD_TABLE_ATTR:
 	case OCS_INVALID_PRDT_ATTR:
@@ -5543,9 +5543,8 @@ static inline int ufshcd_transfer_rsp_status(struct ufs_hba *hba,
 	case OCS_GENERAL_CRYPTO_ERROR:
 	default:
 		result |= DID_ERROR << 16;
-		dev_err(hba->dev,
-				"OCS error from controller = %x for tag %d\n",
-				ocs, lrbp->task_tag);
+		dev_err(hba->dev, "OCS error from controller = %x for tag %d\n",
+			ocs, tag);
 		ufshcd_print_evt_hist(hba);
 		ufshcd_print_host_state(hba);
 		break;
@@ -7660,8 +7659,8 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)
 	u8 resp = 0xF;
 
 	for (poll_cnt = 100; poll_cnt; poll_cnt--) {
-		err = ufshcd_issue_tm_cmd(hba, lrbp->lun, lrbp->task_tag,
-				UFS_QUERY_TASK, &resp);
+		err = ufshcd_issue_tm_cmd(hba, lrbp->lun, tag, UFS_QUERY_TASK,
+					  &resp);
 		if (!err && resp == UPIU_TASK_MANAGEMENT_FUNC_SUCCEEDED) {
 			/* cmd pending in the device */
 			dev_err(hba->dev, "%s: cmd pending in the device. tag = %d\n",
@@ -7694,8 +7693,7 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag)
 	if (!poll_cnt)
 		return -EBUSY;
 
-	err = ufshcd_issue_tm_cmd(hba, lrbp->lun, lrbp->task_tag,
-			UFS_ABORT_TASK, &resp);
+	err = ufshcd_issue_tm_cmd(hba, lrbp->lun, tag, UFS_ABORT_TASK, &resp);
 	if (err || resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) {
 		if (!err) {
 			err = resp; /* service response error */
@@ -7805,7 +7803,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 		goto release;
 	}
 
-	err = ufshcd_try_to_abort_task(hba, lrbp->task_tag);
+	err = ufshcd_try_to_abort_task(hba, tag);
 	if (err) {
 		dev_err(hba->dev, "%s: failed with err %d\n", __func__, err);
 		ufshcd_set_req_abort_skip(hba, hba->outstanding_reqs);
diff --git a/include/ufs/ufshcd.h b/include/ufs/ufshcd.h
index 89dcac34c957..9eb69241e40f 100644
--- a/include/ufs/ufshcd.h
+++ b/include/ufs/ufshcd.h
@@ -188,7 +188,6 @@ struct ufshcd_lrb {
 	int scsi_status;
 
 	int command_type;
-	int task_tag;
 	u8 lun; /* UPIU LUN id field is only 8-bit wide */
 	bool intr_cmd;
 	bool req_abort_skip;

[PATCH v4 26/29] ufs: core: Make blk_mq_tagset_busy_iter() skip reserved requests

[Bart Van Assche]

A later patch will convert hba->reserved_slot into a reserved tag. Make
blk_mq_tagset_busy_iter() skip reserved requests such that device
management commands are skipped.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 0961432e9ee0..2f815f74aa66 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -646,7 +646,8 @@ static bool ufshcd_print_tr_iter(struct request *req, void *priv)
 	struct Scsi_Host *shost = sdev->host;
 	struct ufs_hba *hba = shost_priv(shost);
 
-	ufshcd_print_tr(hba, blk_mq_rq_to_pdu(req), *(bool *)priv);
+	if (!blk_mq_is_reserved_rq(req))
+		ufshcd_print_tr(hba, blk_mq_rq_to_pdu(req), *(bool *)priv);
 
 	return true;
 }
@@ -5741,7 +5742,7 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
 	struct ufs_hba *hba = shost_priv(shost);
 	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
 
-	if (!hwq)
+	if (blk_mq_is_reserved_rq(rq) || !hwq)
 		return true;
 
 	ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
@@ -5768,7 +5769,7 @@ static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)
 	struct ufs_hba *hba = shost_priv(shost);
 	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
 
-	if (hwq)
+	if (!blk_mq_is_reserved_rq(rq) && hwq)
 		ufshcd_mcq_poll_cqe_lock(hba, hwq);
 
 	return true;
@@ -6630,6 +6631,9 @@ static bool ufshcd_abort_one(struct request *rq, void *priv)
 	struct Scsi_Host *shost = sdev->host;
 	struct ufs_hba *hba = shost_priv(shost);
 
+	if (blk_mq_is_reserved_rq(rq))
+		return true;
+
 	*ret = ufshcd_try_to_abort_task(hba, tag);
 	dev_err(hba->dev, "Aborting tag %d / CDB %#02x %s\n", tag,
 		ufshcd_is_scsi_cmd(cmd) ? cmd->cmnd[0] : -1,
@@ -7565,7 +7569,7 @@ static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)
 	const u64 lun = *(u64 *)priv;
 	const u32 tag = req->tag;
 
-	if (sdev->lun != lun)
+	if (blk_mq_is_reserved_rq(req) || sdev->lun != lun)
 		return true;
 
 	if (ufshcd_clear_cmd(hba, tag) < 0) {

[PATCH v4 27/29] ufs: core: Move code out of ufshcd_wait_for_dev_cmd()

[Bart Van Assche]

The ufshcd_dev_cmd_completion() call is useful for some but not for all
ufshcd_wait_for_dev_cmd() callers. Hence, remove the
ufshcd_dev_cmd_completion() call from ufshcd_wait_for_dev_cmd() and move
it past the ufshcd_issue_dev_cmd() calls where appropriate. This makes
it easier to detect timeout errors for UPIU frames submitted through the
BSG interface.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 2f815f74aa66..6e346fe59014 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -3259,8 +3259,6 @@ static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
 
 	if (likely(time_left)) {
 		err = ufshcd_get_tr_ocs(lrbp, NULL);
-		if (!err)
-			err = ufshcd_dev_cmd_completion(hba, lrbp);
 	} else {
 		err = -ETIMEDOUT;
 		dev_dbg(hba->dev, "%s: dev_cmd request timedout, tag %d\n",
@@ -3369,6 +3367,7 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,
 {
 	const u32 tag = hba->reserved_slot;
 	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	int err;
 
 	/* Protects use of hba->reserved_slot. */
@@ -3378,7 +3377,11 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,
 	if (unlikely(err))
 		return err;
 
-	return ufshcd_issue_dev_cmd(hba, cmd, tag, timeout);
+	err = ufshcd_issue_dev_cmd(hba, cmd, tag, timeout);
+	if (err)
+		return err;
+
+	return ufshcd_dev_cmd_completion(hba, lrbp);
 }
 
 /**
@@ -7381,12 +7384,9 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,
 
 	memset(lrbp->ucd_rsp_ptr, 0, sizeof(struct utp_upiu_rsp));
 
-	/*
-	 * ignore the returning value here - ufshcd_check_query_response is
-	 * bound to fail since dev_cmd.query and dev_cmd.type were left empty.
-	 * read the response directly ignoring all errors.
-	 */
-	ufshcd_issue_dev_cmd(hba, cmd, tag, dev_cmd_timeout);
+	err = ufshcd_issue_dev_cmd(hba, cmd, tag, dev_cmd_timeout);
+	if (err)
+		return err;
 
 	/* just copy the upiu response as it is */
 	memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));
@@ -7531,7 +7531,10 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 	memset(lrbp->ucd_rsp_ptr, 0, sizeof(struct utp_upiu_rsp));
 
 	err = ufshcd_issue_dev_cmd(hba, cmd, tag, ADVANCED_RPMB_REQ_TIMEOUT);
+	if (err)
+		return err;
 
+	err = ufshcd_dev_cmd_completion(hba, lrbp);
 	if (!err) {
 		/* Just copy the upiu response as it is */
 		memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));

[PATCH v4 28/29] ufs: core: Rework the ufshcd_issue_dev_cmd() callers

[Bart Van Assche]

Move the code for initializing requests and also the code for extracting
results into new functions. No functionality has been changed. This
change prepares for switching to scsi_execute_cmd() for submitting
device management commands.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufshcd.c | 318 +++++++++++++++++++++++++++-----------
 1 file changed, 226 insertions(+), 92 deletions(-)

diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 6e346fe59014..6b713caba7ea 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -3351,6 +3351,23 @@ static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 	return err;
 }
 
+struct ufshcd_exec_dev_cmd_args {
+	struct scsi_exec_args args;
+	struct ufs_hba *hba;
+	enum dev_cmd_type cmd_type;
+};
+
+static int ufshcd_init_dev_cmd(struct scsi_cmnd *cmd,
+			       const struct scsi_exec_args *args)
+{
+	const struct ufshcd_exec_dev_cmd_args *uea =
+		container_of(args, typeof(*uea), args);
+	struct request *rq = scsi_cmd_to_rq(cmd);
+	unsigned int tag = rq->tag;
+
+	return ufshcd_compose_dev_cmd(uea->hba, cmd, uea->cmd_type, tag);
+}
+
 /**
  * ufshcd_exec_dev_cmd - API for sending device management requests
  * @hba: UFS hba
@@ -3365,6 +3382,10 @@ static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,
 		enum dev_cmd_type cmd_type, int timeout)
 {
+	const struct ufshcd_exec_dev_cmd_args args = {
+		.hba = hba,
+		.cmd_type = cmd_type
+	};
 	const u32 tag = hba->reserved_slot;
 	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
@@ -3373,7 +3394,7 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,
 	/* Protects use of hba->reserved_slot. */
 	lockdep_assert_held(&hba->dev_cmd.lock);
 
-	err = ufshcd_compose_dev_cmd(hba, cmd, cmd_type, tag);
+	err = ufshcd_init_dev_cmd(cmd, &args.args);
 	if (unlikely(err))
 		return err;
 
@@ -7329,6 +7350,88 @@ static int ufshcd_issue_tm_cmd(struct ufs_hba *hba, int lun_id, int task_id,
 	return err;
 }
 
+struct ufshcd_exec_devman_upiu_cmd_args {
+	struct scsi_exec_args args;
+	struct ufs_hba *hba;
+	struct utp_upiu_req *req_upiu;
+	struct utp_upiu_req *rsp_upiu;
+	u8 *desc_buff;
+	int *buff_len;
+	enum dev_cmd_type cmd_type;
+	enum query_opcode desc_op;
+	int *err;
+};
+
+static int ufshcd_init_upiu_cmd(struct scsi_cmnd *cmd,
+				const struct scsi_exec_args *args)
+{
+	const struct ufshcd_exec_devman_upiu_cmd_args *uea =
+		container_of(args, typeof(*uea), args);
+	struct utp_upiu_req *req_upiu = uea->req_upiu;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	enum dev_cmd_type cmd_type = uea->cmd_type;
+	struct request *rq = scsi_cmd_to_rq(cmd);
+	enum query_opcode desc_op = uea->desc_op;
+	struct ufs_hba *hba = uea->hba;
+	u8 *desc_buff = uea->desc_buff;
+	unsigned int tag = rq->tag;
+	u8 upiu_flags;
+
+	ufshcd_setup_dev_cmd(hba, cmd, cmd_type, 0, tag);
+
+	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags, DMA_NONE, 0);
+
+	/* Set the task tag in the request UPIU. */
+	req_upiu->header.task_tag = tag;
+
+	/* Copy the UPIU request into the LRB. */
+	memcpy(lrbp->ucd_req_ptr, req_upiu, sizeof(*lrbp->ucd_req_ptr));
+	if (desc_buff && desc_op == UPIU_QUERY_OPCODE_WRITE_DESC) {
+		/*
+		 * For the WRITE DESCRIPTOR operation, the data segment follows
+		 * right after the UPIU transaction specific fields.
+		 */
+		memcpy(lrbp->ucd_req_ptr + 1, desc_buff, *uea->buff_len);
+		*uea->buff_len = 0;
+	}
+
+	memset(lrbp->ucd_rsp_ptr, 0, sizeof(struct utp_upiu_rsp));
+
+	return 0;
+}
+
+static void ufshcd_copy_upiu_cmd_result(struct scsi_cmnd *cmd,
+					const struct scsi_exec_args *args)
+{
+	const struct ufshcd_exec_devman_upiu_cmd_args *uea =
+		container_of(args, typeof(*uea), args);
+	struct utp_upiu_req *rsp_upiu = uea->rsp_upiu;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	enum query_opcode desc_op = uea->desc_op;
+	u8 *desc_buff = uea->desc_buff;
+	struct ufs_hba *hba = uea->hba;
+
+	/* Copy the UPIU response. */
+	memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));
+
+	if (desc_buff && desc_op == UPIU_QUERY_OPCODE_READ_DESC) {
+		u8 *descp = (u8 *)lrbp->ucd_rsp_ptr + sizeof(*rsp_upiu);
+		u16 resp_len = be16_to_cpu(
+			lrbp->ucd_rsp_ptr->header.data_segment_length);
+
+		if (resp_len <= *uea->buff_len) {
+			memcpy(desc_buff, descp, resp_len);
+			*uea->buff_len = resp_len;
+		} else {
+			dev_warn(hba->dev,
+				"%s: rsp size %d is bigger than buffer size %d",
+				__func__, resp_len, *uea->buff_len);
+			*uea->buff_len = 0;
+			*uea->err = -EINVAL;
+		}
+	}
+}
+
 /**
  * ufshcd_issue_devman_upiu_cmd - API for sending "utrd" type requests
  * @hba:	per-adapter instance
@@ -7355,59 +7458,33 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,
 					enum dev_cmd_type cmd_type,
 					enum query_opcode desc_op)
 {
+	int err, copy_result_err = 0;
+	const struct ufshcd_exec_devman_upiu_cmd_args args = {
+		.hba = hba,
+		.req_upiu = req_upiu,
+		.rsp_upiu = rsp_upiu,
+		.desc_buff = desc_buff,
+		.buff_len = buff_len,
+		.cmd_type = cmd_type,
+		.desc_op = desc_op,
+		.err = &copy_result_err,
+	};
 	const u32 tag = hba->reserved_slot;
 	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
-	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-	int err = 0;
-	u8 upiu_flags;
 
 	/* Protects use of hba->reserved_slot. */
 	lockdep_assert_held(&hba->dev_cmd.lock);
 
-	ufshcd_setup_dev_cmd(hba, cmd, cmd_type, 0, tag);
-
-	ufshcd_prepare_req_desc_hdr(hba, lrbp, &upiu_flags, DMA_NONE, 0);
-
-	/* update the task tag in the request upiu */
-	req_upiu->header.task_tag = tag;
-
-	/* just copy the upiu request as it is */
-	memcpy(lrbp->ucd_req_ptr, req_upiu, sizeof(*lrbp->ucd_req_ptr));
-	if (desc_buff && desc_op == UPIU_QUERY_OPCODE_WRITE_DESC) {
-		/* The Data Segment Area is optional depending upon the query
-		 * function value. for WRITE DESCRIPTOR, the data segment
-		 * follows right after the tsf.
-		 */
-		memcpy(lrbp->ucd_req_ptr + 1, desc_buff, *buff_len);
-		*buff_len = 0;
-	}
-
-	memset(lrbp->ucd_rsp_ptr, 0, sizeof(struct utp_upiu_rsp));
+	err = ufshcd_init_upiu_cmd(cmd, &args.args);
+	WARN_ON_ONCE(err);
 
 	err = ufshcd_issue_dev_cmd(hba, cmd, tag, dev_cmd_timeout);
 	if (err)
 		return err;
 
-	/* just copy the upiu response as it is */
-	memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));
-	if (desc_buff && desc_op == UPIU_QUERY_OPCODE_READ_DESC) {
-		u8 *descp = (u8 *)lrbp->ucd_rsp_ptr + sizeof(*rsp_upiu);
-		u16 resp_len = be16_to_cpu(lrbp->ucd_rsp_ptr->header
-					   .data_segment_length);
+	ufshcd_copy_upiu_cmd_result(cmd, &args.args);
 
-		if (*buff_len >= resp_len) {
-			memcpy(desc_buff, descp, resp_len);
-			*buff_len = resp_len;
-		} else {
-			dev_warn(hba->dev,
-				 "%s: rsp size %d is bigger than buffer size %d",
-				 __func__, resp_len, *buff_len);
-			*buff_len = 0;
-			err = -EINVAL;
-		}
-	}
-
-	return err;
+	return copy_result_err;
 }
 
 /**
@@ -7481,36 +7558,34 @@ int ufshcd_exec_raw_upiu_cmd(struct ufs_hba *hba,
 	return err;
 }
 
-/**
- * ufshcd_advanced_rpmb_req_handler - handle advanced RPMB request
- * @hba:	per adapter instance
- * @req_upiu:	upiu request
- * @rsp_upiu:	upiu reply
- * @req_ehs:	EHS field which contains Advanced RPMB Request Message
- * @rsp_ehs:	EHS field which returns Advanced RPMB Response Message
- * @sg_cnt:	The number of sg lists actually used
- * @sg_list:	Pointer to SG list when DATA IN/OUT UPIU is required in ARPMB operation
- * @dir:	DMA direction
- *
- * Return: zero on success, non-zero on failure.
- */
-int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *req_upiu,
-			 struct utp_upiu_req *rsp_upiu, struct ufs_ehs *req_ehs,
-			 struct ufs_ehs *rsp_ehs, int sg_cnt, struct scatterlist *sg_list,
-			 enum dma_data_direction dir)
+struct ufshcd_rpmb_args {
+	struct scsi_exec_args args;
+	struct ufs_hba *hba;
+	struct utp_upiu_req *req_upiu;
+	struct utp_upiu_req *rsp_upiu;
+	struct ufs_ehs *req_ehs;
+	struct ufs_ehs *rsp_ehs;
+	int sg_cnt;
+	struct scatterlist *sg_list;
+	enum dma_data_direction dir;
+	int *upiu_result;
+};
+
+static int ufshcd_init_rpmb(struct scsi_cmnd *cmd,
+			    const struct scsi_exec_args *args)
 {
-	const u32 tag = hba->reserved_slot;
-	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+	const struct ufshcd_rpmb_args *ura =
+		container_of(args, typeof(*ura), args);
+	struct ufs_hba *hba = ura->hba;
+	int ehs = (hba->capabilities & MASK_EHSLUTRD_SUPPORTED) ? 2 : 0;
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-	int err = 0;
-	int result;
+	struct utp_upiu_req *req_upiu = ura->req_upiu;
+	struct scatterlist *sg_list = ura->sg_list;
+	struct ufs_ehs *req_ehs = ura->req_ehs;
+	enum dma_data_direction dir = ura->dir;
+	u32 tag = scsi_cmd_to_rq(cmd)->tag;
+	int sg_cnt = ura->sg_cnt;
 	u8 upiu_flags;
-	u8 *ehs_data;
-	u16 ehs_len;
-	int ehs = (hba->capabilities & MASK_EHSLUTRD_SUPPORTED) ? 2 : 0;
-
-	/* Protects use of hba->reserved_slot. */
-	ufshcd_dev_man_lock(hba);
 
 	ufshcd_setup_dev_cmd(hba, cmd, DEV_CMD_TYPE_RPMB, UFS_UPIU_RPMB_WLUN,
 			     tag);
@@ -7530,37 +7605,96 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 
 	memset(lrbp->ucd_rsp_ptr, 0, sizeof(struct utp_upiu_rsp));
 
-	err = ufshcd_issue_dev_cmd(hba, cmd, tag, ADVANCED_RPMB_REQ_TIMEOUT);
-	if (err)
-		return err;
+	return 0;
+}
 
-	err = ufshcd_dev_cmd_completion(hba, lrbp);
-	if (!err) {
-		/* Just copy the upiu response as it is */
-		memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));
-		/* Get the response UPIU result */
-		result = (lrbp->ucd_rsp_ptr->header.response << 8) |
-			lrbp->ucd_rsp_ptr->header.status;
+static void ufshcd_copy_rpmb_result(struct scsi_cmnd *cmd,
+				    const struct scsi_exec_args *args)
+{
+	const struct ufshcd_rpmb_args *ura =
+		container_of(args, typeof(*ura), args);
+	struct utp_upiu_req *rsp_upiu = ura->rsp_upiu;
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	struct ufs_ehs *rsp_ehs = ura->rsp_ehs;
+	u8 *ehs_data;
+	u16 ehs_len;
 
-		ehs_len = lrbp->ucd_rsp_ptr->header.ehs_length;
+	/* Copy the upiu response. */
+	memcpy(rsp_upiu, lrbp->ucd_rsp_ptr, sizeof(*rsp_upiu));
+	/* Copy the UPIU response and status fields. */
+	*ura->upiu_result = (lrbp->ucd_rsp_ptr->header.response << 8) |
+		lrbp->ucd_rsp_ptr->header.status;
+
+	ehs_len = lrbp->ucd_rsp_ptr->header.ehs_length;
+	/*
+	 * Since the bLength in the EHS indicates the total size of the EHS
+	 * Header and EHS Data in 32 Byte units, the value of the bLength
+	 * Request/Response for Advanced RPMB Message is 02h.
+	 */
+	if (ehs_len == 2 && rsp_ehs) {
 		/*
-		 * Since the bLength in EHS indicates the total size of the EHS Header and EHS Data
-		 * in 32 Byte units, the value of the bLength Request/Response for Advanced RPMB
-		 * Message is 02h
+		 * ucd_rsp_ptr points to a buffer with a length of 512 bytes
+		 * (ALIGNED_UPIU_SIZE = 512), and the EHS data just starts from
+		 * byte32.
 		 */
-		if (ehs_len == 2 && rsp_ehs) {
-			/*
-			 * ucd_rsp_ptr points to a buffer with a length of 512 bytes
-			 * (ALIGNED_UPIU_SIZE = 512), and the EHS data just starts from byte32
-			 */
-			ehs_data = (u8 *)lrbp->ucd_rsp_ptr + EHS_OFFSET_IN_RESPONSE;
-			memcpy(rsp_ehs, ehs_data, ehs_len * 32);
-		}
+		ehs_data = (u8 *)lrbp->ucd_rsp_ptr + EHS_OFFSET_IN_RESPONSE;
+		memcpy(rsp_ehs, ehs_data, ehs_len * 32);
 	}
+}
 
+/**
+ * ufshcd_advanced_rpmb_req_handler - handle advanced RPMB request
+ * @hba:	per adapter instance
+ * @req_upiu:	upiu request
+ * @rsp_upiu:	upiu reply
+ * @req_ehs:	EHS field which contains Advanced RPMB Request Message
+ * @rsp_ehs:	EHS field which returns Advanced RPMB Response Message
+ * @sg_cnt:	The number of sg lists actually used
+ * @sg_list:	Pointer to SG list when DATA IN/OUT UPIU is required in ARPMB operation
+ * @dir:	DMA direction
+ *
+ * Return: zero on success, non-zero on failure. This function can return a
+ * negative error code, a positive OCS error or a two byte integer with the
+ * most significant byte representing the UTP response value and the least
+ * significant byte representing the RPMB status value. See also section
+ * "12.4.3.7 RPMB Operation Result" in the UFS standard.
+ */
+int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *req_upiu,
+			 struct utp_upiu_req *rsp_upiu, struct ufs_ehs *req_ehs,
+			 struct ufs_ehs *rsp_ehs, int sg_cnt, struct scatterlist *sg_list,
+			 enum dma_data_direction dir)
+{
+	int err, upiu_result = 0;
+	const struct ufshcd_rpmb_args args = {
+		.hba = hba,
+		.req_upiu = req_upiu,
+		.rsp_upiu = rsp_upiu,
+		.req_ehs = req_ehs,
+		.rsp_ehs = rsp_ehs,
+		.sg_cnt = sg_cnt,
+		.sg_list = sg_list,
+		.dir = dir,
+		.upiu_result = &upiu_result,
+	};
+	const u32 tag = hba->reserved_slot;
+	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
+
+	/* Protects use of hba->reserved_slot. */
+	ufshcd_dev_man_lock(hba);
+
+	err = ufshcd_init_rpmb(cmd, &args.args);
+	WARN_ON_ONCE(err);
+
+	err = ufshcd_issue_dev_cmd(hba, cmd, tag, ADVANCED_RPMB_REQ_TIMEOUT);
+	if (err)
+		goto unlock;
+
+	ufshcd_copy_rpmb_result(cmd, &args.args);
+
+unlock:
 	ufshcd_dev_man_unlock(hba);
 
-	return err ? : result;
+	return err ?: upiu_result;
 }
 
 static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)

[PATCH v4 29/29] ufs: core: Switch to scsi_execute_cmd()

[Bart Van Assche]

Use scsi_execute_cmd() instead of calling ufshcd_send_command() and
ufshcd_wait_for_dev_cmd() directly. Add ufshcd_queue_reserved_command()
for submitting reserved commands. Add support in ufshcd_abort() for
device management commands. Remove the code and data structures
that became superfluous. This includes ufshcd_wait_for_dev_cmd(),
hba->reserved_slot and ufs_dev_cmd.complete.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/ufs/core/ufs-mcq.c     |  19 ++-
 drivers/ufs/core/ufshcd-priv.h |  25 +---
 drivers/ufs/core/ufshcd.c      | 223 +++++++++++----------------------
 include/ufs/ufshcd.h           |   6 -
 4 files changed, 83 insertions(+), 190 deletions(-)

diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c
index c6c6cca400de..9303687e38a8 100644
--- a/drivers/ufs/core/ufs-mcq.c
+++ b/drivers/ufs/core/ufs-mcq.c
@@ -471,9 +471,6 @@ int ufshcd_mcq_init(struct ufs_hba *hba)
 		mutex_init(&hwq->sq_mutex);
 	}
 
-	/* The very first HW queue serves device commands */
-	hba->dev_cmd_queue = &hba->uhq[0];
-
 	host->host_tagset = 1;
 	return 0;
 }
@@ -528,6 +525,7 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 {
 	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, task_tag);
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	struct request *rq = scsi_cmd_to_rq(cmd);
 	struct ufs_hw_queue *hwq;
 	void __iomem *reg, *opr_sqd_base;
 	u32 nexus, id, val;
@@ -536,15 +534,12 @@ int ufshcd_mcq_sq_cleanup(struct ufs_hba *hba, int task_tag)
 	if (hba->quirks & UFSHCD_QUIRK_MCQ_BROKEN_RTC)
 		return -ETIMEDOUT;
 
-	if (task_tag != hba->reserved_slot) {
-		if (!cmd)
-			return -EINVAL;
-		hwq = ufshcd_mcq_req_to_hwq(hba, scsi_cmd_to_rq(cmd));
-		if (!hwq)
-			return 0;
-	} else {
-		hwq = hba->dev_cmd_queue;
-	}
+	if (!cmd)
+		return -EINVAL;
+
+	hwq = ufshcd_mcq_req_to_hwq(hba, rq);
+	if (!hwq)
+		return 0;
 
 	id = hwq->id;
 
diff --git a/drivers/ufs/core/ufshcd-priv.h b/drivers/ufs/core/ufshcd-priv.h
index 3222c4d3ceb4..35c3277a4373 100644
--- a/drivers/ufs/core/ufshcd-priv.h
+++ b/drivers/ufs/core/ufshcd-priv.h
@@ -367,30 +367,7 @@ static inline bool ufs_is_valid_unit_desc_lun(struct ufs_dev_info *dev_info, u8
 static inline struct scsi_cmnd *ufshcd_tag_to_cmd(struct ufs_hba *hba, u32 tag)
 {
 	struct blk_mq_tags *tags = hba->host->tag_set.shared_tags;
-	struct request *rq;
-
-	/*
-	 * Handle reserved tags differently because the UFS driver does not
-	 * call blk_mq_alloc_request() for allocating reserved requests.
-	 * Allocating reserved tags with blk_mq_alloc_request() would require
-	 * the following:
-	 * - Allocate an additional request queue from &hba->host->tag_set for
-	 *   allocating reserved requests from.
-	 * - For that request queue, allocate a SCSI device.
-	 * - Calling blk_mq_alloc_request(hba->dev_mgmt_queue, REQ_OP_DRV_OUT,
-	 *   BLK_MQ_REQ_RESERVED) for allocating a reserved request and
-	 *   blk_mq_free_request() for freeing reserved requests.
-	 * - Set the .device pointer for these reserved requests.
-	 * - Submit reserved requests with blk_execute_rq().
-	 * - Modify ufshcd_queuecommand() such that it handles reserved requests
-	 *   in another way than SCSI requests.
-	 * - Modify ufshcd_compl_one_cqe() such that it calls scsi_done() for
-	 *   device management commands.
-	 * - Modify all callback functions called by blk_mq_tagset_busy_iter()
-	 *   calls in the UFS driver and skip device management commands.
-	 */
-	rq = tag < UFSHCD_NUM_RESERVED ? tags->static_rqs[tag] :
-					 blk_mq_tag_to_rq(tags, tag);
+	struct request *rq = blk_mq_tag_to_rq(tags, tag);
 
 	if (WARN_ON_ONCE(!rq))
 		return NULL;
diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 6b713caba7ea..7cd75b3014e1 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -2352,7 +2352,7 @@ static void ufshcd_update_monitor(struct ufs_hba *hba, struct scsi_cmnd *cmd)
  */
 static bool ufshcd_is_scsi_cmd(struct scsi_cmnd *cmd)
 {
-	return blk_mq_request_started(scsi_cmd_to_rq(cmd));
+	return !blk_mq_is_reserved_rq(scsi_cmd_to_rq(cmd));
 }
 
 /**
@@ -2483,7 +2483,6 @@ static inline int ufshcd_hba_capabilities(struct ufs_hba *hba)
 	hba->nutrs = (hba->capabilities & MASK_TRANSFER_REQUESTS_SLOTS_SDB) + 1;
 	hba->nutmrs =
 	((hba->capabilities & MASK_TASK_MANAGEMENT_REQUEST_SLOTS) >> 16) + 1;
-	hba->reserved_slot = 0;
 
 	hba->nortt = FIELD_GET(MASK_NUMBER_OUTSTANDING_RTT, hba->capabilities) + 1;
 
@@ -3111,6 +3110,20 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
 	return err;
 }
 
+static int ufshcd_queue_reserved_command(struct Scsi_Host *host,
+					 struct scsi_cmnd *cmd)
+{
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+	struct request *rq = scsi_cmd_to_rq(cmd);
+	struct ufs_hba *hba = shost_priv(host);
+	struct ufs_hw_queue *hwq =
+		hba->mcq_enabled ? ufshcd_mcq_req_to_hwq(hba, rq) : NULL;
+
+	ufshcd_add_query_upiu_trace(hba, UFS_QUERY_SEND, lrbp->ucd_req_ptr);
+	ufshcd_send_command(hba, cmd, hwq);
+	return 0;
+}
+
 static void ufshcd_setup_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
 				 enum dev_cmd_type cmd_type, u8 lun, int tag)
 {
@@ -3240,84 +3253,6 @@ ufshcd_dev_cmd_completion(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
 	return err;
 }
 
-/*
- * Return: 0 upon success; < 0 upon failure.
- */
-static int ufshcd_wait_for_dev_cmd(struct ufs_hba *hba,
-		struct ufshcd_lrb *lrbp, int max_timeout)
-{
-	struct scsi_cmnd *cmd = (struct scsi_cmnd *)lrbp - 1;
-	const int tag = scsi_cmd_to_rq(cmd)->tag;
-	unsigned long time_left = msecs_to_jiffies(max_timeout);
-	unsigned long flags;
-	bool pending;
-	int err;
-
-retry:
-	time_left = wait_for_completion_timeout(&hba->dev_cmd.complete,
-						time_left);
-
-	if (likely(time_left)) {
-		err = ufshcd_get_tr_ocs(lrbp, NULL);
-	} else {
-		err = -ETIMEDOUT;
-		dev_dbg(hba->dev, "%s: dev_cmd request timedout, tag %d\n",
-			__func__, tag);
-
-		/* MCQ mode */
-		if (hba->mcq_enabled) {
-			/* successfully cleared the command, retry if needed */
-			if (ufshcd_clear_cmd(hba, tag) == 0)
-				err = -EAGAIN;
-			return err;
-		}
-
-		/* SDB mode */
-		if (ufshcd_clear_cmd(hba, tag) == 0) {
-			/* successfully cleared the command, retry if needed */
-			err = -EAGAIN;
-			/*
-			 * Since clearing the command succeeded we also need to
-			 * clear the task tag bit from the outstanding_reqs
-			 * variable.
-			 */
-			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			pending = test_bit(tag, &hba->outstanding_reqs);
-			if (pending)
-				__clear_bit(tag, &hba->outstanding_reqs);
-			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
-
-			if (!pending) {
-				/*
-				 * The completion handler ran while we tried to
-				 * clear the command.
-				 */
-				time_left = 1;
-				goto retry;
-			}
-		} else {
-			dev_err(hba->dev, "%s: failed to clear tag %d\n",
-				__func__, tag);
-
-			spin_lock_irqsave(&hba->outstanding_lock, flags);
-			pending = test_bit(tag, &hba->outstanding_reqs);
-			spin_unlock_irqrestore(&hba->outstanding_lock, flags);
-
-			if (!pending) {
-				/*
-				 * The completion handler ran while we tried to
-				 * clear the command.
-				 */
-				time_left = 1;
-				goto retry;
-			}
-		}
-	}
-
-	WARN_ONCE(err > 0, "Incorrect return value %d > 0\n", err);
-	return err;
-}
-
 static void ufshcd_dev_man_lock(struct ufs_hba *hba)
 {
 	ufshcd_hold(hba);
@@ -3332,25 +3267,6 @@ static void ufshcd_dev_man_unlock(struct ufs_hba *hba)
 	ufshcd_release(hba);
 }
 
-/*
- * Return: 0 upon success; < 0 upon failure.
- */
-static int ufshcd_issue_dev_cmd(struct ufs_hba *hba, struct scsi_cmnd *cmd,
-				const u32 tag, int timeout)
-{
-	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-	int err;
-
-	ufshcd_add_query_upiu_trace(hba, UFS_QUERY_SEND, lrbp->ucd_req_ptr);
-	ufshcd_send_command(hba, cmd, hba->dev_cmd_queue);
-	err = ufshcd_wait_for_dev_cmd(hba, lrbp, timeout);
-
-	ufshcd_add_query_upiu_trace(hba, err ? UFS_QUERY_ERR : UFS_QUERY_COMP,
-				    (struct utp_upiu_req *)lrbp->ucd_rsp_ptr);
-
-	return err;
-}
-
 struct ufshcd_exec_dev_cmd_args {
 	struct scsi_exec_args args;
 	struct ufs_hba *hba;
@@ -3368,13 +3284,24 @@ static int ufshcd_init_dev_cmd(struct scsi_cmnd *cmd,
 	return ufshcd_compose_dev_cmd(uea->hba, cmd, uea->cmd_type, tag);
 }
 
+static void ufshcd_copy_dev_cmd_result(struct scsi_cmnd *cmd,
+				       const struct scsi_exec_args *args)
+{
+	const struct ufshcd_exec_dev_cmd_args *uea =
+		container_of(args, typeof(*uea), args);
+	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
+
+	ufshcd_dev_cmd_completion(uea->hba, lrbp);
+}
+
 /**
  * ufshcd_exec_dev_cmd - API for sending device management requests
  * @hba: UFS hba
  * @cmd_type: specifies the type (NOP, Query...)
  * @timeout: timeout in milliseconds
  *
- * Return: 0 upon success; < 0 upon failure.
+ * Return: 0 upon success; < 0 upon timeout; > 0 in case the UFS device
+ * reported an OCS error.
  *
  * NOTE: Since there is only one available tag for device management commands,
  * it is expected you hold the hba->dev_cmd.lock mutex.
@@ -3383,26 +3310,21 @@ static int ufshcd_exec_dev_cmd(struct ufs_hba *hba,
 		enum dev_cmd_type cmd_type, int timeout)
 {
 	const struct ufshcd_exec_dev_cmd_args args = {
+		.args = {
+			.init_cmd = ufshcd_init_dev_cmd,
+			.copy_result = ufshcd_copy_dev_cmd_result,
+			.req_flags = BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT,
+			.specify_hctx = true,
+		},
 		.hba = hba,
 		.cmd_type = cmd_type
 	};
-	const u32 tag = hba->reserved_slot;
-	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
-	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
-	int err;
 
-	/* Protects use of hba->reserved_slot. */
+	/* Protects use of hba->dev_cmd. */
 	lockdep_assert_held(&hba->dev_cmd.lock);
 
-	err = ufshcd_init_dev_cmd(cmd, &args.args);
-	if (unlikely(err))
-		return err;
-
-	err = ufshcd_issue_dev_cmd(hba, cmd, tag, timeout);
-	if (err)
-		return err;
-
-	return ufshcd_dev_cmd_completion(hba, lrbp);
+	return scsi_execute_cmd(hba->host->pseudo_sdev, NULL, REQ_OP_DRV_OUT,
+				NULL, 0, timeout, 0, &args.args);
 }
 
 /**
@@ -5667,6 +5589,10 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	enum utp_ocs ocs;
 
+	if (WARN_ONCE(!cmd, "cqe->command_desc_base_addr = %#llx\n",
+		      le64_to_cpu(cqe->command_desc_base_addr)))
+		return;
+
 	if (hba->monitor.enabled) {
 		lrbp->compl_time_stamp = ktime_get();
 		lrbp->compl_time_stamp_local_clock = local_clock();
@@ -5677,15 +5603,20 @@ void ufshcd_compl_one_cqe(struct ufs_hba *hba, int task_tag,
 		ufshcd_add_command_trace(hba, cmd, UFS_CMD_COMP);
 		cmd->result = ufshcd_transfer_rsp_status(hba, cmd, cqe);
 		ufshcd_release_scsi_cmd(hba, cmd);
-		/* Do not touch lrbp after scsi done */
-		scsi_done(cmd);
 	} else {
 		if (cqe) {
 			ocs = le32_to_cpu(cqe->status) & MASK_OCS;
 			lrbp->utr_descriptor_ptr->header.ocs = ocs;
+		} else {
+			ocs = lrbp->utr_descriptor_ptr->header.ocs;
 		}
-		complete(&hba->dev_cmd.complete);
+		cmd->result = ocs;
+		ufshcd_add_query_upiu_trace(hba,
+			ocs == OCS_SUCCESS ? UFS_QUERY_COMP : UFS_QUERY_ERR,
+			(struct utp_upiu_req *)lrbp->ucd_rsp_ptr);
 	}
+	/* Do not touch lrbp after scsi_done() has been called. */
+	scsi_done(cmd);
 }
 
 /**
@@ -7460,6 +7391,12 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,
 {
 	int err, copy_result_err = 0;
 	const struct ufshcd_exec_devman_upiu_cmd_args args = {
+		.args = {
+			.init_cmd = ufshcd_init_upiu_cmd,
+			.copy_result = ufshcd_copy_upiu_cmd_result,
+			.req_flags = BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT,
+			.specify_hctx = true,
+		},
 		.hba = hba,
 		.req_upiu = req_upiu,
 		.rsp_upiu = rsp_upiu,
@@ -7469,22 +7406,13 @@ static int ufshcd_issue_devman_upiu_cmd(struct ufs_hba *hba,
 		.desc_op = desc_op,
 		.err = &copy_result_err,
 	};
-	const u32 tag = hba->reserved_slot;
-	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
 
-	/* Protects use of hba->reserved_slot. */
+	/* Protects use of hba->dev_cmd. */
 	lockdep_assert_held(&hba->dev_cmd.lock);
 
-	err = ufshcd_init_upiu_cmd(cmd, &args.args);
-	WARN_ON_ONCE(err);
-
-	err = ufshcd_issue_dev_cmd(hba, cmd, tag, dev_cmd_timeout);
-	if (err)
-		return err;
-
-	ufshcd_copy_upiu_cmd_result(cmd, &args.args);
-
-	return copy_result_err;
+	err = scsi_execute_cmd(hba->host->pseudo_sdev, NULL, REQ_OP_DRV_OUT,
+			       NULL, 0, dev_cmd_timeout, 0, &args.args);
+	return err ?: copy_result_err;
 }
 
 /**
@@ -7666,6 +7594,12 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 {
 	int err, upiu_result = 0;
 	const struct ufshcd_rpmb_args args = {
+		.args = {
+			.init_cmd = ufshcd_init_rpmb,
+			.copy_result = ufshcd_copy_rpmb_result,
+			.req_flags = BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT,
+			.specify_hctx = true,
+		},
 		.hba = hba,
 		.req_upiu = req_upiu,
 		.rsp_upiu = rsp_upiu,
@@ -7676,22 +7610,12 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r
 		.dir = dir,
 		.upiu_result = &upiu_result,
 	};
-	const u32 tag = hba->reserved_slot;
-	struct scsi_cmnd *cmd = ufshcd_tag_to_cmd(hba, tag);
 
-	/* Protects use of hba->reserved_slot. */
 	ufshcd_dev_man_lock(hba);
 
-	err = ufshcd_init_rpmb(cmd, &args.args);
-	WARN_ON_ONCE(err);
-
-	err = ufshcd_issue_dev_cmd(hba, cmd, tag, ADVANCED_RPMB_REQ_TIMEOUT);
-	if (err)
-		goto unlock;
-
-	ufshcd_copy_rpmb_result(cmd, &args.args);
+	err = scsi_execute_cmd(hba->host->pseudo_sdev, NULL, REQ_OP_DRV_OUT,
+			NULL, 0, ADVANCED_RPMB_REQ_TIMEOUT, 0, &args.args);
 
-unlock:
 	ufshcd_dev_man_unlock(hba);
 
 	return err ?: upiu_result;
@@ -7862,7 +7786,8 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 {
 	struct Scsi_Host *host = cmd->device->host;
 	struct ufs_hba *hba = shost_priv(host);
-	int tag = scsi_cmd_to_rq(cmd)->tag;
+	struct request *rq = scsi_cmd_to_rq(cmd);
+	int tag = rq->tag;
 	struct ufshcd_lrb *lrbp = scsi_cmd_priv(cmd);
 	unsigned long flags;
 	int err = FAILED;
@@ -7892,7 +7817,8 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 	 * to reduce repeated printouts. For other aborted requests only print
 	 * basic details.
 	 */
-	scsi_print_command(cmd);
+	if (ufshcd_is_scsi_cmd(cmd))
+		scsi_print_command(cmd);
 	if (!hba->req_abort_count) {
 		ufshcd_update_evt_hist(hba, UFS_EVT_ABORT, tag);
 		ufshcd_print_evt_hist(hba);
@@ -7944,7 +7870,10 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 		goto release;
 	}
 
-	err = ufshcd_try_to_abort_task(hba, tag);
+	if (blk_mq_is_reserved_rq(rq))
+		err = ufshcd_clear_cmd(hba, tag);
+	else
+		err = ufshcd_try_to_abort_task(hba, tag);
 	if (err) {
 		dev_err(hba->dev, "%s: failed with err %d\n", __func__, err);
 		ufshcd_set_req_abort_skip(hba, hba->outstanding_reqs);
@@ -9072,7 +9001,6 @@ static int ufshcd_alloc_mcq(struct ufs_hba *hba)
 
 	hba->host->can_queue = hba->nutrs - UFSHCD_NUM_RESERVED;
 	hba->host->nr_reserved_cmds = UFSHCD_NUM_RESERVED;
-	hba->reserved_slot = 0;
 
 	return 0;
 err:
@@ -9318,6 +9246,7 @@ static const struct scsi_host_template ufshcd_driver_template = {
 	.cmd_size		= sizeof(struct ufshcd_lrb),
 	.init_cmd_priv		= ufshcd_init_cmd_priv,
 	.queuecommand		= ufshcd_queuecommand,
+	.queue_reserved_command	= ufshcd_queue_reserved_command,
 	.mq_poll		= ufshcd_poll,
 	.sdev_init		= ufshcd_sdev_init,
 	.sdev_configure		= ufshcd_sdev_configure,
@@ -10860,8 +10789,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
 						UFS_SLEEP_PWR_MODE,
 						UIC_LINK_HIBERN8_STATE);
 
-	init_completion(&hba->dev_cmd.complete);
-
 	err = ufshcd_hba_init(hba);
 	if (err)
 		goto out_error;
diff --git a/include/ufs/ufshcd.h b/include/ufs/ufshcd.h
index 9eb69241e40f..e07834509182 100644
--- a/include/ufs/ufshcd.h
+++ b/include/ufs/ufshcd.h
@@ -236,13 +236,11 @@ struct ufs_query {
  * struct ufs_dev_cmd - all assosiated fields with device management commands
  * @type: device management command type - Query, NOP OUT
  * @lock: lock to allow one command at a time
- * @complete: internal commands completion
  * @query: Device management query information
  */
 struct ufs_dev_cmd {
 	enum dev_cmd_type type;
 	struct mutex lock;
-	struct completion complete;
 	struct ufs_query query;
 };
 
@@ -838,7 +836,6 @@ enum ufshcd_mcq_opr {
  * @nutrs: Transfer Request Queue depth supported by controller
  * @nortt - Max outstanding RTTs supported by controller
  * @nutmrs: Task Management Queue depth supported by controller
- * @reserved_slot: Used to submit device commands. Protected by @dev_cmd.lock.
  * @ufs_version: UFS Version to which controller complies
  * @vops: pointer to variant specific operations
  * @vps: pointer to variant specific parameters
@@ -929,7 +926,6 @@ enum ufshcd_mcq_opr {
  * @res: array of resource info of MCQ registers
  * @mcq_base: Multi circular queue registers base address
  * @uhq: array of supported hardware queues
- * @dev_cmd_queue: Queue for issuing device management commands
  * @mcq_opr: MCQ operation and runtime registers
  * @ufs_rtc_update_work: A work for UFS RTC periodic update
  * @pm_qos_req: PM QoS request handle
@@ -980,7 +976,6 @@ struct ufs_hba {
 	int nortt;
 	u32 mcq_capabilities;
 	int nutmrs;
-	u32 reserved_slot;
 	u32 ufs_version;
 	const struct ufs_hba_variant_ops *vops;
 	struct ufs_hba_variant_params *vps;
@@ -1098,7 +1093,6 @@ struct ufs_hba {
 	bool mcq_esi_enabled;
 	void __iomem *mcq_base;
 	struct ufs_hw_queue *uhq;
-	struct ufs_hw_queue *dev_cmd_queue;
 	struct ufshcd_mcq_opr_info_t mcq_opr[OPR_MAX];
 
 	struct delayed_work ufs_rtc_update_work;

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[michael.christie]

On 9/12/25 1:21 PM, Bart Van Assche wrote:
> Make the @cmd argument optional. Add .init_cmd() and .copy_result()
> callbacks in struct scsi_exec_args. Support allocating from a specific
> hardware queue. This patch prepares for submitting reserved commands
> with scsi_execute_cmd().
> 
> Cc: John Garry <john.g.garry@oracle.com>
> Cc: Hannes Reinecke <hare@suse.de>
> Cc: Mike Christie <michael.christie@oracle.com>
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>  drivers/scsi/scsi_lib.c    | 28 +++++++++++++++++++++++++---
>  include/scsi/scsi_cmnd.h   |  2 ++
>  include/scsi/scsi_device.h | 37 +++++++++++++++++++++++++++++--------
>  3 files changed, 56 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 5e636e015352..022cd454d658 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -308,7 +308,10 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
>  		return -EINVAL;
>  
>  retry:
> -	req = scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
> +	req = args->specify_hctx ?
> +		scsi_alloc_request_hctx(sdev->request_queue, opf,
> +					args->req_flags, args->hctx_idx) :
> +		scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
>  	if (IS_ERR(req))
>  		return PTR_ERR(req);
>  
> @@ -318,8 +321,12 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
>  			goto out;
>  	}
>  	scmd = blk_mq_rq_to_pdu(req);
> -	scmd->cmd_len = COMMAND_SIZE(cmd[0]);
> -	memcpy(scmd->cmnd, cmd, scmd->cmd_len);
> +	if (cmd) {
> +		scmd->cmd_len = COMMAND_SIZE(cmd[0]);
> +		memcpy(scmd->cmnd, cmd, scmd->cmd_len);
> +	}
> +	if (args->init_cmd)
> +		args->init_cmd(scmd, args);

I didn't follow all the reserved tag discussion so this might be a
duplicate question. What will you do in the callout?

For these types of commands is the scsi_host_template->init_cmd_priv
callout not called, or not called at the right time, or do you want to
do special case initialization?

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[Bart Van Assche]

On 9/12/25 1:03 PM, michael.christie@oracle.com wrote:
> On 9/12/25 1:21 PM, Bart Van Assche wrote:
>> +	if (cmd) {
>> +		scmd->cmd_len = COMMAND_SIZE(cmd[0]);
>> +		memcpy(scmd->cmnd, cmd, scmd->cmd_len);
>> +	}
>> +	if (args->init_cmd)
>> +		args->init_cmd(scmd, args);
> 
> I didn't follow all the reserved tag discussion so this might be a
> duplicate question. What will you do in the callout?
> 
> For these types of commands is the scsi_host_template->init_cmd_priv
> callout not called, or not called at the right time, or do you want to
> do special case initialization?

Hi Mike,

The scsi_execute_cmd() @args pointer is not passed to the SCSI host
template .init_cmd_priv callback. There are several examples in patch
29/29 where that pointer is essential at initialization time.

Patch 29/29 uses scsi_execute_cmd() to submit non-SCSI commands to a UFS
device.

Thanks,

Bart.

Re: [PATCH v4 02/29] scsi: core: Move two statements

[John Garry]

On 12/09/2025 19:21, Bart Van Assche wrote:

About "scsi: core: Move two statements", you could be more specific, 
like "scsi: core: Reorder some sdev initialization"

> Move two statements that will be needed for pseudo SCSI devices in front
> of code that won't be needed for pseudo SCSI devices. No functionality
> has been changed.
> 
> CC: John Garry<john.g.garry@oracle.com>
> Cc: Hannes Reinecke<hare@suse.de>
> Signed-off-by: Bart Van Assche<bvanassche@acm.org>
> ---

This change on its own looks ok, so:

Reviewed-by: John Garry <john.g.garry@oracle.com>

Re: [PATCH v4 04/29] scsi: core: Support allocating a pseudo SCSI device

[John Garry]

On 12/09/2025 19:21, Bart Van Assche wrote:
> From: Hannes Reinecke <hare@suse.de>
> 
> Allocate a pseudo SCSI device if 'nr_reserved_cmds' has been set. Pseudo
> SCSI devices have the SCSI ID <max_id>:U64_MAX so they won't clash with
> any devices the LLD might create. Pseudo SCSI devices are excluded from
> scanning and will not show up in sysfs. Additionally, pseudo SCSI
> devices are skipped by shost_for_each_device(). This prevents that the
> SCSI error handler tries to submit a reset to a non-existent logical unit.
> 
> Do not allocate a budget map for pseudo SCSI devices since the
> cmd_per_lun limit does not apply to pseudo SCSI devices.

IDGI, in v3 series you said that you would allocate the budget map 
https://lore.kernel.org/linux-scsi/20250827000816.2370150-1-bvanassche@acm.org/T/#m13c361e081b886b9318238b6dc05b571840b9698

FWIW, I still think that it is worth allocating the budget map for the 
psuedo sdev and making the queue depth the same as nr_reserved_commands 
via a scsi_change_queue_depth() call.

If we want to optimise budget code handling, then I think that it is 
worth doing later. The whole budget map and cmd_per_lun handling is 
murky IMHO.

> 
> Do not perform queue depth ramp up / ramp down for pseudo SCSI devices.

Are we even ever going to try ramp up or down for the pseudo sdev?

I suppose we could see it if there is some internal reserved command 
error, right?

> 
> Pseudo SCSI devices will be used to send internal commands to a storage
> device.
> 
> Cc: John Garry <john.g.garry@oracle.com>
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> [ bvanassche: edited patch description / renamed host_sdev into
>    pseudo_sdev / unexported scsi_get_host_dev() / modified error path in
>    scsi_get_pseudo_dev() / skip pseudo devices in __scsi_iterate_devices()
>    and also when calling sdev_init(), sdev_configure() and sdev_destroy().
>    See also
>    https://lore.kernel.org/linux-scsi/20211125151048.103910-2-hare@suse.de/ ]
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>   drivers/scsi/hosts.c       |  8 +++++
>   drivers/scsi/scsi.c        |  9 +++--
>   drivers/scsi/scsi_error.c  |  3 ++
>   drivers/scsi/scsi_priv.h   |  2 ++
>   drivers/scsi/scsi_scan.c   | 69 +++++++++++++++++++++++++++++++++++++-
>   drivers/scsi/scsi_sysfs.c  |  5 ++-
>   include/scsi/scsi_device.h | 16 +++++++++
>   include/scsi/scsi_host.h   |  6 ++++
>   8 files changed, 114 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
> index 9bb7f0114763..986586bf67dc 100644
> --- a/drivers/scsi/hosts.c
> +++ b/drivers/scsi/hosts.c
> @@ -307,6 +307,14 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
>   	if (error)
>   		goto out_del_dev;
>   
> +	if (sht->nr_reserved_cmds) {
> +		shost->pseudo_sdev = scsi_get_pseudo_dev(shost);
> +		if (!shost->pseudo_sdev) {
> +			error = -ENOMEM;
> +			goto out_del_dev;
> +		}
> +	}
> +
>   	scsi_proc_host_add(shost);
>   	scsi_autopm_put_host(shost);
>   	return error;
> diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
> index ff6b0973d3b4..2d2a52c3ef49 100644
> --- a/drivers/scsi/scsi.c
> +++ b/drivers/scsi/scsi.c
> @@ -257,6 +257,8 @@ EXPORT_SYMBOL(scsi_change_queue_depth);
>    */
>   int scsi_track_queue_full(struct scsi_device *sdev, int depth)
>   {
> +	if (scsi_device_is_pseudo_dev(sdev))
> +		return 0;
>   
>   	/*
>   	 * Don't let QUEUE_FULLs on the same
> @@ -828,8 +830,11 @@ struct scsi_device *__scsi_iterate_devices(struct Scsi_Host *shost,
>   	spin_lock_irqsave(shost->host_lock, flags);
>   	while (list->next != &shost->__devices) {
>   		next = list_entry(list->next, struct scsi_device, siblings);
> -		/* skip devices that we can't get a reference to */
> -		if (!scsi_device_get(next))
> +		/*
> +		 * Skip pseudo devices and also devices for which
> +		 * scsi_device_get() fails.
> +		 */
> +		if (!scsi_device_is_pseudo_dev(next) && !scsi_device_get(next))
>   			break;

looks ok

>   		next = NULL;
>   		list = list->next;
> diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> index 746ff6a1f309..540d82974529 100644
> --- a/drivers/scsi/scsi_error.c
> +++ b/drivers/scsi/scsi_error.c
> @@ -749,6 +749,9 @@ static void scsi_handle_queue_ramp_up(struct scsi_device *sdev)
>   	const struct scsi_host_template *sht = sdev->host->hostt;
>   	struct scsi_device *tmp_sdev;
>   
> +	if (scsi_device_is_pseudo_dev(sdev))
> +		return;
> +
>   	if (!sht->track_queue_depth ||
>   	    sdev->queue_depth >= sdev->max_queue_depth)
>   		return;
> diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h
> index 5b2b19f5e8ec..da3bc87ac5a6 100644
> --- a/drivers/scsi/scsi_priv.h
> +++ b/drivers/scsi/scsi_priv.h
> @@ -135,6 +135,8 @@ extern int scsi_complete_async_scans(void);
>   extern int scsi_scan_host_selected(struct Scsi_Host *, unsigned int,
>   				   unsigned int, u64, enum scsi_scan_mode);
>   extern void scsi_forget_host(struct Scsi_Host *);
> +struct scsi_device *scsi_get_pseudo_dev(struct Scsi_Host *);
> +bool scsi_device_is_pseudo_dev(struct scsi_device *sdev);
>   
>   /* scsi_sysctl.c */
>   #ifdef CONFIG_SYSCTL
> diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> index de039efef290..a3523f964bc1 100644
> --- a/drivers/scsi/scsi_scan.c
> +++ b/drivers/scsi/scsi_scan.c
> @@ -225,6 +225,8 @@ static int scsi_realloc_sdev_budget_map(struct scsi_device *sdev,
>   	int ret;
>   	struct sbitmap sb_backup;
>   
> +	WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev));
> +
>   	depth = min_t(unsigned int, depth, scsi_device_max_queue_depth(sdev));
>   
>   	/*
> @@ -349,6 +351,9 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
>   
>   	scsi_sysfs_device_initialize(sdev);
>   
> +	if (scsi_device_is_pseudo_dev(sdev))
> +		return sdev;
> +
>   	depth = sdev->host->cmd_per_lun ?: 1;
>   
>   	/*
> @@ -1070,6 +1075,9 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
>   
>   	sdev->sdev_bflags = *bflags;
>   
> +	if (scsi_device_is_pseudo_dev(sdev))
> +		return SCSI_SCAN_LUN_PRESENT;
> +
>   	/*
>   	 * No need to freeze the queue as it isn't reachable to anyone else yet.
>   	 */
> @@ -1213,6 +1221,12 @@ static int scsi_probe_and_add_lun(struct scsi_target *starget,
>   	if (!sdev)
>   		goto out;
>   
> +	if (scsi_device_is_pseudo_dev(sdev)) {
> +		if (bflagsp)
> +			*bflagsp = BLIST_NOLUN;
> +		return SCSI_SCAN_LUN_PRESENT;
> +	}
> +
>   	result = kmalloc(result_len, GFP_KERNEL);
>   	if (!result)
>   		goto out_free_sdev;
> @@ -2084,12 +2098,65 @@ void scsi_forget_host(struct Scsi_Host *shost)
>    restart:
>   	spin_lock_irqsave(shost->host_lock, flags);
>   	list_for_each_entry(sdev, &shost->__devices, siblings) {
> -		if (sdev->sdev_state == SDEV_DEL)
> +		if (scsi_device_is_pseudo_dev(sdev) ||
> +		    sdev->sdev_state == SDEV_DEL)
>   			continue;

maybe this would be neater with seperate if statements

>   		spin_unlock_irqrestore(shost->host_lock, flags);
>   		__scsi_remove_device(sdev);
>   		goto restart;
>   	}
>   	spin_unlock_irqrestore(shost->host_lock, flags);
> +
> +	/*
> +	 * Remove the pseudo device last since it may be needed during removal
> +	 * of other SCSI devices.
> +	 */
> +	if (shost->pseudo_sdev)
> +		__scsi_remove_device(shost->pseudo_sdev);

looks ok

>   }
>   
> +/**
> + * scsi_get_pseudo_dev() - Attach a pseudo SCSI device to a SCSI host
> + * @shost: Host that needs a pseudo SCSI device
> + *
> + * Lock status: None assumed.
> + *
> + * Returns:     The scsi_device or NULL
> + *
> + * Notes:
> + *	Attach a single scsi_device to the Scsi_Host. The primary aim for this
> + *	device is to serve as a container from which SCSI commands can be
> + *	allocated. Each SCSI command will carry a command tag allocated by the
> + *	block layer. These SCSI commands can be used by the LLDD to send
> + *	internal or passthrough commands without having to manage tag allocation
> + *	inside the LLDD.
> + */
> +struct scsi_device *scsi_get_pseudo_dev(struct Scsi_Host *shost)
> +{
> +	struct scsi_device *sdev = NULL;
> +	struct scsi_target *starget;
> +
> +	guard(mutex)(&shost->scan_mutex);
> +
> +	if (!scsi_host_scan_allowed(shost))
> +		goto out;

when would/could this fail? It seems to me that the shost add would also 
fail (if this failed), right?

> +
> +	starget = scsi_alloc_target(&shost->shost_gendev, 0, shost->max_id);
> +	if (!starget)
> +		goto out;
> +
> +	sdev = scsi_alloc_sdev(starget, U64_MAX, NULL);
> +	if (!sdev) {
> +		scsi_target_reap(starget);
> +		goto put_target;
> +	}
> +
> +	sdev->borken = 0;
> +
> +put_target:
> +	/* See also the get_device(dev) call in scsi_alloc_target(). */
> +	put_device(&starget->dev);
> +
> +out:
> +	return sdev;
> +}

looks ok

> diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
> index 169af7d47ce7..22f76a1ca23b 100644
> --- a/drivers/scsi/scsi_sysfs.c
> +++ b/drivers/scsi/scsi_sysfs.c
> @@ -1406,6 +1406,9 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev)
>   	int error;
>   	struct scsi_target *starget = sdev->sdev_target;
>   
> +	if (WARN_ON_ONCE(scsi_device_is_pseudo_dev(sdev)))
> +		return -EINVAL;
> +
>   	error = scsi_target_add(starget);
>   	if (error)
>   		return error;
> @@ -1513,7 +1516,7 @@ void __scsi_remove_device(struct scsi_device *sdev)
>   	kref_put(&sdev->host->tagset_refcnt, scsi_mq_free_tags);
>   	cancel_work_sync(&sdev->requeue_work);
>   
> -	if (sdev->host->hostt->sdev_destroy)
> +	if (!scsi_device_is_pseudo_dev(sdev) && sdev->host->hostt->sdev_destroy)
>   		sdev->host->hostt->sdev_destroy(sdev);
>   	transport_destroy_device(dev);
>   
> diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
> index 6d6500148c4b..3846f5dfc51c 100644
> --- a/include/scsi/scsi_device.h
> +++ b/include/scsi/scsi_device.h
> @@ -589,6 +589,22 @@ static inline unsigned int sdev_id(struct scsi_device *sdev)
>   #define scmd_id(scmd) sdev_id((scmd)->device)
>   #define scmd_channel(scmd) sdev_channel((scmd)->device)
>   
> +/**
> + * scsi_device_is_pseudo_dev() - Whether a device is a pseudo SCSI device.
> + * @sdev: SCSI device to examine
> + *
> + * A pseudo SCSI device can be used to allocate SCSI commands but does not show
> + * up in sysfs. Additionally, the logical unit information in *@sdev is made up.
> + *
> + * This function tests the LUN number instead of comparing @sdev with
> + * @sdev->host->pseudo_sdev because this function may be called before
> + * @sdev->host->pseudo_sdev has been initialized.
> + */
> +static inline bool scsi_device_is_pseudo_dev(struct scsi_device *sdev)
> +{
> +	return sdev->lun == U64_MAX;

could you also check sdev->shost->psuedo_sdev == sdev?

I suppose just checking the lun is simpler

> +}
> +
>   /*
>    * checks for positions of the SCSI state machine
>    */
> diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
> index 91eb3f52b3d0..3bfb53cf5dfc 100644
> --- a/include/scsi/scsi_host.h
> +++ b/include/scsi/scsi_host.h
> @@ -721,6 +721,12 @@ struct Scsi_Host {
>   	/* ldm bits */
>   	struct device		shost_gendev, shost_dev;
>   
> +	/*
> +	 * A SCSI device structure used for sending internal commands to the
> +	 * HBA. There is no corresponding logical unit inside the SCSI device.
> +	 */
> +	struct scsi_device *pseudo_sdev;
> +
>   	/*
>   	 * Points to the transport data (if any) which is allocated
>   	 * separately

Re: [PATCH v4 02/29] scsi: core: Move two statements

[Hannes Reinecke]

On 9/12/25 20:21, Bart Van Assche wrote:
> Move two statements that will be needed for pseudo SCSI devices in front
> of code that won't be needed for pseudo SCSI devices. No functionality
> has been changed.
> 
> CC: John Garry <john.g.garry@oracle.com>
> Cc: Hannes Reinecke <hare@suse.de>
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>   drivers/scsi/scsi_scan.c | 7 ++++---
>   1 file changed, 4 insertions(+), 3 deletions(-)
> 

Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [PATCH v4 03/29] scsi: core: Make the budget map optional

[Hannes Reinecke]

On 9/12/25 20:21, Bart Van Assche wrote:
> Prepare for not allocating a budget map for pseudo SCSI devices by
> checking whether a budget map has been allocated before using a budget
> map.
> 
> Cc: Hannes Reinecke <hare@suse.de>
> Cc: John Garry <john.g.garry@oracle.com>
> Cc: Ming Lei <ming.lei@redhat.com>
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>   drivers/scsi/scsi.c     |  2 ++
>   drivers/scsi/scsi_lib.c | 14 ++++++++++++--
>   2 files changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
> index 9a0f467264b3..ff6b0973d3b4 100644
> --- a/drivers/scsi/scsi.c
> +++ b/drivers/scsi/scsi.c
> @@ -216,6 +216,8 @@ int scsi_device_max_queue_depth(struct scsi_device *sdev)
>    */
>   int scsi_change_queue_depth(struct scsi_device *sdev, int depth)
>   {
> +	WARN_ON_ONCE(!sdev->budget_map.map);
> +
>   	depth = min_t(int, depth, scsi_device_max_queue_depth(sdev));
>   
>   	if (depth > 0) {
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 9c67e04265ce..91a0c7f843c1 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -396,7 +396,8 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)
>   	if (starget->can_queue > 0)
>   		atomic_dec(&starget->target_busy);
>   
> -	sbitmap_put(&sdev->budget_map, cmd->budget_token);
> +	if (sdev->budget_map.map)
> +		sbitmap_put(&sdev->budget_map, cmd->budget_token);
>   	cmd->budget_token = -1;
>   }
>   
> @@ -1360,6 +1361,14 @@ static inline int scsi_dev_queue_ready(struct request_queue *q,
>   {
>   	int token;
>   
> +	/*
> +	 * Do not allocate a budget token for reserved SCSI commands. Budget
> +	 * tokens are used to enforce the cmd_per_lun limit. That limit does not
> +	 * apply to reserved commands.
> +	 */
> +	if (!sdev->budget_map.map)
> +		return INT_MAX;
> +

Strictly speaking it's not related to reserved commands, but rather to 
cmd_per_lun. Wouldn't it be better to introduce a way to disable 
cmd_per_lun (eg by setting it to INT_MAX or somesuch), and then disable
the budget map when cmd_per_lun is disabled?

Other than that I really like the idea of being able to disable the
budget map. I always wondered if we won't be better off with dropping
the budget map for HBAs with a shared host tagset.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [PATCH v4 04/29] scsi: core: Support allocating a pseudo SCSI device

[Hannes Reinecke]

On 9/16/25 10:21, John Garry wrote:
> On 12/09/2025 19:21, Bart Van Assche wrote:
>> From: Hannes Reinecke <hare@suse.de>
>>
>> Allocate a pseudo SCSI device if 'nr_reserved_cmds' has been set. Pseudo
>> SCSI devices have the SCSI ID <max_id>:U64_MAX so they won't clash with
>> any devices the LLD might create. Pseudo SCSI devices are excluded from
>> scanning and will not show up in sysfs. Additionally, pseudo SCSI
>> devices are skipped by shost_for_each_device(). This prevents that the
>> SCSI error handler tries to submit a reset to a non-existent logical 
>> unit.
>>
>> Do not allocate a budget map for pseudo SCSI devices since the
>> cmd_per_lun limit does not apply to pseudo SCSI devices.
> 
> IDGI, in v3 series you said that you would allocate the budget map 
> https://lore.kernel.org/linux-scsi/20250827000816.2370150-1- 
> bvanassche@acm.org/T/#m13c361e081b886b9318238b6dc05b571840b9698
> 
> FWIW, I still think that it is worth allocating the budget map for the 
> psuedo sdev and making the queue depth the same as nr_reserved_commands 
> via a scsi_change_queue_depth() call.
> 
No. budget map is only for resource arbitration between several devices
sharing the same underlying bitmap. Which doesn't apply for the pseudo
device; that is a virtual device pointing to the HBA itself, of which
we only every will have _one_.

> If we want to optimise budget code handling, then I think that it is 
> worth doing later. The whole budget map and cmd_per_lun handling is 
> murky IMHO.
> 
Oh, most definitely. budget map / cmd_per_lun was okay for single queue
devices where we couldn't really tell how many commands we should send.
But for multiqueue we know exactly how many commands should be possible,
so for _real_ multiqueue devices cmd_per_lun is pretty pointless if not
downright detrimental.
_Unless_ we have a multiqueue device with a shared host tagset, then it
sort of would make sense. But then one wonders why sbitmap doesn't do
that internally.

Sounds like a fun topic for ALPSS.

>>
>> Do not perform queue depth ramp up / ramp down for pseudo SCSI devices.
> 
> Are we even ever going to try ramp up or down for the pseudo sdev?
> 
No. Why would you want to restrict the number of eg TMFs you could send?

> I suppose we could see it if there is some internal reserved command 
> error, right?
> 
That would be equivalent to the HBA rejecting TMFs with 'out of 
resources'. At which point I would declare the HBA hosed and do
a full host/PCI reset.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[John Garry]

On 12/09/2025 19:21, Bart Van Assche wrote:
> Make the @cmd argument optional. Add .init_cmd() and .copy_result()
> callbacks in struct scsi_exec_args. Support allocating from a specific
> hardware queue. This patch prepares for submitting reserved commands
> with scsi_execute_cmd().
> 

A comment: I did suggest trying to reuse scsi_execute_cmd() for this. 
However considering the code changes and the comments, below. Maybe it 
is not the best thing. Let's further investigate...

> Cc: John Garry <john.g.garry@oracle.com>
> Cc: Hannes Reinecke <hare@suse.de>
> Cc: Mike Christie <michael.christie@oracle.com>
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>   drivers/scsi/scsi_lib.c    | 28 +++++++++++++++++++++++++---
>   include/scsi/scsi_cmnd.h   |  2 ++
>   include/scsi/scsi_device.h | 37 +++++++++++++++++++++++++++++--------
>   3 files changed, 56 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 5e636e015352..022cd454d658 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -308,7 +308,10 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
>   		return -EINVAL;
>   
>   retry:
> -	req = scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
> +	req = args->specify_hctx ?

Can you check args->hctx_idx is a specific queue or something like 
NVME_QID_ANY?

> +		scsi_alloc_request_hctx(sdev->request_queue, opf,
> +					args->req_flags, args->hctx_idx) :

did you consider passing this hctx info to scsi_alloc_request() and 
allow scsi_alloc_request() contain the logic as to call 
blk_mq_alloc_request_hctx() or blk_mq_alloc_request()?

> +		scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
>   	if (IS_ERR(req))
>   		return PTR_ERR(req);
>   
> @@ -318,8 +321,12 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
>   			goto out;
>   	}
>   	scmd = blk_mq_rq_to_pdu(req);
> -	scmd->cmd_len = COMMAND_SIZE(cmd[0]);
> -	memcpy(scmd->cmnd, cmd, scmd->cmd_len);


> +	if (cmd) {
> +		scmd->cmd_len = COMMAND_SIZE(cmd[0]);
> +		memcpy(scmd->cmnd, cmd, scmd->cmd_len);
> +	}

you could just pass a dummy cmd instead of doing this

> +	if (args->init_cmd)
> +		args->init_cmd(scmd, args);

is it possible to do this in ufshcd_init_cmd_priv? Or too late?

We could have a "is reserved command" check there (in 
ufshcd_init_cmd_priv), and do whatever processing is needed which is 
done in ufshcd_init_dev_cmd

>   	scmd->allowed = ml_retries;
>   	scmd->flags |= args->scmd_flags;
>   	req->timeout = timeout;
> @@ -353,6 +360,9 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
>   				     args->sshdr);
>   
>   	ret = scmd->result;
> +	if (ret == 0 && args->copy_result)
> +		args->copy_result(scmd, args);

can this sort of thing be done in the LLD completion handler?

> +
>    out:
>   	blk_mq_free_request(req);
>   
> @@ -1247,6 +1257,18 @@ struct request *scsi_alloc_request(struct request_queue *q, blk_opf_t opf,
>   }
>   EXPORT_SYMBOL_GPL(scsi_alloc_request);
>   
> +struct request *scsi_alloc_request_hctx(struct request_queue *q, blk_opf_t opf,
> +			blk_mq_req_flags_t flags, unsigned int hctx_idx)
> +{
> +	struct request *rq;
> +
> +	rq = blk_mq_alloc_request_hctx(q, opf, flags, hctx_idx);
> +	if (!IS_ERR(rq))
> +		scsi_initialize_rq(rq);
> +	return rq;
> +}
> +EXPORT_SYMBOL_GPL(scsi_alloc_request_hctx);
> +

Re: [PATCH v4 04/29] scsi: core: Support allocating a pseudo SCSI device

[John Garry]

On 16/09/2025 09:44, Hannes Reinecke wrote:
> On 9/16/25 10:21, John Garry wrote:
>> On 12/09/2025 19:21, Bart Van Assche wrote:
>>> From: Hannes Reinecke <hare@suse.de>
>>>
>>> Allocate a pseudo SCSI device if 'nr_reserved_cmds' has been set. Pseudo
>>> SCSI devices have the SCSI ID <max_id>:U64_MAX so they won't clash with
>>> any devices the LLD might create. Pseudo SCSI devices are excluded from
>>> scanning and will not show up in sysfs. Additionally, pseudo SCSI
>>> devices are skipped by shost_for_each_device(). This prevents that the
>>> SCSI error handler tries to submit a reset to a non-existent logical 
>>> unit.
>>>
>>> Do not allocate a budget map for pseudo SCSI devices since the
>>> cmd_per_lun limit does not apply to pseudo SCSI devices.
>>
>> IDGI, in v3 series you said that you would allocate the budget map 
>> https://urldefense.com/v3/__https://lore.kernel.org/linux- 
>> scsi/20250827000816.2370150-1-__;!!ACWV5N9M2RV99hQ! 
>> PXP56tO8nGuApLOmFXRDRrX5- 
>> UTPdJrsBgtullL3kVHsDxEZ0GtNqiKsGG1CxDiJjcvLSCcwCkBkO9U$ 
>> bvanassche@acm.org/T/#m13c361e081b886b9318238b6dc05b571840b9698
>>
>> FWIW, I still think that it is worth allocating the budget map for the 
>> psuedo sdev and making the queue depth the same as 
>> nr_reserved_commands via a scsi_change_queue_depth() call.
>>
> No. budget map is only for resource arbitration between several devices
> sharing the same underlying bitmap. Which doesn't apply for the pseudo
> device; that is a virtual device pointing to the HBA itself, of which
> we only every will have _one_.

What you write is true. However, I am just suggesting a method to keep 
the core code clean, even if we are not sticking to the tenet of queue 
depths only applying to real sdevs.

> 
>> If we want to optimise budget code handling, then I think that it is 
>> worth doing later. The whole budget map and cmd_per_lun handling is 
>> murky IMHO.
>>
> Oh, most definitely. budget map / cmd_per_lun was okay for single queue
> devices where we couldn't really tell how many commands we should send.
> But for multiqueue we know exactly how many commands should be possible,
> so for _real_ multiqueue devices cmd_per_lun is pretty pointless if not
> downright detrimental.
> _Unless_ we have a multiqueue device with a shared host tagset, then it
> sort of would make sense. But then one wonders why sbitmap doesn't do
> that internally.
> 
> Sounds like a fun topic for ALPSS.
> 
>>>
>>> Do not perform queue depth ramp up / ramp down for pseudo SCSI devices.
>>
>> Are we even ever going to try ramp up or down for the pseudo sdev?
>>
> No. Why would you want to restrict the number of eg TMFs you could send?

My question was really theory vs practice, that being - are we going to 
see "queue full" errors for reserved commands which will lead us to 
entering the queue depth ramping? The answer is below...

> 
>> I suppose we could see it if there is some internal reserved command 
>> error, right?
>>
> That would be equivalent to the HBA rejecting TMFs with 'out of 
> resources'. At which point I would declare the HBA hosed and do
> a full host/PCI reset.

Sure, maybe that is the best way to go.

Thanks,
John

Re: [PATCH v4 05/29] scsi: core: Introduce .queue_reserved_command()

[John Garry]

On 12/09/2025 19:21, Bart Van Assche wrote:
> From: John Garry <john.garry@huawei.com>
> 
> Reserved commands will be used by SCSI LLDs for submitting internal
> commands. Since the SCSI host, target and device limits do not apply to
> the reserved command use cases, bypass the SCSI host limit checks for
> reserved commands. Introduce the .queue_reserved_command() callback for
> reserved commands. Additionally, do not activate the SCSI error handler

This has changed appreciably since I originally authored, so:

Reviewed-by: John Garry <john.g.garry@oracle.com>

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[Bart Van Assche]

On 9/16/25 2:09 AM, John Garry wrote:
> On 12/09/2025 19:21, Bart Van Assche wrote:
>> Make the @cmd argument optional. Add .init_cmd() and .copy_result()
>> callbacks in struct scsi_exec_args. Support allocating from a specific
>> hardware queue. This patch prepares for submitting reserved commands
>> with scsi_execute_cmd().
> 
> A comment: I did suggest trying to reuse scsi_execute_cmd() for this. 
> However considering the code changes and the comments, below. Maybe it 
> is not the best thing. Let's further investigate...

I followed your suggestion because I think it results in cleaner code.
This approach allowed me to drop the patch that introduces the
scsi_get_internal_cmd() and scsi_put_internal_cmd() functions. This
approach also eliminates some code duplication in the scsi_execute_cmd()
callers introduced by this patch series.

>>   retry:
>> -    req = scsi_alloc_request(sdev->request_queue, opf, args->req_flags);
>> +    req = args->specify_hctx ?
> 
> Can you check args->hctx_idx is a specific queue or something like 
> NVME_QID_ANY?

That would require explicit initialization of the .hctx_idx member by
all callers that don't care about which hardware queue a command is
allocated from. I think the current approach (only code that cares about
the hardware queue a command is allocated from has to specify
information related to the hardware queue) is more user friendly
because scsi_execute_cmd() callers won't forget by accident to set
.hctx_idx to e.g. ANY_HCTX.

>> +        scsi_alloc_request_hctx(sdev->request_queue, opf,
>> +                    args->req_flags, args->hctx_idx) :
> 
> did you consider passing this hctx info to scsi_alloc_request() and 
> allow scsi_alloc_request() contain the logic as to call 
> blk_mq_alloc_request_hctx() or blk_mq_alloc_request()?

I'm open to this but I'm not sure this would be an improvement because
there is only one scsi_alloc_request_hctx() caller and there are nine
scsi_alloc_request() calls. An argument like ANY_HCTX would have to be
added to all nine calls. On the other hand, the code duplication that is
the result of the current approach is minimal.

>> @@ -318,8 +321,12 @@ int scsi_execute_cmd(struct scsi_device *sdev, 
>> const unsigned char *cmd,
>>               goto out;
>>       }
>>       scmd = blk_mq_rq_to_pdu(req);
>> -    scmd->cmd_len = COMMAND_SIZE(cmd[0]);
>> -    memcpy(scmd->cmnd, cmd, scmd->cmd_len);
> 
> 
>> +    if (cmd) {
>> +        scmd->cmd_len = COMMAND_SIZE(cmd[0]);
>> +        memcpy(scmd->cmnd, cmd, scmd->cmd_len);
>> +    }
> 
> you could just pass a dummy cmd instead of doing this

Sure, that's possible, but it would make some scsi_execute_cmd() calls 
really confusing. Making the callers pass a CDB that is never used would
make the reader of the callers wonder why e.g. a TEST UNIT READY CDB is
passed to a scsi_execute_cmd() call that submits something that is not a
SCSI command.

>> +    if (args->init_cmd)
>> +        args->init_cmd(scmd, args);
> 
> is it possible to do this in ufshcd_init_cmd_priv? Or too late?
> 
> We could have a "is reserved command" check there (in 
> ufshcd_init_cmd_priv), and do whatever processing is needed which is 
> done in ufshcd_init_dev_cmd

This is not possible because the 'args' pointer is not passed to
ufshcd_init_cmd_priv(). See also the conversions of the 'arg' pointer
into a pointer to the surrounding data structure in the .init_cmd
functions in patch 29/29. Maybe I should rename .init_cmd into
.setup_cmd because all functions in the SCSI disk driver that have a
similar role have "_setup_" in their function name.

>>       scmd->allowed = ml_retries;
>>       scmd->flags |= args->scmd_flags;
>>       req->timeout = timeout;
>> @@ -353,6 +360,9 @@ int scsi_execute_cmd(struct scsi_device *sdev, 
>> const unsigned char *cmd,
>>                        args->sshdr);
>>       ret = scmd->result;
>> +    if (ret == 0 && args->copy_result)
>> +        args->copy_result(scmd, args);
> 
> can this sort of thing be done in the LLD completion handler?
Only if the 'args' pointer would be stored in the SCSI command private
data. Do you perhaps prefer that the 'args' pointer would be stored in
the SCSI command private data instead of adding a .copy_result function
call in scsi_execute_cmd()? This approach is more risky because it may
result in scsi_execute_cmd() callers forgetting to clear the 'args'
pointer in the SCSI command private data after scsi_execute_cmd() has
finished.

Thanks,

Bart.

Re: [PATCH v4 03/29] scsi: core: Make the budget map optional

[Bart Van Assche]

On 9/16/25 1:34 AM, Hannes Reinecke wrote:
> On 9/12/25 20:21, Bart Van Assche wrote:
>> +    if (!sdev->budget_map.map)
>> +        return INT_MAX;
> 
> Strictly speaking it's not related to reserved commands, but rather to 
> cmd_per_lun. Wouldn't it be better to introduce a way to disable 
> cmd_per_lun (eg by setting it to INT_MAX or somesuch), and then disable
> the budget map when cmd_per_lun is disabled?

I will look into this.

Thanks,

Bart.

Re: [PATCH v4 03/29] scsi: core: Make the budget map optional

[Bart Van Assche]

On 9/16/25 1:34 AM, Hannes Reinecke wrote:
> On 9/12/25 20:21, Bart Van Assche wrote:
>> Prepare for not allocating a budget map for pseudo SCSI devices by
>> checking whether a budget map has been allocated before using a budget
>> map.
>
> Strictly speaking it's not related to reserved commands, but rather to 
> cmd_per_lun. Wouldn't it be better to introduce a way to disable 
> cmd_per_lun (eg by setting it to INT_MAX or somesuch), and then disable
> the budget map when cmd_per_lun is disabled?
> 
> Other than that I really like the idea of being able to disable the
> budget map. I always wondered if we won't be better off with dropping
> the budget map for HBAs with a shared host tagset.

Hi Hannes,

There is a cmd_per_lun member in struct scsi_host_template and also in
struct Scsi_Host but not in struct scsi_device. I think we need a
mechanism per SCSI device rather than a host-wide mechanism.

Although adding a new member in struct scsi_device is easy, I prefer to
add a new function that checks whether or not a budget map is necessary.
Even if a new member would be added in struct scsi_device, such a new 
function would be necessary anyway to decide what value to store in that
new structure member.

Thanks,

Bart.

Re: [PATCH v4 07/29] scsi_debug: Allocate a pseudo SCSI device

[John Garry]

On 12/09/2025 19:21, Bart Van Assche wrote:
> Make sure that the code for allocating a pseudo SCSI device gets triggered
> while running blktests.
> 
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>   drivers/scsi/scsi_debug.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
> index 2a8638937d23..3f7884025d19 100644
> --- a/drivers/scsi/scsi_debug.c
> +++ b/drivers/scsi/scsi_debug.c
> @@ -9197,6 +9197,19 @@ static int sdebug_fail_cmd(struct scsi_cmnd *cmnd, int *retval,
>   	return ret;
>   }
>   
> +/*

Hi Bart,

Could you consider integrating the following patch into this series? I 
think maybe some ideas can be applied to the UFS driver. I made this 
change on top of your series. My motivation is that I would like to be 
able to test reserved commands handling.

Thanks.

-------8<------

 From 67bff76aacaffcf8f22c634d5b8afaee8e705424 Mon Sep 17 00:00:00 2001
From: John Garry <john.g.garry@oracle.com>
Date: Wed, 17 Sep 2025 12:58:10 +0100
Subject: [PATCH] scsi_debug: really handle reserved commands

Use the scsi reserved command handling for issuing aborts.

This allows developers without specific HW access to test reserved
commands handling.

In reality, the only thing which we are doing is changing the context
in which we abort commands. But at least we are trying to emulate real
HW handling for such a scenario.

Other users of scsi_debug_abort_cmnd() can we switched over eventually.

Signed-off-by: John Garry <john.g.garry@oracle.com>

diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
index 3f7884025d19..1c5d0b012c65 100644
--- a/drivers/scsi/scsi_debug.c
+++ b/drivers/scsi/scsi_debug.c
@@ -6729,20 +6729,53 @@ static bool scsi_debug_stop_cmnd(struct 
scsi_cmnd *cmnd)
  	return false;
  }

+struct scsi_debug_abort_cmd {
+	u16 tag;
+	u16 hwq;
+};
+
+enum {
+	SCSI_DEBUG_ABORT_CMD,
+};
+
+struct scsi_debug_reserved_data {
+	unsigned int type;
+
+	union {
+		struct scsi_debug_abort_cmd abort_cmd;
+	};
+};
+
  /*
   * Called from scsi_debug_abort() only, which is for timed-out cmd.
   */
  static bool scsi_debug_abort_cmnd(struct scsi_cmnd *cmnd)
  {
-	struct sdebug_scsi_cmd *sdsc = scsi_cmd_priv(cmnd);
-	unsigned long flags;
-	bool res;
+	struct request *rq = scsi_cmd_to_rq(cmnd);
+	u32 unique_tag = blk_mq_unique_tag(rq);
+	u16 hwq = blk_mq_unique_tag_to_hwq(unique_tag);
+	u16 tag = blk_mq_unique_tag_to_tag(unique_tag);
+	struct scsi_device *sdev = cmnd->device;
+	struct Scsi_Host *shost = sdev->host;
+	int err;
+	struct scsi_exec_args args = {
+		.req_flags = BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT,
+	};
+	struct scsi_debug_reserved_data data = {
+		.type = SCSI_DEBUG_ABORT_CMD,
+		.abort_cmd = {
+			.tag = tag,
+			.hwq = hwq,
+		},
+	};

-	spin_lock_irqsave(&sdsc->lock, flags);
-	res = scsi_debug_stop_cmnd(cmnd);
-	spin_unlock_irqrestore(&sdsc->lock, flags);
+	err = scsi_execute_cmd(shost->pseudo_sdev, NULL, REQ_OP_DRV_OUT,
+			&data, sizeof(data), 3000, 0, &args);

-	return res;
+	if (err)
+		return false;
+	return true;
  }

  /*
@@ -9197,14 +9230,41 @@ static int sdebug_fail_cmd(struct scsi_cmnd 
*cmnd, int *retval,
  	return ret;
  }

-/*
- * The only purpose of this function is to make the SCSI core allocate a
- * pseudo SCSI device.
- */
  static int scsi_debug_queue_reserved_command(struct Scsi_Host *shost,
  					     struct scsi_cmnd *scp)
  {
-	WARN_ON_ONCE(true);
+	struct request *rq = scsi_cmd_to_rq(scp);
+	struct scsi_debug_reserved_data *data = bio_data(rq->bio);
+
+	switch (data->type) {
+	case SCSI_DEBUG_ABORT_CMD:
+		struct scsi_debug_abort_cmd *abort_cmd = &data->abort_cmd;
+		struct blk_mq_tag_set *tag_set = &shost->tag_set;
+		unsigned int tag = abort_cmd->tag;
+		unsigned int hwq = abort_cmd->hwq;
+		struct scsi_cmnd *abort_scmd;
+		struct sdebug_scsi_cmd *sdsc;
+		struct request *abort_rq;
+		struct blk_mq_tags *tags;
+		unsigned long flags;
+		bool res;
+
+		tags = tag_set->tags[hwq];
+		abort_rq = blk_mq_tag_to_rq(tags, tag);
+		abort_scmd = blk_mq_rq_to_pdu(abort_rq);
+		sdsc = scsi_cmd_priv(abort_scmd);
+		spin_lock_irqsave(&sdsc->lock, flags);
+		res = scsi_debug_stop_cmnd(abort_scmd);
+		spin_unlock_irqrestore(&sdsc->lock, flags);
+		if (res == true) {
+			scp->result = DID_OK << 16;
+			scsi_done(scp);
+			return 0;
+		}
+		fallthrough;
+		default:
+	}
+
  	scp->result = DID_ERROR << 16;
  	scsi_done(scp);
  	return 0;
@@ -9451,6 +9511,7 @@ static const struct scsi_host_template 
sdebug_driver_template = {
  	.init_cmd_priv = sdebug_init_cmd_priv,
  	.target_alloc =		sdebug_target_alloc,
  	.target_destroy =	sdebug_target_destroy,
+	.nr_reserved_cmds =	1,
  };

  static int sdebug_driver_probe(struct device *dev)
-- 
2.43.0

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[John Garry]

>>>   retry:
>>> -    req = scsi_alloc_request(sdev->request_queue, opf, args- 
>>> >req_flags);
>>> +    req = args->specify_hctx ?
>>
>> Can you check args->hctx_idx is a specific queue or something like 
>> NVME_QID_ANY?
> 
> That would require explicit initialization of the .hctx_idx member by
> all callers that don't care about which hardware queue a command is
> allocated from.

Yeah, doing that is not reliable. You could also add hctx_idx as 
a(nother) new argument to scsi_execute_cmd()

> I think the current approach (only code that cares about
> the hardware queue a command is allocated from has to specify
> information related to the hardware queue) is more user friendly
> because scsi_execute_cmd() callers won't forget by accident to set
> .hctx_idx to e.g. ANY_HCTX.

understood

> 
>>> +        scsi_alloc_request_hctx(sdev->request_queue, opf,
>>> +                    args->req_flags, args->hctx_idx) :
>>
>> did you consider passing this hctx info to scsi_alloc_request() and 
>> allow scsi_alloc_request() contain the logic as to call 
>> blk_mq_alloc_request_hctx() or blk_mq_alloc_request()?
> 
...

> 
>>> @@ -318,8 +321,12 @@ int scsi_execute_cmd(struct scsi_device *sdev, 
>>> const unsigned char *cmd,
>>>               goto out;
>>>       }
>>>       scmd = blk_mq_rq_to_pdu(req);
>>> -    scmd->cmd_len = COMMAND_SIZE(cmd[0]);
>>> -    memcpy(scmd->cmnd, cmd, scmd->cmd_len);
>>
>>
>>> +    if (cmd) {
>>> +        scmd->cmd_len = COMMAND_SIZE(cmd[0]);
>>> +        memcpy(scmd->cmnd, cmd, scmd->cmd_len);
>>> +    }
>>
>> you could just pass a dummy cmd instead of doing this
> 
> Sure, that's possible, but it would make some scsi_execute_cmd() calls 
> really confusing. Making the callers pass a CDB that is never used would
> make the reader of the callers wonder why e.g. a TEST UNIT READY CDB is
> passed to a scsi_execute_cmd() call that submits something that is not a
> SCSI command.

eh, don't we have space for vendor commands which could be used (instead 
of something like TEST UNIT READY)?

> 
>>> +    if (args->init_cmd)
>>> +        args->init_cmd(scmd, args);
>>
>> is it possible to do this in ufshcd_init_cmd_priv? Or too late?
>>
>> We could have a "is reserved command" check there (in 
>> ufshcd_init_cmd_priv), and do whatever processing is needed which is 
>> done in ufshcd_init_dev_cmd
> 
> This is not possible because the 'args' pointer is not passed to
> ufshcd_init_cmd_priv(). See also the conversions of the 'arg' pointer
> into a pointer to the surrounding data structure in the .init_cmd
> functions in patch 29/29. Maybe I should rename .init_cmd into
> .setup_cmd because all functions in the SCSI disk driver that have a
> similar role have "_setup_" in their function name.

I thought that you could pass the data like how I done it in the 
scsi_debug change which I proposed.

> 
>>>       scmd->allowed = ml_retries;
>>>       scmd->flags |= args->scmd_flags;
>>>       req->timeout = timeout;
>>> @@ -353,6 +360,9 @@ int scsi_execute_cmd(struct scsi_device *sdev, 
>>> const unsigned char *cmd,
>>>                        args->sshdr);
>>>       ret = scmd->result;
>>> +    if (ret == 0 && args->copy_result)
>>> +        args->copy_result(scmd, args);
>>
>> can this sort of thing be done in the LLD completion handler?
> Only if the 'args' pointer would be stored in the SCSI command private
> data. Do you perhaps prefer that the 'args' pointer would be stored in
> the SCSI command private data instead of adding a .copy_result function
> call in scsi_execute_cmd()? 

I don't think that passing args around is a good idea. We only have args 
as it minimizes the args to scsi_execute_cmd()

> This approach is more risky because it may
> result in scsi_execute_cmd() callers forgetting to clear the 'args'
> pointer in the SCSI command private data after scsi_execute_cmd() has
> finished.

Is there anywhere else where the result can be stored and passed back?

I know that it is not ideal, but could we use scsi_execute_cmd() @buffer 
arg for both in and out data?

Thanks,
John

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[Bart Van Assche]

On 9/17/25 6:08 AM, John Garry wrote:
> I know that it is not ideal, but could we use scsi_execute_cmd() @buffer 
> arg for both in and out data?

The scsi_execute_cmd() buffer and bufflen arguments are passed to
blk_rq_map_kern(). blk_rq_map_kern() shouldn't be called in this case
because:
- Calling blk_rq_map_kern() is only necessary for data that will be
   transferred with DMA. The 'args' data won't be transferred by DMA.
- The 'args' data structure is typically allocated on the stack.
   blk_rq_map_kern() copies data that has been allocated on the stack.

Thanks,

Bart.

Re: [PATCH v4 07/29] scsi_debug: Allocate a pseudo SCSI device

[Bart Van Assche]

On 9/17/25 5:09 AM, John Garry wrote:
> On 12/09/2025 19:21, Bart Van Assche wrote:
>> Make sure that the code for allocating a pseudo SCSI device gets 
>> triggered
>> while running blktests.
>>
>> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
>> ---
>>   drivers/scsi/scsi_debug.c | 14 ++++++++++++++
>>   1 file changed, 14 insertions(+)
>>
>> diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
>> index 2a8638937d23..3f7884025d19 100644
>> --- a/drivers/scsi/scsi_debug.c
>> +++ b/drivers/scsi/scsi_debug.c
>> @@ -9197,6 +9197,19 @@ static int sdebug_fail_cmd(struct scsi_cmnd 
>> *cmnd, int *retval,
>>       return ret;
>>   }
>> +/*
> 
> Hi Bart,
> 
> Could you consider integrating the following patch into this series? I 
> think maybe some ideas can be applied to the UFS driver. I made this 
> change on top of your series. My motivation is that I would like to be 
> able to test reserved commands handling.

Thanks for the patch! I will look into combining this patch with "[PATCH
v4 07/29] scsi_debug: Allocate a pseudo SCSI device" unless if you
request me to keep this patch as a separate patch.

Bart.

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[Bart Van Assche]

On 9/17/25 11:21 AM, Bart Van Assche wrote:
> On 9/17/25 6:08 AM, John Garry wrote:
>> I know that it is not ideal, but could we use scsi_execute_cmd() 
>> @buffer arg for both in and out data?
> 
> The scsi_execute_cmd() buffer and bufflen arguments are passed to
> blk_rq_map_kern(). blk_rq_map_kern() shouldn't be called in this case
> because:
> - Calling blk_rq_map_kern() is only necessary for data that will be
>    transferred with DMA. The 'args' data won't be transferred by DMA.
> - The 'args' data structure is typically allocated on the stack.
>    blk_rq_map_kern() copies data that has been allocated on the stack.

There is another concern: passing the scsi_exec_args pointer via the
scsi_execute_cmd() @buffer argument makes it impossible to pass both a
scsi_exec_args pointer and a data buffer to scsi_execute_cmd(). This
functionality is not required by this patch series but may be required
by other drivers than the UFS driver.

Thanks,

Bart.

Re: [PATCH v4 07/29] scsi_debug: Allocate a pseudo SCSI device

[John Garry]

On 17/09/2025 22:37, Bart Van Assche wrote:
>> Could you consider integrating the following patch into this series? I 
>> think maybe some ideas can be applied to the UFS driver. I made this 
>> change on top of your series. My motivation is that I would like to be 
>> able to test reserved commands handling.
> 
> Thanks for the patch! I will look into combining this patch with "[PATCH
> v4 07/29] scsi_debug: Allocate a pseudo SCSI device" unless if you
> request me to keep this patch as a separate patch.

I think that it only makes sense to combine. Indeed, my change 
essentially overwrites what was in the original patch.

Thanks,
John

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[John Garry]

On 18/09/2025 00:42, Bart Van Assche wrote:
> On 9/17/25 11:21 AM, Bart Van Assche wrote:
>> On 9/17/25 6:08 AM, John Garry wrote:
>>> I know that it is not ideal, but could we use scsi_execute_cmd() 
>>> @buffer arg for both in and out data?
>>
>> The scsi_execute_cmd() buffer and bufflen arguments are passed to
>> blk_rq_map_kern(). blk_rq_map_kern() shouldn't be called in this case
>> because:
>> - Calling blk_rq_map_kern() is only necessary for data that will be
>>    transferred with DMA. The 'args' data won't be transferred by DMA.

I am just trying to reuse what we already have without extending 
scsi_execute_cmd() further

>> - The 'args' data structure is typically allocated on the stack.
>>    blk_rq_map_kern() copies data that has been allocated on the stack.
> 

what's the problem with that? It's not like we care about performance in 
this case.

> There is another concern: passing the scsi_exec_args pointer via the
> scsi_execute_cmd() @buffer argument makes it impossible to pass both a
> scsi_exec_args pointer and a data buffer to scsi_execute_cmd().

I would not suggest to put any pointers in the data buffer - just copy 
in or out the data which you require, like the example which I had for 
scsi_debug.

> This
> functionality is not required by this patch series but may be required
> by other drivers than the UFS driver.

Thanks,
John

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[Bart Van Assche]

On 9/18/25 1:01 AM, John Garry wrote:
> On 18/09/2025 00:42, Bart Van Assche wrote:
>> There is another concern: passing the scsi_exec_args pointer via the
>> scsi_execute_cmd() @buffer argument makes it impossible to pass both a
>> scsi_exec_args pointer and a data buffer to scsi_execute_cmd().
> 
> I would not suggest to put any pointers in the data buffer - just copy 
> in or out the data which you require, like the example which I had for 
> scsi_debug.

Hi John,

In patch 29/29 of this series several pointers are present in the data
structures in which struct scsi_exec_args is embedded.

It seems like the scsi_execute_cmd() changes in this patch series are
controversial. How about dropping these changes from this patch series
and restoring the approach of v3 of this patch series? That means
calling blk_execute_rq() instead of scsi_execute_cmd().

Thanks,

Bart.

Re: [PATCH v4 06/29] scsi: core: Extend the scsi_execute_cmd() functionality

[John Garry]

On 18/09/2025 20:49, Bart Van Assche wrote:
>>
>> I would not suggest to put any pointers in the data buffer - just copy 
>> in or out the data which you require, like the example which I had for 
>> scsi_debug.
> 
> Hi John,
> 
> In patch 29/29 of this series several pointers are present in the data
> structures in which struct scsi_exec_args is embedded.
> 
> It seems like the scsi_execute_cmd() changes in this patch series are
> controversial. How about dropping these changes from this patch series
> and restoring the approach of v3 of this patch series? That means
> calling blk_execute_rq() instead of scsi_execute_cmd().

I thought that we could make this work for scsi_execute_cmd(), but I 
have no real problem with reverting to having another function which 
calls blk_execute_rq() just for reserved commands.