[PATCH v2 0/3] Improve requeuing behavior

[PATCH v2 0/3] Improve requeuing behavior

[Bart Van Assche]

Hello Martin,

The conclusion of a recent discussion is that .jiffies_at_alloc and .retries
should be set once at the start of a lifetime of a SCSI request instead of
every time a request is requeued. This patch series realizes that. Please
consider these patches for kernel v4.14.

Thanks,

Bart.

Bart Van Assche (3):
  Call scsi_initialize_rq() also for filesystem requests
  Improve requeuing behavior
  Show .retries and .jiffies_at_alloc in debugfs

 drivers/scsi/scsi_debugfs.c |  3 ++-
 drivers/scsi/scsi_lib.c     | 41 +++++++++++++++++++++++++++++++++++------
 include/scsi/scsi_cmnd.h    |  3 +++
 3 files changed, 40 insertions(+), 7 deletions(-)

-- 
2.14.1

[PATCH v2 1/3] Call scsi_initialize_rq() also for filesystem requests

[Bart Van Assche]

If a pass-through request is submitted then blk_get_request()
initializes that request by calling scsi_initialize_rq(). Also
call this function for filesystem requests. Introduce
CMD_INITIALIZED to keep track of whether or not a request has
already been initialized.

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Hannes Reinecke <hare@suse.com>
Cc: Johannes Thumshirn <jthumshirn@suse.de>
---
 drivers/scsi/scsi_lib.c  | 28 ++++++++++++++++++++++++----
 include/scsi/scsi_cmnd.h |  3 +++
 2 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 938a7e398cd4..170a725e33ce 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -642,6 +642,11 @@ static bool scsi_end_request(struct request *req, blk_status_t error,
 	if (blk_queue_add_random(q))
 		add_disk_randomness(req->rq_disk);
 
+	if (!blk_rq_is_scsi(req)) {
+		WARN_ON_ONCE(!(cmd->flags & SCMD_INITIALIZED));
+		cmd->flags &= ~SCMD_INITIALIZED;
+	}
+
 	if (req->mq_ctx) {
 		/*
 		 * In the MQ case the command gets freed by __blk_mq_end_request,
@@ -1110,7 +1115,8 @@ EXPORT_SYMBOL(scsi_init_io);
  * scsi_initialize_rq - initialize struct scsi_cmnd.req
  * @rq: Request associated with the SCSI command to be initialized.
  *
- * Called from inside blk_get_request().
+ * Called from inside blk_get_request() for pass-through requests and from
+ * inside scsi_init_command() for filesystem requests.
  */
 void scsi_initialize_rq(struct request *rq)
 {
@@ -1154,7 +1160,13 @@ void scsi_init_command(struct scsi_device *dev, struct scsi_cmnd *cmd)
 {
 	void *buf = cmd->sense_buffer;
 	void *prot = cmd->prot_sdb;
-	unsigned int unchecked_isa_dma = cmd->flags & SCMD_UNCHECKED_ISA_DMA;
+	struct request *rq = blk_mq_rq_from_pdu(cmd);
+	unsigned int flags = cmd->flags & SCMD_PRESERVED_FLAGS;
+
+	if (!blk_rq_is_scsi(rq) && !(flags & SCMD_INITIALIZED)) {
+		flags |= SCMD_INITIALIZED;
+		scsi_initialize_rq(rq);
+	}
 
 	/* zero out the cmd, except for the embedded scsi_request */
 	memset((char *)cmd + sizeof(cmd->req), 0,
@@ -1163,7 +1175,7 @@ void scsi_init_command(struct scsi_device *dev, struct scsi_cmnd *cmd)
 	cmd->device = dev;
 	cmd->sense_buffer = buf;
 	cmd->prot_sdb = prot;
-	cmd->flags = unchecked_isa_dma;
+	cmd->flags = flags;
 	INIT_DELAYED_WORK(&cmd->abort_work, scmd_eh_abort_handler);
 	cmd->jiffies_at_alloc = jiffies;
 
@@ -1350,6 +1362,8 @@ static int scsi_prep_fn(struct request_queue *q, struct request *req)
 
 	ret = scsi_setup_cmnd(sdev, req);
 out:
+	if (ret != BLKPREP_OK)
+		cmd->flags &= ~SCMD_INITIALIZED;
 	return scsi_prep_return(q, req, ret);
 }
 
@@ -1869,6 +1883,7 @@ static int scsi_mq_prep_fn(struct request *req)
 	struct scsi_device *sdev = req->q->queuedata;
 	struct Scsi_Host *shost = sdev->host;
 	struct scatterlist *sg;
+	int ret;
 
 	scsi_init_command(sdev, cmd);
 
@@ -1902,7 +1917,10 @@ static int scsi_mq_prep_fn(struct request *req)
 
 	blk_mq_start_request(req);
 
-	return scsi_setup_cmnd(sdev, req);
+	ret = scsi_setup_cmnd(sdev, req);
+	if (ret != BLK_STS_OK)
+		cmd->flags &= ~SCMD_INITIALIZED;
+	return ret;
 }
 
 static void scsi_mq_done(struct scsi_cmnd *cmd)
@@ -1938,6 +1956,8 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 		goto out_dec_target_busy;
 
 	if (!(req->rq_flags & RQF_DONTPREP)) {
+		if (!blk_rq_is_scsi(req))
+			WARN_ON_ONCE(cmd->flags & SCMD_INITIALIZED);
 		ret = prep_to_mq(scsi_mq_prep_fn(req));
 		if (ret != BLK_STS_OK)
 			goto out_dec_host_busy;
diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h
index f5afcff8d76f..a9f8f7e79d83 100644
--- a/include/scsi/scsi_cmnd.h
+++ b/include/scsi/scsi_cmnd.h
@@ -57,6 +57,9 @@ struct scsi_pointer {
 /* for scmd->flags */
 #define SCMD_TAGGED		(1 << 0)
 #define SCMD_UNCHECKED_ISA_DMA	(1 << 1)
+#define SCMD_INITIALIZED	(1 << 3)
+/* flags preserved across unprep / reprep */
+#define SCMD_PRESERVED_FLAGS	(SCMD_UNCHECKED_ISA_DMA | SCMD_INITIALIZED)
 
 struct scsi_cmnd {
 	struct scsi_request req;
-- 
2.14.1

Re: [PATCH v2 1/3] Call scsi_initialize_rq() also for filesystem requests

[Christoph Hellwig]

Looks good except for the subject, which will need a little update:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH v2 1/3] Call scsi_initialize_rq() also for filesystem requests

[Bart Van Assche]

On Wed, 2017-08-30 at 10:29 +0200, Christoph Hellwig wrote:
> Looks good except for the subject, which will need a little update:

Hello Christoph,

Thanks for the review. Sorry but I don't see what's wrong with the subject.
Can you clarify your comment?

Bart.

Re: [PATCH v2 1/3] Call scsi_initialize_rq() also for filesystem requests

[hch]

On Wed, Aug 30, 2017 at 09:13:00PM +0000, Bart Van Assche wrote:
> On Wed, 2017-08-30 at 10:29 +0200, Christoph Hellwig wrote:
> > Looks good except for the subject, which will need a little update:
> 
> Hello Christoph,
> 
> Thanks for the review. Sorry but I don't see what's wrong with the subject.
> Can you clarify your comment?

Hi Bart,

I'll take that comment back - I misread scsi_initialize_rq as
.initialize_rq.

Re: [PATCH v2 1/3] Call scsi_initialize_rq() also for filesystem requests

[Brian King]

On 08/29/2017 07:07 PM, Bart Van Assche wrote:
> If a pass-through request is submitted then blk_get_request()
> initializes that request by calling scsi_initialize_rq(). Also
> call this function for filesystem requests. Introduce
> CMD_INITIALIZED to keep track of whether or not a request has
> already been initialized.

Bart,

I applied these patches and was able to then re-apply your now reverted patch:

270065e92 - scsi-mq: Always unprepare before requeuing a request

With this combination, I was still able to boot successfully on
a Power system with ipr.

Tested-by: Brian King <brking@linux.vnet.ibm.com>

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH v2 1/3] Call scsi_initialize_rq() also for filesystem requests

[Bart Van Assche]

On Wed, 2017-08-30 at 17:14 -0500, Brian King wrote:
> On 08/29/2017 07:07 PM, Bart Van Assche wrote:
> > If a pass-through request is submitted then blk_get_request()
> > initializes that request by calling scsi_initialize_rq(). Also
> > call this function for filesystem requests. Introduce
> > CMD_INITIALIZED to keep track of whether or not a request has
> > already been initialized.
> 
> I applied these patches and was able to then re-apply your now reverted patch:
> 
> 270065e92 - scsi-mq: Always unprepare before requeuing a request
> 
> With this combination, I was still able to boot successfully on
> a Power system with ipr.

Hello Brian,

Thanks for testing! Sorry but a few minutes ago I found a bug in this
patch. I will repost this patch series.

Bart.

[PATCH v2 2/3] Improve requeuing behavior

[Bart Van Assche]

Requests are unprepared and reprepared when being requeued.
Avoid that requeuing resets .jiffies_at_alloc and .retries by
initializing these two member variables from inside
scsi_initialize_rq() and by preserving both member variables
when preparing a request. This patch affects the requeuing
behavior of both the legacy scsi and the scsi-mq code paths.

Reported-by: Brian King <brking@linux.vnet.ibm.com>
References: https://lkml.org/lkml/2017/8/18/923 ("Re: [BUG][bisected 270065e] linux-next fails to boot on powerpc")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Brian King <brking@linux.vnet.ibm.com>
Cc: Hannes Reinecke <hare@suse.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Johannes Thumshirn <jthumshirn@suse.de>
---
 drivers/scsi/scsi_lib.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 170a725e33ce..60fff8468620 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1112,9 +1112,13 @@ int scsi_init_io(struct scsi_cmnd *cmd)
 EXPORT_SYMBOL(scsi_init_io);
 
 /**
- * scsi_initialize_rq - initialize struct scsi_cmnd.req
+ * scsi_initialize_rq - initialize struct scsi_cmnd partially
  * @rq: Request associated with the SCSI command to be initialized.
  *
+ * This function initializes the members of struct scsi_cmnd that must be
+ * initialized before request processing starts and that won't be
+ * reinitialized if a SCSI command is requeued.
+ *
  * Called from inside blk_get_request() for pass-through requests and from
  * inside scsi_init_command() for filesystem requests.
  */
@@ -1123,6 +1127,8 @@ void scsi_initialize_rq(struct request *rq)
 	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
 
 	scsi_req_init(&cmd->req);
+	cmd->jiffies_at_alloc = jiffies;
+	cmd->retries = 0;
 }
 EXPORT_SYMBOL(scsi_initialize_rq);
 
@@ -1162,6 +1168,8 @@ void scsi_init_command(struct scsi_device *dev, struct scsi_cmnd *cmd)
 	void *prot = cmd->prot_sdb;
 	struct request *rq = blk_mq_rq_from_pdu(cmd);
 	unsigned int flags = cmd->flags & SCMD_PRESERVED_FLAGS;
+	unsigned long jiffies_at_alloc = cmd->jiffies_at_alloc;
+	int retries = cmd->retries;
 
 	if (!blk_rq_is_scsi(rq) && !(flags & SCMD_INITIALIZED)) {
 		flags |= SCMD_INITIALIZED;
@@ -1177,7 +1185,8 @@ void scsi_init_command(struct scsi_device *dev, struct scsi_cmnd *cmd)
 	cmd->prot_sdb = prot;
 	cmd->flags = flags;
 	INIT_DELAYED_WORK(&cmd->abort_work, scmd_eh_abort_handler);
-	cmd->jiffies_at_alloc = jiffies;
+	cmd->jiffies_at_alloc = jiffies_at_alloc;
+	cmd->retries = retries;
 
 	scsi_add_cmd_to_list(cmd);
 }
-- 
2.14.1

Re: [PATCH v2 2/3] Improve requeuing behavior

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH v2 3/3] Show .retries and .jiffies_at_alloc in debugfs

[Bart Van Assche]

Make these two member variables available in debugfs such that
their value can be verified by kernel developers. An example of
the new output from a system with CONFIG_HZ=100:

ffff8804a41ed480 {.op=READ, .cmd_flags=META|PRIO, .rq_flags=MQ_INFLIGHT|IO_STAT, .atomic_flags=COMPLETE, .tag=17, .internal_tag=-1, .cmd=Read(10) 28 00 09 01 1f a0 00 00 08 00, .retries=0, .jiffies_at_alloc=-25647}

Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Hannes Reinecke <hare@suse.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Johannes Thumshirn <jthumshirn@suse.de>
---
 drivers/scsi/scsi_debugfs.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_debugfs.c b/drivers/scsi/scsi_debugfs.c
index a97c9507103d..282834935e07 100644
--- a/drivers/scsi/scsi_debugfs.c
+++ b/drivers/scsi/scsi_debugfs.c
@@ -9,5 +9,6 @@ void scsi_show_rq(struct seq_file *m, struct request *rq)
 	char buf[80];
 
 	__scsi_format_command(buf, sizeof(buf), cmd->cmnd, cmd->cmd_len);
-	seq_printf(m, ", .cmd=%s", buf);
+	seq_printf(m, ", .cmd=%s, .retries=%d, .jiffies_at_alloc=%ld", buf,
+		   cmd->retries, cmd->jiffies_at_alloc - jiffies);
 }
-- 
2.14.1

Re: [PATCH v2 3/3] Show .retries and .jiffies_at_alloc in debugfs

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>