[PATCH] scsi-mq: fix hw queue hang caused by timeout

[PATCH] scsi-mq: fix hw queue hang caused by timeout

[Ming Lei]

If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.

This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.

Basically this patch maintains same behavior for this situation
with non-mq mode.

Signed-off-by: Ming Lei <ming.lei@canonical.com>
---
 drivers/scsi/scsi_lib.c |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 3b92c39..dfbc028 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -142,6 +142,8 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, int unbusy)
 	struct scsi_device *device = cmd->device;
 	struct request_queue *q = device->request_queue;
 	unsigned long flags;
+	bool restart = false;
+	bool blocked = !!atomic_read(&device->device_blocked);
 
 	SCSI_LOG_MLQUEUE(1, scmd_printk(KERN_INFO, cmd,
 		"Inserting command %p into mlqueue\n", cmd));
@@ -152,9 +154,14 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, int unbusy)
 	 * Decrement the counters, since these commands are no longer
 	 * active on the host/device.
 	 */
-	if (unbusy)
+	if (unbusy) {
 		scsi_device_unbusy(device);
 
+		/* need to restart hw queue if it was stopped */
+		if (!atomic_read(&device->device_busy) && blocked)
+			restart = true;
+	}
+
 	/*
 	 * Requeue this command.  It will go before all other commands
 	 * that are already in the queue. Schedule requeue work under
@@ -164,6 +171,8 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, int unbusy)
 	cmd->result = 0;
 	if (q->mq_ops) {
 		scsi_mq_requeue_cmd(cmd);
+		if (restart)
+			blk_mq_start_stopped_hw_queues(q, true);
 		return;
 	}
 	spin_lock_irqsave(q->queue_lock, flags);
-- 
1.7.9.5

Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout

[Christoph Hellwig]

On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote:
> If there are two requests or more timed out, the dispatch queue
> is put into stopped state and never be recoverd, and there
> is no such problem in non-mq mode.
> 
> This patch trys to recover the stopped queue when the queue
> becomes unbusy, then the following retries can move on.
> 
> Basically this patch maintains same behavior for this situation
> with non-mq mode.

This looks somewhat similar to the issues that Doug reported, and I remember
when he was last running into boot problems it was timeout related, too.

As far as the implementation is concerned I think the correct fix is
to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout

[Jens Axboe]

On 2014-09-18 10:35, Christoph Hellwig wrote:
> On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote:
>> If there are two requests or more timed out, the dispatch queue
>> is put into stopped state and never be recoverd, and there
>> is no such problem in non-mq mode.
>>
>> This patch trys to recover the stopped queue when the queue
>> becomes unbusy, then the following retries can move on.
>>
>> Basically this patch maintains same behavior for this situation
>> with non-mq mode.
>
> This looks somewhat similar to the issues that Doug reported, and I remember
> when he was last running into boot problems it was timeout related, too.
>
> As far as the implementation is concerned I think the correct fix is
> to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

Since that's the kick part of the requeue, auto-starting the queue for 
that makes a lot of sense. I say that's the way we go.

-- 
Jens Axboe

Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout

[Ming Lei]

On Fri, Sep 19, 2014 at 1:03 AM, Jens Axboe <axboe@fb.com> wrote:
> On 2014-09-18 10:35, Christoph Hellwig wrote:
>>
>> On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote:
>>>
>>> If there are two requests or more timed out, the dispatch queue
>>> is put into stopped state and never be recoverd, and there
>>> is no such problem in non-mq mode.
>>>
>>> This patch trys to recover the stopped queue when the queue
>>> becomes unbusy, then the following retries can move on.
>>>
>>> Basically this patch maintains same behavior for this situation
>>> with non-mq mode.
>>
>>
>> This looks somewhat similar to the issues that Doug reported, and I
>> remember
>> when he was last running into boot problems it was timeout related, too.
>>
>> As far as the implementation is concerned I think the correct fix is
>> to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.
>
>
> Since that's the kick part of the requeue, auto-starting the queue for that
> makes a lot of sense. I say that's the way we go.

Yeah, that looks better.

But it doesn't work after the simple change, and I need to
investigate further.

Thanks,

Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout

[Ming Lei]

On Fri, Sep 19, 2014 at 9:07 PM, Ming Lei <ming.lei@canonical.com> wrote:
> On Fri, Sep 19, 2014 at 1:03 AM, Jens Axboe <axboe@fb.com> wrote:
>> On 2014-09-18 10:35, Christoph Hellwig wrote:
>>>
>>> On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote:
>>>>
>>>> If there are two requests or more timed out, the dispatch queue
>>>> is put into stopped state and never be recoverd, and there
>>>> is no such problem in non-mq mode.
>>>>
>>>> This patch trys to recover the stopped queue when the queue
>>>> becomes unbusy, then the following retries can move on.
>>>>
>>>> Basically this patch maintains same behavior for this situation
>>>> with non-mq mode.
>>>
>>>
>>> This looks somewhat similar to the issues that Doug reported, and I
>>> remember
>>> when he was last running into boot problems it was timeout related, too.
>>>
>>> As far as the implementation is concerned I think the correct fix is
>>> to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.
>>
>>
>> Since that's the kick part of the requeue, auto-starting the queue for that
>> makes a lot of sense. I say that's the way we go.
>
> Yeah, that looks better.
>
> But it doesn't work after the simple change, and I need to
> investigate further.

It is because of the timer miss, now it starts to work.

Thanks,

Re: [PATCH] scsi-mq: fix hw queue hang caused by timeout

[Jens Axboe]

On 09/19/2014 08:18 AM, Ming Lei wrote:
> On Fri, Sep 19, 2014 at 9:07 PM, Ming Lei <ming.lei@canonical.com> wrote:
>> On Fri, Sep 19, 2014 at 1:03 AM, Jens Axboe <axboe@fb.com> wrote:
>>> On 2014-09-18 10:35, Christoph Hellwig wrote:
>>>>
>>>> On Thu, Sep 18, 2014 at 11:59:10PM +0800, Ming Lei wrote:
>>>>>
>>>>> If there are two requests or more timed out, the dispatch queue
>>>>> is put into stopped state and never be recoverd, and there
>>>>> is no such problem in non-mq mode.
>>>>>
>>>>> This patch trys to recover the stopped queue when the queue
>>>>> becomes unbusy, then the following retries can move on.
>>>>>
>>>>> Basically this patch maintains same behavior for this situation
>>>>> with non-mq mode.
>>>>
>>>>
>>>> This looks somewhat similar to the issues that Doug reported, and I
>>>> remember
>>>> when he was last running into boot problems it was timeout related, too.
>>>>
>>>> As far as the implementation is concerned I think the correct fix is
>>>> to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.
>>>
>>>
>>> Since that's the kick part of the requeue, auto-starting the queue for that
>>> makes a lot of sense. I say that's the way we go.
>>
>> Yeah, that looks better.
>>
>> But it doesn't work after the simple change, and I need to
>> investigate further.
> 
> It is because of the timer miss, now it starts to work.

Excellent. I think most new issues should be fixed in for-linus for
inclusion in this round. It's much bigger than I hoped for this late in
the cycle, but lots of us have run a lot of testing, so that's not a
huge worry.

-- 
Jens Axboe