[PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[Ming Lei]

blk_mq_unquiesce_queue() is used for unquiescing the
queue explicitly, so replace blk_mq_start_stopped_hw_queues()
with it.

Cc: linux-nvme@lists.infradead.org
Cc: linux-scsi@vger.kernel.org
Cc: dm-devel@redhat.com
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 drivers/md/dm-rq.c       | 2 +-
 drivers/nvme/host/core.c | 2 +-
 drivers/scsi/scsi_lib.c  | 5 ++++-
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
index b639fa7246ee..ea4029de077f 100644
--- a/drivers/md/dm-rq.c
+++ b/drivers/md/dm-rq.c
@@ -71,7 +71,7 @@ static void dm_old_start_queue(struct request_queue *q)
 
 static void dm_mq_start_queue(struct request_queue *q)
 {
-	blk_mq_start_stopped_hw_queues(q, true);
+	blk_mq_unquiesce_queue(q);
 	blk_mq_kick_requeue_list(q);
 }
 
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index a60926410438..c3f189e54d10 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2526,7 +2526,7 @@ void nvme_start_queues(struct nvme_ctrl *ctrl)
 
 	mutex_lock(&ctrl->namespaces_mutex);
 	list_for_each_entry(ns, &ctrl->namespaces, list) {
-		blk_mq_start_stopped_hw_queues(ns->queue, true);
+		blk_mq_unquiesce_queue(ns->queue);
 		blk_mq_kick_requeue_list(ns->queue);
 	}
 	mutex_unlock(&ctrl->namespaces_mutex);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 99e16ac479e3..ffcf05765e2b 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev,
 		return -EINVAL;
 
 	if (q->mq_ops) {
-		blk_mq_start_stopped_hw_queues(q, false);
+		if (blk_queue_quiesced(q))
+			blk_mq_unquiesce_queue(q);
+		else
+			blk_mq_start_stopped_hw_queues(q, false);
 	} else {
 		spin_lock_irqsave(q->queue_lock, flags);
 		blk_start_queue(q);
-- 
2.9.4

Re: [PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[Bart Van Assche]

On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote:
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 99e16ac479e3..ffcf05765e2b 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev,
>  		return -EINVAL;
>  
>  	if (q->mq_ops) {
> -		blk_mq_start_stopped_hw_queues(q, false);
> +		if (blk_queue_quiesced(q))
> +			blk_mq_unquiesce_queue(q);
> +		else
> +			blk_mq_start_stopped_hw_queues(q, false);
>  	} else {
>  		spin_lock_irqsave(q->queue_lock, flags);
>  		blk_start_queue(q);

As I commented on v2, this change is really wrong. All what's needed here is
a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to
blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to
use the STOPPED flag in the SCSI core to make the block layer core stop calling
.queue_rq() if a SCSI LLD returns "busy".

Bart.

Re: [PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[Ming Lei]

On Wed, May 31, 2017 at 03:21:41PM +0000, Bart Van Assche wrote:
> On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote:
> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > index 99e16ac479e3..ffcf05765e2b 100644
> > --- a/drivers/scsi/scsi_lib.c
> > +++ b/drivers/scsi/scsi_lib.c
> > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev,
> >  		return -EINVAL;
> >  
> >  	if (q->mq_ops) {
> > -		blk_mq_start_stopped_hw_queues(q, false);
> > +		if (blk_queue_quiesced(q))
> > +			blk_mq_unquiesce_queue(q);
> > +		else
> > +			blk_mq_start_stopped_hw_queues(q, false);
> >  	} else {
> >  		spin_lock_irqsave(q->queue_lock, flags);
> >  		blk_start_queue(q);
> 
> As I commented on v2, this change is really wrong. All what's needed here is
> a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to
> blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to
> use the STOPPED flag in the SCSI core to make the block layer core stop calling
> .queue_rq() if a SCSI LLD returns "busy".

I am not sure if I understand your idea, could you explain a bit why it is wrong?

Let's see the function of scsi_internal_device_block():

	if (q->mq_ops) {
                if (wait)
                        blk_mq_quiesce_queue(q);
                else
                        blk_mq_stop_hw_queues(q);
	}

So the queue may be put into quiesced if 'wait' is true, or it is
stopped if 'wait' is false.

And this patch just makes the two SCSI APIs symmetrical.

Since we will not stop queue in blk_mq_quiesce_queue() later,
I have to unquiese one queue only if it is quiesced.

So suppose the queue is put into stopped in scsi_internal_device_block(),
do we expect not to restart it in scsi_internal_device_unblock() via
blk_mq_unquiesce_queue()?

Thanks,
Ming

Re: [PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[Bart Van Assche]

On Thu, 2017-06-01 at 08:54 +0800, Ming Lei wrote:
> On Wed, May 31, 2017 at 03:21:41PM +0000, Bart Van Assche wrote:
> > On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote:
> > > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > > index 99e16ac479e3..ffcf05765e2b 100644
> > > --- a/drivers/scsi/scsi_lib.c
> > > +++ b/drivers/scsi/scsi_lib.c
> > > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev,
> > >  		return -EINVAL;
> > >  
> > >  	if (q->mq_ops) {
> > > -		blk_mq_start_stopped_hw_queues(q, false);
> > > +		if (blk_queue_quiesced(q))
> > > +			blk_mq_unquiesce_queue(q);
> > > +		else
> > > +			blk_mq_start_stopped_hw_queues(q, false);
> > >  	} else {
> > >  		spin_lock_irqsave(q->queue_lock, flags);
> > >  		blk_start_queue(q);
> > 
> > As I commented on v2, this change is really wrong. All what's needed here is
> > a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to
> > blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to
> > use the STOPPED flag in the SCSI core to make the block layer core stop calling
> > .queue_rq() if a SCSI LLD returns "busy".
> 
> I am not sure if I understand your idea, could you explain a bit why it is wrong?
> 
> Let's see the function of scsi_internal_device_block():
> 
> 	if (q->mq_ops) {
>                 if (wait)
>                         blk_mq_quiesce_queue(q);
>                 else
>                         blk_mq_stop_hw_queues(q);
> 	}
> 
> So the queue may be put into quiesced if 'wait' is true, or it is
> stopped if 'wait' is false.
> 
> And this patch just makes the two SCSI APIs symmetrical.
> 
> Since we will not stop queue in blk_mq_quiesce_queue() later,
> I have to unquiese one queue only if it is quiesced.
> 
> So suppose the queue is put into stopped in scsi_internal_device_block(),
> do we expect not to restart it in scsi_internal_device_unblock() via
> blk_mq_unquiesce_queue()?

Hello Ming,

My opinion is that scsi_internal_device_block() and scsi_internal_device_unblock()
should be changed as follows for the scsi-mq code path:
* scsi_internal_device_block() should call blk_mq_quiesce_queue(q) if the "wait"
  argument is true and should set the QUEUE_FLAG_QUIESCED flag if the "wait"
  argument is false.
* scsi_internal_device_unblock() should call blk_mq_unquiesce_queue().

I am aware it sounds weird to set the QUEUE_FLAG_QUIESCED flag without waiting
until ongoing .queue_rq() calls have finished. The only driver that triggers
that code path is the mpt3sas driver. I think it's unfortunate that that driver
has ever been allowed to call scsi_internal_device_block() because it's the only
driver that calls that function from a context where sleeping is not allowed.
No matter whether the scsi_internal_device_block() call from the mpt3sas driver
sets the QUEUE_FLAG_QUIESCED or the BLK_MQ_S_STOPPED flag, I don't think that
will have the effect the authors of this driver intended. Unfortunately I'm not
familiar enough with the mpt3sas driver to fix that driver myself.

Note: these changes conflict with a patch SCSI patch series I started working on
about two months ago. See also https://www.spinics.net/lists/linux-scsi/msg109103.html.

Bart.

Re: [PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[Ming Lei]

On Thu, Jun 01, 2017 at 11:09:00PM +0000, Bart Van Assche wrote:
> On Thu, 2017-06-01 at 08:54 +0800, Ming Lei wrote:
> > On Wed, May 31, 2017 at 03:21:41PM +0000, Bart Van Assche wrote:
> > > On Wed, 2017-05-31 at 20:37 +0800, Ming Lei wrote:
> > > > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > > > index 99e16ac479e3..ffcf05765e2b 100644
> > > > --- a/drivers/scsi/scsi_lib.c
> > > > +++ b/drivers/scsi/scsi_lib.c
> > > > @@ -3031,7 +3031,10 @@ scsi_internal_device_unblock(struct scsi_device *sdev,
> > > >  		return -EINVAL;
> > > >  
> > > >  	if (q->mq_ops) {
> > > > -		blk_mq_start_stopped_hw_queues(q, false);
> > > > +		if (blk_queue_quiesced(q))
> > > > +			blk_mq_unquiesce_queue(q);
> > > > +		else
> > > > +			blk_mq_start_stopped_hw_queues(q, false);
> > > >  	} else {
> > > >  		spin_lock_irqsave(q->queue_lock, flags);
> > > >  		blk_start_queue(q);
> > > 
> > > As I commented on v2, this change is really wrong. All what's needed here is
> > > a call to blk_mq_unquiesce_queue() and nothing else. Adding a call to
> > > blk_mq_start_stopped_hw_queues() is wrong because it makes it impossible to
> > > use the STOPPED flag in the SCSI core to make the block layer core stop calling
> > > .queue_rq() if a SCSI LLD returns "busy".
> > 
> > I am not sure if I understand your idea, could you explain a bit why it is wrong?
> > 
> > Let's see the function of scsi_internal_device_block():
> > 
> > 	if (q->mq_ops) {
> >                 if (wait)
> >                         blk_mq_quiesce_queue(q);
> >                 else
> >                         blk_mq_stop_hw_queues(q);
> > 	}
> > 
> > So the queue may be put into quiesced if 'wait' is true, or it is
> > stopped if 'wait' is false.
> > 
> > And this patch just makes the two SCSI APIs symmetrical.
> > 
> > Since we will not stop queue in blk_mq_quiesce_queue() later,
> > I have to unquiese one queue only if it is quiesced.
> > 
> > So suppose the queue is put into stopped in scsi_internal_device_block(),
> > do we expect not to restart it in scsi_internal_device_unblock() via
> > blk_mq_unquiesce_queue()?
> 
> Hello Ming,
> 
> My opinion is that scsi_internal_device_block() and scsi_internal_device_unblock()
> should be changed as follows for the scsi-mq code path:
> * scsi_internal_device_block() should call blk_mq_quiesce_queue(q) if the "wait"
>   argument is true and should set the QUEUE_FLAG_QUIESCED flag if the "wait"
>   argument is false.
> * scsi_internal_device_unblock() should call blk_mq_unquiesce_queue().
> 
> I am aware it sounds weird to set the QUEUE_FLAG_QUIESCED flag without waiting
> until ongoing .queue_rq() calls have finished. The only driver that triggers

This way may break the contract of blk_mq_quiesce_queue() and is really weird.
And the above change shouldn't be related with this patchset, so better
to do whatever you want once this patch is merged.

This patchset won't break current SCSI uses of blk_mq_quiesce_queue() and
won't cause regression, and I setup one iSCSI target yesterday and it works
just fine, how about just keeping this patch as it is?

Once it is merged, you can figure out one perfect solution, but
the contract of blk_mq_quiesce_queue() still shouldn't be broken.

Thanks,
Ming

Re: [PATCH v3 3/9] blk-mq: use the introduced blk_mq_unquiesce_queue()

[Bart Van Assche]

On Fri, 2017-06-02 at 10:00 +0800, Ming Lei wrote:
> On Thu, Jun 01, 2017 at 11:09:00PM +0000, Bart Van Assche wrote:
> > My opinion is that scsi_internal_device_block() and scsi_internal_device_unblock()
> > should be changed as follows for the scsi-mq code path:
> > * scsi_internal_device_block() should call blk_mq_quiesce_queue(q) if the "wait"
> >   argument is true and should set the QUEUE_FLAG_QUIESCED flag if the "wait"
> >   argument is false.
> > * scsi_internal_device_unblock() should call blk_mq_unquiesce_queue().
> > 
> > I am aware it sounds weird to set the QUEUE_FLAG_QUIESCED flag without waiting
> > until ongoing .queue_rq() calls have finished. The only driver that triggers
> 
> This way may break the contract of blk_mq_quiesce_queue() and is really weird.
> And the above change shouldn't be related with this patchset, so better
> to do whatever you want once this patch is merged.

No, what I proposed doesn't break the contract of blk_mq_quiesce_queue().
The contract of that function is that if it is called that all pending
.queue_rq() calls have finished and no new .queue_rq() calls will occur
until blk_mq_unquiesce_queue() is called. Since the wait == false path does
not call blk_mq_quiesce_queue() that contract does not apply.

> This patchset won't break current SCSI uses of blk_mq_quiesce_queue() and
> won't cause regression, and I setup one iSCSI target yesterday and it works
> just fine, how about just keeping this patch as it is?

As I explained in a previous e-mail, the iSCSI initiator driver does not
trigger the wait == false path so your test did not trigger that code path.

Something else I explained in a previous e-mail is that your patch makes it
impossible to use the STOPPED flag in the SCSI core what it was originally
intended for, namely preventing that the block layer keeps trying to queue
requests while the block driver knows that it will have to return "BUSY".
This is why I asked you to modify the SCSI integration of your patch.

Bart.