public inbox for linux-block@vger.kernel.org
 help / color / mirror / Atom feed
* [PATCH v5 00/12] blk-mq: fix & improve queue quiescing
@ 2017-06-06 15:20 Ming Lei
  0 siblings, 0 replies; 7+ messages in thread
From: Ming Lei @ 2017-06-06 15:20 UTC (permalink / raw)
  To: Jens Axboe, linux-block, Christoph Hellwig; +Cc: Bart Van Assche, Ming Lei

Hi,

There is one big issue in current blk_mq_quiesce_queue():

    - in case of direct issue or BLK_MQ_S_START_ON_RUN, dispatch won't
    be prevented after blk_mq_quiesce_queue() is returned.


The 1st two patches fix two problems in direct issue, please consider
it for v4.12.

The other 10 patches improves blk_mq_quiesce_queue(), and
make is easy to use, and avoid race between queue restart and
quiescing. These 10 patches should be for v4.13.

One typical use case of blk_mq_quiesce_queue() is for canceling
requests when queue is dead. Currently, only NVMe uses this API
for canceling requests. Actually we have to quiesce queue first
before canceling requests in other drivers too, such as NBD and
mtip32xx,

Another use case is for freezing device, for example, virtio-blk
uses stopping queue in virtblk_freeze(), but that way isn't safe
becasue dispatch still may happen after blk_mq_stop_hw_queues()
returns.

Unfortunately blk_mq_quiesce_queue() is implemented via stopping queue,
we can't switch to blk_mq_quiesce_queue() simply in above cases because
any queue restart in other pathes may break blk_mq_quiesce_queue().
For example, we sometimes stops queue when hw can't handle too many
ongoing requests and restarts queue after requests are completed.
Meantime when we want to cancel requests if hardware is dead or need
to suspend, quiescing has to be run first, then the restarting
in complete path can break quiescing easily. This patch improves this
interface via removing stopping queue, then it can be easier to use.

V5:
	- pass correct hctx to blk_mq_try_issue_directly()(patch 1)
	- add more comment on blk_mq_stop_hw_queue() as suggested by Bart
	- add Reviewed-by tag

V4:
	- introduce the 1st patch for fixing direct issue
	- take Bart's suggestion to deal with quiescing in SCSI
	- avoid to introduce waitqueue

V3:
	- wait until queue becomes unquiesced in direct issue path, so
	we can avoid to queue the current req into sw queue or scheduler
	queue, then the state of STOPPED needn't to be touched
	- move checking of !blk_queue_quiesced() into blk_mq_sched_dispatch_requests()
	as suggested by Bart
	- NVMe: unquiesce queue in nvme_kill_queues()
	- misc changes(fix grammer issue in commit log or comment, ...)

V2:
	- split patch "blk-mq: fix blk_mq_quiesce_queue" into two and
  	fix one build issue when only applying the 1st two patches.
	- add kernel oops and hang log into commit log
	- add 'Revert "blk-mq: don't use sync workqueue flushing from drivers"'


Ming Lei (12):
  blk-mq: pass correct hctx to blk_mq_try_issue_directly
  blk-mq: fix direct issue
  blk-mq: move blk_mq_quiesce_queue() into include/linux/blk-mq.h
  blk-mq: introduce blk_mq_quiesce_queue_nowait()
  blk-mq: introduce blk_mq_unquiesce_queue
  blk-mq: use the introduced blk_mq_unquiesce_queue()
  nvme: host: unquiesce queue in nvme_kill_queues()
  blk-mq: use QUEUE_FLAG_QUIESCED to quiesce queue
  blk-mq: update comments on blk_mq_quiesce_queue()
  blk-mq: don't stop queue for quiescing
  blk-mq: clarify dispatch may not be drained/blocked by stopping queue
  Revert "blk-mq: don't use sync workqueue flushing from drivers"

 block/blk-mq-sched.c     |  3 +-
 block/blk-mq.c           | 93 ++++++++++++++++++++++++++++++++----------------
 drivers/md/dm-rq.c       |  2 +-
 drivers/nvme/host/core.c |  8 ++++-
 drivers/scsi/scsi_lib.c  |  4 +--
 include/linux/blk-mq.h   | 12 +++++++
 include/linux/blkdev.h   |  3 +-
 7 files changed, 89 insertions(+), 36 deletions(-)

-- 
2.9.4

^ permalink raw reply	[flat|nested] 7+ messages in thread
* [PATCH v5 00/12] blk-mq: fix & improve queue quiescing
@ 2017-06-06 15:21 Ming Lei
  2017-06-06 16:03 ` Jens Axboe
  0 siblings, 1 reply; 7+ messages in thread
From: Ming Lei @ 2017-06-06 15:21 UTC (permalink / raw)
  To: Jens Axboe, linux-block, Christoph Hellwig; +Cc: Bart Van Assche, Ming Lei

Hi,

There is one big issue in current blk_mq_quiesce_queue():

    - in case of direct issue or BLK_MQ_S_START_ON_RUN, dispatch won't
    be prevented after blk_mq_quiesce_queue() is returned.


The 1st two patches fix two problems in direct issue, please consider
it for v4.12.

The other 10 patches improves blk_mq_quiesce_queue(), and
make is easy to use, and avoid race between queue restart and
quiescing. These 10 patches should be for v4.13.

One typical use case of blk_mq_quiesce_queue() is for canceling
requests when queue is dead. Currently, only NVMe uses this API
for canceling requests. Actually we have to quiesce queue first
before canceling requests in other drivers too, such as NBD and
mtip32xx,

Another use case is for freezing device, for example, virtio-blk
uses stopping queue in virtblk_freeze(), but that way isn't safe
becasue dispatch still may happen after blk_mq_stop_hw_queues()
returns.

Unfortunately blk_mq_quiesce_queue() is implemented via stopping queue,
we can't switch to blk_mq_quiesce_queue() simply in above cases because
any queue restart in other pathes may break blk_mq_quiesce_queue().
For example, we sometimes stops queue when hw can't handle too many
ongoing requests and restarts queue after requests are completed.
Meantime when we want to cancel requests if hardware is dead or need
to suspend, quiescing has to be run first, then the restarting
in complete path can break quiescing easily. This patch improves this
interface via removing stopping queue, then it can be easier to use.

V5:
	- pass correct hctx to blk_mq_try_issue_directly()(patch 1)
	- add more comment on blk_mq_stop_hw_queue() as suggested by Bart
	- add Reviewed-by tag

V4:
	- introduce the 1st patch for fixing direct issue
	- take Bart's suggestion to deal with quiescing in SCSI
	- avoid to introduce waitqueue

V3:
	- wait until queue becomes unquiesced in direct issue path, so
	we can avoid to queue the current req into sw queue or scheduler
	queue, then the state of STOPPED needn't to be touched
	- move checking of !blk_queue_quiesced() into blk_mq_sched_dispatch_requests()
	as suggested by Bart
	- NVMe: unquiesce queue in nvme_kill_queues()
	- misc changes(fix grammer issue in commit log or comment, ...)

V2:
	- split patch "blk-mq: fix blk_mq_quiesce_queue" into two and
  	fix one build issue when only applying the 1st two patches.
	- add kernel oops and hang log into commit log
	- add 'Revert "blk-mq: don't use sync workqueue flushing from drivers"'


Ming Lei (12):
  blk-mq: pass correct hctx to blk_mq_try_issue_directly
  blk-mq: fix direct issue
  blk-mq: move blk_mq_quiesce_queue() into include/linux/blk-mq.h
  blk-mq: introduce blk_mq_quiesce_queue_nowait()
  blk-mq: introduce blk_mq_unquiesce_queue
  blk-mq: use the introduced blk_mq_unquiesce_queue()
  nvme: host: unquiesce queue in nvme_kill_queues()
  blk-mq: use QUEUE_FLAG_QUIESCED to quiesce queue
  blk-mq: update comments on blk_mq_quiesce_queue()
  blk-mq: don't stop queue for quiescing
  blk-mq: clarify dispatch may not be drained/blocked by stopping queue
  Revert "blk-mq: don't use sync workqueue flushing from drivers"

 block/blk-mq-sched.c     |  3 +-
 block/blk-mq.c           | 93 ++++++++++++++++++++++++++++++++----------------
 drivers/md/dm-rq.c       |  2 +-
 drivers/nvme/host/core.c |  8 ++++-
 drivers/scsi/scsi_lib.c  |  4 +--
 include/linux/blk-mq.h   | 12 +++++++
 include/linux/blkdev.h   |  3 +-
 7 files changed, 89 insertions(+), 36 deletions(-)

-- 
2.9.4

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-06-18 23:48 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-06-06 15:20 [PATCH v5 00/12] blk-mq: fix & improve queue quiescing Ming Lei
  -- strict thread matches above, loose matches on Subject: below --
2017-06-06 15:21 Ming Lei
2017-06-06 16:03 ` Jens Axboe
2017-06-13  2:15   ` Ming Lei
2017-06-18 20:26     ` Jens Axboe
2017-06-18 23:05       ` Bart Van Assche
2017-06-18 23:48         ` Jens Axboe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox