From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bart Van Assche Subject: Re: [PATCH 05/12] blk-mq: Introduce blk_mq_quiesce_queue() Date: Wed, 26 Oct 2016 19:40:26 -0700 Message-ID: <16074653-5e31-27b0-90db-33d36a9df0bc@acm.org> References: <5143c240-39af-9fe2-d3e6-ed69f9c20531@sandisk.com> <7690b469-f5b7-ab04-4b6f-fa0d679e0f18@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Ming Lei Cc: Jens Axboe , Christoph Hellwig , James Bottomley , "Martin K. Petersen" , Mike Snitzer , Doug Ledford , Keith Busch , Laurence Oberman , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" List-Id: linux-rdma@vger.kernel.org On 10/26/16 19:31, Ming Lei wrote: > On Thu, Oct 27, 2016 at 10:04 AM, Bart Van Assche wrote: >> On 10/26/16 18:30, Ming Lei wrote: >>> >>> On Thu, Oct 27, 2016 at 6:53 AM, Bart Van Assche >>> wrote: >>>> >>>> blk_mq_quiesce_queue() waits until ongoing .queue_rq() invocations >>>> have finished. This function does *not* wait until all outstanding >>>> requests have finished (this means invocation of request.end_io()). >>>> The algorithm used by blk_mq_quiesce_queue() is as follows: >>>> * Hold either an RCU read lock or an SRCU read lock around >>>> .queue_rq() calls. The former is used if .queue_rq() does not >>>> block and the latter if .queue_rq() may block. >>>> * blk_mq_quiesce_queue() calls synchronize_srcu() or >>>> synchronize_rcu() to wait for .queue_rq() invocations that >>>> started before blk_mq_quiesce_queue() was called. >>>> * The blk_mq_hctx_stopped() calls that control whether or not >>>> .queue_rq() will be called are called with the (S)RCU read lock >>>> held. This is necessary to avoid race conditions against >>>> the "blk_mq_stop_hw_queues(q); blk_mq_quiesce_queue(q);" >>>> sequence from another thread. >>>> >>>> Signed-off-by: Bart Van Assche >>>> Cc: Christoph Hellwig >>>> Cc: Ming Lei >>>> Cc: Hannes Reinecke >>>> Cc: Johannes Thumshirn >>>> --- >>>> block/Kconfig | 1 + >>>> block/blk-mq.c | 69 >>>> +++++++++++++++++++++++++++++++++++++++++++++----- >>>> include/linux/blk-mq.h | 3 +++ >>>> include/linux/blkdev.h | 1 + >>>> 4 files changed, 67 insertions(+), 7 deletions(-) >>>> >>>> diff --git a/block/Kconfig b/block/Kconfig >>>> index 1d4d624..0562ef9 100644 >>>> --- a/block/Kconfig >>>> +++ b/block/Kconfig >>>> @@ -5,6 +5,7 @@ menuconfig BLOCK >>>> bool "Enable the block layer" if EXPERT >>>> default y >>>> select SBITMAP >>>> + select SRCU >>>> help >>>> Provide block layer support for the kernel. >>>> >>>> diff --git a/block/blk-mq.c b/block/blk-mq.c >>>> index 0cf21c2..4945437 100644 >>>> --- a/block/blk-mq.c >>>> +++ b/block/blk-mq.c >>>> @@ -115,6 +115,31 @@ void blk_mq_unfreeze_queue(struct request_queue *q) >>>> } >>>> EXPORT_SYMBOL_GPL(blk_mq_unfreeze_queue); >>>> >>>> +/** >>>> + * blk_mq_quiesce_queue() - wait until all ongoing queue_rq calls have >>>> finished >>>> + * @q: request queue. >>>> + * >>>> + * Note: this function does not prevent that the struct request end_io() >>>> + * callback function is invoked. Additionally, it is not prevented that >>>> + * new queue_rq() calls occur unless the queue has been stopped first. >>>> + */ >>>> +void blk_mq_quiesce_queue(struct request_queue *q) >>>> +{ >>>> + struct blk_mq_hw_ctx *hctx; >>>> + unsigned int i; >>>> + bool rcu = false; >>> >>> >>> Before synchronizing SRCU/RCU, we have to set a per-hctx flag >>> or per-queue flag to block comming .queue_rq(), seems I mentioned >>> that before: >>> >>> https://www.spinics.net/lists/linux-rdma/msg41389.html >> >> >> Hello Ming, >> >> Thanks for having included an URL to an archived version of that discussion. >> What I remember about that discussion is that I proposed to use the existing >> flag BLK_MQ_S_STOPPED instead of introducing a >> new QUEUE_FLAG_QUIESCING flag and that you agreed with that proposal. See >> also https://www.spinics.net/lists/linux-rdma/msg41430.html. > > Yes, I am fine with either one, but the flag need to set in > blk_mq_quiesce_queue(), doesnt't it? Hello Ming, If you have a look at the later patches in this series then you will see that the dm core and the NVMe driver have been modified such that blk_mq_stop_hw_queues(q) is called immediately before blk_mq_quiesce_queue(q) is called. Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html