Re: [PATCH V11 00/12] blk-mq: improvement CPU hotplug

[Ming Lei <ming.lei@redhat.com>]

On Wed, May 13, 2020 at 08:34:01AM +0100, John Garry wrote:
> On 13/05/2020 04:47, Ming Lei wrote:
> > Hi,
> > 
> > Thomas mentioned:
> >      "
> >       That was the constraint of managed interrupts from the very beginning:
> >        The driver/subsystem has to quiesce the interrupt line and the associated
> >        queue _before_ it gets shutdown in CPU unplug and not fiddle with it
> >        until it's restarted by the core when the CPU is plugged in again.
> >      "
> > 
> > But no drivers or blk-mq do that before one hctx becomes inactive(all
> > CPUs for one hctx are offline), and even it is worse, blk-mq stills tries
> > to run hw queue after hctx is dead, see blk_mq_hctx_notify_dead().
> > 
> > This patchset tries to address the issue by two stages:
> > 
> > 1) add one new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE
> > 
> > - mark the hctx as internal stopped, and drain all in-flight requests
> > if the hctx is going to be dead.
> > 
> > 2) re-submit IO in the state of CPUHP_BLK_MQ_DEAD after the hctx becomes dead
> > 
> > - steal bios from the request, and resubmit them via generic_make_request(),
> > then these IO will be mapped to other live hctx for dispatch
> > 
> > Thanks John Garry for running lots of tests on arm64 with this patchset
> > and co-working on investigating all kinds of issues.
> > 
> > Thanks Christoph's review on V7 & V8.
> > 
> > Please consider it for v5.8.
> > 
> > https://github.com/ming1/linux/commits/v5.7-rc-blk-mq-improve-cpu-hotplug
> > 
> > V11:
> > 	- drop new callback from blk_mq_all_tag_busy_iter, add new helper
> > 	of blk_mq_all_tag_iter (5/12), as suggested by Bart
> > 	- fix request allocation hang in case of queue freeze(11/12), as
> > 	reported by Bart
> > 
> > V10:
> > 	- fix double bio complete in request resubmission(10/11)
> > 	- add tested-by tag
> > 
> > V9:
> > 	- add Reviewed-by tag
> > 	- document more on memory barrier usage between getting driver tag
> > 	and handling cpu offline(7/11)
> > 	- small code cleanup as suggested by Chritoph(7/11)
> > 	- rebase against for-5.8/block(1/11, 2/11)
> > V8:
> > 	- add patches to share code with blk_rq_prep_clone
> > 	- code re-organization as suggested by Christoph, most of them are
> > 	in 04/11, 10/11
> > 	- add reviewed-by tag
> > 
> > V7:
> > 	- fix updating .nr_active in get_driver_tag
> > 	- add hctx->cpumask check in cpuhp handler
> > 	- only drain requests which tag is >= 0
> > 	- pass more aggressive cpuhotplug&io test
> > 
> > V6:
> > 	- simplify getting driver tag, so that we can drain in-flight
> > 	  requests correctly without using synchronize_rcu()
> > 	- handle re-submission of flush & passthrough request correctly
> > 
> > V5:
> > 	- rename BLK_MQ_S_INTERNAL_STOPPED as BLK_MQ_S_INACTIVE
> > 	- re-factor code for re-submit requests in cpu dead hotplug handler
> > 	- address requeue corner case
> > 
> > V4:
> > 	- resubmit IOs in dispatch list in case that this hctx is dead
> > 
> > V3:
> > 	- re-organize patch 2 & 3 a bit for addressing Hannes's comment
> > 	- fix patch 4 for avoiding potential deadlock, as found by Hannes
> > 
> > V2:
> > 	- patch4 & patch 5 in V1 have been merged to block tree, so remove
> > 	  them
> > 	- address comments from John Garry and Minwoo
> > 
> > 
> > *** BLURB HERE ***
> 
> :)
> 
> So my tested-by tags have been dropped. I'll test again, since the changes
> are non-trivial.
> 
> Tip commit of
> https://github.com/ming1/linux/commits/v5.7-rc-blk-mq-improve-cpu-hotplug at
> this moment is b55e97a4

Hi John,

I just sent out V12 of patch 10/12, which can kill the warning on BLK_MQ_REQ_FORCE,
and the warning is harmless actually. And the above git branch has been
updated too.

Please retest and update with us.

Thanks, 
Ming