Re: [PATCH V2 5/5] blk-mq: Wait for for hctx inflight requests on CPU unplug

[Ming Lei <ming.lei@redhat.com>]

On Tue, May 28, 2019 at 05:50:40PM +0100, John Garry wrote:
> On 27/05/2019 16:02, Ming Lei wrote:
> > Managed interrupts can not migrate affinity when their CPUs are offline.
> > If the CPU is allowed to shutdown before they're returned, commands
> > dispatched to managed queues won't be able to complete through their
> > irq handlers.
> > 
> > Wait in cpu hotplug handler until all inflight requests on the tags
> > are completed or timeout. Wait once for each tags, so we can save time
> > in case of shared tags.
> > 
> > Based on the following patch from Keith, and use simple delay-spin
> > instead.
> > 
> > https://lore.kernel.org/linux-block/20190405215920.27085-1-keith.busch@intel.com/
> > 
> > Some SCSI devices may have single blk_mq hw queue and multiple private
> > completion queues, and wait until all requests on the private completion
> > queue are completed.
> 
> Hi Ming,
> 
> I'm a bit concerned that this approach won't work due to ordering: it seems
> that the IRQ would be shutdown prior to the CPU dead notification for the

Managed IRQ shutdown is run in irq_migrate_all_off_this_cpu(), which is
called in the callback of takedown_cpu(). And the CPU dead notification
is always sent after that CPU becomes offline, see cpuhp_invoke_callback().

> last CPU in the mask (where we attempt to drain the queue associated with
> the IRQ, which would require the IRQ to be still enabled).
> 
> I hope that you can tell me that I'm wrong...

Or you add one line printk in both irq_migrate_all_off_this_cpu() and
blk_mq_hctx_notify_dead(), you will see if you are wrong.


Thanks, 
Ming