Re: [PATCH] blk-mq: respect queue dead via blk_mq_quiesce_queue

[Ming Lei <ming.lei@redhat.com>]

On Sun, Nov 05, 2017 at 08:10:08PM +0800, Ming Lei wrote:
> blk-mq never respects queue dead, and this may cause use-after-free on
> any kind of queue resources. This patch respects the rule by calling
> blk_mq_quiesce_queue() when queue is marked as DEAD.
> 
> This patch fixes the following kernel crash:
> 
> [   42.170824] BUG: unable to handle kernel NULL pointer dereference at           (null)
> [   42.172011] IP: blk_mq_flush_busy_ctxs+0x5a/0xe0
> [   42.172011] PGD 25d8d1067 P4D 25d8d1067 PUD 25d458067 PMD 0
> [   42.172011] Oops: 0000 [#1] PREEMPT SMP
> [   42.172011] Dumping ftrace buffer:
> [   42.172011]    (ftrace buffer empty)
> [   42.172011] Modules linked in: scsi_debug ebtable_filter ebtables ip6table_filter ip6_tables xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c bridge stp llc fuse iptable_filter ip_tables sd_mod sg mptsas mptscsih mptbase crc32c_intel scsi_transport_sas nvme lpc_ich serio_raw ahci virtio_scsi libahci libata nvme_core binfmt_misc dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi null_blk configs
> [   42.172011] CPU: 0 PID: 2750 Comm: fio Not tainted 4.14.0-rc7.blk_mq_io_hang+ #507
> [   42.172011] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.9.3-1.fc25 04/01/2014
> [   42.172011] task: ffff88025dc18000 task.stack: ffffc900028c4000
> [   42.172011] RIP: 0010:blk_mq_flush_busy_ctxs+0x5a/0xe0
> [   42.172011] RSP: 0018:ffffc900028c7be0 EFLAGS: 00010246
> [   42.172011] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8802775907c8
> [   42.172011] RDX: 0000000000000000 RSI: ffffc900028c7c50 RDI: ffff88025de25c00
> [   42.172011] RBP: ffffc900028c7c28 R08: 0000000000000000 R09: ffff8802595692c0
> [   42.172011] R10: ffffc900028c7e50 R11: 0000000000000008 R12: ffffc900028c7c50
> [   42.172011] R13: ffff88025de25cd8 R14: ffffc900028c7d78 R15: ffff88025de25c00
> [   42.172011] FS:  00007faa8653f7c0(0000) GS:ffff88027fc00000(0000) knlGS:0000000000000000
> [   42.172011] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   42.172011] CR2: 0000000000000000 CR3: 00000002593df006 CR4: 00000000003606f0
> [   42.172011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   42.172011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [   42.172011] Call Trace:
> [   42.172011]  blk_mq_sched_dispatch_requests+0x1b4/0x1f0
> [   42.172011]  ? preempt_schedule+0x27/0x30
> [   42.172011]  __blk_mq_run_hw_queue+0x8b/0xa0
> [   42.172011]  __blk_mq_delay_run_hw_queue+0xb7/0x100
> [   42.172011]  blk_mq_run_hw_queue+0x14/0x20
> [   42.172011]  blk_mq_sched_insert_requests+0xda/0x120
> [   42.172011]  blk_mq_flush_plug_list+0x179/0x280
> [   42.172011]  blk_flush_plug_list+0x102/0x290
> [   42.172011]  blk_finish_plug+0x2c/0x40
> [   42.172011]  do_io_submit+0x3fa/0x780
> [   42.172011]  SyS_io_submit+0x10/0x20
> [   42.172011]  ? SyS_io_submit+0x10/0x20
> [   42.172011]  entry_SYSCALL_64_fastpath+0x1a/0xa5
> [   42.172011] RIP: 0033:0x7faa80ff8697
> [   42.172011] RSP: 002b:00007ffe08236de8 EFLAGS: 00000206 ORIG_RAX: 00000000000000d1
> [   42.172011] RAX: ffffffffffffffda RBX: 0000000001f1e600 RCX: 00007faa80ff8697
> [   42.172011] RDX: 000000000208e638 RSI: 0000000000000001 RDI: 00007faa6ecae000
> [   42.172011] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000001f1e0e0
> [   42.172011] R10: 0000000000000001 R11: 0000000000000206 R12: 00007faa614f19e0
> [   42.172011] R13: 00007faa614ff0b0 R14: 00007faa614fef50 R15: 0000000100000000
> [   42.172011] Code: e0 00 00 00 c7 45 bc 00 00 00 00 85 c0 74 76 4c 8d af d8 00 00 00 49 89 ff 44 8b 45 bc 4c 89 c0 49 c1 e0 06 4d 03 87 e8 00 00 00 <49> 83 38 00 4d 89 c6 74 41 41 8b 8f dc 00 00 00 41 89 c4 31 db
> [   42.172011] RIP: blk_mq_flush_busy_ctxs+0x5a/0xe0 RSP: ffffc900028c7be0
> [   42.172011] CR2: 0000000000000000
> [   42.172011] ---[ end trace 2432dfddf9b84061 ]---
> [   42.172011] Kernel panic - not syncing: Fatal exception
> [   42.172011] Dumping ftrace buffer:
> [   42.172011]    (ftrace buffer empty)
> [   42.172011] Kernel Offset: disabled
> [   42.172011] ---[ end Kernel panic - not syncing: Fatal exception
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
>  block/blk-core.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 048be4aa6024..0b121f29e3b1 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -658,6 +658,10 @@ void blk_cleanup_queue(struct request_queue *q)
>  	queue_flag_set(QUEUE_FLAG_DEAD, q);
>  	spin_unlock_irq(lock);
>  
> +	/* respect queue DEAD via quiesce for blk-mq */
> +	if (q->mq_ops)
> +		blk_mq_quiesce_queue(q);
> +
>  	/* for synchronous bio-based driver finish in-flight integrity i/o */
>  	blk_flush_integrity();

Hi Jens,

Could you consider this patch? This issue can be reproduced easily
in heavy IO and device remove test on SCSI.

-- 
Ming