From: Ming Lei <ming.lei@redhat.com>
To: John Garry <john.garry@huawei.com>
Cc: hare@suse.de, bvanassche@acm.org, axboe@kernel.dk, hch@lst.de,
linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
pragalla@codeaurora.org, kashyap.desai@broadcom.com,
yuyufen@huawei.com
Subject: Re: [RFC PATCH v3 2/3] blk-mq: Freeze and quiesce all queues for tagset in elevator_exit()
Date: Thu, 11 Mar 2021 08:58:48 +0800 [thread overview]
Message-ID: <YElrSFGyim3rjDN+@T590> (raw)
In-Reply-To: <1614957294-188540-3-git-send-email-john.garry@huawei.com>
On Fri, Mar 05, 2021 at 11:14:53PM +0800, John Garry wrote:
> A use-after-free may occur if blk_mq_queue_tag_busy_iter() is run on a
> queue when another queue associated with the same tagset is switching IO
> scheduler:
>
> BUG: KASAN: use-after-free in bt_iter+0xa0/0x120
> Read of size 8 at addr ffff0410285e7e00 by task fio/2302
>
> CPU: 24 PID: 2302 Comm: fio Not tainted 5.12.0-rc1-11925-g29a317e228d9 #747
> Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 IT21 Nemo 2.0 RC0 04/18/2018
> Call trace:
> dump_backtrace+0x0/0x2d8
> show_stack+0x18/0x68
> dump_stack+0x124/0x1a0
> print_address_description.constprop.13+0x68/0x30c
> kasan_report+0x1e8/0x258
> __asan_load8+0x9c/0xd8
> bt_iter+0xa0/0x120
> blk_mq_queue_tag_busy_iter+0x348/0x5d8
> blk_mq_in_flight+0x80/0xb8
> part_stat_show+0xcc/0x210
> dev_attr_show+0x44/0x90
> sysfs_kf_seq_show+0x120/0x1c0
> kernfs_seq_show+0x9c/0xb8
> seq_read_iter+0x214/0x668
> kernfs_fop_read_iter+0x204/0x2c0
> new_sync_read+0x1ec/0x2d0
> vfs_read+0x18c/0x248
> ksys_read+0xc8/0x178
> __arm64_sys_read+0x44/0x58
> el0_svc_common.constprop.1+0xc8/0x1a8
> do_el0_svc+0x90/0xa0
> el0_svc+0x24/0x38
> el0_sync_handler+0x90/0xb8
> el0_sync+0x154/0x180
>
> Indeed, blk_mq_queue_tag_busy_iter() already does take a reference to its
> queue usage counter when called, and the queue cannot be frozen to switch
> IO scheduler until all refs are dropped. This ensures no stale references
> to IO scheduler requests will be seen by blk_mq_queue_tag_busy_iter().
>
> However, there is nothing to stop blk_mq_queue_tag_busy_iter() being
> run for another queue associated with the same tagset, and it seeing
> a stale IO scheduler request from the other queue after they are freed.
>
> To stop this happening, freeze and quiesce all queues associated with the
> tagset as the elevator is exited.
I think this way can't be accepted since switching one queue's scheduler
is nothing to do with other request queues attached to same HBA.
This patch will cause performance regression because userspace may
switch scheduler according to medium or workloads, at that time other
LUNs will be affected by this patch.
--
Ming
next prev parent reply other threads:[~2021-03-11 0:59 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-05 15:14 [RFC PATCH v3 0/3] blk-mq: Avoid use-after-free for accessing old requests John Garry
2021-03-05 15:14 ` [RFC PATCH v3 1/3] blk-mq: Clean up references to old requests when freeing rqs John Garry
2021-03-06 2:52 ` Khazhy Kumykov
2021-03-08 11:18 ` John Garry
2021-03-06 18:13 ` Bart Van Assche
2021-03-08 10:37 ` John Garry
2021-03-05 15:14 ` [RFC PATCH v3 2/3] blk-mq: Freeze and quiesce all queues for tagset in elevator_exit() John Garry
2021-03-06 4:32 ` Bart Van Assche
2021-03-08 10:50 ` John Garry
2021-03-08 19:35 ` Bart Van Assche
2021-03-10 15:57 ` Bart Van Assche
2021-03-11 0:58 ` Ming Lei [this message]
2021-03-11 8:21 ` John Garry
2021-03-12 23:05 ` Bart Van Assche
2021-03-16 16:15 ` John Garry
2021-03-16 17:00 ` Bart Van Assche
2021-03-16 17:43 ` John Garry
2021-03-16 19:59 ` Bart Van Assche
2021-03-19 18:19 ` John Garry
2021-03-19 18:32 ` Bart Van Assche
2021-03-05 15:14 ` [RFC PATCH v3 3/3] blk-mq: Lockout tagset iterator when exiting elevator John Garry
2021-03-06 4:43 ` Bart Van Assche
2021-03-08 11:17 ` John Garry
2021-03-08 19:59 ` Bart Van Assche
2021-03-09 17:47 ` John Garry
2021-03-09 19:21 ` Bart Van Assche
2021-03-10 8:52 ` John Garry
2021-03-10 16:00 ` Bart Van Assche
2021-03-10 17:26 ` John Garry
2021-03-18 10:26 ` [RFC PATCH v3 0/3] blk-mq: Avoid use-after-free for accessing old requests Shinichiro Kawasaki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YElrSFGyim3rjDN+@T590 \
--to=ming.lei@redhat.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=john.garry@huawei.com \
--cc=kashyap.desai@broadcom.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=pragalla@codeaurora.org \
--cc=yuyufen@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox