From: Ye Bin <yebin@huaweicloud.com>
To: axboe@kernel.dk, linux-block@vger.kernel.org
Cc: ming.lei@redhat.com
Subject: [PATCH v2] blk-flush: fix possible deadlock when process nvme_timeout()
Date: Thu, 12 Nov 2026 21:50:19 +0800 [thread overview]
Message-ID: <20261112135019.3290131-1-yebin@huaweicloud.com> (raw)
From: Ye Bin <yebin10@huawei.com>
There's when process nvme_timeout():
[ 206.734601][ T8184] nvme nvme0: I/O tag 512 (1200) opcode 0x0 (I/O Cmd) QID 3 timeout, aborting req_op:FLUSH(2) size:0
[ 206.736112][ C0] nvme nvme0: Abort status: 0x0
[ 208.094637][ T8184] nvme nvme0: I/O tag 512 (1200) opcode 0x0 (I/O Cmd) QID 3 timeout, reset controller
[root@localhost ~]# cat /proc/8184/stack
[<0>] msleep+0x37/0x50
[<0>] blk_mq_tagset_wait_completed_request+0x6f/0xe0
[<0>] nvme_cancel_tagset+0x79/0xa0
[<0>] nvme_dev_disable+0x55c/0x7e0
[<0>] nvme_timeout+0x25b/0x1530
[<0>] blk_mq_handle_expired+0x210/0x2c0
[<0>] bt_iter+0x2bb/0x360
[<0>] blk_mq_queue_tag_busy_iter+0x9f8/0x1f30
[<0>] blk_mq_timeout_work+0x5dc/0x7d0
[<0>] process_one_work+0xa08/0x1d00
[<0>] worker_thread+0x698/0xeb0
[<0>] kthread+0x408/0x540
[<0>] ret_from_fork+0xa4d/0xdd0
[<0>] ret_from_fork_asm+0x1a/0x30
Above issue may happen as follows:
nvme_timeout // tag 512 request's flush request the first timeout
iod->aborted = 1;
abort_req = nvme_alloc_request(dev->ctrl.admin_q, &cmd,
BLK_MQ_REQ_NOWAIT, NVME_QID_ANY); // Abort tag 512 flush request
blk_execute_rq_nowait(abort_req->q, NULL, abort_req, 0, abort_endio);
// Abort request completion, will no wait
....
****'abort_req' not complete***
....
nvme_timeout // tag 512 request's flush request the second timeout
if (!nvmeq->qid || (iod->flags & IOD_ABORTED))
nvme_req(req)->flags |= NVME_REQ_CANCELLED;
goto disable;
...
**** tag 512 request's flush request end ****
nvme_try_complete_req // PATH[1]
blk_mq_complete_request_remote(req);
WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
...
nvme_end_req(req);
blk_mq_end_request(req, status);
__blk_mq_end_request(rq, error);
if (rq->end_io)
rq->end_io(rq, error);
flush_end_io(rq, error);
// The timeout process holds the reference count.
// so request keep MQ_RQ_COMPLETE state
if (!refcount_dec_and_test(&flush_rq->ref))
fq->rq_status = error;
return;
**** tag 512 flush request is MQ_RQ_COMPLETE state ****
disable:
nvme_dev_disable(dev, false);
nvme_cancel_tagset(&dev->ctrl);
blk_mq_tagset_busy_iter(&dev->tagset, nvme_cancel_request,
&dev->ctrl);
nvme_cancel_request
if (blk_mq_request_completed(req)) // PATH[2]
return true;
blk_mq_tagset_wait_completed_request(&dev->tagset);
while (true)
blk_mq_tagset_busy_iter(tagset,
blk_mq_tagset_count_completed_rqs, &count);
blk_mq_tagset_count_completed_rqs();
// request is MQ_RQ_COMPLETE state
if (blk_mq_request_completed(rq)) // return true
(*count)++;
if (!count) // So the value of 'count' is never 0, loop endless
break;
msleep(5);
The root cause is that the flush-rq is moved to MQ_RQ_IDLE only when its
reference count drops to zero, which was introduced by commit 9f16a66733c9
("block: mark flush request as IDLE when it is really finished"). When
the last reference is held by a request iterator - here
blk_mq_timeout_work() iterating tags and running nvme_timeout()
- flush_end_io() returns early and leaves the request in MQ_RQ_COMPLETE,
so the very same timeout flow spins forever in
blk_mq_tagset_wait_completed_request().
In fact, the flush request cannot be set to the MQ_RQ_IDLE in the timeout
processing procedure in both the PATH[1] and PATH[2] cases.
Fix this by marking the flush request as IDLE as soon as flush_end_io()
is invoked, i.e. restore the behavior introduced by commit 65ff5cd04551
("blk-mq: mark flush request as IDLE in flush_end_io()"), which aligns
flush requests with normal requests: blk_mq_free_request() also sets
MQ_RQ_IDLE before the last reference is put.
Reverting to that version is safe now because:
- The request can only be freed/reused after its reference count reaches
zero, and whether the final cleanup has to be run is decided by the
reference count below, not by the request state. blk_mq_put_rq_ref()
chooses to call ->end_io() based on is_flush_rq() rather than the
state, so the cleanup (io accounting, releasing the borrowed tags and
advancing the flush state machine) still runs exactly once, on the last
reference. Marking IDLE early therefore cannot double-account or leak
the request.
- The use-after-free that the above commit wanted to avoid predates
the per-request reference counting model, under which a request that
is still referenced can no longer be reallocated beneath its holder.
Fixes: 9f16a66733c9 ("block: mark flush request as IDLE when it is really finished")
Signed-off-by: Ye Bin <yebin10@huawei.com>
---
block/blk-flush.c | 28 ++++++++++++++++++++++------
1 file changed, 22 insertions(+), 6 deletions(-)
diff --git a/block/blk-flush.c b/block/blk-flush.c
index 38bd4ae5d9e7..277c5a9a4bb7 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -211,6 +211,28 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq,
/* release the tag's ownership to the req cloned from */
spin_lock_irqsave(&fq->mq_flush_lock, flags);
+ /*
+ * Mark the flush request as IDLE as soon as its .end_io() is invoked,
+ * aligning it with how a normal request behaves: blk_mq_free_request()
+ * sets MQ_RQ_IDLE before the last reference is put too.
+ *
+ * flush_end_io() is called from both the normal completion path and
+ * from blk_mq_put_rq_ref() when a request iterator drops its reference.
+ * If the last reference is held by such an iterator (e.g. the timeout
+ * work that is currently running nvme_timeout()), the request would
+ * otherwise stay in MQ_RQ_COMPLETE and make the
+ * blk_mq_tagset_wait_completed_request() called from that same path
+ * spin forever.
+ *
+ * This is safe because the request is only freed/reused once its
+ * reference count reaches zero, and whether ->end_io() has to run the
+ * final cleanup is decided by the reference count below, not by the
+ * request state. The cleanup (io accounting, releasing tags and
+ * advancing the flush state machine) still runs exactly once, on the
+ * last reference.
+ */
+ WRITE_ONCE(flush_rq->state, MQ_RQ_IDLE);
+
if (!req_ref_put_and_test(flush_rq)) {
/*
* Remember the status of the flush for whoever drops the last
@@ -224,12 +246,6 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq,
}
blk_account_io_flush(flush_rq);
- /*
- * Flush request has to be marked as IDLE when it is really ended
- * because its .end_io() is called from timeout code path too for
- * avoiding use-after-free.
- */
- WRITE_ONCE(flush_rq->state, MQ_RQ_IDLE);
if (fq->rq_status != BLK_STS_OK) {
error = fq->rq_status;
fq->rq_status = BLK_STS_OK;
--
2.34.1
next reply other threads:[~2026-11-12 13:59 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 13:50 Ye Bin [this message]
2026-11-15 14:41 ` [PATCH v2] blk-flush: fix possible deadlock when process nvme_timeout() James Eversole
2026-11-15 15:06 ` Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261112135019.3290131-1-yebin@huaweicloud.com \
--to=yebin@huaweicloud.com \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=ming.lei@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox