From: sashiko-bot@kernel.org
To: "Srivatsa S. Bhat (Microsoft)" <srivatsa@csail.mit.edu>
Cc: linux-hyperv@vger.kernel.org
Subject: Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
Date: Wed, 18 Nov 2026 19:57:27 +0000 [thread overview]
Message-ID: <sashiko-outbox-221909@kernel.org> (raw)
In-Reply-To: <20261119-fix-blk-mq-null-ptr-deref-v1-1-9bcbcfcfaff8@csail.mit.edu>
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] block: incomplete and racy freed-request check in
blk_mq_complete_request_remote()
- [Medium] nvme: missing in-flight state check in nvme_find_rq() allows
prior crash and corruption
- [Low] block: inaccurate Fixes: trailer in commit message
--
commit 65327008502cbbac3ada696570a4b5ff52feb917
Author: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>
> block: Fix NULL pointer dereference in blk_mq_complete_request_remote
>
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.
>
> When NVMe devices experience hardware failures or firmware bugs,
> completion queue corruption can occur, leading to scenarios where
> drivers attempt to complete already-freed requests. The NVMe genctr
> validation in nvme_find_rq() catches most stale completions, but when
> corruption produces a command_id whose genctr bits happen to match,
> the freed request passes validation and reaches the block layer. The
> block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
> but blk_mq_complete_request_remote() lacks validation before
> dereferencing this pointer.
>
> This manifests as a kernel crash like shown below (the stack trace
> was obtained from a crash observed in a Linux VM on Hyper-V):
>
> nvme nvme0: resetting controller due to AER
> nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
> nvme nvme0: invalid id 227 completed on queue 6
> nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
> nvme nvme0: invalid id 45094 completed on queue 94
>
> BUG: kernel NULL pointer dereference, address: 00000000000000fe
> RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
> Call Trace:
> <IRQ>
> nvme_irq+0x182/0x320 [nvme]
> __handle_irq_event_percpu+0x47/0x180
> handle_irq_event+0x38/0x60
> handle_edge_irq+0x87/0x1a0
> __common_interrupt+0x3f/0xa0
> common_interrupt+0x42/0xa0
>
> The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
> where mq_hctx is NULL.
>
> Add a NULL check that returns true (signaling to callers that the
> request has been handled) to gracefully drop freed requests, with a
> WARN_ONCE to flag the condition for diagnosis.
>
> Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
> Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>
[Severity: Low]
In commit 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API"),
blk_mq_complete_request_remote() did not dereference rq->mq_hctx. The
rq->mq_hctx->nr_ctx dereference was added later, so should the Fixes tag
instead be:
Fixes: f168420c62e7 ("blk-mq: don't redirect completion for hctx withs only one ctx mapping")
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 5ae22d5565404..5dc9d7d806b59 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
>
> bool blk_mq_complete_request_remote(struct request *rq)
> {
> + /*
> + * Validate request is not freed (__blk_mq_free_request() sets
> + * mq_hctx = NULL).
> + */
> + if (unlikely(!rq->mq_hctx)) {
> + WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
> + return true;
> + }
> +
> WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
>
> /*
[Severity: High]
Is checking !rq->mq_hctx sufficient and race-free for detecting requests
that are not in flight?
First, the batched completion fast path used by nvme-pci frees requests
without ever clearing rq->mq_hctx:
block/blk-mq.c:blk_mq_end_request_batch() {
...
WRITE_ONCE(rq->state, MQ_RQ_IDLE);
if (!req_ref_put_and_test(rq))
continue;
blk_crypto_free_request(rq);
blk_pm_mark_last_busy(rq);
if (nr_tags == TAG_COMP_BATCH || cur_hctx != rq->mq_hctx) {
if (cur_hctx)
blk_mq_flush_tag_batch(cur_hctx, tags, nr_tags);
nr_tags = 0;
cur_hctx = rq->mq_hctx;
}
tags[nr_tags++] = rq->tag;
...
}
Requests freed through blk_mq_end_request_batch(), requests reallocated
or cached in a plug via blk_mq_rq_ctx_init() (still in MQ_RQ_IDLE before
blk_mq_start_request()), and requests already in MQ_RQ_COMPLETE awaiting
remote IPI or softirq completion all have a non-NULL rq->mq_hctx. A
spurious or duplicate CQE matching one of those requests bypasses the
!rq->mq_hctx check and completes the request a second time (causing
double DMA unmapping in nvme_pci_unmap_rq(), refcount and sbitmap tag
underflow, or cyclic corruption of the per-CPU blk_cpu_done llist if
rq->ipi_list is queued twice).
Second, because __blk_mq_free_request() clears rq->mq_hctx without
synchronization against a spurious completion on another CPU, the unlocked
check still races:
CPU0 CPU1
blk_mq_free_request(rq)
WRITE_ONCE(rq->state, MQ_RQ_IDLE)
blk_mq_complete_request_remote(rq)
if (unlikely(!rq->mq_hctx))
/* observes non-NULL */
WRITE_ONCE(rq->state, MQ_RQ_COMPLETE)
__blk_mq_free_request(rq)
rq->mq_hctx = NULL
if (rq->mq_hctx->nr_ctx == 1 ...)
<-- NULL pointer dereference
In addition to crashing at rq->mq_hctx->nr_ctx, CPU1 overwrites the freed
request's rq->state back to MQ_RQ_COMPLETE after CPU0 set it to
MQ_RQ_IDLE.
[Severity: Medium]
Should stale completions be rejected earlier in nvme_find_rq() by checking
that blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT (similar to
scsi_host_find_tag() and nbd_read_reply())?
When nvme_find_rq() returns a request whose 4-bit genctr collided with a
corrupted CQE, callers act on that request before
blk_mq_complete_request_remote() is reached:
drivers/nvme/host/pci.c:nvme_handle_cqe() {
...
trace_nvme_sq(req, cqe->sq_head, data_race(nvmeq->sq_tail));
if (!nvme_try_complete_req(req, cqe->status, cqe->result) &&
...
}
If the nvme_sq tracepoint is enabled, trace_nvme_sq() calls
nvme_req_qid(req), which dereferences req->mq_hctx->queue_num and hits the
same NULL pointer dereference when req->mq_hctx is NULL. Next,
nvme_try_complete_req() increments nvme_req(req)->genctr and overwrites
status and result before calling blk_mq_complete_request_remote(), which
corrupts the request if it has already been reallocated or is awaiting
remote completion and causes its real completion to fail genctr
validation.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119-fix-blk-mq-null-ptr-deref-v1-1-9bcbcfcfaff8@csail.mit.edu?part=1
next prev parent reply other threads:[~2026-11-18 19:57 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
2026-11-18 19:57 ` sashiko-bot [this message]
2026-11-18 20:12 ` Jens Axboe
2026-11-19 3:34 ` Srivatsa S. Bhat
2026-11-18 20:53 ` Hamza Mahfooz
2026-11-18 21:43 ` Keith Busch
2026-11-19 3:29 ` Srivatsa S. Bhat
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-221909@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=linux-hyperv@vger.kernel.org \
--cc=sashiko-reviews@lists.linux.dev \
--cc=srivatsa@csail.mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.