Linux-HyperV List
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Srivatsa S. Bhat (Microsoft)" <srivatsa@csail.mit.edu>
Cc: linux-hyperv@vger.kernel.org
Subject: Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
Date: Wed, 18 Nov 2026 19:57:27 +0000	[thread overview]
Message-ID: <sashiko-outbox-221909@kernel.org> (raw)
In-Reply-To: <20261119-fix-blk-mq-null-ptr-deref-v1-1-9bcbcfcfaff8@csail.mit.edu>

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] block: incomplete and racy freed-request check in
  blk_mq_complete_request_remote()
- [Medium] nvme: missing in-flight state check in nvme_find_rq() allows
  prior crash and corruption
- [Low] block: inaccurate Fixes: trailer in commit message
--

commit 65327008502cbbac3ada696570a4b5ff52feb917
Author: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>

> block: Fix NULL pointer dereference in blk_mq_complete_request_remote
>
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.
>
> When NVMe devices experience hardware failures or firmware bugs,
> completion queue corruption can occur, leading to scenarios where
> drivers attempt to complete already-freed requests. The NVMe genctr
> validation in nvme_find_rq() catches most stale completions, but when
> corruption produces a command_id whose genctr bits happen to match,
> the freed request passes validation and reaches the block layer. The
> block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
> but blk_mq_complete_request_remote() lacks validation before
> dereferencing this pointer.
>
> This manifests as a kernel crash like shown below (the stack trace
> was obtained from a crash observed in a Linux VM on Hyper-V):
>
>   nvme nvme0: resetting controller due to AER
>   nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
>   nvme nvme0: invalid id 227 completed on queue 6
>   nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
>   nvme nvme0: invalid id 45094 completed on queue 94
>
>   BUG: kernel NULL pointer dereference, address: 00000000000000fe
>   RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
>   Call Trace:
>    <IRQ>
>    nvme_irq+0x182/0x320 [nvme]
>    __handle_irq_event_percpu+0x47/0x180
>    handle_irq_event+0x38/0x60
>    handle_edge_irq+0x87/0x1a0
>    __common_interrupt+0x3f/0xa0
>    common_interrupt+0x42/0xa0
>
> The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
> where mq_hctx is NULL.
>
> Add a NULL check that returns true (signaling to callers that the
> request has been handled) to gracefully drop freed requests, with a
> WARN_ONCE to flag the condition for diagnosis.
>
> Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
> Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>

[Severity: Low]
In commit 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API"),
blk_mq_complete_request_remote() did not dereference rq->mq_hctx.  The
rq->mq_hctx->nr_ctx dereference was added later, so should the Fixes tag
instead be:

  Fixes: f168420c62e7 ("blk-mq: don't redirect completion for hctx withs only one ctx mapping")

> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 5ae22d5565404..5dc9d7d806b59 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
>  
>  bool blk_mq_complete_request_remote(struct request *rq)
>  {
> +	/*
> +	 * Validate request is not freed (__blk_mq_free_request() sets
> +	 * mq_hctx = NULL).
> +	 */
> +	if (unlikely(!rq->mq_hctx)) {
> +		WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
> +		return true;
> +	}
> +
>  	WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
>  
>  	/*

[Severity: High]
Is checking !rq->mq_hctx sufficient and race-free for detecting requests
that are not in flight?

First, the batched completion fast path used by nvme-pci frees requests
without ever clearing rq->mq_hctx:

block/blk-mq.c:blk_mq_end_request_batch() {
    ...
		WRITE_ONCE(rq->state, MQ_RQ_IDLE);
		if (!req_ref_put_and_test(rq))
			continue;

		blk_crypto_free_request(rq);
		blk_pm_mark_last_busy(rq);

		if (nr_tags == TAG_COMP_BATCH || cur_hctx != rq->mq_hctx) {
			if (cur_hctx)
				blk_mq_flush_tag_batch(cur_hctx, tags, nr_tags);
			nr_tags = 0;
			cur_hctx = rq->mq_hctx;
		}
		tags[nr_tags++] = rq->tag;
    ...
}

Requests freed through blk_mq_end_request_batch(), requests reallocated
or cached in a plug via blk_mq_rq_ctx_init() (still in MQ_RQ_IDLE before
blk_mq_start_request()), and requests already in MQ_RQ_COMPLETE awaiting
remote IPI or softirq completion all have a non-NULL rq->mq_hctx.  A
spurious or duplicate CQE matching one of those requests bypasses the
!rq->mq_hctx check and completes the request a second time (causing
double DMA unmapping in nvme_pci_unmap_rq(), refcount and sbitmap tag
underflow, or cyclic corruption of the per-CPU blk_cpu_done llist if
rq->ipi_list is queued twice).

Second, because __blk_mq_free_request() clears rq->mq_hctx without
synchronization against a spurious completion on another CPU, the unlocked
check still races:

CPU0                                  CPU1
blk_mq_free_request(rq)
  WRITE_ONCE(rq->state, MQ_RQ_IDLE)
                                      blk_mq_complete_request_remote(rq)
                                        if (unlikely(!rq->mq_hctx))
                                          /* observes non-NULL */
                                        WRITE_ONCE(rq->state, MQ_RQ_COMPLETE)
  __blk_mq_free_request(rq)
    rq->mq_hctx = NULL
                                        if (rq->mq_hctx->nr_ctx == 1 ...)
                                          <-- NULL pointer dereference

In addition to crashing at rq->mq_hctx->nr_ctx, CPU1 overwrites the freed
request's rq->state back to MQ_RQ_COMPLETE after CPU0 set it to
MQ_RQ_IDLE.

[Severity: Medium]
Should stale completions be rejected earlier in nvme_find_rq() by checking
that blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT (similar to
scsi_host_find_tag() and nbd_read_reply())?

When nvme_find_rq() returns a request whose 4-bit genctr collided with a
corrupted CQE, callers act on that request before
blk_mq_complete_request_remote() is reached:

drivers/nvme/host/pci.c:nvme_handle_cqe() {
    ...
	trace_nvme_sq(req, cqe->sq_head, data_race(nvmeq->sq_tail));
	if (!nvme_try_complete_req(req, cqe->status, cqe->result) &&
    ...
}

If the nvme_sq tracepoint is enabled, trace_nvme_sq() calls
nvme_req_qid(req), which dereferences req->mq_hctx->queue_num and hits the
same NULL pointer dereference when req->mq_hctx is NULL.  Next,
nvme_try_complete_req() increments nvme_req(req)->genctr and overwrites
status and result before calling blk_mq_complete_request_remote(), which
corrupts the request if it has already been reallocated or is awaiting
remote completion and causes its real completion to fail genctr
validation.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261119-fix-blk-mq-null-ptr-deref-v1-1-9bcbcfcfaff8@csail.mit.edu?part=1

  reply	other threads:[~2026-11-18 19:57 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
2026-11-18 19:57 ` sashiko-bot [this message]
2026-11-18 20:12 ` Jens Axboe
2026-11-18 20:53 ` Hamza Mahfooz
2026-11-18 21:43   ` Keith Busch

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-221909@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=srivatsa@csail.mit.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox