Linux-HyperV List
 help / color / mirror / Atom feed
* [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
@ 2026-11-18 19:28 Srivatsa S. Bhat (Microsoft)
  2026-11-18 19:57 ` sashiko-bot
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Srivatsa S. Bhat (Microsoft) @ 2026-11-18 19:28 UTC (permalink / raw)
  To: Jens Axboe, Daniel Wagner, Christoph Hellwig
  Cc: linux-block, linux-kernel, linux-hyperv,
	Srivatsa S. Bhat (Microsoft)

Add NULL pointer validation in blk_mq_complete_request_remote() to
prevent kernel crashes when block drivers pass freed request structures.

When NVMe devices experience hardware failures or firmware bugs,
completion queue corruption can occur, leading to scenarios where
drivers attempt to complete already-freed requests. The NVMe genctr
validation in nvme_find_rq() catches most stale completions, but when
corruption produces a command_id whose genctr bits happen to match,
the freed request passes validation and reaches the block layer. The
block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
but blk_mq_complete_request_remote() lacks validation before
dereferencing this pointer.

This manifests as a kernel crash like shown below (the stack trace
was obtained from a crash observed in a Linux VM on Hyper-V):

  nvme nvme0: resetting controller due to AER
  nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
  nvme nvme0: invalid id 227 completed on queue 6
  nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
  nvme nvme0: invalid id 45094 completed on queue 94

  BUG: kernel NULL pointer dereference, address: 00000000000000fe
  RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
  Call Trace:
   <IRQ>
   nvme_irq+0x182/0x320 [nvme]
   __handle_irq_event_percpu+0x47/0x180
   handle_irq_event+0x38/0x60
   handle_edge_irq+0x87/0x1a0
   __common_interrupt+0x3f/0xa0
   common_interrupt+0x42/0xa0

The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
where mq_hctx is NULL.

Add a NULL check that returns true (signaling to callers that the
request has been handled) to gracefully drop freed requests, with a
WARN_ONCE to flag the condition for diagnosis.

Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>
---
I'd like to call out a few unresolved aspects about this patch for
review attention:

1. I'm not sure if the NULL ptr check should be placed before or after
the rq->state update to MQ_RQ_COMPLETE. Do we need to update that
state even if we don't proceed with the complete-request operation?

2. While this was a real bug that I hit in one of my systems, I don't
have a way to repro that same scenario anymore (i.e., hardware/firmware
bug that introduces the corruption), so validating this kernel fix
against the original crash has been a bit of a challenge.

Still, I wanted to post this patch to report the bug and seek guidance
on how best to fix it and harden blk-mq against this NULL ptr dereference.

Thank you!
---
 block/blk-mq.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 5ae22d556540..5dc9d7d806b5 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
 
 bool blk_mq_complete_request_remote(struct request *rq)
 {
+	/*
+	 * Validate request is not freed (__blk_mq_free_request() sets
+	 * mq_hctx = NULL).
+	 */
+	if (unlikely(!rq->mq_hctx)) {
+		WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
+		return true;
+	}
+
 	WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
 
 	/*

---
base-commit: 20eef29b7c8465ac98d7dd6d6f2ac07a8d8056ca
change-id: 20261118-fix-blk-mq-null-ptr-deref-7f376cd5ad90

Regards,
Srivatsa


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
  2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
@ 2026-11-18 19:57 ` sashiko-bot
  2026-11-18 20:12 ` Jens Axboe
  2026-11-18 20:53 ` Hamza Mahfooz
  2 siblings, 0 replies; 4+ messages in thread
From: sashiko-bot @ 2026-11-18 19:57 UTC (permalink / raw)
  To: Srivatsa S. Bhat (Microsoft); +Cc: linux-hyperv

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] block: incomplete and racy freed-request check in
  blk_mq_complete_request_remote()
- [Medium] nvme: missing in-flight state check in nvme_find_rq() allows
  prior crash and corruption
- [Low] block: inaccurate Fixes: trailer in commit message
--

commit 65327008502cbbac3ada696570a4b5ff52feb917
Author: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>

> block: Fix NULL pointer dereference in blk_mq_complete_request_remote
>
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.
>
> When NVMe devices experience hardware failures or firmware bugs,
> completion queue corruption can occur, leading to scenarios where
> drivers attempt to complete already-freed requests. The NVMe genctr
> validation in nvme_find_rq() catches most stale completions, but when
> corruption produces a command_id whose genctr bits happen to match,
> the freed request passes validation and reaches the block layer. The
> block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
> but blk_mq_complete_request_remote() lacks validation before
> dereferencing this pointer.
>
> This manifests as a kernel crash like shown below (the stack trace
> was obtained from a crash observed in a Linux VM on Hyper-V):
>
>   nvme nvme0: resetting controller due to AER
>   nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
>   nvme nvme0: invalid id 227 completed on queue 6
>   nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
>   nvme nvme0: invalid id 45094 completed on queue 94
>
>   BUG: kernel NULL pointer dereference, address: 00000000000000fe
>   RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
>   Call Trace:
>    <IRQ>
>    nvme_irq+0x182/0x320 [nvme]
>    __handle_irq_event_percpu+0x47/0x180
>    handle_irq_event+0x38/0x60
>    handle_edge_irq+0x87/0x1a0
>    __common_interrupt+0x3f/0xa0
>    common_interrupt+0x42/0xa0
>
> The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
> where mq_hctx is NULL.
>
> Add a NULL check that returns true (signaling to callers that the
> request has been handled) to gracefully drop freed requests, with a
> WARN_ONCE to flag the condition for diagnosis.
>
> Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
> Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>

[Severity: Low]
In commit 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API"),
blk_mq_complete_request_remote() did not dereference rq->mq_hctx.  The
rq->mq_hctx->nr_ctx dereference was added later, so should the Fixes tag
instead be:

  Fixes: f168420c62e7 ("blk-mq: don't redirect completion for hctx withs only one ctx mapping")

> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 5ae22d5565404..5dc9d7d806b59 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
>  
>  bool blk_mq_complete_request_remote(struct request *rq)
>  {
> +	/*
> +	 * Validate request is not freed (__blk_mq_free_request() sets
> +	 * mq_hctx = NULL).
> +	 */
> +	if (unlikely(!rq->mq_hctx)) {
> +		WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
> +		return true;
> +	}
> +
>  	WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
>  
>  	/*

[Severity: High]
Is checking !rq->mq_hctx sufficient and race-free for detecting requests
that are not in flight?

First, the batched completion fast path used by nvme-pci frees requests
without ever clearing rq->mq_hctx:

block/blk-mq.c:blk_mq_end_request_batch() {
    ...
		WRITE_ONCE(rq->state, MQ_RQ_IDLE);
		if (!req_ref_put_and_test(rq))
			continue;

		blk_crypto_free_request(rq);
		blk_pm_mark_last_busy(rq);

		if (nr_tags == TAG_COMP_BATCH || cur_hctx != rq->mq_hctx) {
			if (cur_hctx)
				blk_mq_flush_tag_batch(cur_hctx, tags, nr_tags);
			nr_tags = 0;
			cur_hctx = rq->mq_hctx;
		}
		tags[nr_tags++] = rq->tag;
    ...
}

Requests freed through blk_mq_end_request_batch(), requests reallocated
or cached in a plug via blk_mq_rq_ctx_init() (still in MQ_RQ_IDLE before
blk_mq_start_request()), and requests already in MQ_RQ_COMPLETE awaiting
remote IPI or softirq completion all have a non-NULL rq->mq_hctx.  A
spurious or duplicate CQE matching one of those requests bypasses the
!rq->mq_hctx check and completes the request a second time (causing
double DMA unmapping in nvme_pci_unmap_rq(), refcount and sbitmap tag
underflow, or cyclic corruption of the per-CPU blk_cpu_done llist if
rq->ipi_list is queued twice).

Second, because __blk_mq_free_request() clears rq->mq_hctx without
synchronization against a spurious completion on another CPU, the unlocked
check still races:

CPU0                                  CPU1
blk_mq_free_request(rq)
  WRITE_ONCE(rq->state, MQ_RQ_IDLE)
                                      blk_mq_complete_request_remote(rq)
                                        if (unlikely(!rq->mq_hctx))
                                          /* observes non-NULL */
                                        WRITE_ONCE(rq->state, MQ_RQ_COMPLETE)
  __blk_mq_free_request(rq)
    rq->mq_hctx = NULL
                                        if (rq->mq_hctx->nr_ctx == 1 ...)
                                          <-- NULL pointer dereference

In addition to crashing at rq->mq_hctx->nr_ctx, CPU1 overwrites the freed
request's rq->state back to MQ_RQ_COMPLETE after CPU0 set it to
MQ_RQ_IDLE.

[Severity: Medium]
Should stale completions be rejected earlier in nvme_find_rq() by checking
that blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT (similar to
scsi_host_find_tag() and nbd_read_reply())?

When nvme_find_rq() returns a request whose 4-bit genctr collided with a
corrupted CQE, callers act on that request before
blk_mq_complete_request_remote() is reached:

drivers/nvme/host/pci.c:nvme_handle_cqe() {
    ...
	trace_nvme_sq(req, cqe->sq_head, data_race(nvmeq->sq_tail));
	if (!nvme_try_complete_req(req, cqe->status, cqe->result) &&
    ...
}

If the nvme_sq tracepoint is enabled, trace_nvme_sq() calls
nvme_req_qid(req), which dereferences req->mq_hctx->queue_num and hits the
same NULL pointer dereference when req->mq_hctx is NULL.  Next,
nvme_try_complete_req() increments nvme_req(req)->genctr and overwrites
status and result before calling blk_mq_complete_request_remote(), which
corrupts the request if it has already been reallocated or is awaiting
remote completion and causes its real completion to fail genctr
validation.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261119-fix-blk-mq-null-ptr-deref-v1-1-9bcbcfcfaff8@csail.mit.edu?part=1

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
  2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
  2026-11-18 19:57 ` sashiko-bot
@ 2026-11-18 20:12 ` Jens Axboe
  2026-11-18 20:53 ` Hamza Mahfooz
  2 siblings, 0 replies; 4+ messages in thread
From: Jens Axboe @ 2026-11-18 20:12 UTC (permalink / raw)
  To: Srivatsa S. Bhat (Microsoft), Daniel Wagner, Christoph Hellwig
  Cc: linux-block, linux-kernel, linux-hyperv

On 11/18/26 12:28 PM, Srivatsa S. Bhat (Microsoft) wrote:
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.

Please stop this nonsense.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
  2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
  2026-11-18 19:57 ` sashiko-bot
  2026-11-18 20:12 ` Jens Axboe
@ 2026-11-18 20:53 ` Hamza Mahfooz
  2 siblings, 0 replies; 4+ messages in thread
From: Hamza Mahfooz @ 2026-11-18 20:53 UTC (permalink / raw)
  To: Srivatsa S. Bhat (Microsoft)
  Cc: Jens Axboe, Daniel Wagner, Christoph Hellwig, linux-block,
	linux-kernel, linux-hyperv

On Thu, Nov 19, 2026 at 12:58:44AM +0530, Srivatsa S. Bhat (Microsoft) wrote:
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.
> 

This looks a lot like a driver bug, so it's probably best to hunt that
down instead of trying to work around it here.

> When NVMe devices experience hardware failures or firmware bugs,
> completion queue corruption can occur, leading to scenarios where
> drivers attempt to complete already-freed requests. The NVMe genctr
> validation in nvme_find_rq() catches most stale completions, but when
> corruption produces a command_id whose genctr bits happen to match,
> the freed request passes validation and reaches the block layer. The
> block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
> but blk_mq_complete_request_remote() lacks validation before
> dereferencing this pointer.
> 
> This manifests as a kernel crash like shown below (the stack trace
> was obtained from a crash observed in a Linux VM on Hyper-V):
> 
>   nvme nvme0: resetting controller due to AER
>   nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
>   nvme nvme0: invalid id 227 completed on queue 6
>   nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
>   nvme nvme0: invalid id 45094 completed on queue 94
> 
>   BUG: kernel NULL pointer dereference, address: 00000000000000fe
>   RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
>   Call Trace:
>    <IRQ>
>    nvme_irq+0x182/0x320 [nvme]
>    __handle_irq_event_percpu+0x47/0x180
>    handle_irq_event+0x38/0x60
>    handle_edge_irq+0x87/0x1a0
>    __common_interrupt+0x3f/0xa0
>    common_interrupt+0x42/0xa0
> 
> The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
> where mq_hctx is NULL.
> 
> Add a NULL check that returns true (signaling to callers that the
> request has been handled) to gracefully drop freed requests, with a
> WARN_ONCE to flag the condition for diagnosis.
> 
> Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
> Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>
> ---
> I'd like to call out a few unresolved aspects about this patch for
> review attention:
> 
> 1. I'm not sure if the NULL ptr check should be placed before or after
> the rq->state update to MQ_RQ_COMPLETE. Do we need to update that
> state even if we don't proceed with the complete-request operation?
> 
> 2. While this was a real bug that I hit in one of my systems, I don't
> have a way to repro that same scenario anymore (i.e., hardware/firmware
> bug that introduces the corruption), so validating this kernel fix
> against the original crash has been a bit of a challenge.
> 
> Still, I wanted to post this patch to report the bug and seek guidance
> on how best to fix it and harden blk-mq against this NULL ptr dereference.
> 
> Thank you!
> ---
>  block/blk-mq.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 5ae22d556540..5dc9d7d806b5 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
>  
>  bool blk_mq_complete_request_remote(struct request *rq)
>  {
> +	/*
> +	 * Validate request is not freed (__blk_mq_free_request() sets
> +	 * mq_hctx = NULL).
> +	 */
> +	if (unlikely(!rq->mq_hctx)) {
> +		WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
> +		return true;
> +	}
> +
>  	WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
>  
>  	/*
> 
> ---
> base-commit: 20eef29b7c8465ac98d7dd6d6f2ac07a8d8056ca
> change-id: 20261118-fix-blk-mq-null-ptr-deref-7f376cd5ad90
> 
> Regards,
> Srivatsa
> 

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-18 20:53 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
2026-11-18 19:57 ` sashiko-bot
2026-11-18 20:12 ` Jens Axboe
2026-11-18 20:53 ` Hamza Mahfooz

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox