* [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
@ 2026-11-18 19:28 Srivatsa S. Bhat (Microsoft)
2026-11-18 20:12 ` Jens Axboe
2026-11-18 20:53 ` Hamza Mahfooz
0 siblings, 2 replies; 4+ messages in thread
From: Srivatsa S. Bhat (Microsoft) @ 2026-11-18 19:28 UTC (permalink / raw)
To: Jens Axboe, Daniel Wagner, Christoph Hellwig
Cc: linux-block, linux-kernel, linux-hyperv,
Srivatsa S. Bhat (Microsoft)
Add NULL pointer validation in blk_mq_complete_request_remote() to
prevent kernel crashes when block drivers pass freed request structures.
When NVMe devices experience hardware failures or firmware bugs,
completion queue corruption can occur, leading to scenarios where
drivers attempt to complete already-freed requests. The NVMe genctr
validation in nvme_find_rq() catches most stale completions, but when
corruption produces a command_id whose genctr bits happen to match,
the freed request passes validation and reaches the block layer. The
block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
but blk_mq_complete_request_remote() lacks validation before
dereferencing this pointer.
This manifests as a kernel crash like shown below (the stack trace
was obtained from a crash observed in a Linux VM on Hyper-V):
nvme nvme0: resetting controller due to AER
nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
nvme nvme0: invalid id 227 completed on queue 6
nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
nvme nvme0: invalid id 45094 completed on queue 94
BUG: kernel NULL pointer dereference, address: 00000000000000fe
RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
Call Trace:
<IRQ>
nvme_irq+0x182/0x320 [nvme]
__handle_irq_event_percpu+0x47/0x180
handle_irq_event+0x38/0x60
handle_edge_irq+0x87/0x1a0
__common_interrupt+0x3f/0xa0
common_interrupt+0x42/0xa0
The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
where mq_hctx is NULL.
Add a NULL check that returns true (signaling to callers that the
request has been handled) to gracefully drop freed requests, with a
WARN_ONCE to flag the condition for diagnosis.
Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>
---
I'd like to call out a few unresolved aspects about this patch for
review attention:
1. I'm not sure if the NULL ptr check should be placed before or after
the rq->state update to MQ_RQ_COMPLETE. Do we need to update that
state even if we don't proceed with the complete-request operation?
2. While this was a real bug that I hit in one of my systems, I don't
have a way to repro that same scenario anymore (i.e., hardware/firmware
bug that introduces the corruption), so validating this kernel fix
against the original crash has been a bit of a challenge.
Still, I wanted to post this patch to report the bug and seek guidance
on how best to fix it and harden blk-mq against this NULL ptr dereference.
Thank you!
---
block/blk-mq.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 5ae22d556540..5dc9d7d806b5 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
bool blk_mq_complete_request_remote(struct request *rq)
{
+ /*
+ * Validate request is not freed (__blk_mq_free_request() sets
+ * mq_hctx = NULL).
+ */
+ if (unlikely(!rq->mq_hctx)) {
+ WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
+ return true;
+ }
+
WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
/*
---
base-commit: 20eef29b7c8465ac98d7dd6d6f2ac07a8d8056ca
change-id: 20261118-fix-blk-mq-null-ptr-deref-7f376cd5ad90
Regards,
Srivatsa
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
@ 2026-11-18 20:12 ` Jens Axboe
2026-11-18 20:53 ` Hamza Mahfooz
1 sibling, 0 replies; 4+ messages in thread
From: Jens Axboe @ 2026-11-18 20:12 UTC (permalink / raw)
To: Srivatsa S. Bhat (Microsoft), Daniel Wagner, Christoph Hellwig
Cc: linux-block, linux-kernel, linux-hyperv
On 11/18/26 12:28 PM, Srivatsa S. Bhat (Microsoft) wrote:
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.
Please stop this nonsense.
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
2026-11-18 20:12 ` Jens Axboe
@ 2026-11-18 20:53 ` Hamza Mahfooz
2026-11-18 21:43 ` Keith Busch
1 sibling, 1 reply; 4+ messages in thread
From: Hamza Mahfooz @ 2026-11-18 20:53 UTC (permalink / raw)
To: Srivatsa S. Bhat (Microsoft)
Cc: Jens Axboe, Daniel Wagner, Christoph Hellwig, linux-block,
linux-kernel, linux-hyperv
On Thu, Nov 19, 2026 at 12:58:44AM +0530, Srivatsa S. Bhat (Microsoft) wrote:
> Add NULL pointer validation in blk_mq_complete_request_remote() to
> prevent kernel crashes when block drivers pass freed request structures.
>
This looks a lot like a driver bug, so it's probably best to hunt that
down instead of trying to work around it here.
> When NVMe devices experience hardware failures or firmware bugs,
> completion queue corruption can occur, leading to scenarios where
> drivers attempt to complete already-freed requests. The NVMe genctr
> validation in nvme_find_rq() catches most stale completions, but when
> corruption produces a command_id whose genctr bits happen to match,
> the freed request passes validation and reaches the block layer. The
> block layer's __blk_mq_free_request() sets rq->mq_hctx to NULL,
> but blk_mq_complete_request_remote() lacks validation before
> dereferencing this pointer.
>
> This manifests as a kernel crash like shown below (the stack trace
> was obtained from a crash observed in a Linux VM on Hyper-V):
>
> nvme nvme0: resetting controller due to AER
> nvme nvme0: request 0xe3 genctr mismatch (got 0x0 expected 0x5)
> nvme nvme0: invalid id 227 completed on queue 6
> nvme nvme0: request 0x26 genctr mismatch (got 0xb expected 0x9)
> nvme nvme0: invalid id 45094 completed on queue 94
>
> BUG: kernel NULL pointer dereference, address: 00000000000000fe
> RIP: 0010:blk_mq_complete_request_remote+0x1b/0x150
> Call Trace:
> <IRQ>
> nvme_irq+0x182/0x320 [nvme]
> __handle_irq_event_percpu+0x47/0x180
> handle_irq_event+0x38/0x60
> handle_edge_irq+0x87/0x1a0
> __common_interrupt+0x3f/0xa0
> common_interrupt+0x42/0xa0
>
> The crash occurs at offset 0xfe when accessing rq->mq_hctx->nr_ctx,
> where mq_hctx is NULL.
>
> Add a NULL check that returns true (signaling to callers that the
> request has been handled) to gracefully drop freed requests, with a
> WARN_ONCE to flag the condition for diagnosis.
>
> Fixes: 40d09b53bfc5 ("blk-mq: add a new blk_mq_complete_request_remote API")
> Signed-off-by: Srivatsa S. Bhat (Microsoft) <srivatsa@csail.mit.edu>
> ---
> I'd like to call out a few unresolved aspects about this patch for
> review attention:
>
> 1. I'm not sure if the NULL ptr check should be placed before or after
> the rq->state update to MQ_RQ_COMPLETE. Do we need to update that
> state even if we don't proceed with the complete-request operation?
>
> 2. While this was a real bug that I hit in one of my systems, I don't
> have a way to repro that same scenario anymore (i.e., hardware/firmware
> bug that introduces the corruption), so validating this kernel fix
> against the original crash has been a bit of a challenge.
>
> Still, I wanted to post this patch to report the bug and seek guidance
> on how best to fix it and harden blk-mq against this NULL ptr dereference.
>
> Thank you!
> ---
> block/blk-mq.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 5ae22d556540..5dc9d7d806b5 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -1307,6 +1307,15 @@ static void blk_mq_raise_softirq(struct request *rq)
>
> bool blk_mq_complete_request_remote(struct request *rq)
> {
> + /*
> + * Validate request is not freed (__blk_mq_free_request() sets
> + * mq_hctx = NULL).
> + */
> + if (unlikely(!rq->mq_hctx)) {
> + WARN_ONCE(1, "%s: attempt to complete freed request", __func__);
> + return true;
> + }
> +
> WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
>
> /*
>
> ---
> base-commit: 20eef29b7c8465ac98d7dd6d6f2ac07a8d8056ca
> change-id: 20261118-fix-blk-mq-null-ptr-deref-7f376cd5ad90
>
> Regards,
> Srivatsa
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote
2026-11-18 20:53 ` Hamza Mahfooz
@ 2026-11-18 21:43 ` Keith Busch
0 siblings, 0 replies; 4+ messages in thread
From: Keith Busch @ 2026-11-18 21:43 UTC (permalink / raw)
To: Hamza Mahfooz
Cc: Srivatsa S. Bhat (Microsoft), Jens Axboe, Daniel Wagner,
Christoph Hellwig, linux-block, linux-kernel, linux-hyperv
On Wed, Nov 18, 2026 at 03:53:31PM -0500, Hamza Mahfooz wrote:
> On Thu, Nov 19, 2026 at 12:58:44AM +0530, Srivatsa S. Bhat (Microsoft) wrote:
> > Add NULL pointer validation in blk_mq_complete_request_remote() to
> > prevent kernel crashes when block drivers pass freed request structures.
> >
>
> This looks a lot like a driver bug, so it's probably best to hunt that
> down instead of trying to work around it here.
It would be better to go another level lower and fix the broken
hardware. The "bug" is describing device behavior too far outside
protocol to be usable.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-18 21:43 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 19:28 [PATCH] block: Fix NULL pointer dereference in blk_mq_complete_request_remote Srivatsa S. Bhat (Microsoft)
2026-11-18 20:12 ` Jens Axboe
2026-11-18 20:53 ` Hamza Mahfooz
2026-11-18 21:43 ` Keith Busch
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox