[PATCH] nvme: avoid missing db ring during reset

[PATCH] nvme: avoid missing db ring during reset

[Linjun Bao]

During nvme reset, there is a rare case, when user admin cmd such
as smart-log and nvme_admin_create_sq from nvme_setup_io_queues
happen to in the same blk_mq dispatch list, and the user cmd is
the last one. nvme_admin_create_sq is dispatched first in
nvme_queue_rq(), nvme_write_sq_db() is called but immediately
returns without writing the doorbell because it's not masked
"last". The subsequent smart-log ioctl fails fast hitting
nvme_fail_nonready_cmd(), skipping both nvme_sq_copy_cmd() and
nvme_write_sq_db(), so no doorbell write ever occurs. The
nvme_admin_create_sq fails timeout finally.

The proposal is that do not treat user admin cmd during
RECONNECTING as non-ready, through it to the drive, thus no
doorbell missing happens in case above.

Signed-off-by: Linjun Bao <meljbao@gmail.com>
---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3cc79817e4d7..fc550226ed77 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -750,7 +750,8 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 	 * sequence. until the controller will be LIVE, fail with
 	 * BLK_STS_RESOURCE so that they will be rescheduled.
 	 */
-	if (rq->q == ctrl->admin_q && (req->flags & NVME_REQ_USERCMD))
+	if (rq->q == ctrl->admin_q && (req->flags & NVME_REQ_USERCMD) &&
+	    (nvme_ctrl_state(ctrl) != NVME_CTRL_CONNECTING))
 		return false;
 
 	if (ctrl->ops->flags & NVME_F_FABRICS) {
-- 
2.25.1

Re: [PATCH] nvme: avoid missing db ring during reset

[Keith Busch]

On Fri, Apr 25, 2025 at 08:01:45PM +0800, Linjun Bao wrote:
> During nvme reset, there is a rare case, when user admin cmd such
> as smart-log and nvme_admin_create_sq from nvme_setup_io_queues
> happen to in the same blk_mq dispatch list, and the user cmd is
> the last one. nvme_admin_create_sq is dispatched first in
> nvme_queue_rq(), nvme_write_sq_db() is called but immediately
> returns without writing the doorbell because it's not masked
> "last". The subsequent smart-log ioctl fails fast hitting
> nvme_fail_nonready_cmd(), skipping both nvme_sq_copy_cmd() and
> nvme_write_sq_db(), so no doorbell write ever occurs. The
> nvme_admin_create_sq fails timeout finally.

The block layer is supposed to call the driver's commit_rqs() function
if anything in the dispatch list wasn't successful, which should notify
the controller of any pending SQEs. Is that not happening here?

Re: [PATCH] nvme: avoid missing db ring during reset

[Linjun Bao]

On 4/25/2025 11:48 PM, Keith Busch wrote:
> On Fri, Apr 25, 2025 at 08:01:45PM +0800, Linjun Bao wrote:
>> During nvme reset, there is a rare case, when user admin cmd such
>> as smart-log and nvme_admin_create_sq from nvme_setup_io_queues
>> happen to in the same blk_mq dispatch list, and the user cmd is
>> the last one. nvme_admin_create_sq is dispatched first in
>> nvme_queue_rq(), nvme_write_sq_db() is called but immediately
>> returns without writing the doorbell because it's not masked
>> "last". The subsequent smart-log ioctl fails fast hitting
>> nvme_fail_nonready_cmd(), skipping both nvme_sq_copy_cmd() and
>> nvme_write_sq_db(), so no doorbell write ever occurs. The
>> nvme_admin_create_sq fails timeout finally.
> 
> The block layer is supposed to call the driver's commit_rqs() function
> if anything in the dispatch list wasn't successful, which should notify
> the controller of any pending SQEs. Is that not happening here?

Yes, in this case, the last user admin cmd will fail nvme_host_path_errror
finally, but ret BLK_STS_OK, which will let blk_mq_dispatch_rq_list skips 
the commit_rqs, thus missing updating the SQ doorbell.