From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECD0136896D for ; Sun, 12 Jul 2026 02:25:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783823141; cv=none; b=dIteH78qAmsw9XSVopKTHJoXI9NGk8pQPy99rF59V+wfWFQGZ+tdFM0jDagydvm5pMV6hWL9SAyleQ4dKT8legHlsjOdi2J80W+fxn2C24EUXNXBNAE1s+XTOqWJidHcoIV4Vn0a+/P8thW8M+WxGTrCARJOBORCKg+dhGdFD/k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783823141; c=relaxed/simple; bh=k9+W91WgE7B0Ip0L9YWDXIyQ4eak1hWpaGk445QcIHk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sICPo8J9cH2Zp2UHJ37ZhpG1WCqKStbfBLcHrFhYAOGU11cEjsK0TJ0e9ATMK0gGx6cmx3QqqDYXVUTWfSc0Eo7MoHXWXP0FVrRCh+kHRUTDFZDS/nC4+ZiOGM0QDVN01uSBHVC4SKF8ahmkMdALxefoeCXHGhdBJAS0EljKgtA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=purestorage.com; spf=pass smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=AbsY92n0; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="AbsY92n0" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-384c94c9414so1998693a91.3 for ; Sat, 11 Jul 2026 19:25:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1783823138; x=1784427938; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=k17NZK3iZiYWXEcv1S1Cll44Lxae4D1042zuaqAzmEs=; b=AbsY92n0fhLtmmZAZKkCCF1m+xrEzV1pv4FyOp7dMQiDNsVPKff04jgu6KWcNLoJcC Cide8Z5ehYWrSxf++5U/aOIAlepk8anYPXCewQRrccbNil8o2TtgvzP51KkLMduVc/MQ 0TvLM6ikThq9TvQuXsslKvZQynlnjJsuIRu/jw54vjIsfyrR/ju66m8ctI8U6eYntVee IbmJD0rNH84brHd3ca2z5bZFeGL/RlayrbZghwUIfkxDPKAq3NtS4YAo9Ia3zg5paHTm nxDCtqpTlkvaG+zgZjcEUrdsA0ioIwQC4DyjO9wN0QOcGjn4iHbM1T3qy/dnQesj+rOD NIOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783823138; x=1784427938; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=k17NZK3iZiYWXEcv1S1Cll44Lxae4D1042zuaqAzmEs=; b=rA4VoheXvsbPQ2FDwJTxrAmw72Bm0lEMbc7ev8J+llH4pZmlzGfUcRHCev2DJOLHRn ipdGopmW8MtsreY47k1yg3EC/kLOyNwp+K55eoexlnubOnUZlGpvYjUSEjIKjzNOH2Ro v6pTvnJJz2Hlv+7EnD73RD40gbv+3oYkBePmVF7Em1uJwoQ27UZkTVLI//8T3Jg1WHck jFwv/n/LeHWUD5ODAwxyAPUNdXzXY/N5hVwUkyeFuRNqfix20SBImf2snR2/6wQssjKv mMFrx7j6zMYwvcAwWVn1JBJY3aCVFmj+njXC8mdn7W7ZOGEXg9McaAIGtnHIKAbaGp/i Yc1A== X-Forwarded-Encrypted: i=1; AHgh+Rqdm0WuxwWDIYyV5XRUwsdPUHha57OQaAbIy/JD3ZNj6C92LSZBMsR47VH2yt0n71ecN2z7Kb4SskbfgqA=@vger.kernel.org X-Gm-Message-State: AOJu0YyTPmLZbm4Wh6Ic3nsetQGGfhP/1OWsgwZRob52C/Tyw0652qTR t2CjG289Yjy590QGccdQFNH0Q1a+g4FgKBltmGuuMot6ZQrYTpFHSFFa70tAxNw78Qo= X-Gm-Gg: AfdE7cnQ/252t4BLWZ1+C1n8sR4AeaRTljzpQJU4WSRZIQ1iZZhSkUNoPVq95zJCVYZ hLkhBotoluYHIQbaWaRG+xUygMfspbxo5ri0r3y1yQBuoQ4euLFmxQiKhO9Q0eCW76VFiKs56F0 9ln5Q/mc0U+OzFMj9gi1OmLg3zG1aDwkmqOmIrkx324nfQw2T6MpCmlSvTr2aUVsIxnGioRDzQe AMV3woGNWfDk8Mihy56nySwrvGVtknfgZoxPYcbuJqAOxKR/L/ZQQ0lo/cxWTSniChIW5wceTha zbW06FLsocrwlsZTI7jLLiajv8PdqVnfiBrlAWvGGeDKuCbarMWICsXdxylL69Ev431OQQc+RUu 7scsKMsNveHaR6bBbxnMEp4op++R92ecUPuxW9ivh/DxrHO+NCkMpKNmbB9JC+5r270IohFZXAT 3vfSb6 X-Received: by 2002:a17:90b:5870:b0:37f:a915:1c29 with SMTP id 98e67ed59e1d1-38dc75faef0mr3582950a91.19.1783823138013; Sat, 11 Jul 2026 19:25:38 -0700 (PDT) Received: from ceto.lan ([2607:fb90:9c20:2264::1c31]) by smtp.googlemail.com with ESMTPSA id 5a478bee46e88-3117462f5c7sm60704693eec.0.2026.07.11.19.25.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 11 Jul 2026 19:25:37 -0700 (PDT) From: Mohamed Khalfella To: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , James Smart , Hannes Reinecke , Randy Jennings , Dhaval Giani Cc: Aaron Dailey , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Mohamed Khalfella Subject: [PATCH v5 12/16] nvme-fc: Refactor IO error recovery Date: Sat, 11 Jul 2026 19:23:33 -0700 Message-ID: <20260712022437.3743117-13-mkhalfella@purestorage.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260712022437.3743117-1-mkhalfella@purestorage.com> References: <20260712022437.3743117-1-mkhalfella@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Added new nvme_fc_start_ioerr_recovery() to trigger error recovery instead of directly queueing ctrl->ioerr_work. nvme_fc_error_recovery() now called only from ctrl->ioerr_work has been updated to not depend on nvme_reset_ctrl() to handle error recovery. nvme_fc_error_recovery() effectively resets the controller and attempts reconnection if needed. This makes nvme-fc ioerr handling similar to other fabric transports. Update nvme_fc_timeout() to not abort timed out IOs. IOs aborted from nvme_fc_timeout() are not accounted for in ctrl->iocnt and this causes nvme_fc_delete_association() not to wait for them. Instead of aborting IOs nvme_fc_timeout() calls nvme_fc_start_ioerr_recovery() to start IO error recovery. Since error recovery runs in ctrl->ioerr_work this change fixes the issue reported in the link below. Link: https://lore.kernel.org/all/20250529214928.2112990-1-mkhalfella@purestorage.com/ Signed-off-by: Mohamed Khalfella --- drivers/nvme/host/fc.c | 119 +++++++++++++++++++++++------------------ 1 file changed, 66 insertions(+), 53 deletions(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 04363b9c4489..54a09db9716d 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -227,6 +227,10 @@ static DEFINE_IDA(nvme_fc_ctrl_cnt); static struct device *fc_udev_device; static void nvme_fc_complete_rq(struct request *rq); +static void nvme_fc_start_ioerr_recovery(struct nvme_fc_ctrl *ctrl, + char *errmsg); +static void __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, + bool start_queues); /* *********************** FC-NVME Port Management ************************ */ @@ -788,7 +792,7 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl) "Reconnect", ctrl->cnum); set_bit(ASSOC_FAILED, &ctrl->flags); - nvme_reset_ctrl(&ctrl->ctrl); + nvme_fc_start_ioerr_recovery(ctrl, "Connectivity Loss"); } /** @@ -985,7 +989,7 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, static void nvme_fc_ctrl_put(struct nvme_fc_ctrl *); static int nvme_fc_ctrl_get(struct nvme_fc_ctrl *); -static void nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg); +static void nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl); static void __nvme_fc_finish_ls_req(struct nvmefc_ls_req_op *lsop) @@ -1567,9 +1571,8 @@ nvme_fc_ls_disconnect_assoc(struct nvmefc_ls_rcv_op *lsop) * for the association have been ABTS'd by * nvme_fc_delete_association(). */ - - /* fail the association */ - nvme_fc_error_recovery(ctrl, "Disconnect Association LS received"); + nvme_fc_start_ioerr_recovery(ctrl, + "Disconnect Association LS received"); /* release the reference taken by nvme_fc_match_disconn_ls() */ nvme_fc_ctrl_put(ctrl); @@ -1871,7 +1874,22 @@ nvme_fc_ctrl_ioerr_work(struct work_struct *work) struct nvme_fc_ctrl *ctrl = container_of(work, struct nvme_fc_ctrl, ioerr_work); - nvme_fc_error_recovery(ctrl, "transport detected io error"); + /* + * if an error (io timeout, etc) while (re)connecting, the remote + * port requested terminating of the association (disconnect_ls) + * or an error (timeout or abort) occurred on an io while creating + * the controller. Abort any ios on the association and let the + * create_association error path resolve things. + */ + if (nvme_ctrl_state(&ctrl->ctrl) == NVME_CTRL_CONNECTING) { + __nvme_fc_abort_outstanding_ios(ctrl, true); + dev_warn(ctrl->ctrl.device, + "NVME-FC{%d}: transport error during (re)connect\n", + ctrl->cnum); + return; + } + + nvme_fc_error_recovery(ctrl); } /* @@ -1892,6 +1910,24 @@ char *nvme_fc_io_getuuid(struct nvmefc_fcp_req *req) } EXPORT_SYMBOL_GPL(nvme_fc_io_getuuid); +static void nvme_fc_start_ioerr_recovery(struct nvme_fc_ctrl *ctrl, + char *errmsg) +{ + enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl); + + if (state == NVME_CTRL_CONNECTING || state == NVME_CTRL_DELETING || + state == NVME_CTRL_DELETING_NOIO) { + queue_work(nvme_reset_wq, &ctrl->ioerr_work); + return; + } + + if (nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING)) { + dev_warn(ctrl->ctrl.device, "NVME-FC{%d}: starting error recovery %s\n", + ctrl->cnum, errmsg); + queue_work(nvme_reset_wq, &ctrl->ioerr_work); + } +} + static void nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) { @@ -2049,9 +2085,8 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) nvme_fc_complete_rq(rq); check_error: - if (terminate_assoc && - nvme_ctrl_state(&ctrl->ctrl) != NVME_CTRL_RESETTING) - queue_work(nvme_reset_wq, &ctrl->ioerr_work); + if (terminate_assoc) + nvme_fc_start_ioerr_recovery(ctrl, "io error"); } static int @@ -2495,39 +2530,6 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues) nvme_unquiesce_admin_queue(&ctrl->ctrl); } -static void -nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg) -{ - enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl); - - /* - * if an error (io timeout, etc) while (re)connecting, the remote - * port requested terminating of the association (disconnect_ls) - * or an error (timeout or abort) occurred on an io while creating - * the controller. Abort any ios on the association and let the - * create_association error path resolve things. - */ - if (state == NVME_CTRL_CONNECTING) { - __nvme_fc_abort_outstanding_ios(ctrl, true); - dev_warn(ctrl->ctrl.device, - "NVME-FC{%d}: transport error during (re)connect\n", - ctrl->cnum); - return; - } - - /* Otherwise, only proceed if in LIVE state - e.g. on first error */ - if (state != NVME_CTRL_LIVE) - return; - - dev_warn(ctrl->ctrl.device, - "NVME-FC{%d}: transport association event: %s\n", - ctrl->cnum, errmsg); - dev_warn(ctrl->ctrl.device, - "NVME-FC{%d}: resetting controller\n", ctrl->cnum); - - nvme_reset_ctrl(&ctrl->ctrl); -} - static enum blk_eh_timer_return nvme_fc_timeout(struct request *rq) { struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(rq); @@ -2536,24 +2538,14 @@ static enum blk_eh_timer_return nvme_fc_timeout(struct request *rq) struct nvme_fc_cmd_iu *cmdiu = &op->cmd_iu; struct nvme_command *sqe = &cmdiu->sqe; - /* - * Attempt to abort the offending command. Command completion - * will detect the aborted io and will fail the connection. - */ dev_info(ctrl->ctrl.device, "NVME-FC{%d.%d}: io timeout: opcode %d fctype %d (%s) w10/11: " "x%08x/x%08x\n", ctrl->cnum, qnum, sqe->common.opcode, sqe->fabrics.fctype, nvme_fabrics_opcode_str(qnum, sqe), sqe->common.cdw10, sqe->common.cdw11); - if (__nvme_fc_abort_op(ctrl, op)) - nvme_fc_error_recovery(ctrl, "io timeout abort failed"); - /* - * the io abort has been initiated. Have the reset timer - * restarted and the abort completion will complete the io - * shortly. Avoids a synchronous wait while the abort finishes. - */ + nvme_fc_start_ioerr_recovery(ctrl, "io timeout"); return BLK_EH_RESET_TIMER; } @@ -3359,6 +3351,27 @@ nvme_fc_reset_ctrl_work(struct work_struct *work) } } +static void +nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl) +{ + nvme_stop_keep_alive(&ctrl->ctrl); + nvme_stop_ctrl(&ctrl->ctrl); + flush_work(&ctrl->ctrl.async_event_work); + + /* will block while waiting for io to terminate */ + nvme_fc_delete_association(ctrl); + + /* Do not reconnect if controller is being deleted */ + if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) + return; + + if (ctrl->rport->remoteport.port_state == FC_OBJSTATE_ONLINE) { + queue_delayed_work(nvme_wq, &ctrl->connect_work, 0); + return; + } + + nvme_fc_reconnect_or_delete(ctrl, -ENOTCONN); +} static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = { .name = "fc", -- 2.54.0