From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18D8610F92E3 for ; Tue, 31 Mar 2026 16:38:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pn0l+rWCRwk5JlaI6GKau18ILhConWqlWtc8SFEQbyc=; b=MP23gtst5dvJw19+O0hJzT81XN UVGHMHHO6Fannh9VWeQdeWy1mSx7C6FQvcJJrMnFGEi3VXku/rJQvylYaAU3XSBGdbwQMKmj5h3TJ q39bYxV0ZexMff5K2lIxvi7tjZvCNp9AQX1o560RDYHMZkTwXxVxxEDzIDCDDhMOkaD/0hASiSUFw rbUgmYpE/jKWUE6V1gz7d1TyR/Kdoa7OpjmdsTkurKkTIEaYHa4cdVcV66lzC4xRhrQQYngQDoo89 KNIv5t6SPGtgU9pbAo4JmhKalKf4YbeDAEKViqqcxTVrBEO40aHVezdtWmq2ZT3vzqklSKsjIKBTW MTJ2oq/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7c6W-0000000DGoU-11OK; Tue, 31 Mar 2026 16:38:12 +0000 Received: from mail-dy1-x132e.google.com ([2607:f8b0:4864:20::132e]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7c6T-0000000DGo9-2pPe for linux-nvme@lists.infradead.org; Tue, 31 Mar 2026 16:38:11 +0000 Received: by mail-dy1-x132e.google.com with SMTP id 5a478bee46e88-2c18af885c0so6550865eec.0 for ; Tue, 31 Mar 2026 09:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1774975088; x=1775579888; darn=lists.infradead.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=pn0l+rWCRwk5JlaI6GKau18ILhConWqlWtc8SFEQbyc=; b=U5XRtZ/DoL5AfTD63s2I4mwkcSEbXBvcjFEHQuse9OCFDjYd+D3HFuJIfQ/GTr/I/o D1dxVMoO0P0A+zYNJFTAa+AqfMOAG8zLClVEEMI+uaH0mLpWuV9Ll60+KfSLTdgsS3FR 384uuka0ANYmW38yCKQOoWIikXXZVKssYazhN2X4Byh1hGrg7GLhIx7K/Ck29qIO8lDf euBYpDnZGP4XLopfTO8RNa+MCaDUC+nJ3VbLYZvXj5lC2G5K1MjhRHqqRKyvwgdxOutq bVm527f6UCtm/XI1Y8X92MSUTDeuEl3lDhU55W27Z0yeVNUonMteOw4mAdSq9AOYbCKo e91Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774975088; x=1775579888; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pn0l+rWCRwk5JlaI6GKau18ILhConWqlWtc8SFEQbyc=; b=Udbp+byK2CYJopVpAgRLrfrAE1lpkjL3FURUN+i5m66OdHFUnVPhlCaMIXyWL0r5yC GsEWWrI3OQO/nEFTS0BH0xAPITQ29vfO++1hxP7TWAJCSbC0/oU/PJbC7Av8ho2EbYr4 I1XiFdzDJHGHFl2RCuXWHUIdVcPOy6goEs5wy7LNxTW6NWq5EQAJ/Q+PzLn8waBHv7HF rU+lDanjqOOXJo9p95vYqxSaSkmRQPrIE/m11G5p0yd1x+OgK0eFYTdMJvUtTO0VYCmx ORxR+xuU6YFDLY53ricT+LU6Ycg7/+SNXzp5DJ67P7RBF8vj15PV2SsIDhUeBR4EMwDA Dnrg== X-Forwarded-Encrypted: i=1; AJvYcCVwPxAvpxbSR7gE8VvUBN9TxaKS7+0ZlguvbmkjfoMYanfKjNfL5B0mZ8eWuM5VHfvETSDUv3qKvujn@lists.infradead.org X-Gm-Message-State: AOJu0YzflRk//RXOF4dHA5dWVSZrlm0gHjtpUTlUjeTfLEFQ0iq2NmpC aBQFTnfP0YMz/Qj8htjKYSx/r0M+tYhUUk8e+zS5ftqbvcdidoUkiBSJJ6c+fzgpFAM= X-Gm-Gg: ATEYQzy1Z2IPQkLS5jAzHuM8BdGWb+BuVmV1SSwzUZ/f5R6wYX8Qw6ItlfM6u+KqzZc 5jOLF22II4NNCCYPY6Vr9orq3Jj7Lq+X4F2Nxo3H+urnhB6cHq+LoKpBgKjul0/ojvnd1cTTIPx dnYqX0JQ3/Lcanr0XytozfVmbDRheJyNOhepqGs1viMTVB98DpAm7MyIyStu7hMWfMNvZUB2lIU 5RfEkUbT7rsOMxAHkdH9YHLgy8eRVtdjwHGLQ/9lRNIvvDGAQ9bQ/mHTV5HJVYhrdmaEH7lYAfL mABOB1/lzL/3avIXbRkxHWKD0BCOV6sslW0VJkIqdKiZ/h2yx3WNzfP3YHBv0IqwwBXkhOlaK8V fa3hp5dQZVCS0UHT74gwngrg74rJJWkR2jsu7pnXICb0wSTxX0C02uPt6yTJVP9XCQFxagJW+jE tS/Xftue4vddAhYPMxMrs6T9w1ECL3w/Th37NvkbS2EU2s X-Received: by 2002:a05:7300:ac90:b0:2c5:fb3f:70f0 with SMTP id 5a478bee46e88-2c5fb3f7427mr4741988eec.19.1774975088073; Tue, 31 Mar 2026 09:38:08 -0700 (PDT) Received: from medusa.lab.kspace.sh ([208.88.152.253]) by smtp.googlemail.com with UTF8SMTPSA id 5a478bee46e88-2c3c68b2ff2sm10662742eec.18.2026.03.31.09.38.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Mar 2026 09:38:07 -0700 (PDT) Date: Tue, 31 Mar 2026 09:38:06 -0700 From: Mohamed Khalfella To: Hannes Reinecke Cc: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Jens Axboe , Keith Busch , Sagi Grimberg , James Smart , Aaron Dailey , Randy Jennings , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v4 03/15] nvmet: Implement CCR nvme command Message-ID: <20260331163806.GB2861-mkhalfella@purestorage.com> References: <20260328004518.1729186-1-mkhalfella@purestorage.com> <20260328004518.1729186-4-mkhalfella@purestorage.com> <204893dd-d256-4f37-955a-1fa724048021@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <204893dd-d256-4f37-955a-1fa724048021@suse.de> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_093809_759410_1BE1769F X-CRM114-Status: GOOD ( 29.22 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org On Mon 2026-03-30 12:45:57 +0200, Hannes Reinecke wrote: > On 3/28/26 01:43, Mohamed Khalfella wrote: > > Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller > > Reset) command is an nvme command issued to source controller by > > initiator to reset impacted controller. Implement CCR command for linux > > nvme target. > > > > Signed-off-by: Mohamed Khalfella > > --- > > drivers/nvme/target/admin-cmd.c | 74 ++++++++++++++++++++++++++++++++ > > drivers/nvme/target/core.c | 76 +++++++++++++++++++++++++++++++++ > > drivers/nvme/target/nvmet.h | 13 ++++++ > > include/linux/nvme.h | 23 ++++++++++ > > 4 files changed, 186 insertions(+) > > > > diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c > > index ec09e30eca18..0a37c0eeebb5 100644 > > --- a/drivers/nvme/target/admin-cmd.c > > +++ b/drivers/nvme/target/admin-cmd.c > > @@ -376,7 +376,9 @@ static void nvmet_get_cmd_effects_admin(struct nvmet_ctrl *ctrl, > > log->acs[nvme_admin_get_features] = > > log->acs[nvme_admin_async_event] = > > log->acs[nvme_admin_keep_alive] = > > + log->acs[nvme_admin_cross_ctrl_reset] = > > cpu_to_le32(NVME_CMD_EFFECTS_CSUPP); > > + > > } > > > > static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log) > > @@ -1613,6 +1615,75 @@ void nvmet_execute_keep_alive(struct nvmet_req *req) > > nvmet_req_complete(req, status); > > } > > > > +void nvmet_execute_cross_ctrl_reset(struct nvmet_req *req) > > +{ > > + struct nvmet_ctrl *ictrl, *sctrl = req->sq->ctrl; > > + struct nvme_command *cmd = req->cmd; > > + struct nvmet_ccr *ccr, *new_ccr; > > + int ccr_active, ccr_total; > > + u16 cntlid, status = NVME_SC_SUCCESS; > > + > > + cntlid = le16_to_cpu(cmd->ccr.icid); > > + if (sctrl->cntlid == cntlid) { > > + req->error_loc = > > + offsetof(struct nvme_cross_ctrl_reset_cmd, icid); > > + status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; > > + goto out; > > + } > > + > > + /* Find and get impacted controller */ > > + ictrl = nvmet_ctrl_find_get_ccr(sctrl->subsys, sctrl->hostnqn, > > + cmd->ccr.ciu, cntlid, > > + le64_to_cpu(cmd->ccr.cirn)); > > + if (!ictrl) { > > + /* Immediate Reset Successful */ > > + nvmet_set_result(req, 1); > > + status = NVME_SC_SUCCESS; > > + goto out; > > + } > > + > > + ccr_total = ccr_active = 0; > > + mutex_lock(&sctrl->lock); > > + list_for_each_entry(ccr, &sctrl->ccr_list, entry) { > > + if (ccr->ctrl == ictrl) { > > + status = NVME_SC_CCR_IN_PROGRESS | NVME_STATUS_DNR; > > + goto out_unlock; > > + } > > + > > + ccr_total++; > > + if (ccr->ctrl) > > + ccr_active++; > > + } > > + > > + if (ccr_active >= NVMF_CCR_LIMIT) { > > + status = NVME_SC_CCR_LIMIT_EXCEEDED; > > + goto out_unlock; > > + } > > + if (ccr_total >= NVMF_CCR_PER_PAGE) { > > + status = NVME_SC_CCR_LOGPAGE_FULL; > > + goto out_unlock; > > + } > > + > > + new_ccr = kmalloc_obj(*new_ccr, GFP_KERNEL); > > + if (!new_ccr) { > > + status = NVME_SC_INTERNAL; > > + goto out_unlock; > > + } > > + > > + new_ccr->ciu = cmd->ccr.ciu; > > + new_ccr->icid = cntlid; > > + new_ccr->ctrl = ictrl; > > + list_add_tail(&new_ccr->entry, &sctrl->ccr_list); > > + > > +out_unlock: > > + mutex_unlock(&sctrl->lock); > > + if (status == NVME_SC_SUCCESS) > > + nvmet_ctrl_fatal_error(ictrl); > > + nvmet_ctrl_put(ictrl); > > +out: > > + nvmet_req_complete(req, status); > > +} > > + > > u32 nvmet_admin_cmd_data_len(struct nvmet_req *req) > > { > > struct nvme_command *cmd = req->cmd; > > @@ -1690,6 +1761,9 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req) > > case nvme_admin_keep_alive: > > req->execute = nvmet_execute_keep_alive; > > return 0; > > + case nvme_admin_cross_ctrl_reset: > > + req->execute = nvmet_execute_cross_ctrl_reset; > > + return 0; > > default: > > return nvmet_report_invalid_opcode(req); > > } > > diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c > > index e8b945a01f35..2e0c31d82bad 100644 > > --- a/drivers/nvme/target/core.c > > +++ b/drivers/nvme/target/core.c > > @@ -117,6 +117,20 @@ u16 nvmet_zero_sgl(struct nvmet_req *req, off_t off, size_t len) > > return 0; > > } > > > > +void nvmet_ctrl_cleanup_ccrs(struct nvmet_ctrl *ctrl, bool all) > > +{ > > + struct nvmet_ccr *ccr, *tmp; > > + > > + lockdep_assert_held(&ctrl->lock); > > + > > + list_for_each_entry_safe(ccr, tmp, &ctrl->ccr_list, entry) { > > + if (all || ccr->ctrl == NULL) { > > + list_del(&ccr->entry); > > + kfree(ccr); > > + } > > + } > > +} > > + > > static u32 nvmet_max_nsid(struct nvmet_subsys *subsys) > > { > > struct nvmet_ns *cur; > > @@ -1399,6 +1413,7 @@ static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl) > > if (!nvmet_is_disc_subsys(ctrl->subsys)) { > > ctrl->ciu = ((u8)(ctrl->ciu + 1)) ? : 1; > > ctrl->cirn = get_random_u64(); > > + nvmet_ctrl_cleanup_ccrs(ctrl, false); > > } > > ctrl->csts = NVME_CSTS_RDY; > > > > @@ -1504,6 +1519,35 @@ struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn, > > return ctrl; > > } > > > > +struct nvmet_ctrl *nvmet_ctrl_find_get_ccr(struct nvmet_subsys *subsys, > > + const char *hostnqn, u8 ciu, > > + u16 cntlid, u64 cirn) > > +{ > > + struct nvmet_ctrl *ctrl, *ictrl = NULL; > > + bool found = false; > > + > > + mutex_lock(&subsys->lock); > > + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) { > > + if (ctrl->cntlid != cntlid) > > + continue; > > + > > + /* Avoid racing with a controller that is becoming ready */ > > + mutex_lock(&ctrl->lock); > > + if (ctrl->ciu == ciu && ctrl->cirn == cirn) > > + found = true; > > + mutex_unlock(&ctrl->lock); > > + > > + if (found) { > > + if (kref_get_unless_zero(&ctrl->ref)) > > + ictrl = ctrl; > > + break; > > + } > > + }; > > + mutex_unlock(&subsys->lock); > > + > > + return ictrl; > > +} > > + > > u16 nvmet_check_ctrl_status(struct nvmet_req *req) > > { > > if (unlikely(!(req->sq->ctrl->cc & NVME_CC_ENABLE))) { > > @@ -1629,6 +1673,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) > > subsys->clear_ids = 1; > > #endif > > > > + INIT_LIST_HEAD(&ctrl->ccr_list); > > INIT_WORK(&ctrl->async_event_work, nvmet_async_event_work); > > INIT_LIST_HEAD(&ctrl->async_events); > > INIT_RADIX_TREE(&ctrl->p2p_ns_map, GFP_KERNEL); > > @@ -1739,12 +1784,43 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) > > } > > EXPORT_SYMBOL_GPL(nvmet_alloc_ctrl); > > > > +static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) > > +{ > > + struct nvmet_subsys *subsys = ctrl->subsys; > > + struct nvmet_ctrl *sctrl; > > + struct nvmet_ccr *ccr; > > + > > + lockdep_assert_held(&subsys->lock); > > + > > + /* Cleanup all CCRs issued by ctrl as source controller */ > > + mutex_lock(&ctrl->lock); > > + nvmet_ctrl_cleanup_ccrs(ctrl, true); > > + mutex_unlock(&ctrl->lock); > > + > > + /* > > + * Find all CCRs targeting ctrl as impacted controller and > > + * set ccr->ctrl to NULL. This tells the source controller > > + * that CCR completed successfully. > > + */ > > + list_for_each_entry(sctrl, &subsys->ctrls, subsys_entry) { > > + mutex_lock(&sctrl->lock); > > + list_for_each_entry(ccr, &sctrl->ccr_list, entry) { > > + if (ccr->ctrl == ctrl) { > > + ccr->ctrl = NULL; > > + break; > > + } > > + } > > + mutex_unlock(&sctrl->lock); > > + } > > +} > > + > > Do I see this correct that with this implementation a CCR is only > complete once the controller resets? IOW the CCR has to wait for > the controller to be reset, but it does not invoke a controller reset > itself? > > Is that intended? nvmet_execute_cross_ctrl_reset() calls nvmet_ctrl_fatal_error() to cause impacted controller to fail. CCR is completed when the impacted controller exits.