From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5945478F4F for ; Thu, 25 Dec 2025 17:33:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766684027; cv=none; b=C4v2AHT3CKFWu8FNzyiN/vaVEz7JN8hB0WF4+nAHuNu+KHwXZ4bre1kCmMlLYmR44laqQBOoiLnwESzR3r3qJu01itQUEuPMjuiFvV/ep8j9foKL+MuIgTbQgfdWR53G2W/aH++xXmDSujRGBUbhs1RkhkJEnQ2tbk027m6pSzQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766684027; c=relaxed/simple; bh=+Ae1ub1HfCSbeRNVjQ0QSu19/zAZuqIi9GV9UIMmtSw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=G9oKN9JcF7ATnfEqZDGk+Ks3wzIld8+/hUCqJcvS5MSAs18NecDjnpUp5rraPMOZjuloP9ZjH5kk1XAmLkVsb/dPWZFaFZRs7DXmF8F6L4py1XbRP455kir4U/g0XRRb+CtBjt0Vs09wCtSoXL9+x9D9LXH2jJURjkHQDWWofvA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=becEqXPh; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="becEqXPh" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2a0d5c365ceso86241825ad.3 for ; Thu, 25 Dec 2025 09:33:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1766684024; x=1767288824; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=j5EC01ePkdkVNX3nDSKOcSuhVJygES9lIvbPYyKJrtk=; b=becEqXPhl4nyBZwFTDs+96jklr/3nMzXLCDpXkZAM0bfU7JEWPu7Fdmc5T+oie7XV3 BF6+rULpMD06DP7/iHybehnYqp63Ia2D+eqLWtmxifXCCeckVzw3IvcsrxIBckrzupsy qM5UwmqCU9iIL2flqa25cxFUfTVsAQQ3F3T4EiklMktMhXEj6ntIPgvl8eZHFdw3xcz3 E77FzYDF5/eADD/PLlxSD8BU5SCpLm7cdSTAhkmZ/DAR9p6jmvOIcp7iOjy2SyVHSrsF mavT7Z0jn2t7n4TmJFKVgbOSBeirxfqHDQ+AUmVsTIxZsy2nQpjaRv2X1vIrSVqK/6YK Q/zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766684024; x=1767288824; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j5EC01ePkdkVNX3nDSKOcSuhVJygES9lIvbPYyKJrtk=; b=iI/sLAqNL8bvytHCBeTF5uTGi8g2fWWxBNGdTS5K239BsGyd5EoDd3/NzUJ9uhXaN/ alygG5GbxQcMfD76Jwc62wEkuuMWCoR2U2isq2CLrR60os57XI4UIMTBWM9j2NsnsjWu LctHOtNfg3ixBVHlBNWoxHwRigZ4IkpPSaPA6UttLHOT/xBwuWiLbGWHqE1MtfgsEOkM I8Xhm8BNB0CBB+Q7PA5ulnkJbxYegnkKjJ+mnUvjTGfXuEPKi/4DDvRV/El67FAOveLB kkgDBM5ZPEV3570NkdRX54xd++LKIr48ivqcuYv5x+TgUgs2aNc07QORsTvmDX3tWuwW MbZA== X-Forwarded-Encrypted: i=1; AJvYcCXp+BnMxXHXflXVg/6zY9Kyqh3eCPhYRLnsNgrisi+9Ac2ll3dWV8nvJTwFY4PncsYS8T92UMHtZDC9KK8=@vger.kernel.org X-Gm-Message-State: AOJu0YwKS00l9Is5bM3qXSoSSqCe8GZOUzpyfXwHDpvXChEL+yZGi9lg 11ZEbT+47wUya2V6sqx943BtMzEzszh29Di17QYpSdnUSb4nAUxCQjC10SECvu7Qsj6DYQEcNL1 VCQIn X-Gm-Gg: AY/fxX7SELmNrmyx8BmHwKXgDzHntdxnuUpWc1/GiS5uOeFZ9E65zJtRw8i1Ydq4WjH LzBueOaZTnQlx74SYdcVB7ZHWRUxqqLPCTHkIMYp1TA5bbWHcNJtQFC8UYAevHe0kRCvun6jSMP EiFUXFSDHp4vYSDxm3atXJ7H5V4XAnyKLPF4Wk68NncAz0+rxqioG24iubbgNhX1PET7G3AN6rv MPXL/Ni1gxB/CECzaDf9PsTrJ4q6ufEgHRnDhIiRw759zrqnRzf8o/3TOk0WMU5x/+2vauYM6G3 yxJpmKKtGXxLrPi8nzp7iSZmaoNa6DpcMvc4nmeRgWCz94G7uZf/rDHmraFpPF7O7ydR2ASjhZf SSnF9ufiT85wj8DiGaRYpYoQ2+q1GJbslAPCYaa9/se6SeQk9BYIK6hIyHTtqdAe1j59LKa9P9w bUha68vRRoElrJ1bM= X-Google-Smtp-Source: AGHT+IHvDPVi7MfkxykM6GkuJHP8TD2suiOScF3/pn4zZdLHiPPU1gilWhZbMT9py8BRxehOWyMs7g== X-Received: by 2002:a05:7022:eac1:b0:11d:c91e:3b58 with SMTP id a92af1059eb24-121722e9e26mr17987214c88.39.1766684024292; Thu, 25 Dec 2025 09:33:44 -0800 (PST) Received: from medusa.lab.kspace.sh ([2601:640:8202:6fb0::1305]) by smtp.googlemail.com with UTF8SMTPSA id a92af1059eb24-1217243bbe3sm81422741c88.0.2025.12.25.09.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Dec 2025 09:33:43 -0800 (PST) Date: Thu, 25 Dec 2025 09:33:42 -0800 From: Mohamed Khalfella To: Sagi Grimberg Cc: Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Aaron Dailey , Randy Jennings , John Meneghini , Hannes Reinecke , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 03/14] nvmet: Implement CCR nvme command Message-ID: <20251225173342.GB8129-mkhalfella@purestorage.com> References: <20251126021250.2583630-1-mkhalfella@purestorage.com> <20251126021250.2583630-4-mkhalfella@purestorage.com> <3f2c8a9e-2e1c-4cb8-b296-8f61ab16b03a@grimberg.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3f2c8a9e-2e1c-4cb8-b296-8f61ab16b03a@grimberg.me> On Thu 2025-12-25 15:14:31 +0200, Sagi Grimberg wrote: > > > On 26/11/2025 4:11, Mohamed Khalfella wrote: > > Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller > > Reset) command is an nvme command the is issued to source controller by > > initiator to reset impacted controller. Implement CCR command for linux > > nvme target. > > > > Signed-off-by: Mohamed Khalfella > > --- > > drivers/nvme/target/admin-cmd.c | 79 +++++++++++++++++++++++++++++++++ > > drivers/nvme/target/core.c | 69 ++++++++++++++++++++++++++++ > > drivers/nvme/target/nvmet.h | 13 ++++++ > > include/linux/nvme.h | 23 ++++++++++ > > 4 files changed, 184 insertions(+) > > > > diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c > > index aaceb697e4d2..a55ca010d34f 100644 > > --- a/drivers/nvme/target/admin-cmd.c > > +++ b/drivers/nvme/target/admin-cmd.c > > @@ -376,7 +376,9 @@ static void nvmet_get_cmd_effects_admin(struct nvmet_ctrl *ctrl, > > log->acs[nvme_admin_get_features] = > > log->acs[nvme_admin_async_event] = > > log->acs[nvme_admin_keep_alive] = > > + log->acs[nvme_admin_cross_ctrl_reset] = > > cpu_to_le32(NVME_CMD_EFFECTS_CSUPP); > > + > > } > > > > static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log) > > @@ -1615,6 +1617,80 @@ void nvmet_execute_keep_alive(struct nvmet_req *req) > > nvmet_req_complete(req, status); > > } > > > > +void nvmet_execute_cross_ctrl_reset(struct nvmet_req *req) > > +{ > > + struct nvmet_ctrl *ictrl, *ctrl = req->sq->ctrl; > > + struct nvme_command *cmd = req->cmd; > > + struct nvmet_ccr *ccr, *new_ccr; > > + int ccr_active, ccr_total; > > + u16 cntlid, status = 0; > > + > > + cntlid = le16_to_cpu(cmd->ccr.icid); > > + if (ctrl->cntlid == cntlid) { > > + req->error_loc = > > + offsetof(struct nvme_cross_ctrl_reset_cmd, icid); > > + status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; > > + goto out; > > + } > > + > > + ictrl = nvmet_ctrl_find_get_ccr(ctrl->subsys, ctrl->hostnqn, > > What does the 'i' stand for? 'i' stands for impacted controller. Also, if you see sctrl the 's' stands for source controller. These terms are from TP8028. > > > + cmd->ccr.ciu, cntlid, > > + le64_to_cpu(cmd->ccr.cirn)); > > + if (!ictrl) { > > + /* Immediate Reset Successful */ > > + nvmet_set_result(req, 1); > > + status = NVME_SC_SUCCESS; > > + goto out; > > + } > > + > > + new_ccr = kmalloc(sizeof(*ccr), GFP_KERNEL); > > + if (!new_ccr) { > > + status = NVME_SC_INTERNAL; > > + goto out_put_ctrl; > > + } > > Allocating this later when you actually use it would probably simplify > error path. Right, it will save us kfree(). Will do that. > > > + > > + ccr_total = ccr_active = 0; > > + mutex_lock(&ctrl->lock); > > + list_for_each_entry(ccr, &ctrl->ccrs, entry) { > > + if (ccr->ctrl == ictrl) { > > + status = NVME_SC_CCR_IN_PROGRESS | NVME_STATUS_DNR; > > + goto out_unlock; > > + } > > + > > + ccr_total++; > > + if (ccr->ctrl) > > + ccr_active++; > > + } > > + > > + if (ccr_active >= NVMF_CCR_LIMIT) { > > + status = NVME_SC_CCR_LIMIT_EXCEEDED; > > + goto out_unlock; > > + } > > + if (ccr_total >= NVMF_CCR_PER_PAGE) { > > + status = NVME_SC_CCR_LOGPAGE_FULL; > > + goto out_unlock; > > + } > > + > > + new_ccr->ciu = cmd->ccr.ciu; > > + new_ccr->icid = cntlid; > > + new_ccr->ctrl = ictrl; > > + list_add_tail(&new_ccr->entry, &ctrl->ccrs); > > + mutex_unlock(&ctrl->lock); > > + > > + nvmet_ctrl_fatal_error(ictrl); > > Don't you need to wait for it to complete? > e.g. flush_work(&ictrl->fatal_err_work); > > Or is that done async? will need to look downstream... No, we do not need to wait for ictrl->fatal_err_work to complete. An AEN will be sent when ictrl exits. It is okay if AEN is sent before CCR request is completed. The initiator should expect this behavior and deal with it.