From: Hannes Reinecke <hare@suse.de>
To: Mohamed Khalfella <mkhalfella@purestorage.com>,
Dmitry Bogdanov <d.bogdanov@yadro.com>
Cc: Mohamed Khalfella <linux-nvme-bounces@lists.infradead.org>,
Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@kernel.dk>,
Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>,
Justin Tee <justin.tee@broadcom.com>,
Naresh Gottumukkala <nareshgottumukkala83@gmail.com>,
Paul Ely <paul.ely@broadcom.com>,
Hannes Reinecke <hare@kernel.org>,
Chaitanya Kulkarni <kch@nvidia.com>,
James Smart <jsmart833426@gmail.com>,
Randy Jennings <randyj@purestorage.com>,
Daniel Wagner <dwagner@suse.de>,
linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v7 03/17] nvmet: Implement CCR nvme command
Date: Tue, 17 Nov 2026 08:09:27 +0100 [thread overview]
Message-ID: <44dcc2bc-e7eb-48c6-9aa0-96f2ad5dcb0b@suse.de> (raw)
In-Reply-To: <20261117000348.GD1670230-mkhalfella@purestorage.com>
On 11/17/26 1:03 AM, Mohamed Khalfella wrote:
> On Mon 2026-11-16 11:45:40 +0300, Dmitry Bogdanov wrote:
>> On Fri, Nov 13, 2026 at 07:04:17PM -0800, Mohamed Khalfella wrote:
>>>
>>> Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller
>>> Reset) command is an nvme command issued to source controller by
>>> initiator to reset impacted controller. Implement CCR command for linux
>>> nvme target.
>>>
>>> Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com>
>>> Reviewed-by: Hannes Reinecke <hare@suse.de>
>>> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
>>> ---
>>> drivers/nvme/target/admin-cmd.c | 76 ++++++++++++++++++++++++++++++
>>> drivers/nvme/target/core.c | 83 +++++++++++++++++++++++++++++++++
>>> drivers/nvme/target/nvmet.h | 13 ++++++
>>> include/linux/nvme.h | 23 +++++++++
>>> 4 files changed, 195 insertions(+)
>>>
>>> diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
>>> index 5710ae53eb83..3df9db454db9 100644
>>> --- a/drivers/nvme/target/admin-cmd.c
>>> +++ b/drivers/nvme/target/admin-cmd.c
>>> @@ -378,6 +378,7 @@ static void nvmet_get_cmd_effects_admin(struct nvmet_ctrl *ctrl,
>>> log->acs[nvme_admin_get_features] =
>>> log->acs[nvme_admin_async_event] =
>>> log->acs[nvme_admin_keep_alive] =
>>> + log->acs[nvme_admin_cross_ctrl_reset] =
>>> cpu_to_le32(NVME_CMD_EFFECTS_CSUPP);
>>> }
>>>
>>> @@ -1629,6 +1630,78 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)
>>> nvmet_req_complete(req, status);
>>> }
>>>
>>> +void nvmet_execute_cross_ctrl_reset(struct nvmet_req *req)
>>> +{
>>> + struct nvmet_ctrl *ictrl, *sctrl = req->sq->ctrl;
>>> + struct nvme_command *cmd = req->cmd;
>>> + struct nvmet_ccr *ccr, *new_ccr;
>>> + int ccr_active, ccr_total;
>>> + u16 cntlid, status = NVME_SC_SUCCESS;
>>> +
>>> + if (!nvmet_check_transfer_len(req, 0))
>>> + return;
>>> +
>>> + cntlid = le16_to_cpu(cmd->ccr.icid);
>>> + if (sctrl->cntlid == cntlid) {
>>> + req->error_loc =
>>> + offsetof(struct nvme_cross_ctrl_reset_cmd, icid);
>>> + status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
>>> + goto out;
>>> + }
>>
>> cntlid has to be validated against the configured range of the controller
>> ids (subsys->cntlid_min <= cntlid <= subsys->cntlid_max).
>> That will allow CCR to work correctly on a distributed target.
>>
>
> Thanks for the review. nvmet_ctrl_find_get_ccr() below will verify
> ictrl->dntlid == cntlid before proceeding with the reset. Is this not
> enough?
>
> Adding this check here means if either cntlid_min or cntlid_max change
> after ictrl has been created, CCR will fail saying the controller was not
> found. Why do we want to CCR to fail in this case when ictrl might
> actually be there?
>
The point is that in a distributed system the cntlid range is
distributed across all nodes in the cluster (as indicated by
min/max_cntlid).
While logically all cntlids belong to the same subsystem,
technically each node can only reach (and reset!) the cntlids
of controllers local on that node.
With CCR each node can receive a CCR for any cntlid, but it
can handle only those cntlids which are local on the node.
But as we're looking up the actual controller structures
(which are only present if the cntlid is local to the node)
I think we should be good here.
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
next prev parent reply other threads:[~2026-11-17 7:09 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 3:04 [PATCH v7 00/17] TP8028 Rapid Path Failure Recovery Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 01/17] nvmet: Rapid Path Failure Recovery set controller identify fields Mohamed Khalfella
2026-11-14 3:17 ` sashiko-bot
2026-11-16 23:17 ` Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 02/17] nvmet/debugfs: Export controller CIU and CIRN via debugfs Mohamed Khalfella
2026-11-14 3:12 ` sashiko-bot
2026-11-16 23:19 ` Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 03/17] nvmet: Implement CCR nvme command Mohamed Khalfella
2026-11-14 3:30 ` sashiko-bot
2026-11-16 23:54 ` Mohamed Khalfella
2026-11-16 8:45 ` Dmitry Bogdanov
2026-11-17 0:03 ` Mohamed Khalfella
2026-11-17 7:09 ` Hannes Reinecke [this message]
2026-11-17 9:38 ` Dmitry Bogdanov
2026-11-14 3:04 ` [PATCH v7 04/17] nvmet: Implement CCR logpage Mohamed Khalfella
2026-11-14 3:20 ` sashiko-bot
2026-11-17 0:16 ` Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 05/17] nvmet: Send an AEN on CCR completion Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 06/17] nvme: Rapid Path Failure Recovery read controller identify fields Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 07/17] nvme: Introduce FENCING and FENCED controller states Mohamed Khalfella
2026-11-14 3:29 ` sashiko-bot
2026-11-16 7:16 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 08/17] nvme: Implement cross-controller reset recovery Mohamed Khalfella
2026-11-14 3:26 ` sashiko-bot
2026-11-14 3:04 ` [PATCH v7 09/17] nvme: Implement cross-controller reset completion Mohamed Khalfella
2026-11-14 3:29 ` sashiko-bot
2026-11-16 7:17 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 10/17] nvme-tcp: Use CCR to recover controller that hits an error Mohamed Khalfella
2026-11-14 3:38 ` sashiko-bot
2026-11-14 3:04 ` [PATCH v7 11/17] nvme-rdma: " Mohamed Khalfella
2026-11-16 7:19 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 12/17] nvme-fc: start error recovery instead of aborting timed out IOs Mohamed Khalfella
2026-11-14 3:46 ` sashiko-bot
2026-11-16 7:31 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 13/17] nvme-fc: perform error recovery directly from ioerr_work Mohamed Khalfella
2026-11-14 3:31 ` sashiko-bot
2026-11-16 7:37 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 14/17] nvme-fc: Use CCR to recover controller that hits an error Mohamed Khalfella
2026-11-14 3:38 ` sashiko-bot
2026-11-16 7:38 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 15/17] nvme-fc: Hold inflight requests while in FENCING state Mohamed Khalfella
2026-11-14 4:06 ` sashiko-bot
2026-11-16 7:52 ` Hannes Reinecke
2026-11-14 3:04 ` [PATCH v7 16/17] nvmet: Add support for CQT to nvme target Mohamed Khalfella
2026-11-14 3:04 ` [PATCH v7 17/17] nvme: Add support for CQT to nvme host Mohamed Khalfella
2026-11-16 7:55 ` Hannes Reinecke
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=44dcc2bc-e7eb-48c6-9aa0-96f2ad5dcb0b@suse.de \
--to=hare@suse.de \
--cc=axboe@kernel.dk \
--cc=d.bogdanov@yadro.com \
--cc=dwagner@suse.de \
--cc=hare@kernel.org \
--cc=hch@lst.de \
--cc=jsmart833426@gmail.com \
--cc=justin.tee@broadcom.com \
--cc=kbusch@kernel.org \
--cc=kch@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme-bounces@lists.infradead.org \
--cc=linux-nvme@lists.infradead.org \
--cc=mkhalfella@purestorage.com \
--cc=nareshgottumukkala83@gmail.com \
--cc=paul.ely@broadcom.com \
--cc=randyj@purestorage.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox