From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f179.google.com (mail-dy1-f179.google.com [74.125.82.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69AFF3EBF10 for ; Tue, 17 Feb 2026 18:25:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771352758; cv=none; b=YOvv0ZPemubL6oUQc2nae9tzLZu2OD4t4wW3yXp9pQb3q9A/NIGHFE+OZ77Jm61PnwVoE4miGNbxs0uYvhJrk8oHjOziG0HK17FpWYVo06jPjeCzrqzjimlCcC7rr7z9poYS+iTq2bkJHGpKg1ZbEAEniD4xQ/s6Dw2GrRHbzew= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771352758; c=relaxed/simple; bh=Cpt13jX3csgFBA2B1JAxspVEf9KaLU5dln58Uo995gY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=vFFnlJXh+nMaQfjHE+f43nMTH7ZmOgC/LYIzYKscUNq17zQRO1j5OF6zDl4SbamExIlcVB4JjtfKuAxz1t6L/iJ01s0boiXzfXIgvuLRfDCN8gb8jE112zNIXAssBkV1KBv+FaIWLwOTRRbzteGKsogrmifpGVIc3/K1tUvmuQM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=MPIhb+wL; arc=none smtp.client-ip=74.125.82.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="MPIhb+wL" Received: by mail-dy1-f179.google.com with SMTP id 5a478bee46e88-2bab70f8c8aso4727043eec.1 for ; Tue, 17 Feb 2026 10:25:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1771352756; x=1771957556; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=HLWRzMtsCpelPtrD3vI1WSb7s2lpiuT1u2xkypQAkdQ=; b=MPIhb+wLf/DPp6fn2pm7dSTk0Txj5UlmsKJfqXH9qrKJZcpA7B5otnUOtlyG2Ryl6+ IhitD3IeCSnSRKPcXJPTgd/Gh+7/vFMDZ5bsWO9DGYERoz40z5hhJxHm+DSZCc+9O2m7 df/5P8ftDijn9MRzrK2HXcWYoNz2WAitDS4J3ozEJMZjZIIALWzRJylbKbvmomqzub4W amVbDbb+T11vg179bMPX5NJiAiz0LksuZwR7Zlf0EidDmxiZy+7DCK733MHhLNqJxEf+ wLqV6cx0mypwDDAB8PvSFKEGKBqYn/Nm+3fX0g4XgDN1RuDv9eHSYh8IshYDRREiI4RW mRww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771352756; x=1771957556; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLWRzMtsCpelPtrD3vI1WSb7s2lpiuT1u2xkypQAkdQ=; b=OV+lSBm7Ks+RMtenIyoDc7Wq3zROyBOm3O+UB7ytJY0dWGo7T77GZlCQkQdwp04FAk VSwXsLNlirY9kbSbHdfW1NiwKK5qQOw84ICJIFi6Fz2b9A8SnHzAwEtd5gulSfSAjBeE 5YabT1R8sqHzbZ2IU7wJ034TFFLROqrNfx6RkK0wSYp7UVTHmi6Huv6mO4eMCFA30pIo 6WLZxwGMFVmor8JKYC+vq/OimY/in6c7uflFzcRPZ3RSFXuRoIH9sCYWEdLvPNj9nMtl mla5DCnjh26DM+2NWBSenGaKycFlwNFk527mwkBjA7ZcBYkg4E7CBBloq46/UKJFUXDJ VD7A== X-Forwarded-Encrypted: i=1; AJvYcCUx2sAmjP4rcnR2LbUc2CDhU57+SI8rVi70YC1FlZrwKEnRTGTe9BIIv2kvScjzqoS/A8q4PftaKD1NI2g=@vger.kernel.org X-Gm-Message-State: AOJu0Yyz/Kr+jn2WtohGjgcA2jq6dwY+KkMnnHZYjRUyJ7XGi09ZUKwf PR7p5SvZa7V94tVIf/BhPE9HUX8n0UcN/OiS9YHnsT6KtBiILY/DbDWZ9BTiYFU9q0Y= X-Gm-Gg: AZuq6aISvW218hCYyy3SjNjkK03dKji9pNpQ2uH8gpRQko47zxOU8xY5wK6/RWjw5WL 1bZDYeXNdW+P8ihblG+79allcMYPxjxeQRO2Dlp62nGoCq3EIolDxQGGgbFeKqGTRD4MVEv6Mye 2Quu9SzniHmX/ccYHeJVIEXkC8x31UAJ8OriN87cK40yAbzHwrQ4vncg0iF6sLo1tI5DHW4WUqG Cbc9OUrHqnMcW6zG1c6jXcLp4jv5YTt2/ynV5GTo/dx5ukD0ZZj6SkUZN9ML8peXrbie3ahFZu4 hY37+N3RUqovnCjY3n98MMln2lhwrgvV6qebFf/97xOOKfLqieuhBfsQIJbqNgT2FHHUOtQG2KC +fDTJq2RoxdJvkVRyVjSve38E6nzNSjlyH/vUd3OuxlF1goPhSTYcqXZuTCTkeOlnFYpW/FtBxc Uai1y1Rm4JpfjFMuspyzbYEBX+cdD2L2Svh1bU5xaByK4= X-Received: by 2002:a05:7301:4085:b0:2ba:6e6e:d8ef with SMTP id 5a478bee46e88-2babc53a802mr6429231eec.28.1771352756278; Tue, 17 Feb 2026 10:25:56 -0800 (PST) Received: from medusa.lab.kspace.sh ([208.88.152.253]) by smtp.googlemail.com with UTF8SMTPSA id a92af1059eb24-12742cadb25sm18674000c88.10.2026.02.17.10.25.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Feb 2026 10:25:55 -0800 (PST) Date: Tue, 17 Feb 2026 10:25:54 -0800 From: Mohamed Khalfella To: Hannes Reinecke Cc: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , James Smart , Aaron Dailey , Randy Jennings , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 09/21] nvme: Implement cross-controller reset completion Message-ID: <20260217182554.GE3435530-mkhalfella@purestorage.com> References: <20260214042753.4073668-1-mkhalfella@purestorage.com> <20260214042753.4073668-10-mkhalfella@purestorage.com> <3b21ccbd-7948-436b-8faa-a5541c65946a@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3b21ccbd-7948-436b-8faa-a5541c65946a@suse.de> On Mon 2026-02-16 13:43:51 +0100, Hannes Reinecke wrote: > On 2/14/26 05:25, Mohamed Khalfella wrote: > > An nvme source controller that issues CCR command expects to receive an > > NVME_AER_NOTICE_CCR_COMPLETED when pending CCR succeeds or fails. Add > > sctrl->ccr_work to read NVME_LOG_CCR logpage and wakeup any thread > > waiting on CCR completion. > > > > Signed-off-by: Mohamed Khalfella > > --- > > drivers/nvme/host/core.c | 49 +++++++++++++++++++++++++++++++++++++++- > > drivers/nvme/host/nvme.h | 1 + > > 2 files changed, 49 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c > > index 765b1524b3ed..a9fcde1b411b 100644 > > --- a/drivers/nvme/host/core.c > > +++ b/drivers/nvme/host/core.c > > @@ -1916,7 +1916,8 @@ EXPORT_SYMBOL_GPL(nvme_set_queue_count); > > > > #define NVME_AEN_SUPPORTED \ > > (NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_FW_ACT | \ > > - NVME_AEN_CFG_ANA_CHANGE | NVME_AEN_CFG_DISC_CHANGE) > > + NVME_AEN_CFG_ANA_CHANGE | NVME_AEN_CFG_CCR_COMPLETE | \ > > + NVME_AEN_CFG_DISC_CHANGE) > > > > static void nvme_enable_aen(struct nvme_ctrl *ctrl) > > { > > @@ -4880,6 +4881,47 @@ static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl) > > kfree(log); > > } > > > > +static void nvme_ccr_work(struct work_struct *work) > > +{ > > + struct nvme_ctrl *ctrl = container_of(work, struct nvme_ctrl, ccr_work); > > + struct nvme_ccr_entry *ccr; > > + struct nvme_ccr_log_entry *entry; > > + struct nvme_ccr_log *log; > > + unsigned long flags; > > + int ret, i; > > + > > + log = kmalloc(sizeof(*log), GFP_KERNEL); > > + if (!log) > > + return; > > + > > + ret = nvme_get_log(ctrl, 0, NVME_LOG_CCR, 0x01, > > + 0x00, log, sizeof(*log), 0); > > + if (ret) > > + goto out; > > + > > + spin_lock_irqsave(&ctrl->lock, flags); > > + for (i = 0; i < le16_to_cpu(log->ne); i++) { > > + entry = &log->entries[i]; > > + if (entry->ccrs == NVME_CCR_STATUS_IN_PROGRESS) > > + continue; > > + > > + list_for_each_entry(ccr, &ctrl->ccr_list, list) { > > + struct nvme_ctrl *ictrl = ccr->ictrl; > > + > > + if (ictrl->cntlid != le16_to_cpu(entry->icid) || > > + ictrl->ciu != entry->ciu) > > + continue; > > + > > + /* Complete matching entry */ > > + ccr->ccrs = entry->ccrs; > > + complete(&ccr->complete); > > + } > > + } > > + spin_unlock_irqrestore(&ctrl->lock, flags); > > +out: > > + kfree(log); > > +} > > + > > static void nvme_fw_act_work(struct work_struct *work) > > { > > struct nvme_ctrl *ctrl = container_of(work, > > @@ -4956,6 +4998,9 @@ static bool nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result) > > case NVME_AER_NOTICE_DISC_CHANGED: > > ctrl->aen_result = result; > > break; > > + case NVME_AER_NOTICE_CCR_COMPLETED: > > + queue_work(nvme_wq, &ctrl->ccr_work); > > + break; > > default: > > dev_warn(ctrl->device, "async event result %08x\n", result); > > } > > @@ -5145,6 +5190,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl) > > nvme_stop_failfast_work(ctrl); > > flush_work(&ctrl->async_event_work); > > cancel_work_sync(&ctrl->fw_act_work); > > + cancel_work_sync(&ctrl->ccr_work); > > if (ctrl->ops->stop_ctrl) > > ctrl->ops->stop_ctrl(ctrl); > > } > > @@ -5268,6 +5314,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, > > ctrl->quirks = quirks; > > ctrl->numa_node = NUMA_NO_NODE; > > INIT_WORK(&ctrl->scan_work, nvme_scan_work); > > + INIT_WORK(&ctrl->ccr_work, nvme_ccr_work); > > INIT_WORK(&ctrl->async_event_work, nvme_async_event_work); > > INIT_WORK(&ctrl->fw_act_work, nvme_fw_act_work); > > INIT_WORK(&ctrl->delete_work, nvme_delete_ctrl_work); > > diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h > > index f3ab9411cac5..af6a4e83053e 100644 > > --- a/drivers/nvme/host/nvme.h > > +++ b/drivers/nvme/host/nvme.h > > @@ -365,6 +365,7 @@ struct nvme_ctrl { > > struct nvme_effects_log *effects; > > struct xarray cels; > > struct work_struct scan_work; > > + struct work_struct ccr_work; > > struct work_struct async_event_work; > > struct delayed_work ka_work; > > struct delayed_work failfast_work; > > We really would need some indicator whether 'ccr' is supported at all. Why do we need this indicator, other than exporting it via sysfs? > Using the number of available CCR commands would be an option, if though > that would require us to keep two counters (one for the number of > possible outstanding CCRs, and one for the number of actual outstanding > CCRs.). Like mentioned above ctrl->ccr_limit gives us the number of ccrs available now. It is not 100% indicator if CCR is supported or not, but it is enough to implement CCR. A second counter can help us skip trying CCR if we know impacted controller does not support it. Do you think it is worth it? Iterating over controllers in the subsystem is not that bad IMO. This is similar to the point raised by James Smart [1]. 1- https://lore.kernel.org/all/05875e07-b908-425a-ba6f-5e060e03241e@gmail.com/