[PATCHv2] nvme-pci: try function level reset on init failure

[PATCHv2] nvme-pci: try function level reset on init failure

[Keith Busch]

From: Keith Busch <kbusch@kernel.org>

NVMe devices from multiple vendors appear to get stuck in a reset state
that we can't get out of with an NVMe level Controller Reset. The kernel
would report these with messages that look like:

  Device not ready; aborting reset, CSTS=0x1

These have historically required a power cycle to make them usable
again, but in many cases, a PCIe FLR is sufficient to restart operation
without a power cycle. Try it if the initial controller reset fails
during any nvme reset attempt.

Cc: Chaitanya Kulkarni <chaitanyak@nvidia.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
v1->v2:

  Added code comment explaining whe escalation

  Add an informational kernel message that this event occured

  Use the "pcie_reset_flr()" API instead of "pcie_flr()" since that one
  checks for quirks and capabilities before writing FLR config bits.
  Note, NVMe PCI Trasnsport Spec mandates FLR capability, so the latter
  should not apply to any compliant device, but you never know...

 drivers/nvme/host/pci.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 4cf87fb5d8573..f8f8cb6a4786a 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2067,8 +2067,28 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
 	 * might be pointing at!
 	 */
 	result = nvme_disable_ctrl(&dev->ctrl, false);
-	if (result < 0)
-		return result;
+	if (result < 0) {
+		struct pci_dev *pdev = to_pci_dev(dev->dev);
+
+		/*
+		 * The NVMe Controller Reset method did not get an expected
+		 * CSTS.RDY transition, so something with the device appears to
+		 * be stuck. Use the lower level and bigger hammer PCIe
+		 * Function Level Reset to attempt restoring the device to its
+		 * initial state, and try again.
+		 */
+		result = pcie_reset_flr(pdev, false);
+		if (result < 0)
+			return result;
+
+		pci_restore_state(pdev);
+		result = nvme_disable_ctrl(&dev->ctrl, false);
+		if (result < 0)
+			return result;
+
+		dev_info(&dev->ctrl.device,
+			"controller reset completed after pcie flr\n");
+	}
 
 	result = nvme_alloc_queue(dev, 0, NVME_AQ_DEPTH);
 	if (result)
-- 
2.47.1

Re: [PATCHv2] nvme-pci: try function level reset on init failure

[Chaitanya Kulkarni]

On 7/15/25 12:16, Keith Busch wrote:
> From: Keith Busch<kbusch@kernel.org>
>
> NVMe devices from multiple vendors appear to get stuck in a reset state
> that we can't get out of with an NVMe level Controller Reset. The kernel
> would report these with messages that look like:
>
>    Device not ready; aborting reset, CSTS=0x1
>
> These have historically required a power cycle to make them usable
> again, but in many cases, a PCIe FLR is sufficient to restart operation
> without a power cycle. Try it if the initial controller reset fails
> during any nvme reset attempt.
>
> Cc: Chaitanya Kulkarni<chaitanyak@nvidia.com>
> Signed-off-by: Keith Busch<kbusch@kernel.org>

Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck

Re: nvme-pci: try function level reset on init failure

[Nitesh Shetty]

[-- Attachment #1: Type: text/plain, Size: 1245 bytes --]

On 15/07/25 12:16PM, Keith Busch wrote:
>From: Keith Busch <kbusch@kernel.org>
>
>NVMe devices from multiple vendors appear to get stuck in a reset state
>that we can't get out of with an NVMe level Controller Reset. The kernel
>would report these with messages that look like:
>
>  Device not ready; aborting reset, CSTS=0x1
>
>These have historically required a power cycle to make them usable
>again, but in many cases, a PCIe FLR is sufficient to restart operation
>without a power cycle. Try it if the initial controller reset fails
>during any nvme reset attempt.
>
>Cc: Chaitanya Kulkarni <chaitanyak@nvidia.com>
>Signed-off-by: Keith Busch <kbusch@kernel.org>
>---
>v1->v2:
>
>  Added code comment explaining whe escalation
>
>  Add an informational kernel message that this event occured
>
>  Use the "pcie_reset_flr()" API instead of "pcie_flr()" since that one
>  checks for quirks and capabilities before writing FLR config bits.
>  Note, NVMe PCI Trasnsport Spec mandates FLR capability, so the latter
>  should not apply to any compliant device, but you never know...
>
> drivers/nvme/host/pci.c | 24 ++++++++++++++++++++++--
> 1 file changed, 22 insertions(+), 2 deletions(-)
>

Reviewed-by: Nitesh Shetty <nj.shetty@samsung.com>

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCHv2] nvme-pci: try function level reset on init failure

[Christoph Hellwig]

Thanks,

applied to nvme-6.17.

Re: [PATCHv2] nvme-pci: try function level reset on init failure

[Keith Busch]

On Tue, Jul 15, 2025 at 12:16:27PM -0700, Keith Busch wrote:
> +		dev_info(&dev->ctrl.device,
> +			"controller reset completed after pcie flr\n");

Urg, sorry, this was the wrong diff: that '&' shouldn't be there, and
build bot will probably flag this.

Christoph, could you squash this in?

-- >8 --
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a7aa8ba4951f4..73d5a5298822a 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2083,7 +2083,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
 		if (result < 0)
 			return result;
 
-		dev_info(&dev->ctrl.device,
+		dev_info(dev->ctrl.device,
 			"controller reset completed after pcie flr\n");
 	}
 
--

Re: [PATCHv2] nvme-pci: try function level reset on init failure

[Christoph Hellwig]

On Thu, Jul 17, 2025 at 07:54:45AM -0600, Keith Busch wrote:
> On Tue, Jul 15, 2025 at 12:16:27PM -0700, Keith Busch wrote:
> > +		dev_info(&dev->ctrl.device,
> > +			"controller reset completed after pcie flr\n");
> 
> Urg, sorry, this was the wrong diff: that '&' shouldn't be there, and
> build bot will probably flag this.
> 
> Christoph, could you squash this in?

In fact I already did, but forgot to push out the result or tell
anyone..

Re: [PATCHv2] nvme-pci: try function level reset on init failure

[Keith Busch]

On Thu, Jul 17, 2025 at 03:57:06PM +0200, Christoph Hellwig wrote:
> On Thu, Jul 17, 2025 at 07:54:45AM -0600, Keith Busch wrote:
> > Christoph, could you squash this in?
> 
> In fact I already did, but forgot to push out the result or tell
> anyone..

Thanks, but the new push appears to have incorporated some unrelated
changes to fs/verity/verify.c

Re: [PATCHv2] nvme-pci: try function level reset on init failure

[Christoph Hellwig]

On Thu, Jul 17, 2025 at 09:44:51AM -0600, Keith Busch wrote:
> On Thu, Jul 17, 2025 at 03:57:06PM +0200, Christoph Hellwig wrote:
> > On Thu, Jul 17, 2025 at 07:54:45AM -0600, Keith Busch wrote:
> > > Christoph, could you squash this in?
> > 
> > In fact I already did, but forgot to push out the result or tell
> > anyone..
> 
> Thanks, but the new push appears to have incorporated some unrelated
> changes to fs/verity/verify.c

Fixes.  That's what you for not commiting thing immediately before
rushing to a meeting.