Re: [PATCH] nvme-pci: skip FLR after a failed controller reset

[Keith Busch <kbusch@kernel.org>]

On Mon, Sep 21, 2026 at 10:07:32PM +0800, Haowen Bai wrote:
> nvme_disable_ctrl() already waits up to CAP.TO for CSTS.RDY to clear.
> If that times out, nvme_pci_configure_admin_queue() currently issues a
> PCIe Function Level Reset and retries.
> 
> FLR is performed with PCI config cycles. Those cycles take
> pci_config_lock, a raw spinlock, and wait for the endpoint to complete
> the transaction. A wedged NVMe function can stall that completion.

Shouldn't PCIe CTO have kicked in to fail the transaction? Do you know
which transaction is failing? Is the stall specific to FLR or could any
config access stall in your setup?

> Other CPUs then spin in pci_conf1_read() -- including ACPI PCI config
> from an unrelated device -- and the NMI watchdog reports a hard lockup.
> 
> This was observed on an x86_64 UGREEN DXP4800 (kernel 6.18.15) with two
> ZHITAI Ti600 NVMe devices used as bcache. Each disk independently:

Are you able to fix the device instead? Maybe add your device to the
"quirk_no_flr" list if you can't fix it. This sounds like a pretty nasty
bug on that side that forces driver to remove one of its recovery
options.
 
>   nvme: I/O timeout, reset controller
>   nvme: Device not ready; aborting reset, CSTS=0x1
>   nvme: Device not ready; aborting reset, CSTS=0x1
>   watchdog: Watchdog detected hard LOCKUP
>   RIP: native_queued_spin_lock_slowpath
>   pci_conf1_read -> acpi_pci_set_power_state -> mmc runtime resume
> 
> The two "aborting reset" messages are nvme_wait_ready() timeouts, 128s
> apart, matching (CAP.TO+1)/2. The lockup is ~11s after the second
> timeout, i.e. on the post-FLR cleanup path, not in the wait loop.
> Linux 6.12 has no FLR fallback here; 6.18 and 7.3 still do.
> 
> Skip FLR when the controller is already in NVME_CTRL_RESETTING (I/O
> timeout recovery). Keep the FLR hammer for initial probe, where the
> device may simply have been left enabled by firmware. Reset work then
> marks namespaces dead instead of hard-locking the host.

While I don't see this happen very often, I've seen FLR recover devices
both on first probe and IO timeout, so skipping for RESETTING will miss
recovering when it was possible for some conditions.