[PATCH] nvme: make firmware activation poll interval configurable

[PATCH] nvme: make firmware activation poll interval configurable

[guzebing]

From: Guzebing <guzebing@bytedance.com>

nvme_fw_act_work() polls the controller processing-paused status every
100 ms while firmware activation is pending. Some devices can complete
online activation in only a few hundred milliseconds, so the fixed
interval can add noticeable latency before the driver observes
completion.

Add an nvme_core.fw_act_poll_interval_ms module parameter to make the
poll interval tunable. Keep the default at 100 ms to preserve existing
behavior, and accept values from 10 ms to 100 ms so systems that need
faster completion detection can opt in to a shorter interval.

Signed-off-by: Guzebing <guzebing@bytedance.com>
---
We recently observed this issue while performing online firmware
updates for Gen5 NVMe SSDs in a production environment.

During firmware activation, the kernel quiesces I/O. Detecting the end
of firmware activation earlier lets the driver unquiesce I/O earlier,
which is important for the long-tail I/O latency of production
workloads.

Operation steps:
  nvme fw-download /dev/nvme13n1 \
    -f ./Inventec_G238B_NVME_LDD5902Q_MZWL61T9HFLT-00B07.bin
  nvme fw-commit /dev/nvme13n1 -s 0 -a 3
  nvme fw-log /dev/nvme13n1

Operation device:
  Samsung PM9D3a Gen5 NVMe SSD, model MZWL67T6HBLC-00B07
  Firmware update: LDD5702Q -> LDD5902Q

Operation results:
  100 ms polling interval: 3 polls, 311 ms
   10 ms polling interval: 24 polls, 266 ms

 drivers/nvme/host/core.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3b7a8f7a35426..784da06fee559 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -58,6 +58,21 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);
 MODULE_PARM_DESC(io_timeout, "timeout in seconds for I/O");
 EXPORT_SYMBOL_GPL(nvme_io_timeout);
 
+#define NVME_FW_ACT_POLL_INTERVAL_MIN_MS	10
+#define NVME_FW_ACT_POLL_INTERVAL_MAX_MS	100
+static int fw_act_poll_interval_ms_set(const char *val,
+				       const struct kernel_param *kp);
+static const struct kernel_param_ops fw_act_poll_interval_ms_ops = {
+	.set = fw_act_poll_interval_ms_set,
+	.get = param_get_uint,
+};
+
+static unsigned int fw_act_poll_interval_ms = 100;
+module_param_cb(fw_act_poll_interval_ms, &fw_act_poll_interval_ms_ops,
+		&fw_act_poll_interval_ms, 0644);
+MODULE_PARM_DESC(fw_act_poll_interval_ms,
+		 "firmware activation polling interval in ms (10-100)");
+
 static unsigned char shutdown_timeout = 5;
 module_param(shutdown_timeout, byte, 0644);
 MODULE_PARM_DESC(shutdown_timeout, "timeout in seconds for controller shutdown");
@@ -106,6 +121,13 @@ module_param(disable_pi_offsets, bool, 0444);
 MODULE_PARM_DESC(disable_pi_offsets,
 	"disable protection information if it has an offset");
 
+static int fw_act_poll_interval_ms_set(const char *val,
+				       const struct kernel_param *kp)
+{
+	return param_set_uint_minmax(val, kp, NVME_FW_ACT_POLL_INTERVAL_MIN_MS,
+				     NVME_FW_ACT_POLL_INTERVAL_MAX_MS);
+}
+
 /*
  * nvme_wq - hosts nvme related works that are not reset or delete
  * nvme_reset_wq - hosts nvme reset works
@@ -4798,7 +4820,7 @@ static void nvme_fw_act_work(struct work_struct *work)
 			nvme_try_sched_reset(ctrl);
 			return;
 		}
-		msleep(100);
+		msleep(READ_ONCE(fw_act_poll_interval_ms));
 	}
 
 	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING) ||
-- 
2.20.1

Re: [PATCH] nvme: make firmware activation poll interval configurable

[guzebing]

Hi Keith, Jens, Christoph, Sagi,

Gentle ping.

The motivation is to reduce the extra time I/O queues remain quiesced
after the controller has finished online firmware activation. In our
Gen5 NVMe SSD update test, using a 10 ms poll interval instead of the
current 100 ms interval reduced the observed completion detection time
from 311 ms to 266 ms.

Could you advise whether this is a latency issue worth addressing
upstream now? If so, would you prefer a fixed shorter polling interval
instead of a module parameter?

If the current benefit is not compelling enough, would it be better to
defer this until firmware activation times become shorter on more
devices, where the fixed 100 ms polling interval would account for a
larger part of the I/O quiesce time?

Thanks,
Guzebing

On 6/27/26 9:06 AM, guzebing wrote:
> From: Guzebing <guzebing@bytedance.com>
> 
> nvme_fw_act_work() polls the controller processing-paused status every
> 100 ms while firmware activation is pending. Some devices can complete
> online activation in only a few hundred milliseconds, so the fixed
> interval can add noticeable latency before the driver observes
> completion.
> 
> Add an nvme_core.fw_act_poll_interval_ms module parameter to make the
> poll interval tunable. Keep the default at 100 ms to preserve existing
> behavior, and accept values from 10 ms to 100 ms so systems that need
> faster completion detection can opt in to a shorter interval.
> 
> Signed-off-by: Guzebing <guzebing@bytedance.com>
> ---
> We recently observed this issue while performing online firmware
> updates for Gen5 NVMe SSDs in a production environment.
> 
> During firmware activation, the kernel quiesces I/O. Detecting the end
> of firmware activation earlier lets the driver unquiesce I/O earlier,
> which is important for the long-tail I/O latency of production
> workloads.
> 
> Operation steps:
>   nvme fw-download /dev/nvme13n1 \
>     -f ./Inventec_G238B_NVME_LDD5902Q_MZWL61T9HFLT-00B07.bin
>   nvme fw-commit /dev/nvme13n1 -s 0 -a 3
>   nvme fw-log /dev/nvme13n1
> 
> Operation device:
>   Samsung PM9D3a Gen5 NVMe SSD, model MZWL67T6HBLC-00B07
>   Firmware update: LDD5702Q -> LDD5902Q
> 
> Operation results:
>   100 ms polling interval: 3 polls, 311 ms
>    10 ms polling interval: 24 polls, 266 ms
> 
>  drivers/nvme/host/core.c | 24 +++++++++++++++++++++++-
>  1 file changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 3b7a8f7a35426..784da06fee559 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -58,6 +58,21 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);
>  MODULE_PARM_DESC(io_timeout, "timeout in seconds for I/O");
>  EXPORT_SYMBOL_GPL(nvme_io_timeout);
>  
> +#define NVME_FW_ACT_POLL_INTERVAL_MIN_MS	10
> +#define NVME_FW_ACT_POLL_INTERVAL_MAX_MS	100
> +static int fw_act_poll_interval_ms_set(const char *val,
> +				       const struct kernel_param *kp);
> +static const struct kernel_param_ops fw_act_poll_interval_ms_ops = {
> +	.set = fw_act_poll_interval_ms_set,
> +	.get = param_get_uint,
> +};
> +
> +static unsigned int fw_act_poll_interval_ms = 100;
> +module_param_cb(fw_act_poll_interval_ms, &fw_act_poll_interval_ms_ops,
> +		&fw_act_poll_interval_ms, 0644);
> +MODULE_PARM_DESC(fw_act_poll_interval_ms,
> +		 "firmware activation polling interval in ms (10-100)");
> +
>  static unsigned char shutdown_timeout = 5;
>  module_param(shutdown_timeout, byte, 0644);
>  MODULE_PARM_DESC(shutdown_timeout, "timeout in seconds for controller shutdown");
> @@ -106,6 +121,13 @@ module_param(disable_pi_offsets, bool, 0444);
>  MODULE_PARM_DESC(disable_pi_offsets,
>  	"disable protection information if it has an offset");
>  
> +static int fw_act_poll_interval_ms_set(const char *val,
> +				       const struct kernel_param *kp)
> +{
> +	return param_set_uint_minmax(val, kp, NVME_FW_ACT_POLL_INTERVAL_MIN_MS,
> +				     NVME_FW_ACT_POLL_INTERVAL_MAX_MS);
> +}
> +
>  /*
>   * nvme_wq - hosts nvme related works that are not reset or delete
>   * nvme_reset_wq - hosts nvme reset works
> @@ -4798,7 +4820,7 @@ static void nvme_fw_act_work(struct work_struct *work)
>  			nvme_try_sched_reset(ctrl);
>  			return;
>  		}
> -		msleep(100);
> +		msleep(READ_ONCE(fw_act_poll_interval_ms));
>  	}
>  
>  	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING) ||

Re: [PATCH] nvme: make firmware activation poll interval configurable

[Christoph Hellwig]

On Sat, Jun 27, 2026 at 09:06:10AM +0800, guzebing wrote:
> From: Guzebing <guzebing@bytedance.com>
> 
> nvme_fw_act_work() polls the controller processing-paused status every
> 100 ms while firmware activation is pending. Some devices can complete
> online activation in only a few hundred milliseconds, so the fixed
> interval can add noticeable latency before the driver observes
> completion.
> 
> Add an nvme_core.fw_act_poll_interval_ms module parameter to make the
> poll interval tunable. Keep the default at 100 ms to preserve existing
> behavior, and accept values from 10 ms to 100 ms so systems that need
> faster completion detection can opt in to a shorter interval.
> 
> Signed-off-by: Guzebing <guzebing@bytedance.com>
> ---
> We recently observed this issue while performing online firmware
> updates for Gen5 NVMe SSDs in a production environment.
> 
> During firmware activation, the kernel quiesces I/O. Detecting the end
> of firmware activation earlier lets the driver unquiesce I/O earlier,
> which is important for the long-tail I/O latency of production
> workloads.

What value does this device report in the Maximum Time for Firmware
Activation (MTFA) field?  It might make sense to scale the polling
time as a fraction of that instead of requiring a manual override.

Re: [PATCH] nvme: make firmware activation poll interval configurable

[guzebing]

On 7/9/26 2:42 PM, Christoph Hellwig wrote:
> On Sat, Jun 27, 2026 at 09:06:10AM +0800, guzebing wrote:
>> From: Guzebing <guzebing@bytedance.com>
>>
>> nvme_fw_act_work() polls the controller processing-paused status every
>> 100 ms while firmware activation is pending. Some devices can complete
>> online activation in only a few hundred milliseconds, so the fixed
>> interval can add noticeable latency before the driver observes
>> completion.
>>
>> Add an nvme_core.fw_act_poll_interval_ms module parameter to make the
>> poll interval tunable. Keep the default at 100 ms to preserve existing
>> behavior, and accept values from 10 ms to 100 ms so systems that need
>> faster completion detection can opt in to a shorter interval.
>>
>> Signed-off-by: Guzebing <guzebing@bytedance.com>
>> ---
>> We recently observed this issue while performing online firmware
>> updates for Gen5 NVMe SSDs in a production environment.
>>
>> During firmware activation, the kernel quiesces I/O. Detecting the end
>> of firmware activation earlier lets the driver unquiesce I/O earlier,
>> which is important for the long-tail I/O latency of production
>> workloads.
> 
> What value does this device report in the Maximum Time for Firmware
> Activation (MTFA) field?  It might make sense to scale the polling
> time as a fraction of that instead of requiring a manual override.
> 

The Samsung PM9D3a Gen5 SSD reports MTFA = 10, i.e. 1000 ms.

I also checked another device, an Intel/Solidigm P5520 Gen4 drive.  It
reports MTFA = 100, i.e. 10000 ms, while the observed online activation
time is about 800 ms.

I agree that deriving the polling interval from MTFA would be better
than adding a module parameter.  Given that MTFA is a conservative upper
bound rather than a good estimate of the common activation time, would
using a small fraction of it, for example MTFA / 100 clamped to 10..100
ms, be a reasonable policy for v2?

That would give 10 ms for the PM9D3a device above, while keeping the
current 100 ms interval for the P5520 case and for large-MTFA devices.

Re: [PATCH] nvme: make firmware activation poll interval configurable

[Keith Busch]

On Thu, Jul 09, 2026 at 05:06:11PM +0800, guzebing wrote:
> The Samsung PM9D3a Gen5 SSD reports MTFA = 10, i.e. 1000 ms.
> 
> I also checked another device, an Intel/Solidigm P5520 Gen4 drive.  It
> reports MTFA = 100, i.e. 10000 ms, while the observed online activation
> time is about 800 ms.
> 
> I agree that deriving the polling interval from MTFA would be better
> than adding a module parameter.  Given that MTFA is a conservative upper
> bound rather than a good estimate of the common activation time, would
> using a small fraction of it, for example MTFA / 100 clamped to 10..100
> ms, be a reasonable policy for v2?
> 
> That would give 10 ms for the PM9D3a device above, while keeping the
> current 100 ms interval for the P5520 case and for large-MTFA devices.

nvme_wait_ready() has a similar polling loop on the csts register, but
it does a udelay_range for 1-2 msecs no matter what the ready timeout
is. Maybe just do the same for consistency?

Re: [PATCH] nvme: make firmware activation poll interval configurable

[guzebing]

On 7/9/26 11:07 PM, Keith Busch wrote:
> On Thu, Jul 09, 2026 at 05:06:11PM +0800, guzebing wrote:
>> The Samsung PM9D3a Gen5 SSD reports MTFA = 10, i.e. 1000 ms.
>>
>> I also checked another device, an Intel/Solidigm P5520 Gen4 drive.  It
>> reports MTFA = 100, i.e. 10000 ms, while the observed online activation
>> time is about 800 ms.
>>
>> I agree that deriving the polling interval from MTFA would be better
>> than adding a module parameter.  Given that MTFA is a conservative upper
>> bound rather than a good estimate of the common activation time, would
>> using a small fraction of it, for example MTFA / 100 clamped to 10..100
>> ms, be a reasonable policy for v2?
>>
>> That would give 10 ms for the PM9D3a device above, while keeping the
>> current 100 ms interval for the P5520 case and for large-MTFA devices.
> 
> nvme_wait_ready() has a similar polling loop on the csts register, but
> it does a udelay_range for 1-2 msecs no matter what the ready timeout
> is. Maybe just do the same for consistency?

Keith, using the same 1-2 ms sleep range as nvme_wait_ready() looks
simpler to me as well.

I measured two long-running nvme_wait_ready() calls in the normal
nvme-pci NVMe controller-reset path for a live Intel P5520
(SSDPF2KX076T1), not the PCIe FLR fallback:

  nvme_disable_ctrl() cleared NVME_CC_ENABLE in ctrl->ctrl_config,
  wrote the cached value to CC, and then waited for CSTS.RDY == 0:
  1353 ms

  nvme_enable_ctrl() set NVME_CC_ENABLE in ctrl->ctrl_config, wrote
  the cached value to CC, and then waited for CSTS.RDY == 1: 1070 ms

Both long-running waits used an effective timeout argument of 30
seconds. The same drive reports MTFA = 100 (10 seconds), while its
observed online firmware activation time was about 800 ms. Thus, on
this device, the ready waits already use the 1-2 ms sleep range for
longer than the observed firmware activation.

Thanks both for the suggestions. Christoph, would this approach work
for you as well? If so, I can respin v2 to drop the module parameter
and replace msleep(100) with usleep_range(1000, 2000) in
nvme_fw_act_work().