* [PATCH 0/2] fio: steady state for latency
@ 2025-07-25 6:17 Luis Chamberlain
2025-07-25 6:17 ` [PATCH 1/2] configure: libnfs + gnutls Luis Chamberlain
2025-07-25 6:17 ` [PATCH 2/2] fio: add latency steady state detection Luis Chamberlain
0 siblings, 2 replies; 6+ messages in thread
From: Luis Chamberlain @ 2025-07-25 6:17 UTC (permalink / raw)
To: vincent.fu, fio; +Cc: mcgrof
I've been testing Claude Code for things lately, I figured one good task
may be to try steady state latency support which we lack support for. It
also fixed a small build bug.
Luis Chamberlain (2):
configure: libnfs + gnutls
fio: add latency steady state detection
HOWTO.rst | 12 +++++
client.c | 1 +
configure | 4 +-
example_latency_steadystate.fio | 47 ++++++++++++++++++++
options.c | 26 ++++++++++-
stat.h | 10 +++++
steadystate.c | 78 ++++++++++++++++++++++++++++-----
steadystate.h | 7 +++
8 files changed, 170 insertions(+), 15 deletions(-)
create mode 100644 example_latency_steadystate.fio
--
2.47.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/2] configure: libnfs + gnutls
2025-07-25 6:17 [PATCH 0/2] fio: steady state for latency Luis Chamberlain
@ 2025-07-25 6:17 ` Luis Chamberlain
2025-07-25 17:44 ` Vincent Fu
2025-07-25 6:17 ` [PATCH 2/2] fio: add latency steady state detection Luis Chamberlain
1 sibling, 1 reply; 6+ messages in thread
From: Luis Chamberlain @ 2025-07-25 6:17 UTC (permalink / raw)
To: vincent.fu, fio; +Cc: mcgrof
The problem is that the configure script tries to get both libnfs and
gnutls flags together, but gnutls is missing. Let me check the configure
script's logic and fix it:
The configure script is trying to link both libnfs and gnutls together,
but gnutls is not installed. Let me fix this by modifying the configure
script to only require libnfs:
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
configure | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/configure b/configure
index 9e69dc4b..1769ef38 100755
--- a/configure
+++ b/configure
@@ -2359,8 +2359,8 @@ print_config "DAOS File System (dfs) Engine" "$dfs"
if test "$libnfs" != "no" ; then
if $(pkg-config libnfs > /dev/null 2>&1); then
libnfs="yes"
- libnfs_cflags=$(pkg-config --cflags libnfs gnutls)
- libnfs_libs=$(pkg-config --libs libnfs gnutls)
+ libnfs_cflags=$(pkg-config --cflags libnfs)
+ libnfs_libs=$(pkg-config --libs libnfs)
else
if test "$libnfs" = "yes" ; then
feature_not_found "libnfs" "libnfs"
--
2.47.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/2] fio: add latency steady state detection
2025-07-25 6:17 [PATCH 0/2] fio: steady state for latency Luis Chamberlain
2025-07-25 6:17 ` [PATCH 1/2] configure: libnfs + gnutls Luis Chamberlain
@ 2025-07-25 6:17 ` Luis Chamberlain
2025-07-28 18:14 ` Vincent Fu
2025-07-28 21:27 ` Sitsofe Wheeler
1 sibling, 2 replies; 6+ messages in thread
From: Luis Chamberlain @ 2025-07-25 6:17 UTC (permalink / raw)
To: vincent.fu, fio; +Cc: mcgrof
Add fio latency steady state support. This is based on the SNIA SSD
Performance Test Specification requirements for latency steady state
detection. The implementation calculates weighted average latency across
all I/O directions and supports both maximum mean deviation and slope-based
detection methods.
Tested successfully against NVMe device with debug output confirming
proper latency calculation and steady state evaluation.
Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
HOWTO.rst | 12 +++++
client.c | 1 +
example_latency_steadystate.fio | 47 ++++++++++++++++++++
options.c | 26 ++++++++++-
stat.h | 10 +++++
steadystate.c | 78 ++++++++++++++++++++++++++++-----
steadystate.h | 7 +++
7 files changed, 168 insertions(+), 13 deletions(-)
create mode 100644 example_latency_steadystate.fio
diff --git a/HOWTO.rst b/HOWTO.rst
index 55ebc388..c5e6a0ad 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -4154,6 +4154,18 @@ Steady state
Collect bandwidth data and calculate the least squares regression
slope. Stop the job if the slope falls below the specified limit.
+ **lat**
+ Collect completion latency data and calculate the maximum mean
+ deviation. Stop the job if the deviation falls below the specified
+ limit. The latency values are weighted by the number of I/O samples
+ in each measurement interval.
+
+ **lat_slope**
+ Collect completion latency data and calculate the least squares
+ regression slope. Stop the job if the slope falls below the
+ specified limit. The latency values are weighted by the number
+ of I/O samples in each measurement interval.
+
.. option:: steadystate_duration=time, ss_dur=time
A rolling window of this duration will be used to judge whether steady
diff --git a/client.c b/client.c
index 923b092e..af37aea1 100644
--- a/client.c
+++ b/client.c
@@ -1077,6 +1077,7 @@ static void convert_ts(struct thread_stat *dst, struct thread_stat *src)
for (i = 0; i < dst->ss_dur; i++ ) {
dst->ss_iops_data[i] = le64_to_cpu(src->ss_iops_data[i]);
dst->ss_bw_data[i] = le64_to_cpu(src->ss_bw_data[i]);
+ dst->ss_lat_data[i] = le64_to_cpu(src->ss_lat_data[i]);
}
}
diff --git a/example_latency_steadystate.fio b/example_latency_steadystate.fio
new file mode 100644
index 00000000..b769ad15
--- /dev/null
+++ b/example_latency_steadystate.fio
@@ -0,0 +1,47 @@
+# Example FIO job file demonstrating latency steady state detection
+# This example shows how to use FIO's latency steady state detection
+# to automatically terminate workloads when latency stabilizes
+#
+# Based on SNIA SSD Performance Test Specification requirements:
+# - Steady state is achieved when latency measurements don't change more than
+# 20% for 5 measurement windows and remain within 5% of a line with 10% slope
+# - This example uses more conservative 5% deviation threshold for demonstration
+
+[global]
+# Basic I/O parameters
+ioengine=libaio
+iodepth=32
+bs=4k
+direct=1
+rw=randread
+numjobs=1
+time_based=1
+runtime=3600 # Max runtime: 1 hour (will terminate early if steady state reached)
+
+# Steady state detection parameters
+steadystate=lat:5% # Stop when latency mean deviation < 5% of average
+steadystate_duration=300 # Use 5-minute rolling window for measurements
+steadystate_ramp_time=60 # Wait 1 minute before starting measurements
+steadystate_check_interval=10 # Take measurements every 10 seconds
+
+# Output options
+write_lat_log=lat_steadystate
+log_avg_msec=10000 # Log average latency every 10 seconds
+
+[latency_steady_test]
+filename=/dev/nvme3n1
+size=10G
+
+# Alternative steady state configurations (uncomment to try):
+
+# Use slope-based detection instead of deviation:
+# steadystate=lat_slope:0.1%
+
+# More aggressive detection (faster convergence):
+# steadystate=lat:2%
+# steadystate_duration=120 # 2-minute window
+# steadystate_check_interval=5 # Check every 5 seconds
+
+# More conservative detection (slower convergence):
+# steadystate=lat:10%
+# steadystate_duration=600 # 10-minute window
diff --git a/options.c b/options.c
index 6295a616..8884dd8a 100644
--- a/options.c
+++ b/options.c
@@ -1370,7 +1370,8 @@ static int str_steadystate_cb(void *data, const char *str)
long long ll;
if (td->o.ss_state != FIO_SS_IOPS && td->o.ss_state != FIO_SS_IOPS_SLOPE &&
- td->o.ss_state != FIO_SS_BW && td->o.ss_state != FIO_SS_BW_SLOPE) {
+ td->o.ss_state != FIO_SS_BW && td->o.ss_state != FIO_SS_BW_SLOPE &&
+ td->o.ss_state != FIO_SS_LAT && td->o.ss_state != FIO_SS_LAT_SLOPE) {
/* should be impossible to get here */
log_err("fio: unknown steady state criterion\n");
return 1;
@@ -1414,6 +1415,21 @@ static int str_steadystate_cb(void *data, const char *str)
return 0;
td->o.ss_limit.u.f = val;
+ } else if (td->o.ss_state & FIO_SS_LAT) {
+ long long tns;
+ if (check_str_time(nr, &tns, 0)) {
+ log_err("fio: steadystate latency threshold parsing failed\n");
+ free(nr);
+ return 1;
+ }
+
+ dprint(FD_PARSE, "set steady state latency threshold to %lld nsec\n", tns);
+ free(nr);
+ if (parse_dryrun())
+ return 0;
+
+ td->o.ss_limit.u.f = (double) tns;
+
} else { /* bandwidth criterion */
if (str_to_decimal(nr, &ll, 1, td, 0, 0)) {
log_err("fio: steadystate BW threshold postfix parsing failed\n");
@@ -5489,6 +5505,14 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
.oval = FIO_SS_BW_SLOPE,
.help = "slope calculated from bandwidth measurements",
},
+ { .ival = "lat",
+ .oval = FIO_SS_LAT,
+ .help = "maximum mean deviation of latency measurements",
+ },
+ { .ival = "lat_slope",
+ .oval = FIO_SS_LAT_SLOPE,
+ .help = "slope calculated from latency measurements",
+ },
},
.category = FIO_OPT_C_GENERAL,
.group = FIO_OPT_G_RUNTIME,
diff --git a/stat.h b/stat.h
index ac74d6c2..fad7e8d3 100644
--- a/stat.h
+++ b/stat.h
@@ -282,6 +282,16 @@ struct thread_stat {
uint64_t pad5;
};
+ union {
+ uint64_t *ss_lat_data;
+ /*
+ * For FIO_NET_CMD_TS, the pointed to data will temporarily
+ * be stored at this offset from the start of the payload.
+ */
+ uint64_t ss_lat_data_offset;
+ uint64_t pad5b;
+ };
+
union {
struct clat_prio_stat *clat_prio[DDIR_RWDIR_CNT];
/*
diff --git a/steadystate.c b/steadystate.c
index 3e3683f3..96924b96 100644
--- a/steadystate.c
+++ b/steadystate.c
@@ -10,8 +10,10 @@ void steadystate_free(struct thread_data *td)
{
free(td->ss.iops_data);
free(td->ss.bw_data);
+ free(td->ss.lat_data);
td->ss.iops_data = NULL;
td->ss.bw_data = NULL;
+ td->ss.lat_data = NULL;
}
static void steadystate_alloc(struct thread_data *td)
@@ -20,6 +22,7 @@ static void steadystate_alloc(struct thread_data *td)
td->ss.bw_data = calloc(intervals, sizeof(uint64_t));
td->ss.iops_data = calloc(intervals, sizeof(uint64_t));
+ td->ss.lat_data = calloc(intervals, sizeof(uint64_t));
td->ss.state |= FIO_SS_DATA;
}
@@ -60,7 +63,7 @@ void steadystate_setup(void)
steadystate_alloc(prev_td);
}
-static bool steadystate_slope(uint64_t iops, uint64_t bw,
+static bool steadystate_slope(uint64_t iops, uint64_t bw, double lat,
struct thread_data *td)
{
int i, j;
@@ -71,11 +74,14 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
ss->bw_data[ss->tail] = bw;
ss->iops_data[ss->tail] = iops;
+ ss->lat_data[ss->tail] = (uint64_t)lat;
if (ss->state & FIO_SS_IOPS)
new_val = iops;
- else
+ else if (ss->state & FIO_SS_BW)
new_val = bw;
+ else
+ new_val = (uint64_t)lat;
if (ss->state & FIO_SS_BUFFER_FULL || ss->tail - ss->head == intervals - 1) {
if (!(ss->state & FIO_SS_BUFFER_FULL)) {
@@ -83,13 +89,17 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
for (i = 0, ss->sum_y = 0; i < intervals; i++) {
if (ss->state & FIO_SS_IOPS)
ss->sum_y += ss->iops_data[i];
- else
+ else if (ss->state & FIO_SS_BW)
ss->sum_y += ss->bw_data[i];
+ else
+ ss->sum_y += ss->lat_data[i];
j = (ss->head + i) % intervals;
if (ss->state & FIO_SS_IOPS)
ss->sum_xy += i * ss->iops_data[j];
- else
+ else if (ss->state & FIO_SS_BW)
ss->sum_xy += i * ss->bw_data[j];
+ else
+ ss->sum_xy += i * ss->lat_data[j];
}
ss->state |= FIO_SS_BUFFER_FULL;
} else { /* easy to update the sums */
@@ -100,8 +110,10 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
if (ss->state & FIO_SS_IOPS)
ss->oldest_y = ss->iops_data[ss->head];
- else
+ else if (ss->state & FIO_SS_BW)
ss->oldest_y = ss->bw_data[ss->head];
+ else
+ ss->oldest_y = ss->lat_data[ss->head];
/*
* calculate slope as (sum_xy - sum_x * sum_y / n) / (sum_(x^2)
@@ -134,7 +146,7 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
return false;
}
-static bool steadystate_deviation(uint64_t iops, uint64_t bw,
+static bool steadystate_deviation(uint64_t iops, uint64_t bw, double lat,
struct thread_data *td)
{
int i;
@@ -146,6 +158,7 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,
ss->bw_data[ss->tail] = bw;
ss->iops_data[ss->tail] = iops;
+ ss->lat_data[ss->tail] = (uint64_t)lat;
if (ss->state & FIO_SS_BUFFER_FULL || ss->tail - ss->head == intervals - 1) {
if (!(ss->state & FIO_SS_BUFFER_FULL)) {
@@ -153,22 +166,28 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,
for (i = 0, ss->sum_y = 0; i < intervals; i++) {
if (ss->state & FIO_SS_IOPS)
ss->sum_y += ss->iops_data[i];
- else
+ else if (ss->state & FIO_SS_BW)
ss->sum_y += ss->bw_data[i];
+ else
+ ss->sum_y += ss->lat_data[i];
}
ss->state |= FIO_SS_BUFFER_FULL;
} else { /* easy to update the sum */
ss->sum_y -= ss->oldest_y;
if (ss->state & FIO_SS_IOPS)
ss->sum_y += ss->iops_data[ss->tail];
- else
+ else if (ss->state & FIO_SS_BW)
ss->sum_y += ss->bw_data[ss->tail];
+ else
+ ss->sum_y += ss->lat_data[ss->tail];
}
if (ss->state & FIO_SS_IOPS)
ss->oldest_y = ss->iops_data[ss->head];
- else
+ else if (ss->state & FIO_SS_BW)
ss->oldest_y = ss->bw_data[ss->head];
+ else
+ ss->oldest_y = ss->lat_data[ss->head];
mean = (double) ss->sum_y / intervals;
ss->deviation = 0.0;
@@ -176,8 +195,10 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,
for (i = 0; i < intervals; i++) {
if (ss->state & FIO_SS_IOPS)
diff = ss->iops_data[i] - mean;
- else
+ else if (ss->state & FIO_SS_BW)
diff = ss->bw_data[i] - mean;
+ else
+ diff = ss->lat_data[i] - mean;
ss->deviation = max(ss->deviation, diff * (diff < 0.0 ? -1.0 : 1.0));
}
@@ -209,13 +230,18 @@ int steadystate_check(void)
unsigned long rate_time;
struct timespec now;
uint64_t group_bw = 0, group_iops = 0;
+ double group_lat_sum = 0.0;
+ uint64_t group_lat_samples = 0;
uint64_t td_iops, td_bytes;
+ double group_lat;
bool ret;
prev_groupid = -1;
for_each_td(td) {
const bool needs_lock = td_async_processing(td);
struct steadystate_data *ss = &td->ss;
+ double td_lat_sum = 0.0;
+ uint64_t td_lat_samples = 0;
if (!ss->dur || td->runstate <= TD_SETTING_UP ||
td->runstate >= TD_EXITED || !ss->state ||
@@ -228,6 +254,8 @@ int steadystate_check(void)
(td->o.group_reporting && td->groupid != prev_groupid)) {
group_bw = 0;
group_iops = 0;
+ group_lat_sum = 0.0;
+ group_lat_samples = 0;
group_ramp_time_over = 0;
}
prev_groupid = td->groupid;
@@ -248,6 +276,9 @@ int steadystate_check(void)
for (ddir = 0; ddir < DDIR_RWDIR_CNT; ddir++) {
td_iops += td->io_blocks[ddir];
td_bytes += td->io_bytes[ddir];
+ td_lat_sum += td->ts.clat_stat[ddir].mean.u.f *
+ td->ts.clat_stat[ddir].samples;
+ td_lat_samples += td->ts.clat_stat[ddir].samples;
}
if (needs_lock)
@@ -261,10 +292,14 @@ int steadystate_check(void)
(ss_check_interval * ss_check_interval / 1000L);
group_iops += rate_time * (td_iops - ss->prev_iops) /
(ss_check_interval * ss_check_interval / 1000L);
+ group_lat_sum += td_lat_sum - ss->prev_lat_sum;
+ group_lat_samples += td_lat_samples - ss->prev_lat_samples;
++group_ramp_time_over;
}
ss->prev_iops = td_iops;
ss->prev_bytes = td_bytes;
+ ss->prev_lat_sum = td_lat_sum;
+ ss->prev_lat_samples = td_lat_samples;
if (td->o.group_reporting && !(ss->state & FIO_SS_DATA))
continue;
@@ -284,10 +319,14 @@ int steadystate_check(void)
(unsigned long long) group_bw,
ss->head, ss->tail);
+ group_lat = 0.0;
+ if (group_lat_samples)
+ group_lat = group_lat_sum / group_lat_samples;
+
if (ss->state & FIO_SS_SLOPE)
- ret = steadystate_slope(group_iops, group_bw, td);
+ ret = steadystate_slope(group_iops, group_bw, group_lat, td);
else
- ret = steadystate_deviation(group_iops, group_bw, td);
+ ret = steadystate_deviation(group_iops, group_bw, group_lat, td);
if (ret) {
if (td->o.group_reporting) {
@@ -382,3 +421,18 @@ uint64_t steadystate_iops_mean(struct thread_stat *ts)
return sum / intervals;
}
+
+uint64_t steadystate_lat_mean(struct thread_stat *ts)
+{
+ int i;
+ uint64_t sum;
+ int intervals = ts->ss_dur / (ss_check_interval / 1000L);
+
+ if (!ts->ss_dur)
+ return 0;
+
+ for (i = 0, sum = 0; i < intervals; i++)
+ sum += ts->ss_lat_data[i];
+
+ return sum / intervals;
+}
diff --git a/steadystate.h b/steadystate.h
index f1ef2b20..fffcb463 100644
--- a/steadystate.h
+++ b/steadystate.h
@@ -9,6 +9,7 @@ extern void steadystate_setup(void);
extern int td_steadystate_init(struct thread_data *);
extern uint64_t steadystate_bw_mean(struct thread_stat *);
extern uint64_t steadystate_iops_mean(struct thread_stat *);
+extern uint64_t steadystate_lat_mean(struct thread_stat *);
extern bool steadystate_enabled;
extern unsigned int ss_check_interval;
@@ -24,6 +25,7 @@ struct steadystate_data {
unsigned int tail;
uint64_t *iops_data;
uint64_t *bw_data;
+ uint64_t *lat_data;
double slope;
double deviation;
@@ -38,6 +40,8 @@ struct steadystate_data {
struct timespec prev_time;
uint64_t prev_iops;
uint64_t prev_bytes;
+ double prev_lat_sum;
+ uint64_t prev_lat_samples;
};
enum {
@@ -49,6 +53,7 @@ enum {
__FIO_SS_DATA,
__FIO_SS_PCT,
__FIO_SS_BUFFER_FULL,
+ __FIO_SS_LAT,
};
enum {
@@ -60,9 +65,11 @@ enum {
FIO_SS_DATA = 1 << __FIO_SS_DATA,
FIO_SS_PCT = 1 << __FIO_SS_PCT,
FIO_SS_BUFFER_FULL = 1 << __FIO_SS_BUFFER_FULL,
+ FIO_SS_LAT = 1 << __FIO_SS_LAT,
FIO_SS_IOPS_SLOPE = FIO_SS_IOPS | FIO_SS_SLOPE,
FIO_SS_BW_SLOPE = FIO_SS_BW | FIO_SS_SLOPE,
+ FIO_SS_LAT_SLOPE = FIO_SS_LAT | FIO_SS_SLOPE,
};
#endif
--
2.47.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] configure: libnfs + gnutls
2025-07-25 6:17 ` [PATCH 1/2] configure: libnfs + gnutls Luis Chamberlain
@ 2025-07-25 17:44 ` Vincent Fu
0 siblings, 0 replies; 6+ messages in thread
From: Vincent Fu @ 2025-07-25 17:44 UTC (permalink / raw)
To: Luis Chamberlain, vincent.fu, fio
On 7/25/25 2:17 AM, Luis Chamberlain wrote:
> The problem is that the configure script tries to get both libnfs and
> gnutls flags together, but gnutls is missing. Let me check the configure
> script's logic and fix it:
>
> The configure script is trying to link both libnfs and gnutls together,
> but gnutls is not installed. Let me fix this by modifying the configure
> script to only require libnfs:
>
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
> configure | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/configure b/configure
> index 9e69dc4b..1769ef38 100755
> --- a/configure
> +++ b/configure
> @@ -2359,8 +2359,8 @@ print_config "DAOS File System (dfs) Engine" "$dfs"
> if test "$libnfs" != "no" ; then
> if $(pkg-config libnfs > /dev/null 2>&1); then
> libnfs="yes"
> - libnfs_cflags=$(pkg-config --cflags libnfs gnutls)
> - libnfs_libs=$(pkg-config --libs libnfs gnutls)
> + libnfs_cflags=$(pkg-config --cflags libnfs)
> + libnfs_libs=$(pkg-config --libs libnfs)
> else
> if test "$libnfs" = "yes" ; then
> feature_not_found "libnfs" "libnfs"
The gnutls requirement was added via
https://lore.kernel.org/fio/4369d6e2-48d4-43cb-965a-b376dc559dad@gmail.com/T/#t
I can think of two ways to resolve this:
- Make sure gnutls is installed when building fio
- Modify the configure script to detect whether or not gnutls really is
needed by libnfs and set up the flags accordingly
Vincent
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] fio: add latency steady state detection
2025-07-25 6:17 ` [PATCH 2/2] fio: add latency steady state detection Luis Chamberlain
@ 2025-07-28 18:14 ` Vincent Fu
2025-07-28 21:27 ` Sitsofe Wheeler
1 sibling, 0 replies; 6+ messages in thread
From: Vincent Fu @ 2025-07-28 18:14 UTC (permalink / raw)
To: Luis Chamberlain, vincent.fu, fio
On 7/25/25 2:17 AM, Luis Chamberlain wrote:
> Add fio latency steady state support. This is based on the SNIA SSD
> Performance Test Specification requirements for latency steady state
> detection. The implementation calculates weighted average latency across
> all I/O directions and supports both maximum mean deviation and slope-based
> detection methods.
>
> Tested successfully against NVMe device with debug output confirming
> proper latency calculation and steady state evaluation.
>
> Generated-by: Claude AI
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Please add some more details about how closely this adheres to SNIA's
definition of steady state. From
https://www.snia.org/sites/default/files/technical-work/pts/release/SNIA-SSS-PTS-2.0.2.pdf
Here is SNIA's definition of steady state:
2.1.24 Steady State: A device is said to be in Steady State when, for
the dependent variable (y)
being tracked:
a) Range(y) is less than 20% of Ave(y): Max(y)-Min(y) within the
Measurement Window
is no more than 20% of the Ave(y) within the Measurement Window; and
b) Slope(y) is less than 10%: Max(y)-Min(y), where Max(y) and Min(y) are
the maximum
and minimum values on the best linear curve fit of the y-values within
the Measurement
Window, is within 10% of Ave(y) value within the Measurement Window.
SNIA requires both slope *and* maximum deviation criteria to
simultaneously be met whereas fio considers only one criterion. It might
be worthwhile to add to fio an option to consider both criteria.
Also b) is unclear to me. "Slope(y) is less than 10%" of what? Perhaps
this is meant to refer to 10% of Ave(y) but it would be better to spell
this out explicitly.
Finally, fio does not consider the second part of the slope criterion
concerning the range of values on the best linear curve fit.
> ---
> HOWTO.rst | 12 +++++
> client.c | 1 +
> example_latency_steadystate.fio | 47 ++++++++++++++++++++
> options.c | 26 ++++++++++-
> stat.h | 10 +++++
> steadystate.c | 78 ++++++++++++++++++++++++++++-----
> steadystate.h | 7 +++
> 7 files changed, 168 insertions(+), 13 deletions(-)
> create mode 100644 example_latency_steadystate.fio
>
> diff --git a/HOWTO.rst b/HOWTO.rst
> index 55ebc388..c5e6a0ad 100644
> --- a/HOWTO.rst
> +++ b/HOWTO.rst
> @@ -4154,6 +4154,18 @@ Steady state
> Collect bandwidth data and calculate the least squares regression
> slope. Stop the job if the slope falls below the specified limit.
>
> + **lat**
> + Collect completion latency data and calculate the maximum mean
> + deviation. Stop the job if the deviation falls below the specified
> + limit. The latency values are weighted by the number of I/O samples
> + in each measurement interval.
> +
This is unstated but it should be made explicit that latency steady
state is determined using mean latency values for each period with
duration specified by ss_interval.
It's not obvious to me how the weigthing is accomplished or whether
weighting is desirable. Having gone through the patch, I don't see any
unexpected weighting happening. Each IO during an interval is given the
same weight. I would just remove the statement about weighting.
> + **lat_slope**
> + Collect completion latency data and calculate the least squares
> + regression slope. Stop the job if the slope falls below the
> + specified limit. The latency values are weighted by the number
> + of I/O samples in each measurement interval.
> +
Same as above.
> .. option:: steadystate_duration=time, ss_dur=time
>
> A rolling window of this duration will be used to judge whether steady
> diff --git a/client.c b/client.c
> index 923b092e..af37aea1 100644
> --- a/client.c
> +++ b/client.c
> @@ -1077,6 +1077,7 @@ static void convert_ts(struct thread_stat *dst, struct thread_stat *src)
> for (i = 0; i < dst->ss_dur; i++ ) {
> dst->ss_iops_data[i] = le64_to_cpu(src->ss_iops_data[i]);
> dst->ss_bw_data[i] = le64_to_cpu(src->ss_bw_data[i]);
> + dst->ss_lat_data[i] = le64_to_cpu(src->ss_lat_data[i]);
> }
> }
>
> diff --git a/example_latency_steadystate.fio b/example_latency_steadystate.fio
> new file mode 100644
> index 00000000..b769ad15
> --- /dev/null
> +++ b/example_latency_steadystate.fio
> @@ -0,0 +1,47 @@
> +# Example FIO job file demonstrating latency steady state detection
> +# This example shows how to use FIO's latency steady state detection
> +# to automatically terminate workloads when latency stabilizes
> +#
> +# Based on SNIA SSD Performance Test Specification requirements:
> +# - Steady state is achieved when latency measurements don't change more than
> +# 20% for 5 measurement windows and remain within 5% of a line with 10% slope
> +# - This example uses more conservative 5% deviation threshold for demonstration
> +
> +[global]
> +# Basic I/O parameters
> +ioengine=libaio
> +iodepth=32
> +bs=4k
> +direct=1
> +rw=randread
> +numjobs=1
> +time_based=1
> +runtime=3600 # Max runtime: 1 hour (will terminate early if steady state reached)
> +
> +# Steady state detection parameters
> +steadystate=lat:5% # Stop when latency mean deviation < 5% of average
> +steadystate_duration=300 # Use 5-minute rolling window for measurements
> +steadystate_ramp_time=60 # Wait 1 minute before starting measurements
> +steadystate_check_interval=10 # Take measurements every 10 seconds
> +
> +# Output options
> +write_lat_log=lat_steadystate
> +log_avg_msec=10000 # Log average latency every 10 seconds
> +
> +[latency_steady_test]
> +filename=/dev/nvme3n1
> +size=10G
> +
> +# Alternative steady state configurations (uncomment to try):
> +
> +# Use slope-based detection instead of deviation:
> +# steadystate=lat_slope:0.1%
> +
> +# More aggressive detection (faster convergence):
> +# steadystate=lat:2%
> +# steadystate_duration=120 # 2-minute window
> +# steadystate_check_interval=5 # Check every 5 seconds
> +
> +# More conservative detection (slower convergence):
> +# steadystate=lat:10%
> +# steadystate_duration=600 # 10-minute window
> diff --git a/options.c b/options.c
> index 6295a616..8884dd8a 100644
> --- a/options.c
> +++ b/options.c
> @@ -1370,7 +1370,8 @@ static int str_steadystate_cb(void *data, const char *str)
> long long ll;
>
> if (td->o.ss_state != FIO_SS_IOPS && td->o.ss_state != FIO_SS_IOPS_SLOPE &&
> - td->o.ss_state != FIO_SS_BW && td->o.ss_state != FIO_SS_BW_SLOPE) {
> + td->o.ss_state != FIO_SS_BW && td->o.ss_state != FIO_SS_BW_SLOPE &&
> + td->o.ss_state != FIO_SS_LAT && td->o.ss_state != FIO_SS_LAT_SLOPE) {
> /* should be impossible to get here */
> log_err("fio: unknown steady state criterion\n");
> return 1;
> @@ -1414,6 +1415,21 @@ static int str_steadystate_cb(void *data, const char *str)
> return 0;
>
> td->o.ss_limit.u.f = val;
> + } else if (td->o.ss_state & FIO_SS_LAT) {
> + long long tns;
> + if (check_str_time(nr, &tns, 0)) {
> + log_err("fio: steadystate latency threshold parsing failed\n");
> + free(nr);
> + return 1;
> + }
> +
> + dprint(FD_PARSE, "set steady state latency threshold to %lld nsec\n", tns);
> + free(nr);
> + if (parse_dryrun())
> + return 0;
> +
> + td->o.ss_limit.u.f = (double) tns;
Why make this a double instead of an integer?
> +
> } else { /* bandwidth criterion */
> if (str_to_decimal(nr, &ll, 1, td, 0, 0)) {
> log_err("fio: steadystate BW threshold postfix parsing failed\n");
> @@ -5489,6 +5505,14 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
> .oval = FIO_SS_BW_SLOPE,
> .help = "slope calculated from bandwidth measurements",
> },
> + { .ival = "lat",
> + .oval = FIO_SS_LAT,
> + .help = "maximum mean deviation of latency measurements",
> + },
> + { .ival = "lat_slope",
> + .oval = FIO_SS_LAT_SLOPE,
> + .help = "slope calculated from latency measurements",
> + },
> },
> .category = FIO_OPT_C_GENERAL,
> .group = FIO_OPT_G_RUNTIME,
> diff --git a/stat.h b/stat.h
> index ac74d6c2..fad7e8d3 100644
> --- a/stat.h
> +++ b/stat.h
> @@ -282,6 +282,16 @@ struct thread_stat {
> uint64_t pad5;
> };
>
> + union {
> + uint64_t *ss_lat_data;
> + /*
> + * For FIO_NET_CMD_TS, the pointed to data will temporarily
> + * be stored at this offset from the start of the payload.
> + */
> + uint64_t ss_lat_data_offset;
> + uint64_t pad5b;
> + };
> +
I don't see any changes in client.c to process this data on the
receiving end or in server.c to send this data from the server.
> union {
> struct clat_prio_stat *clat_prio[DDIR_RWDIR_CNT];
> /*
> diff --git a/steadystate.c b/steadystate.c
> index 3e3683f3..96924b96 100644
> --- a/steadystate.c
> +++ b/steadystate.c
> @@ -10,8 +10,10 @@ void steadystate_free(struct thread_data *td)
> {
> free(td->ss.iops_data);
> free(td->ss.bw_data);
> + free(td->ss.lat_data);
> td->ss.iops_data = NULL;
> td->ss.bw_data = NULL;
> + td->ss.lat_data = NULL;
> }
>
> static void steadystate_alloc(struct thread_data *td)
> @@ -20,6 +22,7 @@ static void steadystate_alloc(struct thread_data *td)
>
> td->ss.bw_data = calloc(intervals, sizeof(uint64_t));
> td->ss.iops_data = calloc(intervals, sizeof(uint64_t));
> + td->ss.lat_data = calloc(intervals, sizeof(uint64_t));
>
> td->ss.state |= FIO_SS_DATA;
> }
> @@ -60,7 +63,7 @@ void steadystate_setup(void)
> steadystate_alloc(prev_td);
> }
>
> -static bool steadystate_slope(uint64_t iops, uint64_t bw,
> +static bool steadystate_slope(uint64_t iops, uint64_t bw, double lat,
> struct thread_data *td)
I would make lat uint64_t instead of double to be consistent with iops
and bw.
> {
> int i, j;
> @@ -71,11 +74,14 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
>
> ss->bw_data[ss->tail] = bw;
> ss->iops_data[ss->tail] = iops;
> + ss->lat_data[ss->tail] = (uint64_t)lat;
>
> if (ss->state & FIO_SS_IOPS)
> new_val = iops;
> - else
> + else if (ss->state & FIO_SS_BW)
> new_val = bw;
> + else
> + new_val = (uint64_t)lat;
>
> if (ss->state & FIO_SS_BUFFER_FULL || ss->tail - ss->head == intervals - 1) {
> if (!(ss->state & FIO_SS_BUFFER_FULL)) {
> @@ -83,13 +89,17 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
> for (i = 0, ss->sum_y = 0; i < intervals; i++) {
> if (ss->state & FIO_SS_IOPS)
> ss->sum_y += ss->iops_data[i];
> - else
> + else if (ss->state & FIO_SS_BW)
> ss->sum_y += ss->bw_data[i];
> + else
> + ss->sum_y += ss->lat_data[i];
> j = (ss->head + i) % intervals;
> if (ss->state & FIO_SS_IOPS)
> ss->sum_xy += i * ss->iops_data[j];
> - else
> + else if (ss->state & FIO_SS_BW)
> ss->sum_xy += i * ss->bw_data[j];
> + else
> + ss->sum_xy += i * ss->lat_data[j];
> }
> ss->state |= FIO_SS_BUFFER_FULL;
> } else { /* easy to update the sums */
> @@ -100,8 +110,10 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
>
> if (ss->state & FIO_SS_IOPS)
> ss->oldest_y = ss->iops_data[ss->head];
> - else
> + else if (ss->state & FIO_SS_BW)
> ss->oldest_y = ss->bw_data[ss->head];
> + else
> + ss->oldest_y = ss->lat_data[ss->head];
>
> /*
> * calculate slope as (sum_xy - sum_x * sum_y / n) / (sum_(x^2)
> @@ -134,7 +146,7 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,
> return false;
> }
>
> -static bool steadystate_deviation(uint64_t iops, uint64_t bw,
> +static bool steadystate_deviation(uint64_t iops, uint64_t bw, double lat,
> struct thread_data *td)
Same comment as above.
> {
> int i;
> @@ -146,6 +158,7 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,
>
> ss->bw_data[ss->tail] = bw;
> ss->iops_data[ss->tail] = iops;
> + ss->lat_data[ss->tail] = (uint64_t)lat;
>
> if (ss->state & FIO_SS_BUFFER_FULL || ss->tail - ss->head == intervals - 1) {
> if (!(ss->state & FIO_SS_BUFFER_FULL)) {
> @@ -153,22 +166,28 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,
> for (i = 0, ss->sum_y = 0; i < intervals; i++) {
> if (ss->state & FIO_SS_IOPS)
> ss->sum_y += ss->iops_data[i];
> - else
> + else if (ss->state & FIO_SS_BW)
> ss->sum_y += ss->bw_data[i];
> + else
> + ss->sum_y += ss->lat_data[i];
> }
> ss->state |= FIO_SS_BUFFER_FULL;
> } else { /* easy to update the sum */
> ss->sum_y -= ss->oldest_y;
> if (ss->state & FIO_SS_IOPS)
> ss->sum_y += ss->iops_data[ss->tail];
> - else
> + else if (ss->state & FIO_SS_BW)
> ss->sum_y += ss->bw_data[ss->tail];
> + else
> + ss->sum_y += ss->lat_data[ss->tail];
> }
>
> if (ss->state & FIO_SS_IOPS)
> ss->oldest_y = ss->iops_data[ss->head];
> - else
> + else if (ss->state & FIO_SS_BW)
> ss->oldest_y = ss->bw_data[ss->head];
> + else
> + ss->oldest_y = ss->lat_data[ss->head];
>
> mean = (double) ss->sum_y / intervals;
> ss->deviation = 0.0;
> @@ -176,8 +195,10 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,
> for (i = 0; i < intervals; i++) {
> if (ss->state & FIO_SS_IOPS)
> diff = ss->iops_data[i] - mean;
> - else
> + else if (ss->state & FIO_SS_BW)
> diff = ss->bw_data[i] - mean;
> + else
> + diff = ss->lat_data[i] - mean;
> ss->deviation = max(ss->deviation, diff * (diff < 0.0 ? -1.0 : 1.0));
> }
>
> @@ -209,13 +230,18 @@ int steadystate_check(void)
> unsigned long rate_time;
> struct timespec now;
> uint64_t group_bw = 0, group_iops = 0;
> + double group_lat_sum = 0.0;
> + uint64_t group_lat_samples = 0;
> uint64_t td_iops, td_bytes;
> + double group_lat;
> bool ret;
>
> prev_groupid = -1;
> for_each_td(td) {
> const bool needs_lock = td_async_processing(td);
> struct steadystate_data *ss = &td->ss;
> + double td_lat_sum = 0.0;
> + uint64_t td_lat_samples = 0;
>
> if (!ss->dur || td->runstate <= TD_SETTING_UP ||
> td->runstate >= TD_EXITED || !ss->state ||
> @@ -228,6 +254,8 @@ int steadystate_check(void)
> (td->o.group_reporting && td->groupid != prev_groupid)) {
> group_bw = 0;
> group_iops = 0;
> + group_lat_sum = 0.0;
> + group_lat_samples = 0;
> group_ramp_time_over = 0;
> }
> prev_groupid = td->groupid;
> @@ -248,6 +276,9 @@ int steadystate_check(void)
> for (ddir = 0; ddir < DDIR_RWDIR_CNT; ddir++) {
> td_iops += td->io_blocks[ddir];
> td_bytes += td->io_bytes[ddir];
> + td_lat_sum += td->ts.clat_stat[ddir].mean.u.f *
> + td->ts.clat_stat[ddir].samples;
> + td_lat_samples += td->ts.clat_stat[ddir].samples;
> }
>
> if (needs_lock)
> @@ -261,10 +292,14 @@ int steadystate_check(void)
> (ss_check_interval * ss_check_interval / 1000L);
> group_iops += rate_time * (td_iops - ss->prev_iops) /
> (ss_check_interval * ss_check_interval / 1000L);
> + group_lat_sum += td_lat_sum - ss->prev_lat_sum;
> + group_lat_samples += td_lat_samples - ss->prev_lat_samples;
> ++group_ramp_time_over;
> }
> ss->prev_iops = td_iops;
> ss->prev_bytes = td_bytes;
> + ss->prev_lat_sum = td_lat_sum;
> + ss->prev_lat_samples = td_lat_samples;
>
> if (td->o.group_reporting && !(ss->state & FIO_SS_DATA))
> continue;
> @@ -284,10 +319,14 @@ int steadystate_check(void)
> (unsigned long long) group_bw,
> ss->head, ss->tail);
>
> + group_lat = 0.0;
> + if (group_lat_samples)
> + group_lat = group_lat_sum / group_lat_samples;
> +
Are there any concerns about numerical precision here? Latency for an
interval is calculated by taking differences in the sum of completion
latency over all IOs at the beginning and the end of the period. The
sums are calculated by multiplying the count by the average latency. For
a long running workload count * (avg latency) can be very large and
there may be few changes in avg latency. Might it be possible that these
numerical issues would bias fio toward positive detection of steady
state because of how avg latency within a period is calculated?
> if (ss->state & FIO_SS_SLOPE)
> - ret = steadystate_slope(group_iops, group_bw, td);
> + ret = steadystate_slope(group_iops, group_bw, group_lat, td);
> else
> - ret = steadystate_deviation(group_iops, group_bw, td);
> + ret = steadystate_deviation(group_iops, group_bw, group_lat, td);
>
> if (ret) {
> if (td->o.group_reporting) {
> @@ -382,3 +421,18 @@ uint64_t steadystate_iops_mean(struct thread_stat *ts)
>
> return sum / intervals;
> }
> +
> +uint64_t steadystate_lat_mean(struct thread_stat *ts)
> +{
> + int i;
> + uint64_t sum;
> + int intervals = ts->ss_dur / (ss_check_interval / 1000L);
> +
> + if (!ts->ss_dur)
> + return 0;
> +
> + for (i = 0, sum = 0; i < intervals; i++)
> + sum += ts->ss_lat_data[i];
> +
> + return sum / intervals;
> +}
There is an opportunity to eliminate duplicate code here with the
steadystate_*_mean functions.
> diff --git a/steadystate.h b/steadystate.h
> index f1ef2b20..fffcb463 100644
> --- a/steadystate.h
> +++ b/steadystate.h
> @@ -9,6 +9,7 @@ extern void steadystate_setup(void);
> extern int td_steadystate_init(struct thread_data *);
> extern uint64_t steadystate_bw_mean(struct thread_stat *);
> extern uint64_t steadystate_iops_mean(struct thread_stat *);
> +extern uint64_t steadystate_lat_mean(struct thread_stat *);
>
> extern bool steadystate_enabled;
> extern unsigned int ss_check_interval;
> @@ -24,6 +25,7 @@ struct steadystate_data {
> unsigned int tail;
> uint64_t *iops_data;
> uint64_t *bw_data;
> + uint64_t *lat_data;
>
> double slope;
> double deviation;
> @@ -38,6 +40,8 @@ struct steadystate_data {
> struct timespec prev_time;
> uint64_t prev_iops;
> uint64_t prev_bytes;
> + double prev_lat_sum;
> + uint64_t prev_lat_samples;
> };
>
> enum {
> @@ -49,6 +53,7 @@ enum {
> __FIO_SS_DATA,
> __FIO_SS_PCT,
> __FIO_SS_BUFFER_FULL,
> + __FIO_SS_LAT,
Put this next to the labels for IOPS and BW since they belong together.
> };
>
> enum {
> @@ -60,9 +65,11 @@ enum {
> FIO_SS_DATA = 1 << __FIO_SS_DATA,
> FIO_SS_PCT = 1 << __FIO_SS_PCT,
> FIO_SS_BUFFER_FULL = 1 << __FIO_SS_BUFFER_FULL,
> + FIO_SS_LAT = 1 << __FIO_SS_LAT,
Same comment as above.
>
> FIO_SS_IOPS_SLOPE = FIO_SS_IOPS | FIO_SS_SLOPE,
> FIO_SS_BW_SLOPE = FIO_SS_BW | FIO_SS_SLOPE,
> + FIO_SS_LAT_SLOPE = FIO_SS_LAT | FIO_SS_SLOPE,
> };
>
> #endif
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] fio: add latency steady state detection
2025-07-25 6:17 ` [PATCH 2/2] fio: add latency steady state detection Luis Chamberlain
2025-07-28 18:14 ` Vincent Fu
@ 2025-07-28 21:27 ` Sitsofe Wheeler
1 sibling, 0 replies; 6+ messages in thread
From: Sitsofe Wheeler @ 2025-07-28 21:27 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: vincent.fu, fio
On Fri, 25 Jul 2025 at 07:20, Luis Chamberlain <mcgrof@kernel.org> wrote:
<snip>
> diff --git a/HOWTO.rst b/HOWTO.rst
> index 55ebc388..c5e6a0ad 100644
> --- a/HOWTO.rst
> +++ b/HOWTO.rst
> @@ -4154,6 +4154,18 @@ Steady state
> Collect bandwidth data and calculate the least squares regression
> slope. Stop the job if the slope falls below the specified limit.
>
> + **lat**
> + Collect completion latency data and calculate the maximum mean
> + deviation. Stop the job if the deviation falls below the specified
> + limit. The latency values are weighted by the number of I/O samples
> + in each measurement interval.
> +
> + **lat_slope**
> + Collect completion latency data and calculate the least squares
> + regression slope. Stop the job if the slope falls below the
> + specified limit. The latency values are weighted by the number
> + of I/O samples in each measurement interval.
> +
You'll need to update the man page (fio.1) too.
--
Sitsofe
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-07-28 21:27 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-25 6:17 [PATCH 0/2] fio: steady state for latency Luis Chamberlain
2025-07-25 6:17 ` [PATCH 1/2] configure: libnfs + gnutls Luis Chamberlain
2025-07-25 17:44 ` Vincent Fu
2025-07-25 6:17 ` [PATCH 2/2] fio: add latency steady state detection Luis Chamberlain
2025-07-28 18:14 ` Vincent Fu
2025-07-28 21:27 ` Sitsofe Wheeler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox