[PATCH 0/3] fio: fix eta

[PATCH 0/3] fio: fix eta

[Keith Busch]

From: Keith Busch <kbusch@kernel.org>

When jobs are overlapping or serialized by the stonewall, the eta
calculation messes up by restarting back to the original calculated
time. Here's a simple example of running multiple 5 second jobs with the
stonewall to serialize them:

  $ fio --eta-newline=1 --name=global --ioengine=io_uring --rw=randread \
      --filename=/dev/zero --time_based --runtime=5 --size=10T --stonewall \
      --name=a --name=b --name=c --name=d --name=e
  ...
  Jobs: 1 (f=1): [r(1),P(4)][15.4%][r=837MiB/s][r=214k IOPS][eta 00m:22s]
  Jobs: 1 (f=1): [r(1),P(4)][19.2%][r=737MiB/s][r=189k IOPS][eta 00m:21s]
  Jobs: 1 (f=1): [_(1),r(1),P(3)][25.0%][r=1569MiB/s][r=402k IOPS][eta 00m:24s]
  Jobs: 1 (f=1): [_(1),r(1),P(3)][31.2%][r=2045MiB/s][r=523k IOPS][eta 00m:22s]
  Jobs: 1 (f=1): [_(1),r(1),P(3)][37.5%][r=1722MiB/s][r=441k IOPS][eta 00m:20s]
  Jobs: 1 (f=1): [_(2),r(1),P(2)][35.1%][r=374MiB/s][r=95.6k IOPS][eta 00m:24s]
  Jobs: 1 (f=1): [_(2),r(1),P(2)][40.5%][r=2193MiB/s][r=561k IOPS][eta 00m:22s]

Literally unplayable! ;)

This series gets the eta reporting the correct value.

Keith Busch (3):
  eta: don't estimate runtime from an unset job epoch
  eta: cap the ETA at the job's own remaining runtime
  eta: remove now unused done_secs

 backend.c |  2 --
 eta.c     | 39 +++++++++++++++++++++++++++++----------
 fio.h     |  1 -
 libfio.c  |  1 -
 4 files changed, 29 insertions(+), 14 deletions(-)

-- 
2.52.0

[PATCH 1/3] eta: don't estimate runtime from an unset job epoch

[Keith Busch]

From: Keith Busch <kbusch@kernel.org>

run_threads() moves a job from TD_INITIALIZED to TD_RUNNING and only
then releases it. The job itself doesn't call set_epoch_time() until
well into thread_main(), after exec_prerun, pre_read_files(),
fio_verify_init() and rate_submit_init() have run. Until that point
td->epoch is still zeroed, so the "elapsed" that thread_eta() derives
from it is the time since the unix epoch rather than the time this job
has been running.

Nothing currently notices, because the one consumer of elapsed that
could care wraps around on the unsigned subtraction, but it is a trap
for anyone using elapsed as a real number of seconds.

Take a local copy of the runstate and demote a TD_RUNNING or
TD_VERIFYING job whose epoch is not set yet to TD_CREATED, so it gets
estimated as a job that is still starting up.

Signed-off-by: Keith Busch <kbusch@kernel.org>
---
 eta.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/eta.c b/eta.c
index c6e3cffb..5dbf4d75 100644
--- a/eta.c
+++ b/eta.c
@@ -162,6 +162,16 @@ static unsigned long thread_eta(struct thread_data *td)
 	unsigned long eta_sec = 0;
 	unsigned long elapsed;
 	uint64_t timeout;
+	int runstate = td->runstate;
+
+	/*
+	 * run_threads() moves a job to TD_RUNNING before the job itself has
+	 * recorded its epoch, so elapsed is meaningless until then. Estimate
+	 * such a job as if it were still starting up.
+	 */
+	if (!td->epoch.tv_sec && !td->epoch.tv_nsec &&
+	    (runstate == TD_RUNNING || runstate == TD_VERIFYING))
+		runstate = TD_CREATED;
 
 	elapsed = (mtime_since_now(&td->epoch) + 999) / 1000;
 	timeout = td->o.timeout / 1000000UL;
@@ -220,7 +230,7 @@ static unsigned long thread_eta(struct thread_data *td)
 		}
 	}
 
-	if (td->runstate == TD_RUNNING || td->runstate == TD_VERIFYING) {
+	if (runstate == TD_RUNNING || runstate == TD_VERIFYING) {
 		double perc, perc_t;
 
 		bytes_done = ddir_rw_sum(td->io_bytes);
@@ -256,11 +266,11 @@ static unsigned long thread_eta(struct thread_data *td)
 		if (td->o.timeout &&
 		    eta_sec > (timeout + done_secs - elapsed))
 			eta_sec = timeout + done_secs - elapsed;
-	} else if (td->runstate == TD_NOT_CREATED || td->runstate == TD_CREATED
-			|| td->runstate == TD_INITIALIZED
-			|| td->runstate == TD_SETTING_UP
-			|| td->runstate == TD_RAMP
-			|| td->runstate == TD_PRE_READING) {
+	} else if (runstate == TD_NOT_CREATED || runstate == TD_CREATED
+			|| runstate == TD_INITIALIZED
+			|| runstate == TD_SETTING_UP
+			|| runstate == TD_RAMP
+			|| runstate == TD_PRE_READING) {
 		int64_t t_eta = 0, r_eta = 0;
 		unsigned long long rate_bytes;
 
@@ -278,7 +288,7 @@ static unsigned long thread_eta(struct thread_data *td)
 				t_eta += ramp_time;
 			t_eta /= 1000000ULL;
 
-			if ((td->runstate == TD_RAMP) && in_ramp_period(td)) {
+			if ((runstate == TD_RAMP) && in_ramp_period(td)) {
 				unsigned long ramp_left;
 
 				ramp_left = mtime_since_now(&td->epoch);
-- 
2.52.0

Re: [PATCH 1/3] eta: don't estimate runtime from an unset job epoch

[Vincent Fu]

On Thu, Sep 3, 2026 at 1:21 PM Keith Busch <kbusch@meta.com> wrote:
>
> From: Keith Busch <kbusch@kernel.org>
>
> run_threads() moves a job from TD_INITIALIZED to TD_RUNNING and only
> then releases it. The job itself doesn't call set_epoch_time() until
> well into thread_main(), after exec_prerun, pre_read_files(),
> fio_verify_init() and rate_submit_init() have run. Until that point
> td->epoch is still zeroed, so the "elapsed" that thread_eta() derives
> from it is the time since the unix epoch rather than the time this job
> has been running.
>
> Nothing currently notices, because the one consumer of elapsed that
> could care wraps around on the unsigned subtraction, but it is a trap
> for anyone using elapsed as a real number of seconds.
>
> Take a local copy of the runstate and demote a TD_RUNNING or
> TD_VERIFYING job whose epoch is not set yet to TD_CREATED, so it gets
> estimated as a job that is still starting up.
>

I think a better fix would be to improve Fio's state machine so that it does
not jump directly into TD_RUNNING from TD_INITIALIZED. Perhaps run_threads
should promote jobs to TD_SETTING_UP instead. And after set_epoch_time()
succeeds, transition to TD_RUNNING.

The current patch is safer, but it doesn't address the underlying problem that
a job is marked as TD_RUNNING when it actually has not attained that state. And
It's this problem which is one of the sources of the ETA issue.

Vincent

Re: [PATCH 1/3] eta: don't estimate runtime from an unset job epoch

[Keith Busch]

On Wed, Sep 09, 2026 at 01:01:32PM -0400, Vincent Fu wrote:
> I think a better fix would be to improve Fio's state machine so that it does
> not jump directly into TD_RUNNING from TD_INITIALIZED. Perhaps run_threads
> should promote jobs to TD_SETTING_UP instead. And after set_epoch_time()
> succeeds, transition to TD_RUNNING.
> 
> The current patch is safer, but it doesn't address the underlying problem that
> a job is marked as TD_RUNNING when it actually has not attained that state. And
> It's this problem which is one of the sources of the ETA issue.

Sounds good. I thinks this suggestion will result in a cleaner result in
the end. Testing a few things now before sending a new version.

[PATCH 2/3] eta: cap the ETA at the job's own remaining runtime

[Keith Busch]

From: Keith Busch <kbusch@kernel.org>

The ETA of a running job is capped at "timeout + done_secs - elapsed".
done_secs is a global accumulator of the runtime of every job reaped so
far, so the cap grows every time a job finishes. Whenever the cap is
what actually gets reported, the ETA jumps back up by the runtime of
everything that has already completed.

The cap is what gets reported when the progress based estimate exceeds
the remaining runtime, that is when perc is below elapsed/timeout. Any
job that is behind on bytes relative to its runtime is in that state,
so this covers the common "size the job to the whole device, bound it
with runtime" pattern. A job that would complete its size early stays
at or above elapsed/timeout and never reaches the cap.

time_based makes no difference either way. It only lowers perc to
min(perc, elapsed/timeout), so a time_based job that cannot finish its
size within the runtime is affected exactly like a size based one.

The problem is most visible with stonewalled jobs, where the ETA climbs
back to the full run time at every batch boundary instead of counting
down. Start stonewalled io_uring jobs, size=10T runtime=5 time_based,
report:

  22 21 20  24 23 22 21 20  24 23 22 21 20  24 23 22 21 20  24 ...

instead of counting down to 0.

It is not specific to stonewall. Two concurrent jobs with runtime=5 and
runtime=20 show the same jump when the short one is reaped at t=5:

  19 18 17 16 15  19 18 17 16 15 14 ...

done_secs made sense when it was introduced: thread_eta() was handed
the global elapsed time back then, so "timeout + done_secs" was this
job's projected finish time relative to the start of the whole run.
b29ee5b3 switched elapsed to be per job, measured from td->epoch, but
left the done_secs term behind.

A job is terminated once utime_since(&td->epoch, now) reaches
td->o.timeout, so with a per job elapsed the cap is simply
"timeout - elapsed". Use that, and clamp at zero instead of relying on
the unsigned subtraction wrapping.

Fixes: b29ee5b3dee4 ("Update ramp_time")
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
 eta.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/eta.c b/eta.c
index 5dbf4d75..29e1bf41 100644
--- a/eta.c
+++ b/eta.c
@@ -263,9 +263,18 @@ static unsigned long thread_eta(struct thread_data *td)
 			eta_sec = (unsigned long) (elapsed * (1.0 / perc)) - elapsed;
 		}
 
-		if (td->o.timeout &&
-		    eta_sec > (timeout + done_secs - elapsed))
-			eta_sec = timeout + done_secs - elapsed;
+		/*
+		 * A job never runs for longer than its own timeout, which is
+		 * measured from its own epoch. Cap the estimate at whatever
+		 * time this job has left.
+		 */
+		if (td->o.timeout) {
+			unsigned long timeout_left;
+
+			timeout_left = timeout > elapsed ? timeout - elapsed : 0;
+			if (eta_sec > timeout_left)
+				eta_sec = timeout_left;
+		}
 	} else if (runstate == TD_NOT_CREATED || runstate == TD_CREATED
 			|| runstate == TD_INITIALIZED
 			|| runstate == TD_SETTING_UP
-- 
2.52.0

[PATCH 3/3] eta: remove now unused done_secs

[Keith Busch]

From: Keith Busch <kbusch@kernel.org>

Capping the ETA at the job's own remaining runtime removed the last
reader of done_secs. Drop the variable, along with the gettime call
that maintained it on every reap.

Signed-off-by: Keith Busch <kbusch@kernel.org>
---
 backend.c | 2 --
 fio.h     | 1 -
 libfio.c  | 1 -
 3 files changed, 4 deletions(-)

diff --git a/backend.c b/backend.c
index 7f41bdfa..50d4394a 100644
--- a/backend.c
+++ b/backend.c
@@ -71,7 +71,6 @@ unsigned int nr_segments = 0;
 unsigned int cur_segment = 0;
 unsigned int stat_number = 0;
 int temp_stall_ts;
-unsigned long done_secs = 0;
 #ifdef PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP
 pthread_mutex_t overlap_check = PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP;
 #else
@@ -2494,7 +2493,6 @@ reaped:
 		if (td->error)
 			exit_value++;
 
-		done_secs += mtime_since_now(&td->epoch) / 1000;
 		profile_td_exit(td);
 		flow_exit_job(td);
 	} end_for_each();
diff --git a/fio.h b/fio.h
index 494959a6..11a0d59d 100644
--- a/fio.h
+++ b/fio.h
@@ -597,7 +597,6 @@ extern bool read_only;
 extern int eta_print;
 extern int eta_new_line;
 extern unsigned int eta_interval_msec;
-extern unsigned long done_secs;
 extern int fio_gtod_offload;
 extern int fio_gtod_cpu;
 extern enum fio_cs fio_clock_source;
diff --git a/libfio.c b/libfio.c
index a57ede4f..c3c17edd 100644
--- a/libfio.c
+++ b/libfio.c
@@ -187,7 +187,6 @@ void reset_fio_state(void)
 	for (i = 0; i < nr_segments; i++)
 		segments[i].nr_threads = 0;
 	stat_number = 0;
-	done_secs = 0;
 }
 
 const char *fio_get_os_string(int nr)
-- 
2.52.0

Re: [PATCH 0/3] fio: fix eta

[fiotestbot]

[-- Attachment #1: Type: text/plain, Size: 144 bytes --]


The result of fio's continuous integration tests was: success

For more details see https://github.com/fiotestbot/fio/actions/runs/33784801874