Re: fio hangs with --status-interval

[Jens Axboe <axboe@kernel.dk>]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On 2014-07-10 00:56, Michael Mattsson wrote:
> Hey,
> I've got 8 identical CentOS 6.5 clients that randomly keeps hanging
> fio when using --status-interval. I've tried fio 2.1.4 and fio 2.1.10
> they both behave the same. I've also tried piping the output to tee
> instead of redirecting to a file. I also tried --output and specified
> output file, still same problem. My fio command runs through its tests
> flawlessly without --status-interval and exits cleanly every time.
> There could be anywhere from 0 to 5 clients that gets affected.
> Running strace on the process that seem hung yields the following
> output:
>
> $ strace -p 31055
> Process 31055 attached - interrupt to quit
> futex(0x7f346ede802c, FUTEX_WAIT, 1, NULL

Strange, it must be stuck on the stat mutex, but I don't immediately see 
why that would happen. Does the attached patch make any difference for 
you, both in getting rid of the hang but still producing output at the 
desired intervals?

-- 
Jens Axboe


[-- Attachment #2: stat.patch --]
[-- Type: text/x-patch, Size: 1790 bytes --]

diff --git a/stat.c b/stat.c
index 979c8100d378..93316a239f7b 100644
--- a/stat.c
+++ b/stat.c
@@ -1466,11 +1466,12 @@ static void *__show_running_run_stats(void fio_unused *arg)
  * in the sig handler, but we should be disturbing the system less by just
  * creating a thread to do it.
  */
-void show_running_run_stats(void)
+int show_running_run_stats(void)
 {
 	pthread_t thread;
 
-	fio_mutex_down(stat_mutex);
+	if (fio_mutex_down_trylock(stat_mutex))
+		return 1;
 
 	if (!pthread_create(&thread, NULL, __show_running_run_stats, NULL)) {
 		int err;
@@ -1479,10 +1480,11 @@ void show_running_run_stats(void)
 		if (err)
 			log_err("fio: DU thread detach failed: %s\n", strerror(err));
 
-		return;
+		return 0;
 	}
 
 	fio_mutex_up(stat_mutex);
+	return 1;
 }
 
 static int status_interval_init;
@@ -1531,8 +1533,8 @@ void check_for_running_stats(void)
 			fio_gettime(&status_time, NULL);
 			status_interval_init = 1;
 		} else if (mtime_since_now(&status_time) >= status_interval) {
-			show_running_run_stats();
-			fio_gettime(&status_time, NULL);
+			if (!show_running_run_stats())
+				fio_gettime(&status_time, NULL);
 			return;
 		}
 	}
diff --git a/stat.h b/stat.h
index 2e46175053e8..82b8e973e4be 100644
--- a/stat.h
+++ b/stat.h
@@ -218,7 +218,7 @@ extern void show_group_stats(struct group_run_stats *rs);
 extern int calc_thread_status(struct jobs_eta *je, int force);
 extern void display_thread_status(struct jobs_eta *je);
 extern void show_run_stats(void);
-extern void show_running_run_stats(void);
+extern int show_running_run_stats(void);
 extern void check_for_running_stats(void);
 extern void sum_thread_stats(struct thread_stat *dst, struct thread_stat *src, int nr);
 extern void sum_group_stats(struct group_run_stats *dst, struct group_run_stats *src);