From: Jens Axboe <axboe@kernel.dk>
To: Vasily Tarasov <tarasov@vasily.name>
Cc: Michael Mattsson <michael.mattsson@gmail.com>,
"fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: fio hangs with --status-interval
Date: Fri, 25 Jul 2014 09:43:36 +0200 [thread overview]
Message-ID: <53D20AA8.6020700@kernel.dk> (raw)
In-Reply-To: <CAFTzLMP5GRx6R2HCykKN4pjkPLwHGafMc2HNh4gUONRRjBgYFg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3920 bytes --]
On 2014-07-21 22:25, Vasily Tarasov wrote:
> Hi Jens,
>
> I tried your patch, but it didn't help. Interestingly, the number of
> threads changes in the end. At first, during the run:
>
> # ps -eLf | grep fio
> root 5224 4274 5224 1 2 11:12 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5231 5224 5231 60 1 11:12 ? 00:00:07 fio
> --status-interval 10 --minimal fios/1.fio
> root 5260 5237 5260 0 1 11:12 pts/0 00:00:00 grep fio
> [root@bison01 vass]# ps -eLf | grep fio
> root 5224 4274 5224 0 2 11:12 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5231 5224 5231 16 1 11:12 ? 00:00:21 fio
> --status-interval 10 --minimal fios/1.fio
> root 5293 5237 5293 0 1 11:14 pts/0 00:00:00 grep fio
> [root@bison01 vass]# ps -eLf | grep fio
> root 5224 4274 5224 0 2 11:12 pts/1 00:00:01 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5231 5224 5231 12 1 11:12 ? 00:01:13 fio
> --status-interval 10 --minimal fios/1.fio
> root 5411 5237 5411 0 1 11:22 pts/0 00:00:00 grep fio
>
> Later, when the threads are stuck:
>
> # ps -eLf | grep fio
> root 5224 4274 5224 0 16 11:12 pts/1 00:00:02 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5225 0 16 11:12 pts/1 00:00:01 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5458 0 16 11:25 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5459 0 16 11:25 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5460 0 16 11:25 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5461 0 16 11:25 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5462 0 16 11:25 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5471 0 16 11:25 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5472 0 16 11:26 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5475 0 16 11:26 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5476 0 16 11:26 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5477 0 16 11:26 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5478 0 16 11:26 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5487 0 16 11:26 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5488 0 16 11:27 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 5224 4274 5489 0 16 11:27 pts/1 00:00:00 fio
> --status-interval 10 --minimal fios/1.fio
> root 6665 5237 6665 0 1 13:21 pts/0 00:00:00 grep fio
>
> Is the number of threads supposed to change?..
Never answered this one... Yes, it'll change, since when you run the
job, you'll have one backend process, a number of IO workers, and one
disk util thread typically. When you get stuck, it's the backend that is
left waiting for that mutex.
In any case, I haven't been able to figure this one out yet. But it
should be safe enough to just ignore the stat mutex for the final
output, since the threads otherwise accessing it are gone. Can you see
if this one makes the issue go away?
--
Jens Axboe
[-- Attachment #2: stat-hang.patch --]
[-- Type: text/x-patch, Size: 812 bytes --]
diff --git a/backend.c b/backend.c
index 30f78b72f9d1..981625b61095 100644
--- a/backend.c
+++ b/backend.c
@@ -2068,7 +2068,7 @@ int fio_backend(void)
run_threads();
if (!fio_abort) {
- show_run_stats();
+ __show_run_stats();
if (write_bw_log) {
int i;
diff --git a/stat.h b/stat.h
index 2e46175053e8..90a7fb31a1bc 100644
--- a/stat.h
+++ b/stat.h
@@ -218,6 +218,7 @@ extern void show_group_stats(struct group_run_stats *rs);
extern int calc_thread_status(struct jobs_eta *je, int force);
extern void display_thread_status(struct jobs_eta *je);
extern void show_run_stats(void);
+extern void __show_run_stats(void);
extern void show_running_run_stats(void);
extern void check_for_running_stats(void);
extern void sum_thread_stats(struct thread_stat *dst, struct thread_stat *src, int nr);
next prev parent reply other threads:[~2014-07-25 7:43 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-09 22:56 fio hangs with --status-interval Michael Mattsson
2014-07-10 8:44 ` Jens Axboe
2014-07-10 14:55 ` Michael Mattsson
2014-07-10 20:07 ` Jens Axboe
2014-07-10 21:27 ` Michael Mattsson
2014-07-11 11:48 ` Jens Axboe
2014-07-11 16:19 ` Michael Mattsson
2014-07-12 8:42 ` Jens Axboe
2014-07-16 16:58 ` Vasily Tarasov
2014-07-21 8:08 ` Jens Axboe
2014-07-21 20:25 ` Vasily Tarasov
2014-07-21 20:54 ` Jens Axboe
2014-07-25 7:43 ` Jens Axboe [this message]
2014-07-25 7:56 ` Jens Axboe
2014-07-25 16:34 ` Vasily Tarasov
2014-07-28 8:56 ` Jens Axboe
2014-07-28 16:05 ` Vasily Tarasov
2014-07-28 19:49 ` Jens Axboe
2014-10-23 15:57 ` Michael Mattsson
2014-10-23 16:01 ` Jens Axboe
2014-10-23 21:58 ` Michael Mattsson
2014-10-24 5:17 ` Jens Axboe
2014-10-24 15:33 ` Michael Mattsson
2014-10-24 15:50 ` Jens Axboe
2014-11-04 15:46 ` Vasily Tarasov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D20AA8.6020700@kernel.dk \
--to=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
--cc=michael.mattsson@gmail.com \
--cc=tarasov@vasily.name \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.