From: Jens Axboe <axboe@kernel.dk>
To: Vasily Tarasov <tarasov@vasily.name>
Cc: Michael Mattsson <michael.mattsson@gmail.com>,
"fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: fio hangs with --status-interval
Date: Mon, 28 Jul 2014 10:56:51 +0200 [thread overview]
Message-ID: <53D61053.9030902@kernel.dk> (raw)
In-Reply-To: <CAFTzLMPE+h_v8ydUP+iuCwJErQLuU=ZNcuKKcuxuZztandp+mQ@mail.gmail.com>
On 2014-07-25 18:34, Vasily Tarasov wrote:
> Hi Jens,
>
> You'll be surprised but it did not help :( I used the latest code from
> git (fio-2.1.11-10-gae7e, commit ae7e050). Still see the same picture.
That's actually good news, since it didn't make a lot of sense. So lets
see if we can't get to the bottom of this...
> I don't know if it helps, but I see this behavior on a machine with
> 96GB of RAM. So, after buffered writes are over, fio waits for a long
> time till all dirty buffers hit the disk. But, even after there is no
> more disk activity, fio is still stuck for as long as I don't kill it.
>
> Regarding the number of threads. I do understand where the 3 threads
> can come from:
>
> 1) Backend thread (sort of a manager)
> 1) Worker thread(s)
> 2) Disk stats thread
>
> I my case I defined only one job instance, so I suppose there always
> should be only one worker thread. I don't understand how the total
> number of threads go to 10 in the end.
>
> <snip starts>
> $ ps -eLf | grep fio
> root 4427 4135 4427 0 15 07:44 pts/1 00:00:02 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4636 0 15 07:56 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4637 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4638 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4647 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4650 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4651 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4652 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4653 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4654 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4663 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4664 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4666 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4668 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4669 0 15 07:59 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> <snip ends>
Can you try and gdb attach to it when it's hung and produce a new
backtrace? It can't be off the final status run, I wonder if it's off
the mutex down and remove instead.
--
Jens Axboe
next prev parent reply other threads:[~2014-07-28 8:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-09 22:56 fio hangs with --status-interval Michael Mattsson
2014-07-10 8:44 ` Jens Axboe
2014-07-10 14:55 ` Michael Mattsson
2014-07-10 20:07 ` Jens Axboe
2014-07-10 21:27 ` Michael Mattsson
2014-07-11 11:48 ` Jens Axboe
2014-07-11 16:19 ` Michael Mattsson
2014-07-12 8:42 ` Jens Axboe
2014-07-16 16:58 ` Vasily Tarasov
2014-07-21 8:08 ` Jens Axboe
2014-07-21 20:25 ` Vasily Tarasov
2014-07-21 20:54 ` Jens Axboe
2014-07-25 7:43 ` Jens Axboe
2014-07-25 7:56 ` Jens Axboe
2014-07-25 16:34 ` Vasily Tarasov
2014-07-28 8:56 ` Jens Axboe [this message]
2014-07-28 16:05 ` Vasily Tarasov
2014-07-28 19:49 ` Jens Axboe
2014-10-23 15:57 ` Michael Mattsson
2014-10-23 16:01 ` Jens Axboe
2014-10-23 21:58 ` Michael Mattsson
2014-10-24 5:17 ` Jens Axboe
2014-10-24 15:33 ` Michael Mattsson
2014-10-24 15:50 ` Jens Axboe
2014-11-04 15:46 ` Vasily Tarasov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D61053.9030902@kernel.dk \
--to=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
--cc=michael.mattsson@gmail.com \
--cc=tarasov@vasily.name \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox