From: Jens Axboe <axboe@kernel.dk>
To: Vasily Tarasov <tarasov@vasily.name>
Cc: Michael Mattsson <michael.mattsson@gmail.com>,
"fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: fio hangs with --status-interval
Date: Mon, 28 Jul 2014 10:56:51 +0200 [thread overview]
Message-ID: <53D61053.9030902@kernel.dk> (raw)
In-Reply-To: <CAFTzLMPE+h_v8ydUP+iuCwJErQLuU=ZNcuKKcuxuZztandp+mQ@mail.gmail.com>
On 2014-07-25 18:34, Vasily Tarasov wrote:
> Hi Jens,
>
> You'll be surprised but it did not help :( I used the latest code from
> git (fio-2.1.11-10-gae7e, commit ae7e050). Still see the same picture.
That's actually good news, since it didn't make a lot of sense. So lets
see if we can't get to the bottom of this...
> I don't know if it helps, but I see this behavior on a machine with
> 96GB of RAM. So, after buffered writes are over, fio waits for a long
> time till all dirty buffers hit the disk. But, even after there is no
> more disk activity, fio is still stuck for as long as I don't kill it.
>
> Regarding the number of threads. I do understand where the 3 threads
> can come from:
>
> 1) Backend thread (sort of a manager)
> 1) Worker thread(s)
> 2) Disk stats thread
>
> I my case I defined only one job instance, so I suppose there always
> should be only one worker thread. I don't understand how the total
> number of threads go to 10 in the end.
>
> <snip starts>
> $ ps -eLf | grep fio
> root 4427 4135 4427 0 15 07:44 pts/1 00:00:02 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4636 0 15 07:56 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4637 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4638 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4647 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4650 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4651 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4652 0 15 07:57 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4653 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4654 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4663 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4664 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4666 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4668 0 15 07:58 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> root 4427 4135 4669 0 15 07:59 pts/1 00:00:00 fio
> --minimal --status-interval 10 1.fio
> <snip ends>
Can you try and gdb attach to it when it's hung and produce a new
backtrace? It can't be off the final status run, I wonder if it's off
the mutex down and remove instead.
--
Jens Axboe
next prev parent reply other threads:[~2014-07-28 8:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-09 22:56 fio hangs with --status-interval Michael Mattsson
2014-07-10 8:44 ` Jens Axboe
2014-07-10 14:55 ` Michael Mattsson
2014-07-10 20:07 ` Jens Axboe
2014-07-10 21:27 ` Michael Mattsson
2014-07-11 11:48 ` Jens Axboe
2014-07-11 16:19 ` Michael Mattsson
2014-07-12 8:42 ` Jens Axboe
2014-07-16 16:58 ` Vasily Tarasov
2014-07-21 8:08 ` Jens Axboe
2014-07-21 20:25 ` Vasily Tarasov
2014-07-21 20:54 ` Jens Axboe
2014-07-25 7:43 ` Jens Axboe
2014-07-25 7:56 ` Jens Axboe
2014-07-25 16:34 ` Vasily Tarasov
2014-07-28 8:56 ` Jens Axboe [this message]
2014-07-28 16:05 ` Vasily Tarasov
2014-07-28 19:49 ` Jens Axboe
2014-10-23 15:57 ` Michael Mattsson
2014-10-23 16:01 ` Jens Axboe
2014-10-23 21:58 ` Michael Mattsson
2014-10-24 5:17 ` Jens Axboe
2014-10-24 15:33 ` Michael Mattsson
2014-10-24 15:50 ` Jens Axboe
2014-11-04 15:46 ` Vasily Tarasov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D61053.9030902@kernel.dk \
--to=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
--cc=michael.mattsson@gmail.com \
--cc=tarasov@vasily.name \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.