From: Jens Axboe <axboe@kernel.dk>
To: Vasily Tarasov <tarasov@vasily.name>
Cc: Michael Mattsson <michael.mattsson@gmail.com>,
"fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: fio hangs with --status-interval
Date: Fri, 25 Jul 2014 09:56:40 +0200 [thread overview]
Message-ID: <53D20DB8.10100@kernel.dk> (raw)
In-Reply-To: <53D20AA8.6020700@kernel.dk>
On 2014-07-25 09:43, Jens Axboe wrote:
> On 2014-07-21 22:25, Vasily Tarasov wrote:
>> Hi Jens,
>>
>> I tried your patch, but it didn't help. Interestingly, the number of
>> threads changes in the end. At first, during the run:
>>
>> # ps -eLf | grep fio
>> root 5224 4274 5224 1 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5231 5224 5231 60 1 11:12 ? 00:00:07 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5260 5237 5260 0 1 11:12 pts/0 00:00:00 grep fio
>> [root@bison01 vass]# ps -eLf | grep fio
>> root 5224 4274 5224 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5231 5224 5231 16 1 11:12 ? 00:00:21 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5293 5237 5293 0 1 11:14 pts/0 00:00:00 grep fio
>> [root@bison01 vass]# ps -eLf | grep fio
>> root 5224 4274 5224 0 2 11:12 pts/1 00:00:01 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5231 5224 5231 12 1 11:12 ? 00:01:13 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5411 5237 5411 0 1 11:22 pts/0 00:00:00 grep fio
>>
>> Later, when the threads are stuck:
>>
>> # ps -eLf | grep fio
>> root 5224 4274 5224 0 16 11:12 pts/1 00:00:02 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 16 11:12 pts/1 00:00:01 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5458 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5459 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5460 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5461 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5462 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5471 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5472 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5475 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5476 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5477 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5478 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5487 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5488 0 16 11:27 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5489 0 16 11:27 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 6665 5237 6665 0 1 13:21 pts/0 00:00:00 grep fio
>>
>> Is the number of threads supposed to change?..
>
> Never answered this one... Yes, it'll change, since when you run the
> job, you'll have one backend process, a number of IO workers, and one
> disk util thread typically. When you get stuck, it's the backend that is
> left waiting for that mutex.
>
> In any case, I haven't been able to figure this one out yet. But it
> should be safe enough to just ignore the stat mutex for the final
> output, since the threads otherwise accessing it are gone. Can you see
> if this one makes the issue go away?
Patch was not compiled, was missing the non-static __show_run_stats().
But just pull current -git, I have committed a variant that does compile :-)
--
Jens Axboe
next prev parent reply other threads:[~2014-07-25 7:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-09 22:56 fio hangs with --status-interval Michael Mattsson
2014-07-10 8:44 ` Jens Axboe
2014-07-10 14:55 ` Michael Mattsson
2014-07-10 20:07 ` Jens Axboe
2014-07-10 21:27 ` Michael Mattsson
2014-07-11 11:48 ` Jens Axboe
2014-07-11 16:19 ` Michael Mattsson
2014-07-12 8:42 ` Jens Axboe
2014-07-16 16:58 ` Vasily Tarasov
2014-07-21 8:08 ` Jens Axboe
2014-07-21 20:25 ` Vasily Tarasov
2014-07-21 20:54 ` Jens Axboe
2014-07-25 7:43 ` Jens Axboe
2014-07-25 7:56 ` Jens Axboe [this message]
2014-07-25 16:34 ` Vasily Tarasov
2014-07-28 8:56 ` Jens Axboe
2014-07-28 16:05 ` Vasily Tarasov
2014-07-28 19:49 ` Jens Axboe
2014-10-23 15:57 ` Michael Mattsson
2014-10-23 16:01 ` Jens Axboe
2014-10-23 21:58 ` Michael Mattsson
2014-10-24 5:17 ` Jens Axboe
2014-10-24 15:33 ` Michael Mattsson
2014-10-24 15:50 ` Jens Axboe
2014-11-04 15:46 ` Vasily Tarasov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D20DB8.10100@kernel.dk \
--to=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
--cc=michael.mattsson@gmail.com \
--cc=tarasov@vasily.name \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox