From: Jens Axboe <axboe@kernel.dk>
To: Vasily Tarasov <tarasov@vasily.name>
Cc: Michael Mattsson <michael.mattsson@gmail.com>,
"fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: fio hangs with --status-interval
Date: Fri, 25 Jul 2014 09:56:40 +0200 [thread overview]
Message-ID: <53D20DB8.10100@kernel.dk> (raw)
In-Reply-To: <53D20AA8.6020700@kernel.dk>
On 2014-07-25 09:43, Jens Axboe wrote:
> On 2014-07-21 22:25, Vasily Tarasov wrote:
>> Hi Jens,
>>
>> I tried your patch, but it didn't help. Interestingly, the number of
>> threads changes in the end. At first, during the run:
>>
>> # ps -eLf | grep fio
>> root 5224 4274 5224 1 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5231 5224 5231 60 1 11:12 ? 00:00:07 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5260 5237 5260 0 1 11:12 pts/0 00:00:00 grep fio
>> [root@bison01 vass]# ps -eLf | grep fio
>> root 5224 4274 5224 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5231 5224 5231 16 1 11:12 ? 00:00:21 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5293 5237 5293 0 1 11:14 pts/0 00:00:00 grep fio
>> [root@bison01 vass]# ps -eLf | grep fio
>> root 5224 4274 5224 0 2 11:12 pts/1 00:00:01 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 2 11:12 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5231 5224 5231 12 1 11:12 ? 00:01:13 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5411 5237 5411 0 1 11:22 pts/0 00:00:00 grep fio
>>
>> Later, when the threads are stuck:
>>
>> # ps -eLf | grep fio
>> root 5224 4274 5224 0 16 11:12 pts/1 00:00:02 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5225 0 16 11:12 pts/1 00:00:01 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5458 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5459 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5460 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5461 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5462 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5471 0 16 11:25 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5472 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5475 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5476 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5477 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5478 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5487 0 16 11:26 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5488 0 16 11:27 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 5224 4274 5489 0 16 11:27 pts/1 00:00:00 fio
>> --status-interval 10 --minimal fios/1.fio
>> root 6665 5237 6665 0 1 13:21 pts/0 00:00:00 grep fio
>>
>> Is the number of threads supposed to change?..
>
> Never answered this one... Yes, it'll change, since when you run the
> job, you'll have one backend process, a number of IO workers, and one
> disk util thread typically. When you get stuck, it's the backend that is
> left waiting for that mutex.
>
> In any case, I haven't been able to figure this one out yet. But it
> should be safe enough to just ignore the stat mutex for the final
> output, since the threads otherwise accessing it are gone. Can you see
> if this one makes the issue go away?
Patch was not compiled, was missing the non-static __show_run_stats().
But just pull current -git, I have committed a variant that does compile :-)
--
Jens Axboe
next prev parent reply other threads:[~2014-07-25 7:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-09 22:56 fio hangs with --status-interval Michael Mattsson
2014-07-10 8:44 ` Jens Axboe
2014-07-10 14:55 ` Michael Mattsson
2014-07-10 20:07 ` Jens Axboe
2014-07-10 21:27 ` Michael Mattsson
2014-07-11 11:48 ` Jens Axboe
2014-07-11 16:19 ` Michael Mattsson
2014-07-12 8:42 ` Jens Axboe
2014-07-16 16:58 ` Vasily Tarasov
2014-07-21 8:08 ` Jens Axboe
2014-07-21 20:25 ` Vasily Tarasov
2014-07-21 20:54 ` Jens Axboe
2014-07-25 7:43 ` Jens Axboe
2014-07-25 7:56 ` Jens Axboe [this message]
2014-07-25 16:34 ` Vasily Tarasov
2014-07-28 8:56 ` Jens Axboe
2014-07-28 16:05 ` Vasily Tarasov
2014-07-28 19:49 ` Jens Axboe
2014-10-23 15:57 ` Michael Mattsson
2014-10-23 16:01 ` Jens Axboe
2014-10-23 21:58 ` Michael Mattsson
2014-10-24 5:17 ` Jens Axboe
2014-10-24 15:33 ` Michael Mattsson
2014-10-24 15:50 ` Jens Axboe
2014-11-04 15:46 ` Vasily Tarasov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D20DB8.10100@kernel.dk \
--to=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
--cc=michael.mattsson@gmail.com \
--cc=tarasov@vasily.name \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.