From: Jens Axboe <axboe@kernel.dk>
To: Akash Verma <akashv@google.com>
Cc: Michael Bella <mbella@google.com>, Caio Villela <caio@google.com>,
Allen Schade <aschade@google.com>, fio <fio@vger.kernel.org>
Subject: Re: Running a separate fio process for each disk?
Date: Thu, 3 Dec 2015 11:58:00 -0700 [thread overview]
Message-ID: <566090B8.60302@kernel.dk> (raw)
In-Reply-To: <CAFFT=Uks8j+K7M0dfT0tS2aiRTOHCqu0S_3-DnKY2zL-K33T7g@mail.gmail.com>
Perfect! Thanks for reporting and re-testing.
On 12/03/2015 11:54 AM, Akash Verma wrote:
> Jens, I confirmed that the issue is not seen with the latest FIO (I used
> version fio-2.2.12-15-gcdab).
>
> On Tue, Nov 24, 2015 at 5:18 PM, Jens Axboe <axboe@kernel.dk
> <mailto:axboe@kernel.dk>> wrote:
>
> No worries, I know this week is a bit more problematic than usual.
> I'll hold off on the new release until I know.
>
>
>
> On 11/24/2015 01:51 PM, Akash Verma wrote:
>
> Sorry for not getting back - I didn't get a chance to try the latest
> git, and I'm off on vacation soon; I'm ccing Michael and Caio who
> might have a chance to try it out before Thursday. Michael or Caio,
> could you try run the two things Jens asked (the cpuclock test using
> the FIO we've been currently using as well as the latest from
> Git; and
> the regular multi-process FIO run with the latest git)?
>
> On Tue, Nov 24, 2015 at 7:51 AM, Jens Axboe <axboe@kernel.dk
> <mailto:axboe@kernel.dk>> wrote:
>
> Did you try current -git yet? I think it should work for
> both scenarios.
> It's a silly bug, would be great to have confirmation that
> it's fixed. Then
> I'll spin a new release.
>
>
>
> On 11/20/2015 05:21 PM, Jens Axboe wrote:
>
>
> And finally, there's a potential fix, if you run commit
> 99afcdb53dc3 or later. So please do try that as well, and
> see if that behaves any better for you.
>
>
> On 11/20/2015 05:03 PM, Jens Axboe wrote:
>
>
> Hi,
>
> OK, I see. Can you pull the latest -git, and then
> run fio
> --cpuclock-test on one of the boxes where you see
> the issue? It should
> have commit 5896d827e1e2 or later.
>
>
> On Fri, Nov 20, 2015 at 3:20 PM, Akash Verma
> <akashv@google.com <mailto:akashv@google.com>
> <mailto:akashv@google.com
> <mailto:akashv@google.com>>> wrote:
>
> Hi Jens,
> The issue is not seen with non-cpu clock
> sources, or when using a
> single process (with individual threads, the
> only config I tried). We
> only see the issue when using multiple
> processes and the cpu clock
> source.
>
> On Fri, Nov 20, 2015 at 11:50 AM, Jens Axboe
> <axboe@kernel.dk <mailto:axboe@kernel.dk>
> <mailto:axboe@kernel.dk
> <mailto:axboe@kernel.dk>>> wrote:
> > On 11/20/2015 12:37 PM, Caio Villela wrote:
> >>
> >> Hello Allen and Jens,
> >>
> >> Sorry for the long output, this is just in
> case you want the
> details.
> >> Here is a simple explanation for the
> problem. I want to run a 15
> minute
> >> random write, using 1 Meg requests, and
> measure throughput and
> latency.
> >> What seems to be the problem is that if
> the test system has a
> large
> >> number of drives - the system that I am
> testing here has 28
> drives -
> >> then the time accounting seems to go bad
> for some of the
> processes.
> >> What you see below is that during the 15
> minutes from start, all
> disks
> >> are getting hit the same, as they should.
> Then, after 15
> minutes, there
> >> are 15 drives that are still running....
> after 5 minutes over the
> >> specified 15 minutes, there is still one
> drive running. Then
> looking at
> >> the amount of IOs sent to each drive, the
> ones that ran on that
> excess
> >> time have much more IOs. FIO still reports
> that all drives ran
> for 15
> >> minutes, although some ran for more than
> 20 minutes.
> >>
> >> We will attempt to run a single process
> instead of 28 instances
> of FIO
> >> to see if this goes away.
> >
> >
> > Could you also check if adding
> clocksource=gettimeofday makes any
> > difference? This sounds very odd.
> >
> > Assuming this was run with fio -git?
> >
> >
> > --
> > Jens Axboe
> >
> > --
> > To unsubscribe from this list: send the line
> "unsubscribe fio" in
> > the body of a message
> tomajordomo@vger.kernel.org
> <mailto:tomajordomo@vger.kernel.org>
> <mailto:majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>>
> > More majordomo info
> athttp://vger.kernel.org/majordomo-info.html
> <http://vger.kernel.org/majordomo-info.html>
>
>
>
>
>
>
> --
> Jens Axboe
>
>
>
> --
> Jens Axboe
>
>
--
Jens Axboe
prev parent reply other threads:[~2015-12-03 18:58 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CADp+U7ibiKciX8_cpzGzob4oL-UF-H+W7kYuiujovD0ba=hM6A@mail.gmail.com>
[not found] ` <56464ACC.9030605@kernel.dk>
2015-11-13 22:04 ` Running a separate fio process for each disk? Allen Schade
2015-11-13 22:06 ` Jens Axboe
2015-11-20 18:28 ` Allen Schade
2015-11-20 19:37 ` Caio Villela
2015-11-20 19:50 ` Jens Axboe
2015-11-20 22:20 ` Akash Verma
2015-11-21 0:03 ` Jens Axboe
2015-11-21 0:21 ` Jens Axboe
2015-11-24 15:51 ` Jens Axboe
2015-11-24 20:51 ` Akash Verma
2015-11-25 1:18 ` Jens Axboe
2015-12-03 18:54 ` Akash Verma
2015-12-03 18:58 ` Jens Axboe [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=566090B8.60302@kernel.dk \
--to=axboe@kernel.dk \
--cc=akashv@google.com \
--cc=aschade@google.com \
--cc=caio@google.com \
--cc=fio@vger.kernel.org \
--cc=mbella@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox