From: Jens Axboe <axboe@kernel.dk>
To: Akash Verma <akashv@google.com>
Cc: Michael Bella <mbella@google.com>, Caio Villela <caio@google.com>,
Allen Schade <aschade@google.com>, fio <fio@vger.kernel.org>
Subject: Re: Running a separate fio process for each disk?
Date: Thu, 3 Dec 2015 11:58:00 -0700 [thread overview]
Message-ID: <566090B8.60302@kernel.dk> (raw)
In-Reply-To: <CAFFT=Uks8j+K7M0dfT0tS2aiRTOHCqu0S_3-DnKY2zL-K33T7g@mail.gmail.com>
Perfect! Thanks for reporting and re-testing.
On 12/03/2015 11:54 AM, Akash Verma wrote:
> Jens, I confirmed that the issue is not seen with the latest FIO (I used
> version fio-2.2.12-15-gcdab).
>
> On Tue, Nov 24, 2015 at 5:18 PM, Jens Axboe <axboe@kernel.dk
> <mailto:axboe@kernel.dk>> wrote:
>
> No worries, I know this week is a bit more problematic than usual.
> I'll hold off on the new release until I know.
>
>
>
> On 11/24/2015 01:51 PM, Akash Verma wrote:
>
> Sorry for not getting back - I didn't get a chance to try the latest
> git, and I'm off on vacation soon; I'm ccing Michael and Caio who
> might have a chance to try it out before Thursday. Michael or Caio,
> could you try run the two things Jens asked (the cpuclock test using
> the FIO we've been currently using as well as the latest from
> Git; and
> the regular multi-process FIO run with the latest git)?
>
> On Tue, Nov 24, 2015 at 7:51 AM, Jens Axboe <axboe@kernel.dk
> <mailto:axboe@kernel.dk>> wrote:
>
> Did you try current -git yet? I think it should work for
> both scenarios.
> It's a silly bug, would be great to have confirmation that
> it's fixed. Then
> I'll spin a new release.
>
>
>
> On 11/20/2015 05:21 PM, Jens Axboe wrote:
>
>
> And finally, there's a potential fix, if you run commit
> 99afcdb53dc3 or later. So please do try that as well, and
> see if that behaves any better for you.
>
>
> On 11/20/2015 05:03 PM, Jens Axboe wrote:
>
>
> Hi,
>
> OK, I see. Can you pull the latest -git, and then
> run fio
> --cpuclock-test on one of the boxes where you see
> the issue? It should
> have commit 5896d827e1e2 or later.
>
>
> On Fri, Nov 20, 2015 at 3:20 PM, Akash Verma
> <akashv@google.com <mailto:akashv@google.com>
> <mailto:akashv@google.com
> <mailto:akashv@google.com>>> wrote:
>
> Hi Jens,
> The issue is not seen with non-cpu clock
> sources, or when using a
> single process (with individual threads, the
> only config I tried). We
> only see the issue when using multiple
> processes and the cpu clock
> source.
>
> On Fri, Nov 20, 2015 at 11:50 AM, Jens Axboe
> <axboe@kernel.dk <mailto:axboe@kernel.dk>
> <mailto:axboe@kernel.dk
> <mailto:axboe@kernel.dk>>> wrote:
> > On 11/20/2015 12:37 PM, Caio Villela wrote:
> >>
> >> Hello Allen and Jens,
> >>
> >> Sorry for the long output, this is just in
> case you want the
> details.
> >> Here is a simple explanation for the
> problem. I want to run a 15
> minute
> >> random write, using 1 Meg requests, and
> measure throughput and
> latency.
> >> What seems to be the problem is that if
> the test system has a
> large
> >> number of drives - the system that I am
> testing here has 28
> drives -
> >> then the time accounting seems to go bad
> for some of the
> processes.
> >> What you see below is that during the 15
> minutes from start, all
> disks
> >> are getting hit the same, as they should.
> Then, after 15
> minutes, there
> >> are 15 drives that are still running....
> after 5 minutes over the
> >> specified 15 minutes, there is still one
> drive running. Then
> looking at
> >> the amount of IOs sent to each drive, the
> ones that ran on that
> excess
> >> time have much more IOs. FIO still reports
> that all drives ran
> for 15
> >> minutes, although some ran for more than
> 20 minutes.
> >>
> >> We will attempt to run a single process
> instead of 28 instances
> of FIO
> >> to see if this goes away.
> >
> >
> > Could you also check if adding
> clocksource=gettimeofday makes any
> > difference? This sounds very odd.
> >
> > Assuming this was run with fio -git?
> >
> >
> > --
> > Jens Axboe
> >
> > --
> > To unsubscribe from this list: send the line
> "unsubscribe fio" in
> > the body of a message
> tomajordomo@vger.kernel.org
> <mailto:tomajordomo@vger.kernel.org>
> <mailto:majordomo@vger.kernel.org
> <mailto:majordomo@vger.kernel.org>>
> > More majordomo info
> athttp://vger.kernel.org/majordomo-info.html
> <http://vger.kernel.org/majordomo-info.html>
>
>
>
>
>
>
> --
> Jens Axboe
>
>
>
> --
> Jens Axboe
>
>
--
Jens Axboe
prev parent reply other threads:[~2015-12-03 18:58 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CADp+U7ibiKciX8_cpzGzob4oL-UF-H+W7kYuiujovD0ba=hM6A@mail.gmail.com>
[not found] ` <56464ACC.9030605@kernel.dk>
2015-11-13 22:04 ` Running a separate fio process for each disk? Allen Schade
2015-11-13 22:06 ` Jens Axboe
2015-11-20 18:28 ` Allen Schade
2015-11-20 19:37 ` Caio Villela
2015-11-20 19:50 ` Jens Axboe
2015-11-20 22:20 ` Akash Verma
2015-11-21 0:03 ` Jens Axboe
2015-11-21 0:21 ` Jens Axboe
2015-11-24 15:51 ` Jens Axboe
2015-11-24 20:51 ` Akash Verma
2015-11-25 1:18 ` Jens Axboe
2015-12-03 18:54 ` Akash Verma
2015-12-03 18:58 ` Jens Axboe [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=566090B8.60302@kernel.dk \
--to=axboe@kernel.dk \
--cc=akashv@google.com \
--cc=aschade@google.com \
--cc=caio@google.com \
--cc=fio@vger.kernel.org \
--cc=mbella@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.