All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: Caio Villela <caio@google.com>, Allen Schade <aschade@google.com>
Cc: fio@vger.kernel.org
Subject: Re: Running a separate fio process for each disk?
Date: Fri, 20 Nov 2015 12:50:34 -0700	[thread overview]
Message-ID: <564F798A.8050009@kernel.dk> (raw)
In-Reply-To: <CAFXh1QgZNghYY9CKJY5BZA4=ARGzXWYVHBBpHxV724qu2Wd5Hw@mail.gmail.com>

On 11/20/2015 12:37 PM, Caio Villela wrote:
> Hello Allen and Jens,
>
> Sorry for the long output, this is just in case you want the details.
> Here is a simple explanation for the problem. I want to run a 15 minute
> random write, using 1 Meg requests, and measure throughput and latency.
> What seems to be the problem is that if the test system has a large
> number of drives - the system that I am testing here has 28 drives -
> then the time accounting seems to go bad for some of the processes.
> What you see below is that during the 15 minutes from start, all disks
> are getting hit the same, as they should. Then, after 15 minutes, there
> are 15 drives that are still running.... after 5 minutes over the
> specified 15 minutes, there is still one drive running. Then looking at
> the amount of IOs sent to each drive, the ones that ran on that excess
> time have much more IOs. FIO still reports that all drives ran for 15
> minutes, although some ran for more than 20 minutes.
>
> We will attempt to run a single process instead of 28 instances of FIO
> to see if this goes away.

Could you also check if adding clocksource=gettimeofday makes any 
difference? This sounds very odd.

Assuming this was run with fio -git?

-- 
Jens Axboe



  reply	other threads:[~2015-11-20 19:50 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CADp+U7ibiKciX8_cpzGzob4oL-UF-H+W7kYuiujovD0ba=hM6A@mail.gmail.com>
     [not found] ` <56464ACC.9030605@kernel.dk>
2015-11-13 22:04   ` Running a separate fio process for each disk? Allen Schade
2015-11-13 22:06     ` Jens Axboe
2015-11-20 18:28       ` Allen Schade
2015-11-20 19:37         ` Caio Villela
2015-11-20 19:50           ` Jens Axboe [this message]
2015-11-20 22:20             ` Akash Verma
2015-11-21  0:03               ` Jens Axboe
2015-11-21  0:21                 ` Jens Axboe
2015-11-24 15:51                   ` Jens Axboe
2015-11-24 20:51                     ` Akash Verma
2015-11-25  1:18                       ` Jens Axboe
2015-12-03 18:54                         ` Akash Verma
2015-12-03 18:58                           ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=564F798A.8050009@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=aschade@google.com \
    --cc=caio@google.com \
    --cc=fio@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.