correct file size to use

correct file size to use

[jvogt]

Hi list

I was wondering:
When i start
fio --name=/srv/mail/test --size=10M --direct=1 --runtime=60 
--time_based --group_reporting --refill_buffer --rw=read --numjobs=1

I get as result:
read : io=1177.4MB, bw=20093KB/s, iops=5023, runt= 60000msec

But when I do a
fio --name=/srv/mail/test --size=10G --direct=1 --runtime=60 
--time_based --group_reporting --refill_buffer --rw=read --numjobs=1

The result is:
read : io=976692KB, bw=16278KB/s, iops=4069, runt= 60001msec
(decrease of 20%, in some scenarios it's even worse with more than 30%)

So how does the file size influence the speed (iops, bandwith)? My 
understanding was that it is not important from which filesize we are 
reading from - important is the block size and if the filesize is not 
big enough, fio just restarts to read the file from the beginning. How 
can that be explained? And which sizes should be used to get reliable 
results?

Thanks
Josef

Re: correct file size to use

[Ming Lei]

On Sat, Mar 15, 2014 at 7:10 PM, jvogt <jvogt@adon.li> wrote:
> Hi list
>
> I was wondering:
> When i start
> fio --name=/srv/mail/test --size=10M --direct=1 --runtime=60 --time_based
> --group_reporting --refill_buffer --rw=read --numjobs=1
>
> I get as result:
> read : io=1177.4MB, bw=20093KB/s, iops=5023, runt= 60000msec

From fio/HOWTO:

time_based  If set, fio will run for the duration of the runtime
        specified even if the file(s) are completely read or
        written. It will simply loop over the same workload
        as many times as the runtime allows.

The same 10M file will be read many times(118), so most
of times it is read from page cache instead of the file, and
the result can't be accurate enough.



Thanks,
-- 
Ming Lei

Re: correct file size to use

[Matthew Eaton]

On Sun, Mar 16, 2014 at 5:12 AM, Ming Lei <tom.leiming@gmail.com> wrote:
> On Sat, Mar 15, 2014 at 7:10 PM, jvogt <jvogt@adon.li> wrote:
>> Hi list
>>
>> I was wondering:
>> When i start
>> fio --name=/srv/mail/test --size=10M --direct=1 --runtime=60 --time_based
>> --group_reporting --refill_buffer --rw=read --numjobs=1
>>
>> I get as result:
>> read : io=1177.4MB, bw=20093KB/s, iops=5023, runt= 60000msec
>
> From fio/HOWTO:
>
> time_based  If set, fio will run for the duration of the runtime
>         specified even if the file(s) are completely read or
>         written. It will simply loop over the same workload
>         as many times as the runtime allows.
>
> The same 10M file will be read many times(118), so most
> of times it is read from page cache instead of the file, and
> the result can't be accurate enough.
>
>
>
> Thanks,
> --
> Ming Lei
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Tested on my system and saw no difference in performance.  Direct io
will bypass the page cache so the only thing that comes to mind would
be an on-device cache, though, I would think the difference in
performance would be more dramatic if that were the case.

RE: correct file size to use

[Carl Zwanzig]

[sorry for the lack of quoting, OWA doesn't do it nicely.]

Which OS & file system? If the storage target is an NFS mount, you'll still have the server cache. Also, IIRC solaris doesn't honor direct=1 anyway. If zfs is involved.... there's no escaping it's optimizations :).

Please give us more info about the setup.

z!

Re: correct file size to use

[Ming Lei]

On Mon, Mar 17, 2014 at 7:16 AM, Matthew Eaton
>
> Tested on my system and saw no difference in performance.  Direct io
> will bypass the page cache so the only thing that comes to mind would

Looks I miss the '--direct=1' parameter, :-(

> be an on-device cache, though, I would think the difference in
> performance would be more dramatic if that were the case.

Another possible cause might be related with storage for the
big file. If the file is stored in lot of fragments of the disk, accessing
the whole file might become slower than just the first 10M, but not
sure if there is so big difference(30%).


Thanks,
-- 
Ming Lei

Re: correct file size to use

[jvogt]

Am 17.03.2014 03:55, schrieb Ming Lei:
> On Mon, Mar 17, 2014 at 7:16 AM, Matthew Eaton
>> Tested on my system and saw no difference in performance.  Direct io
>> will bypass the page cache so the only thing that comes to mind would
> Looks I miss the '--direct=1' parameter, :-(
>
>> be an on-device cache, though, I would think the difference in
>> performance would be more dramatic if that were the case.
> Another possible cause might be related with storage for the
> big file. If the file is stored in lot of fragments of the disk, accessing
> the whole file might become slower than just the first 10M, but not
> sure if there is so big difference(30%).

> Direct io
will bypass the page cache so the only thing that comes to mind would
be an on-device cache, though, I would think the difference in
performance would be more dramatic if that were the case.

This could be possible - I'm not so sure how to check that...
My setup is the following:
RHEL6.5 as virtual machine via VMWare - the storage is a Dell Equallogic 
Storage connected via ISCSI to the host. These tests are not made via NFS.
I just wanted to make some tests before taking it all in production.
Thank you all for your answers!
Josef

Re: correct file size to use

[Sitsofe Wheeler]

On Mon, Mar 17, 2014 at 09:13:47PM +0100, jvogt wrote:
> 
> I just wanted to make some tests before taking it all in production.
> Thank you all for your answers!

If you want raw upper maximum numbers try using direct=1 ioengine=libaio
with an appropriate iodepth against the openiscsi block device. Bear in
mind things like iodepth and whether coalescing can take place can make
a huge difference...

-- 
Sitsofe | http://sucs.org/~sits/

Re: correct file size to use

[jvogt]

Am 17.03.2014 21:34, schrieb Sitsofe Wheeler:
> On Mon, Mar 17, 2014 at 09:13:47PM +0100, jvogt wrote:
>> I just wanted to make some tests before taking it all in production.
>> Thank you all for your answers!
> If you want raw upper maximum numbers try using direct=1 ioengine=libaio
> with an appropriate iodepth against the openiscsi block device. Bear in
> mind things like iodepth and whether coalescing can take place can make
> a huge difference...
Please explain, what do you mean with appropriate iodepth? What would 
you suggest?

RE: correct file size to use

[Carl Zwanzig]

> From: jvogt [mailto:jvogt@adon.li]
> Sent: Monday, March 17, 2014 1:41 PM

> Please explain, what do you mean with appropriate iodepth? What would
> you suggest?

Something higher than 1 :).  I use 512, since AFAIK no storage array out there will actually queue up that many operations.

z!