Fio giving iops results that are equal to RAM speeds

Fio giving iops results that are equal to RAM speeds

[Kim Holmebakken]

Hey,

Im currently working on a bachelor thesis where I need to measure iops 
on my storage solution.

I have set up a "frontend" server(ubuntu server 13.10) which has zfs on 
linux running, and is connected to an iSCSI RAID(SAN).

When I use this command:

fio --directory=/pool  --size=128M --direct=0 --fallocate=none 
--rw=randrw --refill_buffers --norandommap --randrepeat=0 
--ioengine=sync --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 
--runtime=60 --group_reporting --name=4krandreadtest

I get:

4krandreadtest: (groupid=0, jobs=16): err= 0: pid=4101
   read : io=2048.0MB, bw=2158.7MB/s, iops=552463 , runt=   949msec
     clat (usec): min=4 , max=23125 , avg=26.15, stdev=346.02
      lat (usec): min=4 , max=23125 , avg=26.28, stdev=346.07
     clat percentiles (usec):
      |  1.00th=[    5],  5.00th=[    5], 10.00th=[    6], 20.00th=[    8],
      | 30.00th=[    9], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
      | 70.00th=[   11], 80.00th=[   12], 90.00th=[   16], 95.00th=[   19],
      | 99.00th=[   35], 99.50th=[  133], 99.90th=[ 5664], 99.95th=[ 8160],
      | 99.99th=[14528]
     bw (KB/s)  : min=114424, max=159840, per=6.09%, avg=134474.12, 
stdev=13435.6
7
     lat (usec) : 10=51.38%, 20=44.19%, 50=3.71%, 100=0.15%, 250=0.19%
     lat (usec) : 500=0.06%, 750=0.03%, 1000=0.02%
     lat (msec) : 2=0.05%, 4=0.08%, 10=0.12%, 20=0.03%, 50=0.01%
   cpu          : usr=5.76%, sys=37.95%, ctx=5049, majf=0, minf=409
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
 >=64=0.0%

  read : io=2048.0MB, bw=2158.7MB/s, iops=552463 , runt=   949msec
     clat (usec): min=4 , max=23125 , avg=26.15, stdev=346.02
      lat (usec): min=4 , max=23125 , avg=26.28, stdev=346.07
     clat percentiles (usec):
      |  1.00th=[    5],  5.00th=[    5], 10.00th=[    6], 20.00th=[    8],
      | 30.00th=[    9], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
      | 70.00th=[   11], 80.00th=[   12], 90.00th=[   16], 95.00th=[   19],
      | 99.00th=[   35], 99.50th=[  133], 99.90th=[ 5664], 99.95th=[ 8160],
      | 99.99th=[14528]
     bw (KB/s)  : min=114424, max=159840, per=6.09%, avg=134474.12, 
stdev=13435.6
7
     lat (usec) : 10=51.38%, 20=44.19%, 50=3.71%, 100=0.15%, 250=0.19%
     lat (usec) : 500=0.06%, 750=0.03%, 1000=0.02%
     lat (msec) : 2=0.05%, 4=0.08%, 10=0.12%, 20=0.03%, 50=0.01%
   cpu          : usr=5.76%, sys=37.95%, ctx=5049, majf=0, minf=409
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
 >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      issued    : total=r=524288/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
    READ: io=2048.0MB, aggrb=2158.7MB/s, minb=2158.7MB/s, 
maxb=2158.7MB/s, mint=9
49msec, maxt=949msec

These values are extremly high and I suspect that these are the RAM 
speed being returned to me.

Is there anyway to get a more realistic result?

Thanks in advance,
Kim Holmebakken

Re: Fio giving iops results that are equal to RAM speeds

[Erwan Velu]

Le 07/05/2014 09:45, Kim Holmebakken a �crit :
> Hey,
>
> Im currently working on a bachelor thesis where I need to measure iops 
> on my storage solution.
>
> I have set up a "frontend" server(ubuntu server 13.10) which has zfs 
> on linux running, and is connected to an iSCSI RAID(SAN).
>
> When I use this command:
>
> fio --directory=/pool  --size=128M --direct=0 --fallocate=none 
> --rw=randrw --refill_buffers --norandommap --randrepeat=0 
> --ioengine=sync --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 
> --runtime=60 --group_reporting --name=4krandreadtest

[...]
> These values are extremly high and I suspect that these are the RAM 
> speed being returned to me.
>
> Is there anyway to get a more realistic result?

You do cached IOs (as direct=0), only 128M of data and everything into a 
file. As a result, yes everything is getting into the host's cache.

If you want to determine the ISCSI performance byitself, I suggest not 
using file but the block device and do direct IOs (direct=1).

If you want to keep your filesystem you have two options : enabling the 
direct=1 and/or making IOs at twice the size of your host memory.

Hope this helps,
Erwan

Re: Fio giving iops results that are equal to RAM speeds

[puvi]

Is all the reads getting satisifed from your filesystem cache? Is your
memory much larger than the file size? Can you try a run after
unmount/mount to clear the fs cache?

On Wed, May 7, 2014 at 1:15 PM, Kim Holmebakken
<kim.holmebakken@hotmail.com> wrote:
> Hey,
>
> Im currently working on a bachelor thesis where I need to measure iops on my
> storage solution.
>
> I have set up a "frontend" server(ubuntu server 13.10) which has zfs on
> linux running, and is connected to an iSCSI RAID(SAN).
>
> When I use this command:
>
> fio --directory=/pool  --size=128M --direct=0 --fallocate=none --rw=randrw
> --refill_buffers --norandommap --randrepeat=0 --ioengine=sync --bs=4k
> --rwmixread=100 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting
> --name=4krandreadtest
>
> I get:
>
> 4krandreadtest: (groupid=0, jobs=16): err= 0: pid=4101
>   read : io=2048.0MB, bw=2158.7MB/s, iops=552463 , runt=   949msec
>     clat (usec): min=4 , max=23125 , avg=26.15, stdev=346.02
>      lat (usec): min=4 , max=23125 , avg=26.28, stdev=346.07
>     clat percentiles (usec):
>      |  1.00th=[    5],  5.00th=[    5], 10.00th=[    6], 20.00th=[    8],
>      | 30.00th=[    9], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
>      | 70.00th=[   11], 80.00th=[   12], 90.00th=[   16], 95.00th=[   19],
>      | 99.00th=[   35], 99.50th=[  133], 99.90th=[ 5664], 99.95th=[ 8160],
>      | 99.99th=[14528]
>     bw (KB/s)  : min=114424, max=159840, per=6.09%, avg=134474.12,
> stdev=13435.6
> 7
>     lat (usec) : 10=51.38%, 20=44.19%, 50=3.71%, 100=0.15%, 250=0.19%
>     lat (usec) : 500=0.06%, 750=0.03%, 1000=0.02%
>     lat (msec) : 2=0.05%, 4=0.08%, 10=0.12%, 20=0.03%, 50=0.01%
>   cpu          : usr=5.76%, sys=37.95%, ctx=5049, majf=0, minf=409
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>=64=0.0%
>
>  read : io=2048.0MB, bw=2158.7MB/s, iops=552463 , runt=   949msec
>     clat (usec): min=4 , max=23125 , avg=26.15, stdev=346.02
>      lat (usec): min=4 , max=23125 , avg=26.28, stdev=346.07
>     clat percentiles (usec):
>      |  1.00th=[    5],  5.00th=[    5], 10.00th=[    6], 20.00th=[    8],
>      | 30.00th=[    9], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
>      | 70.00th=[   11], 80.00th=[   12], 90.00th=[   16], 95.00th=[   19],
>      | 99.00th=[   35], 99.50th=[  133], 99.90th=[ 5664], 99.95th=[ 8160],
>      | 99.99th=[14528]
>     bw (KB/s)  : min=114424, max=159840, per=6.09%, avg=134474.12,
> stdev=13435.6
> 7
>     lat (usec) : 10=51.38%, 20=44.19%, 50=3.71%, 100=0.15%, 250=0.19%
>     lat (usec) : 500=0.06%, 750=0.03%, 1000=0.02%
>     lat (msec) : 2=0.05%, 4=0.08%, 10=0.12%, 20=0.03%, 50=0.01%
>   cpu          : usr=5.76%, sys=37.95%, ctx=5049, majf=0, minf=409
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>      issued    : total=r=524288/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
>    READ: io=2048.0MB, aggrb=2158.7MB/s, minb=2158.7MB/s, maxb=2158.7MB/s,
> mint=9
> 49msec, maxt=949msec
>
> These values are extremly high and I suspect that these are the RAM speed
> being returned to me.
>
> Is there anyway to get a more realistic result?
>
> Thanks in advance,
> Kim Holmebakken
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fio giving iops results that are equal to RAM speeds

[Erwan Velu]

Le 07/05/2014 10:19, Kim Holmebakken a �crit :
> Also the direct=0 is for the fact that in the documentation to fio the 
> direct= parameter is not supported for zfs. And as a result the test 
> will not complete(it does not even create the test files). 

So you need to make a much more longer run to minimize the cache effect. 
Running like twice the memory of the server could be an interesting 
test. 128M on a cached FS is clearly not enough.

Erwan,

Re: Fio giving iops results that are equal to RAM speeds

[Erwan Velu]

Le 07/05/2014 10:58, Kim Holmebakken a �crit :
> I tried running with the size=24G since the memory on the server is at 
> 12GB, this reduced the iops, however the MB/s is now reduced from 
> 2000+ MB/s, and now running on 850 MB/s
>
> Still i think this might be a bit high? 

If you do random IOs there is still a great chance of having cache hits. 
This number have to be put in comparison with the network bandwidth.

If you have a 1Gbit connection ... 850 is clearly hitting lot of cache.

Re: Fio giving iops results that are equal to RAM speeds

[Jens Axboe]

> Le 07/05/2014 10:19, Kim Holmebakken a �crit :
>> Also the direct=0 is for the fact that in the documentation to fio the
>> direct= parameter is not supported for zfs. And as a result the test
>> will not complete(it does not even create the test files). 

Clarification, this is not a fio limitation, it's zfs that doesn't
support unbuffered IO.


-- 
Jens Axboe