inflated bandwidth numbers with buffered I/O

inflated bandwidth numbers with buffered I/O

[Dallas Clement]

Hi, I hope it's ok to ask usage questions on this mailing list.

I have been using fio version 2.2.4 on a Fedora 22 server host to test
throughput on a 10 GigE LIO iSCSI block device.  Up until this point I
have been testing with Direct I/O (direct=1) and writing directly to
the block device with filename=/dev/blah (no filesystem).  I am seeing
bandwidth measurements that look reasonable.  I next tried to repeat
this test with Buffered I/O (direct=0) and the bandwidth numbers are
way too large.  Here is a comparison of the sequential write results
for various block sizes:

bs=4k => direct=95.945 MB/s, buffered=1475.4 MB/s
bs=512 => direct=495.328 MB/s, buffered=2637.333 MB/s
bs=2048 => direct=502.093 MB/s, buffered=2663.772 MB/s

As you can see the buffered bandwidth measurements are much larger and
don't make sense given that not more than 1200 MB/s can be transmitted
over a 10 Gbps network connection.

Now it's very likely I am doing something stupid and lack
understanding about how fio works.  Someone please enlighten me!

Here is the script I have been using to collect my results for buffered I/O

#!/bin/bash

JOB_FILE=job.fio

cat << EOF > ${JOB_FILE}
[job]
ioengine=libaio
iodepth=\${DEPTH}
prio=0
rw=\${RW}
bs=\${BS}
filename=/dev/sdc
numjobs=1
size=10g
direct=0
invalidate=0
ramp_time=15
runtime=120
time_based
write_bw_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
write_lat_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
write_iops_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
EOF

for RW in read write randread randwrite
do
  for BS in 4k 512k 2048k
  do
    for DEPTH in 4 32 256
    do
      RW=${RW} BS=${BS} DEPTH=${DEPTH} fio ${JOB_FILE}
    done
  done
done

I have also tried running this script without the filename=blah
setting on an XFS formatted block device.  I am still seeing inflated
numbers for this also.

Thanks,

Dallas

Re: inflated bandwidth numbers with buffered I/O

[Matthew Eaton]

On Thu, Jan 14, 2016 at 3:28 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> Hi, I hope it's ok to ask usage questions on this mailing list.
>
> I have been using fio version 2.2.4 on a Fedora 22 server host to test
> throughput on a 10 GigE LIO iSCSI block device.  Up until this point I
> have been testing with Direct I/O (direct=1) and writing directly to
> the block device with filename=/dev/blah (no filesystem).  I am seeing
> bandwidth measurements that look reasonable.  I next tried to repeat
> this test with Buffered I/O (direct=0) and the bandwidth numbers are
> way too large.  Here is a comparison of the sequential write results
> for various block sizes:
>
> bs=4k => direct=95.945 MB/s, buffered=1475.4 MB/s
> bs=512 => direct=495.328 MB/s, buffered=2637.333 MB/s
> bs=2048 => direct=502.093 MB/s, buffered=2663.772 MB/s
>
> As you can see the buffered bandwidth measurements are much larger and
> don't make sense given that not more than 1200 MB/s can be transmitted
> over a 10 Gbps network connection.
>
> Now it's very likely I am doing something stupid and lack
> understanding about how fio works.  Someone please enlighten me!
>
> Here is the script I have been using to collect my results for buffered I/O
>
> #!/bin/bash
>
> JOB_FILE=job.fio
>
> cat << EOF > ${JOB_FILE}
> [job]
> ioengine=libaio
> iodepth=\${DEPTH}
> prio=0
> rw=\${RW}
> bs=\${BS}
> filename=/dev/sdc
> numjobs=1
> size=10g
> direct=0
> invalidate=0
> ramp_time=15
> runtime=120
> time_based
> write_bw_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
> write_lat_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
> write_iops_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
> EOF
>
> for RW in read write randread randwrite
> do
>   for BS in 4k 512k 2048k
>   do
>     for DEPTH in 4 32 256
>     do
>       RW=${RW} BS=${BS} DEPTH=${DEPTH} fio ${JOB_FILE}
>     done
>   done
> done
>
> I have also tried running this script without the filename=blah
> setting on an XFS formatted block device.  I am still seeing inflated
> numbers for this also.
>
> Thanks,
>
> Dallas
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


I don't have much experience in testing devices over the network but
you could try increasing size= or leave it out completely to use the
full device size and see how that affects your throughput.

Re: inflated bandwidth numbers with buffered I/O

[Jens Axboe]

On 01/14/2016 04:28 PM, Dallas Clement wrote:
> Hi, I hope it's ok to ask usage questions on this mailing list.
>
> I have been using fio version 2.2.4 on a Fedora 22 server host to test
> throughput on a 10 GigE LIO iSCSI block device.  Up until this point I
> have been testing with Direct I/O (direct=1) and writing directly to
> the block device with filename=/dev/blah (no filesystem).  I am seeing
> bandwidth measurements that look reasonable.  I next tried to repeat
> this test with Buffered I/O (direct=0) and the bandwidth numbers are
> way too large.  Here is a comparison of the sequential write results
> for various block sizes:
>
> bs=4k => direct=95.945 MB/s, buffered=1475.4 MB/s
> bs=512 => direct=495.328 MB/s, buffered=2637.333 MB/s
> bs=2048 => direct=502.093 MB/s, buffered=2663.772 MB/s
>
> As you can see the buffered bandwidth measurements are much larger and
> don't make sense given that not more than 1200 MB/s can be transmitted
> over a 10 Gbps network connection.
>
> Now it's very likely I am doing something stupid and lack
> understanding about how fio works.  Someone please enlighten me!
>
> Here is the script I have been using to collect my results for buffered I/O
>
> #!/bin/bash
>
> JOB_FILE=job.fio
>
> cat << EOF > ${JOB_FILE}
> [job]
> ioengine=libaio
> iodepth=\${DEPTH}
> prio=0
> rw=\${RW}
> bs=\${BS}
> filename=/dev/sdc
> numjobs=1
> size=10g
> direct=0
> invalidate=0
> ramp_time=15
> runtime=120
> time_based
> write_bw_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
> write_lat_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
> write_iops_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
> EOF
>
> for RW in read write randread randwrite
> do
>    for BS in 4k 512k 2048k
>    do
>      for DEPTH in 4 32 256
>      do
>        RW=${RW} BS=${BS} DEPTH=${DEPTH} fio ${JOB_FILE}
>      done
>    done
> done
>
> I have also tried running this script without the filename=blah
> setting on an XFS formatted block device.  I am still seeing inflated
> numbers for this also.

buffered IO is, well, buffered. So if you don't include any kind of 
sync'ing, then you are merely measuring how long it took you to dirty 
the amount of memory specified. You can set end_fsync=1 and fio will do 
an fsync at the end of the job run. Or you can sprinkle fsync/fdatasync 
to sync for every X blocks. Or you can set sync=1, which would open the 
device O_SYNC.

axboe@xps13:/home/axboe/git/fio $ ./fio --cmdhelp|grep sync
fsync                   : Issue fsync for writes every given number of 
blocks
fdatasync               : Issue fdatasync for writes every given number 
of blocks
sync_file_range         : Use sync_file_range()
   sync                  : Use O_SYNC for buffered writes
   verify_async          : Number of async verifier threads to use
create_fsync            : fsync file after creation
end_fsync               : Include fsync at the end of job
fsync_on_close          : fsync files on close


-- 
Jens Axboe

Re: inflated bandwidth numbers with buffered I/O

[Dallas Clement]

On Fri, Jan 15, 2016 at 9:45 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 01/14/2016 04:28 PM, Dallas Clement wrote:
>>
>> Hi, I hope it's ok to ask usage questions on this mailing list.
>>
>> I have been using fio version 2.2.4 on a Fedora 22 server host to test
>> throughput on a 10 GigE LIO iSCSI block device.  Up until this point I
>> have been testing with Direct I/O (direct=1) and writing directly to
>> the block device with filename=/dev/blah (no filesystem).  I am seeing
>> bandwidth measurements that look reasonable.  I next tried to repeat
>> this test with Buffered I/O (direct=0) and the bandwidth numbers are
>> way too large.  Here is a comparison of the sequential write results
>> for various block sizes:
>>
>> bs=4k => direct=95.945 MB/s, buffered=1475.4 MB/s
>> bs=512 => direct=495.328 MB/s, buffered=2637.333 MB/s
>> bs=2048 => direct=502.093 MB/s, buffered=2663.772 MB/s
>>
>> As you can see the buffered bandwidth measurements are much larger and
>> don't make sense given that not more than 1200 MB/s can be transmitted
>> over a 10 Gbps network connection.
>>
>> Now it's very likely I am doing something stupid and lack
>> understanding about how fio works.  Someone please enlighten me!
>>
>> Here is the script I have been using to collect my results for buffered
>> I/O
>>
>> #!/bin/bash
>>
>> JOB_FILE=job.fio
>>
>> cat << EOF > ${JOB_FILE}
>> [job]
>> ioengine=libaio
>> iodepth=\${DEPTH}
>> prio=0
>> rw=\${RW}
>> bs=\${BS}
>> filename=/dev/sdc
>> numjobs=1
>> size=10g
>> direct=0
>> invalidate=0
>> ramp_time=15
>> runtime=120
>> time_based
>> write_bw_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
>> write_lat_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
>> write_iops_log=\${RW}-bs-\${BS}-depth-\${DEPTH}
>> EOF
>>
>> for RW in read write randread randwrite
>> do
>>    for BS in 4k 512k 2048k
>>    do
>>      for DEPTH in 4 32 256
>>      do
>>        RW=${RW} BS=${BS} DEPTH=${DEPTH} fio ${JOB_FILE}
>>      done
>>    done
>> done
>>
>> I have also tried running this script without the filename=blah
>> setting on an XFS formatted block device.  I am still seeing inflated
>> numbers for this also.
>
>
> buffered IO is, well, buffered. So if you don't include any kind of
> sync'ing, then you are merely measuring how long it took you to dirty the
> amount of memory specified. You can set end_fsync=1 and fio will do an fsync
> at the end of the job run. Or you can sprinkle fsync/fdatasync to sync for
> every X blocks. Or you can set sync=1, which would open the device O_SYNC.
>
> axboe@xps13:/home/axboe/git/fio $ ./fio --cmdhelp|grep sync
> fsync                   : Issue fsync for writes every given number of
> blocks
> fdatasync               : Issue fdatasync for writes every given number of
> blocks
> sync_file_range         : Use sync_file_range()
>   sync                  : Use O_SYNC for buffered writes
>   verify_async          : Number of async verifier threads to use
> create_fsync            : fsync file after creation
> end_fsync               : Include fsync at the end of job
> fsync_on_close          : fsync files on close
>
>
> --
> Jens Axboe
>

Hi Jens.

> So if you don't include any kind of
> sync'ing, then you are merely measuring how long it took you to dirty the
> amount of memory specified.

I suspected that might be the case.  Is this dirty memory on the
client side where fio is running or on the target host where the data
has been written to?

I would like to make fio sync in a similar fashion to what real
applications would do.  Setting sync=1 is probably too aggressive.
Probably a sync after a given number of blocks would seem more
realistic.

Thanks for the help!

Re: inflated bandwidth numbers with buffered I/O

[Sitsofe Wheeler]

On 15 January 2016 at 16:09, Dallas Clement <dallas.a.clement@gmail.com> wrote:
>
> I suspected that might be the case.  Is this dirty memory on the
> client side where fio is running or on the target host where the data
> has been written to?

It will be on the client side - you aren't waiting for anything to
flush so it can even just queue in the kernel's page cache.

> I would like to make fio sync in a similar fashion to what real
> applications would do.  Setting sync=1 is probably too aggressive.
> Probably a sync after a given number of blocks would seem more
> realistic.

Try taking a look at the fsync parameter in the HOWTO:
https://github.com/axboe/fio/blob/fio-2.3/HOWTO#L909 .

-- 
Sitsofe | http://sucs.org/~sits/

Re: inflated bandwidth numbers with buffered I/O

[Dallas Clement]

On Sun, Jan 17, 2016 at 12:00 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 15 January 2016 at 16:09, Dallas Clement <dallas.a.clement@gmail.com> wrote:
>>
>> I suspected that might be the case.  Is this dirty memory on the
>> client side where fio is running or on the target host where the data
>> has been written to?
>

Hi Sitsofe,

> It will be on the client side - you aren't waiting for anything to
> flush so it can even just queue in the kernel's page cache.
>

Thanks for confirming that.  That definitely explains what I'm seeing then.

>> I would like to make fio sync in a similar fashion to what real
>> applications would do.  Setting sync=1 is probably too aggressive.
>> Probably a sync after a given number of blocks would seem more
>> realistic.
>
> Try taking a look at the fsync parameter in the HOWTO:
> https://github.com/axboe/fio/blob/fio-2.3/HOWTO#L909 .

Thanks.  I have been playing around with this fsync parameter.  I have
tried various numbers ranging from 32 to 128.  I do see reasonable
numbers now.  However throughput is quite a bit less than I was seeing
with direct I/O (direct=1).  I was expecting I would actually get
better performance with buffered I/O.  Am I misguided?

Re: inflated bandwidth numbers with buffered I/O

[Sitsofe Wheeler]

On 18 January 2016 at 03:20, Dallas Clement <dallas.a.clement@gmail.com> wrote:
>
> Thanks.  I have been playing around with this fsync parameter.  I have
> tried various numbers ranging from 32 to 128.  I do see reasonable
> numbers now.  However throughput is quite a bit less than I was seeing
> with direct I/O (direct=1).  I was expecting I would actually get
> better performance with buffered I/O.  Am I misguided?

I would expect async direct I/O to be able to go faster (speed is it's
mostly what it's there for...). Direct asynchronous I/O can be faster
because you don't have the overhead of copying things into the page
cache. Additionally, you don't necessarily suffer from head of line
blocking - each I/O can potentially finish independently of any other.

-- 
Sitsofe | http://sucs.org/~sits/

Re: inflated bandwidth numbers with buffered I/O

[Dallas Clement]

> I would expect async direct I/O to be able to go faster (speed is it's
> mostly what it's there for...). Direct asynchronous I/O can be faster
> because you don't have the overhead of copying things into the page
> cache. Additionally, you don't necessarily suffer from head of line
> blocking - each I/O can potentially finish independently of any other.

Okay, makes sense and definitely jives with my measurements.  Thanks
for explaining.