Writing to /dev/null with fio

Writing to /dev/null with fio

[Bart Van Assche]

Hello Jens,

I noticed that writing with fio to /dev/null takes much more time (220
times more for the test below) than with dd. Did I specify the correct
arguments to fio ? If so, is this known behavior of fio ?

$ dd if=/dev/zero of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.121485 s, 8.8 GB/s

$ fio --bs=1M --size=1G --buffered=1 --rw=write --verify=0
--name=dev-null /dev/null
dev-null: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/0K /s] [0/0 iops] [eta 00m:00s]
dev-null: (groupid=0, jobs=1): err= 0: pid=15859
  write: io=1,024MB, bw=39,214KB/s, iops=38, runt= 26740msec
    clat (usec): min=547, max=1,708K, avg=25927.47, stdev=124258.62
    bw (KB/s) : min= 1167, max=732000, per=125.95%, avg=49387.11,
stdev=119413.97
  cpu          : usr=0.01%, sys=4.72%, ctx=780, majf=0, minf=138
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1024, short=0/0
     lat (usec): 750=47.95%, 1000=39.26%
     lat (msec): 2=4.10%, 4=0.49%, 10=0.88%, 20=0.10%, 100=0.29%
     lat (msec): 250=3.12%, 500=2.34%, 750=0.68%, 1000=0.49%, 2000=0.29%

Run status group 0 (all jobs):
  WRITE: io=1,024MB, aggrb=39,213KB/s, minb=40,154KB/s,
maxb=40,154KB/s, mint=26740msec, maxt=26740msec

Bart.

Re: Writing to /dev/null with fio

[Jens Axboe]

On Tue, Feb 02 2010, Bart Van Assche wrote:
> Hello Jens,
> 
> I noticed that writing with fio to /dev/null takes much more time (220
> times more for the test below) than with dd. Did I specify the correct
> arguments to fio ? If so, is this known behavior of fio ?
> 
> $ dd if=/dev/zero of=/dev/null bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 0.121485 s, 8.8 GB/s
> 
> $ fio --bs=1M --size=1G --buffered=1 --rw=write --verify=0
> --name=dev-null /dev/null

You are writing to a file named dev-null and /dev/null is being ignored
as an argument. You want a --filename=/dev/null at the end instead.

-- 
Jens Axboe

Re: Writing to /dev/null with fio

[Bart Van Assche]

On Tue, Feb 2, 2010 at 8:41 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> On Tue, Feb 02 2010, Bart Van Assche wrote:
>> I noticed that writing with fio to /dev/null takes much more time (220
>> times more for the test below) than with dd. Did I specify the correct
>> arguments to fio ? If so, is this known behavior of fio ?
>>
>> $ dd if=/dev/zero of=/dev/null bs=1M count=1024
>> 1024+0 records in
>> 1024+0 records out
>> 1073741824 bytes (1.1 GB) copied, 0.121485 s, 8.8 GB/s
>>
>> $ fio --bs=1M --size=1G --buffered=1 --rw=write --verify=0
>> --name=dev-null /dev/null
>
> You are writing to a file named dev-null and /dev/null is being ignored
> as an argument. You want a --filename=/dev/null at the end instead.

Sorry for this mistake.

The reason I started running such silly tests is because I noticed
that tests with dd and a small block size complete in a shorter time
than tests with fio for a fast storage device (e.g. remote RAM disk
accessed via SRP or iSER). Do the two tests below trigger similar
system calls ? The ratio of fio time / dd time is about 1.50 for block
size 512 and about 1.15 for block size 4096.

$ dd if=/dev/zero of=/dev/null bs=512 count=$((2**34/512))
33554432+0 records in
33554432+0 records out
17179869184 bytes (17 GB) copied, 22.1956 s, 774 MB/s

$ fio --bs=512 --size=$((2**34)) --buffered=1 --rw=write --verify=0
--name=dev-null --filename=/dev/null
dev-null: (g=0): rw=write, bs=512-512/512-512, ioengine=sync, iodepth=1
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/496M /s] [0/992K iops] [eta 00m:00s]
dev-null: (groupid=0, jobs=1): err= 0: pid=15978
  write: io=16,384MB, bw=483MB/s, iops=989K, runt= 33931msec
    clat (usec): min=0, max=182, avg= 0.38, stdev= 0.50
    bw (KB/s) : min=408342, max=496691, per=100.00%, avg=494450.67,
stdev=10697.23
  cpu          : usr=62.81%, sys=37.15%, ctx=491, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/33554432, short=0/0
     lat (usec): 2=99.96%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01%
     lat (usec): 100=0.01%, 250=0.01%

Run status group 0 (all jobs):
  WRITE: io=16,384MB, aggrb=483MB/s, minb=494MB/s, maxb=494MB/s,
mint=33931msec, maxt=33931msec

$ dd if=/dev/zero of=/dev/null bs=4096 count=$((2**34/4096))
4194304+0 records in
4194304+0 records out
17179869184 bytes (17 GB) copied, 3.72608 s, 4.6 GB/s

$ fio --bs=4096 --size=$((2**34)) --buffered=1 --rw=write --verify=0
--name=dev-null --filename=/dev/null
dev-null: (g=0): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/4,005M /s] [0/1M iops] [eta 00m:00s]
dev-null: (groupid=0, jobs=1): err= 0: pid=16600
  write: io=16,384MB, bw=3,809MB/s, iops=975K, runt=  4301msec
    clat (usec): min=0, max=644, avg= 0.39, stdev= 0.60
    bw (KB/s) : min=3146072, max=4002800, per=99.81%, avg=3893298.00,
stdev=301949.65
  cpu          : usr=61.86%, sys=38.09%, ctx=121, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/4194304, short=0/0
     lat (usec): 2=99.96%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01%
     lat (usec): 100=0.01%, 250=0.01%, 750=0.01%

Run status group 0 (all jobs):
  WRITE: io=16,384MB, aggrb=3,809MB/s, minb=3,901MB/s, maxb=3,901MB/s,
mint=4301msec, maxt=4301msec

Bart.

Re: Writing to /dev/null with fio

[Jens Axboe]

On Tue, Feb 02 2010, Bart Van Assche wrote:
> On Tue, Feb 2, 2010 at 8:41 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> > On Tue, Feb 02 2010, Bart Van Assche wrote:
> >> I noticed that writing with fio to /dev/null takes much more time (220
> >> times more for the test below) than with dd. Did I specify the correct
> >> arguments to fio ? If so, is this known behavior of fio ?
> >>
> >> $ dd if=/dev/zero of=/dev/null bs=1M count=1024
> >> 1024+0 records in
> >> 1024+0 records out
> >> 1073741824 bytes (1.1 GB) copied, 0.121485 s, 8.8 GB/s
> >>
> >> $ fio --bs=1M --size=1G --buffered=1 --rw=write --verify=0
> >> --name=dev-null /dev/null
> >
> > You are writing to a file named dev-null and /dev/null is being ignored
> > as an argument. You want a --filename=/dev/null at the end instead.
> 
> Sorry for this mistake.
> 
> The reason I started running such silly tests is because I noticed
> that tests with dd and a small block size complete in a shorter time
> than tests with fio for a fast storage device (e.g. remote RAM disk
> accessed via SRP or iSER). Do the two tests below trigger similar
> system calls ? The ratio of fio time / dd time is about 1.50 for block
> size 512 and about 1.15 for block size 4096.

Fio definitely has more overhead than a simple read() to buf, write buf
to /dev/null. If you switch off the stat calculations, it'll drop
somewhat (use --gtod_reduce=1). But even then it's going to be slower
than dd. Fio is modular and supports different IO engines etc, so the IO
path is going to be a lot longer than with dd. The flexibility of fio
does come at a cost. If you time(1) fio and dd, you'll most likely see a
lot more usr time in fio.

That said, it is probably time to do some profiling and make sure that
fio is as fast as it can be.

-- 
Jens Axboe

Re: Writing to /dev/null with fio

[Bart Van Assche]

On Tue, Feb 2, 2010 at 9:19 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> On Tue, Feb 02 2010, Bart Van Assche wrote:
>> The reason I started running such silly tests is because I noticed
>> that tests with dd and a small block size complete in a shorter time
>> than tests with fio for a fast storage device (e.g. remote RAM disk
>> accessed via SRP or iSER). Do the two tests below trigger similar
>> system calls ? The ratio of fio time / dd time is about 1.50 for block
>> size 512 and about 1.15 for block size 4096.
>
> Fio definitely has more overhead than a simple read() to buf, write buf
> to /dev/null. If you switch off the stat calculations, it'll drop
> somewhat (use --gtod_reduce=1). But even then it's going to be slower
> than dd. Fio is modular and supports different IO engines etc, so the IO
> path is going to be a lot longer than with dd. The flexibility of fio
> does come at a cost. If you time(1) fio and dd, you'll most likely see a
> lot more usr time in fio.
>
> That said, it is probably time to do some profiling and make sure that
> fio is as fast as it can be.

That would definitely be appreciated. I would like to switch from dd
to fio for storage system benchmarking, something I can't do yet
because of the different results reported by the two tools.

Bart.

Re: Writing to /dev/null with fio

[Jens Axboe]

On Tue, Feb 02 2010, Bart Van Assche wrote:
> On Tue, Feb 2, 2010 at 9:19 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> > On Tue, Feb 02 2010, Bart Van Assche wrote:
> >> The reason I started running such silly tests is because I noticed
> >> that tests with dd and a small block size complete in a shorter time
> >> than tests with fio for a fast storage device (e.g. remote RAM disk
> >> accessed via SRP or iSER). Do the two tests below trigger similar
> >> system calls ? The ratio of fio time / dd time is about 1.50 for block
> >> size 512 and about 1.15 for block size 4096.
> >
> > Fio definitely has more overhead than a simple read() to buf, write buf
> > to /dev/null. If you switch off the stat calculations, it'll drop
> > somewhat (use --gtod_reduce=1). But even then it's going to be slower
> > than dd. Fio is modular and supports different IO engines etc, so the IO
> > path is going to be a lot longer than with dd. The flexibility of fio
> > does come at a cost. If you time(1) fio and dd, you'll most likely see a
> > lot more usr time in fio.
> >
> > That said, it is probably time to do some profiling and make sure that
> > fio is as fast as it can be.
> 
> That would definitely be appreciated. I would like to switch from dd
> to fio for storage system benchmarking, something I can't do yet
> because of the different results reported by the two tools.

So the first thing I noticed is that you get an lseek() because fio
doesn't track the sequential nature of that job. How close do you get
for bs=512 with using --gtod_reduce=1 and commenting out the lseek() in
engines/sync.c:fio_syncio_prep()?

Alternatively, using --ioengine=psync would remove that overhead as
well.

But realize that fio will never be as fast as dd completely for plain
sync and sequential IO, it's just not possible.

-- 
Jens Axboe

Re: Writing to /dev/null with fio

[Jens Axboe]

On Tue, Feb 02 2010, Jens Axboe wrote:
> On Tue, Feb 02 2010, Bart Van Assche wrote:
> > On Tue, Feb 2, 2010 at 9:19 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> > > On Tue, Feb 02 2010, Bart Van Assche wrote:
> > >> The reason I started running such silly tests is because I noticed
> > >> that tests with dd and a small block size complete in a shorter time
> > >> than tests with fio for a fast storage device (e.g. remote RAM disk
> > >> accessed via SRP or iSER). Do the two tests below trigger similar
> > >> system calls ? The ratio of fio time / dd time is about 1.50 for block
> > >> size 512 and about 1.15 for block size 4096.
> > >
> > > Fio definitely has more overhead than a simple read() to buf, write buf
> > > to /dev/null. If you switch off the stat calculations, it'll drop
> > > somewhat (use --gtod_reduce=1). But even then it's going to be slower
> > > than dd. Fio is modular and supports different IO engines etc, so the IO
> > > path is going to be a lot longer than with dd. The flexibility of fio
> > > does come at a cost. If you time(1) fio and dd, you'll most likely see a
> > > lot more usr time in fio.
> > >
> > > That said, it is probably time to do some profiling and make sure that
> > > fio is as fast as it can be.
> > 
> > That would definitely be appreciated. I would like to switch from dd
> > to fio for storage system benchmarking, something I can't do yet
> > because of the different results reported by the two tools.
> 
> So the first thing I noticed is that you get an lseek() because fio
> doesn't track the sequential nature of that job. How close do you get
> for bs=512 with using --gtod_reduce=1 and commenting out the lseek() in
> engines/sync.c:fio_syncio_prep()?

I committed a fix for this, it'll do it automatically now if you use
latest -git. A quick test here on my desktop machine:

fio --bs=4k --size=64G --buffered=1 --rw=write --verify=0
--name=/dev/null --gtod_reduce=1 --disk_util=0

10378MB/s

fio --bs=512 --size=8G --buffered=1 --rw=write --verify=0
--name=/dev/null --gtod_reduce=1 --disk_util=0

1306MB/s

dd if=/dev/zero of=/dev/null bs=4k count=16M
68719476736 bytes (69 GB) copied, 9,71235 s, 7,1 GB/s

dd if=/dev/zero of=/dev/null bs=512 count=8M
4294967296 bytes (4,3 GB) copied, 3,3574 s, 1,3 GB/s

So the same for 512b buffers, fio is much quicker for 64k (must be due
to proper aligning). I'll boot the big box and see what that says.

-- 
Jens Axboe

Re: Writing to /dev/null with fio

[Bart Van Assche]

On Tue, Feb 2, 2010 at 9:53 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> I committed a fix for this, it'll do it automatically now if you use
> latest -git. A quick test here on my desktop machine:
>
> fio --bs=4k --size=64G --buffered=1 --rw=write --verify=0
> --name=/dev/null --gtod_reduce=1 --disk_util=0
>
> 10378MB/s
>
> fio --bs=512 --size=8G --buffered=1 --rw=write --verify=0
> --name=/dev/null --gtod_reduce=1 --disk_util=0
>
> 1306MB/s
>
> dd if=/dev/zero of=/dev/null bs=4k count=16M
> 68719476736 bytes (69 GB) copied, 9,71235 s, 7,1 GB/s
>
> dd if=/dev/zero of=/dev/null bs=512 count=8M
> 4294967296 bytes (4,3 GB) copied, 3,3574 s, 1,3 GB/s
>
> So the same for 512b buffers, fio is much quicker for 64k (must be due
> to proper aligning). I'll boot the big box and see what that says.

Thanks, this helps: the ratio of fio to dd runtime is now reduced to
1.25 for a block size of 512 bytes and 0.96 for a block size of 4096
bytes.

$ count=$((2**25)); for bs in 512 4096; do size=$((count*bs)); dd
if=/dev/zero of=/dev/null bs=$bs count=$count 2>&1|grep copied; fio
--bs=$bs --size=$size --buffered=1 --rw=write --verify=0
--name=dev-null --filename=/dev/null 2>&1|grep runt; done
17179869184 bytes (17 GB) copied, 22.1017 s, 777 MB/s
  write: io=16,384MB, bw=621MB/s, iops=1,272K, runt= 26381msec
137438953472 bytes (137 GB) copied, 27.5793 s, 5.0 GB/s
  write: io=128GB, bw=4,964MB/s, iops=1,271K, runt= 26407msec

Bart.

Re: Writing to /dev/null with fio

[Jens Axboe]

On Tue, Feb 02 2010, Jens Axboe wrote:
> So the same for 512b buffers, fio is much quicker for 64k (must be due
> to proper aligning). I'll boot the big box and see what that says.

Same set of tests, only this time using a box with a lot more CPUs and
memory bandwidth. 'fio+' is fio with the fix for eliminating lseek(),
'fio' is without that fix.

App             bs              bandwidth
-----------------------------------------
dd              512              1.7GB/sec
fio             512              1.2GB/sec
fio+            512              1.5GB/sec
dd               4k              6.6GB/sec
fio              4k             10.3GB/sec
fio+             4k             12.4GB/sec

So dd is still faster for small blocks sizes, as expected. I don't think
there's any low hanging fruit left for fixing that. At 4k and higher the
impact should be neglible, so I don't think it's a huge problem. Most
real world testing is 4k or above.

-- 
Jens Axboe