Can dbench be used for benchmarking fs?

Can dbench be used for benchmarking fs?

[E. Gryaznova]

Hello.

I use dbench for benchmarking the file systems and some results are
suspicious for me.

I run 10 times dbench on ext3 and i have the following :

Throughput 16.6754 MB/sec (NB=20.8443 MB/sec  166.754 MBit/sec)
Throughput 22.9772 MB/sec (NB=28.7216 MB/sec  229.772 MBit/sec)
Throughput 22.3698 MB/sec (NB=27.9623 MB/sec  223.698 MBit/sec)
Throughput 19.0533 MB/sec (NB=23.8167 MB/sec  190.533 MBit/sec)
Throughput 21.9662 MB/sec (NB=27.4577 MB/sec  219.662 MBit/sec)
Throughput 23.4062 MB/sec (NB=29.2578 MB/sec  234.062 MBit/sec)
Throughput 21.4233 MB/sec (NB=26.7791 MB/sec  214.233 MBit/sec)
Throughput 20.6202 MB/sec (NB=25.7753 MB/sec  206.202 MBit/sec)
Throughput 15.7005 MB/sec (NB=19.6256 MB/sec  157.005 MBit/sec)
Throughput 19.9631 MB/sec (NB=24.9538 MB/sec  199.631 MBit/sec)

As you can see
the average Throughput value is equal  20.4155 MB/sec
the max  value                         23.4062 MB/sec and
the min  value                         15.7005 MB/sec

As the result: the measuring deviation is equal = 23.4062 - 15.7005 =
7.7057 or about ~38% from average value.

I have dbench-1.1.tar.gz

I run dbench by the following way :
mke2fs -j /dev/xxx
mount /dev/xxx /mnt
cp dbench /mnt/.
cp clients.txt /mnt/.
cd /mnt
for x 1, 2, ... 10
do
    ./dbench 8 >>results
done

So, I have 2 questions :
1. Is there a way to avoid such big deviations on measuring a file
systems throughput and to get more stable results?
2. Can dbench be used for benchmarking the file systems and if it is so
-- what is the predictable error on the measuring?

Thank you very much for helping.
Lena.

Re: Can dbench be used for benchmarking fs?

[Andreas Dilger]

On Oct 07, 2003  16:42 +0400, E. Gryaznova wrote:
> I use dbench for benchmarking the file systems and some results are
> suspicious for me.
> :
> :
> :
> As the result: the measuring deviation is equal = 23.4062 - 15.7005 =
> 7.7057 or about ~38% from average value.
> 
> So, I have 2 questions :
> 1. Is there a way to avoid such big deviations on measuring a file
> systems throughput and to get more stable results?
> 2. Can dbench be used for benchmarking the file systems and if it is so
> -- what is the predictable error on the measuring?

Dbench is not a good filesystem benchmark, because it deletes all of the
files at the end.  Use something else for the filesystem benchmark - there
are lots of them (bonnie, iozone, mongo, etc).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Can dbench be used for benchmarking fs?

[Nikita Danilov]

Andreas Dilger writes:
 > On Oct 07, 2003  16:42 +0400, E. Gryaznova wrote:
 > > I use dbench for benchmarking the file systems and some results are
 > > suspicious for me.
 > > :
 > > :
 > > :
 > > As the result: the measuring deviation is equal = 23.4062 - 15.7005 =
 > > 7.7057 or about ~38% from average value.
 > > 
 > > So, I have 2 questions :
 > > 1. Is there a way to avoid such big deviations on measuring a file
 > > systems throughput and to get more stable results?
 > > 2. Can dbench be used for benchmarking the file systems and if it is so
 > > -- what is the predictable error on the measuring?
 > 
 > Dbench is not a good filesystem benchmark, because it deletes all of the
 > files at the end.  Use something else for the filesystem benchmark - there

Err... What is wrong with deleting all files at the end? Or do you mean
it should mix file operations during run?

 > are lots of them (bonnie, iozone, mongo, etc).

But why variance is so large?

Another probably interesting observation is that dbench works faster on
ext2 if it is run directly after mkfs.ext2, that is, without reboot
between mkfs and mount.

 > 
 > Cheers, Andreas

Nikita.

Re: Can dbench be used for benchmarking fs?

[Andreas Dilger]

On Oct 07, 2003  18:45 +0400, Nikita Danilov wrote:
> Andreas Dilger writes:
>  > On Oct 07, 2003  16:42 +0400, E. Gryaznova wrote:
>  > > I use dbench for benchmarking the file systems and some results are
>  > > suspicious for me.
>  > > :
>  > > :
>  > > :
>  > > As the result: the measuring deviation is equal = 23.4062 - 15.7005 =
>  > > 7.7057 or about ~38% from average value.
>  > > 
>  > > So, I have 2 questions :
>  > > 1. Is there a way to avoid such big deviations on measuring a file
>  > > systems throughput and to get more stable results?
>  > > 2. Can dbench be used for benchmarking the file systems and if it is so
>  > > -- what is the predictable error on the measuring?
>  > 
>  > Dbench is not a good filesystem benchmark, because it deletes all of the
>  > files at the end.  Use something else for the filesystem benchmark - there
> 
> Err... What is wrong with deleting all files at the end? Or do you mean
> it should mix file operations during run?
> 
>  > are lots of them (bonnie, iozone, mongo, etc).
> 
> But why variance is so large?

Dbench is a bad filesystem benchmark because if all (or some large part)
of the files are deleted before the VM flushes them to disk, then you are
not really testing filesystem performance very much.  If you have enough
RAM in your system and a fast enough CPU you could complete the entire
dbench run without ever writing any data to disk.  There is some filesystem
activity (you need to create files and map pages), but it isn't a good test
of filesystem throughput.

The reason there is so much variability in the tests is that if the test
takes even a small amount more time to run this means more data gets flushed
to disk by the VM (instead of being deleted and never written), making the
test run longer and even _more_ data needs to get written, etc.

Dbench is a good multi-threaded filesystem stress program, but it isn't
a good benchmark.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/