Fwd: strange discrepancy

Fwd: strange discrepancy

[Pal, Laszlo]

Hi,

Finally I've figured out how to use FIO :) now, I can measure several
things, so I've started to get IOPS data for our hardware. The graphs
looks good, however I can see a very strange discrepancy at seq_write
part

The HW itself is an appliance built from supermicro components and
using software raid and running ubuntu squeezy based OS

FIO paramaters I'm currently using as the follows

[global]
ioengine=libaio
buffered=0
direct=1
bs=4k
blocksize_range=1k-64k
size=2048m

[seq_read]
rw=read
write_iolog=iops_seq_read
write_bw_log=iops_seq_read
write_lat_log=iops_seq_read
write_iops_log=iops_seq_read

[seq_write]
rw=write
write_iolog=iops_seq_write
write_bw_log=iops_seq_write
write_lat_log=iops_seq_write
write_iops_log=iops_seq_write

[random_read]
rw=randread
write_iolog=iops_rand_read
write_bw_log=iops_rand_read
write_lat_log=iops_rand_read
write_iops_log=iops_rand_read

[random_write]
rw=randwrite
write_iolog=iops_rand_write
write_bw_log=iops_rand_write
write_lat_log=iops_rand_write
write_iops_log=iops_rand_write

[random_rw]
rw=randrw
write_iolog=iops_rand_randrw
write_bw_log=iops_rand_randrw
write_lat_log=iops_rand_randrw
write_iops_log=iops_rand_randrw

------------------------------

the results as the follows

Random Read -- 86
Random Write -- 77
Sequential Read -- 2170
Sequential Write -- 45
Random RW -- 40

---------------------------------------------------------

I have two questions again.

The parameters above are good approach to determine IOPS for this system?

What can be the reason of the huge difference between Seq_read and Seq_write?

Thank you for your help
L:

RE: strange discrepancy

[Carl Zwanzig]

> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Pal, Laszlo

> Finally I've figured out how to use FIO :) now, I can measure several
> things, so I've started to get IOPS data for our hardware. The graphs
> looks good, however I can see a very strange discrepancy at seq_write
> part

> [global]
> ioengine=libaio
Without iodepth >1, you'll effectively be doing synchronous i/o, so there's no benefit to using libaio. If you're interested in throughput (not latency), set iodepth to 512, which will likely be greater than whatever is in the drive/controller/OS.

> buffered=0
Don't need this if you specify-
> direct=1


> bs=4k
> blocksize_range=1k-64k

AFAIK, blocksize_range will supersede bs. Also depending on the drive (what type?), this can cause mis-aligned writes which massively slow the overall process. I've mostly seen this with 4kn and 512e SSDs. You can examine that by trying blockalign=4k.

> size=2048m
I usually use a runtime spec instead of size, and also set fill_device=true.

[snip]

> 
> the results as the follows
(iops or MB/s?)
> Random Read -- 86
> Random Write -- 77
> Sequential Read -- 2170
> Sequential Write -- 45

> I have two questions again.
> The parameters above are good approach to determine IOPS for this system?
See above  :)

> What can be the reason of the huge difference between Seq_read and Seq_write?

Many things. First off, is this a block level test or is there a file system involved? What drives? What controller? RAID? 

z!

Re: strange discrepancy

[Pal, Laszlo]

Hi,

I've modified the .fio file according to the suggestions. Now, it
looks like this

[global]
ioengine=sync
direct=1
blocksize_range=1k-64k
directory=/mnt/drbd/private/
size=5120000000
runtime=7200

[seq_read]
rw=read
write_bw_log=iops_seq_read
write_lat_log=iops_seq_read
write_iops_log=iops_seq_read

[seq_write]
rw=write
write_bw_log=iops_seq_write
write_lat_log=iops_seq_write
write_iops_log=iops_seq_write

[random_read]
rw=randread
write_bw_log=iops_rand_read
write_lat_log=iops_rand_read
write_iops_log=iops_rand_read

[random_write]
rw=randwrite
write_bw_log=iops_rand_write
write_lat_log=iops_rand_write
write_iops_log=iops_rand_write

[random_rw]
rw=randrw
write_bw_log=iops_rand_randrw
write_lat_log=iops_rand_randrw
write_iops_log=iops_rand_randrw



Now, I'm getting much worst results (around 30 IOPS for random). I'm
wondering if the above config is the right approach for measur the
IOPS of this system?

As I written below this is a debian based box, using soft raid with
DRBD and my task is to tell what is the IOPS of this box


Any idea?

Thank you
Lastlo

On 21 February 2014 20:47, Pal, Laszlo <vlad@vlad.hu> wrote:
> Thank you for your answer. I'm quite new in this, so I'm sure my parameters
> maybe not optimal.
>
> The system is a supermicro appliance running an ubuntu variant and using
> 2x7200 RPM disk in raid1 with DRBD
>
> The other box is a bit stronger and includes and adaptec raid controller
> 12x7200 RPM disk in RAID50 but also with DRBD
>
> The goal is to measure the IOPS of this two hardware running this software.
> Basically the IOPS values I've got with these parameters are very close to
> the average IOPS of a 7200 disk which is as far as I know arund 100
>
> I also put some comments inline, so if you can help me with same appropritae
> parameters for this scenario it would be great
>
> Thank you
> Laszlo
>
>
>
> On 21 February 2014 20:31, Carl Zwanzig <cpz@coraid.com> wrote:
>>
>> > From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
>> > Behalf Of Pal, Laszlo
>>
>> > Finally I've figured out how to use FIO :) now, I can measure several
>> > things, so I've started to get IOPS data for our hardware. The graphs
>> > looks good, however I can see a very strange discrepancy at seq_write
>> > part
>>
>> > [global]
>> > ioengine=libaio
>> Without iodepth >1, you'll effectively be doing synchronous i/o, so
>> there's no benefit to using libaio. If you're interested in throughput (not
>> latency), set iodepth to 512, which will likely be greater than whatever is
>> in the drive/controller/OS.
>>
>> > buffered=0
>> Don't need this if you specify-
>> > direct=1
>>
>>
>> > bs=4k
>> > blocksize_range=1k-64k
>>
>> AFAIK, blocksize_range will supersede bs. Also depending on the drive
>> (what type?), this can cause mis-aligned writes which massively slow the
>> overall process. I've mostly seen this with 4kn and 512e SSDs. You can
>> examine that by trying blockalign=4k.
>>
> it is normal 7200K disk
>>
>> > size=2048m
>> I usually use a runtime spec instead of size, and also set
>> fill_device=true.
>
>
>>
>>
>> [snip]
>>
>> >
>> > the results as the follows
>> (iops or MB/s?)
>
> these numbers from the IOPS log generated by fio
>>
>> > Random Read -- 86
>> > Random Write -- 77
>> > Sequential Read -- 2170
>> > Sequential Write -- 45
>>
>> > I have two questions again.
>> > The parameters above are good approach to determine IOPS for this
>> > system?
>> See above  :)
>>
>> > What can be the reason of the huge difference between Seq_read and
>> > Seq_write?
>>
>> Many things. First off, is this a block level test or is there a file
>> system involved? What drives? What controller? RAID?
>
> it is file system (ext3) soft raid + DRBD
>>
>>
>> z!
>>
>

Re: strange discrepancy

[Erwan Velu]

On 24/02/2014 12:55, Pal, Laszlo wrote:
> Hi,
>
> I've modified the .fio file according to the suggestions. Now, it
> looks like this
>
>
I'm pretty sure you are making a mistake here.
You shall use the 'stonewall' instruction of your jobs if you want to 
make them one after the other unless you are running all of them at the 
same time.

My genfio script inside the tools/ directory would have surely help you 
avoiding this mistake.... but I'm not yet supporting the filesystems 
with it... I have to spend some time on it.

At least, generate an output with it and change then after the block 
device by your filesystem stuff.

The nice stuff is that genfio is easy to "pipe" with fio2gnuplot. Btw, 
I'm glad some are using it ;o)))

Erwan,

RE: strange discrepancy

[Carl Zwanzig]

Couple of things-
If you want to test the -storage- system, use async IO with a good size queue depth. Sync will be very slow comparatively.
As mentioned, all tests will run in parallel unless you stonewall them.
Don't make logs that you don't really need; you probably don't need any of those.

Give the HOWTO file a really close read.

z!
writing without full benefit of the morning coffee

RE: strange discrepancy

[Carl Zwanzig]

> From: Pal, Laszlo [mailto:vlad@vlad.hu] 
> Sent: Monday, February 24, 2014 11:27 AM

> thank you. if I stonewalling sync engine is a good approach ? The main goal is to tell our customers what iops they have to provide to get the same or similar io performance as they have when using the appliances.

I'm not sure what you're asking. If you're trying to model a specific workload, well, you have to define what that workload is. If you're trying to see how fast the storage system itself can go, then the tests you've listed are rational -provided- that you don't run them in parallel ('stonewall') and that you don't limit yourself at the client (by using sync i/o).

On the systems I test, I can see a great improvement in the throughput of RAID volumes by increasing the iodepth. Think of it, with sync i/o (iodepth=1) you're doing one operation at a time and to only one drive*. If you have 8 drives in a RAID-0 set, -each- drive can be doing something at the same time, but with iodepth<8, that'll never happen. Likewise, assuming modern drives, each drive can queue up some operations internally and optimize head movement. This also improves performance.  (I usually use iodepth=32 or higher; on linux.)

*I'm ignoring parity RAID configs where each write to the RAID set causes reads and writes to the entire volume. Nothing will help you there :).

z!

Re: strange discrepancy

[Pal, Laszlo]

Thank you. The workload is mixed read / write , small and big packets,
and I have raid1 sets in one of the box and raid50 in the big boy. I
suppose using the iodepth=32 will show me more realistic numbers than
my original tests. Is this a valid assumption if I increase iodepth
until I get bigger IOPS values will get me closer and closer to the
maximum IOPS can provide the config?

Thank you
Laszlo


On 24 February 2014 21:16, Carl Zwanzig <cpz@coraid.com> wrote:
>> From: Pal, Laszlo [mailto:vlad@vlad.hu]
>> Sent: Monday, February 24, 2014 11:27 AM
>
>> thank you. if I stonewalling sync engine is a good approach ? The main goal is to tell our customers what iops they have to provide to get the same or similar io performance as they have when using the appliances.
>
> I'm not sure what you're asking. If you're trying to model a specific workload, well, you have to define what that workload is. If you're trying to see how fast the storage system itself can go, then the tests you've listed are rational -provided- that you don't run them in parallel ('stonewall') and that you don't limit yourself at the client (by using sync i/o).
>
> On the systems I test, I can see a great improvement in the throughput of RAID volumes by increasing the iodepth. Think of it, with sync i/o (iodepth=1) you're doing one operation at a time and to only one drive*. If you have 8 drives in a RAID-0 set, -each- drive can be doing something at the same time, but with iodepth<8, that'll never happen. Likewise, assuming modern drives, each drive can queue up some operations internally and optimize head movement. This also improves performance.  (I usually use iodepth=32 or higher; on linux.)
>
> *I'm ignoring parity RAID configs where each write to the RAID set causes reads and writes to the entire volume. Nothing will help you there :).
>
> z!
>

Re: strange discrepancy

[Matthew Eaton]

On Mon, Feb 24, 2014 at 12:30 PM, Pal, Laszlo <vlad@vlad.hu> wrote:
> Thank you. The workload is mixed read / write , small and big packets,
> and I have raid1 sets in one of the box and raid50 in the big boy. I
> suppose using the iodepth=32 will show me more realistic numbers than
> my original tests. Is this a valid assumption if I increase iodepth
> until I get bigger IOPS values will get me closer and closer to the
> maximum IOPS can provide the config?
>
> Thank you
> Laszlo

You'll want to experiment with different iodepths and use whatever
gives you the best results.  And if you're looking to get a max IOPS
you probably want to stick to 4k transfers.

I would try something like this:

[global]
ioengine=libaio
direct=1
iodepth=128
size=50g
runtime=5m
directory=/mnt/drbd/private/

[random_write]
rw=randwrite
bs=4k
stonewall

[random_read]
rw=randread
bs=4k
stonewall

[seq_write]
rw=write
bs=128k
stonewall

[seq_read]
rw=read
bs=128k
stonewall