multiple block devices question

multiple block devices question

[Yuyang (Alex) Wang]

hi all,

I am using FIO (2.1.7) to profile block level performance of multiple
RAID devices. (see config file below). The test is done on a capable
16-core server. With a sigle raid device, I am getting roughly
1.4GB/s, with two devices in place, I am still getting around 1.4GB/s,
which is weird. Another in-house developed benchmark tool, running
SINGLE-thread suggested we can almost double. I could get double the
performance through FIO by configuring two jobs, with each job
handling one device only. However, it still puzzles me as to why a
single process FIO can't drive the performance of two devices. Did I
do anything wrong as to the configure file?

Thanks

Alex.


[globals]

rw=write
ioengine=libaio
iodepth=16
direct=1
blocksize=1m
runtime=30

[devices]
filename=/dev/mapper/raid-0
filename=/dev/mapper/raid-1

Re: multiple block devices question

[Jens Axboe]

On 07/02/2014 01:42 PM, Yuyang (Alex) Wang wrote:
> hi all,
> 
> I am using FIO (2.1.7) to profile block level performance of multiple
> RAID devices. (see config file below). The test is done on a capable
> 16-core server. With a sigle raid device, I am getting roughly
> 1.4GB/s, with two devices in place, I am still getting around 1.4GB/s,
> which is weird. Another in-house developed benchmark tool, running
> SINGLE-thread suggested we can almost double. I could get double the
> performance through FIO by configuring two jobs, with each job
> handling one device only. However, it still puzzles me as to why a
> single process FIO can't drive the performance of two devices. Did I
> do anything wrong as to the configure file?
> 
> Thanks
> 
> Alex.
> 
> 
> [globals]
> 
> rw=write
> ioengine=libaio
> iodepth=16
> direct=1
> blocksize=1m
> runtime=30
> 
> [devices]
> filename=/dev/mapper/raid-0
> filename=/dev/mapper/raid-1

You are overwriting filename here, hence running with just one device.

-- 
Jens Axboe

Re: multiple block devices question

[Yuyang (Alex) Wang]

Jens, thanks for the reply.

I also tried the following as the HOWTO suggested, and getting similar results
[devices]
filename=/dev/mapper/raid0:/dev/mapper/raid1


This may be silly part on me misreading the manual, what's correct way
of specifying multiple devices?

thanks

Alex





On Wed, Jul 2, 2014 at 3:50 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 07/02/2014 01:42 PM, Yuyang (Alex) Wang wrote:
>> hi all,
>>
>> I am using FIO (2.1.7) to profile block level performance of multiple
>> RAID devices. (see config file below). The test is done on a capable
>> 16-core server. With a sigle raid device, I am getting roughly
>> 1.4GB/s, with two devices in place, I am still getting around 1.4GB/s,
>> which is weird. Another in-house developed benchmark tool, running
>> SINGLE-thread suggested we can almost double. I could get double the
>> performance through FIO by configuring two jobs, with each job
>> handling one device only. However, it still puzzles me as to why a
>> single process FIO can't drive the performance of two devices. Did I
>> do anything wrong as to the configure file?
>>
>> Thanks
>>
>> Alex.
>>
>>
>> [globals]
>>
>> rw=write
>> ioengine=libaio
>> iodepth=16
>> direct=1
>> blocksize=1m
>> runtime=30
>>
>> [devices]
>> filename=/dev/mapper/raid-0
>> filename=/dev/mapper/raid-1
>
> You are overwriting filename here, hence running with just one device.
>
> --
> Jens Axboe
>

RE: multiple block devices question

[Carl Zwanzig]

> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Yuyang (Alex) Wang
> Sent: Wednesday, July 02, 2014 1:36 PM

> I also tried the following as the HOWTO suggested, and getting similar
> results
> [devices]
> filename=/dev/mapper/raid0:/dev/mapper/raid1

I specify them as separate jobs:
[devices0]
filename=/dev/mapper/raid0
[devices1]
filename=/dev/mapper/raid1

without the "stonewall" directive, the jobs will run simultaneously.

z!

Re: multiple block devices question

[Yuyang (Alex) Wang]

Carl,

Specifying them as separate jobs will do what I expect. However, in
this case, I am seeking to know how much performance and how many
devices can a __single__ FIO process can drive. If my above
configuration is correct, then, for some reason, the single process
doesn't increase with more devices at its disposal, not as I have seen
from another benchmark tool - which is puzzling. I am curious to
understand why.

Best,

A.

On Wed, Jul 2, 2014 at 7:38 PM, Carl Zwanzig <cpz@coraid.com> wrote:
>> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
>> Behalf Of Yuyang (Alex) Wang
>> Sent: Wednesday, July 02, 2014 1:36 PM
>
>> I also tried the following as the HOWTO suggested, and getting similar
>> results
>> [devices]
>> filename=/dev/mapper/raid0:/dev/mapper/raid1
>
> I specify them as separate jobs:
> [devices0]
> filename=/dev/mapper/raid0
> [devices1]
> filename=/dev/mapper/raid1
>
> without the "stonewall" directive, the jobs will run simultaneously.
>
> z!
>

Re: multiple block devices question

[David Nellans]

On 07/03/2014 12:28 PM, Yuyang (Alex) Wang wrote:
> Carl,
>
> Specifying them as separate jobs will do what I expect. However, in
> this case, I am seeking to know how much performance and how many
> devices can a __single__ FIO process can drive. If my above
> configuration is correct, then, for some reason, the single process
> doesn't increase with more devices at its disposal, not as I have seen
> from another benchmark tool - which is puzzling. I am curious to
> understand why.
>
> Best,
>
> A.
If you want to benchmark fio throughput or the block layer, not your 
actual raid array,
then use a ramdisk.  There are lots of flash devices out there now which 
will push far
more than 1.4GB/s from a single device,  rest assured fio isn't your 
limitation here.
1.4 GB/s with 1M block sizes isn't even 2k IOPS.  fio pushes 500k IOPS 
through the block layer
without issue, its not your limiting factor.

Re: multiple block devices question

[Yuyang (Alex) Wang]

I get it and agree with you that FIO shouldn't be the limiting factor,
but I don't get it why I am not getting sensible numbers when I
increase the number of devices. Apparently, I am only interested in
actual throughput FIO can drive my raid array, rather than ramdisk in
this case.

Thanks

A.

On Thu, Jul 3, 2014 at 1:45 PM, David Nellans <david@nellans.org> wrote:
> On 07/03/2014 12:28 PM, Yuyang (Alex) Wang wrote:
>>
>> Carl,
>>
>> Specifying them as separate jobs will do what I expect. However, in
>> this case, I am seeking to know how much performance and how many
>> devices can a __single__ FIO process can drive. If my above
>> configuration is correct, then, for some reason, the single process
>> doesn't increase with more devices at its disposal, not as I have seen
>> from another benchmark tool - which is puzzling. I am curious to
>> understand why.
>>
>> Best,
>>
>> A.
>
> If you want to benchmark fio throughput or the block layer, not your actual
> raid array,
> then use a ramdisk.  There are lots of flash devices out there now which
> will push far
> more than 1.4GB/s from a single device,  rest assured fio isn't your
> limitation here.
> 1.4 GB/s with 1M block sizes isn't even 2k IOPS.  fio pushes 500k IOPS
> through the block layer
> without issue, its not your limiting factor.

Re: multiple block devices question

[Jeff Moyer]

"Yuyang (Alex) Wang" <yyalex.wang@gmail.com> writes:

> Jens, thanks for the reply.
>
> I also tried the following as the HOWTO suggested, and getting similar results
> [devices]
> filename=/dev/mapper/raid0:/dev/mapper/raid1
>
>
> This may be silly part on me misreading the manual, what's correct way
> of specifying multiple devices?

That looks ok to me.  What happens if you double your queue depth?

Cheers,
Jeff

RE: multiple block devices question

[Hinson, Roger]

It doesn't work properly with 2.1.7.  it only grabs the first name in filename.  I switched to 2.1.10 to fix that problem for me.

Roger


-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jeff Moyer
Sent: Thursday, July 03, 2014 11:12 AM
To: Yuyang (Alex) Wang
Cc: Jens Axboe; fio@vger.kernel.org
Subject: Re: multiple block devices question

"Yuyang (Alex) Wang" <yyalex.wang@gmail.com> writes:

> Jens, thanks for the reply.
>
> I also tried the following as the HOWTO suggested, and getting similar 
> results [devices]
> filename=/dev/mapper/raid0:/dev/mapper/raid1
>
>
> This may be silly part on me misreading the manual, what's correct way 
> of specifying multiple devices?

That looks ok to me.  What happens if you double your queue depth?

Cheers,
Jeff
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multiple block devices question

[Yuyang (Alex) Wang]

Kudos Roger, that solves the problem.
I didn't realize one release behind, the filename parsing has this bug in it.

Thanks!

A.

On Thu, Jul 3, 2014 at 2:21 PM, Hinson, Roger <Roger_Hinson@intuit.com> wrote:
> It doesn't work properly with 2.1.7.  it only grabs the first name in filename.  I switched to 2.1.10 to fix that problem for me.
>
> Roger
>
>
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jeff Moyer
> Sent: Thursday, July 03, 2014 11:12 AM
> To: Yuyang (Alex) Wang
> Cc: Jens Axboe; fio@vger.kernel.org
> Subject: Re: multiple block devices question
>
> "Yuyang (Alex) Wang" <yyalex.wang@gmail.com> writes:
>
>> Jens, thanks for the reply.
>>
>> I also tried the following as the HOWTO suggested, and getting similar
>> results [devices]
>> filename=/dev/mapper/raid0:/dev/mapper/raid1
>>
>>
>> This may be silly part on me misreading the manual, what's correct way
>> of specifying multiple devices?
>
> That looks ok to me.  What happens if you double your queue depth?
>
> Cheers,
> Jeff
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multiple block devices question

[Yuyang (Alex) Wang]

hi Jeff,

iodepth doesn't seem to make difference - see output (this is with two
raid array)

devices: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
fio-2.1.7
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0KB/1227MB/0KB /s] [0/1227/0 iops]
[eta 00m:00s]
devices: (groupid=0, jobs=1): err= 0: pid=91658: Thu Jul  3 14:20:42 2014
  write: io=41368MB, bw=1376.7MB/s, iops=1376, runt= 30050msec
    slat (usec): min=104, max=8419, avg=328.49, stdev=63.54
    clat (usec): min=4, max=114758, avg=22883.96, stdev=8585.56
     lat (msec): min=2, max=115, avg=23.21, stdev= 8.58
    clat percentiles (msec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   21],
     | 30.00th=[   23], 40.00th=[   23], 50.00th=[   23], 60.00th=[   25],
     | 70.00th=[   25], 80.00th=[   26], 90.00th=[   28], 95.00th=[   35],
     | 99.00th=[   61], 99.50th=[   68], 99.90th=[   80], 99.95th=[   89],
     | 99.99th=[  110]
    bw (MB  /s): min=  926, max= 2790, per=100.00%, avg=1376.74, stdev=420.52
    lat (usec) : 10=0.01%
    lat (msec) : 4=0.03%, 10=0.39%, 20=19.28%, 50=78.27%, 100=1.99%
    lat (msec) : 250=0.04%
  cpu          : usr=30.09%, sys=17.91%, ctx=19628, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=41368/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=41368MB, aggrb=1376.7MB/s, minb=1376.7MB/s,
maxb=1376.7MB/s, mint=30050msec, maxt=30050msec



On Thu, Jul 3, 2014 at 2:12 PM, Jeff Moyer <jmoyer@redhat.com> wrote:
> "Yuyang (Alex) Wang" <yyalex.wang@gmail.com> writes:
>
>> Jens, thanks for the reply.
>>
>> I also tried the following as the HOWTO suggested, and getting similar results
>> [devices]
>> filename=/dev/mapper/raid0:/dev/mapper/raid1
>>
>>
>> This may be silly part on me misreading the manual, what's correct way
>> of specifying multiple devices?
>
> That looks ok to me.  What happens if you double your queue depth?
>
> Cheers,
> Jeff

Re: multiple block devices question

[Jens Axboe]

On 07/03/2014 12:22 PM, Yuyang (Alex) Wang wrote:
> hi Jeff,
> 
> iodepth doesn't seem to make difference - see output (this is with two
> raid array)
> 
> devices: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
> fio-2.1.7
> Starting 1 process
> Jobs: 1 (f=1): [W] [100.0% done] [0KB/1227MB/0KB /s] [0/1227/0 iops]
> [eta 00m:00s]
> devices: (groupid=0, jobs=1): err= 0: pid=91658: Thu Jul  3 14:20:42 2014
>   write: io=41368MB, bw=1376.7MB/s, iops=1376, runt= 30050msec
>     slat (usec): min=104, max=8419, avg=328.49, stdev=63.54
>     clat (usec): min=4, max=114758, avg=22883.96, stdev=8585.56
>      lat (msec): min=2, max=115, avg=23.21, stdev= 8.58
>     clat percentiles (msec):
>      |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   21],
>      | 30.00th=[   23], 40.00th=[   23], 50.00th=[   23], 60.00th=[   25],
>      | 70.00th=[   25], 80.00th=[   26], 90.00th=[   28], 95.00th=[   35],
>      | 99.00th=[   61], 99.50th=[   68], 99.90th=[   80], 99.95th=[   89],
>      | 99.99th=[  110]
>     bw (MB  /s): min=  926, max= 2790, per=100.00%, avg=1376.74, stdev=420.52
>     lat (usec) : 10=0.01%
>     lat (msec) : 4=0.03%, 10=0.39%, 20=19.28%, 50=78.27%, 100=1.99%
>     lat (msec) : 250=0.04%
>   cpu          : usr=30.09%, sys=17.91%, ctx=19628, majf=0, minf=26
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>      issued    : total=r=0/w=41368/d=0, short=r=0/w=0/d=0
>      latency   : target=0, window=0, percentile=100.00%, depth=32
> 
> Run status group 0 (all jobs):
>   WRITE: io=41368MB, aggrb=1376.7MB/s, minb=1376.7MB/s,
> maxb=1376.7MB/s, mint=30050msec, maxt=30050msec

Just tested on a box here, for comparison. It has 16 SCSI disks, each
does roughly 160MB/sec in bandwidth. Adding disks scales bw linearly for
me, no problems observed.

You're submission latency is fairly low, so it should not be a problem
with fio getting stuck in dm on submission. usr+sys is pretty high
though, for such a light workload.

-- 
Jens Axboe