Solaris issues in 1.3.1

Solaris issues in 1.3.1

[Chris Worley]

I'm running fio 1.3.1 in Solaris.� I'm seeing two issues:

1) Direct I/O throws an error.
2) Performance is way too high.

W/o Direct I/O my size is set to 2.5x memory capacity to assure no
caching... yet performance for read/write is up to >7x the theoretical
speed of the storage device (i.e. it's reporting 7GB/s, when
theoretical for the device is 1GB/s).

I'm not seeing the reporting issue in 1.2.1.

Thanks,

Chris

Re: Solaris issues in 1.3.1

[Jens Axboe]

On Wed, Aug 05 2009, Chris Worley wrote:
> I'm running fio 1.3.1 in Solaris.� I'm seeing two issues:
> 
> 1) Direct I/O throws an error.
> 2) Performance is way too high.
> 
> W/o Direct I/O my size is set to 2.5x memory capacity to assure no
> caching... yet performance for read/write is up to >7x the theoretical
> speed of the storage device (i.e. it's reporting 7GB/s, when
> theoretical for the device is 1GB/s).
> 
> I'm not seeing the reporting issue in 1.2.1.

Are you familiar with git? If yes, then you could try and bisect the
issue. You would do something ala:

$ git bisect start
$ git bisect good fio-1.21
$ git bisect bad fio-1.31

and then git would checkout a point in between. You would then compile
and run, checking if it works or not. If the version works, then you
would reply with

$ git bisect good

if not, then you would do

$ git bisect bad

This continues until git tells you which commit is the defective one.
I'll give it a spin on Solaris too, I recently put Solaris 11 on a box
for test purposes.

-- 
Jens Axboe

Re: Solaris issues in 1.3.1

[Jens Axboe]

On Wed, Aug 05 2009, Jens Axboe wrote:
> On Wed, Aug 05 2009, Chris Worley wrote:
> > I'm running fio 1.3.1 in Solaris.� I'm seeing two issues:
> > 
> > 1) Direct I/O throws an error.
> > 2) Performance is way too high.
> > 
> > W/o Direct I/O my size is set to 2.5x memory capacity to assure no
> > caching... yet performance for read/write is up to >7x the theoretical
> > speed of the storage device (i.e. it's reporting 7GB/s, when
> > theoretical for the device is 1GB/s).
> > 
> > I'm not seeing the reporting issue in 1.2.1.
> 
> Are you familiar with git? If yes, then you could try and bisect the
> issue. You would do something ala:
> 
> $ git bisect start
> $ git bisect good fio-1.21
> $ git bisect bad fio-1.31
> 
> and then git would checkout a point in between. You would then compile
> and run, checking if it works or not. If the version works, then you
> would reply with
> 
> $ git bisect good
> 
> if not, then you would do
> 
> $ git bisect bad
> 
> This continues until git tells you which commit is the defective one.
> I'll give it a spin on Solaris too, I recently put Solaris 11 on a box
> for test purposes.

Oh, and if you are not, just wait and I'll double check things here. But
please send me your job file(s) so I can reproduce more easily, thanks.

-- 
Jens Axboe

Re: Solaris issues in 1.3.1

[Chris Worley]

On Wed, Aug 5, 2009 at 1:02 PM, Jens Axboe<jens.axboe@oracle.com> wrote:
> On Wed, Aug 05 2009, Jens Axboe wrote:
>> On Wed, Aug 05 2009, Chris Worley wrote:
>> > I'm running fio 1.3.1 in Solaris.  I'm seeing two issues:
>> >
>> > 1) Direct I/O throws an error.
>> > 2) Performance is way too high.
>> >
>> > W/o Direct I/O my size is set to 2.5x memory capacity to assure no
>> > caching... yet performance for read/write is up to >7x the theoretical
>> > speed of the storage device (i.e. it's reporting 7GB/s, when
>> > theoretical for the device is 1GB/s).
>> >
>> > I'm not seeing the reporting issue in 1.2.1.
>>
<snip>
>
> Oh, and if you are not, just wait and I'll double check things here. But
> please send me your job file(s) so I can reproduce more easily, thanks.

Sequential Reads and Writes seem most effected.  Random and random
mixes seem mostly unaffected, but I've seen them also go wild.  Here's
an example incantation:

fio --rw=write --bs=1m --numjobs=64 --iodepth=64 --sync=0
--randrepeat=0 --norandommap \
     --ioengine=sync --filename=/dev/<your device here> --name=test
--loops=1000 --size=128849018880 \
     --runtime=600 --group_reporting

Here's some example output (not from the above command, but a similar
command w/ 512KB blocks):

test: (g=0): rw=write, bs=512K-512K/512K-512K, ioengine=sync, iodepth=64
...
test: (g=0): rw=write, bs=512K-512K/512K-512K, ioengine=sync, iodepth=64
Starting 64 processes
Jobs: 1 (f=1): [_____________________________________________________W__________]
[1.6% done] [0K/0K /s] [0/0 iops] [eta 10h:29m:52s]     s]
test: (groupid=0, jobs=64): err= 0: pid=1222
  write: io=787299MiB, bw=1312MiB/s, iops=2623, runt=600153msec
    clat (usec): min=173, max=1107K, avg=24145.42, stdev=6786.14
    bw (KiB/s) : min=    0, max=347101, per=1.59%, avg=21306.67, stdev=4105.00
  cpu          : usr=0.04%, sys=3.31%, ctx=6406332, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/1574598, short=0/0
     lat (usec): 250=1.07%, 500=17.56%, 750=28.71%, 1000=4.53%
     lat (msec): 2=8.75%, 4=5.22%, 10=6.40%, 20=7.98%, 50=6.54%
     lat (msec): 100=2.02%, 250=10.58%, 500=0.64%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=787299MiB, aggrb=1312MiB/s, minb=1312MiB/s,
maxb=1312MiB/s, mint=600153msec, maxt=600153msec
test: (g=0): rw=read, bs=512K-512K/512K-512K, ioengine=sync, iodepth=64
...
test: (g=0): rw=read, bs=512K-512K/512K-512K, ioengine=sync, iodepth=64
Starting 64 processes
Jobs: 1 (f=1): [_______________________________________________________________R]
[1.6% done] [2153M/0K /s] [4205/0 iops] [eta 10h:29m:59s]
test: (groupid=0, jobs=64): err= 0: pid=1290
  read : io=145295MiB, bw=7233MiB/s, iops=148, runt=600009msec
    clat (usec): min=381, max=1251K, avg=4390.26, stdev=2714.96
    bw (KiB/s) : min=  538, max=451584, per=1.55%, avg=114760.43, stdev=6304.82
  cpu          : usr=0.15%, sys=24.90%, ctx=4695895, majf=1, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=8679198/0, short=0/0
     lat (usec): 500=0.01%, 750=0.02%, 1000=1.05%
     lat (msec): 2=89.40%, 4=3.57%, 10=1.86%, 20=0.86%, 50=1.29%
     lat (msec): 100=0.90%, 250=0.89%, 500=0.16%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%

Run status group 0 (all jobs):
   READ: io=145295MiB, aggrb=7233MiB/s, minb=7233MiB/s,
maxb=7233MiB/s, mint=600009msec, maxt=600009msec

Maybe if the "B" in "minb=7233MiB/s" meant "bits" it would be
correct... but then the "write" case (above) would be 5x too slow.

Thanks,

Chris
>
> --
> Jens Axboe
>
>