FIO 2.0.9 Windows - Job IOPS limitation

FIO 2.0.9 Windows - Job IOPS limitation

[Keith Orsak]

OS: Windows 2008 R2 Enterprise Edition
CPU: Intel X5660 2.8GHz (6-core)
Memory: 24GB
Device Under Test: 350GB SLC PCIe 2.0 x8

I seem to be running into a Job IOPS limitation in Windows.  When I
run a deep queue (iodepth=66 for example) the IOPS seem to be pegged
out around 200K.  If I split the iodepth between two jobs I can get
around 250K IOPS with the device I am testing which is what I would
expect to see.  Here is the command that I am currently using:

 fio --filename=\\.\PHYSICALDRIVE1 --ioengine=windowsaio --direct=1
--rw=randrw --bs=4K --rwmixread=70 --iodepth=66 --randrepeat=0
--norandommap --name=test --runtime=1000000 --time_based
--group_reporting

When I ran two jobs I simply added the numjobs=2 attribute in and
reduced iodepth to 33.

This is probably more of an enhancement request than an actual issue
but having to split the queue between two jobs makes it difficult to
get datapoints on even numbered queues beyond a certain point.

-Keith

Re: FIO 2.0.9 Windows - Job IOPS limitation

[Jens Axboe]

On 2012-10-05 00:32, Keith Orsak wrote:
> OS: Windows 2008 R2 Enterprise Edition
> CPU: Intel X5660 2.8GHz (6-core)
> Memory: 24GB
> Device Under Test: 350GB SLC PCIe 2.0 x8
> 
> I seem to be running into a Job IOPS limitation in Windows.  When I
> run a deep queue (iodepth=66 for example) the IOPS seem to be pegged
> out around 200K.  If I split the iodepth between two jobs I can get
> around 250K IOPS with the device I am testing which is what I would
> expect to see.  Here is the command that I am currently using:
> 
>  fio --filename=\\.\PHYSICALDRIVE1 --ioengine=windowsaio --direct=1
> --rw=randrw --bs=4K --rwmixread=70 --iodepth=66 --randrepeat=0
> --norandommap --name=test --runtime=1000000 --time_based
> --group_reporting
> 
> When I ran two jobs I simply added the numjobs=2 attribute in and
> reduced iodepth to 33.
> 
> This is probably more of an enhancement request than an actual issue
> but having to split the queue between two jobs makes it difficult to
> get datapoints on even numbered queues beyond a certain point.

Depending on how expensive the issue path is on Windows, it's not
unusual to require more than 1 thread/process to drive that number of
IOPS. In other words, this isn't necessarily a problem or limitation of
fio. I'd need to see some profiles to make an educated guess on that.
But it's a universal problem, not something limited to Windows or fio.

You might be able to squeeze some more IOPS out of a single process by
reducing the tracking that fio does. If you set --gtod_reduce=1 then fio
wont do as much time keeping. On the down side, you don't get good
latency statistics. You can also increase the completion batch count to
reduce the number of times you have to reap events, by playing with the
--iodepth_batch_completion setting.

-- 
Jens Axboe