FIO is CPU bound on small block sizes

FIO is CPU bound on small block sizes

[Bike & Snow]

Hello

I am trying to measure IOPs on a RAID system using FIO.

I've been using FIO for a while and I'm pretty familiar with it.

However, I have found for small blocksizes, the benchmark is CPU bound
as it only works on a single CPU core.

Here's what I mean. All these are sequential read results.

Iometer results:
512B	181,317
1K		166,232
2K		133,753
4K		109,498
8K		 78,276
16K		 51,638

FIO results:
512B	 97,494
1K		100,034
2K		 98,615
4K		 93,859
8K		 83,064
16K		 69,623

I can see when observing the output from "top" that the "fio" thread
is at 99% for the smaller block sizes.
So it appears to be CPU bound for small block sizes - 512B to 4K.
From 8K onwards FIO is actually faster than Iometer.

Software:
Linux 2.6.36.3
FIO 1.50
Iometer 1.1.0

Hardware:
Intel Xeon X3450 @ 2.67GHz
Hardware RAID6 array

FIO command used:
fio -name iops -rw=read -bs=512 -runtime=180 -iodepth 4 -filename
/dev/storage/iometer -ioengine libaio -direct=1

As you can see I'm testing the block device directly without using a
file system.

I have tried different iodepth values but it makes no difference.

Any ideas on how I can measure the IOPs of the RAID device when using
small block sizes?

I really want to use FIO as I can script it.

Re: FIO is CPU bound on small block sizes

[Jens Axboe]

On 2011-02-08 11:30, Bike & Snow wrote:
> Hello
> 
> I am trying to measure IOPs on a RAID system using FIO.
> 
> I've been using FIO for a while and I'm pretty familiar with it.
> 
> However, I have found for small blocksizes, the benchmark is CPU bound
> as it only works on a single CPU core.
> 
> Here's what I mean. All these are sequential read results.
> 
> Iometer results:
> 512B	181,317
> 1K		166,232
> 2K		133,753
> 4K		109,498
> 8K		 78,276
> 16K		 51,638
> 
> FIO results:
> 512B	 97,494
> 1K		100,034
> 2K		 98,615
> 4K		 93,859
> 8K		 83,064
> 16K		 69,623
> 
> I can see when observing the output from "top" that the "fio" thread
> is at 99% for the smaller block sizes.
> So it appears to be CPU bound for small block sizes - 512B to 4K.
> From 8K onwards FIO is actually faster than Iometer.
> 
> Software:
> Linux 2.6.36.3
> FIO 1.50
> Iometer 1.1.0
> 
> Hardware:
> Intel Xeon X3450 @ 2.67GHz
> Hardware RAID6 array
> 
> FIO command used:
> fio -name iops -rw=read -bs=512 -runtime=180 -iodepth 4 -filename
> /dev/storage/iometer -ioengine libaio -direct=1
> 
> As you can see I'm testing the block device directly without using a
> file system.
> 
> I have tried different iodepth values but it makes no difference.
> 
> Any ideas on how I can measure the IOPs of the RAID device when using
> small block sizes?

Are you sure you are comparing the same IO pattern? One way to reduce
the fio overhead is to reduce the time keepnig that it does, in case
that isn't of interest here. Add --gtod_reduce=1 for that. Submit and
complete more in one batch, add --iodepth_batch_submit=2
--iodepth_batch_complete=2 for that for instance.

-- 
Jens Axboe

Re: FIO is CPU bound on small block sizes

[Chris Worley]

On Tue, Feb 8, 2011 at 3:30 AM, Bike & Snow <bikeandsnow@googlemail.com> wrote:
<snip>
> I can see when observing the output from "top" that the "fio" thread
> is at 99% for the smaller block sizes.

Why not increase the thread count?

Re: FIO is CPU bound on small block sizes

[Bike & Snow]

Thanks Chris & Jens.

Sounds really obviously now - just run multiple threads of FIO.

This is what I get now on 512B block size:
203,216 IOPs

That is using 8 threads and an IOdepth of 32.

The RAID array is now being stressed and not the CPU.