REPORT: High CPU utilization since 2.2.5

REPORT: High CPU utilization since 2.2.5

[Kudryavtsev, Andrey O]

Hi Jens,
I�m running NVMe SSD benchmarks which are using multi-job configurations in the most common scenarios.
I have noticed that since 2.2.5 release FIO has 3-4 times higher CPU utilization on the same workload, same system, same kernel 3.18.
In fact I�m not able to achieve 450k IOPS on latest Core-i7 CPU due to 100% utilization.
I tracked down the changes back, 2.1.14 is the most stable for me. Release had 2.2.0 the reporting issues, which were fixed in 2.2.5 but introduced high CPU utilization instead.
Did anyone notice that too?

Configuration details are bellow:

[global]
name=4k random read 4 ios in the queue in 32 queues
filename=/dev/nvme0n1
ioengine=libaio
direct=1
bs=4k
rw=randread
iodepth=4
numjobs=32
buffered=0
size=100%
runtime=0
randrepeat=0
norandommap
refill_buffers
[job1]


--
Andrey Kudryavtsev,
SSD Solution Architect
Intel Corp.
inet: 83564353
work: +1-916-356-4353
mobile: +1-916-221-2281

Re: REPORT: High CPU utilization since 2.2.5

[Sitsofe Wheeler]

On 20 February 2015 at 23:10, Kudryavtsev, Andrey O
<andrey.o.kudryavtsev@intel.com> wrote:
> I’m running NVMe SSD benchmarks which are using multi-job configurations in the most common scenarios.
> I have noticed that since 2.2.5 release FIO has 3-4 times higher CPU utilization on the same workload, same system, same kernel 3.18.
> In fact I’m not able to achieve 450k IOPS on latest Core-i7 CPU due to 100% utilization.
> I tracked down the changes back, 2.1.14 is the most stable for me. Release had 2.2.0 the reporting issues, which were fixed in 2.2.5 but introduced high CPU utilization instead.
> Did anyone notice that too?

Where is the CPU usage going - userspace, device interrupts etc? If
it's vanishing into userspace does gtod_reduce=1 make any difference?
Does iostat confirm that you were getting more IOPs with the older
fio? What's the maximum number of requests reported by
/sys/block/<blockdevice>/queue/nr_requests ?

(PS buffered=0 and direct=1 are synonyms of each other and why runtime=0?)

-- 
Sitsofe | http://sucs.org/~sits/

Re: REPORT: High CPU utilization since 2.2.5

[Andrey Kuzmin]

On Sat, Feb 21, 2015 at 2:10 AM, Kudryavtsev, Andrey O
<andrey.o.kudryavtsev@intel.com> wrote:
> Hi Jens,
> I’m running NVMe SSD benchmarks which are using multi-job configurations in the most common scenarios.
> I have noticed that since 2.2.5 release FIO has 3-4 times higher CPU utilization on the same workload, same system, same kernel 3.18.

It would be definitely more informative if you have attached run
summary or, at least, the excerpt with fio CPU utilization report.

Regards,
Andrey

> In fact I’m not able to achieve 450k IOPS on latest Core-i7 CPU due to 100% utilization.
> I tracked down the changes back, 2.1.14 is the most stable for me. Release had 2.2.0 the reporting issues, which were fixed in 2.2.5 but introduced high CPU utilization instead.
> Did anyone notice that too?
>
> Configuration details are bellow:
>
> [global]
> name=4k random read 4 ios in the queue in 32 queues
> filename=/dev/nvme0n1
> ioengine=libaio
> direct=1
> bs=4k
> rw=randread
> iodepth=4
> numjobs=32
> buffered=0
> size=100%
> runtime=0
> randrepeat=0
> norandommap
> refill_buffers
> [job1]
>
>
> --
> Andrey Kudryavtsev,
> SSD Solution Architect
> Intel Corp.
> inet: 83564353
> work: +1-916-356-4353
> mobile: +1-916-221-2281
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: REPORT: High CPU utilization since 2.2.5

[Jens Axboe]

On 02/20/2015 04:10 PM, Kudryavtsev, Andrey O wrote:
> Hi Jens,
> I’m running NVMe SSD benchmarks which are using multi-job configurations in the most common scenarios.
> I have noticed that since 2.2.5 release FIO has 3-4 times higher CPU utilization on the same workload, same system, same kernel 3.18.
> In fact I’m not able to achieve 450k IOPS on latest Core-i7 CPU due to 100% utilization.
> I tracked down the changes back, 2.1.14 is the most stable for me. Release had 2.2.0 the reporting issues, which were fixed in 2.2.5 but introduced high CPU utilization instead.
> Did anyone notice that too?
> 
> Configuration details are bellow:
> 
> [global]
> name=4k random read 4 ios in the queue in 32 queues
> filename=/dev/nvme0n1
> ioengine=libaio
> direct=1
> bs=4k
> rw=randread
> iodepth=4
> numjobs=32
> buffered=0
> size=100%
> runtime=0
> randrepeat=0
> norandommap
> refill_buffers
> [job1]

Can you try and bisect the issue?

-- 
Jens Axboe

Re: REPORT: High CPU utilization since 2.2.5

[Kudryavtsev, Andrey O]

Thank you for the advise,
Utilization is coming from fio threads, reported I/O is similar to iostat.
Disabling all latency reporting works and returns the CPU utilization back
to normal. Unfortunately this workaround doesnпїЅt work for me, IпїЅm using
latency monitoring a lot for my tests.

I did some more research,
	With latency reporting enabled IпїЅm getпїЅing 350k IOPS instead of 450k.
	With gdot_disable=1 back to normal 450k
	With disable_slat=1 only it gets 420k
	disable_clat=1 doesnпїЅt make any difference and the performace is still
low 350k.

Please, advise. 
-- 
Andrey Kudryavtsev,

SSD Solution Architect
Intel Corp. 
inet: 83564353
work: +1-916-356-4353
mobile: +1-916-221-2281




On 2/21/15, 1:20 AM, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

>On 20 February 2015 at 23:10, Kudryavtsev, Andrey O
><andrey.o.kudryavtsev@intel.com> wrote:
>> IпїЅm running NVMe SSD benchmarks which are using multi-job
>>configurations in the most common scenarios.
>> I have noticed that since 2.2.5 release FIO has 3-4 times higher CPU
>>utilization on the same workload, same system, same kernel 3.18.
>> In fact IпїЅm not able to achieve 450k IOPS on latest Core-i7 CPU due to
>>100% utilization.
>> I tracked down the changes back, 2.1.14 is the most stable for me.
>>Release had 2.2.0 the reporting issues, which were fixed in 2.2.5 but
>>introduced high CPU utilization instead.
>> Did anyone notice that too?
>
>Where is the CPU usage going - userspace, device interrupts etc? If
>it's vanishing into userspace does gtod_reduce=1 make any difference?
>Does iostat confirm that you were getting more IOPs with the older
>fio? What's the maximum number of requests reported by
>/sys/block/<blockdevice>/queue/nr_requests ?
>
>(PS buffered=0 and direct=1 are synonyms of each other and why runtime=0?)
>
>-- 
>Sitsofe | http://sucs.org/~sits/

--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: REPORT: High CPU utilization since 2.2.5

[Jens Axboe]

On 03/09/2015 04:07 PM, Kudryavtsev, Andrey O wrote:
> Thank you for the advise,
> Utilization is coming from fio threads, reported I/O is similar to iostat.
> Disabling all latency reporting works and returns the CPU utilization back
> to normal. Unfortunately this workaround doesn’t work for me, I’m using
> latency monitoring a lot for my tests.
>
> I did some more research,
> 	With latency reporting enabled I’m getеing 350k IOPS instead of 450k.
> 	With gdot_disable=1 back to normal 450k
> 	With disable_slat=1 only it gets 420k
> 	disable_clat=1 doesn’t make any difference and the performace is still
> low 350k.
>
> Please, advise.

Please run a git bisect, as suggested. This will help pinpoint what 
commit made it worse for you. Should be pretty easy to do, since you can 
reproduce quickly and easily. You mentioned that 2.1.14 works, and that 
2.2.5 is broken. So do:

$ git bisect start
$ git bisect good fio-2.1.14
$ git bisect bad fio-2.2.5

Now for each iteration, do:

$ make clean && make -j20
$ ./fio <jobfile>

if the result is slow, do:

$ git bisect bad

and if the result is fast, do:

$ git bisect good

Then go back to the make clean && make and rerun your test case. Repeat 
this until you've bisected your way to the problematic commit, git will 
tell you when it's done.

-- 
Jens Axboe

Re: REPORT: High CPU utilization since 2.2.5

[Kudryavtsev, Andrey O]

Thank you, I’ll do that.

-- 
Andrey Kudryavtsev,
SSD Solution Architect
Intel Corp. 
inet: 83564353
work: +1-916-356-4353
mobile: +1-916-221-2281




On 3/17/15, 4:16 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 03/09/2015 04:07 PM, Kudryavtsev, Andrey O wrote:
>> Thank you for the advise,
>> Utilization is coming from fio threads, reported I/O is similar to
>>iostat.
>> Disabling all latency reporting works and returns the CPU utilization
>>back
>> to normal. Unfortunately this workaround doesn’t work for me, I’m using
>> latency monitoring a lot for my tests.
>>
>> I did some more research,
>> 	With latency reporting enabled I’m getеing 350k IOPS instead of 450k.
>> 	With gdot_disable=1 back to normal 450k
>> 	With disable_slat=1 only it gets 420k
>> 	disable_clat=1 doesn’t make any difference and the performace is still
>> low 350k.
>>
>> Please, advise.
>
>Please run a git bisect, as suggested. This will help pinpoint what
>commit made it worse for you. Should be pretty easy to do, since you can
>reproduce quickly and easily. You mentioned that 2.1.14 works, and that
>2.2.5 is broken. So do:
>
>$ git bisect start
>$ git bisect good fio-2.1.14
>$ git bisect bad fio-2.2.5
>
>Now for each iteration, do:
>
>$ make clean && make -j20
>$ ./fio <jobfile>
>
>if the result is slow, do:
>
>$ git bisect bad
>
>and if the result is fast, do:
>
>$ git bisect good
>
>Then go back to the make clean && make and rerun your test case. Repeat
>this until you've bisected your way to the problematic commit, git will
>tell you when it's done.
>
>-- 
>Jens Axboe
>