Potential Bug: Block size dropped in log files when running with log_avg_msec

Potential Bug: Block size dropped in log files when running with log_avg_msec

[Todd Lawall]

Hello,

I've been doing some test runs with fio using the write_bw_log, write_lat_log and write_iops_log options, and I've needed to use the log_avg_msec option to set my sample interval.�� I've noticed when running with log_avg_msec, the block size is dropped from the logs.� See examples of the BW log:

Without log_avg_msec:

13, 14833, 0, 32768
13, 14780, 1, 32768
13, 14780, 1, 32768
13, 15003, 0, 32768
13, 17702, 1, 32768
13, 17655, 1, 32768

With log_avg_msec:

500, 312320, 0, 0
1090, 343972, 0, 0
1090, 310784, 1, 0
2083, 329861, 0, 0
2083, 324756, 1, 0
3073, 285252, 0, 0
3073, 306290, 1, 0
4040, 320644, 0, 0

-----------------------------------------------------------
Now for the other details:

Environment: Linux Kubuntu 16.04.  Uname output:

����Linux <hostname> 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Fio version:

   fio-2.9-35-g5c8f

GCC version:

    $ gcc --version
    gcc (Ubuntu 5.3.1-14ubuntu2) 5.3.1 20160413

Job file (named fileTest.job):

    [global]
    ioengine=aio
    numjobs=1
    offset=0
    verify=crc32c-intel
    time_based
    runtime=120
    thread

    [1]
    blocksize=32k
    readwrite=rw
    verify=meta
    do_verify=0
    verify_pattern=0xAA55AA55
    iodepth=32
    filename=/tmp/fio.dat
    size=100M

Command line:

    #!/bin/bash
    /opt/build/fio/fio --write_iops_log $1 --write_bw_log $1 --write_lat_log $1 --output-format=json --status-interval=1 fileTest.job

Failure command line is the same as above, but add --log_avg_msec 500

Re: Potential Bug: Block size dropped in log files when running with log_avg_msec

[Jens Axboe]

On 05/12/2016 04:54 PM, Todd Lawall wrote:
> Hello,
>
> I've been doing some test runs with fio using the write_bw_log, write_lat_log and write_iops_log options, and I've needed to use the log_avg_msec option to set my sample interval.   I've noticed when running with log_avg_msec, the block size is dropped from the logs.  See examples of the BW log:
>
> Without log_avg_msec:
>
> 13, 14833, 0, 32768
> 13, 14780, 1, 32768
> 13, 14780, 1, 32768
> 13, 15003, 0, 32768
> 13, 17702, 1, 32768
> 13, 17655, 1, 32768
>
> With log_avg_msec:
>
> 500, 312320, 0, 0
> 1090, 343972, 0, 0
> 1090, 310784, 1, 0
> 2083, 329861, 0, 0
> 2083, 324756, 1, 0
> 3073, 285252, 0, 0
> 3073, 306290, 1, 0
> 4040, 320644, 0, 0

That is correct, fio will do that. I can add the block size, if it's 
consistent. The problem is if you are doing averaging for a multiple 
block size workload, then fio can't add it, as there would be no right 
answer. For a fixed size, it could though. I'll ensure that it does for 
that case.

-- 
Jens Axboe

Re: Potential Bug: Block size dropped in log files when running with log_avg_msec

[Jens Axboe]

On 05/16/2016 10:49 AM, Jens Axboe wrote:
> On 05/12/2016 04:54 PM, Todd Lawall wrote:
>> Hello,
>>
>> I've been doing some test runs with fio using the write_bw_log,
>> write_lat_log and write_iops_log options, and I've needed to use the
>> log_avg_msec option to set my sample interval.   I've noticed when
>> running with log_avg_msec, the block size is dropped from the logs.
>> See examples of the BW log:
>>
>> Without log_avg_msec:
>>
>> 13, 14833, 0, 32768
>> 13, 14780, 1, 32768
>> 13, 14780, 1, 32768
>> 13, 15003, 0, 32768
>> 13, 17702, 1, 32768
>> 13, 17655, 1, 32768
>>
>> With log_avg_msec:
>>
>> 500, 312320, 0, 0
>> 1090, 343972, 0, 0
>> 1090, 310784, 1, 0
>> 2083, 329861, 0, 0
>> 2083, 324756, 1, 0
>> 3073, 285252, 0, 0
>> 3073, 306290, 1, 0
>> 4040, 320644, 0, 0
>
> That is correct, fio will do that. I can add the block size, if it's
> consistent. The problem is if you are doing averaging for a multiple
> block size workload, then fio can't add it, as there would be no right
> answer. For a fixed size, it could though. I'll ensure that it does for
> that case.

http://git.kernel.dk/cgit/fio/commit/?id=66b98c9fe942e8fd417525ace39d0e0b93bcb9e9

This should fix it for you.

-- 
Jens Axboe