Fio 2.1.14 released

Fio 2.1.14 released

[Jens Axboe]

Hi,

I've released version 2.1.14 today. Here's a (rough) list of the 
changes, features, additions that were added/done since 2.1.13:

- Fio now supports arithmetic expressions in job files or on the
   command line. They have to be enclosed in double parentheses. This
   might be relaxed in the future, once it's matured a bit. This means
   you can do things like:

   bs=((4*1024))

   etc. The parser understands the regular +, -, /, *, as well as ^ for
   exponentiation, and % for modulus operations. Fio needs development
   packages of lex/flex and yacc/bison, it'll detect these at configure
   time.

- Some of the test tools are now installed properly when fio is
   installed. Generally they are prefixed with fio-, like fio-dedupe or
   fio-btrace2fio.

- The net IO engine grew some options to support setting socket buffer
   sizes and tcp max segments. It also supports counting dropped packets
   for UDP now.

- Various improvements and fixes to the RBD IO engine.

- The client/server functionally has seen some bug fixes and should be
   improved in stability.

- Fix for a stat related hang on exit, and for a stat related issue
   with periodical dumps through --status-interval.

- Support added for DragonFly BSD.

- The OS provided random functions have been removed. Fio defaulted
   to its own default random generator for years now, since it was
   easier to provide consistent workloads across operating systems
   that way. This means that the 'use_os_rand' option has now been
   deprecated.

- Fix a potential crash on fio exit with data verification enabled,
   which has existed since version 2.1.11.

- Fio will exit a bit faster now, since the disk util thread and the
   helper thread were combined into one, and it'll be immediately torn
   down on exit. It had to wait around 250msecs before this change. You
   might have noticed this delay when doing ctrl-c on a workload.

- Lots of small bug fixes.

-- 
Jens Axboe

Re: Fio 2.1.14 released

[Stephen Cameron]

Double parens?  If a single pair of parens is no longer sufficient,
that is news to me (git top of tree seems to take a single pair of
surrounding parens ok, and I didn't see a recent commit that looked
like a change to require double parens.)

Eg:

sleekness fio # grep 2k examples/latency-profile.fio
bs=(2k+2k)
sleekness fio # ./fio examples/latency-profile.fio
device: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.14-3-g9e311
Starting 1 process
^Cbs: 1 (f=1)
fio: terminating on signal 2

-- steve

On Tue, Nov 11, 2014 at 3:14 PM, Jens Axboe <axboe@kernel.dk> wrote:
> Hi,
>
> I've released version 2.1.14 today. Here's a (rough) list of the changes,
> features, additions that were added/done since 2.1.13:
>
> - Fio now supports arithmetic expressions in job files or on the
>   command line. They have to be enclosed in double parentheses. This
>   might be relaxed in the future, once it's matured a bit. This means
>   you can do things like:
>
>   bs=((4*1024))
>
>   etc. The parser understands the regular +, -, /, *, as well as ^ for
>   exponentiation, and % for modulus operations. Fio needs development
>   packages of lex/flex and yacc/bison, it'll detect these at configure
>   time.
>
> - Some of the test tools are now installed properly when fio is
>   installed. Generally they are prefixed with fio-, like fio-dedupe or
>   fio-btrace2fio.
>
> - The net IO engine grew some options to support setting socket buffer
>   sizes and tcp max segments. It also supports counting dropped packets
>   for UDP now.
>
> - Various improvements and fixes to the RBD IO engine.
>
> - The client/server functionally has seen some bug fixes and should be
>   improved in stability.
>
> - Fix for a stat related hang on exit, and for a stat related issue
>   with periodical dumps through --status-interval.
>
> - Support added for DragonFly BSD.
>
> - The OS provided random functions have been removed. Fio defaulted
>   to its own default random generator for years now, since it was
>   easier to provide consistent workloads across operating systems
>   that way. This means that the 'use_os_rand' option has now been
>   deprecated.
>
> - Fix a potential crash on fio exit with data verification enabled,
>   which has existed since version 2.1.11.
>
> - Fio will exit a bit faster now, since the disk util thread and the
>   helper thread were combined into one, and it'll be immediately torn
>   down on exit. It had to wait around 250msecs before this change. You
>   might have noticed this delay when doing ctrl-c on a workload.
>
> - Lots of small bug fixes.
>
> --
> Jens Axboe
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fio 2.1.14 released

[Jens Axboe]

Oops, I misremembered. But hey, double parentheses work too :-)

But Stephen is correct, a single one is all that is required.

BTW, forgot to mention, the shortlog of changes is here:

http://brick.kernel.dk/snaps/fio-2.1.14.shortlog.txt


On 2014-11-11 14:52, Stephen Cameron wrote:
> Double parens?  If a single pair of parens is no longer sufficient,
> that is news to me (git top of tree seems to take a single pair of
> surrounding parens ok, and I didn't see a recent commit that looked
> like a change to require double parens.)
>
> Eg:
>
> sleekness fio # grep 2k examples/latency-profile.fio
> bs=(2k+2k)
> sleekness fio # ./fio examples/latency-profile.fio
> device: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
> fio-2.1.14-3-g9e311
> Starting 1 process
> ^Cbs: 1 (f=1)
> fio: terminating on signal 2
>
> -- steve
>
> On Tue, Nov 11, 2014 at 3:14 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> Hi,
>>
>> I've released version 2.1.14 today. Here's a (rough) list of the changes,
>> features, additions that were added/done since 2.1.13:
>>
>> - Fio now supports arithmetic expressions in job files or on the
>>    command line. They have to be enclosed in double parentheses. This
>>    might be relaxed in the future, once it's matured a bit. This means
>>    you can do things like:
>>
>>    bs=((4*1024))
>>
>>    etc. The parser understands the regular +, -, /, *, as well as ^ for
>>    exponentiation, and % for modulus operations. Fio needs development
>>    packages of lex/flex and yacc/bison, it'll detect these at configure
>>    time.
>>
>> - Some of the test tools are now installed properly when fio is
>>    installed. Generally they are prefixed with fio-, like fio-dedupe or
>>    fio-btrace2fio.
>>
>> - The net IO engine grew some options to support setting socket buffer
>>    sizes and tcp max segments. It also supports counting dropped packets
>>    for UDP now.
>>
>> - Various improvements and fixes to the RBD IO engine.
>>
>> - The client/server functionally has seen some bug fixes and should be
>>    improved in stability.
>>
>> - Fix for a stat related hang on exit, and for a stat related issue
>>    with periodical dumps through --status-interval.
>>
>> - Support added for DragonFly BSD.
>>
>> - The OS provided random functions have been removed. Fio defaulted
>>    to its own default random generator for years now, since it was
>>    easier to provide consistent workloads across operating systems
>>    that way. This means that the 'use_os_rand' option has now been
>>    deprecated.
>>
>> - Fix a potential crash on fio exit with data verification enabled,
>>    which has existed since version 2.1.11.
>>
>> - Fio will exit a bit faster now, since the disk util thread and the
>>    helper thread were combined into one, and it'll be immediately torn
>>    down on exit. It had to wait around 250msecs before this change. You
>>    might have noticed this delay when doing ctrl-c on a workload.
>>
>> - Lots of small bug fixes.
>>
>> --
>> Jens Axboe
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Jens Axboe

Re: Fio 2.1.14 released

[Matthew Eaton]

> - Fio will exit a bit faster now, since the disk util thread and the
>   helper thread were combined into one, and it'll be immediately torn
>   down on exit. It had to wait around 250msecs before this change. You
>   might have noticed this delay when doing ctrl-c on a workload.

I find this update interesting, as I have a script that calls fio
multiple times, one after the other.

Are there any options that would make fio start or exit faster?  For
example, if I used disk_util=0, would this help fio exit any faster?

Re: Fio 2.1.14 released

[Jens Axboe]

On 2014-11-11 18:11, Matthew Eaton wrote:
>> - Fio will exit a bit faster now, since the disk util thread and the
>>    helper thread were combined into one, and it'll be immediately torn
>>    down on exit. It had to wait around 250msecs before this change. You
>>    might have noticed this delay when doing ctrl-c on a workload.
>
> I find this update interesting, as I have a script that calls fio
> multiple times, one after the other.
>
> Are there any options that would make fio start or exit faster?  For
> example, if I used disk_util=0, would this help fio exit any faster?

It should all be as fast as it could be before. The issue fixed above is 
when we wanted the disk util thread to exit, we set a done flag and just 
waited for it to wakeup as, which it did a few times per second. Now we 
use regular condvars, so it'll wake and exit immediately. So disk_util=0 
should not change more than a msec or so :-)

That said, if you do find things where it seems to start or stop slower 
than it should, let me know. But as far as I know, it's all properly 
driven now.

-- 
Jens Axboe

Re: Fio 2.1.14 released

[Matthew Eaton]

> It should all be as fast as it could be before. The issue fixed above is
> when we wanted the disk util thread to exit, we set a done flag and just
> waited for it to wakeup as, which it did a few times per second. Now we use
> regular condvars, so it'll wake and exit immediately. So disk_util=0 should
> not change more than a msec or so :-)
>
> That said, if you do find things where it seems to start or stop slower than
> it should, let me know. But as far as I know, it's all properly driven now.
>
> --
> Jens Axboe
>

Thanks!  Just to follow up, I wrote a test script looping 20
iterations of fio and tested 2.1.12 vs 2.1.14.

2.1.14 consistently finished ~2.7 seconds faster than 2.1.12, which I
think is great for only 20 iterations.

Just for kicks, I also tried setting disk_util=0, but could not
conclude that it made any difference.

Re: Fio 2.1.14 released

[Jens Axboe]

On 11/12/2014 02:36 PM, Matthew Eaton wrote:
>> It should all be as fast as it could be before. The issue fixed above is
>> when we wanted the disk util thread to exit, we set a done flag and just
>> waited for it to wakeup as, which it did a few times per second. Now we use
>> regular condvars, so it'll wake and exit immediately. So disk_util=0 should
>> not change more than a msec or so :-)
>>
>> That said, if you do find things where it seems to start or stop slower than
>> it should, let me know. But as far as I know, it's all properly driven now.
>>
>> --
>> Jens Axboe
>>
> 
> Thanks!  Just to follow up, I wrote a test script looping 20
> iterations of fio and tested 2.1.12 vs 2.1.14.
> 
> 2.1.14 consistently finished ~2.7 seconds faster than 2.1.12, which I
> think is great for only 20 iterations.

That sounds reasonable. That's 135msec per iteration, which is roughly
what we'd expect the average wait time to be (250msec / 2).



-- 
Jens Axboe