Fio 2.0 coming (was "Re: Recent changes")

Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

Hi,

As you may have noticed from the change email this morning, I pulled the
client-server branch into the master branch. The old master branch now
lives in stable-1.x, which is a maintenance branch for 1.60 and on. Only
bug fixes should go into that branch from now on, all new developments
should go into the 'master' (or 2.x) branch. For now the version is
still 1.99.6, I'll release 2.0 once it's seen a few weeks of testing.

Highlights of changes between stable-1.x and master:

- Basic support for client/server. I wrote about this in here before, I
  wont bore you with more details. See the included documentation for
  how to play with that, if you are interested.

- Completion latency percentiles is a great feature of 1.x, but I don't
  think a lot of people have used it. I've prettied up the output and
  turned it on by default.

- Terse output format has been modified, new additions are:

        Fio version
        IOPS
        Completion latency percentiles
        Disk utilization

  I've wanted to add IOPS and disk utilization for a while, now seems a
  good time to introduce a modified format and get all of them done in
  one sweep.

- Support has been added for write_iops_log. It's similar to
  write_bw_log, but writes the IOPS instead. For mixed block size
  workloads, it's not possible to deduct IOPS from the bandwidth.

- Network IO engine has support for local unix domain sockets as well,
  plus the format for configuring it has been modified to appropriately
  support that.

- Build is clang clean, and works fine with it (for some quick testing,
  clang 2.8 used here). There were in fact only 2-3 warnings prior.

Especially the client/server split-up has been an exercise in
refactoring lots of code into network friendly structures, and splitting
previous "show these results" code into "calculate these results" and
"display these results" parts instead.

Please give the current master branch (or the 1.99.x releases) a good
testing, it should work fine. I care more about generic testing just
moving to the 2.x/master branch than specific client/server
functionality testing, since that feature will be stabilized soonish and
regressions are the worst enemy of something like fio. As you perhaps
saw from the 'Recent changes' email, there's been a few changes since
1.60 in the current branch:

  62 files changed, 5162 insertions(+), 1395 deletions(-)

So we do need a few weeks to ensure that everything is fine before I tag
2.0 and release it.

Martin, can I talk you into taking a look at the man page to see whether
it's clean (it's not, I guarantee it)?

And folks running on non-linux, please try and build it at least (extra
brownie points for actually running it, too).

Thanks!

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Martin Steigerwald]

Sorry, forgot to cc fio mailing list.

Am Freitag, 14. Oktober 2011 schrieben Sie:
> So we do need a few weeks to ensure that everything is fine before I
> tag 2.0 and release it.
>
> Martin, can I talk you into taking a look at the man page to see
> whether it's clean (it's not, I guarantee it)?

I am quite occupied with holding trainings currently. Lets see what I can 
do next week during my Linux performance analysis & tuning training in 
Berlin.

After that I hold another training. So it might take a while.

> And folks running on non-linux, please try and build it at least (extra
> brownie points for actually running it, too).

PS: fio 1.59 has hit Debian unstable. Unlike the version on oss.teamix.org 
this one is also built with the new rdma engine. I don´t think I have the 
hardware available to test it, but you might give it a spin. I hope it 
will build with the two additional dependencies on all platforms.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Fio 2.0 coming (was "Re: Recent changes")

[Bruce Cran]

On 14 Oct 2011, at 07:20, Jens Axboe wrote:

> And folks running on non-linux, please try and build it at least (extra
> brownie points for actually running it, too).

It seems the latest version still has problems on non-Linux platforms when running in client/server mode:

> ./fio -S
fio: server listening on 0.0.0.0:8765

> ./fio -C localhost examples/tiobench-example
hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
>

-- 
Bruce Cran

Re: Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

On 2011-10-14 18:43, Martin Steigerwald wrote:
> Am Freitag, 14. Oktober 2011 schrieben Sie:
>> So we do need a few weeks to ensure that everything is fine before I
>> tag 2.0 and release it.
>>
>> Martin, can I talk you into taking a look at the man page to see
>> whether it's clean (it's not, I guarantee it)?
> 
> I am quite occupied with holding trainings currently. Lets see what I can 
> do next week during my Linux performance analysis & tuning training in 
> Berlin.
> 
> After that I hold another training. So it might take a while.

No rush, just whenever you have time to do it.

>> And folks running on non-linux, please try and build it at least (extra
>> brownie points for actually running it, too).
> 
> PS: fio 1.59 has hit Debian unstable. Unlike the version on oss.teamix.org 
> this one is also built with the new rdma engine. I don�t think I have the 
> hardware available to test it, but you might give it a spin. I hope it 
> will build with the two additional dependencies on all platforms.

Excellent!

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

On 2011-10-14 18:55, Bruce Cran wrote:
> 
> On 14 Oct 2011, at 07:20, Jens Axboe wrote:
> 
>> And folks running on non-linux, please try and build it at least (extra
>> brownie points for actually running it, too).
> 
> It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
> 
>> ./fio -S
> fio: server listening on 0.0.0.0:8765
> 
>> ./fio -C localhost examples/tiobench-example
> hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7

Can you try and add --debug=all before the -S and -C on both ends,
perhaps that'll give a clue as to what is going on. I'll see if I can
find a FreeBSD machine here for a test spin.

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

On 2011-10-15 10:08, Jens Axboe wrote:
> On 2011-10-14 18:55, Bruce Cran wrote:
>>
>> On 14 Oct 2011, at 07:20, Jens Axboe wrote:
>>
>>> And folks running on non-linux, please try and build it at least (extra
>>> brownie points for actually running it, too).
>>
>> It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
>>
>>> ./fio -S
>> fio: server listening on 0.0.0.0:8765
>>
>>> ./fio -C localhost examples/tiobench-example
>> hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
> 
> Can you try and add --debug=all before the -S and -C on both ends,
> perhaps that'll give a clue as to what is going on. I'll see if I can
> find a FreeBSD machine here for a test spin.

It's not OS specific, which did sound odd to me. Looks like the parser
is broken for some comment types when being passed a buffer instead of a
file.

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

On 2011-10-15 10:22, Jens Axboe wrote:
> On 2011-10-15 10:08, Jens Axboe wrote:
>> On 2011-10-14 18:55, Bruce Cran wrote:
>>>
>>> On 14 Oct 2011, at 07:20, Jens Axboe wrote:
>>>
>>>> And folks running on non-linux, please try and build it at least (extra
>>>> brownie points for actually running it, too).
>>>
>>> It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
>>>
>>>> ./fio -S
>>> fio: server listening on 0.0.0.0:8765
>>>
>>>> ./fio -C localhost examples/tiobench-example
>>> hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
>>
>> Can you try and add --debug=all before the -S and -C on both ends,
>> perhaps that'll give a clue as to what is going on. I'll see if I can
>> find a FreeBSD machine here for a test spin.
> 
> It's not OS specific, which did sound odd to me. Looks like the parser
> is broken for some comment types when being passed a buffer instead of a
> file.

Please try current -git, it should work now.

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Bruce Cran]

On 15 Oct 2011, at 13:44, Jens Axboe wrote:

> Please try current -git, it should work now.

Thanks, it's fixed.

I can crash the client by pressing ctrl-c on the server:

> ./fio -S
fio: server listening on 0.0.0.0:8765
 and ^C> 
^^^^ there seems to be a random "and" printed?

> ./fio -C localhost examples/tiobench-example

...

<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> 
Assertion failed: (cmdret->opcode == cmd.opcode), function fio_net_recv_cmd, file server.c, line 187.

Program received signal SIGABRT, Aborted.
[Switching to Thread 28301140 (LWP 100220/initial thread)]
0x281e7207 in kill () from /lib/libc.so.7
(gdb) bt
#0  0x281e7207 in kill () from /lib/libc.so.7
#1  0x280eb377 in raise () from /lib/libthr.so.3
#2  0x281e5d3a in abort () from /lib/libc.so.7
#3  0x281ce0c6 in __assert () from /lib/libc.so.7
#4  0x08074f8e in fio_net_recv_cmd (sk=3) at server.c:187
#5  0x08077f95 in fio_handle_clients () at client.c:779
#6  0x08051033 in exec_run () at fio.c:1779
#7  0x08051be1 in main (argc=4, argv=0xbfbfec04, envp=Error accessing memory address 0xb: Bad address.
) at fio.c:1904
(gdb) 

-- 
Bruce

Re: Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

On 2011-10-15 14:55, Bruce Cran wrote:
> 
> On 15 Oct 2011, at 13:44, Jens Axboe wrote:
> 
>> Please try current -git, it should work now.
> 
> Thanks, it's fixed.
> 
> I can crash the client by pressing ctrl-c on the server:
> 
>> ./fio -S
> fio: server listening on 0.0.0.0:8765
>  and ^C> 
> ^^^^ there seems to be a random "and" printed?
> 
>> ./fio -C localhost examples/tiobench-example
> 
> ...
> 
> <muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
> <muon> 
> Assertion failed: (cmdret->opcode == cmd.opcode), function fio_net_recv_cmd, file server.c, line 187.
> 
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 28301140 (LWP 100220/initial thread)]
> 0x281e7207 in kill () from /lib/libc.so.7
> (gdb) bt
> #0  0x281e7207 in kill () from /lib/libc.so.7
> #1  0x280eb377 in raise () from /lib/libthr.so.3
> #2  0x281e5d3a in abort () from /lib/libc.so.7
> #3  0x281ce0c6 in __assert () from /lib/libc.so.7
> #4  0x08074f8e in fio_net_recv_cmd (sk=3) at server.c:187
> #5  0x08077f95 in fio_handle_clients () at client.c:779
> #6  0x08051033 in exec_run () at fio.c:1779
> #7  0x08051be1 in main (argc=4, argv=0xbfbfec04, envp=Error accessing memory address 0xb: Bad address.
> ) at fio.c:1904
> (gdb) 

OK, I'll take a look at that one. You are a good tester :-)

The assert itself is a fragmented packet, where a later fragment does
not match the initial opcode. Must be due to a read error not being
handled correctly.

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Jens Axboe]

On 2011-10-15 15:59, Jens Axboe wrote:
> On 2011-10-15 14:55, Bruce Cran wrote:
>>
>> On 15 Oct 2011, at 13:44, Jens Axboe wrote:
>>
>>> Please try current -git, it should work now.
>>
>> Thanks, it's fixed.
>>
>> I can crash the client by pressing ctrl-c on the server:
>>
>>> ./fio -S
>> fio: server listening on 0.0.0.0:8765
>>  and ^C> 
>> ^^^^ there seems to be a random "and" printed?
>>
>>> ./fio -C localhost examples/tiobench-example
>>
>> ...
>>
>> <muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
>> <muon> 
>> Assertion failed: (cmdret->opcode == cmd.opcode), function fio_net_recv_cmd, file server.c, line 187.
>>
>> Program received signal SIGABRT, Aborted.
>> [Switching to Thread 28301140 (LWP 100220/initial thread)]
>> 0x281e7207 in kill () from /lib/libc.so.7
>> (gdb) bt
>> #0  0x281e7207 in kill () from /lib/libc.so.7
>> #1  0x280eb377 in raise () from /lib/libthr.so.3
>> #2  0x281e5d3a in abort () from /lib/libc.so.7
>> #3  0x281ce0c6 in __assert () from /lib/libc.so.7
>> #4  0x08074f8e in fio_net_recv_cmd (sk=3) at server.c:187
>> #5  0x08077f95 in fio_handle_clients () at client.c:779
>> #6  0x08051033 in exec_run () at fio.c:1779
>> #7  0x08051be1 in main (argc=4, argv=0xbfbfec04, envp=Error accessing memory address 0xb: Bad address.
>> ) at fio.c:1904
>> (gdb) 
> 
> OK, I'll take a look at that one. You are a good tester :-)
> 
> The assert itself is a fragmented packet, where a later fragment does
> not match the initial opcode. Must be due to a read error not being
> handled correctly.

Added attempt at failing naturally when that happens, does it reproduce
now?

-- 
Jens Axboe

Re: Fio 2.0 coming (was "Re: Recent changes")

[Bruce Cran]

On 15 Oct 2011, at 15:08, Jens Axboe wrote:

> Added attempt at failing naturally when that happens, does it reproduce
> now?

It doesn't crash now - it prints a warning and exits normally:

fio: fragment opcode mismatch (6 != 7)
client: host=localhost disconnected

-- 
Bruce Cran