New tools and changes

New tools and changes

[Jens Axboe]

Hi,

I have committed a few new tools to the fio repo, and enabled the 
building of these tools (and the older ones) by default now. Previously, 
you'd have to do

$ make t/<toolname>

manually to build them, that is no longer the case. Anyway, the new 
tools are:

t/dedupe: this is a dedupe checker. You run this on a device or a file, 
and it will tell you how well the blocks on that device will dedupe. The 
block size defaults to 4KB, but can be set to anything you wish. By 
default this will use a bloom filter, which (for my testing) is within 
1% or better of the actually fully tracked check. You can do a full 
check with -B0 to turn off the bloom, however that can consume large 
amounts of memory on larger devices. If you are truly paranoid, there's 
a -c option as well to do full checking in case there are checksum 
collisions. The checksum used is md5. The output looks like this:

axboe@nelson:/home/axboe/git/fio $ sudo t/dedupe /dev/sda5
Will check </dev/sda5>, size <49999249408>, using 8 threads
Threads(8): 12206848 items processed
Extents=12206848, Unique extents=1625328
De-dupe ratio: 1:6.51
Fio setting: dedupe_percentage=87

dedupe will default to using as many threads as there are CPUs in the 
system, to speed things up. This is also settable. dedupe will also give 
you the fio setting to use to create a write workload with similar 
dedupability. The dedupe_percentage is a new option that allows you to 
set that.

t/btrace2fio: this is a tool that takes a blkparse binary dump and 
generates a fio job file matching that dump. This tools needs a bit more 
development effort, and I'd welcome people joining in and helping out. 
Right now it figures out the queue depth, read vs write mix, block size 
distribution, etc. It makes no attempt to collapse identical threads 
into one job with numjobs=x for readability, however.

I should write up some documentation for these tools, and will do so in 
the not-so-distant future. The dedupe tool probably needs a bit of error 
checking and handling and option range checking, but apart from that, 
it's mostly done. The btrace2fio tool is a bit more rough around the 
edges. I'd welcome feedback, suggestions, improvements for both, as usual.

-- 
Jens Axboe

Re: New tools and changes

[Jens Axboe]

Hi,

I forgot to mention another handy change that went in recently - fio now 
supports including files from job files. This should make it easier to 
manage a test harness of fio files, since you can have more manageable 
files with a global (or per-host, whatever) include fio that has options 
for that particular instance. It's pretty simple to use, here's the 
section that was added to the HOWTO to explain it:

-----------

When fio is utilized as a basis of any reasonably large test suite, it 
might be desirable to share a set of standardized settings across 
multiple job files. Instead of copy/pasting such settings, any section 
may pull in an external .fio file with 'include filename' directive, as 
in the following example:

; -- start job file including.fio --
[global]
filename=/tmp/test
filesize=1m
include glob-include.fio

[test]
rw=randread
bs=4k
time_based=1
runtime=10
include test-include.fio
; -- end job file including.fio --

; -- start job file glob-include.fio --
thread=1
group_reporting=1
; -- end job file glob-include.fio --

; -- start job file test-include.fio --
ioengine=libaio
iodepth=4
; -- end job file test-include.fio --

Settings pulled into a section apply to that section only (except global
section). Include directives may be nested in that any included file may
contain further include directive(s). Include files may not contain []
sections.

--------

This feature courtesy of Andrey Kuzmin <andrey.v.kuzmin@gmail.com> who 
did all the work to make this feasible.

-- 
Jens Axboe