Pull an external [global] section into an .fio file?

Pull an external [global] section into an .fio file?

[Andrey Kuzmin]

When dealing with a growing number of fio test scripts, copy/pasting
global section becomes error-prone and cumbersome. Is it possible (or
how much effort would it be to add such a feature) to pull a named set
of global settings into the .fio file?

Regards,
Andrey

Re: Pull an external [global] section into an .fio file?

[Kulkarni, Vasu]

A workaround is to use separate file with globals and then append just
before you invoke fio
#cat globals.txt >> io.txt | fio io.txt

On Mon, Jul 28, 2014 at 10:42 AM, Andrey Kuzmin
<andrey.v.kuzmin@gmail.com> wrote:
> When dealing with a growing number of fio test scripts, copy/pasting
> global section becomes error-prone and cumbersome. Is it possible (or
> how much effort would it be to add such a feature) to pull a named set
> of global settings into the .fio file?
>
> Regards,
> Andrey
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
A ship is safer in a harbor.  But that's not what ships are built for.

RE: Pull an external [global] section into an .fio file?

[Carl Zwanzig]

Hi,

For the most part, I've stopped using job files and use a script to write/execute a fio command line. I do end up with hideously-long command lines (>1100 chars), but can stash all the parameters in the script or included file and easily iterate over certain ones.

I start with things like this:
set FioCmd { \
/var/tmp/fio --create_serialize=0 --time_based  --norandommap --ioengine=solarisaio --minimal \
--randrepeat=0 --fill_device=1 --iodepth=32 --group_reporting --ramp_time=0 --unlink=0 --overwrite=1 \
--size=${SIZE} --rw=${DIR} --bs=${BS} --ba=${BS}  --runtime=${RUNTIME} }

and 

set FioJobs { \
--name=file__0  --filename=/${MNT}/${PN}/${HN}/fiofile.0  --name=file__1 --filename=/${MNT}/${PN}/${HN}/fiofile.1 \
--name=file__2  --filename=/${MNT}/${PN}/${HN}/fiofile.2  --name=file__3 --filename=/${MNT}/${PN}/${HN}/fiofile.3 \
--name=file__4  --filename=/${MNT}/${PN}/${HN}/fiofile.4  --name=file__5 --filename=/${MNT}/${PN}/${HN}/fiofile.5 \
--name=file__6  --filename=/${MNT}/${PN}/${HN}/fiofile.6  --name=file__7 --filename=/${MNT}/${PN}/${HN}/fiofile.7 \
--name=file__8  --filename=/${MNT}/${PN}/${HN}/fiofile.8  --name=file__9 --filename=/${MNT}/${PN}/${HN}/fiofile.9 \
--name=file__10 --filename=/${MNT}/${PN}/${HN}/fiofile.10 --name=file__11 --filename=/${MNT}/${PN}/${HN}/fiofile.11 }

do variable substitution, bolt on a "--output", and exec the entire string:

set cmdstring [ concat \
    exec -- ssh "${uname}@${thisclient}" [subst ${FioCmd}] \
    [subst $OPTS]  \
    --output ${outDataDir}/${filetag}-${thisclient}.fio \
    [subst $FioJobs] & ]
}

There's a bit more magic involved to run multiple instances at once.

(This, of course, didn't actually answer the OP's question.)

z!

Re: Pull an external [global] section into an .fio file?

[Christian Ehrhardt]

On 28/07/14 20:35, Carl Zwanzig wrote:
> Hi,
>
> For the most part, I've stopped using job files and use a script to write/execute a fio command line. I do end up with hideously-long command lines (>1100 chars), but can stash all the parameters in the script or included file and easily iterate over certain ones.
>

We went a similar way, but not that far to push all into the commandline.
The reason for that is that I liked the job file format, it was easy to 
grasp for everyone - and eventually I have jobfiles I can pass along if 
I identify a bug.

So what I ended up was having a few basic profiles (for totally 
different fio job behaviour) and filling a lot of things via environment 
variables that are set by a wrapper script.

In my opinion that gives you the flexibility to iterate like Carl 
suggested in his solution while keeping the majority of the jobfile 
semantic.

For example our global section often is only something like this:
[global]
bs=${FIO_BS}
[...]

And you can even add non-predefined options with a trick like this:
${FIO_FREEPARM1}
[...]

Just make the variable contain the full specification like "numjobs=4" 
and it will work. In case you don't use those I usually set 
description=n/a to all of them to avoid bugs due to unset variables.

I hope one of the suggestions helps you,
Christian

Re: Pull an external [global] section into an .fio file?

[Jens Axboe]

On 2014-07-28 19:42, Andrey Kuzmin wrote:
> When dealing with a growing number of fio test scripts, copy/pasting
> global section becomes error-prone and cumbersome. Is it possible (or
> how much effort would it be to add such a feature) to pull a named set
> of global settings into the .fio file?

Not currently possible, but I can see it being a useful feature. I'll 
add it to the list. Something ala

fio --shared globals.fio actual-job.fio

I added something similar back in April for command line jobs, where you 
can now do

fio --some-setting=1 --other-setting=2 actual-job.fio

and have that those settings be part of the global namespace.

-- 
Jens Axboe

Re: Pull an external [global] section into an .fio file?

[Andrey Kuzmin]

On Mon, Jul 28, 2014 at 11:47 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2014-07-28 19:42, Andrey Kuzmin wrote:
>>
>> When dealing with a growing number of fio test scripts, copy/pasting
>> global section becomes error-prone and cumbersome. Is it possible (or
>> how much effort would it be to add such a feature) to pull a named set
>> of global settings into the .fio file?
>
>
> Not currently possible, but I can see it being a useful feature. I'll add it
> to the list. Something ala
>
> fio --shared globals.fio actual-job.fio
>
> I added something similar back in April for command line jobs, where you can
> now do
>
> fio --some-setting=1 --other-setting=2 actual-job.fio
>
> and have that those settings be part of the global namespace.

My take would be to focus on the .fio file rather than on the
command-line, as job files lend itself very well to the management and
version control found in the test suite settings. Adding support for
the include filenames... directive in .fio files would make the most
sense to me in this respect.

Regards,
Andrey

>
> --
> Jens Axboe
>

Re: Pull an external [global] section into an .fio file?

[Jens Axboe]

On 2014-07-29 19:30, Andrey Kuzmin wrote:
> On Mon, Jul 28, 2014 at 11:47 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 2014-07-28 19:42, Andrey Kuzmin wrote:
>>>
>>> When dealing with a growing number of fio test scripts, copy/pasting
>>> global section becomes error-prone and cumbersome. Is it possible (or
>>> how much effort would it be to add such a feature) to pull a named set
>>> of global settings into the .fio file?
>>
>>
>> Not currently possible, but I can see it being a useful feature. I'll add it
>> to the list. Something ala
>>
>> fio --shared globals.fio actual-job.fio
>>
>> I added something similar back in April for command line jobs, where you can
>> now do
>>
>> fio --some-setting=1 --other-setting=2 actual-job.fio
>>
>> and have that those settings be part of the global namespace.
>
> My take would be to focus on the .fio file rather than on the
> command-line, as job files lend itself very well to the management and
> version control found in the test suite settings. Adding support for
> the include filenames... directive in .fio files would make the most
> sense to me in this respect.

Yes, an include directive would be useful. After writing the previous 
email I thought a bit more about it, and decided that the way to go 
would not be an extra --shared or --include, but rather just have that 
be the default behavior if a job file only had a global section. But we 
could still allow that, and add an include directive in the parser.

-- 
Jens Axboe