Re: Mixing command line and job file parameters

[Jens Axboe <axboe@kernel.dk>]

On 2014-04-04 21:24, Jens Axboe wrote:
> On Fri, Apr 04 2014, Carl Zwanzig wrote:
>> Hello,
>>
>> This has been bugging me for a while... it appears that we have a
>> choice of using either cmd line parameters xor a job file (but not
>> combining the two) to fully specify a job.
>>
>> For instance:
>> fio --runtime=20 derf.cfg
>> fio: time_based requires a runtime/timeout setting
>> file__0: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> fio-2.1.6.1
>> [...]
>> where derf.cfg starts otherwise fully specified the job, just doesn't
>> have the runtime.
>>
>> There are ways around this for some cases by using environment
>> variables (e.g. "RT=20 fio def.cfg" and runtime=${RT}) but those can
>> get clunky and only allows value substitutions, not additional
>> directives. In my current case, I'd like to have the config file
>> supply default values for certain parameters and override them as
>> needed from the command line (e.g. runtime=20 unless I say otherwise).
>> Right now, I already have 6 or 7 env vars passed in to the job file
>> and that's too many.
>>
>> Is there any way to do this without really hacking up the parsing
>> code? I'm currently looking at init.c, where it looks like the cmd
>> line is parsed first then the job file(s), but I lose the mental
>> thread when I get into parse_jobs_ini(). Ideally, any fio options on
>> the command line before a --name= would be stuck at the end of the
>> global thread data (*td_parent?), similar to including a directive
>> multiple times.
>
> Should be fairly easy to fix - right now the transition would imply a
> job barrier, which is why it doesn't work. We could just assume [global]
> state for that.

Took a quick look. The code basically looks like this:

job_files = parse_cmd_line(argc, argv, type);

if (job_files > 0) {
         for (i = 0; i < job_files; i++) {
                  if (fill_def_thread())
                          [...]

(init.c:2019)

So fio does parse the command line options, and unless they are 
sectioned and form a job, they basically go into the default thread 
options and hence are considered as part of the [global] section for the 
next job file. If you comment out that fill_def_thread() part, it should 
work.

It becomes more complicated if you expect this to work:

fio --some_option=1 jobfile.fio --some_other_option=2 jobfile2.fio

which I'd be surprised if that works.

Fio clears the default options between each job file, since they are 
considered independent, instead of leaking set [global] option between 
job files. That part should remain. But it would be nice to make any 
previous command line options be part of the next job file. The full fix 
will be larger than just commenting out those two lines, but it should 
not be a lot of work.

-- 
Jens Axboe