Mixing command line and job file parameters

Mixing command line and job file parameters

[Carl Zwanzig]

Hello,

This has been bugging me for a while... it appears that we have a choice of using either cmd line parameters xor a job file (but not combining the two) to fully specify a job. 
For instance:
fio --runtime=20 derf.cfg
fio: time_based requires a runtime/timeout setting
file__0: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.6.1
[...]
where derf.cfg starts otherwise fully specified the job, just doesn't have the runtime.

There are ways around this for some cases by using environment variables (e.g. "RT=20 fio def.cfg" and runtime=${RT}) but those can get clunky and only allows value substitutions, not additional directives. In my current case, I'd like to have the config file supply default values for certain parameters and override them as needed from the command line (e.g. runtime=20 unless I say otherwise). Right now, I already have 6 or 7 env vars passed in to the job file and that's too many. 

Is there any way to do this without really hacking up the parsing code? I'm currently looking at init.c, where it looks like the cmd line is parsed first then the job file(s), but I lose the mental thread when I get into parse_jobs_ini(). Ideally, any fio options on the command line before a --name= would be stuck at the end of the global thread data (*td_parent?), similar to including a directive multiple times.

Thanks,

z!

Re: Mixing command line and job file parameters

[Jens Axboe]

On Fri, Apr 04 2014, Carl Zwanzig wrote:
> Hello,
> 
> This has been bugging me for a while... it appears that we have a
> choice of using either cmd line parameters xor a job file (but not
> combining the two) to fully specify a job. 
>
> For instance:
> fio --runtime=20 derf.cfg
> fio: time_based requires a runtime/timeout setting
> file__0: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
> fio-2.1.6.1
> [...]
> where derf.cfg starts otherwise fully specified the job, just doesn't
> have the runtime.
> 
> There are ways around this for some cases by using environment
> variables (e.g. "RT=20 fio def.cfg" and runtime=${RT}) but those can
> get clunky and only allows value substitutions, not additional
> directives. In my current case, I'd like to have the config file
> supply default values for certain parameters and override them as
> needed from the command line (e.g. runtime=20 unless I say otherwise).
> Right now, I already have 6 or 7 env vars passed in to the job file
> and that's too many. 
> 
> Is there any way to do this without really hacking up the parsing
> code? I'm currently looking at init.c, where it looks like the cmd
> line is parsed first then the job file(s), but I lose the mental
> thread when I get into parse_jobs_ini(). Ideally, any fio options on
> the command line before a --name= would be stuck at the end of the
> global thread data (*td_parent?), similar to including a directive
> multiple times.

Should be fairly easy to fix - right now the transition would imply a
job barrier, which is why it doesn't work. We could just assume [global]
state for that.

-- 
Jens Axboe

Re: Mixing command line and job file parameters

[Jens Axboe]

On 2014-04-04 21:24, Jens Axboe wrote:
> On Fri, Apr 04 2014, Carl Zwanzig wrote:
>> Hello,
>>
>> This has been bugging me for a while... it appears that we have a
>> choice of using either cmd line parameters xor a job file (but not
>> combining the two) to fully specify a job.
>>
>> For instance:
>> fio --runtime=20 derf.cfg
>> fio: time_based requires a runtime/timeout setting
>> file__0: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> fio-2.1.6.1
>> [...]
>> where derf.cfg starts otherwise fully specified the job, just doesn't
>> have the runtime.
>>
>> There are ways around this for some cases by using environment
>> variables (e.g. "RT=20 fio def.cfg" and runtime=${RT}) but those can
>> get clunky and only allows value substitutions, not additional
>> directives. In my current case, I'd like to have the config file
>> supply default values for certain parameters and override them as
>> needed from the command line (e.g. runtime=20 unless I say otherwise).
>> Right now, I already have 6 or 7 env vars passed in to the job file
>> and that's too many.
>>
>> Is there any way to do this without really hacking up the parsing
>> code? I'm currently looking at init.c, where it looks like the cmd
>> line is parsed first then the job file(s), but I lose the mental
>> thread when I get into parse_jobs_ini(). Ideally, any fio options on
>> the command line before a --name= would be stuck at the end of the
>> global thread data (*td_parent?), similar to including a directive
>> multiple times.
>
> Should be fairly easy to fix - right now the transition would imply a
> job barrier, which is why it doesn't work. We could just assume [global]
> state for that.

Took a quick look. The code basically looks like this:

job_files = parse_cmd_line(argc, argv, type);

if (job_files > 0) {
         for (i = 0; i < job_files; i++) {
                  if (fill_def_thread())
                          [...]

(init.c:2019)

So fio does parse the command line options, and unless they are 
sectioned and form a job, they basically go into the default thread 
options and hence are considered as part of the [global] section for the 
next job file. If you comment out that fill_def_thread() part, it should 
work.

It becomes more complicated if you expect this to work:

fio --some_option=1 jobfile.fio --some_other_option=2 jobfile2.fio

which I'd be surprised if that works.

Fio clears the default options between each job file, since they are 
considered independent, instead of leaking set [global] option between 
job files. That part should remain. But it would be nice to make any 
previous command line options be part of the next job file. The full fix 
will be larger than just commenting out those two lines, but it should 
not be a lot of work.

-- 
Jens Axboe

Re: Mixing command line and job file parameters

[Jens Axboe]

On 2014-04-04 22:07, Jens Axboe wrote:
> On 2014-04-04 21:24, Jens Axboe wrote:
>> On Fri, Apr 04 2014, Carl Zwanzig wrote:
>>> Hello,
>>>
>>> This has been bugging me for a while... it appears that we have a
>>> choice of using either cmd line parameters xor a job file (but not
>>> combining the two) to fully specify a job.
>>>
>>> For instance:
>>> fio --runtime=20 derf.cfg
>>> fio: time_based requires a runtime/timeout setting
>>> file__0: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
>>> iodepth=32
>>> fio-2.1.6.1
>>> [...]
>>> where derf.cfg starts otherwise fully specified the job, just doesn't
>>> have the runtime.
>>>
>>> There are ways around this for some cases by using environment
>>> variables (e.g. "RT=20 fio def.cfg" and runtime=${RT}) but those can
>>> get clunky and only allows value substitutions, not additional
>>> directives. In my current case, I'd like to have the config file
>>> supply default values for certain parameters and override them as
>>> needed from the command line (e.g. runtime=20 unless I say otherwise).
>>> Right now, I already have 6 or 7 env vars passed in to the job file
>>> and that's too many.
>>>
>>> Is there any way to do this without really hacking up the parsing
>>> code? I'm currently looking at init.c, where it looks like the cmd
>>> line is parsed first then the job file(s), but I lose the mental
>>> thread when I get into parse_jobs_ini(). Ideally, any fio options on
>>> the command line before a --name= would be stuck at the end of the
>>> global thread data (*td_parent?), similar to including a directive
>>> multiple times.
>>
>> Should be fairly easy to fix - right now the transition would imply a
>> job barrier, which is why it doesn't work. We could just assume [global]
>> state for that.
>
> Took a quick look. The code basically looks like this:
>
> job_files = parse_cmd_line(argc, argv, type);
>
> if (job_files > 0) {
>          for (i = 0; i < job_files; i++) {
>                   if (fill_def_thread())
>                           [...]
>
> (init.c:2019)
>
> So fio does parse the command line options, and unless they are
> sectioned and form a job, they basically go into the default thread
> options and hence are considered as part of the [global] section for the
> next job file. If you comment out that fill_def_thread() part, it should
> work.
>
> It becomes more complicated if you expect this to work:
>
> fio --some_option=1 jobfile.fio --some_other_option=2 jobfile2.fio
>
> which I'd be surprised if that works.
>
> Fio clears the default options between each job file, since they are
> considered independent, instead of leaking set [global] option between
> job files. That part should remain. But it would be nice to make any
> previous command line options be part of the next job file. The full fix
> will be larger than just commenting out those two lines, but it should
> not be a lot of work.

Committed a minor fix that allows the basic usage, ala:

$ fio --option_foo=1 --option_bar=2 jobfile.fio

to work as expected. The two command line options will be considered 
part of the global section of jobfile.fio.

http://git.kernel.dk/?p=fio.git;a=commit;h=bc4f5ef67d26ef98f4822d5f798cb8c4e2d2fce5

-- 
Jens Axboe

RE: Mixing command line and job file parameters

[Carl Zwanzig]

Hi,

(Jens- Thanks for the speedy work on this.)

I just pulled from git and built (fio-2.1.7-29-gbc4f5). 

The patch will supply the parameter if it was missing from the config:

works:
./fio --runtime=10 derf.cfg  (derf.cfg has "time_based" but no "runtime")

will throws "no runtime" error:
./fio derf.cfg

but will not replace an existing parameter as would happen if it's twice in the cfg file.  Also tried forcing global "./fio --name=global --runtime=10 derf.cfg", but no luck, either.

I'll try poking around the area of the changes. (I have 7-8 parameters that I want to individually override, and using env vars has gotten ugly and needs a helper script.)

z!

Re: Mixing command line and job file parameters

[Jens Axboe]

On 04/07/2014 02:24 PM, Carl Zwanzig wrote:
> Hi,
>
> (Jens- Thanks for the speedy work on this.)
>
> I just pulled from git and built (fio-2.1.7-29-gbc4f5).
>
> The patch will supply the parameter if it was missing from the config:
>
> works:
> ./fio --runtime=10 derf.cfg  (derf.cfg has "time_based" but no "runtime")
>
> will throws "no runtime" error:
> ./fio derf.cfg
>
> but will not replace an existing parameter as would happen if it's twice in the cfg file.  Also tried forcing global "./fio --name=global --runtime=10 derf.cfg", but no luck, either.
>
> I'll try poking around the area of the changes. (I have 7-8 parameters that I want to individually override, and using env vars has gotten ugly and needs a helper script.)

Yep, it wont replace an existing parameter. It will basically work like 
the option appears before any of the others in the global section, 
similar to if you have:

timeout=10s
foo=1
bar=89
timeout=20s

the last timeout= will override the first one. To make that work would 
be more involved, I'm afraid.

-- 
Jens Axboe

Re: Mixing command line and job file parameters

[Jens Axboe]

On 04/07/2014 02:26 PM, Jens Axboe wrote:
> On 04/07/2014 02:24 PM, Carl Zwanzig wrote:
>> Hi,
>>
>> (Jens- Thanks for the speedy work on this.)
>>
>> I just pulled from git and built (fio-2.1.7-29-gbc4f5).
>>
>> The patch will supply the parameter if it was missing from the config:
>>
>> works:
>> ./fio --runtime=10 derf.cfg  (derf.cfg has "time_based" but no "runtime")
>>
>> will throws "no runtime" error:
>> ./fio derf.cfg
>>
>> but will not replace an existing parameter as would happen if it's
>> twice in the cfg file.  Also tried forcing global "./fio --name=global
>> --runtime=10 derf.cfg", but no luck, either.
>>
>> I'll try poking around the area of the changes. (I have 7-8 parameters
>> that I want to individually override, and using env vars has gotten
>> ugly and needs a helper script.)
>
> Yep, it wont replace an existing parameter. It will basically work like
> the option appears before any of the others in the global section,
> similar to if you have:
>
> timeout=10s
> foo=1
> bar=89
> timeout=20s
>
> the last timeout= will override the first one. To make that work would
> be more involved, I'm afraid.

So the way to make this would... Right now, the options associated with 
each thread/process looks like this:

struct thread_options {
	char *string_option;
	unsigned int uint_option;
	[...]
};

and so forth. We could change that to be more ala:

typedef struct fio_char_opt {
         char *option;
         bool was_set;
};

typedef struct fio_uint_opt {
         unsigned int option;
         bool was_set;
};

and so forth. When we parse an option, we'd set the ->was_set flag to 
true. I have been thinking about doing this before, since there are a 
few option hacks in fio where we deliberately add an option callback 
just to know if an option was set or not. This is done for options where 
checking for 0 isn't the valid check for "was set or not", and it'd be 
nice to get rid of those. Both because it would make the code cleaner, 
but also because the callback variants make life harder for the 
client/server job parsing.

With this infrastructure in place, it'd be easy enough to track and 
support overloading of options like the command line and job file mix.

-- 
Jens Axboe

Re: Mixing command line and job file parameters

[Alireza Haghdoost]

Would it be possible to combine mixed command line and job file
parameters with --client option ? Right now --client option sends only
job file to another server. Would be great if --client can also send
command line options to the remote servers like this:

#fio  --numjobs=1 --iodepth=128 --runtime=10 --ramp_time=10
--rw=randwrite  --client=host.list job_file

Alireza

On Mon, Apr 7, 2014 at 3:51 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 04/07/2014 02:26 PM, Jens Axboe wrote:
>>
>> On 04/07/2014 02:24 PM, Carl Zwanzig wrote:
>>>
>>> Hi,
>>>
>>> (Jens- Thanks for the speedy work on this.)
>>>
>>> I just pulled from git and built (fio-2.1.7-29-gbc4f5).
>>>
>>> The patch will supply the parameter if it was missing from the config:
>>>
>>> works:
>>> ./fio --runtime=10 derf.cfg  (derf.cfg has "time_based" but no "runtime")
>>>
>>> will throws "no runtime" error:
>>> ./fio derf.cfg
>>>
>>> but will not replace an existing parameter as would happen if it's
>>> twice in the cfg file.  Also tried forcing global "./fio --name=global
>>> --runtime=10 derf.cfg", but no luck, either.
>>>
>>> I'll try poking around the area of the changes. (I have 7-8 parameters
>>> that I want to individually override, and using env vars has gotten
>>> ugly and needs a helper script.)
>>
>>
>> Yep, it wont replace an existing parameter. It will basically work like
>> the option appears before any of the others in the global section,
>> similar to if you have:
>>
>> timeout=10s
>> foo=1
>> bar=89
>> timeout=20s
>>
>> the last timeout= will override the first one. To make that work would
>> be more involved, I'm afraid.
>
>
> So the way to make this would... Right now, the options associated with each
> thread/process looks like this:
>
> struct thread_options {
>         char *string_option;
>         unsigned int uint_option;
>         [...]
> };
>
> and so forth. We could change that to be more ala:
>
> typedef struct fio_char_opt {
>         char *option;
>         bool was_set;
> };
>
> typedef struct fio_uint_opt {
>         unsigned int option;
>         bool was_set;
> };
>
> and so forth. When we parse an option, we'd set the ->was_set flag to true.
> I have been thinking about doing this before, since there are a few option
> hacks in fio where we deliberately add an option callback just to know if an
> option was set or not. This is done for options where checking for 0 isn't
> the valid check for "was set or not", and it'd be nice to get rid of those.
> Both because it would make the code cleaner, but also because the callback
> variants make life harder for the client/server job parsing.
>
> With this infrastructure in place, it'd be easy enough to track and support
> overloading of options like the command line and job file mix.
>
>
> --
> Jens Axboe
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html