fio jobs time_based & runtime scope

fio jobs time_based & runtime scope

[Saeed]

If a job file contains x number of job declarations ([jobxx]) how does 
the 'time_based" and 'runtime' scoped?

Is it per job declaration or inclusive of all the job declarations in 
the file?  I'm trying to set a limit in run time to have the jobs run 
until a certain amount of time has been reached and the jobs still run 
over the 'runtime' defined limit

Re: fio jobs time_based & runtime scope

[Sitsofe Wheeler]

On 25 July 2016 at 18:00, Saeed <ionictea@gmail.com> wrote:
> If a job file contains x number of job declarations ([jobxx]) how does the
> 'time_based" and 'runtime' scoped?
>
> Is it per job declaration or inclusive of all the job declarations in the
> file?  I'm trying to set a limit in run time to have the jobs run until a
> certain amount of time has been reached and the jobs still run over the
> 'runtime' defined limit

Could you give a minimal job file that demonstrates the problem and
just to check are you using time_based too
(https://github.com/axboe/fio/blob/fio-2.13/HOWTO#L1200 )?

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio jobs time_based & runtime scope

[Saeed]

Here's the example:

[global]
bs=4k
ioengine=libaio
iodepth=32
size=1g
direct=1
time_based=1
runtime=300
filename=/dev/sdb
group_reporting
numjobs=1

# validation
do_verify=1
verify_fatal=1
verify_dump=1

verify_pattern=0x00000001

[sequential]
rw=readwrite
stonewall

[random]
rw=randrw
stonewall

So with this job file there are two job declarations (sequential & 
random).  Does the time_based & runtime requirement apply to the entire 
job as a whole or per job declaration?

On 2016-07-25 17:13:57 +0000, Sitsofe Wheeler said:

> On 25 July 2016 at 18:00, Saeed 
> <ionictea@gmail.com> wrote:
>> If a job file contains x number of job declarations ([jobxx]) how does the
>> 'time_based" and 'runtime' scoped?
>> 
>> Is it per job declaration or inclusive of all the job declarations in the
>> file?  I'm trying to set a limit in run time to have the jobs run until a
>> certain amount of time has been reached and the jobs still run over the
>> 'runtime' defined limit
> 
> Could you give a minimal job file that demonstrates the problem and
> just to check are you using time_based too
> (https://github.com/axboe/fio/blob/fio-2.13/HOWTO#L1200 )?

Re: fio jobs time_based & runtime scope

[Sitsofe Wheeler]

On 26 July 2016 at 00:15, Saeed <ionictea@gmail.com> wrote:
> Here's the example:
>
> [global]
> bs=4k
> ioengine=libaio
> iodepth=32
> size=1g
> direct=1
> time_based=1
> runtime=300
> filename=/dev/sdb
> group_reporting
> numjobs=1
>
> # validation
> do_verify=1
> verify_fatal=1
> verify_dump=1
>
> verify_pattern=0x00000001
>
> [sequential]
> rw=readwrite
> stonewall
>
> [random]
> rw=randrw
> stonewall
>
> So with this job file there are two job declarations (sequential & random).
> Does the time_based & runtime requirement apply to the entire job as a whole
> or per job declaration?

time_based and runtime are per job (like nearly all fio options) so
this job file above will take 600 seconds to complete. This is easy to
check with a smaller and simpler job file:

[global]
runtime=5
time_based
size=10M
stonewall
filename=/tmp/fiotmp
[job1]
[job2]

The stonewall forces the jobs to run one after the other and the
results will show long each job ran for. If necessary we can easily
use a program like time to see how long the total running time turned
out to be.

Remember that runtime applies to all parts of a job. If a job is
supposed to verify after it has finished writing but the runtime is
exceeded during the writing part then no verification will occur.

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio jobs time_based & runtime scope

[Saeed]

But what about in the case of a verify job where we want verification 
done (do_verify)?  Should we exclude the runtime & time_based settings 
from the validation/verification job?

http://www.coderplay.org/filesysdev/FIO-Data-Integrity-Test.html

"However, if the job file specifies to run based on time rather than 
total number of bytes (setting runtime=int and time_based), then 
do_verify() is not performed. "

On 2016-07-26 06:15:34 +0000, Sitsofe Wheeler said:

> On 26 July 2016 at 00:15, Saeed 
> <ionictea@gmail.com> wrote:
>> Here's the example:
>> 
>> [global]
>> bs=4k
>> ioengine=libaio
>> iodepth=32
>> size=1g
>> direct=1
>> time_based=1
>> runtime=300
>> filename=/dev/sdb
>> group_reporting
>> numjobs=1
>> 
>> # validation
>> do_verify=1
>> verify_fatal=1
>> verify_dump=1
>> 
>> verify_pattern=0x00000001
>> 
>> [sequential]
>> rw=readwrite
>> stonewall
>> 
>> [random]
>> rw=randrw
>> stonewall
>> 
>> So with this job file there are two job declarations (sequential & random).
>> Does the time_based & runtime requirement apply to the entire job as a whole
>> or per job declaration?
> 
> time_based and runtime are per job (like nearly all fio options) so
> this job file above will take 600 seconds to complete. This is easy to
> check with a smaller and simpler job file:
> 
> [global]
> runtime=5
> time_based
> size=10M
> stonewall
> filename=/tmp/fiotmp
> [job1]
> [job2]
> 
> The stonewall forces the jobs to run one after the other and the
> results will show long each job ran for. If necessary we can easily
> use a program like time to see how long the total running time turned
> out to be.
> 
> Remember that runtime applies to all parts of a job. If a job is
> supposed to verify after it has finished writing but the runtime is
> exceeded during the writing part then no verification will occur.

Re: fio jobs time_based & runtime scope

[Sitsofe Wheeler]

On 26 July 2016 at 17:53, Saeed <ionictea@gmail.com> wrote:
> But what about in the case of a verify job where we want verification done
> (do_verify)?  Should we exclude the runtime & time_based settings from the
> validation/verification job?

If I had to guarantee at least one write pass was completed and one
verification pass was also completed then yes I'd ensure that runtime
and time_based were not set on that write job. runtime/time_based is
there so you can guarantee a job will run for given period of time
(and not dramatically more or less time).

> http://www.coderplay.org/filesysdev/FIO-Data-Integrity-Test.html
>
> "However, if the job file specifies to run based on time rather than total
> number of bytes (setting runtime=int and time_based), then do_verify() is
> not performed. "

This is correct and matches my previous warning. If you are asking for
*both runtime and time_based* and the verification job is a write one
then it won't stop writing until runtime is exceeded and at that point
there can't be any time left to do the verify. What are you trying to
achieve by setting runtime/time_based?

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio jobs time_based & runtime scope

[Saeed]

Looking at this example 
https://github.com/axboe/fio/blob/master/examples/surface-scan.fio I 
noticed it didn't have any runtime and/or time_based bound settings so 
I wanted to confirm if its needed or not.


On 2016-07-26 18:11:47 +0000, Sitsofe Wheeler said:

> On 26 July 2016 at 17:53, Saeed 
> <ionictea@gmail.com> wrote:
>> But what about in the case of a verify job where we want verification done
>> (do_verify)?  Should we exclude the runtime & time_based settings from the
>> validation/verification job?
> 
> If I had to guarantee at least one write pass was completed and one
> verification pass was also completed then yes I'd ensure that runtime
> and time_based were not set on that write job. runtime/time_based is
> there so you can guarantee a job will run for given period of time
> (and not dramatically more or less time).
> 
>> http://www.coderplay.org/filesysdev/FIO-Data-Integrity-Test.html
>> 
>> "However, if the job file specifies to run based on time rather than total
>> number of bytes (setting runtime=int and time_based), then do_verify() is
>> not performed. "
> 
> This is correct and matches my previous warning. If you are asking for
> *both runtime and time_based* and the verification job is a write one
> then it won't stop writing until runtime is exceeded and at that point
> there can't be any time left to do the verify. What are you trying to
> achieve by setting runtime/time_based?