Simulating bursts at different rates at irregular intervals?

Simulating bursts at different rates at irregular intervals?

[James Vanns]

Hi. I'm new to fio and though I've read the HOWTO from the git repo several times now, I'm unsure if it is possible to achieve what I want;

I can simulate a mix of sequential and random writes for a duration at an average rate or throughput of say, 5.4MB/s.
I presume that fio will attempt to sustain that rate? My real question is this - is it possible in either the same job
or a different job in the same file to simulate bursts of higher rates? I.e. on average attempt to write at nMB/s but
at random or irregular intervals during this period attempt a 'burst' of 138MB/s for 5 minutes and then return to the 'normal' rate. Then again at some other point a burst of 57MB/s for 30minutes etc.

Is this possible? Presently I've found that I cannot do this. Or rather, thinking about it, I'd have to specify n jobs 
for each different rate and then perhaps use 'ratecycle' and 'thinktime'? I can't see how to skew 'thinktime' randomly though?

Regards,

Jim Vanns

-- 
Jim Vanns
Senior Software Developer
Framestore

Re: Simulating bursts at different rates at irregular intervals?

[Jens Axboe]

On Thu, Jun 27 2013, James Vanns wrote:
> Hi. I'm new to fio and though I've read the HOWTO from the git repo several times now, I'm unsure if it is possible to achieve what I want;
> 
> I can simulate a mix of sequential and random writes for a duration at
> an average rate or throughput of say, 5.4MB/s.  I presume that fio
> will attempt to sustain that rate? My real question is this - is it
> possible in either the same job or a different job in the same file to
> simulate bursts of higher rates? I.e. on average attempt to write at
> nMB/s but at random or irregular intervals during this period attempt
> a 'burst' of 138MB/s for 5 minutes and then return to the 'normal'
> rate. Then again at some other point a burst of 57MB/s for 30minutes
> etc.
> 
> Is this possible? Presently I've found that I cannot do this. Or
> rather, thinking about it, I'd have to specify n jobs for each
> different rate and then perhaps use 'ratecycle' and 'thinktime'? I
> can't see how to skew 'thinktime' randomly though?

That's currently not possible, fio only supports a fixed rate. But if
you have a good idea on how to express your criteria as an option (or
options), then it can surely be added :-)

-- 
Jens Axboe

Re: Simulating bursts at different rates at irregular intervals?

[Roger Sibert]

Just a thought that came to mind.

To work around the problem would the following work?
Run fio to get you max MB/s
Write a script to generate you work/jobs file that has randomizes
length of time for the job to run and % of max for it to run at.

Thanks,
Roger

On Thu, Jun 27, 2013 at 5:02 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On Thu, Jun 27 2013, James Vanns wrote:
>> Hi. I'm new to fio and though I've read the HOWTO from the git repo several times now, I'm unsure if it is possible to achieve what I want;
>>
>> I can simulate a mix of sequential and random writes for a duration at
>> an average rate or throughput of say, 5.4MB/s.  I presume that fio
>> will attempt to sustain that rate? My real question is this - is it
>> possible in either the same job or a different job in the same file to
>> simulate bursts of higher rates? I.e. on average attempt to write at
>> nMB/s but at random or irregular intervals during this period attempt
>> a 'burst' of 138MB/s for 5 minutes and then return to the 'normal'
>> rate. Then again at some other point a burst of 57MB/s for 30minutes
>> etc.
>>
>> Is this possible? Presently I've found that I cannot do this. Or
>> rather, thinking about it, I'd have to specify n jobs for each
>> different rate and then perhaps use 'ratecycle' and 'thinktime'? I
>> can't see how to skew 'thinktime' randomly though?
>
> That's currently not possible, fio only supports a fixed rate. But if
> you have a good idea on how to express your criteria as an option (or
> options), then it can surely be added :-)
>
> --
> Jens Axboe
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Simulating bursts at different rates at irregular intervals?

[Jens Axboe]

On Thu, Jun 27 2013, Roger Sibert wrote:
> Just a thought that came to mind.
> 
> To work around the problem would the following work?
> Run fio to get you max MB/s
> Write a script to generate you work/jobs file that has randomizes
> length of time for the job to run and % of max for it to run at.

That's a bit of a bother, though, since you have to collect data and
output from each individual run. And not only that, shutdown/startup
will make things less than smooth. It could potentially dirty your
results.

We could have something ala:

rate_sequence={10s:20M},{20s:5M}

or whatever, which runs 10s at 20M/sec, then 20s at 5M/sec. Repeat.
Would be fairly easy to do.

-- 
Jens Axboe

Re: Simulating bursts at different rates at irregular intervals?

[James Vanns]

> That's a bit of a bother, though, since you have to collect data and
> output from each individual run. And not only that, shutdown/startup
> will make things less than smooth. It could potentially dirty your
> results.
> 
> We could have something ala:
> 
> rate_sequence={10s:20M},{20s:5M}
> 
> or whatever, which runs 10s at 20M/sec, then 20s at 5M/sec. Repeat.
> Would be fairly easy to do.

Yes, that would be rather useful I think and the parameter format 
is intuitive. Though perhaps another parameter to somehow skew or
randomise the length at which the burst rate runs for? And to also
modify the sequence itself? Perhaps something akin to the following;

; define a list  of rates that occasionally override the default in 'rate'
burst_rates={10s:20M},{20s:5M}
; rate_sequence could default to the order defined above
burst_rate_sequence=random
; intervals (in ms) - defaults to random
burst_rate_interval=2500

Just imagine a spiky chart of write (or read, for that matter) throughput over time ;)

I suppose if interval were to be random then there should be a minimum and maximum.

Regards,

Jim

> --
> Jens Axboe
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Jim Vanns
Senior Software Developer
Framestore