computing percentiles from fio data

computing percentiles from fio data

[Karl Cronburg]

Hello,

In benchmarking ceph I've been using fio / fiologparser, and want to
get out the sort of stats & percentiles fiologparser currently gives
(min, avg, max, percentiles). However I'm concerned the data coming
out of fio is insufficient when I pass it the log_avg_msec argument.
Namely using the average of a possibly asymmetric sample distribution
(the set of I/O request samples over which fio is averaging when I
pass it this argument) will not give accurate percentiles.

Something like this argument is necessary though to keep the log files
a reasonable size. Would it be a good idea to push the sort of
statistics done in the log parser directly into fio? I'm considering
writing some code to compute the quantiles directly in fio, either
brute-force by maintaining a sorted list or implementing something
like the algorithm described here:

http://www.cs.rutgers.edu/~muthu/bquant.pdf

with some acceptable user-defined level of error given to fio when
asked to compute the percentiles on long-running / large data sets.

Is there any interest in having this added directly into fio? If so
where in the code should I be looking?

-Karl Cronburg-

Re: computing percentiles from fio data

[Mark Nelson]

On 06/06/2016 04:00 PM, Karl Cronburg wrote:
> Hello,
>
> In benchmarking ceph I've been using fio / fiologparser, and want to
> get out the sort of stats & percentiles fiologparser currently gives
> (min, avg, max, percentiles). However I'm concerned the data coming
> out of fio is insufficient when I pass it the log_avg_msec argument.
> Namely using the average of a possibly asymmetric sample distribution
> (the set of I/O request samples over which fio is averaging when I
> pass it this argument) will not give accurate percentiles.
>
> Something like this argument is necessary though to keep the log files
> a reasonable size. Would it be a good idea to push the sort of
> statistics done in the log parser directly into fio? I'm considering
> writing some code to compute the quantiles directly in fio, either
> brute-force by maintaining a sorted list or implementing something
> like the algorithm described here:
>
> http://www.cs.rutgers.edu/~muthu/bquant.pdf
>
> with some acceptable user-defined level of error given to fio when
> asked to compute the percentiles on long-running / large data sets.
>
> Is there any interest in having this added directly into fio? If so
> where in the code should I be looking?

I think it would be great!  I confess I haven't read through the paper 
yet (and may not be able to before I leave on holiday for 2 weeks).  The 
big thing will be having some kind of reasonable test case data to make 
sure the results are reasonable and an explanation of how it works. 
It's one of the things missing from fiologparser.py right now.

Potentially this could benefit a lot more folks than just those of us 
doing ceph testing, so I'd say go for it. :)

>
> -Karl Cronburg-
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: computing percentiles from fio data

[Jeff Furlong]

If you add some more logging or sorting options to fio, be sure that the extra overhead does not slow down fio in any way (decrease IOPs, increase latency, increase CPU utilization, etc.).  Having a post process step (within fio) or separate parser script (outside fio) would keep the overhead low when running fio measurements.

Regards,
Jeff


-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Monday, June 6, 2016 2:11 PM
To: Karl Cronburg <kcronbur@redhat.com>; fio@vger.kernel.org
Subject: Re: computing percentiles from fio data

On 06/06/2016 04:00 PM, Karl Cronburg wrote:
> Hello,
>
> In benchmarking ceph I've been using fio / fiologparser, and want to 
> get out the sort of stats & percentiles fiologparser currently gives 
> (min, avg, max, percentiles). However I'm concerned the data coming 
> out of fio is insufficient when I pass it the log_avg_msec argument.
> Namely using the average of a possibly asymmetric sample distribution 
> (the set of I/O request samples over which fio is averaging when I 
> pass it this argument) will not give accurate percentiles.
>
> Something like this argument is necessary though to keep the log files 
> a reasonable size. Would it be a good idea to push the sort of 
> statistics done in the log parser directly into fio? I'm considering 
> writing some code to compute the quantiles directly in fio, either 
> brute-force by maintaining a sorted list or implementing something 
> like the algorithm described here:
>
> http://www.cs.rutgers.edu/~muthu/bquant.pdf
>
> with some acceptable user-defined level of error given to fio when 
> asked to compute the percentiles on long-running / large data sets.
>
> Is there any interest in having this added directly into fio? If so 
> where in the code should I be looking?

I think it would be great!  I confess I haven't read through the paper yet (and may not be able to before I leave on holiday for 2 weeks).  The big thing will be having some kind of reasonable test case data to make sure the results are reasonable and an explanation of how it works. 
It's one of the things missing from fiologparser.py right now.

Potentially this could benefit a lot more folks than just those of us doing ceph testing, so I'd say go for it. :)

>
> -Karl Cronburg-
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in the 
> body of a message to majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.

Re: computing percentiles from fio data

[Karl Cronburg]

Mark: The main insight of the algorithm(s) in the paper is that one
can get logarithmic space and time (per operation) usage in the number
of operations, with input parameters of a set of desired percentiles
and corresponding error bounds (e.g. 99th percentile with a strict
error bound of 1% on the value).

Jeff: Right, will keep that in mind. Outputting slightly different
statistics than what log_avg_msec outputs could solve this without
introducing any more overhead than log_avg_msec. I'm just not sure
what to output to then make it feasible to have a post-processing step
which gives strict error bounds on the percentiles.

-Karl-

On Mon, Jun 6, 2016 at 7:31 PM, Jeff Furlong <jeff.furlong@hgst.com> wrote:
> If you add some more logging or sorting options to fio, be sure that the extra overhead does not slow down fio in any way (decrease IOPs, increase latency, increase CPU utilization, etc.).  Having a post process step (within fio) or separate parser script (outside fio) would keep the overhead low when running fio measurements.
>
> Regards,
> Jeff
>
>
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Mark Nelson
> Sent: Monday, June 6, 2016 2:11 PM
> To: Karl Cronburg <kcronbur@redhat.com>; fio@vger.kernel.org
> Subject: Re: computing percentiles from fio data
>
> On 06/06/2016 04:00 PM, Karl Cronburg wrote:
>> Hello,
>>
>> In benchmarking ceph I've been using fio / fiologparser, and want to
>> get out the sort of stats & percentiles fiologparser currently gives
>> (min, avg, max, percentiles). However I'm concerned the data coming
>> out of fio is insufficient when I pass it the log_avg_msec argument.
>> Namely using the average of a possibly asymmetric sample distribution
>> (the set of I/O request samples over which fio is averaging when I
>> pass it this argument) will not give accurate percentiles.
>>
>> Something like this argument is necessary though to keep the log files
>> a reasonable size. Would it be a good idea to push the sort of
>> statistics done in the log parser directly into fio? I'm considering
>> writing some code to compute the quantiles directly in fio, either
>> brute-force by maintaining a sorted list or implementing something
>> like the algorithm described here:
>>
>> http://www.cs.rutgers.edu/~muthu/bquant.pdf
>>
>> with some acceptable user-defined level of error given to fio when
>> asked to compute the percentiles on long-running / large data sets.
>>
>> Is there any interest in having this added directly into fio? If so
>> where in the code should I be looking?
>
> I think it would be great!  I confess I haven't read through the paper yet (and may not be able to before I leave on holiday for 2 weeks).  The big thing will be having some kind of reasonable test case data to make sure the results are reasonable and an explanation of how it works.
> It's one of the things missing from fiologparser.py right now.
>
> Potentially this could benefit a lot more folks than just those of us doing ceph testing, so I'd say go for it. :)
>
>>
>> -Karl Cronburg-
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in the
>> body of a message to majordomo@vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.

Re: computing percentiles from fio data

[Jens Axboe]

On 06/06/2016 03:00 PM, Karl Cronburg wrote:
> Hello,
>
> In benchmarking ceph I've been using fio / fiologparser, and want to
> get out the sort of stats & percentiles fiologparser currently gives
> (min, avg, max, percentiles). However I'm concerned the data coming
> out of fio is insufficient when I pass it the log_avg_msec argument.
> Namely using the average of a possibly asymmetric sample distribution
> (the set of I/O request samples over which fio is averaging when I
> pass it this argument) will not give accurate percentiles.

The normal stats like percentiles and min/max/avg etc values are not 
averaged, even if log_avg_msec is set. That's only true for the logging, 
if you specify any of the latency (or iops/bw) logging. The stats that 
fio outputs at the end of a run in the normal output is not averaged.

So which problem are you attacking? If you want to improve the logged 
values, then that could be useful. You want to look at 
stat.c:add_log_sample() for that code.

> Something like this argument is necessary though to keep the log files
> a reasonable size. Would it be a good idea to push the sort of
> statistics done in the log parser directly into fio? I'm considering
> writing some code to compute the quantiles directly in fio, either
> brute-force by maintaining a sorted list or implementing something
> like the algorithm described here:
>
> http://www.cs.rutgers.edu/~muthu/bquant.pdf
>
> with some acceptable user-defined level of error given to fio when
> asked to compute the percentiles on long-running / large data sets.

I still don't quite follow this... You already have the percentiles. If 
you re-compute them from a latency log with log_avg_msec set, then yes, 
it won't be completely accurate. But why not just use the percentiles 
directly?

> Is there any interest in having this added directly into fio? If so
> where in the code should I be looking?

It might be, if I know exactly what problem we are trying to solve here: :-)

-- 
Jens Axboe

Re: computing percentiles from fio data

[Karl Cronburg]

On Wed, Jun 8, 2016 at 1:10 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 06/06/2016 03:00 PM, Karl Cronburg wrote:
>>
>> Hello,
>>
>> In benchmarking ceph I've been using fio / fiologparser, and want to
>> get out the sort of stats & percentiles fiologparser currently gives
>> (min, avg, max, percentiles). However I'm concerned the data coming
>> out of fio is insufficient when I pass it the log_avg_msec argument.
>> Namely using the average of a possibly asymmetric sample distribution
>> (the set of I/O request samples over which fio is averaging when I
>> pass it this argument) will not give accurate percentiles.
>
>
> The normal stats like percentiles and min/max/avg etc values are not
> averaged, even if log_avg_msec is set. That's only true for the logging, if
> you specify any of the latency (or iops/bw) logging. The stats that fio
> outputs at the end of a run in the normal output is not averaged.
>
> So which problem are you attacking? If you want to improve the logged
> values, then that could be useful. You want to look at
> stat.c:add_log_sample() for that code.

I'm looking to:
1) Have a log file with min/avg/max and percentiles for each time interval,
2) Be able to (accurately) merge these statistics across threads, and
3) Massage the data into uniform time intervals

So basically what Mark has been trying to do in post-processing with
fiologparser, but directly in fio to both reduce logging overhead of fio
(because I would only need to output a log entry say every second)
and to leverage the finer granularity of the data.

I see you use a buckets / histogram method to maintain and subsequently
compute the percentiles at the end for each thread. Would solving (1) above
be a simple matter of querying this histogram over time?

>
>> Something like this argument is necessary though to keep the log files
>> a reasonable size. Would it be a good idea to push the sort of
>> statistics done in the log parser directly into fio? I'm considering
>> writing some code to compute the quantiles directly in fio, either
>> brute-force by maintaining a sorted list or implementing something
>> like the algorithm described here:
>>
>> http://www.cs.rutgers.edu/~muthu/bquant.pdf
>>
>> with some acceptable user-defined level of error given to fio when
>> asked to compute the percentiles on long-running / large data sets.
>
>
> I still don't quite follow this... You already have the percentiles. If you
> re-compute them from a latency log with log_avg_msec set, then yes, it won't
> be completely accurate. But why not just use the percentiles directly?
>
>> Is there any interest in having this added directly into fio? If so
>> where in the code should I be looking?
>
>
> It might be, if I know exactly what problem we are trying to solve here: :-)
>
> --
> Jens Axboe
>

Re: computing percentiles from fio data

[Jens Axboe]

On 06/08/2016 09:21 AM, Karl Cronburg wrote:
> On Wed, Jun 8, 2016 at 1:10 AM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 06/06/2016 03:00 PM, Karl Cronburg wrote:
>>>
>>> Hello,
>>>
>>> In benchmarking ceph I've been using fio / fiologparser, and want to
>>> get out the sort of stats & percentiles fiologparser currently gives
>>> (min, avg, max, percentiles). However I'm concerned the data coming
>>> out of fio is insufficient when I pass it the log_avg_msec argument.
>>> Namely using the average of a possibly asymmetric sample distribution
>>> (the set of I/O request samples over which fio is averaging when I
>>> pass it this argument) will not give accurate percentiles.
>>
>>
>> The normal stats like percentiles and min/max/avg etc values are not
>> averaged, even if log_avg_msec is set. That's only true for the logging, if
>> you specify any of the latency (or iops/bw) logging. The stats that fio
>> outputs at the end of a run in the normal output is not averaged.
>>
>> So which problem are you attacking? If you want to improve the logged
>> values, then that could be useful. You want to look at
>> stat.c:add_log_sample() for that code.
>
> I'm looking to:
> 1) Have a log file with min/avg/max and percentiles for each time interval,
> 2) Be able to (accurately) merge these statistics across threads, and
> 3) Massage the data into uniform time intervals
>
> So basically what Mark has been trying to do in post-processing with
> fiologparser, but directly in fio to both reduce logging overhead of fio
> (because I would only need to output a log entry say every second)
> and to leverage the finer granularity of the data.
>
> I see you use a buckets / histogram method to maintain and subsequently
> compute the percentiles at the end for each thread. Would solving (1) above
> be a simple matter of querying this histogram over time?

Right, you could solve it that way. Basically you would have two sets of 
states, one for the entire run (what we have now), and one that gets 
cleared for every log_avg_msec. That would solve #1 without needing to 
add any new algorithms.

Merging/summing the percentiles is trivial, so #2 is solvable too 
without much work.

-- 
Jens Axboe

Re: computing percentiles from fio data

[Karl Cronburg]

Jens,

Ultimately we are looking to make a visualization of latency as a
function of time (every couple seconds) with both per-node percentiles
as well as cluster-wide percentiles at scale (with 10's or 100's of
nodes) on fast storage devices (e.g. NVDIMM).

So by merging the percentiles, do you mean exporting the second set of
states (histogram), summing corresponding bins across nodes, and then
computing percentiles from that histogram? We've tried the even more
trivial method of taking a weighted (by iops) average of the
percentiles across nodes, but are in many cases getting wildly
inaccurate percentiles.

-Karl Cronburg-

On Wed, Jun 8, 2016 at 11:29 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 06/08/2016 09:21 AM, Karl Cronburg wrote:
>>
>> On Wed, Jun 8, 2016 at 1:10 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>> On 06/06/2016 03:00 PM, Karl Cronburg wrote:
>>>>
>>>>
>>>> Hello,
>>>>
>>>> In benchmarking ceph I've been using fio / fiologparser, and want to
>>>> get out the sort of stats & percentiles fiologparser currently gives
>>>> (min, avg, max, percentiles). However I'm concerned the data coming
>>>> out of fio is insufficient when I pass it the log_avg_msec argument.
>>>> Namely using the average of a possibly asymmetric sample distribution
>>>> (the set of I/O request samples over which fio is averaging when I
>>>> pass it this argument) will not give accurate percentiles.
>>>
>>>
>>>
>>> The normal stats like percentiles and min/max/avg etc values are not
>>> averaged, even if log_avg_msec is set. That's only true for the logging,
>>> if
>>> you specify any of the latency (or iops/bw) logging. The stats that fio
>>> outputs at the end of a run in the normal output is not averaged.
>>>
>>> So which problem are you attacking? If you want to improve the logged
>>> values, then that could be useful. You want to look at
>>> stat.c:add_log_sample() for that code.
>>
>>
>> I'm looking to:
>> 1) Have a log file with min/avg/max and percentiles for each time
>> interval,
>> 2) Be able to (accurately) merge these statistics across threads, and
>> 3) Massage the data into uniform time intervals
>>
>> So basically what Mark has been trying to do in post-processing with
>> fiologparser, but directly in fio to both reduce logging overhead of fio
>> (because I would only need to output a log entry say every second)
>> and to leverage the finer granularity of the data.
>>
>> I see you use a buckets / histogram method to maintain and subsequently
>> compute the percentiles at the end for each thread. Would solving (1)
>> above
>> be a simple matter of querying this histogram over time?
>
>
> Right, you could solve it that way. Basically you would have two sets of
> states, one for the entire run (what we have now), and one that gets cleared
> for every log_avg_msec. That would solve #1 without needing to add any new
> algorithms.
>
> Merging/summing the percentiles is trivial, so #2 is solvable too without
> much work.
>
> --
> Jens Axboe
>