Re: Missing log entries and other questions

[Mark Nelson <mark.a.nelson@gmail.com>]

On 05/06/2016 12:11 PM, Jens Axboe wrote:
> On 05/05/2016 01:36 PM, Mark Nelson wrote:
>>
>>
>> On 05/04/2016 03:51 PM, Jens Axboe wrote:
>>> On 05/04/2016 11:43 AM, Mark Nelson wrote:
>>>> On 05/04/2016 11:34 AM, Jens Axboe wrote:
>>>>> On 05/04/2016 08:10 AM, Jens Axboe wrote:
>>>>>> On 04/27/2016 02:38 PM, Jens Axboe wrote:
>>>>>>> I'll try and see if I can reproduce this.
>>>>>>
>>>>>> So I took a look at your logged output. The way that fio logs is
>>>>>> that it
>>>>>> does it on IO completion. You have set 1s intervals, so it checks,
>>>>>> when
>>>>>> an IO completes, if we're due for logging the previous interval IOPS:
>>>>>>
>>>>>> spent = mtime_since(&td->iops_sample_time, t);
>>>>>> if (spent < td->o.iops_avg_time)
>>>>>>          return;
>>>>>>
>>>>>> If you are doing lots of IO, it matches pretty evenly. But for slow
>>>>>> workloads, it will end up being skewed a bit. It won't be too hard to
>>>>>> fix the drift, though, but it still won't make it clock precise at
>>>>>> exact
>>>>>> 1s intervals.
>>>>>>
>>>>>> We could move this part into the main thread, and out of the workers.
>>>>>> Then we could make it more precise. Might not be a bad idea in
>>>>>> general,
>>>>>> since it'll move some of this code out of the fast path.
>>>>>>
>>>>>> I'll take a stab at it.
>>>>>
>>>>> Can you try the below patch?
>>>>
>>>> I'll also try to take a look at this as we've seen a fair amount of
>>>> skew
>>>> in the logs doing fio librbd upstream ceph testing.
>>>
>>> That'd be great. More complete version below, fwiw.
>>>
>>>> I actually wrote a
>>>> first stab at a parser to aggregate logs from multiple fio processes
>>>> with the assumption of non-uniform sampling:
>>>>
>>>> https://github.com/ceph/cbt/blob/master/tools/fiologpaser.py
>>>>
>>>> The idea is you have say 8 fio bw logs in a directory you can do:
>>>>
>>>> fiologparser.py -f *bw*
>>>>
>>>> which will return average values from the logs over user defined
>>>> intervals by taking weighted scores from samples that overlap.  You can
>>>> also look at per-interval averages, sums, or just a global average.
>>>>
>>>> various option flags:
>>>>
>>>> -i <interval time in ms>
>>>> -d <divide results by d (say if you want a different unit of output)
>>>> -f <show full interval data>
>>>> -a <show average interval data across logs>
>>>> -s <show sums for interval data across logs>
>>>
>>> That looks useful. Might want to put that in the fio repo as well, if
>>> you are interested.
>>
>> Sure, I'm game.  Might be a little buggy and it's definitely slow since
>> I just threw it together the other day.  Do you have a place in the repo
>> you like throwing parsing scripts?
>
> Usually I put them in the tools/ directory.

Done: https://github.com/axboe/fio/pull/170