Re: Missing log entries and other questions

[Jens Axboe <axboe@kernel.dk>]

On 05/05/2016 01:36 PM, Mark Nelson wrote:
>
>
> On 05/04/2016 03:51 PM, Jens Axboe wrote:
>> On 05/04/2016 11:43 AM, Mark Nelson wrote:
>>> On 05/04/2016 11:34 AM, Jens Axboe wrote:
>>>> On 05/04/2016 08:10 AM, Jens Axboe wrote:
>>>>> On 04/27/2016 02:38 PM, Jens Axboe wrote:
>>>>>> I'll try and see if I can reproduce this.
>>>>>
>>>>> So I took a look at your logged output. The way that fio logs is
>>>>> that it
>>>>> does it on IO completion. You have set 1s intervals, so it checks,
>>>>> when
>>>>> an IO completes, if we're due for logging the previous interval IOPS:
>>>>>
>>>>> spent = mtime_since(&td->iops_sample_time, t);
>>>>> if (spent < td->o.iops_avg_time)
>>>>>          return;
>>>>>
>>>>> If you are doing lots of IO, it matches pretty evenly. But for slow
>>>>> workloads, it will end up being skewed a bit. It won't be too hard to
>>>>> fix the drift, though, but it still won't make it clock precise at
>>>>> exact
>>>>> 1s intervals.
>>>>>
>>>>> We could move this part into the main thread, and out of the workers.
>>>>> Then we could make it more precise. Might not be a bad idea in
>>>>> general,
>>>>> since it'll move some of this code out of the fast path.
>>>>>
>>>>> I'll take a stab at it.
>>>>
>>>> Can you try the below patch?
>>>
>>> I'll also try to take a look at this as we've seen a fair amount of skew
>>> in the logs doing fio librbd upstream ceph testing.
>>
>> That'd be great. More complete version below, fwiw.
>>
>>> I actually wrote a
>>> first stab at a parser to aggregate logs from multiple fio processes
>>> with the assumption of non-uniform sampling:
>>>
>>> https://github.com/ceph/cbt/blob/master/tools/fiologpaser.py
>>>
>>> The idea is you have say 8 fio bw logs in a directory you can do:
>>>
>>> fiologparser.py -f *bw*
>>>
>>> which will return average values from the logs over user defined
>>> intervals by taking weighted scores from samples that overlap.  You can
>>> also look at per-interval averages, sums, or just a global average.
>>>
>>> various option flags:
>>>
>>> -i <interval time in ms>
>>> -d <divide results by d (say if you want a different unit of output)
>>> -f <show full interval data>
>>> -a <show average interval data across logs>
>>> -s <show sums for interval data across logs>
>>
>> That looks useful. Might want to put that in the fio repo as well, if
>> you are interested.
>
> Sure, I'm game.  Might be a little buggy and it's definitely slow since
> I just threw it together the other day.  Do you have a place in the repo
> you like throwing parsing scripts?

Usually I put them in the tools/ directory.

-- 
Jens Axboe