Re: Any way to detect performance in a test case?

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2694 bytes --]



On 2019/1/17 上午8:16, Dave Chinner wrote:
> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2019/1/16 上午11:57, Dave Chinner wrote:
>>> On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote:
>>>> Hi,
>>>>
>>>> Is there any way to detect (huge) performance regression in a test case?
>>>>
>>>> By huge performance regression, I mean some operation takes from less
>>>> than 10s to around 400s.
>>>>
>>>> There is existing runtime accounting, but we can't do it inside a test
>>>> case (or can we?)
>>>>
>>>> So is there any way to detect huge performance regression in a test case?
>>>
>>> Just run your normal performance monitoring tools while the test is
>>> running to see what has changed. Is it IO, memory, CPU, lock
>>> contention or somethign else that is the problem?  pcp, strace, top,
>>> iostat, perf, etc all work just fine for finding perf regressions
>>> reported by test cases...
>>
>> Sorry for the misunderstanding.
>>
>> I mean if it's possible for a test case to just fail when hitting some
>> big performance regression.
> 
> This is part of the reported information in $RESULT_BASE/check.time.
> 
> If you want to keep a history of runtimes for later comparison, then
> you just need to archive contents of that file with the test
> results.
> 
> OR, alternatively, generate an XML test report which reports the
> individual test runtime in each report:
> 
> .....
>         <testcase classname="xfstests.xfs" name="generic/036" time="12">
>         </testcase>
>         <testcase classname="xfstests.xfs" name="generic/112" time="5">
>         </testcase>
>         <testcase classname="xfstests.xfs" name="generic/113" time="4">
>         </testcase>
>         <testcase classname="xfstests.xfs" name="generic/114" time="1">
> .....
> 
> And then post-process these reports to determine runtime
> differences.
> 
>> E.g. one operation should finish in 30s, but when it takes over 300s,
>> it's definitely a big regression.
>>
>> But considering how many different hardware/VM the test may be run on,
>> I'm not really confident if this is possible.
> 
> You can really only determine performance regressions by comparing
> test runtime on kernels with the same features set run on the same
> hardware. Hence you'll need to keep archives from all your test
> machiens and configs and only compare between matching
> configurations.

Thanks, this matches my current understanding of how the testsuite works.

It looks like such regression detection can only be implemented outside
of fstests.

Thanks for all the suggestions!
Qu

> 
> Cheers,
> 
> Dave.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]