Any way to detect performance in a test case?

Any way to detect performance in a test case?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 367 bytes --]

Hi,

Is there any way to detect (huge) performance regression in a test case?

By huge performance regression, I mean some operation takes from less
than 10s to around 400s.

There is existing runtime accounting, but we can't do it inside a test
case (or can we?)

So is there any way to detect huge performance regression in a test case?

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any way to detect performance in a test case?

[Dave Chinner]

On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote:
> Hi,
> 
> Is there any way to detect (huge) performance regression in a test case?
> 
> By huge performance regression, I mean some operation takes from less
> than 10s to around 400s.
> 
> There is existing runtime accounting, but we can't do it inside a test
> case (or can we?)
> 
> So is there any way to detect huge performance regression in a test case?

Just run your normal performance monitoring tools while the test is
running to see what has changed. Is it IO, memory, CPU, lock
contention or somethign else that is the problem?  pcp, strace, top,
iostat, perf, etc all work just fine for finding perf regressions
reported by test cases...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Any way to detect performance in a test case?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1219 bytes --]



On 2019/1/16 上午11:57, Dave Chinner wrote:
> On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote:
>> Hi,
>>
>> Is there any way to detect (huge) performance regression in a test case?
>>
>> By huge performance regression, I mean some operation takes from less
>> than 10s to around 400s.
>>
>> There is existing runtime accounting, but we can't do it inside a test
>> case (or can we?)
>>
>> So is there any way to detect huge performance regression in a test case?
> 
> Just run your normal performance monitoring tools while the test is
> running to see what has changed. Is it IO, memory, CPU, lock
> contention or somethign else that is the problem?  pcp, strace, top,
> iostat, perf, etc all work just fine for finding perf regressions
> reported by test cases...

Sorry for the misunderstanding.

I mean if it's possible for a test case to just fail when hitting some
big performance regression.

E.g. one operation should finish in 30s, but when it takes over 300s,
it's definitely a big regression.

But considering how many different hardware/VM the test may be run on,
I'm not really confident if this is possible.

Thanks,
Qu

> 
> Cheers,
> 
> Dave.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any way to detect performance in a test case?

[Vijaychidambaram Velayudhan Pillai]

> I mean if it's possible for a test case to just fail when hitting some
> big performance regression.
>
> E.g. one operation should finish in 30s, but when it takes over 300s,
> it's definitely a big regression.
>
> But considering how many different hardware/VM the test may be run on,
> I'm not really confident if this is possible.

It is a lot of work, but one-setup-agnostic way of doing this is the following:
- Run test with some older version that is known to be correct/stable,
measure time
- Run test with latest version, measure time
- If newer version is 2x (or some configurable threshold) slower than
old version, warn developer

Since both old and new versions are being run on the same setup, we
don't have to worry about hardware or setup differences.

Re: Any way to detect performance in a test case?

[Dave Chinner]

On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
> 
> 
> On 2019/1/16 上午11:57, Dave Chinner wrote:
> > On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote:
> >> Hi,
> >>
> >> Is there any way to detect (huge) performance regression in a test case?
> >>
> >> By huge performance regression, I mean some operation takes from less
> >> than 10s to around 400s.
> >>
> >> There is existing runtime accounting, but we can't do it inside a test
> >> case (or can we?)
> >>
> >> So is there any way to detect huge performance regression in a test case?
> > 
> > Just run your normal performance monitoring tools while the test is
> > running to see what has changed. Is it IO, memory, CPU, lock
> > contention or somethign else that is the problem?  pcp, strace, top,
> > iostat, perf, etc all work just fine for finding perf regressions
> > reported by test cases...
> 
> Sorry for the misunderstanding.
> 
> I mean if it's possible for a test case to just fail when hitting some
> big performance regression.

This is part of the reported information in $RESULT_BASE/check.time.

If you want to keep a history of runtimes for later comparison, then
you just need to archive contents of that file with the test
results.

OR, alternatively, generate an XML test report which reports the
individual test runtime in each report:

.....
        <testcase classname="xfstests.xfs" name="generic/036" time="12">
        </testcase>
        <testcase classname="xfstests.xfs" name="generic/112" time="5">
        </testcase>
        <testcase classname="xfstests.xfs" name="generic/113" time="4">
        </testcase>
        <testcase classname="xfstests.xfs" name="generic/114" time="1">
.....

And then post-process these reports to determine runtime
differences.

> E.g. one operation should finish in 30s, but when it takes over 300s,
> it's definitely a big regression.
> 
> But considering how many different hardware/VM the test may be run on,
> I'm not really confident if this is possible.

You can really only determine performance regressions by comparing
test runtime on kernels with the same features set run on the same
hardware. Hence you'll need to keep archives from all your test
machiens and configs and only compare between matching
configurations.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Any way to detect performance in a test case?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2694 bytes --]



On 2019/1/17 上午8:16, Dave Chinner wrote:
> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2019/1/16 上午11:57, Dave Chinner wrote:
>>> On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote:
>>>> Hi,
>>>>
>>>> Is there any way to detect (huge) performance regression in a test case?
>>>>
>>>> By huge performance regression, I mean some operation takes from less
>>>> than 10s to around 400s.
>>>>
>>>> There is existing runtime accounting, but we can't do it inside a test
>>>> case (or can we?)
>>>>
>>>> So is there any way to detect huge performance regression in a test case?
>>>
>>> Just run your normal performance monitoring tools while the test is
>>> running to see what has changed. Is it IO, memory, CPU, lock
>>> contention or somethign else that is the problem?  pcp, strace, top,
>>> iostat, perf, etc all work just fine for finding perf regressions
>>> reported by test cases...
>>
>> Sorry for the misunderstanding.
>>
>> I mean if it's possible for a test case to just fail when hitting some
>> big performance regression.
> 
> This is part of the reported information in $RESULT_BASE/check.time.
> 
> If you want to keep a history of runtimes for later comparison, then
> you just need to archive contents of that file with the test
> results.
> 
> OR, alternatively, generate an XML test report which reports the
> individual test runtime in each report:
> 
> .....
>         <testcase classname="xfstests.xfs" name="generic/036" time="12">
>         </testcase>
>         <testcase classname="xfstests.xfs" name="generic/112" time="5">
>         </testcase>
>         <testcase classname="xfstests.xfs" name="generic/113" time="4">
>         </testcase>
>         <testcase classname="xfstests.xfs" name="generic/114" time="1">
> .....
> 
> And then post-process these reports to determine runtime
> differences.
> 
>> E.g. one operation should finish in 30s, but when it takes over 300s,
>> it's definitely a big regression.
>>
>> But considering how many different hardware/VM the test may be run on,
>> I'm not really confident if this is possible.
> 
> You can really only determine performance regressions by comparing
> test runtime on kernels with the same features set run on the same
> hardware. Hence you'll need to keep archives from all your test
> machiens and configs and only compare between matching
> configurations.

Thanks, this matches my current understanding of how the testsuite works.

It looks like such regression detection can only be implemented outside
of fstests.

Thanks for all the suggestions!
Qu

> 
> Cheers,
> 
> Dave.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any way to detect performance in a test case?

[Dave Chinner]

On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote:
> On 2019/1/17 上午8:16, Dave Chinner wrote:
> > On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
> >> E.g. one operation should finish in 30s, but when it takes over 300s,
> >> it's definitely a big regression.
> >>
> >> But considering how many different hardware/VM the test may be run on,
> >> I'm not really confident if this is possible.
> > 
> > You can really only determine performance regressions by comparing
> > test runtime on kernels with the same features set run on the same
> > hardware. Hence you'll need to keep archives from all your test
> > machiens and configs and only compare between matching
> > configurations.
> 
> Thanks, this matches my current understanding of how the testsuite works.
> 
> It looks like such regression detection can only be implemented outside
> of fstests.

That's pretty much by design. Analysis of multiple test run results
and post-processing them is really not something that the test
harness does. The test harness really just runs the tests and
records the results....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Any way to detect performance in a test case?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1709 bytes --]



On 2019/1/17 上午10:25, Dave Chinner wrote:
> On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote:
>> On 2019/1/17 上午8:16, Dave Chinner wrote:
>>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
>>>> E.g. one operation should finish in 30s, but when it takes over 300s,
>>>> it's definitely a big regression.
>>>>
>>>> But considering how many different hardware/VM the test may be run on,
>>>> I'm not really confident if this is possible.
>>>
>>> You can really only determine performance regressions by comparing
>>> test runtime on kernels with the same features set run on the same
>>> hardware. Hence you'll need to keep archives from all your test
>>> machiens and configs and only compare between matching
>>> configurations.
>>
>> Thanks, this matches my current understanding of how the testsuite works.
>>
>> It looks like such regression detection can only be implemented outside
>> of fstests.
> 
> That's pretty much by design. Analysis of multiple test run results
> and post-processing them is really not something that the test
> harness does. The test harness really just runs the tests and
> records the results....

What about using some other telemetry other than time to determine
regreesion?

In my particular case, the correct behavior, some reading like
generation would only increase by a somewhat predictable number.

While when the regression happens, the generation will go way higher
than expectation.

Is it acceptable to craft a test case using such measurement?

As I really hope to get an easy PASS/FAIL test case for QA guys to
notice such huge regression.

Thanks,
Qu

> 
> Cheers,
> 
> Dave.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any way to detect performance in a test case?

[Dave Chinner]

On Wed, Jan 23, 2019 at 08:51:03AM +0800, Qu Wenruo wrote:
> 
> 
> On 2019/1/17 上午10:25, Dave Chinner wrote:
> > On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote:
> >> On 2019/1/17 上午8:16, Dave Chinner wrote:
> >>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
> >>>> E.g. one operation should finish in 30s, but when it takes over 300s,
> >>>> it's definitely a big regression.
> >>>>
> >>>> But considering how many different hardware/VM the test may be run on,
> >>>> I'm not really confident if this is possible.
> >>>
> >>> You can really only determine performance regressions by comparing
> >>> test runtime on kernels with the same features set run on the same
> >>> hardware. Hence you'll need to keep archives from all your test
> >>> machiens and configs and only compare between matching
> >>> configurations.
> >>
> >> Thanks, this matches my current understanding of how the testsuite works.
> >>
> >> It looks like such regression detection can only be implemented outside
> >> of fstests.
> > 
> > That's pretty much by design. Analysis of multiple test run results
> > and post-processing them is really not something that the test
> > harness does. The test harness really just runs the tests and
> > records the results....
> 
> What about using some other telemetry other than time to determine
> regreesion?
> 
> In my particular case, the correct behavior, some reading like
> generation would only increase by a somewhat predictable number.
>
> While when the regression happens, the generation will go way higher
> than expectation.

That's something that would be done inside the test, right? i.e.
this has nothing to do with the test harness itself, but is a
failure criteria for the specific test?

> Is it acceptable to craft a test case using such measurement?

If it's reliable and not prone to false positives from future code
changes, yes.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Any way to detect performance in a test case?

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2149 bytes --]



On 2019/1/23 下午12:18, Dave Chinner wrote:
> On Wed, Jan 23, 2019 at 08:51:03AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2019/1/17 上午10:25, Dave Chinner wrote:
>>> On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote:
>>>> On 2019/1/17 上午8:16, Dave Chinner wrote:
>>>>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
>>>>>> E.g. one operation should finish in 30s, but when it takes over 300s,
>>>>>> it's definitely a big regression.
>>>>>>
>>>>>> But considering how many different hardware/VM the test may be run on,
>>>>>> I'm not really confident if this is possible.
>>>>>
>>>>> You can really only determine performance regressions by comparing
>>>>> test runtime on kernels with the same features set run on the same
>>>>> hardware. Hence you'll need to keep archives from all your test
>>>>> machiens and configs and only compare between matching
>>>>> configurations.
>>>>
>>>> Thanks, this matches my current understanding of how the testsuite works.
>>>>
>>>> It looks like such regression detection can only be implemented outside
>>>> of fstests.
>>>
>>> That's pretty much by design. Analysis of multiple test run results
>>> and post-processing them is really not something that the test
>>> harness does. The test harness really just runs the tests and
>>> records the results....
>>
>> What about using some other telemetry other than time to determine
>> regreesion?
>>
>> In my particular case, the correct behavior, some reading like
>> generation would only increase by a somewhat predictable number.
>>
>> While when the regression happens, the generation will go way higher
>> than expectation.
> 
> That's something that would be done inside the test, right? i.e.
> this has nothing to do with the test harness itself, but is a
> failure criteria for the specific test?

Yep, a failure criteria for that specific test.

> 
>> Is it acceptable to craft a test case using such measurement?
> 
> If it's reliable and not prone to false positives from future code
> changes, yes.

Glad to know that.

Thanks,
Qu
> 
> Cheers,
> 
> Dave.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]