* Any way to detect performance in a test case? @ 2019-01-16 1:59 Qu Wenruo 2019-01-16 3:57 ` Dave Chinner 0 siblings, 1 reply; 10+ messages in thread From: Qu Wenruo @ 2019-01-16 1:59 UTC (permalink / raw) To: fstests [-- Attachment #1.1: Type: text/plain, Size: 367 bytes --] Hi, Is there any way to detect (huge) performance regression in a test case? By huge performance regression, I mean some operation takes from less than 10s to around 400s. There is existing runtime accounting, but we can't do it inside a test case (or can we?) So is there any way to detect huge performance regression in a test case? Thanks, Qu [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-16 1:59 Any way to detect performance in a test case? Qu Wenruo @ 2019-01-16 3:57 ` Dave Chinner 2019-01-16 4:47 ` Qu Wenruo 0 siblings, 1 reply; 10+ messages in thread From: Dave Chinner @ 2019-01-16 3:57 UTC (permalink / raw) To: Qu Wenruo; +Cc: fstests On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote: > Hi, > > Is there any way to detect (huge) performance regression in a test case? > > By huge performance regression, I mean some operation takes from less > than 10s to around 400s. > > There is existing runtime accounting, but we can't do it inside a test > case (or can we?) > > So is there any way to detect huge performance regression in a test case? Just run your normal performance monitoring tools while the test is running to see what has changed. Is it IO, memory, CPU, lock contention or somethign else that is the problem? pcp, strace, top, iostat, perf, etc all work just fine for finding perf regressions reported by test cases... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-16 3:57 ` Dave Chinner @ 2019-01-16 4:47 ` Qu Wenruo 2019-01-16 17:33 ` Vijaychidambaram Velayudhan Pillai 2019-01-17 0:16 ` Dave Chinner 0 siblings, 2 replies; 10+ messages in thread From: Qu Wenruo @ 2019-01-16 4:47 UTC (permalink / raw) To: Dave Chinner; +Cc: fstests [-- Attachment #1.1: Type: text/plain, Size: 1219 bytes --] On 2019/1/16 上午11:57, Dave Chinner wrote: > On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote: >> Hi, >> >> Is there any way to detect (huge) performance regression in a test case? >> >> By huge performance regression, I mean some operation takes from less >> than 10s to around 400s. >> >> There is existing runtime accounting, but we can't do it inside a test >> case (or can we?) >> >> So is there any way to detect huge performance regression in a test case? > > Just run your normal performance monitoring tools while the test is > running to see what has changed. Is it IO, memory, CPU, lock > contention or somethign else that is the problem? pcp, strace, top, > iostat, perf, etc all work just fine for finding perf regressions > reported by test cases... Sorry for the misunderstanding. I mean if it's possible for a test case to just fail when hitting some big performance regression. E.g. one operation should finish in 30s, but when it takes over 300s, it's definitely a big regression. But considering how many different hardware/VM the test may be run on, I'm not really confident if this is possible. Thanks, Qu > > Cheers, > > Dave. > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-16 4:47 ` Qu Wenruo @ 2019-01-16 17:33 ` Vijaychidambaram Velayudhan Pillai 2019-01-17 0:16 ` Dave Chinner 1 sibling, 0 replies; 10+ messages in thread From: Vijaychidambaram Velayudhan Pillai @ 2019-01-16 17:33 UTC (permalink / raw) To: Qu Wenruo; +Cc: Dave Chinner, fstests > I mean if it's possible for a test case to just fail when hitting some > big performance regression. > > E.g. one operation should finish in 30s, but when it takes over 300s, > it's definitely a big regression. > > But considering how many different hardware/VM the test may be run on, > I'm not really confident if this is possible. It is a lot of work, but one-setup-agnostic way of doing this is the following: - Run test with some older version that is known to be correct/stable, measure time - Run test with latest version, measure time - If newer version is 2x (or some configurable threshold) slower than old version, warn developer Since both old and new versions are being run on the same setup, we don't have to worry about hardware or setup differences. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-16 4:47 ` Qu Wenruo 2019-01-16 17:33 ` Vijaychidambaram Velayudhan Pillai @ 2019-01-17 0:16 ` Dave Chinner 2019-01-17 1:30 ` Qu Wenruo 1 sibling, 1 reply; 10+ messages in thread From: Dave Chinner @ 2019-01-17 0:16 UTC (permalink / raw) To: Qu Wenruo; +Cc: fstests On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: > > > On 2019/1/16 上午11:57, Dave Chinner wrote: > > On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote: > >> Hi, > >> > >> Is there any way to detect (huge) performance regression in a test case? > >> > >> By huge performance regression, I mean some operation takes from less > >> than 10s to around 400s. > >> > >> There is existing runtime accounting, but we can't do it inside a test > >> case (or can we?) > >> > >> So is there any way to detect huge performance regression in a test case? > > > > Just run your normal performance monitoring tools while the test is > > running to see what has changed. Is it IO, memory, CPU, lock > > contention or somethign else that is the problem? pcp, strace, top, > > iostat, perf, etc all work just fine for finding perf regressions > > reported by test cases... > > Sorry for the misunderstanding. > > I mean if it's possible for a test case to just fail when hitting some > big performance regression. This is part of the reported information in $RESULT_BASE/check.time. If you want to keep a history of runtimes for later comparison, then you just need to archive contents of that file with the test results. OR, alternatively, generate an XML test report which reports the individual test runtime in each report: ..... <testcase classname="xfstests.xfs" name="generic/036" time="12"> </testcase> <testcase classname="xfstests.xfs" name="generic/112" time="5"> </testcase> <testcase classname="xfstests.xfs" name="generic/113" time="4"> </testcase> <testcase classname="xfstests.xfs" name="generic/114" time="1"> ..... And then post-process these reports to determine runtime differences. > E.g. one operation should finish in 30s, but when it takes over 300s, > it's definitely a big regression. > > But considering how many different hardware/VM the test may be run on, > I'm not really confident if this is possible. You can really only determine performance regressions by comparing test runtime on kernels with the same features set run on the same hardware. Hence you'll need to keep archives from all your test machiens and configs and only compare between matching configurations. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-17 0:16 ` Dave Chinner @ 2019-01-17 1:30 ` Qu Wenruo 2019-01-17 2:25 ` Dave Chinner 0 siblings, 1 reply; 10+ messages in thread From: Qu Wenruo @ 2019-01-17 1:30 UTC (permalink / raw) To: Dave Chinner; +Cc: fstests [-- Attachment #1.1: Type: text/plain, Size: 2694 bytes --] On 2019/1/17 上午8:16, Dave Chinner wrote: > On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: >> >> >> On 2019/1/16 上午11:57, Dave Chinner wrote: >>> On Wed, Jan 16, 2019 at 09:59:40AM +0800, Qu Wenruo wrote: >>>> Hi, >>>> >>>> Is there any way to detect (huge) performance regression in a test case? >>>> >>>> By huge performance regression, I mean some operation takes from less >>>> than 10s to around 400s. >>>> >>>> There is existing runtime accounting, but we can't do it inside a test >>>> case (or can we?) >>>> >>>> So is there any way to detect huge performance regression in a test case? >>> >>> Just run your normal performance monitoring tools while the test is >>> running to see what has changed. Is it IO, memory, CPU, lock >>> contention or somethign else that is the problem? pcp, strace, top, >>> iostat, perf, etc all work just fine for finding perf regressions >>> reported by test cases... >> >> Sorry for the misunderstanding. >> >> I mean if it's possible for a test case to just fail when hitting some >> big performance regression. > > This is part of the reported information in $RESULT_BASE/check.time. > > If you want to keep a history of runtimes for later comparison, then > you just need to archive contents of that file with the test > results. > > OR, alternatively, generate an XML test report which reports the > individual test runtime in each report: > > ..... > <testcase classname="xfstests.xfs" name="generic/036" time="12"> > </testcase> > <testcase classname="xfstests.xfs" name="generic/112" time="5"> > </testcase> > <testcase classname="xfstests.xfs" name="generic/113" time="4"> > </testcase> > <testcase classname="xfstests.xfs" name="generic/114" time="1"> > ..... > > And then post-process these reports to determine runtime > differences. > >> E.g. one operation should finish in 30s, but when it takes over 300s, >> it's definitely a big regression. >> >> But considering how many different hardware/VM the test may be run on, >> I'm not really confident if this is possible. > > You can really only determine performance regressions by comparing > test runtime on kernels with the same features set run on the same > hardware. Hence you'll need to keep archives from all your test > machiens and configs and only compare between matching > configurations. Thanks, this matches my current understanding of how the testsuite works. It looks like such regression detection can only be implemented outside of fstests. Thanks for all the suggestions! Qu > > Cheers, > > Dave. > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-17 1:30 ` Qu Wenruo @ 2019-01-17 2:25 ` Dave Chinner 2019-01-23 0:51 ` Qu Wenruo 0 siblings, 1 reply; 10+ messages in thread From: Dave Chinner @ 2019-01-17 2:25 UTC (permalink / raw) To: Qu Wenruo; +Cc: fstests On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote: > On 2019/1/17 上午8:16, Dave Chinner wrote: > > On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: > >> E.g. one operation should finish in 30s, but when it takes over 300s, > >> it's definitely a big regression. > >> > >> But considering how many different hardware/VM the test may be run on, > >> I'm not really confident if this is possible. > > > > You can really only determine performance regressions by comparing > > test runtime on kernels with the same features set run on the same > > hardware. Hence you'll need to keep archives from all your test > > machiens and configs and only compare between matching > > configurations. > > Thanks, this matches my current understanding of how the testsuite works. > > It looks like such regression detection can only be implemented outside > of fstests. That's pretty much by design. Analysis of multiple test run results and post-processing them is really not something that the test harness does. The test harness really just runs the tests and records the results.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-17 2:25 ` Dave Chinner @ 2019-01-23 0:51 ` Qu Wenruo 2019-01-23 4:18 ` Dave Chinner 0 siblings, 1 reply; 10+ messages in thread From: Qu Wenruo @ 2019-01-23 0:51 UTC (permalink / raw) To: Dave Chinner; +Cc: fstests [-- Attachment #1.1: Type: text/plain, Size: 1709 bytes --] On 2019/1/17 上午10:25, Dave Chinner wrote: > On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote: >> On 2019/1/17 上午8:16, Dave Chinner wrote: >>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: >>>> E.g. one operation should finish in 30s, but when it takes over 300s, >>>> it's definitely a big regression. >>>> >>>> But considering how many different hardware/VM the test may be run on, >>>> I'm not really confident if this is possible. >>> >>> You can really only determine performance regressions by comparing >>> test runtime on kernels with the same features set run on the same >>> hardware. Hence you'll need to keep archives from all your test >>> machiens and configs and only compare between matching >>> configurations. >> >> Thanks, this matches my current understanding of how the testsuite works. >> >> It looks like such regression detection can only be implemented outside >> of fstests. > > That's pretty much by design. Analysis of multiple test run results > and post-processing them is really not something that the test > harness does. The test harness really just runs the tests and > records the results.... What about using some other telemetry other than time to determine regreesion? In my particular case, the correct behavior, some reading like generation would only increase by a somewhat predictable number. While when the regression happens, the generation will go way higher than expectation. Is it acceptable to craft a test case using such measurement? As I really hope to get an easy PASS/FAIL test case for QA guys to notice such huge regression. Thanks, Qu > > Cheers, > > Dave. > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-23 0:51 ` Qu Wenruo @ 2019-01-23 4:18 ` Dave Chinner 2019-01-23 5:08 ` Qu Wenruo 0 siblings, 1 reply; 10+ messages in thread From: Dave Chinner @ 2019-01-23 4:18 UTC (permalink / raw) To: Qu Wenruo; +Cc: fstests On Wed, Jan 23, 2019 at 08:51:03AM +0800, Qu Wenruo wrote: > > > On 2019/1/17 上午10:25, Dave Chinner wrote: > > On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote: > >> On 2019/1/17 上午8:16, Dave Chinner wrote: > >>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: > >>>> E.g. one operation should finish in 30s, but when it takes over 300s, > >>>> it's definitely a big regression. > >>>> > >>>> But considering how many different hardware/VM the test may be run on, > >>>> I'm not really confident if this is possible. > >>> > >>> You can really only determine performance regressions by comparing > >>> test runtime on kernels with the same features set run on the same > >>> hardware. Hence you'll need to keep archives from all your test > >>> machiens and configs and only compare between matching > >>> configurations. > >> > >> Thanks, this matches my current understanding of how the testsuite works. > >> > >> It looks like such regression detection can only be implemented outside > >> of fstests. > > > > That's pretty much by design. Analysis of multiple test run results > > and post-processing them is really not something that the test > > harness does. The test harness really just runs the tests and > > records the results.... > > What about using some other telemetry other than time to determine > regreesion? > > In my particular case, the correct behavior, some reading like > generation would only increase by a somewhat predictable number. > > While when the regression happens, the generation will go way higher > than expectation. That's something that would be done inside the test, right? i.e. this has nothing to do with the test harness itself, but is a failure criteria for the specific test? > Is it acceptable to craft a test case using such measurement? If it's reliable and not prone to false positives from future code changes, yes. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Any way to detect performance in a test case? 2019-01-23 4:18 ` Dave Chinner @ 2019-01-23 5:08 ` Qu Wenruo 0 siblings, 0 replies; 10+ messages in thread From: Qu Wenruo @ 2019-01-23 5:08 UTC (permalink / raw) To: Dave Chinner; +Cc: fstests [-- Attachment #1.1: Type: text/plain, Size: 2149 bytes --] On 2019/1/23 下午12:18, Dave Chinner wrote: > On Wed, Jan 23, 2019 at 08:51:03AM +0800, Qu Wenruo wrote: >> >> >> On 2019/1/17 上午10:25, Dave Chinner wrote: >>> On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote: >>>> On 2019/1/17 上午8:16, Dave Chinner wrote: >>>>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote: >>>>>> E.g. one operation should finish in 30s, but when it takes over 300s, >>>>>> it's definitely a big regression. >>>>>> >>>>>> But considering how many different hardware/VM the test may be run on, >>>>>> I'm not really confident if this is possible. >>>>> >>>>> You can really only determine performance regressions by comparing >>>>> test runtime on kernels with the same features set run on the same >>>>> hardware. Hence you'll need to keep archives from all your test >>>>> machiens and configs and only compare between matching >>>>> configurations. >>>> >>>> Thanks, this matches my current understanding of how the testsuite works. >>>> >>>> It looks like such regression detection can only be implemented outside >>>> of fstests. >>> >>> That's pretty much by design. Analysis of multiple test run results >>> and post-processing them is really not something that the test >>> harness does. The test harness really just runs the tests and >>> records the results.... >> >> What about using some other telemetry other than time to determine >> regreesion? >> >> In my particular case, the correct behavior, some reading like >> generation would only increase by a somewhat predictable number. >> >> While when the regression happens, the generation will go way higher >> than expectation. > > That's something that would be done inside the test, right? i.e. > this has nothing to do with the test harness itself, but is a > failure criteria for the specific test? Yep, a failure criteria for that specific test. > >> Is it acceptable to craft a test case using such measurement? > > If it's reliable and not prone to false positives from future code > changes, yes. Glad to know that. Thanks, Qu > > Cheers, > > Dave. > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-01-23 5:08 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-01-16 1:59 Any way to detect performance in a test case? Qu Wenruo 2019-01-16 3:57 ` Dave Chinner 2019-01-16 4:47 ` Qu Wenruo 2019-01-16 17:33 ` Vijaychidambaram Velayudhan Pillai 2019-01-17 0:16 ` Dave Chinner 2019-01-17 1:30 ` Qu Wenruo 2019-01-17 2:25 ` Dave Chinner 2019-01-23 0:51 ` Qu Wenruo 2019-01-23 4:18 ` Dave Chinner 2019-01-23 5:08 ` Qu Wenruo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox