From: Dave Chinner <david@fromorbit.com>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: fstests <fstests@vger.kernel.org>
Subject: Re: Any way to detect performance in a test case?
Date: Wed, 23 Jan 2019 15:18:30 +1100 [thread overview]
Message-ID: <20190123041830.GQ6173@dastard> (raw)
In-Reply-To: <89cc6e7d-f8d7-23c0-d7ce-8f873ae5c0bc@gmx.com>
On Wed, Jan 23, 2019 at 08:51:03AM +0800, Qu Wenruo wrote:
>
>
> On 2019/1/17 上午10:25, Dave Chinner wrote:
> > On Thu, Jan 17, 2019 at 09:30:19AM +0800, Qu Wenruo wrote:
> >> On 2019/1/17 上午8:16, Dave Chinner wrote:
> >>> On Wed, Jan 16, 2019 at 12:47:21PM +0800, Qu Wenruo wrote:
> >>>> E.g. one operation should finish in 30s, but when it takes over 300s,
> >>>> it's definitely a big regression.
> >>>>
> >>>> But considering how many different hardware/VM the test may be run on,
> >>>> I'm not really confident if this is possible.
> >>>
> >>> You can really only determine performance regressions by comparing
> >>> test runtime on kernels with the same features set run on the same
> >>> hardware. Hence you'll need to keep archives from all your test
> >>> machiens and configs and only compare between matching
> >>> configurations.
> >>
> >> Thanks, this matches my current understanding of how the testsuite works.
> >>
> >> It looks like such regression detection can only be implemented outside
> >> of fstests.
> >
> > That's pretty much by design. Analysis of multiple test run results
> > and post-processing them is really not something that the test
> > harness does. The test harness really just runs the tests and
> > records the results....
>
> What about using some other telemetry other than time to determine
> regreesion?
>
> In my particular case, the correct behavior, some reading like
> generation would only increase by a somewhat predictable number.
>
> While when the regression happens, the generation will go way higher
> than expectation.
That's something that would be done inside the test, right? i.e.
this has nothing to do with the test harness itself, but is a
failure criteria for the specific test?
> Is it acceptable to craft a test case using such measurement?
If it's reliable and not prone to false positives from future code
changes, yes.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2019-01-23 4:18 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-16 1:59 Any way to detect performance in a test case? Qu Wenruo
2019-01-16 3:57 ` Dave Chinner
2019-01-16 4:47 ` Qu Wenruo
2019-01-16 17:33 ` Vijaychidambaram Velayudhan Pillai
2019-01-17 0:16 ` Dave Chinner
2019-01-17 1:30 ` Qu Wenruo
2019-01-17 2:25 ` Dave Chinner
2019-01-23 0:51 ` Qu Wenruo
2019-01-23 4:18 ` Dave Chinner [this message]
2019-01-23 5:08 ` Qu Wenruo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190123041830.GQ6173@dastard \
--to=david@fromorbit.com \
--cc=fstests@vger.kernel.org \
--cc=quwenruo.btrfs@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.