From: Dave Chinner <david@fromorbit.com>
To: Eryu Guan <guaneryu@gmail.com>
Cc: fstests@vger.kernel.org
Subject: Re: [PATCH] check: fail tests if check/dmesg are not clean
Date: Tue, 22 May 2018 18:00:36 +1000 [thread overview]
Message-ID: <20180522080035.GU10363@dastard> (raw)
In-Reply-To: <20180512125108.GS8373@desktop>
On Sat, May 12, 2018 at 08:51:08PM +0800, Eryu Guan wrote:
> On Mon, May 07, 2018 at 08:46:34AM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> >
> > Currently a test passes even if it leaves a corrupt filesystem
> > behind, or a splat in the system logs that should not be there.
>
> It seems that test does fail when post-test fsck fails or dmesg check
> reports failure, but just after the test runtime being recorded &
> reported, which makes the test looks like a PASS.
Yup, that's exactly what I said - the test fails, but the reporting
indicates the test passed before it reports failure.
And because these failures match the "test pass" signature, my results
post-processing script was saying these tests passed, not had a
"post-test failure". The post-test failure messages don't mention
the test directly, either, so it's a bit of a wart when it comes to
processing the output for regression test comparisons...
Hence I thought I'd fix it so they didn't look like a test pass.
> But the test summary
> does report it as a failure, e.g. (I added "echo BUG: > /dev/kmsg" to
> generic/444 manually)
Right, but I don't look at those for comparing run-to-run regression
test results - having a timestamp that is wildly different is a
regression that needs investigation, too, even though we are only
looking at correctness in the test harness...
> generic/443 0s ... 0s
> generic/444 0s ... 0s
> _check_dmesg: something found in dmesg (see /root/workspace/xfstests/results//xfs_2k_crc/generic/444.dmesg)
> generic/445 0s ... 0s
> Ran: generic/443 generic/444 generic/445
> Failures: generic/444
> Failed 1 of 3 tests
>
> And the return value of check is also non-zero.
Yes, that is also true, but it is also not useful to tell me what
test out of the 700+ I've just run failed.
> But the whole rework looks good! Just need to reword the patch summary
> and description?
Ok.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2018-05-22 8:00 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-05-06 22:46 [PATCH] check: fail tests if check/dmesg are not clean Dave Chinner
2018-05-12 12:51 ` Eryu Guan
2018-05-22 8:00 ` Dave Chinner [this message]
-- strict thread matches above, loose matches on Subject: below --
2018-02-23 1:16 Dave Chinner
2018-02-23 4:34 ` Eryu Guan
2018-02-26 10:10 ` Eryu Guan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180522080035.GU10363@dastard \
--to=david@fromorbit.com \
--cc=fstests@vger.kernel.org \
--cc=guaneryu@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.