Re: [PATCH] fstests: report: always save the dmesg as system-err if KEEP_DMESG is set

Qu Wenruo
2022-12-20T00:57:19Z

On 2022/12/20 07:49, Theodore Ts'o wrote:
> On Tue, Dec 20, 2022 at 06:45:27AM +0800, Qu Wenruo wrote:
>>
>>
>> On the other hand, separating dmesg from its test cases means extra parsing
>> to bind the log section to certain test runs.
>>
>> In fact, I'm not even sure if possible to do that.
>> E.g. on the same host with different kernel/fstests configs.
>
> Well, I just have my test runner framework drop a file which contains
> a single line[2] in the /etc/rsyslog.d directory, and then kill -HUP
> rsyslogd. When I'm done, I'll reconfigure rsyslog and HUP it again,
> and then tar up the results directory, and save that as part of my
> test artifacts for that particular test run.
>
> [2] *.*;auth,authpriv.none -/results/syslog
>
> (This could probably be "kern.*" instead, but occasionally it's useful
> to see what other kind of background noise might be running on the
> test machine that might be disturbing the test run.)
>
> The /results directory will contain a copy of the current kernel's
> build config file, the arguments to the xfstests run, the xfstests
> results directory with all of the NNN.full and NNN.out.bad files, and
> of course, the syslog file. I also have things like a copy of
> /proc/slabinfo before and after the run; and I have support for hook
> scripts that can enable/disable ftrace with various tracepoints and
> which will drop the ftrace logs in the /results directory. (I don't
> have support for drgn or BPF scripts yet, but one of these days...)
>
> Anyway, my test framework will compress the /results directory and
> save it away (in my case, in Google Cloud Store). For a full ext4
> test run (twelve "-g auto" runs with different fs configs), the
> compressed test artifact tarball will be about 23 megabytes. In
> contrast, the xunut-quiet XML file will be about 80k compressed, and
> the test run summary which lands in my e-mail inbox is less than 4k. :-)
>
> - Ted
>
> P.S. This is a recent test run on btrfs using my test environment.
> I'm curious whether any of these test failures look familiar to you.
> (Some of them may very well be issues in how I set up the btrfs pool
> of block devices, which is a relatively new feature in gce-xfstests.
> What are you seeing when you run -g auto on btrfs using the default
> file system config?)
>
> TESTRUNID: tytso-20221212170803
> KERNEL: kernel 6.1.0-xfstests #2 SMP PREEMPT_DYNAMIC Mon Dec 12 16:09:40 EST 2022 x86_64
> CMDLINE: -c btrfs/4k -g auto
> CPUS: 2
> MEM: 7680
>
> btrfs/4k: 953 tests, 21 failures, 176 skipped, 12135 seconds

That's way more errors than what we see.

Currently what we expect is, at most 3 from btrfs, zero from generic:

- btrfs/011
If your SSD is too fast, that balance is done before canceling, it can
caused output mismatch.
Always reproducible here if using cache=unsafe for VM.

- btrfs/219 (already in your failure)
The upstream change is not merged, it should be removed from auto.

- btrfs/249 (btrfs-progs bug)
The fix for btrfs-progs is submitted by still needs some polish.

> Failures: btrfs/006 btrfs/012 btrfs/049 btrfs/100 btrfs/162 btrfs/163
> btrfs/184 btrfs/192 btrfs/196 btrfs/197 btrfs/218 btrfs/219 btrfs/235
> btrfs/254 btrfs/277 btrfs/291 generic/455 generic/457
> Flaky: btrfs/028: 40% (2/5) generic/475: 60% (3/5)
> shared/298: 20% (1/5)

btrfs/012 is convert related, thus I'm wondering if some older
btrfs-progs is involved in this case.

For generic/455 and generic/457, we need the full out.bad and dmesg to
determine.

Anyway since our expectation for future fstests runs are really no
errors, I'm pretty happy to help on any of the failed runs.

>
> FSTESTIMG: gce-xfstests/xfstests-amd64-202211301035
> FSTESTPRJ: gce-xfstests
> FSTESTVER: blktests 068bd2a (Fri, 18 Nov 2022 08:38:35 +0900)
> FSTESTVER: fio fio-3.31 (Tue, 9 Aug 2022 14:41:25 -0600)
> FSTESTVER: fsverity v1.5 (Sun, 6 Feb 2022 10:59:13 -0800)
> FSTESTVER: ima-evm-utils v1.3.2 (Wed, 28 Oct 2020 13:18:08 -0400)
> FSTESTVER: nvme-cli v1.16 (Thu, 11 Nov 2021 13:09:06 -0800)
> FSTESTVER: quota v4.05-52-gf7e24ee (Tue, 1 Nov 2022 11:45:06 +0100)
> FSTESTVER: util-linux v2.38.1 (Thu, 4 Aug 2022 11:06:21 +0200)
> FSTESTVER: xfsprogs v6.0.0 (Mon, 14 Nov 2022 12:06:23 +0100)
> FSTESTVER: xfstests-bld 920c0e32 (Mon, 28 Nov 2022 00:37:48 -0500)
> FSTESTVER: xfstests v2022.11.27-8-g3c178050c (Wed, 30 Nov 2022 10:25:39 -0500)
> FSTESTVER: zz_build-distro bullseye

Since the distro is Debian 11, I'm pretty sure the btrfs-progs is
out-of-date.

Mind to add btrfs-progs to the FSTESTVER reporting?
And hopefully to also build a newer progs just like xfsprogs for testing?

Thanks,
Qu

> FSTESTCFG: btrfs/4k
> FSTESTSET: -g auto
> FSTESTOPT: aex
> GCE ID: 1186623270044702885
>
>
> - Ted