* Golden output mismatch from generic/228, fs independent
@ 2025-09-02 10:30 Qu Wenruo
2025-09-02 12:09 ` Theodore Ts'o
0 siblings, 1 reply; 5+ messages in thread
From: Qu Wenruo @ 2025-09-02 10:30 UTC (permalink / raw)
To: fstests@vger.kernel.org, linux-btrfs, linux-xfs
Hi,
Recently I updated my arm64 VM, and now several test cases are failing
due to golden output mismatch.
This time it's fs independent, and I haven't yet updated fstests itself,
so it looks like some updates in my environment is breaking the test.
E.g, generic/228 on ext4 (the same on btrfs)
FSTYP -- ext4
PLATFORM -- Linux/aarch64 btrfs-aarch64 6.17.0-rc3-custom+ #132 SMP
PREEMPT_DYNAMIC Mon Sep 1 08:38:46 ACST 2025
MKFS_OPTIONS -- -F /dev/mapper/test-scratch1
MOUNT_OPTIONS -- -o acl,user_xattr /dev/mapper/test-scratch1 /mnt/scratch
generic/228 0s ... - output mismatch (see
/home/adam/xfstests-dev/results//generic/228.out.bad)
--- tests/generic/228.out 2024-04-25 18:13:45.126552201 +0930
+++ /home/adam/xfstests-dev/results//generic/228.out.bad 2025-09-02
19:51:22.806635177 +0930
@@ -1,6 +1,6 @@
QA output created by 228
File size limit is now set to 100 MB.
Let us try to preallocate 101 MB. This should fail.
-File size limit exceeded
+File size limit exceeded $XFS_IO_PROG -f -c 'falloc 0 101m'
$TEST_DIR/ouch
Let us now try to preallocate 50 MB. This should succeed.
Test over.
...
(Run 'diff -u /home/adam/xfstests-dev/tests/generic/228.out
/home/adam/xfstests-dev/results//generic/228.out.bad' to see the entire
diff)
Ran: generic/228
Failures: generic/228
Failed 1 of 1 tests
Or generic/733 on btrfs:
generic/733 7s ... - output mismatch (see
/home/adam/xfstests-dev/results//generic/733.out.bad)
--- tests/generic/733.out 2024-04-25 18:13:45.203549435 +0930
+++ /home/adam/xfstests-dev/results//generic/733.out.bad 2025-09-02
19:59:07.858861039 +0930
@@ -2,5 +2,5 @@
Format and mount
Create a many-block file
Reflink the big file
-Terminated
+Terminated $here/src/t_reflink_read_race
"$testdir/file1" "$testdir/file2" "$testdir/outcome" &>> $seqres.full
test completed successfully
...
(Run 'diff -u /home/adam/xfstests-dev/tests/generic/733.out
/home/adam/xfstests-dev/results//generic/733.out.bad' to see the entire
diff)
HINT: You _MAY_ be missing kernel fix:
5d6f0e9890ed btrfs: stop locking the source extent range during
reflink
I didn't know why, but the when those commands failed, the full script
line triggering it is also shown.
I checked my log, bash/xfsprogs and a lot of other packages are all
updated, and unfortunately my distro doesn't provide older packages for
me to bisect...
Thanks,
Qu
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Golden output mismatch from generic/228, fs independent 2025-09-02 10:30 Golden output mismatch from generic/228, fs independent Qu Wenruo @ 2025-09-02 12:09 ` Theodore Ts'o 2025-09-02 16:48 ` Theodore Ts'o 2025-09-02 21:40 ` Qu Wenruo 0 siblings, 2 replies; 5+ messages in thread From: Theodore Ts'o @ 2025-09-02 12:09 UTC (permalink / raw) To: Qu Wenruo; +Cc: fstests@vger.kernel.org, linux-btrfs, linux-xfs On Tue, Sep 02, 2025 at 08:00:32PM +0930, Qu Wenruo wrote: > Hi, > > Recently I updated my arm64 VM, and now several test cases are failing due > to golden output mismatch. > > This time it's fs independent, and I haven't yet updated fstests itself, so > it looks like some updates in my environment is breaking the test. > > E.g, generic/228 on ext4 (the same on btrfs) > > I checked my log, bash/xfsprogs and a lot of other packages are all updated, > and unfortunately my distro doesn't provide older packages for me to > bisect... I don't know if this helps, but here's a kvm-xfstests using Debian Trixie and certain updated critical packages (fio, quota, xfsprogs, util-linux, etc.) overriden. How does that differ from your distro package versions? I haven't updated my arm64 image in a bit (this was from dated July 20th). I can try doing an arm64 rebuild and see if a newer version still works on arm64, but here's a data point.... - TEd KERNEL: kernel 6.17.0-rc3-xfstests #1 SMP Tue Sep 2 07:57:28 EDT 2025 aarch64 CMDLINE: --arm64 -c ext4/4k generic/228 CPUS: 2 MEM: 1977.09 ext4/4k: 1 tests, 5 seconds generic/228 Pass 2s Totals: 1 tests, 0 skipped, 0 failures, 0 errors, 2s FSTESTVER: blktests 401420a (Fri, 6 Jun 2025 22:12:43 +0900) FSTESTVER: fio fio-3.40 (Tue, 20 May 2025 12:23:01 -0600) FSTESTVER: fsverity v1.6-2-gee7d74d (Mon, 17 Feb 2025 11:41:58 -0800) FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500) FSTESTVER: libaio libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200) FSTESTVER: ltp 20250130-280-g60656cbbb (Wed, 28 May 2025 15:04:44 +0200) FSTESTVER: quota v4.05-71-g4cd93fc (Sun, 27 Apr 2025 08:24:24 -0400) FSTESTVER: util-linux v2.41-40-g22b91501d (Mon, 26 May 2025 11:27:31 +0200) FSTESTVER: xfsprogs v6.15.0 (Mon, 23 Jun 2025 13:56:41 +0200) FSTESTVER: xfstests v2025.07.13-12-gef63d1368 (Sat, 19 Jul 2025 18:14:29 -0400) FSTESTVER: xfstests-bld gce-xfstests-202504292206-20-g905451c1 (Sun, 20 Jul 2025 03:04:27 +0000) FSTESTVER: zz_build-distro trixie FSTESTCFG: ext4/4k FSTESTSET: generic/228 FSTESTOPT: aex Truncating test artifacts in /results to 31k ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Golden output mismatch from generic/228, fs independent 2025-09-02 12:09 ` Theodore Ts'o @ 2025-09-02 16:48 ` Theodore Ts'o 2025-09-02 21:40 ` Qu Wenruo 1 sibling, 0 replies; 5+ messages in thread From: Theodore Ts'o @ 2025-09-02 16:48 UTC (permalink / raw) To: Qu Wenruo; +Cc: fstests@vger.kernel.org, linux-btrfs, linux-xfs On Tue, Sep 02, 2025 at 08:09:32AM -0400, Theodore Ts'o wrote: > I haven't updated my arm64 image in a bit (this was from dated July 20th). > I can try doing an arm64 rebuild and see if a newer version still > works on arm64, but here's a data point.... And here's an update with the latest xfstests for-next branch, and 6.16 xfsprogs, etc. Still no problems for me.... - Ted KERNEL: kernel 6.17.0-rc3-xfstests #1 SMP Tue Sep 2 07:57:28 EDT 2025 aarch64 CMDLINE: --arm64 -c ext4/4k generic/228 CPUS: 2 MEM: 1977.09 ext4/4k: 1 tests, 5 seconds generic/228 Pass 2s Totals: 1 tests, 0 skipped, 0 failures, 0 errors, 2s FSTESTVER: blktests 8ac6c4f (Sun, 31 Aug 2025 18:34:41 +0900) FSTESTVER: fio fio-3.40 (Tue, 20 May 2025 12:23:01 -0600) FSTESTVER: fsverity v1.6-2-gee7d74d (Mon, 17 Feb 2025 11:41:58 -0800) FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500) FSTESTVER: libaio libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200) FSTESTVER: ltp 20250530-178-g9691c4b2b (Mon, 1 Sep 2025 12:01:57 +0200) FSTESTVER: quota v4.05-77-g22ff3d9 (Tue, 2 Sep 2025 08:12:02 -0400) FSTESTVER: util-linux v2.41.1 (Tue, 24 Jun 2025 09:55:28 +0200) FSTESTVER: xfsprogs v6.16.0 (Tue, 26 Aug 2025 20:27:25 +0200) FSTESTVER: xfstests v2025.08.17-10-g057be3ead-dirty (Mon, 25 Aug 2025 11:59:43 -0400) FSTESTVER: xfstests-bld gce-xfstests-202504292206-22-ge10dc029-dirty (Mon, 25 Aug 2025 12:33:01 -0400) FSTESTVER: zz_build-distro trixie FSTESTCFG: ext4/4k FSTESTSET: generic/228 FSTESTOPT: aex ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Golden output mismatch from generic/228, fs independent 2025-09-02 12:09 ` Theodore Ts'o 2025-09-02 16:48 ` Theodore Ts'o @ 2025-09-02 21:40 ` Qu Wenruo 2025-09-04 1:29 ` Qu Wenruo 1 sibling, 1 reply; 5+ messages in thread From: Qu Wenruo @ 2025-09-02 21:40 UTC (permalink / raw) To: Theodore Ts'o; +Cc: fstests@vger.kernel.org, linux-btrfs, linux-xfs 在 2025/9/2 21:39, Theodore Ts'o 写道: > On Tue, Sep 02, 2025 at 08:00:32PM +0930, Qu Wenruo wrote: >> Hi, >> >> Recently I updated my arm64 VM, and now several test cases are failing due >> to golden output mismatch. >> >> This time it's fs independent, and I haven't yet updated fstests itself, so >> it looks like some updates in my environment is breaking the test. >> >> E.g, generic/228 on ext4 (the same on btrfs) >> >> I checked my log, bash/xfsprogs and a lot of other packages are all updated, >> and unfortunately my distro doesn't provide older packages for me to >> bisect... > > I don't know if this helps, but here's a kvm-xfstests using Debian > Trixie and certain updated critical packages (fio, quota, xfsprogs, > util-linux, etc.) overriden. How does that differ from your distro > package versions? > > I haven't updated my arm64 image in a bit (this was from dated July 20th). > I can try doing an arm64 rebuild and see if a newer version still > works on arm64, but here's a data point.... > > - TEd > > > KERNEL: kernel 6.17.0-rc3-xfstests #1 SMP Tue Sep 2 07:57:28 EDT 2025 aarch64 > CMDLINE: --arm64 -c ext4/4k generic/228 > CPUS: 2 > MEM: 1977.09 > > ext4/4k: 1 tests, 5 seconds > generic/228 Pass 2s > Totals: 1 tests, 0 skipped, 0 failures, 0 errors, 2s > > FSTESTVER: blktests 401420a (Fri, 6 Jun 2025 22:12:43 +0900) > FSTESTVER: fio fio-3.40 (Tue, 20 May 2025 12:23:01 -0600) Fio is in fact newer than mine. Mine is 3.39. > FSTESTVER: fsverity v1.6-2-gee7d74d (Mon, 17 Feb 2025 11:41:58 -0800) > FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500) > FSTESTVER: libaio libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200) Mine is a little newer, 0.3.113. > FSTESTVER: ltp 20250130-280-g60656cbbb (Wed, 28 May 2025 15:04:44 +0200) > FSTESTVER: quota v4.05-71-g4cd93fc (Sun, 27 Apr 2025 08:24:24 -0400) > FSTESTVER: util-linux v2.41-40-g22b91501d (Mon, 26 May 2025 11:27:31 +0200) > FSTESTVER: xfsprogs v6.15.0 (Mon, 23 Jun 2025 13:56:41 +0200) > FSTESTVER: xfstests v2025.07.13-12-gef63d1368 (Sat, 19 Jul 2025 18:14:29 -0400) > FSTESTVER: xfstests-bld gce-xfstests-202504292206-20-g905451c1 (Sun, 20 Jul 2025 03:04:27 +0000) > FSTESTVER: zz_build-distro trixie Others are mostly the same version. And indeed it's bash, after rolling back to v5.2.037 bash, everything is fine. It looks like it's bash 5.3.x starting to fail (5.3.0 tested and failed). I'll try dig a little deeper, and this is why I'm always running Archlinux (ArchlinuxARM in this case), to give some early warnings of some unexpected updates breaking the test cases. Thanks, Qu > FSTESTCFG: ext4/4k > FSTESTSET: generic/228 > FSTESTOPT: aex > Truncating test artifacts in /results to 31k > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Golden output mismatch from generic/228, fs independent 2025-09-02 21:40 ` Qu Wenruo @ 2025-09-04 1:29 ` Qu Wenruo 0 siblings, 0 replies; 5+ messages in thread From: Qu Wenruo @ 2025-09-04 1:29 UTC (permalink / raw) To: Theodore Ts'o; +Cc: fstests@vger.kernel.org, linux-btrfs, linux-xfs 在 2025/9/3 07:10, Qu Wenruo 写道: > > > 在 2025/9/2 21:39, Theodore Ts'o 写道: >> On Tue, Sep 02, 2025 at 08:00:32PM +0930, Qu Wenruo wrote: >>> Hi, >>> >>> Recently I updated my arm64 VM, and now several test cases are >>> failing due >>> to golden output mismatch. >>> >>> This time it's fs independent, and I haven't yet updated fstests >>> itself, so >>> it looks like some updates in my environment is breaking the test. >>> >>> E.g, generic/228 on ext4 (the same on btrfs) >>> >>> I checked my log, bash/xfsprogs and a lot of other packages are all >>> updated, >>> and unfortunately my distro doesn't provide older packages for me to >>> bisect... >> >> I don't know if this helps, but here's a kvm-xfstests using Debian >> Trixie and certain updated critical packages (fio, quota, xfsprogs, >> util-linux, etc.) overriden. How does that differ from your distro >> package versions? >> >> I haven't updated my arm64 image in a bit (this was from dated July >> 20th). >> I can try doing an arm64 rebuild and see if a newer version still >> works on arm64, but here's a data point.... >> >> - TEd >> >> >> KERNEL: kernel 6.17.0-rc3-xfstests #1 SMP Tue Sep 2 07:57:28 >> EDT 2025 aarch64 >> CMDLINE: --arm64 -c ext4/4k generic/228 >> CPUS: 2 >> MEM: 1977.09 >> >> ext4/4k: 1 tests, 5 seconds >> generic/228 Pass 2s >> Totals: 1 tests, 0 skipped, 0 failures, 0 errors, 2s >> >> FSTESTVER: blktests 401420a (Fri, 6 Jun 2025 22:12:43 +0900) >> FSTESTVER: fio fio-3.40 (Tue, 20 May 2025 12:23:01 -0600) > > Fio is in fact newer than mine. Mine is 3.39. > >> FSTESTVER: fsverity v1.6-2-gee7d74d (Mon, 17 Feb 2025 11:41:58 -0800) >> FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500) >> FSTESTVER: libaio libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 >> 13:33:11 +0200) > > Mine is a little newer, 0.3.113. > >> FSTESTVER: ltp 20250130-280-g60656cbbb (Wed, 28 May 2025 >> 15:04:44 +0200) >> FSTESTVER: quota v4.05-71-g4cd93fc (Sun, 27 Apr 2025 >> 08:24:24 -0400) >> FSTESTVER: util-linux v2.41-40-g22b91501d (Mon, 26 May 2025 11:27:31 >> +0200) >> FSTESTVER: xfsprogs v6.15.0 (Mon, 23 Jun 2025 13:56:41 +0200) >> FSTESTVER: xfstests v2025.07.13-12-gef63d1368 (Sat, 19 Jul 2025 >> 18:14:29 -0400) >> FSTESTVER: xfstests-bld gce-xfstests-202504292206-20-g905451c1 (Sun, >> 20 Jul 2025 03:04:27 +0000) >> FSTESTVER: zz_build-distro trixie > > Others are mostly the same version. > > > And indeed it's bash, after rolling back to v5.2.037 bash, everything is > fine. > > It looks like it's bash 5.3.x starting to fail (5.3.0 tested and failed). > > I'll try dig a little deeper, and this is why I'm always running > Archlinux (ArchlinuxARM in this case), to give some early warnings of > some unexpected updates breaking the test cases. It turns out to be the core dump behavior, the following script can easily reproduce it: ``` #!/bin/bash dev="/dev/test/scratch1" mnt="/mnt/btrfs" mkfs.btrfs -f $dev > /dev/null mount $dev $mnt sysctl -w kernel.core_pattern=core &>/dev/null ulimit -c 0 ulimit -f 102400 xfs_io -f -c "falloc 0 101m" /mnt/btrfs/foobar umount $mnt ``` Xfs_io will trigger coredump, and the full cmdline dump belongs to the core dump handling of bash. Thanks, Qu > > Thanks, > Qu > >> FSTESTCFG: ext4/4k >> FSTESTSET: generic/228 >> FSTESTOPT: aex >> Truncating test artifacts in /results to 31k >> > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-09-04 1:29 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-09-02 10:30 Golden output mismatch from generic/228, fs independent Qu Wenruo 2025-09-02 12:09 ` Theodore Ts'o 2025-09-02 16:48 ` Theodore Ts'o 2025-09-02 21:40 ` Qu Wenruo 2025-09-04 1:29 ` Qu Wenruo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox