[PATCHSET 2/2] fstests: integrate with coredump capturing

[PATCHSET 2/2] fstests: integrate with coredump capturing

[Darrick J. Wong]

Hi all,

Integrate fstests with coredump capturing tools such as systemd-coredump.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

With a bit of luck, this should all go splendidly.
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=coredump-capture
---
Commits in this patchset:
 * fsstress: don't abort when stat(".") returns EIO
 * check: collect core dumps from systemd-coredump
---
 README         |   20 ++++++++++++++++++++
 check          |    2 ++
 common/rc      |   44 ++++++++++++++++++++++++++++++++++++++++++++
 ltp/fsstress.c |   29 ++++++++++++++++++++++++++---
 4 files changed, 92 insertions(+), 3 deletions(-)

[PATCH 1/2] fsstress: don't abort when stat(".") returns EIO

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

First, start with the premise that fstests is run with a nonzero limit
on the size of core dumps so that we can capture the state of
misbehaving fs utilities like fsck and scrub if they crash.

When fsstress is compiled with DEBUG defined (which is the default), it
will periodically call check_cwd to ensure that the current working
directory hasn't changed out from underneath it.

If the filesystem is XFS and it shuts down, the stat64() calls will
start returning EIO.  In this case, we follow the out: label and call
abort() to exit the program.  Historically this did not produce any core
dumps because $PWD is on the dead filesystem and the write fails.

However, modern systems are often configured to capture coredumps using
some external mechanism, e.g. abrt/systemd-coredump.   In this case, the
capture tool will succeeds in capturing every crashed process, which
fills the crash dump directory with a lot of useless junk.  Worse, if
the capture tool is configured to pass the dumps to fstests, it will
flag the test as failed because something dumped core.

This is really silly, because basic stat requests for the current
working directory can be satisfied from the inode cache without a disk
access.  In this narrow situation, EIO only happens when the fs has shut
down, so just exit the program.  Apply the same exit-on-EIO logic to
post-operation cleanup if we fail to go up one directory.

We really should have a way to query if a filesystem is shut down that
isn't conflated with (possibly transient) EIO errors.  But for now this
is what we have to do. :(

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 ltp/fsstress.c |   29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)


diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index 8dbfb81f95a538..ae31c6a22d4d93 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -1049,8 +1049,21 @@ check_cwd(void)
 
 	ret = stat64(".", &statbuf);
 	if (ret != 0) {
+		int error = errno;
+
 		fprintf(stderr, "fsstress: check_cwd stat64() returned %d with errno: %d (%s)\n",
-			ret, errno, strerror(errno));
+			ret, error, strerror(error));
+
+		/*
+		 * The current working directory is pinned in memory, which
+		 * means that stat should not have had to do any disk accesses
+		 * to retrieve stat information.  Treat an EIO as an indication
+		 * that the filesystem shut down and exit instead of dumping
+		 * core like the abort() below does.
+		 */
+		if (error == EIO)
+			exit(1);
+
 		goto out;
 	}
 
@@ -1284,14 +1297,24 @@ doproc(void)
 		 */
 		if (errtag != 0 && opno % 100 == 0)  {
 			rval = stat64(".", &statbuf);
-			if (rval == EIO)  {
+			if (rval != 0 && errno == EIO)  {
 				fprintf(stderr, "Detected EIO\n");
 				goto errout;
 			}
 		}
 	}
 errout:
-	assert(chdir("..") == 0);
+	rval = chdir("..");
+	if (rval != 0 && errno == EIO) {
+		/*
+		 * If we can't go up a directory due to EIO, treat that as an
+		 * indication that the filesystem shut down and exit instead of
+		 * dumping core like the abort() below does.
+		 */
+		fprintf(stderr, "Detected EIO, cannot clean up\n");
+		exit(1);
+	}
+	assert(rval == 0);
 	free(homedir);
 	if (cleanup) {
 		int ret;

Re: [PATCH 1/2] fsstress: don't abort when stat(".") returns EIO

[Christoph Hellwig]

On Tue, Oct 21, 2025 at 11:42:20AM -0700, Darrick J. Wong wrote:
> We really should have a way to query if a filesystem is shut down that
> isn't conflated with (possibly transient) EIO errors.  But for now this
> is what we have to do. :(

Please send a patch to send a statx patch..

But for this patch:

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH 2/2] check: collect core dumps from systemd-coredump

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be
installed to capture core dumps from crashed programs.  If this is the
case, we would like to capture core dumps from programs that crash
during the test.  Set up an (admittedly overwrought) pipeline to extract
dumps created during the test and then capture them the same way that we
pick up "core" and "core.$pid" files.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 README    |   20 ++++++++++++++++++++
 check     |    2 ++
 common/rc |   44 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+)


diff --git a/README b/README
index 9e9afe3cbb7ad4..196c79a21bdc0c 100644
--- a/README
+++ b/README
@@ -109,6 +109,11 @@ Ubuntu or Debian
    $ sudo apt-get install exfatprogs f2fs-tools ocfs2-tools udftools xfsdump \
         xfslibs-dev
 
+3. Install packages for optional features:
+
+    systemd coredump capture:
+    $ sudo apt install systemd-coredump systemd jq
+
 Fedora
 ------
 
@@ -124,6 +129,11 @@ Fedora
     $ sudo yum install btrfs-progs exfatprogs f2fs-tools ocfs2-tools xfsdump \
         xfsprogs-devel
 
+3. Install packages for optional features:
+
+    systemd coredump capture:
+    $ sudo yum install systemd systemd-udev jq
+
 RHEL or CentOS
 --------------
 
@@ -159,6 +169,11 @@ RHEL or CentOS
     For ocfs2 build and install:
      - see https://github.com/markfasheh/ocfs2-tools
 
+5. Install packages for optional features:
+
+    systemd coredump capture:
+    $ sudo yum install systemd systemd-udev jq
+
 SUSE Linux Enterprise or openSUSE
 ---------------------------------
 
@@ -176,6 +191,11 @@ SUSE Linux Enterprise or openSUSE
     For XFS install:
      $ sudo zypper install xfsdump xfsprogs-devel
 
+3. Install packages for optional features:
+
+    systemd coredump capture:
+    $ sudo yum install systemd systemd-coredump jq
+
 Build and install test, libs and utils
 --------------------------------------
 
diff --git a/check b/check
index 2d089d351380d2..c897afbb419612 100755
--- a/check
+++ b/check
@@ -930,6 +930,7 @@ function run_section()
 		     $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \
 		     END {printf "%s", lasttime}' "$check.time"
 		rm -f core $seqres.notrun
+		_start_coredumpctl_collection
 
 		start=`_wallclock`
 		$timestamp && _timestamp
@@ -963,6 +964,7 @@ function run_section()
 		# just "core".  Use globbing to find the most common patterns,
 		# assuming there are no other coredump capture packages set up.
 		local cores=0
+		_finish_coredumpctl_collection
 		for i in core core.*; do
 			test -f "$i" || continue
 			if ((cores++ == 0)); then
diff --git a/common/rc b/common/rc
index 462f433197a3c2..3e7d7646cd6868 100644
--- a/common/rc
+++ b/common/rc
@@ -5001,6 +5001,50 @@ _check_kmemleak()
 	fi
 }
 
+# Current timestamp, in a format that systemd likes
+_systemd_now() {
+	timedatectl show --property=TimeUSec --value
+}
+
+# Do what we need to do to capture core dumps from coredumpctl
+_start_coredumpctl_collection() {
+	command -v coredumpctl &>/dev/null || return
+	command -v timedatectl &>/dev/null || return
+	command -v jq &>/dev/null || return
+
+	sysctl kernel.core_pattern | grep -q systemd-coredump || return
+	COREDUMPCTL_START_TIMESTAMP="$(_systemd_now)"
+}
+
+# Capture core dumps from coredumpctl.
+#
+# coredumpctl list only supports json output as a machine-readable format.  The
+# human-readable format intermingles spaces from the timestamp with actual
+# column separators, so we cannot parse that sanely.  The json output is an
+# array of:
+#        {
+#                "time" : 1749744847150926,
+#                "pid" : 2297,
+#                "uid" : 0,
+#                "gid" : 0,
+#                "sig" : 6,
+#                "corefile" : "present",
+#                "exe" : "/run/fstests/e2fsprogs/fuse2fs",
+#                "size" : 47245
+#        },
+# So we use jq to filter out lost corefiles, then print the pid and exe
+# separated by a pipe and hope that nobody ever puts a pipe in an executable
+# name.
+_finish_coredumpctl_collection() {
+	test -n "$COREDUMPCTL_START_TIMESTAMP" || return
+
+	coredumpctl list --since="$COREDUMPCTL_START_TIMESTAMP" --json=short 2>/dev/null | \
+	jq --raw-output 'map(select(.corefile == "present")) | map("\(.pid)|\(.exe)") | .[]' | while IFS='|' read pid exe; do
+		test -e "core.$pid" || coredumpctl dump --output="core.$pid" "$pid" "$exe" &>> $seqres.full
+	done
+	unset COREDUMPCTL_START_TIMESTAMP
+}
+
 # don't check dmesg log after test
 _disable_dmesg_check()
 {

Re: [PATCH 2/2] check: collect core dumps from systemd-coredump

[Christoph Hellwig]

On Tue, Oct 21, 2025 at 11:42:35AM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be

Only KDE?  Most of my test systems don't have any graphic
environment set up.

Anyway, the changes looks fine:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 2/2] check: collect core dumps from systemd-coredump

[Darrick J. Wong]

On Tue, Oct 21, 2025 at 09:24:14PM -0700, Christoph Hellwig wrote:
> On Tue, Oct 21, 2025 at 11:42:35AM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be
> 
> Only KDE?  Most of my test systems don't have any graphic
> environment set up.

Non-KDE Debian doesn't bother installing systemd-coredump.  My actual
test systems have non-graphical Debian, but then I stood up one OL10
system for giggles and <kaboom> coredumps broke. :P

> Anyway, the changes looks fine:
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>

Thanks!

--D

[PATCHSET 3/3] fstests: integrate with coredump capturing

[Darrick J. Wong]

Hi all,

Integrate fstests with coredump capturing tools such as systemd-coredump.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

With a bit of luck, this should all go splendidly.
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=coredump-capture
---
Commits in this patchset:
 * fsstress: don't abort when stat(".") returns EIO
 * check: collect core dumps from systemd-coredump
---
 check          |    2 ++
 common/rc      |   44 ++++++++++++++++++++++++++++++++++++++++++++
 ltp/fsstress.c |   15 ++++++++++++++-
 3 files changed, 60 insertions(+), 1 deletion(-)

[PATCH 1/2] fsstress: don't abort when stat(".") returns EIO

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

First, start with the premise that fstests is run with a nonzero limit
on the size of core dumps so that we can capture the state of
misbehaving fs utilities like fsck and scrub if they crash.

When fsstress is compiled with DEBUG defined (which is the default), it
will periodically call check_cwd to ensure that the current working
directory hasn't changed out from underneath it.

If the filesystem is XFS and it shuts down, the stat64() calls will
start returning EIO.  In this case, we follow the out: label and call
abort() to exit the program.  Historically this did not produce any core
dumps because $PWD is on the dead filesystem and the write fails.

However, modern systems are often configured to capture coredumps using
some external mechanism, e.g. abrt/systemd-coredump.   In this case, the
capture tool will succeeds in capturing every crashed process, which
fills the crash dump directory with a lot of useless junk.  Worse, if
the capture tool is configured to pass the dumps to fstests, it will
flag the test as failed because something dumped core.

This is really silly, because basic stat requests for the current
working directory can be satisfied from the inode cache without a disk
access.  In this narrow situation, EIO only happens when the fs has shut
down, so just exit the program.

We really should have a way to query if a filesystem is shut down that
isn't conflated with (possibly transient) EIO errors.  But for now this
is what we have to do. :(

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 ltp/fsstress.c |   15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)


diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index 8dbfb81f95a538..d4abe561787f19 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -1049,8 +1049,21 @@ check_cwd(void)
 
 	ret = stat64(".", &statbuf);
 	if (ret != 0) {
+		int error = errno;
+
 		fprintf(stderr, "fsstress: check_cwd stat64() returned %d with errno: %d (%s)\n",
-			ret, errno, strerror(errno));
+			ret, error, strerror(error));
+
+		/*
+		 * The current working directory is pinned in memory, which
+		 * means that stat should not have had to do any disk accesses
+		 * to retrieve stat information.  Treat an EIO as an indication
+		 * that the filesystem shut down and exit instead of dumping
+		 * core like the abort() below does.
+		 */
+		if (error == EIO)
+			exit(1);
+
 		goto out;
 	}
 

Re: [PATCH 1/2] fsstress: don't abort when stat(".") returns EIO

[Christoph Hellwig]

On Tue, Jul 29, 2025 at 01:10:50PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> First, start with the premise that fstests is run with a nonzero limit
> on the size of core dumps so that we can capture the state of
> misbehaving fs utilities like fsck and scrub if they crash.

Can you explain what this has to do with core dumping?

I'm just really confused between this patch content and the subject of
this patch and the entire series..

> This is really silly, because basic stat requests for the current
> working directory can be satisfied from the inode cache without a disk
> access.  In this narrow situation, EIO only happens when the fs has shut
> down, so just exit the program.

If we think it's silly we can trivially drop the xfs_is_shutdown check
in xfs_vn_getattr.  But is it really silly?  We've tried to basically
make every file system operation consistently fail on shut down
file systems,

> We really should have a way to query if a filesystem is shut down that
> isn't conflated with (possibly transient) EIO errors.  But for now this
> is what we have to do. :(

Well, a new STATX_ flag would work, assuming stat doesn't actually
fail :)  Otherwise a new ioctl/fcntl would make sense, especially as
the shutdown concept has spread beyond XFS.

Re: [PATCH 1/2] fsstress: don't abort when stat(".") returns EIO

[Darrick J. Wong]

On Wed, Jul 30, 2025 at 07:23:24AM -0700, Christoph Hellwig wrote:
> On Tue, Jul 29, 2025 at 01:10:50PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > First, start with the premise that fstests is run with a nonzero limit
> > on the size of core dumps so that we can capture the state of
> > misbehaving fs utilities like fsck and scrub if they crash.
> 
> Can you explain what this has to do with core dumping?
> 
> I'm just really confused between this patch content and the subject of
> this patch and the entire series..

It's a bugfix ahead of new behaviors introduced in patch 2.  I clearly
didn't explain this well enough, so I'll try again.

Before abrt/systemd-coredump, FS_IOC_SHUTDOWN fsstress tests would do
something like the following:

1. start fsstress, which chdirs to $TEST_DIR
2. shut down the filesystem
3. fsstress tries to stat($TEST_DIR), fails, and calls abort
4. abort triggers coredump
5. kernel fails to write "core" to $TEST_DIR (because fs is shut down)
6. test finishes, no core files written to $here, test passes

Once you install systemd-coredump, that changes to:

same 1-4 above
5. kernel pipes core file to coredumpctl, which writes it to /var/crash
6. test finishes, no core files written to $here, test passes

And then with patch 2 of this series, that becomes:

same 1-4 above
5. kernel pipes core file to coredumpctl, which writes it to /var/crash
6. test finishes, ./check queries coredumpctl for any new coredumps,
   and copies them to $here
7. ./check finds core files written to $here, test fails

Now we've caused a test failure where there was none before, simply
because the crash reporting improved.

Therefore this patch changes fsstress not to call abort() from check_cwd
when it has a reasonable suspicion that the fs has died.

(Did that help?  /me is still pre-coffee...)

> > This is really silly, because basic stat requests for the current
> > working directory can be satisfied from the inode cache without a disk
> > access.  In this narrow situation, EIO only happens when the fs has shut
> > down, so just exit the program.
> 
> If we think it's silly we can trivially drop the xfs_is_shutdown check
> in xfs_vn_getattr.  But is it really silly?  We've tried to basically
> make every file system operation consistently fail on shut down
> file systems,

No no, "really silly" refers to failing tests that we didn't used to
fail.

> > We really should have a way to query if a filesystem is shut down that
> > isn't conflated with (possibly transient) EIO errors.  But for now this
> > is what we have to do. :(
> 
> Well, a new STATX_ flag would work, assuming stat doesn't actually
> fail :)  Otherwise a new ioctl/fcntl would make sense, especially as
> the shutdown concept has spread beyond XFS.

I think we ought to add a new ioctl or something so that callers can
positively identify a shut down filesystem.  bfoster I think was asking
about that for fstests some years back, and ended up coding a bunch of
grep heuristics to work around the lack of a real call.

I think we can't drop the "stat{,x} returns EIO on shutdown fs" behavior
because I know of a few, uh, users whose heartbeat monitor periodically
queries statx($PWD) and reboots the node if it returns errno.

--D