* [PATCHSET v31.0 1/5] fstests: detect deceptive filename extensions
2024-08-27 18:42 [PATCHBOMB v2024.08.25] fstests: catch us up to xfsprogs 6.10 Darrick J. Wong
@ 2024-08-27 18:44 ` Darrick J. Wong
2024-08-27 18:45 ` [PATCH 1/2] generic/453: test confusable name detection with 32-bit unicode codepoints Darrick J. Wong
2024-08-27 18:46 ` [PATCH 2/2] generic/453: check xfs_scrub detection of confusing job offers Darrick J. Wong
2024-08-27 18:44 ` [PATCHSET v31.0 2/5] fstests: test systemd background services Darrick J. Wong
` (3 subsequent siblings)
4 siblings, 2 replies; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:44 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
Hi all,
In early 2023, malware researchers disclosed a phishing attack that was
targeted at people running Linux workstations. The attack vector
involved the use of filenames containing what looked like a file
extension but instead contained a lookalike for the full stop (".")
and a common extension ("pdf"). Enhance xfs_scrub phase 5 to detect
these types of attacks and warn the system administrator. Add
functional testing for this code.
If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy!
Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-detect-deceptive-extensions
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-detect-deceptive-extensions
---
Commits in this patchset:
* generic/453: test confusable name detection with 32-bit unicode codepoints
* generic/453: check xfs_scrub detection of confusing job offers
---
tests/generic/453 | 111 +++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 99 insertions(+), 12 deletions(-)
^ permalink raw reply [flat|nested] 21+ messages in thread* [PATCH 1/2] generic/453: test confusable name detection with 32-bit unicode codepoints
2024-08-27 18:44 ` [PATCHSET v31.0 1/5] fstests: detect deceptive filename extensions Darrick J. Wong
@ 2024-08-27 18:45 ` Darrick J. Wong
2024-08-30 5:55 ` Christoph Hellwig
2024-08-27 18:46 ` [PATCH 2/2] generic/453: check xfs_scrub detection of confusing job offers Darrick J. Wong
1 sibling, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:45 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
Test the confusable name detection when there are 32-bit unicode
sequences in use. In other words, emoji. Change the xfs_scrub test to
dump the output to a file instead of passing huge echo commands around.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
tests/generic/453 | 32 +++++++++++++++++++++-----------
1 file changed, 21 insertions(+), 11 deletions(-)
diff --git a/tests/generic/453 b/tests/generic/453
index a0fb802e9b..930e6408ff 100755
--- a/tests/generic/453
+++ b/tests/generic/453
@@ -148,6 +148,10 @@ setf "combmark_\xe1\x80\x9c\xe1\x80\xaf\xe1\x80\xad.txt" "combining marks"
setd ".\xe2\x80\x8d" "zero width joiners in dot entry"
setd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
+# utf8 sequence mapping to a u32 unicode codepoint that can be confused
+setf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
+setf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
+
ls -la $testdir >> $seqres.full
echo "Test files"
@@ -198,6 +202,9 @@ testf "combmark_\xe1\x80\x9c\xe1\x80\xaf\xe1\x80\xad.txt" "combining marks"
testd ".\xe2\x80\x8d" "zero width joiners in dot entry"
testd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
+testf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
+testf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
+
echo "Uniqueness of inodes?"
stat -c '%i' "${testdir}/"* | sort | uniq -c | while read nr inum; do
if [ "${nr}" -gt 1 ]; then
@@ -208,18 +215,21 @@ done
echo "Test XFS online scrub, if applicable"
if _check_xfs_scrub_does_unicode "$SCRATCH_MNT" "$SCRATCH_DEV"; then
- output="$(LC_ALL="C.UTF-8" ${XFS_SCRUB_PROG} -v -n "${SCRATCH_MNT}" 2>&1 | filter_scrub)"
- echo "${output}" | grep -q "french_" || echo "No complaints about french e accent?"
- echo "${output}" | grep -q "greek_" || echo "No complaints about greek letter mess?"
- echo "${output}" | grep -q "arabic_" || echo "No complaints about arabic expanded string?"
- echo "${output}" | grep -q "mixed_" || echo "No complaints about mixed script confusables?"
- echo "${output}" | grep -q "hyphens_" || echo "No complaints about hyphenation confusables?"
- echo "${output}" | grep -q "dz_digraph_" || echo "No complaints about single script confusables?"
- echo "${output}" | grep -q "inadequate_" || echo "No complaints about inadequate rendering confusables?"
- echo "${output}" | grep -q "prohibition_" || echo "No complaints about prohibited sequence confusables?"
- echo "${output}" | grep -q "zerojoin_" || echo "No complaints about zero-width join confusables?"
+ LC_ALL="C.UTF-8" ${XFS_SCRUB_PROG} -v -n "${SCRATCH_MNT}" 2>&1 | filter_scrub > $tmp.scrub
+
+ grep -q "french_" $tmp.scrub || echo "No complaints about french e accent?"
+ grep -q "greek_" $tmp.scrub || echo "No complaints about greek letter mess?"
+ grep -q "arabic_" $tmp.scrub || echo "No complaints about arabic expanded string?"
+ grep -q "mixed_" $tmp.scrub || echo "No complaints about mixed script confusables?"
+ grep -q "hyphens_" $tmp.scrub || echo "No complaints about hyphenation confusables?"
+ grep -q "dz_digraph_" $tmp.scrub || echo "No complaints about single script confusables?"
+ grep -q "inadequate_" $tmp.scrub || echo "No complaints about inadequate rendering confusables?"
+ grep -q "prohibition_" $tmp.scrub || echo "No complaints about prohibited sequence confusables?"
+ grep -q "zerojoin_" $tmp.scrub || echo "No complaints about zero-width join confusables?"
+ grep -q "toilet_" $tmp.scrub || echo "No complaints about zero-width join confusables with emoji?"
+
echo "Actual xfs_scrub output:" >> $seqres.full
- echo "${output}" >> $seqres.full
+ cat $tmp.scrub >> $seqres.full
fi
# success, all done
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 2/2] generic/453: check xfs_scrub detection of confusing job offers
2024-08-27 18:44 ` [PATCHSET v31.0 1/5] fstests: detect deceptive filename extensions Darrick J. Wong
2024-08-27 18:45 ` [PATCH 1/2] generic/453: test confusable name detection with 32-bit unicode codepoints Darrick J. Wong
@ 2024-08-27 18:46 ` Darrick J. Wong
2024-08-30 5:55 ` Christoph Hellwig
1 sibling, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:46 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
Earlier this year, ESET revealed that Linux users had been tricked into
opening executables containing malware payloads. The trickery came in
the form of a malicious zip file containing a filename with the string
"job offer․pdf". Note that the filename does *not* denote a real pdf
file, since the last four codepoints in the file name are "ONE DOT
LEADER", p, d, and f. Not period (ok, FULL STOP), p, d, f like you'd
normally expect.
Now that xfs_scrub can look for codepoints that could be confused with a
period followed by alphanumerics, let's make sure it actually works.
Link: https://www.welivesecurity.com/2023/04/20/linux-malware-strengthens-links-lazarus-3cx-supply-chain-attack/
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
tests/generic/453 | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 78 insertions(+), 1 deletion(-)
diff --git a/tests/generic/453 b/tests/generic/453
index 930e6408ff..855243a860 100755
--- a/tests/generic/453
+++ b/tests/generic/453
@@ -36,6 +36,15 @@ setf() {
echo "Storing ${key} ($(hexbytes "${key}")) -> ${value}" >> $seqres.full
}
+setchild() {
+ subdir="$1"
+ key="$(echo -e "$2")"
+
+ mkdir -p "${testdir}/${subdir}"
+ echo "$subdir" > "${testdir}/${subdir}/${key}"
+ echo "Storing ${subdir}/${key} ($(hexbytes "${key}")) -> ${subdir}" >> $seqres.full
+}
+
setd() {
key="$(echo -e "$1")"
value="$2"
@@ -63,6 +72,24 @@ testf() {
fi
}
+testchild() {
+ subdir="$1"
+ key="$(echo -e "$2")"
+ fname="${testdir}/${subdir}/${key}"
+
+ echo "Testing ${subdir}/${key} ($(hexbytes "${key}")) -> ${subdir}" >> $seqres.full
+
+ if [ ! -e "${fname}" ]; then
+ echo "Key ${key} does not exist for ${subdir} test??"
+ return
+ fi
+
+ actual_value="$(cat "${fname}")"
+ if [ "${actual_value}" != "${subdir}" ]; then
+ echo "Key ${key} has value ${subdir}, expected ${actual_value}."
+ fi
+}
+
testd() {
key="$(echo -e "$1")"
value="$2"
@@ -152,7 +179,27 @@ setd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
setf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
setf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
-ls -la $testdir >> $seqres.full
+# decoy file extensions used in 3cx malware attack, and similar ones
+setchild "one_dot_leader" "job offer\xe2\x80\xa4pdf"
+setchild "small_full_stop" "job offer\xef\xb9\x92pdf"
+setchild "fullwidth_full_stop" "job offer\xef\xbc\x8epdf"
+setchild "syriac_supralinear" "job offer\xdc\x81pdf"
+setchild "syriac_sublinear" "job offer\xdc\x82pdf"
+setchild "lisu_letter_tone" "job offer\xea\x93\xb8pdf"
+setchild "actual_period" "job offer.pdf"
+setchild "one_dot_leader_zero_width_space" "job offer\xe2\x80\xa4\xe2\x80\x8dpdf"
+
+# again, but this time all in the same directory to trip the confusable
+# detector
+setf "job offer\xe2\x80\xa4pdf" "one dot leader"
+setf "job offer\xef\xb9\x92pdf" "small full stop"
+setf "job offer\xef\xbc\x8epdf" "fullwidth full stop"
+setf "job offer\xdc\x81pdf" "syriac supralinear full stop"
+setf "job offer\xdc\x82pdf" "syriac sublinear full stop"
+setf "job offer\xea\x93\xb8pdf" "lisu letter tone mya ti"
+setf "job offer.pdf" "actual period"
+
+ls -laR $testdir >> $seqres.full
echo "Test files"
testf "french_caf\xc3\xa9.txt" "NFC"
@@ -205,6 +252,23 @@ testd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
testf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
testf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
+testchild "one_dot_leader" "job offer\xe2\x80\xa4pdf"
+testchild "small_full_stop" "job offer\xef\xb9\x92pdf"
+testchild "fullwidth_full_stop" "job offer\xef\xbc\x8epdf"
+testchild "syriac_supralinear" "job offer\xdc\x81pdf"
+testchild "syriac_sublinear" "job offer\xdc\x82pdf"
+testchild "lisu_letter_tone" "job offer\xea\x93\xb8pdf"
+testchild "actual_period" "job offer.pdf"
+testchild "one_dot_leader_zero_width_space" "job offer\xe2\x80\xa4\xe2\x80\x8dpdf"
+
+testf "job offer\xe2\x80\xa4pdf" "one dot leader"
+testf "job offer\xef\xb9\x92pdf" "small full stop"
+testf "job offer\xef\xbc\x8epdf" "fullwidth full stop"
+testf "job offer\xdc\x81pdf" "syriac supralinear full stop"
+testf "job offer\xdc\x82pdf" "syriac sublinear full stop"
+testf "job offer\xea\x93\xb8pdf" "lisu letter tone mya ti"
+testf "job offer.pdf" "actual period"
+
echo "Uniqueness of inodes?"
stat -c '%i' "${testdir}/"* | sort | uniq -c | while read nr inum; do
if [ "${nr}" -gt 1 ]; then
@@ -228,6 +292,19 @@ if _check_xfs_scrub_does_unicode "$SCRATCH_MNT" "$SCRATCH_DEV"; then
grep -q "zerojoin_" $tmp.scrub || echo "No complaints about zero-width join confusables?"
grep -q "toilet_" $tmp.scrub || echo "No complaints about zero-width join confusables with emoji?"
+ # Does xfs_scrub complain at all about the job offer files? Pre-2023
+ # versions did not know to screen for that.
+ if grep -q "job offer" $tmp.scrub; then
+ grep -q 'job offer.xe2.x80.xa4pdf' $tmp.scrub || echo "No complaints about one dot leader?"
+ grep -q "job offer.xef.xb9.x92pdf" $tmp.scrub || echo "No complaints about small full stop?"
+ grep -q "job offer.xef.xbc.x8epdf" $tmp.scrub || echo "No complaints about fullwidth full stop?"
+ grep -q "job offer.xdc.x81pdf" $tmp.scrub || echo "No complaints about syriac supralinear full stop?"
+ grep -q "job offer.xdc.x82pdf" $tmp.scrub || echo "No complaints about syriac sublinear full stop?"
+ grep -q "job offer.xea.x93.xb8pdf" $tmp.scrub || echo "No complaints about lisu letter tone mya ti?"
+ grep -q "job offer.*could be confused with" $tmp.scrub || echo "No complaints about confusing job offers?"
+ grep -q "job offer.xe2.x80.xa4.xe2.x80.x8dpdf" $tmp.scrub || echo "No complaints about one dot leader with invisible space?"
+ fi
+
echo "Actual xfs_scrub output:" >> $seqres.full
cat $tmp.scrub >> $seqres.full
fi
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCHSET v31.0 2/5] fstests: test systemd background services
2024-08-27 18:42 [PATCHBOMB v2024.08.25] fstests: catch us up to xfsprogs 6.10 Darrick J. Wong
2024-08-27 18:44 ` [PATCHSET v31.0 1/5] fstests: detect deceptive filename extensions Darrick J. Wong
@ 2024-08-27 18:44 ` Darrick J. Wong
2024-08-27 18:46 ` [PATCH 1/1] xfs: test xfs_scrub services Darrick J. Wong
2024-08-27 18:45 ` [PATCHSET v31.0 3/5] fstests: use free space histograms to reduce fstrim runtime Darrick J. Wong
` (2 subsequent siblings)
4 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:44 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
Hi all,
Add a couple of new tests to check that the systemd services for
xfs_scrub are at least minimally functional.
If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy!
Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-improvements
---
Commits in this patchset:
* xfs: test xfs_scrub services
---
common/rc | 14 +++++
common/systemd | 73 +++++++++++++++++++++++++++
tests/xfs/1863 | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/1863.out | 5 ++
4 files changed, 232 insertions(+)
create mode 100644 common/systemd
create mode 100755 tests/xfs/1863
create mode 100644 tests/xfs/1863.out
^ permalink raw reply [flat|nested] 21+ messages in thread* [PATCH 1/1] xfs: test xfs_scrub services
2024-08-27 18:44 ` [PATCHSET v31.0 2/5] fstests: test systemd background services Darrick J. Wong
@ 2024-08-27 18:46 ` Darrick J. Wong
2024-08-30 5:56 ` Christoph Hellwig
0 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:46 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
Create a pair of new tests that check that xfs_scrub and xfs_scrub_all
will find and test mounted filesystems.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/rc | 14 +++++
common/systemd | 73 +++++++++++++++++++++++++++
tests/xfs/1863 | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/1863.out | 5 ++
4 files changed, 232 insertions(+)
create mode 100644 common/systemd
create mode 100755 tests/xfs/1863
create mode 100644 tests/xfs/1863.out
diff --git a/common/rc b/common/rc
index 9da9fe1882..5a26669925 100644
--- a/common/rc
+++ b/common/rc
@@ -5480,6 +5480,20 @@ _require_duplicate_fsid()
esac
}
+# Can we find a program in the $PATH?
+_have_program() {
+ command -v "$1" &>/dev/null
+}
+
+# Require that a program can be found via the $PATH, or complain otherwise
+_require_program() {
+ local cmd="$1"
+ local tag="$2"
+
+ test -z "$tag" && tag="$(basename "$cmd")"
+ _have_program "$1" || _notrun "$tag required"
+}
+
init_rc
################################################################################
diff --git a/common/systemd b/common/systemd
new file mode 100644
index 0000000000..b2e24f267b
--- /dev/null
+++ b/common/systemd
@@ -0,0 +1,73 @@
+##/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle. All Rights Reserved.
+#
+# Routines for managing systemd units
+
+# Is systemd installed?
+_systemd_installed() {
+ _have_program systemctl
+}
+
+_require_systemd_installed() {
+ _require_program systemctl systemd
+}
+
+# Is systemd running on this system?
+_systemd_is_running() {
+ _systemd_installed || return 1
+ systemctl show-environment &>/dev/null
+}
+
+_require_systemd_is_running() {
+ _systemd_is_running || \
+ _notrun "systemd is not active"
+}
+
+# Is this unit defined, as according to systemd?
+_systemd_unit_defined() {
+ _systemd_installed || return 1
+ systemctl cat "$1" >/dev/null
+}
+
+_require_systemd_unit_defined() {
+ _require_systemd_installed
+ _systemd_unit_defined "$1" || \
+ _notrun "systemd unit \"$1\" not found"
+}
+
+# Is this unit active, as according to systemd?
+_systemd_unit_active() {
+ _systemd_installed || return 1
+ _systemd_unit_defined "$1" || return 1
+
+ test "$(systemctl is-active "$1")" = "active"
+}
+
+_require_systemd_unit_active() {
+ _require_systemd_unit_defined "$1"
+ _systemd_unit_active "$1" || \
+ _notrun "systemd unit \"$1\" not active"
+}
+
+# Find the path to a systemd unit
+_systemd_unit_path() {
+ systemctl show "$1" | grep FragmentPath= | cut -d '=' -f 2
+}
+
+# Make systemd reload itself after changing unit files or generator sources
+# such as /etc/fstab
+_systemd_reload() {
+ systemctl daemon-reload
+}
+
+# Where is the systemd runtime directory?
+_systemd_runtime_dir() {
+ echo "/run/systemd/system/"
+}
+
+# What is the status of this systemd unit?
+_systemd_unit_status() {
+ _systemd_installed || return 1
+ systemctl status "$1"
+}
diff --git a/tests/xfs/1863 b/tests/xfs/1863
new file mode 100755
index 0000000000..7d41092e2e
--- /dev/null
+++ b/tests/xfs/1863
@@ -0,0 +1,140 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023-2024 Oracle. All Rights Reserved.
+#
+# FS QA Test No. 1863
+#
+# Check that the online fsck systemd services find and check the scratch
+# filesystem, and that we can read the health reports after the fact. IOWs,
+# this is basic testing for the systemd background services.
+#
+. ./common/preamble
+_begin_fstest auto scrub
+
+_cleanup()
+{
+ cd /
+ if [ -n "$new_svcfile" ]; then
+ rm -f "$new_svcfile"
+ systemctl daemon-reload
+ fi
+ rm -r -f $tmp.*
+}
+
+. ./common/filter
+. ./common/populate
+. ./common/fuzzy
+. ./common/systemd
+
+_require_systemd_is_running
+_require_systemd_unit_defined xfs_scrub@.service
+_require_systemd_unit_defined xfs_scrub_all.service
+_require_scratch
+_require_scrub
+_require_xfs_io_command "scrub"
+_require_xfs_spaceman_command "health"
+_require_populate_commands
+
+# xfs_scrub_all requires the python3-dbus library, which is a separate package
+_require_command $PYTHON3_PROG python3
+$PYTHON3_PROG -c 'import dbus' &>/dev/null || \
+ _notrun "test requires python3-dbus"
+
+_require_command $ATTR_PROG "attr"
+
+_xfs_skip_online_rebuild
+_xfs_skip_offline_rebuild
+
+# Back when xfs_scrub was really experimental, the systemd service definitions
+# contained various bugs that resulted in weird problems such as logging
+# messages sometimes dropping slashes from paths, and the xfs_scrub@ service
+# being logged as completing long after the process actually stopped. These
+# problems were all fixed by the time the --auto-media-scan-stamp option was
+# added to xfs_scrub_all, so turn off this test for such old codebases.
+scruball_exe="$(systemctl cat xfs_scrub_all | grep '^ExecStart=' | \
+ sed -e 's/ExecStart=//g' -e 's/ .*$//g')"
+grep -q -- '--auto-media-scan-stamp' "$scruball_exe" || \
+ _notrun "xfs_scrub service too old, skipping test"
+
+orig_svcfile="$(_systemd_unit_path "xfs_scrub_all.service")"
+test -f "$orig_svcfile" || \
+ _notrun "cannot find xfs_scrub_all service file"
+
+new_svcdir="$(_systemd_runtime_dir)"
+test -d "$new_svcdir" || \
+ _notrun "cannot find runtime systemd service dir"
+
+# We need to make some local mods to the xfs_scrub_all service definition
+# so we fork it and create a new service just for this test.
+new_scruball_svc="xfs_scrub_all_fstest.service"
+_systemd_unit_status "$new_scruball_svc" 2>&1 | \
+ grep -E -q '(could not be found|Loaded: not-found)' || \
+ _notrun "systemd service \"$new_scruball_svc\" found, will not mess with this"
+
+find_scrub_trace() {
+ local path="$1"
+
+ $XFS_SPACEMAN_PROG -c "health" "$path" | grep -q ": ok$" || \
+ echo "cannot find evidence that $path was scrubbed"
+}
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+_scratch_mount
+
+# Configure the filesystem for background checks of the filesystem.
+$ATTR_PROG -R -s xfs:autofsck -V check $SCRATCH_MNT >> $seqres.full
+
+run_scrub_service() {
+ systemctl start --wait "$1"
+
+ # Sometimes systemctl start --wait returns early due to some external
+ # event, such as somebody else reloading the daemon, which closes the
+ # socket. The CLI has no way to resume waiting for the service once
+ # the connection breaks, so we'll pgrep for up to 30 seconds until
+ # there are no xfs_scrub processes running on the system.
+ for ((i = 0; i < 30; i++)); do
+ pgrep -f 'xfs_scrub*' > /dev/null 2>&1 || break
+ sleep 1
+ done
+}
+
+echo "Scrub Scratch FS"
+scratch_path=$(systemd-escape --path "$SCRATCH_MNT")
+run_scrub_service xfs_scrub@$scratch_path
+find_scrub_trace "$SCRATCH_MNT"
+
+# Remove the xfs_scrub_all media scan stamp directory (if specified) because we
+# want to leave the regular system's stamp file alone.
+mkdir -p $tmp/stamp
+
+new_svcfile="$new_svcdir/$new_scruball_svc"
+cp "$orig_svcfile" "$new_svcfile"
+
+execstart="$(grep '^ExecStart=' $new_svcfile | \
+ sed -e 's/--auto-media-scan-interval[[:space:]]*[0-9]*[a-z]*//g')"
+sed -e '/ExecStart=/d' -e '/BindPaths=/d' -i $new_svcfile
+cat >> "$new_svcfile" << ENDL
+[Service]
+$execstart
+ENDL
+_systemd_reload
+
+# Emit the results of our editing to the full log.
+systemctl cat "$new_scruball_svc" >> $seqres.full
+
+# Cycle mounts to clear all the incore CHECKED bits.
+_scratch_cycle_mount
+
+echo "Scrub Everything"
+run_scrub_service "$new_scruball_svc"
+
+sleep 2 # give systemd a chance to tear down the service container mount tree
+
+find_scrub_trace "$SCRATCH_MNT"
+
+echo "Scrub Done" | tee -a $seqres.full
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1863.out b/tests/xfs/1863.out
new file mode 100644
index 0000000000..27c2d79a63
--- /dev/null
+++ b/tests/xfs/1863.out
@@ -0,0 +1,5 @@
+QA output created by 1863
+Format and populate
+Scrub Scratch FS
+Scrub Everything
+Scrub Done
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCHSET v31.0 3/5] fstests: use free space histograms to reduce fstrim runtime
2024-08-27 18:42 [PATCHBOMB v2024.08.25] fstests: catch us up to xfsprogs 6.10 Darrick J. Wong
2024-08-27 18:44 ` [PATCHSET v31.0 1/5] fstests: detect deceptive filename extensions Darrick J. Wong
2024-08-27 18:44 ` [PATCHSET v31.0 2/5] fstests: test systemd background services Darrick J. Wong
@ 2024-08-27 18:45 ` Darrick J. Wong
2024-08-27 18:46 ` [PATCH 1/1] xfs/004: fix column extraction code Darrick J. Wong
2024-08-27 18:45 ` [PATCHSET v4.1 4/5] fstests: enable FITRIM for the realtime section Darrick J. Wong
2024-08-27 18:45 ` [PATCHSET v31.0 5/5] fstests: xfs filesystem properties Darrick J. Wong
4 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:45 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
Hi all,
This patchset dramatically reduces the runtime of the FITRIM calls made
during phase 8 of xfs_scrub. It turns out that phase 8 can really get
bogged down if the free space contains a large number of very small
extents. In these cases, the runtime can increase by an order of
magnitude to free less than 1% of the free space. This is not worth the
time, since we're spending a lot of time to do very little work. The
FITRIM ioctl allows us to specify a minimum extent length, so we can use
statistical methods to compute a minlen parameter.
It turns out xfs_db/spaceman already have the code needed to create
histograms of free space extent lengths. We add the ability to compute
a CDF of the extent lengths, which make it easy to pick a minimum length
corresponding to 99% of the free space. In most cases, this results in
dramatic reductions in phase 8 runtime. Hence, move the histogram code
to libfrog, and wire up xfs_scrub, since phase 7 already walks the
fsmap.
We also add a new -o suboption to xfs_scrub so that people who /do/ want
to examine every free extent can do so.
If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy!
Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
---
Commits in this patchset:
* xfs/004: fix column extraction code
---
tests/xfs/004 | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH 1/1] xfs/004: fix column extraction code
2024-08-27 18:45 ` [PATCHSET v31.0 3/5] fstests: use free space histograms to reduce fstrim runtime Darrick J. Wong
@ 2024-08-27 18:46 ` Darrick J. Wong
2024-08-30 5:57 ` Christoph Hellwig
0 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:46 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
Now that the xfs_db freesp command prints a CDF of the free space
histograms, fix the pct column extraction code to handle the two
new columns by <cough> using awk.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
tests/xfs/004 | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
diff --git a/tests/xfs/004 b/tests/xfs/004
index 8b2fd9b320..473b82c945 100755
--- a/tests/xfs/004
+++ b/tests/xfs/004
@@ -82,14 +82,17 @@ then
fi
# check the 'pct' field from freesp command is good
-perl -ne '
- BEGIN { $percent = 0; }
- /free/ && next; # skip over free extent size number
- if (/\s+(\d+\.\d+)$/) {
- $percent += $1;
- }
- END { $percent += 0.5; print int($percent), "\n" } # round up
-' <$tmp.xfs_db >$tmp.ans
+awk '
+{
+ if ($0 ~ /free/) {
+ next;
+ }
+
+ percent += $5;
+}
+END {
+ printf("%d\n", int(percent + 0.5));
+}' < $tmp.xfs_db > $tmp.ans
ans="`cat $tmp.ans`"
echo "Checking percent column yields 100: $ans"
if [ "$ans" != 100 ]
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [PATCH 1/1] xfs/004: fix column extraction code
2024-08-27 18:46 ` [PATCH 1/1] xfs/004: fix column extraction code Darrick J. Wong
@ 2024-08-30 5:57 ` Christoph Hellwig
0 siblings, 0 replies; 21+ messages in thread
From: Christoph Hellwig @ 2024-08-30 5:57 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: zlang, hch, fstests, linux-xfs
On Tue, Aug 27, 2024 at 11:46:39AM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Now that the xfs_db freesp command prints a CDF of the free space
> histograms, fix the pct column extraction code to handle the two
> new columns by <cough> using awk.
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
.. and we really need this ASAP to not fail tests with the latest
xfsprogs.
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCHSET v4.1 4/5] fstests: enable FITRIM for the realtime section
2024-08-27 18:42 [PATCHBOMB v2024.08.25] fstests: catch us up to xfsprogs 6.10 Darrick J. Wong
` (2 preceding siblings ...)
2024-08-27 18:45 ` [PATCHSET v31.0 3/5] fstests: use free space histograms to reduce fstrim runtime Darrick J. Wong
@ 2024-08-27 18:45 ` Darrick J. Wong
2024-08-27 18:46 ` [PATCH 1/2] xfs: refactor statfs field extraction Darrick J. Wong
2024-08-27 18:47 ` [PATCH 2/2] common/xfs: FITRIM now supports realtime volumes Darrick J. Wong
2024-08-27 18:45 ` [PATCHSET v31.0 5/5] fstests: xfs filesystem properties Darrick J. Wong
4 siblings, 2 replies; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:45 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
Hi all,
One thing that's been missing for a long time is the ability to tell
underlying storage that it can unmap the unused space on the realtime
device. This short series exposes this functionality through FITRIM.
Callers that want ranged FITRIM should be aware that the realtime space
exists in the offset range after the data device. However, it is
anticipated that most callers pass in offset=0 len=-1ULL and will not
notice or care.
If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy!
Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-discard
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-discard
---
Commits in this patchset:
* xfs: refactor statfs field extraction
* common/xfs: FITRIM now supports realtime volumes
---
common/xfs | 46 ++++++++++++++++++++++++++++++++++++++++++++--
tests/xfs/176 | 4 ++--
tests/xfs/187 | 6 +++---
tests/xfs/541 | 6 ++----
4 files changed, 51 insertions(+), 11 deletions(-)
^ permalink raw reply [flat|nested] 21+ messages in thread* [PATCH 1/2] xfs: refactor statfs field extraction
2024-08-27 18:45 ` [PATCHSET v4.1 4/5] fstests: enable FITRIM for the realtime section Darrick J. Wong
@ 2024-08-27 18:46 ` Darrick J. Wong
2024-08-30 5:57 ` Christoph Hellwig
2024-08-27 18:47 ` [PATCH 2/2] common/xfs: FITRIM now supports realtime volumes Darrick J. Wong
1 sibling, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:46 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
Prepare for the next patch by refactoring the open-coded bits that call
statfs on a mounted xfs filesystem to extract a status field.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/xfs | 6 ++++++
tests/xfs/176 | 4 ++--
tests/xfs/187 | 6 +++---
tests/xfs/541 | 6 ++----
4 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/common/xfs b/common/xfs
index 6370f17523..7ee6fbec84 100644
--- a/common/xfs
+++ b/common/xfs
@@ -1818,3 +1818,9 @@ _require_xfs_parent()
|| _notrun "kernel does not support parent pointers"
_scratch_unmount
}
+
+# Extract a statfs attribute of the given mounted XFS filesystem.
+_xfs_statfs_field()
+{
+ $XFS_IO_PROG -c 'statfs' "$1" | grep -E "$2" | cut -d ' ' -f 3
+}
diff --git a/tests/xfs/176 b/tests/xfs/176
index db7001a5b9..8d58590fd9 100755
--- a/tests/xfs/176
+++ b/tests/xfs/176
@@ -47,7 +47,7 @@ fi
_scratch_mount
_xfs_force_bdev data $SCRATCH_MNT
-old_dblocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.datablocks)
+old_dblocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.datablocks)
mkdir $SCRATCH_MNT/save/ $SCRATCH_MNT/urk/
sino=$(stat -c '%i' $SCRATCH_MNT/save)
@@ -170,7 +170,7 @@ for ((ino = target_ino; ino >= icluster_ino; ino--)); do
res=$?
# Make sure shrink did not work
- new_dblocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.datablocks)
+ new_dblocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.datablocks)
if [ "$new_dblocks" != "$old_dblocks" ]; then
echo "should not have shrank $old_dblocks -> $new_dblocks"
break
diff --git a/tests/xfs/187 b/tests/xfs/187
index 04ff9a81b6..56a9adc164 100755
--- a/tests/xfs/187
+++ b/tests/xfs/187
@@ -77,8 +77,8 @@ _xfs_force_bdev realtime $SCRATCH_MNT
# Set the extent size hint larger than the realtime extent size. This is
# necessary to exercise the minlen constraints on the realtime allocator.
-fsbsize=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.bsize | awk '{print $3}')
-rtextsize_blks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.rtextsize | awk '{print $3}')
+fsbsize=$(_xfs_statfs_field "$SCRATCH_MNT" geom.bsize)
+rtextsize_blks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextsize)
extsize=$((2 * rtextsize_blks * fsbsize))
echo "rtextsize_blks=$rtextsize_blks extsize=$extsize" >> $seqres.full
@@ -133,7 +133,7 @@ punch_off=$((bigfile_sz - frag_sz))
$here/src/punch-alternating $SCRATCH_MNT/bigfile -o $((punch_off / fsbsize)) -i $((rtextsize_blks * 2)) -s $rtextsize_blks
# Make sure we have some free rtextents.
-free_rtx=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep statfs.f_bavail | awk '{print $3}')
+free_rtx=$(_xfs_statfs_field "$SCRATCH_MNT" statfs.f_bavail)
if [ $free_rtx -eq 0 ]; then
echo "Expected fragmented free rt space, found none."
fi
diff --git a/tests/xfs/541 b/tests/xfs/541
index f18b801cfe..518373fa89 100755
--- a/tests/xfs/541
+++ b/tests/xfs/541
@@ -81,13 +81,11 @@ test $grow_extszhint -eq 0 || \
echo "expected post-grow extszhint 0, got $grow_extszhint"
# Check that we now have rt extents.
-rtextents=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | \
- grep 'geom.rtextents' | cut -d ' ' -f 3)
+rtextents=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextents)
test $rtextents -gt 0 || echo "expected rtextents > 0"
# Check the new rt extent size.
-after_rtextsz_blocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | \
- grep 'geom.rtextsize' | cut -d ' ' -f 3)
+after_rtextsz_blocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextsize)
test $after_rtextsz_blocks -eq $new_rtextsz_blocks || \
echo "expected rtextsize $new_rtextsz_blocks, got $after_rtextsz_blocks"
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 2/2] common/xfs: FITRIM now supports realtime volumes
2024-08-27 18:45 ` [PATCHSET v4.1 4/5] fstests: enable FITRIM for the realtime section Darrick J. Wong
2024-08-27 18:46 ` [PATCH 1/2] xfs: refactor statfs field extraction Darrick J. Wong
@ 2024-08-27 18:47 ` Darrick J. Wong
2024-08-30 5:57 ` Christoph Hellwig
1 sibling, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:47 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
XFS now supports FITRIM to the realtime volume. Detect this support and
enable it.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/xfs | 40 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 38 insertions(+), 2 deletions(-)
diff --git a/common/xfs b/common/xfs
index 7ee6fbec84..de557ebd90 100644
--- a/common/xfs
+++ b/common/xfs
@@ -1777,8 +1777,44 @@ _require_xfs_scratch_atomicswap()
# of 1024 byte blocks.
_xfs_discard_max_offset_kb()
{
- $XFS_IO_PROG -c 'statfs' "$1" | \
- awk '{g[$1] = $3} END {print (g["geom.bsize"] * g["geom.datablocks"] / 1024)}'
+ local statfs
+
+ # Use awk to read the statfs output for the XFS filesystem, compute
+ # the two possible FITRIM offset maximums, and then use some horrid
+ # bash magic to import the five numbers as an indexed array. There's
+ # no better way to do this in bash since you can't readarray to build
+ # an associative array. Elements are as follows:
+ #
+ # 0: fsblock size in bytes
+ # 1: Data volume size in fsblocks.
+ # 2: Realtime volume size in fsblocks.
+ # 3: Max FITRIM offset if we can only trim the data volume
+ # 4: Max FITRIM offset if we can trim the data and rt volumes
+ readarray -t statfs < <($XFS_IO_PROG -c 'statfs' "$1" | \
+ awk '{g[$1] = $3} END {printf("%d\n%d\n%d\n%d\n%d\n",
+ g["geom.bsize"],
+ g["geom.datablocks"],
+ g["geom.rtblocks"],
+ g["geom.bsize"] * g["geom.datablocks"] / 1024,
+ g["geom.bsize"] * (g["geom.datablocks"] + g["geom.rtblocks"]) / 1024);}')
+
+ # If the kernel supports discarding the realtime volume, then it will
+ # not reject a FITRIM for fsblock dblks+1, even if the len/minlen
+ # arguments are absurd.
+ if [ "${statfs[2]}" -gt 0 ]; then
+ if $FSTRIM_PROG -o "$((statfs[0] * statfs[1]))" \
+ -l "${statfs[0]}" \
+ -m "$((statfs[0] * 2))" "$1" &>/dev/null; then
+ # The kernel supports discarding the rt volume, so
+ # print out the second answer from above.
+ echo "${statfs[4]}"
+ return
+ fi
+ fi
+
+ # The kernel does not support discarding the rt volume or there is no
+ # rt volume. Print out the first answer from above.
+ echo "${statfs[3]}"
}
# check if mkfs and the kernel support nocrc (v4) file systems
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCHSET v31.0 5/5] fstests: xfs filesystem properties
2024-08-27 18:42 [PATCHBOMB v2024.08.25] fstests: catch us up to xfsprogs 6.10 Darrick J. Wong
` (3 preceding siblings ...)
2024-08-27 18:45 ` [PATCHSET v4.1 4/5] fstests: enable FITRIM for the realtime section Darrick J. Wong
@ 2024-08-27 18:45 ` Darrick J. Wong
2024-08-27 18:47 ` [PATCH 1/1] xfs: functional testing for " Darrick J. Wong
4 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:45 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
Hi all,
It would be very useful if system administrators could set properties for a
given xfs filesystem to control its behavior. This we can do easily and
extensibly by setting ATTR_ROOT (aka "trusted") extended attributes on the root
directory. To prevent this from becoming a weird free for all, let's add some
library and tooling support so that sysadmins simply run the xfs_property
program to administer these properties.
If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy!
Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=filesystem-properties
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=filesystem-properties
---
Commits in this patchset:
* xfs: functional testing for filesystem properties
---
common/config | 1
common/xfs | 14 ++++-
doc/group-names.txt | 1
tests/generic/062 | 4 +
tests/xfs/1886 | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/1886.out | 53 ++++++++++++++++++++
tests/xfs/1887 | 122 +++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/1887.out | 46 +++++++++++++++++
tests/xfs/1888 | 66 +++++++++++++++++++++++++
tests/xfs/1888.out | 9 +++
tests/xfs/1889 | 63 +++++++++++++++++++++++
tests/xfs/1889.out | 8 +++
12 files changed, 522 insertions(+), 2 deletions(-)
create mode 100755 tests/xfs/1886
create mode 100644 tests/xfs/1886.out
create mode 100755 tests/xfs/1887
create mode 100644 tests/xfs/1887.out
create mode 100755 tests/xfs/1888
create mode 100644 tests/xfs/1888.out
create mode 100755 tests/xfs/1889
create mode 100644 tests/xfs/1889.out
^ permalink raw reply [flat|nested] 21+ messages in thread* [PATCH 1/1] xfs: functional testing for filesystem properties
2024-08-27 18:45 ` [PATCHSET v31.0 5/5] fstests: xfs filesystem properties Darrick J. Wong
@ 2024-08-27 18:47 ` Darrick J. Wong
2024-08-30 5:57 ` Christoph Hellwig
0 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2024-08-27 18:47 UTC (permalink / raw)
To: djwong, zlang; +Cc: hch, fstests, linux-xfs
From: Darrick J. Wong <djwong@kernel.org>
Make sure that fs property storage and retrieval actually work.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/config | 1
common/xfs | 14 ++++-
doc/group-names.txt | 1
tests/generic/062 | 4 +
tests/xfs/1886 | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/1886.out | 53 ++++++++++++++++++++
tests/xfs/1887 | 122 +++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/1887.out | 46 +++++++++++++++++
tests/xfs/1888 | 66 +++++++++++++++++++++++++
tests/xfs/1888.out | 9 +++
tests/xfs/1889 | 63 +++++++++++++++++++++++
tests/xfs/1889.out | 8 +++
12 files changed, 522 insertions(+), 2 deletions(-)
create mode 100755 tests/xfs/1886
create mode 100644 tests/xfs/1886.out
create mode 100755 tests/xfs/1887
create mode 100644 tests/xfs/1887.out
create mode 100755 tests/xfs/1888
create mode 100644 tests/xfs/1888.out
create mode 100755 tests/xfs/1889
create mode 100644 tests/xfs/1889.out
diff --git a/common/config b/common/config
index 307f93fbce..668b185ff0 100644
--- a/common/config
+++ b/common/config
@@ -234,6 +234,7 @@ export GZIP_PROG="$(type -P gzip)"
export BTRFS_IMAGE_PROG="$(type -P btrfs-image)"
export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
export PARTED_PROG="$(type -P parted)"
+export XFS_PROPERTY_PROG="$(type -P xfs_property)"
# use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
# newer systems have udevadm command but older systems like RHEL5 don't.
diff --git a/common/xfs b/common/xfs
index de557ebd90..62e3100ee1 100644
--- a/common/xfs
+++ b/common/xfs
@@ -1323,8 +1323,8 @@ _require_xfs_spaceman_command()
testfile=$TEST_DIR/$$.xfs_spaceman
touch $testfile
case $command in
- "health")
- testio=`$XFS_SPACEMAN_PROG -c "health $param" $TEST_DIR 2>&1`
+ "health"|"listfsprops")
+ testio=`$XFS_SPACEMAN_PROG -c "$command $param" $TEST_DIR 2>&1`
param_checked=1
;;
*)
@@ -1860,3 +1860,13 @@ _xfs_statfs_field()
{
$XFS_IO_PROG -c 'statfs' "$1" | grep -E "$2" | cut -d ' ' -f 3
}
+
+# Wipe all filesystem properties from an xfs filesystem. The sole argument
+# must be the root directory of a filesystem.
+_wipe_xfs_properties()
+{
+ getfattr --match="^trusted.xfs:" --absolute-names --dump --encoding=hex "$1" | \
+ grep '=' | sed -e 's/=.*$//g' | while read name; do
+ setfattr --remove="$name" "$1"
+ done
+}
diff --git a/doc/group-names.txt b/doc/group-names.txt
index 6cf717969d..ed886caac0 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -56,6 +56,7 @@ fiexchange XFS_IOC_EXCHANGE_RANGE ioctl
freeze filesystem freeze tests
fsck general fsck tests
fsmap FS_IOC_GETFSMAP ioctl
+fsproperties Filesystem properties
fsr XFS free space reorganizer
fuzzers filesystem fuzz tests
growfs increasing the size of a filesystem
diff --git a/tests/generic/062 b/tests/generic/062
index 8f4dfcbf55..f0904992d1 100755
--- a/tests/generic/062
+++ b/tests/generic/062
@@ -75,6 +75,10 @@ fi
_require_attrs $ATTR_MODES
+# Wipe all xfs filesystem properties (which are rootdir xattrs) before we dump
+# them all later.
+test $FSTYP = "xfs" && _wipe_xfs_properties $SCRATCH_MNT
+
for nsp in $ATTR_MODES; do
for inode in reg dir lnk dev/b dev/c dev/p; do
diff --git a/tests/xfs/1886 b/tests/xfs/1886
new file mode 100755
index 0000000000..68abfedd15
--- /dev/null
+++ b/tests/xfs/1886
@@ -0,0 +1,137 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle. All Rights Reserved.
+#
+# FS QA Test 1886
+#
+# Functional testing for low level filesystem property manipulation by
+# xfs_{spaceman,db}.
+#
+. ./common/preamble
+_begin_fstest auto fsproperties
+
+. ./common/filter
+. ./common/attr
+
+_require_test
+_require_user fsgqa
+_require_attrs
+_require_xfs_io_command listfsprops
+_require_xfs_db_command attr_list
+
+_cleanup()
+
+{
+ cd /
+ rm -r -f $tmp.*
+ rm -f $TEST_DIR/$seq.somefile
+ rm -r -f $TEST_DIR/$seq.somedir
+ test -n "$propname" && $ATTR_PROG -R -r $propname $TEST_DEV &>/dev/null
+}
+
+filter_inum()
+{
+ sed -e 's/inode [0-9]*/inode XXX/g'
+}
+
+propname="fakeproperty" # must not be an actual property
+propval="1721943740"
+longpropname="$(perl -e 'print "x" x 300;')"
+longpropval="$(perl -e 'print "x" x 80000;')"
+
+echo "*** IO TEST ***"
+
+echo empty get property
+$XFS_IO_PROG -c "getfsprops $propname" $TEST_DIR
+
+echo pointless remove property
+$XFS_IO_PROG -c "removefsprops $propname" $TEST_DIR
+
+echo list property
+$XFS_IO_PROG -c "listfsprops" $TEST_DIR | grep $propname
+
+echo set property
+$XFS_IO_PROG -c "setfsprops $propname=$propval" $TEST_DIR
+
+echo list property
+$XFS_IO_PROG -c "listfsprops" $TEST_DIR | grep $propname
+
+echo dump xattrs
+$ATTR_PROG -R -l $TEST_DIR | grep $propname | _filter_test_dir
+
+echo get property
+$XFS_IO_PROG -c "getfsprops $propname" $TEST_DIR
+
+echo list property
+$XFS_IO_PROG -c "listfsprops" $TEST_DIR | grep $propname
+
+echo child file rejected
+touch $TEST_DIR/$seq.somefile
+$XFS_IO_PROG -c "listfsprops $propname" $TEST_DIR/$seq.somefile 2>&1 | \
+ _filter_test_dir
+
+echo child dir rejected
+mkdir -p $TEST_DIR/$seq.somedir
+$XFS_IO_PROG -c "listfsprops $propname" $TEST_DIR/$seq.somedir 2>&1 | \
+ _filter_test_dir
+
+echo remove property
+$XFS_IO_PROG -c "removefsprops $propname" $TEST_DIR
+
+echo pointless remove property
+$XFS_IO_PROG -c "removefsprops $propname" $TEST_DIR
+
+echo set too long name
+$XFS_IO_PROG -c "setfsprops $longpropname=$propval" $TEST_DIR
+
+echo set too long value
+$XFS_IO_PROG -c "setfsprops $propname=$longpropval" $TEST_DIR
+
+echo not enough permissions
+su - "$qa_user" -c "$XFS_IO_PROG -c \"setfsprops $propname=$propval\" $TEST_DIR" 2>&1 | _filter_test_dir
+
+echo "*** DB TEST ***"
+
+propval=$((propval + 1))
+_test_unmount
+
+echo empty get property
+_test_xfs_db -x -c 'path /' -c "attr_get -Z $propname" 2>&1 | filter_inum
+
+echo pointless remove property
+_test_xfs_db -x -c 'path /' -c "attr_remove -Z $propname" 2>&1 | filter_inum
+
+echo list property
+_test_xfs_db -x -c 'path /' -c "attr_list -Z" | grep $propname
+
+echo set property
+_test_xfs_db -x -c 'path /' -c "attr_set -Z $propname $propval"
+
+echo list property
+_test_xfs_db -x -c 'path /' -c "attr_list -Z" | grep $propname
+
+echo dump xattrs
+_test_mount
+$ATTR_PROG -R -l $TEST_DIR | grep $propname | _filter_test_dir
+_test_unmount
+
+echo get property
+_test_xfs_db -x -c 'path /' -c "attr_get -Z $propname"
+
+echo list property
+_test_xfs_db -x -c 'path /' -c "attr_list -Z" | grep $propname
+
+echo remove property
+_test_xfs_db -x -c 'path /' -c "attr_remove -Z $propname"
+
+echo pointless remove property
+_test_xfs_db -x -c 'path /' -c "attr_remove -Z $propname" 2>&1 | filter_inum
+
+echo set too long name
+_test_xfs_db -x -c 'path /' -c "attr_set -Z $longpropname $propval"
+
+echo set too long value
+_test_xfs_db -x -c 'path /' -c "attr_set -Z $propname $longpropval"
+
+status=0
+exit
diff --git a/tests/xfs/1886.out b/tests/xfs/1886.out
new file mode 100644
index 0000000000..e02d810406
--- /dev/null
+++ b/tests/xfs/1886.out
@@ -0,0 +1,53 @@
+QA output created by 1886
+*** IO TEST ***
+empty get property
+fakeproperty: No data available
+pointless remove property
+fakeproperty: No data available
+list property
+set property
+fakeproperty=1721943740
+list property
+fakeproperty
+dump xattrs
+Attribute "xfs:fakeproperty" has a 10 byte value for TEST_DIR
+get property
+fakeproperty=1721943740
+list property
+fakeproperty
+child file rejected
+TEST_DIR/1886.somefile: Not a XFS mount point.
+child dir rejected
+TEST_DIR/1886.somedir: Not a XFS mount point.
+remove property
+pointless remove property
+fakeproperty: No data available
+set too long name
+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: Invalid argument
+set too long value
+fakeproperty: Invalid argument
+not enough permissions
+TEST_DIR: Operation not permitted
+*** DB TEST ***
+empty get property
+failed to get attr xfs:fakeproperty on inode XXX: No data available
+pointless remove property
+failed to remove attr xfs:fakeproperty from inode XXX: No data available
+list property
+set property
+fakeproperty=1721943741
+list property
+fakeproperty
+dump xattrs
+Attribute "xfs:fakeproperty" has a 10 byte value for TEST_DIR
+get property
+fakeproperty=1721943741
+list property
+fakeproperty
+remove property
+pointless remove property
+failed to remove attr xfs:fakeproperty from inode XXX: No data available
+set too long name
+name too long
+set too long value
+xfs:fakeproperty: value too long
diff --git a/tests/xfs/1887 b/tests/xfs/1887
new file mode 100755
index 0000000000..cd70c42021
--- /dev/null
+++ b/tests/xfs/1887
@@ -0,0 +1,122 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle. All Rights Reserved.
+#
+# FS QA Test 1887
+#
+# Functional testing for xfs_property the wrapper script.
+#
+. ./common/preamble
+_begin_fstest auto fsproperties
+
+. ./common/filter
+. ./common/attr
+
+_require_test
+_require_attrs
+_require_command "$XFS_PROPERTY_PROG" xfs_property
+_require_xfs_io_command listfsprops # actually detect support
+
+_cleanup()
+
+{
+ cd /
+ rm -r -f $tmp.*
+ rm -f $TEST_DIR/$seq.somefile
+ rm -r -f $TEST_DIR/$seq.somedir
+ test -n "$propname" && $ATTR_PROG -R -r $propname $TEST_DEV &>/dev/null
+}
+
+filter_inum()
+{
+ sed -e 's/inode [0-9]*/inode XXX/g'
+}
+
+propname="fakeproperty" # must not be an actual property
+propval="1721943742"
+longpropname="$(perl -e 'print "x" x 300;')"
+longpropval="$(perl -e 'print "x" x 80000;')"
+
+echo "*** OFFLINE XFS_PROPERTY TEST ***"
+
+_test_unmount
+
+echo empty get property
+$XFS_PROPERTY_PROG $TEST_DEV get "$propname" 2>&1 | filter_inum
+
+echo pointless remove property
+$XFS_PROPERTY_PROG $TEST_DEV remove "$propname" 2>&1 | filter_inum
+
+echo list property
+$XFS_PROPERTY_PROG $TEST_DEV list | grep $propname
+
+echo set property
+$XFS_PROPERTY_PROG $TEST_DEV set "$propname=$propval"
+
+echo list property
+$XFS_PROPERTY_PROG $TEST_DEV list | grep $propname
+
+echo dump xattrs
+$ATTR_PROG -R -l $TEST_DEV | grep $propname | _filter_test_dir
+
+echo get property
+$XFS_PROPERTY_PROG $TEST_DEV get "$propname"
+
+echo list property
+$XFS_PROPERTY_PROG $TEST_DEV list | grep $propname
+
+echo remove property
+$XFS_PROPERTY_PROG $TEST_DEV remove "$propname"
+
+echo pointless remove property
+$XFS_PROPERTY_PROG $TEST_DEV remove "$propname" 2>&1 | filter_inum
+
+echo set too long name
+$XFS_PROPERTY_PROG $TEST_DEV set "$longpropname=$propval"
+
+echo set too long value
+$XFS_PROPERTY_PROG $TEST_DEV set "$propname=$longpropval"
+
+echo "*** ONLINE XFS_PROPERTY TEST ***"
+
+propval=$((propval+1))
+_test_mount
+
+echo empty get property
+$XFS_PROPERTY_PROG $TEST_DIR get "$propname"
+
+echo pointless remove property
+$XFS_PROPERTY_PROG $TEST_DIR remove "$propname"
+
+echo list property
+$XFS_PROPERTY_PROG $TEST_DIR list | grep $propname
+
+echo set property
+$XFS_PROPERTY_PROG $TEST_DIR set "$propname=$propval"
+
+echo list property
+$XFS_PROPERTY_PROG $TEST_DIR list | grep $propname
+
+echo dump xattrs
+$ATTR_PROG -R -l $TEST_DIR | grep $propname | _filter_test_dir
+
+echo get property
+$XFS_PROPERTY_PROG $TEST_DIR get "$propname"
+
+echo list property
+$XFS_PROPERTY_PROG $TEST_DIR list | grep $propname
+
+echo remove property
+$XFS_PROPERTY_PROG $TEST_DIR remove "$propname"
+
+echo pointless remove property
+$XFS_PROPERTY_PROG $TEST_DIR remove "$propname"
+
+echo set too long name
+$XFS_PROPERTY_PROG $TEST_DIR set "$longpropname=$propval"
+
+echo set too long value
+$XFS_PROPERTY_PROG $TEST_DIR set "$propname=$longpropval"
+
+status=0
+exit
diff --git a/tests/xfs/1887.out b/tests/xfs/1887.out
new file mode 100644
index 0000000000..2c27206acf
--- /dev/null
+++ b/tests/xfs/1887.out
@@ -0,0 +1,46 @@
+QA output created by 1887
+*** OFFLINE XFS_PROPERTY TEST ***
+empty get property
+failed to get attr xfs:fakeproperty on inode XXX: No data available
+pointless remove property
+failed to remove attr xfs:fakeproperty from inode XXX: No data available
+list property
+set property
+fakeproperty=1721943742
+list property
+fakeproperty
+dump xattrs
+get property
+fakeproperty=1721943742
+list property
+fakeproperty
+remove property
+pointless remove property
+failed to remove attr xfs:fakeproperty from inode XXX: No data available
+set too long name
+name too long
+set too long value
+xfs:fakeproperty: value too long
+*** ONLINE XFS_PROPERTY TEST ***
+empty get property
+fakeproperty: No data available
+pointless remove property
+fakeproperty: No data available
+list property
+set property
+fakeproperty=1721943743
+list property
+fakeproperty
+dump xattrs
+Attribute "xfs:fakeproperty" has a 10 byte value for TEST_DIR
+get property
+fakeproperty=1721943743
+list property
+fakeproperty
+remove property
+pointless remove property
+fakeproperty: No data available
+set too long name
+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: Invalid argument
+set too long value
+fakeproperty: Invalid argument
diff --git a/tests/xfs/1888 b/tests/xfs/1888
new file mode 100755
index 0000000000..f16a1c0675
--- /dev/null
+++ b/tests/xfs/1888
@@ -0,0 +1,66 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle. All Rights Reserved.
+#
+# FS QA Test 1888
+#
+# Functional testing for mkfs applying autofsck fs property.
+#
+. ./common/preamble
+_begin_fstest auto fsproperties
+
+. ./common/filter
+. ./common/attr
+
+_require_test
+_require_xfs_io_command listfsprops # needed for fs props
+_require_xfs_db_command attr_get
+
+_cleanup()
+
+{
+ cd /
+ rm -r -f $tmp.*
+ rm -f $dummyfile
+ rmdir $dummymnt &>/dev/null
+}
+
+dummyfile=$TEST_DIR/$seq.somefile
+dummymnt=$TEST_DIR/$seq.mount
+
+truncate -s 10g $dummyfile
+mkdir -p $dummymnt
+
+filter_inum()
+{
+ sed -e 's/inode [0-9]*/inode XXX/g'
+}
+
+testme() {
+ local mkfs_args=('-f')
+ local value="$1"
+ test -n "$value" && value="=$value"
+
+ if [ $# -gt 0 ]; then
+ mkfs_args+=('-m' "autofsck$value")
+ fi
+
+ echo "testing ${mkfs_args[*]}" >> $seqres.full
+
+ $MKFS_XFS_PROG "${mkfs_args[@]}" $dummyfile >> $seqres.full || \
+ _notrun "mkfs.xfs ${mkfs_args[*]} failed?"
+
+ $XFS_DB_PROG -x -c 'path /' -c "attr_get -Z autofsck" $dummyfile 2>&1 | filter_inum
+}
+
+testme ''
+testme
+testme none
+testme check
+testme optimize
+testme repair
+testme 0
+testme 1
+
+status=0
+exit
diff --git a/tests/xfs/1888.out b/tests/xfs/1888.out
new file mode 100644
index 0000000000..73857cabd4
--- /dev/null
+++ b/tests/xfs/1888.out
@@ -0,0 +1,9 @@
+QA output created by 1888
+autofsck=repair
+failed to get attr xfs:autofsck on inode XXX: No data available
+autofsck=none
+autofsck=check
+autofsck=optimize
+autofsck=repair
+autofsck=none
+autofsck=repair
diff --git a/tests/xfs/1889 b/tests/xfs/1889
new file mode 100755
index 0000000000..be1f447204
--- /dev/null
+++ b/tests/xfs/1889
@@ -0,0 +1,63 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle. All Rights Reserved.
+#
+# FS QA Test 1889
+#
+# Functional testing for mkfs applying autofsck fs property and xfs_scrub
+# changing its behavior accordingly. Or at least claiming to.
+#
+. ./common/preamble
+_begin_fstest auto fsproperties
+
+. ./common/filter
+. ./common/fuzzy
+
+_require_test
+_require_xfs_io_command listfsprops # needed for fs props
+_require_xfs_db_command attr_get
+_require_scrub
+
+_cleanup()
+
+{
+ cd /
+ rm -r -f $tmp.*
+ umount $dummymnt &>/dev/null
+ rmdir $dummymnt &>/dev/null
+ rm -f $dummyfile
+}
+
+dummyfile=$TEST_DIR/$seq.somefile
+dummymnt=$TEST_DIR/$seq.mount
+
+truncate -s 10g $dummyfile
+mkdir -p $dummymnt
+
+testme() {
+ local mkfs_args=('-f' '-m' "$1")
+
+ echo "testing ${mkfs_args[*]}" >> $seqres.full
+
+ $MKFS_XFS_PROG "${mkfs_args[@]}" $dummyfile >> $seqres.full || \
+ echo "mkfs.xfs ${mkfs_args[*]} failed?"
+
+ _mount -o loop $dummyfile $dummymnt
+ XFS_SCRUB_PHASE=7 $XFS_SCRUB_PROG -d -o autofsck $dummymnt 2>&1 | \
+ grep autofsck | _filter_test_dir | \
+ sed -e 's/\(directive.\).*$/\1/g'
+ umount $dummymnt
+}
+
+# We don't test the absence of an autofsck directive because xfs_scrub behaves
+# differently depending on whether or not mkfs adds rmapbt/pptrs to the fs.
+testme 'autofsck'
+testme 'autofsck=none'
+testme 'autofsck=check'
+testme 'autofsck=optimize'
+testme 'autofsck=repair'
+testme 'autofsck=0'
+testme 'autofsck=1'
+
+status=0
+exit
diff --git a/tests/xfs/1889.out b/tests/xfs/1889.out
new file mode 100644
index 0000000000..fd8123b53a
--- /dev/null
+++ b/tests/xfs/1889.out
@@ -0,0 +1,8 @@
+QA output created by 1889
+Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.
+Info: TEST_DIR/1889.mount: Disabling scrub per autofsck directive.
+Info: TEST_DIR/1889.mount: Checking per autofsck directive.
+Info: TEST_DIR/1889.mount: Optimizing per autofsck directive.
+Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.
+Info: TEST_DIR/1889.mount: Disabling scrub per autofsck directive.
+Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.
^ permalink raw reply related [flat|nested] 21+ messages in thread