[xfstests PATCH v5 0/7] overlay: add fsck.overlay basic tests

[xfstests PATCH v5 0/7] overlay: add fsck.overlay basic tests

[zhangyi (F)]

Hi All:

Here is the fifth version of test cases for the fsck.overlay[1].

[1] https://github.com/hisilicon/overlayfs-progs

Changes since v4:
- Add _overlay_check_scratch_dirs helper to check optionally dirs
  for tests use _require_overlay_scratch_dirs.
- Modify _overlay_check_fs, make sure overlay is not mounted when
  checking filesystem.

-----------

Changes since v3:
- Modify minor suggestions in 202.
- Add _require_overlay_scratch_dirs helper.
- Replace _require_scratch helper in test cases that finished with
  corrupt filesystem and use optionally test dirs.

Changes since v2:
- Modify some minor suggestions from Amir and Vivek, and simplify code
  in all three test cases.
- Fix duplicate redirect xattr test in 202.
- Add impure xattr test case of general merge directory in 'no impure'
  parent directory.
- Fix fs check failure in other overlay fs test cases. PS: The
  multi-lowers problem in overlay/010 also appears in many other tests,
  _require_scratch() will mis-check the pre-defined underlying dirs
  (not the real test dirs). I think they also need to be modified as this
  patch(06/07) does to check the correct dirs, and I can fix them if you
  think necessary.

Changes since v1:

- Fix _check_scratch_fs hook.
- Remove valid/invalid opaque xattr test.
- Add whiteout test cases of valid/invalid whiteouts in opaque/redirect
  parent directory.
- Add impure xattr test.

zhangyi (F) (7):
  overlay: add filesystem check helper
  overlay: add fsck.overlay whiteout test
  overlay: add fsck.overlay redirect directory test
  overlay: add fsck.overlay impure xattr test
  overlay/003: fix fs check failure
  overlay: skip check for tests finished with corrupt filesystem
  overlay: replace _require_scratch() in optionally dirs cases

 common/config         |   1 +
 common/overlay        | 145 +++++++++++++++++++++++++++
 common/rc             |   4 +-
 tests/overlay/003     |   1 -
 tests/overlay/005     |   6 +-
 tests/overlay/010     |   6 +-
 tests/overlay/014     |  10 +-
 tests/overlay/019     |   2 +-
 tests/overlay/031     |   2 +-
 tests/overlay/035     |   6 +-
 tests/overlay/036     |   5 +-
 tests/overlay/037     |   6 +-
 tests/overlay/038     |  10 +-
 tests/overlay/041     |  10 +-
 tests/overlay/043     |   6 +-
 tests/overlay/044     |   6 +-
 tests/overlay/201     | 232 +++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/201.out |  10 ++
 tests/overlay/202     | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/202.out |  11 +++
 tests/overlay/203     | 143 +++++++++++++++++++++++++++
 tests/overlay/203.out |   4 +
 tests/overlay/group   |   3 +
 23 files changed, 883 insertions(+), 15 deletions(-)
 create mode 100755 tests/overlay/201
 create mode 100644 tests/overlay/201.out
 create mode 100755 tests/overlay/202
 create mode 100644 tests/overlay/202.out
 create mode 100755 tests/overlay/203
 create mode 100644 tests/overlay/203.out

-- 
2.5.0

[xfstests PATCH v5 1/7] overlay: add filesystem check helper

[zhangyi (F)]

Add filesystem check helpers for the upcoming fsck.overlay utility.
Hook these helpers to _check_test_fs and _check_scratch_fs for constants
underlying dirs of overlay filesystem, and introduce scratch check helpers
for optionally lower/upper/work dirs. These helpers works only if
fsck.overlay exists.

[ _check_test_fs/_check_scratch_fs part picked from Amir's patch, thanks ]

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
---
 common/config  |   1 +
 common/overlay | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 common/rc      |   4 +-
 3 files changed, 148 insertions(+), 2 deletions(-)

diff --git a/common/config b/common/config
index 5f40413..71115bd 100644
--- a/common/config
+++ b/common/config
@@ -236,6 +236,7 @@ case "$HOSTOS" in
         export MKFS_REISER4_PROG="`set_prog_path mkfs.reiser4`"
 	export E2FSCK_PROG="`set_prog_path e2fsck`"
 	export TUNE2FS_PROG="`set_prog_path tune2fs`"
+	export FSCK_OVERLAY_PROG="`set_prog_path fsck.overlay`"
         ;;
 esac
 
diff --git a/common/overlay b/common/overlay
index 1da4ab1..c611a49 100644
--- a/common/overlay
+++ b/common/overlay
@@ -151,3 +151,148 @@ _require_scratch_overlay_feature()
 	        _notrun "${FSTYP} feature '${feature}' cannot be enabled on ${SCRATCH_DEV}"
 	_scratch_unmount
 }
+
+# Require the same scratch device as _require_scratch, but do not check
+# the constants OVL_LOWER/OVL_UPPER/OVL_WORK dirs, should use together
+# with optionally lower/upper/work dirs and do check explicitly after test.
+_require_overlay_scratch_dirs()
+{
+	_require_scratch_nocheck
+}
+
+_overlay_fsck_dirs()
+{
+	local lowerdir=$1
+	local upperdir=$2
+	local workdir=$3
+	local options=$4
+
+	[[ ! -x "$FSCK_OVERLAY_PROG" ]] && return 0
+
+	$FSCK_OVERLAY_PROG -o lowerdir=$lowerdir -o upperdir=$upperdir \
+			   -o workdir=$workdir $options
+}
+
+_overlay_check_dirs()
+{
+	local lowerdir=$1
+	local upperdir=$2
+	local workdir=$3
+	local err=0
+
+	_overlay_fsck_dirs $* $FSCK_OPTIONS >>$tmp.fsck 2>&1
+	if [ $? -ne 0 ]; then
+		_log_err "_overlay_check_fs: overlayfs on $lowerdir,$upperdir,$workdir is inconsistent"
+
+		echo "*** fsck.overlay output ***"	>>$seqres.full
+		cat $tmp.fsck				>>$seqres.full
+		echo "*** end fsck.overlay output"	>>$seqres.full
+
+		echo "*** mount output ***"		>>$seqres.full
+		_mount					>>$seqres.full
+		echo "*** end mount output"		>>$seqres.full
+
+		err=1
+	fi
+	rm -f $tmp.fsck
+
+	return $err
+}
+
+# Check the same mnt/dev of _check_overlay_scratch_fs, but check optionally
+# lower/upper/work dirs of overlay filesystem, should use together with
+# _require_overlay_scratch_dirs
+_overlay_check_scratch_dirs()
+{
+	local lowerdir=$1
+	local upperdir=$2
+	local workdir=$3
+
+	# Need to umount overlay for scratch dir check
+	local type=`_fs_type $OVL_BASE_SCRATCH_MNT`
+
+	# If type is set, overlay is mounted
+	[ "$type" = "$FSTYP" ] && $UMOUNT_PROG $SCRATCH_MNT
+
+	_overlay_check_dirs $lowerdir $upperdir $workdir
+	local ret=$?
+
+	if [ $ret = 0 -a "$type" = "$FSTYP" ]; then
+		# overlay was mounted...
+
+		# If overlayfs have to recover with last mount options,
+		# OVERLAY_MOUNT_OPTIONS should be clarified before last
+		# mount
+		_overlay_scratch_mount_dirs $lowerdir $upperdir \
+			$workdir $OVERLAY_MOUNT_OPTIONS
+		ret=$?
+	fi
+
+	return $ret
+}
+
+_overlay_check_fs()
+{
+	local ovl_mnt=$1
+	local base_dev=$4
+	local base_mnt=$5
+	shift 1
+
+	[ "$FSTYP" = overlay ] || return 0
+
+	# Base fs needs to be mounted to check overlay dirs
+	local mounted=""
+	[ -z "$base_dev" ] || mounted=`_is_mounted $base_dev`
+
+	if [ -z "$mounted" ]; then
+		_overlay_base_mount $*
+	else
+		# Need to umount overlay for dir check
+		local type=`_fs_type $base_mnt`
+
+		# If type is set, overlay is mounted
+		[ "$type" = "$FSTYP" ] && $UMOUNT_PROG $ovl_mnt
+	fi
+
+	_overlay_check_dirs $base_mnt/$OVL_LOWER $base_mnt/$OVL_UPPER \
+			    $base_mnt/$OVL_WORK
+	local ret=$?
+
+	if [ -z "$mounted" ]; then
+		_overlay_base_unmount "$base_dev" "$base_mnt"
+	elif [ $ret = 0 -a "$type" = "$FSTYP" ]; then
+		# overlay was mounted...
+
+		# If overlayfs have to recover with last mount options,
+		# OVERLAY_MOUNT_OPTIONS should be clarified before last
+		# mount
+		_overlay_mount $base_mnt $ovl_mnt $OVERLAY_MOUNT_OPTIONS
+		ret=$?
+	fi
+
+	if [ $ret != 0 ]; then
+		status=1
+		if [ "$iam" != "check" ]; then
+			exit 1
+		fi
+		return 1
+	fi
+
+	return 0
+}
+
+_check_overlay_test_fs()
+{
+	_overlay_check_fs "$TEST_DIR" \
+		OVL_BASE_TEST_DEV OVL_BASE_TEST_DIR \
+		"$OVL_BASE_TEST_DEV" "$OVL_BASE_TEST_DIR" \
+		$TEST_FS_MOUNT_OPTS $SELINUX_MOUNT_OPTIONS
+}
+
+_check_overlay_scratch_fs()
+{
+	_overlay_check_fs "$SCRATCH_MNT" \
+		OVL_BASE_SCRATCH_DEV OVL_BASE_SCRATCH_MNT \
+		"$OVL_BASE_SCRATCH_DEV" "$OVL_BASE_SCRATCH_MNT" \
+		$OVL_BASE_MOUNT_OPTIONS $SELINUX_MOUNT_OPTIONS
+}
diff --git a/common/rc b/common/rc
index 77a4eb4..99aa092 100644
--- a/common/rc
+++ b/common/rc
@@ -2590,7 +2590,7 @@ _check_test_fs()
 	# no way to check consistency for GlusterFS
 	;;
     overlay)
-	# no way to check consistency for overlay
+	_check_overlay_test_fs
 	;;
     pvfs2)
 	;;
@@ -2648,7 +2648,7 @@ _check_scratch_fs()
 	# no way to check consistency for GlusterFS
 	;;
     overlay)
-	# no way to check consistency for overlay
+	_check_overlay_scratch_fs
 	;;
     pvfs2)
 	;;
-- 
2.5.0

[xfstests PATCH v5 2/7] overlay: add fsck.overlay whiteout test

[zhangyi (F)]

Add fsck.overlay test case to test it how to deal with orphan/valid
whiteouts in underlying directories of overlayfs.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/201     | 232 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/201.out |  10 +++
 tests/overlay/group   |   1 +
 3 files changed, 243 insertions(+)
 create mode 100755 tests/overlay/201
 create mode 100644 tests/overlay/201.out

diff --git a/tests/overlay/201 b/tests/overlay/201
new file mode 100755
index 0000000..3198c14
--- /dev/null
+++ b/tests/overlay/201
@@ -0,0 +1,232 @@
+#! /bin/bash
+# FS QA Test 201
+#
+# Test fsck.overlay how to deal with whiteouts in overlayfs.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Huawei.  All Rights Reserved.
+# Author: zhangyi (F) <yi.zhang@huawei.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+_require_overlay_scratch_dirs
+_require_command "$FSCK_OVERLAY_PROG" fsck.overlay
+
+# remove all files from previous tests
+_scratch_mkfs
+
+OVL_REDIRECT_XATTR="trusted.overlay.redirect"
+OVL_OPAQUE_XATTR="trusted.overlay.opaque"
+OVL_OPAQUE_XATTR_VAL="y"
+
+# Check whiteout
+check_whiteout()
+{
+	for arg in $*
+	do
+		local ttype=`stat -c "%F:%t,%T" $arg`
+
+		[[ $ttype == "character special file:0,0" ]] || \
+			echo "Valid whiteout removed incorrectly"
+	done
+}
+
+# Create a whiteout
+make_whiteout()
+{
+	for arg in $*
+	do
+		mknod $arg c 0 0
+	done
+}
+
+# Create an opaque directory
+make_opaque_dir()
+{
+	local target=$1
+
+	mkdir -p $target
+	$SETFATTR_PROG -n $OVL_OPAQUE_XATTR -v $OVL_OPAQUE_XATTR_VAL $target
+}
+
+# Create a redirect directory
+make_redirect_dir()
+{
+	local target=$1
+	local value=$2
+
+	mkdir -p $target
+	$SETFATTR_PROG -n $OVL_REDIRECT_XATTR -v $value $target
+}
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+
+make_test_dirs()
+{
+	rm -rf $lowerdir $lowerdir2 $upperdir $workdir
+	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir
+}
+
+# Test orphan whiteout in lower and upper layer, should remove
+echo "+ Orphan whiteout"
+make_test_dirs
+make_whiteout $lowerdir/foo $upperdir/{foo,bar}
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+ls $lowerdir
+ls $upperdir
+
+# Test valid whiteout covering lower target, should not remove
+echo "+ Valid whiteout"
+make_test_dirs
+touch $lowerdir2/{foo,bar}
+make_whiteout $upperdir/foo $lowerdir/bar
+
+_overlay_fsck_dirs "$lowerdir:$lowerdir2" $upperdir $workdir -p >> \
+	 $seqres.full 2>&1 || _fail "fsck should not fail"
+check_whiteout $upperdir/foo $lowerdir/bar
+
+# Test orphan whiteout in opaque directory, should remove
+echo "+ Orphan whiteout(2)"
+make_test_dirs
+mkdir $lowerdir/testdir
+touch $lowerdir/testdir/foo
+make_opaque_dir $upperdir/testdir
+make_whiteout $upperdir/testdir/foo
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+ls $upperdir/testdir
+
+# Test orphan whiteout whose parent path is not an merged directory,
+# should remove
+echo "+ Orphan whiteout(3)"
+make_test_dirs
+mkdir $lowerdir2/{testdir1,testdir2,testdir3}
+touch $lowerdir2/{testdir1/foo,testdir2/foo,testdir3/foo}
+mkdir $upperdir/{testdir1,testdir2,testdir3,testdir4}
+touch $lowerdir/testdir1
+make_whiteout $lowerdir/testdir2
+make_opaque_dir $lowerdir/testdir3
+make_whiteout $upperdir/{testdir1/foo,/testdir2/foo,testdir3/foo,testdir4/foo}
+
+_overlay_fsck_dirs "$lowerdir:$lowerdir2" $upperdir $workdir -p >> \
+	$seqres.full 2>&1 || _fail "fsck should not fail"
+ls $upperdir/testdir1
+ls $upperdir/testdir2
+ls $upperdir/testdir3
+ls $upperdir/testdir4
+
+# Test orphan whiteout in redirect directory, should remove
+echo "+ Orphan whiteout(4)"
+make_test_dirs
+mkdir $lowerdir/{testdir,origin}
+touch $lowerdir/testdir/foo
+make_redirect_dir $upperdir/testdir "origin"
+make_whiteout $upperdir/testdir/foo
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+ls $upperdir/testdir
+
+# Test valid whiteout in redirect directory cover file in lower
+# redirect origin directory, should not remove
+echo "+ Valid whiteout(2)"
+make_test_dirs
+mkdir $lowerdir/origin
+touch $lowerdir/origin/foo
+make_redirect_dir $upperdir/testdir "origin"
+make_whiteout $upperdir/testdir/foo
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_whiteout $upperdir/testdir/foo
+
+# Test valid whiteout covering lower target whose parent directory
+# merge with a redirect directory in the middle layer, should not remove.
+echo "+ Valid whiteout(3)"
+make_test_dirs
+mkdir -p $lowerdir2/origin/subdir
+touch $lowerdir2/origin/subdir/foo
+make_redirect_dir $lowerdir/testdir "origin"
+mkdir -p $upperdir/testdir/subdir
+make_whiteout $upperdir/testdir/subdir/foo
+
+_overlay_fsck_dirs "$lowerdir:$lowerdir2" $upperdir $workdir -p \
+	>> $seqres.full 2>&1 || _fail "fsck should not fail"
+check_whiteout $upperdir/testdir/subdir/foo
+
+# Test invalid whiteout in opaque subdirectory in a redirect directory,
+# should remove
+echo "+ Orphan whiteout(5)"
+make_test_dirs
+mkdir -p $lowerdir/origin/subdir
+touch $lowerdir/origin/subdir/foo
+make_redirect_dir $upperdir/testdir "origin"
+make_opaque_dir $upperdir/testdir/subdir
+make_whiteout $upperdir/testdir/subdir/foo
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+ls $upperdir/testdir/subdir
+
+# Test valid whiteout in reidrect subdirectory in a opaque directory
+# covering lower target, should not remove
+echo "+ Valid whiteout(4)"
+make_test_dirs
+mkdir $lowerdir/origin
+touch $lowerdir/origin/foo
+make_opaque_dir $upperdir/testdir
+make_redirect_dir $upperdir/testdir/subdir "/origin"
+make_whiteout $upperdir/testdir/subdir/foo
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+        _fail "fsck should not fail"
+check_whiteout $upperdir/testdir/subdir/foo
+
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/201.out b/tests/overlay/201.out
new file mode 100644
index 0000000..157bb85
--- /dev/null
+++ b/tests/overlay/201.out
@@ -0,0 +1,10 @@
+QA output created by 201
++ Orphan whiteout
++ Valid whiteout
++ Orphan whiteout(2)
++ Orphan whiteout(3)
++ Orphan whiteout(4)
++ Valid whiteout(2)
++ Valid whiteout(3)
++ Orphan whiteout(5)
++ Valid whiteout(4)
diff --git a/tests/overlay/group b/tests/overlay/group
index 7e541e4..7c5fcbb 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -49,3 +49,4 @@
 044 auto quick copyup hardlink nonsamefs
 047 auto quick copyup hardlink
 048 auto quick copyup hardlink
+201 auto quick fsck
-- 
2.5.0

[xfstests PATCH v5 3/7] overlay: add fsck.overlay redirect directory test

[zhangyi (F)]

Add fsck.overlay test case to test it how to deal with invalid/valid/
duplicate redirect xattr in underlying directories of overlayfs.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/202     | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/202.out |  11 +++
 tests/overlay/group   |   1 +
 3 files changed, 281 insertions(+)
 create mode 100755 tests/overlay/202
 create mode 100644 tests/overlay/202.out

diff --git a/tests/overlay/202 b/tests/overlay/202
new file mode 100755
index 0000000..43e17bc
--- /dev/null
+++ b/tests/overlay/202
@@ -0,0 +1,269 @@
+#! /bin/bash
+# FS QA Test 202
+#
+# Test fsck.overlay how to deal with redirect xattr in overlayfs.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Huawei.  All Rights Reserved.
+# Author: zhangyi (F) <yi.zhang@huawei.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+_require_overlay_scratch_dirs
+_require_attrs
+_require_command "$FSCK_OVERLAY_PROG" fsck.overlay
+
+# remove all files from previous tests
+_scratch_mkfs
+
+OVL_REDIRECT_XATTR="trusted.overlay.redirect"
+OVL_OPAQUE_XATTR="trusted.overlay.opaque"
+OVL_OPAQUE_XATTR_VAL="y"
+
+# Create a redirect directory
+make_redirect_dir()
+{
+	local target=$1
+	local value=$2
+
+	mkdir -p $target
+	$SETFATTR_PROG -n $OVL_REDIRECT_XATTR -v $value $target
+}
+
+# Check redirect xattr
+check_redirect()
+{
+	local target=$1
+	local expect=$2
+
+	value=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_REDIRECT_XATTR $target)
+
+	[[ $value == $expect ]]	|| echo "Redirect xattr incorrect"
+}
+
+check_no_redirect()
+{
+	local target=$1
+
+	value=$($GETFATTR_PROG --absolute-names -d -m \
+		$OVL_REDIRECT_XATTR $target)
+
+	[[ -z "$value" ]] || echo "Redirect xattr not empty"
+}
+
+# Check opaque xattr
+check_opaque()
+{
+	local target=$1
+
+	value=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_OPAQUE_XATTR $target)
+
+	[[ $value == $OVL_OPAQUE_XATTR_VAL ]] || echo "Opaque xattr incorrect"
+}
+
+# Create a whiteout
+make_whiteout()
+{
+	for arg in $*
+	do
+		mknod $arg c 0 0
+	done
+}
+
+# Check whiteout
+check_whiteout()
+{
+	for arg in $*
+	do
+		local ttype=`stat -c "%F:%t,%T" $arg`
+
+		[[ $ttype == "character special file:0,0" ]] || \
+			echo "Valid whiteout removed incorrectly"
+	done
+}
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+
+make_test_dirs()
+{
+	rm -rf $lowerdir $lowerdir2 $upperdir $workdir
+	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir
+}
+
+# Test invalid redirect xattr point to a nonexistent origin, should remove
+echo "+ Invalid redirect"
+make_test_dirs
+make_redirect_dir $upperdir/testdir "invalid"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_no_redirect $upperdir/testdir
+
+# Test invalid redirect xattr point to a file origin, should remove
+echo "+ Invalid redirect(2)"
+make_test_dirs
+touch $lowerdir/origin
+make_redirect_dir $upperdir/testdir "origin"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_no_redirect $upperdir/testdir
+
+# Test valid redirect xattr point to a directory origin in the same directory,
+# should not remove
+echo "+ Valid redirect"
+make_test_dirs
+mkdir $lowerdir/origin
+make_whiteout $upperdir/origin
+make_redirect_dir $upperdir/testdir "origin"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_redirect $upperdir/testdir "origin"
+
+# Test valid redirect xattr point to a directory origin in different directories
+# should not remove
+echo "+ Valid redirect(2)"
+make_test_dirs
+mkdir $lowerdir/origin
+make_whiteout $upperdir/origin
+make_redirect_dir $upperdir/testdir1/testdir2 "/origin"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_redirect $upperdir/testdir1/testdir2 "/origin"
+
+# Test valid redirect xattr but missing whiteout to cover lower target,
+# should fix whiteout
+echo "+ Missing whiteout"
+make_test_dirs
+mkdir $lowerdir/origin
+make_redirect_dir $upperdir/testdir "origin"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_redirect $upperdir/testdir "origin"
+check_whiteout $upperdir/origin
+
+# Test valid redirect xattrs exchanged by rename, should not remove
+echo "+ Valid redirect(3)"
+make_test_dirs
+mkdir $lowerdir/{testdir1,testdir2}
+make_redirect_dir $upperdir/testdir1 "testdir2"
+make_redirect_dir $upperdir/testdir2 "testdir1"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_redirect $upperdir/testdir1 "testdir2"
+check_redirect $upperdir/testdir2 "testdir1"
+
+# Test invalid redirect xattr with lower same name directory exists,
+# should remove invalid redirect xattr and set opaque in yes mode
+echo "+ Invalid redirect(3)"
+make_test_dirs
+mkdir $lowerdir/testdir
+make_redirect_dir $upperdir/testdir "invalid"
+
+# Question get yes answer: Should set opaque dir ?
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -y >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_no_redirect $upperdir/testdir
+check_opaque $upperdir/testdir
+
+# Test duplicate redirect xattrs point to one origin, should fail in
+# auto mode, and should remove either of the duplicates in yes mode
+echo "+ Duplicate redirect"
+make_test_dirs
+mkdir $lowerdir2/origin
+make_redirect_dir $lowerdir/testdir1 "origin"
+make_redirect_dir $lowerdir/testdir2 "origin"
+make_redirect_dir $upperdir/testdir3 "origin"
+
+_overlay_fsck_dirs "$lowerdir:$lowerdir2" $upperdir $workdir -p >> \
+	$seqres.full 2>&1 && _fail "fsck should fail"
+
+# Question get yes answer: Duplicate redirect directory, remove xattr ?
+_overlay_fsck_dirs "$lowerdir:$lowerdir2" $upperdir $workdir -y >> \
+	$seqres.full 2>&1 || _fail "fsck should not fail"
+redirect_1=`check_redirect $lowerdir/testdir1 "origin" 2>/dev/null`
+redirect_2=`check_redirect $lowerdir/testdir2 "origin" 2>/dev/null`
+[[ $redirect_1 == $redirect_2 ]] && echo "Redirect xattr incorrect"
+check_no_redirect $upperdir/testdir3
+
+# Test duplicate redirect xattr duplicate with merge directory, should
+# fail in auto mode, and should remove the redirect xattr in yes mode
+echo "+ Duplicate redirect(2)"
+make_test_dirs
+mkdir $lowerdir/origin $upperdir/origin
+make_redirect_dir $upperdir/testdir "origin"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 && \
+	_fail "fsck should fail"
+
+# Question get yes answer: Duplicate redirect directory, remove xattr ?
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -y >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_no_redirect $upperdir/testdir
+
+# Test duplicate redirect xattr with lower same name directory exists,
+# should remove the duplicate redirect xattr and set opaque in yes mode
+echo "+ Duplicate redirect(3)"
+make_test_dirs
+mkdir $lowerdir/{origin,testdir} $upperdir/origin
+make_redirect_dir $upperdir/testdir "invalid"
+
+# Question one get yes answer: Duplicate redirect directory, remove xattr?
+# Question two get yes answer: Should set opaque dir ?
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -y >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_no_redirect $upperdir/testdir
+check_opaque $upperdir/testdir
+
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/202.out b/tests/overlay/202.out
new file mode 100644
index 0000000..99d2b32
--- /dev/null
+++ b/tests/overlay/202.out
@@ -0,0 +1,11 @@
+QA output created by 202
++ Invalid redirect
++ Invalid redirect(2)
++ Valid redirect
++ Valid redirect(2)
++ Missing whiteout
++ Valid redirect(3)
++ Invalid redirect(3)
++ Duplicate redirect
++ Duplicate redirect(2)
++ Duplicate redirect(3)
diff --git a/tests/overlay/group b/tests/overlay/group
index 7c5fcbb..e39b5e0 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -50,3 +50,4 @@
 047 auto quick copyup hardlink
 048 auto quick copyup hardlink
 201 auto quick fsck
+202 auto quick fsck
-- 
2.5.0

[xfstests PATCH v5 4/7] overlay: add fsck.overlay impure xattr test

[zhangyi (F)]

Add fsck.overlay test case to test it how to deal with impure
xattr in underlying directories of overlayfs.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/203     | 143 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/203.out |   4 ++
 tests/overlay/group   |   1 +
 3 files changed, 148 insertions(+)
 create mode 100755 tests/overlay/203
 create mode 100644 tests/overlay/203.out

diff --git a/tests/overlay/203 b/tests/overlay/203
new file mode 100755
index 0000000..75b2097
--- /dev/null
+++ b/tests/overlay/203
@@ -0,0 +1,143 @@
+#! /bin/bash
+# FS QA Test 203
+#
+# Test fsck.overlay how to deal with impure xattr in overlayfs.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Huawei.  All Rights Reserved.
+# Author: zhangyi (F) <yi.zhang@huawei.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+_require_overlay_scratch_dirs
+_require_command "$FSCK_OVERLAY_PROG" fsck.overlay
+
+# remove all files from previous tests
+_scratch_mkfs
+
+OVL_REDIRECT_XATTR="trusted.overlay.redirect"
+OVL_IMPURE_XATTR="trusted.overlay.impure"
+OVL_IMPURE_XATTR_VAL="y"
+
+# Create a redirect directory
+make_redirect_dir()
+{
+	local target=$1
+	local value=$2
+
+	mkdir -p $target
+	$SETFATTR_PROG -n $OVL_REDIRECT_XATTR -v $value $target
+}
+
+# Remove impure xattr
+remove_impure()
+{
+	local target=$1
+
+	$SETFATTR_PROG -x $OVL_IMPURE_XATTR $target
+}
+
+# Check impure xattr
+check_impure()
+{
+	local target=$1
+
+	value=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_IMPURE_XATTR $target)
+
+	[[ $value == $OVL_IMPURE_XATTR_VAL ]] || echo "Missing impure xattr"
+}
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+
+make_test_dirs()
+{
+	rm -rf $lowerdir $lowerdir2 $upperdir $workdir
+	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir
+}
+
+# Test missing impure xattr in directory which has origin targets, should fix
+echo "+ Missing impure"
+make_test_dirs
+mkdir $lowerdir/{testdir1,testdir2}
+mkdir $upperdir/{testdir1,testdir2}
+touch $lowerdir/testdir1/foo
+mkdir $lowerdir/testdir2/subdir
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir
+touch $SCRATCH_MNT/testdir1/foo
+touch $SCRATCH_MNT/testdir2/subdir
+$UMOUNT_PROG $SCRATCH_MNT
+remove_impure $upperdir/testdir1
+remove_impure $upperdir/testdir2
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_impure $upperdir/testdir1
+check_impure $upperdir/testdir2
+
+# Test missing impure xattr in directory which has redirect directories,
+# should fix
+echo "+ Missing impure(2)"
+make_test_dirs
+mkdir $lowerdir/origin
+make_redirect_dir $upperdir/testdir/subdir "/origin"
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_impure $upperdir/testdir
+
+# Test missing impure xattr in directory which has merge directories,
+# should fix
+echo "+ Missing impure(3)"
+make_test_dirs
+mkdir $lowerdir/testdir $upperdir/testdir
+
+_overlay_fsck_dirs $lowerdir $upperdir $workdir -p >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+check_impure $upperdir
+
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/203.out b/tests/overlay/203.out
new file mode 100644
index 0000000..dd81788
--- /dev/null
+++ b/tests/overlay/203.out
@@ -0,0 +1,4 @@
+QA output created by 203
++ Missing impure
++ Missing impure(2)
++ Missing impure(3)
diff --git a/tests/overlay/group b/tests/overlay/group
index e39b5e0..f99d89e 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -51,3 +51,4 @@
 048 auto quick copyup hardlink
 201 auto quick fsck
 202 auto quick fsck
+203 auto quick fsck
-- 
2.5.0

[xfstests PATCH v5 5/7] overlay/003: fix fs check failure

[zhangyi (F)]

_check_overlay_scratch_fs() will check lowerdir of overlay filesystem,
this case remove this directory after test will lead to check failure,
and it is not really necessary to remove this directory, so keep this
directory.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/003 | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/overlay/003 b/tests/overlay/003
index f980edb..154531e 100755
--- a/tests/overlay/003
+++ b/tests/overlay/003
@@ -92,7 +92,6 @@ ls ${SCRATCH_MNT}/
 # unmount overlayfs but not base fs
 $UMOUNT_PROG $SCRATCH_MNT
 
-rm -rf $lowerdir
 echo "Silence is golden"
 # success, all done
 status=0
-- 
2.5.0

[xfstests PATCH v5 6/7] overlay: skip check for tests finished with corrupt filesystem

[zhangyi (F)]

No post-test check of the overlay dirs is required if case leaves
corrupt filesystem after test. We shoud use _require_scratch_nocheck()
instead of _require_scratch() in these cases.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/019 | 2 +-
 tests/overlay/031 | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/overlay/019 b/tests/overlay/019
index 3e2bc4e..575aaf7 100755
--- a/tests/overlay/019
+++ b/tests/overlay/019
@@ -46,7 +46,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_scratch_nocheck
 
 # Remove all files from previous tests
 _scratch_mkfs
diff --git a/tests/overlay/031 b/tests/overlay/031
index 186b409..90a06af 100755
--- a/tests/overlay/031
+++ b/tests/overlay/031
@@ -67,7 +67,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_scratch_nocheck
 
 # remove all files from previous runs
 _scratch_mkfs
-- 
2.5.0

[xfstests PATCH v5 7/7] overlay: correct scratch dirs check

[zhangyi (F)]

Tests that use _overlay_scratch_mount_dirs instead of _scratch_mount
should use _require_overlay_scratch_dirs instead of _require_scratch.
Those tests can optionally call _overlay_check_scratch_dirs at the end
of the test or before _scratch_umount to umount $SCRATCH_MNT and
run the checker.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
---
 tests/overlay/005 |  6 +++++-
 tests/overlay/010 |  6 +++++-
 tests/overlay/014 | 10 +++++++++-
 tests/overlay/035 |  6 +++++-
 tests/overlay/036 |  5 ++++-
 tests/overlay/037 |  6 +++++-
 tests/overlay/038 | 10 +++++++++-
 tests/overlay/041 | 10 +++++++++-
 tests/overlay/043 |  6 +++++-
 tests/overlay/044 |  6 +++++-
 10 files changed, 61 insertions(+), 10 deletions(-)

diff --git a/tests/overlay/005 b/tests/overlay/005
index 17992a3..f502d3f 100755
--- a/tests/overlay/005
+++ b/tests/overlay/005
@@ -54,7 +54,7 @@ rm -f $seqres.full
 # Modify as appropriate.
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_loop
 
 # Remove all files from previous tests
@@ -102,6 +102,10 @@ $XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \
 # unmount overlayfs
 $UMOUNT_PROG $SCRATCH_MNT
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after test
+_overlay_check_scratch_dirs $lowerd $upperd $workd
+
 # unmount undelying xfs, this tiggers panic if memleak happens
 $UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/uppermnt
 $UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/lowermnt
diff --git a/tests/overlay/010 b/tests/overlay/010
index f55ebec..f3b50d7 100755
--- a/tests/overlay/010
+++ b/tests/overlay/010
@@ -48,7 +48,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 
 # Remove all files from previous tests
 _scratch_mkfs
@@ -70,6 +70,10 @@ mknod $lowerdir2/testdir/a c 0 0
 _overlay_scratch_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir
 rm -rf $SCRATCH_MNT/testdir
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after test
+_overlay_check_scratch_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir
+
 # success, all done
 echo "Silence is golden"
 status=0
diff --git a/tests/overlay/014 b/tests/overlay/014
index 9f308d3..069d8b6 100755
--- a/tests/overlay/014
+++ b/tests/overlay/014
@@ -53,7 +53,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 
 # Remove all files from previous tests
 _scratch_mkfs
@@ -78,6 +78,10 @@ mkdir -p $SCRATCH_MNT/testdir/visibledir
 # unmount overlayfs but not base fs
 $UMOUNT_PROG $SCRATCH_MNT
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after each test
+_overlay_check_scratch_dirs $lowerdir1 $lowerdir2 $workdir2
+
 # mount overlay again, with lowerdir1 and lowerdir2 as multiple lowerdirs,
 # and create a new file in testdir, triggers copyup from lowerdir,
 # copyup should not copy overlayfs private xattr
@@ -90,6 +94,10 @@ $UMOUNT_PROG $SCRATCH_MNT
 _overlay_scratch_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir
 ls $SCRATCH_MNT/testdir
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after each test
+_overlay_check_scratch_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir
+
 # success, all done
 status=0
 exit
diff --git a/tests/overlay/035 b/tests/overlay/035
index 0544774..bc73013 100755
--- a/tests/overlay/035
+++ b/tests/overlay/035
@@ -51,7 +51,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_chattr i
 
 # Remove all files from previous tests
@@ -81,6 +81,10 @@ _overlay_scratch_mount_dirs $lowerdir2 $upperdir $workdir
 touch $SCRATCH_MNT/bar 2>&1 | _filter_scratch
 _scratch_remount rw 2>&1 | _filter_ro_mount
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after test
+_overlay_check_scratch_dirs $lowerdir2 $upperdir $workdir
+
 # success, all done
 status=0
 exit
diff --git a/tests/overlay/036 b/tests/overlay/036
index e0c13ae..ca5adbd 100755
--- a/tests/overlay/036
+++ b/tests/overlay/036
@@ -69,7 +69,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_scratch_feature index
 
 # Remove all files from previous tests
@@ -110,6 +110,9 @@ _overlay_mount_dirs $lowerdir2 $upperdir $workdir2 \
 _overlay_mount_dirs $lowerdir2 $upperdir2 $workdir \
 	    overlay3 $SCRATCH_MNT -oindex=on 2>&1 | _filter_busy_mount
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# but which will not lead to inconsistency after this mount test,
+# so do not check optionally dirs after test
 
 # success, all done
 status=0
diff --git a/tests/overlay/037 b/tests/overlay/037
index 6710dda..60290a1 100755
--- a/tests/overlay/037
+++ b/tests/overlay/037
@@ -55,7 +55,7 @@ rm -f $seqres.full
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_scratch_feature index
 
 # Remove all files from previous tests
@@ -87,6 +87,10 @@ $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
 # Mount overlay with original lowerdir, upperdir, workdir and index=on - expect success
 _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -oindex=on
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# but which will not lead to inconsistency after this mount test,
+# so do not check optionally dirs after test
+
 # success, all done
 status=0
 exit
diff --git a/tests/overlay/038 b/tests/overlay/038
index bd87156..aa20cbc 100755
--- a/tests/overlay/038
+++ b/tests/overlay/038
@@ -46,7 +46,7 @@ _cleanup()
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_attrs
 _require_test_program "t_dir_type"
 
@@ -161,6 +161,10 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 [[ $subdir_d == "subdir d" ]] || \
 	echo "Merged dir: Invalid d_ino reported for subdir"
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check after each test
+_check_scratch_fs
+
 _scratch_unmount
 
 # Verify pure lower residing in dir which has another lower layer
@@ -202,6 +206,10 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 [[ $subdir_d == "subdir d" ]] || \
 	echo "Pure lower in dir which has another lower layer: Invalid d_ino reported for subdir"
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after each test
+_overlay_check_scratch_dirs "$middir:$lowerdir" $upperdir $workdir
+
 echo "Silence is golden"
 status=0
 exit
diff --git a/tests/overlay/041 b/tests/overlay/041
index 4152107..607908a 100755
--- a/tests/overlay/041
+++ b/tests/overlay/041
@@ -48,7 +48,7 @@ _cleanup()
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_test
 _require_attrs
 _require_test_program "t_dir_type"
@@ -167,6 +167,10 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 [[ $subdir_d == "subdir d" ]] || \
 	echo "Merged dir: Invalid d_ino reported for subdir"
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after each test
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir
+
 _scratch_unmount
 
 # Verify pure lower residing in dir which has another lower layer
@@ -208,6 +212,10 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 [[ $subdir_d == "subdir d" ]] || \
 	echo "Pure lower in dir which has another lower layer: Invalid d_ino reported for subdir"
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after each test
+_overlay_check_scratch_dirs "$middir:$lowerdir" $upperdir $workdir
+
 echo "Silence is golden"
 status=0
 exit
diff --git a/tests/overlay/043 b/tests/overlay/043
index 858b6a9..76e0dd9 100755
--- a/tests/overlay/043
+++ b/tests/overlay/043
@@ -56,7 +56,7 @@ _cleanup()
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_test
 _require_test_program "af_unix"
 _require_test_program "t_dir_type"
@@ -155,6 +155,10 @@ _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir \
 # Compare inode numbers before/after mount cycle
 check_inode_numbers $testdir $tmp.after_move $tmp.after_cycle
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after test
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir
+
 echo "Silence is golden"
 status=0
 exit
diff --git a/tests/overlay/044 b/tests/overlay/044
index 9c0ff04..e929068 100755
--- a/tests/overlay/044
+++ b/tests/overlay/044
@@ -49,7 +49,7 @@ _cleanup()
 # real QA test starts here
 _supported_fs overlay
 _supported_os Linux
-_require_scratch
+_require_overlay_scratch_dirs
 _require_test
 _require_scratch_feature index
 _require_test_program "t_dir_type"
@@ -141,5 +141,9 @@ echo "== After write two =="
 cat $FILES
 check_ino_nlink $SCRATCH_MNT $tmp.after_one $tmp.after_two
 
+# We called _require_overlay_scratch_dirs instead of _require_scratch,
+# do check optionally dirs after test
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir
+
 status=0
 exit
-- 
2.5.0

Re: [xfstests PATCH v5 1/7] overlay: add filesystem check helper

[Amir Goldstein]

On Tue, Jan 23, 2018 at 9:34 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
> Add filesystem check helpers for the upcoming fsck.overlay utility.
> Hook these helpers to _check_test_fs and _check_scratch_fs for constants
> underlying dirs of overlay filesystem, and introduce scratch check helpers
> for optionally lower/upper/work dirs. These helpers works only if
> fsck.overlay exists.
>
> [ _check_test_fs/_check_scratch_fs part picked from Amir's patch, thanks ]
>
> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
> ---
>  common/config  |   1 +
>  common/overlay | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  common/rc      |   4 +-
>  3 files changed, 148 insertions(+), 2 deletions(-)
>
> diff --git a/common/config b/common/config
> index 5f40413..71115bd 100644
> --- a/common/config
> +++ b/common/config
> @@ -236,6 +236,7 @@ case "$HOSTOS" in
>          export MKFS_REISER4_PROG="`set_prog_path mkfs.reiser4`"
>         export E2FSCK_PROG="`set_prog_path e2fsck`"
>         export TUNE2FS_PROG="`set_prog_path tune2fs`"
> +       export FSCK_OVERLAY_PROG="`set_prog_path fsck.overlay`"
>          ;;
>  esac
>
> diff --git a/common/overlay b/common/overlay
> index 1da4ab1..c611a49 100644
> --- a/common/overlay
> +++ b/common/overlay
> @@ -151,3 +151,148 @@ _require_scratch_overlay_feature()
>                 _notrun "${FSTYP} feature '${feature}' cannot be enabled on ${SCRATCH_DEV}"
>         _scratch_unmount
>  }
> +
> +# Require the same scratch device as _require_scratch, but do not check
> +# the constants OVL_LOWER/OVL_UPPER/OVL_WORK dirs, should use together
> +# with optionally lower/upper/work dirs and do check explicitly after test.
> +_require_overlay_scratch_dirs()
> +{
> +       _require_scratch_nocheck
> +}
> +
> +_overlay_fsck_dirs()
> +{
> +       local lowerdir=$1
> +       local upperdir=$2
> +       local workdir=$3
> +       local options=$4
> +
> +       [[ ! -x "$FSCK_OVERLAY_PROG" ]] && return 0
> +
> +       $FSCK_OVERLAY_PROG -o lowerdir=$lowerdir -o upperdir=$upperdir \
> +                          -o workdir=$workdir $options
> +}
> +
> +_overlay_check_dirs()
> +{
> +       local lowerdir=$1
> +       local upperdir=$2
> +       local workdir=$3
> +       local err=0
> +
> +       _overlay_fsck_dirs $* $FSCK_OPTIONS >>$tmp.fsck 2>&1
> +       if [ $? -ne 0 ]; then
> +               _log_err "_overlay_check_fs: overlayfs on $lowerdir,$upperdir,$workdir is inconsistent"
> +
> +               echo "*** fsck.overlay output ***"      >>$seqres.full
> +               cat $tmp.fsck                           >>$seqres.full
> +               echo "*** end fsck.overlay output"      >>$seqres.full
> +
> +               echo "*** mount output ***"             >>$seqres.full
> +               _mount                                  >>$seqres.full
> +               echo "*** end mount output"             >>$seqres.full
> +
> +               err=1
> +       fi
> +       rm -f $tmp.fsck
> +
> +       return $err
> +}
> +
> +# Check the same mnt/dev of _check_overlay_scratch_fs, but check optionally
> +# lower/upper/work dirs of overlay filesystem, should use together with
> +# _require_overlay_scratch_dirs
> +_overlay_check_scratch_dirs()
> +{
> +       local lowerdir=$1
> +       local upperdir=$2
> +       local workdir=$3
> +
> +       # Need to umount overlay for scratch dir check
> +       local type=`_fs_type $OVL_BASE_SCRATCH_MNT`

This test is fragile because it relies that the mount device name
specified in mount command was OVL_BASE_SCRATCH_MNT.
That is true for tests using _require_overlay_scratch_dirs, but not
always true for tests using  _overlay_mount_dirs.
For example, you did not end up calling _overlay_check_scratch_dirs
in overlay/036, but for different reason. If you would have called
this helper it would do the wrong thing, because device name used
in this test in 'overlay3'.

One way to deal with this is to check whether $SCRATCH_MNT
is an 'overlay' type mount point (instead of checking device name).

> +
> +       # If type is set, overlay is mounted
> +       [ "$type" = "$FSTYP" ] && $UMOUNT_PROG $SCRATCH_MNT

This is an overlay specific helper, so I rather see [ "$type" = overlay ]

> +
> +       _overlay_check_dirs $lowerdir $upperdir $workdir
> +       local ret=$?
> +
> +       if [ $ret = 0 -a "$type" = "$FSTYP" ]; then

more clear to read if there was an auxiliary variable ovl_mounted

> +               # overlay was mounted...
> +
> +               # If overlayfs have to recover with last mount options,
> +               # OVERLAY_MOUNT_OPTIONS should be clarified before last
> +               # mount
> +               _overlay_scratch_mount_dirs $lowerdir $upperdir \
> +                       $workdir $OVERLAY_MOUNT_OPTIONS


This part is a bit obfuscated in check -overlay, but OVERLAY_MOUNT_OPTIONS
should already be assigned to MOUNT_OPTIONS and included from
_overlay_mount_dirs -> _common_dev_mount_options

If anything you should allow to pass in extra mount option variables
and pass them into _overlay_scratch_mount_dirs in the end ($*)
for tests that would like to continue testing on a mounted overlay after
calling _overlay_check_scratch_dirs and have provide specialized
mount options to _overlay_scratch_mount_dirs.
But _check_generic_filesystem doesn't handle this case, so I don't
think you should handle it as well.

> +               ret=$?
> +       fi
> +
> +       return $ret
> +}
> +
> +_overlay_check_fs()
> +{
> +       local ovl_mnt=$1
> +       local base_dev=$4
> +       local base_mnt=$5
> +       shift 1
> +
> +       [ "$FSTYP" = overlay ] || return 0
> +
> +       # Base fs needs to be mounted to check overlay dirs
> +       local mounted=""
> +       [ -z "$base_dev" ] || mounted=`_is_mounted $base_dev`
> +
> +       if [ -z "$mounted" ]; then
> +               _overlay_base_mount $*
> +       else
> +               # Need to umount overlay for dir check
> +               local type=`_fs_type $base_mnt`
> +
> +               # If type is set, overlay is mounted
> +               [ "$type" = "$FSTYP" ] && $UMOUNT_PROG $ovl_mnt

Same comments as above

> +       fi
> +
> +       _overlay_check_dirs $base_mnt/$OVL_LOWER $base_mnt/$OVL_UPPER \
> +                           $base_mnt/$OVL_WORK
> +       local ret=$?
> +
> +       if [ -z "$mounted" ]; then
> +               _overlay_base_unmount "$base_dev" "$base_mnt"
> +       elif [ $ret = 0 -a "$type" = "$FSTYP" ]; then
> +               # overlay was mounted...
> +
> +               # If overlayfs have to recover with last mount options,
> +               # OVERLAY_MOUNT_OPTIONS should be clarified before last
> +               # mount
> +               _overlay_mount $base_mnt $ovl_mnt $OVERLAY_MOUNT_OPTIONS

Same comments as above

Thanks,
Amir.

Re: [xfstests PATCH v5 1/7] overlay: add filesystem check helper

[zhangyi (F)]

On 2018/1/23 17:16, Amir Goldstein 写道:
> On Tue, Jan 23, 2018 at 9:34 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
>> Add filesystem check helpers for the upcoming fsck.overlay utility.
>> Hook these helpers to _check_test_fs and _check_scratch_fs for constants
>> underlying dirs of overlay filesystem, and introduce scratch check helpers
>> for optionally lower/upper/work dirs. These helpers works only if
>> fsck.overlay exists.
>>
>> [ _check_test_fs/_check_scratch_fs part picked from Amir's patch, thanks ]
>>
>> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
>> ---
>>  common/config  |   1 +
>>  common/overlay | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  common/rc      |   4 +-
>>  3 files changed, 148 insertions(+), 2 deletions(-)
>>
>> diff --git a/common/config b/common/config
>> index 5f40413..71115bd 100644
>> --- a/common/config
>> +++ b/common/config
>> @@ -236,6 +236,7 @@ case "$HOSTOS" in
>>          export MKFS_REISER4_PROG="`set_prog_path mkfs.reiser4`"
>>         export E2FSCK_PROG="`set_prog_path e2fsck`"
>>         export TUNE2FS_PROG="`set_prog_path tune2fs`"
>> +       export FSCK_OVERLAY_PROG="`set_prog_path fsck.overlay`"
>>          ;;
>>  esac
>>
>> diff --git a/common/overlay b/common/overlay
>> index 1da4ab1..c611a49 100644
>> --- a/common/overlay
>> +++ b/common/overlay
>> @@ -151,3 +151,148 @@ _require_scratch_overlay_feature()
>>                 _notrun "${FSTYP} feature '${feature}' cannot be enabled on ${SCRATCH_DEV}"
>>         _scratch_unmount
>>  }
>> +
>> +# Require the same scratch device as _require_scratch, but do not check
>> +# the constants OVL_LOWER/OVL_UPPER/OVL_WORK dirs, should use together
>> +# with optionally lower/upper/work dirs and do check explicitly after test.
>> +_require_overlay_scratch_dirs()
>> +{
>> +       _require_scratch_nocheck
>> +}
>> +
>> +_overlay_fsck_dirs()
>> +{
>> +       local lowerdir=$1
>> +       local upperdir=$2
>> +       local workdir=$3
>> +       local options=$4
>> +
>> +       [[ ! -x "$FSCK_OVERLAY_PROG" ]] && return 0
>> +
>> +       $FSCK_OVERLAY_PROG -o lowerdir=$lowerdir -o upperdir=$upperdir \
>> +                          -o workdir=$workdir $options
>> +}
>> +
>> +_overlay_check_dirs()
>> +{
>> +       local lowerdir=$1
>> +       local upperdir=$2
>> +       local workdir=$3
>> +       local err=0
>> +
>> +       _overlay_fsck_dirs $* $FSCK_OPTIONS >>$tmp.fsck 2>&1
>> +       if [ $? -ne 0 ]; then
>> +               _log_err "_overlay_check_fs: overlayfs on $lowerdir,$upperdir,$workdir is inconsistent"
>> +
>> +               echo "*** fsck.overlay output ***"      >>$seqres.full
>> +               cat $tmp.fsck                           >>$seqres.full
>> +               echo "*** end fsck.overlay output"      >>$seqres.full
>> +
>> +               echo "*** mount output ***"             >>$seqres.full
>> +               _mount                                  >>$seqres.full
>> +               echo "*** end mount output"             >>$seqres.full
>> +
>> +               err=1
>> +       fi
>> +       rm -f $tmp.fsck
>> +
>> +       return $err
>> +}
>> +
>> +# Check the same mnt/dev of _check_overlay_scratch_fs, but check optionally
>> +# lower/upper/work dirs of overlay filesystem, should use together with
>> +# _require_overlay_scratch_dirs
>> +_overlay_check_scratch_dirs()
>> +{
>> +       local lowerdir=$1
>> +       local upperdir=$2
>> +       local workdir=$3
>> +
>> +       # Need to umount overlay for scratch dir check
>> +       local type=`_fs_type $OVL_BASE_SCRATCH_MNT`
> 
> This test is fragile because it relies that the mount device name
> specified in mount command was OVL_BASE_SCRATCH_MNT.
> That is true for tests using _require_overlay_scratch_dirs, but not
> always true for tests using  _overlay_mount_dirs.
> For example, you did not end up calling _overlay_check_scratch_dirs
> in overlay/036, but for different reason. If you would have called
> this helper it would do the wrong thing, because device name used
> in this test in 'overlay3'.
> 
> One way to deal with this is to check whether $SCRATCH_MNT
> is an 'overlay' type mount point (instead of checking device name).
> 
>> +
>> +       # If type is set, overlay is mounted
>> +       [ "$type" = "$FSTYP" ] && $UMOUNT_PROG $SCRATCH_MNT
> 
> This is an overlay specific helper, so I rather see [ "$type" = overlay ]
> 
>> +
>> +       _overlay_check_dirs $lowerdir $upperdir $workdir
>> +       local ret=$?
>> +
>> +       if [ $ret = 0 -a "$type" = "$FSTYP" ]; then
> 
> more clear to read if there was an auxiliary variable ovl_mounted
> 
>> +               # overlay was mounted...
>> +
>> +               # If overlayfs have to recover with last mount options,
>> +               # OVERLAY_MOUNT_OPTIONS should be clarified before last
>> +               # mount
>> +               _overlay_scratch_mount_dirs $lowerdir $upperdir \
>> +                       $workdir $OVERLAY_MOUNT_OPTIONS
> 
> 
> This part is a bit obfuscated in check -overlay, but OVERLAY_MOUNT_OPTIONS
> should already be assigned to MOUNT_OPTIONS and included from
> _overlay_mount_dirs -> _common_dev_mount_options
> 
> If anything you should allow to pass in extra mount option variables
> and pass them into _overlay_scratch_mount_dirs in the end ($*)
> for tests that would like to continue testing on a mounted overlay after
> calling _overlay_check_scratch_dirs and have provide specialized
> mount options to _overlay_scratch_mount_dirs.
> But _check_generic_filesystem doesn't handle this case, so I don't
> think you should handle it as well.

Right, I had planned to use OVERLAY_MOUNT_OPTIONS to pass extra mount options($*)
as overlay/044 does, but it seems not necessary because we always check after test
or subtest. For simplicity, I will remove this option.

Thanks,
Yi.

> 
>> +               ret=$?
>> +       fi
>> +
>> +       return $ret
>> +}
>> +
>> +_overlay_check_fs()
>> +{
>> +       local ovl_mnt=$1
>> +       local base_dev=$4
>> +       local base_mnt=$5
>> +       shift 1
>> +
>> +       [ "$FSTYP" = overlay ] || return 0
>> +
>> +       # Base fs needs to be mounted to check overlay dirs
>> +       local mounted=""
>> +       [ -z "$base_dev" ] || mounted=`_is_mounted $base_dev`
>> +
>> +       if [ -z "$mounted" ]; then
>> +               _overlay_base_mount $*
>> +       else
>> +               # Need to umount overlay for dir check
>> +               local type=`_fs_type $base_mnt`
>> +
>> +               # If type is set, overlay is mounted
>> +               [ "$type" = "$FSTYP" ] && $UMOUNT_PROG $ovl_mnt
> 
> Same comments as above
> 
>> +       fi
>> +
>> +       _overlay_check_dirs $base_mnt/$OVL_LOWER $base_mnt/$OVL_UPPER \
>> +                           $base_mnt/$OVL_WORK
>> +       local ret=$?
>> +
>> +       if [ -z "$mounted" ]; then
>> +               _overlay_base_unmount "$base_dev" "$base_mnt"
>> +       elif [ $ret = 0 -a "$type" = "$FSTYP" ]; then
>> +               # overlay was mounted...
>> +
>> +               # If overlayfs have to recover with last mount options,
>> +               # OVERLAY_MOUNT_OPTIONS should be clarified before last
>> +               # mount
>> +               _overlay_mount $base_mnt $ovl_mnt $OVERLAY_MOUNT_OPTIONS
> 
> Same comments as above
> 
> Thanks,
> Amir.
> 
> .
>