[PATCH v2 0/4] Fix failing overlay nonsamefs fstests

[PATCH v2 0/4] Fix failing overlay nonsamefs fstests

[Amir Goldstein]

Eryu,

xino feature was introduced in kernel v4.17-rc1 to fix constant inode
numbers on non samefs setup. This patch set converts the relevant tests
to notrun instead of failing on old kernels and opts-in for xino with
new kernels.

The first patch is a re-factoring of helpers from v1 patches I ended
not using in v2 patches, but seems useful, so you may take it or leave
it out as you wish.

Thanks,
Amir.

Changes since v1:
- Fix fsck run failure in 041
- Fix missing redirect_dir requirement in 043
- Fix check for xino enabled

Amir Goldstein (4):
  common/rc: factor out _require_scratch_fs_option
  overlay/041: fix call to _overlay_check_scratch_dirs
  overlay/043: require and enable redirect_dir
  overlay/04{1,3,4}: enable xino feature

 common/rc         | 29 ++++++++++++++++++++++-------
 tests/overlay/041 | 21 +++++++++++++--------
 tests/overlay/043 | 17 ++++++++++++++---
 tests/overlay/044 | 16 ++++++++++++----
 4 files changed, 61 insertions(+), 22 deletions(-)

-- 
2.7.4

[PATCH v2 1/4] common/rc: factor out _require_scratch_fs_option

[Amir Goldstein]

Factor out _require_scratch_fs_option and _check_scratch_fs_option
from the helper _require_scratch_dax.
Those are generic helpers to test if mount option is supported and
if mount option was enabled.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 common/rc | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/common/rc b/common/rc
index 366489b..78bc59d 100644
--- a/common/rc
+++ b/common/rc
@@ -3036,20 +3036,35 @@ _require_scratch_shutdown()
 	_scratch_unmount
 }
 
-# Does dax mount option work on this dev/fs?
-_require_scratch_dax()
+_check_scratch_fs_option()
+{
+	local opt=$1
+
+	_fs_options $SCRATCH_DEV | grep -qw "$opt"
+}
+
+# Does mount option work on this dev/fs?
+_require_scratch_fs_option()
 {
+	local opt=$1
+
 	_require_scratch
 	_scratch_mkfs > /dev/null 2>&1
-	_try_scratch_mount -o dax || \
-		_notrun "mount $SCRATCH_DEV with dax failed"
-	# Check options to be sure. XFS ignores dax option
+	_try_scratch_mount -o "$opt" || \
+		_notrun "mount $SCRATCH_DEV with $opt failed"
+	# Check options to be sure. For example, XFS ignores dax option
 	# and goes on if dev underneath does not support dax.
-	_fs_options $SCRATCH_DEV | grep -qw "dax" || \
-		_notrun "$SCRATCH_DEV $FSTYP does not support -o dax"
+	_check_scratch_fs_option "$opt" || \
+		_notrun "$SCRATCH_DEV $FSTYP does not support -o $opt"
 	_scratch_unmount
 }
 
+# Does dax mount option work on this dev/fs?
+_require_scratch_dax()
+{
+	_require_scratch_fs_option "dax"
+}
+
 # Does norecovery support by this fs?
 _require_norecovery()
 {
-- 
2.7.4

[PATCH v2 2/4] overlay/041: fix call to _overlay_check_scratch_dirs

[Amir Goldstein]

_overlay_check_scratch_dirs needs to base scratch fs to be mounted,
so only unmount overlay before check.

Remove redundant definition of upperdir/workdir path, which also
uses hardcoded path instead of the config vars OVL_UPPER/OVL_WORK.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/041 | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/overlay/041 b/tests/overlay/041
index 4e72348..f3be746 100755
--- a/tests/overlay/041
+++ b/tests/overlay/041
@@ -168,7 +168,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 [[ $subdir_d == "subdir d" ]] || \
 	echo "Merged dir: Invalid d_ino reported for subdir"
 
-_scratch_unmount
+$UMOUNT_PROG $SCRATCH_MNT
 
 # check overlayfs
 _overlay_check_scratch_dirs $lowerdir $upperdir $workdir
@@ -186,9 +186,6 @@ mkdir -p $lowerdir/test_dir/pure_lower_dir/subdir
 
 _scratch_mkfs
 
-upperdir=$OVL_BASE_SCRATCH_MNT/ovl-upper
-workdir=$OVL_BASE_SCRATCH_MNT/ovl-work
-
 _overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir
 
 # Copy up test_dir
-- 
2.7.4

[PATCH v2 3/4] overlay/043: require and enable redirect_dir

[Amir Goldstein]

This test renames a merge directory so it needs to enable
redirect_dir feature, which is not enabled by default.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/043 | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/overlay/043 b/tests/overlay/043
index 46df686..f70ee7f 100755
--- a/tests/overlay/043
+++ b/tests/overlay/043
@@ -62,6 +62,8 @@ _require_scratch_nocheck
 _require_test
 _require_test_program "af_unix"
 _require_test_program "t_dir_type"
+# Require redirect_dir for renaming a merge directory
+_require_scratch_feature redirect_dir
 
 rm -f $seqres.full
 
@@ -83,7 +85,8 @@ _scratch_mkfs >>$seqres.full 2>&1
 upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
 workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
 
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir
+# Enable redirect_dir for renaming a merge directory
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
 
 FILES="dir file symlink chrdev blkdev fifo socket"
 
@@ -150,13 +153,13 @@ check_inode_numbers $testdir $tmp.after_copyup $tmp.after_move
 
 # Verify that the inode numbers survive a mount cycle
 $UMOUNT_PROG $SCRATCH_MNT
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
 
 # Compare inode numbers before/after mount cycle
 check_inode_numbers $testdir $tmp.after_move $tmp.after_cycle
 
 # check overlayfs
-_overlay_check_scratch_dirs $lowerdir $upperdir $workdir
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
 
 echo "Silence is golden"
 status=0
-- 
2.7.4

[PATCH v2 4/4] overlay/04{1,3,4}: enable xino feature

[Amir Goldstein]

These tests check for constant inode number on copy up with nonsamefs
layer configuration. This problem is fixes only when opting-in with the
xino=on mount option, so let the tests enable the mount option on new
kernels and notrun on old kernels.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/041 | 16 ++++++++++++----
 tests/overlay/043 | 16 ++++++++++++----
 tests/overlay/044 | 16 ++++++++++++----
 3 files changed, 36 insertions(+), 12 deletions(-)

diff --git a/tests/overlay/041 b/tests/overlay/041
index f3be746..78f8179 100755
--- a/tests/overlay/041
+++ b/tests/overlay/041
@@ -70,7 +70,15 @@ _scratch_mkfs
 upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
 workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
 
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir
+# Enabling xino in this test requires that base filesystem inode numbers will
+# not use bit 63 in inode number of the test files, because bit 63 is used by
+# overlayfs to indicate the layer. Let's just assume that this is true for all
+# tested filesystems and if we are wrong, the test may fail.
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o xino=on ||
+	_notrun "mount overlay with xino failed"
+# Not using _check_scratch_fs_option because it uses grep -w
+_fs_options $SCRATCH_DEV | grep -q "xino=on" || \
+	_notrun "failed to enable xino"
 
 test_dir=$SCRATCH_MNT/test_dir/
 
@@ -171,7 +179,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 $UMOUNT_PROG $SCRATCH_MNT
 
 # check overlayfs
-_overlay_check_scratch_dirs $lowerdir $upperdir $workdir
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o xino=on
 
 # Verify pure lower residing in dir which has another lower layer
 middir=$OVL_BASE_TEST_DIR/$seq-ovl-mid
@@ -186,7 +194,7 @@ mkdir -p $lowerdir/test_dir/pure_lower_dir/subdir
 
 _scratch_mkfs
 
-_overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir
+_overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir -o xino=on
 
 # Copy up test_dir
 touch $test_dir/test_file
@@ -209,7 +217,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 	echo "Pure lower in dir which has another lower layer: Invalid d_ino reported for subdir"
 
 # check overlayfs
-_overlay_check_scratch_dirs "$middir:$lowerdir" $upperdir $workdir
+_overlay_check_scratch_dirs "$middir:$lowerdir" $upperdir $workdir -o xino=on
 
 echo "Silence is golden"
 status=0
diff --git a/tests/overlay/043 b/tests/overlay/043
index f70ee7f..c865b51 100755
--- a/tests/overlay/043
+++ b/tests/overlay/043
@@ -85,8 +85,16 @@ _scratch_mkfs >>$seqres.full 2>&1
 upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
 workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
 
-# Enable redirect_dir for renaming a merge directory
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
+# Enable redirect_dir for renaming a merge directory.
+# Enabling xino in this test requires that base filesystem inode numbers will
+# not use bit 63 in inode number of the test files, because bit 63 is used by
+# overlayfs to indicate the layer. Let's just assume that this is true for all
+# tested filesystems and if we are wrong, the test may fail.
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on || \
+	_notrun "mount overlay with xino failed"
+# Not using _check_scratch_fs_option because it uses grep -w
+_fs_options $SCRATCH_DEV | grep -q "xino=on" || \
+	_notrun "failed to enable xino"
 
 FILES="dir file symlink chrdev blkdev fifo socket"
 
@@ -153,13 +161,13 @@ check_inode_numbers $testdir $tmp.after_copyup $tmp.after_move
 
 # Verify that the inode numbers survive a mount cycle
 $UMOUNT_PROG $SCRATCH_MNT
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on
 
 # Compare inode numbers before/after mount cycle
 check_inode_numbers $testdir $tmp.after_move $tmp.after_cycle
 
 # check overlayfs
-_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on
 
 echo "Silence is golden"
 status=0
diff --git a/tests/overlay/044 b/tests/overlay/044
index 2ab3035..bcc30ac 100755
--- a/tests/overlay/044
+++ b/tests/overlay/044
@@ -99,8 +99,16 @@ _scratch_mkfs >>$seqres.full 2>&1
 upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
 workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
 
-# Enable overlay index feature to prevent breaking hardlinks on copy up
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on
+# Enable overlay index feature to prevent breaking hardlinks on copy up.
+# Enabling xino in this test requires that base filesystem inode numbers will
+# not use bit 63 in inode number of the test files, because bit 63 is used by
+# overlayfs to indicate the layer. Let's just assume that this is true for all
+# tested filesystems and if we are wrong, the test may fail.
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on,xino=on || \
+	_notrun "mount overlay with xino failed"
+# Not using _check_scratch_fs_option because it uses grep -w
+_fs_options $SCRATCH_DEV | grep -q "xino=on" || \
+	_notrun "failed to enable xino"
 
 rm -f $tmp.*
 
@@ -124,8 +132,8 @@ check_ino_nlink $SCRATCH_MNT $tmp.before $tmp.after_one
 
 # Verify that the hardlinks survive a mount cycle
 $UMOUNT_PROG $SCRATCH_MNT
-_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o index=on
-_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on
+_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
+_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
 
 echo "== After mount cycle =="
 cat $FILES
-- 
2.7.4

Re: [PATCH v2 4/4] overlay/04{1,3,4}: enable xino feature

[Eryu Guan]

On Wed, Apr 25, 2018 at 11:53:27PM -0600, Amir Goldstein wrote:
> These tests check for constant inode number on copy up with nonsamefs
> layer configuration. This problem is fixes only when opting-in with the
> xino=on mount option, so let the tests enable the mount option on new
> kernels and notrun on old kernels.
> 
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
>  tests/overlay/041 | 16 ++++++++++++----
>  tests/overlay/043 | 16 ++++++++++++----
>  tests/overlay/044 | 16 ++++++++++++----
>  3 files changed, 36 insertions(+), 12 deletions(-)
> 
> diff --git a/tests/overlay/041 b/tests/overlay/041
> index f3be746..78f8179 100755
> --- a/tests/overlay/041
> +++ b/tests/overlay/041
> @@ -70,7 +70,15 @@ _scratch_mkfs
>  upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
>  workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
>  
> -_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir
> +# Enabling xino in this test requires that base filesystem inode numbers will
> +# not use bit 63 in inode number of the test files, because bit 63 is used by
> +# overlayfs to indicate the layer. Let's just assume that this is true for all
> +# tested filesystems and if we are wrong, the test may fail.
> +_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o xino=on ||
> +	_notrun "mount overlay with xino failed"

The _notrun message seems to indicate there's a failure (I can see that
it's copied from _require_scratch_dax), which may confuse users. I can
change it to something like below on commit, if you have better messages
please advise :)

	_notrun "cannot mount overlay with xino=on option"

> +# Not using _check_scratch_fs_option because it uses grep -w
> +_fs_options $SCRATCH_DEV | grep -q "xino=on" || \
> +	_notrun "failed to enable xino"

	_notrun "cannot enable xino feature"

Thanks,
Eryu

>  
>  test_dir=$SCRATCH_MNT/test_dir/
>  
> @@ -171,7 +179,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
>  $UMOUNT_PROG $SCRATCH_MNT
>  
>  # check overlayfs
> -_overlay_check_scratch_dirs $lowerdir $upperdir $workdir
> +_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o xino=on
>  
>  # Verify pure lower residing in dir which has another lower layer
>  middir=$OVL_BASE_TEST_DIR/$seq-ovl-mid
> @@ -186,7 +194,7 @@ mkdir -p $lowerdir/test_dir/pure_lower_dir/subdir
>  
>  _scratch_mkfs
>  
> -_overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir
> +_overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir -o xino=on
>  
>  # Copy up test_dir
>  touch $test_dir/test_file
> @@ -209,7 +217,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
>  	echo "Pure lower in dir which has another lower layer: Invalid d_ino reported for subdir"
>  
>  # check overlayfs
> -_overlay_check_scratch_dirs "$middir:$lowerdir" $upperdir $workdir
> +_overlay_check_scratch_dirs "$middir:$lowerdir" $upperdir $workdir -o xino=on
>  
>  echo "Silence is golden"
>  status=0
> diff --git a/tests/overlay/043 b/tests/overlay/043
> index f70ee7f..c865b51 100755
> --- a/tests/overlay/043
> +++ b/tests/overlay/043
> @@ -85,8 +85,16 @@ _scratch_mkfs >>$seqres.full 2>&1
>  upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
>  workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
>  
> -# Enable redirect_dir for renaming a merge directory
> -_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
> +# Enable redirect_dir for renaming a merge directory.
> +# Enabling xino in this test requires that base filesystem inode numbers will
> +# not use bit 63 in inode number of the test files, because bit 63 is used by
> +# overlayfs to indicate the layer. Let's just assume that this is true for all
> +# tested filesystems and if we are wrong, the test may fail.
> +_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on || \
> +	_notrun "mount overlay with xino failed"
> +# Not using _check_scratch_fs_option because it uses grep -w
> +_fs_options $SCRATCH_DEV | grep -q "xino=on" || \
> +	_notrun "failed to enable xino"
>  
>  FILES="dir file symlink chrdev blkdev fifo socket"
>  
> @@ -153,13 +161,13 @@ check_inode_numbers $testdir $tmp.after_copyup $tmp.after_move
>  
>  # Verify that the inode numbers survive a mount cycle
>  $UMOUNT_PROG $SCRATCH_MNT
> -_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
> +_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on
>  
>  # Compare inode numbers before/after mount cycle
>  check_inode_numbers $testdir $tmp.after_move $tmp.after_cycle
>  
>  # check overlayfs
> -_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o redirect_dir=on
> +_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on
>  
>  echo "Silence is golden"
>  status=0
> diff --git a/tests/overlay/044 b/tests/overlay/044
> index 2ab3035..bcc30ac 100755
> --- a/tests/overlay/044
> +++ b/tests/overlay/044
> @@ -99,8 +99,16 @@ _scratch_mkfs >>$seqres.full 2>&1
>  upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
>  workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
>  
> -# Enable overlay index feature to prevent breaking hardlinks on copy up
> -_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on
> +# Enable overlay index feature to prevent breaking hardlinks on copy up.
> +# Enabling xino in this test requires that base filesystem inode numbers will
> +# not use bit 63 in inode number of the test files, because bit 63 is used by
> +# overlayfs to indicate the layer. Let's just assume that this is true for all
> +# tested filesystems and if we are wrong, the test may fail.
> +_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on,xino=on || \
> +	_notrun "mount overlay with xino failed"
> +# Not using _check_scratch_fs_option because it uses grep -w
> +_fs_options $SCRATCH_DEV | grep -q "xino=on" || \
> +	_notrun "failed to enable xino"
>  
>  rm -f $tmp.*
>  
> @@ -124,8 +132,8 @@ check_ino_nlink $SCRATCH_MNT $tmp.before $tmp.after_one
>  
>  # Verify that the hardlinks survive a mount cycle
>  $UMOUNT_PROG $SCRATCH_MNT
> -_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o index=on
> -_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on
> +_overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
> +_overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
>  
>  echo "== After mount cycle =="
>  cat $FILES
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/4] Fix failing overlay nonsamefs fstests

[Eryu Guan]

On Wed, Apr 25, 2018 at 11:53:23PM -0600, Amir Goldstein wrote:
> Eryu,
> 
> xino feature was introduced in kernel v4.17-rc1 to fix constant inode
> numbers on non samefs setup. This patch set converts the relevant tests
> to notrun instead of failing on old kernels and opts-in for xino with
> new kernels.

overlay/04{1,3,4} all passed for me on v4.17-rc2 now, thanks! Looking
forward to passing all the overlay tests :)

> 
> The first patch is a re-factoring of helpers from v1 patches I ended
> not using in v2 patches, but seems useful, so you may take it or leave
> it out as you wish.

I'd drop it for now, and fix the corresponding comments in patch 4/4.

Thanks,
Eryu

P.S.
I'm not using eguan@redhat.com anymore, please use guaneryu@gmail.com :)

> 
> Thanks,
> Amir.
> 
> Changes since v1:
> - Fix fsck run failure in 041
> - Fix missing redirect_dir requirement in 043
> - Fix check for xino enabled
> 
> Amir Goldstein (4):
>   common/rc: factor out _require_scratch_fs_option
>   overlay/041: fix call to _overlay_check_scratch_dirs
>   overlay/043: require and enable redirect_dir
>   overlay/04{1,3,4}: enable xino feature
> 
>  common/rc         | 29 ++++++++++++++++++++++-------
>  tests/overlay/041 | 21 +++++++++++++--------
>  tests/overlay/043 | 17 ++++++++++++++---
>  tests/overlay/044 | 16 ++++++++++++----
>  4 files changed, 61 insertions(+), 22 deletions(-)
> 
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html