[PATCH 0/3] Overlayfs orphan index cleanup tests

[PATCH 0/3] Overlayfs orphan index cleanup tests

[Amir Goldstein]

Eryu,

Vivek found a bug with the index nlink accounting of the index feature
that was introduced in kernel v4.13.

The bug was there from the first implementation of index and the
tests I wrote to verify nlink accounting (overlay/033,034) did not
cover the bug use case.

The first regression test (overlay/047) demonstrates the effects
of the bug on end user.

The second regression test (overlay/048) adds the missing test coverage
of the existing nlink accounting tests.

The kernel bug was hiding a bug in test overlay/042, so Vivek's
kernel fix broke the test. The last patch fixes the wrong/broken test.

Amir.

Amir Goldstein (3):
  overlay: regression test for hardlink breakage after unlink and mount
  overlay: regression test for orphan index cleanup on mount
  overlay/042: remove wrong check for empty index

 tests/overlay/042     |   4 --
 tests/overlay/047     | 107 ++++++++++++++++++++++++++++++++++++++
 tests/overlay/047.out |  16 ++++++
 tests/overlay/048     | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/048.out |  18 +++++++
 tests/overlay/group   |   2 +
 6 files changed, 284 insertions(+), 4 deletions(-)
 create mode 100755 tests/overlay/047
 create mode 100644 tests/overlay/047.out
 create mode 100755 tests/overlay/048
 create mode 100644 tests/overlay/048.out

-- 
2.7.4

[PATCH 1/3] overlay: regression test for hardlink breakage after unlink and mount cycle

[Amir Goldstein]

- file A and B are hardlinked in lower
- modify A to trigger copy up and index lower
- unlink A and mount cycle
- check that B still contains the modified data

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/047     | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/047.out |  16 ++++++++
 tests/overlay/group   |   1 +
 3 files changed, 124 insertions(+)
 create mode 100755 tests/overlay/047
 create mode 100644 tests/overlay/047.out

diff --git a/tests/overlay/047 b/tests/overlay/047
new file mode 100755
index 0000000..ad1aabe
--- /dev/null
+++ b/tests/overlay/047
@@ -0,0 +1,107 @@
+#! /bin/bash
+# FSQA Test No. 047
+#
+# Test hardlink breakage after unlink and mount cycle
+#
+# - file A and B are hardlinked in lower
+# - modify A to trigger copy up and index lower
+# - unlink A and mount cycle
+# - check that B still contains the modified data
+#
+#-----------------------------------------------------------------------
+#
+# Copyright (C) 2016-2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+_require_scratch_feature index
+
+rm -f $seqres.full
+
+_scratch_mkfs >>$seqres.full 2>&1
+
+# Create 2 hardlinked files in lower
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+mkdir -p $lowerdir
+echo "zero" >> $lowerdir/foo
+ln $lowerdir/foo $lowerdir/bar
+
+
+# Enable overlay index feature to prevent breaking hardlinks on copy up
+_scratch_mount -o index=on
+
+
+rm -f $tmp.*
+
+foo=$SCRATCH_MNT/foo
+bar=$SCRATCH_MNT/bar
+
+FILES="$foo $bar"
+
+echo "== Before copy up =="
+cat $foo
+
+# Modify content of one of the hardlinks
+echo "one" >> $bar
+
+echo "== After write one =="
+cat $foo
+
+# Unlink the copied up hardlink
+rm $bar
+
+echo "== After unlink one =="
+cat $foo
+
+# Verify that the hardlinks survive a mount cycle
+_scratch_cycle_mount index=on
+
+echo "== After mount cycle =="
+cat $foo
+
+# Drop caches to get the copied up hardlink out of cache
+echo 3 > /proc/sys/vm/drop_caches
+
+# Modify content of the other hardlink
+echo "two" >> $foo
+
+echo "== After write two =="
+cat $foo
+
+status=0
+exit
diff --git a/tests/overlay/047.out b/tests/overlay/047.out
new file mode 100644
index 0000000..298e116
--- /dev/null
+++ b/tests/overlay/047.out
@@ -0,0 +1,16 @@
+QA output created by 047
+== Before copy up ==
+zero
+== After write one ==
+zero
+one
+== After unlink one ==
+zero
+one
+== After mount cycle ==
+zero
+one
+== After write two ==
+zero
+one
+two
diff --git a/tests/overlay/group b/tests/overlay/group
index b32c6cd..3e31d2d 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -47,3 +47,4 @@
 042 auto quick copyup hardlink
 043 auto quick copyup nonsamefs
 044 auto quick copyup hardlink nonsamefs
+047 auto quick copyup hardlink
-- 
2.7.4

[PATCH 2/3] overlay: regression test for orphan index cleanup on mount

[Amir Goldstein]

Test nlink accounting of overlay hardlinks with offline modifications.

nlink of overlay inode should account for the union of lower and upper
hardlinks. Orphan index inodes with union nlink 0 should be cleaned on
mount.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/048     | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/048.out |  18 +++++++
 tests/overlay/group   |   1 +
 3 files changed, 160 insertions(+)
 create mode 100755 tests/overlay/048
 create mode 100644 tests/overlay/048.out

diff --git a/tests/overlay/048 b/tests/overlay/048
new file mode 100755
index 0000000..4b2c58f
--- /dev/null
+++ b/tests/overlay/048
@@ -0,0 +1,141 @@
+#! /bin/bash
+# FS QA Test 048
+#
+# Test nlink accounting of overlay hardlinks with offline modifications.
+#
+# nlink of overlay inode should account for the union of lower and upper
+# hardlinks. Orphan index inodes with union nlink 0 should be cleaned on
+# mount.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+_require_scratch_feature index
+
+report_nlink()
+{
+	when=$1
+
+	# mount and check nlink after overlay offline modification
+	_scratch_mount -o index=on
+
+	# List <nlink> <name>
+	echo "== $when offline =="
+	for f in $HARDLINKS; do
+		_ls_l $SCRATCH_MNT/$f | awk '{ print $2, $9 }' | _filter_scratch
+	done
+
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+# Create lower hardlinks
+create_hardlinks()
+{
+	mkdir -p $lowerdir
+	touch $lowerdir/0
+	ln $lowerdir/0 $lowerdir/1
+	ln $lowerdir/0 $lowerdir/2
+}
+
+test_hardlinks_offline()
+{
+	HARDLINKS=`seq 0 2`
+	report_nlink "all upper"
+
+	# Unlink copied up hardlink
+	rm $upperdir/0
+	HARDLINKS=`seq 1 2`
+	report_nlink "unlink upper"
+
+	# Link to copied up hardlink
+	ln $upperdir/2 $upperdir/3
+	HARDLINKS=`seq 1 3`
+	report_nlink "link upper"
+
+	# Rename over copied up hardlink
+	touch $upperdir/new
+	mv $upperdir/new $upperdir/1
+	HARDLINKS=`seq 2 3`
+	report_nlink "rename over upper"
+
+	# Unlink new upper hardlink
+	rm $upperdir/3
+	HARDLINKS=2
+	report_nlink "unlink new upper"
+
+	# Unlink last upper and drop union nlink to zero
+	rm $upperdir/2
+
+	HARDLINKS=
+	report_nlink "unlink last lower"
+
+	# Verify that orphan index is cleaned when dropping nlink to zero
+	ls $workdir/index
+}
+
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
+workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
+
+# Remove all files from previous tests
+_scratch_mkfs
+
+# Create lower hardlinks
+create_hardlinks
+
+# Enable overlay index feature to prevent breaking hardlinks on copy up
+_scratch_mount -o index=on
+
+# Copy up and index hardlinks
+touch $SCRATCH_MNT/0
+touch $SCRATCH_MNT/1
+touch $SCRATCH_MNT/2
+
+# Perform the rest of the changes offline
+$UMOUNT_PROG $SCRATCH_MNT
+
+test_hardlinks_offline
+
+status=0
+exit
diff --git a/tests/overlay/048.out b/tests/overlay/048.out
new file mode 100644
index 0000000..0527dec
--- /dev/null
+++ b/tests/overlay/048.out
@@ -0,0 +1,18 @@
+QA output created by 048
+== all upper offline ==
+3 SCRATCH_MNT/0
+3 SCRATCH_MNT/1
+3 SCRATCH_MNT/2
+== unlink upper offline ==
+2 SCRATCH_MNT/1
+2 SCRATCH_MNT/2
+== link upper offline ==
+3 SCRATCH_MNT/1
+3 SCRATCH_MNT/2
+3 SCRATCH_MNT/3
+== rename over upper offline ==
+2 SCRATCH_MNT/2
+2 SCRATCH_MNT/3
+== unlink new upper offline ==
+1 SCRATCH_MNT/2
+== unlink last lower offline ==
diff --git a/tests/overlay/group b/tests/overlay/group
index 3e31d2d..7e541e4 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -48,3 +48,4 @@
 043 auto quick copyup nonsamefs
 044 auto quick copyup hardlink nonsamefs
 047 auto quick copyup hardlink
+048 auto quick copyup hardlink
-- 
2.7.4

[PATCH 3/3] overlay/042: remove wrong check for empty index

[Amir Goldstein]

The check for empty index in this test was wrongly copied from test
overlay/034. In test overlay/034 lower file starts as a hardlink, so
nlink accounting is done from the first copy up and index can be cleaned
on last upper hardlink unlink. In this test, lower starts as
non-hardlink, so first copy up does not perform nlink accounting and
therefore, union nlink count does not drop to 0 at the end of the test
and the index is not expected to be cleaned.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/042 | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/tests/overlay/042 b/tests/overlay/042
index dae5198..8967ff3 100755
--- a/tests/overlay/042
+++ b/tests/overlay/042
@@ -129,10 +129,6 @@ rm $SCRATCH_MNT/2
 rm $SCRATCH_MNT/3
 rm $SCRATCH_MNT/4
 
-# Verify that orphan index is cleaned on mount
-_scratch_cycle_mount index=on
-ls $OVL_BASE_SCRATCH_MNT/$OVL_WORK/index
-
 echo "Silence is golden"
 status=0
 exit
-- 
2.7.4