[PATCH] overlay: test basic whiteout

[PATCH] overlay: test basic whiteout

[Xiong Zhou]

Overlayfs whiteout can be visible if underlying xfs was created
with ftype=0. Kernel commit
	45aebea (ovl: Ensure upper filesystem supports d_type)
prevents mounting overlayfs like this since v4.6-rc1. No harm to
keep this test as a sanity check.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---
 tests/overlay/003     | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/003.out |  2 ++
 tests/overlay/group   |  1 +
 3 files changed, 81 insertions(+)
 create mode 100755 tests/overlay/003
 create mode 100644 tests/overlay/003.out

diff --git a/tests/overlay/003 b/tests/overlay/003
new file mode 100755
index 0000000..2401075
--- /dev/null
+++ b/tests/overlay/003
@@ -0,0 +1,78 @@
+#! /bin/bash
+# FS QA Test 003
+#
+# Test overlayfs whiteout.
+#
+# Overlayfs whiteout can be visible if underlying xfs was created
+# with ftype=0. Kernel commit
+#	45aebea (ovl: Ensure upper filesystem supports d_type)
+# prevents mounting overlayfs like this since v4.6-rc1. No harm to
+# keep this test as a sanity check.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+
+# Remove all files from previous tests
+_scratch_mkfs
+
+# Create test file in lower dir
+lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR}
+mkdir -p $lowerdir
+touch ${lowerdir}/test_file
+
+_scratch_mount
+
+# whiteout file in overlay mount
+rm -f ${SCRATCH_MNT}/test_file
+# it should not be listed
+ls -l ${SCRATCH_MNT}/test_file 2>&1 | awk -F: '{print $NF}'
+
+_scratch_unmount
+
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/003.out b/tests/overlay/003.out
new file mode 100644
index 0000000..83bafee
--- /dev/null
+++ b/tests/overlay/003.out
@@ -0,0 +1,2 @@
+QA output created by 003
+ No such file or directory
diff --git a/tests/overlay/group b/tests/overlay/group
index 84d164e..f671c67 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -5,3 +5,4 @@
 #
 001 auto quick copyup
 002 auto quick metadata
+003 auto quick
-- 
2.5.5

Re: [PATCH] overlay: test basic whiteout

[Eryu Guan]

On Thu, Apr 21, 2016 at 08:23:58PM +0800, Xiong Zhou wrote:
> Overlayfs whiteout can be visible if underlying xfs was created
> with ftype=0. Kernel commit
> 	45aebea (ovl: Ensure upper filesystem supports d_type)
> prevents mounting overlayfs like this since v4.6-rc1. No harm to
> keep this test as a sanity check.

It turns out that the test harness won't even start if underlying XFS
has no d_type support :)

mount: wrong fs type, bad option, bad superblock on /mnt/ovl/test,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
common/rc: retrying test device mount with external set

But I still think it's good to have a whiteout sanity test.

> 
> Signed-off-by: Xiong Zhou <xzhou@redhat.com>
> ---
>  tests/overlay/003     | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/overlay/003.out |  2 ++
>  tests/overlay/group   |  1 +
>  3 files changed, 81 insertions(+)
>  create mode 100755 tests/overlay/003
>  create mode 100644 tests/overlay/003.out
> 
> diff --git a/tests/overlay/003 b/tests/overlay/003
> new file mode 100755
> index 0000000..2401075
> --- /dev/null
> +++ b/tests/overlay/003
> @@ -0,0 +1,78 @@
> +#! /bin/bash
> +# FS QA Test 003
> +#
> +# Test overlayfs whiteout.
> +#
> +# Overlayfs whiteout can be visible if underlying xfs was created
> +# with ftype=0. Kernel commit
> +#	45aebea (ovl: Ensure upper filesystem supports d_type)
> +# prevents mounting overlayfs like this since v4.6-rc1. No harm to
> +# keep this test as a sanity check.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs overlay
> +_supported_os Linux
> +_require_scratch
> +
> +# Remove all files from previous tests
> +_scratch_mkfs
> +
> +# Create test file in lower dir
> +lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR}
> +mkdir -p $lowerdir
> +touch ${lowerdir}/test_file

I think as a whiteout sanity test, more file types can be tested here,
not only regular file, things like dir, hardlink, symlink, chardev,
blkdev and named pipe should be tested as well.

> +
> +_scratch_mount
> +
> +# whiteout file in overlay mount
> +rm -f ${SCRATCH_MNT}/test_file

Then all files can be removed here
rm -rf ${SCRATCH_MNT}/*

> +# it should not be listed
> +ls -l ${SCRATCH_MNT}/test_file 2>&1 | awk -F: '{print $NF}'

And no output expected here from ls
ls ${SCRATCH_MNT}/

> +
> +_scratch_unmount
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/overlay/003.out b/tests/overlay/003.out
> new file mode 100644
> index 0000000..83bafee
> --- /dev/null
> +++ b/tests/overlay/003.out
> @@ -0,0 +1,2 @@
> +QA output created by 003
> + No such file or directory

And probably we can change this to a simple "Silence is golden" and
print it from the test.

> diff --git a/tests/overlay/group b/tests/overlay/group
> index 84d164e..f671c67 100644
> --- a/tests/overlay/group
> +++ b/tests/overlay/group
> @@ -5,3 +5,4 @@
>  #
>  001 auto quick copyup
>  002 auto quick metadata
> +003 auto quick

Add a new "whiteout" group?

Thanks,
Eryu

Re: [PATCH] overlay: test basic whiteout

[Xiong Zhou]

On Thu, Apr 21, 2016 at 08:57:32PM +0800, Eryu Guan wrote:
> On Thu, Apr 21, 2016 at 08:23:58PM +0800, Xiong Zhou wrote:
> > Overlayfs whiteout can be visible if underlying xfs was created
> > with ftype=0. Kernel commit
> > 	45aebea (ovl: Ensure upper filesystem supports d_type)
> > prevents mounting overlayfs like this since v4.6-rc1. No harm to
> > keep this test as a sanity check.
> 
> It turns out that the test harness won't even start if underlying XFS
> has no d_type support :)

Yes, so do other overlay test cases. Maybe we should check underlying
fs d_type support situation before running overlay cases, mark as
not run if need.

> 
> mount: wrong fs type, bad option, bad superblock on /mnt/ovl/test,
>        missing codepage or helper program, or other error
> 
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
> common/rc: retrying test device mount with external set
> 
> But I still think it's good to have a whiteout sanity test.
> 
> > 
> > Signed-off-by: Xiong Zhou <xzhou@redhat.com>
> > ---
> >  tests/overlay/003     | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/overlay/003.out |  2 ++
> >  tests/overlay/group   |  1 +
> >  3 files changed, 81 insertions(+)
> >  create mode 100755 tests/overlay/003
> >  create mode 100644 tests/overlay/003.out
> > 
> > diff --git a/tests/overlay/003 b/tests/overlay/003
> > new file mode 100755
> > index 0000000..2401075
> > --- /dev/null
> > +++ b/tests/overlay/003
> > @@ -0,0 +1,78 @@
> > +#! /bin/bash
> > +# FS QA Test 003
> > +#
> > +# Test overlayfs whiteout.
> > +#
> > +# Overlayfs whiteout can be visible if underlying xfs was created
> > +# with ftype=0. Kernel commit
> > +#	45aebea (ovl: Ensure upper filesystem supports d_type)
> > +# prevents mounting overlayfs like this since v4.6-rc1. No harm to
> > +# keep this test as a sanity check.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -f $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# remove previous $seqres.full before test
> > +rm -f $seqres.full
> > +
> > +# real QA test starts here
> > +
> > +# Modify as appropriate.
> > +_supported_fs overlay
> > +_supported_os Linux
> > +_require_scratch
> > +
> > +# Remove all files from previous tests
> > +_scratch_mkfs
> > +
> > +# Create test file in lower dir
> > +lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR}
> > +mkdir -p $lowerdir
> > +touch ${lowerdir}/test_file
> 
> I think as a whiteout sanity test, more file types can be tested here,
> not only regular file, things like dir, hardlink, symlink, chardev,
> blkdev and named pipe should be tested as well.

Yes! I missed this. Great catch!

> 
> > +
> > +_scratch_mount
> > +
> > +# whiteout file in overlay mount
> > +rm -f ${SCRATCH_MNT}/test_file
> 
> Then all files can be removed here
> rm -rf ${SCRATCH_MNT}/*
> 
> > +# it should not be listed
> > +ls -l ${SCRATCH_MNT}/test_file 2>&1 | awk -F: '{print $NF}'
> 
> And no output expected here from ls
> ls ${SCRATCH_MNT}/
> 
> > +
> > +_scratch_unmount
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/overlay/003.out b/tests/overlay/003.out
> > new file mode 100644
> > index 0000000..83bafee
> > --- /dev/null
> > +++ b/tests/overlay/003.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 003
> > + No such file or directory
> 
> And probably we can change this to a simple "Silence is golden" and
> print it from the test.

en.. yes, this looks better.

> 
> > diff --git a/tests/overlay/group b/tests/overlay/group
> > index 84d164e..f671c67 100644
> > --- a/tests/overlay/group
> > +++ b/tests/overlay/group
> > @@ -5,3 +5,4 @@
> >  #
> >  001 auto quick copyup
> >  002 auto quick metadata
> > +003 auto quick
> 
> Add a new "whiteout" group?

Good idea.  Thanks very much for reviewing!

--
Xiong

> 
> Thanks,
> Eryu

[PATCH 1/2] overlay: test basic whiteout

[Xiong Zhou]

Once a while, overlayfs whiteout can be visible if upper fs
does not support d_type(ftype/filetype). Test as a sanity
check on whiteout of regular files, links, dirs, devices
and pipes.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---
 tests/overlay/003     | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/003.out |  2 ++
 tests/overlay/group   |  1 +
 3 files changed, 101 insertions(+)
 create mode 100755 tests/overlay/003
 create mode 100644 tests/overlay/003.out

diff --git a/tests/overlay/003 b/tests/overlay/003
new file mode 100755
index 0000000..fbc242a
--- /dev/null
+++ b/tests/overlay/003
@@ -0,0 +1,98 @@
+#! /bin/bash
+# FS QA Test 003
+#
+# Test overlayfs basic whiteout.
+#
+# Overlayfs whiteout can be visible if underlying upper fs does not
+# support d_type. Kernel commit
+#	45aebea (ovl: Ensure upper filesystem supports d_type)
+# prevents mounting overlayfs like this since v4.6-rc1.
+# No harm to keep this test as a sanity check.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+
+# Remove all files from previous tests
+_scratch_mkfs
+
+# Create test files dirs in lower dir
+lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR}
+mkdir -p $lowerdir
+
+touch ${lowerdir}/test_file
+ln -s ${lowerdir}/test_file ${lowerdir}/test_softlink_file
+ln ${lowerdir}/test_file ${lowerdir}/test_hardlink_file
+
+mkdir -p ${lowerdir}/test_dir
+ln -s ${lowerdir}/test_dir ${lowerdir}/test_softlink_dir
+
+mknod ${lowerdir}/test_cdev c 2 3
+ln -s ${lowerdir}/test_cdev ${lowerdir}/test_softlink_cdev
+ln ${lowerdir}/test_cdev ${lowerdir}/test_hardlink_cdev
+
+mknod ${lowerdir}/test_bdev b 2 4
+ln -s ${lowerdir}/test_bdev ${lowerdir}/test_softlink_bdev
+ln ${lowerdir}/test_bdev ${lowerdir}/test_hardlink_bdev
+
+mknod ${lowerdir}/test_pipe p
+ln -s ${lowerdir}/test_pipe ${lowerdir}/test_softlink_pipe
+ln ${lowerdir}/test_pipe ${lowerdir}/test_hardlink_pipe
+
+_scratch_mount
+
+# whiteout all files in overlay mountpoint
+rm -rf ${SCRATCH_MNT}/*
+# nothing should be listed
+ls ${SCRATCH_MNT}/
+
+_scratch_unmount
+
+rm -rf $lowerdir
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/003.out b/tests/overlay/003.out
new file mode 100644
index 0000000..6895fc8
--- /dev/null
+++ b/tests/overlay/003.out
@@ -0,0 +1,2 @@
+QA output created by 003
+Silence is golden
diff --git a/tests/overlay/group b/tests/overlay/group
index 84d164e..9735f40 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -5,3 +5,4 @@
 #
 001 auto quick copyup
 002 auto quick metadata
+003 auto quick whiteout
-- 
2.5.5

[PATCH 2/2] overlay: notrun if upper fs does not support d_type

[Xiong Zhou]

Overlayfs whiteout can be visible if underlying upper fs does not
support d_type. Kernel commit
	45aebea (ovl: Ensure upper filesystem supports d_type)
prevents mounting overlayfs like this since v4.6-rc1.

Check upper fs before mounting overlay, mark as not run if needs.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---
 common/rc | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/common/rc b/common/rc
index 8bec836..a25d30e 100644
--- a/common/rc
+++ b/common/rc
@@ -273,6 +273,18 @@ _overlay_mount()
 	local dir=$1
 	local mnt=$2
 	shift 2
+
+	if [ "$(df --output=fstype $dir | tail -1)" == "xfs" ] && \
+	   ! xfs_info $dir | grep -q "ftype=1" ; then
+		_notrun "upper fs needs to support d_type"
+	fi
+
+	if [[ "$(df --output=fstype $dir | tail -1)" =~ ext ]] && \
+	   ! tune2fs -l $(df --output=source $dir | tail -1) | \
+	   grep -q filetype ; then
+		_notrun "upper fs needs to support d_type"
+	fi
+
 	mkdir -p $dir/$OVERLAY_UPPER_DIR
 	mkdir -p $dir/$OVERLAY_LOWER_DIR
 	mkdir -p $dir/$OVERLAY_WORK_DIR
-- 
2.5.5

[PATCH v2] overlay: notrun if upper fs does not support d_type

[Xiong Zhou]

Overlayfs whiteout can be visible if underlying upper fs does not
support d_type. Kernel commit
	45aebea (ovl: Ensure upper filesystem supports d_type)
prevents mounting overlayfs like this since v4.6-rc1.

Check upper fs before mounting overlay, mark as not run if needs.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---

v2: use case to make this less ugly..


 common/rc | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/common/rc b/common/rc
index 8bec836..946b354 100644
--- a/common/rc
+++ b/common/rc
@@ -273,6 +273,22 @@ _overlay_mount()
 	local dir=$1
 	local mnt=$2
 	shift 2
+
+	local upper_fst=$(df --output=fstype $dir | tail -1)
+	case "$upper_fst" in
+	xfs)
+		if ! xfs_info $dir | grep -q "ftype=1" ; then
+			_notrun "upper fs needs to support d_type"
+		fi
+		;;
+	ext2|ext3|ext4)
+		if ! tune2fs -l $(df --output=source $dir | tail -1) | \
+		   grep -q filetype ; then
+			_notrun "upper fs needs to support d_type"
+		fi
+		;;
+	esac
+
 	mkdir -p $dir/$OVERLAY_UPPER_DIR
 	mkdir -p $dir/$OVERLAY_LOWER_DIR
 	mkdir -p $dir/$OVERLAY_WORK_DIR
-- 
2.5.5

Re: [PATCH 1/2] overlay: test basic whiteout

[Eryu Guan]

On Fri, Apr 22, 2016 at 04:11:54PM +0800, Xiong Zhou wrote:
> Once a while, overlayfs whiteout can be visible if upper fs
> does not support d_type(ftype/filetype). Test as a sanity
> check on whiteout of regular files, links, dirs, devices
> and pipes.
> 
> Signed-off-by: Xiong Zhou <xzhou@redhat.com>

Looks good to me, thanks!

Reviewed-by: Eryu Guan <eguan@redhat.com>

Re: [PATCH v2] overlay: notrun if upper fs does not support d_type

[Eryu Guan]

On Fri, Apr 22, 2016 at 05:21:51PM +0800, Xiong Zhou wrote:
> Overlayfs whiteout can be visible if underlying upper fs does not
> support d_type. Kernel commit
> 	45aebea (ovl: Ensure upper filesystem supports d_type)
> prevents mounting overlayfs like this since v4.6-rc1.
> 
> Check upper fs before mounting overlay, mark as not run if needs.
> 
> Signed-off-by: Xiong Zhou <xzhou@redhat.com>

I'm not sure if this really necessary, but looks fine to me.

Reviewed-by: Eryu Guan <eguan@redhat.com>

[PATCH] overlay: test copy up by changing mode bits

[Xiong Zhou]

Changing file attr in overlayfs triggers copy up.
Test this by changing mode bits then check both mode
bits and copy up results.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---

This patch depends on another patch i post on Apr 22:
  overlay: test basic whiteout
for the named number and tests/overlay/group .

 tests/overlay/004     | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/004.out |  5 +++
 tests/overlay/group   |  1 +
 3 files changed, 103 insertions(+)
 create mode 100755 tests/overlay/004
 create mode 100644 tests/overlay/004.out

diff --git a/tests/overlay/004 b/tests/overlay/004
new file mode 100755
index 0000000..4075094
--- /dev/null
+++ b/tests/overlay/004
@@ -0,0 +1,97 @@
+#! /bin/bash
+# FS QA Test 004
+#
+# Test file copy up on overlayfs by changing mode bits.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# Modify as appropriate.
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+_require_user
+
+# Remove all files from previous tests
+_scratch_mkfs
+
+# Create test file in lower dir
+lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR}
+upperdir=${SCRATCH_DEV}/${OVERLAY_UPPER_DIR}
+mkdir -p $lowerdir
+touch ${lowerdir}/attr_file1
+chmod 000 ${lowerdir}/attr_file1
+touch ${lowerdir}/attr_file2
+chmod 000 ${lowerdir}/attr_file2
+
+_scratch_mount
+
+# copy up by change mode bits and check.
+chmod a+x ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod o-w ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod g+w ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod u-r ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod a+X ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod o-s ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod g+t ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+chmod u-X ${SCRATCH_MNT}/attr_file1 > /dev/null 2>&1
+stat -c %a ${SCRATCH_MNT}/attr_file1
+
+# unprivileged user can't do this.
+_user_do "chmod a+x ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod o-w ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod g+w ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod u-r ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod a+X ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod o-s ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod g+t ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+_user_do "chmod u-X ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
+stat -c %a ${SCRATCH_MNT}/attr_file2
+
+_scratch_unmount
+
+# check mode bits of the file that has been copied up, and
+# the file that should not have been copied up.
+stat -c %a ${upperdir}/attr_file1
+ls ${upperdir}/
+
+# success, all done
+status=0
+exit
diff --git a/tests/overlay/004.out b/tests/overlay/004.out
new file mode 100644
index 0000000..2429f38
--- /dev/null
+++ b/tests/overlay/004.out
@@ -0,0 +1,5 @@
+QA output created by 004
+31
+0
+31
+attr_file1
diff --git a/tests/overlay/group b/tests/overlay/group
index 9735f40..fcf6d63 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -6,3 +6,4 @@
 001 auto quick copyup
 002 auto quick metadata
 003 auto quick whiteout
+004 attr auto copyup quick
-- 
2.5.5

Re: [PATCH] overlay: test copy up by changing mode bits

[Eryu Guan]

On Wed, May 04, 2016 at 02:53:16PM +0800, Xiong Zhou wrote:
> Changing file attr in overlayfs triggers copy up.
> Test this by changing mode bits then check both mode
> bits and copy up results.
> 
> Signed-off-by: Xiong Zhou <xzhou@redhat.com>

Looks good to me.

Reviewed-by: Eryu Guan <eguan@redhat.com>

[PATCH] overlay: test memleak in copy up error handling path

[Xiong Zhou]

This memleak may trigger panic when unmount the underlying fs.
Kernel commit ab79efab0 fixed this issue.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---

This patch depends on another patch i post on Apr 22:
  overlay: test basic whiteout
for the named number and tests/overlay/group .

 tests/overlay/005     | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/005.out |   2 +
 tests/overlay/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/overlay/005
 create mode 100644 tests/overlay/005.out

diff --git a/tests/overlay/005 b/tests/overlay/005
new file mode 100755
index 0000000..a61ef45
--- /dev/null
+++ b/tests/overlay/005
@@ -0,0 +1,111 @@
+#! /bin/bash
+# FS QA Test 005
+#
+# Memleak in overlayfs copy up error handling path may trigger panic
+# when unmount the underlying fs.
+# Kernel commit ab79efab0 fixed this issue.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+_require_loop
+
+# backup old scrach dev and mnt
+OLD_SCRATCH_DEV=$SCRATCH_DEV
+OLD_SCRATCH_MNT=$SCRATCH_MNT
+
+# setup loop image fs for overlayfs
+fs_img=${SCRATCH_MNT}/${seq}.$$.img
+$XFS_IO_PROG -f -c "truncate 5120m" $fs_img >>$seqres.full 2>&1
+loop_dev=`_create_loop_device $fs_img`
+mkfs.xfs -f -n ftype=1 $loop_dev >>$seqres.full 2>&1
+
+# mount underlying xfs
+mkdir -p ${SCRATCH_MNT}/xfsmnt
+_mount $loop_dev ${SCRATCH_MNT}/xfsmnt
+
+# prepare dirs and set new SCRATCH_MNT/_DEV
+mkdir -p ${SCRATCH_MNT}/xfsmnt/schmnt
+mkdir -p ${SCRATCH_MNT}/xfsmnt/schdev
+export SCRATCH_MNT=${SCRATCH_MNT}/xfsmnt/schmnt
+export SCRATCH_DEV=${SCRATCH_MNT}/xfsmnt/schdev
+
+# make newly overlayfs
+_scratch_mkfs
+
+# Create test file in lower dir
+lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR}
+mkdir -p $lowerdir
+$XFS_IO_PROG -f -c "truncate 2048m"  ${lowerdir}/test_file >>$seqres.full 2>&1
+
+# mount new overlayfs
+_scratch_mount
+
+# the open call triggers copy up and it will take some time
+src/writemod $SCRATCH_MNT/test_file >>$seqres.full 2>&1 &
+pid=$!
+
+# 3 seconds till cancel it
+sleep 3
+kill -2 $pid 2>/dev/null
+wait $pid 2>/dev/null
+pid=""
+
+# unmount overlayfs
+_scratch_unmount
+
+# restore old values and cleanup
+export SCRATCH_MNT=$OLD_SCRATCH_MNT
+export SCRATCH_DEV=$OLD_SCRATCH_DEV
+
+# unmount undelying xfs, this tiggers a panic if mem leak happens
+umount -d ${SCRATCH_MNT}/xfsmnt
+rm -f $fs_img
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/overlay/005.out b/tests/overlay/005.out
new file mode 100644
index 0000000..a5027f1
--- /dev/null
+++ b/tests/overlay/005.out
@@ -0,0 +1,2 @@
+QA output created by 005
+Silence is golden
diff --git a/tests/overlay/group b/tests/overlay/group
index fcf6d63..bc2474e 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -7,3 +7,4 @@
 002 auto quick metadata
 003 auto quick whiteout
 004 attr auto copyup quick
+005 auto copyup dangerous quick
-- 
2.5.5

[PATCH v2] overlay: test memleak in copy-up error handling path

[Xiong Zhou]

This memleak leads to panic when unmount the underlying fs.
Kernel commit ab79efab0 fixed this issue.

Signed-off-by: Xiong Zhou <xzhou@redhat.com>
---

Diff to v1:
	set up underlying fs in $SCRATCH_DEV instead of $SCRATCH_MNT
	xfs_io open test_file instead of src/writemod
	use little upper dir to trigger ENOSP failure instead of kill
	clean up in cleanup

Thanks Eryu for reviewing!

 tests/overlay/005     | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/005.out |   2 +
 tests/overlay/group   |   1 +
 3 files changed, 115 insertions(+)
 create mode 100755 tests/overlay/005
 create mode 100644 tests/overlay/005.out

diff --git a/tests/overlay/005 b/tests/overlay/005
new file mode 100755
index 0000000..3816bf1
--- /dev/null
+++ b/tests/overlay/005
@@ -0,0 +1,112 @@
+#! /bin/bash
+# FS QA Test 005
+#
+# Memleak in overlayfs copy-up error handling path leads to panic
+# when unmount the underlying fs.
+# Kernel commit ab79efab0 fixed this issue.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	_destroy_loop_device $fs_loop_dev
+	_destroy_loop_device $upper_loop_dev
+	rm -f $fs_img
+	rm -f $upper_img
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs overlay
+_supported_os Linux
+_require_scratch
+_require_loop
+
+# setup loop images fs for overlayfs
+fs_img=${SCRATCH_DEV}/${seq}.$$.img
+$XFS_IO_PROG -f -c "truncate 120m" $fs_img >>$seqres.full 2>&1
+fs_loop_dev=`_create_loop_device $fs_img`
+$MKFS_XFS_PROG -f -n ftype=1 $fs_loop_dev >>$seqres.full 2>&1
+
+# only 20m for upper dir
+upper_img=${SCRATCH_DEV}/$$.${seq}.img
+$XFS_IO_PROG -f -c "truncate 20m" $upper_img >>$seqres.full 2>&1
+upper_loop_dev=`_create_loop_device $upper_img`
+$MKFS_XFS_PROG -f -n ftype=1 $upper_loop_dev >>$seqres.full 2>&1
+
+# mount underlying xfs
+mkdir -p ${SCRATCH_DEV}/xfsmnt
+mkdir -p ${SCRATCH_DEV}/uppermnt
+$MOUNT_PROG $fs_loop_dev ${SCRATCH_DEV}/xfsmnt
+$MOUNT_PROG $upper_loop_dev ${SCRATCH_DEV}/uppermnt
+
+# prepare dirs
+mkdir -p ${SCRATCH_DEV}/xfsmnt/lower
+mkdir -p ${SCRATCH_DEV}/uppermnt/upper
+mkdir -p ${SCRATCH_DEV}/uppermnt/work
+
+lowerd=${SCRATCH_DEV}/xfsmnt/lower
+upperd=${SCRATCH_DEV}/uppermnt/upper
+workd=${SCRATCH_DEV}/uppermnt/work
+
+# Create test file in lower dir, with too big a size for
+# upper dir to copy up.
+$XFS_IO_PROG -f -c "truncate 48m" ${lowerd}/test_file \
+	>>$seqres.full 2>&1
+
+# mount new overlayfs
+$MOUNT_PROG -t overlay $SELINUX_MOUNT_OPTIONS \
+	    -o lowerdir=$lowerd -o upperdir=$upperd \
+	    -o workdir=$workd \
+	    $SCRATCH_DEV $SCRATCH_MNT
+
+# the open call triggers copy-up and it will fail ENOSP
+$XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \
+	>>$seqres.full 2>&1
+
+# unmount overlayfs
+$UMOUNT_PROG $SCRATCH_MNT
+
+# unmount undelying xfs, this tiggers panic if memleak happens
+$UMOUNT_PROG ${SCRATCH_DEV}/uppermnt
+$UMOUNT_PROG ${SCRATCH_DEV}/xfsmnt
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/overlay/005.out b/tests/overlay/005.out
new file mode 100644
index 0000000..a5027f1
--- /dev/null
+++ b/tests/overlay/005.out
@@ -0,0 +1,2 @@
+QA output created by 005
+Silence is golden
diff --git a/tests/overlay/group b/tests/overlay/group
index fcf6d63..bc2474e 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -7,3 +7,4 @@
 002 auto quick metadata
 003 auto quick whiteout
 004 attr auto copyup quick
+005 auto copyup dangerous quick
-- 
2.5.5

Re: [PATCH v2] overlay: test memleak in copy-up error handling path

[Eryu Guan]

On Tue, May 10, 2016 at 10:32:45AM +0800, Xiong Zhou wrote:
> This memleak leads to panic when unmount the underlying fs.
> Kernel commit ab79efab0 fixed this issue.
> 
> Signed-off-by: Xiong Zhou <xzhou@redhat.com>
> ---
> 
> Diff to v1:
> 	set up underlying fs in $SCRATCH_DEV instead of $SCRATCH_MNT
> 	xfs_io open test_file instead of src/writemod
> 	use little upper dir to trigger ENOSP failure instead of kill
> 	clean up in cleanup

Thanks for the update! And sorry for late review.. Some minor issues
inline

> 
> Thanks Eryu for reviewing!
> 
>  tests/overlay/005     | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/overlay/005.out |   2 +
>  tests/overlay/group   |   1 +
>  3 files changed, 115 insertions(+)
>  create mode 100755 tests/overlay/005
>  create mode 100644 tests/overlay/005.out
> 
> diff --git a/tests/overlay/005 b/tests/overlay/005
> new file mode 100755
> index 0000000..3816bf1
> --- /dev/null
> +++ b/tests/overlay/005
> @@ -0,0 +1,112 @@
> +#! /bin/bash
> +# FS QA Test 005
> +#
> +# Memleak in overlayfs copy-up error handling path leads to panic
> +# when unmount the underlying fs.
> +# Kernel commit ab79efab0 fixed this issue.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2016 Red Hat Inc. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +	_destroy_loop_device $fs_loop_dev
> +	_destroy_loop_device $upper_loop_dev
> +	rm -f $fs_img
> +	rm -f $upper_img
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs overlay
> +_supported_os Linux
> +_require_scratch
> +_require_loop
> +

call _scratch_mkfs before creating fs images? It removes all existing
files in $SCRATCH_DEV and make sure no interference from previous test.

> +# setup loop images fs for overlayfs
> +fs_img=${SCRATCH_DEV}/${seq}.$$.img

"lower_img" seems better, we know what it's for from its name.

> +$XFS_IO_PROG -f -c "truncate 120m" $fs_img >>$seqres.full 2>&1
> +fs_loop_dev=`_create_loop_device $fs_img`
> +$MKFS_XFS_PROG -f -n ftype=1 $fs_loop_dev >>$seqres.full 2>&1
> +
> +# only 20m for upper dir
> +upper_img=${SCRATCH_DEV}/$$.${seq}.img
> +$XFS_IO_PROG -f -c "truncate 20m" $upper_img >>$seqres.full 2>&1
> +upper_loop_dev=`_create_loop_device $upper_img`
> +$MKFS_XFS_PROG -f -n ftype=1 $upper_loop_dev >>$seqres.full 2>&1
> +
> +# mount underlying xfs
> +mkdir -p ${SCRATCH_DEV}/xfsmnt

So "lowermnt" seems better here.

> +mkdir -p ${SCRATCH_DEV}/uppermnt
> +$MOUNT_PROG $fs_loop_dev ${SCRATCH_DEV}/xfsmnt
> +$MOUNT_PROG $upper_loop_dev ${SCRATCH_DEV}/uppermnt
> +
> +# prepare dirs
> +mkdir -p ${SCRATCH_DEV}/xfsmnt/lower
> +mkdir -p ${SCRATCH_DEV}/uppermnt/upper
> +mkdir -p ${SCRATCH_DEV}/uppermnt/work
> +
> +lowerd=${SCRATCH_DEV}/xfsmnt/lower
> +upperd=${SCRATCH_DEV}/uppermnt/upper
> +workd=${SCRATCH_DEV}/uppermnt/work
> +
> +# Create test file in lower dir, with too big a size for
> +# upper dir to copy up.
> +$XFS_IO_PROG -f -c "truncate 48m" ${lowerd}/test_file \
> +	>>$seqres.full 2>&1
> +
> +# mount new overlayfs
> +$MOUNT_PROG -t overlay $SELINUX_MOUNT_OPTIONS \
> +	    -o lowerdir=$lowerd -o upperdir=$upperd \
> +	    -o workdir=$workd \
> +	    $SCRATCH_DEV $SCRATCH_MNT
> +
> +# the open call triggers copy-up and it will fail ENOSP
                                                     ^^^^^ ENOSPC :)

> +$XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \
> +	>>$seqres.full 2>&1
> +
> +# unmount overlayfs
> +$UMOUNT_PROG $SCRATCH_MNT
> +
> +# unmount undelying xfs, this tiggers panic if memleak happens
> +$UMOUNT_PROG ${SCRATCH_DEV}/uppermnt
> +$UMOUNT_PROG ${SCRATCH_DEV}/xfsmnt
> +
> +# success, all done
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/overlay/005.out b/tests/overlay/005.out
> new file mode 100644
> index 0000000..a5027f1
> --- /dev/null
> +++ b/tests/overlay/005.out
> @@ -0,0 +1,2 @@
> +QA output created by 005
> +Silence is golden
> diff --git a/tests/overlay/group b/tests/overlay/group
> index fcf6d63..bc2474e 100644
> --- a/tests/overlay/group
> +++ b/tests/overlay/group
> @@ -7,3 +7,4 @@
>  002 auto quick metadata
>  003 auto quick whiteout
>  004 attr auto copyup quick
> +005 auto copyup dangerous quick

This bug has been fixed, 'dangerous' group can be removed.

Thanks,
Eryu

Re: [PATCH v2] overlay: test memleak in copy-up error handling path

[Xiong Zhou]

On Thu, May 26, 2016 at 03:24:26PM +0800, Eryu Guan wrote:
> On Tue, May 10, 2016 at 10:32:45AM +0800, Xiong Zhou wrote:
> > This memleak leads to panic when unmount the underlying fs.
> > Kernel commit ab79efab0 fixed this issue.
> > 
> > Signed-off-by: Xiong Zhou <xzhou@redhat.com>
> > ---
> > 
> > Diff to v1:
> > 	set up underlying fs in $SCRATCH_DEV instead of $SCRATCH_MNT
> > 	xfs_io open test_file instead of src/writemod
> > 	use little upper dir to trigger ENOSP failure instead of kill
> > 	clean up in cleanup
> 
> Thanks for the update! And sorry for late review.. Some minor issues
> inline

Thank you very much for the tips! Sending v3.

--
Xiong