[PATCH v3 0/6] fstests: copy_file_range() tests

[PATCH v3 0/6] fstests: copy_file_range() tests

[Amir Goldstein]

Eryu,

This is a re-work of Dave Chinner's copy_file_range() tests which
I used to verify the kernel fixes of the syscall [1].

The 2 first tests fix bugs in the interface, so they are appropriate
for merge IMO.

The cross-device copy test checks a new functionality, so you may
want to wait with merging it till after the work is merged upstream.

The bounds check test depend on a change that was only posted to
xfsprogs [2]. Without two changes that were merge to xfsprogs v4.20,
the original test (v1, v2) would hang. Requiring the new copy_range
flag (copy_range -f) mitigates this problem.

You may want to wait until the xfs_io change is merged before merging
the check for the new flag.

Thanks,
Amir.

[1] https://lore.kernel.org/linux-fsdevel/20190531164701.15112-1-amir73il@gmail.com/
[2] https://marc.info/?l=linux-xfs&m=155912482124038&w=2

Changes from v2:
- Change blockdev in test to loop and _require_loop (Olga)
- Implement and use _require_xfs_io_command copy_range -f

Changes from v1:
- Remove patch to test EINVAL behavior instead of short copy
- Remove 'chmod -r' permission drop test case
- Split out test for swap/immutable file copy
- Split of cross-device copy test


Amir Goldstein (6):
  generic: create copy_range group
  generic: copy_file_range immutable file test
  generic: copy_file_range swapfile test
  common/rc: check support for xfs_io copy_range -f N
  generic: copy_file_range bounds test
  generic: cross-device copy_file_range test

 common/rc             |   9 ++-
 tests/generic/434     |   2 +
 tests/generic/988     |  59 +++++++++++++++++++
 tests/generic/988.out |   5 ++
 tests/generic/989     |  56 ++++++++++++++++++
 tests/generic/989.out |   4 ++
 tests/generic/990     | 132 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/990.out |  37 ++++++++++++
 tests/generic/991     |  56 ++++++++++++++++++
 tests/generic/991.out |   4 ++
 tests/generic/group   |  14 +++--
 11 files changed, 372 insertions(+), 6 deletions(-)
 create mode 100755 tests/generic/988
 create mode 100644 tests/generic/988.out
 create mode 100755 tests/generic/989
 create mode 100644 tests/generic/989.out
 create mode 100755 tests/generic/990
 create mode 100644 tests/generic/990.out
 create mode 100755 tests/generic/991
 create mode 100644 tests/generic/991.out

-- 
2.17.1

[PATCH v3 1/6] generic: create copy_range group

[Amir Goldstein]

Move some tests to the copy_range group so they are distinct
from the copy group which refers to xfs_copy tests.

[Amir] Revert copy past EOF behavior change

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/434   |  2 ++
 tests/generic/group | 10 +++++-----
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/tests/generic/434 b/tests/generic/434
index 032f933d..edbf49d3 100755
--- a/tests/generic/434
+++ b/tests/generic/434
@@ -46,10 +46,12 @@ $XFS_IO_PROG -f -c "copy_range -s 1000 -l 100 $testdir/file" "$testdir/copy"
 md5sum $testdir/copy | _filter_test_dir
 
 echo "Try to copy to a read-only file"
+rm -f $testdir/copy
 $XFS_IO_PROG -r -f -c "copy_range -s 0 -l 100 $testdir/file" "$testdir/copy"
 md5sum $testdir/copy | _filter_test_dir
 
 echo "Try to copy to an append-only file"
+rm -f $testdir/copy
 $XFS_IO_PROG -a -f -c "copy_range -s 0 -l 100 $testdir/file" "$testdir/copy"
 md5sum $testdir/copy | _filter_test_dir
 
diff --git a/tests/generic/group b/tests/generic/group
index 49639fc9..b498eb56 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -432,11 +432,11 @@
 427 auto quick aio rw
 428 auto quick dax
 429 auto encrypt
-430 auto quick copy
-431 auto quick copy
-432 auto quick copy
-433 auto quick copy
-434 auto quick copy
+430 auto quick copy_range
+431 auto quick copy_range
+432 auto quick copy_range
+433 auto quick copy_range
+434 auto quick copy_range
 435 auto encrypt
 436 auto quick rw seek prealloc
 437 auto quick dax
-- 
2.17.1

[PATCH v3 2/6] generic: copy_file_range immutable file test

[Amir Goldstein]

This test case was split out of Dave Chinner's copy_file_range bounds
check test to reduce the requirements for running the bounds check.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/988     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/988.out |  5 ++++
 tests/generic/group   |  1 +
 3 files changed, 65 insertions(+)
 create mode 100755 tests/generic/988
 create mode 100644 tests/generic/988.out

diff --git a/tests/generic/988 b/tests/generic/988
new file mode 100755
index 00000000..0f4ee4ea
--- /dev/null
+++ b/tests/generic/988
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 988
+#
+# Check that we cannot copy_file_range() to/from an immutable file
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	$CHATTR_PROG -i $testdir/immutable > /dev/null 2>&1
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs generic
+
+rm -f $seqres.full
+
+_require_test
+_require_chattr i
+_require_xfs_io_command "copy_range"
+_require_xfs_io_command "chattr"
+
+testdir="$TEST_DIR/test-$seq"
+rm -rf $testdir
+mkdir $testdir
+
+rm -f $seqres.full
+
+$XFS_IO_PROG -f -c "pwrite -S 0x61 0 128k" $testdir/file >> $seqres.full 2>&1
+
+# we have to open the file to be immutable rw and hold it open over the
+# chattr command to set it immutable, otherwise we won't be able to open it for
+# writing after it's been made immutable. (i.e. would exercise file mode checks,
+# not immutable inode flag checks).
+echo immutable file returns EPERM
+$XFS_IO_PROG -f -c "pwrite -S 0x61 0 64k" -c fsync $testdir/immutable | _filter_xfs_io
+$XFS_IO_PROG -f -c "chattr +i" -c "copy_range -l 32k $testdir/file" $testdir/immutable
+$XFS_IO_PROG -f -r -c "chattr -i" $testdir/immutable
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/988.out b/tests/generic/988.out
new file mode 100644
index 00000000..e74a96bf
--- /dev/null
+++ b/tests/generic/988.out
@@ -0,0 +1,5 @@
+QA output created by 988
+immutable file returns EPERM
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+copy_range: Operation not permitted
diff --git a/tests/generic/group b/tests/generic/group
index b498eb56..20b95c14 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -550,3 +550,4 @@
 545 auto quick cap
 546 auto quick clone enospc log
 547 auto quick log
+988 auto quick copy_range
-- 
2.17.1

[PATCH v3 3/6] generic: copy_file_range swapfile test

[Amir Goldstein]

This test case was split out of Dave Chinner's copy_file_range bounds
check test to reduce the requirements for running the bounds check.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/989     | 56 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/989.out |  4 ++++
 tests/generic/group   |  1 +
 3 files changed, 61 insertions(+)
 create mode 100755 tests/generic/989
 create mode 100644 tests/generic/989.out

diff --git a/tests/generic/989 b/tests/generic/989
new file mode 100755
index 00000000..27c10296
--- /dev/null
+++ b/tests/generic/989
@@ -0,0 +1,56 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 989
+#
+# Check that we cannot copy_file_range() to/from a swapfile
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs generic
+
+rm -f $seqres.full
+
+_require_scratch
+_require_xfs_io_command "copy_range"
+_require_scratch_swapfile
+
+_scratch_mkfs 2>&1 >> $seqres.full
+_scratch_mount
+
+testdir=$SCRATCH_MNT
+
+rm -f $seqres.full
+
+$XFS_IO_PROG -f -c "pwrite -S 0x61 0 128k" $testdir/file >> $seqres.full 2>&1
+
+echo swap files return ETXTBUSY
+_format_swapfile $testdir/swapfile 16m
+swapon $testdir/swapfile
+$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/file" $testdir/swapfile
+$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/swapfile" $testdir/copy
+swapoff $testdir/swapfile
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/989.out b/tests/generic/989.out
new file mode 100644
index 00000000..32da0ce9
--- /dev/null
+++ b/tests/generic/989.out
@@ -0,0 +1,4 @@
+QA output created by 989
+swap files return ETXTBUSY
+copy_range: Text file busy
+copy_range: Text file busy
diff --git a/tests/generic/group b/tests/generic/group
index 20b95c14..4c100781 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -551,3 +551,4 @@
 546 auto quick clone enospc log
 547 auto quick log
 988 auto quick copy_range
+989 auto quick copy_range swap
-- 
2.17.1

[PATCH v3 4/6] common/rc: check support for xfs_io copy_range -f N

[Amir Goldstein]

Implement "_require_xfs_io_command copy_range -f" to check for
the option added by following xfsprogs commit:

  xfs_io: allow passing an open file to copy_range

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 common/rc | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/common/rc b/common/rc
index 17b89d5d..0d26b0fc 100644
--- a/common/rc
+++ b/common/rc
@@ -2086,9 +2086,16 @@ _require_xfs_io_command()
 		;;
 	"copy_range")
 		local testcopy=$TEST_DIR/$$.copy.xfs_io
+		local copy_opts=$testfile
+		if [ "$param" == "-f" ]; then
+			# source file is the open destination file
+			testcopy=$testfile
+			copy_opts="0 -d 4k"
+		fi
 		$XFS_IO_PROG -F -f -c "pwrite 0 4k" $testfile > /dev/null 2>&1
-		testio=`$XFS_IO_PROG -F -f -c "copy_range $testfile" $testcopy 2>&1`
+		testio=`$XFS_IO_PROG -F -f -c "copy_range $param $copy_opts" $testcopy 2>&1`
 		rm -f $testcopy > /dev/null 2>&1
+		param_checked=1
 		;;
 	"falloc" )
 		testio=`$XFS_IO_PROG -F -f -c "falloc $param 0 1m" $testfile 2>&1`
-- 
2.17.1

[PATCH v3 5/6] generic: copy_file_range bounds test

[Amir Goldstein]

Test that copy_file_range will return the correct errors for various
error conditions and boundary constraints.

[Amir] Split out cross-device copy_range test and use only scratch dev.
Split out immutable/swapfile test cases to reduce the requirements to
run the bounds check to minimum and get coverage for more filesystems.
Remove the tests for read past EOF and write after chmod -r,
because we decided to stick with read(2)/write(2) semantics.
Add requirements needed for large size copy tests and fifo test.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/990     | 132 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/990.out |  37 ++++++++++++
 tests/generic/group   |   1 +
 3 files changed, 170 insertions(+)
 create mode 100755 tests/generic/990
 create mode 100644 tests/generic/990.out

diff --git a/tests/generic/990 b/tests/generic/990
new file mode 100755
index 00000000..821aa10b
--- /dev/null
+++ b/tests/generic/990
@@ -0,0 +1,132 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 990
+#
+# Exercise copy_file_range() syscall error conditions.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs generic
+
+rm -f $seqres.full
+
+_require_test
+# For blockdev copy test case
+_require_loop
+#
+# This test effectively requires xfs_io with these commits
+#  2a42470b xfs_io: copy_file_range length is a size_t
+#  1a05efba io: open pipes in non-blocking mode
+#
+# Without those commits test will hang on old kernel when copying
+# very large size and when copying from a pipe.
+#
+# We require a new xfs_io feature of passing an open file as the
+# copy source, as an indication that the test can run without hanging
+# with large size argument and to avoid opening pipe in blocking mode.
+#
+_require_xfs_io_command "copy_range" "-f"
+
+testdir="$TEST_DIR/test-$seq"
+rm -rf $testdir
+mkdir $testdir
+
+rm -f $seqres.full
+
+$XFS_IO_PROG -f -c "pwrite -S 0x61 0 128k" $testdir/file >> $seqres.full 2>&1
+
+echo source range overlaps destination range in same file returns EINVAL
+$XFS_IO_PROG -f -c "copy_range -s 32k -d 48k -l 32k $testdir/file" $testdir/file
+
+echo
+echo destination file O_RDONLY returns EBADF
+$XFS_IO_PROG -f -r -c "copy_range -l 32k $testdir/file" $testdir/copy
+
+echo
+echo destination file O_APPEND returns EBADF
+$XFS_IO_PROG -f -a -c "copy_range -l 32k $testdir/file" $testdir/copy
+
+echo
+echo source/destination as directory returns EISDIR
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $testdir
+$XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
+
+echo
+echo source/destination as blkdev returns EINVAL
+# We need to use an existing major number otherwise we get ENXIO
+# so we _require_loop and use a loop device major number.
+# We are not going to do I/O so minor number doesn't matter.
+mknod $testdir/dev1 b 7 123
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $testdir/dev1
+$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/dev1" $testdir/copy
+
+echo
+echo source/destination as chardev returns EINVAL
+mknod $testdir/dev2 c 1 3
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $testdir/dev2
+$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/dev2" $testdir/copy
+
+echo
+echo source/destination as FIFO returns EINVAL
+mkfifo $testdir/fifo
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $testdir/fifo
+# Pass input pipe as non-blocking open file to avoid old xfs_io (<4.20)
+# opening the pipe in blocking mode and causing the test to hang
+$XFS_IO_PROG -r -n -f -c "open $testdir/copy" -C "copy_range -l 32k -f 0" $testdir/fifo
+
+max_off=$((8 * 2**60 - 65536 - 1))
+min_off=65537
+
+echo
+echo length beyond 8EiB wraps around 0 returns EOVERFLOW
+$XFS_IO_PROG -f -c "copy_range -l 10e -s $max_off $testdir/file" $testdir/copy
+$XFS_IO_PROG -f -c "copy_range -l 10e -d $max_off $testdir/file" $testdir/copy
+
+echo
+echo source range beyond 8TiB returns 0
+$XFS_IO_PROG -c "copy_range -s $max_off -l $min_off -d 0 $testdir/file" $testdir/copy
+
+echo
+echo destination range beyond 8TiB returns EFBIG
+$XFS_IO_PROG -c "copy_range -l $min_off -s 0 -d $max_off $testdir/file" $testdir/copy
+
+echo
+echo destination larger than rlimit returns EFBIG
+rm -f $testdir/copy
+$XFS_IO_PROG -c "truncate 128k" $testdir/file
+
+# need a wrapper so the "File size limit exceeded" error can be filtered
+do_rlimit_copy()
+{
+	$XFS_IO_PROG -f -c "copy_range -l 32k -s 0 -d 16m $testdir/file" $testdir/copy
+}
+
+ulimit -f $((8 * 1024))
+ulimit -c 0
+do_rlimit_copy 2>&1 | grep -o "File size limit exceeded"
+ulimit -f unlimited
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/990.out b/tests/generic/990.out
new file mode 100644
index 00000000..05d137de
--- /dev/null
+++ b/tests/generic/990.out
@@ -0,0 +1,37 @@
+QA output created by 990
+source range overlaps destination range in same file returns EINVAL
+copy_range: Invalid argument
+
+destination file O_RDONLY returns EBADF
+copy_range: Bad file descriptor
+
+destination file O_APPEND returns EBADF
+copy_range: Bad file descriptor
+
+source/destination as directory returns EISDIR
+copy_range: Is a directory
+copy_range: Is a directory
+
+source/destination as blkdev returns EINVAL
+copy_range: Invalid argument
+copy_range: Invalid argument
+
+source/destination as chardev returns EINVAL
+copy_range: Invalid argument
+copy_range: Invalid argument
+
+source/destination as FIFO returns EINVAL
+copy_range: Invalid argument
+copy_range: Invalid argument
+
+length beyond 8EiB wraps around 0 returns EOVERFLOW
+copy_range: Value too large for defined data type
+copy_range: Value too large for defined data type
+
+source range beyond 8TiB returns 0
+
+destination range beyond 8TiB returns EFBIG
+copy_range: File too large
+
+destination larger than rlimit returns EFBIG
+File size limit exceeded
diff --git a/tests/generic/group b/tests/generic/group
index 4c100781..86802d54 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -552,3 +552,4 @@
 547 auto quick log
 988 auto quick copy_range
 989 auto quick copy_range swap
+990 auto quick copy_range
-- 
2.17.1

[PATCH v3 6/6] generic: cross-device copy_file_range test

[Amir Goldstein]

Old kernels do not support cross-device copy_file_range.
A new patch set is aimed at allowing cross-device copy_file_range
using the default in-kernel copy implementation.

[Amir] Split out cross-device copy_range test to a new test.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/991     | 56 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/991.out |  4 ++++
 tests/generic/group   |  1 +
 3 files changed, 61 insertions(+)
 create mode 100755 tests/generic/991
 create mode 100644 tests/generic/991.out

diff --git a/tests/generic/991 b/tests/generic/991
new file mode 100755
index 00000000..94266e89
--- /dev/null
+++ b/tests/generic/991
@@ -0,0 +1,56 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 991
+#
+# Exercise copy_file_range() across devices
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs generic
+
+rm -f $seqres.full
+
+_require_test
+_require_scratch
+_require_xfs_io_command "copy_range"
+
+_scratch_mkfs 2>&1 >> $seqres.full
+_scratch_mount
+
+
+testdir=$TEST_DIR/test-$seq
+rm -rf $testdir
+mkdir $testdir
+rm -f $seqres.full
+
+$XFS_IO_PROG -f -c "pwrite -S 0x61 0 128k" $testdir/file >> $seqres.full 2>&1
+
+$XFS_IO_PROG -f -c "copy_range -l 128k $testdir/file" $SCRATCH_MNT/copy
+cmp $testdir/file $SCRATCH_MNT/copy
+echo "md5sums after xdev copy:"
+md5sum $testdir/file $SCRATCH_MNT/copy | _filter_test_dir | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/991.out b/tests/generic/991.out
new file mode 100644
index 00000000..19c22dfe
--- /dev/null
+++ b/tests/generic/991.out
@@ -0,0 +1,4 @@
+QA output created by 991
+md5sums after xdev copy:
+81615449a98aaaad8dc179b3bec87f38  TEST_DIR/test-991/file
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/copy
diff --git a/tests/generic/group b/tests/generic/group
index 86802d54..bc5dac3b 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -553,3 +553,4 @@
 988 auto quick copy_range
 989 auto quick copy_range swap
 990 auto quick copy_range
+991 auto quick copy_range
-- 
2.17.1

Re: [PATCH v3 2/6] generic: copy_file_range immutable file test

[Eryu Guan]

On Sun, Jun 02, 2019 at 03:41:10PM +0300, Amir Goldstein wrote:
> This test case was split out of Dave Chinner's copy_file_range bounds
> check test to reduce the requirements for running the bounds check.

I think this description should go below "---" and not be in the commit
log. I copied the test description from test here instead.

> 
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
>  tests/generic/988     | 59 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/988.out |  5 ++++
>  tests/generic/group   |  1 +
>  3 files changed, 65 insertions(+)
>  create mode 100755 tests/generic/988
>  create mode 100644 tests/generic/988.out
> 
> diff --git a/tests/generic/988 b/tests/generic/988
> new file mode 100755
> index 00000000..0f4ee4ea
> --- /dev/null
> +++ b/tests/generic/988
> @@ -0,0 +1,59 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Red Hat, Inc.  All Rights Reserved.
> +#
> +# FS QA Test No. 988
> +#
> +# Check that we cannot copy_file_range() to/from an immutable file
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	$CHATTR_PROG -i $testdir/immutable > /dev/null 2>&1
> +	cd /
> +	rm -rf $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs generic
> +
> +rm -f $seqres.full
> +
> +_require_test
> +_require_chattr i
> +_require_xfs_io_command "copy_range"
> +_require_xfs_io_command "chattr"
> +
> +testdir="$TEST_DIR/test-$seq"

I renamed "testdir" to "workdir" to avoid confusing with TEST_DIR, and
moved the definition before _cleanup so we have a valid workdir
definition if any of the requires are not met.

Thanks,
Eryu

> +rm -rf $testdir
> +mkdir $testdir
> +
> +rm -f $seqres.full
> +
> +$XFS_IO_PROG -f -c "pwrite -S 0x61 0 128k" $testdir/file >> $seqres.full 2>&1
> +
> +# we have to open the file to be immutable rw and hold it open over the
> +# chattr command to set it immutable, otherwise we won't be able to open it for
> +# writing after it's been made immutable. (i.e. would exercise file mode checks,
> +# not immutable inode flag checks).
> +echo immutable file returns EPERM
> +$XFS_IO_PROG -f -c "pwrite -S 0x61 0 64k" -c fsync $testdir/immutable | _filter_xfs_io
> +$XFS_IO_PROG -f -c "chattr +i" -c "copy_range -l 32k $testdir/file" $testdir/immutable
> +$XFS_IO_PROG -f -r -c "chattr -i" $testdir/immutable
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/988.out b/tests/generic/988.out
> new file mode 100644
> index 00000000..e74a96bf
> --- /dev/null
> +++ b/tests/generic/988.out
> @@ -0,0 +1,5 @@
> +QA output created by 988
> +immutable file returns EPERM
> +wrote 65536/65536 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +copy_range: Operation not permitted
> diff --git a/tests/generic/group b/tests/generic/group
> index b498eb56..20b95c14 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -550,3 +550,4 @@
>  545 auto quick cap
>  546 auto quick clone enospc log
>  547 auto quick log
> +988 auto quick copy_range
> -- 
> 2.17.1
> 

Re: [PATCH v3 0/6] fstests: copy_file_range() tests

[Eryu Guan]

On Sun, Jun 02, 2019 at 03:41:08PM +0300, Amir Goldstein wrote:
> Eryu,
> 
> This is a re-work of Dave Chinner's copy_file_range() tests which
> I used to verify the kernel fixes of the syscall [1].
> 
> The 2 first tests fix bugs in the interface, so they are appropriate
> for merge IMO.
> 
> The cross-device copy test checks a new functionality, so you may
> want to wait with merging it till after the work is merged upstream.
> 
> The bounds check test depend on a change that was only posted to
> xfsprogs [2]. Without two changes that were merge to xfsprogs v4.20,
> the original test (v1, v2) would hang. Requiring the new copy_range
> flag (copy_range -f) mitigates this problem.
> 
> You may want to wait until the xfs_io change is merged before merging
> the check for the new flag.

Thanks a lot for the detailed explanation! I applied the first three
patches for this week's update.

Thanks,
Eryu

> 
> Thanks,
> Amir.
> 
> [1] https://lore.kernel.org/linux-fsdevel/20190531164701.15112-1-amir73il@gmail.com/
> [2] https://marc.info/?l=linux-xfs&m=155912482124038&w=2
> 
> Changes from v2:
> - Change blockdev in test to loop and _require_loop (Olga)
> - Implement and use _require_xfs_io_command copy_range -f
> 
> Changes from v1:
> - Remove patch to test EINVAL behavior instead of short copy
> - Remove 'chmod -r' permission drop test case
> - Split out test for swap/immutable file copy
> - Split of cross-device copy test
> 
> 
> Amir Goldstein (6):
>   generic: create copy_range group
>   generic: copy_file_range immutable file test
>   generic: copy_file_range swapfile test
>   common/rc: check support for xfs_io copy_range -f N
>   generic: copy_file_range bounds test
>   generic: cross-device copy_file_range test
> 
>  common/rc             |   9 ++-
>  tests/generic/434     |   2 +
>  tests/generic/988     |  59 +++++++++++++++++++
>  tests/generic/988.out |   5 ++
>  tests/generic/989     |  56 ++++++++++++++++++
>  tests/generic/989.out |   4 ++
>  tests/generic/990     | 132 ++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/990.out |  37 ++++++++++++
>  tests/generic/991     |  56 ++++++++++++++++++
>  tests/generic/991.out |   4 ++
>  tests/generic/group   |  14 +++--
>  11 files changed, 372 insertions(+), 6 deletions(-)
>  create mode 100755 tests/generic/988
>  create mode 100644 tests/generic/988.out
>  create mode 100755 tests/generic/989
>  create mode 100644 tests/generic/989.out
>  create mode 100755 tests/generic/990
>  create mode 100644 tests/generic/990.out
>  create mode 100755 tests/generic/991
>  create mode 100644 tests/generic/991.out
> 
> -- 
> 2.17.1
> 

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Theodore Ts'o]

On Sun, Jun 02, 2019 at 03:41:11PM +0300, Amir Goldstein wrote:
> This test case was split out of Dave Chinner's copy_file_range bounds
> check test to reduce the requirements for running the bounds check.
> 
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>

I've just updated to the latest fstests, where this has landed as
generic/554.  This test is failing on ext4, and should fail on all
file systems which do not support copy_file_range (ext4, nfsv3, etc.),
since xfs_io will fall back to emulating this via reading and writing
the file, and this causes a test failure because:

> +echo swap files return ETXTBUSY
> +_format_swapfile $testdir/swapfile 16m
> +swapon $testdir/swapfile
> +$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/file" $testdir/swapfile
> +$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/swapfile" $testdir/copy
> +swapoff $testdir/swapfile

Currently, the VFS doesn't prevent us from reading a swap file.
Perhaps it shouldn't, for security (theatre) reasons, but root can
read the raw block device anyway, so it's really kind of pointless.

I'm not sure what's the best way fix this, but I'm going to exclude
this test in my test appliance builds for now.

     	     	     	       	      	  - Ted

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Amir Goldstein]

On Mon, Jun 10, 2019 at 6:58 AM Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Sun, Jun 02, 2019 at 03:41:11PM +0300, Amir Goldstein wrote:
> > This test case was split out of Dave Chinner's copy_file_range bounds
> > check test to reduce the requirements for running the bounds check.
> >
> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>
> I've just updated to the latest fstests, where this has landed as
> generic/554.  This test is failing on ext4, and should fail on all
> file systems which do not support copy_file_range (ext4, nfsv3, etc.),
> since xfs_io will fall back to emulating this via reading and writing

Why do you think this is xfs_io fall back and not kernel fall back to
do_splice_direct()? Anyway, both cases allow read from swapfile
on upstream.

> the file, and this causes a test failure because:
>
> > +echo swap files return ETXTBUSY
> > +_format_swapfile $testdir/swapfile 16m
> > +swapon $testdir/swapfile
> > +$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/file" $testdir/swapfile
> > +$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/swapfile" $testdir/copy
> > +swapoff $testdir/swapfile
>
> Currently, the VFS doesn't prevent us from reading a swap file.
> Perhaps it shouldn't, for security (theatre) reasons, but root can
> read the raw block device anyway, so it's really kind of pointless.
>

Hmm, my intention with the copy_file_range() behavior was that
it mostly follows user copy limitations/semantics.
I guess preventing copy *from* swapfile doesn't make much sense
and we should relax this check in the new c_f_r bounds check in VFS.

> I'm not sure what's the best way fix this, but I'm going to exclude
> this test in my test appliance builds for now.
>

Trying to understand the desired flow of tests and fixes.
I agree that generic/554 failure may be a test/interface bug that
we should fix in a way that current upstream passes the test for
ext4. Unless there is objection, I will send a patch to fix the test
to only test copy *to* swapfile.

generic/553, OTOH, is expected to fail on upstream kernel.
Are you leaving 553 in appliance build in anticipation to upstream fix?
I guess the answer is in the ext4 IS_IMMUTABLE patch that you
posted and plan to push to upstream/stable sooner than VFS patches.

In any case, the VFS c_f_r patches are aiming for v5.3 and
I will make sure to promote them for stable as well.

Do you think that should there be a different policy w.r.t timing of
merging xfstests tests that fail on upstream kernel?

Thanks,
Amir.

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Amir Goldstein]

On Mon, Jun 10, 2019 at 9:37 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Mon, Jun 10, 2019 at 6:58 AM Theodore Ts'o <tytso@mit.edu> wrote:
> >
> > On Sun, Jun 02, 2019 at 03:41:11PM +0300, Amir Goldstein wrote:
> > > This test case was split out of Dave Chinner's copy_file_range bounds
> > > check test to reduce the requirements for running the bounds check.
> > >
> > > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> >
> > I've just updated to the latest fstests, where this has landed as
> > generic/554.  This test is failing on ext4, and should fail on all
> > file systems which do not support copy_file_range (ext4, nfsv3, etc.),
> > since xfs_io will fall back to emulating this via reading and writing
>
> Why do you think this is xfs_io fall back and not kernel fall back to
> do_splice_direct()? Anyway, both cases allow read from swapfile
> on upstream.
>
> > the file, and this causes a test failure because:
> >
> > > +echo swap files return ETXTBUSY
> > > +_format_swapfile $testdir/swapfile 16m
> > > +swapon $testdir/swapfile
> > > +$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/file" $testdir/swapfile
> > > +$XFS_IO_PROG -f -c "copy_range -l 32k $testdir/swapfile" $testdir/copy
> > > +swapoff $testdir/swapfile
> >
> > Currently, the VFS doesn't prevent us from reading a swap file.
> > Perhaps it shouldn't, for security (theatre) reasons, but root can
> > read the raw block device anyway, so it's really kind of pointless.
> >
>
> Hmm, my intention with the copy_file_range() behavior was that
> it mostly follows user copy limitations/semantics.
> I guess preventing copy *from* swapfile doesn't make much sense
> and we should relax this check in the new c_f_r bounds check in VFS.
>
> > I'm not sure what's the best way fix this, but I'm going to exclude
> > this test in my test appliance builds for now.
> >
>
> Trying to understand the desired flow of tests and fixes.
> I agree that generic/554 failure may be a test/interface bug that
> we should fix in a way that current upstream passes the test for
> ext4. Unless there is objection, I will send a patch to fix the test
> to only test copy *to* swapfile.

I made this change and test still failed on upstream ext4, because
kernel performs copy_file_range() to swapfile.
To me that seems like a real kernel bug, which is addressed by vfs
c_f_r patches, so I don't know if you should be excluding the test
from test appliance after all ?

Thanks,
Amir.

>
> generic/553, OTOH, is expected to fail on upstream kernel.
> Are you leaving 553 in appliance build in anticipation to upstream fix?
> I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> posted and plan to push to upstream/stable sooner than VFS patches.
>
> In any case, the VFS c_f_r patches are aiming for v5.3 and
> I will make sure to promote them for stable as well.
>
> Do you think that should there be a different policy w.r.t timing of
> merging xfstests tests that fail on upstream kernel?
>
> Thanks,
> Amir.

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Theodore Ts'o]

On Mon, Jun 10, 2019 at 09:37:32AM +0300, Amir Goldstein wrote:
>
>Why do you think thhis is xfs_io fall back and not kernel fall back to
>do_splice_direct()? Anyway, both cases allow read from swapfile
>on upstream.

Ah, I had assumed this was changed that was made because if you are
implementing copy_file_range in terms of some kind of reflink-like
mechanism, it becomes super-messy since you know have to break tons
and tons of COW sharing each time the kernel swaps to the swap file.

I didn't think we had (or maybe we did, and I missed it) a discussion
about whether reading from a swap file should be prohibited.
Personally, I think it's security theatre, and not worth the
effort/overhead, but whatever.... my main complaint was with the
unnecessary test failures with upstream kernels.

> Trying to understand the desired flow of tests and fixes. 
> I agree that generic/554 failure may be a test/interface bug that
> we should fix in a way that current upstream passes the test for
> ext4. Unless there is objection, I will send a patch to fix the test
> to only test copy *to* swapfile.
> 
> generic/553, OTOH, is expected to fail on upstream kernel.
> Are you leaving 553 in appliance build in anticipation to upstream fix?
> I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> posted and plan to push to upstream/stable sooner than VFS patches.

So I find it kind of annoying when tests land before the fixes do
upstream.  I still have this in my global_exclude file:

# The proposed fix for generic/484, "locks: change POSIX lock
# ownership on execve when files_struct is displaced" would break NFS
# Jeff Layton and Eric Biederman have some ideas for how to address it
# but fixing it is non-trivial
generic/484

The generic/484 test landed in August 2018, and as far as I know, the
issue which it is testing for *still* hasn't been fixed upstream.
(There were issues raised with the proposed fix, and it looks like the
people looking at the kernel change have lost interest.)

The problem is that there are people who are trying to use xfstests to
look for failures, and unless they start digging into the kernel
archives from last year, they won't understand that generic/484 is a
known failing test, and it will get fixed....someday.

For people in the know, or for people who use my kvm-xfstests,
gce-xfstests, it's not a big problem, since I've already blacklisted
the test.  But not everyone (and in fact, probably most people don't)
use my front end scripts.

For generic/553, I have a fix in ext4 so it will clear the failure,
and that's fine, since I think we've all agreed in principle what the
correct fix will be, and presumably it will get fixed soon.  At that
point, I might revert the commit from ext4, and rely on the VFS to
catch the error, but the overhead of a few extra unlikely() tests
aren't that big.  But yeah, I did that mainly because unnecessary test
failures because doing an ext4-specific fix didn't have many
downsides, and one risk of adding tests to the global exclude file is
that I then have to remember to remove it from the global exclude file
when the issue is finally fixed upstream.

> Do you think that should there be a different policy w.r.t timing of
> merging xfstests tests that fail on upstream kernel?

That's my opinion, and generic/484 is the best argument for why we
should wait.  Other people may have other opinions though, and I have
a workaround, so I don't feel super-strong about it.  (generic/454 is
now the second test in my global exclude file.  :-)

At the very *least* there should be a comment in the test that fix is
pending, and might not be applied yet, with a URL to the mailing list
discussion.  That will save effort when months (years?) go by, and the
fix still hasn't landed the upstream kernel....

	       	       	      	      - Ted

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Darrick J. Wong]

On Mon, Jun 10, 2019 at 09:31:31AM -0400, Theodore Ts'o wrote:
> On Mon, Jun 10, 2019 at 09:37:32AM +0300, Amir Goldstein wrote:
> >
> >Why do you think thhis is xfs_io fall back and not kernel fall back to
> >do_splice_direct()? Anyway, both cases allow read from swapfile
> >on upstream.
> 
> Ah, I had assumed this was changed that was made because if you are
> implementing copy_file_range in terms of some kind of reflink-like
> mechanism, it becomes super-messy since you know have to break tons
> and tons of COW sharing each time the kernel swaps to the swap file.
> 
> I didn't think we had (or maybe we did, and I missed it) a discussion
> about whether reading from a swap file should be prohibited.
> Personally, I think it's security theatre, and not worth the
> effort/overhead, but whatever.... my main complaint was with the
> unnecessary test failures with upstream kernels.
> 
> > Trying to understand the desired flow of tests and fixes. 
> > I agree that generic/554 failure may be a test/interface bug that
> > we should fix in a way that current upstream passes the test for
> > ext4. Unless there is objection, I will send a patch to fix the test
> > to only test copy *to* swapfile.
> > 
> > generic/553, OTOH, is expected to fail on upstream kernel.
> > Are you leaving 553 in appliance build in anticipation to upstream fix?
> > I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> > posted and plan to push to upstream/stable sooner than VFS patches.
> 
> So I find it kind of annoying when tests land before the fixes do
> upstream.  I still have this in my global_exclude file:

Yeah, it's awkward for VFS fixes because on the one hand we don't want
to have multiyear regressions like generic/484, but OTOH stuffing tests
in before code goes upstream enables broader testing by the other fs
maintainers.

In any case, the fixes are in the copy-range-fixes branch which I'm
finally publishing...

> # The proposed fix for generic/484, "locks: change POSIX lock
> # ownership on execve when files_struct is displaced" would break NFS
> # Jeff Layton and Eric Biederman have some ideas for how to address it
> # but fixing it is non-trivial

Also, uh, can we remove this from the auto and quick groups for now?

--D

> generic/484
> 
> The generic/484 test landed in August 2018, and as far as I know, the
> issue which it is testing for *still* hasn't been fixed upstream.
> (There were issues raised with the proposed fix, and it looks like the
> people looking at the kernel change have lost interest.)
> 
> The problem is that there are people who are trying to use xfstests to
> look for failures, and unless they start digging into the kernel
> archives from last year, they won't understand that generic/484 is a
> known failing test, and it will get fixed....someday.
> 
> For people in the know, or for people who use my kvm-xfstests,
> gce-xfstests, it's not a big problem, since I've already blacklisted
> the test.  But not everyone (and in fact, probably most people don't)
> use my front end scripts.
> 
> For generic/553, I have a fix in ext4 so it will clear the failure,
> and that's fine, since I think we've all agreed in principle what the
> correct fix will be, and presumably it will get fixed soon.  At that
> point, I might revert the commit from ext4, and rely on the VFS to
> catch the error, but the overhead of a few extra unlikely() tests
> aren't that big.  But yeah, I did that mainly because unnecessary test
> failures because doing an ext4-specific fix didn't have many
> downsides, and one risk of adding tests to the global exclude file is
> that I then have to remember to remove it from the global exclude file
> when the issue is finally fixed upstream.
> 
> > Do you think that should there be a different policy w.r.t timing of
> > merging xfstests tests that fail on upstream kernel?
> 
> That's my opinion, and generic/484 is the best argument for why we
> should wait.  Other people may have other opinions though, and I have
> a workaround, so I don't feel super-strong about it.  (generic/454 is
> now the second test in my global exclude file.  :-)
> 
> At the very *least* there should be a comment in the test that fix is
> pending, and might not be applied yet, with a URL to the mailing list
> discussion.  That will save effort when months (years?) go by, and the
> fix still hasn't landed the upstream kernel....
> 
> 	       	       	      	      - Ted

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Amir Goldstein]

On Mon, Jun 10, 2019 at 7:06 PM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Mon, Jun 10, 2019 at 09:31:31AM -0400, Theodore Ts'o wrote:
> > On Mon, Jun 10, 2019 at 09:37:32AM +0300, Amir Goldstein wrote:
> > >
> > >Why do you think thhis is xfs_io fall back and not kernel fall back to
> > >do_splice_direct()? Anyway, both cases allow read from swapfile
> > >on upstream.
> >
> > Ah, I had assumed this was changed that was made because if you are
> > implementing copy_file_range in terms of some kind of reflink-like
> > mechanism, it becomes super-messy since you know have to break tons
> > and tons of COW sharing each time the kernel swaps to the swap file.
> >
> > I didn't think we had (or maybe we did, and I missed it) a discussion
> > about whether reading from a swap file should be prohibited.
> > Personally, I think it's security theatre, and not worth the
> > effort/overhead, but whatever.... my main complaint was with the
> > unnecessary test failures with upstream kernels.
> >
> > > Trying to understand the desired flow of tests and fixes.
> > > I agree that generic/554 failure may be a test/interface bug that
> > > we should fix in a way that current upstream passes the test for
> > > ext4. Unless there is objection, I will send a patch to fix the test
> > > to only test copy *to* swapfile.
> > >
> > > generic/553, OTOH, is expected to fail on upstream kernel.
> > > Are you leaving 553 in appliance build in anticipation to upstream fix?
> > > I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> > > posted and plan to push to upstream/stable sooner than VFS patches.
> >
> > So I find it kind of annoying when tests land before the fixes do
> > upstream.  I still have this in my global_exclude file:
>
> Yeah, it's awkward for VFS fixes because on the one hand we don't want
> to have multiyear regressions like generic/484, but OTOH stuffing tests
> in before code goes upstream enables broader testing by the other fs
> maintainers.

And to prove this point, Ted pointed out a test bug in 554, which also
affects the kernel and man pages fixes, so it was really worth it ;-)

>
> In any case, the fixes are in the copy-range-fixes branch which I'm
> finally publishing...
>
> > # The proposed fix for generic/484, "locks: change POSIX lock
> > # ownership on execve when files_struct is displaced" would break NFS
> > # Jeff Layton and Eric Biederman have some ideas for how to address it
> > # but fixing it is non-trivial
>
> Also, uh, can we remove this from the auto and quick groups for now?
>

I am not opposed to removing these test from auto,quick, although removing
from quick is a bit shady. I would like to mark them explicitly with group
known_issues, so that users can run ./check -g quick -x known_issues.
BTW, overlay/061 is also a known_issue that is going to be hard to fix.

But anyway, neither 553 nor 554 fall into that category.

Thanks,
Amir.

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Darrick J. Wong]

On Mon, Jun 10, 2019 at 07:54:52PM +0300, Amir Goldstein wrote:
> On Mon, Jun 10, 2019 at 7:06 PM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> >
> > On Mon, Jun 10, 2019 at 09:31:31AM -0400, Theodore Ts'o wrote:
> > > On Mon, Jun 10, 2019 at 09:37:32AM +0300, Amir Goldstein wrote:
> > > >
> > > >Why do you think thhis is xfs_io fall back and not kernel fall back to
> > > >do_splice_direct()? Anyway, both cases allow read from swapfile
> > > >on upstream.
> > >
> > > Ah, I had assumed this was changed that was made because if you are
> > > implementing copy_file_range in terms of some kind of reflink-like
> > > mechanism, it becomes super-messy since you know have to break tons
> > > and tons of COW sharing each time the kernel swaps to the swap file.
> > >
> > > I didn't think we had (or maybe we did, and I missed it) a discussion
> > > about whether reading from a swap file should be prohibited.
> > > Personally, I think it's security theatre, and not worth the
> > > effort/overhead, but whatever.... my main complaint was with the
> > > unnecessary test failures with upstream kernels.
> > >
> > > > Trying to understand the desired flow of tests and fixes.
> > > > I agree that generic/554 failure may be a test/interface bug that
> > > > we should fix in a way that current upstream passes the test for
> > > > ext4. Unless there is objection, I will send a patch to fix the test
> > > > to only test copy *to* swapfile.
> > > >
> > > > generic/553, OTOH, is expected to fail on upstream kernel.
> > > > Are you leaving 553 in appliance build in anticipation to upstream fix?
> > > > I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> > > > posted and plan to push to upstream/stable sooner than VFS patches.
> > >
> > > So I find it kind of annoying when tests land before the fixes do
> > > upstream.  I still have this in my global_exclude file:
> >
> > Yeah, it's awkward for VFS fixes because on the one hand we don't want
> > to have multiyear regressions like generic/484, but OTOH stuffing tests
> > in before code goes upstream enables broader testing by the other fs
> > maintainers.
> 
> And to prove this point, Ted pointed out a test bug in 554, which also
> affects the kernel and man pages fixes, so it was really worth it ;-)

:D

> >
> > In any case, the fixes are in the copy-range-fixes branch which I'm
> > finally publishing...
> >
> > > # The proposed fix for generic/484, "locks: change POSIX lock
> > > # ownership on execve when files_struct is displaced" would break NFS
> > > # Jeff Layton and Eric Biederman have some ideas for how to address it
> > > # but fixing it is non-trivial
> >
> > Also, uh, can we remove this from the auto and quick groups for now?
> >
> 
> I am not opposed to removing these test from auto,quick, although removing
> from quick is a bit shady. I would like to mark them explicitly with group
> known_issues, so that users can run ./check -g quick -x known_issues.
> BTW, overlay/061 is also a known_issue that is going to be hard to fix.
> 
> But anyway, neither 553 nor 554 fall into that category.

Sorry, I was unclear -- I was asking to remove g/484 from auto/quick.

--D

> Thanks,
> Amir.

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Eryu Guan]

On Mon, Jun 10, 2019 at 09:31:31AM -0400, Theodore Ts'o wrote:
> On Mon, Jun 10, 2019 at 09:37:32AM +0300, Amir Goldstein wrote:
> >
> >Why do you think thhis is xfs_io fall back and not kernel fall back to
> >do_splice_direct()? Anyway, both cases allow read from swapfile
> >on upstream.
> 
> Ah, I had assumed this was changed that was made because if you are
> implementing copy_file_range in terms of some kind of reflink-like
> mechanism, it becomes super-messy since you know have to break tons
> and tons of COW sharing each time the kernel swaps to the swap file.
> 
> I didn't think we had (or maybe we did, and I missed it) a discussion
> about whether reading from a swap file should be prohibited.
> Personally, I think it's security theatre, and not worth the
> effort/overhead, but whatever.... my main complaint was with the
> unnecessary test failures with upstream kernels.
> 
> > Trying to understand the desired flow of tests and fixes. 
> > I agree that generic/554 failure may be a test/interface bug that
> > we should fix in a way that current upstream passes the test for
> > ext4. Unless there is objection, I will send a patch to fix the test
> > to only test copy *to* swapfile.
> > 
> > generic/553, OTOH, is expected to fail on upstream kernel.
> > Are you leaving 553 in appliance build in anticipation to upstream fix?
> > I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> > posted and plan to push to upstream/stable sooner than VFS patches.
> 
> So I find it kind of annoying when tests land before the fixes do
> upstream.  I still have this in my global_exclude file:
> 
> # The proposed fix for generic/484, "locks: change POSIX lock
> # ownership on execve when files_struct is displaced" would break NFS
> # Jeff Layton and Eric Biederman have some ideas for how to address it
> # but fixing it is non-trivial
> generic/484
> 
> The generic/484 test landed in August 2018, and as far as I know, the
> issue which it is testing for *still* hasn't been fixed upstream.
> (There were issues raised with the proposed fix, and it looks like the
> people looking at the kernel change have lost interest.)

I usually push "known failing" tests only when there's a known & pending
fix which is expected to be merged into mainline kernel soon. And as
Darrick stated, "enables broader testing by the other fs maintainers."
and could bring broader attention of the failure.

But generic/484 is a bit unfortunate. It was in that exact situation
back then (or at least gave me the impression that the fix would be
merged soon), but apparaently things changed after test being applied..

> 
> The problem is that there are people who are trying to use xfstests to
> look for failures, and unless they start digging into the kernel
> archives from last year, they won't understand that generic/484 is a
> known failing test, and it will get fixed....someday.
> 
> For people in the know, or for people who use my kvm-xfstests,
> gce-xfstests, it's not a big problem, since I've already blacklisted
> the test.  But not everyone (and in fact, probably most people don't)
> use my front end scripts.
> 
> For generic/553, I have a fix in ext4 so it will clear the failure,
> and that's fine, since I think we've all agreed in principle what the
> correct fix will be, and presumably it will get fixed soon.  At that
> point, I might revert the commit from ext4, and rely on the VFS to
> catch the error, but the overhead of a few extra unlikely() tests
> aren't that big.  But yeah, I did that mainly because unnecessary test
> failures because doing an ext4-specific fix didn't have many
> downsides, and one risk of adding tests to the global exclude file is
> that I then have to remember to remove it from the global exclude file
> when the issue is finally fixed upstream.
> 
> > Do you think that should there be a different policy w.r.t timing of
> > merging xfstests tests that fail on upstream kernel?
> 
> That's my opinion, and generic/484 is the best argument for why we
> should wait.  Other people may have other opinions though, and I have
> a workaround, so I don't feel super-strong about it.  (generic/454 is
> now the second test in my global exclude file.  :-)

I don't see generic/454 failing with ext4 (I'm testing with default
mkfs/mount options, kernel is 5.2-rc2). But IMHO, I think generic/454 is
different, it's not a targeted regression test, it's kind of generic
test that should work for all filesystems.

> 
> At the very *least* there should be a comment in the test that fix is
> pending, and might not be applied yet, with a URL to the mailing list
> discussion.  That will save effort when months (years?) go by, and the
> fix still hasn't landed the upstream kernel....

Agreed, I've been making sure there's a comment referring to the fix or
pending fix (e.g. only commit summary no hash ID) for such targeted
regression tests.

Thanks,
Eryu

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Eryu Guan]

On Mon, Jun 10, 2019 at 09:06:16AM -0700, Darrick J. Wong wrote:
> On Mon, Jun 10, 2019 at 09:31:31AM -0400, Theodore Ts'o wrote:
> > On Mon, Jun 10, 2019 at 09:37:32AM +0300, Amir Goldstein wrote:
> > >
> > >Why do you think thhis is xfs_io fall back and not kernel fall back to
> > >do_splice_direct()? Anyway, both cases allow read from swapfile
> > >on upstream.
> > 
> > Ah, I had assumed this was changed that was made because if you are
> > implementing copy_file_range in terms of some kind of reflink-like
> > mechanism, it becomes super-messy since you know have to break tons
> > and tons of COW sharing each time the kernel swaps to the swap file.
> > 
> > I didn't think we had (or maybe we did, and I missed it) a discussion
> > about whether reading from a swap file should be prohibited.
> > Personally, I think it's security theatre, and not worth the
> > effort/overhead, but whatever.... my main complaint was with the
> > unnecessary test failures with upstream kernels.
> > 
> > > Trying to understand the desired flow of tests and fixes. 
> > > I agree that generic/554 failure may be a test/interface bug that
> > > we should fix in a way that current upstream passes the test for
> > > ext4. Unless there is objection, I will send a patch to fix the test
> > > to only test copy *to* swapfile.
> > > 
> > > generic/553, OTOH, is expected to fail on upstream kernel.
> > > Are you leaving 553 in appliance build in anticipation to upstream fix?
> > > I guess the answer is in the ext4 IS_IMMUTABLE patch that you
> > > posted and plan to push to upstream/stable sooner than VFS patches.
> > 
> > So I find it kind of annoying when tests land before the fixes do
> > upstream.  I still have this in my global_exclude file:
> 
> Yeah, it's awkward for VFS fixes because on the one hand we don't want
> to have multiyear regressions like generic/484, but OTOH stuffing tests
> in before code goes upstream enables broader testing by the other fs
> maintainers.
> 
> In any case, the fixes are in the copy-range-fixes branch which I'm
> finally publishing...
> 
> > # The proposed fix for generic/484, "locks: change POSIX lock
> > # ownership on execve when files_struct is displaced" would break NFS
> > # Jeff Layton and Eric Biederman have some ideas for how to address it
> > # but fixing it is non-trivial
> 
> Also, uh, can we remove this from the auto and quick groups for now?

I'm fine with that :)

Thanks,
Eryu

Re: [PATCH v3 3/6] generic: copy_file_range swapfile test

[Eryu Guan]

On Tue, Jun 11, 2019 at 10:12:22AM +0800, Eryu Guan wrote:
[snip]
> > 
> > > Do you think that should there be a different policy w.r.t timing of
> > > merging xfstests tests that fail on upstream kernel?
> > 
> > That's my opinion, and generic/484 is the best argument for why we
> > should wait.  Other people may have other opinions though, and I have
> > a workaround, so I don't feel super-strong about it.  (generic/454 is
> > now the second test in my global exclude file.  :-)
> 
> I don't see generic/454 failing with ext4 (I'm testing with default
> mkfs/mount options, kernel is 5.2-rc2). But IMHO, I think generic/454 is

Oh, I see, I think you meant generic/554 not generic/454 (thanks Darrick
for pointing that out :)

> different, it's not a targeted regression test, it's kind of generic
> test that should work for all filesystems.
> 
> > 
> > At the very *least* there should be a comment in the test that fix is
> > pending, and might not be applied yet, with a URL to the mailing list
> > discussion.  That will save effort when months (years?) go by, and the
> > fix still hasn't landed the upstream kernel....
> 
> Agreed, I've been making sure there's a comment referring to the fix or
> pending fix (e.g. only commit summary no hash ID) for such targeted
> regression tests.

And I took generic/55[34] as generic tests not targeted regression test.
But looks like it's better to reference the fixes anyway.

Amir, would you mind adding such references to generic/55[34] as well?

Thanks,
Eryu