[PATCH v2 00/16] generic: add some mmap CoW tests

[PATCH v2 00/16] generic: add some mmap CoW tests

[Shiyang Ruan]

Changes since v1:
  - Replace 'xxx' in each header with real sequence number
  - Combine two patchsets together because they both aim to add mmap tests

This series is to add missing mmap CoW tests in some sets of testcases:
Set 1. CoW on shared blocks which are interleaving reflinked by two files.
  - P1:            add mmap test
Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
   the destination file.
  - P2 ~ P6:       add mmap test
Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
   the source file.
  - P11, P12:      add missing buffered-io and direct-io test for mixed blocks
  - P7 ~ P10, P13: add mmap test
Set 4. races tests
  - P14: race between reflink and mmap reading
  - P15: race between mmap writing to and reflink source file
  - P16: race between mmap writing to and reflink target file


Shiyang Ruan (16):
  generic: add mmap CoW test for ranges of two shared files
  generic: add mmap CoW test for regular&destination extents
  generic: add mmap CoW test for unwritten&destination extents
  generic: add mmap CoW test for holes&destination extents
  generic: add mmap CoW test for delalloc&destination extents
  generic: add mmap CoW test for mixed&destination extents
  generic: add mmap CoW test for regular&source extents
  generic: add mmap CoW test for unwritten&source extents
  generic: add mmap CoW test for holes&source extents
  generic: add mmap CoW test for delalloc&source extents
  generic: add buffered-io CoW test for mixed&source extents
  generic: add direct-io CoW test for mixed&source extents
  generic: add mmap CoW test for mixed&source extents
  generic: add race test between reflink and mmap read
  generic: add race test that mmap write to source of reflink
  generic: add race test that mmap write to target of reflink

 common/reflink        | 74 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/900     | 69 ++++++++++++++++++++++++++++++++++++++++
 tests/generic/900.out | 14 ++++++++
 tests/generic/901     | 59 ++++++++++++++++++++++++++++++++++
 tests/generic/901.out | 12 +++++++
 tests/generic/902     | 59 ++++++++++++++++++++++++++++++++++
 tests/generic/902.out | 12 +++++++
 tests/generic/903     | 59 ++++++++++++++++++++++++++++++++++
 tests/generic/903.out | 12 +++++++
 tests/generic/904     | 61 +++++++++++++++++++++++++++++++++++
 tests/generic/904.out | 12 +++++++
 tests/generic/905     | 68 +++++++++++++++++++++++++++++++++++++++
 tests/generic/905.out | 12 +++++++
 tests/generic/906     | 60 +++++++++++++++++++++++++++++++++++
 tests/generic/906.out | 12 +++++++
 tests/generic/907     | 62 ++++++++++++++++++++++++++++++++++++
 tests/generic/907.out | 12 +++++++
 tests/generic/908     | 62 ++++++++++++++++++++++++++++++++++++
 tests/generic/908.out | 12 +++++++
 tests/generic/909     | 64 +++++++++++++++++++++++++++++++++++++
 tests/generic/909.out | 12 +++++++
 tests/generic/910     | 66 ++++++++++++++++++++++++++++++++++++++
 tests/generic/910.out | 12 +++++++
 tests/generic/911     | 66 ++++++++++++++++++++++++++++++++++++++
 tests/generic/911.out | 12 +++++++
 tests/generic/912     | 67 +++++++++++++++++++++++++++++++++++++++
 tests/generic/912.out | 12 +++++++
 tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++
 tests/generic/913.out |  5 +++
 tests/generic/914     | 64 +++++++++++++++++++++++++++++++++++++
 tests/generic/914.out |  4 +++
 tests/generic/915     | 64 +++++++++++++++++++++++++++++++++++++
 tests/generic/915.out |  4 +++
 33 files changed, 1267 insertions(+)
 create mode 100755 tests/generic/900
 create mode 100644 tests/generic/900.out
 create mode 100755 tests/generic/901
 create mode 100644 tests/generic/901.out
 create mode 100755 tests/generic/902
 create mode 100644 tests/generic/902.out
 create mode 100755 tests/generic/903
 create mode 100644 tests/generic/903.out
 create mode 100755 tests/generic/904
 create mode 100644 tests/generic/904.out
 create mode 100755 tests/generic/905
 create mode 100644 tests/generic/905.out
 create mode 100755 tests/generic/906
 create mode 100644 tests/generic/906.out
 create mode 100755 tests/generic/907
 create mode 100644 tests/generic/907.out
 create mode 100755 tests/generic/908
 create mode 100644 tests/generic/908.out
 create mode 100755 tests/generic/909
 create mode 100644 tests/generic/909.out
 create mode 100755 tests/generic/910
 create mode 100644 tests/generic/910.out
 create mode 100755 tests/generic/911
 create mode 100644 tests/generic/911.out
 create mode 100755 tests/generic/912
 create mode 100644 tests/generic/912.out
 create mode 100755 tests/generic/913
 create mode 100644 tests/generic/913.out
 create mode 100755 tests/generic/914
 create mode 100644 tests/generic/914.out
 create mode 100755 tests/generic/915
 create mode 100644 tests/generic/915.out

-- 
2.34.0

[PATCH v2 01/16] generic: add mmap CoW test for ranges of two shared files

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents. (MMAP version of generic/185,183)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/900     | 69 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/900.out | 14 +++++++++
 2 files changed, 83 insertions(+)
 create mode 100755 tests/generic/900
 create mode 100644 tests/generic/900.out

diff --git a/tests/generic/900 b/tests/generic/900
new file mode 100755
index 00000000..de76cd00
--- /dev/null
+++ b/tests/generic/900
@@ -0,0 +1,69 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 900
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents. (MMAP version of generic/185,183)
+#   - Create two files
+#   - Reflink the odd blocks of the first file into a third file.
+#   - Reflink the even blocks of the second file into the third file.
+#   - mmap CoW across the halfway mark.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_pwrite_byte 0x61 0 $filesize $testdir/file1 >> $seqres.full
+_pwrite_byte 0x62 0 $filesize $testdir/file2 >> $seqres.full
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range $testdir/file1 $((blksz * f)) $testdir/file3 $((blksz * f)) $blksz >> $seqres.full
+	_pwrite_byte 0x61 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
+done
+seq 1 2 $((nr-1)) | while read f; do
+	_reflink_range $testdir/file2 $((blksz * f)) $testdir/file3 $((blksz * f)) $blksz >> $seqres.full
+	_pwrite_byte 0x62 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
+done
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/900.out b/tests/generic/900.out
new file mode 100644
index 00000000..b2569ab0
--- /dev/null
+++ b/tests/generic/900.out
@@ -0,0 +1,14 @@
+QA output created by 900
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-900/file1
+b83f9394092e15bdcda585cd8e776dc6  SCRATCH_MNT/test-900/file2
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-900/file3
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-900/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-900/file1
+b83f9394092e15bdcda585cd8e776dc6  SCRATCH_MNT/test-900/file2
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-900/file3
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-900/file3.chk
-- 
2.34.0

[PATCH v2 02/16] generic: add mmap CoW test for regular&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some regular, some not.
(MMAP version of generic/197,196)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/901     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/901.out | 12 +++++++++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/generic/901
 create mode 100644 tests/generic/901.out

diff --git a/tests/generic/901 b/tests/generic/901
new file mode 100755
index 00000000..b9f4fbc0
--- /dev/null
+++ b/tests/generic/901
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 901
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some regular, some not.
+# (MMAP version of generic/197,196)
+#   - Create two files.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/901.out b/tests/generic/901.out
new file mode 100644
index 00000000..c3983e87
--- /dev/null
+++ b/tests/generic/901.out
@@ -0,0 +1,12 @@
+QA output created by 901
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-901/file1
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-901/file3
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-901/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-901/file1
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-901/file3
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-901/file3.chk
-- 
2.34.0

[PATCH v2 03/16] generic: add mmap CoW test for unwritten&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some unwritten, some not.
(MMAP version of generic/189,188)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/902     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/902.out | 12 +++++++++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/generic/902
 create mode 100644 tests/generic/902.out

diff --git a/tests/generic/902 b/tests/generic/902
new file mode 100755
index 00000000..457b039a
--- /dev/null
+++ b/tests/generic/902
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 902
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some unwritten, some not.
+# (MMAP version of generic/189,188)
+#   - Create a file and fallocate a second file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/902.out b/tests/generic/902.out
new file mode 100644
index 00000000..bc951255
--- /dev/null
+++ b/tests/generic/902.out
@@ -0,0 +1,12 @@
+QA output created by 902
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-902/file1
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-902/file3
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-902/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-902/file1
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-902/file3
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-902/file3.chk
-- 
2.34.0

[PATCH v2 04/16] generic: add mmap CoW test for holes&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some holes, some not.
(MMAP version of generic/191,190)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/903     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/903.out | 12 +++++++++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/generic/903
 create mode 100644 tests/generic/903.out

diff --git a/tests/generic/903 b/tests/generic/903
new file mode 100755
index 00000000..d0de3f26
--- /dev/null
+++ b/tests/generic/903
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 903
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some holes, some not.
+# (MMAP version of generic/191,190)
+#   - Create a file and truncate a second file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/903.out b/tests/generic/903.out
new file mode 100644
index 00000000..bf57acc7
--- /dev/null
+++ b/tests/generic/903.out
@@ -0,0 +1,12 @@
+QA output created by 903
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-903/file1
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-903/file3
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-903/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-903/file1
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-903/file3
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-903/file3.chk
-- 
2.34.0

[PATCH v2 05/16] generic: add mmap CoW test for delalloc&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some delalloc, some not.
(MMAP version of generic/195,194)

Also, add a check to make sure the delalloc block is accutally exist.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        |  3 +++
 tests/generic/904     | 61 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/904.out | 12 +++++++++
 3 files changed, 76 insertions(+)
 create mode 100755 tests/generic/904
 create mode 100644 tests/generic/904.out

diff --git a/common/reflink b/common/reflink
index ee60398e..38210673 100644
--- a/common/reflink
+++ b/common/reflink
@@ -296,6 +296,9 @@ _weave_reflink_holes_delalloc() {
 		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
 		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
 	done
+	# make sure we actually got delalloc block
+	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
+		_notrun "test requires delayed allocation writes"
 }
 
 # Create a file of interleaved regular blocks and reflinked blocks
diff --git a/tests/generic/904 b/tests/generic/904
new file mode 100755
index 00000000..69703d70
--- /dev/null
+++ b/tests/generic/904
@@ -0,0 +1,61 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 904
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some delalloc, some not.
+# (MMAP version of generic/195,194)
+#   - Create a file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - Buffered write the even blocks of the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_weave_reflink_holes_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/904.out b/tests/generic/904.out
new file mode 100644
index 00000000..cc993394
--- /dev/null
+++ b/tests/generic/904.out
@@ -0,0 +1,12 @@
+QA output created by 904
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3.chk
-- 
2.34.0

[PATCH v2 06/16] generic: add mmap CoW test for mixed&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, mixed with reflinked, unwritten, hole, regular and
delalloc blocks.
(MMAP version of generic/200,199)

Also, add a check to make sure the delalloc block is accutally exist.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        |  3 ++
 tests/generic/905     | 68 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/905.out | 12 ++++++++
 3 files changed, 83 insertions(+)
 create mode 100755 tests/generic/905
 create mode 100644 tests/generic/905.out

diff --git a/common/reflink b/common/reflink
index 38210673..84173d59 100644
--- a/common/reflink
+++ b/common/reflink
@@ -362,6 +362,9 @@ _weave_reflink_rainbow_delalloc() {
 		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
 		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
 	done
+	# make sure we actually got delalloc block
+	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
+		_notrun "test requires delayed allocation writes"
 }
 
 # Make the source file have interleaved regular blocks and reflinked blocks
diff --git a/tests/generic/905 b/tests/generic/905
new file mode 100755
index 00000000..56cdcd27
--- /dev/null
+++ b/tests/generic/905
@@ -0,0 +1,68 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 905
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+# (MMAP version of generic/200,199)
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+# now cow
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/905.out b/tests/generic/905.out
new file mode 100644
index 00000000..6fb0be5f
--- /dev/null
+++ b/tests/generic/905.out
@@ -0,0 +1,12 @@
+QA output created by 905
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3.chk
-- 
2.34.0

[PATCH v2 07/16] generic: add mmap CoW test for regular&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers regular unshared and regular shared blocks.
(MMAP version of generic/284,287)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/906     | 60 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/906.out | 12 +++++++++
 2 files changed, 72 insertions(+)
 create mode 100755 tests/generic/906
 create mode 100644 tests/generic/906.out

diff --git a/tests/generic/906 b/tests/generic/906
new file mode 100755
index 00000000..953348e8
--- /dev/null
+++ b/tests/generic/906
@@ -0,0 +1,60 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 906
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers regular unshared and regular shared blocks.
+# (MMAP version of generic/284,287)
+#   - Create two files.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/906.out b/tests/generic/906.out
new file mode 100644
index 00000000..4949769b
--- /dev/null
+++ b/tests/generic/906.out
@@ -0,0 +1,12 @@
+QA output created by 906
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-906/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-906/file3
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-906/file1.chk
+mmap CoW across the transition
+Compare files
+2dc2b131303b2f70ddb480778caac8b7  SCRATCH_MNT/test-906/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-906/file3
+2dc2b131303b2f70ddb480778caac8b7  SCRATCH_MNT/test-906/file1.chk
-- 
2.34.0

[PATCH v2 08/16] generic: add mmap CoW test for unwritten&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers unwritten and regular shared blocks.
(MMAP version of generic/289,290)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/907     | 62 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/907.out | 12 +++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/generic/907
 create mode 100644 tests/generic/907.out

diff --git a/tests/generic/907 b/tests/generic/907
new file mode 100755
index 00000000..61c9293a
--- /dev/null
+++ b/tests/generic/907
@@ -0,0 +1,62 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 907
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers unwritten and regular shared blocks.
+# (MMAP version of generic/289,290)
+#   - Create two files.
+#   - fallocate the first file.
+#   - Write the odd blocks of the first file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/907.out b/tests/generic/907.out
new file mode 100644
index 00000000..c18b849c
--- /dev/null
+++ b/tests/generic/907.out
@@ -0,0 +1,12 @@
+QA output created by 907
+Format and mount
+Create the original files
+Compare files
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-907/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-907/file3
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-907/file1.chk
+mmap CoW across the transition
+Compare files
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-907/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-907/file3
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-907/file1.chk
-- 
2.34.0

[PATCH v2 09/16] generic: add mmap CoW test for holes&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers holes and regular shared blocks.
(MMAP version of generic/291,292)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/908     | 62 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/908.out | 12 +++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/generic/908
 create mode 100644 tests/generic/908.out

diff --git a/tests/generic/908 b/tests/generic/908
new file mode 100755
index 00000000..ad8de699
--- /dev/null
+++ b/tests/generic/908
@@ -0,0 +1,62 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 908
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers holes and regular shared blocks.
+# (MMAP version of generic/291,292)
+#   - Create two files.
+#   - Truncate the first file.
+#   - Write the odd blocks of the first file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/908.out b/tests/generic/908.out
new file mode 100644
index 00000000..c8c670ac
--- /dev/null
+++ b/tests/generic/908.out
@@ -0,0 +1,12 @@
+QA output created by 908
+Format and mount
+Create the original files
+Compare files
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-908/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-908/file3
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-908/file1.chk
+mmap CoW across the transition
+Compare files
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-908/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-908/file3
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-908/file1.chk
-- 
2.34.0

[PATCH v2 10/16] generic: add mmap CoW test for delalloc&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers delalloc blocks and regular shared blocks.
(MMAP version of generic/293,295)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        |  3 ++
 tests/generic/909     | 64 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/909.out | 12 ++++++++
 3 files changed, 79 insertions(+)
 create mode 100755 tests/generic/909
 create mode 100644 tests/generic/909.out

diff --git a/common/reflink b/common/reflink
index 84173d59..e304ddc8 100644
--- a/common/reflink
+++ b/common/reflink
@@ -431,4 +431,7 @@ _sweave_reflink_holes_delalloc() {
 		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile
 		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile.chk
 	done
+	# make sure we actually got delalloc block
+	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
+		_notrun "test requires delayed allocation writes"
 }
diff --git a/tests/generic/909 b/tests/generic/909
new file mode 100755
index 00000000..ef1233ea
--- /dev/null
+++ b/tests/generic/909
@@ -0,0 +1,64 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 909
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers delalloc blocks and regular shared blocks.
+# (MMAP version of generic/293,295)
+#   - Create two files.
+#   - Truncate the first file.
+#   - Write the odd blocks of the first file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - Write the even blocks of the first file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_holes_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/909.out b/tests/generic/909.out
new file mode 100644
index 00000000..4f9a5d33
--- /dev/null
+++ b/tests/generic/909.out
@@ -0,0 +1,12 @@
+QA output created by 909
+Format and mount
+Create the original files
+Compare files
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1.chk
+mmap CoW across the transition
+Compare files
+35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
+35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1.chk
-- 
2.34.0

[PATCH v2 11/16] generic: add buffered-io CoW test for mixed&source extents

[Shiyang Ruan]

Ensuring that copy on write in buffered mode works when the CoW
range originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        | 54 +++++++++++++++++++++++++++++++++++
 tests/generic/910     | 66 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/910.out | 12 ++++++++
 3 files changed, 132 insertions(+)
 create mode 100755 tests/generic/910
 create mode 100644 tests/generic/910.out

diff --git a/common/reflink b/common/reflink
index e304ddc8..cca43393 100644
--- a/common/reflink
+++ b/common/reflink
@@ -435,3 +435,57 @@ _sweave_reflink_holes_delalloc() {
 	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
 		_notrun "test requires delayed allocation writes"
 }
+
+# Create a file of interleaved holes, unwritten blocks, regular blocks, and
+# reflinked blocks
+_sweave_reflink_rainbow() {
+	blksz=$1
+	nr=$2
+	sfile=$3
+	dfile=$4
+
+	$XFS_IO_PROG -f -c "truncate $((blksz * nr))" $sfile
+	_pwrite_byte 0x00 0 $((blksz * nr)) $sfile.chk
+	_pwrite_byte 0x61 0 $((blksz * nr)) $dfile
+	seq 0 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile
+		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 0 blocks are reflinked
+	seq 0 5 $((nr - 1)) | while read i; do
+		_reflink_range $sfile $((blksz * i)) $dfile $((blksz * i)) $blksz
+		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 1 blocks are unwritten
+	seq 1 5 $((nr - 1)) | while read i; do
+		$XFS_IO_PROG -f -c "falloc $((blksz * i)) $blksz" $sfile
+		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 2 blocks are holes
+	seq 2 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 3 blocks are regular
+	seq 3 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile
+		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 4 blocks will be delalloc later
+}
+
+# For a file created with _sweave_reflink_rainbow, fill the holes with delalloc
+# extents
+_sweave_reflink_rainbow_delalloc() {
+	blksz=$1
+	nr=$2
+	dfile=$3
+
+	# 4 blocks are delalloc (do later)
+	seq 4 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile
+		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile.chk
+	done
+	# make sure we actually got delalloc block
+	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
+		_notrun "test requires delayed allocation writes"
+}
diff --git a/tests/generic/910 b/tests/generic/910
new file mode 100755
index 00000000..8d731a2d
--- /dev/null
+++ b/tests/generic/910
@@ -0,0 +1,66 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 910
+#
+# Ensuring that copy on write in buffered mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+# now cow
+$XFS_IO_PROG -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/910.out b/tests/generic/910.out
new file mode 100644
index 00000000..12956584
--- /dev/null
+++ b/tests/generic/910.out
@@ -0,0 +1,12 @@
+QA output created by 910
+Format and mount
+Create the original files
+Compare files
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1.chk
+CoW across the transition
+Compare files
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1.chk
-- 
2.34.0

[PATCH v2 12/16] generic: add direct-io CoW test for mixed&source extents

[Shiyang Ruan]

Ensuring that copy on write in buffered mode works when the CoW
range originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/911     | 66 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/911.out | 12 ++++++++
 2 files changed, 78 insertions(+)
 create mode 100755 tests/generic/911
 create mode 100644 tests/generic/911.out

diff --git a/tests/generic/911 b/tests/generic/911
new file mode 100755
index 00000000..57de8754
--- /dev/null
+++ b/tests/generic/911
@@ -0,0 +1,66 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 911
+#
+# Ensuring that copy on write in direct-io mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - directio CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "directio CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+# now cow
+$XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/911.out b/tests/generic/911.out
new file mode 100644
index 00000000..e098fe3c
--- /dev/null
+++ b/tests/generic/911.out
@@ -0,0 +1,12 @@
+QA output created by 911
+Format and mount
+Create the original files
+Compare files
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-911/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-911/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-911/file1.chk
+directio CoW across the transition
+Compare files
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-911/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-911/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-911/file1.chk
-- 
2.34.0

[PATCH v2 13/16] generic: add mmap CoW test for mixed&source extents

[Shiyang Ruan]

Ensuring that copy on write in buffered mode works when the CoW
range originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/912     | 67 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/912.out | 12 ++++++++
 2 files changed, 79 insertions(+)
 create mode 100755 tests/generic/912
 create mode 100644 tests/generic/912.out

diff --git a/tests/generic/912 b/tests/generic/912
new file mode 100755
index 00000000..4f0d94dd
--- /dev/null
+++ b/tests/generic/912
@@ -0,0 +1,67 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 912
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+# now cow
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/912.out b/tests/generic/912.out
new file mode 100644
index 00000000..32f25a1d
--- /dev/null
+++ b/tests/generic/912.out
@@ -0,0 +1,12 @@
+QA output created by 912
+Format and mount
+Create the original files
+Compare files
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1.chk
+mmap CoW across the transition
+Compare files
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1.chk
-- 
2.34.0

[PATCH v2 14/16] generic: add race test between reflink and mmap read

[Shiyang Ruan]

Test for races or FS corruption between reflink and mmap reading the
target file. (MMAP version of generic/164,165)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        | 11 +++++++
 tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/913.out |  5 +++
 3 files changed, 88 insertions(+)
 create mode 100755 tests/generic/913
 create mode 100644 tests/generic/913.out

diff --git a/common/reflink b/common/reflink
index cca43393..7b94864e 100644
--- a/common/reflink
+++ b/common/reflink
@@ -186,6 +186,17 @@ _read_range() {
 	$XFS_IO_PROG $xfs_io_args -f -c "pread -q -v $offset $len" "$file" | cut -d ' ' -f '3-18'
 }
 
+# Prints a range of a file as a hex dump
+_mread_range() {
+	file="$1"
+	offset="$2"
+	len="$3"
+	xfs_io_args="$4"
+
+	$XFS_IO_PROG $xfs_io_args -f -c "mmap -rw 0 $((offset + len))" \
+		-c "mread -v $offset $len" "$file" | cut -d ' ' -f '3-18'
+}
+
 # Compare ranges of two files
 _compare_range() {
 	file1="$1"
diff --git a/tests/generic/913 b/tests/generic/913
new file mode 100755
index 00000000..f709c36c
--- /dev/null
+++ b/tests/generic/913
@@ -0,0 +1,72 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 913
+#
+# Test for races or FS corruption between reflink and mmap reading the
+# target file. (MMAP version of generic/164,165)
+#
+. ./common/preamble
+_begin_fstest auto clone
+
+_register_cleanup "_cleanup" BUS
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_cp_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+finished_file=/tmp/finished
+rm -rf $finished_file
+mkdir $testdir
+
+loops=512
+nr_loops=$((loops - 1))
+blksz=65536
+
+echo "Initialize files"
+echo >> $seqres.full
+_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
+_cp_reflink $testdir/file1 $testdir/file3
+_scratch_cycle_mount
+
+fbytes() {
+	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62)'
+}
+
+reader() {
+	while [ ! -e $finished_file ]; do
+		_mread_range $testdir/file3 0 $((loops * blksz)) | fbytes
+	done
+}
+
+echo "Reflink and mmap reread the files!"
+reader &
+for i in `seq 1 2`; do
+	seq $nr_loops -1 0 | while read i; do
+		_reflink_range  $testdir/file1 $((i * blksz)) \
+				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
+		[ $? -ne 0 ] && break
+	done
+	seq $nr_loops -1 0 | while read i; do
+		_reflink_range  $testdir/file2 $((i * blksz)) \
+				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
+		[ $? -ne 0 ] && break
+	done
+done
+echo "Finished reflinking"
+touch $finished_file
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/913.out b/tests/generic/913.out
new file mode 100644
index 00000000..a34df6ce
--- /dev/null
+++ b/tests/generic/913.out
@@ -0,0 +1,5 @@
+QA output created by 913
+Format and mount
+Initialize files
+Reflink and mmap reread the files!
+Finished reflinking
-- 
2.34.0

[PATCH v2 15/16] generic: add race test that mmap write to source of reflink

[Shiyang Ruan]

Test for races or FS corruption when mmap writing to a file that's also
the source of a reflink operation. (MMAP version of generic/167,166)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/914     | 64 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/914.out |  4 +++
 2 files changed, 68 insertions(+)
 create mode 100755 tests/generic/914
 create mode 100644 tests/generic/914.out

diff --git a/tests/generic/914 b/tests/generic/914
new file mode 100755
index 00000000..c51c44c7
--- /dev/null
+++ b/tests/generic/914
@@ -0,0 +1,64 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 914
+#
+# Test for races or FS corruption when mmap writing to a file that's also
+# the source of a reflink operation. (MMAP version of generic/167,166)
+#
+. ./common/preamble
+_begin_fstest auto clone
+
+_register_cleanup "_cleanup" BUS
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_cp_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+finished_file=/tmp/finished
+rm -rf $finished_file
+mkdir $testdir
+
+loops=1024
+nr_loops=$((loops - 1))
+blksz=65536
+
+echo "Initialize file"
+echo >> $seqres.full
+_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
+_scratch_cycle_mount
+
+# Snapshot creator...
+snappy() {
+	n=0
+	while [ ! -e $finished_file ]; do
+		out="$(_cp_reflink $testdir/file1 $testdir/snap_$n 2>&1)"
+		res=$?
+		echo "$out" | grep -q "No space left" && break
+		test -n "$out" && echo "$out"
+		test $res -ne 0 && break
+		n=$((n + 1))
+	done
+}
+
+echo "Snapshot a file undergoing mmap rewrite"
+snappy &
+seq $nr_loops -1 0 | while read i; do
+	$XFS_IO_PROG -f -c "mmap -rw $((i * blksz)) $blksz" \
+		-c "mwrite -S 0x63 $((i * blksz)) $blksz" $testdir/file1 >> $seqres.full
+done
+touch $finished_file
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/914.out b/tests/generic/914.out
new file mode 100644
index 00000000..1472055f
--- /dev/null
+++ b/tests/generic/914.out
@@ -0,0 +1,4 @@
+QA output created by 914
+Format and mount
+Initialize file
+Snapshot a file undergoing mmap rewrite
-- 
2.34.0

[PATCH v2 16/16] generic: add race test that mmap write to target of reflink

[Shiyang Ruan]

Test for races or FS corruption when mmap writing to a file that's also
the target of a reflink operation. (MMAP version of generic/168,170)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/915     | 64 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/915.out |  4 +++
 2 files changed, 68 insertions(+)
 create mode 100755 tests/generic/915
 create mode 100644 tests/generic/915.out

diff --git a/tests/generic/915 b/tests/generic/915
new file mode 100755
index 00000000..365f304c
--- /dev/null
+++ b/tests/generic/915
@@ -0,0 +1,64 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 915
+#
+# Test for races or FS corruption when mmap writing to a file that's also
+# the target of a reflink operation. (MMAP version of generic/168,170)
+#
+. ./common/preamble
+_begin_fstest auto clone
+
+_register_cleanup "_cleanup" BUS
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+finished_file=/tmp/finished
+rm -rf $finished_file
+mkdir $testdir
+
+loops=1024
+nr_loops=$((loops - 1))
+blksz=65536
+
+echo "Initialize files"
+echo >> $seqres.full
+_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
+_scratch_cycle_mount
+
+# mmap overwriter...
+overwrite() {
+	while [ ! -e $finished_file ]; do
+		seq $nr_loops -1 0 | while read i; do
+			$XFS_IO_PROG -f -c "mmap -rw $((i * blksz)) $blksz" \
+				-c "mwrite -S 0x63 $((i * blksz)) $blksz" $testdir/file2 >> $seqres.full
+		done
+	done
+}
+
+echo "Reflink and mmap write the target"
+overwrite &
+seq 1 10 | while read j; do
+	seq 0 $nr_loops | while read i; do
+		_reflink_range  $testdir/file1 $((i * blksz)) \
+				$testdir/file2 $((i * blksz)) $blksz >> $seqres.full
+		[ $? -ne 0 ] && exit
+	done
+done
+touch $finished_file
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/915.out b/tests/generic/915.out
new file mode 100644
index 00000000..4e77847b
--- /dev/null
+++ b/tests/generic/915.out
@@ -0,0 +1,4 @@
+QA output created by 915
+Format and mount
+Initialize files
+Reflink and mmap write the target
-- 
2.34.0

Re: [PATCH v2 00/16] generic: add some mmap CoW tests

[Shiyang Ruan]

Ping

> This series is to add missing mmap CoW tests in some sets of testcases:
> Set 1. CoW on shared blocks which are interleaving reflinked by two files.
>   - P1:            add mmap test
> Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
>    the destination file.
>   - P2 ~ P6:       add mmap test
> Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
>    the source file.
>   - P11, P12:      add missing buffered-io and direct-io test for mixed blocks
>   - P7 ~ P10, P13: add mmap test
> Set 4. races tests
>   - P14: race between reflink and mmap reading
>   - P15: race between mmap writing to and reflink source file
>   - P16: race between mmap writing to and reflink target file

Re: [PATCH v2 00/16] generic: add some mmap CoW tests

[Eryu Guan]

On Fri, Dec 03, 2021 at 03:30:53PM +0800, Shiyang Ruan wrote:
> Ping

The patchset is in my to-review queue, but just didn't get enough time
to review them.. Will try to find some time to review them next week.

Thanks,
Eryu

> 
> > This series is to add missing mmap CoW tests in some sets of testcases:
> > Set 1. CoW on shared blocks which are interleaving reflinked by two files.
> >   - P1:            add mmap test
> > Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
> >    the destination file.
> >   - P2 ~ P6:       add mmap test
> > Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
> >    the source file.
> >   - P11, P12:      add missing buffered-io and direct-io test for mixed blocks
> >   - P7 ~ P10, P13: add mmap test
> > Set 4. races tests
> >   - P14: race between reflink and mmap reading
> >   - P15: race between mmap writing to and reflink source file
> >   - P16: race between mmap writing to and reflink target file
> 

Re: [PATCH v2 00/16] generic: add some mmap CoW tests

[Shiyang Ruan]

在 2021/12/5 22:57, Eryu Guan 写道:
> On Fri, Dec 03, 2021 at 03:30:53PM +0800, Shiyang Ruan wrote:
>> Ping
> 
> The patchset is in my to-review queue, but just didn't get enough time
> to review them.. Will try to find some time to review them next week.

OK, Thanks~


--
Ruan

> 
> Thanks,
> Eryu
> 
>>
>>> This series is to add missing mmap CoW tests in some sets of testcases:
>>> Set 1. CoW on shared blocks which are interleaving reflinked by two files.
>>>    - P1:            add mmap test
>>> Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
>>>     the destination file.
>>>    - P2 ~ P6:       add mmap test
>>> Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
>>>     the source file.
>>>    - P11, P12:      add missing buffered-io and direct-io test for mixed blocks
>>>    - P7 ~ P10, P13: add mmap test
>>> Set 4. races tests
>>>    - P14: race between reflink and mmap reading
>>>    - P15: race between mmap writing to and reflink source file
>>>    - P16: race between mmap writing to and reflink target file
>>

Re: [PATCH v2 00/16] generic: add some mmap CoW tests

[Darrick J. Wong]

On Mon, Dec 06, 2021 at 09:18:16AM +0800, Shiyang Ruan wrote:
> 
> 在 2021/12/5 22:57, Eryu Guan 写道:
> > On Fri, Dec 03, 2021 at 03:30:53PM +0800, Shiyang Ruan wrote:
> > > Ping
> > 
> > The patchset is in my to-review queue, but just didn't get enough time
> > to review them.. Will try to find some time to review them next week.
> 
> OK, Thanks~

Same here, I didn't have any time to look at this set last week, too
many other regressions going on. :(

--D

> 
> --
> Ruan
> 
> > 
> > Thanks,
> > Eryu
> > 
> > > 
> > > > This series is to add missing mmap CoW tests in some sets of testcases:
> > > > Set 1. CoW on shared blocks which are interleaving reflinked by two files.
> > > >    - P1:            add mmap test
> > > > Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
> > > >     the destination file.
> > > >    - P2 ~ P6:       add mmap test
> > > > Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
> > > >     the source file.
> > > >    - P11, P12:      add missing buffered-io and direct-io test for mixed blocks
> > > >    - P7 ~ P10, P13: add mmap test
> > > > Set 4. races tests
> > > >    - P14: race between reflink and mmap reading
> > > >    - P15: race between mmap writing to and reflink source file
> > > >    - P16: race between mmap writing to and reflink target file
> > > 
> 
> 

Re: [PATCH v2 16/16] generic: add race test that mmap write to target of reflink

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:52PM +0800, Shiyang Ruan wrote:
> Test for races or FS corruption when mmap writing to a file that's also
> the target of a reflink operation. (MMAP version of generic/168,170)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Looks pretty straightforward,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/915     | 64 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/915.out |  4 +++
>  2 files changed, 68 insertions(+)
>  create mode 100755 tests/generic/915
>  create mode 100644 tests/generic/915.out
> 
> diff --git a/tests/generic/915 b/tests/generic/915
> new file mode 100755
> index 00000000..365f304c
> --- /dev/null
> +++ b/tests/generic/915
> @@ -0,0 +1,64 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 915
> +#
> +# Test for races or FS corruption when mmap writing to a file that's also
> +# the target of a reflink operation. (MMAP version of generic/168,170)
> +#
> +. ./common/preamble
> +_begin_fstest auto clone
> +
> +_register_cleanup "_cleanup" BUS
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +finished_file=/tmp/finished
> +rm -rf $finished_file
> +mkdir $testdir
> +
> +loops=1024
> +nr_loops=$((loops - 1))
> +blksz=65536
> +
> +echo "Initialize files"
> +echo >> $seqres.full
> +_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
> +_scratch_cycle_mount
> +
> +# mmap overwriter...
> +overwrite() {
> +	while [ ! -e $finished_file ]; do
> +		seq $nr_loops -1 0 | while read i; do
> +			$XFS_IO_PROG -f -c "mmap -rw $((i * blksz)) $blksz" \
> +				-c "mwrite -S 0x63 $((i * blksz)) $blksz" $testdir/file2 >> $seqres.full
> +		done
> +	done
> +}
> +
> +echo "Reflink and mmap write the target"
> +overwrite &
> +seq 1 10 | while read j; do
> +	seq 0 $nr_loops | while read i; do
> +		_reflink_range  $testdir/file1 $((i * blksz)) \
> +				$testdir/file2 $((i * blksz)) $blksz >> $seqres.full
> +		[ $? -ne 0 ] && exit
> +	done
> +done
> +touch $finished_file
> +wait
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/915.out b/tests/generic/915.out
> new file mode 100644
> index 00000000..4e77847b
> --- /dev/null
> +++ b/tests/generic/915.out
> @@ -0,0 +1,4 @@
> +QA output created by 915
> +Format and mount
> +Initialize files
> +Reflink and mmap write the target
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 15/16] generic: add race test that mmap write to source of reflink

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:51PM +0800, Shiyang Ruan wrote:
> Test for races or FS corruption when mmap writing to a file that's also
> the source of a reflink operation. (MMAP version of generic/167,166)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  tests/generic/914     | 64 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/914.out |  4 +++
>  2 files changed, 68 insertions(+)
>  create mode 100755 tests/generic/914
>  create mode 100644 tests/generic/914.out
> 
> diff --git a/tests/generic/914 b/tests/generic/914
> new file mode 100755
> index 00000000..c51c44c7
> --- /dev/null
> +++ b/tests/generic/914
> @@ -0,0 +1,64 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 914
> +#
> +# Test for races or FS corruption when mmap writing to a file that's also
> +# the source of a reflink operation. (MMAP version of generic/167,166)
> +#
> +. ./common/preamble
> +_begin_fstest auto clone
> +
> +_register_cleanup "_cleanup" BUS
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_cp_reflink
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +finished_file=/tmp/finished
> +rm -rf $finished_file
> +mkdir $testdir
> +
> +loops=1024
> +nr_loops=$((loops - 1))
> +blksz=65536
> +
> +echo "Initialize file"
> +echo >> $seqres.full
> +_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
> +_scratch_cycle_mount
> +
> +# Snapshot creator...
> +snappy() {
> +	n=0
> +	while [ ! -e $finished_file ]; do
> +		out="$(_cp_reflink $testdir/file1 $testdir/snap_$n 2>&1)"

Just out of curiosity, did you notice any of the slow-writes effects
that caused Dave to add throttling to generic/166?  I'm kinda assuming
no since he only changed the directio write tester, but <shrug> now
you're the expert in this area. ;)

In any case, looks correct to me, so
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> +		res=$?
> +		echo "$out" | grep -q "No space left" && break
> +		test -n "$out" && echo "$out"
> +		test $res -ne 0 && break
> +		n=$((n + 1))
> +	done
> +}
> +
> +echo "Snapshot a file undergoing mmap rewrite"
> +snappy &
> +seq $nr_loops -1 0 | while read i; do
> +	$XFS_IO_PROG -f -c "mmap -rw $((i * blksz)) $blksz" \
> +		-c "mwrite -S 0x63 $((i * blksz)) $blksz" $testdir/file1 >> $seqres.full
> +done
> +touch $finished_file
> +wait
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/914.out b/tests/generic/914.out
> new file mode 100644
> index 00000000..1472055f
> --- /dev/null
> +++ b/tests/generic/914.out
> @@ -0,0 +1,4 @@
> +QA output created by 914
> +Format and mount
> +Initialize file
> +Snapshot a file undergoing mmap rewrite
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 14/16] generic: add race test between reflink and mmap read

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:50PM +0800, Shiyang Ruan wrote:
> Test for races or FS corruption between reflink and mmap reading the
> target file. (MMAP version of generic/164,165)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        | 11 +++++++
>  tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/913.out |  5 +++
>  3 files changed, 88 insertions(+)
>  create mode 100755 tests/generic/913
>  create mode 100644 tests/generic/913.out
> 
> diff --git a/common/reflink b/common/reflink
> index cca43393..7b94864e 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -186,6 +186,17 @@ _read_range() {
>  	$XFS_IO_PROG $xfs_io_args -f -c "pread -q -v $offset $len" "$file" | cut -d ' ' -f '3-18'
>  }
>  
> +# Prints a range of a file as a hex dump
> +_mread_range() {
> +	file="$1"
> +	offset="$2"
> +	len="$3"
> +	xfs_io_args="$4"
> +
> +	$XFS_IO_PROG $xfs_io_args -f -c "mmap -rw 0 $((offset + len))" \
> +		-c "mread -v $offset $len" "$file" | cut -d ' ' -f '3-18'
> +}

Hm.  Any reason why you didn't introduce a _mwrite_range helper for
some of these tests?

(This test looks fine otherwise...)

--D

> +
>  # Compare ranges of two files
>  _compare_range() {
>  	file1="$1"
> diff --git a/tests/generic/913 b/tests/generic/913
> new file mode 100755
> index 00000000..f709c36c
> --- /dev/null
> +++ b/tests/generic/913
> @@ -0,0 +1,72 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 913
> +#
> +# Test for races or FS corruption between reflink and mmap reading the
> +# target file. (MMAP version of generic/164,165)
> +#
> +. ./common/preamble
> +_begin_fstest auto clone
> +
> +_register_cleanup "_cleanup" BUS
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_cp_reflink
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +finished_file=/tmp/finished
> +rm -rf $finished_file
> +mkdir $testdir
> +
> +loops=512
> +nr_loops=$((loops - 1))
> +blksz=65536
> +
> +echo "Initialize files"
> +echo >> $seqres.full
> +_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
> +_cp_reflink $testdir/file1 $testdir/file3
> +_scratch_cycle_mount
> +
> +fbytes() {
> +	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62)'
> +}
> +
> +reader() {
> +	while [ ! -e $finished_file ]; do
> +		_mread_range $testdir/file3 0 $((loops * blksz)) | fbytes
> +	done
> +}
> +
> +echo "Reflink and mmap reread the files!"
> +reader &
> +for i in `seq 1 2`; do
> +	seq $nr_loops -1 0 | while read i; do
> +		_reflink_range  $testdir/file1 $((i * blksz)) \
> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
> +		[ $? -ne 0 ] && break
> +	done
> +	seq $nr_loops -1 0 | while read i; do
> +		_reflink_range  $testdir/file2 $((i * blksz)) \
> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
> +		[ $? -ne 0 ] && break
> +	done
> +done
> +echo "Finished reflinking"
> +touch $finished_file
> +wait
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/913.out b/tests/generic/913.out
> new file mode 100644
> index 00000000..a34df6ce
> --- /dev/null
> +++ b/tests/generic/913.out
> @@ -0,0 +1,5 @@
> +QA output created by 913
> +Format and mount
> +Initialize files
> +Reflink and mmap reread the files!
> +Finished reflinking
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 13/16] generic: add mmap CoW test for mixed&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:49PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in buffered mode works when the CoW
> range originally covers multiple extents, mixed with reflinked, unwritten,
> hole, regular and delalloc blocks.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Looks ok,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/912     | 67 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/912.out | 12 ++++++++
>  2 files changed, 79 insertions(+)
>  create mode 100755 tests/generic/912
>  create mode 100644 tests/generic/912.out
> 
> diff --git a/tests/generic/912 b/tests/generic/912
> new file mode 100755
> index 00000000..4f0d94dd
> --- /dev/null
> +++ b/tests/generic/912
> @@ -0,0 +1,67 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 912
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, mixed with reflinked, unwritten,
> +# hole, regular and delalloc blocks.
> +#   - Create a file with the following repeating sequence of blocks:
> +#     1. reflinked
> +#     2. unwritten
> +#     3. hole
> +#     4. regular block
> +#     5. delalloc
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone punch
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fpunch"
> +_require_cp_reflink
> +_require_odirect
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> +# now cow
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> diff --git a/tests/generic/912.out b/tests/generic/912.out
> new file mode 100644
> index 00000000..32f25a1d
> --- /dev/null
> +++ b/tests/generic/912.out
> @@ -0,0 +1,12 @@
> +QA output created by 912
> +Format and mount
> +Create the original files
> +Compare files
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1.chk
> +mmap CoW across the transition
> +Compare files
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 01/16] generic: add mmap CoW test for ranges of two shared files

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:37PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents. (MMAP version of generic/185,183)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Looks good,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/900     | 69 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/900.out | 14 +++++++++
>  2 files changed, 83 insertions(+)
>  create mode 100755 tests/generic/900
>  create mode 100644 tests/generic/900.out
> 
> diff --git a/tests/generic/900 b/tests/generic/900
> new file mode 100755
> index 00000000..de76cd00
> --- /dev/null
> +++ b/tests/generic/900
> @@ -0,0 +1,69 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 900
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents. (MMAP version of generic/185,183)
> +#   - Create two files
> +#   - Reflink the odd blocks of the first file into a third file.
> +#   - Reflink the even blocks of the second file into the third file.
> +#   - mmap CoW across the halfway mark.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_pwrite_byte 0x61 0 $filesize $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x62 0 $filesize $testdir/file2 >> $seqres.full
> +seq 0 2 $((nr-1)) | while read f; do
> +	_reflink_range $testdir/file1 $((blksz * f)) $testdir/file3 $((blksz * f)) $blksz >> $seqres.full
> +	_pwrite_byte 0x61 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
> +done
> +seq 1 2 $((nr-1)) | while read f; do
> +	_reflink_range $testdir/file2 $((blksz * f)) $testdir/file3 $((blksz * f)) $blksz >> $seqres.full
> +	_pwrite_byte 0x62 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
> +done
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/900.out b/tests/generic/900.out
> new file mode 100644
> index 00000000..b2569ab0
> --- /dev/null
> +++ b/tests/generic/900.out
> @@ -0,0 +1,14 @@
> +QA output created by 900
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-900/file1
> +b83f9394092e15bdcda585cd8e776dc6  SCRATCH_MNT/test-900/file2
> +d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-900/file3
> +d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-900/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-900/file1
> +b83f9394092e15bdcda585cd8e776dc6  SCRATCH_MNT/test-900/file2
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-900/file3
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-900/file3.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 02/16] generic: add mmap CoW test for regular&destination extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:38PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, some regular, some not.
> (MMAP version of generic/197,196)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Pretty straightforward port.
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/901     | 59 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/901.out | 12 +++++++++
>  2 files changed, 71 insertions(+)
>  create mode 100755 tests/generic/901
>  create mode 100644 tests/generic/901.out
> 
> diff --git a/tests/generic/901 b/tests/generic/901
> new file mode 100755
> index 00000000..b9f4fbc0
> --- /dev/null
> +++ b/tests/generic/901
> @@ -0,0 +1,59 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 901
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, some regular, some not.
> +# (MMAP version of generic/197,196)
> +#   - Create two files.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/901.out b/tests/generic/901.out
> new file mode 100644
> index 00000000..c3983e87
> --- /dev/null
> +++ b/tests/generic/901.out
> @@ -0,0 +1,12 @@
> +QA output created by 901
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-901/file1
> +d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-901/file3
> +d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-901/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-901/file1
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-901/file3
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-901/file3.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 03/16] generic: add mmap CoW test for unwritten&destination extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:39PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, some unwritten, some not.
> (MMAP version of generic/189,188)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

At first I was a little concerned that _mwrite'ing file3.chk could
result in us failing to notice a bug in the mmap write paths that hits
both reflink and cow writes, but then I realized that this is one of
those tests that encode the md5sum in the golden output, so of course
we'd notice.  I think this looks good, so:

Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/902     | 59 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/902.out | 12 +++++++++
>  2 files changed, 71 insertions(+)
>  create mode 100755 tests/generic/902
>  create mode 100644 tests/generic/902.out
> 
> diff --git a/tests/generic/902 b/tests/generic/902
> new file mode 100755
> index 00000000..457b039a
> --- /dev/null
> +++ b/tests/generic/902
> @@ -0,0 +1,59 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 902
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, some unwritten, some not.
> +# (MMAP version of generic/189,188)
> +#   - Create a file and fallocate a second file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/902.out b/tests/generic/902.out
> new file mode 100644
> index 00000000..bc951255
> --- /dev/null
> +++ b/tests/generic/902.out
> @@ -0,0 +1,12 @@
> +QA output created by 902
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-902/file1
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-902/file3
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-902/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-902/file1
> +09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-902/file3
> +09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-902/file3.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 04/16] generic: add mmap CoW test for holes&destination extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:40PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, some holes, some not.
> (MMAP version of generic/191,190)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/903     | 59 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/903.out | 12 +++++++++
>  2 files changed, 71 insertions(+)
>  create mode 100755 tests/generic/903
>  create mode 100644 tests/generic/903.out
> 
> diff --git a/tests/generic/903 b/tests/generic/903
> new file mode 100755
> index 00000000..d0de3f26
> --- /dev/null
> +++ b/tests/generic/903
> @@ -0,0 +1,59 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 903
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, some holes, some not.
> +# (MMAP version of generic/191,190)
> +#   - Create a file and truncate a second file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/903.out b/tests/generic/903.out
> new file mode 100644
> index 00000000..bf57acc7
> --- /dev/null
> +++ b/tests/generic/903.out
> @@ -0,0 +1,12 @@
> +QA output created by 903
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-903/file1
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-903/file3
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-903/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-903/file1
> +09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-903/file3
> +09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-903/file3.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 05/16] generic: add mmap CoW test for delalloc&destination extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:41PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, some delalloc, some not.
> (MMAP version of generic/195,194)
> 
> Also, add a check to make sure the delalloc block is accutally exist.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        |  3 +++
>  tests/generic/904     | 61 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/904.out | 12 +++++++++
>  3 files changed, 76 insertions(+)
>  create mode 100755 tests/generic/904
>  create mode 100644 tests/generic/904.out
> 
> diff --git a/common/reflink b/common/reflink
> index ee60398e..38210673 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -296,6 +296,9 @@ _weave_reflink_holes_delalloc() {
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
>  	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"

Heh, I forgot this check way back when.

However, I think this is going to require a:

_require_command "$FILEFRAG_PROG" filefrag

for all the tests that use _weave_reflink_holes_delalloc.  AFAICT that
is:

generic/194 generic/195 xfs/223 xfs/224

The actual generic/904 test looks good.

--D


>  }
>  
>  # Create a file of interleaved regular blocks and reflinked blocks
> diff --git a/tests/generic/904 b/tests/generic/904
> new file mode 100755
> index 00000000..69703d70
> --- /dev/null
> +++ b/tests/generic/904
> @@ -0,0 +1,61 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 904
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, some delalloc, some not.
> +# (MMAP version of generic/195,194)
> +#   - Create a file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - Buffered write the even blocks of the second file.
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_weave_reflink_holes_delalloc $blksz $nr $testdir/file3 >> $seqres.full
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/904.out b/tests/generic/904.out
> new file mode 100644
> index 00000000..cc993394
> --- /dev/null
> +++ b/tests/generic/904.out
> @@ -0,0 +1,12 @@
> +QA output created by 904
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 06/16] generic: add mmap CoW test for mixed&destination extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:42PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, mixed with reflinked, unwritten, hole, regular and
> delalloc blocks.
> (MMAP version of generic/200,199)
> 
> Also, add a check to make sure the delalloc block is accutally exist.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        |  3 ++
>  tests/generic/905     | 68 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/905.out | 12 ++++++++
>  3 files changed, 83 insertions(+)
>  create mode 100755 tests/generic/905
>  create mode 100644 tests/generic/905.out
> 
> diff --git a/common/reflink b/common/reflink
> index 38210673..84173d59 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -362,6 +362,9 @@ _weave_reflink_rainbow_delalloc() {
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
>  	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"

Same comment as the previous patch.  generic/905 itself looks like a
straightforward translation of the existing generic/200.

--D

>  }
>  
>  # Make the source file have interleaved regular blocks and reflinked blocks
> diff --git a/tests/generic/905 b/tests/generic/905
> new file mode 100755
> index 00000000..56cdcd27
> --- /dev/null
> +++ b/tests/generic/905
> @@ -0,0 +1,68 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 905
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, mixed with reflinked, unwritten,
> +# hole, regular and delalloc blocks.
> +# (MMAP version of generic/200,199)
> +#   - Create a file with the following repeating sequence of blocks:
> +#     1. reflinked
> +#     2. unwritten
> +#     3. hole
> +#     4. regular block
> +#     5. delalloc
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone punch
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fpunch"
> +_require_cp_reflink
> +_require_odirect
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
> +# now cow
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> diff --git a/tests/generic/905.out b/tests/generic/905.out
> new file mode 100644
> index 00000000..6fb0be5f
> --- /dev/null
> +++ b/tests/generic/905.out
> @@ -0,0 +1,12 @@
> +QA output created by 905
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 07/16] generic: add mmap CoW test for regular&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:43PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode to the source file when the CoW range
> covers regular unshared and regular shared blocks.
> (MMAP version of generic/284,287)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Looks good,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/906     | 60 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/906.out | 12 +++++++++
>  2 files changed, 72 insertions(+)
>  create mode 100755 tests/generic/906
>  create mode 100644 tests/generic/906.out
> 
> diff --git a/tests/generic/906 b/tests/generic/906
> new file mode 100755
> index 00000000..953348e8
> --- /dev/null
> +++ b/tests/generic/906
> @@ -0,0 +1,60 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 906
> +#
> +# Ensuring that copy on write in mmap mode to the source file when the
> +# CoW range covers regular unshared and regular shared blocks.
> +# (MMAP version of generic/284,287)
> +#   - Create two files.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - mmap CoW the first file across the halfway mark, starting with the
> +#     regular extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/906.out b/tests/generic/906.out
> new file mode 100644
> index 00000000..4949769b
> --- /dev/null
> +++ b/tests/generic/906.out
> @@ -0,0 +1,12 @@
> +QA output created by 906
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-906/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-906/file3
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-906/file1.chk
> +mmap CoW across the transition
> +Compare files
> +2dc2b131303b2f70ddb480778caac8b7  SCRATCH_MNT/test-906/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-906/file3
> +2dc2b131303b2f70ddb480778caac8b7  SCRATCH_MNT/test-906/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 08/16] generic: add mmap CoW test for unwritten&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:44PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode to the source file when the CoW range
> covers unwritten and regular shared blocks.
> (MMAP version of generic/289,290)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/907     | 62 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/907.out | 12 +++++++++
>  2 files changed, 74 insertions(+)
>  create mode 100755 tests/generic/907
>  create mode 100644 tests/generic/907.out
> 
> diff --git a/tests/generic/907 b/tests/generic/907
> new file mode 100755
> index 00000000..61c9293a
> --- /dev/null
> +++ b/tests/generic/907
> @@ -0,0 +1,62 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 907
> +#
> +# Ensuring that copy on write in mmap mode to the source file when the
> +# CoW range covers unwritten and regular shared blocks.
> +# (MMAP version of generic/289,290)
> +#   - Create two files.
> +#   - fallocate the first file.
> +#   - Write the odd blocks of the first file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - mmap CoW the first file across the halfway mark, starting with the
> +#     regular extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/907.out b/tests/generic/907.out
> new file mode 100644
> index 00000000..c18b849c
> --- /dev/null
> +++ b/tests/generic/907.out
> @@ -0,0 +1,12 @@
> +QA output created by 907
> +Format and mount
> +Create the original files
> +Compare files
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-907/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-907/file3
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-907/file1.chk
> +mmap CoW across the transition
> +Compare files
> +e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-907/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-907/file3
> +e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-907/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 09/16] generic: add mmap CoW test for holes&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:45PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode to the source file when the CoW range
> covers holes and regular shared blocks.
> (MMAP version of generic/291,292)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>

Looks good, and thanks for mentioning which test each of these new ones
came from!

Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> ---
>  tests/generic/908     | 62 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/908.out | 12 +++++++++
>  2 files changed, 74 insertions(+)
>  create mode 100755 tests/generic/908
>  create mode 100644 tests/generic/908.out
> 
> diff --git a/tests/generic/908 b/tests/generic/908
> new file mode 100755
> index 00000000..ad8de699
> --- /dev/null
> +++ b/tests/generic/908
> @@ -0,0 +1,62 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 908
> +#
> +# Ensuring that copy on write in mmap mode to the source file when the
> +# CoW range covers holes and regular shared blocks.
> +# (MMAP version of generic/291,292)
> +#   - Create two files.
> +#   - Truncate the first file.
> +#   - Write the odd blocks of the first file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - mmap CoW the first file across the halfway mark, starting with the
> +#     regular extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/908.out b/tests/generic/908.out
> new file mode 100644
> index 00000000..c8c670ac
> --- /dev/null
> +++ b/tests/generic/908.out
> @@ -0,0 +1,12 @@
> +QA output created by 908
> +Format and mount
> +Create the original files
> +Compare files
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-908/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-908/file3
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-908/file1.chk
> +mmap CoW across the transition
> +Compare files
> +e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-908/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-908/file3
> +e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-908/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 10/16] generic: add mmap CoW test for delalloc&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:46PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode to the source file when the CoW range
> covers delalloc blocks and regular shared blocks.
> (MMAP version of generic/293,295)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        |  3 ++
>  tests/generic/909     | 64 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/909.out | 12 ++++++++
>  3 files changed, 79 insertions(+)
>  create mode 100755 tests/generic/909
>  create mode 100644 tests/generic/909.out
> 
> diff --git a/common/reflink b/common/reflink
> index 84173d59..e304ddc8 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -431,4 +431,7 @@ _sweave_reflink_holes_delalloc() {
>  		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile
>  		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile.chk
>  	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"

Same comment about _require_command $FILEFRAG_PROG "filefrag" as before.

generic/909 looks like a spot-on translation from pwrite to mmap write.

--D

>  }
> diff --git a/tests/generic/909 b/tests/generic/909
> new file mode 100755
> index 00000000..ef1233ea
> --- /dev/null
> +++ b/tests/generic/909
> @@ -0,0 +1,64 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 909
> +#
> +# Ensuring that copy on write in mmap mode to the source file when the
> +# CoW range covers delalloc blocks and regular shared blocks.
> +# (MMAP version of generic/293,295)
> +#   - Create two files.
> +#   - Truncate the first file.
> +#   - Write the odd blocks of the first file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - Write the even blocks of the first file.
> +#   - mmap CoW the first file across the halfway mark, starting with the
> +#     regular extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_sweave_reflink_holes_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/909.out b/tests/generic/909.out
> new file mode 100644
> index 00000000..4f9a5d33
> --- /dev/null
> +++ b/tests/generic/909.out
> @@ -0,0 +1,12 @@
> +QA output created by 909
> +Format and mount
> +Create the original files
> +Compare files
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1.chk
> +mmap CoW across the transition
> +Compare files
> +35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
> +35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 11/16] generic: add buffered-io CoW test for mixed&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:47PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in buffered mode works when the CoW
> range originally covers multiple extents, mixed with reflinked, unwritten,
> hole, regular and delalloc blocks.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        | 54 +++++++++++++++++++++++++++++++++++
>  tests/generic/910     | 66 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/910.out | 12 ++++++++
>  3 files changed, 132 insertions(+)
>  create mode 100755 tests/generic/910
>  create mode 100644 tests/generic/910.out
> 
> diff --git a/common/reflink b/common/reflink
> index e304ddc8..cca43393 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -435,3 +435,57 @@ _sweave_reflink_holes_delalloc() {
>  	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
>  		_notrun "test requires delayed allocation writes"
>  }
> +
> +# Create a file of interleaved holes, unwritten blocks, regular blocks, and
> +# reflinked blocks
> +_sweave_reflink_rainbow() {
> +	blksz=$1
> +	nr=$2
> +	sfile=$3
> +	dfile=$4
> +
> +	$XFS_IO_PROG -f -c "truncate $((blksz * nr))" $sfile
> +	_pwrite_byte 0x00 0 $((blksz * nr)) $sfile.chk
> +	_pwrite_byte 0x61 0 $((blksz * nr)) $dfile
> +	seq 0 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile
> +		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 0 blocks are reflinked
> +	seq 0 5 $((nr - 1)) | while read i; do
> +		_reflink_range $sfile $((blksz * i)) $dfile $((blksz * i)) $blksz
> +		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 1 blocks are unwritten
> +	seq 1 5 $((nr - 1)) | while read i; do
> +		$XFS_IO_PROG -f -c "falloc $((blksz * i)) $blksz" $sfile
> +		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 2 blocks are holes
> +	seq 2 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 3 blocks are regular
> +	seq 3 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile
> +		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 4 blocks will be delalloc later
> +}
> +
> +# For a file created with _sweave_reflink_rainbow, fill the holes with delalloc
> +# extents
> +_sweave_reflink_rainbow_delalloc() {
> +	blksz=$1
> +	nr=$2
> +	dfile=$3
> +
> +	# 4 blocks are delalloc (do later)
> +	seq 4 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile
> +		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"
> +}
> diff --git a/tests/generic/910 b/tests/generic/910
> new file mode 100755
> index 00000000..8d731a2d
> --- /dev/null
> +++ b/tests/generic/910
> @@ -0,0 +1,66 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 910
> +#
> +# Ensuring that copy on write in buffered mode works when the CoW
> +# range originally covers multiple extents, mixed with reflinked, unwritten,
> +# hole, regular and delalloc blocks.
> +#   - Create a file with the following repeating sequence of blocks:
> +#     1. reflinked
> +#     2. unwritten
> +#     3. hole
> +#     4. regular block
> +#     5. delalloc
> +#   - CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone punch
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fpunch"
> +_require_cp_reflink
> +_require_odirect

I don't see any directio going on in this test?

This test is also missing a _require_command $FILEFRAG_PROG filefrag.

The rest looks good.  Thanks for adding a new combination!

--D

> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> +# now cow
> +$XFS_IO_PROG -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> diff --git a/tests/generic/910.out b/tests/generic/910.out
> new file mode 100644
> index 00000000..12956584
> --- /dev/null
> +++ b/tests/generic/910.out
> @@ -0,0 +1,12 @@
> +QA output created by 910
> +Format and mount
> +Create the original files
> +Compare files
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1.chk
> +CoW across the transition
> +Compare files
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 12/16] generic: add direct-io CoW test for mixed&source extents

[Darrick J. Wong]

On Thu, Nov 25, 2021 at 03:47:48PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in buffered mode works when the CoW
> range originally covers multiple extents, mixed with reflinked, unwritten,
> hole, regular and delalloc blocks.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  tests/generic/911     | 66 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/911.out | 12 ++++++++
>  2 files changed, 78 insertions(+)
>  create mode 100755 tests/generic/911
>  create mode 100644 tests/generic/911.out
> 
> diff --git a/tests/generic/911 b/tests/generic/911
> new file mode 100755
> index 00000000..57de8754
> --- /dev/null
> +++ b/tests/generic/911
> @@ -0,0 +1,66 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 911
> +#
> +# Ensuring that copy on write in direct-io mode works when the CoW
> +# range originally covers multiple extents, mixed with reflinked, unwritten,
> +# hole, regular and delalloc blocks.
> +#   - Create a file with the following repeating sequence of blocks:
> +#     1. reflinked
> +#     2. unwritten
> +#     3. hole
> +#     4. regular block
> +#     5. delalloc
> +#   - directio CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone punch
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fpunch"
> +_require_cp_reflink
> +_require_odirect

_requires_command $FILEFRAG_PROG filefrag

Otherwise, this looks like it's ready to go.

--D

> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "directio CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> +# now cow
> +$XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> diff --git a/tests/generic/911.out b/tests/generic/911.out
> new file mode 100644
> index 00000000..e098fe3c
> --- /dev/null
> +++ b/tests/generic/911.out
> @@ -0,0 +1,12 @@
> +QA output created by 911
> +Format and mount
> +Create the original files
> +Compare files
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-911/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-911/file3
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-911/file1.chk
> +directio CoW across the transition
> +Compare files
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-911/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-911/file3
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-911/file1.chk
> -- 
> 2.34.0
> 
> 
> 

Re: [PATCH v2 13/16] generic: add mmap CoW test for mixed&source extents

[Darrick J. Wong]

On Mon, Dec 06, 2021 at 04:40:40PM -0800, Darrick J. Wong wrote:
> On Thu, Nov 25, 2021 at 03:47:49PM +0800, Shiyang Ruan wrote:
> > Ensuring that copy on write in buffered mode works when the CoW
> > range originally covers multiple extents, mixed with reflinked, unwritten,
> > hole, regular and delalloc blocks.
> > 
> > Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> 
> Looks ok,

...with a _require_command $FILEFRAG_PROG filefrag added.

(Serves me right for reviewing in backwards order.)

--D

> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> 
> --D
> 
> > ---
> >  tests/generic/912     | 67 +++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/912.out | 12 ++++++++
> >  2 files changed, 79 insertions(+)
> >  create mode 100755 tests/generic/912
> >  create mode 100644 tests/generic/912.out
> > 
> > diff --git a/tests/generic/912 b/tests/generic/912
> > new file mode 100755
> > index 00000000..4f0d94dd
> > --- /dev/null
> > +++ b/tests/generic/912
> > @@ -0,0 +1,67 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 912
> > +#
> > +# Ensuring that copy on write in mmap mode works when the CoW
> > +# range originally covers multiple extents, mixed with reflinked, unwritten,
> > +# hole, regular and delalloc blocks.
> > +#   - Create a file with the following repeating sequence of blocks:
> > +#     1. reflinked
> > +#     2. unwritten
> > +#     3. hole
> > +#     4. regular block
> > +#     5. delalloc
> > +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> > +#   - Check that the files are now different where we say they're different.
> > +#
> > +. ./common/preamble
> > +_begin_fstest auto quick clone punch
> > +
> > +# Import common functions.
> > +. ./common/filter
> > +. ./common/reflink
> > +
> > +# real QA test starts here
> > +_require_scratch_reflink
> > +_require_xfs_io_command "falloc"
> > +_require_xfs_io_command "fpunch"
> > +_require_cp_reflink
> > +_require_odirect
> > +
> > +echo "Format and mount"
> > +_scratch_mkfs > $seqres.full 2>&1
> > +_scratch_mount >> $seqres.full 2>&1
> > +
> > +testdir=$SCRATCH_MNT/test-$seq
> > +mkdir $testdir
> > +
> > +echo "Create the original files"
> > +blksz=65536
> > +nr=64
> > +filesize=$((blksz * nr))
> > +_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> > +_scratch_cycle_mount
> > +
> > +echo "Compare files"
> > +md5sum $testdir/file1 | _filter_scratch
> > +md5sum $testdir/file3 | _filter_scratch
> > +md5sum $testdir/file1.chk | _filter_scratch
> > +
> > +echo "mmap CoW across the transition"
> > +cowoff=$((filesize / 4))
> > +cowsz=$((filesize / 2))
> > +_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> > +# now cow
> > +mmapsz=$((cowoff + cowsz))
> > +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> > +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> > +_scratch_cycle_mount
> > +
> > +echo "Compare files"
> > +md5sum $testdir/file1 | _filter_scratch
> > +md5sum $testdir/file3 | _filter_scratch
> > +md5sum $testdir/file1.chk | _filter_scratch
> > +
> > +# success, all done
> > +status=0
> > diff --git a/tests/generic/912.out b/tests/generic/912.out
> > new file mode 100644
> > index 00000000..32f25a1d
> > --- /dev/null
> > +++ b/tests/generic/912.out
> > @@ -0,0 +1,12 @@
> > +QA output created by 912
> > +Format and mount
> > +Create the original files
> > +Compare files
> > +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1
> > +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
> > +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1.chk
> > +mmap CoW across the transition
> > +Compare files
> > +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1
> > +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
> > +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1.chk
> > -- 
> > 2.34.0
> > 
> > 
> > 

Re: [PATCH v2 05/16] generic: add mmap CoW test for delalloc&destination extents

[Eryu Guan]

On Thu, Nov 25, 2021 at 03:47:41PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, some delalloc, some not.
> (MMAP version of generic/195,194)
> 
> Also, add a check to make sure the delalloc block is accutally exist.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        |  3 +++
>  tests/generic/904     | 61 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/904.out | 12 +++++++++
>  3 files changed, 76 insertions(+)
>  create mode 100755 tests/generic/904
>  create mode 100644 tests/generic/904.out
> 
> diff --git a/common/reflink b/common/reflink
> index ee60398e..38210673 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -296,6 +296,9 @@ _weave_reflink_holes_delalloc() {
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
>  	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"

I think we need a new _require rule to check delalloc status explicitly,
not do the check and call _notrun in a helper function that does the
real work not check.

Perhaps we could turn the delalloc check in xfs/172 into a helper like
_require_scratch_delalloc, and call it in each test that needs dealloc
writes.

Thanks,
Eryu

>  }
>  
>  # Create a file of interleaved regular blocks and reflinked blocks
> diff --git a/tests/generic/904 b/tests/generic/904
> new file mode 100755
> index 00000000..69703d70
> --- /dev/null
> +++ b/tests/generic/904
> @@ -0,0 +1,61 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 904
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, some delalloc, some not.
> +# (MMAP version of generic/195,194)
> +#   - Create a file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - Buffered write the even blocks of the second file.
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_weave_reflink_holes_delalloc $blksz $nr $testdir/file3 >> $seqres.full
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/904.out b/tests/generic/904.out
> new file mode 100644
> index 00000000..cc993394
> --- /dev/null
> +++ b/tests/generic/904.out
> @@ -0,0 +1,12 @@
> +QA output created by 904
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3
> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3
> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3.chk
> -- 
> 2.34.0
> 
> 

Re: [PATCH v2 06/16] generic: add mmap CoW test for mixed&destination extents

[Eryu Guan]

On Thu, Nov 25, 2021 at 03:47:42PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode works when the CoW range originally
> covers multiple extents, mixed with reflinked, unwritten, hole, regular and
> delalloc blocks.
> (MMAP version of generic/200,199)
> 
> Also, add a check to make sure the delalloc block is accutally exist.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        |  3 ++
>  tests/generic/905     | 68 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/905.out | 12 ++++++++
>  3 files changed, 83 insertions(+)
>  create mode 100755 tests/generic/905
>  create mode 100644 tests/generic/905.out
> 
> diff --git a/common/reflink b/common/reflink
> index 38210673..84173d59 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -362,6 +362,9 @@ _weave_reflink_rainbow_delalloc() {
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
>  		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
>  	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"

Same comments as to _weave_reflink_holes_dealloc().

Thanks,
Eryu

>  }
>  
>  # Make the source file have interleaved regular blocks and reflinked blocks
> diff --git a/tests/generic/905 b/tests/generic/905
> new file mode 100755
> index 00000000..56cdcd27
> --- /dev/null
> +++ b/tests/generic/905
> @@ -0,0 +1,68 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 905
> +#
> +# Ensuring that copy on write in mmap mode works when the CoW
> +# range originally covers multiple extents, mixed with reflinked, unwritten,
> +# hole, regular and delalloc blocks.
> +# (MMAP version of generic/200,199)
> +#   - Create a file with the following repeating sequence of blocks:
> +#     1. reflinked
> +#     2. unwritten
> +#     3. hole
> +#     4. regular block
> +#     5. delalloc
> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone punch
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fpunch"
> +_require_cp_reflink
> +_require_odirect
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
> +# now cow
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file3.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> diff --git a/tests/generic/905.out b/tests/generic/905.out
> new file mode 100644
> index 00000000..6fb0be5f
> --- /dev/null
> +++ b/tests/generic/905.out
> @@ -0,0 +1,12 @@
> +QA output created by 905
> +Format and mount
> +Create the original files
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3.chk
> +mmap CoW across the transition
> +Compare files
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3.chk
> -- 
> 2.34.0
> 
> 

Re: [PATCH v2 10/16] generic: add mmap CoW test for delalloc&source extents

[Eryu Guan]

On Thu, Nov 25, 2021 at 03:47:46PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in mmap mode to the source file when the CoW range
> covers delalloc blocks and regular shared blocks.
> (MMAP version of generic/293,295)
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        |  3 ++
>  tests/generic/909     | 64 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/909.out | 12 ++++++++
>  3 files changed, 79 insertions(+)
>  create mode 100755 tests/generic/909
>  create mode 100644 tests/generic/909.out
> 
> diff --git a/common/reflink b/common/reflink
> index 84173d59..e304ddc8 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -431,4 +431,7 @@ _sweave_reflink_holes_delalloc() {
>  		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile
>  		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile.chk
>  	done
> +	# make sure we actually got delalloc block
> +	$FILEFRAG_PROG -v $sfile 2>&1 | grep -q delalloc || \
> +		_notrun "test requires delayed allocation writes"

Same here, need a new helper to check delalloc status. And in test
generic/910 as well.

Thanks,
Eryu

>  }
> diff --git a/tests/generic/909 b/tests/generic/909
> new file mode 100755
> index 00000000..ef1233ea
> --- /dev/null
> +++ b/tests/generic/909
> @@ -0,0 +1,64 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 909
> +#
> +# Ensuring that copy on write in mmap mode to the source file when the
> +# CoW range covers delalloc blocks and regular shared blocks.
> +# (MMAP version of generic/293,295)
> +#   - Create two files.
> +#   - Truncate the first file.
> +#   - Write the odd blocks of the first file.
> +#   - Reflink the odd blocks of the first file into the second file.
> +#   - Write the even blocks of the first file.
> +#   - mmap CoW the first file across the halfway mark, starting with the
> +#     regular extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_xfs_io_command "falloc"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "mmap CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_sweave_reflink_holes_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> +mmapsz=$((cowoff + cowsz))
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/909.out b/tests/generic/909.out
> new file mode 100644
> index 00000000..4f9a5d33
> --- /dev/null
> +++ b/tests/generic/909.out
> @@ -0,0 +1,12 @@
> +QA output created by 909
> +Format and mount
> +Create the original files
> +Compare files
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
> +b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1.chk
> +mmap CoW across the transition
> +Compare files
> +35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1
> +ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
> +35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1.chk
> -- 
> 2.34.0
> 
> 

Re: [PATCH v2 14/16] generic: add race test between reflink and mmap read

[Shiyang Ruan]

在 2021/12/7 8:32, Darrick J. Wong 写道:
> On Thu, Nov 25, 2021 at 03:47:50PM +0800, Shiyang Ruan wrote:
>> Test for races or FS corruption between reflink and mmap reading the
>> target file. (MMAP version of generic/164,165)
>>
>> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
>> ---
>>   common/reflink        | 11 +++++++
>>   tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
>>   tests/generic/913.out |  5 +++
>>   3 files changed, 88 insertions(+)
>>   create mode 100755 tests/generic/913
>>   create mode 100644 tests/generic/913.out
>>
>> diff --git a/common/reflink b/common/reflink
>> index cca43393..7b94864e 100644
>> --- a/common/reflink
>> +++ b/common/reflink
>> @@ -186,6 +186,17 @@ _read_range() {
>>   	$XFS_IO_PROG $xfs_io_args -f -c "pread -q -v $offset $len" "$file" | cut -d ' ' -f '3-18'
>>   }
>>   
>> +# Prints a range of a file as a hex dump
>> +_mread_range() {
>> +	file="$1"
>> +	offset="$2"
>> +	len="$3"
>> +	xfs_io_args="$4"
>> +
>> +	$XFS_IO_PROG $xfs_io_args -f -c "mmap -rw 0 $((offset + len))" \
>> +		-c "mread -v $offset $len" "$file" | cut -d ' ' -f '3-18'
>> +}
> 
> Hm.  Any reason why you didn't introduce a _mwrite_range helper for
> some of these tests?

I think _mwrite_byte() is enough, since "mwrite" doesn't have a '-v' 
option to show what has been written.

--
Thanks,
Ruan.

> 
> (This test looks fine otherwise...)
> 
> --D
> 
>> +
>>   # Compare ranges of two files
>>   _compare_range() {
>>   	file1="$1"
>> diff --git a/tests/generic/913 b/tests/generic/913
>> new file mode 100755
>> index 00000000..f709c36c
>> --- /dev/null
>> +++ b/tests/generic/913
>> @@ -0,0 +1,72 @@
>> +#! /bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +#
>> +# FS QA Test No. 913
>> +#
>> +# Test for races or FS corruption between reflink and mmap reading the
>> +# target file. (MMAP version of generic/164,165)
>> +#
>> +. ./common/preamble
>> +_begin_fstest auto clone
>> +
>> +_register_cleanup "_cleanup" BUS
>> +
>> +# Import common functions.
>> +. ./common/filter
>> +. ./common/reflink
>> +
>> +# real QA test starts here
>> +_require_scratch_reflink
>> +_require_cp_reflink
>> +
>> +echo "Format and mount"
>> +_scratch_mkfs > $seqres.full 2>&1
>> +_scratch_mount >> $seqres.full 2>&1
>> +
>> +testdir=$SCRATCH_MNT/test-$seq
>> +finished_file=/tmp/finished
>> +rm -rf $finished_file
>> +mkdir $testdir
>> +
>> +loops=512
>> +nr_loops=$((loops - 1))
>> +blksz=65536
>> +
>> +echo "Initialize files"
>> +echo >> $seqres.full
>> +_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
>> +_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
>> +_cp_reflink $testdir/file1 $testdir/file3
>> +_scratch_cycle_mount
>> +
>> +fbytes() {
>> +	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62)'
>> +}
>> +
>> +reader() {
>> +	while [ ! -e $finished_file ]; do
>> +		_mread_range $testdir/file3 0 $((loops * blksz)) | fbytes
>> +	done
>> +}
>> +
>> +echo "Reflink and mmap reread the files!"
>> +reader &
>> +for i in `seq 1 2`; do
>> +	seq $nr_loops -1 0 | while read i; do
>> +		_reflink_range  $testdir/file1 $((i * blksz)) \
>> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
>> +		[ $? -ne 0 ] && break
>> +	done
>> +	seq $nr_loops -1 0 | while read i; do
>> +		_reflink_range  $testdir/file2 $((i * blksz)) \
>> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
>> +		[ $? -ne 0 ] && break
>> +	done
>> +done
>> +echo "Finished reflinking"
>> +touch $finished_file
>> +wait
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/913.out b/tests/generic/913.out
>> new file mode 100644
>> index 00000000..a34df6ce
>> --- /dev/null
>> +++ b/tests/generic/913.out
>> @@ -0,0 +1,5 @@
>> +QA output created by 913
>> +Format and mount
>> +Initialize files
>> +Reflink and mmap reread the files!
>> +Finished reflinking
>> -- 
>> 2.34.0
>>
>>
>>

Re: [PATCH v2 05/16] generic: add mmap CoW test for delalloc&destination extents

[Shiyang Ruan]

在 2021/12/12 21:41, Eryu Guan 写道:
> On Thu, Nov 25, 2021 at 03:47:41PM +0800, Shiyang Ruan wrote:
>> Ensuring that copy on write in mmap mode works when the CoW range originally
>> covers multiple extents, some delalloc, some not.
>> (MMAP version of generic/195,194)
>>
>> Also, add a check to make sure the delalloc block is accutally exist.
>>
>> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
>> ---
>>   common/reflink        |  3 +++
>>   tests/generic/904     | 61 +++++++++++++++++++++++++++++++++++++++++++
>>   tests/generic/904.out | 12 +++++++++
>>   3 files changed, 76 insertions(+)
>>   create mode 100755 tests/generic/904
>>   create mode 100644 tests/generic/904.out
>>
>> diff --git a/common/reflink b/common/reflink
>> index ee60398e..38210673 100644
>> --- a/common/reflink
>> +++ b/common/reflink
>> @@ -296,6 +296,9 @@ _weave_reflink_holes_delalloc() {
>>   		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile
>>   		_pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk
>>   	done
>> +	# make sure we actually got delalloc block
>> +	$FILEFRAG_PROG -v $dfile 2>&1 | grep -q delalloc || \
>> +		_notrun "test requires delayed allocation writes"
> 
> I think we need a new _require rule to check delalloc status explicitly,
> not do the check and call _notrun in a helper function that does the
> real work not check.
> 
> Perhaps we could turn the delalloc check in xfs/172 into a helper like
> _require_scratch_delalloc, and call it in each test that needs dealloc
> writes.

OK, thanks for the advice. I think this is better.


--
Thanks,
Ruan.

> 
> Thanks,
> Eryu
> 
>>   }
>>   
>>   # Create a file of interleaved regular blocks and reflinked blocks
>> diff --git a/tests/generic/904 b/tests/generic/904
>> new file mode 100755
>> index 00000000..69703d70
>> --- /dev/null
>> +++ b/tests/generic/904
>> @@ -0,0 +1,61 @@
>> +#! /bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +#
>> +# FS QA Test No. 904
>> +#
>> +# Ensuring that copy on write in mmap mode works when the CoW
>> +# range originally covers multiple extents, some delalloc, some not.
>> +# (MMAP version of generic/195,194)
>> +#   - Create a file.
>> +#   - Reflink the odd blocks of the first file into the second file.
>> +#   - Buffered write the even blocks of the second file.
>> +#   - mmap CoW across the halfway mark, starting with the unwritten extent.
>> +#   - Check that the files are now different where we say they're different.
>> +#
>> +. ./common/preamble
>> +_begin_fstest auto quick clone
>> +
>> +# Import common functions.
>> +. ./common/filter
>> +. ./common/reflink
>> +
>> +# real QA test starts here
>> +_require_scratch_reflink
>> +_require_xfs_io_command "falloc"
>> +
>> +echo "Format and mount"
>> +_scratch_mkfs > $seqres.full 2>&1
>> +_scratch_mount >> $seqres.full 2>&1
>> +
>> +testdir=$SCRATCH_MNT/test-$seq
>> +mkdir $testdir
>> +
>> +echo "Create the original files"
>> +blksz=65536
>> +nr=64
>> +filesize=$((blksz * nr))
>> +_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
>> +_scratch_cycle_mount
>> +
>> +echo "Compare files"
>> +md5sum $testdir/file1 | _filter_scratch
>> +md5sum $testdir/file3 | _filter_scratch
>> +md5sum $testdir/file3.chk | _filter_scratch
>> +
>> +echo "mmap CoW across the transition"
>> +cowoff=$((filesize / 4))
>> +cowsz=$((filesize / 2))
>> +_weave_reflink_holes_delalloc $blksz $nr $testdir/file3 >> $seqres.full
>> +mmapsz=$((cowoff + cowsz))
>> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
>> +_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
>> +_scratch_cycle_mount
>> +
>> +echo "Compare files"
>> +md5sum $testdir/file1 | _filter_scratch
>> +md5sum $testdir/file3 | _filter_scratch
>> +md5sum $testdir/file3.chk | _filter_scratch
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/904.out b/tests/generic/904.out
>> new file mode 100644
>> index 00000000..cc993394
>> --- /dev/null
>> +++ b/tests/generic/904.out
>> @@ -0,0 +1,12 @@
>> +QA output created by 904
>> +Format and mount
>> +Create the original files
>> +Compare files
>> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
>> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3
>> +fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3.chk
>> +mmap CoW across the transition
>> +Compare files
>> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
>> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3
>> +55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3.chk
>> -- 
>> 2.34.0
>>
>>