[fstests PATCH v4 0/4] add test for DAX MAP_SYNC support

[fstests PATCH v4 0/4] add test for DAX MAP_SYNC support

[Ross Zwisler]

The purpose of this series is to exercise the new MAP_SYNC mmap()
functionality [1].  It adds a test which uses dm-log-writes to try and
replay filesystem metadata operations for a file that is being written via
mmap().

If MAP_SYNC is active the dm-log-writes replay will recreate the file's
block allocations and you'll end up with a test file which is a known
size.

If MAP_SYNC is not active the metadata writes will most likely be lost and
the replay will either fail to create the test file at all or it will may
be smaller.  In all of my testing the file simply doesn't exist on the
replay if MAP_SYNC is ommited.

This test relies on a kernel with both the MAP_SYNC mmap() functionality
and the DAX enabling in dm-log-writes.  These will both appear in kernel
v4.15-rc1.  For ease of testing I've posted a kernel that is v4.14 plus
just those two patch series [2].

This test also relies on xfsprogs having support for MAP_SYNC and for
dm-log-writes.  I've posed a tree adding that support [3].  This xfsprogs
series is still under review so if the xfs_io interfaces change this
test will of course need to be updated as well.

Lastly, I've also posted a working version of this series [4].

Changes since v3:
 - Enhanced xfs_io with MAP_SYNC and dm-log-writes functionality instead of
   creating a one-off test program.  (Dave Chinner)

 - Improved dm target version checking. (Amir)

 - Fixed dm-log-writes replay issue, some general cleanup, broke changes
   out into a series.

[1]: https://lists.01.org/pipermail/linux-nvdimm/2017-November/013164.html
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/zwisler/linux.git/log/?h=map_sync_dm_log_writes
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/zwisler/xfsprogs-dev.git/log/?h=map_sync
[4]: https://git.kernel.org/pub/scm/linux/kernel/git/zwisler/xfstests-dev.git/log/?h=map_sync_test_v4

Ross Zwisler (4):
  common/rc: add _scratch_has_mount_option()
  dm-log-writes: only replay log to marks that exist
  dm-log-writes: allow DAX to be used when possible
  generic: add test for DAX MAP_SYNC support

 common/dmlogwrites           |  9 +++--
 common/rc                    | 33 ++++++++++++++++--
 doc/requirement-checking.txt |  5 +--
 tests/generic/468            | 83 ++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/468.out        |  3 ++
 tests/generic/group          |  1 +
 6 files changed, 128 insertions(+), 6 deletions(-)
 create mode 100755 tests/generic/468
 create mode 100644 tests/generic/468.out

-- 
2.9.5

[fstests PATCH v4 1/4] common/rc: add _scratch_has_mount_option()

[Ross Zwisler]

Allow tests to inquire about whether a mount option is set, rather than
just disallowing it via _exclude_scratch_mount_option().

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Suggested-by: Amir Goldstein <amir73il@gmail.com>
---
 common/rc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/common/rc b/common/rc
index 0cda9da..701b1ff 100644
--- a/common/rc
+++ b/common/rc
@@ -3105,6 +3105,16 @@ _exclude_scratch_mount_option()
 	done
 }
 
+_scratch_has_mount_option()
+{
+	local mnt_opts=$(_normalize_mount_options)
+
+	if echo $mnt_opts | grep -qw "$1"; then
+		return 0
+	fi
+	return 1
+}
+
 _require_atime()
 {
 	_exclude_scratch_mount_option "noatime"
-- 
2.9.5

[fstests PATCH v4 2/4] dm-log-writes: only replay log to marks that exist

[Ross Zwisler]

The 'replay-log' executable will replay the dm-log-writes log until the
given mark, or until the end of the log if the mark isn't found.

This means that if the mark you're looking for was never inserted in the
log or if you give garbage to _log_writes_replay_log() the entire log will
be replayed.  This can cause unexpected test results.

Fix this by making sure that the mark we're given actually exists in the
log before we allow the replay.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
---
 common/dmlogwrites | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/common/dmlogwrites b/common/dmlogwrites
index 247c744..80ed4fc 100644
--- a/common/dmlogwrites
+++ b/common/dmlogwrites
@@ -72,6 +72,10 @@ _log_writes_replay_log()
 {
 	_mark=$1
 
+	$here/src/log-writes/replay-log --log $LOGWRITES_DEV --find \
+		--end-mark $_mark >> $seqres.full 2>&1
+	[ $? -ne 0 ] && _fail "mark does not exist"
+
 	$here/src/log-writes/replay-log --log $LOGWRITES_DEV --replay $SCRATCH_DEV \
 		--end-mark $_mark >> $seqres.full 2>&1
 	[ $? -ne 0 ] && _fail "replay failed"
-- 
2.9.5

Re: [fstests PATCH v4 2/4] dm-log-writes: only replay log to marks that exist

[Eryu Guan]

On Fri, Nov 17, 2017 at 01:28:26PM -0700, Ross Zwisler wrote:
> The 'replay-log' executable will replay the dm-log-writes log until the
> given mark, or until the end of the log if the mark isn't found.
> 
> This means that if the mark you're looking for was never inserted in the
> log or if you give garbage to _log_writes_replay_log() the entire log will
> be replayed.  This can cause unexpected test results.
> 
> Fix this by making sure that the mark we're given actually exists in the
> log before we allow the replay.
> 
> Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> ---
>  common/dmlogwrites | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/common/dmlogwrites b/common/dmlogwrites
> index 247c744..80ed4fc 100644
> --- a/common/dmlogwrites
> +++ b/common/dmlogwrites
> @@ -72,6 +72,10 @@ _log_writes_replay_log()
>  {
>  	_mark=$1
>  
> +	$here/src/log-writes/replay-log --log $LOGWRITES_DEV --find \
> +		--end-mark $_mark >> $seqres.full 2>&1
> +	[ $? -ne 0 ] && _fail "mark does not exist"

Print out $_mark too?

Thanks,
Eryu

> +
>  	$here/src/log-writes/replay-log --log $LOGWRITES_DEV --replay $SCRATCH_DEV \
>  		--end-mark $_mark >> $seqres.full 2>&1
>  	[ $? -ne 0 ] && _fail "replay failed"
> -- 
> 2.9.5
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[fstests PATCH v4 3/4] dm-log-writes: allow DAX to be used when possible

[Ross Zwisler]

Enhance _require_dm_target so that a user can request a minimum version of a
given dm target.

DAX support was added to v1.1.0 of the dm-log-writes kernel module, so
allow the DAX mount option starting with that version.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Suggested-by: Amir Goldstein <amir73il@gmail.com>
---
 common/dmlogwrites           |  5 +++--
 common/rc                    | 23 +++++++++++++++++++++--
 doc/requirement-checking.txt |  5 +++--
 3 files changed, 27 insertions(+), 6 deletions(-)

diff --git a/common/dmlogwrites b/common/dmlogwrites
index 80ed4fc..2cb804c 100644
--- a/common/dmlogwrites
+++ b/common/dmlogwrites
@@ -23,8 +23,9 @@ _require_log_writes()
 	[ -z "$LOGWRITES_DEV" -o ! -b "$LOGWRITES_DEV" ] && \
 		_notrun "This test requires a valid \$LOGWRITES_DEV"
 
-	_exclude_scratch_mount_option dax
-	_require_dm_target log-writes
+	local z=0
+	_scratch_has_mount_option dax && z=1
+	_require_dm_target log-writes 1 $z 0
 	_require_test_program "log-writes/replay-log"
 }
 
diff --git a/common/rc b/common/rc
index 701b1ff..c1f24ed 100644
--- a/common/rc
+++ b/common/rc
@@ -1782,6 +1782,18 @@ _require_sane_bdev_flush()
 	fi
 }
 
+_compare_dm_target_versions()
+{
+	for i in $(seq 0 2); do
+		if [[ ${_actual_ver[$i]} > ${_required_ver[$i]} ]]; then
+			return
+		fi
+		if [[ ${_actual_ver[$i]} < ${_required_ver[$i]} ]]; then
+			_notrun "dm $_target version ${_required_ver[@]} required"
+		fi
+	done
+}
+
 # this test requires a specific device mapper target
 _require_dm_target()
 {
@@ -1795,8 +1807,15 @@ _require_dm_target()
 
 	modprobe dm-$_target >/dev/null 2>&1
 
-	$DMSETUP_PROG targets 2>&1 | grep -q ^$_target
-	if [ $? -ne 0 ]; then
+	local _version=$($DMSETUP_PROG targets 2>&1 | grep ^$_target)
+	if [[ $_version =~ .*v([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then
+		# check for a required minimum version?
+		if [[ $# == 4 ]]; then
+			local _actual_ver=(${BASH_REMATCH[@]:1})
+			local _required_ver=($2 $3 $4)
+			_compare_dm_target_versions
+		fi
+	else
 		_notrun "This test requires dm $_target support"
 	fi
 }
diff --git a/doc/requirement-checking.txt b/doc/requirement-checking.txt
index 4e01b1f..bbfc34f 100644
--- a/doc/requirement-checking.txt
+++ b/doc/requirement-checking.txt
@@ -112,10 +112,11 @@ _require_statx
 DEVICE MAPPER REQUIREMENTS
 ==========================
 
-_require_dm_target <name>
+_require_dm_target <name> [<version 1> <version 2> <version 3>]
 
      The test requires the use of the device mapper target and will be skipped
-     if it isn't available in the kernel.
+     if it isn't available in the kernel.  Optionally specify the minimum
+     three part version number of the dm target that is required.
 
 _require_log_writes
 
-- 
2.9.5

Re: [fstests PATCH v4 3/4] dm-log-writes: allow DAX to be used when possible

[Eryu Guan]

On Fri, Nov 17, 2017 at 01:28:27PM -0700, Ross Zwisler wrote:
> Enhance _require_dm_target so that a user can request a minimum version of a
> given dm target.
> 
> DAX support was added to v1.1.0 of the dm-log-writes kernel module, so
> allow the DAX mount option starting with that version.

Is is possible not relying on the version numbers but actually trying
what you want to do and _notrun if that fails? Currently fstests does no
version number checking at all, and it'd be great if we keep that
tradition.

Thanks,
Eryu

Re: [fstests PATCH v4 3/4] dm-log-writes: allow DAX to be used when possible

[Ross Zwisler]

On Sat, Nov 18, 2017 at 02:17:03PM +0800, Eryu Guan wrote:
> On Fri, Nov 17, 2017 at 01:28:27PM -0700, Ross Zwisler wrote:
> > Enhance _require_dm_target so that a user can request a minimum version of a
> > given dm target.
> > 
> > DAX support was added to v1.1.0 of the dm-log-writes kernel module, so
> > allow the DAX mount option starting with that version.
> 
> Is is possible not relying on the version numbers but actually trying
> what you want to do and _notrun if that fails? Currently fstests does no
> version number checking at all, and it'd be great if we keep that
> tradition.

Yep, and I agree that this is better.  I'll be fixed in the next version.
Thanks for the feedback.

[fstests PATCH v4 4/4] generic: add test for DAX MAP_SYNC support

[Ross Zwisler]

This test creates a file and writes to it via an mmap(), but never syncs
via fsync/msync.  This process is tracked via dm-log-writes, then replayed.

If MAP_SYNC is working the dm-log-writes replay will show the test file
with 1 MiB of on-media block allocations.  This is because each allocating
page fault included an implicit metadata sync.  If MAP_SYNC isn't working
(which you can test by removing the "-S" flag to xfs_io mmap) the file
will be smaller or missing entirely.

Note that dm-log-writes doesn't track the data that we write via the
mmap(), so we can't do any data integrity checking.  We can only verify
that the metadata writes for the page faults happened.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
---
 tests/generic/468     | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/468.out |  3 ++
 tests/generic/group   |  1 +
 3 files changed, 87 insertions(+)
 create mode 100755 tests/generic/468
 create mode 100644 tests/generic/468.out

diff --git a/tests/generic/468 b/tests/generic/468
new file mode 100755
index 0000000..fcbbf34
--- /dev/null
+++ b/tests/generic/468
@@ -0,0 +1,83 @@
+#! /bin/bash
+# FS QA Test No. 468
+#
+# Use dm-log-writes to verify that MAP_SYNC actually syncs metadata during
+# page faults.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Intel Corporation.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_log_writes_cleanup
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_log_writes
+_require_scratch_dax
+_require_xfs_io_command "log_writes"
+
+_log_writes_init
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mount -o dax
+
+LEN=$((1024 * 1024)) # 1 MiB
+
+xfs_io -t -c "truncate $LEN" -c "mmap -S 0 $LEN" -c "mwrite 0 $LEN" \
+	-c "log_writes -d $LOGWRITES_NAME -m preunmap" \
+	-f $SCRATCH_MNT/test > /dev/null 2>&1
+
+# Unmount the scratch dir and tear down the log writes target
+_log_writes_unmount
+_log_writes_remove
+_check_scratch_fs
+
+# destroy previous filesystem so we can be sure our rebuild works
+_scratch_mkfs >> $seqres.full 2>&1
+
+# check pre-unmap state
+_log_writes_replay_log preunmap
+_scratch_mount
+
+# We should see $SCRATCH_MNT/test as having 1 MiB in block allocations
+stat -c "%s" $SCRATCH_MNT/test
+
+_scratch_unmount
+_check_scratch_fs
+
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/generic/468.out b/tests/generic/468.out
new file mode 100644
index 0000000..5f897af
--- /dev/null
+++ b/tests/generic/468.out
@@ -0,0 +1,3 @@
+QA output created by 468
+1048576
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 9183950..c061842 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -470,3 +470,4 @@
 465 auto rw quick aio
 466 auto quick rw
 467 auto quick exportfs
+468 auto quick dax
-- 
2.9.5