[PATCH 0/4] fstests/btrfs: add a test case for send v2 and an update

[PATCH 0/4] fstests/btrfs: add a test case for send v2 and an update

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

This adds a test case specific for send v2 with some preparatory work in
order to allow for that. Support for v2 send streams was added in
btrfs-progs v5.19 and the kernel support introduced with kernel 6.0.

More send v2 specific tests will be added later.

Filipe Manana (4):
  btrfs: add a _require_btrfs_send_v2 helper
  common: make _filter_fiemap_flags optionally print the encoded flag
  btrfs/280: also verify that fiemap reports extents as encoded
  btrfs: test a case with compressed send stream and a shared extent

 common/btrfs        | 14 +++++++
 common/punch        | 40 ++++++++++++++++++--
 tests/btrfs/280     |  4 +-
 tests/btrfs/280.out | 12 +++---
 tests/btrfs/281     | 89 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/281.out | 17 +++++++++
 6 files changed, 164 insertions(+), 12 deletions(-)
 create mode 100755 tests/btrfs/281
 create mode 100644 tests/btrfs/281.out

-- 
2.35.1

[PATCH 1/4] btrfs: add a _require_btrfs_send_v2 helper

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Add a helper to check that both btrfs-progs and kernel support the v2 send
stream, so that we can have tests specific for send v2 stream.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 common/btrfs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/common/btrfs b/common/btrfs
index d27d3384..ee673a93 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -585,3 +585,17 @@ _require_btrfs_corrupt_block()
 {
 	_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs-corrupt-block
 }
+
+_require_btrfs_send_v2()
+{
+	# Check first if btrfs-progs supports the v2 stream.
+	_require_btrfs_command send --compressed-data
+
+	# Now check the kernel support. If send_stream_version does not exists,
+	# then it's a kernel that only supports v1.
+	[ -f /sys/fs/btrfs/features/send_stream_version ] || \
+		_notrun "kernel does not support send stream v2"
+
+	[ $(cat /sys/fs/btrfs/features/send_stream_version) -gt 1 ] || \
+		_notrun "kernel does not support send stream v2"
+}
-- 
2.35.1

[PATCH 2/4] common: make _filter_fiemap_flags optionally print the encoded flag

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

We'd like to have some btrfs test cases in the future to verify that
extents are compressed when using fiemap. For that we can just check if
the FIEMAP_EXTENT_ENCODED (0x8) flag is set for an extent. Currently
_filter_fiemap_flags does not print that flag, so this changes it to
print the flag.

However printing the encoded flag is optional, because some tests use
the filter and use its output to match the golden output. So always
printing the flag would make the tests fail on btrfs when they are run
with "-o compress" (or compress-force) set in MOUNT_OPTIONS due to a
mismatch with the golden output. The tests that can be run with or
without compression on btrfs are generic/352, generic/353 and btrfs/279.
Since those tests don't care about the encoded flag, there is no need to
change them, just make the output of the flag optional, and any future
tests that want to check the presence of the encoded flag, will just pass
a parameter to _filter_fiemap_flags to tell it that the encoded flag
should be printed.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 common/punch | 40 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 36 insertions(+), 4 deletions(-)

diff --git a/common/punch b/common/punch
index 94599c35..3b8be21a 100644
--- a/common/punch
+++ b/common/punch
@@ -109,7 +109,23 @@ _filter_fiemap()
 
 _filter_fiemap_flags()
 {
-	$AWK_PROG '
+	local include_encoded_flag=0
+
+	# Unless a first argument is passed and with a value of 1, the fiemap
+	# encoded flag is filtered out.
+	# This is because tests that use this filter's output in their golden
+	# output may get the encoded flag set or not depending on the filesystem
+	# and its configuration. For example, running btrfs with "-o compress"
+	# (or compress-force) set in MOUNT_OPTIONS, then extents that get
+	# compressed are reported with the encoded flag, otherwise that flag is
+	# not reported. Like this the fs configuration does not cause a mismatch
+	# with the golden output, and tests that exercise specific configurations
+	# can explicitly ask for the encoded flag to be printed.
+	if [ ! -z "$1" ] && [ $1 -eq 1 ]; then
+		include_encoded_flag=1
+	fi
+
+	local awk_script='
 		$3 ~ /hole/ {
 			print $1, $2, $3;
 			next;
@@ -126,7 +142,22 @@ _filter_fiemap_flags()
 			if (and(flags, 0x2000)) {
 				flag_str = "shared";
 				set = 1;
-			}
+			}'
+
+	if [ $include_encoded_flag -eq 1 ]; then
+		awk_script=$awk_script'
+			if (and(flags, 0x8)) {
+				if (set) {
+					flag_str = flag_str"|";
+				} else {
+					flag_str = "";
+				}
+				flag_str = flag_str"encoded";
+				set = 1;
+			}'
+	fi
+
+	awk_script=$awk_script'
 			if (and(flags, 0x1)) {
 				if (set) {
 					flag_str = flag_str"|";
@@ -136,8 +167,9 @@ _filter_fiemap_flags()
 				flag_str = flag_str"last";
 			}
 			print $1, $2, flag_str
-		}' |
-	_coalesce_extents
+		}'
+
+	$AWK_PROG -e "$awk_script" | _coalesce_extents
 }
 
 # Filters fiemap output to only print the 
-- 
2.35.1

[PATCH 3/4] btrfs/280: also verify that fiemap reports extents as encoded

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Now that _filter_fiemap_flags() optionally reports the encoded flag and
since btrfs/280 explicitly uses and tests compression, make it check that
fiemap reports the compressed extents with the encoded flag set.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 tests/btrfs/280     |  4 ++--
 tests/btrfs/280.out | 12 ++++++------
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/tests/btrfs/280 b/tests/btrfs/280
index 06ef221e..fc049adb 100755
--- a/tests/btrfs/280
+++ b/tests/btrfs/280
@@ -42,7 +42,7 @@ $BTRFS_UTIL_PROG subvolume snapshot $SCRATCH_MNT $SCRATCH_MNT/snap | _filter_scr
 echo
 echo "File foo fiemap before COWing extent:"
 echo
-$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags 1
 
 echo
 echo "Overwriting file range [120M, 120M + 128K) in the snapshot"
@@ -57,7 +57,7 @@ echo "File foo fiemap after COWing extent in the snapshot:"
 echo
 # Now we should have all extents marked as shared except the 128K extent in the
 # file range [120M, 120M + 128K).
-$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags 1
 
 # success, all done
 status=0
diff --git a/tests/btrfs/280.out b/tests/btrfs/280.out
index c3f82966..5371f3b0 100644
--- a/tests/btrfs/280.out
+++ b/tests/btrfs/280.out
@@ -5,8 +5,8 @@ Create a snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap'
 
 File foo fiemap before COWing extent:
 
-0: [0..261887]: shared
-1: [261888..262143]: shared|last
+0: [0..261887]: shared|encoded
+1: [261888..262143]: shared|encoded|last
 
 Overwriting file range [120M, 120M + 128K) in the snapshot
 
@@ -15,7 +15,7 @@ XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 File foo fiemap after COWing extent in the snapshot:
 
-0: [0..245759]: shared
-1: [245760..246015]: none
-2: [246016..261887]: shared
-3: [261888..262143]: shared|last
+0: [0..245759]: shared|encoded
+1: [245760..246015]: encoded
+2: [246016..261887]: shared|encoded
+3: [261888..262143]: shared|encoded|last
-- 
2.35.1

[PATCH 4/4] btrfs: test a case with compressed send stream and a shared extent

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Test that if we have a snapshot with a compressed extent that is partially
shared between two files, one of them has a size that is not sector size
aligned, we create a v2 send stream for the snapshot with compressed data,
and then apply that stream to another filesystem, the operation succeeds
and no data is missing. Also check that the file that had a reference to
the whole extent gets two compressed extents in the new filesystem, with
only one of them being shared (reflinked).

This tests a recent patch that landed in kernel 6.1-rc7:

  a11452a3709e ("btrfs: send: avoid unaligned encoded writes when attempting to clone range")

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 tests/btrfs/281     | 89 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/281.out | 17 +++++++++
 2 files changed, 106 insertions(+)
 create mode 100755 tests/btrfs/281
 create mode 100644 tests/btrfs/281.out

diff --git a/tests/btrfs/281 b/tests/btrfs/281
new file mode 100755
index 00000000..63fb89ea
--- /dev/null
+++ b/tests/btrfs/281
@@ -0,0 +1,89 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 281
+#
+# Test that if we have a snapshot with a compressed extent that is partially
+# shared between two files, one of them has a size that is not sector size
+# aligned, we create a v2 send stream for the snapshot with compressed data,
+# and then apply that stream to another filesystem, the operation succeeds and
+# no data is missing. Also check that the file that had a reference to the whole
+# extent gets two compressed extents in the new filesystem, with only one of
+# them being shared (reflinked).
+#
+. ./common/preamble
+_begin_fstest auto quick send compress clone fiemap
+
+. ./common/filter
+. ./common/reflink
+. ./common/punch # for _filter_fiemap_flags
+
+_supported_fs btrfs
+_require_test
+_require_scratch_reflink
+_require_btrfs_send_v2
+_require_xfs_io_command "fiemap"
+_require_fssum
+
+_fixed_by_kernel_commit a11452a3709e \
+	"btrfs: send: avoid unaligned encoded writes when attempting to clone range"
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+send_stream=$send_files_dir/snap.stream
+snap_fssum=$send_files_dir/snap.fssum
+
+rm -fr $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount -o compress
+
+# File foo has a size of 65K, which is not sector size aligned for any
+# supported sector size on btrfs.
+$XFS_IO_PROG -f -c "pwrite -S 0xab 0 65K" $SCRATCH_MNT/foo | _filter_xfs_io
+
+# File bar has a compressed extent (and its size is sector size aligned).
+$XFS_IO_PROG -f -c "pwrite -S 0xcd 0 128K" $SCRATCH_MNT/bar | _filter_xfs_io
+
+# Now clone only half of bar's extent into foo.
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 0 64K" $SCRATCH_MNT/foo \
+	| _filter_xfs_io
+
+echo "Creating snapshot and a send stream for it..."
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/snap \
+	| _filter_scratch
+$BTRFS_UTIL_PROG send --compressed-data -f $send_stream $SCRATCH_MNT/snap 2>&1 \
+	| _filter_scratch
+
+$FSSUM_PROG -A -f -w $snap_fssum $SCRATCH_MNT/snap
+
+echo "Creating a new filesystem to receive the send stream..."
+_scratch_unmount
+_scratch_mkfs >> $seqres.full 2>&1
+# Mount without compression, we created the stream with data compression enabled
+# so we want to verify that applying the stream preserves the compression.
+_scratch_mount
+
+$BTRFS_UTIL_PROG receive -f $send_stream $SCRATCH_MNT
+
+echo "Verifying data matches the original filesystem..."
+$FSSUM_PROG -r $snap_fssum $SCRATCH_MNT/snap
+
+# Now check that fiemap reports two extents for file bar:
+#
+# 1) The first extent should be encoded, because compression was enabled in the
+#    original filesystem, and should also be flagged as shared, since that file
+#    range was reflinked with file foo in the original filesystem;
+#
+# 2) The second extent should also be encoded (compression was enabled in the
+#    original filesystem), but not shared since that file range was not
+#    reflinked in the original filesystem. It should also have the "last" flag
+#    set, as it's the last extent in the file.
+#
+echo "File bar fiemap output in the new filesystem:"
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap/bar | _filter_fiemap_flags 1
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/281.out b/tests/btrfs/281.out
new file mode 100644
index 00000000..2585e3e5
--- /dev/null
+++ b/tests/btrfs/281.out
@@ -0,0 +1,17 @@
+QA output created by 281
+wrote 66560/66560 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+linked 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Creating snapshot and a send stream for it...
+Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap'
+At subvol SCRATCH_MNT/snap
+Creating a new filesystem to receive the send stream...
+At subvol snap
+Verifying data matches the original filesystem...
+OK
+File bar fiemap output in the new filesystem:
+0: [0..127]: shared|encoded
+1: [128..255]: encoded|last
-- 
2.35.1

Re: [PATCH 0/4] fstests/btrfs: add a test case for send v2 and an update

[Josef Bacik]

On Mon, Nov 28, 2022 at 12:07:20PM +0000, fdmanana@kernel.org wrote:
> From: Filipe Manana <fdmanana@suse.com>
> 
> This adds a test case specific for send v2 with some preparatory work in
> order to allow for that. Support for v2 send streams was added in
> btrfs-progs v5.19 and the kernel support introduced with kernel 6.0.
> 
> More send v2 specific tests will be added later.
> 

You can add

Reviewed-by: Josef Bacik <josef@toxicpanda.com>

for the whole series, thanks,

Josef