[PATCH 1/2] btrfs: test incremental send after a succession of rename and link operations

[PATCH 1/2] btrfs: test incremental send after a succession of rename and link operations

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Test that an incremental send operation emits the correct path for link
and rename operation after swapping the names and locations of several
inodes in a way that creates a nasty dependency of rename and link
operations. Notably one file has its name and location swapped with a
directory for which it used to have a directory entry in it.

This test currently fails but a kernel patch for it exists and has the
following subject:

  "btrfs: send, orphanize first all conflicting inodes when processing references"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 tests/btrfs/221     | 119 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/221.out |   6 +++
 tests/btrfs/group   |   1 +
 3 files changed, 126 insertions(+)
 create mode 100755 tests/btrfs/221
 create mode 100644 tests/btrfs/221.out

diff --git a/tests/btrfs/221 b/tests/btrfs/221
new file mode 100755
index 00000000..a482d0c5
--- /dev/null
+++ b/tests/btrfs/221
@@ -0,0 +1,119 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. btrfs/221
+#
+# Test that an incremental send operation emits the correct path for link and
+# rename operation after swapping the names and locations of several inodes in
+# a way that creates a nasty dependency of rename and link operations. Notably
+# one file has its name and location swapped with a directory for which it used
+# to have a directory entry in it.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -fr $send_files_dir
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs btrfs
+_require_test
+_require_scratch
+_require_fssum
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+
+rm -f $seqres.full
+rm -fr $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+touch $SCRATCH_MNT/a
+touch $SCRATCH_MNT/b
+mkdir $SCRATCH_MNT/testdir
+# We want "a" to have a lower inode number than its parent directory, so it
+# was created before the directory and then moved into it.
+mv $SCRATCH_MNT/a $SCRATCH_MNT/testdir/a
+
+# Filesystem looks like:
+#
+# .                                                      (ino 256)
+# |----- testdir/                                        (ino 259)
+# |          |----- a                                    (ino 257)
+# |
+# |----- b                                               (ino 258)
+#
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
+	$SCRATCH_MNT/mysnap1 > /dev/null
+
+$BTRFS_UTIL_PROG send -f $send_files_dir/1.snap \
+	$SCRATCH_MNT/mysnap1 2>&1 1>/dev/null | _filter_scratch
+
+# Now rename 259 to "testdir_2", then change the name of 257 to "testdir" and
+# make it a direct descendant of the root inode (256). Also create a new link
+# for inode 257 with the old name of inode 258. By swapping the names and
+# location of several inodes and create a nasty dependency chain of rename and
+# link operations.
+mv $SCRATCH_MNT/testdir/a $SCRATCH_MNT/a2
+touch $SCRATCH_MNT/testdir/a
+mv $SCRATCH_MNT/b $SCRATCH_MNT/b2
+ln $SCRATCH_MNT/a2 $SCRATCH_MNT/b
+mv $SCRATCH_MNT/testdir $SCRATCH_MNT/testdir_2
+mv $SCRATCH_MNT/a2 $SCRATCH_MNT/testdir
+
+# Filesystem now looks like:
+#
+# .                                                      (ino 256)
+# |----- testdir_2/                                      (ino 259)
+# |          |----- a                                    (ino 260)
+# |
+# |----- testdir                                         (ino 257)
+# |----- b                                               (ino 257)
+# |----- b2                                              (ino 258)
+#
+
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
+		 $SCRATCH_MNT/mysnap2 > /dev/null
+$BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 -f $send_files_dir/2.snap \
+		 $SCRATCH_MNT/mysnap2 2>&1 1>/dev/null | _filter_scratch
+
+$FSSUM_PROG -A -f -w $send_files_dir/1.fssum $SCRATCH_MNT/mysnap1
+$FSSUM_PROG -A -f -w $send_files_dir/2.fssum \
+	-x $SCRATCH_MNT/mysnap2/mysnap1 $SCRATCH_MNT/mysnap2
+
+# Now recreate the filesystem by receiving both send streams and verify we get
+# the same content that the original filesystem had.
+_scratch_unmount
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+$BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT > /dev/null
+
+# The receive operation below used to fail because when attemping to create the
+# hard link named "b" for inode 257, the link operation contained a target path
+# of "o259-6-0/a", which caused the receiver process to fail because inode 259
+# was not yet orphanized (renamed to "o259-6-0"), it still had the name "testdir"
+# when the link operation was issued.
+$BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT > /dev/null
+
+$FSSUM_PROG -r $send_files_dir/1.fssum $SCRATCH_MNT/mysnap1
+$FSSUM_PROG -r $send_files_dir/2.fssum $SCRATCH_MNT/mysnap2
+
+status=0
+exit
diff --git a/tests/btrfs/221.out b/tests/btrfs/221.out
new file mode 100644
index 00000000..4c887062
--- /dev/null
+++ b/tests/btrfs/221.out
@@ -0,0 +1,6 @@
+QA output created by 221
+At subvol SCRATCH_MNT/mysnap1
+At subvol SCRATCH_MNT/mysnap2
+At subvol mysnap1
+OK
+OK
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 1b5fa695..8b285dbb 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -222,3 +222,4 @@
 218 auto quick volume
 219 auto quick volume
 220 auto quick
+221 auto quick send
-- 
2.26.2

[PATCH 2/2] btrfs: test incremental send after swapping same file with two directories

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Test an incremental send operation after doing a series of changes in a
tree such that one inode gets two hardlinks with names and locations
swapped with two other inodes that correspond to different directories,
and one of these directories is the parent of the other directory.

This currently fails on btrfs, the receive of the incremental send stream
fails. This is fixed by a patchset for btrfs which has two patches with the
following subjects:

  "btrfs: send, orphanize first all conflicting inodes when processing references"
  "btrfs: send, recompute reference path after orphanization of a directory"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 tests/btrfs/222     | 142 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/222.out |   6 ++
 tests/btrfs/group   |   1 +
 3 files changed, 149 insertions(+)
 create mode 100755 tests/btrfs/222
 create mode 100644 tests/btrfs/222.out

diff --git a/tests/btrfs/222 b/tests/btrfs/222
new file mode 100755
index 00000000..51fd052a
--- /dev/null
+++ b/tests/btrfs/222
@@ -0,0 +1,142 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. btrfs/222
+#
+# Test an incremental send operation after doing a series of changes in a tree
+# such that one inode gets two hardlinks with names and locations swapped with
+# two other inodes that correspond to different directories, and one of these
+# directories is the parent of the other directory.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -fr $send_files_dir
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs btrfs
+_require_test
+_require_scratch
+_require_fssum
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+
+rm -f $seqres.full
+rm -fr $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+touch $SCRATCH_MNT/f1
+touch $SCRATCH_MNT/f2
+mkdir $SCRATCH_MNT/d1
+mkdir $SCRATCH_MNT/d1/d2
+
+# Filesystem looks like:
+#
+# .                                     (ino 256)
+# |----- f1                             (ino 257)
+# |----- f2                             (ino 258)
+# |----- d1/                            (ino 259)
+#        |----- d2/                     (ino 260)
+#
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
+	$SCRATCH_MNT/mysnap1 > /dev/null
+
+$BTRFS_UTIL_PROG send -f $send_files_dir/1.snap \
+	$SCRATCH_MNT/mysnap1 2>&1 1>/dev/null | _filter_scratch
+
+# Now do a series of changes such that:
+#
+# *) inode 258 has one new hardlink and the previous name changed
+#
+# *) both names conflict with the old names of two other inodes:
+#
+#    1) the new name "d1" conflicts with the old name of inode 259, under
+#       directory inode 256 (root)
+#
+#    2) the new name "d2" conflicts with the old name of inode 260 under
+#       directory inode 259
+#
+# *) inodes 259 and 260 now have the old names of inode 258
+#
+# *) inode 257 is now located under inode 260 - an inode with a number
+#    smaller than the inode (258) for which we created a second hard link
+#    and swapped its names with inodes 259 and 260
+#
+ln $SCRATCH_MNT/f2 $SCRATCH_MNT/d1/f2_link
+mv $SCRATCH_MNT/f1 $SCRATCH_MNT/d1/d2/f1
+
+# Swap d1 and f2.
+mv $SCRATCH_MNT/d1 $SCRATCH_MNT/tmp
+mv $SCRATCH_MNT/f2 $SCRATCH_MNT/d1
+mv $SCRATCH_MNT/tmp $SCRATCH_MNT/f2
+
+# Swap d2 and f2_link
+mv $SCRATCH_MNT/f2/d2 $SCRATCH_MNT/tmp
+mv $SCRATCH_MNT/f2/f2_link $SCRATCH_MNT/f2/d2
+mv $SCRATCH_MNT/tmp $SCRATCH_MNT/f2/f2_link
+
+# Filesystem now looks like:
+#
+# .                                     (ino 256)
+# |----- d1                             (ino 258)
+# |----- f2/                            (ino 259)
+#        |----- f2_link/                (ino 260)
+#        |       |----- f1              (ino 257)
+#        |
+#        |----- d2                      (ino 258)
+#
+
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
+		 $SCRATCH_MNT/mysnap2 > /dev/null
+$BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 -f $send_files_dir/2.snap \
+		 $SCRATCH_MNT/mysnap2 2>&1 1>/dev/null | _filter_scratch
+
+$FSSUM_PROG -A -f -w $send_files_dir/1.fssum $SCRATCH_MNT/mysnap1
+$FSSUM_PROG -A -f -w $send_files_dir/2.fssum \
+	-x $SCRATCH_MNT/mysnap2/mysnap1 $SCRATCH_MNT/mysnap2
+
+# Now recreate the filesystem by receiving both send streams and verify we get
+# the same content that the original filesystem had.
+_scratch_unmount
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+$BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT > /dev/null
+
+# The receive operation below used to fail with the following error:
+#
+#    ERROR: rename d1/d2 -> o260-6-0 failed: No such file or directory
+#
+# Because when processing inode 258, the kernel orphanized first inode 259,
+# renaming it from "d1" to "o259-6-0", since it the new reference named "d1"
+# for inode 258 conflicts with the old one of inode 259, which was not yet
+# processed. After that, because the new reference named "d2" for inode 258
+# conflicts with the old reference of the not yet processed inode 260, it
+# tried to orphanize inode 260 - however the corresponding rename operation
+# used a source path of "d1/d2", instead of "o259-6-0/d2".
+#
+$BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT > /dev/null
+
+$FSSUM_PROG -r $send_files_dir/1.fssum $SCRATCH_MNT/mysnap1
+$FSSUM_PROG -r $send_files_dir/2.fssum $SCRATCH_MNT/mysnap2
+
+status=0
+exit
diff --git a/tests/btrfs/222.out b/tests/btrfs/222.out
new file mode 100644
index 00000000..4fea113d
--- /dev/null
+++ b/tests/btrfs/222.out
@@ -0,0 +1,6 @@
+QA output created by 222
+At subvol SCRATCH_MNT/mysnap1
+At subvol SCRATCH_MNT/mysnap2
+At subvol mysnap1
+OK
+OK
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 8b285dbb..3b009220 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -223,3 +223,4 @@
 219 auto quick volume
 220 auto quick
 221 auto quick send
+222 auto quick send
-- 
2.26.2

Re: [PATCH 1/2] btrfs: test incremental send after a succession of rename and link operations

[Josef Bacik]

On 9/21/20 9:15 AM, fdmanana@kernel.org wrote:
> From: Filipe Manana <fdmanana@suse.com>
> 
> Test that an incremental send operation emits the correct path for link
> and rename operation after swapping the names and locations of several
> inodes in a way that creates a nasty dependency of rename and link
> operations. Notably one file has its name and location swapped with a
> directory for which it used to have a directory entry in it.
> 
> This test currently fails but a kernel patch for it exists and has the
> following subject:
> 
>    "btrfs: send, orphanize first all conflicting inodes when processing references"
> 
> Signed-off-by: Filipe Manana <fdmanana@suse.com>

Reviewed-by: Josef Bacik <josef@toxicpanda.com>

Thanks,

Josef

Re: [PATCH 2/2] btrfs: test incremental send after swapping same file with two directories

[Josef Bacik]

On 9/21/20 9:15 AM, fdmanana@kernel.org wrote:
> From: Filipe Manana <fdmanana@suse.com>
> 
> Test an incremental send operation after doing a series of changes in a
> tree such that one inode gets two hardlinks with names and locations
> swapped with two other inodes that correspond to different directories,
> and one of these directories is the parent of the other directory.
> 
> This currently fails on btrfs, the receive of the incremental send stream
> fails. This is fixed by a patchset for btrfs which has two patches with the
> following subjects:
> 
>    "btrfs: send, orphanize first all conflicting inodes when processing references"
>    "btrfs: send, recompute reference path after orphanization of a directory"
> 
> Signed-off-by: Filipe Manana <fdmanana@suse.com>

Reviewed-by: Josef Bacik <josef@toxicpanda.com>

Thanks,

Josef