[PATCH 0/2] fstests: btrfs: the lost levels

[PATCH 0/2] fstests: btrfs: the lost levels

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

Inspired by a conversation a few of us had at LSF about forgotten
fstests, I went back and checked if I had let any tests disappear into
the ether. Turns out that I did, so here are the tests for Btrfs replace
missing that I never got merged, updated with everyone's comments from
last time around ([1], [2]). Patch 1/3 in that original series did get
merged, these two didn't fare so well.

Thanks!

1: https://patchwork.kernel.org/patch/6856101/
2: https://patchwork.kernel.org/patch/6856051/

Omar Sandoval (2):
  btrfs: add replace missing and replace RAID 5/6 to profile configs
  btrfs: add test for replacing a missing device

 common/rc           |  95 ++++++++++++++++++++---------------------
 tests/btrfs/027     | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/027.out |   2 +
 tests/btrfs/group   |   1 +
 4 files changed, 170 insertions(+), 47 deletions(-)
 create mode 100755 tests/btrfs/027
 create mode 100644 tests/btrfs/027.out

-- 
2.8.0

[PATCH 1/2] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

Replacing and scrubbing RAID 5/6 is now supported on Btrfs. Enable it in
_btrfs_get_profile_configs while making it more generic to also support
replace missing.

Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 common/rc | 95 ++++++++++++++++++++++++++++++++-------------------------------
 1 file changed, 48 insertions(+), 47 deletions(-)

diff --git a/common/rc b/common/rc
index 8bec836805c6..91e8f1c8e693 100644
--- a/common/rc
+++ b/common/rc
@@ -3207,60 +3207,61 @@ _btrfs_get_profile_configs()
 		return
 	fi
 
-	# no user specified btrfs profile configs, export the default configs
 	if [ -z "$BTRFS_PROFILE_CONFIGS" ]; then
-		# default configs
-		_btrfs_profile_configs=(
-			"-m single -d single"
-			"-m dup -d single"
-			"-m raid0 -d raid0"
-			"-m raid1 -d raid0"
-			"-m raid1 -d raid1"
-			"-m raid10 -d raid10"
-			"-m raid5 -d raid5"
-			"-m raid6 -d raid6"
+		# Default configurations to test.
+		local configs=(
+			"single:single"
+			"dup:single"
+			"raid0:raid0"
+			"raid1:raid0"
+			"raid1:raid1"
+			"raid10:raid10"
+			"raid5:raid5"
+			"raid6:raid6"
 		)
+	else
+		# User-provided configurations.
+		local configs=(${BTRFS_PROFILE_CONFIGS[@]})
+	fi
 
-		# remove dup/raid5/raid6 profiles if we're doing device replace
-		# dup profile indicates only one device being used (SCRATCH_DEV),
-		# but we don't want to replace SCRATCH_DEV, which will be used in
-		# _scratch_mount/_check_scratch_fs etc.
-		# and raid5/raid6 doesn't support replace yet
+	_btrfs_profile_configs=()
+	for cfg in "${configs[@]}"; do
+		local supported=true
+		local profiles=(${cfg/:/ })
 		if [ "$1" == "replace" ]; then
-			_btrfs_profile_configs=(
-				"-m single -d single"
-				"-m raid0 -d raid0"
-				"-m raid1 -d raid0"
-				"-m raid1 -d raid1"
-				"-m raid10 -d raid10"
-				# add these back when raid5/6 is working with replace
-				#"-m raid5 -d raid5"
-				#"-m raid6 -d raid6"
+			# We can't do replace with these profiles because they
+			# imply only one device ($SCRATCH_DEV), and we need to
+			# keep $SCRATCH_DEV around for _scratch_mount
+			# and _check_scratch_fs.
+			local unsupported=(
+				"dup"
 			)
+		elif [ "$1" == "replace-missing" ]; then
+			# We can't replace missing devices with these profiles
+			# because there isn't enough redundancy.
+			local unsupported=(
+				"single"
+				"dup"
+				"raid0"
+			)
+		else
+			local unsupported=()
 		fi
-		export _btrfs_profile_configs
-		return
-	fi
-
-	# parse user specified btrfs profile configs
-	local i=0
-	local cfg=""
-	for cfg in $BTRFS_PROFILE_CONFIGS; do
-		# turn "metadata:data" format to "-m metadata -d data"
-		# and assign it to _btrfs_profile_configs array
-		cfg=`echo "$cfg" | sed -e 's/^/-m /' -e 's/:/ -d /'`
-		_btrfs_profile_configs[$i]="$cfg"
-		let i=i+1
-	done
-
-	if [ "$1" == "replace" ]; then
-		if echo ${_btrfs_profile_configs[*]} | grep -q raid[56]; then
-			_notrun "RAID5/6 doesn't support btrfs device replace yet"
-		fi
-		if echo ${_btrfs_profile_configs[*]} | grep -q dup; then
-			_notrun "Do not set dup profile in btrfs device replace test"
+		for unsupp in "${unsupported[@]}"; do
+			if [ "${profiles[0]}" == "$unsupp" -o "${profiles[1]}" == "$unsupp" ]; then
+			     if [ -z "$BTRFS_PROFILE_CONFIGS" ]; then
+				     # For the default config, just omit it.
+				     supported=false
+			     else
+				     # For user-provided config, don't run the test.
+				     _notrun "Profile $unsupp not supported for $1"
+			     fi
+			fi
+		done
+		if "$supported"; then
+			_btrfs_profile_configs+=("-m ${profiles[0]} -d ${profiles[1]}")
 		fi
-	fi
+	done
 	export _btrfs_profile_configs
 }
 
-- 
2.8.0

[PATCH 2/2] btrfs: add test for replacing a missing device

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

Now that _btrfs_get_profile_configs supports replace missing and the
kernel doesn't crash when replacing a missing RAID 5/6 device, test it.
Based on an earlier test from Wang Yanfeng.

Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 tests/btrfs/027     | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/027.out |   2 +
 tests/btrfs/group   |   1 +
 3 files changed, 122 insertions(+)
 create mode 100755 tests/btrfs/027
 create mode 100644 tests/btrfs/027.out

diff --git a/tests/btrfs/027 b/tests/btrfs/027
new file mode 100755
index 000000000000..f0844a14f8e6
--- /dev/null
+++ b/tests/btrfs/027
@@ -0,0 +1,119 @@
+#! /bin/bash
+# FS QA Test 027
+#
+# Test replace of a missing device on various data and metadata profiles.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Facebook.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs generic
+_supported_os IRIX Linux
+# we check scratch dev after each loop
+_require_scratch_nocheck
+_require_scratch_dev_pool 5
+_require_scratch_dev_pool_equal_size
+_btrfs_get_profile_configs replace-missing
+_require_command "$WIPEFS_PROG" wipefs
+
+run_test()
+{
+	local mkfs_opts=$1
+	local saved_scratch_dev_pool=$SCRATCH_DEV_POOL
+	local replace_dev=`echo $SCRATCH_DEV_POOL | awk '{print $NF}'`
+
+	echo "Test $mkfs_opts" >>$seqres.full
+
+	SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | sed -e "s# *$replace_dev *##"`
+	_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
+	# make sure we created btrfs with desired options
+	if [ $? -ne 0 ]; then
+		echo "mkfs $mkfs_opts failed"
+		SCRATCH_DEV_POOL=$saved_scratch_dev_pool
+		return
+	fi
+	_scratch_mount >>$seqres.full 2>&1
+
+	local missing_dev=`echo $SCRATCH_DEV_POOL | awk '{print $2}'`
+	local missing_dev_id=`$BTRFS_UTIL_PROG fi show $SCRATCH_MNT | grep $missing_dev | awk '{print $2}'`
+
+	# get some data on the filesystem so there's something to replace
+	dd if=/dev/urandom of="$SCRATCH_MNT"/file1 bs=1M count=1 \
+		>>$seqres.full 2>&1
+	dd if=/dev/urandom of="$SCRATCH_MNT"/file2 bs=1M count=2 \
+		>>$seqres.full 2>&1
+	dd if=/dev/urandom of="$SCRATCH_MNT"/file3 bs=1M count=4 \
+		>>$seqres.full 2>&1
+
+	# nuke a device and remount in degraded mode
+	_scratch_unmount
+	$WIPEFS_PROG -a "$missing_dev" >>$seqres.full 2>&1
+	_scratch_mount -o degraded >>$seqres.full 2>&1
+
+	# replace $missing_dev with $replace_dev and scrub it to double-check
+	$BTRFS_UTIL_PROG replace start -B -r $missing_dev_id $replace_dev \
+		$SCRATCH_MNT -f >>$seqres.full 2>&1
+	if [ $? -ne 0 ]; then
+		echo "btrfs replace failed"
+		SCRATCH_DEV_POOL=$saved_scratch_dev_pool
+		return
+	fi
+	$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1
+	if [ $? -ne 0 ]; then
+		echo "btrfs scrub failed"
+		SCRATCH_DEV_POOL=$saved_scratch_dev_pool
+		return
+	fi
+
+	_scratch_unmount
+	# we called _require_scratch_nocheck instead of _require_scratch
+	# do check after test for each profile config
+	_check_scratch_fs
+	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
+}
+
+echo "Silence is golden"
+for t in "${_btrfs_profile_configs[@]}"; do
+	run_test "$t"
+done
+
+status=0
+exit
diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
new file mode 100644
index 000000000000..ef78cdae50e8
--- /dev/null
+++ b/tests/btrfs/027.out
@@ -0,0 +1,2 @@
+QA output created by 027
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 13aa1e57d61e..90af8fc2d106 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -29,6 +29,7 @@
 024 auto quick compress
 025 auto quick send clone
 026 auto quick compress prealloc
+027 auto replace
 029 auto quick clone
 030 auto quick send
 031 auto quick subvol clone
-- 
2.8.0

Re: [PATCH 2/2] btrfs: add test for replacing a missing device

[Josef Bacik]

On 04/28/2016 12:31 AM, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
>
> Now that _btrfs_get_profile_configs supports replace missing and the
> kernel doesn't crash when replacing a missing RAID 5/6 device, test it.
> Based on an earlier test from Wang Yanfeng.
>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
>  tests/btrfs/027     | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/027.out |   2 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 122 insertions(+)
>  create mode 100755 tests/btrfs/027
>  create mode 100644 tests/btrfs/027.out
>
> diff --git a/tests/btrfs/027 b/tests/btrfs/027
> new file mode 100755
> index 000000000000..f0844a14f8e6
> --- /dev/null
> +++ b/tests/btrfs/027
> @@ -0,0 +1,119 @@
> +#! /bin/bash
> +# FS QA Test 027
> +#
> +# Test replace of a missing device on various data and metadata profiles.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2016 Facebook.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs generic
> +_supported_os IRIX Linux

Pretty sure we don't support btrfs on IRIX.  Fix that up and you can add

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef