[PATCH 0/3] xfstests: test btrfs replace on RAID 5/6

[PATCH 0/3] xfstests: test btrfs replace on RAID 5/6

[Omar Sandoval]

Btrfs has supported replace on RAID 5 and 6 since 3.19, but xfstests
hasn't been updated to reflect that. Patches 1 and 2 in this series fix
that.

Additionally, I'm including Wang Yanfeng's test for my patch series
"Btrfs: RAID 5/6 missing device scrub+replace", updated to use the
infrastructure added by patch 2.

Thanks!

Omar Sandoval (2):
  btrfs/011: test replace on RAID 5/6 now that it's supported
  btrfs: add replace missing and replace RAID 5/6 to profile configs

Wang Yanfeng (1):
  btrfs: add a test of replace missing dev in diff raid

 common/rc           | 96 +++++++++++++++++++++++++++--------------------------
 tests/btrfs/011     |  4 +--
 tests/btrfs/095     | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/095.out |  2 ++
 tests/btrfs/group   |  1 +
 5 files changed, 147 insertions(+), 49 deletions(-)
 create mode 100755 tests/btrfs/095
 create mode 100644 tests/btrfs/095.out

-- 
2.4.6

[PATCH 1/3] btrfs/011: test replace on RAID 5/6 now that it's supported

[Omar Sandoval]

btrfs replace has been supported on RAID 5/6 since Linux 3.19.

Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 tests/btrfs/011 | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/btrfs/011 b/tests/btrfs/011
index f4f2fbed68d8..c7d35fa46062 100755
--- a/tests/btrfs/011
+++ b/tests/btrfs/011
@@ -282,8 +282,8 @@ workout "-m dup -d single" 1 cancel quick
 workout "-m dup -d dup -M" 1 no quick
 workout "-m raid0 -d raid0" 2 no quick
 workout "-m raid1 -d raid1" 2 no thorough
-#workout "-m raid5 -d raid5" 2 no quick # not yet supported for btrfs replace
-#workout "-m raid6 -d raid6" 3 no quick # not yet supported for btrfs replace
+workout "-m raid5 -d raid5" 2 no quick
+workout "-m raid6 -d raid6" 3 no quick
 workout "-m raid10 -d raid10" 4 no quick
 
 echo "*** done"
-- 
2.4.6

[PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Omar Sandoval]

Replacing and scrubbing RAID 5/6 is now supported on Btrfs. Enable it in
_btrfs_get_profile_configs while making it more generic to also support
replace missing.

Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 common/rc | 96 ++++++++++++++++++++++++++++++++-------------------------------
 1 file changed, 49 insertions(+), 47 deletions(-)

diff --git a/common/rc b/common/rc
index 610045eab304..3e6fdb6ebcfa 100644
--- a/common/rc
+++ b/common/rc
@@ -2748,60 +2748,62 @@ _btrfs_get_profile_configs()
 		return
 	fi
 
-	# no user specified btrfs profile configs, export the default configs
 	if [ -z "$BTRFS_PROFILE_CONFIGS" ]; then
-		# default configs
-		_btrfs_profile_configs=(
-			"-m single -d single"
-			"-m dup -d single"
-			"-m raid0 -d raid0"
-			"-m raid1 -d raid0"
-			"-m raid1 -d raid1"
-			"-m raid10 -d raid10"
-			"-m raid5 -d raid5"
-			"-m raid6 -d raid6"
+		# Default configurations to test.
+		local configs=(
+			"single:single"
+			"dup:single"
+			"raid0:raid0"
+			"raid1:raid0"
+			"raid1:raid1"
+			"raid10:raid10"
+			"raid5:raid5"
+			"raid6:raid6"
 		)
+	else
+		# User-provided configurations.
+		local configs=(${BTRFS_PROFILE_CONFIGS[@]})
+	fi
 
-		# remove dup/raid5/raid6 profiles if we're doing device replace
-		# dup profile indicates only one device being used (SCRATCH_DEV),
-		# but we don't want to replace SCRATCH_DEV, which will be used in
-		# _scratch_mount/_check_scratch_fs etc.
-		# and raid5/raid6 doesn't support replace yet
+	_btrfs_profile_configs=()
+	for cfg in "${configs[@]}"; do
+		local supported=true
+		local profiles=(${cfg/:/ })
 		if [ "$1" == "replace" ]; then
-			_btrfs_profile_configs=(
-				"-m single -d single"
-				"-m raid0 -d raid0"
-				"-m raid1 -d raid0"
-				"-m raid1 -d raid1"
-				"-m raid10 -d raid10"
-				# add these back when raid5/6 is working with replace
-				#"-m raid5 -d raid5"
-				#"-m raid6 -d raid6"
+			# We can't do replace with these profiles because they
+			# imply only one device ($SCRATCH_DEV), and we need to
+			# keep $SCRATCH_DEV around for _scratch_mount
+			# and _check_scratch_fs.
+			local unsupported=(
+				"single"
+				"dup"
 			)
+		elif [ "$1" == "replace-missing" ]; then
+			# We can't replace missing devices with these profiles
+			# because there isn't enough redundancy.
+			local unsupported=(
+				"single"
+				"dup"
+				"raid0"
+			)
+		else
+			local unsupported=()
 		fi
-		export _btrfs_profile_configs
-		return
-	fi
-
-	# parse user specified btrfs profile configs
-	local i=0
-	local cfg=""
-	for cfg in $BTRFS_PROFILE_CONFIGS; do
-		# turn "metadata:data" format to "-m metadata -d data"
-		# and assign it to _btrfs_profile_configs array
-		cfg=`echo "$cfg" | sed -e 's/^/-m /' -e 's/:/ -d /'`
-		_btrfs_profile_configs[$i]="$cfg"
-		let i=i+1
-	done
-
-	if [ "$1" == "replace" ]; then
-		if echo ${_btrfs_profile_configs[*]} | grep -q raid[56]; then
-			_notrun "RAID5/6 doesn't support btrfs device replace yet"
-		fi
-		if echo ${_btrfs_profile_configs[*]} | grep -q dup; then
-			_notrun "Do not set dup profile in btrfs device replace test"
+		for unsupp in "${unsupported[@]}"; do
+			if [ "${profiles[0]}" == "$unsupp" -o "${profiles[1]}" == "$unsupp" ]; then
+			     if [ -z "$BTRFS_PROFILE_CONFIGS" ]; then
+				     # For the default config, just omit it.
+				     supported=false
+			     else
+				     # For user-provided config, don't run the test.
+				     _notrun "Profile $unsupp not supported for $1"
+			     fi
+			fi
+		done
+		if "$supported"; then
+			_btrfs_profile_configs+=("-m ${profiles[0]} -d ${profiles[1]}")
 		fi
-	fi
+	done
 	export _btrfs_profile_configs
 }
 
-- 
2.4.6

[PATCH 3/3] btrfs: add a test of replace missing dev in diff raid

[Omar Sandoval]

From: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>

Test of missing device replace in different raid modes.  This
test requires SCRATCH_DEV_POOL contain 5 same size devices.

This issue has been fixed by Omar's patch:
	Btrfs: RAID 5/6 missing device scrub+replace

Signed-off-by: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 tests/btrfs/095     | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/095.out |  2 ++
 tests/btrfs/group   |  1 +
 3 files changed, 96 insertions(+)
 create mode 100755 tests/btrfs/095
 create mode 100644 tests/btrfs/095.out

diff --git a/tests/btrfs/095 b/tests/btrfs/095
new file mode 100755
index 000000000000..2dfb919fe4f6
--- /dev/null
+++ b/tests/btrfs/095
@@ -0,0 +1,93 @@
+#! /bin/bash
+# FS QA Test No. btrfs/095
+#
+# Test of missing device replace in different raid mode
+#
+# Be sure $SCRATCH_DEV_POOL including 5 devices. And all
+# devices in pool must be in the same size.
+#
+# To check the fs after replacing a dev, a scrub run is performed.
+#
+# This issue has been fixed by Omar Sandoval's patch:
+# 	Btrfs: RAID 5/6 missing device scrub+replace
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 Fujitsu.  All Rights Reserved.
+# Author: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+status=1
+trap "exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_scratch_dev_pool 5
+_btrfs_get_profile_configs replace-missing
+
+rm -f $seqres.full
+
+REPLACE_DEV=`echo $SCRATCH_DEV_POOL | awk '{print $5}'`
+export SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | awk '{print $1,$2,$3,$4}'`
+
+run_test () {
+	local mkfs_opts=$1
+	_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1 || _fail "mkfs failed"
+	_scratch_mount
+
+	PRUNE_DEV=`echo $SCRATCH_DEV_POOL | awk '{print $2}'`
+	PRUNE_DEV_ID=`$BTRFS_UTIL_PROG fi show $SCRATCH_MNT | grep $PRUNE_DEV | awk '{print $2}'`
+
+	# dd some data
+	dd if=/dev/urandom of="$SCRATCH_MNT"/file1 bs=1M count=1 \
+		>>$seqres.full 2>&1 || _fail "dd failed"
+	dd if=/dev/urandom of="$SCRATCH_MNT"/file2 bs=1M count=2 \
+		>>$seqres.full 2>&1 || _fail "dd failed"
+	dd if=/dev/urandom of="$SCRATCH_MNT"/file3 bs=1M count=4 \
+		>>$seqres.full 2>&1 || _fail "dd failed"
+
+	# prune the device PRUNE_DEV && remount by degraded mode
+	umount $SCRATCH_MNT
+	dd if=/dev/zero of=$PRUNE_DEV bs=1M count=1 >>$seqres.full 2>&1 \
+		|| _fail "dd failed"
+	mount -o degraded $SCRATCH_DEV $SCRATCH_MNT
+
+	# replace the missing dev $PRUNE_DEV with $REPLACE_DEV and scrub it
+	$BTRFS_UTIL_PROG replace start -B -r $PRUNE_DEV_ID $REPLACE_DEV \
+		$SCRATCH_MNT -f >>$seqres.full 2>&1 || _fail "replace failed"
+	$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 \
+		|| _fail "scrub failed"
+
+	umount $SCRATCH_MNT
+}
+
+echo "Silence is golden"
+for t in "${_btrfs_profile_configs[@]}"; do
+	run_test "$t"
+done
+
+status=0
+exit
diff --git a/tests/btrfs/095.out b/tests/btrfs/095.out
new file mode 100644
index 000000000000..80ad3b9c2a7b
--- /dev/null
+++ b/tests/btrfs/095.out
@@ -0,0 +1,2 @@
+QA output created by 095
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ffe18bff0d21..f20a191c24be 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -96,3 +96,4 @@
 092 auto quick send
 093 auto quick clone
 094 auto quick send
+095 auto
-- 
2.4.6

Re: [PATCH 1/3] btrfs/011: test replace on RAID 5/6 now that it's supported

[Eryu Guan]

On Thu, Jul 23, 2015 at 01:51:49PM -0700, Omar Sandoval wrote:
> btrfs replace has been supported on RAID 5/6 since Linux 3.19.
> 
> Signed-off-by: Omar Sandoval <osandov@fb.com>

Reviewed-by: Eryu Guan <eguan@redhat.com>

> ---
>  tests/btrfs/011 | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/btrfs/011 b/tests/btrfs/011
> index f4f2fbed68d8..c7d35fa46062 100755
> --- a/tests/btrfs/011
> +++ b/tests/btrfs/011
> @@ -282,8 +282,8 @@ workout "-m dup -d single" 1 cancel quick
>  workout "-m dup -d dup -M" 1 no quick
>  workout "-m raid0 -d raid0" 2 no quick
>  workout "-m raid1 -d raid1" 2 no thorough
> -#workout "-m raid5 -d raid5" 2 no quick # not yet supported for btrfs replace
> -#workout "-m raid6 -d raid6" 3 no quick # not yet supported for btrfs replace
> +workout "-m raid5 -d raid5" 2 no quick
> +workout "-m raid6 -d raid6" 3 no quick
>  workout "-m raid10 -d raid10" 4 no quick
>  
>  echo "*** done"
> -- 
> 2.4.6
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Eryu Guan]

On Thu, Jul 23, 2015 at 01:51:50PM -0700, Omar Sandoval wrote:
> Replacing and scrubbing RAID 5/6 is now supported on Btrfs. Enable it in
> _btrfs_get_profile_configs while making it more generic to also support
> replace missing.
> 
> Signed-off-by: Omar Sandoval <osandov@fb.com>

Looks great! Tested with default configs and user-defined configs.

Reviewed-by: Eryu Guan <eguan@redhat.com>

> ---
>  common/rc | 96 ++++++++++++++++++++++++++++++++-------------------------------
>  1 file changed, 49 insertions(+), 47 deletions(-)
> 
> diff --git a/common/rc b/common/rc
> index 610045eab304..3e6fdb6ebcfa 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -2748,60 +2748,62 @@ _btrfs_get_profile_configs()
>  		return
>  	fi
>  
> -	# no user specified btrfs profile configs, export the default configs
>  	if [ -z "$BTRFS_PROFILE_CONFIGS" ]; then
> -		# default configs
> -		_btrfs_profile_configs=(
> -			"-m single -d single"
> -			"-m dup -d single"
> -			"-m raid0 -d raid0"
> -			"-m raid1 -d raid0"
> -			"-m raid1 -d raid1"
> -			"-m raid10 -d raid10"
> -			"-m raid5 -d raid5"
> -			"-m raid6 -d raid6"
> +		# Default configurations to test.
> +		local configs=(
> +			"single:single"
> +			"dup:single"
> +			"raid0:raid0"
> +			"raid1:raid0"
> +			"raid1:raid1"
> +			"raid10:raid10"
> +			"raid5:raid5"
> +			"raid6:raid6"
>  		)
> +	else
> +		# User-provided configurations.
> +		local configs=(${BTRFS_PROFILE_CONFIGS[@]})
> +	fi
>  
> -		# remove dup/raid5/raid6 profiles if we're doing device replace
> -		# dup profile indicates only one device being used (SCRATCH_DEV),
> -		# but we don't want to replace SCRATCH_DEV, which will be used in
> -		# _scratch_mount/_check_scratch_fs etc.
> -		# and raid5/raid6 doesn't support replace yet
> +	_btrfs_profile_configs=()
> +	for cfg in "${configs[@]}"; do
> +		local supported=true
> +		local profiles=(${cfg/:/ })
>  		if [ "$1" == "replace" ]; then
> -			_btrfs_profile_configs=(
> -				"-m single -d single"
> -				"-m raid0 -d raid0"
> -				"-m raid1 -d raid0"
> -				"-m raid1 -d raid1"
> -				"-m raid10 -d raid10"
> -				# add these back when raid5/6 is working with replace
> -				#"-m raid5 -d raid5"
> -				#"-m raid6 -d raid6"
> +			# We can't do replace with these profiles because they
> +			# imply only one device ($SCRATCH_DEV), and we need to
> +			# keep $SCRATCH_DEV around for _scratch_mount
> +			# and _check_scratch_fs.
> +			local unsupported=(
> +				"single"
> +				"dup"
>  			)
> +		elif [ "$1" == "replace-missing" ]; then
> +			# We can't replace missing devices with these profiles
> +			# because there isn't enough redundancy.
> +			local unsupported=(
> +				"single"
> +				"dup"
> +				"raid0"
> +			)
> +		else
> +			local unsupported=()
>  		fi
> -		export _btrfs_profile_configs
> -		return
> -	fi
> -
> -	# parse user specified btrfs profile configs
> -	local i=0
> -	local cfg=""
> -	for cfg in $BTRFS_PROFILE_CONFIGS; do
> -		# turn "metadata:data" format to "-m metadata -d data"
> -		# and assign it to _btrfs_profile_configs array
> -		cfg=`echo "$cfg" | sed -e 's/^/-m /' -e 's/:/ -d /'`
> -		_btrfs_profile_configs[$i]="$cfg"
> -		let i=i+1
> -	done
> -
> -	if [ "$1" == "replace" ]; then
> -		if echo ${_btrfs_profile_configs[*]} | grep -q raid[56]; then
> -			_notrun "RAID5/6 doesn't support btrfs device replace yet"
> -		fi
> -		if echo ${_btrfs_profile_configs[*]} | grep -q dup; then
> -			_notrun "Do not set dup profile in btrfs device replace test"
> +		for unsupp in "${unsupported[@]}"; do
> +			if [ "${profiles[0]}" == "$unsupp" -o "${profiles[1]}" == "$unsupp" ]; then
> +			     if [ -z "$BTRFS_PROFILE_CONFIGS" ]; then
> +				     # For the default config, just omit it.
> +				     supported=false
> +			     else
> +				     # For user-provided config, don't run the test.
> +				     _notrun "Profile $unsupp not supported for $1"
> +			     fi
> +			fi
> +		done
> +		if "$supported"; then
> +			_btrfs_profile_configs+=("-m ${profiles[0]} -d ${profiles[1]}")
>  		fi
> -	fi
> +	done
>  	export _btrfs_profile_configs
>  }
>  
> -- 
> 2.4.6
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/3] btrfs: add a test of replace missing dev in diff raid

[Eryu Guan]

On Thu, Jul 23, 2015 at 01:51:51PM -0700, Omar Sandoval wrote:
> From: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
> 
> Test of missing device replace in different raid modes.  This
> test requires SCRATCH_DEV_POOL contain 5 same size devices.
> 
> This issue has been fixed by Omar's patch:
> 	Btrfs: RAID 5/6 missing device scrub+replace
> 
> Signed-off-by: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
>  tests/btrfs/095     | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/095.out |  2 ++
>  tests/btrfs/group   |  1 +
>  3 files changed, 96 insertions(+)
>  create mode 100755 tests/btrfs/095
>  create mode 100644 tests/btrfs/095.out
> 
> diff --git a/tests/btrfs/095 b/tests/btrfs/095
> new file mode 100755
> index 000000000000..2dfb919fe4f6
> --- /dev/null
> +++ b/tests/btrfs/095
> @@ -0,0 +1,93 @@
> +#! /bin/bash
> +# FS QA Test No. btrfs/095
> +#
> +# Test of missing device replace in different raid mode
> +#
> +# Be sure $SCRATCH_DEV_POOL including 5 devices. And all
> +# devices in pool must be in the same size.
> +#
> +# To check the fs after replacing a dev, a scrub run is performed.
> +#
> +# This issue has been fixed by Omar Sandoval's patch:
> +# 	Btrfs: RAID 5/6 missing device scrub+replace
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2015 Fujitsu.  All Rights Reserved.
> +# Author: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +status=1
> +trap "exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_need_to_be_root
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_scratch_dev_pool 5

Need a call of _require_scratch_dev_pool_equal_size here

> +_btrfs_get_profile_configs replace-missing
> +
> +rm -f $seqres.full
> +
> +REPLACE_DEV=`echo $SCRATCH_DEV_POOL | awk '{print $5}'`
> +export SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | awk '{print $1,$2,$3,$4}'`
> +
> +run_test () {
> +	local mkfs_opts=$1

echo the $mkfs_opts to $seqres.full is useful for debug purpose, so if
some profile config fails we know what config is failing.

echo "Test $mkfs_opts" >>$seqres.full

> +	_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1 || _fail "mkfs failed"

Continue to test next profile config if mkfs failed, _fail breaks out
the whole test.

> +	_scratch_mount
> +
> +	PRUNE_DEV=`echo $SCRATCH_DEV_POOL | awk '{print $2}'`
> +	PRUNE_DEV_ID=`$BTRFS_UTIL_PROG fi show $SCRATCH_MNT | grep $PRUNE_DEV | awk '{print $2}'`
> +
> +	# dd some data
> +	dd if=/dev/urandom of="$SCRATCH_MNT"/file1 bs=1M count=1 \
> +		>>$seqres.full 2>&1 || _fail "dd failed"
> +	dd if=/dev/urandom of="$SCRATCH_MNT"/file2 bs=1M count=2 \
> +		>>$seqres.full 2>&1 || _fail "dd failed"
> +	dd if=/dev/urandom of="$SCRATCH_MNT"/file3 bs=1M count=4 \
> +		>>$seqres.full 2>&1 || _fail "dd failed"

I don't think the _fail here is necessary either.

> +
> +	# prune the device PRUNE_DEV && remount by degraded mode
> +	umount $SCRATCH_MNT

_scratch_unmount

> +	dd if=/dev/zero of=$PRUNE_DEV bs=1M count=1 >>$seqres.full 2>&1 \
> +		|| _fail "dd failed"

$WIPEFS_PROG -a $PRUNE_DEV ?

> +	mount -o degraded $SCRATCH_DEV $SCRATCH_MNT

_scratch_mount "-o degraded"

> +
> +	# replace the missing dev $PRUNE_DEV with $REPLACE_DEV and scrub it
> +	$BTRFS_UTIL_PROG replace start -B -r $PRUNE_DEV_ID $REPLACE_DEV \
> +		$SCRATCH_MNT -f >>$seqres.full 2>&1 || _fail "replace failed"

Don't _fail here, just echo error messages to break golden image. Better
to append the error message to $seqres.full too, for debug purpose.

> +	$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 \
> +		|| _fail "scrub failed"

Here too.

> +
> +	umount $SCRATCH_MNT

_scratch_unmount

> +}
> +
> +echo "Silence is golden"
> +for t in "${_btrfs_profile_configs[@]}"; do
> +	run_test "$t"
> +done
> +
> +status=0
> +exit
> diff --git a/tests/btrfs/095.out b/tests/btrfs/095.out
> new file mode 100644
> index 000000000000..80ad3b9c2a7b
> --- /dev/null
> +++ b/tests/btrfs/095.out
> @@ -0,0 +1,2 @@
> +QA output created by 095
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index ffe18bff0d21..f20a191c24be 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -96,3 +96,4 @@
>  092 auto quick send
>  093 auto quick clone
>  094 auto quick send
> +095 auto

Should also be in "replace" group.

Thanks,
Eryu
> -- 
> 2.4.6
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[David Sterba]

On Thu, Jul 23, 2015 at 01:51:50PM -0700, Omar Sandoval wrote:
> +			# We can't do replace with these profiles because they
> +			# imply only one device ($SCRATCH_DEV), and we need to
> +			# keep $SCRATCH_DEV around for _scratch_mount
> +			# and _check_scratch_fs.
> +			local unsupported=(
> +				"single"
> +				"dup"

DUP does imply single device, but why does 'single' ?

Re: [PATCH 1/3] btrfs/011: test replace on RAID 5/6 now that it's supported

[David Sterba]

On Thu, Jul 23, 2015 at 01:51:49PM -0700, Omar Sandoval wrote:
> btrfs replace has been supported on RAID 5/6 since Linux 3.19.
> 
> Signed-off-by: Omar Sandoval <osandov@fb.com>
Reviewed-by: David Sterba <dsterba@suse.com>

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Omar Sandoval]

On Fri, Jul 24, 2015 at 02:09:46PM +0200, David Sterba wrote:
> On Thu, Jul 23, 2015 at 01:51:50PM -0700, Omar Sandoval wrote:
> > +			# We can't do replace with these profiles because they
> > +			# imply only one device ($SCRATCH_DEV), and we need to
> > +			# keep $SCRATCH_DEV around for _scratch_mount
> > +			# and _check_scratch_fs.
> > +			local unsupported=(
> > +				"single"
> > +				"dup"
> 
> DUP does imply single device, but why does 'single' ?

It does not, I apparently forgot that you could use single to
concatenate multiple devices. I'll fix that in v2.

Thanks for reviewing!
-- 
Omar

Re: [PATCH 3/3] btrfs: add a test of replace missing dev in diff raid

[Omar Sandoval]

On Fri, Jul 24, 2015 at 11:28:32AM +0800, Eryu Guan wrote:
> On Thu, Jul 23, 2015 at 01:51:51PM -0700, Omar Sandoval wrote:
> > From: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
> > 
> > Test of missing device replace in different raid modes.  This
> > test requires SCRATCH_DEV_POOL contain 5 same size devices.
> > 
> > This issue has been fixed by Omar's patch:
> > 	Btrfs: RAID 5/6 missing device scrub+replace
> > 
> > Signed-off-by: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
> > Signed-off-by: Omar Sandoval <osandov@fb.com>
> > ---
> >  tests/btrfs/095     | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/btrfs/095.out |  2 ++
> >  tests/btrfs/group   |  1 +
> >  3 files changed, 96 insertions(+)
> >  create mode 100755 tests/btrfs/095
> >  create mode 100644 tests/btrfs/095.out
> > 
> > diff --git a/tests/btrfs/095 b/tests/btrfs/095
> > new file mode 100755
> > index 000000000000..2dfb919fe4f6
> > --- /dev/null
> > +++ b/tests/btrfs/095
> > @@ -0,0 +1,93 @@
> > +#! /bin/bash
> > +# FS QA Test No. btrfs/095
> > +#
> > +# Test of missing device replace in different raid mode
> > +#
> > +# Be sure $SCRATCH_DEV_POOL including 5 devices. And all
> > +# devices in pool must be in the same size.
> > +#
> > +# To check the fs after replacing a dev, a scrub run is performed.
> > +#
> > +# This issue has been fixed by Omar Sandoval's patch:
> > +# 	Btrfs: RAID 5/6 missing device scrub+replace
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2015 Fujitsu.  All Rights Reserved.
> > +# Author: Wang Yanfeng <wangyf-fnst@cn.fujitsu.com>
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +status=1
> > +trap "exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# real QA test starts here
> > +_need_to_be_root
> > +_supported_fs btrfs
> > +_supported_os Linux
> > +_require_scratch
> > +_require_scratch_dev_pool 5
> 
> Need a call of _require_scratch_dev_pool_equal_size here
> 
> > +_btrfs_get_profile_configs replace-missing
> > +
> > +rm -f $seqres.full
> > +
> > +REPLACE_DEV=`echo $SCRATCH_DEV_POOL | awk '{print $5}'`
> > +export SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | awk '{print $1,$2,$3,$4}'`
> > +
> > +run_test () {
> > +	local mkfs_opts=$1
> 
> echo the $mkfs_opts to $seqres.full is useful for debug purpose, so if
> some profile config fails we know what config is failing.
> 
> echo "Test $mkfs_opts" >>$seqres.full
> 
> > +	_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1 || _fail "mkfs failed"
> 
> Continue to test next profile config if mkfs failed, _fail breaks out
> the whole test.
> 
> > +	_scratch_mount
> > +
> > +	PRUNE_DEV=`echo $SCRATCH_DEV_POOL | awk '{print $2}'`
> > +	PRUNE_DEV_ID=`$BTRFS_UTIL_PROG fi show $SCRATCH_MNT | grep $PRUNE_DEV | awk '{print $2}'`
> > +
> > +	# dd some data
> > +	dd if=/dev/urandom of="$SCRATCH_MNT"/file1 bs=1M count=1 \
> > +		>>$seqres.full 2>&1 || _fail "dd failed"
> > +	dd if=/dev/urandom of="$SCRATCH_MNT"/file2 bs=1M count=2 \
> > +		>>$seqres.full 2>&1 || _fail "dd failed"
> > +	dd if=/dev/urandom of="$SCRATCH_MNT"/file3 bs=1M count=4 \
> > +		>>$seqres.full 2>&1 || _fail "dd failed"
> 
> I don't think the _fail here is necessary either.
> 
> > +
> > +	# prune the device PRUNE_DEV && remount by degraded mode
> > +	umount $SCRATCH_MNT
> 
> _scratch_unmount
> 
> > +	dd if=/dev/zero of=$PRUNE_DEV bs=1M count=1 >>$seqres.full 2>&1 \
> > +		|| _fail "dd failed"
> 
> $WIPEFS_PROG -a $PRUNE_DEV ?
> 
> > +	mount -o degraded $SCRATCH_DEV $SCRATCH_MNT
> 
> _scratch_mount "-o degraded"
> 
> > +
> > +	# replace the missing dev $PRUNE_DEV with $REPLACE_DEV and scrub it
> > +	$BTRFS_UTIL_PROG replace start -B -r $PRUNE_DEV_ID $REPLACE_DEV \
> > +		$SCRATCH_MNT -f >>$seqres.full 2>&1 || _fail "replace failed"
> 
> Don't _fail here, just echo error messages to break golden image. Better
> to append the error message to $seqres.full too, for debug purpose.
> 
> > +	$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >>$seqres.full 2>&1 \
> > +		|| _fail "scrub failed"
> 
> Here too.
> 
> > +
> > +	umount $SCRATCH_MNT
> 
> _scratch_unmount
> 
> > +}
> > +
> > +echo "Silence is golden"
> > +for t in "${_btrfs_profile_configs[@]}"; do
> > +	run_test "$t"
> > +done
> > +
> > +status=0
> > +exit
> > diff --git a/tests/btrfs/095.out b/tests/btrfs/095.out
> > new file mode 100644
> > index 000000000000..80ad3b9c2a7b
> > --- /dev/null
> > +++ b/tests/btrfs/095.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 095
> > +Silence is golden
> > diff --git a/tests/btrfs/group b/tests/btrfs/group
> > index ffe18bff0d21..f20a191c24be 100644
> > --- a/tests/btrfs/group
> > +++ b/tests/btrfs/group
> > @@ -96,3 +96,4 @@
> >  092 auto quick send
> >  093 auto quick clone
> >  094 auto quick send
> > +095 auto
> 
> Should also be in "replace" group.
> 
> Thanks,
> Eryu

Thanks for reviewing, I'll fix this all up in a v2.

-- 
Omar

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Brendan Hide]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2015/07/24 07:50 PM, Omar Sandoval wrote:
> On Fri, Jul 24, 2015 at 02:09:46PM +0200, David Sterba wrote:
>> On Thu, Jul 23, 2015 at 01:51:50PM -0700, Omar Sandoval wrote:
>>> +			# We can't do replace with these profiles because they +
>>> # imply only one device ($SCRATCH_DEV), and we need to +			#
>>> keep $SCRATCH_DEV around for _scratch_mount +			# and
>>> _check_scratch_fs. +			local unsupported=( +				"single" +
>>> "dup"
>> 
>> DUP does imply single device, but why does 'single' ?
> 
> It does not, I apparently forgot that you could use single to 
> concatenate multiple devices. I'll fix that in v2.
> 
> Thanks for reviewing!
> 
Late to the party. DUP *implies* single device but there are cases
where dup is used on a multi-device fs. Even if the use-cases aren't
good or intended to be long-term, they are still valid, right?


- -- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJVtq9AAAoJEE+uni74c4qN5eYIAJAGznsi3RD1tchbSLwhMXJk
bJJ4ORB9taLXHykSfYTsHIaUoVpcVR6tT/I1jz5070DY3mKkQ16a8nwtSxPba4Lv
QiS8YRegFiHMYzZbH1T7Tnm6R9g/RZsaU7GS3JhP9HUYG7hIWGRRuoiOjYn/hoLw
uMXuIFOkPKGYDgyAhDIp3KDYlBjMHT6Oun7CcpvTjXiOnzJFFp3MSt3b6mmmdMVV
YKWpWyKVh7qlENEoqKb4exqr1WGYKU+kBLXRs4wdm3xb66EcWYs0Er1u6v+K1trx
nryFrfUxYtMJsSuR9ZJm88DOsXKAuX1LEdRKVOlq7krsIK8HlTizccMXAl10gKk=
=ndkL
-----END PGP SIGNATURE-----

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

On Tue, Jul 28, 2015 at 12:22:56AM +0200, Brendan Hide wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 2015/07/24 07:50 PM, Omar Sandoval wrote:
> > On Fri, Jul 24, 2015 at 02:09:46PM +0200, David Sterba wrote:
> >> On Thu, Jul 23, 2015 at 01:51:50PM -0700, Omar Sandoval wrote:
> >>> +			# We can't do replace with these profiles because they +
> >>> # imply only one device ($SCRATCH_DEV), and we need to +			#
> >>> keep $SCRATCH_DEV around for _scratch_mount +			# and
> >>> _check_scratch_fs. +			local unsupported=( +				"single" +
> >>> "dup"
> >> 
> >> DUP does imply single device, but why does 'single' ?
> > 
> > It does not, I apparently forgot that you could use single to 
> > concatenate multiple devices. I'll fix that in v2.
> > 
> > Thanks for reviewing!
> > 
> Late to the party. DUP *implies* single device but there are cases
> where dup is used on a multi-device fs. Even if the use-cases aren't
> good or intended to be long-term, they are still valid, right?

   Yes, but bear in mind that any new (presumably metadata) chunks
allocated on the multi-device FS will end up as RAID-1, because of the
automatic upgrade from DUP to RAID-1. Any balance will also do the
upgrade.

   Hugo.

PS. Can we get rid of that upgrade? It's a pain the arse in so many
ways.

> - -- 
> __________
> Brendan Hide
> http://swiftspirit.co.za/
> http://www.webafrica.co.za/?AFF1E97
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (MingW32)
> 
> iQEcBAEBAgAGBQJVtq9AAAoJEE+uni74c4qN5eYIAJAGznsi3RD1tchbSLwhMXJk
> bJJ4ORB9taLXHykSfYTsHIaUoVpcVR6tT/I1jz5070DY3mKkQ16a8nwtSxPba4Lv
> QiS8YRegFiHMYzZbH1T7Tnm6R9g/RZsaU7GS3JhP9HUYG7hIWGRRuoiOjYn/hoLw
> uMXuIFOkPKGYDgyAhDIp3KDYlBjMHT6Oun7CcpvTjXiOnzJFFp3MSt3b6mmmdMVV
> YKWpWyKVh7qlENEoqKb4exqr1WGYKU+kBLXRs4wdm3xb66EcWYs0Er1u6v+K1trx
> nryFrfUxYtMJsSuR9ZJm88DOsXKAuX1LEdRKVOlq7krsIK8HlTizccMXAl10gKk=
> =ndkL
> -----END PGP SIGNATURE-----

-- 
Hugo Mills             | I'm on a 30-day diet. So far I've lost 18 days.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[David Sterba]

On Tue, Jul 28, 2015 at 12:22:56AM +0200, Brendan Hide wrote:
> > It does not, I apparently forgot that you could use single to 
> > concatenate multiple devices. I'll fix that in v2.
> > 
> > Thanks for reviewing!
> > 
> Late to the party. DUP *implies* single device but there are cases
> where dup is used on a multi-device fs. Even if the use-cases aren't
> good or intended to be long-term, they are still valid, right?

You're right, DUP is reported by 'fi df' after 2nd device is added and
this state (even if it's temporary) has to be taken into account.

Re: [PATCH 2/3] btrfs: add replace missing and replace RAID 5/6 to profile configs

[Omar Sandoval]

On Tue, Jul 28, 2015 at 06:52:07PM +0200, David Sterba wrote:
> On Tue, Jul 28, 2015 at 12:22:56AM +0200, Brendan Hide wrote:
> > > It does not, I apparently forgot that you could use single to 
> > > concatenate multiple devices. I'll fix that in v2.
> > > 
> > > Thanks for reviewing!
> > > 
> > Late to the party. DUP *implies* single device but there are cases
> > where dup is used on a multi-device fs. Even if the use-cases aren't
> > good or intended to be long-term, they are still valid, right?
> 
> You're right, DUP is reported by 'fi df' after 2nd device is added and
> this state (even if it's temporary) has to be taken into account.

I wouldn't read too much into this, _btrfs_get_profile_configs is just
for tests that mkfs a bunch of new filesystems for testing, and in that
case, DUP is invalid for multiple devices. It's not meant to be an
authoritative source of truth regarding what metadata/data profiles
could be legal.

Thanks,
-- 
Omar