[PATCH 0/5] New btrfsck test framework

[PATCH 0/5] New btrfsck test framework

[Qu Wenruo]

This patchset introduce the new btrfsck test framework.
The main changes are:
1) File layout change
The new file layout is based on dir.
Every corruption type has its own dir, and it can contain several
different images for different cases, like bad root items type, it has
the default case and the skinny metadata case.

2) Image format
The framework support 2 image format.
One is the original btrfs-image dump.
The other one is custom script.

Any other type will need a small script to extract the image and use the
framework routine or other codes to check it.

3) New test case for leaf-corruption recovery
Added a new type dir for leaf-corruption, and add verification codes to
check the recovered data.

Minor changes include:
1) fsck-test.sh can be called from anywhere
Now fsck-test.sh can determine the btrfs-progs top dir according to its
path, no need to be called from btrfs-progs dir.

2) default value and auto-detect for environment variant
Now fsck-test.sh can accept environment variants, no need to edit the
script anymore.
And if TEST_MNT is not set, it will use <btrfs-progs>/tests/mnt as
default mount point.

3) better prompt for needed environment variant/privilege
When some case is not run due to missing environment variant or
privilege, fsck-test.sh will prompt it.

Qu Wenruo (5):
  btrfs-progs: New btrfsck test infrastructure
  btrfs-progs: Move btrfs-image dump to corresponding dir
  btrfs-progs: Move bad root items test cases to its corresponding dir
  btrfs-progs: Move leaf-corruption no extent data case and add    
    verification script
  btrfs-progs: Move extent tree rebuild test to its dir

 tests/common                                       |  49 +++++++++
 tests/fsck-tests.sh                                | 122 +++++++--------------
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.tar.xz}                           | Bin
 .../skinny_case.tar.xz}                            | Bin
 tests/fsck-tests/006-bad-root-items/test.sh        |  15 +++
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../default_case.img}                              | Bin
 .../no_data_extent.tar.xz}                         | Bin
 tests/fsck-tests/012-leaf-corruption/test.sh       | 119 ++++++++++++++++++++
 tests/fsck-tests/013-extent-tree-rebuild/test.sh   |  47 ++++++++
 18 files changed, 272 insertions(+), 80 deletions(-)
 create mode 100644 tests/common
 mode change 100644 => 100755 tests/fsck-tests.sh
 rename tests/fsck-tests/{001-bad-file-extent-bytenr.img => 001-bad-file-extent-bytenr/default_case.img} (100%)
 rename tests/fsck-tests/{002-bad-transid.img => 002-bad-transid/default_case.img} (100%)
 rename tests/fsck-tests/{003-shift-offsets.img => 003-shift-offsets/default_case.img} (100%)
 rename tests/fsck-tests/{004-no-dir-index.img => 004-no-dir-index/default_case.img} (100%)
 rename tests/fsck-tests/{005-bad-item-offset.img => 005-bad-item-offset/default_case.img} (100%)
 rename tests/fsck-tests/{006-bad_root_items_fs.tar.xz => 006-bad-root-items/default_case.tar.xz} (100%)
 rename tests/fsck-tests/{007-bad_root_items_fs_skinny.tar.xz => 006-bad-root-items/skinny_case.tar.xz} (100%)
 create mode 100755 tests/fsck-tests/006-bad-root-items/test.sh
 rename tests/fsck-tests/{008-bad-offset-snapshots.img => 007-bad-offset-snapshots/default_case.img} (100%)
 rename tests/fsck-tests/{009-bad-dir-index-name.img => 008-bad-dir-index-name/default_case.img} (100%)
 rename tests/fsck-tests/{010-no-dir-item-or-index.img => 009-no-dir-item-or-index/default_case.img} (100%)
 rename tests/fsck-tests/{011-no-rootdir-inode-item.img => 010-no-rootdir-inode-item/default_case.img} (100%)
 rename tests/fsck-tests/{012-no-inode-item.img => 011-no-inode-item/default_case.img} (100%)
 rename tests/fsck-tests/{013-leaf-corruption-no-extent-data.tar.xz => 012-leaf-corruption/no_data_extent.tar.xz} (100%)
 create mode 100755 tests/fsck-tests/012-leaf-corruption/test.sh
 create mode 100755 tests/fsck-tests/013-extent-tree-rebuild/test.sh

-- 
2.2.1

[PATCH 1/5] btrfs-progs: New btrfsck test infrastructure

[Qu Wenruo]

Change the old btrfsck test infrastructure (btrfs-image dump or xz raw
dump) to the new test infrastructure.

1) Test case layout
The new infrastructure is dir based, each dir is one test type, and can
contain multiple images/scripts for different corner cases.
So layout will be the following:
btrfs-progs
	|-tests
	    |-fsck-tests
		|-001-SOME-CORRUPT-TYPE
			|-IMAGE-FOR-CASE1
			|-IMAGE-FOR-CASE2

2) Test case image types
Only 2 types for test case images.
a) btrfs-image dump
   This one is the simplest case, one only needs to add the image to
   corresponding dir.

b) custom script
   This one is for all the resting cases which can't fit btrfs-image,
   like csum error or script can generate the image (this reduces the
   size obviously and good for review)
   The old binary dump also belongs to this type, so need to add script
   to extract them.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/common        | 49 ++++++++++++++++++++++++++
 tests/fsck-tests.sh | 99 +++++++++++++++++++++++++++--------------------------
 2 files changed, 99 insertions(+), 49 deletions(-)
 create mode 100644 tests/common
 mode change 100644 => 100755 tests/fsck-tests.sh

diff --git a/tests/common b/tests/common
new file mode 100644
index 0000000..80a5381
--- /dev/null
+++ b/tests/common
@@ -0,0 +1,49 @@
+#!/bin/bash
+#
+# Common routines for all tests
+#
+
+_fail()
+{
+	echo "$*" | tee -a $RESULT
+	exit 1
+}
+
+run_check()
+{
+	echo "############### $@" >> $RESULT 2>&1
+	"$@" >> $RESULT 2>&1 || _fail "failed: $@"
+}
+
+check_prereq()
+{
+	if ! [ -f $top/$1 ]; then
+		_fail "Failed prerequisities: $1";
+	fi
+}
+
+check_image()
+{
+	image=$1
+	echo "testing image $(basename $image)" >> $RESULT
+	$top/btrfs check $image >> $RESULT 2>&1
+	[ $? -eq 0 ] && _fail "btrfs check should have detected corruption"
+
+	run_check $top/btrfs check --repair $image
+	run_check $top/btrfs check $image
+}
+
+check_all_images()
+{
+	dir=$1
+	for i in $(find $dir -iname '*.img')
+	do
+		echo "extracting image $(basename $i)" >> $RESULT
+		$top/btrfs-image -r $i $i.restored || \
+			_fail "failed to extract image $i"
+
+		check_image $i.restored
+
+		rm $i.restored
+	done
+}
diff --git a/tests/fsck-tests.sh b/tests/fsck-tests.sh
old mode 100644
new mode 100755
index 24fb109..a992b45
--- a/tests/fsck-tests.sh
+++ b/tests/fsck-tests.sh
@@ -5,60 +5,60 @@
 # It's GPL, same as everything else in this tree.
 #
 
-here=`pwd`
-TEST_DEV=
-TEST_MNT=
-RESULT="fsck-tests-results.txt"
-
-_fail()
-{
-	echo "$*" | tee -a $RESULT
-	exit 1
-}
-
-run_check()
-{
-	echo "############### $@" >> $RESULT 2>&1
-	"$@" >> $RESULT 2>&1 || _fail "failed: $@"
-}
-
-check_prereq()
-{
-	if ! [ -f $here/$1 ]; then
-		_fail "Failed prerequisities: $1";
-	fi
-}
+unset top
+unset LANG
+LANG=C
+script_dir=$(dirname $(realpath $0))
+top=$(realpath $script_dir/../)
+TEST_DEV=${TEST_DEV:-}
+TEST_MNT=${TEST_MNT:-$top/tests/mnt}
+RESULT="$top/tests/fsck-tests-results.txt"
+
+source $top/tests/common
+
+# Allow child test to use $top and $RESULT
+export top
+export RESULT
+# For custom script needs to verfiy recovery
+export TEST_MNT
+export LANG
 
 rm -f $RESULT
+mkdir -p $TEST_MNT || _fail "unable to create mount point on $TEST_MNT"
 
 # test rely on corrupting blocks tool
 check_prereq btrfs-corrupt-block
 check_prereq btrfs-image
 check_prereq btrfs
 
-# Some broken filesystem images are kept as .img files, created by the tool
-# btrfs-image, and others are kept as .tar.xz files that contain raw filesystem
-# image (the backing file of a loop device, as a sparse file). The reason for
-# keeping some as tarballs of raw images is that for these cases btrfs-image
-# isn't able to preserve all the (bad) filesystem structure for some reason.
-for i in $(find $here/tests/fsck-tests -name '*.img' -o -name '*.tar.xz' | sort)
+# Each dir contains one type of error for btrfsck test.
+# Each dir must be one of the following 2 types:
+# 1) Only btrfs-image dump
+#    Only contains one or several btrfs-image dumps (.img)
+#    Each image will be tested by generic test routine
+#    (btrfsck --repair and btrfsck).
+#    This is for case that btree-healthy images.
+# 2) Custom test script
+#    This dir contains test.sh which will do custom image
+#    generation/check/verification.
+#    This is for case btrfs-image can't dump or case needs extra
+#    check/verify
+
+for i in $(find $top/tests/fsck-tests -maxdepth 1 -mindepth 1 -type d | sort)
 do
 	echo "     [TEST]    $(basename $i)"
-	echo "testing image $i" >> $RESULT
-
-	extension=${i#*.}
-
-	if [ $extension == "img" ]; then
-		run_check $here/btrfs-image -r $i test.img
+	cd $i
+	if [ -x test.sh ]; then
+		# Type 2
+		./test.sh
+		if [ $? -ne 0 ]; then
+			_fail "test failed for case $(basename $i)"
+		fi
 	else
-		run_check tar xJf $i
+		# Type 1
+		check_all_images `pwd`
 	fi
-
-	$here/btrfs check test.img >> $RESULT 2>&1
-	[ $? -eq 0 ] && _fail "btrfs check should have detected corruption"
-
-	run_check $here/btrfs check --repair test.img
-	run_check $here/btrfs check test.img
+	cd $top
 done
 
 if [ -z $TEST_DEV ] || [ -z $TEST_MNT ];then
@@ -70,31 +70,32 @@ fi
 test_extent_tree_rebuild()
 {
 	echo "     [TEST]    extent tree rebuild"
-	$here/mkfs.btrfs -f $TEST_DEV >> /dev/null 2>&1 || _fail "fail to mkfs"
+	$top/mkfs.btrfs -f $TEST_DEV >> /dev/null 2>&1 || _fail "fail to mkfs"
 
 	run_check mount $TEST_DEV $TEST_MNT
 	cp -aR /lib/modules/`uname -r`/ $TEST_MNT 2>&1
 
 	for i in `seq 1 100`;do
-		$here/btrfs sub snapshot $TEST_MNT \
+		$top/btrfs sub snapshot $TEST_MNT \
 			$TEST_MNT/snapaaaaaaa_$i >& /dev/null
 	done
 	run_check umount $TEST_DEV
 
 	# get extent root bytenr
-	extent_root_bytenr=`$here/btrfs-debug-tree -r $TEST_DEV | grep extent | awk '{print $7}'`
+	extent_root_bytenr=`$top/btrfs-debug-tree -r $TEST_DEV | \
+			    grep extent | awk '{print $7}'`
 	if [ -z $extent_root_bytenr ];then
 		_fail "fail to get extent root bytenr"
 	fi
 
 	# corrupt extent root node block
-	run_check $here/btrfs-corrupt-block -l $extent_root_bytenr \
+	run_check $top/btrfs-corrupt-block -l $extent_root_bytenr \
 		-b 4096 $TEST_DEV
 
-	$here/btrfs check $TEST_DEV >& /dev/null && \
+	$top/btrfs check $TEST_DEV >& /dev/null && \
 			_fail "btrfs check should detect failure"
-	run_check $here/btrfs check --init-extent-tree $TEST_DEV
-	run_check $here/btrfs check $TEST_DEV
+	run_check $top/btrfs check --init-extent-tree $TEST_DEV
+	run_check $top/btrfs check $TEST_DEV
 }
 
 test_extent_tree_rebuild
-- 
2.2.1

[PATCH 2/5] btrfs-progs: Move btrfs-image dump to corresponding dir

[Qu Wenruo]

Move these obvious btrfs-image to its corresponding dir to use the new
infrastructure.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 .../default_case.img}                                       | Bin
 10 files changed, 0 insertions(+), 0 deletions(-)
 rename tests/fsck-tests/{001-bad-file-extent-bytenr.img => 001-bad-file-extent-bytenr/default_case.img} (100%)
 rename tests/fsck-tests/{002-bad-transid.img => 002-bad-transid/default_case.img} (100%)
 rename tests/fsck-tests/{003-shift-offsets.img => 003-shift-offsets/default_case.img} (100%)
 rename tests/fsck-tests/{004-no-dir-index.img => 004-no-dir-index/default_case.img} (100%)
 rename tests/fsck-tests/{005-bad-item-offset.img => 005-bad-item-offset/default_case.img} (100%)
 rename tests/fsck-tests/{008-bad-offset-snapshots.img => 007-bad-offset-snapshots/default_case.img} (100%)
 rename tests/fsck-tests/{009-bad-dir-index-name.img => 008-bad-dir-index-name/default_case.img} (100%)
 rename tests/fsck-tests/{010-no-dir-item-or-index.img => 009-no-dir-item-or-index/default_case.img} (100%)
 rename tests/fsck-tests/{011-no-rootdir-inode-item.img => 010-no-rootdir-inode-item/default_case.img} (100%)
 rename tests/fsck-tests/{012-no-inode-item.img => 011-no-inode-item/default_case.img} (100%)

diff --git a/tests/fsck-tests/001-bad-file-extent-bytenr.img b/tests/fsck-tests/001-bad-file-extent-bytenr/default_case.img
similarity index 100%
rename from tests/fsck-tests/001-bad-file-extent-bytenr.img
rename to tests/fsck-tests/001-bad-file-extent-bytenr/default_case.img
diff --git a/tests/fsck-tests/002-bad-transid.img b/tests/fsck-tests/002-bad-transid/default_case.img
similarity index 100%
rename from tests/fsck-tests/002-bad-transid.img
rename to tests/fsck-tests/002-bad-transid/default_case.img
diff --git a/tests/fsck-tests/003-shift-offsets.img b/tests/fsck-tests/003-shift-offsets/default_case.img
similarity index 100%
rename from tests/fsck-tests/003-shift-offsets.img
rename to tests/fsck-tests/003-shift-offsets/default_case.img
diff --git a/tests/fsck-tests/004-no-dir-index.img b/tests/fsck-tests/004-no-dir-index/default_case.img
similarity index 100%
rename from tests/fsck-tests/004-no-dir-index.img
rename to tests/fsck-tests/004-no-dir-index/default_case.img
diff --git a/tests/fsck-tests/005-bad-item-offset.img b/tests/fsck-tests/005-bad-item-offset/default_case.img
similarity index 100%
rename from tests/fsck-tests/005-bad-item-offset.img
rename to tests/fsck-tests/005-bad-item-offset/default_case.img
diff --git a/tests/fsck-tests/008-bad-offset-snapshots.img b/tests/fsck-tests/007-bad-offset-snapshots/default_case.img
similarity index 100%
rename from tests/fsck-tests/008-bad-offset-snapshots.img
rename to tests/fsck-tests/007-bad-offset-snapshots/default_case.img
diff --git a/tests/fsck-tests/009-bad-dir-index-name.img b/tests/fsck-tests/008-bad-dir-index-name/default_case.img
similarity index 100%
rename from tests/fsck-tests/009-bad-dir-index-name.img
rename to tests/fsck-tests/008-bad-dir-index-name/default_case.img
diff --git a/tests/fsck-tests/010-no-dir-item-or-index.img b/tests/fsck-tests/009-no-dir-item-or-index/default_case.img
similarity index 100%
rename from tests/fsck-tests/010-no-dir-item-or-index.img
rename to tests/fsck-tests/009-no-dir-item-or-index/default_case.img
diff --git a/tests/fsck-tests/011-no-rootdir-inode-item.img b/tests/fsck-tests/010-no-rootdir-inode-item/default_case.img
similarity index 100%
rename from tests/fsck-tests/011-no-rootdir-inode-item.img
rename to tests/fsck-tests/010-no-rootdir-inode-item/default_case.img
diff --git a/tests/fsck-tests/012-no-inode-item.img b/tests/fsck-tests/011-no-inode-item/default_case.img
similarity index 100%
rename from tests/fsck-tests/012-no-inode-item.img
rename to tests/fsck-tests/011-no-inode-item/default_case.img
-- 
2.2.1

[PATCH 3/5] btrfs-progs: Move bad root items test cases to its corresponding dir

[Qu Wenruo]

Now 006-bad-root-items has two cases, one for default case, and one for
skinny metadata case.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 .../default_case.tar.xz}                                 | Bin
 .../skinny_case.tar.xz}                                  | Bin
 tests/fsck-tests/006-bad-root-items/test.sh              |  15 +++++++++++++++
 3 files changed, 15 insertions(+)
 rename tests/fsck-tests/{006-bad_root_items_fs.tar.xz => 006-bad-root-items/default_case.tar.xz} (100%)
 rename tests/fsck-tests/{007-bad_root_items_fs_skinny.tar.xz => 006-bad-root-items/skinny_case.tar.xz} (100%)
 create mode 100755 tests/fsck-tests/006-bad-root-items/test.sh

diff --git a/tests/fsck-tests/006-bad_root_items_fs.tar.xz b/tests/fsck-tests/006-bad-root-items/default_case.tar.xz
similarity index 100%
rename from tests/fsck-tests/006-bad_root_items_fs.tar.xz
rename to tests/fsck-tests/006-bad-root-items/default_case.tar.xz
diff --git a/tests/fsck-tests/007-bad_root_items_fs_skinny.tar.xz b/tests/fsck-tests/006-bad-root-items/skinny_case.tar.xz
similarity index 100%
rename from tests/fsck-tests/007-bad_root_items_fs_skinny.tar.xz
rename to tests/fsck-tests/006-bad-root-items/skinny_case.tar.xz
diff --git a/tests/fsck-tests/006-bad-root-items/test.sh b/tests/fsck-tests/006-bad-root-items/test.sh
new file mode 100755
index 0000000..2c0e1c6
--- /dev/null
+++ b/tests/fsck-tests/006-bad-root-items/test.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+source $top/tests/common
+
+echo "extracting image default_case.tar.xz" >> $RESULT
+tar xJf default_case.tar.xz || \
+	_fail "failed to extract default_case.tar.xz"
+check_image test.img
+
+echo "extracting image skinny_case.tar.xz" >> $RESULT
+tar xJf skinny_case.tar.xz || \
+	_fail "failed to extract skinny_case.tar.xz"
+check_image test.img
+
+rm test.img
-- 
2.2.1

[PATCH 4/5] btrfs-progs: Move leaf-corruption no extent data case and add verification script

[Qu Wenruo]

Move leaf-corruption type no extent data case to its dir, and add
verification script in test.sh

The verification script is based on manual btrfs-debug-tree check.

The image can also be reused to other leaf-corruption type, like
corrupted leaf contains regular file extent data case.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 .../no_data_extent.tar.xz}                         | Bin
 tests/fsck-tests/012-leaf-corruption/test.sh       | 119 +++++++++++++++++++++
 2 files changed, 119 insertions(+)
 rename tests/fsck-tests/{013-leaf-corruption-no-extent-data.tar.xz => 012-leaf-corruption/no_data_extent.tar.xz} (100%)
 create mode 100755 tests/fsck-tests/012-leaf-corruption/test.sh

diff --git a/tests/fsck-tests/013-leaf-corruption-no-extent-data.tar.xz b/tests/fsck-tests/012-leaf-corruption/no_data_extent.tar.xz
similarity index 100%
rename from tests/fsck-tests/013-leaf-corruption-no-extent-data.tar.xz
rename to tests/fsck-tests/012-leaf-corruption/no_data_extent.tar.xz
diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh b/tests/fsck-tests/012-leaf-corruption/test.sh
new file mode 100755
index 0000000..edf7719
--- /dev/null
+++ b/tests/fsck-tests/012-leaf-corruption/test.sh
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+source $top/tests/common
+
+# Check file list for leaf corruption, no regular/preallocated
+# file extent case.
+# Corrupted leaf is 20832256, which contains inode 1862~1872
+#
+# 1862, ref from leaf 20828160 key 24(DIR_ITEM)
+# 1863, ref from leaf 605388 item key 11(DIR_ITEM)
+# 1864, no ref to rebuild, no need to rebuild
+# 1865, ref from leaf 19767296 key 23(DIR_ITEM)
+# 1866-1868 no ref to rebuild, all refs in corrupted leaf
+# 1869, ref from leaf 4976640 key 22(DIR_ITEM)
+# 1870 no ref to rebuild, all refs in corrupted leaf
+# 1871, ref from leaf 19746816 key 38(DIR_ITEM)
+# 1872, ref from leaf 19767296 key 14(DIR_ITEM)
+# The list format is:
+# INO SIZE MODE NAME
+# INO: inode number
+# SIZE: file size, only checked for regular file
+# MODE: raw file mode, get from stat
+# NAME: file name
+leaf_no_data_ext_list=(
+	1862 0 40700 "install.d"
+	1862 0 40700 "install.d"
+	1863 0 40700 "gdb"
+	1865 0 40700 "iptables"
+	1869 0 40700 "snmp"
+	1871 0 100700 "machine-id"
+	1872 0 100700 "adjtime"
+)
+
+generate_leaf_corrupt_no_data_ext()
+{
+	dest=$1
+	echo "generating leaf_corrupt_no_data_ext.btrfs-image" >> $RESULT
+	tar xJf ./no_data_extent.tar.xz || \
+		_fail "failed to extract leaf_corrupt_no_data_ext.btrfs-image"
+	btrfs-image -r test.img.btrfs-image $dest || \
+		_fail "failed to extract leaf_corrupt_no_data_ext.btrfs-image"
+
+	# leaf at 20832256 contains no regular data extent, clear its csum to
+	# corrupt the leaf.
+	dd if=/dev/zero of=$dest bs=1 count=32 conv=notrunc seek=20832256 \
+		1>/dev/null 2>&1
+}
+
+check_inode()
+{
+	path=$1
+	ino=$2
+	size=$3
+	mode=$4
+	name=$5
+
+	# Check whether the inode exists
+	exists=$(find $path -inum $ino)
+	if [ -z "$exists" ]; then
+		_fail "inode $ino not recovered correctly"
+	fi
+
+	# Check inode type
+	found_mode=$(printf "%o" 0x$(stat $exists -c %f))
+	if [ $found_mode -ne $mode ]; then
+		echo "$found_mode"
+		_fail "inode $ino modes not recovered"
+	fi
+
+	# Check inode size
+	found_size=$(stat $exists -c %s)
+	if [ $mode -ne 41700 -a $found_size -ne $size ]; then
+		_fail "inode $ino size not recovered correctly"
+	fi
+
+	# Check inode name
+	if [ "$(basename $exists)" != "$name" ]; then
+		_fail "inode $ino name not recovered correctly"
+	else
+		return 0
+	fi
+}
+
+# Check salvaged data in the recovered image
+check_leaf_corrupt_no_data_ext()
+{
+	image=$1
+	if [ $UID -ne 0 ]; then
+		echo "     [NOTRUN] verify recovery. need root privilege"
+		exit 0
+	fi
+	if [ -z $TEST_MNT ]; then
+		echo "\$TEST_MNT not set, use $(pwd)/tmp as fallback"
+		TEST_MNT="$(pwd)/tmp"
+	fi
+	mkdir -p $TEST_MNT || _fail "failed to create mount point"
+	mount $image -o ro $TEST_MNT
+
+	i=0
+	while [ $i -lt ${#leaf_no_data_ext_list[@]} ]; do
+		check_inode $TEST_MNT/lost+found \
+			    ${leaf_no_data_ext_list[i]} \
+			    ${leaf_no_data_ext_list[i + 1]} \
+			    ${leaf_no_data_ext_list[i + 2]} \
+			    ${leaf_no_data_ext_list[i + 3]} \
+			    ${leaf_no_data_ext_list[i + 4]}
+			    ((i+=4))
+	done
+	umount $TEST_MNT
+}
+
+generate_leaf_corrupt_no_data_ext test.img
+check_image test.img
+check_leaf_corrupt_no_data_ext test.img
+
+rm test.img
+rm test.img.btrfs-image
+# Not used, its function is the same as generate_leaf_corrupt_no_data_ext()
+rm generate_image.sh
-- 
2.2.1

[PATCH 5/5] btrfs-progs: Move extent tree rebuild test to its dir

[Qu Wenruo]

Move extent tree rebuild teset to its dir.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/fsck-tests.sh                              | 39 --------------------
 tests/fsck-tests/013-extent-tree-rebuild/test.sh | 47 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+), 39 deletions(-)
 create mode 100755 tests/fsck-tests/013-extent-tree-rebuild/test.sh

diff --git a/tests/fsck-tests.sh b/tests/fsck-tests.sh
index a992b45..df21423 100755
--- a/tests/fsck-tests.sh
+++ b/tests/fsck-tests.sh
@@ -60,42 +60,3 @@ do
 	fi
 	cd $top
 done
-
-if [ -z $TEST_DEV ] || [ -z $TEST_MNT ];then
-	echo "     [NOTRUN] extent tree rebuild"
-	exit 0
-fi
-
-# test whether fsck can rebuild a corrupted extent tree
-test_extent_tree_rebuild()
-{
-	echo "     [TEST]    extent tree rebuild"
-	$top/mkfs.btrfs -f $TEST_DEV >> /dev/null 2>&1 || _fail "fail to mkfs"
-
-	run_check mount $TEST_DEV $TEST_MNT
-	cp -aR /lib/modules/`uname -r`/ $TEST_MNT 2>&1
-
-	for i in `seq 1 100`;do
-		$top/btrfs sub snapshot $TEST_MNT \
-			$TEST_MNT/snapaaaaaaa_$i >& /dev/null
-	done
-	run_check umount $TEST_DEV
-
-	# get extent root bytenr
-	extent_root_bytenr=`$top/btrfs-debug-tree -r $TEST_DEV | \
-			    grep extent | awk '{print $7}'`
-	if [ -z $extent_root_bytenr ];then
-		_fail "fail to get extent root bytenr"
-	fi
-
-	# corrupt extent root node block
-	run_check $top/btrfs-corrupt-block -l $extent_root_bytenr \
-		-b 4096 $TEST_DEV
-
-	$top/btrfs check $TEST_DEV >& /dev/null && \
-			_fail "btrfs check should detect failure"
-	run_check $top/btrfs check --init-extent-tree $TEST_DEV
-	run_check $top/btrfs check $TEST_DEV
-}
-
-test_extent_tree_rebuild
diff --git a/tests/fsck-tests/013-extent-tree-rebuild/test.sh b/tests/fsck-tests/013-extent-tree-rebuild/test.sh
new file mode 100755
index 0000000..ce51e07
--- /dev/null
+++ b/tests/fsck-tests/013-extent-tree-rebuild/test.sh
@@ -0,0 +1,47 @@
+#!/bin/bash
+
+source $top/tests/common
+
+if [ -z $TEST_DEV ]; then
+	echo "     [NOTRUN] extent tree rebuild, need TEST_DEV variant"
+	exit 0
+fi
+
+if [ -z $TEST_MNT ];then
+	echo "     [NOTRUN] extent tree rebuild, need TEST_MNT variant"
+	exit 0
+fi
+
+# test whether fsck can rebuild a corrupted extent tree
+test_extent_tree_rebuild()
+{
+	echo "     [TEST]    extent tree rebuild"
+	$top/mkfs.btrfs -f $TEST_DEV >> /dev/null 2>&1 || _fail "fail to mkfs"
+
+	run_check mount $TEST_DEV $TEST_MNT
+	cp -aR /lib/modules/`uname -r`/ $TEST_MNT 2>&1
+
+	for i in `seq 1 100`;do
+		$top/btrfs sub snapshot $TEST_MNT \
+			$TEST_MNT/snapaaaaaaa_$i >& /dev/null
+	done
+	run_check umount $TEST_DEV
+
+	# get extent root bytenr
+	extent_root_bytenr=`$top/btrfs-debug-tree -r $TEST_DEV | \
+			    grep extent | awk '{print $7}'`
+	if [ -z $extent_root_bytenr ];then
+		_fail "fail to get extent root bytenr"
+	fi
+
+	# corrupt extent root node block
+	run_check $top/btrfs-corrupt-block -l $extent_root_bytenr \
+		-b 4096 $TEST_DEV
+
+	$top/btrfs check $TEST_DEV >& /dev/null && \
+			_fail "btrfs check should detect failure"
+	run_check $top/btrfs check --init-extent-tree $TEST_DEV
+	run_check $top/btrfs check $TEST_DEV
+}
+
+test_extent_tree_rebuild
-- 
2.2.1

[PATCH 1/2] btrfs-progs: tests, add support for running commands under root

[David Sterba]

Most of the checks run fine without root, but some of them may need to
do a mount test or access the data. Add the support to selectively run
commands under root, hardcoded to sudo for now.

Signed-off-by: David Sterba <dsterba@suse.cz>
---

 tests/common | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/tests/common b/tests/common
index 80a538165949..84a4b9d335e2 100644
--- a/tests/common
+++ b/tests/common
@@ -47,3 +47,18 @@ check_all_images()
 		rm $i.restored
 	done
 }
+
+# some tests need to mount the recovered image and do verifications call
+# 'setup_root_helper' and then check for have_root_helper == 1 if the test
+# needs to fail otherwise; using sudo by default for now
+sudo=
+have_root_helper=0
+export sudo
+export have_root_helper
+setup_root_helper()
+{
+	if [ $UID != 0 ]; then
+		sudo=sudo
+	fi
+	have_root_helper=1
+}
-- 
2.1.3

[PATCH 2/2] btrfs-progs: teests, use the root helper in 012

[David Sterba]

We need it to mount/umount, to traverse lost+found and stat the results.

Signed-off-by: David Sterba <dsterba@suse.cz>
---
 tests/fsck-tests/012-leaf-corruption/test.sh | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh b/tests/fsck-tests/012-leaf-corruption/test.sh
index edf77193a08d..896f717a883c 100755
--- a/tests/fsck-tests/012-leaf-corruption/test.sh
+++ b/tests/fsck-tests/012-leaf-corruption/test.sh
@@ -55,20 +55,20 @@ check_inode()
 	name=$5
 
 	# Check whether the inode exists
-	exists=$(find $path -inum $ino)
+	exists=$($sudo find $path -inum $ino)
 	if [ -z "$exists" ]; then
 		_fail "inode $ino not recovered correctly"
 	fi
 
 	# Check inode type
-	found_mode=$(printf "%o" 0x$(stat $exists -c %f))
+	found_mode=$(printf "%o" 0x$($sudo stat $exists -c %f))
 	if [ $found_mode -ne $mode ]; then
 		echo "$found_mode"
 		_fail "inode $ino modes not recovered"
 	fi
 
 	# Check inode size
-	found_size=$(stat $exists -c %s)
+	found_size=$($sudo stat $exists -c %s)
 	if [ $mode -ne 41700 -a $found_size -ne $size ]; then
 		_fail "inode $ino size not recovered correctly"
 	fi
@@ -85,8 +85,8 @@ check_inode()
 check_leaf_corrupt_no_data_ext()
 {
 	image=$1
-	if [ $UID -ne 0 ]; then
-		echo "     [NOTRUN] verify recovery. need root privilege"
+	if [ $have_root_helper -ne 1 ]; then
+		echo "     [NOTRUN] root privileges needed to verify recovery"
 		exit 0
 	fi
 	if [ -z $TEST_MNT ]; then
@@ -94,7 +94,7 @@ check_leaf_corrupt_no_data_ext()
 		TEST_MNT="$(pwd)/tmp"
 	fi
 	mkdir -p $TEST_MNT || _fail "failed to create mount point"
-	mount $image -o ro $TEST_MNT
+	$sudo mount $image -o ro $TEST_MNT
 
 	i=0
 	while [ $i -lt ${#leaf_no_data_ext_list[@]} ]; do
@@ -106,9 +106,11 @@ check_leaf_corrupt_no_data_ext()
 			    ${leaf_no_data_ext_list[i + 4]}
 			    ((i+=4))
 	done
-	umount $TEST_MNT
+	$sudo umount $TEST_MNT
 }
 
+setup_root_helper
+
 generate_leaf_corrupt_no_data_ext test.img
 check_image test.img
 check_leaf_corrupt_no_data_ext test.img
-- 
2.1.3

Re: [PATCH 0/5] New btrfsck test framework

[David Sterba]

On Thu, Dec 25, 2014 at 09:32:10AM +0800, Qu Wenruo wrote:
> This patchset introduce the new btrfsck test framework.

Thanks, merged to the 3.18.x branch, with the two patches and one
cosmetic adjustment to the output. The file layout seems flexible enough
to do what we may need in the future.

Re: [PATCH 1/2] btrfs-progs: tests, add support for running commands under root

[Qu Wenruo]

-------- Original Message --------
Subject: [PATCH 1/2] btrfs-progs: tests, add support for running 
commands under root
From: David Sterba <dsterba@suse.cz>
To: <linux-btrfs@vger.kernel.org>
Date: 2015年01月15日 01:25
> Most of the checks run fine without root, but some of them may need to
> do a mount test or access the data. Add the support to selectively run
> commands under root, hardcoded to sudo for now.
I'm a little worried about using sudo in such test.
If the test is ran by a script, test will be stuck waiting for input and 
timeout after a long time.

I prefer NOT RUN just like xfstests.

Thanks
Qu
>
> Signed-off-by: David Sterba <dsterba@suse.cz>
> ---
>
>   tests/common | 15 +++++++++++++++
>   1 file changed, 15 insertions(+)
>
> diff --git a/tests/common b/tests/common
> index 80a538165949..84a4b9d335e2 100644
> --- a/tests/common
> +++ b/tests/common
> @@ -47,3 +47,18 @@ check_all_images()
>   		rm $i.restored
>   	done
>   }
> +
> +# some tests need to mount the recovered image and do verifications call
> +# 'setup_root_helper' and then check for have_root_helper == 1 if the test
> +# needs to fail otherwise; using sudo by default for now
> +sudo=
> +have_root_helper=0
> +export sudo
> +export have_root_helper
> +setup_root_helper()
> +{
> +	if [ $UID != 0 ]; then
> +		sudo=sudo
> +	fi
> +	have_root_helper=1
> +}

Re: [PATCH 1/2] btrfs-progs: tests, add support for running commands under root

[David Sterba]

> > Most of the checks run fine without root, but some of them may need to
> > do a mount test or access the data. Add the support to selectively run
> > commands under root, hardcoded to sudo for now.
> I'm a little worried about using sudo in such test.

Opposed to running the whole testsuite as root? Most tests are fine to
be run under the user privileges.

> If the test is ran by a script, test will be stuck waiting for input and 
> timeout after a long time.

Good point, I'll add --non-interactive that will abort in that case.

> I prefer NOT RUN just like xfstests.

But xfstests do not run if there's a missing implementation or
incomplete testing setup, ie. something that cannot be fixed by other
means.