[PATCH 1/6] xfstests: add fio requirement V2

[PATCH 1/6] xfstests: add fio requirement V2

[Dmitry Monakhov]

FIO is very flexible io generator, i would call it IO swiss knife.
Currently we have tonns of hardcoded application which reproduces
some predefined scenario. This approach has obvious dissadvantages
1) Lack of flexability: one written it is hard to modify it in future
2) Code base is large, many routines written again and again

At the same time add new fio based tast is just add simle INI file.
This greatly simplify code review. I do beleve that some day we will
replace most of hardcoded io binaries with fio.

One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example:  _require_fio 286-job.fio

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common.config |    1 +
 common.rc     |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/common.config b/common.config
index 7bed1c5..d5c8956 100644
--- a/common.config
+++ b/common.config
@@ -158,6 +158,7 @@ export XFS_QUOTA_PROG="`set_prog_path xfs_quota`"
 export KILLALL_PROG="`set_prog_path killall`"
 export INDENT_PROG="`set_prog_path indent`"
 export XFS_COPY_PROG="`set_prog_path xfs_copy`"
+export FIO_PROG="`set_prog_path fio`"
 
 # Generate a comparable xfsprogs version number in the form of
 # major * 10000 + minor * 100 + release
diff --git a/common.rc b/common.rc
index 602513a..f5027e7 100644
--- a/common.rc
+++ b/common.rc
@@ -1758,6 +1758,21 @@ _require_btrfs()
 	[ $? -eq 0 ] || _notrun "$BTRFS_UTIL_PROG too old (must support $cmd)"
 }
 
+# Check that fio is present, and it is able to execute given jobfile
+_require_fio()
+{
+	job=$1
+
+	_require_command $FIO_PROG
+	if [ -z "$1" ]; then
+		return 1;
+	fi
+	
+	$FIO_PROG --warnings-fatal --showcmd $job >/dev/null 2>&1
+	[ $? -eq 0 ] || _notrun "$FIO_PROG too old"
+}
+
+
 # arg 1 is dev to remove and is output of the below eg.
 # ls -l /sys/class/block/sdd | rev | cut -d "/" -f 3 | rev
 _devmgt_remove()
-- 
1.7.7.6

[PATCH 2/6] xfstest: add configurable load factors

[Dmitry Monakhov]

Most stress test has probable behaviour, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not speficied both variables defined to 1, so original behaviour
preserved.

TODO: Change all stress tests to use this variables

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common.config |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/common.config b/common.config
index d5c8956..cfa7bde 100644
--- a/common.config
+++ b/common.config
@@ -253,5 +253,13 @@ if [ ! -z "$SCRATCH_MNT" -a ! -d "$SCRATCH_MNT" ]; then
     exit 1
 fi
 
+if [ -z "$LOAD_FACTOR" ]; then
+    LOAD_FACTOR=1
+fi
+
+if [ -z "$TIME_FACTOR" ]; then
+    TIME_FACTOR=1
+fi
+
 # make sure this script returns success
 /bin/true
-- 
1.7.7.6

Re: [PATCH 2/6] xfstest: add configurable load factors

[Dave Chinner]

On Mon, Sep 24, 2012 at 06:23:17PM +0400, Dmitry Monakhov wrote:
> Most stress test has probable behaviour, the longer test run the
> larger corner cases will be cover. It is reasonable to allow
> user to provide some sort of system load factor.
> This patch introduce two global variables
> LOAD_FACTOR: Usually means factor number of running tasks
> TIME_FACTOR: Usually means factor of run time, or number of operations
> If not speficied both variables defined to 1, so original behaviour
> preserved.
> 
> TODO: Change all stress tests to use this variables
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>  common.config |    8 ++++++++
>  1 files changed, 8 insertions(+), 0 deletions(-)
> 
> diff --git a/common.config b/common.config
> index d5c8956..cfa7bde 100644
> --- a/common.config
> +++ b/common.config
> @@ -253,5 +253,13 @@ if [ ! -z "$SCRATCH_MNT" -a ! -d "$SCRATCH_MNT" ]; then
>      exit 1
>  fi
>  
> +if [ -z "$LOAD_FACTOR" ]; then
> +    LOAD_FACTOR=1
> +fi
> +
> +if [ -z "$TIME_FACTOR" ]; then
> +    TIME_FACTOR=1
> +fi

They probably need to be exparted variables set up through the
check script, not somthing sourced via common.config. i.e. something
like this in check where the environment is being set up:

export TIME_FACTOR=${TIME_FACTOR:=1}
export LOAD_FACTOR=${LOAD_FACTOR:=1}

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[PATCH 3/6] xfstest: allow fsstress to use load factor where appropriate

[Dmitry Monakhov]

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 017 |    5 ++++-
 068 |    4 ++--
 070 |    4 +++-
 076 |    5 ++++-
 083 |    4 ++--
 087 |   10 ++++++----
 104 |    4 ++--
 114 |    5 +++--
 167 |    4 ++--
 232 |    5 +++--
 233 |    5 +++--
 269 |    7 ++++---
 270 |    7 ++++---
 276 |   11 ++++++-----
 14 files changed, 48 insertions(+), 32 deletions(-)

diff --git a/017 b/017
index 9ca0e72..8d35ee8 100755
--- a/017
+++ b/017
@@ -67,7 +67,10 @@ echo "*** test"
 for l in 0 1 2 3 4
 do
         echo "    *** test $l"
-        $FSSTRESS_PROG -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID >>$seq.full
+	NUM=$((1000 * TIME_FACTOR))
+	CPU=$((1 * LOAD_FACTOR))
+        $FSSTRESS_PROG -d $SCRATCH_MNT -n $NUM -p $CPU \
+	    $FSSTRESS_AVOID >>$seq.full
 
         _scratch_mount -o remount,ro \
             || _fail "remount ro failed"
diff --git a/068 b/068
index b595d1d..9a01100 100755
--- a/068
+++ b/068
@@ -75,8 +75,8 @@ touch $tmp.running
     STRESS_DIR="$SCRATCH_MNT/fsstress_test_dir"
     mkdir "$STRESS_DIR"
 
-    procs=2
-    nops=200
+    procs=$((2 * LOAD_FACTOR))
+    nops=$((200 * TIME_FACTOR))
     while [ -f "$tmp.running" ]
       do
       # We do both read & write IO - not only is this more realistic,
diff --git a/070 b/070
index f48c33c..286ae90 100755
--- a/070
+++ b/070
@@ -62,7 +62,9 @@ $FSSTRESS_PROG \
 	-f unresvsp=0 \
 	-f attr_set=100 \
 	-f attr_remove=100 \
-        -p 1 -n 10000 -S c >$seq.full 2>&1
+        -p $((LOAD_FACTOR)) \
+    	-n $((10000 * TIME_FACTOR)) \
+    	-S c >$seq.full 2>&1
 
 status=$?
 exit
diff --git a/076 b/076
index e472b26..fa1a916 100755
--- a/076
+++ b/076
@@ -74,7 +74,10 @@ echo "*** test concurrent block/fs access"
 cat $SCRATCH_DEV >/dev/null &
 pid=$!
 
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID >>$seq.full
+num=$((2000 * TIME_FACTOR))
+proc=$((2 * LOAD_FACTOR))
+
+$FSSTRESS_PROG -d $SCRATCH_MNT -p $proc -n $num $FSSTRESS_AVOID >>$seq.full
 
 _lets_get_pidst
 _check_scratch_fs
diff --git a/083 b/083
index e0670b9..7af7c08 100755
--- a/083
+++ b/083
@@ -66,8 +66,8 @@ workout()
 {
 	fsz=$1
 	ags=$2
-	procs=$3
-	nops=$4
+	procs=$(($3 * LOAD_FACTOR))
+	nops=$(($4 * TIME_FACTOR))
 
 	umount $SCRATCH_DEV >/dev/null 2>&1
 	echo "*** mkfs -dsize=$fsz,agcount=$ags"    >>$seq.full
diff --git a/087 b/087
index 48e5eaa..5c67f5e 100755
--- a/087
+++ b/087
@@ -43,11 +43,13 @@ trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
 _do_meta()
 {
     out=$SCRATCH_MNT/fsstress
-    count=10000
-    param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
+    count=$((10000 * TIME_FACTOR))
+    proc=$((4 * LOAD_FACTOR))
+    param="-z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
            -f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
-    _echofull "calling fsstress $param -m8 -n $count"
-    if ! $FSSTRESS_PROG $param $FSSTRESS_AVOID -m 8 -n $count -d $out >>$seq.full 2>&1
+    _echofull "calling fsstress $param -m8 -p $proc -n $count"
+    if ! $FSSTRESS_PROG $param $FSSTRESS_AVOID -m 8 -p $proc \
+		-n $count -d $out >>$seq.full 2>&1
     then
 	_echofull "fsstress failed"
     fi
diff --git a/104 b/104
index 14f2669..abc9705 100755
--- a/104
+++ b/104
@@ -61,8 +61,8 @@ _fill_scratch()
 
 _stress_scratch()
 {
-	procs=3
-	nops=1000
+	procs=$((3 * LOAD_FACTOR))
+	nops=$((1000 * TIME_FACTOR))
 	# -w ensures that the only ops are ones which cause write I/O
 	$FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID > /dev/null &
 }
diff --git a/114 b/114
index 7679222..348f992 100755
--- a/114
+++ b/114
@@ -245,12 +245,13 @@ _test_fsstress()
 	echo ""
 
 	out=$SCRATCH_MNT/fsstress.$$
-	count=1000
+	count=$((1000 * TIME_FACTOR))
+	proc=$((3 * LOAD_FACTOR))
 	args="-z \
 -f rmdir=10 -f link=10 -f creat=10 \
 -f mkdir=10 -f rename=30 -f unlink=10 \
 -f symlink=10 \
--n $count -d $out -p 3"
+-n $count -d $out -p $proc"
 
 	echo "fsstress $args" | sed -e "s#$out#outdir#"
 	if ! $FSSTRESS_PROG $args | _filter_num
diff --git a/167 b/167
index ccb6c2a..ad5fcd0 100755
--- a/167
+++ b/167
@@ -42,8 +42,8 @@ _cleanup()
 
 workout()
 {
-	procs=100
-	nops=15000
+	procs=$((100 * LOAD_FACTOR))
+	nops=$((15000 * TIME_FACTOR))
 	$FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n $nops $FSSTRESS_AVOID \
 		>>$seq.full &
 	sleep 2
diff --git a/232 b/232
index 2795da7..6cd0171 100755
--- a/232
+++ b/232
@@ -53,8 +53,9 @@ _fsstress()
 	echo ""
 
 	out=$SCRATCH_MNT/fsstress.$$
-	count=2000
-	args="-n $count -d $out -p 7"
+	count=$((2000*TIME_FACTOR))
+	CPU=$((7 * LOAD_FACTOR))
+	args="-n $count -d $out -p $CPU"
 
 	echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
 	if ! $FSSTRESS_PROG $args | tee -a $here/$seq.full | _filter_num
diff --git a/233 b/233
index 28e6ac7..e30d9cc 100755
--- a/233
+++ b/233
@@ -57,11 +57,12 @@ _fsstress()
 	echo ""
 
 	out=$SCRATCH_MNT/fsstress.$$
-	count=5000
+	count=$((5000 * TIME_FACTOR))
+	proc=$((7 * LOAD_FACTOR))
 	args="-z \
 -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 \
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
--n $count -d $out -p 7"
+-n $count -d $out -p $proc"
 
 	echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
 	if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $here/$seq.full | _filter_num
diff --git a/269 b/269
index 7e13ed9..ca2700c 100755
--- a/269
+++ b/269
@@ -42,10 +42,11 @@ _workout()
 	echo ""
 	echo "Run fsstress"
 	echo ""
-	num_iterations=10
-	enospc_time=2
+	num_iterations=$((10 * TIME_FACTOR))
+	enospc_time=$((2 * TIME_FACTOR))
+	proc=$((128 * LOAD_FACTOR))
 	out=$SCRATCH_MNT/fsstress.$$
-	args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+	args="-p $proc -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
 	echo "fsstress $args" >> $here/$seq.full
 	$FSSTRESS_PROG $args > /dev/null 2>&1 &
 	pid=$!
diff --git a/270 b/270
index b9ada27..5197910 100755
--- a/270
+++ b/270
@@ -45,10 +45,11 @@ _workout()
 	echo ""
 	echo "Run fsstress"
 	echo ""
-	num_iterations=10
-	enospc_time=2
+	num_iterations=$((10 * TIME_FACTOR))
+	enospc_time=$((2 * TIME_FACTOR))
+	proc=$((128 * LOAD_FACTOR))
 	out=$SCRATCH_MNT/fsstress.$$
-	args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+	args="-p$proc -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
 	echo "fsstress $args" >> $here/$seq.full
 	# Grant chown capability 
 	cp $FSSTRESS_PROG  $tmp.fsstress.bin
diff --git a/276 b/276
index 082f943..551bef7 100755
--- a/276
+++ b/276
@@ -175,7 +175,8 @@ workout()
 {
 	fsz=$1
 	nfiles=$2
-	procs=$3
+	procs=$(($3 * LOAD_FACTOR))
+	num=$((1000* TIME_FACTOR))
 	snap_name=$4
 
 	umount $SCRATCH_DEV >/dev/null 2>&1
@@ -185,8 +186,8 @@ workout()
 		|| _fail "size=$fsz mkfs failed"
 	run_check _scratch_mount
 	# -w ensures that the only ops are ones which cause write I/O
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n 1000 \
-		$FSSTRESS_AVOID
+	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $num \
+	    	$FSSTRESS_AVOID
 
 	run_check $BTRFS_UTIL_PROG subvol snap $SCRATCH_MNT \
 		$SCRATCH_MNT/$snap_name
@@ -196,13 +197,13 @@ workout()
 
 	# make some noise but ensure we're not touching existing data
 	# extents.
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n 2000 \
+	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n $((num*2)) \
 		-z -f chown=3 -f link=1 -f mkdir=2 -f mknod=2 \
 		-f rename=2 -f setxattr=1 -f symlink=2
 	clean_dir="$SCRATCH_MNT/next"
 	mkdir $clean_dir
 	# now make more files to get a higher tree
-	run_check $FSSTRESS_PROG -d $clean_dir -w -p $procs -n 2000 \
+	run_check $FSSTRESS_PROG -d $clean_dir -w -p $procs -n $((num*2)) \
 		$FSSTRESS_AVOID
 	run_check umount $SCRATCH_DEV >/dev/null 2>&1
 	run_check _scratch_mount "-o atime"
-- 
1.7.7.6

Re: [PATCH 3/6] xfstest: allow fsstress to use load factor where appropriate

[Dave Chinner]

On Mon, Sep 24, 2012 at 06:23:18PM +0400, Dmitry Monakhov wrote:
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>  017 |    5 ++++-
>  068 |    4 ++--
>  070 |    4 +++-
>  076 |    5 ++++-
>  083 |    4 ++--
>  087 |   10 ++++++----
>  104 |    4 ++--
>  114 |    5 +++--
>  167 |    4 ++--
>  232 |    5 +++--
>  233 |    5 +++--
>  269 |    7 ++++---
>  270 |    7 ++++---
>  276 |   11 ++++++-----
>  14 files changed, 48 insertions(+), 32 deletions(-)
> 
> diff --git a/017 b/017
> index 9ca0e72..8d35ee8 100755
> --- a/017
> +++ b/017
> @@ -67,7 +67,10 @@ echo "*** test"
>  for l in 0 1 2 3 4
>  do
>          echo "    *** test $l"
> -        $FSSTRESS_PROG -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID >>$seq.full
> +	NUM=$((1000 * TIME_FACTOR))

$TIME_FACTOR, perhaps?

> +	CPU=$((1 * LOAD_FACTOR))
> +        $FSSTRESS_PROG -d $SCRATCH_MNT -n $NUM -p $CPU \
> +	    $FSSTRESS_AVOID >>$seq.full

I'd much prefer a wrapper around fsstress than have to code this
into every test. Something like:

run_fstress()
{
	args=""

	while [ $# -gt 0 ]; do
		case "$1" in
		-n) args="$args $1 $(($2 * $TIME_FACTOR))"; shift ;;
		-p) args="$args $1 $(($2 * $LOAD_FACTOR))"; shift ;;
		*) args="$args $1" ;;
		esac

		shift
	done

	$FSSTRESS_PROG $args
}

And so all you need to do is something like:

$ sed -i 's/\$FSSTRESS_PROG/run_fsstress/' [0-9][0-9][0-9]

To convert all tests that use fsstress to do this.

You might also want to do this for the tests that run dirstress as
well, and create a "stress" group so that you can run just
the stress tests easily....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[PATCH 4/6] add fallocate/truncate vs AIO/DIO stress test

[Dmitry Monakhov]

Run  DIO, fallocate and truncate threads on a common file in parallel.
If race exist old dio request may rewrite blocks after it was allocated
to another file, we will catch that by verifying blocks content.

this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 286     |  154 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 286.out |    5 ++
 group   |    1 +
 3 files changed, 160 insertions(+), 0 deletions(-)
 create mode 100755 286
 create mode 100644 286.out

diff --git a/286 b/286
new file mode 100755
index 0000000..8802c56
--- /dev/null
+++ b/286
@@ -0,0 +1,154 @@
+#! /bin/bash
+# FSQA Test No. 286
+#
+# Test various aio dio vs truncate
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2006 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+run_check()
+{
+	echo "# $@" >> $seq.full 2>&1
+	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+iodepth=128*${LOAD_FACTOR}
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio]
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+
+# Perform direct aio and verify data
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randrw
+filename=aio-dio-verifier
+
+# Perform buffered aio and verify data
+[buffered-aio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=0
+buffered=1
+bs=4k
+rw=randrw
+filename=buffered-aio-verifier
+EOF
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio >>  $seq.full &
+	pid=$!
+	echo "Start fallocate/truncate loop"
+	for ((i=0; ; i++))
+	do
+	    for ((k=1; k <= NUM_JOBS; k++))
+	    do
+		fallocate -l $FILE_SIZE $SCRATCH_MNT/direct_aio.$k.0 \
+		    	>> $seq.full 2>&1
+	    done
+	    for ((k=1; k <= NUM_JOBS; k++))
+	    do
+		truncate -s 0 $SCRATCH_MNT/direct_aio.$k.0
+	    done
+	    # One fio exit we can stop fallocate/truncate loop
+	    kill -0 $pid > /dev/null 2>&1 || break
+	done
+	wait $pid
+}
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/286.out b/286.out
new file mode 100644
index 0000000..d721996
--- /dev/null
+++ b/286.out
@@ -0,0 +1,5 @@
+QA output created by 286
+
+Run fio with random aio-dio pattern
+
+Start fallocate/truncate loop
diff --git a/group b/group
index 697269b..2469f80 100644
--- a/group
+++ b/group
@@ -404,3 +404,4 @@ deprecated
 283 dump ioctl auto quick
 284 auto
 285 auto dump quota quick
+286 auto rw enospc aio
-- 
1.7.7.6

Re: [PATCH 4/6] add fallocate/truncate vs AIO/DIO stress test

[Dave Chinner]

On Mon, Sep 24, 2012 at 06:23:19PM +0400, Dmitry Monakhov wrote:
> Run  DIO, fallocate and truncate threads on a common file in parallel.
> If race exist old dio request may rewrite blocks after it was allocated
> to another file, we will catch that by verifying blocks content.
> 
> this patch known to catch deadlock for ext4
> http://lists.openwall.net/linux-ext4/2012/09/06/3
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>  286     |  154 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  286.out |    5 ++
>  group   |    1 +
>  3 files changed, 160 insertions(+), 0 deletions(-)
>  create mode 100755 286
>  create mode 100644 286.out
> 
> diff --git a/286 b/286
> new file mode 100755
> index 0000000..8802c56
> --- /dev/null
> +++ b/286
> @@ -0,0 +1,154 @@
> +#! /bin/bash
> +# FSQA Test No. 286
> +#
> +# Test various aio dio vs truncate

Can you add a better description of what the test is supposed to do
here? The commit message above would be a good start.

....
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +run_check()
> +{
> +	echo "# $@" >> $seq.full 2>&1
> +	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
> +}

run-check is being copied from test to test. Can you move it to
common.rc?

> +
> +
> +NUM_JOBS=$((4*LOAD_FACTOR))
> +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> +FILE_SIZE=$((BLK_DEV_SIZE * 512))
> +
> +cat >$tmp-$seq.fio <<EOF

Before this, please move all the other test setup stuff like the
_require directives before this. That way I don't have to scroll
past a hundred lines of test specific setup to find out what the
test constraints are. i.e. these:

> +_supported_fs generic
> +_supported_os Linux
> +_need_to_be_root
> +_require_scratch

.....
> +## Perform direct aio, to files which may be truncated
> +## by external task
> +[direct_aio]
> +direct=1
> +buffered=0
> +numjobs=${NUM_JOBS}
> +rw=randwrite
> +runtime=100*${TIME_FACTOR}
> +time_based
.....
> +_workout()
> +{
> +	echo ""
> +	echo "Run fio with random aio-dio pattern"
> +	echo ""
> +	cat $tmp-$seq.fio >>  $seq.full
> +	run_check $FIO_PROG $tmp-$seq.fio >>  $seq.full &
> +	pid=$!
> +	echo "Start fallocate/truncate loop"
> +	for ((i=0; ; i++))
> +	do
> +	    for ((k=1; k <= NUM_JOBS; k++))
> +	    do
> +		fallocate -l $FILE_SIZE $SCRATCH_MNT/direct_aio.$k.0 \
> +		    	>> $seq.full 2>&1
> +	    done

It's not clear to me where the file names come from. I think it's
because the FIO job takes the name of the file if it's not specified
from the job name (i.e. [direct_aio]).  If so, can you make that a
variable and comment to the effect that the filenames are derived from
the FIO job definition?

> +	    for ((k=1; k <= NUM_JOBS; k++))
> +	    do
> +		truncate -s 0 $SCRATCH_MNT/direct_aio.$k.0
> +	    done
> +	    # One fio exit we can stop fallocate/truncate loop

	      Once?

> +	    kill -0 $pid > /dev/null 2>&1 || break

/me has to look up what kill -0 does, as 0 is not a valid signal
number.

The kill(1) man page doesn't document it, kill -l doesn't list 0
as a valid signal, and signal(7) doesn't document it either.
Wonderful!

Oh, there it is - in the syscall documentation (i.e. kill(2)):

"this can be used to check for the existence of a process ID"

Right, OK, now I understand the loop. :)

....
> +_scratch_mkfs >> $seq.full 2>&1
> +_scratch_mount
> +
> +if ! _workout; then
> +	umount $SCRATCH_DEV 2>/dev/null
> +	exit
> +fi

No need to unmount the SCRATCH_DEV on exit - that will happen if
necessary next time _require_scratch is called by a test....

> +
> +if ! _scratch_unmount; then
> +	echo "failed to umount"
> +	status=1
> +	exit
> +fi

No need to unmount before checking, the _check_scratch_fs function
does that for you and will leave unmount errors in the log if it
fails. i.e. the test will fail the check and the reason will be
obvious from the output...

> +_check_scratch_fs
> +status=$?
> +exit
> diff --git a/286.out b/286.out
> new file mode 100644
> index 0000000..d721996
> --- /dev/null
> +++ b/286.out
> @@ -0,0 +1,5 @@
> +QA output created by 286
> +
> +Run fio with random aio-dio pattern
> +
> +Start fallocate/truncate loop
> diff --git a/group b/group
> index 697269b..2469f80 100644
> --- a/group
> +++ b/group
> @@ -404,3 +404,4 @@ deprecated
>  283 dump ioctl auto quick
>  284 auto
>  285 auto dump quota quick
> +286 auto rw enospc aio

Why the ENOSPC group? I don't see anything that exercises ENOSPC
behaviour in the test, and none of the comments indicate that it 

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[PATCH 5/6] add fallocate/punch_hole vs AIO/DIO stress test

[Dmitry Monakhov]

Run  DIO, fallocate and punch_hole threads on a common file in parallel.
If race exist old dio request may rewrite punched block after it was
allocated to another file, we will catch that by verifying blocks content.

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 287   |  156 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 group |    1 +
 2 files changed, 157 insertions(+), 0 deletions(-)
 create mode 100755 287

diff --git a/287 b/287
new file mode 100755
index 0000000..7f6823f
--- /dev/null
+++ b/287
@@ -0,0 +1,156 @@
+#! /bin/bash
+# FSQA Test No. 287
+#
+# Test various races AIO/DIO vs fallocate/punch_hole
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2006 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+run_check()
+{
+	echo "# $@" >> $seq.full 2>&1
+	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+if [ $((BLK_DEV_SIZE)) -gt 1048576 ]
+then
+    BLK_DEV_SIZE=1048576
+fi 
+FS_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Run  DIO, fallocate and punch_hole threads on a single in parallel 
+#
+# If race exist old dio request may rewrite punched block after it was
+# allocated to another file, we will catch that by verifying blocks content
+#
+[global]
+directory=${SCRATCH_MNT}
+filesize=${FS_SIZE}
+size=999G
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio_raicer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR} 
+bs=128k 
+direct=1
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+filename=racer
+
+# Run falloc and punch_hole threads in parallel
+# After activity file will be highly fragmented
+[falloc_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+iodepth=1
+bssplit=128k/80:512k/10:32k/10
+rw=randwrite
+numjobs=1
+filename=racer
+
+[punch_hole_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+bs=4k
+time_based=10
+rw=randtrim
+numjobs=2
+filename=racer
+time_based
+
+# Verifier thread continiously write to newly allcated blocks
+# and veryfy written content
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randwrite
+filename=aio-dio-verifier
+EOF
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio >>  $seq.full
+}
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs_sized $FS_SIZE >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/group b/group
index 2469f80..37f3256 100644
--- a/group
+++ b/group
@@ -405,3 +405,4 @@ deprecated
 284 auto
 285 auto dump quota quick
 286 auto rw enospc aio
+287 auto rw enospc aio prealloc
-- 
1.7.7.6

Re: [PATCH 5/6] add fallocate/punch_hole vs AIO/DIO stress test

[Dave Chinner]

On Mon, Sep 24, 2012 at 06:23:20PM +0400, Dmitry Monakhov wrote:
> Run  DIO, fallocate and punch_hole threads on a common file in parallel.
> If race exist old dio request may rewrite punched block after it was
> allocated to another file, we will catch that by verifying blocks content.
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Same comments as for the previous patch.

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com

[PATCH 6/6] add defragmentation stress test for ext4

[Dmitry Monakhov]

Perform various regression tests for ext4defrag  subsystem

 Test1: Defragment file while other task does direct AIO
 Test2: Perform defragmentation on file under buffered AIO
 	while third task does direct AIO to donor file
 Test3: Two defrag tasks use common donor file.
 Test4: Stress defragmentation. Several threads pefrorm
 	fragmentation at random position use inplace=1 will
	allocate and free blocks inside defrag event improve
	load pressure.

This test known to caught most known e4defrag bugs.

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 288     |  278 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 288.out |    4 +
 group   |    1 +
 3 files changed, 283 insertions(+), 0 deletions(-)
 create mode 100755 288
 create mode 100644 288.out

diff --git a/288 b/288
new file mode 100755
index 0000000..c51f563
--- /dev/null
+++ b/288
@@ -0,0 +1,278 @@
+#! /bin/bash
+# FSQA Test No. 288
+#
+# Ext4 defragmentatio stress test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2006 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+run_check()
+{
+	echo "# $@" >> $seq.full 2>&1
+	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 8 files 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 /12)))
+
+cat >$tmp-$seq.fio <<EOF
+# Various e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test1
+# Defragment file while other task does direct io
+
+# Continious sequential defrag activity
+[defrag-4k]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test1.def
+filename=test1
+inplace=0
+rw=write
+numjobs=1
+runtime=30
+time_based
+
+# Verifier
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=randwrite
+filename=test1
+runtime=30
+time_based
+
+##########################################
+# Test2 
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#
+# Continious sequential defrag activity
+[defrag-4k]
+stonewall
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test2.def
+filename=test2
+inplace=0
+rw=write
+numjobs=1
+runtime=30
+time_based
+
+# Run DIO/AIO for donor file
+[donor-file-fuzzer]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=0
+direct=1
+bs=64k
+rw=randwrite
+filename=test2.def
+runtime=30
+time_based
+
+# Verifier thread
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test2
+runtime=30
+time_based
+
+#################################
+# Test3
+# Two defrag tasks use common donor file
+
+[defrag-1]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test31
+inplace=0
+rw=write
+numjobs=1
+runtime=30
+time_based
+
+[defrag-2]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test32
+inplace=0
+rw=write
+numjobs=1
+runtime=30
+time_based
+
+[aio-dio-verifier-1]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test31
+runtime=30
+time_based
+
+[aio-buffer-verifier-2]
+ioengine=libaio 
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test32
+runtime=30
+time_based
+
+#################################
+# Test4
+# Stress test defragmentation
+# Several threads pefrorm defragmentatin at random position
+# use inplace=1 will allocate and free blocks inside defrag event
+# improve load pressure
+[defrag-fuzzer]
+ioengine=e4defrag
+iodepth=1
+bs=8k
+donorname=test4.def
+filename=test4
+inplace=1
+rw=randwrite
+numjobs=4*${LOAD_FACTOR}
+runtime=30
+time_based
+
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128
+iomem_align=4k
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test4
+runtime=30
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio >>  $seq.full
+}
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/288.out b/288.out
new file mode 100644
index 0000000..b215a3f
--- /dev/null
+++ b/288.out
@@ -0,0 +1,4 @@
+QA output created by 288
+
+ Start defragment activity 
+
diff --git a/group b/group
index 37f3256..8bd5c21 100644
--- a/group
+++ b/group
@@ -406,3 +406,4 @@ deprecated
 285 auto dump quota quick
 286 auto rw enospc aio
 287 auto rw enospc aio prealloc
+288 auto rw aio
-- 
1.7.7.6

Re: [PATCH 6/6] add defragmentation stress test for ext4

[Dave Chinner]

On Mon, Sep 24, 2012 at 06:23:21PM +0400, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag  subsystem
> 
>  Test1: Defragment file while other task does direct AIO
>  Test2: Perform defragmentation on file under buffered AIO
>  	while third task does direct AIO to donor file
>  Test3: Two defrag tasks use common donor file.
>  Test4: Stress defragmentation. Several threads pefrorm
>  	fragmentation at random position use inplace=1 will
> 	allocate and free blocks inside defrag event improve
> 	load pressure.

Please do these as 4 separate tests so that it is easy to
isolate/reproduce failures caused by a specific test that is being
run.

Same comments as previous two patches, as well.

> +_workout()
> +{
> +	echo ""
> +	echo " Start defragment activity "
> +	echo ""
> +	cat $tmp-$seq.fio >>  $seq.full
> +	run_check $FIO_PROG $tmp-$seq.fio >>  $seq.full
> +}
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_supported_fs ext4
> +_need_to_be_root
> +_require_scratch
> +_require_fio $tmp-$seq.fio

Even though e4defrag is not used, it's probably still a good idea to
call _require_defrag to check for systems that have a recent enough
ext4 to run defrag on. e.g. a recent fio on an old distro might
result in fio saying "I understand this job file" but then not be
able to run defrag...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH 1/6] xfstests: add fio requirement V2

[Dave Chinner]

On Mon, Sep 24, 2012 at 06:23:16PM +0400, Dmitry Monakhov wrote:
> FIO is very flexible io generator, i would call it IO swiss knife.
> Currently we have tonns of hardcoded application which reproduces
> some predefined scenario. This approach has obvious dissadvantages
> 1) Lack of flexability: one written it is hard to modify it in future
> 2) Code base is large, many routines written again and again
> 
> At the same time add new fio based tast is just add simle INI file.
> This greatly simplify code review. I do beleve that some day we will
> replace most of hardcoded io binaries with fio.
> 
> One who is planning to run $FIO_PROG should first check that system
> contains appropriate version which is able to handle jobfile
> for example:  _require_fio 286-job.fio
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Looks good.

Reviewed-by: Dave Chinner <dchinner@redhat.com>

-- 
Dave Chinner
david@fromorbit.com