Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Rich Johnston <rjohnston@sgi.com>]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag  subsystem
>
>   299'th Test1: Defragment file while other task does direct AIO
>   300'th Test2: Perform defragmentation on file under buffered AIO
>   	while third task does direct AIO to donor file
>   301'th Test3: Two defrag tasks use common donor file.
>   302'th Test4: Stress defragmentation. Several threads pefrorm
typo 	                                                  perform

>   	fragmentation at random position use inplace=1 will
> 	allocate and free blocks inside defrag event improve
> 	load pressure.
>
> This tests are considered dengerous because 300'th and 301'th are known
typo                        dangerous

> to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>   299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
>   299.out       |    4 ++
>   300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>   300.out       |    4 ++
>   301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   301.out       |    4 ++
>   302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
>   302.out       |    4 ++
>   common.defrag |    4 +-
>   group         |    6 ++-
>   10 files changed, 575 insertions(+), 3 deletions(-)
>   create mode 100755 299
>   create mode 100644 299.out
>   create mode 100755 300
>   create mode 100644 300.out
>   create mode 100755 301
>   create mode 100644 301.out
>   create mode 100755 302
>   create mode 100644 302.out
>
> diff --git a/299 b/299
> new file mode 100755
> index 0000000..4a7a98c
> --- /dev/null
> +++ b/299
> @@ -0,0 +1,126 @@
> +#! /bin/bash
> +# FSQA Test No. 299
> +#
> +# Ext4 defragmentation stress test
> +# Defragment file while other task does direct io
> +#-----------------------------------------------------------------------
> +# (c) 2013 Dmitry Monakhov
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#
> +#-----------------------------------------------------------------------
> +#
> +# creator
> +owner=dmonakhov@openvz.org
> +
> +seq=`basename $0`
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.defrag
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs ext4
> +_need_to_be_root
> +_require_scratch
> +_require_defrag
> +
I think you meant to set NUM_JOBS here like you did in [Patch 08/10], 
something like:

+NUM_JOBS=$((4*LOAD_FACTOR))

otherwise you get

###	 [not run] /usr/local/bin/fio too old

> +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> +# We need space for 2 files (test file, and donor one)
> +# reserve 30% in order to avoid ENOSPC
> +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> +
> +cat >$tmp-$seq.fio <<EOF
> +# Common e4defrag regression tests
> +[global]
> +ioengine=ioe_e4defrag
> +iodepth=1
> +directory=${SCRATCH_MNT}
> +filesize=${FILE_SIZE}
> +size=999G
> +buffered=0
> +fadvise_hint=0
> +group_reporting
> +
> +#################################
> +# Test1
> +# Defragment file while other task does direct io
> +
> +# Continious sequential defrag activity
> +[defrag-4k]
> +ioengine=e4defrag
> +iodepth=1
> +bs=128k
> +donorname=test1.def
> +filename=test1
> +inplace=0
> +rw=write
> +numjobs=${NUM_JOBS}
> +runtime=30*${TIME_FACTOR}
> +time_based
> +
> +# Verifier
> +[aio-dio-verifier]
> +ioengine=libaio
> +iodepth=128*${LOAD_FACTOR}
> +numjobs=1
> +verify=crc32c-intel
> +verify_fatal=1
> +verify_dump=1
> +verify_backlog=1024
> +verify_async=1
> +verifysort=1
> +direct=1
> +bs=64k
> +rw=randwrite
> +filename=test1
> +runtime=30*${TIME_FACTOR}
> +time_based
> +EOF
> +
> +_workout()
> +{
> +	echo ""
> +	echo " Start defragment activity "
git does not like trailing whitespace   ^
(same for the other 3 tests)
> +	echo ""
> +	cat $tmp-$seq.fio >>  $seq.full
> +	run_check $FIO_PROG $tmp-$seq.fio
> +}
> +
> +_require_fio $tmp-$seq.fio
> +
> +_scratch_mkfs  >> $seq.full 2>&1
> +_scratch_mount
> +
> +if ! _workout; then
> +	umount $SCRATCH_DEV 2>/dev/null
> +	exit
> +fi
> +
> +if ! _scratch_unmount; then
> +	echo "failed to umount"
> +	status=1
> +	exit
> +fi
> +_check_scratch_fs
> +status=$?
> +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> diff --git a/299.out b/299.out
> new file mode 100644
> index 0000000..c2b0b4c
> --- /dev/null
> +++ b/299.out
> @@ -0,0 +1,4 @@
> +QA output created by 299
> +
> + Start defragment activity
remove trailing whitespace   ^
(same for the other 3 tests)

Let me know if you agree with these changes and I will make them at 
commit time.

Thanks
--Rich