[PATCH 1/7] Create environment setup files

[PATCH 1/7] Create environment setup files

[Jan Ťulák]

First from a batch of patches for adding an environment support.  This
description is rather long, as it describes the goal of all set, so a
TLDR version at first:

- Allows to separate preparation of the environment (full fs,
  damaged fs, ...) from a test itself. So multiple tests can use
  exactly the same conditions.
- A single test can be run in multiple environments.
- Disabled by default for backward compatibility (it changes output).
- I expect it will cause some debate. It is my first bigger patch
  at all. :-)

Long version:

The goal of this set is to allow a single test to be run in different
sitations, for example on empty filesystem, full fs, or damaged fs.
It provides an interface for scripts that can prepare the requested
environments and takes care of starting the test for each one.

Currently, this functionality needs to be enabled explicitely by a
flag -e. It changes output slightly, so I saw this as neccessity.  The
output change is because one test can be run multiple times in
different environments, to note the combination. So when enabled,
[env-name] is added: "xfs/001 [some-environment] 123s ... 456s"

If the test is not aware of this new functionality, nothing changes
for it, the test will run as usuall.

This is a part of my work on performance tests (they needs this sort
of functionality), but is independent on them, so I'm proposing it
now.

Of the seven patches, first three creates new files. Patches four to
six modifies ./check script, but keeps the changes out of existing
code as much as possible (patch four is only exception).  Patch seven
is integrating it all together and is enabling the functionality.

To sum how it works:
New file "environment", similar to "group" file, is created in each
test category. It uses similar syntax, but it ortogonal to groups. In
this file, each test can have specified one or more environments. When
environments are enabled (./check -e ), list of tests is compiled as
before (so -g, -x and other arguments works as usually) and for the
enabled tests, environments are found.

If one test has multiple environments (and the selection is not
limited for only some env.), the test is duplicated for each specified
environment. Each run is then reported independently, as a combination
of the test and the environment. When the test is not found in the
file, it is added implicitly with "none" environment. The none
environment do nothing and can be stated explicitly in the file also.

The tests has to be aware of this. The same way as test requires
TEST_DIR or SCRATCH_DIR, it also needs to require environment setup
for one of these dirs. Some example is:

_require_test
_require_environment $TEST_DIR

If the test is not aware of environments, it runs just as before.

It is possible to share the environment between multiple tests.  For
some longer-running setups (like filling the filesystem with lots of
data), it is usefull to be able to keep the created files.  So in the
environment file, prefixing an environment with underscore switches
between this persistent environment and a "prepare it always from
scratch" mode. Right now, the default is to "prepare once, run
multiple times." That is good for the performance testing that will
follow, but I can change it if it would help to other tests.

For now, did not edited existing tests to make them environment-aware.
In existing tests, as opossed to performance testing, there is not
much places where this is useful, as they checks for specific things
in specific conditions. So these patches are primary infrastructure
preparation work for perfromance tests.

So an example of how to run tests with environments is:
./check -e -g performance
for running all tests in performance group with all environments 
assigned in an environment file. Text arguments of -eo/-ex can be
used to explicitly select just test/environment combinations with
or without given environment.

Further details are in each patch.

THIS PATCH SPECIFIC DESCRIPTION:
>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8>8

This patch creates four example scripts for setting up an environment
in "environments" directory. Except an empty template and two dummy
files (used for a demonstration of how "more environments for one
test"), there is "fill90-dvd script, which fills a target dir
(TEST_DIR/SCRATCH_DIR) from 90 percents with files of a DVD size
(4 GiB).

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 environments/_template  | 100 +++++++++++++++++++++++++++++++++++++++++
 environments/dummy1     |  99 +++++++++++++++++++++++++++++++++++++++++
 environments/dummy2     |  99 +++++++++++++++++++++++++++++++++++++++++
 environments/fill90-dvd | 115 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 413 insertions(+)
 create mode 100644 environments/_template
 create mode 100644 environments/dummy1
 create mode 100644 environments/dummy2
 create mode 100644 environments/fill90-dvd

diff --git a/environments/_template b/environments/_template
new file mode 100644
index 0000000..937bf7d
--- /dev/null
+++ b/environments/_template
@@ -0,0 +1,100 @@
+#!/bin/bash
+# FS QA Environment setup
+#-----------------------------------------------------------------------
+# Copyright 2014 (C) Red Hat, Inc., Jan Tulak <jtulak@redhat.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# arguments: prepare-once, prepare-always, clean
+
+
+THIS_ENVIRONMENT="$(basename $BASH_SOURCE)"
+
+
+# This function should prepare the environment
+prepare()
+{
+	target="$1"
+	echo "ENV ---- Really preparing test directory '$target'."
+}
+
+# This function should clean files created by prepare()
+clean()
+{
+	target="$1"
+	echo "ENV ---- cleaning '$target'"
+}
+
+
+
+
+# ----------------------------------------------------------------------
+# The following code usually don't need any changes for your environment
+# ----------------------------------------------------------------------
+
+prepare_once()
+{
+	target="$1"
+	# check if this environment is already created (was the last one
+	# running)
+	if [ "$env_last" != "$THIS_ENVIRONMENT" ];then
+		echo "ENV ---- There was some other environment, so lets do something in $target. ($THIS_ENVIRONMENT)"
+		prepare "$target"
+	else
+		echo "ENV ---- Stop there, I'm already prepared! ($THIS_ENVIRONMENT)"
+	fi
+
+}
+
+prepare_always()
+{
+	target="$1"
+	# We still need to check the previous environment,
+	# because if it is the same as now, it wasn't cleaned yet!
+	if [ "$env_last" = "$THIS_ENVIRONMENT" ];then
+		echo "ENV ---- I'm preparing again... but I need to clean!"
+		clean "$target"
+	else
+		echo "ENV ---- I'm preparing 'again'... but I wasn't run yet."
+	fi
+	prepare "$target"
+}
+
+usage()
+{
+echo "Usage: 
+$0 OPTION TARGET_DIR
+Where OPTION is one (and only one) of the following:
+    prepare-once, prepare-always, clean"
+}
+
+# arguments...
+if [ $# -ne 2 ];then
+	usage
+#	exit 1
+fi
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+		prepare-once) prepare_once "$2"; shift ;;
+		prepare-always) prepare_always "$2"; shift ;;
+		clean) unset ENVIRONMENT_LAST; clean "$2"; shift ;;
+		*) usage ; exit 1 ;;
+	esac
+	shift
+done
+
+exit 0
diff --git a/environments/dummy1 b/environments/dummy1
new file mode 100644
index 0000000..de6dcf0
--- /dev/null
+++ b/environments/dummy1
@@ -0,0 +1,99 @@
+#!/bin/bash
+# FS QA Environment setup - a dummy file, does nothing
+#-----------------------------------------------------------------------
+# Copyright 2014 (C) Red Hat, Inc., Jan Tulak <jtulak@redhat.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# arguments: prepare-once, prepare-always, clean
+
+THIS_ENVIRONMENT="$(basename $BASH_SOURCE)"
+
+
+# This function should prepare the environment
+prepare()
+{
+	target="$1"
+#	echo "ENV ---- Really preparing test directory '$target'."
+}
+
+# This function should clean files created by prepare()
+clean()
+{
+	target="$1"
+#	echo "ENV ---- cleaning '$target'"
+}
+
+
+
+
+# ----------------------------------------------------------------------
+# The following code usually don't need any changes for your environment
+# ----------------------------------------------------------------------
+
+prepare_once()
+{
+	target="$1"
+	# check if this environment is already created (was the last one
+	# running)
+	if [ "$env_last" != "$THIS_ENVIRONMENT" ];then
+#		echo "ENV ---- There was some other environment, so lets do something in $target. ($THIS_ENVIRONMENT)"
+		prepare "$target"
+#	else
+#		echo "ENV ---- Stop there, I'm already prepared! ($THIS_ENVIRONMENT)"
+	fi
+
+}
+
+prepare_always()
+{
+	target="$1"
+	# We still need to check the previous environment,
+	# because if it is the same as now, it wasn't cleaned yet!
+	if [ "$env_last" = "$THIS_ENVIRONMENT" ];then
+#		echo "ENV ---- I'm preparing again... but I need to clean!"
+		clean "$target"
+#	else
+#		echo "ENV ---- I'm preparing 'again'... but I wasn't run yet."
+	fi
+	prepare "$target"
+}
+
+usage()
+{
+echo "Usage: 
+$0 OPTION TARGET_DIR
+Where OPTION is one (and only one) of the following:
+    prepare-once, prepare-always, clean"
+}
+
+# arguments...
+if [ $# -ne 2 ];then
+	usage
+#	exit 1
+fi
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+		prepare-once) prepare_once "$2"; shift ;;
+		prepare-always) prepare_always "$2"; shift ;;
+		clean) unset ENVIRONMENT_LAST; clean "$2"; shift ;;
+		*) usage ; exit 1 ;;
+	esac
+	shift
+done
+
+exit 0
diff --git a/environments/dummy2 b/environments/dummy2
new file mode 100644
index 0000000..de6dcf0
--- /dev/null
+++ b/environments/dummy2
@@ -0,0 +1,99 @@
+#!/bin/bash
+# FS QA Environment setup - a dummy file, does nothing
+#-----------------------------------------------------------------------
+# Copyright 2014 (C) Red Hat, Inc., Jan Tulak <jtulak@redhat.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# arguments: prepare-once, prepare-always, clean
+
+THIS_ENVIRONMENT="$(basename $BASH_SOURCE)"
+
+
+# This function should prepare the environment
+prepare()
+{
+	target="$1"
+#	echo "ENV ---- Really preparing test directory '$target'."
+}
+
+# This function should clean files created by prepare()
+clean()
+{
+	target="$1"
+#	echo "ENV ---- cleaning '$target'"
+}
+
+
+
+
+# ----------------------------------------------------------------------
+# The following code usually don't need any changes for your environment
+# ----------------------------------------------------------------------
+
+prepare_once()
+{
+	target="$1"
+	# check if this environment is already created (was the last one
+	# running)
+	if [ "$env_last" != "$THIS_ENVIRONMENT" ];then
+#		echo "ENV ---- There was some other environment, so lets do something in $target. ($THIS_ENVIRONMENT)"
+		prepare "$target"
+#	else
+#		echo "ENV ---- Stop there, I'm already prepared! ($THIS_ENVIRONMENT)"
+	fi
+
+}
+
+prepare_always()
+{
+	target="$1"
+	# We still need to check the previous environment,
+	# because if it is the same as now, it wasn't cleaned yet!
+	if [ "$env_last" = "$THIS_ENVIRONMENT" ];then
+#		echo "ENV ---- I'm preparing again... but I need to clean!"
+		clean "$target"
+#	else
+#		echo "ENV ---- I'm preparing 'again'... but I wasn't run yet."
+	fi
+	prepare "$target"
+}
+
+usage()
+{
+echo "Usage: 
+$0 OPTION TARGET_DIR
+Where OPTION is one (and only one) of the following:
+    prepare-once, prepare-always, clean"
+}
+
+# arguments...
+if [ $# -ne 2 ];then
+	usage
+#	exit 1
+fi
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+		prepare-once) prepare_once "$2"; shift ;;
+		prepare-always) prepare_always "$2"; shift ;;
+		clean) unset ENVIRONMENT_LAST; clean "$2"; shift ;;
+		*) usage ; exit 1 ;;
+	esac
+	shift
+done
+
+exit 0
diff --git a/environments/fill90-dvd b/environments/fill90-dvd
new file mode 100644
index 0000000..cc3286f
--- /dev/null
+++ b/environments/fill90-dvd
@@ -0,0 +1,115 @@
+#!/bin/bash
+# FS QA Environment setup
+# This environment fills the target device from 90 percents 
+# with dvd-sized files.
+#-----------------------------------------------------------------------
+# Copyright 2014 (C) Red Hat, Inc., Jan Tulak <jtulak@redhat.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# arguments: prepare-once, prepare-always, clean
+
+
+THIS_ENVIRONMENT="$(basename $BASH_SOURCE)"
+
+files_prefix="env_dummy_"
+# maximum size of a file in kiB
+file_size=4194304 # 4 GiB
+
+# This function should prepare the environment
+prepare()
+{
+	target="$1"
+
+	available=$(df -P "$target"|tail -n1 |awk '{print $4}')
+	available=$(printf "%.0f" $(echo "$available*0.9"|bc )) # get 90 percent size
+
+	full_files=$((available / file_size)) # number of full-size files
+	last_file=$((available - (full_files * file_size) ))
+
+	for i in $(seq $full_files );do
+		f="$target/$files_prefix$i"
+		dd if=/dev/zero of="$f" count=$file_size bs=1024 &>/dev/null
+	done	
+
+	# last file to fill size smaller than $file_size
+	f="$target/$files_prefix""0"
+	dd if=/dev/zero of="$f" count=$last_file bs=1024 &>/dev/null
+}
+
+# This function should clean files created by prepare()
+clean()
+{
+	target="$1"
+	for f in $(ls "$target"|grep -E "^$files_prefix");do
+		rm "$target/$f"
+	done
+}
+
+
+
+
+# ----------------------------------------------------------------------
+# The following code usually don't need any changes for your environment
+# ----------------------------------------------------------------------
+
+prepare_once()
+{
+	target="$1"
+	# check if this environment is already created (was the last one
+	# running)
+	if [ "$env_last" != "$THIS_ENVIRONMENT" ];then
+		prepare "$target"
+	fi
+
+}
+
+prepare_always()
+{
+	target="$1"
+	# We still need to check the previous environment,
+	# because if it is the same as now, it wasn't cleaned yet!
+	if [ "$env_last" = "$THIS_ENVIRONMENT" ];then
+		clean "$target"
+	fi
+	prepare "$target"
+}
+
+usage()
+{
+echo "Usage: 
+$0 OPTION TARGET_DIR
+Where OPTION is one (and only one) of the following:
+    prepare-once, prepare-always, clean"
+}
+
+# arguments...
+if [ $# -ne 2 ];then
+	usage
+#	exit 1
+fi
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+		prepare-once) prepare_once "$2"; shift ;;
+		prepare-always) prepare_always "$2"; shift ;;
+		clean) unset ENVIRONMENT_LAST; clean "$2"; shift ;;
+		*) usage ; exit 1 ;;
+	esac
+	shift
+done
+
+exit 0
-- 
1.9.3

[PATCH 2/7] Created empty environment files in test categories

[Jan Ťulák]

This patch creates empty enviornment files in test categories, because
part of the underlying infrastructure is common with parsing group
files. So nonexistence of these files causes error messages.  (Althoug
it should not break anything)

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 tests/btrfs/environment   | 7 +++++++
 tests/ext4/environment    | 7 +++++++
 tests/generic/environment | 7 +++++++
 tests/shared/environment  | 7 +++++++
 tests/udf/environment     | 7 +++++++
 tests/xfs/environment     | 7 +++++++
 6 files changed, 42 insertions(+)
 create mode 100644 tests/btrfs/environment
 create mode 100644 tests/ext4/environment
 create mode 100644 tests/generic/environment
 create mode 100644 tests/shared/environment
 create mode 100644 tests/udf/environment
 create mode 100644 tests/xfs/environment

diff --git a/tests/btrfs/environment b/tests/btrfs/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/btrfs/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
diff --git a/tests/ext4/environment b/tests/ext4/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/ext4/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
diff --git a/tests/generic/environment b/tests/generic/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/generic/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
diff --git a/tests/shared/environment b/tests/shared/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/shared/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
diff --git a/tests/udf/environment b/tests/udf/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/udf/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
diff --git a/tests/xfs/environment b/tests/xfs/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/xfs/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
-- 
1.9.3

[PATCH 3/7] Created new performance test category

[Jan Ťulák]

This tests category is currently used only for demonstration of the environment setup, but later there are going to be performance tests.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check                         |  2 +-
 tests/performance/001         | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/performance/001.out     |  1 +
 tests/performance/002         | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/performance/002.out     |  1 +
 tests/performance/003         | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/performance/003.out     |  1 +
 tests/performance/004         | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/performance/004.out     |  1 +
 tests/performance/Makefile    | 21 +++++++++++++++
 tests/performance/environment | 11 ++++++++
 tests/performance/group       |  9 +++++++
 12 files changed, 298 insertions(+), 1 deletion(-)
 create mode 100755 tests/performance/001
 create mode 100644 tests/performance/001.out
 create mode 100755 tests/performance/002
 create mode 100644 tests/performance/002.out
 create mode 100755 tests/performance/003
 create mode 100644 tests/performance/003.out
 create mode 100755 tests/performance/004
 create mode 100644 tests/performance/004.out
 create mode 100644 tests/performance/Makefile
 create mode 100644 tests/performance/environment
 create mode 100644 tests/performance/group

diff --git a/check b/check
index 42a1ac2..a246d92 100755
--- a/check
+++ b/check
@@ -59,7 +59,7 @@ then
 fi
 
 SUPPORTED_TESTS="[0-9][0-9][0-9] [0-9][0-9][0-9][0-9]"
-SRC_GROUPS="generic shared"
+SRC_GROUPS="generic shared performance"
 export SRC_DIR="tests"
 
 usage()
diff --git a/tests/performance/001 b/tests/performance/001
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/001
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+	_environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/001.out b/tests/performance/001.out
new file mode 100644
index 0000000..097d046
--- /dev/null
+++ b/tests/performance/001.out
@@ -0,0 +1 @@
+QA output created by 001
diff --git a/tests/performance/002 b/tests/performance/002
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/002
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+	_environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/002.out b/tests/performance/002.out
new file mode 100644
index 0000000..c57ca23
--- /dev/null
+++ b/tests/performance/002.out
@@ -0,0 +1 @@
+QA output created by 002
diff --git a/tests/performance/003 b/tests/performance/003
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/003
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+	_environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/003.out b/tests/performance/003.out
new file mode 100644
index 0000000..e8cf537
--- /dev/null
+++ b/tests/performance/003.out
@@ -0,0 +1 @@
+QA output created by 003
diff --git a/tests/performance/004 b/tests/performance/004
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/004
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+	_environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/004.out b/tests/performance/004.out
new file mode 100644
index 0000000..8e2d391
--- /dev/null
+++ b/tests/performance/004.out
@@ -0,0 +1 @@
+QA output created by 004
diff --git a/tests/performance/Makefile b/tests/performance/Makefile
new file mode 100644
index 0000000..5a50b07
--- /dev/null
+++ b/tests/performance/Makefile
@@ -0,0 +1,21 @@
+#
+# Copyright (c) 2003-2005 Silicon Graphics, Inc.  All Rights Reserved.
+#
+
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+
+PERFORMANCE_DIR = performance
+TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(PERFORMANCE_DIR)
+
+include $(BUILDRULES)
+
+install:
+	$(INSTALL) -m 755 -d $(TARGET_DIR)
+	$(INSTALL) -m 755 [0-9]?? $(TARGET_DIR)
+	$(INSTALL) -m 644 group $(TARGET_DIR)
+	$(INSTALL) -m 644 environment $(TARGET_DIR)
+	$(INSTALL) -m 644 [0-9]??.* $(TARGET_DIR)
+
+# Nothing.
+install-dev install-lib:
diff --git a/tests/performance/environment b/tests/performance/environment
new file mode 100644
index 0000000..9fb2e06
--- /dev/null
+++ b/tests/performance/environment
@@ -0,0 +1,11 @@
+# QA setups control file
+# Defines test environments 
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+#   clean again, even if previously run test had the same environment. 
+#   So big-files and _big-files are the same, only the first one is
+#   prepared only once, while the second one is prepared every time.
+001 none fill90-dvd dummy1
+002 dummy1 dummy2
+003 dummy1 _dummy2
+004 _dummy1 _dummy2
diff --git a/tests/performance/group b/tests/performance/group
new file mode 100644
index 0000000..f5854b9
--- /dev/null
+++ b/tests/performance/group
@@ -0,0 +1,9 @@
+# QA groups control file
+# Defines test groups and nominal group owners
+# - do not start group names with a digit
+# - comment line before each group is "new" description
+#
+001 auto performance
+002 auto performance
+003 auto performance
+004 auto performance
-- 
1.9.3

[PATCH 4/7] get_group_list changed to more universal get_list_from_file

[Jan Ťulák]

This patch changes name of get_group_list and adds one argument to
make it useful not only for groups, but also for environments.  It
works the same, just it takes the file name as an argument instead of
hardcoded 'group'.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/check b/check
index a246d92..bdfd877 100755
--- a/check
+++ b/check
@@ -88,12 +88,15 @@ testlist options
 	    exit 0
 }
 
-get_group_list()
+# Get all tests tagged with specified group/environment
+# from a specified file.
+get_list_from_file()
 {
 	grp=$1
+	file=$2
 
 	for d in $SRC_GROUPS $FSTYP; do
-		l=$(sed -n < $SRC_DIR/$d/group \
+		l=$(sed -n < $SRC_DIR/$d/$file \
 			-e 's/#.*//' \
 			-e 's/$/ /' \
 			-e "s;\(^[0-9][0-9][0-9]\).* $grp .*;$SRC_DIR/$d/\1;p")
@@ -162,7 +165,7 @@ _prepare_test_list()
 
 	# Specified groups to include
 	for group in $GROUP_LIST; do
-		list=$(get_group_list $group)
+		list=$(get_list_from_file $group "group")
 		if [ -z "$list" ]; then
 			echo "Group \"$group\" is empty or not defined?"
 			exit 1
@@ -181,7 +184,7 @@ _prepare_test_list()
 
 	# Specified groups to exclude
 	for xgroup in $XGROUP_LIST; do
-		list=$(get_group_list $xgroup)
+		list=$(get_list_from_file $xgroup "group")
 		if [ -z "$list" ]; then
 			echo "Group \"$xgroup\" is empty or not defined?"
 			exit 1
-- 
1.9.3

[PATCH 5/7] Creates new functions for the entire environments functionality

[Jan Ťulák]

This patch contanins new functions for the environments and exports
directory with environment scripts. The most important function is
sort_tests_by_environment(), where the logic of pairing tests and
environments and sorting them happens.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 169 insertions(+)

diff --git a/check b/check
index bdfd877..777c7a2 100755
--- a/check
+++ b/check
@@ -61,6 +61,7 @@ fi
 SUPPORTED_TESTS="[0-9][0-9][0-9] [0-9][0-9][0-9][0-9]"
 SRC_GROUPS="generic shared performance"
 export SRC_DIR="tests"
+export ENV_DIR="environments"
 
 usage()
 {
@@ -88,6 +89,159 @@ testlist options
 	    exit 0
 }
 
+
+# Get intersect of two lists. Can work with any list of single-worded values.
+get_lists_intersect()
+{
+	local a="$1"
+	local b="$2"
+	local intersect=""
+
+	for item in $a;do
+        if [ $(echo $b|grep -cwE "_?$item") -gt 0 ];then
+			intersect="$intersect $item"
+		fi
+	done
+	echo $intersect
+}
+
+# Get symetric difference of two lists ($1 - $2).
+# Can work with any list of single-worded values.
+# Ignore prefixing underscore
+get_lists_difference()
+{
+	local a="$1"
+	local b="$2"
+	local difference=""
+
+	a=$(echo "$a" | sed -e "s/\b_//g")
+	b=$(echo "$b" | sed -e "s/\b_//g")
+	for item in $a;do
+        if [ $(echo $b|grep -cw $item) -eq 0 ];then
+			difference="$difference $item"
+		fi
+	done
+	echo $difference
+}
+
+# Find all environments with setup files in given sour group directory.
+get_environments()
+{
+	local env_list=""
+
+	# ommit "none", this one always has to exists and we want it add later
+	env_list=$(ls $ENV_DIR|grep -v "none")
+
+	echo "none $env_list"
+}
+
+# Check if all tests passed in first argument exists in a list passed
+# as a second argument.
+environments_test_existence()
+{
+	local specified  existing
+	specified="$1"
+	existing="$2"
+
+	nonexisting=$(get_lists_difference "$specified" "$existing")
+	if [ "$nonexisting" != "" ];then
+		echo "Unknown environment(s) were passed as an argument: $nonexisting"
+		exit 1
+	fi
+}
+
+
+# Sort tests by environment.
+# Duplicate tests that are in multiple environments.
+# Write the sorted tests into $tmp.list file,
+# and their environments into $tmp.list_env
+sort_tests_by_environment()
+{
+	active_tests="$*"
+	sorted_tests=""
+	sorted_envs=""
+	include_implicit=false
+	existing_environments=$(get_environments)
+
+	required_environments="${ENVIRONMENT_LIST/,/ }"
+	excluded_environments="${XENVIRONMENT_LIST/,/ }"
+
+	# test for nonexisting required
+	environments_test_existence \
+		"$required_environments $excluded_environments" \
+		"$existing_environments"
+
+	# filter environments based on -ex or -eo arguments
+	if [ "$required_environments" = "" ];then
+		# we have no explicit list of envs, so include all
+		required_environments="$existing_environments"
+		include_implicit=true
+
+	elif [ $(echo "$required_environments"|grep -cw "none") -gt 0 ]; then
+		# If there is an explicit list, but "none" is listed there,
+		# include implicit "none" too.
+		# Otherwise "none" is ignored.
+		include_implicit=true
+
+	fi
+
+	for xenv in $excluded_environments; do
+		required_environments=$(echo "$existing_environments" |\
+			sed "s/\b$xenv\b//g")
+
+		# do not include implicit none if explicitly blocked
+		if [ "$xenv" = "none" ];then
+			include_implicit=false
+		fi
+	done
+
+	# find nonexisting requested environments in files
+	from_files=$(get_all_groups_from_file "environment")
+	nonexisting=$(get_lists_difference "$from_files" "$existing_environments")
+	if [ "$nonexisting" != "" ];then
+		echo -n  "These environments are specified in some 'environment' file,"
+		echo " but do not exists: $nonexisting"
+		exit 1
+	fi
+
+	# sort tests by explicit environments
+	for e in $required_environments; do
+		# prefix is here for differentiate between -once and -always options
+		for prefix in "" "_";do
+			# get tests for the environment from the file
+			env_tests=$(get_list_from_file "$prefix$e" "environment") 2>/dev/null
+
+			# Get all active tests that are in this environment and put it 
+			# into a list.
+			env_active_tests=$(get_lists_intersect "$env_tests" "$active_tests")
+			sorted_tests="$sorted_tests $env_active_tests"
+			count=$(echo "$env_active_tests"|wc -w)
+
+			if [ "$count" -gt 0 ];then
+				my_tmp=$(printf "$prefix$e %.0s" $(seq $count))
+				sorted_envs="$sorted_envs $my_tmp"
+			fi
+		done
+	done
+
+
+	if $include_implicit ;then
+		unused=$(get_lists_difference "$active_tests" "$sorted_tests")
+		unused_count=$(echo "$unused"|wc -w)
+		if [ "$unused_count" -gt 0 ]; then
+			sorted_tests="$unused $sorted_tests"
+			my_tmp=$(printf "none %.0s" $(seq $unused_count))
+			sorted_envs="$my_tmp $sorted_envs"
+		fi
+	fi
+
+#	printf "Sorted environments: %s\n" "$sorted_envs"
+#	printf "Count of tests: %d, environments: %d\n" \
+#		$(echo "$sorted_tests"|wc -w) $(echo "$sorted_envs"|wc -w)
+	echo "$sorted_envs" > "$tmp.list_env" 2>/dev/null
+	echo "$sorted_tests" > "$tmp.list" 2>/dev/null
+}
+
 # Get all tests tagged with specified group/environment
 # from a specified file.
 get_list_from_file()
@@ -105,6 +259,21 @@ get_list_from_file()
 	echo $grpl
 }
 
+# Get all groups/environments from a specified file.
+get_all_groups_from_file()
+{
+	file="$1"
+
+	for d in $SRC_GROUPS $FSTYP; do
+		l=$(sed -n < $SRC_DIR/$d/$file \
+			-e 's/#.*//' \
+			-e 's/$/ /' \
+			-e "s;^[0-9][0-9][0-9]\(.*\);\1;p")
+		grpl="$grpl $l"
+	done
+	echo $grpl
+}
+
 # find all tests, excluding files that are test metadata such as group files.
 # This assumes that tests are defined purely by alphanumeric filenames with no
 # ".xyz" extensions in the name.
-- 
1.9.3

[PATCH 6/7] Integrates environment functionality into existing code

[Jan Ťulák]

This patch:
1) adds new arguments to --help and to the parser:
   -e                                 use testing environments
   -eo environment[,environment...]   test only in these environments
   -ex environment[,environment...]   exclude these environments

2) Adds call of sort_tests_by_environment() when -e is supplied.
   Currently, randomization of tests order and environments are
   mutually exclusive. I plan to add this later.

3) It slightly changes the main loop where tests are run. The change
is done because the tests and environments are paired in two variables
and it needs to access both of them. Here the required environment
name is exported and then when the test requires an environment, the
exported variable is used to determine the current one.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 51 insertions(+), 2 deletions(-)

diff --git a/check b/check
index 777c7a2..ca5082b 100755
--- a/check
+++ b/check
@@ -34,6 +34,7 @@ diff="diff -u"
 showme=false
 have_test_arg=false
 randomize=false
+environments_used=false
 export here=`pwd`
 xfile=""
 
@@ -76,12 +77,15 @@ check options
     -n			show me, do not run tests
     -T			output timestamps
     -r			randomize test order
+    -e			use testing environments
     --large-fs		optimise scratch device for large filesystems
     -s section		run only specified section from config file
 
 testlist options
     -g group[,group...]	include tests from these groups
     -x group[,group...]	exclude tests from these groups
+    -eo environment[,environment...]	test only in these environments
+    -ex environment[,environment...]	exclude these environments
     -X file		exclude individual tests
     -E external_file	exclude individual tests
     [testlist]		include tests matching names in testlist
@@ -283,7 +287,7 @@ get_all_tests()
 	for d in $SRC_GROUPS $FSTYP; do
 		ls $SRC_DIR/$d/* | \
 			grep -v "\..*" | \
-			grep -v "group\|Makefile" >> $tmp.list 2>/dev/null
+			grep -v "group\|Makefile\|environment" >> $tmp.list 2>/dev/null
 	done
 }
 
@@ -325,6 +329,7 @@ _timestamp()
 _prepare_test_list()
 {
 	unset list
+	unset list_env
 	# Tests specified on the command line
 	if [ -s $tmp.arglist ]; then
 		cat $tmp.arglist > $tmp.list
@@ -366,6 +371,15 @@ _prepare_test_list()
 	list=`sort -n $tmp.list | uniq`
 	rm -f $tmp.list $tmp.tmp $tmp.grep
 
+	if $environments_used
+	then
+		sort_tests_by_environment $list
+		list=$(cat $tmp.list)
+		list_env=$(cat $tmp.list_env)
+		rm -f $tmp.list $tmp.list_env
+		# TODO add randomizing also for environments
+	fi
+
 	if $randomize
 	then
 		list=`echo $list | awk -f randomize.awk`
@@ -389,6 +403,16 @@ while [ $# -gt 0 ]; do
 		XGROUP_LIST="$XGROUP_LIST $xgroup"
 		;;
 
+	-e)	environments_used=true ;;
+
+	-eo)	environment=$2 ; shift ;
+		ENVIRONMENT_LIST="$ENVIRONMENT_LIST $environment"
+		;;
+
+	-ex)	xenvironment=$2 ; shift ;
+		XENVIRONMENT_LIST="$XENVIRONMENT_LIST $xenvironment"
+		;;
+
 	-X)	xfile=$2; shift ;
 		for d in $SRC_GROUPS $FSTYP; do
 			[ -f $SRC_DIR/$d/$xfile ] || continue
@@ -429,6 +453,13 @@ while [ $# -gt 0 ]; do
 	shift
 done
 
+# Test we don't have test randomizing and environment testing at the same time
+if [ $environments_used = true -a $randomize = true ];then
+	>&2 echo "Environments can't be used at the same time"\
+		" as randomized test order!"
+	exit 1
+fi
+
 # Process tests from command line now.
 if $have_test_arg; then
 	while [ $# -gt 0 ]; do
@@ -665,8 +696,20 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	seqres="$check"
 	_check_test_fs
 
-	for seq in $list
+	read -a list_a <<< $list
+	read -a list_env_a <<< $list_env
+	for i in "${!list_a[@]}"
 	do
+		seq="${list_a[$i]}"
+		export env_last="$env"
+		export env="${list_env_a[$i]}"
+		export env_next="${list_env_a[$(($i+1))]}"
+		export env_force=false
+		if [ "${env:0:1}" = "_" ];then
+			env_force=true
+			env="${env#?}"
+		fi
+
 	    err=false
 
 	    # the filename for the test and the name output are different.
@@ -688,6 +731,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
 
 	    echo -n "$seqnum"
 
+		# If an environment is requested, print it.
+		# When not requested, output is not changed.
+		if [ "$env" != "" ];then
+			echo -n " [$env]"
+		fi
+
 	    if $showme
 	    then
 		echo
-- 
1.9.3

[PATCH 7/7] Adds functions for tests to call environment setup/cleaning

[Jan Ťulák]

This patch adds two functions into common/rc. These two functions are
called by tests to setup and clean environment. They have a single
argument - a target directory. So everything required in a test to
support environments is to call _require_environment $target after
requireing test or scratch dir, and then call the clean function at
the end.

Specific environment name is taken from variables exported by check
script.

Calling of the cleaning function is not critical as before setup,
cleaning is called automatically.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 common/rc | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/common/rc b/common/rc
index d5e3aff..14dbb1f 100644
--- a/common/rc
+++ b/common/rc
@@ -1037,6 +1037,55 @@ _supported_os()
     _notrun "not suitable for this OS: $HOSTOS"
 }
 
+_require_environment()
+{
+	target="$1"
+	if [ "$target" = "" ];then
+		1>&2 echo "Target directory for environment can't be empty! Aborting."
+		exit 1
+	fi
+
+	if [ "$env" != "" -a "$env" != "none" ];then
+		prepare_method="prepare-once"
+		if $env_force;then
+			prepare_method="prepare-always"
+		fi
+		bash ./environments/$env $prepare_method $target 
+		sts=$?
+		if [ "$sts" != 0 ]; then
+			echo "       [skipped]"
+			1>&2 echo "Failed to prepare environment $env "\
+				" (will skip the test)!"
+			1>&2 echo ""
+			exit 1
+		fi
+	fi
+}
+
+
+
+_environment_clean()
+{
+	target="$1"
+	if [ "$target" = "" ];then
+		1>&2 echo "Target directory for environment cleaning can't be empty!"\
+		" Aborting."
+		exit 1
+	fi
+	# Clean environment if there was some
+	# and the next environment is different.
+	# The variables are exported from check script.
+	if [ "$env" != "" -a "$env" != "none" -a "$env" != "$env_next" ];then
+		bash ./environments/$env clean $target
+		sts=$?
+		if [ "$sts" != "0" ];then
+			1>&2 echo "An error happened when cleaning environment $env!"
+			1>&2 echo ""
+		fi
+	fi
+
+}
+
 # this test needs a scratch partition - check we're ok & unmount it
 # No post-test check of the device is required. e.g. the test intentionally
 # finishes the test with the filesystem in a corrupt state
-- 
1.9.3

Re: [PATCH 1/7] Create environment setup files

[Dave Chinner]

On Fri, Nov 14, 2014 at 02:27:41PM +0100, Jan Ťulák wrote:
> First from a batch of patches for adding an environment support.  This
> description is rather long, as it describes the goal of all set, so a
> TLDR version at first:
> 
> - Allows to separate preparation of the environment (full fs,
>   damaged fs, ...) from a test itself. So multiple tests can use
>   exactly the same conditions.
> - A single test can be run in multiple environments.
> - Disabled by default for backward compatibility (it changes output).
> - I expect it will cause some debate. It is my first bigger patch
>   at all. :-)

I've had a bit of a look at the patchset. Very interesting but will
need a bit of work.

Seeing as this is your first major patchset, a few hints on how to
structure a large patchset to make it easier for reviewers to read:

	- this overall description belongs in a "patch 0" header

	- put simple, obvious fixes and refactoring patches first

	- don't add things before they are used (e.g. the dummy
	  files in the first patch) because reviewers can't see how
	  they fit into the overall picture until they've applied
	  later patches.

	- it's better to have actual functionality rather than dummy
	  place holders and templates. The code will change
	  significantly as you start to make actual use of it and
	  you solve all the problems a dummy or template don't
	  expose.

	- separate out new chunks of functionality into new files
	  e.g. all the list manipulation functions might be better
	  located in common/list where they can be shared rather
	  than in check.

Couple of things about the code:

	- please try to stick to 80 columns if possible.

	- some of the code uses 4 space tabs. When adding code into
	  such functions, please use 4 space tabs. New code should
	  use 8 space tabs, but only if it's not surrounded by code
	  that is using 4 space tabs.

	- really verbose variable names make the code hard to read.
	  e.g. $THIS_ENVIRONMENT is a long name, but I simply can't
	  tell what it's for from either it's name or it's usage.
	  $TEST_ENV is just as good, really...

	- using "_" prefixes in config files to change the behaviour
	  of the referenced test is pretty nasty. If there are
	  different behaviours needed, then the config file needs
	  to use explicit keywords for those behaviours. The only
	  use of the "_" prefix in xfstests is for prefixing
	  functions defined in the common/ code...

	- "2>&1 echo <foo>". What could echo possibly be sending to
	  stderr?



> Long version:
> 
> The goal of this set is to allow a single test to be run in different
> sitations, for example on empty filesystem, full fs, or damaged fs.
> It provides an interface for scripts that can prepare the requested
> environments and takes care of starting the test for each one.
> 
> Currently, this functionality needs to be enabled explicitely by a
> flag -e. It changes output slightly, so I saw this as neccessity.  The
> output change is because one test can be run multiple times in
> different environments, to note the combination. So when enabled,
> [env-name] is added: "xfs/001 [some-environment] 123s ... 456s"

Scope creep?

i.e. this isn't really what we discussed originally - we don't need
"environments" for the existing regression tests, and even if we do
this is not the way to go about grouping them. e.g. xfs/557, xfs/558
and xfs/559 might require the same setup, but as regression tests
they should not take more than a couple of minutes to run. Hence
the right way to do this is a generic setup function and, if
necessary, use the TEST_DIR to maintain a persistent environment
across tests.

> If the test is not aware of this new functionality, nothing changes
> for it, the test will run as usuall.
> 
> This is a part of my work on performance tests (they needs this sort
> of functionality), but is independent on them, so I'm proposing it
> now.
> 
> Of the seven patches, first three creates new files. Patches four to
> six modifies ./check script, but keeps the changes out of existing
> code as much as possible (patch four is only exception).  Patch seven
> is integrating it all together and is enabling the functionality.
> 
> To sum how it works:
> New file "environment", similar to "group" file, is created in each
> test category. It uses similar syntax, but it ortogonal to groups. In
> this file, each test can have specified one or more environments. When
> environments are enabled (./check -e ), list of tests is compiled as
> before (so -g, -x and other arguments works as usually) and for the
> enabled tests, environments are found.
> 
> If one test has multiple environments (and the selection is not
> limited for only some env.), the test is duplicated for each specified
> environment. Each run is then reported independently, as a combination
> of the test and the environment. When the test is not found in the
> file, it is added implicitly with "none" environment. The none
> environment do nothing and can be stated explicitly in the file also.

Hmm - yes, it is very different to what I thought we talked about.
I'll try to explain the way I see persistent performance test
environments fit into the existing infrastructure so you can see
the direction I was thinking of.

All we really require is a way of setting up a filesystem for
multiple performance tests, where setting up the test context might
take significantly longer than running the tests. I can see what you
are trying to do with the environment code, I'm just thinking that
it's a little over-engineered and trying to do too much.

Lets start with how a test would define the initial filesystem setup
it requires, and how it would trigger it to build and when we should
start our timing for measurement of the workload being benchmarked.
e.g.

....
. ./common/rc
. ./common/fsmark

FSMARK_FILES=10000
FSMARK_FILESIZE=4096
FSMARK_DIRS=100
FSMARK_THREADS=10
_scratch_build_fsmark_env

# real test starts now
_start_timing
.....

And _build_fsmark_env() does all the work of checking the
SCRATCH_MNT for existing test environment. e.g. the root directory
of the $SCRATCH_MNT contains a file created by the
_scratch_build_fsmark_env() function that contains the config used
to build it. It sources the config file, see if it matches the
config passed in by the test, and if it doesn't then we need to
rebuild the scratch device and the test environment according to the
current specification.

Indeed, we can turn the above into a create performance test via:

....
FSMARK_FILES=10000
FSMARK_FILESIZE=4096
FSMARK_DIRS=100
FSMARK_THREADS=10
FORCE_ENV_BUILD=true

# real test starts now
_start_timing
_scratch_build_fsmark_env
_stop_timing

status=0
exit

This doesn't require lots of new infrastructure and is way more
flexible than defining how tests are run/prepared in an external
file.  e.g. as you build tests it's trivial to simply group tests
that use the same environment together manually. Tests can still be
run randomly; it's just that they will need to create the
environment accordingly and so take longer to run.

In the longer term, I think it's better to change the common
infrastructure to support test names that aren't numbers and then
grouping of tests that use the same environment can all use the same
name prefix. e.g.

	peformance/fsmark-small-files-001
	peformance/fsmark-small-files-002
	peformance/fsmark-small-files-003
	peformance/fsmark-large-files-001
	peformance/fsmark-large-files-002
	peformance/fsmark-1m-empty-files-001
	peformance/fsmark-10m-empty-files-001
	peformance/fsmark-100m-empty-files-001
	peformance/fsmark-100m-empty-files-002
	.....

This makes sorting tests that use the same environment a very simple
thing whilst also providing other wishlist functionality we have for
the regression test side of fstests.  If we need common test setups
for regressions tests, then we can simply add the new regression
tests in exactly the same way.

As a result of this, we still use the existing group infrastructure
to control what performance tests are run. Hence there's no need for
explicit environments, CLI parameters to run them, cross-product
matrices of tests running in differnet environments, etc. i.e.

performance/group:
fsmark-small-files-001		fsmark small_files rw sequential
fsmark-small-files-002		fsmark small_files rw random
fsmark-small-files-003		fsmark small_files traverse
fsmark-small-files-004		fsmark small_files unlink
fsmark-large-files-001		fsmark large_files rw
fsmark-large-files-002		fsmark large_files unlink
fsmark-1m-empty-files-001	fsmark metadata scale create
fsmark-10m-empty-files-001	fsmark metadata scale create
fsmark-100m-empty-files-001	fsmark metadata scale create
fsmark-100m-empty-files-002	fsmark metadata scale traverse
fsmark-100m-empty-files-003	fsmark metadata scale unlink
.....

Hence:

# ./check -g fsmark

will run all those fsmark tests.

# ./check -g small_files

will run just the small file tests

# ./check -g fsmark -x scale

will run all the fsmark tests that aren't scalability tests.

That's how I've been thinking we should integrate persistent
filesystem state for performance tests, as well as the test script
interface and management should work. It is not as generic as your
environment concept, but I think it's simpler, more flexible and
easier to manage them a new set of wrappers around the outside of
the existing test infrastructure. I'm interested to see what you
think, Jan...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH 1/7] Create environment setup files

[Jan Tulak]

On Thu, 2014-11-20 at 10:06 +1100, Dave Chinner wrote:
> On Fri, Nov 14, 2014 at 02:27:41PM +0100, Jan Ťulák wrote:
> > First from a batch of patches for adding an environment support.  This
> > description is rather long, as it describes the goal of all set, so a
> > TLDR version at first:
> > 
> > - Allows to separate preparation of the environment (full fs,
> >   damaged fs, ...) from a test itself. So multiple tests can use
> >   exactly the same conditions.
> > - A single test can be run in multiple environments.
> > - Disabled by default for backward compatibility (it changes output).
> > - I expect it will cause some debate. It is my first bigger patch
> >   at all. :-)
> 
> I've had a bit of a look at the patchset. Very interesting but will
> need a bit of work.
> 
Thank you for your reply. :-)

> Seeing as this is your first major patchset, a few hints on how to
> structure a large patchset to make it easier for reviewers to read:
> 
> 	- this overall description belongs in a "patch 0" header
> 
> 	- put simple, obvious fixes and refactoring patches first
> 
> 	- don't add things before they are used (e.g. the dummy
> 	  files in the first patch) because reviewers can't see how
> 	  they fit into the overall picture until they've applied
> 	  later patches.
> 
> 	- it's better to have actual functionality rather than dummy
> 	  place holders and templates. The code will change
> 	  significantly as you start to make actual use of it and
> 	  you solve all the problems a dummy or template don't
> 	  expose.
> 
> 	- separate out new chunks of functionality into new files
> 	  e.g. all the list manipulation functions might be better
> 	  located in common/list where they can be shared rather
> 	  than in check.

I hope my next patch will have these things addressed. 

> 
> Couple of things about the code:
> 
> 	- please try to stick to 80 columns if possible.
I see I missed some lines. Sorry.

> 
> 	- some of the code uses 4 space tabs. When adding code into
> 	  such functions, please use 4 space tabs. New code should
> 	  use 8 space tabs, but only if it's not surrounded by code
> 	  that is using 4 space tabs.
> 
> 	- really verbose variable names make the code hard to read.
> 	  e.g. $THIS_ENVIRONMENT is a long name, but I simply can't
> 	  tell what it's for from either it's name or it's usage.
> 	  $TEST_ENV is just as good, really...
OK, I will watch for it.

> 
> 	- using "_" prefixes in config files to change the behaviour
> 	  of the referenced test is pretty nasty. If there are
> 	  different behaviours needed, then the config file needs
> 	  to use explicit keywords for those behaviours. The only
> 	  use of the "_" prefix in xfstests is for prefixing
> 	  functions defined in the common/ code...
I see it. I have to find a new way how to do it then. Maybe it could be
left directly on test - something like to call
_prepare_env_persistent/_fresh. 

> 	- "2>&1 echo <foo>". What could echo possibly be sending to
> 	  stderr?
I can't find this line. What I'm using on few places is 1>&2 to put some
error messages on stderr. Where did you found it inverted?

> 
> 
> 
> > Long version:
> > 
> > The goal of this set is to allow a single test to be run in different
> > sitations, for example on empty filesystem, full fs, or damaged fs.
> > It provides an interface for scripts that can prepare the requested
> > environments and takes care of starting the test for each one.
> > 
> > Currently, this functionality needs to be enabled explicitely by a
> > flag -e. It changes output slightly, so I saw this as neccessity.  The
> > output change is because one test can be run multiple times in
> > different environments, to note the combination. So when enabled,
> > [env-name] is added: "xfs/001 [some-environment] 123s ... 456s"
> 
> Scope creep?
> 
> i.e. this isn't really what we discussed originally - we don't need
> "environments" for the existing regression tests, and even if we do
> this is not the way to go about grouping them. e.g. xfs/557, xfs/558
> and xfs/559 might require the same setup, but as regression tests
> they should not take more than a couple of minutes to run. Hence
> the right way to do this is a generic setup function and, if
> necessary, use the TEST_DIR to maintain a persistent environment
> across tests.

Initially I didn't plan to use it also for existing regression tests,
just the result seemed to be simply applicable for them too. But I see
your point I think. For the regression tests, if something at all, it is
enough to simply pass few arguments to the generic setup function.
Everything more specific should stay inside of the tests.

> 
> > If the test is not aware of this new functionality, nothing changes
> > for it, the test will run as usuall.
> > 
> > This is a part of my work on performance tests (they needs this sort
> > of functionality), but is independent on them, so I'm proposing it
> > now.
> > 
> > Of the seven patches, first three creates new files. Patches four to
> > six modifies ./check script, but keeps the changes out of existing
> > code as much as possible (patch four is only exception).  Patch seven
> > is integrating it all together and is enabling the functionality.
> > 
> > To sum how it works:
> > New file "environment", similar to "group" file, is created in each
> > test category. It uses similar syntax, but it ortogonal to groups. In
> > this file, each test can have specified one or more environments. When
> > environments are enabled (./check -e ), list of tests is compiled as
> > before (so -g, -x and other arguments works as usually) and for the
> > enabled tests, environments are found.
> > 
> > If one test has multiple environments (and the selection is not
> > limited for only some env.), the test is duplicated for each specified
> > environment. Each run is then reported independently, as a combination
> > of the test and the environment. When the test is not found in the
> > file, it is added implicitly with "none" environment. The none
> > environment do nothing and can be stated explicitly in the file also.
> 
> Hmm - yes, it is very different to what I thought we talked about.
> I'll try to explain the way I see persistent performance test
> environments fit into the existing infrastructure so you can see
> the direction I was thinking of.
> 
> All we really require is a way of setting up a filesystem for
> multiple performance tests, where setting up the test context might
> take significantly longer than running the tests. I can see what you
> are trying to do with the environment code, I'm just thinking that
> it's a little over-engineered and trying to do too much.
> 
> Lets start with how a test would define the initial filesystem setup
> it requires, and how it would trigger it to build and when we should
> start our timing for measurement of the workload being benchmarked.
> e.g.
> 
> ....
> . ./common/rc
> . ./common/fsmark
> 
> FSMARK_FILES=10000
> FSMARK_FILESIZE=4096
> FSMARK_DIRS=100
> FSMARK_THREADS=10
> _scratch_build_fsmark_env
> 
> # real test starts now
> _start_timing
> .....
> 
> And _build_fsmark_env() does all the work of checking the
> SCRATCH_MNT for existing test environment. e.g. the root directory
> of the $SCRATCH_MNT contains a file created by the
> _scratch_build_fsmark_env() function that contains the config used
> to build it. It sources the config file, see if it matches the
> config passed in by the test, and if it doesn't then we need to
> rebuild the scratch device and the test environment according to the
> current specification.
> 
> Indeed, we can turn the above into a create performance test via:
> 
> ....
> FSMARK_FILES=10000
> FSMARK_FILESIZE=4096
> FSMARK_DIRS=100
> FSMARK_THREADS=10
> FORCE_ENV_BUILD=true
> 
> # real test starts now
> _start_timing
> _scratch_build_fsmark_env
> _stop_timing
> 
> status=0
> exit
> 
> This doesn't require lots of new infrastructure and is way more
> flexible than defining how tests are run/prepared in an external
> file.  e.g. as you build tests it's trivial to simply group tests
> that use the same environment together manually. Tests can still be
> run randomly; it's just that they will need to create the
> environment accordingly and so take longer to run.
> 
> In the longer term, I think it's better to change the common
> infrastructure to support test names that aren't numbers and then
> grouping of tests that use the same environment can all use the same
> name prefix. e.g.
> 
> 	peformance/fsmark-small-files-001
> 	peformance/fsmark-small-files-002
> 	peformance/fsmark-small-files-003
> 	peformance/fsmark-large-files-001
> 	peformance/fsmark-large-files-002replied
> 	peformance/fsmark-1m-empty-files-001
> 	peformance/fsmark-10m-empty-files-001
> 	peformance/fsmark-100m-empty-files-001
> 	peformance/fsmark-100m-empty-files-002
> 	.....
> 
> This makes sorting tests that use the same environment a very simple
> thing whilst also providing other wishlist functionality we have for
> the regression test side of fstests.  If we need common test setups
> for regressions tests, then we can simply add the new regression
> tests in exactly the same way.
> 
> As a result of this, we still use the existing group infrastructure
> to control what performance tests are run. Hence there's no need for
> explicit environments, CLI parameters to run them, cross-product
> matrices of tests running in differnet environments, etc. i.e.
> 
> performance/group:
> fsmark-small-files-001		fsmark small_files rw sequential
> fsmark-small-files-002		fsmark small_files rw random
> fsmark-small-files-003		fsmark small_files traverse
> fsmark-small-files-004		fsmark small_files unlink
> fsmark-large-files-001		fsmark large_files rw
> fsmark-large-files-002		fsmark large_files unlink
> fsmark-1m-empty-files-001	fsmark metadata scale create
> fsmark-10m-empty-files-001	fsmark metadata scale create
> fsmark-100m-empty-files-001	fsmark metadata scale create
> fsmark-100m-empty-files-002	fsmark metadata scale traverse
> fsmark-100m-empty-files-003	fsmark metadata scale unlink
> .....
> 
> Hence:
> 
> # ./check -g fsmark
> 
> will run all those fsmark tests.
> 
> # ./check -g small_files
> 
> will run just the small file tests
> 
> # ./check -g fsmark -x scale
> 
> will run all the fsmark tests that aren't scalability tests.
> 
> That's how I've been thinking we should integrate persistent
> filesystem state for performance tests, as well as the test script
> interface and management should work. It is not as generic as your
> environment concept, but I think it's simpler, more flexible and
> easier to manage them a new set of wrappers around the outside of
> the existing test infrastructure. I'm interested to see what you
> think, Jan...

Using fsmark for creating the files is a good idea. It brings a
dependency, but for the performance testing it is required as well, so
it shouldn't be a problem, right? And making customizable values for the
environments is good idea to. I think I even planned something like
this, just didn't thought about some standardized options, rather just
environment-specific.

About the random tests order - I planned to edit the awk shuffling
script once the environment code is settled. It is not a hard limit,
just it is not implemented yet.

Your idea is simpler than what I have done, but I'm missing one thing:
how to test something in multiple situations/environments without having
the test code duplicated? 

For what I see in your text, if I want to test the same thing with
small-files and big-files environment, I have to make it for one setup
and then make a copy and edit it... I don't like this duplicity. This is
the main point that lead me to my approach, to write everything only
once and then create any combination as is needed. But it is possible I
see what the tests should do from a different angle and I'm trying to
move some test responsibility to the environment. Can it be this?

I hope I answered to everything. :-)

Regards
Jan