* [PATCH 2/7] Created empty environment files in test categories
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
@ 2014-11-14 13:27 ` Jan Ťulák
2014-11-14 13:27 ` [PATCH 3/7] Created new performance test category Jan Ťulák
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Jan Ťulák @ 2014-11-14 13:27 UTC (permalink / raw)
To: fstests; +Cc: Jan Ťulák, lczerner, david
This patch creates empty enviornment files in test categories, because
part of the underlying infrastructure is common with parsing group
files. So nonexistence of these files causes error messages. (Althoug
it should not break anything)
Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
tests/btrfs/environment | 7 +++++++
tests/ext4/environment | 7 +++++++
tests/generic/environment | 7 +++++++
tests/shared/environment | 7 +++++++
tests/udf/environment | 7 +++++++
tests/xfs/environment | 7 +++++++
6 files changed, 42 insertions(+)
create mode 100644 tests/btrfs/environment
create mode 100644 tests/ext4/environment
create mode 100644 tests/generic/environment
create mode 100644 tests/shared/environment
create mode 100644 tests/udf/environment
create mode 100644 tests/xfs/environment
diff --git a/tests/btrfs/environment b/tests/btrfs/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/btrfs/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
diff --git a/tests/ext4/environment b/tests/ext4/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/ext4/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
diff --git a/tests/generic/environment b/tests/generic/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/generic/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
diff --git a/tests/shared/environment b/tests/shared/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/shared/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
diff --git a/tests/udf/environment b/tests/udf/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/udf/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
diff --git a/tests/xfs/environment b/tests/xfs/environment
new file mode 100644
index 0000000..238a8c2
--- /dev/null
+++ b/tests/xfs/environment
@@ -0,0 +1,7 @@
+# QA environments control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
--
1.9.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 3/7] Created new performance test category
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
2014-11-14 13:27 ` [PATCH 2/7] Created empty environment files in test categories Jan Ťulák
@ 2014-11-14 13:27 ` Jan Ťulák
2014-11-14 13:27 ` [PATCH 4/7] get_group_list changed to more universal get_list_from_file Jan Ťulák
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Jan Ťulák @ 2014-11-14 13:27 UTC (permalink / raw)
To: fstests; +Cc: Jan Ťulák, lczerner, david
This tests category is currently used only for demonstration of the environment setup, but later there are going to be performance tests.
Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
check | 2 +-
tests/performance/001 | 63 +++++++++++++++++++++++++++++++++++++++++++
tests/performance/001.out | 1 +
tests/performance/002 | 63 +++++++++++++++++++++++++++++++++++++++++++
tests/performance/002.out | 1 +
tests/performance/003 | 63 +++++++++++++++++++++++++++++++++++++++++++
tests/performance/003.out | 1 +
tests/performance/004 | 63 +++++++++++++++++++++++++++++++++++++++++++
tests/performance/004.out | 1 +
tests/performance/Makefile | 21 +++++++++++++++
tests/performance/environment | 11 ++++++++
tests/performance/group | 9 +++++++
12 files changed, 298 insertions(+), 1 deletion(-)
create mode 100755 tests/performance/001
create mode 100644 tests/performance/001.out
create mode 100755 tests/performance/002
create mode 100644 tests/performance/002.out
create mode 100755 tests/performance/003
create mode 100644 tests/performance/003.out
create mode 100755 tests/performance/004
create mode 100644 tests/performance/004.out
create mode 100644 tests/performance/Makefile
create mode 100644 tests/performance/environment
create mode 100644 tests/performance/group
diff --git a/check b/check
index 42a1ac2..a246d92 100755
--- a/check
+++ b/check
@@ -59,7 +59,7 @@ then
fi
SUPPORTED_TESTS="[0-9][0-9][0-9] [0-9][0-9][0-9][0-9]"
-SRC_GROUPS="generic shared"
+SRC_GROUPS="generic shared performance"
export SRC_DIR="tests"
usage()
diff --git a/tests/performance/001 b/tests/performance/001
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/001
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+ _environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/001.out b/tests/performance/001.out
new file mode 100644
index 0000000..097d046
--- /dev/null
+++ b/tests/performance/001.out
@@ -0,0 +1 @@
+QA output created by 001
diff --git a/tests/performance/002 b/tests/performance/002
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/002
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+ _environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/002.out b/tests/performance/002.out
new file mode 100644
index 0000000..c57ca23
--- /dev/null
+++ b/tests/performance/002.out
@@ -0,0 +1 @@
+QA output created by 002
diff --git a/tests/performance/003 b/tests/performance/003
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/003
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+ _environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/003.out b/tests/performance/003.out
new file mode 100644
index 0000000..e8cf537
--- /dev/null
+++ b/tests/performance/003.out
@@ -0,0 +1 @@
+QA output created by 003
diff --git a/tests/performance/004 b/tests/performance/004
new file mode 100755
index 0000000..d8f2dad
--- /dev/null
+++ b/tests/performance/004
@@ -0,0 +1,63 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Random file copier to produce chains of identical files so the head
+# and the tail can be diff'd at the end of each iteration.
+#
+# Exercises creat, write and unlink for a variety of directory sizes, and
+# checks for data corruption.
+#
+# run [config]
+#
+# config has one line per file with filename and byte size, else use
+# the default one below.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2000-2001 Silicon Graphics, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+tmp=/tmp/$$
+here=`pwd`
+status=1
+done_cleanup=false
+trap "_cleanup; rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# real QA test starts here
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_require_environment $TEST_DIR
+
+verbose=true
+
+
+
+_cleanup(){
+ _environment_clean $TEST_DIR
+}
+
+status=0
+exit
diff --git a/tests/performance/004.out b/tests/performance/004.out
new file mode 100644
index 0000000..8e2d391
--- /dev/null
+++ b/tests/performance/004.out
@@ -0,0 +1 @@
+QA output created by 004
diff --git a/tests/performance/Makefile b/tests/performance/Makefile
new file mode 100644
index 0000000..5a50b07
--- /dev/null
+++ b/tests/performance/Makefile
@@ -0,0 +1,21 @@
+#
+# Copyright (c) 2003-2005 Silicon Graphics, Inc. All Rights Reserved.
+#
+
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+
+PERFORMANCE_DIR = performance
+TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(PERFORMANCE_DIR)
+
+include $(BUILDRULES)
+
+install:
+ $(INSTALL) -m 755 -d $(TARGET_DIR)
+ $(INSTALL) -m 755 [0-9]?? $(TARGET_DIR)
+ $(INSTALL) -m 644 group $(TARGET_DIR)
+ $(INSTALL) -m 644 environment $(TARGET_DIR)
+ $(INSTALL) -m 644 [0-9]??.* $(TARGET_DIR)
+
+# Nothing.
+install-dev install-lib:
diff --git a/tests/performance/environment b/tests/performance/environment
new file mode 100644
index 0000000..9fb2e06
--- /dev/null
+++ b/tests/performance/environment
@@ -0,0 +1,11 @@
+# QA setups control file
+# Defines test environments
+# - Do not start environment names with a digit.
+# - If an environment is prefixed with underscore, the environment is prepared
+# clean again, even if previously run test had the same environment.
+# So big-files and _big-files are the same, only the first one is
+# prepared only once, while the second one is prepared every time.
+001 none fill90-dvd dummy1
+002 dummy1 dummy2
+003 dummy1 _dummy2
+004 _dummy1 _dummy2
diff --git a/tests/performance/group b/tests/performance/group
new file mode 100644
index 0000000..f5854b9
--- /dev/null
+++ b/tests/performance/group
@@ -0,0 +1,9 @@
+# QA groups control file
+# Defines test groups and nominal group owners
+# - do not start group names with a digit
+# - comment line before each group is "new" description
+#
+001 auto performance
+002 auto performance
+003 auto performance
+004 auto performance
--
1.9.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 4/7] get_group_list changed to more universal get_list_from_file
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
2014-11-14 13:27 ` [PATCH 2/7] Created empty environment files in test categories Jan Ťulák
2014-11-14 13:27 ` [PATCH 3/7] Created new performance test category Jan Ťulák
@ 2014-11-14 13:27 ` Jan Ťulák
2014-11-14 13:27 ` [PATCH 5/7] Creates new functions for the entire environments functionality Jan Ťulák
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Jan Ťulák @ 2014-11-14 13:27 UTC (permalink / raw)
To: fstests; +Cc: Jan Ťulák, lczerner, david
This patch changes name of get_group_list and adds one argument to
make it useful not only for groups, but also for environments. It
works the same, just it takes the file name as an argument instead of
hardcoded 'group'.
Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
check | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/check b/check
index a246d92..bdfd877 100755
--- a/check
+++ b/check
@@ -88,12 +88,15 @@ testlist options
exit 0
}
-get_group_list()
+# Get all tests tagged with specified group/environment
+# from a specified file.
+get_list_from_file()
{
grp=$1
+ file=$2
for d in $SRC_GROUPS $FSTYP; do
- l=$(sed -n < $SRC_DIR/$d/group \
+ l=$(sed -n < $SRC_DIR/$d/$file \
-e 's/#.*//' \
-e 's/$/ /' \
-e "s;\(^[0-9][0-9][0-9]\).* $grp .*;$SRC_DIR/$d/\1;p")
@@ -162,7 +165,7 @@ _prepare_test_list()
# Specified groups to include
for group in $GROUP_LIST; do
- list=$(get_group_list $group)
+ list=$(get_list_from_file $group "group")
if [ -z "$list" ]; then
echo "Group \"$group\" is empty or not defined?"
exit 1
@@ -181,7 +184,7 @@ _prepare_test_list()
# Specified groups to exclude
for xgroup in $XGROUP_LIST; do
- list=$(get_group_list $xgroup)
+ list=$(get_list_from_file $xgroup "group")
if [ -z "$list" ]; then
echo "Group \"$xgroup\" is empty or not defined?"
exit 1
--
1.9.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 5/7] Creates new functions for the entire environments functionality
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
` (2 preceding siblings ...)
2014-11-14 13:27 ` [PATCH 4/7] get_group_list changed to more universal get_list_from_file Jan Ťulák
@ 2014-11-14 13:27 ` Jan Ťulák
2014-11-14 13:27 ` [PATCH 6/7] Integrates environment functionality into existing code Jan Ťulák
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Jan Ťulák @ 2014-11-14 13:27 UTC (permalink / raw)
To: fstests; +Cc: Jan Ťulák, lczerner, david
This patch contanins new functions for the environments and exports
directory with environment scripts. The most important function is
sort_tests_by_environment(), where the logic of pairing tests and
environments and sorting them happens.
Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
check | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 169 insertions(+)
diff --git a/check b/check
index bdfd877..777c7a2 100755
--- a/check
+++ b/check
@@ -61,6 +61,7 @@ fi
SUPPORTED_TESTS="[0-9][0-9][0-9] [0-9][0-9][0-9][0-9]"
SRC_GROUPS="generic shared performance"
export SRC_DIR="tests"
+export ENV_DIR="environments"
usage()
{
@@ -88,6 +89,159 @@ testlist options
exit 0
}
+
+# Get intersect of two lists. Can work with any list of single-worded values.
+get_lists_intersect()
+{
+ local a="$1"
+ local b="$2"
+ local intersect=""
+
+ for item in $a;do
+ if [ $(echo $b|grep -cwE "_?$item") -gt 0 ];then
+ intersect="$intersect $item"
+ fi
+ done
+ echo $intersect
+}
+
+# Get symetric difference of two lists ($1 - $2).
+# Can work with any list of single-worded values.
+# Ignore prefixing underscore
+get_lists_difference()
+{
+ local a="$1"
+ local b="$2"
+ local difference=""
+
+ a=$(echo "$a" | sed -e "s/\b_//g")
+ b=$(echo "$b" | sed -e "s/\b_//g")
+ for item in $a;do
+ if [ $(echo $b|grep -cw $item) -eq 0 ];then
+ difference="$difference $item"
+ fi
+ done
+ echo $difference
+}
+
+# Find all environments with setup files in given sour group directory.
+get_environments()
+{
+ local env_list=""
+
+ # ommit "none", this one always has to exists and we want it add later
+ env_list=$(ls $ENV_DIR|grep -v "none")
+
+ echo "none $env_list"
+}
+
+# Check if all tests passed in first argument exists in a list passed
+# as a second argument.
+environments_test_existence()
+{
+ local specified existing
+ specified="$1"
+ existing="$2"
+
+ nonexisting=$(get_lists_difference "$specified" "$existing")
+ if [ "$nonexisting" != "" ];then
+ echo "Unknown environment(s) were passed as an argument: $nonexisting"
+ exit 1
+ fi
+}
+
+
+# Sort tests by environment.
+# Duplicate tests that are in multiple environments.
+# Write the sorted tests into $tmp.list file,
+# and their environments into $tmp.list_env
+sort_tests_by_environment()
+{
+ active_tests="$*"
+ sorted_tests=""
+ sorted_envs=""
+ include_implicit=false
+ existing_environments=$(get_environments)
+
+ required_environments="${ENVIRONMENT_LIST/,/ }"
+ excluded_environments="${XENVIRONMENT_LIST/,/ }"
+
+ # test for nonexisting required
+ environments_test_existence \
+ "$required_environments $excluded_environments" \
+ "$existing_environments"
+
+ # filter environments based on -ex or -eo arguments
+ if [ "$required_environments" = "" ];then
+ # we have no explicit list of envs, so include all
+ required_environments="$existing_environments"
+ include_implicit=true
+
+ elif [ $(echo "$required_environments"|grep -cw "none") -gt 0 ]; then
+ # If there is an explicit list, but "none" is listed there,
+ # include implicit "none" too.
+ # Otherwise "none" is ignored.
+ include_implicit=true
+
+ fi
+
+ for xenv in $excluded_environments; do
+ required_environments=$(echo "$existing_environments" |\
+ sed "s/\b$xenv\b//g")
+
+ # do not include implicit none if explicitly blocked
+ if [ "$xenv" = "none" ];then
+ include_implicit=false
+ fi
+ done
+
+ # find nonexisting requested environments in files
+ from_files=$(get_all_groups_from_file "environment")
+ nonexisting=$(get_lists_difference "$from_files" "$existing_environments")
+ if [ "$nonexisting" != "" ];then
+ echo -n "These environments are specified in some 'environment' file,"
+ echo " but do not exists: $nonexisting"
+ exit 1
+ fi
+
+ # sort tests by explicit environments
+ for e in $required_environments; do
+ # prefix is here for differentiate between -once and -always options
+ for prefix in "" "_";do
+ # get tests for the environment from the file
+ env_tests=$(get_list_from_file "$prefix$e" "environment") 2>/dev/null
+
+ # Get all active tests that are in this environment and put it
+ # into a list.
+ env_active_tests=$(get_lists_intersect "$env_tests" "$active_tests")
+ sorted_tests="$sorted_tests $env_active_tests"
+ count=$(echo "$env_active_tests"|wc -w)
+
+ if [ "$count" -gt 0 ];then
+ my_tmp=$(printf "$prefix$e %.0s" $(seq $count))
+ sorted_envs="$sorted_envs $my_tmp"
+ fi
+ done
+ done
+
+
+ if $include_implicit ;then
+ unused=$(get_lists_difference "$active_tests" "$sorted_tests")
+ unused_count=$(echo "$unused"|wc -w)
+ if [ "$unused_count" -gt 0 ]; then
+ sorted_tests="$unused $sorted_tests"
+ my_tmp=$(printf "none %.0s" $(seq $unused_count))
+ sorted_envs="$my_tmp $sorted_envs"
+ fi
+ fi
+
+# printf "Sorted environments: %s\n" "$sorted_envs"
+# printf "Count of tests: %d, environments: %d\n" \
+# $(echo "$sorted_tests"|wc -w) $(echo "$sorted_envs"|wc -w)
+ echo "$sorted_envs" > "$tmp.list_env" 2>/dev/null
+ echo "$sorted_tests" > "$tmp.list" 2>/dev/null
+}
+
# Get all tests tagged with specified group/environment
# from a specified file.
get_list_from_file()
@@ -105,6 +259,21 @@ get_list_from_file()
echo $grpl
}
+# Get all groups/environments from a specified file.
+get_all_groups_from_file()
+{
+ file="$1"
+
+ for d in $SRC_GROUPS $FSTYP; do
+ l=$(sed -n < $SRC_DIR/$d/$file \
+ -e 's/#.*//' \
+ -e 's/$/ /' \
+ -e "s;^[0-9][0-9][0-9]\(.*\);\1;p")
+ grpl="$grpl $l"
+ done
+ echo $grpl
+}
+
# find all tests, excluding files that are test metadata such as group files.
# This assumes that tests are defined purely by alphanumeric filenames with no
# ".xyz" extensions in the name.
--
1.9.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 6/7] Integrates environment functionality into existing code
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
` (3 preceding siblings ...)
2014-11-14 13:27 ` [PATCH 5/7] Creates new functions for the entire environments functionality Jan Ťulák
@ 2014-11-14 13:27 ` Jan Ťulák
2014-11-14 13:27 ` [PATCH 7/7] Adds functions for tests to call environment setup/cleaning Jan Ťulák
2014-11-19 23:06 ` [PATCH 1/7] Create environment setup files Dave Chinner
6 siblings, 0 replies; 9+ messages in thread
From: Jan Ťulák @ 2014-11-14 13:27 UTC (permalink / raw)
To: fstests; +Cc: Jan Ťulák, lczerner, david
This patch:
1) adds new arguments to --help and to the parser:
-e use testing environments
-eo environment[,environment...] test only in these environments
-ex environment[,environment...] exclude these environments
2) Adds call of sort_tests_by_environment() when -e is supplied.
Currently, randomization of tests order and environments are
mutually exclusive. I plan to add this later.
3) It slightly changes the main loop where tests are run. The change
is done because the tests and environments are paired in two variables
and it needs to access both of them. Here the required environment
name is exported and then when the test requires an environment, the
exported variable is used to determine the current one.
Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
check | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 2 deletions(-)
diff --git a/check b/check
index 777c7a2..ca5082b 100755
--- a/check
+++ b/check
@@ -34,6 +34,7 @@ diff="diff -u"
showme=false
have_test_arg=false
randomize=false
+environments_used=false
export here=`pwd`
xfile=""
@@ -76,12 +77,15 @@ check options
-n show me, do not run tests
-T output timestamps
-r randomize test order
+ -e use testing environments
--large-fs optimise scratch device for large filesystems
-s section run only specified section from config file
testlist options
-g group[,group...] include tests from these groups
-x group[,group...] exclude tests from these groups
+ -eo environment[,environment...] test only in these environments
+ -ex environment[,environment...] exclude these environments
-X file exclude individual tests
-E external_file exclude individual tests
[testlist] include tests matching names in testlist
@@ -283,7 +287,7 @@ get_all_tests()
for d in $SRC_GROUPS $FSTYP; do
ls $SRC_DIR/$d/* | \
grep -v "\..*" | \
- grep -v "group\|Makefile" >> $tmp.list 2>/dev/null
+ grep -v "group\|Makefile\|environment" >> $tmp.list 2>/dev/null
done
}
@@ -325,6 +329,7 @@ _timestamp()
_prepare_test_list()
{
unset list
+ unset list_env
# Tests specified on the command line
if [ -s $tmp.arglist ]; then
cat $tmp.arglist > $tmp.list
@@ -366,6 +371,15 @@ _prepare_test_list()
list=`sort -n $tmp.list | uniq`
rm -f $tmp.list $tmp.tmp $tmp.grep
+ if $environments_used
+ then
+ sort_tests_by_environment $list
+ list=$(cat $tmp.list)
+ list_env=$(cat $tmp.list_env)
+ rm -f $tmp.list $tmp.list_env
+ # TODO add randomizing also for environments
+ fi
+
if $randomize
then
list=`echo $list | awk -f randomize.awk`
@@ -389,6 +403,16 @@ while [ $# -gt 0 ]; do
XGROUP_LIST="$XGROUP_LIST $xgroup"
;;
+ -e) environments_used=true ;;
+
+ -eo) environment=$2 ; shift ;
+ ENVIRONMENT_LIST="$ENVIRONMENT_LIST $environment"
+ ;;
+
+ -ex) xenvironment=$2 ; shift ;
+ XENVIRONMENT_LIST="$XENVIRONMENT_LIST $xenvironment"
+ ;;
+
-X) xfile=$2; shift ;
for d in $SRC_GROUPS $FSTYP; do
[ -f $SRC_DIR/$d/$xfile ] || continue
@@ -429,6 +453,13 @@ while [ $# -gt 0 ]; do
shift
done
+# Test we don't have test randomizing and environment testing at the same time
+if [ $environments_used = true -a $randomize = true ];then
+ >&2 echo "Environments can't be used at the same time"\
+ " as randomized test order!"
+ exit 1
+fi
+
# Process tests from command line now.
if $have_test_arg; then
while [ $# -gt 0 ]; do
@@ -665,8 +696,20 @@ for section in $HOST_OPTIONS_SECTIONS; do
seqres="$check"
_check_test_fs
- for seq in $list
+ read -a list_a <<< $list
+ read -a list_env_a <<< $list_env
+ for i in "${!list_a[@]}"
do
+ seq="${list_a[$i]}"
+ export env_last="$env"
+ export env="${list_env_a[$i]}"
+ export env_next="${list_env_a[$(($i+1))]}"
+ export env_force=false
+ if [ "${env:0:1}" = "_" ];then
+ env_force=true
+ env="${env#?}"
+ fi
+
err=false
# the filename for the test and the name output are different.
@@ -688,6 +731,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
echo -n "$seqnum"
+ # If an environment is requested, print it.
+ # When not requested, output is not changed.
+ if [ "$env" != "" ];then
+ echo -n " [$env]"
+ fi
+
if $showme
then
echo
--
1.9.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 7/7] Adds functions for tests to call environment setup/cleaning
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
` (4 preceding siblings ...)
2014-11-14 13:27 ` [PATCH 6/7] Integrates environment functionality into existing code Jan Ťulák
@ 2014-11-14 13:27 ` Jan Ťulák
2014-11-19 23:06 ` [PATCH 1/7] Create environment setup files Dave Chinner
6 siblings, 0 replies; 9+ messages in thread
From: Jan Ťulák @ 2014-11-14 13:27 UTC (permalink / raw)
To: fstests; +Cc: Jan Ťulák, lczerner, david
This patch adds two functions into common/rc. These two functions are
called by tests to setup and clean environment. They have a single
argument - a target directory. So everything required in a test to
support environments is to call _require_environment $target after
requireing test or scratch dir, and then call the clean function at
the end.
Specific environment name is taken from variables exported by check
script.
Calling of the cleaning function is not critical as before setup,
cleaning is called automatically.
Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
common/rc | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)
diff --git a/common/rc b/common/rc
index d5e3aff..14dbb1f 100644
--- a/common/rc
+++ b/common/rc
@@ -1037,6 +1037,55 @@ _supported_os()
_notrun "not suitable for this OS: $HOSTOS"
}
+_require_environment()
+{
+ target="$1"
+ if [ "$target" = "" ];then
+ 1>&2 echo "Target directory for environment can't be empty! Aborting."
+ exit 1
+ fi
+
+ if [ "$env" != "" -a "$env" != "none" ];then
+ prepare_method="prepare-once"
+ if $env_force;then
+ prepare_method="prepare-always"
+ fi
+ bash ./environments/$env $prepare_method $target
+ sts=$?
+ if [ "$sts" != 0 ]; then
+ echo " [skipped]"
+ 1>&2 echo "Failed to prepare environment $env "\
+ " (will skip the test)!"
+ 1>&2 echo ""
+ exit 1
+ fi
+ fi
+}
+
+
+
+_environment_clean()
+{
+ target="$1"
+ if [ "$target" = "" ];then
+ 1>&2 echo "Target directory for environment cleaning can't be empty!"\
+ " Aborting."
+ exit 1
+ fi
+ # Clean environment if there was some
+ # and the next environment is different.
+ # The variables are exported from check script.
+ if [ "$env" != "" -a "$env" != "none" -a "$env" != "$env_next" ];then
+ bash ./environments/$env clean $target
+ sts=$?
+ if [ "$sts" != "0" ];then
+ 1>&2 echo "An error happened when cleaning environment $env!"
+ 1>&2 echo ""
+ fi
+ fi
+
+}
+
# this test needs a scratch partition - check we're ok & unmount it
# No post-test check of the device is required. e.g. the test intentionally
# finishes the test with the filesystem in a corrupt state
--
1.9.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 1/7] Create environment setup files
2014-11-14 13:27 [PATCH 1/7] Create environment setup files Jan Ťulák
` (5 preceding siblings ...)
2014-11-14 13:27 ` [PATCH 7/7] Adds functions for tests to call environment setup/cleaning Jan Ťulák
@ 2014-11-19 23:06 ` Dave Chinner
2014-11-21 13:34 ` Jan Tulak
6 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2014-11-19 23:06 UTC (permalink / raw)
To: Jan Ťulák; +Cc: fstests, lczerner
On Fri, Nov 14, 2014 at 02:27:41PM +0100, Jan Ťulák wrote:
> First from a batch of patches for adding an environment support. This
> description is rather long, as it describes the goal of all set, so a
> TLDR version at first:
>
> - Allows to separate preparation of the environment (full fs,
> damaged fs, ...) from a test itself. So multiple tests can use
> exactly the same conditions.
> - A single test can be run in multiple environments.
> - Disabled by default for backward compatibility (it changes output).
> - I expect it will cause some debate. It is my first bigger patch
> at all. :-)
I've had a bit of a look at the patchset. Very interesting but will
need a bit of work.
Seeing as this is your first major patchset, a few hints on how to
structure a large patchset to make it easier for reviewers to read:
- this overall description belongs in a "patch 0" header
- put simple, obvious fixes and refactoring patches first
- don't add things before they are used (e.g. the dummy
files in the first patch) because reviewers can't see how
they fit into the overall picture until they've applied
later patches.
- it's better to have actual functionality rather than dummy
place holders and templates. The code will change
significantly as you start to make actual use of it and
you solve all the problems a dummy or template don't
expose.
- separate out new chunks of functionality into new files
e.g. all the list manipulation functions might be better
located in common/list where they can be shared rather
than in check.
Couple of things about the code:
- please try to stick to 80 columns if possible.
- some of the code uses 4 space tabs. When adding code into
such functions, please use 4 space tabs. New code should
use 8 space tabs, but only if it's not surrounded by code
that is using 4 space tabs.
- really verbose variable names make the code hard to read.
e.g. $THIS_ENVIRONMENT is a long name, but I simply can't
tell what it's for from either it's name or it's usage.
$TEST_ENV is just as good, really...
- using "_" prefixes in config files to change the behaviour
of the referenced test is pretty nasty. If there are
different behaviours needed, then the config file needs
to use explicit keywords for those behaviours. The only
use of the "_" prefix in xfstests is for prefixing
functions defined in the common/ code...
- "2>&1 echo <foo>". What could echo possibly be sending to
stderr?
> Long version:
>
> The goal of this set is to allow a single test to be run in different
> sitations, for example on empty filesystem, full fs, or damaged fs.
> It provides an interface for scripts that can prepare the requested
> environments and takes care of starting the test for each one.
>
> Currently, this functionality needs to be enabled explicitely by a
> flag -e. It changes output slightly, so I saw this as neccessity. The
> output change is because one test can be run multiple times in
> different environments, to note the combination. So when enabled,
> [env-name] is added: "xfs/001 [some-environment] 123s ... 456s"
Scope creep?
i.e. this isn't really what we discussed originally - we don't need
"environments" for the existing regression tests, and even if we do
this is not the way to go about grouping them. e.g. xfs/557, xfs/558
and xfs/559 might require the same setup, but as regression tests
they should not take more than a couple of minutes to run. Hence
the right way to do this is a generic setup function and, if
necessary, use the TEST_DIR to maintain a persistent environment
across tests.
> If the test is not aware of this new functionality, nothing changes
> for it, the test will run as usuall.
>
> This is a part of my work on performance tests (they needs this sort
> of functionality), but is independent on them, so I'm proposing it
> now.
>
> Of the seven patches, first three creates new files. Patches four to
> six modifies ./check script, but keeps the changes out of existing
> code as much as possible (patch four is only exception). Patch seven
> is integrating it all together and is enabling the functionality.
>
> To sum how it works:
> New file "environment", similar to "group" file, is created in each
> test category. It uses similar syntax, but it ortogonal to groups. In
> this file, each test can have specified one or more environments. When
> environments are enabled (./check -e ), list of tests is compiled as
> before (so -g, -x and other arguments works as usually) and for the
> enabled tests, environments are found.
>
> If one test has multiple environments (and the selection is not
> limited for only some env.), the test is duplicated for each specified
> environment. Each run is then reported independently, as a combination
> of the test and the environment. When the test is not found in the
> file, it is added implicitly with "none" environment. The none
> environment do nothing and can be stated explicitly in the file also.
Hmm - yes, it is very different to what I thought we talked about.
I'll try to explain the way I see persistent performance test
environments fit into the existing infrastructure so you can see
the direction I was thinking of.
All we really require is a way of setting up a filesystem for
multiple performance tests, where setting up the test context might
take significantly longer than running the tests. I can see what you
are trying to do with the environment code, I'm just thinking that
it's a little over-engineered and trying to do too much.
Lets start with how a test would define the initial filesystem setup
it requires, and how it would trigger it to build and when we should
start our timing for measurement of the workload being benchmarked.
e.g.
....
. ./common/rc
. ./common/fsmark
FSMARK_FILES=10000
FSMARK_FILESIZE=4096
FSMARK_DIRS=100
FSMARK_THREADS=10
_scratch_build_fsmark_env
# real test starts now
_start_timing
.....
And _build_fsmark_env() does all the work of checking the
SCRATCH_MNT for existing test environment. e.g. the root directory
of the $SCRATCH_MNT contains a file created by the
_scratch_build_fsmark_env() function that contains the config used
to build it. It sources the config file, see if it matches the
config passed in by the test, and if it doesn't then we need to
rebuild the scratch device and the test environment according to the
current specification.
Indeed, we can turn the above into a create performance test via:
....
FSMARK_FILES=10000
FSMARK_FILESIZE=4096
FSMARK_DIRS=100
FSMARK_THREADS=10
FORCE_ENV_BUILD=true
# real test starts now
_start_timing
_scratch_build_fsmark_env
_stop_timing
status=0
exit
This doesn't require lots of new infrastructure and is way more
flexible than defining how tests are run/prepared in an external
file. e.g. as you build tests it's trivial to simply group tests
that use the same environment together manually. Tests can still be
run randomly; it's just that they will need to create the
environment accordingly and so take longer to run.
In the longer term, I think it's better to change the common
infrastructure to support test names that aren't numbers and then
grouping of tests that use the same environment can all use the same
name prefix. e.g.
peformance/fsmark-small-files-001
peformance/fsmark-small-files-002
peformance/fsmark-small-files-003
peformance/fsmark-large-files-001
peformance/fsmark-large-files-002
peformance/fsmark-1m-empty-files-001
peformance/fsmark-10m-empty-files-001
peformance/fsmark-100m-empty-files-001
peformance/fsmark-100m-empty-files-002
.....
This makes sorting tests that use the same environment a very simple
thing whilst also providing other wishlist functionality we have for
the regression test side of fstests. If we need common test setups
for regressions tests, then we can simply add the new regression
tests in exactly the same way.
As a result of this, we still use the existing group infrastructure
to control what performance tests are run. Hence there's no need for
explicit environments, CLI parameters to run them, cross-product
matrices of tests running in differnet environments, etc. i.e.
performance/group:
fsmark-small-files-001 fsmark small_files rw sequential
fsmark-small-files-002 fsmark small_files rw random
fsmark-small-files-003 fsmark small_files traverse
fsmark-small-files-004 fsmark small_files unlink
fsmark-large-files-001 fsmark large_files rw
fsmark-large-files-002 fsmark large_files unlink
fsmark-1m-empty-files-001 fsmark metadata scale create
fsmark-10m-empty-files-001 fsmark metadata scale create
fsmark-100m-empty-files-001 fsmark metadata scale create
fsmark-100m-empty-files-002 fsmark metadata scale traverse
fsmark-100m-empty-files-003 fsmark metadata scale unlink
.....
Hence:
# ./check -g fsmark
will run all those fsmark tests.
# ./check -g small_files
will run just the small file tests
# ./check -g fsmark -x scale
will run all the fsmark tests that aren't scalability tests.
That's how I've been thinking we should integrate persistent
filesystem state for performance tests, as well as the test script
interface and management should work. It is not as generic as your
environment concept, but I think it's simpler, more flexible and
easier to manage them a new set of wrappers around the outside of
the existing test infrastructure. I'm interested to see what you
think, Jan...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH 1/7] Create environment setup files
2014-11-19 23:06 ` [PATCH 1/7] Create environment setup files Dave Chinner
@ 2014-11-21 13:34 ` Jan Tulak
0 siblings, 0 replies; 9+ messages in thread
From: Jan Tulak @ 2014-11-21 13:34 UTC (permalink / raw)
To: Dave Chinner; +Cc: fstests, lczerner
On Thu, 2014-11-20 at 10:06 +1100, Dave Chinner wrote:
> On Fri, Nov 14, 2014 at 02:27:41PM +0100, Jan Ťulák wrote:
> > First from a batch of patches for adding an environment support. This
> > description is rather long, as it describes the goal of all set, so a
> > TLDR version at first:
> >
> > - Allows to separate preparation of the environment (full fs,
> > damaged fs, ...) from a test itself. So multiple tests can use
> > exactly the same conditions.
> > - A single test can be run in multiple environments.
> > - Disabled by default for backward compatibility (it changes output).
> > - I expect it will cause some debate. It is my first bigger patch
> > at all. :-)
>
> I've had a bit of a look at the patchset. Very interesting but will
> need a bit of work.
>
Thank you for your reply. :-)
> Seeing as this is your first major patchset, a few hints on how to
> structure a large patchset to make it easier for reviewers to read:
>
> - this overall description belongs in a "patch 0" header
>
> - put simple, obvious fixes and refactoring patches first
>
> - don't add things before they are used (e.g. the dummy
> files in the first patch) because reviewers can't see how
> they fit into the overall picture until they've applied
> later patches.
>
> - it's better to have actual functionality rather than dummy
> place holders and templates. The code will change
> significantly as you start to make actual use of it and
> you solve all the problems a dummy or template don't
> expose.
>
> - separate out new chunks of functionality into new files
> e.g. all the list manipulation functions might be better
> located in common/list where they can be shared rather
> than in check.
I hope my next patch will have these things addressed.
>
> Couple of things about the code:
>
> - please try to stick to 80 columns if possible.
I see I missed some lines. Sorry.
>
> - some of the code uses 4 space tabs. When adding code into
> such functions, please use 4 space tabs. New code should
> use 8 space tabs, but only if it's not surrounded by code
> that is using 4 space tabs.
>
> - really verbose variable names make the code hard to read.
> e.g. $THIS_ENVIRONMENT is a long name, but I simply can't
> tell what it's for from either it's name or it's usage.
> $TEST_ENV is just as good, really...
OK, I will watch for it.
>
> - using "_" prefixes in config files to change the behaviour
> of the referenced test is pretty nasty. If there are
> different behaviours needed, then the config file needs
> to use explicit keywords for those behaviours. The only
> use of the "_" prefix in xfstests is for prefixing
> functions defined in the common/ code...
I see it. I have to find a new way how to do it then. Maybe it could be
left directly on test - something like to call
_prepare_env_persistent/_fresh.
> - "2>&1 echo <foo>". What could echo possibly be sending to
> stderr?
I can't find this line. What I'm using on few places is 1>&2 to put some
error messages on stderr. Where did you found it inverted?
>
>
>
> > Long version:
> >
> > The goal of this set is to allow a single test to be run in different
> > sitations, for example on empty filesystem, full fs, or damaged fs.
> > It provides an interface for scripts that can prepare the requested
> > environments and takes care of starting the test for each one.
> >
> > Currently, this functionality needs to be enabled explicitely by a
> > flag -e. It changes output slightly, so I saw this as neccessity. The
> > output change is because one test can be run multiple times in
> > different environments, to note the combination. So when enabled,
> > [env-name] is added: "xfs/001 [some-environment] 123s ... 456s"
>
> Scope creep?
>
> i.e. this isn't really what we discussed originally - we don't need
> "environments" for the existing regression tests, and even if we do
> this is not the way to go about grouping them. e.g. xfs/557, xfs/558
> and xfs/559 might require the same setup, but as regression tests
> they should not take more than a couple of minutes to run. Hence
> the right way to do this is a generic setup function and, if
> necessary, use the TEST_DIR to maintain a persistent environment
> across tests.
Initially I didn't plan to use it also for existing regression tests,
just the result seemed to be simply applicable for them too. But I see
your point I think. For the regression tests, if something at all, it is
enough to simply pass few arguments to the generic setup function.
Everything more specific should stay inside of the tests.
>
> > If the test is not aware of this new functionality, nothing changes
> > for it, the test will run as usuall.
> >
> > This is a part of my work on performance tests (they needs this sort
> > of functionality), but is independent on them, so I'm proposing it
> > now.
> >
> > Of the seven patches, first three creates new files. Patches four to
> > six modifies ./check script, but keeps the changes out of existing
> > code as much as possible (patch four is only exception). Patch seven
> > is integrating it all together and is enabling the functionality.
> >
> > To sum how it works:
> > New file "environment", similar to "group" file, is created in each
> > test category. It uses similar syntax, but it ortogonal to groups. In
> > this file, each test can have specified one or more environments. When
> > environments are enabled (./check -e ), list of tests is compiled as
> > before (so -g, -x and other arguments works as usually) and for the
> > enabled tests, environments are found.
> >
> > If one test has multiple environments (and the selection is not
> > limited for only some env.), the test is duplicated for each specified
> > environment. Each run is then reported independently, as a combination
> > of the test and the environment. When the test is not found in the
> > file, it is added implicitly with "none" environment. The none
> > environment do nothing and can be stated explicitly in the file also.
>
> Hmm - yes, it is very different to what I thought we talked about.
> I'll try to explain the way I see persistent performance test
> environments fit into the existing infrastructure so you can see
> the direction I was thinking of.
>
> All we really require is a way of setting up a filesystem for
> multiple performance tests, where setting up the test context might
> take significantly longer than running the tests. I can see what you
> are trying to do with the environment code, I'm just thinking that
> it's a little over-engineered and trying to do too much.
>
> Lets start with how a test would define the initial filesystem setup
> it requires, and how it would trigger it to build and when we should
> start our timing for measurement of the workload being benchmarked.
> e.g.
>
> ....
> . ./common/rc
> . ./common/fsmark
>
> FSMARK_FILES=10000
> FSMARK_FILESIZE=4096
> FSMARK_DIRS=100
> FSMARK_THREADS=10
> _scratch_build_fsmark_env
>
> # real test starts now
> _start_timing
> .....
>
> And _build_fsmark_env() does all the work of checking the
> SCRATCH_MNT for existing test environment. e.g. the root directory
> of the $SCRATCH_MNT contains a file created by the
> _scratch_build_fsmark_env() function that contains the config used
> to build it. It sources the config file, see if it matches the
> config passed in by the test, and if it doesn't then we need to
> rebuild the scratch device and the test environment according to the
> current specification.
>
> Indeed, we can turn the above into a create performance test via:
>
> ....
> FSMARK_FILES=10000
> FSMARK_FILESIZE=4096
> FSMARK_DIRS=100
> FSMARK_THREADS=10
> FORCE_ENV_BUILD=true
>
> # real test starts now
> _start_timing
> _scratch_build_fsmark_env
> _stop_timing
>
> status=0
> exit
>
> This doesn't require lots of new infrastructure and is way more
> flexible than defining how tests are run/prepared in an external
> file. e.g. as you build tests it's trivial to simply group tests
> that use the same environment together manually. Tests can still be
> run randomly; it's just that they will need to create the
> environment accordingly and so take longer to run.
>
> In the longer term, I think it's better to change the common
> infrastructure to support test names that aren't numbers and then
> grouping of tests that use the same environment can all use the same
> name prefix. e.g.
>
> peformance/fsmark-small-files-001
> peformance/fsmark-small-files-002
> peformance/fsmark-small-files-003
> peformance/fsmark-large-files-001
> peformance/fsmark-large-files-002replied
> peformance/fsmark-1m-empty-files-001
> peformance/fsmark-10m-empty-files-001
> peformance/fsmark-100m-empty-files-001
> peformance/fsmark-100m-empty-files-002
> .....
>
> This makes sorting tests that use the same environment a very simple
> thing whilst also providing other wishlist functionality we have for
> the regression test side of fstests. If we need common test setups
> for regressions tests, then we can simply add the new regression
> tests in exactly the same way.
>
> As a result of this, we still use the existing group infrastructure
> to control what performance tests are run. Hence there's no need for
> explicit environments, CLI parameters to run them, cross-product
> matrices of tests running in differnet environments, etc. i.e.
>
> performance/group:
> fsmark-small-files-001 fsmark small_files rw sequential
> fsmark-small-files-002 fsmark small_files rw random
> fsmark-small-files-003 fsmark small_files traverse
> fsmark-small-files-004 fsmark small_files unlink
> fsmark-large-files-001 fsmark large_files rw
> fsmark-large-files-002 fsmark large_files unlink
> fsmark-1m-empty-files-001 fsmark metadata scale create
> fsmark-10m-empty-files-001 fsmark metadata scale create
> fsmark-100m-empty-files-001 fsmark metadata scale create
> fsmark-100m-empty-files-002 fsmark metadata scale traverse
> fsmark-100m-empty-files-003 fsmark metadata scale unlink
> .....
>
> Hence:
>
> # ./check -g fsmark
>
> will run all those fsmark tests.
>
> # ./check -g small_files
>
> will run just the small file tests
>
> # ./check -g fsmark -x scale
>
> will run all the fsmark tests that aren't scalability tests.
>
> That's how I've been thinking we should integrate persistent
> filesystem state for performance tests, as well as the test script
> interface and management should work. It is not as generic as your
> environment concept, but I think it's simpler, more flexible and
> easier to manage them a new set of wrappers around the outside of
> the existing test infrastructure. I'm interested to see what you
> think, Jan...
Using fsmark for creating the files is a good idea. It brings a
dependency, but for the performance testing it is required as well, so
it shouldn't be a problem, right? And making customizable values for the
environments is good idea to. I think I even planned something like
this, just didn't thought about some standardized options, rather just
environment-specific.
About the random tests order - I planned to edit the awk shuffling
script once the environment code is settled. It is not a hard limit,
just it is not implemented yet.
Your idea is simpler than what I have done, but I'm missing one thing:
how to test something in multiple situations/environments without having
the test code duplicated?
For what I see in your text, if I want to test the same thing with
small-files and big-files environment, I have to make it for one setup
and then make a copy and edit it... I don't like this duplicity. This is
the main point that lead me to my approach, to write everything only
once and then create any combination as is needed. But it is possible I
see what the tests should do from a different angle and I'm trying to
move some test responsibility to the environment. Can it be this?
I hope I answered to everything. :-)
Regards
Jan
^ permalink raw reply [flat|nested] 9+ messages in thread