[Qemu-devel] [PATCH v2 0/4] iotests: skip tests with unchanged dependencies

[Qemu-devel] [PATCH v2 0/4] iotests: skip tests with unchanged dependencies

[John Snow]

This patchset is a little off the beaten path, but:

Add the ability to skip tests that have already been run whose external
dependencies have remained unchanged since the last test run.

This is useful for, say, re-running a test set to see if a failure was
a one-off or is reproducible by just re-running the last command, which
will skip all the tests that completed, but hitting all the ones that
failed or were not run.

It's also useful for testing patchsets for publication against bisect
problems by only running tests after each patch that have changed anything
worth testing -- for instance, a patch that only changes documentation
is not going to change any of the binaries, so we can skip a large swath
of tests here.

The new feature is, of course, optional and can be invoked with -ts.

===
v2:
===

- Replaced grep script for awk in patch 01
- Added qemu dependency in patch 02

==
For convenience, this branch is available at:
https://github.com/jnsnow/qemu.git branch ch3ck
https://github.com/jnsnow/qemu/tree/ch3ck

This version is tagged ch3ck-v2:
https://github.com/jnsnow/qemu/releases/tag/ch3ck-v2
==

John Snow (4):
  iotests: fix exclusion option
  iotests: Add dependency info to groups list
  iotests: add timestamp skip feature
  iotests: clarify help text

 tests/qemu-iotests/check  | 56 +++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/common | 15 ++++++++++--
 tests/qemu-iotests/group  | 62 +++++++++++++++++++++++------------------------
 3 files changed, 99 insertions(+), 34 deletions(-)

-- 
2.1.0

[Qemu-devel] [PATCH v2 1/4] iotests: fix exclusion option

[John Snow]

If you are running out-of-tree, the -x option to exclude
a certain iotest is broken.

Replace porcelain usage of ls with a sturdier grep command.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/common | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 1e556bb..3949d18 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -82,7 +82,8 @@ s/ .*//p
     elif $xgroup
     then
         # arg after -x
-        [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] >$tmp.list 2>/dev/null
+        # Populate $tmp.list with all tests
+        awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null
         group_list=`sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
 s/ .*//p
 }'`
-- 
2.1.0

[Qemu-devel] [PATCH v2 2/4] iotests: Add dependency info to groups list

[John Snow]

Add a few external dependency groups to the iotests groups list,
such as "qemu", "nbd", and "scm".

This will assist us in knowing which tests need to be re-run
when those dependencies are updated, or by helping us to avoid
tests that we know are broken for a certain file.

qemu-img and qemu-io are so ubiquitous that it's not worth
amending this information to the groups file.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/group | 62 ++++++++++++++++++++++++------------------------
 1 file changed, 31 insertions(+), 31 deletions(-)

diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 34b16cb..1e74451 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -13,7 +13,7 @@
 004 rw auto quick
 005 img auto quick
 # 006 was removed, do not reuse
-007 snapshot auto
+007 snapshot auto qemu
 008 rw auto quick
 009 rw auto quick
 010 rw auto quick
@@ -34,9 +34,9 @@
 025 rw auto quick
 026 rw blkdbg auto
 027 rw auto quick
-028 rw backing auto quick
+028 rw backing auto quick qemu
 029 rw auto quick
-030 rw auto backing
+030 rw auto backing qemu
 031 rw auto quick
 032 rw auto quick
 033 rw auto quick
@@ -46,38 +46,38 @@
 037 rw auto backing quick
 038 rw auto backing quick
 039 rw auto quick
-040 rw auto
-041 rw auto backing
+040 rw auto qemu
+041 rw auto backing qemu
 042 rw auto quick
 043 rw auto backing
 044 rw auto
-045 rw auto quick
+045 rw auto quick scm qemu
 046 rw auto aio quick
 047 rw auto quick
 048 img auto quick
 049 rw auto
 050 rw auto backing quick
-051 rw auto
+051 rw auto qemu
 052 rw auto backing quick
 053 rw auto quick
 054 rw auto quick
-055 rw auto
-056 rw auto backing
-057 rw auto
-058 rw auto quick
+055 rw auto qemu
+056 rw auto backing qemu
+057 rw auto qemu
+058 rw auto quick nbd
 059 rw auto quick
 060 rw auto quick
 061 rw auto
 062 rw auto quick
 063 rw auto quick
 064 rw auto quick
-065 rw auto quick
+065 rw auto quick qemu
 066 rw auto quick
-067 rw auto quick
-068 rw auto quick
+067 rw auto quick qemu
+068 rw auto quick qemu
 069 rw auto quick
 070 rw auto quick
-071 rw auto quick
+071 rw auto quick qemu
 072 rw auto quick
 073 rw auto quick
 074 rw auto quick
@@ -87,45 +87,45 @@
 078 rw auto quick
 079 rw auto
 080 rw auto
-081 rw auto quick
+081 rw auto quick qemu
 082 rw auto quick
-083 rw auto
+083 rw auto nbd
 084 img auto quick
-085 rw auto
+085 rw auto qemu
 086 rw auto quick
-087 rw auto quick
+087 rw auto quick qemu
 088 rw auto quick
 089 rw auto quick
 090 rw auto quick
-091 rw auto
+091 rw auto qemu
 092 rw auto quick
-093 auto
-094 rw auto quick
-095 rw auto quick
+093 auto qemu
+094 rw auto quick qemu nbd
+095 rw auto quick qemu
 097 rw auto backing
 098 rw auto backing quick
-099 rw auto quick
+099 rw auto quick qemu
 100 rw auto quick
 101 rw auto quick
-102 rw auto quick
+102 rw auto quick qemu
 103 rw auto quick
 104 rw auto
 105 rw auto quick
 107 rw auto quick
 108 rw auto quick
-109 rw auto
+109 rw auto qemu
 110 rw auto backing quick
 111 rw auto quick
 112 rw auto
-113 rw auto quick
+113 rw auto quick nbd
 114 rw auto quick
 115 rw auto
 116 rw auto quick
 121 rw auto
 122 rw auto
-123 rw auto quick
-124 rw auto backing
+123 rw auto quick nbd
+124 rw auto backing qemu
 128 rw auto quick
-129 rw auto quick
-130 rw auto quick
+129 rw auto quick qemu
+130 rw auto quick qemu
 131 rw auto quick
-- 
2.1.0

[Qemu-devel] [PATCH v2 3/4] iotests: add timestamp skip feature

[John Snow]

Like a makefile, try to skip tests if we know they have already been
executed using the current set of external dependencies.

If a user passes the -ts option to ./check, if a test or its output
or its dependencies (qemu, qemu-nbd, qemu-io, qemu-img,
socket-scm-helper) have been modified, the test will be skipped.

This is primarily to speed up running bisectability tests for larger
series where each interim patch may or may not change all of the
dependencies, so some tests can be skipped.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/common |  6 +++++
 2 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index baeae80..0db40e2 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -141,6 +141,51 @@ _timestamp()
     echo -n " [$now]"
 }
 
+_okfile()
+{
+    echo "ts/${IMGPROTO}/${IMGFMT}/${1}.ok"
+}
+
+# _hasdep <seq> <dep>
+_hasdep()
+{
+    grep -q "^${1}.*${2}" "${source_iotests}/group"
+}
+
+# _skippable <seq>
+_skippable()
+{
+    tsfile=$(_okfile "${1}")
+    deps=(${QEMU_IMG} \
+              ${QEMU_IO} \
+              "${source_iotests}/${1}" \
+              "${source_iotests}/${1}.out" \
+              "${source_iotests}/common" \
+              "${source_iotests}/common.rc" \
+              "${source_iotests}/common.env" \
+              "${source_iotests}/common.config")
+
+    if _hasdep "${1}" "scm"; then
+        deps+=("${SOCKET_SCM_HELPER}")
+    fi
+
+    if _hasdep "${1}" "nbd" || [ "$IMGPROTO" = "nbd" ]; then
+        deps+=("${QEMU_NBD}")
+    fi
+
+    if _hasdep "${1}" "qemu"; then
+        deps+=("${QEMU}")
+    fi
+
+    if [ ! -e "${tsfile}" ]; then return 1; fi
+    for dep in "${deps[@]}"; do
+        if [ "${tsfile}" -ot "${dep}" ]; then return 1; fi
+    done
+
+    # Looks skippable, .ok file is newer than all external dependencies.
+    return 0
+}
+
 _wrapup()
 {
     # for hangcheck ...
@@ -268,6 +313,10 @@ do
         echo " - expunged"
         rm -f $seq.out.bad
         echo "/^$seq\$/d" >>$tmp.expunged
+
+    elif [ "${ts_skip}" == "true" ] && _skippable "${seq}"; then
+        echo "         [skip] (Results are recent)"
+
     elif [ ! -f "$source_iotests/$seq" ]
     then
         echo " - no such test?"
@@ -275,7 +324,8 @@ do
     else
         # really going to try and run this one
         #
-        rm -f $seq.out.bad
+        tsfile="$(_okfile ${seq})"
+        rm -f $seq.out.bad "${tsfile}"
         lasttime=`sed -n -e "/^$seq /s/.* //p" <check.time`
         if [ "X$lasttime" != X ]; then
                 echo -n " ${lasttime}s ..."
@@ -340,6 +390,10 @@ do
                     then
                         :
                     else
+                        if [ ! -d "$(dirname ${tsfile})" ]; then
+                            mkdir -p "$(dirname ${tsfile})"
+                        fi
+                        touch "${tsfile}"
                         echo "$seq `expr $stop - $start`" >>$tmp.time
                     fi
                 else
diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 3949d18..93cef9f 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -161,6 +161,8 @@ check options
     -T                  output timestamps
     -r                  randomize test order
     -c mode             cache mode
+    -ts                 timestamp skip: skip tests when already ran and
+                                   no dependencies have changed on disk.
 
 testlist options
     -g group[,group...]        include tests from these groups
@@ -319,6 +321,10 @@ testlist options
             xpand=false
             ;;
 
+        -ts)
+            ts_skip=true
+            xpand=false
+            ;;
         -v)
             verbose=true
             xpand=false
-- 
2.1.0

[Qemu-devel] [PATCH v2 4/4] iotests: clarify help text

[John Snow]

Split the help text to highlight the groups of options
a little better, carving out a clear "format" and
"protocols" section.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/common | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 93cef9f..d865ad9 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -134,7 +134,7 @@ s/ .*//p
 common options
     -v                  verbose
 
-check options
+image format options
     -raw                test raw (default)
     -bochs              test bochs
     -cloop              test cloop
@@ -146,6 +146,8 @@ check options
     -vpc                test vpc
     -vhdx               test vhdx
     -vmdk               test vmdk
+
+image protocol options
     -file               test file (default)
     -rbd                test rbd
     -sheepdog           test sheepdog
@@ -153,6 +155,8 @@ check options
     -ssh                test ssh
     -nfs                test nfs
     -archipelago        test archipelago
+
+other options
     -xdiff              graphical mode diff
     -nocache            use O_DIRECT on backing file
     -misalign           misalign memory allocations
-- 
2.1.0

Re: [Qemu-devel] [PATCH v2 1/4] iotests: fix exclusion option

[John Snow]

On 05/28/2015 12:45 PM, John Snow wrote:
> If you are running out-of-tree, the -x option to exclude
> a certain iotest is broken.
> 
> Replace porcelain usage of ls with a sturdier grep command.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/common | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
> index 1e556bb..3949d18 100644
> --- a/tests/qemu-iotests/common
> +++ b/tests/qemu-iotests/common
> @@ -82,7 +82,8 @@ s/ .*//p
>      elif $xgroup
>      then
>          # arg after -x
> -        [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] >$tmp.list 2>/dev/null
> +        # Populate $tmp.list with all tests
> +        awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null
>          group_list=`sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
>  s/ .*//p
>  }'`
> 

s/grep/awk/

Does this look good otherwise?

--js