[PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg

[PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg

[Kevin Wolf]

This adds a small script that allows to conveniently run all test cases and
that reports back one "PASS" or "FAIL" line for each test case; it also creates
a test.log file with the full output.

It parses the unittests.cfg file used by autotest, so I'm also making some
updates to this file and hopefully make some improvements that autotest can
make use of as well.

I checked with Lucas that adding new keys isn't a problem. One thing that is
likely to cause failures for autotext, though, is that I'm including i386-only
test cases in unittests.cfg. It should probably parse the 'arch' key in the
future and skip them (and possibly introduce an i386 run besides the x86_64 one
so that they actually get tested).

Kevin Wolf (4):
  Add run_tests.sh
  x86/unittests.cfg: Add arch for x86_64-only tests
  x86/unittests.cfg: Add missing test cases
  x86/unittests.cfg: Create test case groups

 run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 x86-run           |   9 +++-
 x86/unittests.cfg |  49 +++++++++++++++++++++-
 3 files changed, 178 insertions(+), 3 deletions(-)
 create mode 100755 run_tests.sh

-- 
1.8.1.4

[PATCH kvm-unittests v2 1/4] Add run_tests.sh

[Kevin Wolf]

This adds a convenient way to run all tests without having to set up
Autotest.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 x86-run           |   9 +++-
 x86/unittests.cfg |   2 +
 3 files changed, 132 insertions(+), 2 deletions(-)
 create mode 100755 run_tests.sh

diff --git a/run_tests.sh b/run_tests.sh
new file mode 100755
index 0000000..55ecac5
--- /dev/null
+++ b/run_tests.sh
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+testroot=x86
+config=$testroot/unittests.cfg
+qemu=${qemu:-qemu-system-x86_64}
+verbose=0
+
+function run()
+{
+    local testname="$1"
+    local groups="$2"
+    local smp="$3"
+    local kernel="$4"
+    local opts="$5"
+    local arch="$6"
+
+    if [ -z "$testname" ]; then
+        return
+    fi
+
+    if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then
+        return
+    fi
+
+    if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then
+        echo "skip $1 ($arch only)"
+        return
+    fi
+
+	cmdline="./x86-run $kernel -smp $smp -display none $opts"
+    if [ $verbose != 0 ]; then
+        echo $cmdline
+    fi
+
+    # extra_params in the config file may contain backticks that need to be
+    # expanded, so use eval to start qemu
+    eval $cmdline >> test.log
+
+    if [ $? -le 1 ]; then
+        echo -e "\e[32mPASS\e[0m $1"
+    else
+        echo -e "\e[31mFAIL\e[0m $1"
+    fi
+}
+
+function run_all()
+{
+    local config="$1"
+    local testname
+    local smp
+    local kernel
+    local opts
+    local groups
+    local arch
+
+    exec {config_fd}<$config
+
+    while read -u $config_fd line; do
+        if [[ "$line" =~ ^\[(.*)\]$ ]]; then
+            run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+            testname=${BASH_REMATCH[1]}
+            smp=1
+            kernel=""
+            opts=""
+            groups=""
+            arch=""
+        elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then
+            kernel=$testroot/${BASH_REMATCH[1]}
+        elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
+            smp=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
+            opts=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
+            groups=${BASH_REMATCH[1]}
+        elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then
+            arch=${BASH_REMATCH[1]}
+        fi
+    done
+
+    run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+
+    exec {config_fd}<&-
+}
+
+function usage()
+{
+cat <<EOF
+
+Usage: $0 [-g group] [-h] [-v]
+
+    -g: Only execute tests in the given group
+    -h: Output this help text
+    -v: Enables verbose mode
+
+Set the environment variable QEMU=/path/to/qemu-system-x86_64 to allow the
+internally used x86-run to pick up the right qemu binary.
+
+EOF
+}
+
+# As it happens, config.mak is valid shell script code, too :-)
+source config.mak
+
+echo > test.log
+while getopts "g:hv" opt; do
+    case $opt in
+        g)
+            only_group=$OPTARG
+            ;;
+        h)
+            usage
+            exit
+            ;;
+        v)
+            verbose=1
+            ;;
+        *)
+            exit
+            ;;
+    esac
+done
+
+run_all $config
diff --git a/x86-run b/x86-run
index e395a70..19e5dec 100755
--- a/x86-run
+++ b/x86-run
@@ -13,7 +13,7 @@ else
 		qemu="${qemusystem}"
 	else
 		echo QEMU binary ${QEMU} has no support for test device. Exiting.
-		exit 1
+		exit 2
 	fi
 fi
 
@@ -24,4 +24,9 @@ then
 else
 	command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel"
 fi
-exec ${command} "$@"
+
+echo ${command} "$@"
+${command} "$@"
+ret=$?
+echo Return value from qemu: $ret
+exit $ret
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 5e08c55..7d0fa73 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -3,6 +3,8 @@
 # file = foo.flat # Name of the flat file to be used
 # smp = 2 # Number of processors the VM will use during this test
 # extra_params = -cpu qemu64,+x2apic # Additional parameters used
+# arch = i386/x86_64 # Only if the test case works only on one of them
+# groups = group1 group2 # Used to identify test cases with run_tests -g ...
 
 [apic]
 file = apic.flat
-- 
1.8.1.4

[PATCH kvm-unittests v2 2/4] x86/unittests.cfg: Add arch for x86_64-only tests

[Kevin Wolf]

Their kernel binaries would be missing when the tests are built for
i386.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 x86/unittests.cfg | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 7d0fa73..f2336bb 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -10,6 +10,7 @@
 file = apic.flat
 smp = 2
 extra_params = -cpu qemu64,+x2apic
+arch = x86_64
 
 [smptest]
 file = smptest.flat
@@ -55,15 +56,18 @@ extra_params = -append 'ple_round_robin'
 
 [access]
 file = access.flat
+arch = x86_64
 
 [emulator]
 file = emulator.flat
+arch = x86_64
 
 [hypercall]
 file = hypercall.flat
 
 [idt_test]
 file = idt_test.flat
+arch = x86_64
 
 [msr]
 file = msr.flat
@@ -82,19 +86,23 @@ file = tsc.flat
 
 [xsave]
 file = xsave.flat
+arch = x86_64
 
 [rmap_chain]
 file = rmap_chain.flat
+arch = x86_64
 
 [svm]
 file = svm.flat
 smp = 2
 extra_params = -cpu qemu64,+svm
+arch = x86_64
 
 [svm-disabled]
 file = svm.flat
 smp = 2
 extra_params = -cpu qemu64,-svm
+arch = x86_64
 
 [kvmclock_test]
 file = kvmclock_test.flat
@@ -103,4 +111,5 @@ extra_params = --append "10000000 `date +%s`"
 
 [pcid]
 file = pcid.flat
-extra_params = -cpu qemu64,+pcid
\ No newline at end of file
+extra_params = -cpu qemu64,+pcid
+arch = x86_64
-- 
1.8.1.4

[PATCH kvm-unittests v2 3/4] x86/unittests.cfg: Add missing test cases

[Kevin Wolf]

Some test cases seem to have been added without updating the
configuration file. This adds them, and leaves cases commented out that
don't seem to complete.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 x86/unittests.cfg | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index f2336bb..11e8077 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -58,10 +58,16 @@ extra_params = -append 'ple_round_robin'
 file = access.flat
 arch = x86_64
 
+#[asyncpf]
+#file = asyncpf.flat
+
 [emulator]
 file = emulator.flat
 arch = x86_64
 
+[eventinj]
+file = eventinj.flat
+
 [hypercall]
 file = hypercall.flat
 
@@ -69,21 +75,33 @@ file = hypercall.flat
 file = idt_test.flat
 arch = x86_64
 
+#[init]
+#file = init.flat
+
 [msr]
 file = msr.flat
 
+[pmu]
+file = pmu.flat
+
 [port80]
 file = port80.flat
 
 [realmode]
 file = realmode.flat
 
+[s3]
+file = s3.flat
+
 [sieve]
 file = sieve.flat
 
 [tsc]
 file = tsc.flat
 
+[tsc_adjust]
+file = tsc_adjust.flat
+
 [xsave]
 file = xsave.flat
 arch = x86_64
@@ -104,6 +122,14 @@ smp = 2
 extra_params = -cpu qemu64,-svm
 arch = x86_64
 
+[taskswitch]
+file = taskswitch.flat
+arch = i386
+
+[taskswitch2]
+file = taskswitch2.flat
+arch = i386
+
 [kvmclock_test]
 file = kvmclock_test.flat
 smp = 2
-- 
1.8.1.4

[PATCH kvm-unittests v2 4/4] x86/unittests.cfg: Create test case groups

[Kevin Wolf]

Put all vmexit test cases and all task switch test cases into a group,
so that you can use something like ./run_tests -g tasks

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 x86/unittests.cfg | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 11e8077..bc9643e 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -23,36 +23,44 @@ smp = 3
 [vmexit_cpuid]
 file = vmexit.flat
 extra_params = -append 'cpuid'
+groups = vmexit
 
 [vmexit_vmcall]
 file = vmexit.flat
 extra_params = -append 'vmcall'
+groups = vmexit
 
 [vmexit_mov_from_cr8]
 file = vmexit.flat
 extra_params = -append 'mov_from_cr8'
+groups = vmexit
 
 [vmexit_mov_to_cr8]
 file = vmexit.flat
 extra_params = -append 'mov_to_cr8'
+groups = vmexit
 
 [vmexit_inl_pmtimer]
 file = vmexit.flat
 extra_params = -append 'inl_from_pmtimer'
+groups = vmexit
 
 [vmexit_ipi]
 file = vmexit.flat
 smp = 2
 extra_params = -append 'ipi'
+groups = vmexit
 
 [vmexit_ipi_halt]
 file = vmexit.flat
 smp = 2
 extra_params = -append 'ipi_halt'
+groups = vmexit
 
 [vmexit_ple_round_robin]
 file = vmexit.flat
 extra_params = -append 'ple_round_robin'
+groups = vmexit
 
 [access]
 file = access.flat
@@ -125,10 +133,12 @@ arch = x86_64
 [taskswitch]
 file = taskswitch.flat
 arch = i386
+groups = tasks
 
 [taskswitch2]
 file = taskswitch2.flat
 arch = i386
+groups = tasks
 
 [kvmclock_test]
 file = kvmclock_test.flat
-- 
1.8.1.4

Re: [PATCH kvm-unittests v2 1/4] Add run_tests.sh

[Lucas Meneghel Rodrigues]

On Fri, 2013-04-12 at 13:27 +0200, Kevin Wolf wrote:
> This adds a convenient way to run all tests without having to set up
> Autotest.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  x86-run           |   9 +++-
>  x86/unittests.cfg |   2 +
>  3 files changed, 132 insertions(+), 2 deletions(-)
>  create mode 100755 run_tests.sh
> 
> diff --git a/run_tests.sh b/run_tests.sh
> new file mode 100755
> index 0000000..55ecac5
> --- /dev/null
> +++ b/run_tests.sh
> @@ -0,0 +1,123 @@
> +#!/bin/bash
> +
> +testroot=x86
> +config=$testroot/unittests.cfg
> +qemu=${qemu:-qemu-system-x86_64}
> +verbose=0
> +
> +function run()
> +{
> +    local testname="$1"
> +    local groups="$2"
> +    local smp="$3"
> +    local kernel="$4"
> +    local opts="$5"
> +    local arch="$6"
> +
> +    if [ -z "$testname" ]; then
> +        return
> +    fi
> +
> +    if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then
> +        return
> +    fi
> +
> +    if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then
> +        echo "skip $1 ($arch only)"
> +        return
> +    fi
> +
> +       cmdline="./x86-run $kernel -smp $smp -display none $opts"
> +    if [ $verbose != 0 ]; then
> +        echo $cmdline
> +    fi
> +
> +    # extra_params in the config file may contain backticks that need to be
> +    # expanded, so use eval to start qemu
> +    eval $cmdline >> test.log
> +
> +    if [ $? -le 1 ]; then
> +        echo -e "\e[32mPASS\e[0m $1"
> +    else
> +        echo -e "\e[31mFAIL\e[0m $1"
> +    fi
> +}
> +
> +function run_all()
> +{
> +    local config="$1"
> +    local testname
> +    local smp
> +    local kernel
> +    local opts
> +    local groups
> +    local arch
> +
> +    exec {config_fd}<$config
> +
> +    while read -u $config_fd line; do
> +        if [[ "$line" =~ ^\[(.*)\]$ ]]; then
> +            run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
> +            testname=${BASH_REMATCH[1]}
> +            smp=1
> +            kernel=""
> +            opts=""
> +            groups=""
> +            arch=""
> +        elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then
> +            kernel=$testroot/${BASH_REMATCH[1]}
> +        elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
> +            smp=${BASH_REMATCH[1]}
> +        elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
> +            opts=${BASH_REMATCH[1]}
> +        elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
> +            groups=${BASH_REMATCH[1]}
> +        elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then
> +            arch=${BASH_REMATCH[1]}
> +        fi
> +    done

^ This looks good to me, although using python and the ConfigParser
library would be less work (no need to explicitly use regexp based
parsing).

I'm currently taking a look at the new fields and how autotest could
make use of them... Thanks!

Re: [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg

[Cole Robinson]

On 04/12/2013 07:27 AM, Kevin Wolf wrote:
> This adds a small script that allows to conveniently run all test cases and
> that reports back one "PASS" or "FAIL" line for each test case; it also creates
> a test.log file with the full output.
> 
> It parses the unittests.cfg file used by autotest, so I'm also making some
> updates to this file and hopefully make some improvements that autotest can
> make use of as well.
> 
> I checked with Lucas that adding new keys isn't a problem. One thing that is
> likely to cause failures for autotext, though, is that I'm including i386-only
> test cases in unittests.cfg. It should probably parse the 'arch' key in the
> future and skip them (and possibly introduce an i386 run besides the x86_64 one
> so that they actually get tested).
> 

This has some overlap with a series I sent a few weeks back, enhancing x86-run
to read unittests.cfg, and rewriting it in python for native cfg parsing and,
cli handling, etc:

http://www.spinics.net/lists/kvm/msg88030.html
http://www.spinics.net/lists/kvm/msg88033.html

Thread went silent though...

- Cole

Re: [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg

[Kevin Wolf]

Am 14.04.2013 um 01:26 hat Cole Robinson geschrieben:
> On 04/12/2013 07:27 AM, Kevin Wolf wrote:
> > This adds a small script that allows to conveniently run all test cases and
> > that reports back one "PASS" or "FAIL" line for each test case; it also creates
> > a test.log file with the full output.
> > 
> > It parses the unittests.cfg file used by autotest, so I'm also making some
> > updates to this file and hopefully make some improvements that autotest can
> > make use of as well.
> > 
> > I checked with Lucas that adding new keys isn't a problem. One thing that is
> > likely to cause failures for autotext, though, is that I'm including i386-only
> > test cases in unittests.cfg. It should probably parse the 'arch' key in the
> > future and skip them (and possibly introduce an i386 run besides the x86_64 one
> > so that they actually get tested).
> > 
> 
> This has some overlap with a series I sent a few weeks back, enhancing x86-run
> to read unittests.cfg, and rewriting it in python for native cfg parsing and,
> cli handling, etc:
> 
> http://www.spinics.net/lists/kvm/msg88030.html
> http://www.spinics.net/lists/kvm/msg88033.html
> 
> Thread went silent though...

I don't really mind which patch is applied as long as I have an easy way
to run my tests (with my scripts, ./run_tests.sh -g tasks) and get
simple results (one line per testcase, indicating PASS or FAIL) as long
as I'm not debugging.

The problem that I see with your patch (5/5 is the interesting one,
right?) is that you mix up the kernel file name and the test name; I
don't think you can use it with something like the vmexit tests for
which several unittests.cfg sections with different options exist. And
of course, it's missing some features in comparison.

Is there some feature your Python code has that I don't have yet? I can
try to implement it then.

Kevin

Re: [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg

[Marcelo Tosatti]

On Fri, Apr 12, 2013 at 01:27:36PM +0200, Kevin Wolf wrote:
> This adds a small script that allows to conveniently run all test cases and
> that reports back one "PASS" or "FAIL" line for each test case; it also creates
> a test.log file with the full output.
> 
> It parses the unittests.cfg file used by autotest, so I'm also making some
> updates to this file and hopefully make some improvements that autotest can
> make use of as well.
> 
> I checked with Lucas that adding new keys isn't a problem. One thing that is
> likely to cause failures for autotext, though, is that I'm including i386-only
> test cases in unittests.cfg. It should probably parse the 'arch' key in the
> future and skip them (and possibly introduce an i386 run besides the x86_64 one
> so that they actually get tested).
> 
> Kevin Wolf (4):
>   Add run_tests.sh
>   x86/unittests.cfg: Add arch for x86_64-only tests
>   x86/unittests.cfg: Add missing test cases
>   x86/unittests.cfg: Create test case groups
> 
>  run_tests.sh      | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  x86-run           |   9 +++-
>  x86/unittests.cfg |  49 +++++++++++++++++++++-
>  3 files changed, 178 insertions(+), 3 deletions(-)
>  create mode 100755 run_tests.sh

Applied, thanks.