* [PATCH kvm-unittests v2 1/4] Add run_tests.sh
2013-04-12 11:27 [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Kevin Wolf
@ 2013-04-12 11:27 ` Kevin Wolf
2013-04-12 12:24 ` Lucas Meneghel Rodrigues
2013-04-12 11:27 ` [PATCH kvm-unittests v2 2/4] x86/unittests.cfg: Add arch for x86_64-only tests Kevin Wolf
` (4 subsequent siblings)
5 siblings, 1 reply; 9+ messages in thread
From: Kevin Wolf @ 2013-04-12 11:27 UTC (permalink / raw)
To: kvm; +Cc: kwolf, gleb, mtosatti, lmr
This adds a convenient way to run all tests without having to set up
Autotest.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
x86-run | 9 +++-
x86/unittests.cfg | 2 +
3 files changed, 132 insertions(+), 2 deletions(-)
create mode 100755 run_tests.sh
diff --git a/run_tests.sh b/run_tests.sh
new file mode 100755
index 0000000..55ecac5
--- /dev/null
+++ b/run_tests.sh
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+testroot=x86
+config=$testroot/unittests.cfg
+qemu=${qemu:-qemu-system-x86_64}
+verbose=0
+
+function run()
+{
+ local testname="$1"
+ local groups="$2"
+ local smp="$3"
+ local kernel="$4"
+ local opts="$5"
+ local arch="$6"
+
+ if [ -z "$testname" ]; then
+ return
+ fi
+
+ if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then
+ return
+ fi
+
+ if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then
+ echo "skip $1 ($arch only)"
+ return
+ fi
+
+ cmdline="./x86-run $kernel -smp $smp -display none $opts"
+ if [ $verbose != 0 ]; then
+ echo $cmdline
+ fi
+
+ # extra_params in the config file may contain backticks that need to be
+ # expanded, so use eval to start qemu
+ eval $cmdline >> test.log
+
+ if [ $? -le 1 ]; then
+ echo -e "\e[32mPASS\e[0m $1"
+ else
+ echo -e "\e[31mFAIL\e[0m $1"
+ fi
+}
+
+function run_all()
+{
+ local config="$1"
+ local testname
+ local smp
+ local kernel
+ local opts
+ local groups
+ local arch
+
+ exec {config_fd}<$config
+
+ while read -u $config_fd line; do
+ if [[ "$line" =~ ^\[(.*)\]$ ]]; then
+ run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+ testname=${BASH_REMATCH[1]}
+ smp=1
+ kernel=""
+ opts=""
+ groups=""
+ arch=""
+ elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then
+ kernel=$testroot/${BASH_REMATCH[1]}
+ elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
+ smp=${BASH_REMATCH[1]}
+ elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
+ opts=${BASH_REMATCH[1]}
+ elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
+ groups=${BASH_REMATCH[1]}
+ elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then
+ arch=${BASH_REMATCH[1]}
+ fi
+ done
+
+ run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
+
+ exec {config_fd}<&-
+}
+
+function usage()
+{
+cat <<EOF
+
+Usage: $0 [-g group] [-h] [-v]
+
+ -g: Only execute tests in the given group
+ -h: Output this help text
+ -v: Enables verbose mode
+
+Set the environment variable QEMU=/path/to/qemu-system-x86_64 to allow the
+internally used x86-run to pick up the right qemu binary.
+
+EOF
+}
+
+# As it happens, config.mak is valid shell script code, too :-)
+source config.mak
+
+echo > test.log
+while getopts "g:hv" opt; do
+ case $opt in
+ g)
+ only_group=$OPTARG
+ ;;
+ h)
+ usage
+ exit
+ ;;
+ v)
+ verbose=1
+ ;;
+ *)
+ exit
+ ;;
+ esac
+done
+
+run_all $config
diff --git a/x86-run b/x86-run
index e395a70..19e5dec 100755
--- a/x86-run
+++ b/x86-run
@@ -13,7 +13,7 @@ else
qemu="${qemusystem}"
else
echo QEMU binary ${QEMU} has no support for test device. Exiting.
- exit 1
+ exit 2
fi
fi
@@ -24,4 +24,9 @@ then
else
command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel"
fi
-exec ${command} "$@"
+
+echo ${command} "$@"
+${command} "$@"
+ret=$?
+echo Return value from qemu: $ret
+exit $ret
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 5e08c55..7d0fa73 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -3,6 +3,8 @@
# file = foo.flat # Name of the flat file to be used
# smp = 2 # Number of processors the VM will use during this test
# extra_params = -cpu qemu64,+x2apic # Additional parameters used
+# arch = i386/x86_64 # Only if the test case works only on one of them
+# groups = group1 group2 # Used to identify test cases with run_tests -g ...
[apic]
file = apic.flat
--
1.8.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH kvm-unittests v2 1/4] Add run_tests.sh
2013-04-12 11:27 ` [PATCH kvm-unittests v2 1/4] Add run_tests.sh Kevin Wolf
@ 2013-04-12 12:24 ` Lucas Meneghel Rodrigues
0 siblings, 0 replies; 9+ messages in thread
From: Lucas Meneghel Rodrigues @ 2013-04-12 12:24 UTC (permalink / raw)
To: Kevin Wolf; +Cc: kvm, gleb, mtosatti
On Fri, 2013-04-12 at 13:27 +0200, Kevin Wolf wrote:
> This adds a convenient way to run all tests without having to set up
> Autotest.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
> run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> x86-run | 9 +++-
> x86/unittests.cfg | 2 +
> 3 files changed, 132 insertions(+), 2 deletions(-)
> create mode 100755 run_tests.sh
>
> diff --git a/run_tests.sh b/run_tests.sh
> new file mode 100755
> index 0000000..55ecac5
> --- /dev/null
> +++ b/run_tests.sh
> @@ -0,0 +1,123 @@
> +#!/bin/bash
> +
> +testroot=x86
> +config=$testroot/unittests.cfg
> +qemu=${qemu:-qemu-system-x86_64}
> +verbose=0
> +
> +function run()
> +{
> + local testname="$1"
> + local groups="$2"
> + local smp="$3"
> + local kernel="$4"
> + local opts="$5"
> + local arch="$6"
> +
> + if [ -z "$testname" ]; then
> + return
> + fi
> +
> + if [ -n "$only_group" ] && ! grep -q "$only_group" <<<$groups; then
> + return
> + fi
> +
> + if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then
> + echo "skip $1 ($arch only)"
> + return
> + fi
> +
> + cmdline="./x86-run $kernel -smp $smp -display none $opts"
> + if [ $verbose != 0 ]; then
> + echo $cmdline
> + fi
> +
> + # extra_params in the config file may contain backticks that need to be
> + # expanded, so use eval to start qemu
> + eval $cmdline >> test.log
> +
> + if [ $? -le 1 ]; then
> + echo -e "\e[32mPASS\e[0m $1"
> + else
> + echo -e "\e[31mFAIL\e[0m $1"
> + fi
> +}
> +
> +function run_all()
> +{
> + local config="$1"
> + local testname
> + local smp
> + local kernel
> + local opts
> + local groups
> + local arch
> +
> + exec {config_fd}<$config
> +
> + while read -u $config_fd line; do
> + if [[ "$line" =~ ^\[(.*)\]$ ]]; then
> + run "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch"
> + testname=${BASH_REMATCH[1]}
> + smp=1
> + kernel=""
> + opts=""
> + groups=""
> + arch=""
> + elif [[ $line =~ ^file\ *=\ *(.*)$ ]]; then
> + kernel=$testroot/${BASH_REMATCH[1]}
> + elif [[ $line =~ ^smp\ *=\ *(.*)$ ]]; then
> + smp=${BASH_REMATCH[1]}
> + elif [[ $line =~ ^extra_params\ *=\ *(.*)$ ]]; then
> + opts=${BASH_REMATCH[1]}
> + elif [[ $line =~ ^groups\ *=\ *(.*)$ ]]; then
> + groups=${BASH_REMATCH[1]}
> + elif [[ $line =~ ^arch\ *=\ *(.*)$ ]]; then
> + arch=${BASH_REMATCH[1]}
> + fi
> + done
^ This looks good to me, although using python and the ConfigParser
library would be less work (no need to explicitly use regexp based
parsing).
I'm currently taking a look at the new fields and how autotest could
make use of them... Thanks!
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH kvm-unittests v2 2/4] x86/unittests.cfg: Add arch for x86_64-only tests
2013-04-12 11:27 [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Kevin Wolf
2013-04-12 11:27 ` [PATCH kvm-unittests v2 1/4] Add run_tests.sh Kevin Wolf
@ 2013-04-12 11:27 ` Kevin Wolf
2013-04-12 11:27 ` [PATCH kvm-unittests v2 3/4] x86/unittests.cfg: Add missing test cases Kevin Wolf
` (3 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Kevin Wolf @ 2013-04-12 11:27 UTC (permalink / raw)
To: kvm; +Cc: kwolf, gleb, mtosatti, lmr
Their kernel binaries would be missing when the tests are built for
i386.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
x86/unittests.cfg | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 7d0fa73..f2336bb 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -10,6 +10,7 @@
file = apic.flat
smp = 2
extra_params = -cpu qemu64,+x2apic
+arch = x86_64
[smptest]
file = smptest.flat
@@ -55,15 +56,18 @@ extra_params = -append 'ple_round_robin'
[access]
file = access.flat
+arch = x86_64
[emulator]
file = emulator.flat
+arch = x86_64
[hypercall]
file = hypercall.flat
[idt_test]
file = idt_test.flat
+arch = x86_64
[msr]
file = msr.flat
@@ -82,19 +86,23 @@ file = tsc.flat
[xsave]
file = xsave.flat
+arch = x86_64
[rmap_chain]
file = rmap_chain.flat
+arch = x86_64
[svm]
file = svm.flat
smp = 2
extra_params = -cpu qemu64,+svm
+arch = x86_64
[svm-disabled]
file = svm.flat
smp = 2
extra_params = -cpu qemu64,-svm
+arch = x86_64
[kvmclock_test]
file = kvmclock_test.flat
@@ -103,4 +111,5 @@ extra_params = --append "10000000 `date +%s`"
[pcid]
file = pcid.flat
-extra_params = -cpu qemu64,+pcid
\ No newline at end of file
+extra_params = -cpu qemu64,+pcid
+arch = x86_64
--
1.8.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH kvm-unittests v2 3/4] x86/unittests.cfg: Add missing test cases
2013-04-12 11:27 [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Kevin Wolf
2013-04-12 11:27 ` [PATCH kvm-unittests v2 1/4] Add run_tests.sh Kevin Wolf
2013-04-12 11:27 ` [PATCH kvm-unittests v2 2/4] x86/unittests.cfg: Add arch for x86_64-only tests Kevin Wolf
@ 2013-04-12 11:27 ` Kevin Wolf
2013-04-12 11:27 ` [PATCH kvm-unittests v2 4/4] x86/unittests.cfg: Create test case groups Kevin Wolf
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Kevin Wolf @ 2013-04-12 11:27 UTC (permalink / raw)
To: kvm; +Cc: kwolf, gleb, mtosatti, lmr
Some test cases seem to have been added without updating the
configuration file. This adds them, and leaves cases commented out that
don't seem to complete.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
x86/unittests.cfg | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index f2336bb..11e8077 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -58,10 +58,16 @@ extra_params = -append 'ple_round_robin'
file = access.flat
arch = x86_64
+#[asyncpf]
+#file = asyncpf.flat
+
[emulator]
file = emulator.flat
arch = x86_64
+[eventinj]
+file = eventinj.flat
+
[hypercall]
file = hypercall.flat
@@ -69,21 +75,33 @@ file = hypercall.flat
file = idt_test.flat
arch = x86_64
+#[init]
+#file = init.flat
+
[msr]
file = msr.flat
+[pmu]
+file = pmu.flat
+
[port80]
file = port80.flat
[realmode]
file = realmode.flat
+[s3]
+file = s3.flat
+
[sieve]
file = sieve.flat
[tsc]
file = tsc.flat
+[tsc_adjust]
+file = tsc_adjust.flat
+
[xsave]
file = xsave.flat
arch = x86_64
@@ -104,6 +122,14 @@ smp = 2
extra_params = -cpu qemu64,-svm
arch = x86_64
+[taskswitch]
+file = taskswitch.flat
+arch = i386
+
+[taskswitch2]
+file = taskswitch2.flat
+arch = i386
+
[kvmclock_test]
file = kvmclock_test.flat
smp = 2
--
1.8.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH kvm-unittests v2 4/4] x86/unittests.cfg: Create test case groups
2013-04-12 11:27 [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Kevin Wolf
` (2 preceding siblings ...)
2013-04-12 11:27 ` [PATCH kvm-unittests v2 3/4] x86/unittests.cfg: Add missing test cases Kevin Wolf
@ 2013-04-12 11:27 ` Kevin Wolf
2013-04-13 23:26 ` [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Cole Robinson
2013-04-17 22:26 ` Marcelo Tosatti
5 siblings, 0 replies; 9+ messages in thread
From: Kevin Wolf @ 2013-04-12 11:27 UTC (permalink / raw)
To: kvm; +Cc: kwolf, gleb, mtosatti, lmr
Put all vmexit test cases and all task switch test cases into a group,
so that you can use something like ./run_tests -g tasks
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
x86/unittests.cfg | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 11e8077..bc9643e 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -23,36 +23,44 @@ smp = 3
[vmexit_cpuid]
file = vmexit.flat
extra_params = -append 'cpuid'
+groups = vmexit
[vmexit_vmcall]
file = vmexit.flat
extra_params = -append 'vmcall'
+groups = vmexit
[vmexit_mov_from_cr8]
file = vmexit.flat
extra_params = -append 'mov_from_cr8'
+groups = vmexit
[vmexit_mov_to_cr8]
file = vmexit.flat
extra_params = -append 'mov_to_cr8'
+groups = vmexit
[vmexit_inl_pmtimer]
file = vmexit.flat
extra_params = -append 'inl_from_pmtimer'
+groups = vmexit
[vmexit_ipi]
file = vmexit.flat
smp = 2
extra_params = -append 'ipi'
+groups = vmexit
[vmexit_ipi_halt]
file = vmexit.flat
smp = 2
extra_params = -append 'ipi_halt'
+groups = vmexit
[vmexit_ple_round_robin]
file = vmexit.flat
extra_params = -append 'ple_round_robin'
+groups = vmexit
[access]
file = access.flat
@@ -125,10 +133,12 @@ arch = x86_64
[taskswitch]
file = taskswitch.flat
arch = i386
+groups = tasks
[taskswitch2]
file = taskswitch2.flat
arch = i386
+groups = tasks
[kvmclock_test]
file = kvmclock_test.flat
--
1.8.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg
2013-04-12 11:27 [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Kevin Wolf
` (3 preceding siblings ...)
2013-04-12 11:27 ` [PATCH kvm-unittests v2 4/4] x86/unittests.cfg: Create test case groups Kevin Wolf
@ 2013-04-13 23:26 ` Cole Robinson
2013-04-15 8:11 ` Kevin Wolf
2013-04-17 22:26 ` Marcelo Tosatti
5 siblings, 1 reply; 9+ messages in thread
From: Cole Robinson @ 2013-04-13 23:26 UTC (permalink / raw)
To: Kevin Wolf; +Cc: kvm, gleb, mtosatti, lmr
On 04/12/2013 07:27 AM, Kevin Wolf wrote:
> This adds a small script that allows to conveniently run all test cases and
> that reports back one "PASS" or "FAIL" line for each test case; it also creates
> a test.log file with the full output.
>
> It parses the unittests.cfg file used by autotest, so I'm also making some
> updates to this file and hopefully make some improvements that autotest can
> make use of as well.
>
> I checked with Lucas that adding new keys isn't a problem. One thing that is
> likely to cause failures for autotext, though, is that I'm including i386-only
> test cases in unittests.cfg. It should probably parse the 'arch' key in the
> future and skip them (and possibly introduce an i386 run besides the x86_64 one
> so that they actually get tested).
>
This has some overlap with a series I sent a few weeks back, enhancing x86-run
to read unittests.cfg, and rewriting it in python for native cfg parsing and,
cli handling, etc:
http://www.spinics.net/lists/kvm/msg88030.html
http://www.spinics.net/lists/kvm/msg88033.html
Thread went silent though...
- Cole
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg
2013-04-13 23:26 ` [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Cole Robinson
@ 2013-04-15 8:11 ` Kevin Wolf
0 siblings, 0 replies; 9+ messages in thread
From: Kevin Wolf @ 2013-04-15 8:11 UTC (permalink / raw)
To: Cole Robinson; +Cc: kvm, gleb, mtosatti, lmr
Am 14.04.2013 um 01:26 hat Cole Robinson geschrieben:
> On 04/12/2013 07:27 AM, Kevin Wolf wrote:
> > This adds a small script that allows to conveniently run all test cases and
> > that reports back one "PASS" or "FAIL" line for each test case; it also creates
> > a test.log file with the full output.
> >
> > It parses the unittests.cfg file used by autotest, so I'm also making some
> > updates to this file and hopefully make some improvements that autotest can
> > make use of as well.
> >
> > I checked with Lucas that adding new keys isn't a problem. One thing that is
> > likely to cause failures for autotext, though, is that I'm including i386-only
> > test cases in unittests.cfg. It should probably parse the 'arch' key in the
> > future and skip them (and possibly introduce an i386 run besides the x86_64 one
> > so that they actually get tested).
> >
>
> This has some overlap with a series I sent a few weeks back, enhancing x86-run
> to read unittests.cfg, and rewriting it in python for native cfg parsing and,
> cli handling, etc:
>
> http://www.spinics.net/lists/kvm/msg88030.html
> http://www.spinics.net/lists/kvm/msg88033.html
>
> Thread went silent though...
I don't really mind which patch is applied as long as I have an easy way
to run my tests (with my scripts, ./run_tests.sh -g tasks) and get
simple results (one line per testcase, indicating PASS or FAIL) as long
as I'm not debugging.
The problem that I see with your patch (5/5 is the interesting one,
right?) is that you mix up the kernel file name and the test name; I
don't think you can use it with something like the vmexit tests for
which several unittests.cfg sections with different options exist. And
of course, it's missing some features in comparison.
Is there some feature your Python code has that I don't have yet? I can
try to implement it then.
Kevin
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg
2013-04-12 11:27 [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Kevin Wolf
` (4 preceding siblings ...)
2013-04-13 23:26 ` [PATCH kvm-unittests v2 0/4] Add run_tests.sh script and update unittests.cfg Cole Robinson
@ 2013-04-17 22:26 ` Marcelo Tosatti
5 siblings, 0 replies; 9+ messages in thread
From: Marcelo Tosatti @ 2013-04-17 22:26 UTC (permalink / raw)
To: Kevin Wolf; +Cc: kvm, gleb, lmr
On Fri, Apr 12, 2013 at 01:27:36PM +0200, Kevin Wolf wrote:
> This adds a small script that allows to conveniently run all test cases and
> that reports back one "PASS" or "FAIL" line for each test case; it also creates
> a test.log file with the full output.
>
> It parses the unittests.cfg file used by autotest, so I'm also making some
> updates to this file and hopefully make some improvements that autotest can
> make use of as well.
>
> I checked with Lucas that adding new keys isn't a problem. One thing that is
> likely to cause failures for autotext, though, is that I'm including i386-only
> test cases in unittests.cfg. It should probably parse the 'arch' key in the
> future and skip them (and possibly introduce an i386 run besides the x86_64 one
> so that they actually get tested).
>
> Kevin Wolf (4):
> Add run_tests.sh
> x86/unittests.cfg: Add arch for x86_64-only tests
> x86/unittests.cfg: Add missing test cases
> x86/unittests.cfg: Create test case groups
>
> run_tests.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> x86-run | 9 +++-
> x86/unittests.cfg | 49 +++++++++++++++++++++-
> 3 files changed, 178 insertions(+), 3 deletions(-)
> create mode 100755 run_tests.sh
Applied, thanks.
^ permalink raw reply [flat|nested] 9+ messages in thread