Re: [PATCH kvm-unit-tests v2 00/12] Improve the output of test runners

Andrew Jones
2015-12-17T20:05:01Z

On Thu, Dec 17, 2015 at 06:53:31PM +0100, Radim Krčmář wrote:
> v1: http://www.spinics.net/lists/kvm/msg125202.html
>
> Drew brought up the existence of scripts/mkstandalone.sh, which
> significantly expanded v2 (and my set of curses) ...
> I didn't want to do the same twice, so first part of this series,
> [1-4/12], reuses run() from run_tests.sh and does some non-conservative
> changes to scripts/mkstandalone.sh. scripts/mkstandalone.sh is lacking
> behind run_tests.sh, but should be good enough to fulfill its purpose.
>
> The output of run_tests.sh has also changed a bit and now looks like
> this (you'll again need to imagine colours):
>
> > PASS apic (14 tests)
> > PASS ioapic (19 tests)
> > PASS smptest (1 tests)
> > PASS smptest3 (1 tests)
> > PASS vmexit_cpuid
> > PASS vmexit_vmcall

Why do some tests, which have only 1 test, say (1 tests), but other
tests don't say anything?

> > PASS vmexit_mov_from_cr8
> > PASS vmexit_mov_to_cr8
> > PASS vmexit_inl_pmtimer
> > PASS vmexit_ipi
> > PASS vmexit_ipi_halt
> > PASS vmexit_ple_round_robin
> > PASS access
> > SKIP smap (0 tests)
> > SKIP pku (0 tests)
> > PASS emulator (132 tests, 1 skipped)
> > PASS eventinj (13 tests)
> > PASS hypercall (2 tests)
> > PASS idt_test (4 tests)
> > PASS msr (13 tests)
> > PASS pmu (67 tests, 1 expected failures)
> > PASS port80
> > PASS realmode
> > PASS s3
> > PASS sieve
> > PASS tsc (3 tests)
> > PASS tsc_adjust (5 tests)
> > PASS xsave (17 tests)
> > PASS rmap_chain
> > SKIP svm (0 tests)
> > SKIP svm-disabled (0 tests)
> > SKIP taskswitch (i386 only)
> > SKIP taskswitch2 (i386 only)
> > PASS kvmclock_test
> > PASS pcid (3 tests)
> > SKIP vmx (0 tests)
> > PASS debug (7 tests)
> > SKIP hyperv_synic (failed check)

Some nice improvements with this series. I'm not sure I like depending
on bash in standalone tests, but then that could just be cause I worked
pretty hard at avoiding the dependency, and thus I'll have to cry at the
loss of it...

Please review the series I'll send in about 2 minutes, so we can discuss
how to integrate them.

Thanks,
drew

>
>
> Radim Krčmář (12):
> run_tests: move run() to scripts/
> run_tests: prepare for changes in scripts/mkstandalone
> scripts/mkstandalone: use common run function
> scripts/mkstandalone: improve exit paths
> lib/report: allow test skipping
> x86/*: report skipped tests
> x86/pmu: expect failure with nmi_watchdog
> scripts/run: generalize check
> x86/hyperv_synic: check for support before testing
> run_tests: print summary
> wrappers: consolidate skip output
> run_tests: suppress stderr
>
> lib/libcflat.h | 1 +
> lib/report.c | 44 +++++++++++++++++++++++-----------
> run_tests.sh | 58 +++++---------------------------------------
> scripts/mkstandalone.sh | 64 +++++++++++++++++++++----------------------------
> scripts/run.bash | 62 +++++++++++++++++++++++++++++++++++++++++++++++
> x86/apic.c | 7 +++---
> x86/emulator.c | 2 +-
> x86/hyperv_synic.c | 2 +-
> x86/pku.c | 2 +-
> x86/pmu.c | 11 +++++++--
> x86/smap.c | 2 +-
> x86/svm.c | 2 +-
> x86/tsc.c | 2 +-
> x86/unittests.cfg | 4 ++--
> 14 files changed, 146 insertions(+), 117 deletions(-)
> create mode 100644 scripts/run.bash
>
> --
> 2.6.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html