Re: [PATCH kvm-unit-tests v2 00/12] Improve the output of test runners

[Andrew Jones <drjones@redhat.com>]

On Thu, Dec 17, 2015 at 06:53:31PM +0100, Radim Krčmář wrote:
> v1: http://www.spinics.net/lists/kvm/msg125202.html
> 
> Drew brought up the existence of scripts/mkstandalone.sh, which
> significantly expanded v2 (and my set of curses) ...
> I didn't want to do the same twice, so first part of this series,
> [1-4/12], reuses run() from run_tests.sh and does some non-conservative
> changes to scripts/mkstandalone.sh.  scripts/mkstandalone.sh is lacking
> behind run_tests.sh, but should be good enough to fulfill its purpose.
> 
> The output of run_tests.sh has also changed a bit and now looks like
> this (you'll again need to imagine colours):
> 
> > PASS apic (14 tests)
> > PASS ioapic (19 tests)
> > PASS smptest (1 tests)
> > PASS smptest3 (1 tests)
> > PASS vmexit_cpuid
> > PASS vmexit_vmcall

Why do some tests, which have only 1 test, say (1 tests), but other
tests don't say anything?

> > PASS vmexit_mov_from_cr8
> > PASS vmexit_mov_to_cr8
> > PASS vmexit_inl_pmtimer
> > PASS vmexit_ipi
> > PASS vmexit_ipi_halt
> > PASS vmexit_ple_round_robin
> > PASS access
> > SKIP smap (0 tests)
> > SKIP pku (0 tests)
> > PASS emulator (132 tests, 1 skipped)
> > PASS eventinj (13 tests)
> > PASS hypercall (2 tests)
> > PASS idt_test (4 tests)
> > PASS msr (13 tests)
> > PASS pmu (67 tests, 1 expected failures)
> > PASS port80
> > PASS realmode
> > PASS s3
> > PASS sieve
> > PASS tsc (3 tests)
> > PASS tsc_adjust (5 tests)
> > PASS xsave (17 tests)
> > PASS rmap_chain
> > SKIP svm (0 tests)
> > SKIP svm-disabled (0 tests)
> > SKIP taskswitch (i386 only)
> > SKIP taskswitch2 (i386 only)
> > PASS kvmclock_test
> > PASS pcid (3 tests)
> > SKIP vmx (0 tests)
> > PASS debug (7 tests)
> > SKIP hyperv_synic (failed check)

Some nice improvements with this series. I'm not sure I like depending
on bash in standalone tests, but then that could just be cause I worked
pretty hard at avoiding the dependency, and thus I'll have to cry at the
loss of it...

Please review the series I'll send in about 2 minutes, so we can discuss
how to integrate them.

Thanks,
drew

> 
> 
> Radim Krčmář (12):
>   run_tests: move run() to scripts/
>   run_tests: prepare for changes in scripts/mkstandalone
>   scripts/mkstandalone: use common run function
>   scripts/mkstandalone: improve exit paths
>   lib/report: allow test skipping
>   x86/*: report skipped tests
>   x86/pmu: expect failure with nmi_watchdog
>   scripts/run: generalize check
>   x86/hyperv_synic: check for support before testing
>   run_tests: print summary
>   wrappers: consolidate skip output
>   run_tests: suppress stderr
> 
>  lib/libcflat.h          |  1 +
>  lib/report.c            | 44 +++++++++++++++++++++++-----------
>  run_tests.sh            | 58 +++++---------------------------------------
>  scripts/mkstandalone.sh | 64 +++++++++++++++++++++----------------------------
>  scripts/run.bash        | 62 +++++++++++++++++++++++++++++++++++++++++++++++
>  x86/apic.c              |  7 +++---
>  x86/emulator.c          |  2 +-
>  x86/hyperv_synic.c      |  2 +-
>  x86/pku.c               |  2 +-
>  x86/pmu.c               | 11 +++++++--
>  x86/smap.c              |  2 +-
>  x86/svm.c               |  2 +-
>  x86/tsc.c               |  2 +-
>  x86/unittests.cfg       |  4 ++--
>  14 files changed, 146 insertions(+), 117 deletions(-)
>  create mode 100644 scripts/run.bash
> 
> -- 
> 2.6.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html