RE: [kvm-unit-tests PATCHv2] unittests.cfg: Increase timeout for apic test

["Christopherson, Sean J" <sean.j.christopherson@intel.com>]

On Mon, Oct 19, 2020 at 01:32:00PM +0200, Vitaly Kuznetsov wrote:
> Sean Christopherson <sean.j.christopherson@intel.com> writes:
> >> > diff --git a/x86/unittests.cfg b/x86/unittests.cfg
> >> > index 872d679..c72a659 100644
> >> > --- a/x86/unittests.cfg
> >> > +++ b/x86/unittests.cfg
> >> > @@ -41,7 +41,7 @@ file = apic.flat
> >> >  smp = 2
> >> >  extra_params = -cpu qemu64,+x2apic,+tsc-deadline
> >> >  arch = x86_64
> >> > -timeout = 30
> >> > +timeout = 240
> >> >
> >> >  [ioapic]
> >> >  file = ioapic.flat
> >>
> >> AFAIR the default timeout for tests where timeout it not set explicitly
> >> is 90s so don't you need to also modify it for other tests like
> >> 'apic-split', 'ioapic', 'ioapic-split', ... ?
> >>
> >> I was thinking about introducing a 'timeout multiplier' or something to
> >> run_tests.sh for running in slow (read: nested) environments, doing that
> >> would allow us to keep reasonably small timeouts by default. This is
> >> somewhat important as tests tend to hang and waiting for 4 minutes every
> >> time is not great.
> >
> > I would much prefer to go in the other direction and make tests like APIC not
> > do so many loops (in a nested environment?). The port80 test in particular is
> > an absolute waste of time.
>
> I don't think these two suggestions are opposite. Yes, making tests run fast
> is good, however, some of the tests are doomed to be slow. E.g. running
> VMX testsuite while nested (leaving aside the question about who needs
> three level nesting) is always going to be much slower than on bare metal.

Ya, I was specifically referring to tests that arbitrarily choose a high loop
count, without any real/documented justification for running millions of loops.

> > E.g. does running 1M loops in test_multiple_nmi() really add value versus
> > say 10k or 100k loops?
>
> Oddly enough, I vaguely remember this particular test hanging
> *sometimes* after a few thousand loops but I don't remember any
> details.

Thousands still ain't millions :-D.

IMO, the unit tests should sit between a smoke test and a long running,
intensive stress test, i.e. the default config shouldn't be trying to find
literal one-in-a-million bugs on every run.