Re: The whole round of i-g-t testing cost too long running time

[Daniel Vetter <daniel.vetter@intel.com>]

On 16/04/2014 17:42, Jesse Barnes wrote:
> On Tue, 15 Apr 2014 19:17:59 +0200
> Daniel Vetter <daniel.vetter@intel.com> wrote:
>> Ok there are a few cases where we can indeed make tests faster, but it
>> will be work for us. And that won't really speed up much since we're
>> adding piles more testcases at a pretty quick rate. And many of these
>> new testcases are CRC based, so inheritely take some time to run.
> But each test should run very quickly in general; I think we have too
> many tests that take much longer than they need to.  Adding some
> hooks to the driver via debugfs may let us trigger specific cases
> directly rather than trying to induce them through massive threading
> and memory pressure for example.
>
> And can you elaborate on the CRC tests?  It doesn't seem like those
> should take more than a few frames to verify we're getting what we
> expect...

Well they don't take more than a few frames, but we have a _lot_ of 
them, and there's a lot of cominations to test. It adds up quickly. Iirc 
we have over 150 kms_flip testcases alone ...

Like I've said I agree that we could speed tests up, but besides me 
doing the occasional tuning and improvement in that regard I have seen 0 
patches from developers in this area. Which lets me conclude that 
apparently it's not reallly that bad an isssue ;-) If people _really_ 
care about this I have a list of things to knock down. But first someone 
needs to find some time and resources for this.
>
>> So I think longer-term we simply need to throw more machines at the
>> problem and run testcases in parallel on identical machines.
>>
>> Wrt analyzing issues I think the right approach for moving forward is:
>> a) switch to piglit to run tests, not just enumerate them. This will
>> allow QA and developers to share testcase analysis.
>> b) add automated analysis for time-consuming and error prone cases like
>> dmesg warnings and backtraces. Thomas&I have just discussed a few ideas
>> in this are in our 1:1 today.
>>
>> Reducing the set of igt tests we run is imo pointless: The goal of igt
>> is to hit corner-cases, arbitrarily selecting which kinds of
>> corner-cases we test just means that we have a nice illusion about our
>> test coverage.
> My goal is still to get full test coverage before patches get
> committed, and that means having quick (<1hr) turnaround for testing
> from the automated patch test system.  It seems like we'll need to
> approach that from all angles: speeding up tests, parallelizing
> execution, adding hooks to the driver, etc.
>
>
Currently <1h and "full test coverage" are rather mutually exclusive 
unfortunately :( I agree that having it would be extremely useful for 
developers, but if this happens with any cost/service reduction for 
nightly testing on my branches I'm really opposed. Atm we already have a 
_really_ hard time keeping track of all the various regressions and bugs.
-Daniel
Intel Semiconductor AG
Registered No. 020.30.913.786-7
Registered Office: Badenerstrasse 549, 8048 Zurich, Switzerland